Multidimensional and Color imaging Projections.
Colorization by Multidimensional Projection
I. INTRODUCTION
Colorization is a computer-assisted process by which color is imparted to black-and-white films or to gray scale images. It has been widely used in photo processing and scientific illustration, to modernize black-and-white films and to restore damaged color films. Traditionally, colorization is tedious, time consuming and requires artistic skills to precisely add appropriate colors to a gray scale image.
Aiming at making the colorization process simpler and less laborious, several computational systems and tools have been proposed in the last decade, which can be roughly divided into two classes: example-based and scribble-based. Example-based methods accomplish the colorization process by matching the luminance of the gray scale image with the luminance of a source color image used to drive the colorization. In practice, user picks out regions (swatches) on the target gray scale image and matches those regions with color patterns in the source color image. A typical example based colorization scheme was proposed by Welsh etal. which combines image analogies and color transfer to colorize a gray scale image. Irony etal. improve Welsh’s method by using a segmented reference image and a texture matching scheme to better colorize regions with similar textures. In a more recent work, Liuetal. make use of multiple reference images retrieved from the Internet to guide the colorization process and mitigate problems related to environment illumination. Example-based methods end up being very sensitive to the source color image employed to drive the colorization process while still performing poorly in images with complex textured patterns.
In scribble-based methods the user drives the colorization by defining colored strokes onto the gray scale image. The classical work by Levin et al. is a good representative of scribble-based approach. Levin’s method aims at optimizing the color of all image pixels using the scribbles as constraints. Although it shows good results for various types of images, Levin’s method tends to propagate colors beyond the texture boundaries, thus resulting in unpleasant colorizations. The technique proposed by Huang et al. employs adaptive edge detection so as to prevent colors from going beyond region boundaries. Further improvements have been proposed by Yatziv and Sapiro who present a faster scribble-based color optimization technique that relies on chrominance blending to perform the colorization, and by the authors of which employ texture continuity to colorize manga-cartoons and natural images, respectively. Despite the good results, all existing scribble-based approach require intensive user intervention, especially when the image contains complex structures or has different kinds of textures, which can demand lots of scribbles until acceptable outcomes are reached.
In this work we propose a new approach for colorizing gray scale images that relies on interactive multidimensional projection to replace the intensive user-driven scribble mechanism by a simple drag-and-drop manipulation of the badly colorized pixels. This novel interactive scheme, called Projcolor, operates from example-based as well as color scribbles, demanding just a few user interventions to correct imperfections in regions poorly colorized. Moreover, as the multidimensional projection allows for visualizing the neighborhood structure of the pixels under colorization, the user can control color propagation in an accurate and intuitive manner, a trait not present in other methods proposed in the literature. As we shall show, by interacting with the multidimensional projection the user can colorize complex textured regions quite easily, respecting region edges and avoiding introducing new scribbles. Since the user can accurately control the colorization process, our approach is less sensitive to the source color image.
II. MULTIDIMENSIONAL PROJECTION-BASED COLORIZATION
The proposed colorization pipeline comprises four main steps, namely, input color data, feature extraction/selection, multidimensional projection, and image colorization. In our framework, the color data can be provided in two distinct ways: color swatches picked out from a RGB image, or color scribbles brushed by the user. Similar to Welsh et al. the color data and the gray scale image are converted from RBG color and Gray scale space to the independent lαβ space, where l is the luminance channel and α, β are two chromatic channels. During the feature extraction stage, each pixel of the gray scale image is mapped to a point in a high dimensional space whose coordinates correspond to statistics measures and texture attributes computed from luminance and gray values in the neighborhood of the pixel. The high dimensional data is project to a two-dimensional visual space so as to preserve neighborhood structures as much as possible. Colorization is then performed based on information computed from the projected data. Badly colored regions can be updated by interacting with the projected data in the visual space. In other words, neighborhood structures in the visual space can be modified by the user so as to generate better colorization. Details of each stage of the pipeline are presented in the following subsections.
A. Feature Extraction/Selection Given a gray scale image H, the feature extraction step aims at mapping each pixel from H to a point in a high dimensional feature space X ⊂ R N . In more mathematical terms, we build a mapping f : H → X that assigns to each pixel p ∈ H a point f(p) ∈ R N called feature vector. The coordinates of f(p) are given by local feature descriptors computed in a neighborhood of p. In our implementation we are using Gabor first order moment, local binary patterns and their variants, scale invariant feature transforms , and ordinary statistical/morphological measures.
In practice, features resulting from distinct extractors can be highly correlated and it is not necessary to keep all of them in the feature vector. Removing unnecessary data reduces the dimensionality of the feature space while still untangling clusters, thus impacting positively in the result of the colorization process. We handle unnecessary features by applying the well known learning-based feature selection tool called WEKA . WEKA computes the correlation among features and ranks the more relevant ones. In our tests, about 20% of features are kept after filtering them out using WEKA.
B. Multidimensional Projection
The multidimensional projection step is responsible to map high-dimensional data generated during the feature extraction/classification step onto a two-dimensional visual space. In our implementation we are employing the multidimensional projection technique called LAMP (Local Affine Multidimensional Projection) . Besides preserving neighborhood structures quite well, the LAMP technique enables a very flexible mechanism to interactively modify the projection according to user interventions, which is an important characteristic in the context of this work.
LAMP uses a set of control points to perform the mapping of a set of high-dimensional data X to the visual two dimensional space. The set of control points is typically a small subset XS ⊂ X whose basic information in the visual space is known a priori. The mapping of each instance x in X to visual space is carried out based on the linear mapping Tx that minimizes :
where ||·||F is the Frobenious norm, T ⊥ is the transpose of T, I is the identity matrix, and A and B are matrices whose rows correspond to the coordinates of each control point in the high-dimensional and visual space, respectively. The constraint T T ⊥ = I is imposed to ensure that distances are preserved as much as possible during projection. When a user manipulates control points in the visual space s/he also changes entries in the rows of matrix B, thus tuning the mapping Tx to cope with the user intervention. More precisely, when a control point is moved in the visual space the data in its neighborhood is moved together, what allows for controlling neighborhood structures in the visual space.
In our colorization framework, the input control points are assigned through the pixels selected by the user-defined strokes or swatches. User can then interactively manipulate control points in the visual space based on the resulting colorization. The flexibility to change projection according to user manipulation of control points is exploited by the proposed colorization application as described below.
C. Image Colorization
The colorization of a gray scale image relies on pixel classification carried out in the visual space, that is, after mapping the high-dimensional data to the visual space using LAMP, a clustering mechanism is employed to group pixels with similar features (we apply the KNN classifier on the projected data). Once “similar pixels” have been clustered, colors are associated to each cluster from the information initially provided by the user-defined strokes or swatches. More precisely, the user associates a color pattern to a subset of pixels and that color pattern is then propagated to all pixels in the cluster. The propagation mechanism is accomplished as follows: given a gray scale pixel pg and a user provided set of colored pixels Pc, the lαβ color of pg is given by taking the α and β components from pixel pc ∈ Pc that minimizes the distance function dl defined by:
where lc and lg are the luminance values of pc and pg, respectively. The values stdc and stdg are the standard deviations of lc and lg, respectively. After processing the pixels in lαβ color system, pixel conversion to RGB system is then performed in order to produce the final color result. In our experiments we use a pixel neighborhood of size 5 × 5 to compute standard deviations of the pixel luminance.
D. User Manipulation of Clusters
One of the main contributions of Projcolor is to exploit the flexibility provided by the multidimensional projection scheme towards interactively manipulating projected data so as to modifying clusters. When features extracted from distinct regions of an image are similar, pixels belonging to those regions cannot be clearly separated either in the high dimensional space or in the projected space, since the multidimensional projection preserves neighborhood structures as much as possible. LAMP, however, enables to user with an interactive tool that allows for untangling data by manipulating control points. In fact, if the coloring result is not satisfactory, the user can select badly colored pixels, turning them into new control points that can be interactively moved in the visual space to better define clusters and thus to improve the result of the colorization. Fig. 3 illustrates the user intervention process where control points are defined by the user and then dragged to different clusters within visual space. Since LAMP also drags the neighborhood of the control points after user interaction it is necessary to define just a few control points to achieve satisfactory colorizations.
III. RESULTS AND COMPARISONS
the result of colorizing the picture in taking as input the set of color pixels within the two boxes (drawn by the user) in . The result of projecting the high-dimensional data onto the visual space is. Notice that two clusters corresponding to the sky and the trees are formed in the visual space. The clear separation of clusters enables an accurate colorization without user intervention as presented. The colorization example presented illustrates the need of user intervention. The source color image (and the color examples took from it) and the gray scale image to be colored. The result of the multidimensional projection from which the color image in is generated. Notice that pixel close the edge between the panther and the background grass are not colored properly. User can then select some of those badly colorized pixels, which are made control points for the multidimensional projection. Dragging the new control points to the correct cluster changes the projection layout and improves the resulting colorization.
the capability of Projcolor to perform colorization based only on colored strokes, that is, the input is color scribbles drawn by the user on each region s/he wants to define a color. Color scribbles are simpler than example-based colorization, as the user does not need to provide a colored source image not a correspondence between them. Despite its simplicity and ease of use, the stroke-based scheme results in quite satisfactory colorizations.
Projcolor with both techniques mentioned above. Colorization method proposed by Welsh et al. works properly in images that do not contain complex texture patterns and fails drastically when dealing with textures. Otherwise, Projcolor generated a clear and pleasant result gray scale image and two user defined scribbles on it.Are the colorization produced by Levin’s method and Projcolor respectively. Notice that our approach is much more robust when dealing with textures. It is important to say that the results presented were obtained using the MATLAB code available on authors website (http://www. cs.huji.ac.il/˜yweiss/Colorization/).
Another comparison between Projcolor and Levin’s method.The resulting colorization obtained from Levin’s method and Projcolor using only the two color strokes depicted in . Notice that Levin’s method does not generate a good colorization due to the complexity of textured regions while Projcolor results in a pleasant colorization. Levin’s approach produces better results when several strokes are provided by the user. Projcolor, however, demanded just a couple of user intervention to update clusters and reach the nice result , proving that Projcolor is more robust when dealing with complex textured images. Regarding computational times, Projcolor is quite similar to the Welsh’s and Levin’s methods. The computational cost increases proportionally to the number of swatches or scribbles employed to color images, neighborhood sizes in feature extraction step and the image size. In our experiments almost all images were colorized in a few tens of seconds (our code was implemented in MATLAB with support to Java routines).
IV. DISCUSSION AND LIMITATIONS
The comparisons presented in Section III clearly show the effectiveness of the proposed coloring method, surpassing, in requisites such as accuracy and flexibility, existing methods. The local control of neighborhood structures enables the user with a sharp control of color propagation within pixel clusters, a characteristic not present in any other colorization method.
The good results when dealing with highly textured images such as the robustness of the proposed approach. In fact, just a few interventions were needed to reach a pleasant colorization result. As mentioned above, regarding computational times, our method is comparable against two other approaches. In fact, our approach is a bit faster than the compared methods. Despite the good properties and results, Projcolor also has limitations. Although LAMP allows for controlling the extent neighborhoods are affected when manipulating control points, properly setting the number of control points that affects the projection is not a straightforward task, being this an issue to be further investigated.
I. INTRODUCTION
Colorization is a computer-assisted process by which color is imparted to black-and-white films or to gray scale images. It has been widely used in photo processing and scientific illustration, to modernize black-and-white films and to restore damaged color films. Traditionally, colorization is tedious, time consuming and requires artistic skills to precisely add appropriate colors to a gray scale image.
Aiming at making the colorization process simpler and less laborious, several computational systems and tools have been proposed in the last decade, which can be roughly divided into two classes: example-based and scribble-based. Example-based methods accomplish the colorization process by matching the luminance of the gray scale image with the luminance of a source color image used to drive the colorization. In practice, user picks out regions (swatches) on the target gray scale image and matches those regions with color patterns in the source color image. A typical example based colorization scheme was proposed by Welsh etal. which combines image analogies and color transfer to colorize a gray scale image. Irony etal. improve Welsh’s method by using a segmented reference image and a texture matching scheme to better colorize regions with similar textures. In a more recent work, Liuetal. make use of multiple reference images retrieved from the Internet to guide the colorization process and mitigate problems related to environment illumination. Example-based methods end up being very sensitive to the source color image employed to drive the colorization process while still performing poorly in images with complex textured patterns.
In scribble-based methods the user drives the colorization by defining colored strokes onto the gray scale image. The classical work by Levin et al. is a good representative of scribble-based approach. Levin’s method aims at optimizing the color of all image pixels using the scribbles as constraints. Although it shows good results for various types of images, Levin’s method tends to propagate colors beyond the texture boundaries, thus resulting in unpleasant colorizations. The technique proposed by Huang et al. employs adaptive edge detection so as to prevent colors from going beyond region boundaries. Further improvements have been proposed by Yatziv and Sapiro who present a faster scribble-based color optimization technique that relies on chrominance blending to perform the colorization, and by the authors of which employ texture continuity to colorize manga-cartoons and natural images, respectively. Despite the good results, all existing scribble-based approach require intensive user intervention, especially when the image contains complex structures or has different kinds of textures, which can demand lots of scribbles until acceptable outcomes are reached.
In this work we propose a new approach for colorizing gray scale images that relies on interactive multidimensional projection to replace the intensive user-driven scribble mechanism by a simple drag-and-drop manipulation of the badly colorized pixels. This novel interactive scheme, called Projcolor, operates from example-based as well as color scribbles, demanding just a few user interventions to correct imperfections in regions poorly colorized. Moreover, as the multidimensional projection allows for visualizing the neighborhood structure of the pixels under colorization, the user can control color propagation in an accurate and intuitive manner, a trait not present in other methods proposed in the literature. As we shall show, by interacting with the multidimensional projection the user can colorize complex textured regions quite easily, respecting region edges and avoiding introducing new scribbles. Since the user can accurately control the colorization process, our approach is less sensitive to the source color image.
II. MULTIDIMENSIONAL PROJECTION-BASED COLORIZATION
The proposed colorization pipeline comprises four main steps, namely, input color data, feature extraction/selection, multidimensional projection, and image colorization. In our framework, the color data can be provided in two distinct ways: color swatches picked out from a RGB image, or color scribbles brushed by the user. Similar to Welsh et al. the color data and the gray scale image are converted from RBG color and Gray scale space to the independent lαβ space, where l is the luminance channel and α, β are two chromatic channels. During the feature extraction stage, each pixel of the gray scale image is mapped to a point in a high dimensional space whose coordinates correspond to statistics measures and texture attributes computed from luminance and gray values in the neighborhood of the pixel. The high dimensional data is project to a two-dimensional visual space so as to preserve neighborhood structures as much as possible. Colorization is then performed based on information computed from the projected data. Badly colored regions can be updated by interacting with the projected data in the visual space. In other words, neighborhood structures in the visual space can be modified by the user so as to generate better colorization. Details of each stage of the pipeline are presented in the following subsections.
A. Feature Extraction/Selection Given a gray scale image H, the feature extraction step aims at mapping each pixel from H to a point in a high dimensional feature space X ⊂ R N . In more mathematical terms, we build a mapping f : H → X that assigns to each pixel p ∈ H a point f(p) ∈ R N called feature vector. The coordinates of f(p) are given by local feature descriptors computed in a neighborhood of p. In our implementation we are using Gabor first order moment, local binary patterns and their variants, scale invariant feature transforms , and ordinary statistical/morphological measures.
In practice, features resulting from distinct extractors can be highly correlated and it is not necessary to keep all of them in the feature vector. Removing unnecessary data reduces the dimensionality of the feature space while still untangling clusters, thus impacting positively in the result of the colorization process. We handle unnecessary features by applying the well known learning-based feature selection tool called WEKA . WEKA computes the correlation among features and ranks the more relevant ones. In our tests, about 20% of features are kept after filtering them out using WEKA.
B. Multidimensional Projection
The multidimensional projection step is responsible to map high-dimensional data generated during the feature extraction/classification step onto a two-dimensional visual space. In our implementation we are employing the multidimensional projection technique called LAMP (Local Affine Multidimensional Projection) . Besides preserving neighborhood structures quite well, the LAMP technique enables a very flexible mechanism to interactively modify the projection according to user interventions, which is an important characteristic in the context of this work.
LAMP uses a set of control points to perform the mapping of a set of high-dimensional data X to the visual two dimensional space. The set of control points is typically a small subset XS ⊂ X whose basic information in the visual space is known a priori. The mapping of each instance x in X to visual space is carried out based on the linear mapping Tx that minimizes :
where ||·||F is the Frobenious norm, T ⊥ is the transpose of T, I is the identity matrix, and A and B are matrices whose rows correspond to the coordinates of each control point in the high-dimensional and visual space, respectively. The constraint T T ⊥ = I is imposed to ensure that distances are preserved as much as possible during projection. When a user manipulates control points in the visual space s/he also changes entries in the rows of matrix B, thus tuning the mapping Tx to cope with the user intervention. More precisely, when a control point is moved in the visual space the data in its neighborhood is moved together, what allows for controlling neighborhood structures in the visual space.
In our colorization framework, the input control points are assigned through the pixels selected by the user-defined strokes or swatches. User can then interactively manipulate control points in the visual space based on the resulting colorization. The flexibility to change projection according to user manipulation of control points is exploited by the proposed colorization application as described below.
C. Image Colorization
The colorization of a gray scale image relies on pixel classification carried out in the visual space, that is, after mapping the high-dimensional data to the visual space using LAMP, a clustering mechanism is employed to group pixels with similar features (we apply the KNN classifier on the projected data). Once “similar pixels” have been clustered, colors are associated to each cluster from the information initially provided by the user-defined strokes or swatches. More precisely, the user associates a color pattern to a subset of pixels and that color pattern is then propagated to all pixels in the cluster. The propagation mechanism is accomplished as follows: given a gray scale pixel pg and a user provided set of colored pixels Pc, the lαβ color of pg is given by taking the α and β components from pixel pc ∈ Pc that minimizes the distance function dl defined by:
where lc and lg are the luminance values of pc and pg, respectively. The values stdc and stdg are the standard deviations of lc and lg, respectively. After processing the pixels in lαβ color system, pixel conversion to RGB system is then performed in order to produce the final color result. In our experiments we use a pixel neighborhood of size 5 × 5 to compute standard deviations of the pixel luminance.
D. User Manipulation of Clusters
One of the main contributions of Projcolor is to exploit the flexibility provided by the multidimensional projection scheme towards interactively manipulating projected data so as to modifying clusters. When features extracted from distinct regions of an image are similar, pixels belonging to those regions cannot be clearly separated either in the high dimensional space or in the projected space, since the multidimensional projection preserves neighborhood structures as much as possible. LAMP, however, enables to user with an interactive tool that allows for untangling data by manipulating control points. In fact, if the coloring result is not satisfactory, the user can select badly colored pixels, turning them into new control points that can be interactively moved in the visual space to better define clusters and thus to improve the result of the colorization. Fig. 3 illustrates the user intervention process where control points are defined by the user and then dragged to different clusters within visual space. Since LAMP also drags the neighborhood of the control points after user interaction it is necessary to define just a few control points to achieve satisfactory colorizations.
III. RESULTS AND COMPARISONS
the result of colorizing the picture in taking as input the set of color pixels within the two boxes (drawn by the user) in . The result of projecting the high-dimensional data onto the visual space is. Notice that two clusters corresponding to the sky and the trees are formed in the visual space. The clear separation of clusters enables an accurate colorization without user intervention as presented. The colorization example presented illustrates the need of user intervention. The source color image (and the color examples took from it) and the gray scale image to be colored. The result of the multidimensional projection from which the color image in is generated. Notice that pixel close the edge between the panther and the background grass are not colored properly. User can then select some of those badly colorized pixels, which are made control points for the multidimensional projection. Dragging the new control points to the correct cluster changes the projection layout and improves the resulting colorization.
the capability of Projcolor to perform colorization based only on colored strokes, that is, the input is color scribbles drawn by the user on each region s/he wants to define a color. Color scribbles are simpler than example-based colorization, as the user does not need to provide a colored source image not a correspondence between them. Despite its simplicity and ease of use, the stroke-based scheme results in quite satisfactory colorizations.
Projcolor with both techniques mentioned above. Colorization method proposed by Welsh et al. works properly in images that do not contain complex texture patterns and fails drastically when dealing with textures. Otherwise, Projcolor generated a clear and pleasant result gray scale image and two user defined scribbles on it.Are the colorization produced by Levin’s method and Projcolor respectively. Notice that our approach is much more robust when dealing with textures. It is important to say that the results presented were obtained using the MATLAB code available on authors website (http://www. cs.huji.ac.il/˜yweiss/Colorization/).
Another comparison between Projcolor and Levin’s method.The resulting colorization obtained from Levin’s method and Projcolor using only the two color strokes depicted in . Notice that Levin’s method does not generate a good colorization due to the complexity of textured regions while Projcolor results in a pleasant colorization. Levin’s approach produces better results when several strokes are provided by the user. Projcolor, however, demanded just a couple of user intervention to update clusters and reach the nice result , proving that Projcolor is more robust when dealing with complex textured images. Regarding computational times, Projcolor is quite similar to the Welsh’s and Levin’s methods. The computational cost increases proportionally to the number of swatches or scribbles employed to color images, neighborhood sizes in feature extraction step and the image size. In our experiments almost all images were colorized in a few tens of seconds (our code was implemented in MATLAB with support to Java routines).
IV. DISCUSSION AND LIMITATIONS
The comparisons presented in Section III clearly show the effectiveness of the proposed coloring method, surpassing, in requisites such as accuracy and flexibility, existing methods. The local control of neighborhood structures enables the user with a sharp control of color propagation within pixel clusters, a characteristic not present in any other colorization method.
The good results when dealing with highly textured images such as the robustness of the proposed approach. In fact, just a few interventions were needed to reach a pleasant colorization result. As mentioned above, regarding computational times, our method is comparable against two other approaches. In fact, our approach is a bit faster than the compared methods. Despite the good properties and results, Projcolor also has limitations. Although LAMP allows for controlling the extent neighborhoods are affected when manipulating control points, properly setting the number of control points that affects the projection is not a straightforward task, being this an issue to be further investigated.
Comments
Post a Comment