IMAGE ABSTRACTION PAINTING OF FLOW-LIKE STYLIZATION

Original scientific paper This paper presents a non-photorealistic rendering technique for producing flow-like abstraction stylization from a photograph. Based on anisotropic Kuwahara filtering in conjunction with line integral convolution, our method abstracts shapes and colors simultaneously while preserving features of images. In particular, we develop an edge detection and dilation method, to draw attention to salient features and image boundaries. This proposed algorithm is incremental and iterative, and therefore the degree of flow and abstraction can be controlled. Experimental results demonstrate that the effectiveness of our method in producing a coherent and flow-like abstraction stylization is satisfactory yet preserving features and directions from photographs.


Introduction
Automatic generation of different artistic illustrations from photographs is one of the most fundamental and important assignments of Non-photorealistic rendering (NPR).Photorealistic images often include more information than necessary to communicate intended information.Therefore, many researches remove some details and abstract the images to effective and stereoscopic visual effects.A simple method to automatically generate stylized abstractions is edgepreserving filter.The bilateral filter [1] and mean-shift [2] are very famous examples of edge-preserving filter to image abstraction.However, because these two methods use the thresholds to restrict local regions of low-contrast and high-contrast, they may fail either no abstraction has performed border on different quantization regions or too much edge detail is eliminated.The Kuwahara filter [3] can be used to overcome these limitations of edgepreserving filter.Though the Kuwahara filter quantizes regions of high-contrast, it also protects boundaries of low-contrast regions.Therefore, it provides an overall stereoscopic painting visual and maintains a uniform level of abstraction across the image.Unfortunately, the Kuwahara filter can result in clustering artifacts during filter process.In addition, the edge between different regions will be weakened because color is permeated and diffused from one region to another.Several extensions and improved methods have been proposed to modify the original Kuwahara filter.Papari introduced a new weighting windows and a new combination rule [4].Kyprianidis generated a flattening effect along the local feature directions while preserving shape boundaries [5].Even though these improve the output quality significantly, the smooth stylized surface cannot preserve both spatial and temporal coherence.
Flow-like structures and directional features are harmonic, interesting and pleasant to some people.Directional coherence and flow structure in the artistic work helps to salience region features and boundaries.It also helps to evoke viewer's mood.Such artists as van Gogh and Munch, they emphasized these features in their paintings.In this paper, we present a new image abstraction technique that places emphasis on enhancing the directional coherence and flow-like structure.The famous related work about this category is image abstraction and stylization based on partial differential equations (PDE), such as shape-simplifying method of Kang [6], coherence-enhancing shock filter of Gilboa [7].However, PDE methods may require a large number of iterations and running time [8].
We build upon the idea of combining diffusion with anisotropic Kuwahara filter and line integral convolution for image abstraction.Contrary to shape-simplifying abstraction [6], our technique outperforms in terms of salience outline, temporal coherence and flow-like effects.We aim to preserve the shape by Kuwahara filter to the local structure, and then directional edge and image features are better emphasized.This results in overall sharper edges and a smoother flow-like painterly artistic effect.Compared to the conventional abstraction methods [5,9,10], our technique creates stronger contrast and provides a flow-like effect, which also preserves and enhances directional feature better.

Related work
Segmentation is the common technique to abstract image.For example, DeCarlo and Santella [12] used eyetracking data to guide image abstraction based on the mean shift segmentation at different scales.Another popular method to abstract image is the use of edgepreserving filters.The filter technique often removes details in the low-contrast regions, neglecting the overall structure of the image unaffected.The bilateral filter [1] is a well-known method to abstract image.Winnemoller [12] improved bilateral filter technique and applied colour quantization to the color of input images and videos.Based on the local image structure, Kyprianidis [6] proposed separable implementations of the bilateral and difference of Gaussians filters.Kang et al. [13] presented the filter shapes based on the vector field which derived from the salient image features.Though bilateral filter method preserves edges of the high-contrast region, it fails for these regions where either too much detail is removed or no abstraction is performed.So, this technique displays an inconsistent and incoherent abstraction effect.
Another popular edge-preserving filter is Kuwahara filter [3].This technique is unstable if region has the same variance or large noise exists in local region.So, the filter results have clearly noticeable artifacts.Some researchers address the limitations of the Kuwahara filter.Papari et al [4] presented a generalized variant filter.They apply weighted sum of the means of the regions to define filter functions.The filter results are fewer artifacts and have smoother region.Kyprianidis et al [5] proposed the anisotropic Kuwahara filter to address the shape of weighting functions.The anisotropic Kuwahara filter is very stable against the small changes of variance and robust against large noise.So the directional features are better preserved and the artifacts are avoided.
As the same as the bilateral filter and the Kuwahara filter, Kang and Lee [6] used shock filter and mean curvature flow to generate edge-preserving abstraction effect.Though they smooth color variations and protect boundaries, the boundaries of the shape are not salient and simplified.Because directional image features cannot be protected based on the mean curvature flow, Kang and Lee proposed a technique of shape-simplifying constrained mean curvature flow.They applied a user interactive method to define a mask to protect important image features.However, if there are small color and curvature changes in the input image, this technique is not stable.
Other recently presented image abstraction approaches include image abstraction processing based on the gradient domain of Orzan [10], progressing image abstraction based on the weighted least squares framework of Farbmanz [14], a robust optimization framework of constrains for pixel values and pixel gradients of Bhat [15].These methods require some extensive processing to generate abstraction effect.Further, a smooth and flow-like effect is not revealed.

Method
Our processing method is performed iteratively, and the iterative numbers can be used to control the strength and degree of the abstraction.The overview of the algorithm is shown in Fig. 1.We abstract the input image beginning with the anisotropic Kuwahara filtering, and the structure tensor can be calculated with the Gaussian filter.Based on the eigenvalues and eigenvectors of the structure tensor, local orientation and anisotropy feature is acquired to guide filter operation.Then the abstraction effect is performed using a nonlinear filter based on the local orientation and anisotropy.Secondly, coherent shape boundaries of the input photographs can be captured based on DoG filter along edge tangent flow, and we preserve the salient of the image features.In addition, we apply dilation method to expand width of these edges to map to the anisotropic Kuwahara filter image.Finally, we generate the coherent and flow-like abstraction effects with the technique of improved line integral convolution.

Local orientation calculation
Based on the eigenvalues and eigenvectors of the structure tensor, we use Brox's method to acquire the local orientation and anisotropy [16].The structure tensor can be calculated from the input photograph [6].Let f be the input image with standard deviation σ.G σ,x and G σ,y denote the spatial derivatives in x-direction and ydirection, then: The gradient vector of one pixel is: G = (G x , G y ) T .Let * denote convolution operation, the partial derivatives of f can be calculated as: . , , , The structure tensor of input image f can be defined as: .
The feature values of T x,y have minimum and maximum change rate to the square filter window, and the structure tensor field is decided by the brightness of input image.Let d 1 and d 2 be the minimum and maximum eigenvalue of T, which represent gray scale variation degree of the vector v 1 and v 2 : where the parameter σ denotes Gauss kernel function variance, r denotes filter radius and calculates using d 1 / d 2 .If d 1 and d 2 is similar, the filter function tends to the circular field structure.The parameter θ represents edge direction of pixel (x, y), which reflects curve direction of eigenvalue.Then anisotropic structure tensor can be expressed as T': (5) Generally, the vector field is confirmed by the eigenvectors that correspond to the minimum rate of change.Because this field is discontinuous, we also utilize Gaussian filter technique to smooth this vector field, and the eigenvalues of the structure tensor are given as: Then, the minimum rate change of the eigenvector in direction can be given: .12 11 1 The local orientation can be defined arg t ϕ = , and anisotropy can be calculated using Yang's method [17]: .
The parameter A ranges between 0 to 1, which corresponds to isotropic and anisotropic regions respectively.

Anisotropic filtering calculation
When we present our filter calculation, each pixel (i,j) is determined by the spatial distance from the center pixel (x,y), as well as its relative difference of intensity.The filtering output Q at a pixel (i,j) is expressed as: .) , ( The parameter Ω denotes filter spatial space.G i,j denotes the local intensity or color similarity with weighted w i,j , and a i,j denotes the squared standard deviations.m and a can be defined: ( , ) ( , ) ( , ) ( , ) , , , ( , ) Let ϕ be the local orientation, A be the anisotropy, and Rϕ be the matrix defining a rotation, the filter spatial space Ω can be set: The parameter h = 2σ r .The parameter S is used to adjust the value of anisotropy A. So, SR ϕ − is a linear coordinate transform that maps Ω to a disc of radius h.This anisotropic filter ensures that more weight is given to local regions with low standard deviation.
During the filter process, we can apply parameter t to control iterative filter times.The prior filtering output Q can be regarded as f to precede another anisotropic filter next time.Fig. 2 shows our filter results with different iterative times (t = 3 ÷ 8).The smaller iterative times, the clearer the filter result.Compared to other filter methods, the anisotropic Kuwahara filter results remove artifacts by adapting direction shape, orientation to the local structure of the input images.Then directional image's feature can be better preserved and emphasized.

Edge detection and dilation
Conventional edge or contour detectors from image are often employed and extracted, such as Canny's [10,11], mean-shift segmentation [18,19], DoG filtering [12] and so on.As shown by Kang and Lee [13], a set of coherent, smooth, and stylistic lines can be captured and convey important shapes in the image.We improve the DoG filter along the image's edge tangent flow (ETF) to enhance the quality of edges.
Let c x (s) be the flow curve at pixel x, s is length parameter, l x,s denotes a line segment that intersects c x (s) and is perpendicular to t(c x (s)).l x,s (t) is parallel to the gradient vector g(c x (s)).Then the DoG filtering can be formulated [13]: where I(l x,s (t)) denotes the value of the input image I at l x,s (t).Parameter m is applied to determine the size of S. G σ is a Gaussian function with variance σ 2 : .e 2π 1 ) ( Based on the DoG filter model [12], the filter parameter H can be calculated as: The parameters c σ and s σ are used to control the size of the center interval and the surrounding interval, and the parameter ρ decides the level of noise detected.Because of the line image always in black and white illustration, the filter results H should be transfered using the threshold: ( ) where the parameter t is a threshold and its value is between 0 and 1.
Because the filter result image H is not discontinuous, to smooth and enhance image's contour, we apply a guided filter to achieve these purposes.General to this filter, the key edge smooth and enhancement technique is a local linear model between the image H and the filter output F. Let r be the square filter radius, the output F is a linear transform of H in a square window ω c centered at pixel c: ( , ) ( , ) where a c and b c are linear coefficients assumed to be constant in ω c .This linear model ensures that F has an edge only if H has an edge.We also can employ the average value after computing a c and b c for all patches ω c in the H to smooth couturiers.Then the edge enhancement results can be obtained through difference between the H and F: where the parameter m is used to control the degree of edge enhancement.If m is larger, the detail is more prominent and exquisite.This process may let the edge image reach a satisfactory level of line connectivity and illustration quality.
After we obtain successful and coherent edge, if the saliency values of image's edge should be further emphasized, we attempt to clarify and enhance this contour information by mathematical morphology operation of image dilation.
where B is a structuring element.We also use parameter g to denote the center size of B. a and b are local elements of F' and B respectively.When the anisotropic filtering and contours results are generated, we use Liu's method to merge them [20], and the anisotropic filtering image can be mapped to the edge image based on the illumination model of deviation mapping.Fig. 4 shows our mapping results with illumination model.From the mapping effects, we can find the salient outline and flow-like effects are not reflected.

Line Integral Convolution optimization
We benefit from our efficient stylization painting not only in anisotropic filtering, but also in flow-like effect.The local fluid abstract effect can be approximated based on line integral convolution method.This technique calculates a local stream line that moves out from the positive and the negative directions.Let I out present the fluid output image, σ(s) is one stream line with length of num, p 0 denotes each pixel in this line, we can obtain p 0 = σ(s 0 ).k(s) denotes convolution kernel, and we utilize Hanning window function in this paper.I(σ(s)) denotes all of pixels in this line, the convolution implement is defined as [21]: .
The parameter L denotes convolution length.If L is too large, the output will return values very close and the result will show regular.However, if L is too small, then an insufficient amount of filtering occurs.So the convolution length L should be adjusted self-adaptively.Because variance can reflect local statistic information very well, this feature just exactly reflects the change of local convolution length: where L max and L min are the largest and smallest convolution lengths which can be assigned by the user.σ x,y denotes the value of local area's variance.shows two near stream lines x 1 and x 2 .During the convolution process, there is some redundant calculating between x 1 and x 2 .Let x m denote original position of one stream line (red dot), I(Δ 1 ) be convolution results of area Δ 1 , and I(Δ 2 ) denotes results of area Δ 2 .The convolution of positive I(x m+1 ) and negative I(x m−1 ) can be obtained through the difference between x 1 and x 2 : Let k be convolution kernel, after obtaining the convolution value of I(x m ), the positive I(x m+1 ) and negative I(x m−1 ) can be simplified as: where we set k=1/(1+2L).This improvement is particularly useful for accelerating convolution operation.It may also be used to advance abstract quality.

Experiment results
We use our proposed method for producing abstraction results.User can control the iteration times of convolution to generate different effects.Fig. 6 shows different abstraction results obtained by adjusting parameters, such as L max and L min (length of convolution), K (size of convolution kernel), c (iteration times).Note that small L max conveys fine details, large K expands the influence regions, and big iteration times c express more fluid effects.Fig. 7 shows some additional abstraction results with our algorithm.The parameters are used: L max = 8, and 4 ÷ 10 iterations of line integral convolution calculation.Note our method simplifies the local shapes, yet successfully conveys visual information in a fluid abstraction fashion.shows different abstract effects by the vulnerability of the bilateral filter, Kuwahara filter and Gaussian filter.The bilateral filter removes away most of the interesting information.Kuwahara filter is unstable in the presence of noise and suffers from artifacts, and Papari et al. [4] improves the output quality, artifacts are still noticeable.
We have implemented the proposed algorithm using C++, and we tested our algorithm on an Inter 2.8GHz PC with 2GB memory.For a 400×400 color image, our method took about 50 seconds to complete 10 iterations.Our proposed algorithm is based on local operations, so the processing time depends mainly on the iterations number and the image size.If GPU is applied to implement our ideal, it would bring about huge acceleration.

Conclusions
In this paper, we have presented an automatic approach for abstraction and flow-like stylization paint rendering.Based on anisotropic filtering and iterative constrained line integral convolution, our algorithm provides merits such as controlling of the degree of abstraction, simplification of colors and shapes, flow-like stylization effects and the simplicity of implementation.
An obvious limitation of our method is that the local details cannot be protected very well.In addition, some strategies should be used to protect highly important area.We may further extend the GPU implementation of our algorithm to process video in real-time.

Figure 1
Figure 1 Schematic overview of our proposed method

Figure 2
Figure 2 Comparison of different filters

Fig. 3
shows our edge detection results.Fig. 3(b) and Fig. 3(f) are DoG filter results, and the smooth, continuous edge is generated.Further, after enhancement of contour information, Fig. 3(c), Fig. 3(g) show contour dilation results with g = 2, Fig. 3(d) and Fig. 3(h) show contour dilation results with g = 4.

Figure 3 Figure 4
Figure 3 Our edge detection results

Fig 5 .
Fig 5.  shows two near stream lines x 1 and x 2 .During the convolution process, there is some redundant calculating between x 1 and x 2 .Let x m denote original position of one stream line (red dot), I(Δ 1 ) be convolution results of area Δ 1 , and I(Δ 2 ) denotes results of area Δ 2 .The convolution of positive I(x m+1 ) and negative I(x m−1 ) can be obtained through the difference between x 1 and x 2 :