KR101660447B1 - Multi directional weighted interpolation method for Bayer pattern CFA demosaicking - Google Patents

Abstract

The present invention relates to a multidirectional interpolation method for demosaicing a Bayer pattern color filter array (CFA-Color Filter Array), and a multi-directional weighted interpolation method for Bayer pattern color filter array demosaicing according to the present invention a) interpolating missing green elements in eight directions including vertical, horizontal, and diagonal directions using a green interpolation part, and (b) using a red-blue interpolation part, And (c) a step of improving the green surface, the red surface, and the blue surface by using the color value difference using the post-processing unit.

Description

[0001] The present invention relates to a multi-directional weighted interpolation method for Bayer pattern color filter array demosaicing,

The present invention relates to a multi-directional weighted interpolation algorithm for color filter array interpolation, and more particularly to a multi-directional interpolation method for demosaicing a Bayer pattern color filter array (CFA).

For reasons of cost, when currently available digital still color cameras capture a color image, for each pixel, only one of the color information of the three color channels is sent to one CCD (charge-coupled device) sensor ≪ / RTI > In general, only one 8-bit number that measures the amount of RGB is captured at the pixel. This sensor is included in a color filter array (CFA), where each pixel in the camera is captured in only one color channel. This is because only one color element is available for each pixel, and two missing colors must be estimated from adjacent pixels. This process is called the CFC interpolation method or demosaiking. [1], [2] Figure 1 shows the Bayer pattern, the most common CFA scheme.

The color reconstruction quality depends on the CFA sample and the selected demosaicing algorithm. Based on the Bayer pattern, various demosaicing algorithms have been proposed. The prior art has proposed a method of obtaining a full color image using the color difference between the green and red / blue planes in the spatial range or the frequency range. The advantage of these schemes is that they are efficient in the smooth domain. However, these schemes are aggravated by the reproduced, uneven edges and the much less noise in the texture details, if the interpolation direction is not accurately estimated.

[1] B. E. Bayer, "Color imaging array," Patent 3 971 065, July 1976. [2] H. J. Trussell and R. E. Hartwig, "Mathematics for demosaicking," IEEE Trans. Image Processing, vol. 11, no. 4, pp. 485-492, Apr. 2002. [3] L. Zhang and X. Wu, "Color demosaicking via directional linear minimum mean square-error estimation," IEEE Trans. Image Processing, vol. 14, no. 12, pp. 2167-2178, Dec. 2005. [4] I. Pekkucuksen and Y. Altunbasak, "Edge strength filter based color filter array interpolation," IEEE Trans. Image Processing, vol. 21, no. 1, pp. 393-397, Jan. 2012. [5] D. Paliy, V. Katkovnik, R. Bilcu, S. Alenius, and K. Egiazarian, "Spatially adaptive color filter array interpolation for noiseless and noisy data," Int. J. Imaging Sys. Tech., Sp. Iss. Appl. Color Image Process., Vol. 17, no. 3, pp. 105-122, Oct 2007. [6] L. Zhang, A. Wu, A. Buades, and X. Li, "Color demosaicking by local directional interpolation and non-local adaptive thresholding," Journal of Electronic Imaging, vol. 20, no. 2, pp. 023016, 2011. [7] W. Lu and Y. Tan, "Color filter array demosaicking: new method and performance measures," IEEE Trans. Image Processing, vol. 12, no. 10, pp. 1194-1210, 2003. [8] A. Buades, B. Coll, J.-M. Morel, and C. Sbert, "Self-similarity driven color demosaicking," IEEE Trans. Image Processing, vol. 18, no. 6, pp. 1192-1202, June 2009. [9] C. Hu, L. Cheng, and Y. M. Lu, "Graph-based regularization for color image demosaicking," in Proc. IEEE ICIP2012, 2012. [10] N. X. Lian, L. Chang, Y. -P. Tan, and V. Zagorodnov, "Adaptive filtering for color filter array demosaicking," IEEE Trans. Image Processing, vol. 16, no. 10, pp. 2515-2525, Oct. 2007. [11] B. Leung, G. Jeon, and E. Dubois, "Least-squares luma-chroma demultiplexing algorithm for Bayer demosaicking," IEEE Trans. Image Processing, vol. 20, no. 7, pp. 1885-1894, Jul. 2011. [12] Y. Fang, "DAC spectrum of binary sources with equally-likely symbols," IEEE Trans. Communications, vol. 61, no. 4, pp. 1584-1594, Apr. 2013. [13] Y. Fang, "LDPC-based lossless compression of nonstationary binary sources using sliding-window belief propagation," IEEE Trans. Communications, vol. 60, no. 11, pp. 3161-3166, Nov. 2012. [14] Y. Fang, "Joint source-channel estimation using accumulated LDPC syndrome," IEEE Communications Letters, vol. 14, no. 11, pp. 1044-1046, Nov. 2010. [15] Y. Fang, "EREC-based length coding of variable-length data blocks," IEEE Trans. Circuits and Systems for Video Technology, vol. 20, no. 10, pp. 1358-1366, Oct. 2010. [16] Y. Fang, "Distribution of distributed arithmetic codewords for equiprobable binary sources," IEEE Signal Processing Letters, vol. 16, no. 12, pp. 1079-1082, Dec. 2009. [17] Y. Fang, "Crossover probability estimation using mean-intrinsic-LLR of LDPC syndrome," IEEE Communications Letters, vol. 13, no. 9, pp. 679-681, Sep. 2009. [18] G. Jeon, M. Anisetti, V. Bellandi, and J. Jeong, "Fuzzy rule-based edge-restoration algorithm in HDTV interlaced sequences, "IEEE Trans. Consumer Electronics, vol. 53, No. 2, pp. 725-731, May 2007. [19] G. Jeon, M. Anisetti, V. Bellandi, E. Damiani, and J. Jeong, "Rough set-assisted subfield optimization for alternating current plasma display panel," IEEE Trans. Consumer Electronics, vol. 53, no. 3, pp. 825-832, Aug. 2007. [20] G. Jeon, M. Anisetti, V. Bellandi, E. Damiani, and J. Jeong, "Fuzzy weighted approach to improve edge-based filtering," IEEE Trans. Consumer Electronics, vol. 53, no. 4, pp. 1661-1667, Nov. 2007. [21] G. Jeon, M. Anisetti, D. Kim, V. Bellandi, E. Damiani, and J. Jeong, "Fuzzy rough sets hybrid scheme for motion and scene complexity adaptive deinterlacing," Image and Vision Computing, vol. 27, no. 4, pp. 425-436, March 2009. [22] G. Jeon, M. Anisetti, V. Bellandi, E. Damiani, and J. Jeong, "Designing of a Type-2 Fuzzy Logic Filter for Improved Edge-preserving Restoration of Interlaced-to-Progressive Conversion" Sciences, vol. 179, no. 13, pp. 2194-2207, June 2009. [23] G. Jeon, M. Anisetti, and S. Kang, "A rank-ordered marginal filter for deinterlacing," Sensors, vol. 13, no. 3, pp. 3056-3065, March 2013. [24] G. Jeon, M. Anisetti, J. Lee, V. Bellandi, E. Damiani, and J. Jeong, "Concept of linguistic variable-based fuzzy ensemble approach: application to interlaced HDTV sequences," IEEE Trans. Fuzzy Systems, vol. 17, no. 6, pp. 1245-1258, Dec. 2009. [25] G. Jeon, S.J. Park, Y. Fang, M. Anisetti, V. Bellandi, E. Damiani, and J. Jeong, "Specification of Efficient Block Matching Scheme for Motion Estimation in Video Compression," Optical Engineering, vol. 48, no. 12, Dec. 2009, pp. 127005. [26] G. Jeon, M.Y. Jung, M. Anisetti, V. Bellandi, E. Damiani, and J. Jeong, "Specification of the geometric regularity model for fuzzy if-then rulebased deinterlacing," IEEE / OSA Journal of Display Technology, vol. 6, no. 6, pp. 235-243, June 2010. [27] K. H. Chung and Y. H. Chan, "Color demosaicing using variance of color differences," IEEE Trans. Image Processing, vol. 15, no. 10, pp. 2944-2955, Oct. 2006. [28] O. Werner, "Drift analysis and drift reduction for multi-resolution hybrid video coding," Signal Processing: Image Communication, vol. 8, no. 5, pp. 387-409, July 1996. [29] H. Kimata and Y. Yashima, "3D motion vector coding with block base adaptive interpolation filter on H.264, in Proc. IEEE ICASSP 2003, vol. 3, 2003, pp. 633-636. [30] X. Chen, G. Jeon, and J. Jeong, "A gradient inverse weighted filtering approach for video deinterlacing," IEICE Trans. Communications, vol. E95-B, no. 12, pp. 3933-3936, Dec. 2012. [31] Kodak color image dataset, http://r0k.us/graphics/kodak/ [32] X. Li, B. Gunturk, and L. Zhang, "Image demosaicing: a systematic survey," in Proc. of SPIE, vol. 6822, pp. 68221J, 2008. [33] X. Zhang and B. A. Wandell. "A spatial extension of cielab for digital color image reproduction," Journal of Society for Informaiton Display, vol. 5, no. 1, pp. 61-63, March 1997. [34] L. Zhang, L. Zhang, X. Mou, and D. Zhang, "FSIM: a Feature Similarity Index for Image Quality Assessment," IEEE Trans. Image Processing, vol. 20, no. 8, pp. 2378-2386, 2011. [35] Y. Yun, J. Bae, and J. Kim, "Multidirectional edge-directed interpolation with region division for natural images, Opt. Eng., Vol. 51, no. 4, pp. 040503, April 2012. [36] X. Chen, G. Jeon, and J. Jeong, "Voting-based directional interpolation method and its application to still color image demosaicking," IEEE Trans. Circuits and Systems for Video Technology, DOI: 10.1109 / TCSVT.2013.2255421.

It is an object of the present invention to provide a method of estimating missing green elements in diagonal directions as well as in horizontal and vertical directions based on the color correlation between color surfaces, To improve performance and to reduce interpolation noise with an anti-aliasing RIR filter.

A multi-directional weighted interpolation method for demodulating a Bayer pattern color filter array according to the present invention comprises the steps of (a) interpolating missing green elements in eight directions including vertical, horizontal and diagonal directions using a green interpolation part (b) interpolating the red and blue elements with a correlation of the interpolated green planes using the red-blue interpolation unit, and (c) using the post-processing unit, , And improving the blue surface.

As described above, the multi-directionally weighted interpolation method for the Bayer pattern color filter array demosaicing according to the present invention estimates the missing green elements in the diagonal direction as well as the horizontal and vertical directions based on the color correlation between color surfaces The interpolation performance can be improved, and there is a technical effect that the interpolation noise can be reduced by the anti-aliasing RIR filter.

Brief Description of the Drawings Figure 1 shows a typical 9x9 Bayer CFA block.
Figure 2 shows the pre-interpolation of missing center green elements along the four vertical and horizontal directions (north (N), south (S), east (E), west Fig.
Figure 3 shows the pre-interpolation of the missing green elements in the center according to four diagonal directions (northeast (NE), southeast (SE), northwest (NE) drawing.
4 shows the frequency response of a filter. (a) an average filter, and (b) an 8-tap fixed-coefficient Wiener interpolation filter.
Figure 5 shows sub-images in which Figure 1 is divided to interpolate the missing red (R) and blue (B) elements at the green (G) sampling location of the CFA. (a) missing red (R) elements at the red (R) sampling location, (b) red (R) elements missing at the blue (B) sampling location, R) element, (d) Green (G) Blue (B) element missing at the sampling location.
Figure 6 shows the 5x5 color difference plane used in the post-processing step according to the invention. (a) green and the difference between the red (D _GR), (b) green and the difference between the blue (D _GB), (c) the difference between red and green (D _RG), (d) difference (D _BG between the blue and green ).
Fig. 7 is an adjacent color difference value used to improve the median D _GR in the 5x5 color difference plane according to the four directions in the post-processing step according to the present invention.
Figure 8 is a sub-image of the segmented McMaster (McM) used in the experiment. Sub-images arranged from top to bottom and from left to right are labeled # 1 to # 19.
Figure 9 shows (a) the # 5 original image and the result of demosaicing it in the following manner: (b) DWI, (c) MEDI, (d) ESF, (e) SSD, (f) LPA-ICI, (g) LDI-NAT, (h) VDI, (i) MDWI _N and (j) MDWI.
Figure 10 shows (a) the # 13 original image and the result of demosaicing it in the following manner: (b) DWI, (c) MEDI, (d) ESF, (e) SSD, (f) LPA-ICI, (g) LDI-NAT, (h) VDI, (i) MDWI _N and (j) MDWI.
11 shows (a) the original image of # 17 and the result of demosaicing it in the following manner. (b) DWI, (c) MEDI, (d) ESF, (e) SSD, (f) LPA-ICI, (g) LDI-NAT, (h) VDI, (i) MDWI _N and (j) MDWI.

The multi-directional weighted interpolation method for demosaicing a Bayer pattern color filter array according to the present invention will now be described in detail.

It can be intuitively observed that the green (G) plane is twice as much as the red (R) or blue (B) plane sample from the Bayer CFA pattern of Fig. Thus, the green surface retains most of the spatial information of the image in interpolation. In addition, for human visual systems, it is most sensitive to green color. Thus, the missing green color elements in most conventional demosaicing methods are interpolated beforehand because they have a large impact on the perception quality of the image. Once the green color elements are fully recovered, the green side can use the spectral correlation entirely in the channel to guide the interpolation of the red and blue sides in the future. Therefore, the MDWI multi directional weighted interpolation method according to the present invention is mostly focused on improving the interpolation performance of the green surface.

The present invention is similar to the DWI-4 direction weighted interpolation method [7], but the interpolation method according to the present invention has a variable spatial correlation of neighboring pixels along different directions from the central missing pixel Estimation based. This estimation can easily explain that a more spatial correlation is established between adjacent pixels along the interpolation direction, and a missing color value more accurately estimated by the direction can be obtained.

First, the first feature of the present invention is to estimate the missing green elements in the diagonal direction using the fixed interpolation direction of the green surface as well as the horizontal and vertical directions based on the color correlation between the color surfaces. That is, the existing green surface is based on observation that it is consistent along the diagonal line as shown in FIG. 3, and the missing green elements in the center are obtained by using the existing green color elements and using the correlation along the diagonal direction Can be estimated. The multi-directional weighted interpolation technique is used to interpolate the missing color elements in the center according to the directional variable correlation, after estimating all eight directions of the center missing color element.

A second aspect of the present invention is to provide an effective post-processing procedure by re-registering three initial color elements. This is done in the color difference plane with an anti-aliasing FIR filter. Similar to the interpolation method, the present invention also uses a method of improving the direction weighting to change the color difference plane to reduce interpolation noise.

A multi-directional weighted interpolation method for demoicing a Bayer pattern color filter array according to the present invention comprises the steps of interpolating a missing green element in eight directions including vertical, horizontal and diagonal directions using a green interpolation part, ) Interpolating the red and blue elements with the correlation of the interpolated green planes using the red-blue interpolation unit, and (c) using the post-processing unit, And a step of improving the quality of the image.

1. Interpolation of Missing Green (G) Elements

는 8개 방향(N, S, W, E, NW, NE, SW, SE) 중에 가장 먼저 추정된다. 여기서, N, S, W, E 방향의 추정은 수학식 1과 같이 종래 DWI 기법과 동일하며, 도 2에 도시된 바와 같이 유사한 인접 픽셀을 갖는다.In the present invention, a missing green element has to be interpolated first because the green face in the Bayer CFA pattern contains more information than the red and blue faces. As in the 9x9 Bayer CFA block shown in Figure 1, the missing green elements at point B ₁₃

Is estimated first among eight directions (N, S, W, E, NW, NE, SW, SE). Here, the estimation of the N, S, W, and E directions is the same as that of the conventional DWI technique as shown in Equation 1, and has similar adjacent pixels as shown in FIG.

And the more reliable direction gradient elements in the four directions are calculated by the following equation (2).

의 추정은 다음 관찰을 기반으로 한다. 베이어 패턴에 있어서 그린 면은 연속적으로 분포되며, 대각선 방향의 그린 인접 픽셀들의 상관관계는 중앙의 누락된 그린 요소로 추정하기 위해 사용될 수 있다. 여기서, 8-탭 고정 방향 보간 필터는 NW, NE, SW, SE 방향에 따라 중앙의 누락된 그린 요소

를 근사화하는데 적용된다. 도 3에 도시된 바와 같이, 오리지널 인접 그린 픽셀의 4개 세트는

를 추정하는데 사용된다. 이러한 4개의 세트는 다음 수학식 3으로 산출되며, 방향 정보와 대응된다. Here, epsilon is a small positive element for preventing the slope from becoming zero. NW, NE, SW, SE

Is based on the following observations. In the Bayer pattern, the green planes are distributed continuously and the correlation of the green adjacent pixels in the diagonal direction can be used to estimate the missing green elements in the center. Here, the 8-tap fixed direction interpolation filter has a center missing green element according to the NW, NE, SW, SE directions

Lt; / RTI > As shown in Figure 3, the four sets of original adjacent green pixels

. These four sets are calculated by the following equation (3), and correspond to the direction information.

And the 8-tap fixed direction interpolation filter is applied to approximate the missing green element in the center of the fixed coefficient.

These 8-tap fixed-coefficient Wiener filters are mainly used to interpolate subpixels of video codecs for up-sampling of MPEG-4, H.264 / AVC, and HEVC. The Wiener filter is known as an ideal anti-aliasing filter for upsampling, and the frequency response shape is rectangular as shown in Fig. 4 (b) and has the advantage of interpolation without extreme aliasing, There is a more accurate advantage than the conventional average filter shown in Fig. Especially in high frequency regions such as irregular edges or texture details, and the missing pixels can be reconstructed more accurately with the method according to the invention. Utilizing the advantages of the demosaicking application, the missing green elements in the center B13 can be estimated along the diagonal direction. There are several other similar anti-aliasing fixed to the tap interpolation filter, such as the 6-tap interpolation filter of [29], which can be used for robust interpolation performance with high temporal efficiency but somewhat non-robustness. In the present invention, an 8-tap interpolation filter is applied, and the interpolation equation is expressed by the following equation (5).

를 이용하여 X 방향을 따라 누락된 그린 요소를 추정치를 대표한다. 따라서

의 추정값은 대각선 방향에 따라 다음과 같이 산출된다. Where X E {NW, NE, SW, SE} and G ^X are adjacent green pixel sets

Is used to represent the missing green factor along the X direction. therefore

Is calculated as follows according to the diagonal direction.

의 추정값에 대응하는 방향 기울기 요소는 다음 수학식 7에 정의된다. And along the diagonal

Is defined by the following equation (7). &Quot; (7) "

의 8개 추정값을 위한 관련 가중치는 각 방향의 역 그래디언트 사용에 의해 산출된다. 일반적으로 하나의 방향 수단에 따른 스몰 그래디언트는 그 방향에서 보다 더 많은 상관관계를 가지고 있으며, 해당 방향의 사전 추정치

에 더 높은 가중치를 하는 것이 타당하다. 따라서 본 발명에 따른 8 방향 기반의 보간 스키마는 방향 그래디언트의 조인트 분배와 컬러 차이를 반영해야 한다.

의 방향 추정치에 할당된 가중치 비율은 다음 8개 방향으로 정의된다.

The associated weights for the eight estimates of < RTI ID = 0.0 > E < / RTI > Generally, a small gradient according to one directional means has more correlation than in that direction, and a prior estimate of that direction

It is reasonable to put a higher weight on. Therefore, the 8-way based interpolation scheme according to the present invention should reflect the joint distribution and color difference of direction gradient.

The weighting factors assigned to the direction estimates of < RTI ID = 0.0 >

의 모든 공동 기여를 이용한다. 정규화 보간 방정식은 다음 수학식 9로 정의 된다. It is estimated along eight directions to avoid interpolation errors. In other words, in the present invention, in order to ensure interpolation accuracy with weighting factors of ω ^N , ω ^S , ω ^E , ω ^NW , ω ^NE , ω ^SW , and ω ^SE ,

All co-contributions of The normalization interpolation equation is defined by the following equation (9).

Where ω ^T = ω ^N + ω ^S + ω ^W + ω ^E + ω ^NW + ω ^NE + ω ^SW + ω ^SE .

By applying the above procedure at all red and blue locations, the green surface can be reconstructed.

2. Interpolation steps of missing red and blue elements

The pixels drawn from the Bayer CFA sample were initially interpolated using the approach described above. However, the 8-way based interpolation method is not suitable for interpolating missed red and blue color elements when interpolation conditions or available samples are insufficient. Fortunately, however, the red and green blues are highly correlated, so that the interpolation procedure for missing red and blue elements can use the reconstructed green side correlation to prevent color misregistration problems. The present invention improves the conventional DWI using an improvement step to interpolate missing red and blue elements. Particularly interpolated under the following two conditions. (1) missing blue (or red) elements at the red (or blue) sampling position of the CFA shown in FIGS. 5 (a) and 5 (b) Red and blue elements missing from sampling position.

를 추정할 수 있다.First, the interpolation condition of the missing blue element at the red sampling position shown in FIG. 5 (a) is considered. A two-step strategy was applied to reconstruct it to interpolate missing blue elements at the R ₆ position. The first step is to take advantage of the color difference between the green surface and blue to early interpolated blue component missing from the R ₆ position. All green elements are recovered and, if available, the color difference between blue and green in the four diagonal directions at the R ₆ position

Can be estimated.

는 B7, B8, B12, B13 각 위치에서 정제된 그린 픽셀 값이다. here

Is the green pixel value at the B7, B8, B12, and B13 positions.

에 다양한 가중치를 할당한다. 그린 면은 완전히 채워졌으므로, 네 개의 대각선 방향에 따른 그린 픽셀들의 그래디언트는 보간 절차를 가이드하는 가중치 요소로 사용된다. 4개의 방향에 따른 그래디언트는 다음 수학식 12와 같이 산출된다. For better robust estimation of spatial correlation with the application of DWI, four direction estimates

Quot; < / RTI > Since the green surface is completely filled, the gradients of the green pixels along the four diagonal directions are used as weighting elements to guide the interpolation procedure. The gradient along the four directions is calculated by the following equation (12).

는 각 R₁, R₃, R₆, R₉, R₁₁, B₇, B₈, B₁₂, B₁₃ 위치에서 보간된 그린 픽셀 값이다. 역 그래디언트는 다음과 같은 가중치 요소를 사용한다.Here, epsilon is used as a small positive factor to prevent the gradient from becoming zero.

Is the interpolated green pixel value at each of the R ₁ , R ₃ , R ₆ , R ₉ , R ₁₁ , B ₇ , B ₈ , B ₁₂ , B ₁₃ positions. The reverse gradient uses the following weighting factors:

The missing blue pixels due to the DWI combination according to the color difference between the blue and green surfaces are initially interpolated as shown in Equation (14).

이다. 두 번째 단계는

와 높은 상관관계를 가진 블루 컬러 면을 포함하고, 대각선 방향에 따라 4개의 가장 인접한 픽셀 B₇, B₈, B₁₂, B₁₃ 의 그래디언트 역 가중 필터링 방법(GIWF)을 이용한다.

를 정제하기 위해

는 다음과 같은 수학식 15로 얻을 수 있다. here

to be. The second step is

And a gradient inverse weighted filtering method (GIWF) of the four closest pixels B ₇ , B ₈ , B ₁₂ , and B ₁₃ along the diagonal direction.

To purify

Can be obtained by the following expression (15).

Where k = {7, 8, 12 , 13} , and, k indicates the four existing blue components R ₆ to close. The constraint factor δ is a positive number between 0 and 1 and is used to control the improvement performance. Theoretically, this can be achieved by training the image content. However, the present invention has been experimentally set to a constant value between 0.5 and 0.7 to simplify the general optimization performance. Interpolating performance beyond this range will be sensitive to image content.

The processing method according to the present invention is well suited for interpolating and improving missing red elements at the blue sampling position, as shown in Figure 5 (b), because it has the same sampling conditions.

The conditions of the missing blue element in the green sampling position for the second interpolation are as shown in (c, d) of FIG.

After the missing blue (or red) element is recovered at the red (or blue) sampling location of the CFA, the interpolation method is the same as handling the elements drawn along the four directions N, S, W,

는

를 개선하는데 사용되고, 가장 인접한 블루 픽셀 세트

는

를 개선하는데 사용된다. 이러한 개선 접근법은 도 5의 (d)에 도시된 보간 상황에서도 적합하다. 모든 컬러 요소를 보간하고 개선한 후에 전체 풀 컬러 영상은 디모자이크된다. 편의상 채워진 컬러 요소는 추후 절차를 위해

로 표시된다.
Of course, the GIWF is used to improve the pre-interpolated red (or blue) element at the green sampling position with the four nearest neighboring color elements in the same color plane. In the case (c) of Fig. 5, the closest red pixel set

The

The nearest blue pixel set < RTI ID = 0.0 >

The

Lt; / RTI > This improvement approach is also suitable for the interpolation situation shown in Fig. 5 (d). After interpolating and refining all color elements, the entire full color image is demarcated. For convenience, filled-in color elements are

.

3. Post-processing step

가 우선적으로 보간 성능이 개선될 수 있도록 처리되며, 5×5 슬라이딩 윈도우에서 제안된 고정된 탭 보간 필터가 있다. 여기서 후처리 단계의 출력은

로 부른다. 후 처리된 그린 면에 의한 후 처리 단계는 동일한 보간 필터의 레드와 블루 샘플을 보간하는데 수행되며, 도 6의 (c, d)에 도시된 레드와 그린 사이의 컬러 차(D_RG)와 블루와 그린 간의 컬러 차(D_BG)를 이용한다. The post-processing approach according to the present invention is mainly used to enhance the color channel correlation of the demosaicing method [4, 27]. The interpolation noise is inevitable as the red and blue sides are interpolated according to the color difference between the color sides and the unreliable and loose correlation is used between the mosaicked red and blue sides. In order to reduce the interpolation error, a re-aligning procedure based on the direction weighting improvement method is applied continuously to soften the color difference value and apply a local constant in the object. Such a post-processing schema process may be performed by using a green sample interpolated by utilizing the color difference between green and red (D _GR ) and green and blue (D _GB ), as shown in (a, b)

Is processed so that the interpolation performance can be improved first, and there is a fixed tap interpolation filter proposed in the 5x5 sliding window. Here, the output of the post-

. The post-processing step with the post-processed green surface is performed to interpolate the red and blue samples of the same interpolation filter, and the color difference (D _RG ) between red and green shown in (c, d) Green color difference (D _BG ) is used.

For robust color difference estimation, a 5-tap anti-aliasing FIR filter {h ₅ = [- 5 15 44 15 -5] / 64} is applied to improve the accuracy of adjacent color difference values along four directions in the other plane . Color Difference D _GR For example, the central color difference D _GR along each direction is defined as: < EMI ID = 17.0 >

는 도 7에 도시된 바와 같이, X 방향에 따른 중앙 값 D_GR 의 5개 인접 컬러 차이 값이다. 강건한 중앙 값

을 추정하기 위해서 대각선(P), 안티 대각선(Q), 수평(H), 수직(V)의 네 개의 방향이 고려된다. Here, X∈ {P, Q, H, V} and

Are five adjacent color difference values of the median value D _GR along the X direction, as shown in Fig. A robust median

The four directions of the diagonal line P, the anti-diagonal line Q, the horizontal line H, and the vertical line V are considered to estimate the horizontal direction.

Similar to the interpolation step, the DWI strategy also applies to the post-processing stage to enhance the co-contributions in each direction depending on the directional correlation.

The direction gradient is calculated using the absolute value of the adjacent color difference D _GR to show directional correlation along each direction. The gradient along the four directions is calculated by the following equation (18).

의 기여도를 조절하기 위한 방향 가중 요소로 사용되며, 가중치는 다음 수학식 19로 산출된다.And the inverse gradient is finally shifted to the improved color difference D _GR

Is used as a direction weighting element for adjusting the contribution of the weighting factor, and the weight is calculated by the following equation (19).

Here, Xε {P, Q, H, V} and ε are used as small positive elements to prevent the gradient from becoming zero.

와 개선 방향

이 계산되면 최종 후 처리된 중앙 D_{GR ,P}은 다음 수학식 20으로 산출된다.weight

And improvement direction

The final post-processed center D _{GR , P} is calculated by the following equation (20).

및

와 함께 최종 개선된 컬러 차 D_GR,P와 D_GB,P을 이용하여, 그린 컬러 요소를 최종적으로 개선할 수 있다.This procedure is suitable for improving the color difference value of D _GB , D _RG , D _BG to enhance interpolation accuracy. Initially demapped

And

And the final improved color differences D _{GR, P} and D _{GB, P} together with the final color difference.

In particular, the green color surface is improved as follows in the post-processing step.

The green color facet can be used to improve the red and blue facets together with the improved color differences D _{GR, P} and D _{GB, P} after the final improvement.

이 본 발명에 따른 필터를 이용하여 연속적으로 처리된 후에 풀 컬러 이미지는 마침내 채워진다.
All color faces

After being continuously processed using the filter according to the present invention, the full color image is finally filled.

Experiments to verify the performance of the multi-directional weighted interpolation method for demoicing a Bayer pattern color filter array according to the present invention are as follows.

A. Evaluation Method

In this experiment, the conventional techniques (CPSNR [27], S-CIELAB [33], and FSIMc [34]) were applied to evaluate objective image quality. First, the color peak signal-to-noise ratio (CPSNR) is applied to determine the intensity difference between the original image and the reconstructed image color plane.

This CPSNR is obtained by the following equation (23).

간의 전체 컬러(R, G, B) 평균 제곱 에러를 나타낸다. CMSE가 제로에 접근함에 따라 CPSNR은 무한대로 접근하며, 이는 높은 CMSE 값은 높은 이미지 품질을 제공함을 보여준다. 스케일의 타단에서 CPSNR의 작은 값은 I와

간의 높은 수치 차이를 의미하며, 이는 재구성된 이미지의 품질이 떨어짐을 의미한다.Where CMSE is a W × H sized original image I and a reconstructed image

(R, G, B) mean squared error between the colors of the pixels. As the CMSE approaches zero, the CPSNR approaches infinity, indicating that high CMSE values provide high image quality. The smaller value of CPSNR at the other end of the scale is I and

This means that the quality of the reconstructed image is deteriorated.

As evidenced by the prior art, sometimes the CMSE and the CPSNR do not coincide with the human perception of the color difference, so they must use a perceptually uniform color space. In this case, the color between the Euclidean distances yields a perceptually uniform space at S-CIELAB DELTA E * as DELTA E. The ΔE calculation first requires conversion from the RGB color space to the CIELAB color space, and calculates the Euclidean distance average between the pixel colors in this space. The equation is shown in Equation 25 below.

는 각각 참조 이미지와 디모자이크된 이미지를 나타내고, 두 가지 모두 CIELAB 색 공간에 있다. 이러한 방법은 휴먼 비주얼 시스템(HVS)의 인식으로 보다 더 나은 컬러 차를 추정할 수 있으며, 따라서 많은 디모자이킹 레퍼런스 중에서 이미지 품질의 글로벌 추정에 권장할 수 있다.Where W and H are the width and height, respectively, and I and

Represent the reference image and the demapped image, respectively, both of which are in the CIELAB color space. This method can estimate a better color difference with the recognition of Human Visual System (HVS), and thus can be recommended for global estimation of image quality among many demosaicing references.

In order to evaluate the performance of the present invention, the FSIM-feature-similarity index measure (FSIM) quality matrix is used as a third index for measuring the similarity between two images. It was developed by Zhang and considered in connection with the quality perception of HVS. Instead of using the conventional error summing method, the FSIM model, which is image distortion such as gradient correlation, luminance distortion, and contrast distortion, is used. Here, the partial FISM index for color image quality evaluation (FISMc) was proposed to evaluate demosaicing performance. If the value of FISMc is close to 1, the reconstructed image is very similar to the original image.

Additionally, the computational complexity is determined to evaluate the time efficiency of the present invention. In this experiment, all calculations were performed after removing the border of 10 pixel width of the image.

B. Performance Comparison

It provides an objective and subjective assessment of the MDWI according to the present invention based on various demosaicing methods. To perform the experiment, we first perform the mosaic procedure using Bayer CFA in the target test image, and apply different demosaicing methods to reconstruct the full three color channels using the mosaic image. Finally, the MDWI according to the present invention is compared with the prior art AFD [10], DL [3], ESF [4], SSD [8], LPA-ICI [5], and LDI-NAT [6].

In order to demonstrate the benefits of the invention in previous studies of VDI, VDI is also listed in the comparison method. In addition, the method of the present invention without the pretreatment step was also compared to evaluate the improvement result from the pretreatment approach of the present invention, which was expressed as MDWI _N. To verify the method according to the present invention, the Intel (R) Core (TM) i5 CPU M460 @ 2.53GHZ. Simulation was performed using the MATLAB processor.

First, the decibel CPSNR was used to measure the difference between the original image and the reconstructed image. The following Table 1 shows a comparison of objective image quality by CPSNR performance using the McM image set of FIG.

The mean of CPSNR was calculated for intuitive comparison. Based on the results of Table 1, it can be concluded that the MWDI method according to the present invention achieved a higher CPSNR than other prior art for most test images. Compared to ESF with the worst performance in CPSNR performance, 2.597 dB improved. LDI-NAT has the second highest CPSNR in the average. In contrast, the recently developed ESF obtained the lowest CPSNR in this experimental data set. Because only the green color plane is estimated to estimate the missing color component in the center, MEDI has the second lowest CPSNR performance in all comparison methods, even if interpolating missing color components along 12 directions. As for the other comparison methods, the SSD method outperformed the other two methods DWI and LPA-ICI. In particular, MDWI has a higher CPSNR than MDWI _N in all test images, indicating that the preprocessing approach according to the present invention is effective and powerful for demosaicing applications.

Euclidean distance averages between pixel colors in the CIELAB color space were further measured to further verify the performance according to the present invention. This is because the S-CIELAB DELTA E ^* performance is determined as a positive score with a better measurement approaching zero.

Using the McM image data set in this experiment, we calculated S-CIELAB △ E ^* as shown in Table 2, whereby MDWI and LDI-NAT are compared with other methods to determine the best It can clearly be seen that it exhibits the highest performance with a low S-CIELAB DELTA E ^* value. ESF generally has the highest S-CIELAB DELTA E ^* value and other methods are performed appropriately on these index items. In the comparison of the mean S-CIELAB △ E ^* , MDWI showed an improved factor of 0.519 in the ESF and S-CIELAB △ E ^* comparisons and regained the best performance of all comparative methods.

As is well known in the art, CPSNR is not a good indicator of demosaicing performance because most interpolation errors appear around a color edge that occupies only a small portion of the image pixel. To further evaluate the performance according to the present invention, we adopted a new measure of feature similarity index measure for color image (FISMc).

In this experiment, the FISMc values of each comparison method using the McM image data set are calculated as shown in Table 3, where MDWI and LDI-NAT have the highest FISMc values (except # 7 and # 8) Can be clearly identified.

The ESF has the lowest FISMc value in most of the experimental images, and other methods have been performed appropriately on these two items. In the mean FISMc comparison, the MDWI has a 0.0013 improvement compared to the ESF, providing evidence that previous research VIDs have the advantage of preserving image features, in addition to finding the correct interpolation direction. And between the other comparison methods. Similar to the results of the CPSNR matrix, the MDWI has significantly better performance than the MDWI _N in terms of FSIMc indexes, indicating that the post-processing step is valid and effective.

To evaluate the efficiency of the post-processing procedure according to the present invention in the demosaicing application, we also applied a post-processing procedure to the comparison method to determine the performance. As a result, the CPSNR, S-CIELAB △ E ^* and FSIMc indices are improved. The average CPSNR, S-CIELAB? E ^* , and FSIMc of the comparative method with and without the post-processing method according to the present invention using the McM image data set are summarized in Table 4 below.

From Table 4 it can be clearly seen that the post-processing steps and the comparison method according to the present invention have obtained overall improvements with the aid of these three measurement matrices.

For the objective image quality evaluation, three images were selected from the McM image data set and used for time comparison. Figures 9, 10 and 11 show the selection of # 5, # 13 and # 17, respectively, in the original image. The Bayer CFA pattern was used to perform the demosaicing simulation. The demosaicing was performed to reconstruct the full color image using the method of the present invention and the conventional method mentioned above. To illustrate objective demosaicing performance, a partially enlarged image from three selected images was used to determine the detail preservation capabilities of the MDWI.

Figure 9 (a) shows a partially enlarged image of a T-shirt divided from # 5, showing normal detail retention capability. By comparison, the MDWI was executed with the best visual effects in delicate edges and closed detail areas. In particular, the reconstructed diagonal edge with the measurement of missing color elements along the extra diagonal direction had minimal visual noise. For example, as compared to (i, j) other methods in Figure 9, the edge portions appeared very smooth after demosaicing of MDWI _N and MDWI. While other methods have been implemented with noise, such as noise in edges and other complex areas. When evaluating the weak detail retention capability, Image # 13 was specifically selected and used in the McM image data set. The special portion includes irregularly scribed portions as shown in Fig. 10 (a) enlarged for visual comparison. Reconstructed irregular edges and weak details using LDI-NAT, MDWI _N, and MDWI can preserve most of the degree, which can be seen in Figure 10 (gj). Conversely, other methods were implemented with zipper noise and false color visual color noise along weak edges with sudden color changes, as shown in Figure 10 (bf). This comparable method also exacerbated salt-like noise. In addition, a # 17 image was selected for comparison to further evaluate the irregular detail preservation capability of the MDWI according to the present invention. Partially enlarged portions of flowers and leaves were divided as shown in Fig. 11 (a). It is clear that the demosaicing image reconstructed by the VDI, MDWI _N and MDWI after the demosaicing process causes minimal noise in the compared method as shown in (h, i, j) in FIG. However, when such a demosaicing method is used, the joint edge between the flower and the leaf has dropped extremely, as shown in Fig. 11 (bg).

This objective and subjective evaluation shows that the MDWI according to the present invention has the advantage of preserving edge detail and generating minimum noise compared to other prior arts. In particular, it can be seen that the MDWI according to the present invention has the ability to adjust the maximum false color noise along the diagonal direction.

Although the present invention has been described with reference to the exemplary embodiments of the present invention, the technical idea of the present invention is not limited to the above-described embodiments, and a multi-directionally weighted interpolation method for various Bayer pattern color filter array demosaicing in a range not departing from the technical idea of the present invention .

G: Green B: Blue
R: Red N: north
S: south E: east
W: west NE: northeast
SE: southeast NE: northwest
SW: southwest

Claims (11)

(a) interpolating missing green elements in eight directions including vertical, horizontal, and diagonal directions using a green interpolation part;
(b) interpolating the red and blue elements with a correlation of the interpolated green surface using a red-blue interpolator, and
(c) a step of improving the green surface, the red surface and the blue surface by using the color value difference using the post-processing unit,
In the step (a), an interpolation equation using an 8-tap interpolation filter in a 9x9 Bayer CFA block is defined as Equation (5)
&Quot; (5) "

Where X E {NW, NE, SW, SE} and G ^X are adjacent green pixel sets

To represent estimates of missing green elements along the X direction.
In step (a), the missing green element

Is calculated by Equation (6)
&Quot; (6) "

In the step (a)

The directional gradient component corresponding to the estimated value of < RTI ID = 0.0 >
&Quot; (7) "

In the step (a)

The weighted ratio assigned to the direction estimate of < EMI ID = 8.0 >
&Quot; (8) "

The step (a) may include estimating a weighting factor of ω ^N , ω ^S , ω ^E , ω ^NW , ω ^NE , ω ^SW , ω ^SE in advance in eight directions to ensure interpolation accuracy

Wherein the normalized interpolation equation is defined by equation (9). ≪ RTI ID = 0.0 > 8. < / RTI >
&Quot; (9) "

delete delete delete delete delete The method according to claim 1,
The step (b)
interpolating the missing blue (or red) element at the red (or blue) sampling location of (b-1) CFA and
(b-2) interpolating missing red and blue elements at a green sampling location of the CFA. < Desc / Clms Page number 20 > The method according to claim 1,
The step (c) applies a color difference (D _RG ) between red and green and a color difference (D _BG ) between blue and green with a fixed 5-tap anti-aliasing FIR filter in a 5x5 sliding window, And the inverse gradient is finally calculated as the improved color difference D _GR in the direction < RTI ID = 0.0 >

Is used as a direction weighting element for adjusting the contribution of the Bayer pattern color filter array demosaicing, and the weight is calculated by the following equation (19).
&Quot; (18) "

&Quot; (19) "

Here, Xε {P, Q, H, V} and ε are used as small positive elements to prevent the gradient from becoming zero. 9. The method of claim 8,
In the step (c)
weight

And improvement direction

Is calculated, the final post-processed center D _{GR , P} is computed as: < EMI ID = 20.0 >
&Quot; (20) "

10. The method of claim 9,
Wherein the green surface color of step (c) is improved as shown in equation (21).
&Quot; (21) "

11. The method of claim 10,
Wherein the red and blue face colors of step (c) are modified as shown in equation (22). ≪ EMI ID = 22.0 >
&Quot; (22) "

Images