KR102116095B1 - Method for minimizing color interpolation error - Google Patents

Abstract

As a preferred embodiment of the present invention, the color interpolation improvement method comprises: generating a base channel from R, G, B and N input channels; Calculating a difference value between the generated base channel and each of the R, G, B and N input channels; And minimizing a color interpolation (CI) error of the calculated difference value.

Description

Method for minimizing color interpolation error}

The present invention relates to an image processing apparatus, and more particularly to a color interpolation method suitable for the characteristics of the MFA sensor.

In order to simultaneously acquire a visible light band image and an invisible band NIR image, images of each channel are simultaneously acquired using a multi-spectral filter array.

KR 2010-0084458

In the process of performing color interpolation (CI) from a pattern image converted to a Bayer pattern after quincuncial interpolation, we aim to maximize the resolution of the MFA sensor while solving the problem of large performance degradation.

As a preferred embodiment of the present invention, the color interpolation improvement method comprises: generating a base channel from R, G, B and N input channels; Calculating a difference value between the generated base channel and each of the R, G, B and N input channels; And minimizing a CI (Color Interpolation) error of the calculated difference value, wherein the minimizing the error weights each pixel in each local window of the R, G, B, and N input channels. It is characterized by adding a weighted sum by adding a weight to a difference value between a sum value and a center pixel (m, n) in the specific local window of the base channel and each of the surrounding pixels.

Preferably, the base channel is generated by dividing the sum of R, G, B, and N input channels by 4.

Preferably, the weight is a distance value between each pixel in a local window of each of the R, G, B and N input channels and a central pixel (m, n) in a local window of each of the R, G, B and N input channels. It is characterized by being set based on.

Preferably, the weight is the distance between each pixel in the local window of each of the R, G, B and N input channels and the center pixel (m, n) in the local window of each of the R, G, B and N input channels. It is characterized by decreasing as it goes further.

Preferably, the R, G, B and N input channels are characterized in that the color interpolation is performed by removing aliasing caused by overlapping of the luminance channel and the luminance channel.

The method for improving color interpolation disclosed in a preferred embodiment of the present invention has an effect of solving a problem of color aliasing and overshoot of a color patch portion.

In addition, there is an effect of solving a problem in that performance deterioration was large in the process of performing color interpolation (CI) from a pattern image that has been converted into a Bayer pattern after Quincuncial interpolation.

1 is a block diagram schematically showing a camera system as a preferred embodiment of the present invention.
2 shows an example of converting an MFA pattern image into a quincuncial pattern image as one preferred embodiment of the present invention.
3 to 6 illustrate an example in which an error occurs even after color interpolation removing aliasing caused by overlapping of the luminance channel and the luminance channel using the quincuncial pattern image of FIG. 2.
7 shows the Gaussian distribution of the errors that occur in this case.
8 is a flowchart of a color interpolation improvement method as a preferred embodiment of the present invention.

The terminology used herein is for describing the embodiments and is not intended to limit the present invention. In the present specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, "comprises" and / or "comprising" refers to the components, steps, operations and / or elements mentioned above, the presence of one or more other components, steps, operations and / or elements. Or do not exclude additions.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings commonly understood by those skilled in the art to which the present invention pertains. In addition, terms defined in the commonly used dictionary are not ideally or excessively interpreted unless explicitly defined.

1 is a block diagram schematically showing a camera system as a preferred embodiment of the present invention.

The present invention can be applied to a digital camera system using a sensor having an RGB + NIR structure.

Referring to FIG. 1, the camera system 1 includes an image sensor 10, an image processing device 30, and a display device 50. The camera system 1 may be a video shooting system such as a digital camera, a camcorder, or a surveillance camera, or may be mounted on a computer, PDA (Personal Digital Assistant), PMP (Portable Multimedia Player), mobile phone, or the like.

The image sensor 10 uses a photoelectric conversion element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The image sensor 10 includes a pixel array in which a plurality of pixels are formed to convert an optical signal into an electrical image signal by a photoelectric conversion element.

On the image sensor 10, a multi-spectral filter array for passing light in visible and invisible bands is disposed. The MFA includes a color filter that transmits light components in the visible light band ranging from 400 nm to 700 nm, such as RGB, and a near infrared filter that transmits light components in the near-infrared region of 700 nm to 1100 nm in the invisible band.

The color filters (RCF, GCF, BCF) of the MFA and the near infrared filter (NIRF) are arranged to correspond to each pixel of the image sensor 10, so that each pixel of the image sensor 10 is a color channel signal R passing through the MFA , G, B) and near infrared channel signals (NIR, hereinafter "N"). A lens (not shown) for receiving an optical signal may be provided at the front end of the image sensor 10.

The present invention is premised to further improve color interpolation when an error occurs after color interpolation removing aliasing caused by overlapping of the luminance channel and the luminance channel.

In detail, in one preferred embodiment of the present invention, in order to improve color interpolation, aliasing is removed from the MFA image, and when R, G, B, and N channels on which color interpolation has already been performed are input, respectively, each of the input R and G , Base channels are created from B and N channels. Then, the difference values of the generated base channel and each of the R, G, B and N channels are calculated.

Then, in order to minimize the error included in the difference value, each pixel in a local window corresponding to a specific area of each of the R, G, B and N input channels and the center pixel (m, n) of the local window The weighted sum of the difference values between is calculated, and the calculated weighted sum is calculated and summed with the pixel values at the center pixel (m, n) positions of the base channel generated above.

In this case, the weighted sum is weighted according to the distance between each pixel in the local window corresponding to a specific area of each of the R, G, B, and N input channels and the center pixel (m, n) of the local window. It is created by multiplying the values. Preferably, the distance and weight values are inversely proportional. Therefore, as the distance increases, the weight value decreases. In this regard, it will be described in more detail below.

2 is a preferred embodiment of the present invention, showing an example of converting the MFA pattern image to a quincuncial pattern image.

In a preferred embodiment of the present invention, to apply the existing color interpolation algorithm to the R, G, B, NIR channel interpolation method used in the present invention by approximating the MFA pattern image with the Bayer pattern, the difference value in the diagonal pixel direction Based on the MFA pattern image 210 is changed to a quincuncial pattern image (220).

As a preferred embodiment, when interpolating the channel value of the center pixel (m, n) 200 position of the MFA pattern image, one cell in a diagonal direction from the center pixel (m, n) 200 of the MFA pattern image The first weighted average value of the pixel values 211, 212, 213, and 214 at a distant location and the center pixel (m, n) of the MFA pattern image and the position (221, 222) of two spaces diagonally away from the center pixel of the MFA pattern , 223, 224) is a sum of a second weighted average value weighted by a difference value of each pixel value. Each weight used to obtain the first weighted average value and the second weighted average value is inversely proportional to the difference value for each pixel direction .

In a preferred embodiment of the present invention, the color channel and the near infrared channel constituting the MFA pattern image are interpolated at the maximum resolution based on the difference value in the vertical and horizontal pixel direction of the changed quincuncial pattern image.

3 to 6 illustrate an example in which an error occurs even after color interpolation removing aliasing caused by overlapping of the luminance channel and the luminance channel using the quincuncial pattern image of FIG. 2.

FIG. 3 shows an example in which a NIR longitudinal defocus problem occurs in a double aspherical surface lens with IR coating.

4 shows an example in which a longitudinal defocus problem occurs even in a red channel in a general lens. This error phenomenon is caused by different refractive indices for each wavelength band, and is caused by a different focus on the sensor surface.

When the IR cut filters (IRCFs) 310 and 410 are not used as shown in FIGS. 3 to 4, referring to FIG. 5, even in the case of a dual aspheric lens with an IR coating, a longitudinal defocus problem occurs in both the NIR channel and the red channel. do. Referring to FIG. 6, in the case of a normal lens, when an IR cut filter (IRCF) 310 or 410 is not used, a longitudinal defocus problem occurs in all channels.

Due to the phenomena as shown in FIGS. 3 to 6, an error occurs, and as the distance from the center of the image increases, the edge portions do not coincide.

7 shows a Gaussian distribution of CI (Color Interpolation) errors occurring in this case. In a preferred embodiment of the present invention, it is intended to minimize this CI error value.

To do this, first create a base channel in the following way.

Base channel = (R channel + G channel + B channel + N channel) / 4

,

,

및

과 같이 표시할 수 있다. 이하에서는 설명의 편의를 위해 G 채널의 일 예를 들어 설명하면 아래와 같다. Then, the difference value between the generated base channel and each of the R, G, B and N input channels is calculated. The difference between the base channel and each of the R, G, B and N input channels

,

,

And

Can be displayed as Hereinafter, an example of the G channel will be described for convenience of description.

Equation 1 shows the difference value between the G channel and the Base channel.

라고 할 수 있다. 이 경우, CI(Color Interpolation) 에러를

로 정의하면, 도 7 과 같은 형태의 Gaussian형태를 갖는 정규분포 형태(Probability Density Function)로 표시된다. 바람직한 일 실시예로서, CI(Color Interpolation) 에러는 실험적으로 측정한 결과 0에 가까운 평균값을 갖고 2.76의 표준편차를 갖는다. The value when there is no CI (Color Interpolation) error in the difference between the base channel and the G input channel

Can be said. In this case, CI (Color Interpolation) error

If is defined as, it is represented by a normal distribution form (Probability Density Function) having a Gaussian form as shown in FIG. As a preferred embodiment, the CI (Color Interpolation) error has an average value close to 0 as a result of empirical measurements and a standard deviation of 2.76.

을 표시하면 수학식 2와 같다.G-channel with minimal CI error to reduce longitudinal defocus caused by lenses

Is represented by Equation 2.

는 G 채널의 특정 영역(local window) 내의 각 픽셀 (m+a,n+b)에 부여되는 가중치를 나타낸다. 이 경우 가중치값은 local window의 중심 픽셀(m,n)에서 거리가 멀어질수록 크기가 작아진다. In Equation 2

Denotes a weight assigned to each pixel (m + a, n + b) in a specific region (local window) of the G channel. In this case, the weight value becomes smaller as the distance from the center pixel (m, n) of the local window increases.

Looking at Equation 2, in order to generate a G channel with minimized errors, a weight is assigned to each pixel included in the local window of the G channel, and the summation value and (m, n) pixels and surroundings within a specific local window of the Base channel. The weighted value is added to the difference value from each of the pixels (m + a, n + b) to add a weighted value. The methods disclosed in Equations 1 to 2 apply substantially the same to R, B and N channels.

8 is a flowchart of a color interpolation improvement method as a preferred embodiment of the present invention.

The base channel is generated from the R, G, B and N input channels on which color interpolation is performed by removing aliasing caused by overlapping of the luminance channel and the luminance channel (S810). An example of generating a base channel is a method of dividing the sum of R, G, B and N input channels by 4.

Thereafter, a difference value between the generated base channel and each of the R, G, B, and N input channels is calculated (S820). Thereafter, the color interpolation error (CI (Color Interpolation) error) of the calculated difference value is minimized (S830). In this case, refer to Equation 2 for a method of minimizing the color interpolation error.

In detail, weights are assigned to each pixel in the local window of each of the R, G, B, and N input channels, and a weight is added to the difference between the sum of the center pixel (m, n) and each of the surrounding pixels in a specific local window of the base channel. To minimize the color interpolation error, add the weighted sum.

Meanwhile, the present invention can be embodied in computer readable codes on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data readable by a computer system are stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. In addition, the computer-readable recording medium can be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention pertains.

So far, the present invention has been focused on preferred embodiments. Those skilled in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an explanatory point of view rather than a restrictive point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent range should be construed as being included in the present invention.

Claims (5)

Generating base channels from R, G, B and N input channels;
Calculating a difference value between the generated base channel and each of the R, G, B and N input channels; And
And minimizing the CI (Color Interpolation) error of the calculated difference value.
The R, G, B, and N input channels are weighted for each pixel in each local window, and the sum is added to the difference between the center pixel (m, n) in the specific local window of the base channel and each of the surrounding pixels. A method of improving color interpolation, which is generated by dividing the sum of R, G, B, and N input channels by 4 in the base channel. delete The method of claim 1, wherein the weight assigned to each pixel is
It is set based on the distance value between each pixel in the local window of each of the R, G, B and N input channels and the center pixel (m, n) in the local window of each of the R, G, B and N input channels, In addition, the distance between the pixels in the local window of each of the R, G, B, and N input channels and the center pixel (m, n) in the local window of each of the R, G, B, and N input channels are increased. A method for improving color interpolation, which is characterized by decreasing. delete The method of claim 1, wherein the R, G, B and N input channels are
A color interpolation improvement method characterized by being an input channel in which color interpolation is performed by removing aliasing caused by overlapping of the luminance channel and the luminance channel.

Images