JP4302854B2 - Image processing method and apparatus, and recording medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing method and apparatus for estimating signal values at all pixel positions of an imaging device using signal values obtained in an imaging device such as a single-plate CCD, and to cause a computer to execute the image processing method. The present invention relates to a computer-readable recording medium on which a program is recorded.
[0002]
[Prior art]
2. Description of the Related Art As an imaging device such as a CCD used in a digital camera, a device in which a plurality of types of photoelectric conversion elements having different spectral sensitivities are alternately arranged on the same plane is known (hereinafter referred to as a single plate CCD). ). Here, in the case of a single-plate CCD in which photoelectric conversion elements having spectral sensitivities in R, G, and B, that is, photoelectric conversion elements in R, G, and B channels are alternately arranged, continuous R, G, B A set of three photoelectric conversion elements in the channel constitutes one pixel. However, in such a single-plate CCD, since the R, G, B signal values of each pixel cannot be obtained at the same pixel position, color shift or false color may occur. Further, since the number of photoelectric conversion elements in each channel is smaller than the total number of elements constituting the single-plate CCD, a high-resolution image cannot be obtained. For example, in a single-chip CCD in which photoelectric conversion elements for R, G, and B channels are alternately arranged, the number of photoelectric conversion elements for each channel is only 1/3 of the total number of elements. In comparison, the resolution becomes 1/3. For this reason, there has been proposed a method for obtaining the signal value in the portion where the photoelectric conversion elements of the R, G, and B channels do not exist by interpolation processing. However, in the portion where the signal value changes greatly only by performing the interpolation processing. False colors may occur. In this case, the generation of false colors can be prevented by using an optical low-pass filter in the imaging system or by performing smoothing processing on the imaging signal using the low-pass filter. However, in this case, there is a problem that the resolution deteriorates. is there.
[0003]
Here, human visual characteristics are more sensitive to luminance than to color. For this reason, a high-frequency luminance signal representing the luminance of each pixel and a low-frequency color signal obtained by the above-described interpolation processing and smoothing processing using a low-pass filter are generated from the color imaging signal obtained by the single CCD. However, a method has been proposed in which a color image signal is reconstructed using a luminance signal and a color signal (Japanese Patent Laid-Open No. 10-200906, etc.). According to this method, more information is given to the luminance component that is highly sensitive in human visual characteristics, so that it is possible to obtain a color image signal that can reproduce an image with an apparently high resolution.
[0004]
By the way, as a single-plate CCD, for example, a CCD having a honeycomb array array structure in which pixels are arranged in a checkered pattern as shown in FIG. 33 is known (for example, Japanese Patent Laid-Open No. 10-136391). This may be referred to as a checkered pixel array. As shown in FIG. 34, a CCD having a Bayer array structure in which pixels are arranged in a square shape is also known. This may be referred to as a square pixel array. A single-plate CCD having such an array structure, that is, a pixel array also has a false color problem as described above. For this reason, in order to remove the false color from the light quantity-based signal obtained in the CCD with the Bayer array, the ratio of the r, g, and b signals is substantially constant in a local region in the image. Based on the assumption, in a vertical or horizontal line in a CCD with a Bayer array, the g signal is multiplied by the ratio of the r signal to the g signal in the adjacent line to calculate the r signal in that line. This method is known (Japanese Patent Laid-Open No. 9-214989). Specifically, in this method, in order to obtain the r signal r12 at the g12 pixel position in the pixel array shown in FIG. 35, first, the g11 signal at the r11 pixel position is calculated by the calculation of (g6 + g16) / 2, and r11: Based on the assumption that g11 = r12: g12, the r12 signal is calculated by the calculation of r12 = g12 × r11 / g11.
[0005]
However, in the method described in Japanese Patent Laid-Open No. 10-200906 and the like, the high-frequency component in the actual image is obtained no matter what the low-pass filter is used to smooth the image pickup signal obtained in the single-plate CCD. Has already been folded in the image, the moire caused by the folding distortion cannot be removed, and as a result, the false color cannot be sufficiently removed.
[0006]
On the other hand, according to the method described in JP-A-9-214989, false colors can be effectively removed. In particular, this method is based on the assumption that the ratio of the light quantity of r: g: b is constant in a local region of the image, and the analog signal in which the ratio of the obtained RGB signals is proportional to the light quantity. In this case, the false color in the image signal obtained by the Bayer array CCD can be efficiently removed. However, an image signal obtained by a digital camera is input to a video circuit of a computer system in order to reduce quantization error when A / D conversion is performed to convert a light amount rgb into a digital RGB signal. For example, R = r ^0.45 , R = log (r), the signal value is expressed so as to be an exponent value and a logarithmic value with respect to the light amount, and therefore r: g: b = R: G: B is not satisfied. For this reason, the method described in Japanese Patent Laid-Open No. 9-214989 can remove false colors from analog signals whose signal values are proportional to the amount of light, but the signal value is an exponent value or logarithmic value of the amount of light. The false color cannot be removed. The false color is generated not only in the Bayer array single CCD but also in the honeycomb array single CCD.
[0007]
For this reason, when the signal value obtained in the imaging device is represented by an exponent value or logarithmic value with respect to the light amount, based on the premise that the difference between the RGB signal values is constant in the local region of the image, A method for calculating the signal value at each pixel position in consideration of the direction in which the signal value at the pixel position for calculating the signal value changes has been proposed (Japanese Patent Application No. 11-212202). Hereinafter, this method will be specifically described.
[0008]
First, the pixel arrangement of the CCD as an imaging device is the honeycomb arrangement shown in FIG. 33, and reference numbers are given to the respective pixel positions as shown in FIG. In addition, when this pixel arrangement is regarded as a square arrangement, pixel positions having no signal value are indicated by * in FIG. The signal value obtained in each channel is assumed to be 8 bits, the direction extending from the upper left to the lower right of the page is indicated by the arrow A direction, and the direction extending from the upper right to the lower left of the page is indicated by the arrow B. In the book, the directions of arrow A and arrow B are unchanged. Here, calculation of the R signal at the B24 pixel position (hereinafter referred to as R (B24) signal) will be described. First, in the direction of arrow A, the R (B24) A signal at the B24 pixel position is obtained by the following equation (1).
[0009]
R (B24) A = G (B24) A + ((R (G33) A-G33) + (R (G15) A-G15)) / 2 (1)
However, the G (B24) A signal is a one-dimensional interpolation such as one-dimensional Cubic spline interpolation calculation in the direction of arrow A with respect to the G signal at the G pixel position on the line where the pixels are aligned with B02, G13, B24, G35. This is the G signal at the B24 pixel position obtained by performing the calculation. Further, the R (G33) A signal is obtained by performing one-dimensional interpolation in the direction of arrow A on the R signal at the R pixel position on the line where the pixels are aligned with R22, G33, R44, G55. This is an R signal at a pixel position. Further, the R (G15) A signal is a G15 pixel obtained by performing a one-dimensional interpolation operation in the arrow A direction on the R signal at the R pixel position on the line where the pixels are aligned with R04, G15, R26, G37. It is a position R signal.
[0010]
Next, in the direction of arrow B, the R (B24) B signal at the B24 pixel position is obtained by the following equation (2).
[0011]
R (B24) B = G (B24) B + ((R (G13) B-G13) + (R (G35) B-G35)) / 2 (2)
However, the G (B24) B signal is a B24 pixel obtained by performing one-dimensional interpolation operation in the direction of arrow B on the G signal at the G pixel position on the line where the pixels are aligned with B06, G15, B24, G33. This is the G signal of the position. Further, the R (G13) B signal is obtained by performing a one-dimensional interpolation operation in the arrow B direction on the R signal at the R pixel position on the line where the pixels are aligned with R04, G13, R22, G31. It is a position R signal. Further, the R (G35) B signal is obtained by performing a one-dimensional interpolation operation in the direction of arrow B on the R signal at the R pixel position on the line where the pixels are aligned with R26, G35, R44, G53. This is an R signal at a pixel position.
[0012]
Then, the following formulas (3a) and (3b) are used to calculate fluctuation values SA and SB representing the fluctuation amount of the signal value in the arrow A direction and the arrow B direction at the B24 pixel position. Is used to calculate a weighting coefficient Sj for weighted addition of the R (B24) A signal and the R (B24) B signal by the equation (3c).
[0013]
SA = | B24-G (B24) A | / (B24 + G (B24) A) (3a)
SB = | B24-G (B24) B | / (B24 + G (B24) B) (3b)
Sj = SA / (SA + SB) (If SA + SB = 0, Sj = 0.5) (3c)
When the weighting coefficient Sj is obtained in this way, the R (B24) signal is calculated by the following equation (4).
[0014]
R (B24) = (1-Sj) * R (B24) A + Sj * R (B24) B (4)
According to this method, even if the signal value obtained in the imaging device is represented by an exponent value or a logarithmic value, the signal value at all pixel positions can be obtained without generating false colors. Further, in this method, according to the direction in which the signal value changes, the signal value calculated for the predetermined direction on the imaging device and the orthogonal direction orthogonal thereto is weighted and added as shown in the above equation (4), By calculating the signal value at each pixel position, generation of false colors can be prevented regardless of the change direction of the signal value, and the occurrence of artifacts can also be prevented.
[0015]
[Problems to be solved by the invention]
However, in the method of Japanese Patent Application No. 11-212202, when the pixel position where the signal value is estimated is on the color edge of the image, the signal value can be calculated without generating a false color. If the pixel whose value is to be estimated is on the gray edge, the false color cannot be reduced sufficiently. Hereinafter, this problem will be described. First, in the pixel array shown in FIG. 36, signal values are obtained as shown in FIG. 16, and white (signal value is 255) and black (signal value is 0 or close to 0) gray edges are represented. It shall be. Here, calculation of the B (R44) signal at the R44 pixel position will be described. First, in the direction of arrow A, a B (R44) A signal at the R44 pixel position is obtained by the following equation (5).
[0016]
B (R44) A = G (R44) A + ((B (G35) A-G35) + (B (G53) A-G53)) / 2
= 0 + ((2-0) + (255-255)) / 2 = 1 (5)
However, G (R44) A = 0, B (G35) A = 2, B (G53) A = 255
Further, in the direction of arrow B, the B (R44) B signal at the R44 pixel position is obtained by the following equation (6).
[0017]
B (R44) B = G (R44) B + ((B (G33) B-G33) + (B (G55) B-G55)) / 2
= 103 + ((105-0) + (105-0)) / 2 = 208 (6)
However, G (R44) B ≒ 103, B (G33) B ≒ 105, B (G55) B ≒ 105
Then, a weighting coefficient for weighting and adding the B (R44) A signal and the B (R44) B signal is calculated by the following equation (7).
[0018]
SA = | R44-G (R44) A | / (R44 + G (R44) A) = | 1-0 | / (1 + 0) = 1
SB = | R44-G (R44) B | / (R44 + G (R44) B) = | 1-103 | / (1 + 103) ≒ 0.98 (7)
Sj = SA / (SA + SB) = 1 / (1 + 0.98) ≒ 0.5
When the weighting coefficient Sj is obtained in this way, a B (R44) signal is calculated by the following equation (8).
[0019]
B (R44) = (1-Sj) * B (R44) A + Sj * B (R44) B = 0.5 * 1 + 0.5 * 208 ≒ 105 (8)
Here, since the R44 pixel position is on the black edge, the value of the B (R44) signal is preferably close to zero. However, if the weighting coefficient Sj is calculated as shown in the above equation (7) and the value of the B (R44) signal is calculated by the equation (8), the value becomes 105, and a false color is generated here. It becomes.
[0020]
The present invention has been made in view of the above circumstances, and records an image processing method and apparatus capable of estimating a signal value without causing a false color even for a gray edge, and a program for causing a computer to execute the image processing method. An object of the present invention is to provide a computer-readable recording medium.
[0021]
[Means for Solving the Problems]
An image processing method according to the present invention is an imaging device in which first to third photoelectric conversion elements having different spectral sensitivities are arranged on a single surface to form an imaging surface, wherein the first and second The photoelectric conversion elements are alternately arranged in a predetermined direction to form a first line, the first and third photoelectric conversion elements are alternately arranged in the predetermined direction to form a second line, and the predetermined direction The first and second lines are arranged so that the first and second photoelectric conversion elements are alternately arranged in the orthogonal direction orthogonal to the first and third photoelectric conversion elements. Based on the first to third signal values obtained in the imaging device in which the imaging surface including the first to third pixels is formed by alternately arranging in the orthogonal direction, the first at all pixel positions. To the third signal value In out image processing method for estimating at least one signal value,
Determine whether the pixel position where the signal value is estimated is at the gray edge or the color edge,
A predetermined direction estimated value that is an estimated value of the signal value for the predetermined direction and an orthogonal direction estimated value that is an estimated value of the signal value for the orthogonal direction;
By calculating differently depending on the result of the determination, a predetermined direction fluctuation value representing the fluctuation amount of the signal value in the predetermined direction and an orthogonal direction fluctuation value representing the fluctuation amount of the signal value in the orthogonal direction are calculated,
Based on the predetermined direction variation value and the orthogonal direction variation value, set a weighting coefficient for weighted addition of the predetermined direction estimated value and the orthogonal direction estimated value,
Based on the weighting coefficient, the predetermined direction estimated value and the orthogonal direction estimated value are weighted and added to estimate at least one signal value of the first to third signal values. .
[0022]
Here, in the single-chip CCD having the honeycomb arrangement as shown in FIG. 33, assuming that the first to third pixels and the first to third signal values correspond to G, R, and B, respectively, the first line is 33 is a GR line in which GR pixels are alternately arranged from the upper left to the lower right (this is a predetermined direction), and the second line is a GB line in which GB pixels are alternately arranged in the same direction as the first line. Become. In addition, GR lines and GB lines are alternately arranged in a direction orthogonal to the predetermined direction. In the image processing method according to the present invention, when a signal value obtained in an imaging device such as a single CCD is represented by an exponent value or a logarithmic value with respect to the amount of light, each signal value is obtained in a local region on the image. Based on the premise that the difference values are equal, all signal values at each pixel position are estimated from the first to third signal values obtained in such an imaging device. In the present invention, the pixel arrangement of the imaging device is not limited to this.
[0023]
The “predetermined direction estimated value” and the “orthogonal direction estimated value” are at least one signal value of the first to third signal values estimated for the predetermined direction and the orthogonal direction, respectively.
[0024]
“Calculating the predetermined direction variation value and the orthogonal direction variation value by a different calculation method according to the determination result” refers to, for example, the above formulas (3a) and (3b) when the color edge is determined. When the fluctuation value SA and fluctuation value SB are obtained as the predetermined direction fluctuation value and the orthogonal direction fluctuation value by the calculation method shown in FIG. 6 and determined to be a gray edge, for example, the following equations (9a) and (9b) are given. It means obtaining the fluctuation value SA and the fluctuation value SB as a predetermined direction fluctuation value and an orthogonal direction fluctuation value by a calculation method.
[0025]
SA = | B24-G (B24) A | (9a)
SB = | B24-G (B24) B | (9b)
“Set weighting coefficient” means that if a gray edge is determined, a weighting coefficient for weighting the predetermined direction estimated value and the orthogonal direction estimated value is set so that a false color does not occur, and the color edge is determined. In this case, the weighting coefficient is set so that a fine color can be left while the false color is reduced.
[0026]
In the image processing method according to the present invention, an addition value between the predetermined direction variation value and the orthogonal direction variation value is calculated, and a ratio of the predetermined direction variation value or the orthogonal direction variation value to the addition value is calculated. The ratio is preferably set as a weighting coefficient when the predetermined direction estimation value and the orthogonal direction estimation value are weighted and added.
[0027]
Further, in the image processing method according to the present invention, a ratio of the predetermined direction variation value or the orthogonal direction variation value to an addition value of the predetermined direction variation value and the orthogonal direction variation value is set as the predetermined direction estimated value and the orthogonal direction. When setting as a weighting coefficient for weighted addition of estimated values,
Based on the relationship between the predetermined direction variation value and the orthogonal direction variation value when the weighting factor is constant, a correspondence relationship between the predetermined direction variation value and the orthogonal direction variation value for a plurality of weighting factors is calculated in advance. With reference to the storage means stored in the above, based on the predetermined direction variation value and the orthogonal direction variation value, one weighting factor is selected from the plurality of weighting factors, and the selected one weighting factor is It is preferable that the predetermined direction estimated value and the orthogonal direction estimated value are set as weighting coefficients when weighted addition is performed.
[0028]
In this case, the ratio of the predetermined direction variation value or the orthogonal direction variation value to the addition value is ½ with respect to the plurality of weighting coefficients in the correspondence relationship stored in the storage unit. ⁿ (N is a natural number)
The weighted addition is preferably performed by bit shift based on the weighting coefficient.
[0029]
Here, the ratio of the predetermined direction variation value or the orthogonal direction variation value to the added value is 1/2. ⁿ When the weighting coefficient is set so that (n is a natural number), when the predetermined direction estimated value and the orthogonal direction estimated value are represented by binary numbers, they are reduced to 1/2. ⁿ Multiplying by means that an operation equivalent to shifting these estimated values to the right by n bits is performed. Therefore, “bit shift based on weighting coefficient” means that the value of one selected weighting coefficient is 1/2. ⁿ The weighted addition is performed by shifting the predetermined direction estimation value and the orthogonal direction estimation value to the right by n bits.
[0030]
In the image processing method according to the present invention, it is determined whether or not the pixel position from which the signal value is estimated is in a monochromatic flat portion including only one of the first to third signal values. ,
If it is determined that the pixel is in the monochrome flat part, at least one signal value of the first to third signal values is obtained by linear interpolation based on the first to third signal values in the monochrome flat part. Estimate
When it is determined that the monochrome flat portion is not present, it is determined whether the gray edge or the color edge is present, calculation of the predetermined direction estimated value and the orthogonal direction estimated value, the predetermined direction variation value, and the It is preferable to calculate the orthogonal direction variation value, set the weighting coefficient, and estimate the signal value based on the weighting coefficient.
[0031]
In the image processing method according to the present invention, each of the first to third photoelectric conversion elements has a color of G (green), B (blue), or R (red), or Y (yellow). , G (green), and C (cyan) are preferably those having spectral sensitivity.
[0032]
An image processing apparatus according to the present invention is an imaging device in which imaging surfaces are formed by arranging first to third photoelectric conversion elements having different spectral sensitivities on a single surface, wherein the first and second The photoelectric conversion elements are alternately arranged in a predetermined direction to form a first line, the first and third photoelectric conversion elements are alternately arranged in the predetermined direction to form a second line, and the predetermined direction The first and second lines are arranged so that the first and second photoelectric conversion elements are alternately arranged in the orthogonal direction orthogonal to the first and third photoelectric conversion elements. Based on the first to third signal values obtained in the imaging device in which the imaging surface including the first to third pixels is formed by alternately arranging in the orthogonal direction, the first at all pixel positions. To the third signal value In out image processing apparatus for estimating at least one signal value,
Determining means for determining whether a pixel position for estimating a signal value is at a gray edge or a color edge;
An estimated value calculating means for calculating a predetermined direction estimated value that is an estimated value of the signal value for the predetermined direction and an orthogonal direction estimated value that is an estimated value of the signal value for the orthogonal direction;
A variation value for calculating a predetermined direction variation value that represents the variation amount of the signal value in the predetermined direction and an orthogonal direction variation value that represents the variation amount of the signal value in the orthogonal direction by different calculation methods according to the determination result. A calculation means;
Weighting coefficient setting means for setting a weighting coefficient for weighted addition of the predetermined direction estimation value and the orthogonal direction estimation value based on the predetermined direction fluctuation value and the orthogonal direction fluctuation value;
Weighting addition means for estimating the at least one signal value among the first to third signal values by weighting and adding the predetermined direction estimated value and the orthogonal direction estimated value based on the weighting coefficient. It is characterized by.
[0033]
In the image processing apparatus according to the present invention, the weighting coefficient setting unit calculates an addition value of the predetermined direction variation value and the orthogonal direction variation value, and the predetermined direction variation value or the orthogonal direction with respect to the addition value. A ratio calculation means for calculating the ratio of the fluctuation value is provided,
It is preferable that the ratio is a means for setting a weighting coefficient for weighted addition of the predetermined direction estimated value and the orthogonal direction estimated value.
[0034]
In the image processing apparatus according to the present invention, the weighting coefficient setting unit may calculate a ratio of the predetermined direction variation value or the orthogonal direction variation value to an addition value of the predetermined direction variation value and the orthogonal direction variation value. Based on the relationship between the predetermined direction fluctuation value and the orthogonal direction fluctuation value when the weighting coefficient is constant when setting the predetermined direction estimation value and the orthogonal direction estimation value as a weighting coefficient when weighted addition is performed. Storage means for calculating in advance the correspondence relationship between the predetermined direction variation value and the orthogonal direction variation value for a plurality of weighting coefficients, and storing the correspondence relationship;
Weighting coefficient selecting means for selecting one weighting coefficient from the plurality of weighting coefficients with reference to the storage means based on the predetermined direction variation value and the orthogonal direction variation value;
Preferably, the selected one weighting coefficient is a means for setting a weighting coefficient for weighted addition of the predetermined direction estimated value and the orthogonal direction estimated value.
[0035]
In this case, the ratio of the predetermined direction variation value or the orthogonal direction variation value to the addition value is ½ with respect to the plurality of weighting coefficients in the correspondence relationship stored in the storage unit. ⁿ (N is a natural number)
The weighted addition means is preferably means for performing the weighted addition by bit shift based on the weighting coefficient.
[0036]
In the image processing apparatus according to the present invention, it is determined whether or not the pixel position from which the signal value is estimated is in a monochromatic flat portion including only one of the first to third signal values. Flat part determining means;
When it is determined to be in the monochrome flat part, at least one signal value is estimated from the first to third signal values by linear interpolation based on the first to third signal values in the monochrome flat part And linear interpolation means for
When it is determined that the monochrome flat portion is not present, the determination unit determines whether it is a gray edge or a color edge, the estimation value calculation unit calculates the predetermined direction estimation value and the orthogonal direction estimation value, Calculation of the predetermined direction fluctuation value and the orthogonal direction fluctuation value by the fluctuation value calculation means, setting of the weighting coefficient by the weighting coefficient setting means, and estimation of a signal value based on the weighting coefficient by the weighting addition means. Is preferred.
[0037]
Furthermore, in the image processing apparatus according to the present invention, the first to third photoelectric conversion elements are each spectrally divided into any one of G, B, and R colors or any one of Y, G, and C colors. It is preferable that it has a sensitivity.
[0038]
The image processing method according to the present invention may be provided by being recorded on a computer-readable recording medium as a program for causing a computer to execute the image processing method.
[0039]
The image processing apparatus according to the present invention may be provided by being mounted on an imaging apparatus such as a digital camera or an image reading apparatus such as a scanner.
[0040]
【The invention's effect】
According to the present invention, it is determined whether the pixel position from which the signal value is estimated is at the gray edge or the color edge, and the variation amount of the signal value in the predetermined direction is determined by different calculation methods according to the determination result. The predetermined direction fluctuation value and the orthogonal direction fluctuation value representing the fluctuation amount of the signal value in the orthogonal direction are calculated, and the predetermined direction estimation value and the orthogonal direction estimation value are weighted and added based on the predetermined direction fluctuation value and the orthogonal direction fluctuation value. In this case, the weighting coefficient is set. Here, if the edge extends in a predetermined direction, the predetermined direction estimation value of a certain pixel position on the edge is a value that substantially matches the signal value of another pixel position on the edge, but the orthogonal direction estimation The value is different from the signal values at other pixel positions on the edge under the influence of the signal value at the pixel position existing in the direction orthogonal to the edge. When the edge is a gray edge, it is preferable to set the weighting coefficient so that the influence of the orthogonal direction estimation value is eliminated. On the other hand, when the edge is a color edge, it is preferable to eliminate the influence of the orthogonal direction estimation value, but if the signal value is estimated without the influence of the orthogonal direction estimation value completely, the fine color near the edge is lost. As a result, the image becomes unnatural.
[0041]
Here, when an edge extends in a predetermined direction, SA and SB calculated by the above formulas (3a) and (3b) and formulas (9a) and (9b) are set as a predetermined direction variation value and an orthogonal direction variation value, respectively. When these are compared, in the case of formulas (9a) and (9b), no division is performed, so the ratio of the orthogonal direction fluctuation value to the predetermined direction fluctuation value is calculated by the formulas (9a) and (9b). Becomes larger. Therefore, the weighting coefficient obtained by the predetermined direction fluctuation value and the orthogonal direction fluctuation value calculated by the equations (9a) and (9b) is very high compared with the weighting for the orthogonal direction estimation value compared to the weighting for the predetermined direction estimation value. Get smaller. On the other hand, the weighting coefficient obtained by the predetermined direction fluctuation value and the orthogonal direction fluctuation value calculated by the equations (3a) and (3b) is smaller in weighting for the orthogonal direction estimation value than the weighting for the predetermined direction estimation value. , (9a) and (9b) are not so small.
[0042]
Therefore, the signal value can be estimated with almost no false color in the case of a gray edge by setting the weighting coefficient so that the calculation method of the predetermined direction variation value and the orthogonal direction variation value differs depending on the determination result. In the case of a color edge, the signal value can be estimated so as to leave a fine color while reducing the false color.
[0043]
Although the weighting coefficient may be calculated as shown in the above equation (3c), in this equation (3c), since SA is divided by (SA + SB), an apparatus for performing this calculation This increases the size and the amount of calculation. Therefore, as in the third, eighth, and thirteenth aspects of the invention, when the ratio of the predetermined direction fluctuation value or the orthogonal direction fluctuation value to the sum of the predetermined direction fluctuation value and the orthogonal direction fluctuation value is set as the weighting coefficient, Based on the relationship between the predetermined direction variation value and the orthogonal direction variation value when the coefficient is constant, the correspondence relationship between the predetermined direction variation value and the orthogonal direction variation value for a plurality of weighting coefficients is calculated in advance, Based on the calculated predetermined direction fluctuation value and orthogonal direction fluctuation value stored in the storage means, one weighting coefficient is selected with reference to the storage means, and based on the one weighting coefficient, the predetermined direction is selected. It is preferable to perform weighted addition of the estimated value and the orthogonal direction estimated value.
[0044]
Here, when the weighting coefficient is calculated by the above equation (3c), when this is solved for the variation amount SB,
SB = ((1-Sj) / Sj) * SA (3c ')
It becomes. This indicates that when the weighting coefficient Sj is constant, the relationship between the variation value SA and the variation value SB is proportional. Accordingly, when a two-dimensional plane is set in which the variation value SA is associated with the predetermined direction variation value, the variation value SB is associated with the orthogonal direction variation value, the x-axis is the predetermined direction variation value, and the y-axis is the orthogonal direction variation value. On the plane, the plurality of weighting coefficients are represented as straight lines that pass through an origin having an inclination corresponding to the weighting coefficient (strictly, the origin is not included).
[0045]
Thus, for example, a two-dimensional plane is divided into a plurality of regions by a plurality of straight lines having inclinations corresponding to the weighting coefficients, and this is obtained as a correspondence relationship between the predetermined direction variation values and the orthogonal direction variation values for the plurality of weighting coefficients. This is stored in the storage means, and it is determined in which region on the two-dimensional plane the calculated predetermined direction variation value and orthogonal direction variation value are included, and in the region where these are determined to be located The corresponding weighting factor can be selected as one weighting factor.
[0046]
Therefore, according to the third, eighth, and thirteenth aspects of the invention, in order to obtain the weighting coefficient, the predetermined direction fluctuation value or the orthogonal direction fluctuation value is converted into the predetermined direction fluctuation value and the orthogonal direction fluctuation value as shown in the equation (3c). Therefore, it is not necessary to divide by the added value, thereby reducing the amount of calculation and simplifying the configuration of the circuit for performing the calculation. Therefore, it becomes easy to mount an apparatus for carrying out the present invention on a small-sized imaging apparatus such as a digital camera or an image reading apparatus such as a scanner.
[0047]
In addition, the weighting coefficient is ½ of the predetermined direction variation value or the orthogonal direction variation value with respect to the added value ⁿ When n is a natural number, the predetermined direction estimation value and the orthogonal direction estimation value are expressed in binary numbers as 1/2. ⁿ Multiplying by means that an operation equivalent to shifting the value to the right by n bits is performed. Therefore, by performing the weighted addition of the predetermined direction estimated value and the orthogonal direction estimated value by bit shift, the calculation can be performed easily and the processing time can be shortened.
[0048]
In addition, when the pixel position where the signal value is estimated is in a single color flat portion composed of only a single color signal value, the signal value can be estimated without generating false color even if simple linear interpolation calculation is performed. Therefore, it is determined whether or not the pixel position for estimating the signal value is in the monochromatic flat part, and if it is in the monochromatic flat part, the signal value is estimated by linear interpolation, thereby calculating the signal value. Time can be shortened.
[0049]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of the image processing apparatus according to the first embodiment of the present invention. As shown in FIG. 1, the image processing apparatus according to the first embodiment of the present invention performs interpolation processing on the signal values obtained in each photoelectric conversion element constituting the single CCD 1, and outputs signals at all pixel positions. A value is obtained, and an interpolation unit 2 is provided for performing interpolation processing on the image data S0 constituted by each signal value to obtain the interpolated image data S1. The interpolating means 2 includes a G interpolating means 3 for calculating the G signal QG by an interpolation operation as will be described later, an RB interpolating means 4 for calculating the R signal QR and the B signal QB, and the G signal QG, the R signal QR and B And a checkered square interpolation means 5 for calculating a signal value of a hole pixel position * shown in FIG. 3 described later from the signal QB. In the single-chip CCD 1 shown in FIG. 1, an analog signal can be obtained from the photoelectric conversion element constituting the single-chip CCD 1, but the image data S0 in this embodiment is a digital signal obtained by A / D converting the analog signal. In addition, it is assumed that the digital signal is converted so as to be 0.45 or logarithmic value of the amount of light input to the single CCD 1.
[0050]
Note that the image processing apparatus according to the present embodiment may be provided in an image capturing apparatus such as a digital camera or a reading apparatus such as a scanner that reads an image from a film, and reproduces an image signal obtained by these apparatuses. It may be provided in a playback device such as a monitor or a printer. Alternatively, the image processing apparatus may be used alone.
[0051]
FIG. 2 is a diagram showing a pixel array of the single CCD 1 used in this embodiment. The pixel array shown in FIG. 2A is a first line in which pixels corresponding to the R and G channels are alternately arranged in the y direction, and pixels corresponding to the G and B channels are alternately arranged in the y direction. 2 lines are alternately arranged in the x direction, and each line in the x direction has a square pixel array in which R, G channels and G, B channels are alternately arranged. As such a square pixel arrangement, the Bayer arrangement shown in FIG. 34 is known, and in this embodiment, the pixel arrangement shown in FIG. 2A is referred to as a Bayer arrangement.
[0052]
2B includes a line in which pixels corresponding to the R and B channels are alternately arranged in the y direction, and a line in which pixels corresponding to the G channel are arranged in the y direction. The arrangement interval of the pixels of the line is arranged so as to be shifted by about ½ in the y direction with respect to the pixel arrangement of the other lines, and is a checkered pixel arrangement. As such a checkered pixel arrangement, the honeycomb arrangement (FIG. 33) described in JP-A-10-136391 is known. In this embodiment, the pixel arrangement shown in FIG. It shall be called an array. In this honeycomb arrangement, in the direction inclined 45 degrees with respect to the x direction, the lines of the R and G channel pixels are alternately arranged, and the G and B channel pixels are alternately arranged in the direction inclined 45 degrees. The arranged lines are alternately arranged in a direction orthogonal to this direction. Note that the honeycomb arrangement has a relationship in which the Bayer arrangement and the pixel arrangement are rotated by 45 degrees as shown in FIG. In addition, the honeycomb array is formed by arranging pixels in a checkered pattern as described above, and can be expressed in a square shape as shown in FIG. 3 using hole pixel positions * having no signal value. It is. In the present embodiment, a description will be given assuming that processing is performed on the image data S0 obtained in the single-panel CCD 1 having the honeycomb arrangement shown in FIG.
[0053]
FIG. 4 is a diagram showing pixel positions of the single-chip CCD 1 in the honeycomb arrangement, and each pixel position is given a reference number. Here, the direction from the upper left to the lower right in FIG. 4, that is, the direction in which the pixels are aligned with R00, G11, R22, G33, R44... In FIG. The direction orthogonal to is the direction of arrow B. In this embodiment, the reference number given to the pixel position is also used as the reference number of the signal value.
[0054]
FIG. 5 is a schematic block diagram showing the configuration of the RB interpolation unit 4. As shown in FIG. 5, the RB interpolation unit 4 includes a determination unit 10 that determines whether the pixel position from which the signal value is estimated is a gray edge or a color edge, and all pixels in the direction of arrow A in FIG. First interpolation means 11 for calculating R and B signals at positions (generically referred to as signal Q1), and R signals and B signals at all pixel positions in the direction of arrow B (collectively referred to as signal Q2) Based on the second interpolation means 12 for calculating the value and the determination information H indicating the determination result in the determination means 10, values indicating the fluctuation amount of the signal value in the arrow A direction and the arrow B direction are calculated as the fluctuation values SA and SB. Based on the fluctuation value calculation means 13 and the fluctuation values SA and SB calculated by the fluctuation value calculation means 13, the signals Q1 and Q2 obtained by the first interpolation means 11 and the second interpolation means 12 are weighted and interpolated. Weighting coefficient calculating means 14 for calculating the weighting coefficient Sj and the weighting coefficient Sj calculated by the weighting coefficient calculating means 14 are used for weighted addition of the signals Q1 and Q2 obtained by the first interpolation means 11 and the second interpolation means. Weighting addition means 15 for obtaining the R signal QR and the B signal QB. Note that the weighting coefficient calculating means 14 in FIG. 5 is considered as the weighting coefficient setting means in the present invention.
[0055]
Next, calculation of the G signal QG in the G interpolation unit 3 and the R signal QR and the B signal QB in the RB interpolation unit 4 of the present embodiment will be described.
[0056]
(1) First, calculation processing of the G signal at the R and B pixel positions will be described. The calculation of the G signal is performed by the G interpolation means 3. The G signal at the R and B pixel positions can be calculated by performing a spline interpolation operation on the G signal obtained at the G pixel positions around the pixel position. FIG. 6 is a diagram illustrating an example of a two-dimensional Cubic spline interpolation filter that performs a spline interpolation operation. The two-dimensional Cubic spline interpolation filter shown in FIG. 6 performs an interpolation operation on the G signal at the G pixel position of 16 pixels near the pixel position where the G signal is calculated. Accordingly, the G (R44) signal at the R44 pixel position surrounded by the solid line in FIG. 7 is the 16 G pixel positions (G11, G13, G15, G17, G31, G33, G35, G37, G51, G53, G55, G57, G71, G73, G75, G77) are calculated by subjecting the G signal to a filtering process using a two-dimensional Cubic spline interpolation filter shown in FIG. On the other hand, the G (B46) signal at the B46 pixel position is a G pixel position (G13, G15, G17, G19, G33, G35, G37, G39, G53, G55, G57, G59, G73) surrounded by a broken line. , G75, G77, G79) is similarly calculated by performing a filtering process. Thereby, it is possible to perform interpolation without damaging the frequency component of the G signal before interpolation, and as a result, it is possible to obtain a G signal that retains the original frequency information at all pixel positions.
[0057]
Here, as a method of the interpolation calculation, any method can be applied as long as it is an interpolation calculation in the vertical and horizontal two-dimensional directions in FIG. Note that the G signal at the R and G pixel positions may be calculated by performing a simple linear interpolation operation on the G signal obtained at the G pixel position around the pixel position where the G signal is calculated. For example, the G (R44) signal at the R44 pixel position and the G (B46) signal at the B46 pixel position surrounded by the solid line in FIG. 7 are expressed by the following equations (10) using the G signals at the four surrounding G pixel positions: ) And (11).
[0058]
G (R44) = (G33 + G35 + G53 + G55) / 4 (10)
G (B46) = (G35 + G37 + G55 + G57) / 4 (11)
(2) Next, calculation of the R signal QR and the B signal QB in the RB interpolation unit 4 will be described. First, calculation of the signals Q1 and Q2 in the first and second interpolation means 11 and 12 will be described.
[0059]
1. First, an R signal calculation process in the arrow A direction at a G pixel position in a line in which R and G pixels are arranged in the arrow A direction (hereinafter referred to as a first line) will be described. The calculation of the signal in the direction of arrow A is performed by the first interpolation means 11. In the following description, a signal calculated in the direction of arrow A is denoted by (A), and a signal calculated in the direction of arrow B is denoted by (B). Also, a line in which R and G pixels are arranged in the direction of arrow A is a first line, a line in which B and G pixels are arranged in the direction of arrow A is a second line, and a line in which R and G pixels are arranged in the direction of arrow B is a third line. The fourth line is a line in which B and G pixels are arranged in the arrow B direction.
[0060]
The calculation process of the R signal at the G pixel position in the direction of arrow A in the first line is performed by performing a one-dimensional interpolation operation on the R signal on the first line including the G pixel position. For example, the R (G33) A signal at the G33 pixel position surrounded by the solid line in FIG. 8 is the four R pixels (R00, R22, R44) around the G33 pixel position on the first line where the G33 pixel position exists. , R66) is calculated by performing one-dimensional interpolation calculation such as one-dimensional Cubic spline interpolation calculation shown in the following equation (12) on the R signal obtained in R66).
[0061]
R (G33) A = (-3 * ROO + 19 * R22 + 19 * R44-3 * R66) / 32 (12)
It may be calculated by the following equation (13) using R22 and R44 signals at two R pixel positions adjacent to each other in the direction of arrow A in the first line.
[0062]
R (G33) A = (R22 + R44) / 2 (13)
Any interpolation calculation method can be used as long as it is a one-dimensional interpolation calculation in the direction of arrow A on the first line. Thereby, the R signal at the G pixel position on the first line can be calculated.
[0063]
Second, the calculation process of the R signal in the arrow B direction at the G pixel position in the first line in which the R and G pixels are arranged in the arrow A direction will be described. The calculation of the signal in the direction of arrow B is performed in the second interpolation means 12 as follows. First, a pixel position adjacent to the G pixel position where the R signal is calculated in the direction of arrow A is obtained. For example, assuming that the R (G33) B signal at the G33 pixel position surrounded by the solid line shown in FIG. 9 is calculated here, the pixel positions adjacent to the G33 pixel position are the R22 pixel position and R44 pixel position surrounded by the broken line. It becomes. Then, the average value of the difference between the R22 signal and the G (R22) signal at the R22 pixel position and the difference between the R44 signal and the G (R44) signal at the R44 pixel position is added to the G33 signal at the G33 pixel position, and the addition The result is an R (G33) B signal at the G33 pixel position.
[0064]
Here, when only the R22 signal is used, the R (G33) B signal at the G33 pixel position is calculated by the following equation (14).
[0065]
R (G33) B = G33 + (R22-G (R22)) (14)
Expression (14) is determined on the assumption that the difference between the R signal and the G signal in the local region of the image is equal. For example, the difference between the R signal and the G signal at the R22 pixel position and the G33 pixel position is equal.
R (G33) B-G33 = R22-G (R22) (15)
Equation (15) is solved for the R (G33) B signal to obtain Equation (14). Note that in Equation (14), the G (R22) signal is not calculated by the processing of (1) above, but in the direction of arrow B with respect to the G signal on the third line where the R22 pixel position exists. This is calculated by performing a one-dimensional interpolation operation such as a linear interpolation operation shown in the following equation (16) or a one-dimensional Cubic spline interpolation operation shown in equation (12). Therefore, the reference sign B is attached to the G (R22) signal.
[0066]
G (R22) B = (G31 + G13) / 2 (16)
That is, in the equation (14), the signal value is calculated on the assumption that the difference between the R signal and the G signal in the local region of the image is equal, and the signal value is calculated using the G33 in the direction of the arrow B. The fourth line where the pixel position exists and the third line where the R22 pixel position exists. This means that the signal value is calculated based on the correlation between the pixel values on the third and fourth lines, and the change in the signal value in the direction of arrow B is reflected for the unknown signal value. It is necessary to calculate the signal value. When only the R22 signal is used in the processing of 2, the R (G33) B signal at the G33 pixel position is calculated based on the relationship shown in the above equation (15), but the G (R22) signal is unknown. Therefore, it is necessary to estimate this. Here, since equation (15) represents the correlation between the third and fourth lines, in order to estimate the G (R22) signal, the change in the signal value in the direction of arrow B on the third line It is necessary to reflect. In this case, it is conceivable to use the G signal calculated by the method of (1) above, but this G signal is calculated using the signal value of another line as shown in equations (10) and (11). Therefore, it does not reflect the change in the signal value in the direction of arrow B on the third line. Therefore, in order to reflect the change in the signal value in the direction of arrow B, a one-dimensional interpolation operation is performed in the direction of arrow B on the third line as shown in equation (16) to obtain the G (R22) B signal. It is what is calculated.
[0067]
On the other hand, when only the R44 signal is used, the R (G44) B signal at the G33 pixel position is calculated by the following equation (17) based on the relationship of the above equation (15).
[0068]
R (G33) B = G33 + (R44-G (R44) B) (17)
Note that the G (R44) B signal is calculated in the same manner as Equation (16). Here, in this embodiment, the R (G33) B signal may be obtained by either of the equations (14) or (17), but the signal value of the pixel position adjacent to only one side of the G pixel position is used. If so, the phase of the image approaches that direction. In order to prevent this, as shown in the following equation (18), the R signal and the G signal at the pixel positions (R22, R44 pixel positions) adjacent to the G pixel position (for example, the G33 pixel position) where the R signal is calculated. Is obtained by adding the average value of the differences to the G signal at the G pixel position where the R signal is calculated as the R signal at the G pixel position.
[0069]
R (G33) B = G33 + ((R22-G (R22) B) + (R44-G (R44) B)) / 2 (18)
Thereby, the R (G33) B signal at the G pixel position in the arrow B direction can be calculated.
[0070]
3. Next, the calculation process of the R signal in the arrow A direction at the G pixel position in the second line in which the B and G pixels are arranged in the arrow A direction will be described. The calculation of the signal in the direction of arrow A is performed by the first interpolation means 11. First, a pixel position adjacent to the G pixel position for calculating the R signal in the arrow B direction is obtained. For example, assuming that the R (G35) A signal at the G35 pixel position surrounded by the solid line shown in FIG. 10 is calculated here, the pixel positions adjacent to the G35 pixel position are the R26 pixel position and R44 pixel position surrounded by the broken line. It becomes. Similar to the above processing 2, the difference between the R26 signal and the G (R26) A signal at the R26 pixel position and the R44 signal and the G (R44) A signal at the R44 pixel position as shown in the following equation (19). Is added to the G35 signal at the G35 pixel position, and the addition result is used as the R (G35) A signal at the G35 pixel position.
[0071]
R (G35) A = G35 + ((R26-G (R26) A) + (R44-G (R44) A)) / 2 (19)
The G (R26) A signal and the G (R44) A signal are calculated by performing a one-dimensional interpolation operation in the direction of arrow A on the G signal on the first line where the R26 and R44 pixel positions exist. It is a thing. Thereby, the R (G35) A signal at the G pixel position in the direction of arrow A can be calculated.
[0072]
4 Next, the calculation process of the R signal in the arrow B direction at the G pixel position in the second line in which the B and G pixels are arranged in the arrow A direction will be described. The calculation of the signal in the direction of the arrow B is performed by performing one-dimensional interpolation operation on the R signal on the third line including the G pixel position in the second interpolation unit 12. For example, the R (G35) B signal at the G35 pixel position surrounded by the solid line in FIG. 11 is the four R pixels (R08, R26, R44) around the G35 pixel position on the third line where the G35 pixel position exists. , R62) is calculated by performing a one-dimensional interpolation operation on the R signal obtained in R62) in the same manner as in the first process. An example of calculation by one-dimensional Cubic spline interpolation calculation is shown in the following equation (20).
[0073]
R (G35) B = (-3 * RO8 + 19 * R26 + 19 * R44-3 * R62) / 32 (20)
Next, the calculation process of the R signal in the arrow A direction at the B pixel position in the second line in which the B and G pixels are arranged in the arrow A direction will be described. The calculation of the signal in the direction of arrow A is performed by the first interpolation means 11. First, a pixel position adjacent to the B pixel position for calculating the R signal in the arrow B direction is obtained. For example, assuming that the R (B24) A signal at the B24 pixel position surrounded by the solid line shown in FIG. 12 is calculated here, the pixel positions adjacent to the B24 pixel position are the G15 pixel position and the G33 pixel position surrounded by the broken line. It becomes. Similarly to the above processing 2, the difference between the R (G15) A signal and the G15 signal at the G15 pixel position and the R (G33) A signal and the G33 signal at the G33 pixel position as shown in the following equation (21). Is added to the G (B24) A signal at the B24 pixel position, and the addition result is used as the R (B24) A signal at the B24 pixel position.
[0074]
R (B24) A = G (B24) A + ((R (G33) A-G33) + (R (G15) A-G15)) / 2 (21)
The R (G15) A signal and the R (G33) A signal are calculated by performing a one-dimensional interpolation operation in the direction of arrow A on the R signal on the first line where the G15 and G33 pixel positions exist. This is the R signal itself calculated in one process. The G (B24) A signal is calculated by performing a one-dimensional interpolation operation in the arrow A direction on the G signal on the second line where the B24 pixel position exists. As a result, the R (B24) A signal at the B pixel position in the arrow A direction can be calculated.
[0075]
6 Next, the calculation process of the R signal in the arrow B direction at the B pixel position in the second line in which the B and G pixels are arranged in the arrow A direction will be described. The calculation of the signal in the direction of arrow B is performed by the second interpolation means 12. First, a pixel position adjacent to the B pixel position where the R signal is calculated in the direction of arrow A is obtained. For example, assuming that the R (B24) B signal at the B24 pixel position surrounded by the solid line shown in FIG. 13 is calculated here, the pixel positions adjacent to the B24 pixel position are the G13 pixel position and the G35 pixel position surrounded by the broken line. It becomes. Similar to the above processing 2, the difference between the R (G13) B signal and the G13 signal at the G13 pixel position and the R (G35) B signal and the G35 signal at the G35 pixel position as shown in the following equation (22). Is added to the G (B24) B signal at the B24 pixel position, and the addition result is used as the R (B24) B signal at the B24 pixel position.
[0076]
R (B24) B = G (B24) B + ((R (G35) B-G35) + (R (G13) B-G13)) / 2 (22)
The R (G13) B signal and the R (G35) B signal are calculated by performing a one-dimensional interpolation operation in the arrow B direction on the R signal on the third line where the G13 and G35 pixel positions exist. This is the R signal itself calculated in the process of 4. The G (B24) B signal is calculated by performing a one-dimensional interpolation operation in the arrow B direction on the G signal on the fourth line where the B24 pixel position exists. Thereby, the R (B24) B signal at the B pixel position in the arrow B direction can be calculated.
[0077]
As described above, the R signal for the arrow A direction at the G pixel position in the first line in which the R and G pixels are arranged in the arrow A direction is calculated by the process 1, and the R, G in the arrow A direction is calculated by the process 2. An R signal in the direction of arrow B at the G pixel position in the first line where the pixels are arranged is calculated.
[0078]
The R signal for the arrow A direction at the G pixel position in the second line in which the B and G pixels are arranged in the arrow A direction is calculated by the process 3, and the B, G in the arrow A direction is calculated by the process 4. An R signal in the direction of arrow B at the G pixel position in the second line where the pixels are arranged is calculated.
[0079]
Further, the R signal for the arrow A direction at the B pixel position in the second line in which the B and G pixels are arranged in the arrow A direction is calculated by the process 5, and the B, G in the arrow A direction is calculated by the process 6. An R signal in the direction of arrow B at the B pixel position in the second line in which the pixels are arranged is calculated.
[0080]
The calculation of the R signal in both directions of the arrows A and B at each pixel position has been described above, but the B signal can also be calculated in the same manner as the R signal as described in the processes 1 to 6 above.
[0081]
The determination means 10 determines whether the pixel position whose signal value is estimated is on the gray edge or the color edge as follows. FIG. 14 is a diagram illustrating an example of a color edge composed of red (R) and green (G), and FIG. 15 is a diagram illustrating an example of a gray edge composed of white and black. 14 and 15, the pixel position where the signal value is 0 is shown in white, and the other pixels will be described as having a signal value of 255. Here, the determination at the R44 pixel position will be described. First, the determination means 10 obtains edge determination values Dat1 to Dat4 from the following equations (23) to (26).
[0082]
Dat1 = | R44-G33 | + | R44-G55 | + | G35-B24 | + | G35-B46 | (23)
Dat2 = | R44-G33 | + | R44-G55 | + | G53-B42 | + | G53-B64 | (24)
Dat3 = | R44-G35 | + | R44-G53 | + | G33-B24 | + | G33-B42 | (25)
Dat4 = | R44-G35 | + | R44-G53 | + | G55-B46 | + | G55-B64 | (26)
Here, the equation (23) calculates a determination value Dat1 for determining the color of the upper right edge of the R44 pixel position in the arrow A direction, and the equation (24) calculates the R44 pixel in the arrow A direction. The determination value Dat2 for determining the color of the edge at the lower left part of the position is calculated. Expression (25) calculates a determination value Dat3 for determining the color of the edge of the upper left portion of the R44 pixel position in the arrow B direction. Expression (26) is an R44 pixel position in the arrow B direction. The determination value Dat4 for determining the color of the edge of the lower right part of is calculated.
[0083]
When the pixel position from which the signal value is estimated is at the gray edge, any one of the determination values Dat1 to Dat4 is necessarily small. For example, in the case of the gray edge shown in FIG. 15, in the determination values Dat1 to Dat4 calculated in the above formulas (23) to (26), the determination values Dat1 and Dat2 are 0. On the other hand, when the pixel position from which the signal value is estimated is at a color edge as shown in FIG. 14, any of the determination values Dat1 to Dat4 increases. Therefore, the minimum value Dmin of the determination values Dat1 to Dat4 is compared with a predetermined threshold value T. When the minimum value Dmin is equal to or less than the threshold value T, that is,
Dmin = min (Dat1, Dat2, Dat3, Dat4) ≦ T (27)
The pixel position is determined to be a gray edge, otherwise, that is,
Dmin = min (Dat1, Dat2, Dat3, Dat4)> T (28)
If it is, it is determined that it is a color edge. Here, the threshold value T is set to a value of about 128 when the signal value is represented by 8 bits. Then, the determination unit 10 inputs the determination result as determination information H to the fluctuation value calculation unit 13.
[0084]
The fluctuation value calculation means 13 is a weighting coefficient for weighting and adding the signal values in the directions of arrows A and B calculated by the processing of 1 and 2, 3 and 4 and 5 and 6 in the weighting coefficient calculation means 14 described later. The variation values SA and SB necessary for the calculation are calculated. Here, the fluctuation values SA and SB are values representing the fluctuation amount of the signal value at the pixel position where the signal value is estimated. First, calculation of the fluctuation values SA and SB for calculating the weighting coefficient of the R signal in the directions of arrows A and B calculated by the processes 1 and 2 will be described.
[0085]
First, when it is determined by the determination information H that the pixel position for calculating the signal value is at the color edge, the fluctuation value SA in the arrow A direction and the fluctuation value SB in the arrow B direction are expressed by the following equations (29) and ( 30). Here, similarly to the processes 1 and 2, it is assumed that the fluctuation values SA and SB of the signal value at the G33 pixel position shown in FIG. 8 are calculated.
[0086]
SA = | R (G33) A-G33 | / (R (G33) A + G33) (29)
SB = | B (G33) B-G33 | / (B (G33) B + G33) (30)
Here, R (G33) A in Equation (29) is the signal value calculated by Equation (12) in the processing of 1, and B (G33) B in Equation (30) is the G33 pixel position periphery on the fourth line. These are signal values calculated by performing a one-dimensional interpolation operation on the B signals obtained at the four B pixel positions (B06, B24, B42, B60). The variation value SA calculated in this way represents a change in the signal value in the arrow A direction at the G33 pixel position, and the variation value SB represents a change in the signal value in the arrow B direction at the G33 pixel position.
[0087]
On the other hand, when it is determined by the determination information H that the pixel position for calculating the signal value is at the gray edge, the variation value SA and the variation value SB are expressed by the following equation (31) with the denominators of equations (29) and (30) omitted. ) And (32).
[0088]
SA = | R (G33) A-G33 | (31)
SB = | B (G33) B-G33 | (32)
Next, calculation of the fluctuation value for calculating the weighting coefficient of the R signal in the directions of arrows A and B calculated by the processes of 3 and 4 will be described.
[0089]
First, when it is determined by the determination information H that the pixel position for calculating the signal value is at the color edge, the variation value SA in the arrow A direction and the variation value SB in the arrow B direction are expressed by the following equations (33), ( 34). Here, similarly to the processes of 3 and 4, the fluctuation value of the signal value at the G35 pixel position shown in FIG. 10 is calculated.
[0090]
SA = | B (G35) A-G35 | / (B (G35) A + G35) (33)
SB = | R (G35) B-G35 | / (R (G35) B + G35) (34)
Here, B (G35) A in Expression (33) is 1 for the B signal obtained at the four B pixel positions (B02, B24, B46, B68) around the G35 pixel position on the second line. The signal value calculated by performing the dimensional interpolation calculation, R (G35) B in the equation (34), is the signal value calculated by the equation (20) in the above-described processing of 4. The variation value SA calculated in this way represents a change in the signal value in the direction of arrow A at the G35 pixel position, and the variation value SB represents a change in the signal value in the direction of arrow B at the G35 pixel position.
[0091]
On the other hand, when it is determined by the determination information H that the pixel position for calculating the signal value is at the gray edge, the variation value SA and the variation value SB are expressed by the following equation (35) in which the denominators of the equations (33) and (34) are omitted. ) And (36).
[0092]
SA = | B (G35) A-G35 | (35)
SB = | R (G35) B-G35 | (36)
Next, calculation of the fluctuation value for calculating the weighting coefficient of the R signal in the directions of arrows A and B calculated by the processes of 5 and 6 will be described.
[0093]
First, when it is determined by the determination information H that the pixel position for calculating the signal value is at the color edge, the variation value SA in the arrow A direction and the variation value SB in the arrow B direction are expressed by the following equations (37), ( 38). Here, similarly to the processing of 5 and 6, the fluctuation value of the signal value at the B24 pixel position shown in FIGS. 12 and 13 is calculated.
[0094]
SA = | B24-G (B24) A | / (B24 + G (B24) A) (37)
SB = | B24-G (B24) B | / (B24 + G (B24) B) (38)
Here, G (B24) A in Expression (37) is obtained by performing a one-dimensional interpolation operation in the direction of arrow A on the G signal on the second line where the B24 pixel position exists in the process of 5 above. The calculated signal value, G (B24) B in equation (38), is subjected to a one-dimensional interpolation operation in the direction of arrow B on the G signal on the fourth line where the B24 pixel position exists in the process of 6 above. This is the signal value calculated. The variation value SA calculated in this way represents a change in the signal value in the arrow A direction at the B24 pixel position, and the variation value SB represents a change in the signal value in the arrow B direction at the B24 pixel position.
[0095]
On the other hand, when it is determined by the determination information H that the pixel position for calculating the signal value is at the gray edge, the variation value SA and the variation value SB are expressed by the following equation (39) and the denominator of (38) are omitted (39 ) And (40).
[0096]
SA = | B24-G (B24) A | (39)
SB = | B24-G (B24) B | (40)
The fluctuation values SA and SB calculated as described above are input to the weighting coefficient calculating means 14, and the weighting coefficient Sj is calculated by the following equation (41). The weighting coefficient Sj is input to the weighting addition means 15.
[0097]
Sj = SA / (SA + SB) (if SA + SB = 0 then Sj = 0.5) (41)
In Expression (41), SA + SB = 0 is satisfied when SA = SB = 0 because SA and SB ≧ 0.
[0098]
In the weighted addition means 15, signal values are calculated by the following formulas (42) to (44) for the cases of 1 and 2, 3 and 4, and 5 and 6. Expressions (42) to (44) represent the R (G33) signal at the G33 pixel position in the first line in which the R and G pixels are arranged in the arrow A direction, and the B and G pixels in the arrow A direction. The R (G35) signal at the G35 pixel position in the second line and the R (B24) signal at the B24 pixel position in the second line in which the B and G pixels are arranged in the direction of arrow A are calculated.
[0099]
R (G33) = (1-Sj) * R (G33) A + Sj * R (G33) B (42)
R (G35) = (1-Sj) * R (G35) A + Sj * R (G35) B (43)
R (B24) = (1-Sj) * R (B24) A + Sj * R (B24) B (44)
A general formula using signals Q1 and Q2 is shown in formula (45).
[0100]
QR, QB = (1-Sj) * Q1 + Sj * Q2 (45)
As a result, the smaller the signal value changes, the greater the weighting is made, and the R and B signals QR and QB are calculated.
[0101]
Here, differences in the calculation methods of the variation values SA and SB in the case of the color edge and the gray edge will be described. FIG. 16 is a diagram for explaining the difference between the calculation methods of the fluctuation values SA and SB. In the pixel array shown in FIG. 16, it is assumed that gray edges of white (signal value is 255) and black (signal value is 0 or a value close to 0) are represented. Here, calculation of the B (R44) signal in the R44 pixel will be described. First, in the direction of arrow A, a B (R44) A signal at the R44 pixel position is obtained by the following equation (46).
[0102]
B (R44) A = G (R44) A + ((B (G35) A-G35) + (B (G53) A-G53)) / 2
= 0 + ((2-0) + (255-255)) / 2 = 1 (46)
However, G (R44) A = 0, B (G35) A = 2, and B (G53) A = 255, and these values are calculated in one dimension as shown in equations (12) and (20). Cubic spline interpolation operation was used.
[0103]
Further, in the direction of arrow B, the B (R44) B signal at the R44 pixel position is obtained by the following equation (47).
[0104]
B (R44B) = G (R44) B + ((B (G33) B-G33) + (B (G55) B-G55)) / 2
= 103 + ((105-0) + (105-0)) / 2 = 208 (47)
However, G (R44) B≈103, B (G33) B≈105, and B (G55) B≈105, and these values are calculated in one dimension as shown in equations (12) and (20). Cubic spline interpolation operation was used.
[0105]
Then, the fluctuation values SA and SB are obtained by the following equation (48), and the weighting coefficient Sj for weighting and adding the B (R44) A signal and the B (R44) B signal is calculated by the equation (49).
[0106]
SA = | R44-G (R44) A | / (R44 + G (R44) A) = | 1-0 | / (1 + 0) = 1
SB = | R44-G (R44) B | / (R44 + G (R44) B) = | 1-103 | / (1 + 103) ≒ 0.98 (48)
Sj = SA / (SA + SB) = 1 / (1 + 0.98) ≒ 0.5 (49)
When the weighting coefficient Sj is obtained in this way, a B (R44) signal is calculated by the following equation (50).
[0107]
B (R44) = (1-Sj) * B (R44) A + Sj * B (R44) B = 0.5 * 1 + 0.5 * 208 ≒ 105 (50)
Here, since the R44 pixel position is on the black edge, the value of the B (R44) signal is preferably close to zero. However, if the weighting coefficient Sj is calculated as shown in the above equations (48) and (49) and the value of the B (R44) signal is obtained by the equation (50), the value becomes 105. Will occur. This is because the value of the weighting coefficient Sj becomes closer to 0 as the signal value varies less in the direction of the arrow A, but depending on the signal value at the pixel position around the pixel position where the signal value is calculated, as shown in Expression (50). This is because the weighting coefficient Sj may not be a value close to zero.
[0108]
On the other hand, in this embodiment, when the pixel position for calculating the signal value is at the gray edge, the denominator is omitted in the equation (48) to calculate the variation values SA and SB. When the fluctuation values SA and SB are calculated in this way,
SA = | R44-G (R44) A | = | 1-0 | = 1
SB = | R44-G (R44) B | = | 1-103 | = 102 (51)
And the weighting coefficient Sj is
Sj = SA / (SA + SB) = 1 / (1 + 102) ≒ 0.01 (52)
It becomes. When the B (R44) signal is calculated by the weighting coefficient Sj calculated in this way,
B (R44) = (1-Sj) * B (R44) A + Sj * B (R44) B = 0.99 * 1 + 0.01 * 208 ≒ 2.99 (53)
It becomes. Thereby, the B (R44) signal becomes a value close to 0, and as a result, false colors can be reduced.
[0109]
On the other hand, if it is determined that the edge is a color edge, the false color can be reduced by calculating the variation values SA and SB according to the equation (51). However, since the fine color near the edge is lost, the image is It becomes unnatural. Therefore, in this embodiment, the calculation method of the variation values SA and SB is changed according to whether the pixel position for calculating the signal value is a gray edge or a color edge.
[0110]
In the Bayer array single-chip CCD 1 shown in FIG. 2A, when the pixel array is rotated 45 degrees, the pixel array is the same except for the honeycomb array and the hole pixel position * shown in FIG. It will be a thing. Therefore, in the above description, the arrow A direction is the vertical direction of the paper surface in FIG. 2, and the arrow B direction is the horizontal direction of the paper surface in FIG. An RGB signal can be calculated.
[0111]
When all the RGB signal values QR, QG, QB are obtained at all the pixel positions in this way, the checkered square interpolation means 5 calculates the RGB signal values at the hole pixel position * in FIG. 3 by interpolation calculation. Thus, interpolated image data S1 in which pixels are arranged in a square shape is obtained. This interpolation calculation includes an interpolation filter using signal values at four pixel positions around the vacant pixel positions * as shown in FIG. 17, and a two-dimensional Cubic spline interpolation calculation for 4 × 4 pixels as shown in FIG. The interpolation coefficient can be obtained by performing an interpolation operation using an interpolation filter having an interpolation coefficient arrangement inclined by 45 degrees. Note that the interpolation calculation for calculating the signal value at the hole pixel position * is a checkered square interpolation calculation. Further, the interpolation calculation is not limited to these, and the RGB signal at each pixel position obtained as described above is converted into a YCC luminance / color difference space, and an interpolation calculation using a different interpolation filter is performed for each YCC. Any method can be adopted as long as it is an interpolation calculation for calculating the signal value at the hole pixel position *.
[0112]
Next, the operation of the first embodiment will be described. FIG. 19 is a flowchart showing the operation of the first embodiment. First, a subject is photographed to obtain image data S0 in the single CCD 1 (step S1). Next, the G interpolation means 3 of the interpolation means 2 calculates the G signal QG at the R or B pixel position in the predetermined direction by the process (1) (step S2). Then, the interpolation means 2 sets the signal to be calculated to the R signal (step S3), and the RB interpolation means 4 calculates the R signal QR (step S4).
[0113]
FIG. 20 is a flowchart showing the process of step S4. First, a signal value is calculated by the processes 1 and 2 (step S11), a signal value is calculated by steps S3 and 4 (step S12), and a signal value is calculated by processes 5 and 6 (step S13). On the other hand, in the determination means 10, it is determined whether or not the pixel position for calculating the signal value is at the gray edge (step S 14), and determination information H representing the determination result is input to the fluctuation value calculation means 13. In the fluctuation value calculation means 13, when the pixel position for calculating the signal value is at the gray edge, the gray edge fluctuation values SA and SB are calculated as shown in the above formulas (31) and (32). (Step S15). On the other hand, if it is a color edge, the variation values SA and SB for the color edge are calculated as shown in the above equations (29) and (30) (step S16). Then, the fluctuation values SA and SB are input to the weighting coefficient calculating means 14, where the weighting coefficient Sj is calculated (step S17), and the signal values calculated by the processing of steps S11, S12, and S13 are weighted and added. The R signal QR is calculated (step S18).
[0114]
Returning to FIG. 19, it is determined whether or not all R and B signals have been calculated (step S5). If step S5 is negative, the signal to be calculated in step S6 is set to the B signal, and step S4. Returning to step S4, the processes of steps S4 and S5 are repeated to calculate the B signal QB.
[0115]
If the determination in step S5 is affirmative, it is assumed that the RGB signals QR, QG, and QB have been calculated at all pixel positions other than the hole pixel positions. A checkered square interpolation calculation is performed by the interpolation filter shown in FIG. 18 to calculate a signal value at the hole pixel position (step S7), and the process is terminated.
[0116]
As a result, RGB signals can be obtained at all pixel positions including the vacant pixel positions of the single CCD 1 having the honeycomb arrangement shown in FIG. 2B, and the interpolated image data S1 having RGB signals at all pixel positions is obtained. Obtainable.
[0117]
In step S3 of the flowchart shown in FIG. 19, the signal to be calculated is set to the R signal, but may be set to the B signal first. In this case, if step S5 is denied, the signal to be calculated is switched from the B signal to the R signal in step S6.
[0118]
As described above, according to the first embodiment, it is determined whether the pixel position for calculating the signal value is at the gray edge or the color edge, and the variation value for calculating the weighting coefficient Sj according to the determination result. Since the SA and SB calculation methods are changed, the false color can be reduced even when the pixel position for calculating the signal value is at the gray edge. In addition, when the pixel position for calculating the signal value is at the color edge, it is possible to obtain an image having a natural feeling while leaving a fine color near the edge.
[0119]
Next, a second embodiment of the present invention will be described.
[0120]
FIG. 21 is a schematic block diagram showing the configuration of the RB interpolation means of the image processing apparatus according to the second embodiment of the present invention. In FIG. 21, the same components as those in FIG. 5 are denoted by the same reference numerals, and detailed description thereof is omitted. In the second embodiment, instead of the determination means 10, in addition to the determination of whether the pixel position for estimating the signal value is at the gray edge or the color edge, it is in the single-color flat portion made up of single-color signal values. A linear interpolation unit that calculates an R signal QR and a B signal QB at the pixel position by a linear interpolation operation when it is determined that the pixel position is in a monochromatic flat portion. The point provided with 22 is different from the first embodiment.
[0121]
The determination process in the determination unit 21 regarding the gray edge or the color edge is the same as the determination process in the determination unit 10 in the first embodiment. Determination information H representing the determination result is input to the fluctuation value calculating means 13, the fluctuation values SA and SB are calculated in the same manner as in the first embodiment, and the weighting coefficient calculating means 14 further calculates the weighting coefficient Sj. In this case, as in the first embodiment, the first interpolation means 11 and the second interpolation means 12 calculate the signals Q1 and Q2 in the arrow A direction and the arrow B direction at each pixel position, and the signal values are weighted and added. The R signal QR and the B signal QB are calculated by being input to the means 15 and weighted by the weighting coefficient Sj.
[0122]
Hereinafter, determination of whether or not the determination unit 21 is in the monochrome flat portion will be described. FIG. 22 is a diagram showing signal values of the green (G) monochromatic flat portion, FIG. 23 is a diagram showing signal values of the blue (B) monochromatic flat portion, and FIG. 24 is a signal value of the red (R) monochromatic flat portion. FIG. 22, only the G pixel position has a signal value of 255, only the B pixel position has a signal value of 255 in FIG. 23, and only the R pixel position has a signal value of 255 in FIG. It has become. The determination unit 21 obtains the average value of the pixel position where the signal value is calculated and the R signal around the pixel position as Dat11, the average value of the G signal as Dat12, and the average value of the B signal as Dat13. For example, in the pixel array shown in FIGS.
Dat11 = (R22 + R26 + R44 + R62 + R66) / 5
Dat12 = (G33 + G35 + G53 + G55) / 4 (54)
Dat13 = (B24 + B42 + B46 + B64) / 4
Based on this, the average values Dat11 to Dat13 are calculated.
[0123]
For the B42 pixel position,
Dat11 = (R22 + R40 + R44 + R62) / 4
Dat12 = (G31 + G33 + G51 + G53) / 4 (55)
Dat13 = (B20 + B24 + B42 + B60 + B64) / 5
Based on this, the average values Dat11 to Dat13 are calculated.
[0124]
Furthermore, for the G33 pixel position,
Dat11 = (R04 + R22 + R44 + R62) / 4
Dat12 = (G11 + G13 + G15 + G31 + G33 + G35 + G51 + G53 + G55) / 9 (56)
Dat13 = (B02 + B24 + B42 + B64) / 4
Based on this, the average values Dat11 to Dat13 are calculated.
[0125]
Then, absolute values | Dat11-Dat12 |, | Dat12-Dat13 |, | Dat13-Dat11 | of the difference values of the average values Dat11 to Dat13 are calculated, and whether or not the sum of these is greater than a predetermined threshold T1. If it is larger, that is, if the condition of the following equation (57) is satisfied, the pixel position is regarded as being in a monochromatic flat portion.
[0126]
| Dat11-Dat12 | + | Dat12-Dat13 | + | Dat13-Dat11 |> T1 (57)
That is, in the case of the G single-color flat portion shown in FIG. 22, only the G pixel position has a signal value, so in the formulas (54) to (56), only the average value Dat12 has a value, and Dat11 and Dat13 are Has no value. Therefore, the values of | Dat11-Dat12 | and | Dat12-Dat13 | are very large. 23, since only the B pixel position has a signal value in the B monochrome flat portion shown in FIG. 23, only the average value Dat13 has a value in Expressions (54) to (56), and Dat11 and Dat12 are Has no value. Therefore, the values of | Dat12-Dat13 | and | Dat13-Dat11 | are very large. Further, in the case of the R monochromatic flat portion shown in FIG. 24, only the R pixel position has a signal value. Therefore, in the equations (54) to (56), only the average value Dat11 has a value, and Dat12 and Dat13 are Has no value. Therefore, the values of | Dat11-Dat12 | and | Dat13-Dat11 | are very large.
[0127]
In this way, when the pixel position for calculating the signal value is in the monochrome flat portion, two of | Dat11-Dat12 |, | Dat12-Dat13 |, | Dat13-Dat11 | The sum of is also a large value. Therefore, it is possible to determine whether or not the pixel position is in the monochromatic flat portion by appropriately setting the threshold T1 and determining whether or not the relationship represented by the above formula (57) is satisfied.
[0128]
Here, when the pixel position for calculating the signal value is in the monochromatic flat portion, the linear interpolation calculation is simply performed without calculating the signal value in the first interpolation calculation 11 and the second interpolation calculation 12. No false color is generated. Therefore, the determination unit 21 inputs the image data S0 to the linear interpolation unit 22 when it is determined that the pixel position for calculating the signal value is in the monochrome flat portion. The linear interpolation means 22 calculates a signal value by performing a linear interpolation operation using the signal values of the pixel position where the signal value is calculated and the surrounding pixel positions.
[0129]
For example, when calculating the R signal at the G pixel position, if the pixel position for calculating the signal value is the G33 pixel position,
R (G33) = (R22 + R24) / 2 (58)
To calculate the R (G33) signal. When calculating the B (G33) signal at the G33 pixel position,
B (G33) = (B24 + B42) / 2 (59)
To calculate the B (G33) signal.
[0130]
Also, when calculating the B signal at the R pixel position, assuming that the pixel position for calculating the signal value is R44,
B (R44) = (B24 + B42 + B46 + B64) / 4 (60)
To calculate the B (R44) signal.
[0131]
Further, when calculating the R signal at the B pixel position, if the pixel position for calculating the signal value is B24,
R (B24) = (R04 + R22 + R26 + R44) / 4 (61)
To calculate the R (B24) signal.
[0132]
Next, the operation of the second embodiment will be described. The operation of the second embodiment is basically the same as that of the first embodiment shown in FIG. 19, and only the processing in step S4 in FIG. 19 is different. Only the signal calculation process will be described. FIG. 25 is a flowchart showing the operation of signal calculation processing in the second embodiment. First, the determination means 21 determines whether or not the pixel position for calculating the signal value is in the monochromatic flat portion by the above equation (57) (step S21). If it is in the monochromatic flat portion, step S21 is affirmed, and the R signal QR is calculated by the linear interpolation means 22 shown in the above equations (58) and (61) in the linear interpolation means 22 (step S30), and the processing is terminated. The B signal QB is calculated by the linear interpolation calculation shown in the above equations (59) and (60).
[0133]
If step S21 is negative, the process proceeds to step S22, and similarly to the first embodiment, the signal value is calculated by the processing 1 and 2 (step S22), and the signal value is calculated by the processing 3 and 4 (step S23). , Calculation of signal values by processing 5 and 6 (step S24), determination of whether or not a gray edge is detected (step S25), calculation of variation value for gray edge (step S26), calculation of variation value for color edge (step S27), weighting Coefficient calculation (step S28) and weighted addition (step S29) are performed to calculate the R signal QR and the B signal QB.
[0134]
As described above, in the second embodiment, when calculating the R and B signals QR and QB, it is first determined whether or not the pixel position where the signal value is calculated is in the monochromatic flat portion. In some cases, since the signal value is calculated by linear interpolation calculation, the pixel position of the monochromatic flat portion is calculated compared to the case where the signal value is calculated by the method of the first embodiment for all the pixels. The amount can be reduced, thereby enabling high-speed processing.
[0135]
Here, in the imaging apparatus using the single CCD 1, it is necessary to use an optical low-pass filter in the imaging system for obtaining image data in order to reduce false colors. FIG. 26 is a diagram showing an image pickup system of a digital camera using a honeycomb array single-plate CCD, and FIG. 27 is a diagram showing frequency characteristics of signal values obtained in the honeycomb array single-plate CCD. In FIG. 27, fs / 2 is the Nyquist frequency, the solid line in FIG. 27A is a frequency band that can be reproduced by conventional processing, and the solid line in FIG. 27B is a frequency that can be reproduced by the processing of this embodiment. Each band is shown. As shown in FIG. 26, the imaging system of the digital camera includes a photographing lens 31, an optical low-pass filter 32, and a honeycomb CCD single-plate CCD 33 arranged in order from the subject. The optical axis is indicated by X. In the methods described in JP-A-10-20906, 10-136391, etc., the effect of reducing false colors is insufficient, so the frequency band outside the solid line shown in FIG. Therefore, it is necessary to use an optical low-pass filter 32 having a narrow band. However, when such an optical low-pass filter having a narrow band is used, the resolution of the obtained image is lowered. Further, in order to create such an optical low-pass filter 32 having a narrow frequency band, it is necessary to attach three quarter-wave plates having a center wavelength of about 630 to 785 nm as shown in FIG. The configuration of the imaging system is increased.
[0136]
According to the present invention, since the false color can be reduced as compared with the conventional method, the frequency band of the optical low-pass filter 32 can be expanded as shown in FIG. The resolution of the image represented by the image data S0 obtained in the CCD 33 can be improved. Also, as shown in FIG. 28B, the characteristics shown in FIG. 27B can be obtained by bonding two quarter-wave plates together, so that the imaging system can be reduced in size.
[0137]
In the first and second embodiments, the weighting coefficient calculation means 14 performs the division by (SA + SB) on the variation value SA as shown in the equation (41), thereby obtaining the weighting coefficient Sj. Calculated.
[0138]
Sj = SA / (SA + SB) (if SA + SB = 0 then Sj = 0.5) (41)
Here, in an arithmetic circuit that performs various arithmetic operations, when the arithmetic operation is only addition, subtraction, and multiplication, the circuit can be configured with relatively few arithmetic elements. However, if division is included as in the above equation (41), the calculation becomes complicated, so a large number of calculation elements are required, the circuit configuration becomes complicated, and the calculation takes a long time.
[0139]
Here, when the weighting coefficient Sj is calculated by the equation (41), when this is solved for the variation SB,
SB = ((1-Sj) / Sj) * SA (41 ')
It becomes. This indicates that when the weighting coefficient Sj is constant, the relationship between the variation value SA and the variation value SB is proportional. Therefore, when a two-dimensional plane having the variation value SA as the x-axis and the variation value SB as the y-axis is set, the weighting coefficient Sj is inclined according to the weighting coefficient on the two-dimensional plane as shown in FIG. Will be represented as a straight line that passes through the origin with (not strictly including the origin). Hereinafter, an embodiment in which the division process in Expression (41) is omitted using this relationship will be described as a third embodiment.
[0140]
FIG. 30 is a diagram for explaining the processing of the third embodiment, and FIG. 31 is a schematic block diagram showing the configuration of the RB interpolation means of the image processing apparatus according to the third embodiment. In FIG. 31, the same components as those in FIG. 5 are denoted by the same reference numerals, and detailed description thereof is omitted. In the third embodiment, instead of the weighting coefficient calculating means 14, the variation values SA, with respect to the weighting coefficients Sj when the signals Q1 and Q2 obtained by the first interpolation means 11 and the second interpolation means 12 are weighted and interpolated. Based on the memory 14A storing the correspondence relationship of SB with respect to a plurality of weighting coefficients and the variation values SA and SB calculated by the variation value calculation means 13, a plurality of weightings are referred to with reference to the correspondence relationship stored in the memory 14A. Weighting coefficient selection means 24 for selecting one weighting coefficient Sj from the coefficients, and the weighting addition means 15 performs weighted addition using the weighting coefficient Sj selected by the weighting coefficient selection means 24. . In the third embodiment, the weighting coefficient selection means 24 and the memory 14A are considered as weighting coefficient setting means in the present invention.
[0141]
First, how to obtain the correspondence between the variation values SA and SB with respect to the weighting coefficient Sj will be described. In the third embodiment, when the weighting coefficient Sj is constant, the fluctuation value SA and the fluctuation value SB are in a proportional relationship. In the SA-SB plane, the weighting coefficient Sj is a straight line having a slope corresponding to the value. First, the weighting coefficient Sj is quantized into five stages of Sj = 0, 0.25, 0.5, 0.75, and 1.0 as shown in FIG. Then, as shown in FIG. 30, five areas A1 to A5 are set on the SA-SB plane with Sj = 0, 0.125, 0.375, 0.625, 0.875, 1 as boundaries, and each area is set. A1 to A5 are associated with Sj = 0, 0.25, 0.5, 0.75, and 1.0, respectively. Here, when Sj = 0.125, SB = 7 SA, when Sj = 0.375, SB = 5/3 SA, when Sj = 0.625, SB = 3/5 SA, when Sj = 0.875, SB = 1 / 7SA.
[0142]
Therefore, as shown in FIG. 31, a memory 14A is provided, the relationship shown in FIG. 30 is stored in the memory 14A as a table, and the weighting coefficient selection means 24 refers to this table based on the variation values SA and SB. Thus, the weighting coefficient Sj can be obtained without performing division as shown in the equation (41). For example, when the calculated fluctuation values SA and SB have a relationship of 3/5 SA <SB ≦ 5/3 SA, that is, 3SA <5SB and 5SA ≧ 3SB, the fluctuation values SA and SB are shown in FIG. Since it is plotted in the area A3 of the graph shown, the weighting coefficient can be determined as Sj = 0.5. If the relationship is 7SA <SB, the plot is in the area A1, so Sj = 0, 5 / 3SA <SB ≦ 7SA, that is, if 5SA <3SB and 7SA ≧ SB, the plot is Since Sj = 0.25 and 1 / 7SA <SB ≦ 3 / 5SA because of being in the area A2, that is, when SA <7SB and 3SA ≧ 5SB, the plot is in the area A4, so Sj = 0.75 , SB ≦ 1/7 SA, that is, if SA ≧ 7SB, the plot is in the region A5 and can be obtained as Sj = 1.
[0143]
It is preferable that the weighting coefficient Sj is more finely quantized, but it is preferable that the weighting coefficient Sj is about five stages in consideration of the calculation time for determining the region where the fluctuation values SA and SB exist. According to the experiment conducted by the present applicant, even if the quantization is performed in five steps, the false color reduction effect can be sufficiently obtained.
[0144]
Then, the weighted addition means 15 performs the weighted addition shown in the above equations (42) to (44) using the weighting coefficient Sj to calculate the signal value. For example, the calculation of the B (R44) signal in the R44 pixel is performed when the weighting coefficient Sj = 0.5.
B (R44) = (1-Sj) * B (R44) A + Sj * B (R44) B = 0.5 * B (R44) A + 0.5 * B (R44) B (62)
In the third embodiment, Sj or (1-Sj) = 1/2 ⁿ Since the weighting coefficient is set so that (n is a natural number), it is also possible to perform an arithmetic operation by converting it into an integer using a bit shift as shown in the following equation (63).
[0145]
B (R44) = (B (R44) A + B (R44) B + 1) >> 1 (63)
Here, >> 1 indicates that when the signal value is expressed in binary, it is shifted to the right by 1 bit. Thus, by shifting the right one bit, the signal value is halved, so that the same calculation as in equation (62) is performed. The reason why 1 is added in the equation (63) is to round off the calculation result.
[0146]
Furthermore, in the first and second embodiments, the fluctuation value calculation means 13 calculates the fluctuation values SA and SB in the case of a color edge according to the above equations (29) and (30). Since these expressions include division, it is preferable not to perform division in order to reduce the amount of calculation. In this case, there is no problem in calculating the weighting coefficient Sj unless the ratio of the fluctuation values SA and SB changes. Therefore, in the case of the above formulas (29) and (30), the right side is multiplied by (R (G33) A + G33) * (B (G33) B + G33), and the following formulas (29 ′) and (30 ′ ) To calculate the fluctuation values SA and SB.
[0147]
SA = | R (G33) A-G33 | * (B (G33) B + G33) (29 ')
SB = | B (G33) B-G33 | * (R (G33) A + G33) (30 ')
The above formulas (33) and (34) are multiplied by (B (G35) A + G35) * (R (G35) B + G35) to obtain the following formulas (33 ′) and (34 ′). The fluctuation values SA and SB may be calculated by
[0148]
SA = | B (G35) A-G35 | * (R (G35) B + G35) (33 ')
SB = | R (G35) B-G35 | * (B (G35) A + G35) (34 ')
In addition, the above formulas (37) and (38) are multiplied by (B24 + G (B24) A) * (B24 + G (B24) B) to obtain the following formulas (37 ′) and (38 ′). The fluctuation values SA and SB may be calculated by
[0149]
SA = | B24-G (B24) A | * (B24 + G (B24) B) (37 ')
SB = | B24-G (B24) B | * (B24 + G (B24) A) (38 ')
Hereinafter, an operation for obtaining a signal value using a bit shift in the third embodiment will be described. FIG. 32 is a flowchart showing the operation of the third embodiment. In FIG. 32, since the processes up to the calculation of the fluctuation values SA and SB are the same as those in the first and second embodiments, detailed description is omitted, and the fluctuation values SA and SB are calculated. Only the calculation of the weighting coefficient Sj and the weighted addition process will be described. FIG. 32 shows the calculation of the B (R44) signal at the R44 pixel position. First, it is determined whether or not the sum of the fluctuation values SA and SB is 0 (step S41). If step S41 is affirmed, a B (R44) signal is calculated by equation (63) using SA = SB = 0.5 and bit shift (step S42).
[0150]
B (R44) = (B (R44) A + B (R44) B + 1) >> 1 (63)
If step S41 is negative, it is determined whether SA ≧ 7SB (step S43). If step S43 is affirmed, the relationship between the fluctuation values SA and SB is in the region A5 in FIG. 30, so Sj = 1, and the B (R44) signal is calculated by the following equation (64) (step S44). ).
[0151]
B (R44) = B (R44) B (64)
If step S43 is negative, it is determined whether 3SA ≧ 5SB (step S45). If step S45 is affirmed, the relationship between the fluctuation values SA and SB is in the region A4 in FIG. 30, so Sj = 0.75, and the B (R44) signal is calculated by the following equation (65) ( Step S46).
[0152]
B (R44) = (B (R44) A + 3B (R44) B + 2) >> 2 (65)
If step S45 is negative, it is determined whether or not 5SA ≧ 3SB (step S47). If step S47 is affirmed, the relationship between the fluctuation values SA and SB is in the region A3 in FIG. 30, so Sj = 0.5, and the B (R44) signal is calculated by equation (63) (step S48). ).
[0153]
If step S47 is negative, it is determined whether or not 7SA ≧ SB (step S49). If step S49 is affirmed, the relationship between the fluctuation values SA and SB is in the region A2 in FIG. 30, so Sj = 0.25, and the B (R44) signal is calculated by the following equation (66) ( Step S50).
[0154]
B (R44) = (3 * B (R44) A + B (R44) B + 2) >> 2 (66)
If step S49 is negative, the relationship between the fluctuation values SA and SB is in the region A1 in FIG. 30, so Sj = 0, and the B (R44) signal is calculated by the following equation (67) ( Step S51).
[0155]
B (R44) = B (R44) A (67)
Thus, according to the third embodiment, since no division is performed when calculating the weighting coefficient Sj, the configuration of the circuit that calculates the weighting coefficient Sj can be simplified, and as a result, The configuration of the apparatus for carrying out this embodiment can be simplified, and the calculation time can be shortened. In addition, by performing bit shift, it is possible to perform an arithmetic operation with an integer value. Therefore, it is possible to simplify the arithmetic operation and reduce the processing time. Therefore, it becomes easy to mount an apparatus for carrying out the present invention on an imaging apparatus such as a digital camera, an image reading apparatus such as a scanner, a printer, or the like, and thereby a high-function apparatus can be created.
[0156]
In each of the above embodiments, the single CCD 1 has been described in which the G pixel has double the density of the R and B pixels. However, the R pixel is for the G and B pixels, or the B pixel is R. , G pixels may have double the density. Further, the description has been given of the one having spectral sensitivity with respect to R, G, and B. However, the single-plate CCD 1 is not limited to this, and with respect to Y (yellow), G (green), and C (cyan). May have spectral sensitivity, or may have spectral sensitivity with respect to Y, W (white), and C.
[0157]
In each of the above embodiments, the signal values for all three colors are calculated, but the signal values for only one color or only two colors may be calculated.
[0158]
Furthermore, in the first and second embodiments, the weighting coefficient Sj is calculated by the equation (41). However, the weighting coefficient Sj may be calculated by the following equation (68).
[0159]
Sj = SB / (SA + SB) (if SA + SB = 0 then Sj = 0.5) (68)
In this case, the R and B signals QR and QB are calculated by the following equation (69).
[0160]
QR, QB = Sj * Q1 + (1-Sj) * Q2 (69)
[Brief description of the drawings]
FIG. 1 is a schematic block diagram showing the configuration of an image processing apparatus according to a first embodiment of the present invention.
FIG. 2 is a diagram showing a pixel arrangement in a single CCD.
FIG. 3 is a diagram showing a pixel arrangement of a single-panel CCD having a honeycomb arrangement.
FIG. 4 is a diagram showing each pixel position of a single-panel CCD arranged in a honeycomb arrangement with reference numerals attached.
FIG. 5 is a schematic block diagram showing a configuration of RB interpolation means.
FIG. 6 is a diagram showing an example of a Cubic spline interpolation filter.
FIG. 7 is a diagram for explaining calculation of signal values (part 1);
FIG. 8 is a diagram for explaining calculation of a signal value (part 2);
FIG. 9 is a diagram for explaining calculation of signal values (No. 3);
FIG. 10 is a diagram for explaining calculation of signal values (No. 4);
FIG. 11 is a diagram for explaining calculation of signal values (No. 5);
FIG. 12 is a diagram for explaining calculation of signal values (No. 6);
FIG. 13 is a diagram for explaining calculation of signal values (No. 7);
FIG. 14 is a diagram illustrating an example of a color edge composed of red (R) and green (G).
FIG. 15 is a diagram illustrating an example of a gray edge
FIG. 16 is a diagram for explaining a difference between calculation methods of fluctuation values SA and SB;
FIG. 17 is a diagram illustrating an example of a checkered square filter (part 1);
FIG. 18 is a diagram showing an example of a checkered square filter (No. 2)
FIG. 19 is a flowchart showing the operation of the first embodiment.
FIG. 20 is a flowchart showing an operation of signal calculation processing.
FIG. 21 is a schematic block diagram showing the configuration of RB interpolation means in the image processing apparatus according to the second embodiment of the present invention;
FIG. 22 is a diagram illustrating signal values of a single color flat portion of green (G).
FIG. 23 is a diagram illustrating signal values of a single color flat portion of blue (B).
FIG. 24 is a diagram showing signal values of a single color flat portion of red (R).
FIG. 25 is a flowchart showing the operation of signal calculation processing in the second embodiment.
FIG. 26 is a diagram showing an imaging system of a digital camera using a honeycomb CCD single-plate CCD.
FIG. 27 is a diagram showing frequency characteristics of signal values obtained in a single-panel CCD having a honeycomb arrangement;
FIG. 28 is a diagram showing the arrangement of optical low-pass filters.
FIG. 29 is a diagram showing the relationship between the variation values SA and SB and the weighting coefficient Sj.
FIG. 30 is a diagram for explaining processing of the third embodiment;
FIG. 31 is a schematic block diagram showing the configuration of RB interpolation means in an image processing apparatus according to a third embodiment of the present invention;
FIG. 32 is a flowchart showing the operation of the third embodiment.
Fig. 33 shows a honeycomb arrangement
FIG. 34 shows a Bayer array.
FIG. 35 is a diagram showing pixel positions in the Bayer array with reference numerals attached thereto;
FIG. 36 is a diagram showing pixel positions of the honeycomb array with reference numbers attached.
[Explanation of symbols]
1,33 single CCD
2 Interpolation means
3 G interpolation means
4 RB interpolation means
5 checkered square interpolation means
10, 21 judgment means
11 First interpolation means
12 Second interpolation means
13 Fluctuation value calculation means
14 Weighting coefficient calculation means
14A memory
15 Weighted addition means
22 Linear interpolation means
24 Weighting coefficient selection means
31 Shooting lens
32 Optical low-pass filter

Claims (17)

An imaging device in which imaging surfaces are formed by arranging first to third photoelectric conversion elements having different spectral sensitivities on a single plane, wherein the first and second photoelectric conversion elements are alternately arranged in a predetermined direction. To form a first line, to alternately arrange the first and third photoelectric conversion elements in the predetermined direction to form a second line, and in the orthogonal direction perpendicular to the predetermined direction, the first line The first and second lines are alternately arranged in the orthogonal direction so that the first and second photoelectric conversion elements are alternately arranged, and the first and third photoelectric conversion elements are alternately arranged. Thus, based on the first to third signal values obtained in the imaging device in which the imaging surface including the first to third pixels is formed, at least of the first to third signal values at all pixel positions. One signal value An image processing method for estimating,
Determine whether the pixel position where the signal value is estimated is at the gray edge or the color edge,
A predetermined direction estimated value that is an estimated value of the signal value for the predetermined direction and an orthogonal direction estimated value that is an estimated value of the signal value for the orthogonal direction;
By calculating differently depending on the result of the determination, a predetermined direction fluctuation value representing the fluctuation amount of the signal value in the predetermined direction and an orthogonal direction fluctuation value representing the fluctuation amount of the signal value in the orthogonal direction are calculated,
Based on the predetermined direction variation value and the orthogonal direction variation value, set a weighting coefficient for weighted addition of the predetermined direction estimated value and the orthogonal direction estimated value,
An image processing method characterized in that, based on the weighting coefficient, the predetermined direction estimation value and the orthogonal direction estimation value are weighted and added to estimate at least one signal value of the first to third signal values. .   An addition value between the predetermined direction fluctuation value and the orthogonal direction fluctuation value is calculated, a ratio of the predetermined direction fluctuation value or the orthogonal direction fluctuation value to the addition value is calculated, and the ratio is calculated as the predetermined direction estimation value and the predetermined direction fluctuation value. 2. The image processing method according to claim 1, wherein the orthogonal direction estimation value is set as a weighting coefficient for weighted addition. The ratio of the predetermined direction fluctuation value or the orthogonal direction fluctuation value to the addition value of the predetermined direction fluctuation value and the orthogonal direction fluctuation value is used as a weighting coefficient when the predetermined direction estimation value and the orthogonal direction estimation value are weighted and added. When setting,
On the two-dimensional plane defined by the predetermined direction variation value and the orthogonal direction variation value based on the proportional relationship between the predetermined direction variation value and the orthogonal direction variation value when the weighting coefficient is constant. A plurality of predetermined lines generated by setting a plurality of straight lines having inclinations of the plurality of weighting coefficients and setting a plurality of inclination ranges based on the respective straight lines on the two-dimensional plane. Referring to a table for determining weighting coefficients for the direction variation value and various orthogonal direction variation values, one weighting factor is selected from the plurality of weighting factors based on the predetermined direction variation value and the orthogonal direction variation value And setting the selected one weighting coefficient as a weighting coefficient for weighted addition of the predetermined direction estimated value and the orthogonal direction estimated value. The image processing method according to claim 1 wherein symptoms. The plurality of weighting coefficients in the table are set such that a ratio of the predetermined direction variation value or the orthogonal direction variation value to the addition value is 1 / 2n (n is a natural number),
The image processing method according to claim 3, wherein the weighted addition is performed by bit shift based on the weighting coefficient. Determining whether or not the pixel position for estimating the signal value is in a monochromatic flat portion consisting of only one of the first to third signal values;
If it is determined that the pixel is in the monochrome flat part, at least one signal value of the first to third signal values is obtained by linear interpolation based on the first to third signal values in the monochrome flat part. Estimate
When it is determined that the monochrome flat portion is not present, it is determined whether the gray edge or the color edge is present, calculation of the predetermined direction estimated value and the orthogonal direction estimated value, the predetermined direction variation value, and the 5. The image processing method according to claim 1, wherein calculation of an orthogonal direction variation value, setting of the weighting coefficient, and estimation of a signal value based on the weighting coefficient are performed. An imaging device in which imaging surfaces are formed by arranging first to third photoelectric conversion elements having different spectral sensitivities on a single plane, wherein the first and second photoelectric conversion elements are alternately arranged in a predetermined direction. To form a first line, to alternately arrange the first and third photoelectric conversion elements in the predetermined direction to form a second line, and in the orthogonal direction perpendicular to the predetermined direction, the first line The first and second lines are alternately arranged in the orthogonal direction so that the first and second photoelectric conversion elements are alternately arranged, and the first and third photoelectric conversion elements are alternately arranged. Thus, based on the first to third signal values obtained in the imaging device in which the imaging surface including the first to third pixels is formed, at least of the first to third signal values at all pixel positions. One signal value An image processing apparatus for estimating,
Determining means for determining whether a pixel position for estimating a signal value is at a gray edge or a color edge;
An estimated value calculating means for calculating a predetermined direction estimated value that is an estimated value of the signal value for the predetermined direction and an orthogonal direction estimated value that is an estimated value of the signal value for the orthogonal direction;
A variation value for calculating a predetermined direction variation value that represents the variation amount of the signal value in the predetermined direction and an orthogonal direction variation value that represents the variation amount of the signal value in the orthogonal direction by different calculation methods according to the determination result. A calculation means;
Weighting coefficient setting means for setting a weighting coefficient for weighted addition of the predetermined direction estimation value and the orthogonal direction estimation value based on the predetermined direction fluctuation value and the orthogonal direction fluctuation value;
Weighting addition means for estimating the at least one signal value among the first to third signal values by weighting and adding the predetermined direction estimated value and the orthogonal direction estimated value based on the weighting coefficient. An image processing apparatus. The weighting coefficient setting means calculates a sum of the predetermined direction fluctuation value and the orthogonal direction fluctuation value, and calculates a ratio of the predetermined direction fluctuation value or the orthogonal direction fluctuation value with respect to the addition value. Prepared,
The image processing apparatus according to claim 6, wherein the ratio is a means for setting the ratio as a weighting coefficient for weighted addition of the predetermined direction estimated value and the orthogonal direction estimated value. The weighting coefficient setting means determines the ratio of the predetermined direction variation value or the orthogonal direction variation value to the sum of the predetermined direction variation value and the orthogonal direction variation value, and the predetermined direction estimated value and the orthogonal direction estimated value. When setting as a weighting coefficient for weighted addition, the predetermined direction fluctuation value and the orthogonal direction fluctuation value when the weighting coefficient is constant are proportional to each other. A plurality of straight lines with inclinations of a plurality of weighting factors are set on a two-dimensional plane defined by the variation value in the orthogonal direction, and a plurality of inclinations based on the plurality of straight lines are set on the two-dimensional plane. It was produced by setting the range of, for storing a table for determining the weighting factors for various predetermined direction variation value and various direction orthogonal variation serial And means,
Weighting coefficient selecting means for selecting one weighting coefficient from the plurality of weighting coefficients with reference to the table based on the predetermined direction variation value and the orthogonal direction variation value;
7. The image processing apparatus according to claim 6, wherein the selected one weighting coefficient is a means for setting a weighting coefficient for weighted addition of the predetermined direction estimated value and the orthogonal direction estimated value. The plurality of weighting coefficients in the table are set such that a ratio of the predetermined direction variation value or the orthogonal direction variation value to the addition value is 1 / 2n (n is a natural number),
9. The image processing apparatus according to claim 8, wherein the weighted addition means is means for performing the weighted addition by bit shift based on the weighting coefficient. Flat part determining means for determining whether or not the pixel position for estimating the signal value is in a single color flat part including only one of the first to third signal values;
When it is determined to be in the monochrome flat part, at least one signal value is estimated from the first to third signal values by linear interpolation based on the first to third signal values in the monochrome flat part And linear interpolation means for
When it is determined that the monochrome flat portion is not present, the determination unit determines whether there is a gray edge or a color edge, the estimation value calculation unit calculates the predetermined direction estimation value and the orthogonal direction estimation value, Calculation of the predetermined direction fluctuation value and the orthogonal direction fluctuation value by the fluctuation value calculation means, setting of the weighting coefficient by the weighting coefficient setting means, and estimation of a signal value based on the weighting coefficient by the weighting addition means. The image processing apparatus according to claim 6, wherein: An imaging device in which imaging surfaces are formed by arranging first to third photoelectric conversion elements having different spectral sensitivities on a single plane, wherein the first and second photoelectric conversion elements are alternately arranged in a predetermined direction. To form a first line, to alternately arrange the first and third photoelectric conversion elements in the predetermined direction to form a second line, and in the orthogonal direction perpendicular to the predetermined direction, the first line The first and second lines are alternately arranged in the orthogonal direction so that the first and second photoelectric conversion elements are alternately arranged, and the first and third photoelectric conversion elements are alternately arranged. Thus, based on the first to third signal values obtained in the imaging device in which the imaging surface including the first to third pixels is formed, at least of the first to third signal values at all pixel positions. One signal value In computer-readable recording medium storing a program for executing the image processing method of estimating in a computer,
The program determines whether a pixel position from which a signal value is estimated is a gray edge or a color edge;
Calculating a predetermined direction estimated value that is an estimated value of the signal value for the predetermined direction and an orthogonal direction estimated value that is an estimated value of the signal value for the orthogonal direction;
A procedure for calculating a predetermined direction fluctuation value representing the fluctuation amount of the signal value in the predetermined direction and an orthogonal direction fluctuation value representing the fluctuation amount of the signal value in the orthogonal direction by different calculation methods according to the determination result; ,
A step of setting a weighting coefficient for weighted addition of the predetermined direction estimation value and the orthogonal direction estimation value based on the predetermined direction variation value and the orthogonal direction variation value;
And a step of weighting and adding the predetermined direction estimation value and the orthogonal direction estimation value based on the weighting coefficient to estimate at least one signal value from the first to third signal values. A computer-readable recording medium. The step of setting the weighting factor includes a step of calculating an addition value of the predetermined direction variation value and the orthogonal direction variation value and calculating a ratio of the predetermined direction variation value or the orthogonal direction variation value to the addition value. Have
12. The computer-readable recording medium according to claim 11, wherein the ratio is a procedure for setting the ratio as a weighting coefficient for weighted addition of the predetermined direction estimated value and the orthogonal direction estimated value. The step of setting the weighting factor includes: a ratio of the predetermined direction variation value or the orthogonal direction variation value to an addition value of the predetermined direction variation value and the orthogonal direction variation value; and the predetermined direction estimated value and the orthogonal direction estimated value. Is set as a weighting coefficient for weighted addition,
On the two-dimensional plane defined by the predetermined direction variation value and the orthogonal direction variation value based on the fact that the predetermined direction variation value and the orthogonal direction variation value are proportional to each other when the weighting factor is constant. A plurality of predetermined lines generated by setting a plurality of straight lines having inclinations of the plurality of weighting coefficients and setting a plurality of inclination ranges based on the plurality of straight lines on the two-dimensional plane. Referring to a table for determining weighting coefficients for the direction variation value and various orthogonal direction variation values, one weighting factor is selected from the plurality of weighting factors based on the predetermined direction variation value and the orthogonal direction variation value Has a procedure to
12. The computer-readable recording according to claim 11, wherein the selected one weighting coefficient is a procedure for setting a weighting coefficient for weighted addition of the predetermined direction estimated value and the orthogonal direction estimated value. Medium. The plurality of weighting coefficients in the table are set such that a ratio of the predetermined direction variation value or the orthogonal direction variation value to the addition value is 1 / 2n (n is a natural number),
14. The computer-readable recording medium according to claim 13, wherein the step of performing the weighted addition is a step of performing the weighted addition by bit shift based on the weighting coefficient. A procedure for determining whether or not the pixel position for estimating the signal value is in a monochromatic flat portion including only one of the first to third signal values;
If it is determined that the pixel is in the monochrome flat part, at least one signal value of the first to third signal values is obtained by linear interpolation based on the first to third signal values in the monochrome flat part. The estimation procedure;
When it is determined that the monochrome flat portion is not present, it is determined whether the gray edge or the color edge is present, calculation of the predetermined direction estimated value and the orthogonal direction estimated value, the predetermined direction variation value, and the The computer-readable program according to claim 11, further comprising a procedure for calculating an orthogonal direction variation value, setting the weighting coefficient, and estimating a signal value based on the weighting coefficient. Recording medium.   An image pickup apparatus comprising the image processing apparatus according to claim 6.   An image reading apparatus comprising the image processing apparatus according to claim 6.

Images