JP6274744B2 - Image processing apparatus and image processing method - Google Patents

Description

The present invention is particularly directed to a suitable image processing apparatus and how be used to create a high-resolution image using the plurality of low-resolution images.

  As a technique for creating a single high-resolution image using a plurality of low-resolution images, a multi-image super-resolution is generally known. This multiple-image super-resolution technique is used for applications such as improving the image quality of electronic zoom in a digital camera. As a method of super-resolution of multiple images, an image having the same resolution as a high-resolution image is created for each color component using a low-resolution image obtained by an image sensor having a Bayer array color filter, and multiple-image super-resolution is obtained. A high resolution image is created by processing each image.

  However, as shown in FIG. 16, when a high resolution image is generated using low resolution images taken continuously, areas 1601 to 1604 of the low resolution image corresponding to the area 1605 are image heights from the center of the optical axis. Are all different. Therefore, when a high resolution image is generated by inserting a plurality of low resolution image pixels, there is a problem that the image quality deteriorates due to the influence of lateral chromatic aberration and distortion of each low resolution image.

  Also, each pixel of the high resolution image has a different image height from the center of the optical axis when adjacent pixels are inserted, so the magnification chromatic aberration correction and the distortion aberration correction are performed after the high resolution image is generated. It is very difficult. On the other hand, if lateral chromatic aberration correction and distortion correction are performed at the low-resolution image stage, a filtering means such as bilinear interpolation is used, so the frequency band of the low-resolution image drops, which is preferable for generating a high-resolution image. Absent. Therefore, as a countermeasure for the above, Patent Document 1 discloses a method of performing demosaicing processing after performing coordinate conversion based on optical characteristics such as lateral chromatic aberration and distortion.

JP 2009-1000041 A JP 2009-301181 A

  However, even if a high resolution image is generated by the method disclosed in Patent Document 1, the frequency band is reduced by the demosaicing process, which is not preferable for generating a high resolution image.

  In view of the above-described problems, an object of the present invention is to generate a high-resolution image in which aberration based on optical characteristics is corrected without reducing the frequency band of a low-resolution image.

An image processing apparatus according to the present invention is an image processing apparatus that generates a high-resolution image using a plurality of low-resolution images, and for each coordinate of the low-resolution image, coordinate conversion based on optical characteristics of each color component A coordinate conversion means for performing the detection, a detection means for detecting a displacement amount between the plurality of low resolution images, and a low resolution image coordinate-converted by the coordinate conversion means according to the displacement amount detected by the detection means. And an alignment unit for generating the high-resolution image by inserting a pixel value of the low-resolution image based on a result of the alignment by the alignment unit, and the detection The means further includes generating means for generating an image for alignment after coordinate conversion by the coordinate conversion means after reducing the low-resolution image. And detecting the positional deviation amount by using the image for positioning generated by the generating means.

  According to the present invention, it is possible to generate a high-resolution image in which aberration based on optical characteristics is corrected without dropping the frequency band of the low-resolution image.

It is a block diagram which shows the internal structural example of the image processing apparatus which concerns on embodiment. It is a block diagram which shows the specific structural example of the super-resolution process control part in an image processing part. It is a block diagram which shows the specific structural example of the coordinate transformation coefficient calculation part in the super-resolution process control part in 1st Embodiment. It is a block diagram which shows the specific structural example of the super-resolution process part in a super-resolution process control part. It is a flowchart which shows an example of the super-resolution process procedure by the super-resolution process control part which concerns on embodiment. It is a figure for demonstrating the process which isolate | separates a Bayer image into a mosaic image. 6 is a flowchart illustrating an example of a specific processing procedure by a coordinate conversion coefficient calculation unit in the first embodiment. 5 is a flowchart illustrating an example of a specific processing procedure by a super-resolution processing unit in the embodiment. It is a figure for demonstrating the pixel insertion process in R image. 5 is a flowchart illustrating an example of a specific processing procedure performed by a pixel insertion processing unit in the embodiment. It is a figure which shows the position of the reference coordinate in the super-resolution image R. It is a figure which shows an example of the super-resolution image which expanded the number of pixels k times in the horizontal direction and the vertical direction with respect to the input low-resolution image. It is a block diagram which shows the specific structural example of the coordinate transformation coefficient calculation part in the super-resolution process control part in 2nd Embodiment. In 2nd Embodiment, it is a flowchart which shows an example of the specific process sequence by a coordinate transformation coefficient calculation part. It is a figure for demonstrating the procedure which produces | generates a thinning image from the mosaic image of G of the horizontal pixel number W and the vertical pixel number H. It is a figure for demonstrating the method to produce | generate a high resolution image using the low resolution image image | photographed continuously.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. In the present embodiment, a super-resolution process for generating a high-resolution image using a plurality of low-resolution images and expanding the number of pixels in the horizontal and vertical directions by k times will be described.

FIG. 1 is a block diagram illustrating an internal configuration example of an image processing apparatus 100 according to the present embodiment.
In FIG. 1, the optical system 101 includes a lens group including a zoom lens and a focus lens, a diaphragm adjusting device, and a shutter device. The optical system 101 adjusts the magnification, focus position, or light amount of the subject image that reaches the image sensor 102. The imaging element 102 is a photoelectric conversion element such as a CCD or a CMOS sensor that photoelectrically converts a light beam of a subject that has passed through the optical system 101 into an electric signal.

  The A / D converter 103 converts the input analog video signal into image data that is a digital signal. The image processing unit 104 performs normal signal processing such as white balance and γ correction. Furthermore, the image processing unit 104 includes a super-resolution processing control unit 111, and performs super-resolution processing using a plurality of input image data (Bayer images). The image processing unit 104 can perform similar image processing not only on the image data output from the A / D conversion unit 103 but also on the image data read from the recording unit 108. The system control unit 105 is a control function unit that controls and controls the overall operation of the image processing apparatus 100 according to the present embodiment. The system control unit 105 also performs drive control of the optical system 101 and the image sensor 102 based on the luminance value obtained from the image data processed by the image processing unit 104 and the instruction transmitted from the operation unit 106.

  The display unit 107 includes a liquid crystal display or an organic EL (Electro Luminescence) display, and displays image data generated by the image sensor 102 and an image related to the image data read from the recording unit 108. The recording unit 108 has a function of recording image data on the recording medium 110. The recording medium 110 may be, for example, an information recording medium using a package containing a rotary recording body such as a memory card on which a semiconductor memory is mounted or a magneto-optical disk, and may be further detachable. A bus 109 is used to exchange data among the image processing unit 104, the system control unit 105, the display unit 107, and the recording unit 108.

Here, a specific configuration of the super-resolution processing control unit 111 in the image processing unit 104 which is a characteristic part of the present embodiment will be described. FIG. 2 is a block diagram illustrating a specific configuration example of the super-resolution processing control unit 111.
As shown in FIG. 2, the super-resolution processing control unit 111 according to the present embodiment includes a reference image selection unit 201, a Bayer separation unit 202, a coordinate conversion coefficient calculation unit 203, a memory unit 204, a super-resolution processing unit 205, And a YUV conversion unit 206. 3 shows a specific configuration example of the coordinate conversion coefficient calculation unit 203, and FIG. 4 shows a specific configuration example of the super-resolution processing unit 205. The configuration shown in FIGS. 3 and 4 will be described later. Hereinafter, processing performed by each configuration in FIG. 2 will be described with reference to the flowchart in FIG. 5.

FIG. 5 is a flowchart illustrating an example of a super-resolution processing procedure by the super-resolution processing control unit 111 according to the present embodiment.
First, in step S501, the reference image selection unit 201 acquires a plurality of input Bayer images in order. In step S502, a Bayer image serving as a reference for performing super-resolution processing is selected from a plurality of Bayer images input. In this embodiment, the input first image is selected as a reference image. Hereinafter, the reference Bayer image is a reference image, and the other Bayer images are non-reference images.

  Next, in step S503, control is performed so that the reference image is first output to the Bayer separation unit 202, and then a plurality of non-reference images are output. When N images are input, the reference image is output to the first image, and then the second to Nth non-reference images are output.

  Next, in step S504, the Bayer separation unit 202 separates the Bayer image output in step S503 into three mosaic images of R, G, and B, and the coordinate transformation coefficient calculation unit 203 and the super-resolution processing, respectively. The data is output to the unit 205. Here, the process of separating the Bayer image into a mosaic image will be described in detail with reference to FIG.

  FIG. 6A is a diagram illustrating an example of a Bayer image with the number of horizontal images W and the number of vertical pixels H output from the reference image selection unit 201. FIG. 6B is a diagram illustrating an example of three mosaic images obtained by separating the Bayer image illustrated in FIG. 6A into R, G, and B color components. The mosaic image includes non-missing pixels 601 having pixel value information of the Bayer image and missing pixels 602 having no value information of the Bayer image and having been input with a value of 0.

  Next, in step S505, a coordinate conversion coefficient necessary for aligning the non-reference image with the reference image is calculated. In the present embodiment, distortion correction is performed on an image that has been demosaiced from a G mosaic image to create a G image for alignment. By using the G image, a projective transformation coefficient for aligning the images is calculated.

FIG. 3 is a block diagram illustrating a specific configuration example of the coordinate conversion coefficient calculation unit 203 of FIG.
In FIG. 3, the coordinate conversion coefficient calculation unit 203 includes a demosaic processing unit 301, a distortion correction processing unit 302, and a projective conversion coefficient calculation unit 303. Hereinafter, specific processing of step S505 performed by each configuration illustrated in FIG. 3 will be described with reference to the flowchart of FIG.

FIG. 7 is a flowchart showing an example of a specific processing procedure by the coordinate conversion coefficient calculation unit 203 in step S505 of FIG.
First, in step S <b> 701, the demosaic processing unit 301 acquires a G mosaic image currently being processed. In step S702, a demosaic process is performed on the G mosaic image acquired in step S701. As the demosaicing method, a known method such as linear interpolation or adaptive interpolation is used.

  In step S703, the distortion correction processing unit 302 performs distortion correction processing on the G image that has been demosaiced in step S702. As a distortion correction method, a known method such as correction by image deformation using lens information is used. In step S704, it is determined whether or not the G image subjected to distortion correction in step S703 is the first image. As a result of this determination, if the image is the first image, that is, the reference image, the process proceeds to step S709. If the image is not the first image but the non-reference image, the process proceeds to step S705.

  In step S709, the projective transformation coefficient calculation unit 303 stores the G image subjected to distortion correction in step S703 in the memory unit 204 as the first alignment G image. On the other hand, in step S705, the projective transformation coefficient calculation unit 303 sets the G image subjected to distortion correction in step S703 as a G image for alignment, and separates the first alignment image from the G image for alignment. A G image is acquired from the memory unit 204.

Next, in step S706, the projective transformation coefficient calculation unit 303 detects the amount of misalignment between the images using the two alignment G images aligned in step S705. At this time, the amount of positional deviation of the alignment G image to be processed corresponding to the first alignment G image is detected at a plurality of locations. A known method such as a method using pattern matching is used as a method for detecting the amount of displacement. In step S707, the projection conversion coefficient calculation unit 303 calculates a coordinate conversion coefficient from the amount of positional deviation detected in step S706. In this embodiment, the coefficients a _{0 to} a ₇ of the projective transformation equation shown in the following equation (1) are calculated as coordinate transformation factors. As a method for calculating the coordinate conversion coefficient, a known method using a least square method or the like is used.

  Next, in step S708, the coordinate conversion coefficient calculated in step S707 is stored in the memory unit 204. The coordinate conversion coefficient is used in the super-resolution processing unit 205 later.

  Returning to the description of FIG. 5, in step S506, the super-resolution processing unit 205 creates a high-resolution image using the low-resolution image. In this process, in order to generate a k-times high-resolution image, coordinate conversion is performed on which coordinates of the target pixel of the low-resolution image correspond to the high-resolution image, and the pixel value is inserted into the high-resolution image. Perform the process. In the present embodiment, for G, distortion correction and coordinate conversion for alignment between images are performed at once, and for R and B, magnification chromatic aberration correction, distortion correction, and coordinate conversion for alignment between images are performed at once.

FIG. 4 is a block diagram illustrating a specific configuration example of the super-resolution processing unit 205 in FIG.
In FIG. 4, the super-resolution processing unit 205 includes a magnification chromatic aberration coordinate correction unit 401, a distortion coordinate correction unit 402, an alignment coordinate conversion unit 403, a pixel insertion processing unit 404, and a missing interpolation processing unit 405. . Hereinafter, specific processing of step S506 performed by each configuration illustrated in FIG. 4 will be described with reference to the flowchart of FIG.

FIG. 8 is a flowchart illustrating an example of a specific processing procedure performed by the super-resolution processing unit 205 in step S506 of FIG.
First, in step S801, the super-resolution processing unit 205 performs coordinates R (x, y), G (x, X, R, G, B) in order to perform super-resolution processing for each coordinate. y) and B (x, y) are all initialized to (0, 0).

Subsequently, in step S802, the magnification chromatic aberration coordinate correction unit 401 performs coordinate correction by magnification chromatic aberration correction on the coordinates R (x, y) and B (x, y). In this embodiment, coordinate correction is performed based on lens information. Here, the lens information is information regarding the shift amount of R and B with respect to G based on the image height from the center of the optical axis, and is held by the system control unit 105 in order to correct only R and B by magnification chromatic aberration correction. Information. Hereinafter, the coordinates of R and B after the coordinate correction by the lateral chromatic aberration correction are R (x _r , y _r ) and B (x _b , y _b ), respectively.

Next, in step S803, the distortion coordinate correction unit 402 performs coordinate correction by distortion aberration correction on the coordinates R (x _r , y _r ), G (x, y), and B (x _b , y _b ). . In the present embodiment, the coordinate correction is performed based on lens information as in step S802. The lens information used here is information relating to the shift amount of each pixel based on the image height from the center of the optical axis, and is used to correct all the R, G, and B coordinates by correcting distortion. Hereinafter, the coordinates of R, G, and B after the coordinate correction by distortion aberration correction are R (x _rd , y _rd ), G (x _d , y _d ), and B (x _bd , y _bd ), respectively.

  In step S804, it is determined whether the currently processed image is the first image (reference image) or a non-reference image. If it is determined that the image is a reference image, the process proceeds to step S805. If the image is a non-reference image, the process proceeds to step S806.

In step S805, the alignment coordinate conversion unit 403 performs coordinate enlargement processing on the reference image. Specifically, for the coordinates R (x _rd , y _rd ), G (x _d , y _d ), and B (x _bd , y _bd ), the x component and y component are expressed using the following equation (2). Enlarge the coordinates by k times.
R (x _rd , y _rd ) · k = R (x _rd · k, y _rd · k)
G (x _d , y _d ) · k = G (x _d · k, y _d · k) (2)
B (x _bd , y _bd ) · k = B (x _bd · k, y _bd · k)

  Here, k represents a parameter for generating a high-resolution image. In the present embodiment, a high-resolution image is generated by expanding the number of pixels by a factor of k in the horizontal direction and k times in the vertical direction. Hereinafter, the coordinates of R, G, and B after the coordinate enlargement are R (x ′, y ′), G (x ′, y ′), and B (x ′, y ′), respectively.

On the other hand, in step S806, the alignment coordinate conversion unit 403 acquires the coordinate conversion coefficients a _{0 to} a ₇ for alignment calculated by the coordinate conversion coefficient calculation unit 203 in step S707 of FIG. 7 from the memory unit 204. The coordinate conversion process is performed on the non-reference image. In this embodiment, the projective transformation formula is used in the coordinate transformation process. However, since the high-resolution image is generated by expanding the number of pixels by k times in the horizontal direction and k times in the vertical direction, the corresponding points must be enlarged by k times. I must. Therefore, the projective transformation formula shown in the following formula (3) is used.

The coordinates (x, y) input here are the coordinates R (x _rd , y _rd ), G (x _d , y _d ), and B (x _bd , y _bd ) output in step S803. In addition, the coordinates of R, G, and B after the coordinate conversion are R (x ′, y ′), G (x ′, y ′), and B (x ′, y ′), respectively.

Next, in step S807, the pixel insertion processing unit 404 performs R, G corresponding to coordinates R (x, y), G (x, y), and B (x, y) before coordinate correction and coordinate conversion processing, respectively. , B pixel values R _pix , G _pix , B _pix of the mosaic image are acquired from the memory unit 204. In step S808, the pixel insertion processing unit 404 determines the coordinates R (x ′, y ′), G (x ′, y ′), B (x ′, y ′) of R, G, and B and the pixel value R. Pixel insertion processing is performed for each of R, G, and B using _pix , G _pix , and B _pix .

Hereinafter, the pixel insertion process will be described with reference to FIG. FIG. 9 is a diagram for explaining pixel insertion processing in an R image.
First, with respect to the super-resolution image shown in FIG. 9A, pixel insertion processing is performed using a plurality of low-resolution images having the number of horizontal pixels W and the number of vertical pixels H, for example, 2 in the horizontal direction and the vertical direction. A high-resolution image in which the number of pixels is doubled is generated. Here, the super-resolution image is an image having a horizontal pixel number of 2 W and a vertical pixel number of 2 H, and is a base image for generating a high-resolution image. In the super-resolution image shown in FIG. 9A, all pixel values include a value of 0 indicating that the pixel is a missing pixel.

  The pixel insertion process refers to a process of inserting the value of the non-missing pixel 901 of the low resolution image into the missing pixel 902 of the super-resolution image as shown in FIG. By performing the pixel insertion process using a plurality of low resolution images, a high resolution image shown in FIG. 9B can be generated. In the pixel insertion process of the present embodiment, a super-resolution image is generated from the first reference image, and a process of replacing a missing pixel of the super-resolution image with a non-missing pixel by the second and subsequent non-reference images. .

Next, details of the pixel insertion processing in step S808 will be described with reference to the flowchart of FIG. FIG. 10 is a flowchart illustrating an example of a specific processing procedure performed by the pixel insertion processing unit 404 in step S808 of FIG. Hereinafter, the R pixel insertion process will be described, but the process is the same for G and B, and the description thereof will be omitted.
First, in step S1001, reference coordinates R ₀ (x ₀ , y ₀ ), which are integer coordinates closest to the coordinates R (x ′, y ′), are calculated. Specifically, the reference coordinates are calculated by the following equation (4).

Next, in step S1002, it is determined whether or not the pixel value R _{pix is} a value other than zero. If it is determined that the value of the pixel values R _pix is not zero, since the pixel value R _pix is non-missing pixel, the process proceeds to step S1003. On the other hand, when the value of the pixel value R _pix is 0, the pixel value R _pix is a missing pixel, and thus the process proceeds to step S1004.

In step S1003, a distance d (R) between the coordinate R (x ′, y ′) and the reference coordinate R ₀ (x ₀ , y ₀ ) is calculated. The distance d (R) calculated in this process is used in subsequent determinations in steps S1008 and S1010.

Here, the distance d (R) will be described with reference to FIG. FIG. 11 is a diagram illustrating the positions of the coordinates R (x ′, y ′) and the reference coordinates R ₀ (x ₀ , y ₀ ) with respect to the super-resolution image R. The reference coordinate R ₀ (x ₀ , y ₀ ) that is closest to the coordinate R (x ′, y ′) is a coordinate that indicates the center of the pixel of the super-resolution image R, and the reference coordinate R ₀ (x ₀ , A distance d (R) is calculated as an index for evaluating how close the coordinates R (x ′, y ′) are to y ₀ ). Then, as will be described later, it is determined whether or not to insert a pixel using the value of the distance d (R). The distance d (R) is calculated by the following equation (5).

On the other hand, in step S1004, since the pixel value R _pix is 0, it is determined that the pixel is a missing pixel. Therefore, the maximum value that can be expressed in the distance d (R) is input as an exception process. For example, when the distance d (R) can be expressed only up to 8 bits, 255 is input as a value.

  Next, in step S1005, it is determined whether the image currently being processed is the second or later non-reference image or the first reference image. As a result of this determination, if it is the second or subsequent non-reference image, the process proceeds to step S1008. If it is the first reference image, the process proceeds to step S1006.

In step S1006, the distance d (R) is stored in the memory unit 204 as the distance d ₀ (R) of the reference coordinates R ₀ (x ₀ , y ₀ ). Here, the distance d ₀ (R) is used for comparison with the value of the distance d (R) calculated in the second and subsequent non-reference images, and the distance d _{0 is determined as} needed depending on the value of the distance d (R). Update the value of (R). In step S1007, the pixel value R _pix is stored in the memory unit 204 as the pixel value R _pix0 of the reference coordinate R ₀ (x ₀ , y ₀ ). First, a super-resolution image R is newly created using only the first reference image using the pixel value R _pix0 , and the super-resolution image R is updated by the second and subsequent pixel insertion processes in step S1012 described later. I will do it.

On the other hand, in step S1008, it is determined whether the distance d (R) is less than a preset threshold value T _H (R). The result of this determination, the distance d when (R) is less than the threshold T _H (R) the process proceeds to step S1009, the threshold T _H (R) it can be advantageously utilized, the coordinates that are currently targeted for processing (x, y ) Ends the pixel insertion process.

In step S1009, the distance d ₀ (R) stored as the distance of the reference coordinates R ₀ (x ₀ , y ₀ ) is read from the memory unit 204. In step S1010, it is determined whether or not the value of the distance d (R) is smaller than the distance d ₀ (R). As a result of the determination, if the distance d ₀ (R) is smaller than the flow proceeds to step S1011, the distance d ₀ when (R) or in which the coordinates are the object of the current process (x, y) the pixel insertion process finish.

In step S1011, the value of the distance d ₀ (R) is updated to the distance d (R) and stored in the memory unit 204. In step S1012, the pixel value R _pix corresponding to the distance d (R) is updated as the pixel value R _pix0 and stored in the memory unit 204, as in step S1011. After saving in the memory unit 204, the pixel insertion processing for the currently processed coordinates (x, y) is terminated.

  Returning to the description of FIG. 8, in step S809, it is determined whether or not the processing from step S802 to step S808 has been performed on all coordinates of the image currently being processed. As a result of the determination, if all the coordinates are not processed, the process proceeds to step S810, and if all the coordinates are processed, the process proceeds to step S811. In step S810, the coordinates R (x, y), G (x, y), and B (x, y) currently processed are updated to the next coordinates, and the process returns to step S802.

  On the other hand, in step S811, it is determined whether or not the image currently being processed is the last Nth image. As a result of this determination, if it is the last Nth image, the process proceeds to step S812, and if it is not the last Nth image, the process proceeds to step S813.

  In step S812, the missing interpolation processing unit 405 performs missing interpolation processing for the super-resolution images R, G, and B. The missing interpolation process is a process of replacing pixels that are still missing pixels and have a value of 0 in R, G, and B super-resolution images with pixel values that have undergone linear interpolation from surrounding non-missing pixels. . After the processing in step S812, all the pixels of the R, G, B super-resolution image are composed of non-missing pixels. In this embodiment, linear interpolation is used for the missing interpolation processing, but adaptive interpolation that considers the direction of the edge may be used.

  In addition, before performing the missing interpolation process, the ratio of the missing pixels occupying the image may be calculated to control the process. FIG. 12 is a diagram illustrating an example of a super-resolution image in which the number of pixels is expanded k times in both the horizontal direction and the vertical direction with respect to the input low-resolution image. In FIG. 12, pixels shown in white are non-missing pixels, and gray portions are missing pixels. As shown in FIG. 12, if there are no non-missing pixels of 50% or more with respect to the total number of pixels of the super-resolution image, the super-resolution effect is accurately applied to the missing pixels with the edge direction taken into account. Therefore, the super-resolution effect is reduced. Therefore, the ratio shown in the following formula (6) is calculated.

  Here, W and H represent the number of horizontal pixels and the number of vertical pixels of the input low-resolution image, respectively. K represents the magnification of the super-resolution image with respect to the low-resolution image. This ratio is calculated for all of the super-resolution images R, G, and B. If even one of the ratios is a predetermined value (for example, 50%) or more, it is output as an error and the super-resolution processing is stopped. Or you may return to step S801 and perform the process which produces | generates a k / 2 times high resolution image. When outputting as an error, an image obtained by enlarging R, G, B of the first reference image k times in the horizontal direction and the vertical direction may be output.

  In step S813, the R, G, and B super-resolution images are stored in the memory unit 204, and the super-resolution processing ends.

  Returning to the description of FIG. 5, in step S <b> 507, it is determined whether or not the current image to be processed is the Nth last image. As a result of this determination, if the image currently being processed is the last image, the process proceeds to step S509. If not, the process proceeds to step S508. In step S508, the image is updated to the next image, and the process returns to step S504.

On the other hand, in step S509, the YUV conversion unit 206 reads the created super-resolution images R, G, and B from the memory unit 204, and creates a high-resolution YUV image. When converting from an RGB signal to a YUV signal, conversion is performed according to the following equation (7) by a known method. Then, the super-resolution processing is terminated using the high-resolution YUV image as an output signal.
Y = 0.299R + 0.587G + 0.114B
U = −0.169R−0.331G + 0.500B (7)
V = 0.500R-0.419G-0.081B

  As described above, according to the present embodiment, the lateral chromatic aberration correction and distortion correction related to the optical characteristics are performed only on the coordinates, and after alignment, the pixel values of the low-resolution image before these corrections are converted to the high resolution. Added to the image. As a result, it is possible to generate a high-resolution image in which the lateral chromatic aberration and distortion are corrected without reducing the frequency band of the low-resolution image.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to the drawings. Also in this embodiment, the calculation method of the coordinate conversion coefficient is different, but using a plurality of Bayer array low-resolution images, a high-resolution image in which the number of pixels is expanded k times in each of the horizontal direction and the vertical direction is generated. Realize resolution. Note that the difference from the configuration of the first embodiment is only the configuration in the coordinate conversion coefficient calculation unit 203 of FIG. 2, and the other configurations are the same as those of the first embodiment. The overall processing procedure is the same as in FIG. Only differences from the first embodiment will be described below.

FIG. 13 is a block diagram illustrating a specific configuration example of the coordinate conversion coefficient calculation unit 203 according to the present embodiment.
In FIG. 13, the coordinate conversion coefficient calculation unit 203 includes a thinning processing unit 1301, a distortion correction processing unit 1302, and a projective conversion coefficient calculation unit 1303. Hereinafter, the specific process of step S505 performed by each configuration illustrated in FIG. 13 will be described with reference to the flowchart of FIG.

  In the first embodiment, distortion correction is performed on an image that has been demosaiced from a G mosaic image, a G image for alignment is created, and then the G image is used to align the images. The coordinate conversion coefficient for performing the calculation was calculated. On the other hand, in the present embodiment, with the memory and processing time in mind, a coordinate conversion coefficient for performing alignment between images is obtained by using a G image having a resolution lower than that of the G image subjected to demosaic processing. calculate.

FIG. 14 is a flowchart illustrating an example of a specific processing procedure by the coordinate conversion coefficient calculation unit 203 in step S505 of FIG.
First, in step S <b> 1401, the thinning processing unit 1301 acquires a G mosaic image to be processed from the memory unit 204. In step S1402, 1/2 thinning processing is performed on the G mosaic image acquired in step S1401.

  Here, the half-thinning process will be described with reference to FIG. FIG. 15A is a diagram illustrating an example of a G mosaic image having the number of horizontal pixels W and the number of vertical pixels H acquired in step S1401. As shown in FIG. 15A, the pixels described as G1 and G2 are G pixels. By acquiring only G1 pixels surrounded by a thick line from this mosaic image, as shown in FIG. 15B, a thinned image reduced to a horizontal pixel number W / 2 and a vertical pixel number H / 2 is obtained. Generated. Note that the pixel at the position G1 may be acquired to generate a thinned image, or the pixel at the position G2 may be acquired to generate a thinned image.

  Next, in step S1403, the distortion correction processing unit 1302 performs a distortion correction process on the G image that has undergone the thinning process in step S1402. The distortion correction method may be a known method such as correction by image deformation using lens information. However, the number of horizontal pixels and the number of vertical pixels are both halved and the pixel of the mosaic image acquired in step S1401. Correction is performed while considering the position. The processes in the next steps S1404 and S1405 are the same as the processes in steps S704 and S705 in FIG. 7, respectively.

  In the next step S1406, the projective transformation coefficient calculation unit 1303 detects the amount of misalignment between the images using the two alignment G images aligned in step S1405. At this time, as in step S706 in FIG. 7, the amount of positional deviation of the alignment G image to be processed corresponding to the first alignment G image is detected at a plurality of locations. In the method of detecting the amount of misalignment, after performing pattern matching based on the SAD value, the amount of misalignment is detected to a decimal accuracy using equiangular straight line fitting or parabolic fitting. In addition, said method is disclosed by patent document 2 grade | etc., For example.

  Next, in step S1407, the projective transformation coefficient calculation unit 1303 expands the motion vector to the resolution equal to the positional deviation amount detected in step S1406. In this embodiment, since the alignment G image thinned out to ½ is used, the x and y components of the motion vector are doubled. In step S1408, the projective transformation coefficient calculation unit 1303 calculates a coordinate transformation coefficient for alignment using the plurality of motion vectors expanded in step S1407. The calculation method of the coordinate conversion coefficient is the same as that in step S707 in FIG. Note that the processes in steps S1409 and S1410 are the same as steps S708 and S709 in FIG. 7, respectively.

  As described above, according to the present embodiment, even when the G image for alignment is generated by performing the half-thinning process, the chromatic aberration of magnification and distortion are not reduced without reducing the frequency band of the low-resolution image. Can be generated.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

401 Chromatic Aberration Coordinate Correction Unit 402 Distorted Coordinate Correction Unit 403 Position Coordinate Conversion Unit 404 Image Insertion Processing Unit

Claims (7)

An image processing apparatus that generates a high-resolution image using a plurality of low-resolution images,
Coordinate conversion means for performing coordinate conversion based on the optical characteristics of each color component for each coordinate of the low resolution image,
Detecting means for detecting a displacement amount between the plurality of low-resolution images;
Alignment means for aligning the low-resolution image coordinate-transformed by the coordinate transformation means according to the amount of displacement detected by the detection means;
An insertion unit that generates the high-resolution image by inserting a pixel value of the low-resolution image based on a result of the alignment by the alignment unit ;
The detection means further includes a generation means for generating an image for alignment after coordinate conversion by the coordinate conversion means after reducing the low-resolution image,
An image processing apparatus for detecting a positional shift amount by using an alignment image generated by the generating means . The image processing apparatus according to claim 1 , wherein the coordinate conversion unit performs coordinate conversion related to distortion. The image processing apparatus according to claim 2 , wherein the coordinate conversion unit further performs coordinate conversion related to lateral chromatic aberration. It said alignment means, an image processing apparatus according to any one of claim 1 to 3, wherein the aligning performs projective transformation. The said insertion means interpolates from a surrounding pixel with respect to the pixel in which the pixel value of the said low resolution image in the said high resolution image is not inserted, The any one of Claims 1-4 characterized by the above-mentioned. Image processing apparatus. The insertion unit may stop the processing or generate an image having a lower resolution than the high-resolution image when the ratio of the pixel values of the low-resolution image inserted is smaller than a predetermined value. the image processing apparatus according to any one of claims 1-5. An image processing method for generating a high-resolution image using a plurality of low-resolution images,
A coordinate conversion step for performing coordinate conversion based on the optical characteristics of each color component for each coordinate of the low-resolution image;
A detection step of detecting a displacement amount between the plurality of low resolution images;
In accordance with the amount of displacement detected in the detection step, an alignment step for aligning the low resolution image coordinate-converted in the coordinate conversion step;
An insertion step of generating the high resolution image by inserting a pixel value of the low resolution image based on a result of the alignment in the alignment step ;
The detection step further includes a generation step of generating an image for alignment that has undergone coordinate transformation by the coordinate transformation step after reducing the low-resolution image,
An image processing method, comprising: detecting a positional deviation amount using an alignment image generated in the generation step .

Images