JP6579859B2 - Imaging apparatus, image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an imaging device, an image processing device, an image processing method, and a program.

  In an imaging apparatus using a single-plate sensor in which color filters such as a Bayer array are arranged, color signals that can be generated for each pixel at the time of imaging differ, and aliasing (distortion) due to the arrangement of the color filters occurs. Image quality may be impaired. In addition, there has been proposed an imaging apparatus that can divide the exit pupil of the photographing lens into a plurality of regions and simultaneously photograph a plurality of parallax images corresponding to the divided pupil regions. For example, Patent Document 1 includes one microlens and a plurality of photoelectric conversion units (hereinafter referred to as subpixels) for each pixel, and any one of RGB color filters is arranged in the subpixel. An imaging apparatus using a two-dimensional imaging element has been proposed.

Japanese Patent No. 5359465

  However, in Patent Document 1, in the case where a positional shift due to parallax occurs between sub-pixel signals in one pixel, when generating a color signal using the sub-pixel, color misalignment or a resolution feeling due to interpolation is detected. There is a problem that a decrease occurs.

  An object of the present invention is to provide an image processing apparatus, an image processing method, a program, and an imaging apparatus that can generate a high-quality image signal.

The imaging device of the present invention has a pixel array composed of a plurality of pixels, and a plurality of color filters regularly arranged on the pixel array, and each pixel captures one microlens and a subject image. an imaging device having a plurality of photoelectric conversion unit for photoelectrically converting, in each of the plurality of image generation position for generating the output image, within a predetermined range from the position on the pixel array corresponding to the image generation position Color information corresponding to the color filter is set by weighted addition of a signal from a certain photoelectric conversion unit with a weight corresponding to the distance from the position on the pixel array corresponding to the image generation position to each photoelectric conversion unit image processing means for, characterized that you have a.

  According to the present invention, it is possible to provide an image processing device, an image processing method, a program, and an imaging device that can generate a high-quality image signal.

It is a block diagram which shows an example of a structure of the imaging device of this invention. (Examples 1, 2, and 3) It is a schematic diagram for demonstrating the pixel arrangement | sequence of the image pick-up element shown in FIG. (Examples 1, 2, and 3) It is a schematic diagram which shows the divided pupil area | region in the imaging optical system shown in FIG. (Examples 1, 2, and 3) It is the schematic diagram which showed a mode that the light beam which passes through the divided pupil area | region shown in FIG. 3 injects. (Examples 1, 2, and 3) FIG. 2 is a block diagram of color signal generation processing means in the camera signal processing circuit shown in FIG. 1. (Examples 1, 2, and 3) It is the schematic diagram which showed the sampling position of the subpixel, and the sampling position of the output image. (Examples 1, 2, and 3) It is the schematic diagram which showed the G subpixel and the sampling position of the output image. (Examples 1, 2, and 3) It is the schematic diagram which showed the sampling position of R subpixel and an output image. (Examples 1, 2, and 3) It is the schematic diagram which showed the B subpixel and the sampling position of the output image. (Examples 1, 2, and 3) It is a schematic diagram which shows the effect of a folding reduction process with the imaging device of this invention. (Example 1) It is a schematic diagram which shows the mode of the folding | turnback in the conventional imaging device. It is a block diagram of the camera signal processing circuit shown in FIG. (Example 2) It is a schematic diagram which shows the sampling position when the phase difference between subpixel images is small. (Example 2) It is a flowchart which shows the signal processing method which the camera signal processing circuit shown in FIG. 12 performs. (Example 2) It is a block diagram of the camera signal processing circuit shown in FIG. (Example 3) It is a flowchart which shows the signal processing method which the camera signal processing circuit shown in FIG. 15 performs. (Example 3) It is a figure for demonstrating S1600 of FIG. (Example 3)

  FIG. 1 is a block diagram illustrating an example of the configuration of an imaging apparatus according to an embodiment of the present invention. The imaging apparatus 100 may be a camera body in a single-lens reflex camera or a non-reflex camera (mirrorless camera), or may be a lens-integrated type. The imaging apparatus 100 may be a digital still camera, a digital video camera, a television camera, or the like.

  In FIG. 1, an imaging apparatus 100 includes an imaging optical system 101, an imaging element 102, a camera signal processing circuit 103, a frame memory 104, a drive control means 105 for the imaging optical system 101, a display unit 106, a recording unit 107, and an operation system. 108 and system control means 109. The camera signal processing circuit 103, the frame memory 104, the display unit 106, the recording unit 107, and the operation system 108 are connected to each other via a bus line 110.

  The imaging optical system 101 forms a subject image. That is, the imaging optical system 101 includes optical elements such as a focus lens, a zoom lens, a diaphragm, and a camera shake correction lens, and takes in incident light (image light) from a subject and forms an image on the imaging surface of the imaging element 102. . The optical element of the imaging optical system 101 is driven and controlled based on a control signal from the drive control means 105. The drive control unit 105 is controlled by the system control unit 109. The focus lens is moved in the optical axis direction to adjust the focus. In focus adjustment, the image sensor 102 may be moved in the optical axis direction. The driving means for the focus lens and the image sensor 102 can be constituted by an actuator such as a DC motor, a stepping motor, or a voice coil motor. The drive control means 105 controls the drive of the drive means.

  The image sensor 102 is an image sensor composed of a two-dimensional CMOS photosensor and a peripheral circuit, and is disposed on the image forming surface of the image forming optical system. The imaging element 102 converts the amount of incident light imaged on the imaging surface by the imaging optical system 101 into an electrical signal in units of pixels and outputs it as a pixel signal. As described above, the image sensor 102 photoelectrically converts the subject image formed by the imaging optical system 101, and the detailed structure thereof will be described later.

  The camera signal processing circuit 103 temporarily stores, for example, an image signal for one frame output from the image sensor 102 in the frame memory 104, and performs various signal processing (white balance, gamma) on the stored image signal. Correction, demosaic processing, etc.). The camera signal processing circuit 103 may be composed of a digital signal processor (DSP). In general, demosaic processing interpolates missing color information (luminance information, pixel value information) with respect to a signal obtained from an image sensor, so that each pixel has a plurality of colors set in a color filter. Color interpolation processing for setting color information for all.

  The image processing method (signal processing method) of this embodiment may be performed inside the imaging apparatus 100, or a dedicated image processing device (signal processing device) or a program of the image processing method of this embodiment is installed. It may be performed outside the imaging apparatus 100 such as a personal computer. Such an image processing apparatus, an image processing method, and a program also constitute one aspect of the present invention. At this time, each step of the image processing method may be executed by software by a computer or by dedicated hardware.

  The display unit 106 includes a panel type display unit such as a liquid crystal display unit, and displays a moving image or a still image captured by the image sensor 102. The image processing method of the present invention may be applied when the display unit 106 generates a live view image that sequentially displays images processed by the image processing unit. The recording unit 107 records a moving image or a still image captured by the image sensor 102 on a recording medium such as an SD card.

  The operation system 108 includes buttons, dials, levers, touch panels, and the like that issue various operation commands in accordance with user operations. The operation system 108 includes a focus selection switch. Such a focus selection switch (focus selection means) can select a manual focus (MF) mode, an autofocus (automatic focus adjustment, AF) mode, and a defocus mode. In the MF mode, the user operates a focus ring (not shown) to move the focus lens, and manually performs focus adjustment. In the AF mode, focus detection is performed using a phase difference detection method, a contrast detection method, or the like, and the focus lens is automatically moved to the in-focus position to perform focus adjustment. In the defocus mode, the focus lens is automatically moved to a position defocused by a predetermined amount from the in-focus position. In the defocus state, the effect of the present embodiment described later can be obtained. However, the focus lens often does not completely coincide with the focal position even in the AF mode due to a manufacturing error or the like, and the effect of the present embodiment described later can be obtained. There is a case.

  The system control unit 109 is a control unit that controls the operation of each unit in the imaging apparatus 100. The system control unit 109 may control processing of the camera signal processing circuit 103 for the entire image (entire screen).

  Next, the configuration of the image sensor 102 and the process for acquiring the pupil-divided image will be described. The image sensor 102 includes a pixel array including a plurality of pixels, a plurality of color filters regularly arranged on the pixel array, and a plurality of microlenses. Each pixel includes a plurality of sub-pixels as photoelectric conversion units that receive light beams that have passed through different pupil regions of the imaging optical system 101 and photoelectrically convert a subject image formed by the imaging optical system 101. .

  FIG. 2 shows a part of the pixel array of the image sensor 102. FIG. 2A is a schematic diagram of the imaging surface of the imaging element 102 as viewed from above. A plurality of pixels 200, 201, 202, and 203 are signals of the same color component corresponding to one pixel of the output image. Is generated. The pixel outputs an image signal in the horizontal direction and the vertical direction at a sampling period of ΔX. ΔX is an interval larger than the subpixel interval (ΔX / 2). Reference numerals 300 and 302 denote the center positions of the corresponding pixels.

  Each of the pixels 200, 201, 202, and 203 is provided with one microlens, and FIG. 2 shows the microlens as a circle. Each pixel has a total of four subpixels, two in the horizontal direction and two in the vertical direction. With such a configuration, by reading out signals picked up for each sub-pixel, a plurality of regions (shown by broken lines) pupil regions A, B, which are obtained by dividing the exit pupil of the imaging optical system 101 shown by solid lines in FIG. The parallax images corresponding to C and D can be taken simultaneously. One of the R, G, and B color filters is disposed in the sub-pixel, and the same color filter is disposed in the sub-pixel associated with the common microlens. For example, a Bayer array color filter shown in FIG. 2B is arranged.

  Next, with reference to FIG. 4, the sampling position of the subject image sampled by the sub-pixel at the time of imaging will be described. 4A and 4B are schematic diagrams illustrating a state in which light beams 400 (solid lines) and 401 (broken lines) passing through the divided pupil regions are incident on the imaging surface 30 of the imaging element 102. Light beams 400 and 401 indicate light beams that pass through pupil regions A and D, respectively.

  Here, when the light from the subject forms an image on the imaging surface 30 of the image sensor 102, as shown in FIG. 4A, the light beams 400 and 401 that enter the image sensor 102 through the divided pupil regions. Gather at one point on the imaging surface 30. Therefore, as shown in FIG. 4C, the centroids of the sub-pixels corresponding to the divided pupil regions A, B, C, and D acquired by the image sensor 102 coincide. In addition, the direction E of FIG. 4 (A) and (C) respond | corresponds, and the direction E of FIG. 4 (B) and (D) corresponds. The positions at which the sub-pixels Ra, Rb, Rc, and Rd in the pixel 200 shown in FIG. 2B receive the light beam coincide with the center position 300 of the pixel 200 in FIG. The positions at which the sub-pixels Ba, Bb, Bc, and Bd in the pixel 203 shown in FIG. 2B receive the light beam coincide with the center position 302 of the pixel 203 in FIG. As described above, the position where the sub-pixel generates an image may be referred to as a “sampling position” or a “centroid position”.

  In practice, however, the image is often taken at a position deviated from the in-focus position due to a control error of the imaging optical system 101 or the like. In this case, the light beams that have passed through the divided pupil regions and are incident on the image sensor 102 do not converge at one point, as shown in FIG.

  In this case, as shown in FIG. 4D, the position of the center of gravity of the sub-pixel is shifted from the position of the center of gravity of the pixel. For example, the positions 311 to 314 where the sub-pixels Ra, Rb, Rc, and Rd receive the light flux have deviation amounts within ΔX / 2 from the center position 300 of the pixel in the vertical and horizontal directions, respectively. Hereinafter, the amount of deviation between the center-of-gravity position of the sub-pixel and the center position of the pixel may be referred to as “phase difference”.

  As described above, when the subject image is acquired by dividing the pupil, and when the image is not completely focused at the time of imaging, the sub-pixels are individually read out to obtain an interval smaller than the pixel interval ΔX (for example, ΔX / 4). It becomes possible to sample a subject image. The camera signal processing circuit 103 generates (sets) all the RGB color signals at a position different from the center of the pixel, using the signal oversampled by the sub-pixel.

  FIG. 5 is a block diagram showing a configuration of color signal generation processing means (demosaic processing means) 1030 included in the camera signal processing circuit 103. The color signal generation processing unit 1030 sets a plurality of image generation positions in the pixel array, selects a sub-pixel to be used from among the plurality of pixels for each image generation position, and selects from the selected sub-pixel. All color information is set based on the signal. The image generation position is a position set in the pixel array in order to generate an output image, and may be hereinafter referred to as “output image sampling position”. Conventionally, the sampling position of the output image is the center position of each pixel.

  When the present invention is applied to an image processing apparatus, the image processing apparatus includes a unit for setting the plurality of image generation positions, a unit for selecting the sub-pixel, and a unit from the selected sub-pixel. And means for setting all color information based on the signal. When the present invention is applied to an image processing method, the image processing method includes the steps of setting the plurality of image generation positions, selecting the subpixels, and selecting from the selected subpixels. Setting all color information based on the signal.

  The color signal generation processing unit 1030 includes an R signal interpolation processing unit 1031, a B signal interpolation processing unit 1032, and a G signal interpolation processing unit 1033 inside.

  FIG. 6 is a schematic diagram illustrating a sampling position of a sub-pixel output from the image sensor 102 and a sampling position of an output image generated by the color signal generation processing unit 1030 using the sub-pixel. For example, the pixel 250 in the upper left corner of FIG. 6 includes four subpixels 251 to 254, and the gravity center positions of the subpixels 251 to 254 are 261 to 264.

  In FIG. 6, a sampling position indicated by a black circle is a sampling position of a sub-pixel output from the image sensor 102, and a sampling position indicated by a white circle is a sampling position of an output image. Since any one of R, G, and B color filters is arranged in each sub-pixel, the color signal that can be acquired differs depending on the position. At the position of the white circle, RGB signal interpolation processing means 1031, B signal interpolation processing means 1032, and G signal interpolation processing means 1033 respectively perform RGB color interpolation processing using color information of neighboring subpixels.

  Here, the arrangement of the color filters arranged in the neighboring subpixels differs depending on the sampling positions (white circle positions) P00, P01, P10, and P11 of the output image. For example, the arrangement of the color filters in the subpixel region of 4 × 4 pixels centered on P00 is 2 × 2 as one unit, and R, G, G, B in the order of upper left, upper right, lower left, and lower right. Become. The arrangement of the color filters in the 4 × 4 pixel sub-pixel region centered on P01 is G, R, B, and G in the order of upper left, upper right, lower left, and lower right when 2 × 2 is taken as one unit. The arrangement of the color filters in the 4 × 4 pixel sub-pixel region centered on P10 is G, B, R, and G in the order of upper left, upper right, lower left, and lower right when 2 × 2 is taken as one unit. The arrangement of the color filters in the 4 × 4 pixel sub-pixel region centered on P11 is B, G, G, and R in the order of upper left, upper right, lower left, and lower right when 2 × 2 is taken as one unit.

  Therefore, the color signal generation processing unit 1030 corresponds to the signal (color information) from the subpixel to be used and the distance between the sampling position and the subpixel according to the sampling positions P00, P01, P10, and P11 of the output image. Color information is set based on the weight.

  FIG. 7 is a schematic diagram showing the G subpixel used by the G signal interpolation processing unit 1033 and the sampling position of the output image.

  FIG. 7A shows an example in the case of the output image sampling position P00 or P11. For example, when the G signal is generated at the position 600, the G subpixels 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 612, and 613 are referred to according to the distance from the position 600. The weighted addition is performed according to the weight.

  FIG. 7B shows an example of the output image sampling position P01 or P10. For example, when the G signal is generated at the position 620, the G subpixels 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, and 632 are referred to, and depending on the distance from the position 620 The weighted addition is performed according to the weight.

  FIG. 8 is a schematic diagram showing the R subpixels used by the R signal interpolation processing means 1031 and the sampling position of the output image. In FIG. 8, positions 700, 701, 702, and 703 correspond to the sampling positions P00, P01, P10, and P11 of the output image, respectively. When R signals are generated at positions 700, 701, 702, and 703, R subpixels 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725 is referred to. Then, weighted addition is performed with weights corresponding to the distances from the positions 700, 701, 702, and 703, respectively.

  FIG. 9 is a schematic diagram showing the B subpixels used by the B signal interpolation processing means 1032 and the sampling position of the output image. In FIG. 9, positions 800, 801, 802, and 803 correspond to sampling positions P00, P01, P10, and P11 of the output image, respectively. When the B signal is generated at the positions 800, 801, 802, and 803, the B sub-pixels 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, See 825. Then, weighted addition is performed with weights corresponding to the distances from the positions 800, 801, 802, and 803, respectively.

  Thus, the camera signal processing circuit 103 uses color information obtained from a part of subpixels (for example, the subpixel 601) included in one pixel in the demosaic processing for G (for example, the subpixel 601 belongs). Do not use color information obtained from the remaining subpixels (in the pixel). The camera signal processing circuit 103 selects subpixels (for example, subpixels 601 to 613) within a predetermined range from the set sampling position of the output image in the demosaic processing for each of R, G, and B. The camera signal processing circuit 103 also outputs the output image based on the color information of each sub-pixel selected by the selection unit and the weight according to the distance from the sampling position of the output image to each selected sub-pixel. Set color information at the sampling position.

  Finally, with reference to FIGS. 10 and 11, effects of the image pickup apparatus 100 of the present embodiment will be described. In general, in an image sensor having a Bayer array color filter, G is a checkered pattern, and R and B are sampled at a pixel density of 1/4 with respect to G, as shown in FIG. To do. At this time, as shown in FIG. 4B, the gravity center positions of the four sub-pixels are in the center position of the pixel. Therefore, as shown in FIG. 11B, aliasing components of R, G, and B are generated in the band of the Nyquist frequency fn determined from the pixel pitch ΔX of the image sensor 102, and the image quality is impaired. In FIG. 11B, the horizontal axis represents the frequency in the horizontal direction (H), and the vertical axis represents the frequency in the vertical direction (V). White circles represent positions where G signal folding occurs, and hatched circles represent positions where R signal and B signal folding occur.

  However, the imaging apparatus 100 oversamples the subject image with the sub-pixels as shown in FIG. That is, as shown in FIG. 4D, the number of the gravity center positions of the sub-pixels is doubled in the horizontal direction and the vertical direction. Therefore, as shown in FIG. 10B, it is possible to reduce the aliasing components of R, G, and B in the band of the Nyquist frequency fn determined from the pixel pitch ΔX of the image sensor 102, and a high-quality video signal Can be output. The definition in FIG. 10B is the same as that in FIG.

  In the present embodiment, for a specific color (G), a signal obtained from a part of subpixels (may be one subpixel) included in one pixel without using color information of the entire pixel ( Pixel value information) is used, and signals (pixel value information) obtained from the remaining subpixels are not used. As a result, the signal obtained from the sub-pixel far from the sampling point of the output image is not used in the demosaic process, and the image quality of the output image is improved. It should be noted that some of the subpixels need not be all of the subpixels. In this embodiment, the sampling position of the output image is shifted from the center of each pixel, but this is not essential.

  The configuration of the imaging apparatus according to the second embodiment is the same as that illustrated in FIG. 1, but the configuration of the camera signal processing circuit 103 is different from that illustrated in FIG.

  FIG. 12 is a block diagram showing the configuration of the camera signal processing circuit 103 of this embodiment. The camera signal processing circuit 103 includes the color signal generation processing unit 1030, the correlation calculation unit 1034, and the control unit 1035, but the system control unit 109 may serve as the correlation calculation unit 1034 and the control unit 1035. Further, the correlation calculation means 1034 and the control means 1035 may be integrated. The control unit 1035 controls the processing of the color signal generation processing unit 1030 for the entire image based on the output of the correlation calculation unit 1034. “With respect to the entire screen” means that the sampling point of the output image is generated by the color signal generation processing unit 1030 over the entire screen. The present embodiment is different from the first embodiment in that the processing in the color signal generation processing unit 1030 is controlled based on the absolute value of the phase difference between the sub-pixel images acquired by the image sensor 102.

  In the imaging apparatus, when the subpixel signal is acquired by the image sensor 102, the sampling position of the subpixel signal is set in the subpixel area sharing the microlens as shown in FIG. 13 due to the control error of the imaging optical system 101. In some cases, the image is taken with a bias toward the center. In such a case, since the correlation of the signals between the sub-pixels of the same color is high, the sub-pixel signals of the same color are added to generate an image signal of a Bayer array with a pixel interval ΔX, and then demosaic processing is performed. An RGB signal is generated at each pixel position. On the other hand, when the sub-pixel signal is acquired by the image sensor 102, if the sub-pixel signal sampling position is as shown in FIG. 6, the method described in the first embodiment is used to obtain each pixel position at the pixel interval ΔX. An RGB signal is generated.

  FIG. 14 is a flowchart showing an image processing method (signal processing method) executed by the camera signal processing circuit 103 of this embodiment, and “S” represents a step (process). The flowchart shown in FIG. 14 can be embodied as a program for causing a computer to execute each step.

  First, in S1400, a subpixel image is acquired by reading from the image sensor 102. Next, in S1401, the correlation calculation unit 1034 obtains the phase difference between the sub-pixel images. The correlation calculation means 1034 performs a band-pass filter process for detecting a predetermined frequency for each sub-pixel image. The smaller the difference between the output results of the band-pass filters, the higher the correlation, that is, the smaller the phase difference between the sub-pixel images. Is determined. As described above, the correlation calculation unit 1034 functions as an acquisition unit that acquires information regarding the amount of deviation between the center of gravity position of the sub-pixel and the center position of the corresponding pixel. Of course, the correlation calculation means 1034 may acquire information on the phase difference between two sub-pixels of the same pixel.

  In step S1402, the control unit 1035 determines whether the absolute value of the phase difference detected by the correlation calculation unit 1034 is greater than a predetermined value. When the absolute value of the phase difference is larger than the predetermined value, the process proceeds to S1403, and when it is equal to or smaller than the predetermined value, the process proceeds to S1404.

  In step S1403, as in the first embodiment, the color signal generation processing unit 1030 performs R, R, with a pixel interval of ΔX at a position shifted by ΔX / 2 pixels in the horizontal and vertical directions from the center of gravity of the sub-pixel region. Each color signal of G and B is generated. That is, the control unit 1035 causes the color signal generation processing unit 1030 to set all color information based on the signal from the selected subpixel.

  In step S <b> 1404, the color signal generation processing unit 1030 generates one of R, G, and B color signals in a Bayer array at the center of gravity of the subpixel area. After that, in S1405, demosaic processing is performed by referring to the R, G, and B color signals arranged in the Bayer array, and R, G, and B color signals are generated at a pixel interval of ΔX. That is, the control unit 1035 causes the color signal generation processing unit 1030 to select a pixel to be used from among a plurality of pixels for each of a plurality of image generation positions, and based on a signal obtained from the selected pixel. To set all color information. In this case, the color information of the entire pixel is used, and only the color information of some subpixels of the pixel is not used.

  As described above, the imaging apparatus according to the present embodiment switches the processing method in the color signal generation processing unit 1030 according to the phase difference between the sub-pixel images acquired by the imaging element 102. Thereby, when the phase difference between the sub-pixel images is large, by referring to the sub-pixel image and over-sampling the subject image, a signal of all RGB colors is generated at each pixel position of the output image, A high-quality video signal can be output. When a landscape is imaged, a part is in focus and the rest is not in focus, and the output image includes a region where the phase difference satisfies the condition of S1402 and a region where the phase difference is not satisfied. In such a case, a determination is made between S1402 and S1403 as to whether the ratio of the area satisfying the condition of S1402 to the entire output image is greater than the threshold, and if greater than the threshold, the process proceeds to S1403. You may transfer to S1404. This is also true for Example 3.

  Next, the schematic configuration of the imaging apparatus according to the third embodiment of the present invention is the same as that shown in FIG. 1, but the configuration of the camera signal processing circuit 103 is different from that shown in FIGS.

  FIG. 15 is a block diagram showing the configuration of the camera signal processing circuit 103 of this embodiment. The camera signal processing circuit 103 includes a color signal generation processing unit 1030, a correlation calculation unit 1034, a control unit 1035, and a main subject region detection unit 1036. The system control unit 109 may also serve as the correlation calculation unit 1034, the control unit 1035, and the main subject area detection unit 1036. Further, two or more of the correlation calculation means 1034, the control means 1035, and the main subject area detection means 1036 may be integrally configured. The main subject area detection means 1036 can use a face recognition means or the like. The control means 1035 controls the processing of the color signal generation processing means 1030 for the entire image based on the outputs of the correlation calculation means 1034 and the main subject area detection means 1036. “With respect to the entire screen” means that the sampling point of the output image is generated by the color signal generation processing unit 1030 over the entire screen.

  The control means 1035 detects the main subject area in advance in order to improve the image quality of the main subject area. When the main subject area is detected, the main subject area is captured at a position shifted from the in-focus position. The driving means is controlled.

  FIG. 16 is a flowchart showing an image processing method (signal processing method) executed by the camera signal processing circuit 103 of this embodiment, and “S” represents a step (step). The flowchart shown in FIG. 16 can be embodied as a program for causing a computer to execute each step.

  First, in S1600, feature analysis of the input image is performed to detect a main subject area. For example, in the input image shown in FIG. 17, it is determined that the area 1700 is the main subject area and the area 1701 is the background area. In step S <b> 1601, the imaging optical system 101 is controlled via the drive control unit 105 so that the phase difference between the sub-pixel images is increased in the main subject area. In step S <b> 1602, a subpixel image including subpixels is acquired by reading from the image sensor 102.

  In step S1603, the correlation calculation unit 1034 performs a band pass filter process for detecting a predetermined frequency for each sub-pixel image, and obtains a phase difference between the sub-pixel images based on a difference between the output results of the band pass filters.

  Next, in step S1604, the control unit 1035 determines whether or not the main subject area. If it is the main subject area, the process proceeds to S1606, and if it is not the main subject area, the process proceeds to S1605. In step S1605, the control unit 1035 determines whether the absolute value of the phase difference between the sub-pixel images is larger than a predetermined value. If the absolute value of the phase difference is larger than the predetermined value, the process proceeds to S1606, and if it is equal to or smaller than the predetermined value, the process proceeds to S1607.

  In step S1606, the color signal generation processing unit 1030 outputs the R, G, and B color signals at ΔX pixel intervals at positions shifted by ΔX / 2 pixels in the horizontal and vertical directions from the center of gravity of the sub-pixel region. Generate. In step S <b> 1607, the color signal generation processing unit 1030 generates any one of R, G, and B color signals in a Bayer arrangement at the barycentric position of the sub-pixel region. Thereafter, in S1608, demosaic processing is performed with reference to each of the R, G, and B color signals arranged in the Bayer array, and R, G, and B color signals are generated at a pixel interval of ΔX.

  According to the imaging apparatus of the present embodiment, the color signal generation processing is switched for each subject area of the input image according to the detection result of the main subject area and the phase difference between the sub-pixel images acquired by the image sensor 102. . Thereby, in the main subject region, imaging is performed so that the phase difference between the sub-pixel images becomes large. When the phase difference between the sub-pixel images is large, by oversampling the subject image with reference to the sub-pixel image, a signal of all RGB colors is generated at each pixel position of the output image, and high quality A video signal can be output.

  In addition, since the sub-pixel of the present invention has a function of photoelectrically converting a subject image, it does not include a focus detection pixel for performing a phase difference detection method on the imaging surface. Therefore, the present invention for performing demosaic processing using signals (pixel value information) obtained from a smaller number of subpixels (ie, some subpixels) than subpixels included in one pixel is used for focus detection. The present invention can also be applied to an image sensor including a pixel and an imaging pixel.

  The present invention can be applied to applications such as an imaging apparatus such as a digital camera and an image processing apparatus. In particular, the present invention is suitable for an imaging apparatus for simultaneously capturing a plurality of parallax images corresponding to divided pupil regions and generating an image signal.

DESCRIPTION OF SYMBOLS 100 ... Imaging device, 101 ... Imaging optical system, 103 ... Camera signal processing circuit (image processing means), 250 ... Pixel, 251-254 ... Subpixel (photoelectric conversion part), P00, P01, P10, P11 ... Output image Sampling position (image generation position)

Claims (10)

A pixel array including a plurality of pixels; and a plurality of color filters regularly arranged on the pixel array, each pixel having a plurality of photoelectric conversions for photoelectric conversion of one microlens and a subject image An image sensor comprising a section;
In each of the plurality of image generation position for generating the output image, the signal of the photoelectric conversion unit from the position on the pixel array corresponding to the image generation position within a predetermined range, corresponding to the image generation position Image processing means for setting color information corresponding to the color filter by performing weighted addition with a weight according to the distance from the position on the pixel array to each photoelectric conversion unit ;
Imaging apparatus characterized that you have a. The imaging apparatus according to claim 1 , wherein a position on the pixel array corresponding to the image generation position is deviated from any central position of the plurality of pixels . The imaging apparatus according to claim 1 , wherein the image processing unit sets all color information of the plurality of colors at each of the plurality of image generation positions . A control means to control the processing of the image processing unit for the whole images,
The plurality of photoelectric conversion units included in each pixel of the image pickup device receive light beams that have passed through different pupil regions of the imaging optical system, and photoelectrically convert a subject image formed by the imaging optical system. There,
The control means includes
Obtain information on the amount of deviation between the position where the photoelectric conversion unit receives the light beam and the center position of the corresponding pixel,
If the absolute value of the previous SL shift amount is greater than a predetermined value, the image processing unit, the signal of the photoelectric conversion unit from the position on the pixel array corresponding to the image generation position within a predetermined range, the image generation By performing weighted addition with a weight according to the distance from the position on the pixel array corresponding to the position to each photoelectric conversion unit, color information corresponding to the color filter is set,
If the absolute value of the previous SL deviation amount is equal to or less than a predetermined value, the image processing means, for each of the plurality of image generation position, to select the pixels used from among the plurality of pixels are selected 4. The imaging apparatus according to claim 1, wherein color information corresponding to the color filter is set based on a signal obtained from the pixel. 5. When the output image includes a region where the absolute value of the shift amount is larger than the predetermined value and a region where the absolute value of the shift amount is less than or equal to the predetermined value, the absolute value of the shift amount If the ratio of the area larger than the predetermined value to the entire output image is larger than the threshold value, the image processing means may have the photoelectric conversion unit within a predetermined range from the position on the pixel array corresponding to the image generation position. The signal is weighted and added with a weight corresponding to the distance from the position on the pixel array corresponding to the image generation position to each photoelectric conversion unit, so that color information corresponding to the color filter is set, and the ratio is if below the threshold, the image processing unit, wherein the plurality of to select a pixel to be used from among the pixel pairs in the color filter based on a signal obtained from the pixels which are selected The imaging apparatus according to claim 4, characterized in that for setting the color information. An imaging optical system for forming the subject image or driving means for driving the imaging device;
Control means for controlling the drive means so that an image is taken at a position shifted from the in-focus position;
The imaging apparatus according to claim 1, wherein the imaging apparatus includes: Has a main object area detecting means to detect a main object area,
The control means controls the driving means so that the main subject area is imaged at a position deviated from a focus position when the main subject area detection means detects the main subject area. The imaging device according to claim 6 . It has a pixel array including a plurality of pixels, and a plurality of color filters which are regularly arranged on the pixel array, and each pixel has a plurality of photoelectric conversion for photoelectrically converting one microlens and the object image An image processing apparatus that processes a signal obtained from an image sensor including a unit,
In each of the plurality of image generation position for generating the output image, the signal of the photoelectric conversion unit from the position on the pixel array corresponding to the image generation position within a predetermined range, corresponding to the image generation position Means for setting color information corresponding to the color filter by weighted addition with a weight according to a distance from a position on the pixel array to each photoelectric conversion unit ;
The image processing apparatus according to claim that you have a. A plurality of photoelectric conversions that have a pixel array composed of a plurality of pixels and a plurality of color filters regularly arranged on the pixel array, each pixel photoelectrically converting one microlens and a subject image An image processing method for processing a signal obtained from an image pickup device including a unit,
In each of the plurality of image generation position for generating the output image, the signal of the photoelectric conversion unit from the position on the pixel array corresponding to the image generation position within a predetermined range, corresponding to the image generation position by weighted addition by weight according to the distance to the photoelectric conversion portion from a position on the pixel array, and setting the color information corresponding to the color filter,
An image processing method characterized that you have a. A program for causing a computer to execute the image processing method according to claim 9 .