JP7039221B2 - Control method of image processing device, image pickup device and image processing device - Google Patents

Description

The present invention relates to an image processing device, an image pickup device, and a control method for the image processing device.

In an image pickup device such as a digital camera, there are known techniques for increasing the number of shots taken during continuous shooting, performing high-speed moving image shooting, and suppressing the battery of a camera when reading a signal. Patent Document 1 discloses a configuration in which pixel thinning drive and readout (resampling) accompanied by addition of pixel signals can be performed to capture an image at a low resolution.

Japanese Unexamined Patent Publication No. 2008-278453 Japanese Unexamined Patent Publication No. 2008-05754

However, in Patent Document 1, the image signal after addition thinning has an alias signal (folding component) having a false frequency, which causes "moire" in which a part of the image is altered, and the quality of the image is deteriorated. .. Further, Patent Document 2 is an image processing apparatus that detects a defocus amount distribution representing distance information from a relative deviation amount of a plurality of subject images in which light flux arriving from different regions of the pupil of a photographing optical system is generated to the subject image. Is disclosed. When the defocus distribution in Patent Document 2 is detected using an image signal containing a folding component, an erroneous detection of the defocus amount occurs depending on the false-resolution subject.

An object of the present invention is to provide an image processing apparatus capable of calculating the influence of resampling on an image signal as a reliability.

In order to solve the above problems, the image processing apparatus of the present invention allocates reliability to the extraction means for extracting a predetermined frequency band from the signal output from the image pickup means and the pixel of interest according to the output of the extraction means. The first reliability generation means, the sampling means for resample the signal output from the image pickup means, the calculation means for calculating the distribution information corresponding to the distance based on the signal output from the sampling means, and the distribution. A second reliability generating means for calculating the reliability of information, an integrated means for integrating the reliability output by the first reliability generating means and the reliability output by the second reliability generating means. When the frequency of the signal output from the image pickup means is higher than the threshold value, the first reliability generation means assigns a lower reliability than when the frequency is lower than the threshold value.

According to the present invention, it is possible to provide an image processing apparatus capable of calculating the influence of resampling on an image signal as a reliability.

It is a block diagram which shows the structure of an image pickup apparatus. It is a figure which shows the arrangement of the pixel of the image pickup part. It is a figure explaining the relationship between the light flux emitted from an exit pupil and the unit pixel of an image pickup part. It is a block diagram which showed the functional structure of an image pickup part and an image processing part. It is a flowchart explaining the operation of the image pickup unit and the image processing unit. It is a figure explaining the horizontal pixel addition thinning process. It is a figure explaining the filter coefficient of addition processing. It is a figure explaining the frequency characteristic of the filter coefficient of luminance addition. It is a figure explaining the subject of the band of the normalized frequency of 0.08. It is a figure explaining the subject of the band of a normalized frequency of 0.26. It is a figure explaining the frequency characteristic of a high-pass filter. It is a figure explaining the frequency characteristic of a bandpass filter. It is a figure explaining the frequency characteristic of a low-pass filter. It is a figure explaining the extraction interval of the wrapping reliability map. It is a figure explaining the image addition process. It is a block diagram which showed the functional structure of an image pickup part and an image processing part.

(First Embodiment)
FIG. 1 is a block diagram showing a functional configuration of the image pickup apparatus 100. The image pickup apparatus 100 is, for example, a digital camera. The image pickup apparatus 100 includes a system control unit 101, a ROM 102, a RAM 103, an optical system 104, an image pickup unit 105, an image processing unit 106, a recording medium 107, and a bus 108. The set of each block except the optical system 104 may be realized as an image processing device.

The system control unit 101 controls the operation of each block included in the image pickup apparatus 100 by reading the operation program of each block included in the image pickup apparatus 100 from the ROM 102, expanding the program into the RAM 103, and executing the operation program. The system control unit 101 is, for example, a CPU (Central Processing Unit). The ROM (Read only memory) 102 is a rewritable non-volatile memory, and stores, in addition to the operation program of each block included in the image pickup apparatus 100, parameters and the like necessary for the operation of each block. The exit pupil distance is also stored in the ROM 102 as lens information necessary for focus detection and the like. The RAM (Random Access Memory) 103 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each block included in the image pickup apparatus 100.

The optical system 104 forms a subject image on the image pickup unit 105. The optical system 104 also includes a diaphragm, and the amount of light at the time of photographing is adjusted by adjusting the aperture diameter of the optical system. The image pickup unit 105 is an image pickup element, and an optical image formed on the image pickup element by the optical system 104 is photoelectrically converted, an A / D conversion process is applied to the obtained analog image signal, and the digital image signal is transferred to the RAM 103. Output and memorize. The image pickup unit 105 is, for example, a CCD sensor, a CMOS sensor, or the like. Further, the image pickup unit 105 of the present embodiment has a configuration capable of performing reading with pixel thinning drive and addition of pixel signals, which is disclosed in Japanese Patent Application Laid-Open No. 2008-278453. Further, the image pickup unit 105 can also apply a filter process such as a high-pass filter or a band-pass filter to the digital image signal after the A / D conversion process.

The image processing unit 106 performs various processing such as white balance adjustment, color interpolation, reduction / enlargement, and filtering, and various image processing such as creation of a defocus map on the image data stored in the RAM 103. The recording medium 107 is a detachable memory card or the like, and an image stored in the RAM 103 and processed by the image processing unit 106, an image A / D converted by the image pickup unit 105, and the like are recorded as recorded images. Bus 108 is used to exchange signals between each block.

FIG. 2 is a diagram showing the arrangement of pixels of the image pickup unit 105. The image pickup device of the image pickup unit 105 includes a plurality of microlenses and a plurality of pixels corresponding to each of the plurality of microlenses. As shown in FIG. 2, unit pixels 200 (one pixel) are two-dimensionally arranged in a matrix in the image pickup unit 105. R (Red) / G (Green) / B (Blue) color filters are arranged on each pixel in a Bayer arrangement. Further, in the unit pixel 200, a sub-pixel a which is a first photoelectric conversion unit and a sub-pixel b which is a second photoelectric conversion unit are arranged, respectively. An FD (photodiode) 201a is arranged in the sub-pixel a, and an FD201b is arranged in the sub-pixel b. The output signals of the sub-pixel a and the sub-pixel b are used for defocus amount detection. Further, the signal (a / b addition signal) obtained by adding the output signals of the sub-pixel a and the sub-pixel b is used to generate an image for photographing.

FIG. 3 is a schematic diagram showing the relationship between the luminous flux emitted from the exit pupil of the optical system 104 and the unit pixel 200. A color filter 301 and a microlens 302 are formed on the unit pixel 200. The optical axis 304 indicates the center of the light flux emitted from the exit pupil 303 with respect to the pixel (unit pixel 200) having the microlens 302. The luminous flux that has passed through the exit pupil 303 is incident on the unit pixel 200 about the optical axis 304.

The pupil region 305 and the pupil region 306 are different pupil regions (partial regions) of the exit pupils. The luminous flux passing through the pupil region 305 is received by the sub-pixel a (FD201a) via the microlens 302. On the other hand, the luminous flux passing through the pupil region 306 is received by the sub-pixel b (FD201b) via the microlens 302. As described above, the sub-pixel a and the sub-pixel b each receive light in different regions of the exit pupil of the photographing lens. Therefore, by comparing the output signal of the sub-pixel a and the output signal of the sub-pixel b, it is possible to detect the defocus amount by the phase difference method.

FIG. 4 is a block diagram for explaining the processing contents of the image pickup unit 105 and the image processing unit 106. The image pickup unit 105 includes an A image / B image acquisition unit (hereinafter referred to as an acquisition unit) 401, a horizontal pixel addition thinning unit (hereinafter referred to as a thinning unit) 402, and a folding component reliability generation unit (hereinafter referred to as a reliability generation unit). It has a 403 and a reliability map extraction unit (hereinafter referred to as an extraction unit) 404. The image processing unit 106 includes an image addition processing unit 405, a defocus amount detection unit (hereinafter referred to as a detection unit) 406, and a reliability map integration unit (hereinafter referred to as an integration unit) 407. FIG. 5 is a flowchart for explaining the operation of the image pickup unit 105 and the image processing unit 106. Hereinafter, description will be given with reference to FIGS. 4 and 5.

In step S501, the acquisition unit 401 acquires the A image Bayer signal and the B image Bayer signal, which are the output signals of the image sensor. The A image Bayer signal is an image signal obtained from the sub-pixel a, and the B image Bayer signal is an image signal obtained from the sub-pixel b. For the following description, the number of pixels of the signal acquired by the acquisition unit 401 is referred to as full size. Further, in the present embodiment, the number of horizontal pixels is 6000 pixels.

In step S502, the thinning unit 402 performs horizontal pixel addition thinning processing on each of the full-size A image Bayer signal and B image Bayer signal acquired in step S501 to generate a pixel signal. The horizontal pixel addition thinning process is an example of resampling. In the present embodiment, the number of horizontal pixels of the full-size Bayer signal is 6000 pixels, but the number of horizontal pixels of the pixel signal becomes 2000 pixels by the horizontal pixel addition thinning process.

FIG. 6 is a schematic diagram for explaining the horizontal pixel addition thinning process. The RG row 601 indicates the RG row in which the R pixel is arranged, and the GB row 602 indicates the GB row in which the B pixel is arranged. In the horizontal pixel addition thinning process, the pixel values are added at the black-painted pixel positions, and the number of pixels after the addition thinning is reduced to 1/3. The number of horizontal pixels when 1/3 addition is thinned out is 6000 × 1/3 = 2000 pixels. The filter coefficient 607 is a coefficient of the 3-pixel addition filter in the R pixel 603 of the RG row 601 and the G pixel 604 of the GB row 602. Further, the filter coefficient 608 is a coefficient of the 3-pixel addition filter in the G pixel 605 of the RG row 601 and the B pixel 606 of the GB row 602. For example, the addition of the pixel values in the R pixel 603 is performed according to the filter coefficient 607.

In step S503, the reliability generation unit 403 generates the reliability for the folded component as a folded reliability map. Specifically, the full-size A image Bayer signal and B image Bayer signal (output signal) are filtered to generate an A image reliability map and a B image reliability map, and two reliability maps are generated. To generate a wrap confidence map.

First, the folding component will be described. The folding component changes according to the number of pixels thinned out by addition. In the present embodiment, the thinning of pixels by addition is performed twice in total by the thinning unit 402 and the image addition processing unit 405. The image addition processing unit 405 adds pixel signals in Bayer units of 2 × 2 pixels to generate a luminance signal. By this processing, the number of pixels is reduced by 1/2 in the horizontal direction and 1/2 in the vertical direction. That is, the full-size Bayer signal acquired in step S501 is added by the thinning unit 402 to become a pixel signal having 2000 horizontal pixels. Further, the pixel signal having 2000 horizontal pixels is added by the image addition processing unit 405, so that the luminance signal (Y) has 1000 horizontal pixels.

FIG. 7 is a diagram illustrating a filter coefficient of the addition process. The filter coefficient 701 is a luminance addition filter coefficient. The filter coefficient 701 is a filter coefficient indicating the addition process (step S505) in the image addition processing unit 405. In the addition process for generating the luminance signal (Y), the pixels subjected to the addition thinning process in the thinning section 402 are further added. For example, in the horizontal RG line 601 the image addition processing unit 405 adds the R pixel 603 and the G pixel 605 after the addition thinning process.
FIG. 8 is a diagram showing the frequency characteristics of the filter coefficient 701. The horizontal axis is the normalized frequency normalized by the sampling frequency, and 0.5 is the Nyquist frequency. The vertical axis is the normalized amplitude normalized by the amplitude of the normalized frequency = 0. Since the number of horizontal pixels of the brightness signal for detecting the defocus amount is reduced to 1/6 of the number of full-size horizontal pixels, the signal having a normalized frequency higher than 1/6 = 0.166, that is, The gray area in FIG. 8 is the folding component. In the folded component, a part of the image is altered to generate moire, and the quality of the image is deteriorated.

Next, the effect of the folding component on the defocus amount detection will be described. FIG. 9 describes the case of a frequency band that does not become a folding component, and FIG. 10 describes a case of a frequency band that does not become a folding component.
FIG. 9A is a diagram showing a line profile of a sine wave subject having a band having a normalized frequency of 0.08 in FIG. The band with a normalized frequency of 0.08 is in the white region in FIG. 8, i.e., the region that is not the wrapping component. The horizontal axis represents horizontal coordinates, and the vertical axis represents signal values. The solid line 901 represents the output signal of the sub-pixel a, and the broken line 902 represents the output signal of the sub-pixel b. From the solid line 901 and the broken line 902, it can be seen that the sub-pixel a and the sub-pixel b are out of phase by one pixel in the horizontal direction.

FIG. 9B is a diagram showing a luminance signal obtained by performing addition thinning processing on the output signal shown in FIG. 9A. The solid line 903 represents the luminance signal generated from the output signal of the sub-pixel a, and the broken line 904 represents the luminance signal generated from the output signal of the sub-pixel b. Since the number of pixels is reduced to 1/6 of the full size by the addition thinning process, the phase shift amount is also degenerated. That is, when the phase shift is detected in sub-pixel units, the shift amount becomes 1/6 pixel. As shown in FIG. 9B, in the frequency band that does not become a folding component, the phase shift amount of the sub-pixels can be accurately extracted even if the addition thinning process is performed. Therefore, the correct defocus amount can be detected. As a method for calculating the defocus amount from the phase shift amount of the sub-pixel, for example, the method disclosed in Japanese Patent Application Laid-Open No. 2008-15754 may be used.

On the other hand, FIG. 10A is a diagram showing a line profile of a sine wave subject having a band with a normalized frequency of 0.26 in FIG. The band having a normalized frequency of 0.26 is the gray region in FIG. 8, that is, the region of the folding component. The horizontal axis represents horizontal coordinates, and the vertical axis represents signal values. The solid line 1001 represents the output signal of the sub-pixel a, and the broken line 1002 represents the output signal of the sub-pixel b. From the solid line 1001 and the broken line 1002, it can be seen that the sub-pixel a and the sub-pixel b are one pixel out of phase in the horizontal direction.

FIG. 10B is a diagram showing a luminance signal obtained by performing addition thinning processing on the output signal shown in FIG. 10A. The solid line 1003 represents the luminance signal generated from the output signal of the sub-pixel a, and the broken line 1004 represents the luminance signal generated from the output signal of the sub-pixel b. As shown in FIG. 10B, the luminance signal obtained by adding and thinning out the output signal of the frequency band to be the folding component has a phase change due to the folding, and the phase shift is 1/6 pixel which is the correct shift amount. is not.

That is, when the band of the normalized frequency in the full size state is higher than 0.166, the correct defocus amount cannot be detected due to the folding component, while when it is lower than 0.166, the correct defocus amount is detected. can do. Therefore, in order to determine whether or not the correct defocus amount can be detected after the addition thinning process, it is sufficient to detect whether the band of the normalized frequency in the full size state is higher or lower than 0.166.

Therefore, a folded reliability map is generated based on whether the band of the normalized frequency in the full size state is higher or lower than 0.166. Specifically, a high-pass filter (HPF) is applied to each of the full-size A image Bayer signal and B image Bayer signal, and if the output value is lower than the preset threshold value, high reliability is given to the output value. If is high, assign low confidence and generate a confidence map. The high-reliability region corresponds to the white region of FIG. 8, and the low-reliability region corresponds to the gray region of FIG. 8 showing the folded-back component. Then, the reliability map of the A image and the reliability map of the B image are integrated to generate a folded reliability map.

FIG. 11 is a diagram illustrating the frequency characteristics of the high-pass filter. In this embodiment, a high-pass filter having a frequency characteristic of 1101 is used, which attenuates the frequency in the high-reliability region whose frequency is lower than 0.166 and passes the frequency in the low-reliability region whose frequency is higher than 0.166. The high-pass filter having the frequency characteristic 1101 is, for example, a high-pass filter having a filter coefficient of [-12 -1], and the threshold value is set to 40 for the output signals of FIGS. 9 (A) and 10 (A). By doing so, the reliability can be assigned.

A bandpass filter (BPF) may be used instead of the highpass filter to generate the wrap reliability map. By using the bandpass filter, the calculation scale becomes large, but more accurate reliability can be assigned. FIG. 12 is a diagram illustrating the frequency characteristics of the bandpass filter. In this embodiment, a bandpass filter having a frequency characteristic of 1201 is used, which cuts off the frequency in the high reliability region where the frequency is lower than 0.166 and passes the frequency in the low reliability region where the frequency is higher than 0.166. By providing 50 taps of the filter coefficient, it is possible to construct a bandpass filter having a frequency characteristic 1201 having a steep filter characteristic. When generating a wrap-around reliability map using a bandpass filter, the output value with the bandpass filter should be highly reliable if the output value is lower than the preset threshold value. If is high, assign low reliability.

A low pass filter (LPF) may be used instead of the high pass filter to generate the wrap reliability map. FIG. 13 is a diagram illustrating the frequency characteristics of the low-pass filter. In this embodiment, a low-pass filter having a frequency characteristic of 1301 is used, which passes through a frequency in a high-reliability region whose frequency is lower than 0.166 and cuts off a frequency in a low-reliability region whose frequency is higher than 0.166. When a wrap-around reliability map is generated using a low-pass filter, high reliability is set for output values higher than the preset threshold value, and high reliability is set for output values lower than the preset threshold value for output values that have been subjected to the low-pass filter. Assign low reliability.

In the present embodiment, a high-pass filter (or band-pass filter, low-pass filter) is applied to each of the full-size A image Bayer signal and B image Bayer signal to generate a folding reliability map in the A image and a folding reliability map in the B image. do. Furthermore, a folded reliability map that integrates the two generated reliability maps by ORing is output. That is, in the integrated folding reliability map, only when the reliability in both the A image and the B image is high reliability is output as high reliability, and the other combinations are output as low reliability. The gradation of the reliability map in the present embodiment has two values of high reliability and low reliability, but is not limited to this, and is a multi-valued reliability according to the output value of the above filter. May be good. Further, in the present embodiment, the case of adding and thinning out to 1/6 has been described, but the present invention is not limited to this. The extraction band of the filter that extracts the reliability may be appropriately changed according to the addition thinning number.

In step S504, the extraction unit 404 extracts a value at predetermined intervals from the folding reliability map generated in step S503. The predetermined interval is set according to the defocus map (or the reliability map of the defocus map) described in step S506. FIG. 14 is a diagram illustrating an extraction interval of the folding reliability map. Pixel 1401 represents one pixel in the folding reliability map generated in step S503. The extraction unit 404 extracts the information of the folding reliability map at intervals of the black pixels of interest 1402 according to the position of the defocus amount detected by the detection unit 406 in step S506. In the present embodiment, for example, the information of the folded reliability map is extracted every 120 pixels. Assuming that the number of horizontal pixels of the folding reliability map generated in step S503 is 6000 pixels, the number of horizontal pixels of the folding reliability map after extraction is 6000/120 = 50 pixels. It should be noted that the position of the pixel of interest for calculating the reliability of the folding component in step S503 may be calculated discretely with the same coordinates as at the time of calculating the defocus amount, and the extraction unit 404 may do nothing.

In step S505, the image addition processing unit 405 adds the pixel signals processed by the horizontal pixel addition thinning process by the thinning unit 402 in step S502 to generate a luminance signal. FIG. 15 is a diagram illustrating an image addition process. As shown in FIG. 15, the image addition processing unit 405 adds pixel signals in Bayer units of 2 × 2 pixels to generate a luminance signal. For example, the image addition processing unit 405 adds the signals of each pixel of Ra, Ga, Ga, and Ba surrounded by a thick line to generate a luminance signal of Ya. By this processing, the number of pixels is reduced by 1/2 in the horizontal direction and 1/2 in the vertical direction. That is, the full-size Bayer signal acquired in step S501 is added by the thinning unit 402 (step S502) and further added by the image addition processing unit 405 (step S505) to obtain a horizontal 1000 pixel luminance signal. Become. In the present embodiment, an example in which the addition thinning process is performed in steps S502 and S505 has been described, but the present invention is not limited to this, and the pixels may be reduced or reduced by resampling by the interpolation process.

In step S506, the detection unit 406 generates a defocus map and a reliability map of the defocus map based on the luminance signal of the A image and the luminance signal of the B image generated in step S505. Then, the detection unit 406 outputs the generated defocus map to the RAM 103. The defocus map is distribution information corresponding to the distance. The defocus amount and the position for calculating the reliability thereof are discrete, and in the present embodiment, the defocus amount is detected every 20 pixels. Therefore, the number of horizontal pixels of the generated defocus map and the reliability map of the defocus map is 1000/20 = 50 pixels.

The detection unit 406 calculates the defocus amount from the correlation amount of the pixel signal by using, for example, the method disclosed in Japanese Patent Application Laid-Open No. 2008-15754, and generates a defocus map. Further, the detection unit 406 evaluates the contrast between the A image and the B image, assigns a low reliability to the defocus amount when the contrast is low, and assigns a high reliability to the defocus amount when the contrast is high. Generate a confidence map. The method of generating the reliability map is not limited to this. For example, it is determined whether the minimum value when the correlation calculation between the A image and the B image is performed is a predetermined value or more, and if it is a predetermined value or more, it is low. A reliability may be assigned, and if it is less than a predetermined value, a high reliability may be assigned.

In step S507, the integration unit 407 integrates the folding reliability map generated in step S504 and the reliability map of the defocus map generated in step S506 by logical sum. The operation of the logical sum is the same as in step S503. When the reliability is multi-valued, for example, a method such as a weighted average is used for integration. The integration unit 407 outputs the integrated reliability map to the RAM 103.

The defocus map output to the RAM 103 and the reliability map after integration are used for image processing such as autofocus and adding background blur to the captured image. When the pixel of interest is low-reliability, by referring to the defocus amount of the surrounding high-reliability pixels, it is possible to avoid using the defocus amount that is erroneously detected by addition decimation. High-precision image processing can be performed. In the present embodiment, a case where a reliability map of the wrapping component is generated as a reliability map corresponding to the defocus map generated by referring to the image signal after the addition and thinning has been described, but the present invention is limited to this. is not it. The reliability map of the folding component may be applied to evaluate whether the motion vector is accurate or to determine whether the quality of the image quality is deteriorated by "moire".

As described above, according to the present embodiment, the imaging unit can calculate from the full-size signal before the addition thinning out whether or not the folding back component is included after the addition thinning out as the reliability for the folding back component. Therefore, with respect to the defocus amount detected by the phase difference method from the image signal after the addition thinning, it is possible to detect with high accuracy the region where erroneous detection occurs due to the influence of the folding component due to the addition thinning.

(Second Embodiment)
The imaging unit 105 of the first embodiment includes a thinning unit 402, a reliability generation unit 403, and an extraction unit 404, and calculates the reliability for the addition thinning of pixels and the folding component. In this embodiment, a case where these processes are performed by the image processing unit 106 will be described. The configuration of the image pickup apparatus 100 in the present embodiment is the same as each configuration described with reference to FIG. 1 in the first embodiment. Further, the pixel arrangement of the image pickup unit 105 is the same as the pixel arrangement described with reference to FIG. 2 in the first embodiment. The image processing unit 106 may be realized as an image processing device.

FIG. 16 is a block diagram for explaining the processing contents of the image pickup unit 105 and the image processing unit 106. In this embodiment, in addition to generating the defocus map and the reliability map, the image signal AB Bayer image is also output to the RAM 103 for 4K recording.
The image pickup unit 105 includes an acquisition unit 401. The image pickup unit 105 acquires a full-size A image Bayer signal and a B image Bayer signal having 6000 horizontal pixels in the acquisition unit 401. In the first embodiment, the image pickup unit 105 has a configuration capable of performing addition thinning of pixels and performing filter processing to calculate reliability, but in the present embodiment, these are reduced to form an image pickup unit. It's simplified.

The image processing unit 106 includes a thinning unit 402, a reliability generation unit 403, an extraction unit 404, an image addition processing unit 405, a detection unit 406, an integration unit 407, an AB image generation unit 1601, and an adaptive interpolation reduction unit 1602. The operation of the image processing unit 106 from the horizontal pixel addition thinning unit 402 to the integration unit 407 is the same as that of the first embodiment. Here, too, when the defocus amount is detected, the region where erroneous detection occurs due to the influence of the folding component due to the addition thinning can be detected as low reliability. Therefore, when applying such as determining the subject area to be tracked according to the defocus amount, it is possible to suppress erroneous detection of the defocus amount and determine the area of the tracked subject with higher accuracy.

Next, the 4K recording operation of the image processing unit 106 will be described. The AB image generation unit 1601 adds the A image Bayer signal and the B image Bayer signal to generate the AB image Bayer signal. The number of horizontal pixels of the generated AB image Bayer signal is 6000 pixels. The adaptive interpolation reduction unit 1602 reduces the AB image Bayer signal to generate a 4K size AB image Bayer signal. The adaptive interpolation reduction unit 1602 uses a method for obtaining a reduced image with better quality, instead of reduction by simple pixel thinning as in the horizontal pixel addition thinning unit 402. For example, as disclosed in Japanese Patent Application Laid-Open No. 2016-103977, the direction in which the signal correlation is high is determined, and the method of interpolating the signal of the pixel of interest is changed based on the determination result, so that the quality is further improved. Use a method that gives a good reduced image. In the present embodiment, the adaptive interpolation reduction unit 1602 reduces the AB image Bayer signal having 6000 horizontal pixels by interpolation processing to generate a 4K size AB image Bayer signal having 4000 horizontal pixels.

In the present embodiment, the case where the image processing unit performs from the addition thinning process to the generation of the reliability map of the wrapping component as the reliability map corresponding to the defocus map has been described, but the present invention is not limited to this. do not have. For example, as disclosed in Japanese Patent Application Laid-Open No. 2016-58963, a digital front-end unit may be provided, and the digital front-end unit may generate a reliability map by addition thinning or filtering.

As described above, according to the present embodiment, the image processing unit can calculate the reliability of the folded-back component. Therefore, when the image processing unit detects the defocus amount of the phase difference method from the image signal after the addition thinning, it is possible to determine the region where erroneous detection occurs due to the addition thinning as low reliability.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

100 Image pickup device 101 System control unit 105 Image pickup unit 106 Image processing unit

Claims (9)

An extraction means that extracts a predetermined frequency band from a signal output from the image pickup means, and an extraction means.
A first reliability generating means that assigns reliability to the pixel of interest according to the output of the extraction means,
A sampling means that resamples the signal output from the image pickup means,
A calculation means for calculating distribution information corresponding to a distance based on a signal output from the sampling means, and a calculation means.
A second reliability generation means for calculating the reliability of the distribution information,
The reliability generated by the first reliability generating means and the integrated means for integrating the reliability output by the second reliability generating means are provided.
The first reliability generation means is an image processing apparatus characterized in that when the frequency of a signal output from an image pickup means is higher than a threshold value, lower reliability is assigned as compared with the case where the frequency is lower than the threshold value. The image processing apparatus according to claim 1 , wherein the resampling is an addition thinning process or an interpolation process. The extraction means is a high-pass filter or a band-pass filter.
The image according to claim 1 or 2 , wherein the first reliability generation means compares an output of the extraction means with a threshold value, and assigns a low reliability when the output is larger than the threshold value. Processing device. The extraction means is a low-pass filter.
The image according to claim 1 or 2 , wherein the first reliability generation means compares an output of the extraction means with a threshold value, and assigns a low reliability when the output is smaller than the threshold value. Processing device. The image processing apparatus according to any one of claims 1 to 4, wherein the distribution information corresponding to the distance is the distribution information of the defocus amount. An imaging means having a photoelectric conversion unit and
An image pickup apparatus comprising the image processing apparatus according to any one of claims 1 to 5. The sixth aspect of claim 6, wherein the image pickup means includes a plurality of pixels corresponding to each of the plurality of microlenses, and each pixel includes a first photoelectric conversion unit and a second photoelectric conversion unit. Imaging device. The imaging device according to claim 6 or 7, wherein the imaging unit adds and outputs a signal output from the photoelectric conversion unit. It is a control method for image processing equipment.
An extraction process that extracts a predetermined frequency band from a signal output from an imaging means,
A first reliability generation step of assigning reliability to the pixel of interest according to the output of the extraction step, and
A sampling process that resamples the signal output from the imaging means,
A calculation step of calculating distribution information corresponding to a distance based on a signal output in the sampling step, and a calculation step.
A second reliability generation step for calculating the reliability of the distribution information, and
It has a reliability output in the first reliability generation step and an integration step for integrating the reliability output in the second reliability generation step .
The first reliability generation step is a control method characterized in that when the frequency of a signal output from an imaging means is higher than a threshold value, lower reliability is assigned as compared with the case where the frequency is lower than the threshold value.

Images