JP7009219B2 - Image processing method, image processing device, image pickup device, image processing program, and storage medium - Google Patents

Description

The present invention relates to an image processing method for estimating an image in which the pupil of an optical system is divided from a captured image.

In Patent Document 1, in an image pickup element having two photoelectric conversion units in one pixel, the addition signal of the two photoelectric conversion units and the signal of one photoelectric conversion unit are read out, and the difference between the two signals is used as the other signal. A method for obtaining a signal of a photoelectric conversion unit is disclosed.

Japanese Patent No. 469930

However, in the method disclosed in Patent Document 1, when the addition signals of the two photoelectric conversion units are saturated in luminance, the other is obtained from the difference between the addition signal of the two photoelectric conversion units and the signal of one photoelectric conversion unit. The signal of the photoelectric conversion unit cannot be obtained accurately. That is, in this case, it is not possible to estimate the image in which the pupil is divided with high accuracy.

Therefore, the present invention provides an image processing method, an image processing device, an image pickup device, a program, and a storage medium capable of estimating an image in which the pupil is divided with high accuracy even when luminance saturation occurs. With the goal.

The image processing method as one aspect of the present invention includes a first image obtained by imaging the subject space through the first pupil of the optical system and a second image which is a part of the first pupil. Using a step of acquiring an input image based on a second image obtained by imaging the subject space through the pupil of the eye, and a neural network having a plurality of intermediate layers between the input layer and the output layer. The third pupil comprises a step of generating a third image corresponding to an image obtained by imaging the subject space through the third pupil of the optical system from the input image. It is a part of the first pupil and is different from the second pupil.

The image processing apparatus as another aspect of the present invention includes a first image obtained by capturing an image of the subject space through the first pupil of the optical system, and a part of the first pupil. A second image obtained by imaging the subject space through the second pupil, an acquisition means for acquiring an input image based on the second image, and a neural network having a plurality of intermediate layers between an input layer and an output layer. The third has a generation means for generating a third image corresponding to an image obtained by imaging the subject space through the third pupil of the optical system from the input image. The pupil is a part of the first pupil and is different from the second pupil.

The image pickup apparatus as another aspect of the present invention includes an image pickup element that photoelectrically converts an optical image formed by an optical system, and the image processing apparatus.

An image processing program as another aspect of the present invention causes a computer to execute the image processing method.

The storage medium as another aspect of the present invention stores the image processing program.

The objects and features of the present invention will be described in the following examples.

According to the present invention, an image processing method, an image processing device, an image pickup device, an image processing program, and a storage medium capable of estimating an image in which pupils are divided with high accuracy even when luminance saturation occurs are provided. Can be provided.

It is a figure which shows the network structure of image generation in each Example. It is a block diagram of the image pickup apparatus in Example 1. FIG. It is an external view of the image pickup apparatus in Example 1 and Example 3. FIG. It is explanatory drawing of the image pickup part in Example 1. FIG. It is a flowchart of the image estimation process in Example 1 and Example 2. It is explanatory drawing of the image estimation processing in Example 1 and Example 3. It is a figure which shows the relationship between the split pupil, the image height and vignetting in Example 1. FIG. It is a flowchart about learning of coefficient data in each Example. It is explanatory drawing of each image height in Example 1 and pupil division in Azymuth. It is a block diagram of the image processing system in Example 2. It is an external view of the image processing system in Example 2. FIG. It is a block diagram of the image pickup element in Example 2. FIG. It is a flowchart of the image estimation process in Example 2. It is a block diagram of the image pickup apparatus in Example 3. FIG. It is an external view of the image pickup apparatus in Example 3. FIG.

Hereinafter, examples of the present invention will be described in detail with reference to the drawings. In each figure, the same members are designated by the same reference numerals, and duplicate description will be omitted.

A gist of the present invention will be described before going into a specific description of the examples. In the present invention, an image (first image) captured by an image (second image) captured by a certain pupil (second pupil) and a pupil (first pupil) in which the pupil and another pupil are combined (first pupil). An image (third image) captured by the other pupil (third pupil) is estimated from the image) using deep learning. At this time, by using the learning data in which the luminance saturation occurs during the deep learning learning, the image (third image) can be estimated with high accuracy even when the luminance saturation occurs.

First, the image pickup apparatus according to the first embodiment of the present invention will be described with reference to FIGS. 2 and 3. FIG. 2 is a block diagram of the image pickup apparatus 100. FIG. 3 is an external view of the image pickup apparatus 100. First, the outline of each part of the image pickup apparatus 100 will be described, and the details thereof will be described later.

As shown in FIG. 2, the image pickup apparatus 100 has an image pickup unit 101 that acquires an image of a subject space as a captured image (input image). The image pickup unit 101 includes an imaging optical system 101a that collects incident light from the subject space, and an image pickup element 101b having a plurality of pixels. The image pickup device 101b is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor.

FIG. 4 is an explanatory diagram of the image pickup unit 101. FIG. 4A shows a cross-sectional view of the image pickup unit 101, and the alternate long and short dash line represents an axial luminous flux. FIG. 4B is a top view of the image pickup device 101b. The image pickup device 101b has a microlens array 122 and a plurality of pixels 121. The microlens array 122 is arranged at a position conjugate with the subject surface 120 via the imaging optical system 101a. As shown in FIG. 4B, the microlens 122 (only the microlens 122a is shown and omitted after 122b) constituting the microlens array 122 has a plurality of pixels 121 (only the pixel 121a is shown and 121b and later are shown). It corresponds to each of (omitted). Here, when a plurality of parts are collectively designated, only a number is attached, and when one of them is indicated, a number and a symbol such as a are attached.

Each of the plurality of pixels 121 has a first photoelectric conversion unit 123 and a second photoelectric conversion unit 124 that photoelectrically convert an optical image formed via the imaging optical system 101a. As a result, for example, the light incident on the pixel 121a is separated and received by the first photoelectric conversion unit 123a and the second photoelectric conversion unit 124a depending on the incident angle (first photoelectric conversion unit). The 123a and the second photoelectric conversion unit 124a receive light incident at different angles of incidence). The incident angle of the light is determined by which position of the pupil in the imaging optical system 101a the light has passed. Therefore, the pupil of the imaging optical system 101a is divided into two partial pupils by two photoelectric conversion units, and the two photoelectric conversion units in one pixel observe the subject space from different viewpoints (pupil positions). Get information. In this embodiment, the pupil division direction is the horizontal direction, but the present invention is not limited to this, and other directions such as a vertical direction and an oblique direction may be used.

The image pickup element 101b has a signal (second captured image, A image) acquired by the first photoelectric conversion unit 123, this signal (A image), and a signal (second) acquired by the second photoelectric conversion unit 124. An addition signal (first captured image, A + B image) with the captured image (B image) of 3 is output. The A image and the A + B image are output to the image processing unit 102. The image processing unit (image processing apparatus) 102 has an information acquisition unit (acquisition means) 102a and an image generation unit (generation means) 102b, and executes the image processing method of the present embodiment. At this time, the image processing unit 102 uses the coefficient data stored in the storage unit (storage means) 103, and the details of this processing will be described later. As a result, the image processing unit 102 can estimate the B image and acquire the phase difference information from the A image and the estimated B image. The system controller 106 controls the in-focus position of the image pickup unit 101 based on the phase difference information acquired by the image processing unit 102.

When the user gives an instruction to release the image, the image pickup unit 101 executes an image pickup at the in-focus position at that time, and the obtained A image and the A + B image are stored in the recording medium 105. When the user gives an instruction regarding the display of the captured image, the system controller 106 reads out the data stored in the recording medium 105 and displays the data on the display unit 104. At this time, the image processing unit 102 generates an image to be displayed on the display unit 104 according to the conditions specified by the user. When the same focusing position as at the time of imaging is specified, the display unit 104 displays the A + B image as it is. Further, when a focusing position different from that at the time of imaging is specified, the image processing unit 102 generates a refocused image. The refocused image is obtained by estimating the B image using the image processing method of this embodiment based on the A + B image and the A image, and spatially shifting and synthesizing the A image and the estimated B image. Be done. The above series of control is performed by the system controller 106.

Next, the image estimation process (B image estimation process) executed by the image processing unit 102 will be described with reference to FIGS. 5 and 6. In the estimation process of the B image, the image processing unit 102 uses the coefficient data learned in advance, and the details of this learning will be described later. FIG. 5 is a flowchart relating to the estimation process of the B image. FIG. 6 is an explanatory diagram of the estimation process of the B image. Each step in FIG. 5 is executed by the image processing unit 102 based on the command of the system controller 106. In this embodiment, the information acquisition unit 102a of the image processing unit 102 executes steps S101 to S104 of FIG. 5, and the image generation unit 102b executes steps S105 to S108.

First, in step S101, the image processing unit 102 acquires an A + B image (first captured image) 201 and an A image (second captured image) 202. The A image 202 is an image obtained by imaging the subject space based on the luminous flux passing through the partial pupil (second pupil) which is a part of the pupil of the imaging optical system 101a. The A + B image 201 is an image obtained by imaging the subject space based on the luminous flux passing through the pupil (first pupil) of the imaging optical system 101a. In this embodiment, the second pupil is included in the first pupil and is a part of the first pupil.

Subsequently, in step S102, the image processing unit 102 divides each of the A + B image 201 and the A image 202 into two regions based on the luminance saturation of the A + B image 201. In this embodiment, as shown in FIG. 6, the image processing unit 102 divides the A + B image 201 into a first region 204 and a second region 211. Further, the image processing unit 102 divides the A image 202 into a first region 205 and a second region 212. In FIG. 6, the shaded portion in the A + B image 201 represents the luminance saturated region 203 in which the luminance is saturated in the A + B image 201. The first region 204 in the A + B image 201 is set to include the luminance saturation region 203. When the luminance-saturated regions exist in a discrete manner (when a plurality of luminance-saturated regions separated from each other exist), the first region 204 may be similarly set to the discrete regions, and the regions are continuously distributed. It doesn't have to be. The second region 211 is set so as not to include the luminance saturation region 203. The first region 205 and the second region 212 in the A image 202 are set to match the first region 204 and the second region 211, respectively.

When the image sensor 101b is a color sensor having a Bayer array, the A + B image and the A image may remain in the Bayer array, or may be images rearranged into four channels of R, G1, G2, and B. In the case of a 4-channel image, the saturation region is different for each color, so each color may be processed individually. Alternatively, the 4-channel image may be processed at once by setting the first region so as to include luminance saturation for all colors.

Subsequently, in step S103, the image processing unit 102 extracts the first image 206 from the first region 204 of the A + B image 201. Further, the image processing unit 102 extracts the second image 207 from the first region 205 of the A image 202. The first image 206 is a partial region of the A + B image 201, and the second image 207 is a partial region of the A image 202. Then, the image processing unit 102 sets an input image based on the first image 206 and the second image 207. In this embodiment, the input images are the first image 206 and the second image 207. However, this embodiment is not limited to this. For example, an image obtained by subtracting the second image 207 from the first image 206 may be used as the input image.

Subsequently, in step S104, the image processing unit 102 selects and acquires the coefficient data corresponding to the input image. The coefficient data is used in the neural network 208 described later for estimating a third image corresponding to a part of the B image. In this embodiment, a plurality of types of coefficient data are stored in the storage unit 103, and the image processing unit 102 acquires desired coefficient data from the plurality of types of coefficient data stored in the storage unit 103. Here, the image processing unit 102 determines the positions of the first image 206 and the second image 207 extracted in step S103 (the first image 206 in each of the first captured image 201 and the second captured image 202). And the position of the second image 207) to select the coefficient data. The coefficient data is switched based on the positions of the first image 206 and the second image 207 because the vignetting (vignetting) and aberration of the imaging optical system 101a change according to the image height.

Here, the influence of vignetting will be described with reference to FIG. 7. FIG. 7 is a diagram showing the relationship between the split pupil, the image height, and vignetting. FIG. 7A shows the pupil on the optical axis of the imaging optical system 101a. The broken line in FIG. 7 represents the dividing line of the pupil divided by the two photoelectric conversion units. FIG. 7 (B) shows the pupil at an image height different from that in the case of FIG. 7 (A). In FIG. 7A, the light amounts of the two split pupils are uniform, but in FIG. 7B, the light amount ratio of the two is biased due to vignetting. Therefore, comparing FIGS. 7 (A) and 7 (B), it is found that it is difficult to estimate an accurate B image from the brightness-saturated A + B image and the A image using the same coefficient data. .. FIG. 7 (C) shows Azimuth (from the optical axis to the outer periphery in the plane perpendicular to the optical axis) at the same image height (position at the same distance from the optical axis in the plane perpendicular to the optical axis) as in FIG. 7 (B). This is the case when the azimuth to which the head is heading is different. At this time as well, the light intensity ratio of the partial pupil changes. Similarly, regarding aberrations, the relationship between the two partial pupils changes depending on the image height and the azimuth. Therefore, it is preferable that the coefficient data is selected (determined) based on the image heights of the first image 206 and the second image 207 and the azimuth.

If the vignetting of the imaging optical system 101a can be ignored (telephoto lens) or the aberration change due to the image height can be ignored (small aperture), the same coefficient data can be used for the entire captured image. good. Further, FIG. 7 shows a case where the pupil of the imaging optical system 101a is divided into two by a broken line and the divided pupils do not intersect each other, but the present embodiment is not limited to this. The split pupils may overlap each other in some areas (in the overlapping areas, the split pupils share the amount of light). Further, the corresponding coefficient data is provided for some patterns (such as 10: 1, 8: 1, ..., 1: 1, ..., 1:10) of the light amount ratio between the A image and the B image. A plurality of storage units 103 may be stored. Further, the image processing unit 102 acquires information on the vignetting of the imaging optical system 101a in the first image 206 and the second image 207, and selects the corresponding coefficient data of the light amount ratio based on the information on the vignetting. It can also be configured as follows.

Subsequently, in step S105 of FIG. 5, the image processing unit 102 generates a third image 209 from the input images (first image 206 and second image 207). The third image 209 is an image corresponding to a partial region of an image (B image) in which the subject space is captured based on the luminous flux passing through the third pupil of the imaging optical system 101a. The third pupil is a part of the first pupil and is different from the second pupil. In this embodiment, the third pupil is a component obtained by removing the second pupil from the first pupil, and the first pupil is represented by the sum of the second pupil and the third pupil. The image processing unit 102 generates a third image 209 by using a neural network 208 having a plurality of intermediate layers between the input layer and the output layer. In this embodiment, a convolutional neural network (CNN) is used. However, this embodiment is not limited to this, and other neural networks such as GAN (Generative Adversarial Network) may be used.

Here, with reference to FIG. 1, the step of generating the third image 209 by CNN will be described in detail. FIG. 1 is a diagram showing a network structure of image generation. In this embodiment, the input image 221 is an image in which the first image 206 and the second image 207 are stacked in the channel direction. When each of the first image 206 and the second image 207 has a plurality of color channels, the image has twice the number of channels.

The CNN has a plurality of layer structures, and linear transformation and non-linear transformation using the coefficient data learned in each layer are executed. The linear transformation is represented by the sum of the input data, the convolution of the filter, and the bias (bias in FIG. 1). The filter and bias values for each layer are determined by the coefficient data. The non-linear transformation is a transformation by a non-linear function called an activation function (AF in FIG. 1). Examples of activation functions include sigmoid functions and hyperbolic tangent functions. In this embodiment, ReLU (Rectifier Unit) represented by the following equation (1) is used as the activation function.

In the equation (1), max represents a MAX function that outputs the maximum value among the arguments.

The input image 221 input to the input layer is summed with the bias and the convolution with each of the plurality of filters 222 in the first convolution layer. The number of channels of each of the filters 222 matches the input image 221 and becomes a three-dimensional filter when the number of channels of the input image 221 is two or more (the third dimension represents the number of channels). The result of the convolution and the sum is subjected to a non-linear transformation by the activation function, and the first feature map 223 is output to the first intermediate layer.

Next, the first feature map 223 is input to the second convolution layer, and the sum of the convolution with each of the plurality of filters 224 and the bias is taken as described above. The result is non-linearly transformed and repeated for the number of convolution layers in the same manner. Finally, the N-1 feature map 231 of the N-1 intermediate layer is input to the N-convolution layer to obtain a third image 209. Here, N is an integer of 3 or more. The number of filters 232 in the Nth layer matches the number of channels in the third image 209. Also, the final convolutional layer that produces the third image 209 does not have to perform a non-linear transformation. In FIG. 1, the image size of the third image 209 is smaller than that of the input image 221. This is because the convolution is executed only in the region where the data of the input image 221 (or the feature map) exists in the convolution layer. It is possible to make the image size invariant by filling the periphery of the input image 221 (or feature map) with zeros or the like or by using a deconvolution layer.

Subsequently, in step S106 of FIG. 5, the image processing unit 102 determines whether or not the third image 209 has been generated for the predetermined area. If the generation of the third image 209 is not completed, the process returns to step S103, and the image processing unit 102 newly extracts the first image 206 and the second image 207 from the predetermined area. On the other hand, when the generation of the third image is completed, the process proceeds to step S107. When generating a refocused image or the like, a B image of the entire captured image is required, so it is necessary to generate a third image 209 for all of the luminance saturation regions 203. When the focus detection is the purpose, the third image 209 may be generated only in the vicinity of the designated focus point. When the third image 209 is generated for all of the predetermined areas, the first area 210 in the B image (estimated B image) is generated as shown in FIG.

Subsequently, in step S107, the image processing unit 102 generates a fourth image 213 based on the difference between the second region 211 of the A + B image 201 and the second region 212 of the A image 202. That is, the image processing unit 102 acquires the fourth image 213 corresponding to the second region in the B image by subtracting the second region 212 from the second region 211. Since there is no luminance saturation in the second region 211 of the A + B image 201, the image processing unit 102 can obtain the B image by subtraction processing. By estimating only the region where the luminance saturation exists by using the neural network 208, the calculation load can be reduced. Note that step S107 may be executed at any time between steps S102 and S108.

Subsequently, in step S108, the image processing unit 102 generates a fifth image 214 by synthesizing the third image 209 (first region 210) and the fourth image 213 (second region). do. The fifth image 214 is an estimated B image.

In step S107, the difference may be taken over the entire A + B image 201 and A image 202 regardless of the second regions 211 and 212. Although the B image cannot be obtained correctly in the region where the luminance is saturated, the estimated B image (fifth image 214) can be obtained by substituting the signal in that region with the first region 210.

By the above processing, the B image can be estimated with high accuracy even when the A + B image has luminance saturation. By using the A image and the estimated B image, it is possible to perform focus detection by phase difference AF, estimation of depth map by parallax, refocusing, and the like.

Next, the learning of the coefficient data will be described with reference to FIG. FIG. 8 is a flowchart for learning the coefficient data. In this embodiment, the learning is executed in advance by an image processing device other than the image pickup device 100, and the result (a plurality of coefficient data) is stored in the storage unit 103. However, the present embodiment is not limited to this, and a portion for executing learning may exist in the image pickup apparatus 100.

First, in step S201, the image processing apparatus acquires a plurality of learning pairs. The learning pair is a known A + B image, A image, B image, and a first image, a second image, and a third correct answer image extracted from the known A + B image, A image, and B image. The sizes of the first image and the second image are the same as those of the third image generation process shown in FIGS. 5 and 6. The third correct image has the same size as the third image. The A + B image, the A image, and the B image may be an image of an actual subject or an image using CG (computer graphics). It is preferable that the learning pair is extracted from an image captured with substantially the same configuration as the image imaging system (imaging unit 101 in this embodiment) that actually estimates the B image (imaging can be performed by CG simulation). .. Further, it is preferable that all the learning pairs used for calculating the same coefficient data have substantially the same vignetting and aberration. In order to estimate the B image when the A + B image is saturated in brightness, the learning pair must always include the first image in which the brightness is saturated.

Subsequently, in step S202, the image processing device generates coefficient data from a plurality of learning pairs. During learning, the same network structure as the generation of the third image in step S105 is used. In this embodiment, the first image and the second image are input to the network structure shown in FIG. 1, and the error between the output result (estimated third image) and the third correct image is calculated. calculate. In order to minimize this error, the coefficient and bias (coefficient data) of the filter used in each layer are updated and optimized by using, for example, backpropagation. Arbitrary values can be used for the coefficient of the filter and the initial value of the bias, and are determined from, for example, a random number. Alternatively, pre-training such as Auto Encoder that pre-learns the initial value for each layer may be performed.

The method of inputting all the learning pairs into the network structure and updating the coefficient data using all the information is called batch learning. However, in this learning method, the computational load becomes enormous as the number of learning pairs increases. Conversely, a learning method that uses only one learning pair to update the coefficient data and uses a different learning pair for each update is called online learning. This method has the advantage that the amount of calculation does not increase even if the number of learning pairs increases, but instead, it is greatly affected by the noise existing in one learning pair. Therefore, it is preferable to study using the mini-batch method located between these two methods. The mini-batch method extracts a small number from all the training pairs and uses them to update the coefficient data. In the next update, we will extract and use different fractional learning pairs. By repeating this, the drawbacks of batch learning and online learning can be reduced, and the accuracy of depth estimation can be easily improved.

Subsequently, in step S203, the image processing device outputs the learned coefficient data. By repeating the same learning for various pupil sizes, vignetting, and aberrations, a plurality of coefficient data can be acquired. In this embodiment, the coefficient data is stored in the storage unit 103.

Next, preferable conditions for enhancing the effect of the present invention will be described. In step S102 of FIG. 5, it is preferable that the region where both the A + B image and the A image are saturated in luminance is set as the third region and different processes are executed. In step S105, the brightness saturation of the first image is hinted at by the color of the second image (partial region of the A image) or the first image (partial region of the A + B image) that is not saturated in brightness. Estimate a third image in the region. Therefore, when all the input images are saturated in brightness, there is no hint for estimating the third image in the corresponding region. Therefore, the network of FIG. 1 cannot estimate the third image with high accuracy. Further, even if the B image can be estimated, the focus detection cannot be performed because the A image is saturated in brightness. In this case, in order to reduce the calculation load, it is preferable to generate a third image with a luminance saturation value in the third region. Alternatively, the signal in the third region (having a value exceeding the luminance saturation value) can be estimated from the periphery of the luminance saturation by using inpainting by CNN or the like. At this time, a partial region is extracted from the A image so as to include a region other than the luminance saturation, and inpainting is performed. By inpainting the A + B image in the same manner, the B image can be estimated from the difference.

Further, it is preferable to perform denoising of the second image at the same time as estimating the third image. The A + B image has lower noise than the A image because of the sum of the A image and the B image. Therefore, the A image can be denoised by referring to the A + B image. In this case, when learning the coefficient data, a known low-noise A image and B image are prepared. From here, an A image and a B image to which noise is added are generated by simulation, and an A + B image is generated by adding these. The data to be input to the neural network is acquired by extracting the second image and the first image from each of the noise-added A image and the A + B image. The output of the neural network is two images, a third image and a denoized second image, and the third correct image and the second correct image extracted from the low noise A image and the B image are compared. The error is calculated by this. By using the coefficient data learned in this way, it is possible to realize a network that generates a second image denoized at the same time as estimating the third image.

More preferably, the coefficient data is changed according to the noise level. This makes it possible to realize more accurate denoising. The noise level (information about noise) of the first image and the second image can be estimated from the ISO sensitivity at the time of imaging and the like. It can also be estimated from the dispersion of signals in the flat portion in the image. Information on noise is acquired, and the corresponding coefficient data is selected and used from a plurality of coefficient data learned for each of a plurality of noise levels.

Further, in this embodiment, the amount of coefficient data can be reduced by dividing the captured image into two divided regions (a first divided region and a second divided region) and inverting one of them. This will be described with reference to FIG. FIG. 9 is an explanatory diagram of each image height and pupil division in Azymuth. FIG. 9 shows an A + B image, and the image height of the x mark and the split pupil in Azymuth are drawn next to the x mark. The broken line in FIG. 9 is a dividing line (dividing straight line) of the pupil. As shown in FIG. 9, in this embodiment, when either the upper or lower side of the A + B image is inverted with the alternate long and short dash line as the axis, it overlaps with the pupil division of the other and becomes line symmetric. Therefore, if the coefficient data is retained for either the upper or lower region of the alternate long and short dash line, the third image (a part of the B image) can be estimated by inverting the image of the other.

In this embodiment, since the pupil is divided in the horizontal direction, the axis of symmetry is a horizontal straight line, but if the pupil is divided in the vertical direction, the axis of symmetry is also a vertical straight line. This can be expressed more generally as follows. The axis in which the relationship between the divided pupils is line-symmetrical with respect to the entire image passes through the optical axis of the imaging optical system 101a, and the common axis (A image and) in which each divided pupil is line-symmetrical on the optical axis. It is parallel to the axis of line symmetry common to the pupil on each optical axis of the B image). The A + B image and the A image are each divided into two with this axis of symmetry as the dividing line, and the extracted first image and the second image are inverted and input with respect to the dividing line in one of the divided areas. Get an image. As the coefficient data, the coefficient data in which Azymuth is positive or negative inverted at the same image height is used. By re-inverting the generated third image, a third image corresponding to the first image and the second image can be estimated. This eliminates the need to hold the coefficient data for all azimuths (-180 ° to 180 °), and the data capacity can be halved.

The image handled when learning the coefficient data and generating the third image may be a RAW image or a developed image. When the A + B image and the A image are encoded, the learning and generation are performed after decoding. If the image used for training and the generated input image have different gamma corrections and gamma values, it is preferable to process the input image to match the training image. Further, it is preferable that the signal values of the A + B image and the A image (also the B image at the time of learning) are standardized before being input to the neural network. In the case of no standardization, if the number of bits is different at the time of learning and generation, the third image cannot be estimated correctly. In addition, since the scale changes according to the number of bits, optimization during learning may affect convergence. For normalization, the maximum value (luminance saturation value) that the signal can actually take is used. For example, even if the A + B image is stored in 16 bits, the luminance saturation value may be 12 bits, and in this case, the signal range cannot be 0 to 1 unless it is standardized by the maximum value (4095) of 12 bits. Further, it is preferable to subtract the value of optical black at the time of standardization. As a result, the range of signals that can actually be captured by the image can be made closer to 0 to 1. Specifically, it is preferable to standardize according to the following formula (2).

In equation (2), s is the signal of the A + B image (or A image or B image), s _OB is the signal value of optical black (minimum value of the signal that the image can take), s _satu is the luminance saturation value of the signal, and s. _nor indicates a standardized signal.

According to this embodiment, it is possible to provide an image processing method, an image processing device, and an image pickup device capable of estimating an image in which the pupil is divided with high accuracy even when luminance saturation occurs.

Next, the image processing system according to the second embodiment of the present invention will be described. In this embodiment, there are individually an image processing device that estimates a third image, an image pickup device that acquires a captured image, and a server that performs learning.

The image processing system in this embodiment will be described with reference to FIGS. 10 and 11. FIG. 10 is a block diagram of the image processing system 300. FIG. 11 is an external view of the image processing system 300. As shown in FIGS. 10 and 11, the image processing system 300 includes an image pickup device 301, an image processing device 302, a server 306, a display device 309, a recording medium 310, and an output device 311.

The basic configuration of the image pickup apparatus 301 is the same as that of the image pickup apparatus 100 shown in FIG. 2, except for the image processing unit that generates the third image and the image pickup unit. The image pickup device of the image pickup device 301 is configured as shown in FIG. FIG. 12 is a block diagram of the image pickup device in this embodiment. In FIG. 12, the dashed line indicates a microlens. Each of the pixels 320 (omitted after a and b) is provided with four photoelectric conversion units 321, 322, 323, 324 (omitted after a and b), and the pupil of the imaging optical system is 2 × 2. It is divided into two. The images acquired by the photoelectric conversion units 321 to 324 are, in order, an A image, a B image, a C image, and a D image, and the addition result thereof is used as an ABCD image. The image sensor outputs four images, an ABCD image, an A image, a C image, and a D image, as captured images.

When the image pickup device 301 and the image processing device 302 are connected, the captured image is stored in the storage unit 303. The image processing device 302 generates an estimated B image (a set of third images) from the captured image by the image generation unit 304. At this time, the image processing device 302 accesses the server 306 via the network 305 and reads out the coefficient data used for generation. The coefficient data is learned in advance in the learning unit 308 and stored in the storage unit 307. The coefficient data is individually learned by a plurality of lenses, focal lengths, F-numbers, and the like, and there are a plurality of coefficient data.

The image processing device 302 selects coefficient data under conditions that match the input captured image, acquires it in the storage unit 303, and generates a third image. The generated estimated B image is used for refocus processing and the like, and the captured image after the processing is output to at least one of the display device 309, the recording medium 310, and the output device 311. The display device 309 is, for example, a liquid crystal display or a projector. The user can perform the work while confirming the image in the process of processing via the display device 309. The recording medium 310 is, for example, a semiconductor memory, a hard disk, a server on a network, or the like. The output device 311 is a printer or the like. The image processing device 302 has a function of performing development processing and other image processing as needed.

Next, with reference to FIG. 13, an image estimation process (a third image (B image) generation process) executed by the image generation unit 304 of the image processing device 302 will be described. FIG. 13 is a flowchart relating to the image estimation process (B image estimation process). Each step of FIG. 13 is mainly executed by the image processing device 302 (image generation unit 304).

First, in step S301, the image processing device 302 acquires the first captured image and the second captured image. In this embodiment, the first captured image is an ABCD image, and the second captured image is three images, an A image, a C image, and a D image. Subsequently, in step S302, the image processing device 302 acquires an input image based on the first image and the second image. In this embodiment, the first image is extracted from the ABCD image and the second image is extracted from each of the A image, the C image, and the D image. Therefore, the second image is three images. In this embodiment, a 4-channel image in which the first image and the second image are stacked in the channel direction is used as an input image.

Subsequently, in step S303, the image processing device 302 selects and acquires the coefficient data corresponding to the input image. Subsequently, in step S304, the image processing device 302 generates a third image. In this embodiment, the convolutional neural network CNN shown in FIG. 1 is used as the network used to generate the third image.

Subsequently, in step S305, the image processing device 302 determines whether or not the third image has been generated for the predetermined area. In this embodiment, the predetermined area is the entire captured image. If the generation of the third image is not completed, the process returns to step S302, and the image processing device 302 acquires a new input image. On the other hand, when the generation of the third image is completed, the process proceeds to step S306. In step S306, the image processing device 302 generates an estimated B image from the plurality of generated third images.

In this embodiment, the learning of the coefficient data by the learning unit 308 is performed according to the flowchart shown in FIG. 8, as in the first embodiment. Since aberrations and vignetting differ depending on the lens (imaging optical system 101a), a learning pair is created for each lens type and coefficient data is learned. If changes in aberration and vignetting cannot be ignored due to imaging conditions (focal length, F value, etc.) and image height, learning pairs are created for each of a plurality of imaging conditions and image height to learn coefficient data. In this embodiment, an example is given in which the second captured image is three images, but conversely, the second captured image is one A image, and the third image is the B image and C. There may be a configuration in which there are three images as a part of each of the image and the D image.

According to this embodiment, it is possible to provide an image processing system capable of estimating an image in which the pupil is divided with high accuracy even when luminance saturation occurs.

Next, the image pickup apparatus according to the third embodiment of the present invention will be described. The image pickup device of this embodiment is a multi-eye image pickup device. FIG. 14 is a block diagram of the image pickup apparatus 400. FIG. 15 is an external view of the image pickup apparatus 400.

The image pickup apparatus 400 has an image pickup unit 401, and the image pickup unit 401 has two imaging optical systems 401a and 401b. The images (subject image, optical image) formed by each of the two imaging optical systems 401a and 401b are received by one image sensor 401c. At this time, the two images are received in different regions of the image sensor 401c. In this embodiment, the images captured via the imaging optical systems 401a and 401b are referred to as an A image and a B image, respectively. The image sensor 401c outputs an A + B image obtained by adding an A image and a B image, and an A image. At this time, the pupil corresponding to the A + B image (first pupil) is the pupil obtained by adding the pupils of the imaging optical systems 401a and 401b. The description of other parts is the same as that of the first embodiment. Further, in this embodiment, the image sensor 401c is one, but two image pickup elements corresponding to each of the image pickup optical systems 401a and 401b may be arranged. In this case, the image processing unit 402 acquires an addition signal obtained by adding the output signals of the two image pickup elements as the first image pickup image. The generation of the third image (a part of the B image) and the learning of the coefficient data are the same as in the first embodiment.

The image pickup apparatus 400 has an image processing unit (image processing apparatus) 402 that executes the image processing method of the present embodiment, and the image processing unit 402 includes an information acquisition unit (acquisition means) 402a and an image generation unit (generation means) 402b. Have. Further, the image pickup apparatus 400 has a storage unit 403, a display unit 404, a recording medium 405, and a system controller 406.

According to this embodiment, it is possible to provide an image pickup apparatus capable of estimating an image in which the pupil is divided with high accuracy even when luminance saturation occurs.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

According to each embodiment, an image processing method, an image processing device, an image pickup device, an image processing program, and a storage medium capable of estimating an image in which the pupil is divided with high accuracy even when luminance saturation occurs. Can be provided.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these examples, and various modifications and modifications can be made within the scope of the gist thereof.

102 Image processing unit 102a Information acquisition unit 102b Image generation unit

Claims (16)

To image the subject space through a first image obtained by imaging the subject space through the first pupil of the optical system and a second pupil which is a part of the first pupil. The process of acquiring the input image based on the second image obtained in
A third image corresponding to an image obtained by imaging the subject space through the third pupil of the optical system using a neural network having a plurality of intermediate layers between an input layer and an output layer is obtained. It has a step of generating from the input image and
An image processing method characterized in that the third pupil is a part of the first pupil and is different from the second pupil. The image processing method according to claim 1, further comprising a step of selecting coefficient data to be used in the neural network from a plurality of coefficient data. The first image and the second image are partial regions of the first captured image and the second captured image obtained by imaging the subject space through different pupils, respectively.
In the step of selecting the coefficient data, the coefficient data is selected based on the position of the first image in the first captured image and the position of the second image in the second captured image. 2. The image processing method according to claim 2. The first image and the second image are partial regions of the first captured image and the second captured image obtained by imaging the subject space through different pupils, respectively.
2. The image processing method described in. Further comprising a step of acquiring information regarding the vignetting of the optical system used for capturing the first image and the second image.
The image processing method according to any one of claims 2 to 4, wherein in the step of selecting the coefficient data, the coefficient data is selected based on the information regarding the vignetting. Further comprising a step of acquiring information regarding noise of each of the first image and the second image.
The image processing method according to any one of claims 2 to 5, wherein in the step of selecting the coefficient data, the coefficient data is selected based on the information regarding the noise. The noise contained in the first image is smaller than the noise contained in the second image.
Claims 1 to 6 are characterized in that, in the step of generating the third image, the neural network is used to generate the third image and the denosized second image from the input image. The image processing method according to any one of the above items. The first pupil is the sum of the second pupil and the third pupil.
The image processing method according to any one of claims 1 to 7, wherein the input image includes an image obtained by subtracting a second image from the first image. A step of extracting the first image and the second image from each of the first captured image and the second captured image obtained by imaging the subject space through different pupils.
A step of subtracting a second captured image from the first captured image to generate a fourth image,
Further comprising a step of synthesizing the third image and the fourth image to generate a fifth image.
The first pupil is the sum of the second pupil and the third pupil.
A claim characterized in that, in the step of extracting the first image and the second image, the first image is extracted so as to include a region in which the luminance is saturated in the first captured image. Item 6. The image processing method according to any one of Items 1 to 7. A step of extracting the first image and the second image from each of the first captured image and the second captured image obtained by capturing the subject space with different pupils.
A step of dividing each of the first captured image and the second captured image into a first divided region and a second divided region by a dividing straight line.
It further comprises a step of extracting the first image and the second image from each of the first captured image and the second captured image.
The dividing straight line passes through the optical axis of the optical system and is parallel to the axis of line symmetry common to the pupils on the respective optical axes of the first captured image and the second captured image.
In the step of acquiring the input image, the first image extracted from the first divided region or the second divided region and the second image are inverted to acquire the input image. The image processing method according to any one of claims 1 to 9. To image the subject space through the first image obtained by imaging the subject space through the first pupil of the optical system and the second pupil which is a part of the first pupil. The acquisition means for acquiring the input image based on the second image obtained in
A third image corresponding to an image obtained by imaging the subject space through the third pupil of the optical system using a neural network having a plurality of intermediate layers between an input layer and an output layer is obtained. It has a generation means for generating from the input image, and has.
An image processing apparatus characterized in that the third pupil is a part of the first pupil and is different from the second pupil. The image processing apparatus according to claim 11, further comprising a storage means for storing coefficient data used in the neural network. An image pickup element that photoelectrically converts an optical image formed by an optical system,
An image pickup apparatus comprising the image processing apparatus according to claim 11 or 12. The image pickup device has a plurality of pixels and has a plurality of pixels.
Each of the plurality of pixels includes a first and second photoelectric conversion unit that receives light incident at different angles of incidence and generates first and second signals.
The image pickup device includes a first image pickup image corresponding to an addition signal obtained by adding the first and second signals, and a second image pickup image corresponding to one of the first and second signals. Is output,
13. The image pickup apparatus according to claim 13, wherein the first image is a partial region of the first captured image, and the second image is a partial region of the second captured image. An image processing program comprising causing a computer to execute the image processing method according to any one of claims 1 to 10. A storage medium for storing the image processing program according to claim 15.

Images