JP6562650B2 - Image processing apparatus, imaging apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing technique for performing a sharpening process on an image generated by an imaging apparatus such as a digital camera.

  An unsharp mask process (sharpening process) is known as a process for improving the sharpness of an image (input image) generated by imaging with an imaging device such as a digital camera. Unsharp mask processing takes the difference between the blurred image and the original input image by applying an unsharp mask to the input image, and sharpens it by adding the difference data to the input image. Realize. A filter for blurring an image such as a smoothing filter is used for the unsharp mask, and an image region having a larger difference between the blurred image and the input image is sharpened.

  Further, in Patent Document 1, a point spread function (PSF) of an optical system is applied by applying an asymmetric one-dimensional filter to pixel signal sequences arranged in the image height direction (meridional direction). A method for correcting false colors and unnecessary coloring due to the influence of the above is disclosed.

JP 2010-81263 A

  However, in the conventional unsharp mask processing, a rotationally symmetric filter is used as the unsharp mask. For this reason, the input image deteriorated due to the influence of the PSF having a complicated shape such as asymmetric aberration or sagittal halo cannot be sharpened correctly. Correct sharpening means that if you try to correct aberrations in the azimuth direction where the aberration is large, undershoot occurs in the azimuth direction where the aberration is small. It means you can't.

  Further, in the method disclosed in Patent Document 1, although the PSF asymmetry and changes to the image height are taken into consideration, only the PSF asymmetry in the image height direction is considered. For this reason, the correction filter used in the method is also a one-dimensional filter. However, this cannot improve asymmetry in directions other than the image height direction. Furthermore, in the method disclosed in Patent Document 1, the asymmetry of the filter is adjusted by the number of minus tap coefficients. However, even in the image height direction, if the actual asymmetry is different from the blurring caused by the PSF of the optical system, it cannot be sharpened sufficiently.

  Further, in order to cope with a change due to the image height of the PSF, it is necessary to divide the input image into a plurality of areas and to store in advance the PSF and optical information corresponding to each area. In particular, in order to perform unsharp mask processing with high accuracy, it is necessary to finely divide the input image into a large number of regions and hold PSF and optical information for each region. However, this increases the amount of data to be held.

  The present invention provides an image processing apparatus, an image processing method, and the like that enable high-precision sharpening processing while suppressing an increase in the amount of data to be held.

An image processing apparatus according to one aspect of the present invention generates a sharpened image by performing correction processing for sharpening on an input image generated by imaging through an optical system. The apparatus relates to a PSF acquisition means for acquiring information related to a point image intensity distribution function at a plurality of specific image heights, which is a point image intensity distribution function of an optical system according to an imaging condition, and a point image intensity distribution function data generating filter generating means for generating a plurality of filters which respond respectively are paired, a plurality of first data by applying a plurality of filters respectively to the input image into a plurality of specific image height with information Generation means, weight acquisition means for acquiring weight information according to the image height, and second data that is a weighted composition of a plurality of first data using the weight information and is used for correction processing or a second sharpened image have a synthesizing means for generating a data filter generation unit generates a plurality of filters using the difference between the respectively ideal point image of a point image intensity distribution function of a plurality of specific image height, data generation Means A plurality of filters are convolved and integrated with the force image to generate a plurality of first data, and the sharpened image is obtained by correcting the correction data generated using the second data in the correction process with respect to the input image. It is generated by adding .

  Note that an imaging apparatus including the image processing apparatus also constitutes another aspect of the present invention.

An image processing method according to another aspect of the present invention is a method of generating a sharpened image by performing a correction process for sharpening an input image generated by imaging through an optical system. . The method obtains information on a point image intensity distribution function of an optical system according to an imaging condition of an image, and the information on the point image intensity distribution function at a plurality of specific image heights, and uses the information on the point image intensity distribution function generating a plurality of filters which respond respectively are paired to a plurality of specific image height, by applying a plurality of filters each generate a plurality of first data to the input image, the weight information corresponding to the image height Obtaining and performing weighted synthesis of a plurality of first data using weight information to generate data used for correction processing or second data that is a sharpened image, and the plurality of filters have a plurality of specific image heights. Is generated by using the difference between each of the point image intensity distribution functions and the ideal point image, and a plurality of first data are generated by convolution integration of a plurality of filters with respect to the input image, respectively, and are sharpened. Image is corrected Characterized in that it is produced by adding the correction data generated for the input image using the second data.

  An image processing program that causes a computer to perform the image processing method also constitutes another aspect of the present invention.

  According to the present invention, it is possible to perform correction processing for sharpening with high accuracy while reducing the amount of data of the point image intensity distribution function to be prepared (held).

1 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 1 of the present invention. 3 is a flowchart illustrating image processing performed in the first embodiment. FIG. 3 is a diagram illustrating unsharp mask processing (sharpening processing) performed in the first embodiment. FIG. 3 is a diagram illustrating a PSF of an imaging optical system in the imaging apparatus according to the first embodiment. The figure which shows the unsharp mask process using a rotationally symmetrical unsharp mask. The figure which shows the unsharp mask process using a non-rotationally symmetrical unsharp mask. FIG. 3 is a diagram illustrating an unsharp mask used in the first embodiment. Sectional drawing of the said unsharp mask. 5 is a flowchart illustrating details of image processing performed in the first embodiment. The figure which shows a Bayer arrangement | sequence. FIG. 6 is a diagram illustrating an input image dividing method according to the first embodiment. FIG. 3 is a diagram illustrating an interpolation method in the image height direction in Embodiment 1. 6 is another flowchart showing details of image processing performed in the first embodiment. 9 is another flowchart showing details of image processing performed in the first embodiment. 9 is a flowchart showing details of image processing performed in Embodiment 2 of the present invention. 10 is a flowchart showing details of image processing performed in Embodiment 3 of the present invention. The figure explaining the determination method of the image height position holding PSF. 6 is a flowchart showing processing for determining an image height position that holds a PSF. The figure which shows the change of the asymmetry of a point image intensity distribution function.

  Embodiments of the present invention will be described below with reference to the drawings.

  First, an unsharp mask process as a sharpening process performed in a specific example described later will be described with reference to FIG. A solid line in FIG. 3A indicates a captured image generated by imaging through an imaging optical system (hereinafter simply referred to as an optical system), and a broken line indicates an image obtained by blurring the captured image with an unsharp mask (unsharp image). ). Moreover, a dotted line shows the image (sharpening image) after sharpening. A solid line in FIG. 3B indicates a correction component (also referred to as a correction signal) that is a difference between the captured image and the unsharp image.

Here, when the captured image is f (x, y) and the correction component is h (x, y), the sharpened image g (x, y) can be expressed by the following equation (1).
g (x, y) = f (x, y) + m × h (x, y) (1)
In Expression (1), m is an adjustment coefficient for changing the strength of correction, and the correction amount can be adjusted by changing the value of the adjustment coefficient m. m may be a fixed constant regardless of the position of the input image, or may be adjusted according to the position of the input image by varying it according to the position of the input image. Further, the adjustment coefficient m can be varied according to imaging conditions such as the focal length, aperture value, and subject distance of the photographing optical system.
The correction component h (x, y) is assumed to be USM as an unsharp mask.
h (x, y) = f (x, y) -f (x, y) * USM (x, y) (2)
Or, transform the right side of Equation (2),
h (x, y) = f (x, y) * (δ (x, y) −USM (x, y)) (3)
It can be expressed. * Represents a convolution integral (convolution), and δ represents a delta function (ideal point image). The delta function is data in which the number of taps is equal to USM (x, y), the value of the center tap is 1, and the values of all other taps are 0.

  Expression (3) differs from Expression (2) in the calculation method as a process, but Expression (3) can be expressed by modifying Expression (2), so Expression (2) and Expression (3) are equivalent. It can be said that it is a processing. Here, generation of a correction component will be described using Expression (2).

  In Expression (2), the correction component h (x, y) is obtained by taking the difference between the captured image f (x, y) and the unsharp image obtained by blurring the captured image f (x, y) with the unsharp mask USM. Is generated.

  In general unsharp mask processing, smoothing filters such as a Gaussian filter, a median filter, and a moving average filter are used for the unsharp mask USM. For example, when a Gaussian filter is used as an unsharp mask USM for a captured image f (x, y) that can be represented by a solid line in FIG. 3A, the captured image f (x, y) is blurred. The unsharp image is shown by a dotted line in FIG. At this time, the correction component h (x, y) is the difference between the captured image f (x, y) and the unsharp image as shown in the equation (2), and therefore, from the solid line in FIG. By subtracting the broken line in the figure, the correction component represented by the solid line in FIG. 3B can be obtained. By performing the calculation of Expression (1) using the correction component calculated in this way, the captured image f (x, y) indicated by a solid line in FIG. 3A is indicated by a dotted line in FIG. It can be sharpened as shown.

  Next, an unsharp mask process (correction process) using a point image intensity distribution function (hereinafter referred to as PSF) of the optical system as an unsharp mask USM is performed on a captured image deteriorated by imaging through the optical system. A method for obtaining a sharpened image will be described.

The photographed image f (x, y) obtained through the optical system has a pre-imaging image (subject image) as I (x, y) and the PSF of the optical system as psf (x, y). ,
f (x, y) = I (x, y) * psf (x, y) (4)
It can be expressed as.

  Here, if the optical system is a rotationally symmetric coaxial optical system, the PSF corresponding to the central portion of the captured image is rotationally symmetric. For this reason, a sharpening process that brings the captured image f (x, y) closer to the original image I (x, y) is realized by applying a rotationally symmetric unsharp mask USM to the center of the captured image. be able to. Since the correction amount at this time is a difference value between the captured image and the unsharp image, in order to correct with higher accuracy, a simple smoothing filter is not used as the unsharp mask USM, but more psf ( It is better to use a mask having a shape close to x, y). For example, when an image deteriorates due to spherical aberration, the use of the spherical aberration as a mask affects the rotational symmetry, but a mask having a different distribution shape from the PSF due to spherical aberration, such as a smoothing filter such as a Gaussian filter, is used. This is because correction can be performed with higher accuracy than in the case. As described above, even when rotationally symmetric degradation (blur) is reduced, correction can be performed with higher accuracy by using the PSF of the optical system as the unsharp mask USM.

  For this reason, in Example 1 described later, PSF is used as the unsharp mask USM. In FIG. 3A used in the description of the unsharp mask process, the captured image f (x, y) has a symmetric shape, but it does not necessarily have a symmetric shape. Even if the shape of the original image I (x, y) is rotationally asymmetric, if the deterioration function corresponding to psf (x, y) is rotationally symmetric, sharpening using a rotationally symmetric unsharp mask USM is performed. It can be carried out.

  On the other hand, even if the optical system is a rotationally symmetric coaxial optical system at positions other than the center of the image, the PSF usually has a rotationally asymmetric shape. 4 shows the PSF of the optical system in the xy plane, FIG. 4A shows the on-axis PSF, and FIG. 4B shows the off-axis PSF. For example, if the original image (subject) is an ideal point image, the photographed image f (x, y) is the PSF of the optical system from the above equation (4). If there is an ideal point image at the angle of view corresponding to FIG. 4B, and the original image (subject) has deteriorated due to the influence of the PSF of the optical system, the image obtained as the photographed image is as shown in FIG. ) Is a blurred image. A case where sharpening by unsharp mask processing is performed on an image that is thus rotationally asymmetric will be described below.

  FIGS. 5 and 6 show unsharp mask processing for a rotationally asymmetrically deteriorated image. FIG. 5 shows a case where a rotationally symmetric unsharp mask is used, and FIG. 6 shows a case where a rotationally asymmetric unsharp mask is used. Each case is shown. A solid line in FIGS. 5A and 6A represents a cross section in the y-axis direction of a blurred photographed image as shown in FIG. 4B, and a dotted line represents the photographed image using an unsharp mask. Represents a blurred unsharp image. A Gaussian filter is used as the rotationally symmetric unsharp mask in FIG. 5, and an optical PSF is used as the non-rotationally symmetric unsharp mask in FIG. FIGS. 5B and 6B show the difference between the unsharp image in the y-axis direction and the original captured image, that is, the correction component. Here, of the captured images shown in FIGS. 5A and 6A, the one that is more blurred by PSF and has a wider skirt is defined as the plus side in the y-axis direction.

  First, when the rotationally symmetric unsharp mask shown in FIG. 5 is used, the difference between the blurred image and the original image on the plus side with respect to the peak position of the solid line in FIG. On the side, the difference value between the blurred image and the original image is large. For this reason, the extreme value of the correction component shown in FIG. 5B is also smaller on the minus side than on the plus side with respect to the central peak position. As can be seen from the comparison of the curves shown in FIG. 5A and FIG. 5B, the correction amount of the correction component is small on the plus side of the photographed image, and the correction amount is large on the minus side with a narrow skirt. For this reason, with the correction component shown in FIG. 5B, rotationally asymmetric blur cannot be corrected even if sharpening is performed according to equation (4). For example, as a method of adjusting the correction amount without changing the unsharp mask, a method of changing the adjustment coefficient m in Expression (4) is conceivable.

  However, if the adjustment coefficient m is increased in order to sufficiently correct the plus side of the photographed image, the minus side of the photographed image becomes overcorrected (undershoot). Conversely, when the value of the adjustment coefficient m is set so that the negative correction amount of the captured image is appropriate, sufficient correction is not performed on the positive side of the captured image, resulting in insufficient correction.

  Thus, even if unsharp mask processing is performed using a rotationally symmetric unsharp mask on a rotationally asymmetric blurred image, it is difficult to improve and sharpen the asymmetry. In addition, although the case where the Gaussian filter is used as the rotationally symmetric unsharp mask has been described here, the same tendency is observed even when other types of rotationally symmetric filters are used, and the photographed image that is rotationally asymmetrically blurred can be sufficiently obtained. It cannot be sharpened.

  On the other hand, the case where the rotationally asymmetric unsharp mask shown in FIG. 6 is used will be described. In this case, the difference between the image blurred on the plus side and the original image with respect to the peak position of the solid line in FIG. 6A is large, and the difference between the image blurred on the minus side and the original image is large. This is the reverse of FIG. Therefore, the extreme value of the correction component shown in FIG. 6B is also smaller on the plus side than on the minus side with respect to the center peak position. If the correction component shown in FIG. 6B is applied to the photographed image indicated by the solid line in FIG. 6A, the correction amount for the larger plus side blur with respect to the peak position increases, and the minus side. The correction amount for the smaller blur is smaller.

  Accordingly, when such a rotationally asymmetric unsharp mask is used, the magnitude of the blur of the photographed image on the plus side and the minus side coincides with the magnitude of the correction component (correction amount), and thus a rotationally symmetric unsharp mask is used. In this case, too much or insufficient correction, which is a problem, is less likely to occur. In addition, since it is less likely to be overcorrected as compared to the case of using a rotationally symmetric unsharp mask, the value of the adjustment coefficient m in equation (4) can be set relatively large. For this reason, it is possible to obtain a sharper image while reducing asymmetry. From the above, also in the embodiment, a filter having a rotationally asymmetric coefficient in the filter plane is used.

  Further, in order to perform correction with higher accuracy, the magnitude of the correction component (correction amount) is determined by the difference between the blurred image and the original image, and therefore the portion that is greatly blurred by the PSF of the optical system is determined by the unsharp mask. It needs to be more blurred than the other parts. Thus, in order to perform correction with higher accuracy, it is desirable to use the PSF of the optical system as an unsharp mask.

  Next, how to obtain the position of the image height using PSF as an unsharp mask will be described. As described above, since the PSF changes depending on the position of the image height, the unsharp mask and the correction component must be changed according to the image height. In order to perform correction with higher accuracy, if the image height is divided finely and PSF data at each image height is held, the amount of data to be held increases. For this reason, in this embodiment, the image height (specific image height) holding the PSF data is selected based on the optical information (information such as PSF and aberration) of the optical system. Enable unsharp mask processing.

  FIG. 17A shows the change of the PSF with respect to the image height. The horizontal axis represents the image height, and the vertical axis represents the mean square error (MSE) between the PSF on the axis and the PSF at each image height other than on the axis. In this figure, the image height is divided into 10, and the MSE at each image height is indicated by white circles. The performance of the optical system is basically inferior to the axis on the peripheral side, but this is not necessarily the case, and the performance deteriorates at the intermediate image height, or conversely, the performance improves at the intermediate image height. Sometimes.

When the PSF on the axis is P ₀ (x, y) and the PSF at the image height i is Pi (x, y), the mean square error MSE of the PSF on the axis and the image height i is expressed by the following equation (11). Can be expressed as

N represents the total number of divisions of PSF.

  If the PSF generated by linear approximation using a PSF on the axis and a 100% image height is PLi (x, y), the mean square error MSE of the PSF and PLi (x, y) on the axis is Equation (12) is obtained.

  In FIG. 17A, the mean square error MSE of PSF and PLi (x, y) on the axis shown in Expression (12) is represented by a black circle. For example, a PSF generated by linear approximation of P10 (x, y) and P10 (x, y) corresponding to 50% image height is P5 (x, y) as PL5 (x, y). Then, PL5 (x, y) is expressed by the following equation (13).

Furthermore, by substituting equation (13) into equation (12),

It becomes. From this equation (14), the mean square error MSE of P0 (x, y) and PL5 (x, y) may be half of the mean square error MSE of P0 (x, y) and P10 (x, y). Recognize. That is, the mean square error MSE of P0 (x, y) and PLi (x, y) corresponding to the image height i is on the dotted line connecting the on-axis and the white circle of 100% image height as shown in FIG. Is plotted in

  Further, there are three image height areas A from the on-axis to 30% image height, image height area B from 30% image height to 50% image height, and image height area C from 50% image height to 100% image height. A case where the image is divided into regions and the PSF of each image height is linearly approximated from four PSFs will be described. As shown in FIG. 17A, in the image height region A, the distance (deviation amount) between the white circle and the black circle of each image height is large, and the actual PSF corresponding to the white circle is the axis corresponding to the black circle and the PSF of 100% image height. It can be seen that it is closer to the on-axis PSF than the PSF generated by linear approximation.

  On the other hand, when interpolation by linear approximation is performed for each image height region between four image heights of 30%, 50%, and 100% on the axis indicated by the solid line, the actual value is higher than that indicated by the dotted line. There is little deviation from the PSF, and it becomes possible to accurately interpolate PSFs at other image heights other than the above four specific image heights. In this way, the interpolation accuracy can be improved by increasing the number of specific image heights (number of image height regions), but in order to reduce the amount of PSF data to be retained as much as possible, the number of specific image heights should be as small as possible. It is necessary to select a small number. As a method for selecting a specific image height, for example, as shown in FIG. 17A, an MSE of an on-axis PSF and an actual PSF at an image height i, an on-axis PSF, and a PSF generated by linear approximation are used. The difference in MSE is defined as the divergence amount di. A method of calculating the image height i that minimizes the sum of di from the on-axis to the 100% image height and holding the minimum image height i is conceivable. In this example, the MSE is calculated as a relative value to the PSF on the axis for the actual PSF and the approximate PSF, and the difference is used as the divergence amount. May be.

  The flowchart in FIG. 18 shows a specific flow of the specific image height selection process. Here, description will be made assuming that the selection process of the specific image height is performed by the computer (image height selection means) according to the computer program.

  First, in step S011, the computer acquires the number of specific image heights holding the PSF (hereinafter referred to as specific image height number). A process may be performed in which only the maximum value is set as the specific image height number, and a specific image height number equal to or less than the maximum value is determined in the process. Here, as an example, the specific image height number N is four.

  In step S012, the computer provisionally determines a specific image height. Since N = 4 is set in step S011, here, for example, on-axis, 30% image height, 50% image height, and 100% image height are provisionally determined as specific image heights. The specific image height temporarily determined here is used as an initial value (initial selection pattern) when performing an iterative operation to search for the optimum specific image height, and is evaluated while changing the specific image height in the iterative operation. To search for the optimum specific image height. Here, the optimal specific image height is searched with the on-axis and 100% image height fixed, and the 30% image height and 50% image height as variables.

  Subsequently, in step S013, the computer calculates an evaluation function based on the specific image height provisionally determined in step S012. Here, as shown in FIG. 17A, for image heights other than the temporarily determined specific image height, a solid line is obtained by linear approximation, and the sum (distance) di between the solid line and the white circle is taken and evaluated. Value. Note that, in FIG. 17A, the divergence amount di indicates the interval between the dotted line and the white circle, but in this process, it is the interval between the solid line and the white circle.

  Next, in step S014, the computer determines whether or not the evaluation value obtained in step S013 is the minimum. Since there is only one evaluation value obtained from the evaluation function in the first process, this is the minimum value. If the evaluation value calculated in the second and subsequent processes is smaller than the already calculated evaluation value, the process proceeds to step S015, and if larger, the process skips step S015 and proceeds to step S016.

  In step S015, since the evaluation value is determined to be the minimum value in step S014, the computer updates the specific image height as a variable. If multiple specific image heights are used as variables, reset them. Then, control goes to a step S016.

  In step S016, the computer determines whether evaluation values have been calculated for all selected patterns having a specific image height as a variable. If the calculation has been completed for all the selected patterns, the specific image height updated in the last step S015 is set as the official specific image height, and this processing ends. If the calculation has not been completed for all the selected patterns, the computer returns to step S012, selects a specific image height with another selected pattern, and repeats the processes of steps S013 to S016 again.

  In the case shown in FIG. 17A, the on-axis and 100% image heights are fixed, and two image heights are selected from the remaining nine image heights. The processes in steps S013 to S016 are repeatedly performed on the selected pattern. In the example shown in FIG. 17A, the image height from the axis to 10% image height is divided into 10 at 10% image height intervals. However, the number of image height divisions may be increased or decreased. Alternatively, the image height from the axis up to 100% may be divided at unequal intervals.

  In addition, the evaluation function is not limited to the MSE, and other evaluation functions such as a peak signal-to-noise ratio (PSNR) may be used. Further, instead of evaluating with PSF, evaluation may be performed with wavefront aberration or geometric aberration of the optical system.

  In this way, by performing the selection process shown in FIG. 18, an appropriate (optimum) specific image height can be selected in order to perform highly accurate unsharp mask processing with a small amount of retained data.

  As the evaluation value, the sum of the divergence amounts di of PSF (MSE) at each image height with respect to the axis is used. For example, FIG. 19 shows PSFs at 50% image height and 10% image height on the axis in a certain optical system. This optical system is an optical system that has rotationally asymmetric aberrations such as coma aberration, and in which the direction of coma aberration is reversed depending on the image height. As shown in the figure, the PSFs at the 50% image height and the 100% image height are just 180 degrees rotated. In this case, since the evaluation value described in FIG. 18 uses a difference (deviation amount di) from the on-axis MSE, the evaluation value is equal between the 50% image height and the 100% image height. Also, when the PSF is the same at the 50% image height and the 100% image height, the evaluation values are the same, so it is impossible to distinguish them.

  As a countermeasure against such a problem, not only the PSF MSE on the axis and each image height, but also, for example, the PSF is calculated for every 10% image height, and the MSE of the nth image height and the (n + 1) th image height PSF. It is good to calculate and plot. Thus, even when the PSF rotates or reverses as shown in FIG. 19, it is possible to cope with it. That is, in the example shown in FIG. 19, the change in the evaluation value becomes large between the 50% image height and the 100% image height, and the change in the PSF distribution can be detected. Thus, by using the two evaluation functions (indexes) of the PSF MSE of the nth image height and the n + 1th image height and the PSE MSE on the original axis and each image height, a more appropriate specific image is obtained. Can lead to high.

  Further, as a method of selecting a specific image height other than the method using such an evaluation function, the PSF of each image height region is fitted with a function, the inflection point of the function is obtained, and the image height located at the inflection point is calculated. You may make it hold | maintain as specific image height. Such a method may be used in combination with the method described in FIG.

  Next, data interpolation processing (weighting synthesis) in the embodiment will be described. In FIG. 17A, the case where interpolation (linear interpolation) by linear approximation is performed on the data between the selected specific image heights (image height regions) has been described. However, the interpolation method does not necessarily need to be linear interpolation, and by performing nonlinear interpolation according to the performance of the optical system, it is possible to further reduce the data amount and increase the accuracy of correction. The target of data to be interpolated is not limited to the PSF described above, but may be a filter, an unsharp image, a correction component (correction signal), or a sharpened image. Here, the case of interpolating a sharpened image will be described.

  In FIG. 17B, the 50% image height and the 100% image height on the axis are selected as the specific image height, the image height region A from the axis to the 50% image height, and the 100% image height from the 50% image height. An image height region B up to high is provided. The image height area A corresponds to the image height areas A and B shown in FIG. 17A, and the image height area B corresponds to the image height area C shown in FIG. In FIG. 17B, nonlinear interpolation is performed in the image height region A, and linear interpolation is performed in the image height region B as in FIG.

In the image height region A from the on-axis to the 50% image height, a sharpened image generated by performing the unsharp mask process using the on-axis PSF as the unsharp mask is assumed to be Xa. A sharpened image generated by performing unsharp mask processing using a PSF at 50% image height as an unsharp mask is denoted by Xb. At this time, the sharpened image Xw in the image height region A is expressed by the following equation (101):
Xw = w × Xb + (1−w) × Xa (101)
It can be expressed as. Here, when the variable h corresponding to the image height (0 ≦ h ≦ 1) is used and the weight w = h, linear interpolation is performed. A sharp image on the axis is obtained when h = 0, a sharp image corresponding to 2.5% image height is obtained when h = 0.5, and a 50% image height when h = 1. A sharpened image can be obtained.

  On the other hand, in order to change Xw nonlinearly (that is, to perform interpolation processing nonlinearly with respect to the image height) as shown in the image height region A in FIG. , Weight w

And in the form of a power series. As a result, any non-linear interpolation processing can be realized. In the formula (102), _{c n} is a constant term, N is the largest degree, it is necessary to normalized so that they _c n, N to w = 1 when h = 1. For example, if N = 2, C ₁ = 0, and C ₂ > 0, w becomes a quadratic function, and the sharpened image corresponding to the intermediate image height according to the equation (101) is the whole as compared with the case of linear interpolation. Shift to the sharpened image side on the axis. In particular, the performance of the optical system is basically good in the vicinity of the axis and often changes little, and the SPF tends to decrease in the vicinity of the axis similarly. From this, it is possible to generate a sharpened image corresponding to the intermediate image height more efficiently by performing non-linear processing according to equations (101) and (102). Note that the method of expressing w is not limited to the power series as shown in Equation (102), but may be expressed in another form (polynomial).

  Next, interpolation processing other than the above-described interpolation processing, which considers the optical characteristics (particularly PSF) of the optical system, will be described. In FIG. 17C, the performance of the optical system deteriorates in the vicinity of 70% image height than the 100% image height, and each performance for the PSF on the axis when the performance improves from 70% image height to 100% image height. The change of MSE of PSF of image height is shown. Here, the specific image height is three on the axis, that is, 70% image height and 100% image height.

  When a sharpened image corresponding to an intermediate image height is generated by unsharp mask processing using PSF obtained by linear interpolation from the PSF on the axis and at a 100% image height as indicated by the dotted line, the original PSF is used. As a result, unsharp mask processing is performed with the PSF of an optical system with good performance. For this reason, this unsharp mask process is insufficiently corrected.

  On the other hand, the solid line has the same specific image height at the three image heights, 70% image height and 100% image height on the axis, but by implementing an interpolation method, interpolation processing closer to actual optical characteristics is realized. doing. In the interpolation processing indicated by a solid line in FIG. 17C, the image height region A and the image height region B are divided into the image height region A and the image height region B with the 70% image height as a boundary. For the image height region A, the sharpened image corresponding to the intermediate image height is obtained by processing Xa in the formula (101) as a sharpened image on the axis and Xb as a sharpened image having a 70% image height. You can get it.

  On the other hand, with respect to the image height region B, Xa and Xb are interchanged, and Xa is a sharpened image with a 70% image height, and Xb is a sharpened image on the axis. An approximate function as indicated by a solid line in the figure can be obtained even on the axis and 70% image height. In the image height region B, when the variable w = 1, an on-axis sharpened image is obtained. Further, as shown in FIG. 17C, in order to fit a sharpened image corresponding to the 100% image height between the 70% image height and the on-axis sharpened image, for example, 0 ≦ w ≦ 0.5 In this way, the maximum value of the variable may be set to a value smaller than 1.

  As described above, the specific image height calculation method and the data interpolation processing method considering the optical characteristics have been described. However, the specific image height calculation method and the data interpolation processing method are interdependent, and which method is independent of each other. Rather than deciding whether to adopt, it is desirable to decide on the same premise. For example, if the interpolation process in a certain image height region is performed non-linearly, the evaluation function for calculating the specific image height is assumed to be a non-linear function. The accuracy is further improved, and a more appropriate specific image height can be selected.

Moreover, regarding the interpolation processing, the description of FIGS. 17A to 17C has described the interpolation processing using data corresponding to two specific image heights, but data corresponding to a larger number of specific image heights. Interpolation processing may be performed using. When interpolation processing is performed using data at three specific image heights, the interpolation processing may be performed by transforming the equation (101) into the following equation (103).
Xw = w1 * Xc + w2 * Xb + (1-w1-w2) * Xa (103)
Here, the weights w1 and w2 in the equation (103) are expressed as the equation (102), respectively, and can be made non-linear when N ≧ 2 or can be changed linearly when N = 1. By performing the interpolation process using the equation (103), the interpolation process using the data of three specific image heights can be performed. Similarly, when data of four specific image heights are used, it can be realized by increasing the number of terms. According to these, since the degree of freedom increases as compared with the case of using data at two specific image heights, the interpolation accuracy can be improved.

  Next, a filter, a correction signal (correction data), and unsharp mask processing in the first to third embodiments used in the first to third embodiments will be described.

In the first embodiment, unsharp mask processing is performed using the following equation (5) derived from the equations (1) and (2) described above.
g (x, y) = f (x, y) + m × {f (x, y) −f (x, y) * USM (x, y)}
... (5)
In the second embodiment, unsharp mask processing is performed using the following equation (6) derived from the equations (1) and (3) described above.
g (x, y) = f (x, y) + m × f (x, y) * {δ (x, y) −USM (x, y)}
... (6)
In the third embodiment, unsharp mask processing is performed using the following equation (7) obtained by further modifying equation (6).
g (x, y) = f (x, y) * {δ (x, y) + m × (δ (x, y) −USM (x, y))} (7)
USM (x, y) in equations (5) to (7) is an unsharp mask.

  In the first embodiment, USM (x, y) is used as a filter, and in the second embodiment, the difference (δ (x, y) −USM (x, y)) between the unsharp mask (PSF) and the ideal point image is used as a filter. Use.

  In the third embodiment, the difference between the unsharp mask (PSF) and the ideal point image is multiplied by the adjustment coefficient m, and the result is added to the ideal point image {δ (x, y) + m × (δ (X, y) -USM (x, y))}, that is, a correction signal is used as a filter.

  FIG. 1 illustrates the configuration of the imaging apparatus according to the first embodiment. The imaging apparatus 100 is provided with an image processing unit (image processing apparatus) 104 therein. The image processing unit 104 executes unsharp mask processing (image processing method) as sharpening processing on a captured image (input image) according to an image processing program as a computer program.

  The imaging apparatus 100 includes an imaging optical system (hereinafter referred to as an imaging lens) 101 and an imaging apparatus main body (hereinafter referred to as a camera main body). The imaging lens 101 includes a diaphragm 101a and a focus lens 101b. In this embodiment, a lens-integrated image pickup apparatus in which the image pickup lens 101 is provided integrally with the camera body will be described. It may be an imaging device.

  The imaging element (imaging means) 102 is a two-dimensional photoelectric conversion element such as a CCD sensor or a CMOS sensor. The image sensor 102 photoelectrically converts a subject image (imaging light) formed by the imaging lens 101 to generate an analog electric signal. The analog electrical signal output from the image sensor 102 is converted into a digital signal by the A / D converter 103, and this digital signal is input to the image processing unit 104 as an image signal.

  The image processing unit 104 performs predetermined processing on the input imaging signal to generate a captured image, and performs unsharp mask processing on the captured image. First, the image processing unit 104 acquires information on an imaging condition that is a state of the imaging lens 101 at the time of imaging for acquiring a captured image from the state detection unit 107. The imaging conditions include an aperture value and subject distance (the position of the focus lens 101b), and when the imaging lens 101 is a zoom lens, includes the focal length. The state detection unit 107 may acquire information on imaging conditions from the system controller 106 or may be acquired from the lens control unit 105 that controls the imaging lens 101.

  The image processing unit 104 includes a PSF selection unit 201, a correction signal generation unit 202, a correction signal application unit 203, a weight information acquisition unit 204, and a signal synthesis unit 205 in order to perform unsharp mask processing. The PSF selection unit 201 corresponds to an image height selection unit and a PSF acquisition unit, and the correction signal generation unit 202 corresponds to a filter generation unit and a data generation unit. The correction signal application unit 203 corresponds to a correction unit that performs unsharp mask processing (correction processing). The weight information acquisition unit 204 corresponds to a weight acquisition unit, and the signal synthesis unit 205 corresponds to a synthesis unit. Then, the image processing unit 104 converts the sharpened image generated by the unsharp mask process into data of a predetermined format and stores it in the image recording unit 108.

  The image display unit 112 displays a display image obtained by performing a predetermined display process on the sharpened image. The control of each unit described so far is performed by the system controller 106. The mechanical control of the imaging lens 101 is controlled by the lens control unit 105 based on an instruction from the system controller 106.

  An optical element such as a low-pass filter or an infrared cut filter may be inserted into the imaging lens 101. When using an optical element that affects the characteristics of the PSF, such as a low-pass filter, more accurate unsharp mask processing can be performed by considering the influence of the optical element at the stage of creating the unsharp mask. In addition, since the infrared cut filter affects each PSF of the RGB channel, particularly the PSF of the R channel, which is an integral value of the PSF of the spectral wavelength, the influence of the infrared cut filter is considered at the stage of creating the unsharp mask. It is preferable to do.

  Next, the basic flow of unsharp mask processing in the present embodiment will be described using the flowchart of FIG. This process is executed by the image processing unit 104 as a computer that receives an instruction from the system controller 106 in accordance with the above-described image processing program.

  In step S11, the image processing unit 104 acquires a captured image as an input image.

  Next, in step S12, the PSF selection unit 201 first selects a specific image height by the process shown in FIG. Next, the PSF selection unit 201 selects and acquires information regarding the PSF of the imaging lens 101 corresponding to the specific image height from the memory 109. At this time, the information about the PSF acquired by the PSF selection unit 201 may be data of a two-dimensional tap indicating the PSF, or a plurality of one-dimensional taps that are constituent elements of the PSF in order to compress the data amount or speed up the processing. Data may be used. Further, PSF coefficient information may be acquired as information about PSF. In the following description, information related to PSF is referred to as PSF information.

  Next, in step S13, the correction signal generation unit 202 generates an unsharp mask and a correction signal (correction data) using the PSF information acquired in step S12.

  7 and 8 show examples of unsharp masks. The number of taps of the unsharp mask is determined according to the aberration characteristics of the imaging lens 101 and the required sharpening accuracy. FIG. 7 shows an unsharp mask having a filter size of 11 × 11 taps arranged two-dimensionally as an example of the unsharp mask. In FIG. 7, the value of each tap (filter coefficient: hereinafter referred to as tap value) is omitted, but the tap value in one section of this unsharp mask is shown in FIG. The tap value distribution in the unsharp mask is a distribution close to the shape of the signal value spread by the aberration (that is, the PSF of the imaging lens 101), that is, a rotationally asymmetric tap value distribution around the center tap shown by hatching in FIG. Ideal. In this way, the unsharp mask may be generated using the PSF information, or the PSF indicated by the PSF information acquired by the PSF selection unit 201 may be used as it is as the unsharp mask.

  Subsequently, the correction signal generation unit 202 generates a correction signal using an unsharp mask. The correction signal generation process will be described in detail later.

  Next, in step S14, the correction signal application unit 203 applies the correction signal generated by the correction signal generation unit 202 in step S13 to the input image. That is, unsharp mask processing is performed on the input image. The contents of the processing in step S14 will be described in detail later.

  When the unsharp mask is generated using the PSF information in step S13, the PSF changes according to the image height (position in the image) of the input image. Therefore, the unsharp mask needs to be changed according to the image height. . In the present embodiment, any one of PSF information, an unsharp image after applying an unsharp mask to the input image, a correction signal, and a sharpened image after unsharp mask processing at a plurality of specific image heights (first data). 1 data). Then, by performing an interpolation process using intermediate data corresponding to these specific image heights, intermediate data corresponding to all image height PSFs is obtained. The interpolation processing at this time will also be described in detail later.

  Next, the flow of the unsharp mask process described with reference to FIG. 2 will be described in more detail with reference to the flowchart of FIG.

  First, in step S111, the image processing unit 104 acquires a captured image as an input image. The data actually used as the input image in the unsharp mask process among the captured images is, for example, G channel image data after demosaicing. However, R channel and B channel image data may be used as the input image, or image data of all RGB channels may be used as the input image. Further, image data before demosaicing may be used as an input image.

  FIG. 10 shows a Bayer array used as a pixel array of a general image sensor. Image data of a specific channel (for example, G channel) may be simply extracted from the captured image generated based on the image pickup signal from the image pickup device having such a Bayer array, or may be input image data. May be used as an input image. In FIG. 10, the G channel is divided into two channels G1 and G2. By dividing the G channel into two, the resolution of the image data of each of the R, G1, G2, and B channels becomes equal, so that processing and data processing are easy.

  Subsequently, in step S112, the PSF selection unit 201 selects a specific image height by the processing illustrated in FIG. 18 and further acquires PSF information of the imaging lens 101 corresponding to the specific image height from the memory 109. As described in step S12 of FIG. 2, the PSF information to be acquired may be two-dimensional tap data indicating the PSF or data of a plurality of one-dimensional taps that are constituent elements of the PSF. As a method of decomposing two-dimensional data into a plurality of one-dimensional data, for example, there is a singular value decomposition theorem. The principal component decomposed using such a theorem may be recorded in the memory 109, and data of a plurality of one-dimensional taps corresponding to the principal component of the PSF may be acquired according to the imaging condition of the imaging lens 101. .

  Since the PSF changes depending on the image height as described above, in order to perform unsharp mask processing with higher accuracy, an unsharp mask is generated using the PSF corresponding to the image height, and this unsharp mask is used. It is necessary to generate a correction signal corresponding to the image height. However, since the data capacity that can be recorded in the memory 109 is limited, it is not preferable to store PSF information corresponding to each of a large number of image heights in the memory 109. For this reason, in this embodiment, the input image is divided into a plurality of image areas, and interpolation processing is performed using PSF information at at least two specific image heights for each image area. As a result, it is possible to generate an unsharp image blurred by applying an unsharp mask or an unsharp mask corresponding to an image height other than the specific image height. The method of interpolation processing will be described in detail in later steps S114 and S115.

  Here, the division of the input image into a plurality of image regions will be described. FIG. 11 shows an input image. Here, the long side direction of the input image is the x-axis direction, the short side direction is the y-axis direction, and the center of the input image is the origin of coordinates. In this embodiment, the input image is divided into eight image areas A to H, and PSF information of the origin and the peripheral part of each image area is acquired. Note that the number of divisions of the image area is not limited to eight, and needs to be determined in consideration of the imaging conditions of the imaging lens 101 at the time of imaging, the optical characteristics of the imaging lens 101, the data capacity of the memory, and the like.

  Next, in step S113, the correction signal generation unit 202 performs filtering using the PSF information acquired in step S112. In this embodiment, an unsharp mask is generated from the PSF obtained from the PSF information (in other words, the PSF is directly used as an unsharp mask), and the unsharp mask is convolved (convolution integration) with the input image. Perform filtering. The PSFs acquired in step S112 include nine PSFs corresponding to the center image height (on the axis) that is a specific image height and eight peripheral image heights. For this reason, nine unsharp images are obtained by convolving these nine PSFs as unsharp masks with the input image. These nine unsharp images correspond to intermediate data for unsharp mask processing (correction processing).

  Next, in step S114, the weight information acquisition unit 204 acquires, from the memory 109, weight information used in the image height direction interpolation processing performed in step S115 described later. The weight information is w information in the equations (101) and (102).

  Next, in step S115, the signal synthesis unit 205 uses the weight information acquired in step S114 to weight and synthesize a plurality (nine) of unsharp images (first data) generated in step S113. Process. In this weighting synthesis process, an interpolation process in the image height direction of the unsharp image is performed.

  Here, the interpolation processing in the image height direction will be described with reference to FIG. In FIG. 11, if the direction in which the image area C is located with respect to the origin is the positive side in the x-axis direction, and the direction in which the image area A is located is the positive side in the y-axis direction, FIG. , Y are positive quadrant regions where both are positive. Here, P0 indicates the origin (center image height), and P1, P2, and P3 indicate peripheral image heights in the image areas A, B, and C, respectively. Assume that the PSF selection unit 201 acquires PSF information (hereinafter simply referred to as PSF) of these image heights (specific image heights) P0, P1, P2, and P3 in step S112. In FIG. 12, Pn indicated by a white circle represents an arbitrary image height in the image area B.

  For the arbitrary image height Pn, an unsharp image as intermediate data is obtained by filtering (convolution) the unsharp mask for the arbitrary image height Pn created in step S113 using the PSF of the image height P0 and the image height P2 into the input image. Get. Similarly, for an arbitrary image height in the image area A, an unsharp image as intermediate data is obtained by filtering an unsharp mask for the arbitrary image height created using the image height P0 and the PSF of the image height P1 into an input image. obtain. Furthermore, at an arbitrary image height in the image area C, an unsharp mask as intermediate data is obtained by filtering an unsharp mask for the arbitrary image height created using the PSF having the image height P0 and the image height P3 into the input image. .

Here, a method for generating intermediate data corresponding to a filtering result at an arbitrary image height Pn by an interpolation process using intermediate data at two image heights will be described. As shown in FIG. 12, it is assumed that the distance from the origin P0 to the arbitrary image height Pn in the image area B is d0 and the distance from the image height P2 is d2. In step S113, the unsharp images as the filtering results using the PSFs having the image heights P0 and P2 are set as F0 and F2, respectively. At this time, the data (unsharp image) Fn at the arbitrary image height Pn is
Fn = F0 * (1-d0) + F2 * d2 (8)
It is expressed. This equation (8) becomes equivalent to these equations (101) and (102) by setting Xa = F0, Xb = F2 and w = d0 in the above-mentioned equations (101) and (102). Further, in equation (102), if N = 1, linear interpolation processing is performed, and if N> 1, nonlinear interpolation processing is performed.

  The interpolation processing method and the positions of the specific image heights P1, P2, and P3 holding the PSF are selected in consideration of the optical characteristics of the imaging lens 101 as described above. By such interpolation processing, intermediate data at an arbitrary image height in each image region can be generated. As a result, the unsharp as one intermediate data (second data) corresponding to the PSF for every image height from the nine unsharp images as the first intermediate data (first data) generated in step S113. An image (hereinafter referred to as an interpolated unsharp image) is generated.

  In this way, the interpolation unsharp image corresponding to the filtering result of the input image by the unsharp mask obtained from the PSF that varies from image height to image height is retained, and a limited number of specific image height PSF data with a small amount of data is retained. It is possible to generate by just doing. In this method, there is a possibility that the accuracy may be slightly lowered as compared with the case where the PSF of all image heights is prepared and the input image is filtered with an unsharp mask obtained from the PSF for each image height. However, the amount of retained data can be greatly reduced, and the processing speed (memory data reading speed) can be increased.

  Regarding the interpolation processing in the image height direction, equation (8) shows the calculation formula for the image area B in the first quadrant, but is interpolated by performing similar calculations for other image areas and other quadrants. Intermediate data can be created.

  Next, in step S116, the correction signal generation unit 202 generates a correction signal as final correction data using the interpolated unsharp image generated by the signal synthesis unit 205 in step S115. In the present embodiment, the correction signal (correction component) is represented by Expression (2), and is generated by taking the difference between the input image and the interpolated unsharp image.

  Next, in step S117, the correction signal applying unit 203 generates a target sharpened image by applying the correction signal generated in step S116 to the input image. In this embodiment, the correction signal is applied by adding the correction signal multiplied by m to the input image according to the equation (1). At this time, the value of the adjustment coefficient m is determined in consideration of image noise, overcorrection and undercorrection in sharpening. Here, the expression (1) is expressed in the form of adding the first term and the second term. This is a case where the adjustment coefficient m is negative, and subtraction is performed when the adjustment coefficient m is positive. However, addition and subtraction are simply due to the difference in the sign of the adjustment coefficient m, which means essentially the same thing, and in the present embodiment, these are collectively referred to as addition.

  As described above, in the present embodiment, the PSF of the imaging lens 101 is used as an unsharp mask in the unsharp mask process, so that the input image degraded by the rotationally asymmetric PSF of the imaging lens 101 is sharpened with high accuracy. A sharpened image can be obtained.

  In this embodiment, the case where the interpolation processing in the image height direction is performed after the unsharp mask filtering on the input image has been described. However, the interpolation process in the image height direction may be performed on the correction signal, or may be performed on the sharpened image after the correction signal is applied to the input image. Hereinafter, the unsharp mask process in these cases will be described using the flowcharts shown in FIGS. 13 and 14, respectively.

  FIG. 13 shows a flow of unsharp mask processing when interpolation processing in the image height direction is performed on the correction signal. In FIG. 13, the same steps as those shown in FIG. 9 are denoted by the same reference numerals as in FIG. In FIG. 13, steps S124, S125, and S126 are performed instead of steps S114, S115, and S116 shown in FIG.

  In step S124, the correction signal generation unit 202 generates a plurality of correction signals (first data) as intermediate data from each of the unsharp images corresponding to the PSFs having the specific image heights generated in step S113. To do. Each correction signal is generated by taking the difference between the input image and the unsharp image according to the above-described equation (2).

  Next, in step S125, the weight information acquisition unit 204 acquires weight information for performing interpolation processing in the image height direction from the memory 109 in the next step S126.

  Next, in step S126, the signal synthesis unit 205 performs interpolation processing of the correction signal in the image height direction using the plurality of correction signals generated in step S124 and the weight information acquired in step S125. As a result, a correction signal (second data) as final correction data is generated. The interpolation process performed in step S126 is different from the correction signal and step S115 in FIG. 9, but the basic process flow is the same as that described in step S115. In step S115, PSF was used as an unsharp mask, and an unsharp image obtained by convolving this unsharp mask with an input image was interpolated in the image height direction. On the other hand, in step S126, interpolation processing in the image height direction is performed on the correction signal that is the difference between the input image and the unsharp image. In equation (2), “f (x, y) * USM” is interpolated in the image height direction in step S115, and h (x, y) is interpolated in the image height direction in step S126. It becomes. Therefore, by replacing the data of f (x, y) * USM in step S115 with h (x, y), it is possible to perform a process of interpolating the correction signal in the image height direction in step S126.

Further, the formula (101), when expressed by (102), Xa, Xb is a correction component of each image height, _{c n} in the formula (102) is weight information. The correction signal applying unit 203 applies the correction signal as the correction data obtained by the correction signal interpolation processing in this way to the input image in step S127. The correction signal application processing in step S127 is the same as that in step S117 described above.

  FIG. 14 shows the flow of unsharp mask processing when performing interpolation processing in the image height direction on a sharpened image (first data) obtained by applying a correction signal to an input image. In FIG. 14, the same steps as those shown in FIG. 9 are denoted by the same reference numerals as those in FIG. 14, steps S134, S135, S136, and S137 are performed instead of steps S114, S115, S116, and step S117 shown in FIG. In the process of FIG. 14, the correction signal application unit 203 corresponds to a data generation unit, and the signal synthesis unit 205 corresponds to a correction unit.

  In step S134, the correction signal generation unit 202 generates a plurality of correction signals from each of the unsharp images corresponding to the plurality of specific image height PSFs generated in step S113. Each correction signal is generated by taking the difference between the input image and the unsharp image according to the above-described equation (2).

  Next, in step S135, the correction signal application unit 203 applies the plurality of correction signals generated in step S134 to the input image, respectively, and generates a sharpened image (first data) as a plurality of intermediate data. . Here, a sharpened image is generated by applying each correction signal to the input image according to Equation (1).

  Next, in step S136, the weight information acquisition unit 204 acquires weight information for performing interpolation processing in the image height direction from the memory 109 in the next step S137.

Next, in step S137, the signal synthesis unit 205 performs interpolation processing in the image height direction of the sharpened image using the plurality of sharpened images generated in step S135 and the weight information acquired in step S136. To produce a desired sharpened image. The interpolation process performed in step S137 is different from the sharpened image and step S115 in FIG. 9 in the processing target, but the basic process flow is the same as step S115 and step S126 in FIG. Comparing step S126 and step S137, in the equation, step S126 interpolates h (x, y) in equation (1) in the image height direction, and g (x, y) in the image height direction. In step S137, interpolation is performed. Further, the formula (101), is expressed using a (102), Xa, Xb becomes sharpened image with the PSF for each image height, _{c n} in the formula (102) is weight information.

  According to the present embodiment, by performing interpolation in the image height direction using PSF data at a specific image height, it is possible to cope with a change in the image height of the optical characteristics (PSF) even though the amount of retained data is small. High-precision sharpening processing (unsharp mask processing) can be performed.

  Next, an image pickup apparatus that is Embodiment 2 of the present invention will be described. The imaging apparatus of the present embodiment is different from the imaging apparatus of the first embodiment only in the contents of image processing (unsharp mask processing), and the configuration of the imaging apparatus is the same as that of the imaging apparatus of the first embodiment. For this reason, description of the configuration of the imaging apparatus of the present embodiment is omitted.

  The flowchart of FIG. 15 shows the flow of unsharp mask processing in the present embodiment. Also in this embodiment, the image processing unit 104 that has received an instruction from the system controller 106 performs this processing according to the image processing program.

  The present embodiment is different from the first embodiment in the correction signal generation method. Specifically, the correction signal is generated by the expression (2) in the first embodiment, whereas the correction signal is generated by the expression (3) in the present embodiment. Note that step S211 and step S212 are the same as step S111 and step S112 of the first embodiment (FIGS. 9, 13, and 14), and description thereof is omitted.

  In step S213, the correction signal generation unit 202 generates a plurality of filters by taking the difference between the ideal point image and the PSF having the specific image height selected by the PSF selection unit 201 in step S212.

  Next, in step S214, the correction signal generation unit 202 generates a plurality of correction signals by convolving the plurality of filters generated in step S213 with respect to the input image, respectively.

  Next, in step S215, the weight information acquisition unit 204 acquires weight information for performing interpolation processing in the image height direction from the memory 109 in the next step S216.

  Next, in step S216, the correction signal generation unit 202 performs interpolation processing of the correction signal in the image height direction using the plurality of correction signals generated in step S214 and the weight information acquired in step S125. Since the processing of step S216 and the next step S217 is the same as the processing of step S126 and step S127 described in FIG. 13, detailed description thereof will be omitted. Also in this embodiment, the specific image height and the interpolation method are selected in consideration of the optical characteristics of the imaging lens 101 as described above.

  By performing the above process, the unsharp mask process by Formula (6) can be performed.

  Note that the interpolation processing in the image height direction may be performed on the correction signal as described in the first embodiment, or may be performed on the sharpened image after the correction signal is applied to the input image. Good.

  Next, an image pickup apparatus that is Embodiment 3 of the present invention will be described. The imaging apparatus of the present embodiment is different from the imaging apparatus of the first embodiment only in the contents of image processing (unsharp mask processing), and the configuration of the imaging apparatus is the same as that of the imaging apparatus of the first embodiment. For this reason, description of the configuration of the imaging apparatus of the present embodiment is omitted.

  The flowchart of FIG. 16 shows the flow of unsharp mask processing in the present embodiment. Also in this embodiment, the image processing unit 104 that has received an instruction from the system controller 106 performs this processing according to the image processing program.

  This embodiment is different from the first and second embodiments in the method of generating a correction signal and the method of applying the correction signal to the input image. Specifically, in this embodiment, a correction signal is generated according to Expression (7), and this correction signal is applied to the input image. For this reason, in the flowchart of FIG. 16, step S313 (generation of the correction signal) and step S314 (application of the correction signal) are different from the first and second embodiments. Note that steps S311 and S312 are the same as steps S111 and S112 of the first embodiment (FIGS. 9, 13, and 14), and a description thereof is omitted.

  In step S313, the correction signal generation unit 202 uses each of the plurality of PSFs having specific image heights selected in step S312 as an unsharp mask, and uses a plurality of correction signals corresponding to the portion in the braces of Expression (7). Is generated. In this embodiment, the correction signal generated by the correction signal generation unit 202 is equivalent to a filter.

  Next, in step S314, the correction signal applying unit 203 convolves each of the plurality of correction signals (filters) generated in step S313 with respect to the input image, thereby obtaining a sharpened image (a plurality of intermediate data). 1st data) is generated. As described above, in this embodiment, a sharpened image is generated by convolving the correction signal as a filter generated in advance using the PSF of the imaging lens 101 as an unsharp mask with respect to the input image only once. Can do.

  Step S315 (acquisition of weight information) and step S316 (signal synthesis) are the same as steps S136 and S137 described with reference to FIG. 14, and thus detailed description thereof is omitted. Also in the present embodiment, a target sharpened image (second data) is obtained by weighted synthesis (interpolation processing) in the signal synthesis unit 205, as in the processing described with reference to FIG.

  Also in this embodiment, the specific image height and the interpolation method are selected in consideration of the optical characteristics of the imaging lens 101 as described above.

  By performing the above process, the unsharp mask process by Formula (7) can be performed.

In each of the above embodiments, the case where the image processing unit 104 as an image processing device is incorporated in the imaging device has been described. However, the image processing device may be configured as a device other than the imaging device, such as a personal computer. Good. In this case, the image processing apparatus acquires the captured image acquired by the imaging apparatus as an input image via a storage medium such as a semiconductor memory, communication via the Internet, LAN, and the like, and performs unsharp mask processing on the input image. Do.
(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

101 Imaging Lens 104 Image Processing Unit 201 PSF Selection Unit 202 Correction Signal Generation Unit 203 Correction Signal Application Unit 204 Weight Information Acquisition Unit 205 Signal Synthesis Unit

Claims (12)

An image processing apparatus that generates a sharpened image by performing a correction process for sharpening an input image generated by imaging through an optical system,
PSF acquisition means for acquiring information on the point image intensity distribution function of the optical system according to the imaging condition of the imaging, the point image intensity distribution function at a plurality of specific image heights;
A filter generation means for generating a plurality of filters which respond respectively are paired with the plurality of specific image height using information on the point image intensity distribution function,
Data generating means for generating a plurality of first data by applying each of the plurality of filters to the input image;
Weight acquisition means for acquiring weight information according to image height;
Wherein by weighting synthesis of the first data of the plurality using a weight information, it said correction data also used for processing have a synthesizing means for generating a second data which is the sharpened image,
The filter generation means generates the plurality of filters using a difference between each of the point image intensity distribution functions at the plurality of specific image heights and an ideal point image,
The data generation means generates the plurality of first data by convolving and integrating each of the plurality of filters with respect to the input image,
The sharpening image is generated by adding correction data generated by using the second data in the correction processing to the input image . An image processing apparatus that generates a sharpened image by performing a correction process for sharpening an input image generated by imaging through an optical system,
PSF acquisition means for acquiring information on the point image intensity distribution function of the optical system according to the imaging condition of the imaging, the point image intensity distribution function at a plurality of specific image heights;
Filter generating means for generating a plurality of filters respectively corresponding to the plurality of specific image heights using information on the point image intensity distribution function;
Data generating means for generating a plurality of first data by applying each of the plurality of filters to the input image;
Weight acquisition means for acquiring weight information according to image height;
Synthesis means for performing weighted synthesis of the plurality of first data using the weight information to generate data used for the correction process or second data that is the sharpened image;
The filter generation means multiplies the difference between each of the point image intensity distribution functions at the plurality of specific image heights and the ideal point image by an adjustment coefficient, and obtains the result obtained by the multiplication as the ideal point image. Generating the plurality of filters by adding to
The data generation means generates the plurality of first data by convolving and integrating the plurality of filters with respect to the input image,
The sharpened image, the correction process the plurality of first said combined weighted images processor you characterized in that it is produced by the data as the second data in. The image processing apparatus according to claim 1 or 2, characterized in that it has an image height selecting means for selecting based on the specific image height on the optical information of the optical system. The said PSF acquisition means acquires this information from the memory | storage means which memorize | stored the information regarding the point image intensity distribution function in these specific image heights, The information as described in any one of Claim 1 to 3 characterized by the above-mentioned. Image processing device. Each of the plurality of filters has a plurality of taps arranged two-dimensionally,
Wherein the plurality of filters, the image processing device according to claim 1, wherein in any one of 4 to include a filter tap values are distributed rotationally asymmetric with respect to the center tap. The combining means is in said weighted combination, images according to any one of claims 1 5, characterized synthesizing nonlinearly first data of said plurality with respect to the image height in response to the weight information Processing equipment. In the weighted synthesis, the synthesizing unit generates data corresponding to another image height different from any of the plurality of specific image heights in the second data, at least two images of the plurality of specific image heights. the image processing apparatus according to any one of claims 1 to 6, characterized in that generated by interpolation according to the distance from the high to the other image height. Imaging means for imaging through an optical system;
Imaging apparatus characterized by comprising an image processing apparatus according to any one of claims 1 to 7 for the correction process on the acquired input image by the imaging means. An image processing method for generating a sharpened image by performing a correction process for sharpening an input image generated by imaging through an optical system,
Information regarding the point image intensity distribution function at a plurality of specific image heights, which is a point image intensity distribution function of the optical system according to the imaging conditions of the imaging,
Generating a plurality of filters which respond respectively are paired with the plurality of specific image height using information on the point image intensity distribution function,
Applying each of the plurality of filters to the input image to generate a plurality of first data;
Get weight information according to image height,
Wherein by weighting synthesis of the first data of the plurality using a weight information, and generates the second data is data or the sharpened image used in the correction process,
The plurality of filters are generated using a difference between each of the point image intensity distribution functions at the plurality of specific image heights and an ideal point image,
The plurality of first data is generated by convolving and integrating each of the plurality of filters with respect to the input image,
The sharpening image is generated by adding correction data generated by using the second data in the correction processing to the input image . An image processing method for generating a sharpened image by performing a correction process for sharpening an input image generated by imaging through an optical system,
Information regarding the point image intensity distribution function at a plurality of specific image heights, which is a point image intensity distribution function of the optical system according to the imaging conditions of the imaging,
Generating a plurality of filters respectively corresponding to the plurality of specific image heights using information on the point image intensity distribution function;
Applying each of the plurality of filters to the input image to generate a plurality of first data;
Get weight information according to image height,
Performing weighted synthesis of the plurality of first data using the weight information to generate data used for the correction process or second data that is the sharpened image;
The plurality of filters multiply the difference between each of the point image intensity distribution functions at the plurality of specific image heights and the ideal point image by an adjustment coefficient, and obtain a result obtained by the multiplication as the ideal point image. Is generated by adding to
The plurality of first data is generated by convolving and integrating the plurality of filters with respect to the input image,
The sharpening image is generated by the weighted synthesis of the plurality of first data as the second data in the correction process. A computer program for causing a computer to generate a sharpened image by performing correction processing for sharpening on an input image generated by imaging through an optical system,
In the computer,
It is a point image intensity distribution function of the optical system according to the imaging conditions of the imaging, and obtains information on the point image intensity distribution function at a plurality of specific image heights,
To produce a plurality of filters which respond respectively are paired with the plurality of specific image height using information on the point image intensity distribution function,
Generating a plurality of first data by applying each of the plurality of filters to the input image;
Get weight information according to image height,
By using weighted synthesis of the plurality of first data using the weight information, data used for the correction process or second data that is the sharpened image is generated ,
The plurality of filters are generated using a difference between each of the point image intensity distribution functions at the plurality of specific image heights and an ideal point image,
The plurality of first data is generated by convolving and integrating each of the plurality of filters with respect to the input image,
The sharpening image is generated by adding correction data generated using the second data in the correction processing to the input image . A computer program for causing a computer to generate a sharpened image by performing correction processing for sharpening on an input image generated by imaging through an optical system,
In the computer,
It is a point image intensity distribution function of the optical system according to the imaging conditions of the imaging, and obtains information on the point image intensity distribution function at a plurality of specific image heights,
Generating a plurality of filters respectively corresponding to the plurality of specific image heights using information on the point image intensity distribution function;
Generating a plurality of first data by applying each of the plurality of filters to the input image;
Get weight information according to image height,
By using weighted synthesis of the plurality of first data using the weight information, data used for the correction process or second data that is the sharpened image is generated,
The plurality of filters multiply the difference between each of the point image intensity distribution functions at the plurality of specific image heights and the ideal point image by an adjustment coefficient, and obtain a result obtained by the multiplication as the ideal point image. Is generated by adding to
The plurality of first data is generated by convolving and integrating the plurality of filters with respect to the input image,
The sharpening image is generated by the weighted synthesis of the plurality of first data as the second data in the correction process.