JP2019103132A - Image processing device, image processing method, and program - Google Patents

Abstract

To provide an image processing device capable of correcting image quality degradation due to aberration and diffraction while achieving highly accurate vibration reduction processing.SOLUTION: An image processing circuit 7 is configured to acquire a piece of filter information corresponding to two or more kinds of anti-vibration processing used at the time of photography, to perform a series of predetermined arithmetic processing on the basis of the filter information corresponding to two or more kinds of anti-vibration processing, and to generate a correction filter used for correction of a pick-image. The image processing circuit 7 performs a series of correction processing on an image picked up using two or more anti-vibration processing by using the generated correction filter.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technology for processing a captured image acquired by performing anti-vibration processing.

  At the time of shooting by the imaging device, for example, if there is a camera shake (shake) or shaking of the imaging device body, the image blurring phenomenon that the subject image etc. is blurred on the imaging surface lowers the resolution and clarity of the photographed image. There are times when As a technique for improving such an image blurring phenomenon, image stabilization techniques such as optical image stabilization and electronic image stabilization are known. As a configuration for realizing the optical image stabilization, by moving the image stabilizing lens group constituting the whole or a part of the imaging optical system in the direction orthogonal to the optical axis of the imaging optical system according to the shake of the imaging device etc. A configuration is known that reduces image blurring by suppressing movement of a subject image or the like on an imaging surface. In addition, as another configuration of optical image stabilization, the imaging element is moved (shifted) in a direction perpendicular to the optical axis of the imaging optical system, rotated (rolled) around the optical axis, etc. There is also a configuration in which the relative movement of the subject image and the like with respect to the imaging surface is suppressed. In the electronic image stabilization, image blurring is corrected in a pseudo manner by changing the cut-out range of a captured image in accordance with the shake or the like of the imaging apparatus main body. Conventionally, only one of these vibration isolation processes has been performed. However, in recent years, by performing a plurality of vibration isolation processes simultaneously, the vibration isolation effect is larger and high-precision vibration isolation is realized. It is suggested that you do.

  Further, as factors that deteriorate the captured image other than the image blur, there are aberration due to lens aberration, diffraction due to an aperture, and the like. As a technique for reducing the image quality deterioration of the photographed image due to the image blur and the aberration as described above, for example, in Patent Document 1, a part of the imaging optical system is moved in a direction different from the optical axis to A technique is disclosed that realizes correction of curvature of field. In this patent document 1, the first correction lens that corrects image blurring optically by moving in a direction different from the optical axis and the first correction lens are generated by moving in a direction different from the optical axis. And a second correction lens for correcting the aberration. In addition, Patent Document 2 discloses a technique for selecting a vibration isolation method according to the frequency of vibration at the time of vibration isolation by cooperative control of optical image stabilization and electronic image stabilization. In this patent document 2, when optical image stabilization and electronic image stabilization are performed, a configuration is disclosed in which high frequency vibration components are corrected by optical image stabilization and low frequency vibration components are corrected by electronic image stabilization. ing.

JP, 2016-045488, A JP, 2010-4370, A

  For example, when a part of the optical system is moved by the image stabilization processing, spherical aberration and coma aberration also change in addition to astigmatism and curvature of field. On the other hand, in the technique disclosed in Patent Document 1, correction of spherical aberration and coma aberration, and correction of the influence of diffraction are not performed. Further, in the case of the technique disclosed in Patent Document 2, although image stabilization processing is performed according to the frequency of vibration at the time of image stabilization, image quality deterioration caused by aberration or diffraction when optical image stabilization is performed is corrected. It is not possible. Furthermore, for example, in the case where a plurality of image stabilization processes are performed simultaneously, it is necessary to consider the deterioration of the image quality due to the combination of aberrations and the like generated respectively in the plurality of image stabilization processes. The second technology can not cope with it and can not correct the deterioration. For example, it is conceivable to prepare in advance data on the optical characteristics of the imaging optical system and correct the deterioration by image processing using the data, but the amount of data that can correspond to all possible shakes and aberrations etc. Will be huge.

  Therefore, it is an object of the present invention to make it possible to favorably correct image quality deterioration due to aberration and diffraction without preparing a vast amount of data etc. in advance while realizing highly accurate vibration reduction processing.

  The image processing apparatus according to the present invention is acquired corresponding to the two or more types of image stabilization processing, with an acquisition unit for acquiring filter information individually corresponding to the two or more types of image stabilization processing used in photographing. Processing means for performing predetermined arithmetic processing on the basis of the filter information to generate a correction filter used for correcting the photographed image, and the photographed image photographed using the two or more types of image stabilization processing. And processing means for performing correction processing using the generated correction filter.

  According to the present invention, it is possible to satisfactorily correct image quality deterioration due to aberration or diffraction without preparing a vast amount of data in advance while realizing highly accurate vibration reduction processing.

It is a figure which shows the center cross section and electrical block of an imaging device. It is an exploded perspective view of a blurring amendment part. 5 is an operation flowchart at the time of image shooting in the imaging device. It is a related figure of the inverse filter of OTF, and the gain characteristic of an image restoration filter. It is explanatory drawing of an image restoration filter. It is explanatory drawing of PSF. It is explanatory drawing of the amplitude component of an optical transfer function, and a phase component. It is a flowchart of a production | generation process of an image restoration filter.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the drawings.
The detailed configuration and operation of an imaging apparatus as an application example of the image processing apparatus according to the present embodiment will be described.
The image pickup apparatus according to the present embodiment has a configuration that performs the above-described image stabilization processing in order to reduce image shake due to camera shake or shaking of the image pickup apparatus main body. In the present embodiment, an example will be described in which high vibration isolation effects and high accuracy vibration isolation can be realized by simultaneously performing two or more types of vibration isolation processes among a plurality of vibration isolation processes. In this embodiment, the vibration reduction processing for moving the entire imaging optical system or the vibration reduction lens group in the direction orthogonal to the optical axis of the imaging optical system, and the imaging element in the direction orthogonal to the optical axis of the imaging optical system ), Anti-vibration processing to rotate around the optical axis (roll) as an example. In the following description, optical image stabilization by moving the image stabilizing lens group is referred to as lens image stabilization, and optical image stabilization by causing the imaging element to shift or roll is referred to as imager image stabilization. In the present embodiment, the shake of the imaging device or the like is detected by, for example, a shake detection sensor such as an angular velocity sensor or a vibration gyro sensor provided in the imaging device. Then, in the imaging device, the image blur is reduced by performing the image stabilization processing by the lens stabilization and the imager stabilization based on the shake detection signal output from the shake detection sensor.

  FIG. 1A is a central cross-sectional view of the imaging device 1 of the present embodiment, and FIG. 1B is a block diagram showing the electrical configuration of the imaging device 1 of the present embodiment. Those denoted with the same reference numerals in FIG. 1A and FIG. 1B correspond to each other.

  In FIG. 1A and FIG. 1B, the lens unit 2 is a so-called interchangeable lens, and can be detachably attached to the imaging device 1 via a lens mount (not shown) provided in the housing of the imaging device 1 It has been made. FIGS. 1A and 1B show a state in which the lens unit 2 is attached to the imaging device 1, and the imaging device 1 in this state and the lens unit 2 are electrically connected by the electrical contact 11. Be done. The lens unit 2 has an imaging optical system 3 composed of a plurality of lenses, and a part of the lens groups is provided as a vibration reduction lens group. An alternate long and short dash line in the drawing represents the optical axis 4 of the imaging optical system 3. The lens unit 2 also includes a lens drive unit 13 that drives a vibration reduction lens group, a focus lens, an aperture, and the like, and a lens system control circuit 12 that controls the lens drive unit 13. The lens drive unit 13 includes drive mechanisms of the vibration reduction lens group, the focus lens, and the diaphragm, and drive circuits of these drive mechanisms. The lens system control circuit 12 is a processor such as a CPU or an MPU, and is electrically connected to the camera system control circuit 5 of the imaging device 1 through the electrical contact 11. In the present embodiment, an example in which the imaging optical system 3 is provided in the lens unit 2 which is a removable interchangeable lens has been described. However, the present invention is not limited to this. The imaging optical system 3 is integral with the imaging device May be configured.

  The imaging device 1 includes a camera system control circuit 5, an imaging element 6, an image processing circuit 7, a memory circuit 8, a display device 9, an operation detection circuit 10, a shutter mechanism 16, a shake detection circuit 15, a shake correction unit 14 and the like. Is configured. The electrical contact 11 is composed of a contact on the imaging device 1 side and a contact on the lens unit 2 side. The camera system control circuit 5 controls each part in the imaging device 1 and exchanges various information with the lens system control circuit 12 of the lens unit 2 connected via the electric contact 11.

  The shake detection circuit 15 includes a shake detection sensor capable of detecting a shake in each direction of the pitch direction, the yaw direction, and the roll direction around the optical axis in the imaging device 1, and these shake detection sensors are an angular velocity sensor, a vibration gyro sensor, etc. It consists of The shake detection signal output from the shake detection circuit 15 is sent to the camera system control circuit 5. The shake correction unit 14 drives the imaging device 6 of the imaging device 1 so as to move (shift) in a plane orthogonal to the optical axis 4 of the imaging optical system 3 and to rotate (roll) around the optical axis It has a mechanism and the drive circuit of the said drive mechanism, and is comprised. The camera system control circuit 5 is a processor such as a CPU or MPU, generates a target value for shake correction based on the shake detection signal, generates a shake correction control signal based on the target value, and sends it to the shake correction unit 14 . The drive circuit of the shake correction unit 14 generates a drive signal for moving (shifting) the imaging device 6 in a direction orthogonal to the optical axis 4 and rotating (rolling) around the optical axis based on the shake correction control signal. Drive the drive mechanism. Thereby, the image pickup device 6 is moved (shifted), rotated (rolled), or the like in a direction to correct the shake detected by the shake detection circuit 15, and the imager image stabilization processing is realized. The configuration of the drive mechanism of the shake correction unit 14 and the details of the control at the time of image stabilization are described later.

  Further, the camera system control circuit 5 generates a target value of shake correction by the vibration reduction lens group of the lens unit 2 based on the shake detection signal from the shake detection circuit 15. Then, the camera system control circuit 5 generates a control signal for moving (shifting) the anti-vibration lens group in the direction orthogonal to the optical axis 4 based on the target value. A control signal for moving (shifting) the vibration reduction lens group in the direction orthogonal to the optical axis 4 is sent to the lens system control circuit 12 via the electrical contact 11. The lens system control circuit 12 generates a drive control signal for driving the lens drive unit 13 based on the control signal sent from the camera system control circuit 5. Then, the lens drive unit 13 drives the anti-vibration lens group based on the drive control signal from the lens system control circuit 12. As a result, the anti-vibration lens group of the lens unit 2 is moved (shifted) in a direction to correct the shake detected by the shake detection circuit 15, and lens anti-shake processing is realized. Details of drive control of the anti-vibration lens group will be described later.

  In addition, when both the lens stabilization and the imager stabilization are performed simultaneously, the target value of the shake correction for moving the imaging device 6 and the target value for the shake correction for moving the vibration reduction lens group are mutually shake correction Is set as a target value for realizing vibration reduction. Therefore, when both the lens stabilization and the imager stabilization are performed simultaneously, the movement (shift) of the anti-vibration lens group and the movement (shift or roll) of the imaging device 6 are performed based on the target values. Thus, high vibration isolation effect and high accuracy vibration isolation processing are realized.

  The imaging device 6 of the imaging device 1 includes a CCD sensor, a CMOS sensor, and the like, and captures an object image and the like formed on the imaging surface through the lens unit 2. In the imaging device 6, an optical low pass filter and an RGB color filter corresponding to a so-called Bayer arrangement are provided in front of the imaging surface. The imaging signal output from the imaging element 6 is sent to the image processing circuit 7. The imaging signal output from the imaging device 6 is also sent to the camera system control circuit 5 via the image processing circuit 7.

  The camera system control circuit 5 acquires a focus evaluation value used for focus control and an exposure evaluation value used for exposure control based on the imaging signal. Then, the camera system control circuit 5 sends a focus control command to the lens system control circuit 12 based on the focus evaluation value, and the lens system control circuit 12 controls the lens drive unit 13 to drive the focus lens based on the focus control command. Send a signal. The lens drive unit 13 drives the focus lens based on a drive control signal of the focus lens, whereby focusing on an object or the like is performed. Further, the camera system control circuit 5 sends an aperture control command to the lens system control circuit 12 based on the exposure evaluation value, and sends a shutter speed control signal to the shutter mechanism 16. The lens system control circuit 12 sends an aperture drive control signal to the lens drive unit 13 based on the aperture control command. The lens drive unit 13 controls the diaphragm based on the diaphragm drive control signal. The shutter mechanism 16 controls the opening and closing operation of the shutter mechanism based on a shutter speed control signal. More specifically, the shutter mechanism 16 controls whether the subject image reaches the image pickup device 6 by causing the shutter curtain to travel. The shutter mechanism 16 includes at least a curtain (mechanical rear curtain) for blocking an object image, and the completion of the exposure is performed by the shutter mechanism 16. The imaging device 6 also has a mode (electronic front curtain) for controlling the timing of the start of exposure by resetting the electric charge for each line prior to the trailing movement of the shutter mechanism 16. In the electronic front curtain mode, exposure control is performed by operating the charge reset (electronic front curtain) of the image pickup device 6 described above in synchronization with the trailing curtain traveling of the shutter mechanism 16. As a result, in the imaging device 6, imaging with an appropriate exposure amount is performed on a subject or the like to be imaged. As described above, the camera system control circuit 5 performs photometric measurement and ranging operation based on the signal from the image pickup device 6, and determines focusing and exposure conditions (F number, shutter speed, etc.).

  The image processing circuit 7 is an arithmetic circuit equipped with a plurality of ALUs (Arithmetic and Logic Units). The image processing circuit 7 performs correction processing using a correction filter described later in addition to the configuration for general image processing such as an A / D converter, a white balance adjustment circuit, a gamma correction circuit, and an interpolation operation circuit It also has a configuration for performing image restoration processing). Alternatively, instead of the image processing circuit 7, the camera system control circuit 5 may execute a program stored in the memory circuit 8 to process these functions on software. Details of the correction processing (image restoration processing) performed by the image processing circuit 7 will be described later. The interpolation operation circuit includes a color interpolation circuit, and the color interpolation circuit performs color interpolation (demosaicing) processing on the RGB signal corresponding to the color filter of the Bayer array of the imaging device 6 to generate a color image signal. The image processing circuit 7 also compresses images, moving images, sounds and the like using a predetermined method. The image processing circuit 7 applies image processing to the image pickup signal supplied from the image pickup device 6 to generate image data for recording.

  The memory circuit 8 includes an internal memory such as a ROM and a RAM, and an external memory such as a removable semiconductor memory. In addition to the program according to the present embodiment, PSF information and various setting data etc. described later are stored in the ROM of the internal memory, and the program read out from the ROM and expanded during the various processing etc. is stored in the RAM. It is memorized. The program may be read from, for example, a removable semiconductor memory (external memory), or may be downloaded via a network such as the Internet (not shown), in addition to the case where the program is prepared in advance in the ROM. . The external memory stores the image data for recording after the processing by the image processing circuit 7 and the like.

The display unit 9 includes a back display unit 9a and an electronic view finder (EVF 9b). An image, a user interface screen, etc. are displayed on the display device 9 under the control of the camera system control circuit 5.
The operation detection circuit 10 detects a user operation on a power switch, a release button, a menu button, and various other switches and buttons provided in the imaging device 1. In addition, a touch panel is provided on the back surface display device 9 a of the display device 9, and the operation detection circuit 10 also detects a user operation on the touch panel. Then, the operation detection circuit 10 sends those operation detection signals to the camera system control circuit 5.

  The camera system control circuit 5 responds to the user's operation detected by the operation detection circuit 10, and the respective sections related to image pickup as described above, the image processing circuit 7, recording and reproduction, display of the display device 9, etc. Control. In addition to these controls, the camera system control circuit 5 also generates and outputs a timing signal or the like at the time of imaging. For example, when receiving a detection signal of the user's pressing operation on the release button from the operation detection circuit 10, the camera system control circuit 5 generates various timing signals related to imaging, drives the imaging device 6, and the image processing circuit 7 Control the operation etc. As described above, in the imaging device 1 according to the present embodiment, the camera system control circuit 5 controls the operation of each unit of the imaging device 1 in accordance with the user operation detected by the operation detection circuit 10, whereby a still image or a moving image is obtained. And so on.

  Next, the detailed configuration and operation of the drive mechanism provided in the shake correction unit 14 of the imaging device 1 of the present embodiment will be described using FIG. 2. FIG. 2 is an exploded perspective view of a drive mechanism of the shake correction unit 14 for moving the imaging device 6 to reduce shake. The shake correction unit 14 also includes an electrical drive circuit (not shown) for operating the drive mechanism of FIG. 2 in accordance with a control signal from the camera system control circuit 5, but FIG. It shows.

  In FIG. 2, an alternate long and short dash line is a line parallel to the optical axis of the imaging optical system of the lens unit 2. Further, in FIG. 2, the members to which the reference numeral 100 is attached are members that do not move (that is, the members fixed to the housing etc. of the imaging device 1), while the members to which the reference numeral 200 is attached are A movable member (that is, a movable member which is not fixed to the housing or the like of the imaging device 1). Further, the members to which the reference numeral 300 is attached are balls held by the fixed portion and the movable portion.

  The vibration reduction unit 14 shown in FIG. 2 includes, as main components of the fixed portion, upper yokes 101, upper magnets 103a to 103f, fixed portion rolling plates 106a to 106c, lower magnets 107a to 107f, lower yoke 108, and base plate. And 110. The components of the fixing portion also include auxiliary spacers 104a and 104b, main spacers 105a to 105c, and screws 102a to 102c and 109a to 109c. The shake correction unit 14 also has an FPC 201, movable portion rolling plates 204a to 204c, coils 205a to 205c, a movable frame 206, and balls 301a to 301c as main components of the movable portion. Position detection element attachment positions 202a to 202c are determined on the FPC 201, and the position detection elements are attached to these positions.

  The upper yoke 101, the upper magnets 103a to 103f, the lower magnets 107a to 107f, and the lower yoke 108 form a magnetic circuit, forming a so-called closed magnetic path. The upper magnets 103 a to 103 f are adhesively fixed to the upper yoke 101. Similarly, the lower magnets 107 a to 107 f are adhesively fixed to the lower yoke 108. The upper magnets 103a to 103f and the lower magnets 107a to 107f are respectively magnetized in the optical axis direction (vertical direction in FIG. 2), and adjacent magnets (having positional relationship between the magnets 103a and 103b) are different from each other. It is magnetized. Further, the opposing magnets (having the positional relationship between the magnets 103a and 107a) are magnetized in the same direction. By doing this, a strong magnetic flux density is generated between the upper yoke 101 and the lower yoke 108 in the optical axis direction.

  Since a strong suction force is generated between the upper yoke 101 and the lower yoke 108, the main spacers 105a to 105c and the auxiliary spacers 104a and 104b are configured to maintain an appropriate distance. The appropriate distance referred to here is a distance such that the coils 205a to 205c and the FPC 201 can be disposed between the upper magnets 103a to 103f and the lower magnets 107a to 107f and an appropriate gap can be secured. The main spacers 105a to 105c are provided with screw holes, and the upper yoke 101 is fixed to the main spacers 105a to 105c by screws 102a to 102c.

Rubber is placed on the trunks of the main spacers 105a to 105c to form a mechanical end (a so-called stopper) of the movable part.
A hole is provided in the base plate 110 so as to turn the lower magnets 107a to 107f, and the surface of the magnet protrudes from the hole. That is, the base plate 110 and the lower yoke 108 are fixed by the screws 109 a to 109 c, and the lower magnets 107 a to 107 f whose dimension in the thickness direction is larger than the base plate 110 are fixed so as to protrude from the base plate 110.

  The movable frame 203 is formed by magnesium die casting or aluminum die casting, and is lightweight and highly rigid. Each element of the movable portion is fixed to the movable frame 203 to form a movable portion. Position detection elements are attached to the surface of the FPC 201 which can not be seen in FIG. 2 at each position indicated by the position detection element attachment positions 202a to 202c. For example, a Hall element or the like is disposed so that the position can be detected using the above-described magnetic circuit. Because the Hall element is small, it is arranged to be nested inside the windings of the coils 205a-205c.

The movable frame 203 is connected to the imaging device 6, coils 205a to 205c, and a Hall element (not shown) in FIG. The movable frame 203 electrically communicates with the outside through the connector.
The fixed portion rolling plates 106a to 106c are adhered and fixed to the base plate 110, and the movable portion rolling plates 204a to 204c are adhered and fixed to the movable frame 203, and rolling surfaces of the balls 301a to 301c are formed. In addition, it becomes easy to design a surface roughness, hardness, etc. in a preferable state by providing a rolling plate separately.

  In the drive mechanism of the shake correction unit 14 shown in FIG. 2, by passing a current through the coil, a force according to Fleming's left-hand rule is generated to move the movable part. Further, in the case of the shake correction unit 14, it is possible to perform feedback control by using the signal of the Hall element which is the position detection element described above. Further, in the shake correction unit 14, by appropriately controlling the value of the Hall element signal, the image pickup element 6 can be shifted to translate in a plane orthogonal to the optical axis and rotated (rolled) around the optical axis It is also possible.

  In the present embodiment, the rotation (roll) about the optical axis becomes important in the imager anti-vibration blur correction as described later, so this will be described. The drive mechanism of the shake correction unit 14 drives the Hall element signals at the positions 202b and 202c in opposite phase while keeping the signal of the Hall element at the position 202a constant, so that the rotational movement around the approximate optical axis is achieved. It can produce.

  In the image pickup apparatus 1 according to the present embodiment, as described above, the camera system control circuit 5 controls the shake correction unit 14 based on the shake detection signal from the shake detection circuit 15 to realize the imager image stabilization. The vibration reduction lens group of the lens unit 2 is controlled to realize lens vibration reduction. Then, in the shake correction by the imager image stabilization, the image pickup device 6 is also rotated (rolled) around the optical axis.

  FIG. 3 is a flowchart showing a flow of a series of operations from power-on to main shooting in the imaging device 1 of the present embodiment. Hereinafter, with reference to FIG. 3, a series of operations from power on of the imaging device 1 to shooting execution will be described in order. The process of the flowchart in FIG. 3 may be realized by either the hardware configuration or the software configuration or a part of the software configuration and the rest by the hardware configuration. For example, when the memory circuit 8 includes a RAM and a ROM, the program stored in the ROM is expanded in the RAM, and the camera system control circuit 5 which is a CPU executes the program, etc. Implement the processing of the flowchart. The same applies to the other flowcharts described later.

When the power on operation is detected by the operation detection circuit 10, the camera system control circuit 5 starts the control shown in the operation flowchart of FIG.
Next, in step S301, the camera system control circuit 5 reads the shooting conditions set by the user or the shooting conditions stored in advance in the memory circuit 8. The set values of the shooting conditions read in step S301 are, for example, set values indicating whether to use an electronic front curtain, and various set values such as so-called aperture priority or shutter speed priority.

  Next, in step S302, the camera system control circuit 5 determines whether the user has performed the power-off operation. If the camera system control circuit 5 determines that the power-off operation has been performed, the processing of the flowchart of FIG. On the other hand, if it is determined in step S302 that the power-off operation has not been performed, the camera system control circuit 5 advances the process to step S303. Although steps S302, S305, S308, and the like corresponding to the user operation are originally interrupt processing, they are described as a flow in FIG.

  In step S303, the camera system control circuit 5 performs control regarding photometry and distance measurement at the time of live view display described above based on a signal captured by the imaging device 6 and passed through the image processing circuit 7. Thereby, the focus position, the exposure condition, and the like when capturing a live view image are determined.

  In the next step S304, the camera system control circuit 5 causes the display device 9 to display the image data imaged by the imaging device 6 under the conditions of photometry and distance measurement at the time of live view display and image processed by the image processing circuit 7. Send it for live view display. In the live view display, an image is displayed on the rear display 9 a or the EVF 9 b of the display 9.

  Next, in step S305, the camera system control circuit 5 determines whether the operation detection circuit 10 has detected an S1 operation by the user. Then, when the S1 operation is not detected, the camera system control circuit 5 returns the process to step S301, and on the other hand. If it is determined that the S1 operation has been detected, the process proceeds to step S306. The S1 operation is a half-press operation of the release button by the user.

  When the process proceeds to step S306, the camera system control circuit 5 starts control (vibration control) regarding the above-described lens image stabilization and imager image stabilization processing.

  Next, in step S307, the camera system control circuit 5 performs control related to photometry and distance measurement based on a signal captured by the imaging device 6 after the operation of S1 and through the image processing circuit 7. The camera system control circuit 5 at this time determines the aperture value and the shutter speed (Tv value) based on the signal picked up by the imaging device 6 and passed through the image processing circuit 7.

  Next, in step S308, the camera system control circuit 5 determines whether the operation detection circuit 10 has detected the S2 operation by the user. Then, if the camera system control circuit 5 determines that the S2 operation is detected, the process proceeds to step S309, and if the S2 operation is not detected, the process proceeds to step S310. The S2 operation is an operation of fully pressing the release button (pushing it all the way to the end).

  In step S309, the camera system control circuit 5 controls the shutter mechanism 16 to cause the shutter curtain to travel, and causes the imaging device 6 to capture an object or the like. After this step S309, the camera system control circuit 5 returns the process to S302.

  When the process proceeds to step S310, the camera system control circuit 5 determines whether the S1 operation is maintained. If it is determined that the S1 operation is maintained, the camera system control circuit 5 returns the process to step S308, while if it is determined that the S1 operation is not maintained, the process returns to step S302.

  Although not shown in FIG. 3, the camera system control circuit 5 turns off the anti-shake control when a predetermined time elapses after the release operation of the release button by the user is released and the S1 operation is released. Do.

  Next, in the present embodiment, an outline of a correction process for correcting a deterioration component occurring in a photographed image due to the influence of aberration or diffraction due to the above-described vibration reduction processing and the deterioration component thereof will be described. The methods of the image stabilization processing and the correction processing described here are appropriately used in the imaging device 1 of the present embodiment.

  The image pickup optical system 3 of the image pickup apparatus 1 of the present embodiment is designed to be able to correct each aberration such as spherical aberration, coma aberration, curvature of field, astigmatism and the like with high accuracy. If this is done, the aberration may change and the photographed image may deteriorate. For example, when the anti-vibration lens group is moved in a direction orthogonal to the optical axis 4 of the imaging optical system 3 due to lens anti-vibration, the aberration changes due to the decentration of the optical axis generated between the imaging optical system 3 and the anti-vibration lens group And the photographed image may be degraded.

  Although the aberration of the lens changes in relation to the aperture diameter of the stop or the like, the lateral aberration is generally suppressed as the aperture diameter of the stop is smaller. The diffraction phenomenon is less affected when the aperture diameter is large, for example, f / 2.8. On the other hand, the effect is large when the aperture diameter is small, for example, f / 11 or f / 22. Become. This diffraction phenomenon may also be a factor that degrades the captured image, as in the case of aberration.

These aberrations and diffraction can be described by a point spread function (hereinafter referred to as PSF) or an optical transfer function (hereinafter referred to as OTF). The image quality deterioration component due to aberration or diffraction is caused, for example, by the fact that light emitted from one point of the object should spread again at one point on the imaging surface when there is no aberration and there is no influence of diffraction. Can be represented. Further, OTF (optical transfer function) obtained by Fourier-transforming PSF is frequency component information of aberration, and is represented by a complex number. Furthermore, the absolute value of OTF, that is, the amplitude component is called MTF (Modulation Transfer Function), and the phase component is called PTF (Phase Transfer Function). These MTF (amplitude component) and PTF (phase component) are respectively frequency characteristics of the amplitude component and the phase component of the image quality deterioration due to the aberration, and the phase component is represented by the following equation (1) as a phase angle. Re (OTF) in equation (1) represents the real part of OTF (optical transfer function), and Im (OTF) represents the imaginary part of OTF.
PTF = tan ^-1 (Im (OTF) / Re (OTF)) Formula (1)

  Correction of deterioration of a photographed image due to aberration or diffraction can be realized by correcting deterioration components in MTF (amplitude component) and PTF (phase component). As a method of correcting degradation components in MTF (amplitude component) and PTF (phase component), for example, there is a method of correcting based on information of OTF or PSF of an imaging optical system. In this embodiment, an image restoration process is performed as a correction process for correcting the deterioration of a captured image based on the OTF or PSF information of the imaging optical system. One of the methods for realizing the image restoration process is a method of convolving a function having the inverse characteristic of OTF with an input image (captured image). The function having the inverse characteristic of the OTF is made as a correction filter (an image restoration filter to be described later in the image restoration process) for correcting the deterioration of the image.

  Next, an image restoration process which is a correction process for the deteriorated component of the photographed image described above and a correction filter (image restoration filter) used in the image restoration process will be described. In the case of the imaging device 1 of the present embodiment, the image restoration processing is performed in the image processing circuit 7.

  In the present embodiment, the input image to the image restoration process is a captured image captured by the imaging device 6 through the imaging optical system 3, and the output image is an image after the image recovery process. The captured image (input image) is obtained from the OTF (optical transfer function) by the aberration of the lens included in the imaging optical system 3 and various optical filters (not shown), the diaphragm included in the imaging optical system 3 and the optical member (not illustrated) It is degraded by the OTF due to the diffraction of In the following description, the degraded captured image (input image) is referred to as a degraded image. The imaging optical system may be configured using not only a lens but also a mirror (reflection surface) having a curvature. The input image is, for example, a RAW image having information of RGB color components, but is not limited to this. The input image and the output image may be accompanied by photographing conditions such as the presence or absence of the optical member, the focal length of the lens, the aperture value, and the photographing distance, and various correction information for correcting the image.

First, an outline of the image restoration process will be described.
Here, a degraded image (captured image) is g (x, y), an original image is f (x, y), and a PSF (point spread function) that is a Fourier pair of OTF (optical transfer function) is h ( Assuming that x, y), the following equation (2) is established. In equation (2), * is convolution (convolution integral, product-sum), and (x, y) are coordinates on the photographed image.
g (x, y) = h (x, y) * f (x, y) Formula (2)

Further, when Equation (2) is subjected to Fourier transform to convert it into a display form on the frequency plane, Equation (3) expressed by a product for each frequency is obtained. In Equation (3), H is an OTF (optical transfer function) obtained by subjecting the point spread function PSF (h) to Fourier transform, and G is a function obtained by subjecting the degraded image g to Fourier transform , F is a function obtained by Fourier transforming the original image f. Also, (u, v) are coordinates in a two-dimensional frequency plane, that is, frequency.
G (u, v) = H (u, v) · F (u, v) Formula (3)

In order to obtain the original image f from the degraded image g obtained by photographing, both sides may be divided by the optical transfer function H as in the following equation (4).
G (u, v) / H (u, v) = F (u, v) Formula (4)
Then, the original image f (x, y) is obtained as a restored image by performing inverse Fourier transform on F (u, v), that is, G (u, v) / H (u, v) to return to the real surface. Be

In addition, ^assuming that inverse Fourier transform of H ⁻¹ is R, the original image f (x, y) is similarly processed by performing convolution processing on the image in the real plane as in the following equation (5). You can get
g (x, y) * R (x, y) = f (x, y) Formula (5)

  Here, R (x, y) in the equation (5) is called an image restoration filter. When the image is a two-dimensional image, generally, the image restoration filter R also has a tap (cell) corresponding to each pixel of the image and has a distribution of two-dimensional filter values. In addition, as the number of taps (number of cells) of the image restoration filter R is generally larger, the restoration accuracy improves. Therefore, the number of taps that can be realized is set according to the required image quality, the image processing capability, the spread width of the point spread function (PSF), the characteristics of the aberration, and the like. Since the image restoration filter R needs to reflect at least the characteristics of aberration and diffraction, it differs from a general horizontal and vertical three-tap edge emphasis filter (high-pass filter) or the like. Since the image restoration filter R is set based on OTF (optical transfer function), both the deterioration of the amplitude component and the phase component can be corrected with high accuracy.

  Also, since the actual image contains a noise component, using the image restoration filter R generated by taking the reciprocal of OTF (a scientific transfer function) as described above, the noise component is greatly amplified along with the recovery of the degraded image. It will be done. This means that if the MTF (amplitude component) of the optical system is normalized to 1 while the amplitude component of the noise is added to the amplitude component of the image, the MTF will be returned to 1 across all frequencies. As such, it is to lift the MTF. Although the MTF which is the amplitude degradation due to the optical system returns to 1, the power spectrum of noise is also raised at the same time, and as a result, the noise is amplified according to the degree of lifting the MTF (recovery gain).

Therefore, when the image contains noise, a high quality image that can be used for appreciation can not be obtained. This is represented by the following formulas (6-1) and (6-2). Here, N in the equation is a noise component.
G (u, v) = H (u, v) · F (u, v) + N (u, v) Formula (6-1)
G (u, v) / H (u, v) = F (u, v) + N (u, v) / H (u, v) Formula (6-2)

For an image including a noise component, there is a method of controlling the degree of recovery according to the intensity ratio (SNR) of the image signal and the noise signal, as in the Wiener filter represented by the following equation (7), for example.

  In equation (7), M (u, v) is the frequency characteristic of the Wiener filter, and | H (u, v) | is the absolute value (absolute value of MTF) of OTF (optical transfer function). In the present embodiment, M (u, v) corresponds to the frequency characteristic of the image restoration filter. In this method, for each frequency, the smaller the MTF, the smaller the recovery gain (recovery degree), and the larger the MTF, the larger the recovery gain. Generally, the MTF of the imaging optical system is high on the low frequency side and low on the high frequency side, so this method substantially reduces the recovery gain on the high frequency side of the image.

Further, the gain characteristic of the image restoration filter changes due to the SNR term of equation (7). Therefore, the following equation (8) is used as a parameter C for simply controlling the recovery gain (degree of recovery) for the SNR term.

In the equation (8), when the parameter C is C = 0, M (u, v) matches the inverse filter (reciprocal of the MTF) of the OTF (optical transfer function), and the image recovery is performed as the parameter C is increased. Filter gain is reduced. Further, when C> H (u, v) -H (u, v) ² , the gain of the image restoration filter becomes 1 or less.

  Figures 4 (a) to 4 (c) schematically show this. FIGS. 4A to 4C are diagrams showing the relationship between the gain characteristic of the inverse filter of the OTF (optical transfer function) and the gain characteristic of the image restoration filter. In FIG. 4A to FIG. 4C, the vertical axis represents the gain, and the horizontal axis represents the spatial frequency. Further, in FIG. 4A to FIG. 4C, the dotted line indicates the gain characteristic of the inverse filter of the OTF, and the solid line indicates the gain characteristic of the image restoration filter.

Here, when the aperture value Fn is less than the predetermined value Th1 (Fn <Th1), the gain of the image restoration filter does not become larger than the predetermined maximum gain, so the parameter C can be set to C = 0. Therefore, the reciprocal of the MTF (gain of the inverse filter of the OTF) and the gain of the image restoration filter coincide with each other (FIG. 4A). When the aperture value Fn is equal to or greater than the predetermined value Th1 and smaller than the predetermined value Th2 (Th1 ≦ Fn <Th2), the gain on the high frequency side of the inverse filter becomes larger than the predetermined maximum gain. Therefore, the parameter C is increased to suppress the gain on the high frequency side of the image restoration filter (FIG. 4B). When the aperture value Fn becomes Th2 or more (Th2 ≦ Fn) and C> | H (u, v) |-| H (u, v) | ² , the gain of the image restoration filter becomes 1 or less (see FIG. 4 (c). The predetermined values Th1 and Th2 (diaphragm value) are parameters for controlling the pixel pitch of the image sensor 6 and the characteristics of the optical low pass filter, the maximum value of the recovery gain (recovery degree) of the image restoration filter, and the recovery gain (recovery degree). It is decided according to C etc.

  As described above, (the tendency of) the gain characteristic of the image restoration filter largely changes in accordance with the aperture value. In the present embodiment, in consideration of this tendency, the data amount of the image restoration filter can be reduced by calculating the image restoration filter held by the imaging device 1 as described later.

Next, with reference to FIG. 5, an image restoration filter used in the present embodiment will be described.
FIG. 5 is a diagram showing the characteristics of the image restoration filter. The number of taps of the image restoration filter is determined according to the spread of PSF (point spread function) due to the aberration and diffraction of the imaging optical system 3 and the restoration accuracy required for the image restoration processing. In the present embodiment, the image restoration filter is assumed to be, for example, an 11 × 11 tap two-dimensional filter. However, the present invention is not limited to this, and an image restoration filter with a larger number of taps may be used from the relationship between the spread width of the PSF and the pixel pitch. The distribution of the values (coefficient values) of the taps of the image restoration filter has a function of returning the signal value or pixel value spatially expanded due to the aberration to an original one point. Here, in the case where diffraction by a stop that can be approximated to rotational symmetry is targeted, PSF by the diffraction is rotational symmetry. Therefore, the characteristics of the image restoration filter are also made symmetrical as shown in FIG.

  Further, in the present embodiment, at each tap of the image restoration filter, convolution processing (convolution integration, product-sum) is performed in the image restoration processing step corresponding to each pixel of the image. In convolution processing, in order to improve the signal value of a predetermined pixel, that pixel is matched with the center of the image restoration filter, and the product of the signal value of the image and the value of each tap is calculated for each corresponding pixel of the image restoration filter. , Replace the sum as the signal value of the central pixel.

Subsequently, characteristics in the real space and the frequency space in the image restoration processing will be described with reference to FIGS. 6 and 7.
6 (a) and 6 (b) are explanatory diagrams of PSF (point spread function), and FIG. 6 (a) is PSF before image restoration and FIG. 6 (b) is PSF after image restoration. It shows. FIG. 7 (a) is an explanatory diagram of MTF (amplitude component) of OTF (optical transfer function), and FIG. 7 (b) is an explanatory diagram of PTF (phase component). The broken line G in FIG. 7A indicates the MTF before image restoration, and the alternate long and short dash line E indicates the MTF after image restoration. Further, a broken line G in FIG. 7B indicates a PTF before image restoration, and an alternate long and short dash line E indicates a PTF after image restoration.

  As shown in FIG. 6A, PSF (point spread function) before image restoration has an asymmetrical spread due to the influence of aberration, and this asymmetry causes PTF (phase component) to be non-relative to the frequency. It has a linear value. In the image restoration process, the MTF is amplified and corrected so that the PTF becomes zero, so that the PSF after image restoration has a symmetrical and sharp shape.

  In addition, in the case of diffraction by a stop that can be approximated as rotational symmetry, PSF by diffraction becomes rotational symmetry. Therefore, the broken line E in FIG. 7B is 0. In other words, the diffraction handled in this embodiment has no phase shift. Further, regardless of the presence or absence of the phase shift, the principle of the above-described image restoration functions, so the image restoration is effective also in the present embodiment in which the diffraction is to be corrected.

  Thus, the image restoration filter can be obtained by inverse Fourier transform of a function designed based on the inverse function of the OTF (optical transfer function) of the imaging optical system 3. The image restoration filter used in the present embodiment can be changed as appropriate, and, for example, a Wiener filter as described above can be used. When a Wiener filter is used, it is possible to create a real-space image restoration filter that is actually convoluted into an image by performing the inverse Fourier transform on Equation (7) described above.

  Further, OTF (optical transfer function) changes in accordance with the image height (the position of the image) of the imaging optical system 3 even in one imaging state (imaging condition). Therefore, the image restoration filter is used in accordance with the image height. On the other hand, with respect to OTFs in which the influence of diffraction becomes dominant as the f-number increases, if the influence of vignetting on the optical system is small, the OTF may be uniform (constant) with respect to the image height. It can be handled.

  In the present embodiment, diffraction (diffraction blurring) is also targeted. The image restoration filter depends on the aperture value, the wavelength of light, and the image height (the position of the image) when the aperture value is small. For this reason, a uniform (constant) image restoration filter can not be used in one image. That is, the image restoration filter of the present embodiment is generated by calculation using an optical transfer function that also includes a diffraction blur generated according to the aperture value. The calculation method of the image restoration filter will be described later. For wavelengths, an optical transfer function at a plurality of wavelengths can be calculated, and an optical transfer function for each color component can be generated by weighting for each wavelength based on the spectral characteristics of the light source assumed and the light receiving sensitivity information of the imaging device. . Alternatively, it may be calculated using a representative wavelength for each color component determined in advance. Then, an image restoration filter can be generated based on the optical transfer function for each color component.

  When the image restoration process is performed, the image processing circuit 7 first acquires the above-described shooting condition information from the camera system control circuit 5. Further, the image processing circuit 7 selects one or more image restoration filters for the on-axis light beam and the off-axis light beam according to the aperture value at the time of photographing in the photographing condition information. The image processing circuit 7 generates a restored image (output image) from the captured image (input image) using the selected image restoration filter.

  The restored image generated by the image processing circuit 7 is recorded in the memory circuit 8 in a predetermined format. Further, the display device 9 displays an image obtained by performing predetermined processing for display on the image after image processing by the image processing circuit 7. The display device 9 may display an image subjected to a simple process for high-speed display.

  In addition to the lens unit 2, when an optical element that affects the characteristics of the OTF (optical transfer function) is included, it is preferable to consider the influence when generating the image restoration filter. For example, the influence by the optical low pass filter provided in front of the image sensor 6 can be considered. In this case, the image processing circuit 7 generates an image restoration filter based on the optical transfer function by the optical low pass filter. In addition, when there is an infrared cut filter in the shooting light path, an image restoration filter is generated because it affects each PSF of RGB channel that is an integral value of PSF (point spread function) of spectral wavelength, especially PSF of R channel. In doing so, it is preferable to consider the effect. In this case, the image processing circuit 7 generates an image restoration filter based on the optical transfer function by the infrared cut filter. Further, since the shape of the pixel aperture also affects the optical transfer function, it is more preferable to consider the influence. In this case, the image processing circuit 7 generates an image restoration filter based on the optical transfer function by the pixel aperture.

  FIG. 8 is a flowchart showing a flow of image restoration processing performed by the image processing circuit 7 under the control of the camera system control circuit 5.

  Here, in order to perform the image restoration process effectively, it is necessary to obtain accurate OTF or PSF information in the imaging optical system 3. For example, if there is design value information of the imaging optical system 3, the OTF or PSF information can be obtained from the information by calculation. In addition, PSF or OTF information can also be determined by subjecting the intensity distribution of light at the time of photographing a point light source, or by subjecting the intensity distribution to Fourier transform. Furthermore, in the case of diffraction, OTF or PSF information can be determined from theoretically derived equations.

  Therefore, when image restoration processing is performed to reduce image blurring by image stabilization processing and to correct image quality deterioration caused by image stabilization processing, necessary OTF or PSF information can be a considerable amount of data There is sex. That is, for example, by moving the anti-vibration lens group in the direction orthogonal to the optical axis 4 of the imaging optical system 3 by lens anti-vibration, the eccentricity state in which the optical axis is eccentric and eccentricity does not occur in the optical axis In the uncentered state, aberrations generated in the light image on the imaging surface of the imaging device 6 are different. Therefore, in order to perform a good image restoration process, it is necessary to hold the OTF or PSF information in the eccentric state separately from the OTF or PSF information in the eccentric state. Further, at the time of imager image stabilization, the imaging device 6 is shifted in a direction orthogonal to the optical axis 4 of the imaging optical system 3 and rotated (rolled) around the optical axis. For this reason, the image circle of the imaging optical system 3 compatible with the imager image stabilization is made considerably larger than the image circle of the imaging optical system not compatible with the imager image stabilization. Therefore, when imager image stabilization is performed, it is necessary to hold OTF or PSF information up to the image height in a large image circle that can cope with imager image stabilization.

  Furthermore, if, for example, two or more vibration isolation processes are used simultaneously, the required OTF or PSF information will further increase. In the case of this embodiment, for non-vibration processing (no eccentricity), for lens vibration (decentered), for imager vibration, for lens vibration and for imager vibration at the same time. The respective OTF or PSF information, if used, is required. In addition, when lens image stabilization and imager image stabilization are used at the same time, OTF or PSF information up to an image height higher than that for imager image stabilization is required to cope with the eccentricity state due to lens image stabilization. . That is, in the case of simultaneously using two or more types of image stabilization processing, the amount of OTF or PSF information that must be held in advance is enormous.

  The flowchart in FIG. 8 is an image for recovering the deterioration occurring in the captured image due to the aberration and the diffraction by the imaging optical system 3 when both the lens stabilization and the imager stabilization are performed as the image stabilization processing. It shows the recovery process. This image restoration process may be performed immediately after shooting or may be performed when reproducing a recorded image. In the description of FIG. 8, an example in which PSF information is held as filter information of the image restoration filter will be described.

  The camera system control circuit 5 starts the operation of the flowchart of FIG. 8 when the image stabilization control is turned on. When the operation of the flowchart of FIG. 8 is started, first, in step S801, the camera system control circuit 5 causes the image pickup device 6 to take an image while the image stabilization processing is being performed. The shooting at this time may be still image shooting or moving image shooting.

  Next, in step S802, the camera system control circuit 5 determines whether both the lens stabilization and the imager stabilization have been used simultaneously. If the camera system control circuit 5 determines that the lens stabilization and the imager stabilization are simultaneously used, the process proceeds to step S 803, and it is determined that the lens stabilization and the imager stabilization are not simultaneously used. The process advances to step S810.

  In step S803, the camera system control circuit 5 detects the imager image stabilization at the time of imaging based on the imaging condition that the lens stabilization and the imager stabilization are simultaneously used in step S802. Get movement data. The movement data acquired at this time includes a shift amount for moving the imaging device 6 in a direction orthogonal to the optical axis of the imaging optical system 3, a roll amount for rotating around the optical axis, and a tilt amount. The movement data of the imaging element 6 is used when measuring the coordinates of the image height in the image stabilization processing later.

  Subsequently, at step S804, the camera system control circuit 5 obtains a plurality of pieces of filter information corresponding to the imager image stabilization from the plurality of pieces of filter information held in the memory circuit 8 with respect to the image processing circuit 7. Let me do it. In this case, the image processing circuit 7 performs the image stabilization processing based on the movement data of the image sensor 6 at the time of imager stabilization, and the image height of the captured image captured and the imaging optical system 3 at the decentered state. Calculate the relationship with the image height by. Then, the image processing circuit 7 selects a plurality of desired filter information from the filter information stored in the memory circuit 8 based on the relationship between the image heights and the imaging conditions.

  Next, in step S805, the camera system control circuit 5 performs the image stabilization lens group in the lens stabilization at the time of shooting based on the shooting condition that the lens stabilization and the imager stabilization are simultaneously used in step S802. Get movement data of The movement data acquired at this time includes not only the shift amount of the vibration reduction lens group with respect to the optical axis of the imaging optical system 3 but also the tilt amount.

  Subsequently, in step S806, the camera system control circuit 5 causes the image processing circuit 7 to acquire a plurality of pieces of filter information based on lens stabilization from the filter information held in the memory circuit 8. In this case, the image processing circuit 7 calculates the eccentricity amount and the eccentric direction of the vibration reduction lens group with respect to the optical axis of the imaging optical system based on the movement data of the vibration reduction lens group. Then, the image processing circuit 7 selects a plurality of desired pieces of filter information from the filter information held in the memory circuit 8 based on the eccentricity amount, the eccentricity direction, and the photographing condition.

  Next, in step S 807, the camera system control circuit 5 causes the image processing circuit 7 to correct the correction amount (image height change amount) at the time of image stabilization and the eccentricity amount at the time of lens stabilization. The degree of contribution is calculated to indicate how much each of the movement amounts of In this case, the image processing circuit 7 uses the correction amount (image height change amount) at the time of image stabilization and the eccentricity amount (ecentricity amount and direction) at the time of lens stabilization. Calculate the degree of contribution to the amount of movement of the center of the optical axis of. The image processing circuit 7 changes the PSF (point spread function) when comparing before and after moving the movement amount and direction at the time of image stabilization and the movement amount and direction at the time of lens stabilization. The degree of contribution to the quantity may be calculated. The degree of contribution calculated in step 807 is used as a coefficient when convolving a plurality of PSF information.

Here, a method of calculating the degree of contribution will be described.
The amount of change in PSF (point spread function) is regarded as the amount of change in MTF (optical transfer function). As the image height changes due to the imager image stabilization, the MTF changes. The amount of change in MTF at this time is taken as the amount of change in the first MTF. In addition, the MTF changes due to the occurrence of decentering due to lens vibration isolation. The amount of change in MTF at this time is taken as the amount of change in the second MTF. The degree of contribution is determined according to the ratio of the first MTF change amount to the second MTF change amount.

  For example, if the original MTF is 60% and the image height changes, the MTF value at the same evaluation image height becomes 50%, the change amount of the first MTF is 10 (= 60− 50) Also, if the original MTF is 60% and the MTF value at the same evaluation image height becomes 40% by moving the anti-vibration lens group, the change amount of the second MTF is 20 (= 60-40). In this case, the PSF at the time of image stabilization and the PSF at the time of lens stabilization contribute at a ratio of 1: 2.

  Further, the decentering direction at the time of lens stabilization will be described. Depending on the direction of decentration of the anti-vibration lens group, the MTF may improve or deteriorate. In many cases, the decentering direction of the anti-vibration lens group whose MTF improves and the decentering direction of the anti-vibration lens whose MTF deteriorates become point-symmetrical with respect to the center of the optical axis. At this time, the contribution of PSF is changed so that the correction is strongly applied to the direction in which the MTF deteriorates, and the contribution of the PSF is changed such that the correction is weakly applied to the direction in which the MTF improves. . However, in the case where both directions are lower than the original MTF after decentration of the anti-vibration lens group, correction is similarly applied to both image heights.

  For example, if the original MTF is 60% and the MTF value improves to 70% due to the decentration of the anti-vibration lens group, the variation of the second MTF is -10 (= 60-70 ). Also, if the original MTF is 60% and the MTF value is degraded to 50% due to the decentration of the anti-vibration lens group, the change amount of the second MTF is 10 (= 60-50) It becomes. Since the contribution rate is proportional to this ratio, PSF at the time of lens anti-vibration when MTF value is improved contributes at a ratio of 10/60 gain down and at lens anti-vibration when MTF value is deteriorated The PSF will contribute at a 10/60 gain up ratio.

  Subsequently, in step S808, the camera system control circuit 5 causes the image processing circuit 7 to calculate PSF at a certain image height based on PSF of the filter information selected as described above. That is, the image processing circuit 7 selects a PSF selected according to the image height moved by the imager stabilization, and a PSF selected according to the decentering optical system by the movement of the anti-vibration lens group during lens stabilization. Is multiplied by a coefficient to perform convolution, and the PSF of the image height is calculated.

  Subsequently, in step S809, the camera system control circuit 5 causes the image processing circuit 7 to create an image restoration filter. In this case, the image processing circuit 7 Fourier-transforms the PSF calculated in step S 808 to obtain OTF, and further calculates an image restoration filter which is a function obtained by inverse Fourier-transforming the OTF.

  If it is determined in step S802 that the lens image stabilization and the imager image stabilization are not simultaneously used, and the process proceeds to step S810, the camera system control circuit 5 determines that the image stabilization process used is the lens image stabilization and the imager. It is detected which one of the vibration reduction is.

  Next, in step S811, the camera system control circuit 5 causes the image processing circuit 7 to acquire a plurality of pieces of filter information used in the image stabilization processing detected in step S810. At this time, the image processing circuit 7 selects, from the memory circuit 8, a plurality of pieces of filter information corresponding to the image stabilization processing detected in step S810. Thereafter, the camera system control circuit 5 advances the process to step S809, and thereafter, the creation of the image restoration filter is performed in the same manner as described above.

  As described above, the imaging apparatus 1 according to the present embodiment uses a plurality of pieces of filter information according to the image stabilization processing that can be used at the time of imaging, for non-anti-vibration mode, for lens stabilization mode, and for imager stabilization mode. Holds only filter information corresponding individually to each of. That is, in the case of the present embodiment, since the filter information corresponding to the simultaneous use of the lens stabilization and the imager stabilization is not prepared in advance and is not stored, an increase in data for the image restoration filter is suppressed. ing. In addition, when lens image stabilization and imager image stabilization are used at the same time, using the image restoration filter individually calculated from both filter information for lens image stabilization and imager image stabilization, deterioration occurs. It can not be corrected well. For this reason, in the present embodiment, an image restoration filter corresponding to simultaneous use of lens stabilization and imager stabilization is generated by performing predetermined arithmetic processing based on filter information held in advance.

  That is, in the case of generating an image restoration filter adapted for simultaneous use of lens stabilization and imager stabilization, the imaging apparatus 1 according to the present embodiment moves the movement amount during lens stabilization and the movement amount during imager stabilization. To get Further, the imaging device 1 acquires filter information according to the conditions of the image stabilization processing used at the time of imaging from among a plurality of pieces of filter information held in advance. That is, the filter information held corresponding to the lens image stabilization and the imager image stabilization is acquired. Further, the imaging apparatus 1 performs predetermined arithmetic processing based on the movement amount at the time of lens stabilization and imager stabilization and the filter information, thereby simultaneously using the lens stabilization and the imager stabilization. The image restoration filter used for the image restoration process of is calculated. At this time, the predetermined arithmetic processing is performed by multiplying the filter information individually acquired by the lens stabilization and the imager stabilization by a coefficient according to the contribution of the lens stabilization and the imager stabilization to the image stabilization processing. It is done as an operation to perform convolution. Then, using the created image restoration filter, the imaging device 1 of the present embodiment performs an image restoration process on a captured image acquired by simultaneously using the lens stabilization and the imager stabilization.

  As described above, in the imaging apparatus 1 of the present embodiment, the image restoration filter adapted to the case where two image stabilization processes of lens stabilization and imager stabilization are simultaneously used is created. Therefore, according to the imaging apparatus 1 of the present embodiment, high-accuracy image stabilization processing can be performed while reducing the amount of data to be held in advance, and high-accuracy image recovery processing is performed on the deteriorated image after the image stabilization processing. be able to. That is, in the present embodiment, the PSF due to the eccentricity of the optical system is combined in a matrix in the amount of eccentricity and the direction, and data is held discretely. By holding it, it is possible to reduce the amount of data. In the above description, although the filter information to be held is PSF information, it is not limited to this, and it is data of any format as long as it is information representing optical transfer functions such as OTF, wavefront, MTF and PTF. May be

<Other Embodiments>
The imaging apparatus according to the present embodiment is applicable not only to digital cameras whose lenses can be replaced, but also to digital cameras and video cameras whose lenses are fixed to the camera body, other industrial cameras, in-vehicle cameras, medical cameras, etc. is there.

  The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  The above-described embodiments are merely examples of implementation for practicing the present invention, and the technical scope of the present invention should not be interpreted limitedly by these. That is, the present invention can be implemented in various forms without departing from the technical concept or the main features thereof.

1: Imaging device 2: Lens unit 3: Imaging optical system 5: Camera system control circuit 6: Imaging device 7: Image processing circuit 9: Display device 10: Operation detection circuit 12: Lens system control Circuit, 13: lens drive unit, 14: blur correction unit, 15: blur detection circuit

Claims (15)

An acquisition unit for acquiring filter information individually corresponding to two or more types of image stabilization processing used in photographing;
Generation means for performing predetermined arithmetic processing based on the filter information acquired corresponding to the two or more types of image stabilization processing, and generating a correction filter used for correction of a photographed image;
A processing unit that performs correction processing using the generated correction filter on a captured image captured using the two or more types of image stabilization processing;
An image processing apparatus comprising: And holding means for holding a plurality of pieces of filter information individually corresponding to a plurality of image stabilization processes that can be used when taking an image,
The acquisition means acquires filter information individually corresponding to two or more types of image stabilization processing used in the photographing from among the plurality of held filter information. The image processing apparatus according to   3. The image processing system according to claim 1, wherein the two or more types of image stabilization processing include image stabilization processing in which the whole or a part of the imaging optical system is decentered with respect to the optical axis of the imaging optical system. Image processing device.   The image processing apparatus according to claim 1, wherein the two or more types of image stabilization processing include image stabilization processing for moving an imaging element with respect to an optical axis of an imaging optical system.   The two or more types of image stabilization processing move the image pickup element with respect to the optical axis of the imaging optical system, and the image stabilization processing in which the whole or a part of the imaging optical system is eccentric with respect to the optical axis of the imaging optical system. The image processing apparatus according to claim 1, further comprising: anti-vibration processing. The acquisition means may include a point spread function in a state of being decentered with respect to an optical axis of the imaging optical system when a vibration reduction process that decenters the whole or a part of the imaging optical system is used. Acquired as information,
The image processing apparatus according to claim 3, wherein the generation unit generates the correction filter based on the acquired point spread function. The acquisition means is a point image distribution in an image circle larger than an image circle corresponding to a state decentered with respect to the optical axis of the imaging optical system when vibration reduction processing for moving the imaging element is used. Get a function as the filter information,
The image processing apparatus according to claim 4, wherein the generation unit generates the correction filter based on the acquired point spread function. A point spread function in a state of being decentered with respect to an optical axis of the imaging optical system when the image acquisition optical system is used in an anti-vibration process for decentering the whole or a part of the imaging optical system; A point spread function in an image circle larger than an image circle corresponding to a state decentered with respect to the optical axis of the imaging optical system when vibration reduction processing for moving the imaging element is used; Acquired as information,
The image processing apparatus according to claim 5, wherein the generation unit generates the correction filter based on the acquired point spread function.   The generation means calculates a coefficient representing a contribution to the movement of the imaging optical system from the center of the optical axis of the used image stabilization processing, multiplies the point spread function by the coefficient, and convolutes the point spread function. The image processing apparatus according to any one of claims 6 to 8, wherein a correction filter is generated. The generation means calculates a coefficient based on the amount of change of the point spread function before and after movement of the imaging optical system from the center of the optical axis of the used image stabilization processing,
The image processing apparatus according to any one of claims 6 to 8, wherein the point spread function is multiplied by the coefficient to perform convolution to generate the correction filter. Acquisition means for acquiring a plurality of point spread functions individually corresponding to two or more types of image stabilization processing used in photographing;
Calculating means for calculating a second point spread function by performing convolution on the plurality of point spread functions corresponding to the two or more types of image stabilization processing;
Processing means for performing correction processing using a correction filter generated from the second point spread function on a captured image captured using the two or more types of image stabilization processing;
An image processing apparatus comprising: An image processing apparatus according to any one of claims 1 to 11.
An imaging unit for capturing the captured image;
An imaging apparatus characterized by having: An image processing method performed by an image processing apparatus, comprising:
An acquisition step of acquiring filter information individually corresponding to two or more types of image stabilization processing used in photographing;
A generation step of performing a predetermined arithmetic processing based on the filter information acquired corresponding to the two or more types of image stabilization processing, and generating a correction filter used for correcting the photographed image;
A processing step of performing correction processing using the generated correction filter on a captured image captured using the two or more types of image stabilization processing;
An image processing method comprising: An image processing method performed by an image processing apparatus, comprising:
An acquiring step of acquiring a plurality of point spread functions individually corresponding to two or more types of image stabilization processing used in photographing;
Calculating the second point spread function by performing convolution on the plurality of point spread functions corresponding to the two or more types of image stabilization processing;
A processing step of performing correction processing using a correction filter generated from the second point spread function on a photographed image captured using the two or more types of image stabilization processing;
An image processing method comprising:   A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 11.

Images