JP2009296561A - Imaging apparatus and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of efficiently performing blur correction with precision corresponding to a required image quality, and to provide an imaging method like this. <P>SOLUTION: Each time the blurring amount of a subject image, from exposure start, formed on the imaging plane of an imaging element 5 via an imaging lens 1 reaches a predetermined amount depending on the focal distance of the imaging lens 1 and parameters determining image quality, image data are read from pixels of the imaging element 5. Image data of a plurality of frames read from the pixels of the imaging element 5 several times are associated with one another and combined in such a way that the same portions of images represented by the image data of the plurality of frames are overlapped, thereby generating a blur-reduced image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus capable of obtaining an image in which blurring is corrected, and such an imaging method.

  In an imaging device such as a digital camera, it is widely known that a subject image formed on an image sensor blurs due to a camera shake of a photographer or a movement of a subject during imaging, particularly when imaging is performed for a long time. ing.

As one of the methods for correcting blur, imaging is performed in a time-division manner with an exposure time that can suppress the blur amount to be equal to or less than an allowable amount, and is expressed by a plurality of image data obtained as a result of each imaging. Patent Document 1, Patent Document 2, and the like have disclosed a technique for generating an image in which one blur is corrected by combining the plurality of image data so that the mutual blur between the respective images is reduced. Proposed. In Patent Document 1, exposure is performed until the amount of blurring from the start of exposure reaches a predetermined amount of blurring that allows blurring, and the operation of reading out and synthesizing image data from the image sensor is repeatedly performed. A reduced image is generated. In Patent Document 2, image data read out in a time-sharing manner from a single-plate image sensor in which a color filter is arranged is subjected to address correction with an integer multiple of the number of repeated pixels of a unit pattern in color coding of the color filter. However, image data with reduced blur is generated by combining with previously captured image data.
Japanese Patent Laid-Open No. 2003-32540 JP 2005-328326 A

  In an imaging device that captures a still image, it is required to capture all scenes with certainty. Therefore, it is an essential requirement to perform imaging at high speed. In addition, since imaging in every scene is required, it is an essential requirement to withstand long-time imaging. In order to enable long-time imaging, it is desirable to reduce power consumption during imaging.

  What has been described above is no exception in the camera shake prevention processing, and high speed and low power consumption are also required in the camera shake prevention processing.

  Here, as in Patent Document 1 and Patent Document 2, in the method of correcting the shake by correcting the shake of a plurality of image data obtained by time-division imaging, the shake in the image obtained after the synthesis is increased. In order to correct the accuracy, it is necessary to shorten the exposure time of each imaging in the time-division imaging so as to reduce the amount of image blur in each imaging. In order to make the S / N ratio of the combined image data substantially constant irrespective of the exposure time in each time-division imaging, it is necessary to keep the exposure time of the entire combined image data substantially constant. Therefore, it is necessary to increase the number of times of time-division imaging as the exposure time in time-division imaging becomes shorter. However, when the number of time-division imaging is increased, the number of images that have to be subjected to image processing increases, resulting in an increase in the amount of overall image processing, which adversely affects other processing. When processing time is increased or high-speed processing is performed in order to shorten the processing time, power consumption increases and the above-described basic requirements of the imaging apparatus cannot be satisfied.

  On the other hand, depending on the image quality mode, the exposure time is generally 1 / f seconds (where f is the focal length of an imaging lens for a 35 mm film camera and the unit is “mm”), which is an exposure time in which blurring is not noticeable. Even if time-division imaging is performed, a sufficient blur correction effect may not be obtained.

  As described above, the number of time-division imaging is a very important factor that affects the function and performance of the imaging device. However, in the conventional technology, the number of times of time-division imaging is set regardless of the image quality mode. In some cases, high-accuracy shake correction is performed or the accuracy is insufficient.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image pickup apparatus and such an image pickup method capable of efficiently performing shake correction with high accuracy according to required image quality. To do.

  In order to achieve the above object, the imaging apparatus according to the first aspect of the present invention reduces mutual blurring of images represented by each of a plurality of frames of image data obtained by time-division imaging. An imaging device that synthesizes the image data of the plurality of frames, the imaging device having an imaging surface in which a plurality of pixels are two-dimensionally arranged, and an imaging lens that forms a subject image on the imaging surface of the imaging device A blur amount detection unit that detects a blur amount of the imaged subject image from a reference position on the imaging surface; an image quality parameter setting unit that sets parameters related to the image quality of the images of the plurality of frames; Every time the amount of blurring from the start of exposure reaches a predetermined amount depending on the parameter relating to the image quality and the focal length of the imaging lens, an operation of reading image data from the pixels of the imaging element is performed a plurality of times. The image data of the plurality of frames is associated and synthesized so that the same portion of the image represented by the element reading control unit and each of the image data of the plurality of frames read from the pixel of the image sensor a plurality of times overlap each other. And an image composition unit.

  In order to achieve the above object, in the imaging method according to the second aspect of the present invention, mutual blurring of images represented by each of a plurality of frames of image data obtained by time-division imaging is reduced. As described above, in the imaging method for combining the image data of the plurality of frames, a step of detecting a blur amount from a reference position on the imaging surface of the subject image formed on the imaging surface of the imaging element; The step of setting parameters related to the image quality of the frame image, and the amount of blurring from the reference position on the imaging surface from the start of exposure depends on the parameter related to the image quality and the focal length of the imaging lens. Each time a fixed amount is reached, the step of reading the image data from the pixels of the image sensor a plurality of times and the image data of a plurality of frames read from the pixels of the image sensor a plurality of times To overlap the same portion of the image represented me, and having the steps of: synthesizing in association with image data of said plurality of frames.

  In order to achieve the above object, in the imaging method according to the third aspect of the present invention, mutual blurring of images represented by each of a plurality of frames of image data obtained by time-division imaging is reduced. In this way, the image pickup method combines the image data of the plurality of frames, and each time the amount of blur from the start of exposure reaches a predetermined amount that depends on the parameter related to the image quality and the focal length of the image pickup lens, the image pickup device The step of repeatedly reading the image data from the pixels of the image and the image data of the plurality of frames are combined and associated so that the same portion of the images of the plurality of frames represented by the image data read a plurality of times overlap. And a step of performing.

  ADVANTAGE OF THE INVENTION According to this invention, the imaging device which can perform the blurring correction of the precision according to the required image quality efficiently, and such an imaging method can be provided.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating a configuration of a digital camera as an example of an imaging apparatus according to an embodiment of the present invention. The digital camera shown in FIG. 1 includes an imaging lens 1, a lens driving system 2, an aperture 3, a diaphragm driving system 4, an imaging element 5, an imaging element driver 6, and a timing generator (TG) circuit 7. A double sampling (CDS) circuit 8, an amplifier circuit 9, an analog / digital (A / D) converter 10, a data bus 11, a built-in memory 12, an image processor 13, an AE processor 14, AF processing unit 15, compression / decompression unit 16, removable memory 17, display unit 18, nonvolatile memory 19, main CPU 20, input unit 21, power supply unit 22, sub CPU 23, A / D A conversion unit 24 and angular velocity sensors 25 and 26 are provided.

  The imaging lens 1 is an optical system for forming an optical image of a subject (subject image) on the imaging surface of the imaging element 5. The lens driving system 2 is a driving system for driving a focus adjustment lens included in the imaging lens 1. The focus adjustment of the imaging lens 1 can be performed by driving a focus adjustment lens via the lens drive system 2. If the imaging lens 1 is a zoom lens, zooming can be performed by driving the imaging lens 1 by the lens driving system 2.

  The diaphragm 3 is provided to adjust the amount of light incident on the image sensor 5 via the imaging lens 1. The aperture drive system 4 is a drive system for driving the aperture 3. The amount of light incident on the image sensor 5 is adjusted by opening and closing the aperture 3 via the aperture drive system 4. Thereby, the exposure amount of the image sensor 5 can be controlled.

  The imaging element 5 is configured by disposing a plurality of different color filter segments in a predetermined periodic array (for example, a Bayer array) on the front surface of an imaging surface configured by two-dimensionally arranging a plurality of pixels. Has been. The image sensor 5 converts a subject image formed on the imaging surface into an electric signal (image data). The image sensor driver 6 performs an image capturing operation by the image sensor 5 and a signal reading operation from a pixel of the image sensor 5. The TG circuit 7 supplies a timing signal for determining the operation timing of the image sensor 5, the CDS circuit 8, and the A / D converter 10 to the image sensor driver 6, the CDS circuit 8, and the A / D converter 10.

  The CDS circuit 8 removes noise in the image data by performing CDS processing on the image data output from the image sensor 5 in accordance with the timing signal from the TG circuit 7. The amplifier circuit 9 amplifies the image data output from the CDS circuit 8 at a predetermined amplification factor specified by the main CPU 20. The A / D converter 10 converts the image data output as an analog signal from the amplifier circuit 9 into a digital signal.

  The data bus 11 is a transfer path for transferring various data generated inside the digital camera to each unit in the digital camera. The built-in memory 12 is a storage unit that temporarily stores various data such as image data obtained by the A / D conversion unit 10 and image data processed by the image processing unit 13 and the compression / decompression unit 16. .

  The image processing unit 13 performs image processing such as synchronization processing, white balance correction processing, YC separation processing, color conversion processing, and gradation conversion processing. In addition, the image processing unit 13 includes an image composition unit 13a. The image compositing unit 13a performs multiple-frame images such that the same portions of the multiple-frame images represented by the pixel signals (image data) of the image sensor 5 read out multiple times are overlapped by time-division imaging described later. The image data is combined in association with each other to generate one image data with reduced blur.

  The AE processing unit 14 performs a photometric process for measuring the luminance of the subject. Note that the luminance of the subject may be detected using a dedicated sensor, or may be detected from image data. The AF processing unit 15 measures the distance from the digital camera to the subject. The subject distance may be detected based on the output of a dedicated sensor, or may be calculated from the amount of extension of the imaging lens 1 at the time of focus adjustment.

  When recording the image data, the compression / decompression unit 16 compresses the image data processed by the image processing unit 13 according to a predetermined compression method such as a JPEG compression method. When reproducing the image data, the compression / decompression unit 16 reads and decompresses the compressed image data recorded in the removable memory 17.

  The detachable memory 17 is a recording medium composed of a memory that can be detachably attached to the digital camera body, and stores image data compressed by the compression / decompression unit 16. In FIG. 1, the detachable memory 17 is used as a recording medium for recording image data, but a detachable memory is not necessarily used as a recording medium.

  The display unit 18 is, for example, a liquid crystal display (LCD), and displays various images based on the image data processed by the image processing unit 13.

  The nonvolatile memory 19 stores various parameters necessary for the operation of the digital camera of FIG. 1 and various programs executed by the main CPU 20 and the sub CPU 23. The main CPU 20 comprehensively controls various sequences of the digital camera body. The main CPU 20 reads parameters necessary for various sequences from the nonvolatile memory 19 according to a program stored in the nonvolatile memory 19 and executes each process. Further, the main CPU 20 has a timer counter 20a for measuring the exposure time and the like.

  The input unit 21 is an operation member including a release button 21a and an image quality input unit 21b for inputting parameters related to image quality (image quality parameters). The image quality parameters input by the image quality input unit 21b are read into the main CPU 20. When one of the operation members of the input unit 21 is operated by the user, the main CPU 20 executes various sequences according to the user's operation. The release button 21a has a two-stage switch including a first release switch and a second release switch. When the release button 21a is half-pressed and the first release switch is closed, the main CPU 20 performs an imaging preparation sequence such as AE processing and AF processing. Further, when the release button 21a is fully pressed and the second release switch is closed, the main CPU 20 executes an imaging sequence to perform imaging. The image quality input unit 21b is an operation member such as a cross key. The image quality input unit 21b can set the image quality (image size, compression rate) of the compressed image data recorded in the removable memory 17. Therefore, the image quality input unit 21b constitutes an image quality parameter setting unit.

  The power supply unit 22 is a power supply for the digital camera shown in FIG. The main CPU 20 supplies power supplied from the power supply unit 22 to each block shown in FIG.

  The sub CPU 23 comprehensively controls a sequence related to blur correction. The A / D converter 24 converts the shake signal (angular velocity signal) output as an analog signal from the angular velocity sensors 25 and 26 into a digital signal and outputs the digital signal to the sub CPU 23. The angular velocity sensor 25 is an angular velocity sensor for detecting an angular velocity which is a change amount per unit time of the rotation angle θx around the X axis of the digital camera shown in FIG. The angular velocity sensor 26 is for detecting an angular velocity which is a change amount per unit time of the rotation angle θy around the Y axis of the digital camera shown in FIG.

  Here, the angular velocity sensors 25 and 26 will be further described with reference to FIG. FIG. 2 is a diagram illustrating the relationship between the coordinate axes set in the digital camera and the arrangement of the two angular velocity sensors 25 and 26.

  In FIG. 2, the Z axis is set in a direction along the optical axis O of the imaging lens 1 at a certain time. Also, the X axis is set in the left-right direction when the digital camera 100 is viewed from the subject side on a plane perpendicular to the Z axis. Further, the Y axis is set in the vertical direction when the digital camera 100 is viewed from the subject side through the intersection of the Z axis and the X axis. The positive direction of the Z axis is the subject side. The positive direction of the X axis is the right direction when the digital camera 100 is viewed from the subject side. Further, the positive direction of the Y axis is the upward direction. Note that, as shown in FIG. 2, the optical axis O and the Z axis of the imaging lens 1 coincide with each other at a certain time, but when a shake occurs at another time, the optical axis of the imaging lens 1. O generally does not coincide with the Z axis.

  The rotation angles around the X, Y, and Z axes set as described above are θx, θy, and θz, respectively. As shown in FIG. 2, the two angular velocity sensors 25 and 26 are provided so as to be arranged in relation to the X axis, the Y axis, and the Z axis.

  Hereinafter, the operation of the digital camera shown in FIG. 1 will be described. FIG. 3 is a flowchart showing the flow of the imaging operation of the digital camera including the imaging method of the present embodiment.

  First, the main CPU 20 determines whether or not the release button 21a is half-pressed and the first release switch is closed (step S101). In the determination in step S101, the main CPU 20 continues the determination in step S101 until the first release switch is closed. On the other hand, if it is determined in step S101 that the first release switch is closed, the main CPU 20 executes photometric processing (measurement of subject brightness) by the AE processing unit 14 (step S102). Thereafter, the main CPU 20 calculates an exposure time (Texp) by apex calculation based on the photometric result in the AE processing unit 14 (step S103). Here, the exposure time Texp is a standard exposure time in normal imaging. This exposure time Texp is substantially equal to the combined exposure time for imaging a plurality of frames in time-division imaging described later. Next, the main CPU 20 adjusts the focus of the imaging lens 1 based on the output of the AF processing unit 15 (step S104). At this time, the main CPU 20 holds the subject distance calculated by the AF processing unit 15.

  Next, the main CPU 20 determines whether or not the release button 21a is fully pressed and the second release switch is closed (step S105). If it is determined in step S105 that the second release switch is not closed, the process returns to step S102. In this case, the main CPU 20 performs the photometry by the AE processing unit 14 again. On the other hand, if it is determined in step S105 that the second release switch is closed, the main CPU 20 starts exposure of the image sensor 5 via the TG circuit 7 (step S106). Thereby, accumulation of electric charge according to the subject image is started in each pixel of the image sensor 5.

  Further, after the second release switch is closed, the amount of shake is calculated independently of the processing of FIG. In the processing after step S106, the shake amount calculation processing is greatly involved, and here, the shake amount calculation will be described.

  After the second release switch is closed, the sub CPU 23 takes in the detection signals output from the angular velocity sensors 25 and 26 as digital data by the A / D converter 24, respectively. Further, the sub CPU 23 acquires information regarding the focal length f of the imaging lens 1 via the main CPU 20 (for example, when the imaging lens 1 is a zoom lens, the main CPU 20 transmits information via the lens driving system 2). Or when the imaging lens 1 is an interchangeable lens barrel, information is acquired via a communication contact or the like). Further, the sub CPU 23 acquires subject distance information from the main CPU 20. The information on the focal length f and the subject distance information are used for the calculation of the shake amount in the X direction and the calculation of the shake amount in the Y direction, which will be described later.

Next, the relationship between the shake amount and the shake correction amount will be described with reference to FIG.
FIG. 4 is a diagram illustrating a moving state of the image of the subject 63 on the imaging surface when the digital camera 100 is shaken by the rotation angle θx.

  Assuming that the digital camera 100 is rotated by the rotation angle θx due to shaking or the like, the imaging lens 1 is rotated and moved to the position indicated by reference numeral 1 ′ along with this, and the imaging surface 61 of the imaging element 5 is also moved by the angle θx. Move to the tilted CD plane position. Further, the image of the subject 63 that was at the position indicated by reference numeral 62 when no blur has occurred is located at the position indicated by reference numeral 62 ′ on the imaging surface CD after the blur of the rotation angle θx occurs. Moving.

Here, the focal length of the imaging lens 1 is f, the distance from the object space focal point of the imaging lens 1 to the subject 63 when no blurring occurs, and the image space of the imaging lens 1 when blurring does not occur. When the distance from the focal point to the image position is L ′ and the amount of movement of the image position due to blur is ΔY, respectively, the geometric positional relationship as shown in FIG. 4 and the Newton shown in the following (Expression 1) Imaging formula

Is used to calculate the movement amount ΔY as shown in the following (Equation 2).

  Here, β in (Expression 2) indicates an imaging magnification, and β = f / L. In calculating (Expression 2), it is assumed that θx is a minute amount, and approximation to the first order of θx is performed. Further, as described above, the focal length f in (Expression 2) is acquired via the main CPU 20 and input to the sub CPU 23 as lens information of the imaging lens 1. Furthermore, the subject distance L necessary for calculating β can be acquired from the AF processing unit 15. Further, the rotation angle θx can be calculated based on the output from the angular velocity sensor 25.

  Even if blurring occurs in the digital camera by performing substantial correction related to the movement amount ΔY obtained based on (Expression 2), the image represented by the image data output through the image sensor 5 is displayed. It is possible to prevent the influence of shaking from occurring.

  As described above, since the angle θx is a minute amount, even if the imaging surface CD is inclined about the X axis by the angle θx around the X axis as shown in FIG. The effect on the generated image is not a problem other than the movement amount ΔY described above.

Further, the amount of movement ΔX of the image position when the digital camera 100 rotates about the Y axis by the rotation angle θy is also obtained as shown in the following (Expression 3), similarly to (Expression 2).

Here, the following (Expression 4) is obtained by differentiating both sides of (Expression 2) with respect to time.

  Since the right side d (θx) / dt of (Expression 4) is the angular velocity itself around the X axis, the output of the angular velocity sensor 25 can be used as it is. Further, the left side d (ΔY) / dt in (Expression 4) is the image movement speed Vy in the Y-axis direction when an angular velocity of d (θx) / dt occurs.

Similarly, with respect to the movement amount ΔX of the image position in the X-axis direction when a shake occurs around the Y-axis by the rotation angle θy, the following (Expression 5) is obtained by differentiating both sides of (Expression 3) with respect to time. Is obtained.

  Since the right side d (θy) / dt of (Expression 5) is the angular velocity itself around the Y axis, the output of the angular velocity sensor 26 can be used as it is. Further, the left side d (ΔX) / dt of (Expression 5) is the image moving speed Vx in the X-axis direction when an angular velocity of d (θy) / dt occurs.

Now, the predetermined time ΔT (ΔT is a sampling interval for converting the output of the A / D conversion unit 24 into a digital signal. This ΔT is the focal length of the photographing lens 1 converted to the photographing lens of a 35 mm film camera f [ mm], the time is preferably equal to or shorter than 1 / f [s]. The output d (θx) / dt of the angular velocity sensor 25 detected in the period is ωx1, ωx2, ωx3. ,..., .Omega.x (n-1), .omega.xn, the amount of movement .DELTA.Y of the image position in the Y-axis direction after elapse of n.times..DELTA.T is given as shown in the following (Expression 6). .

Similarly, if the output d (θy) / dt of the angular velocity sensor 26 detected at every predetermined time ΔT (in a predetermined time ΔT cycle) is ωy1, ωy2, ωy3,..., Ωy (n−1), ωyn. The amount of movement ΔX of the image position in the X-axis direction after elapse of n × ΔT is given as shown in the following (Expression 7).

  As described above, the moving amount of the image (that is, the amount of blur) in the imaging of a plurality of frames exposed at the time interval of n × ΔT by the imaging device 5 can be calculated by (Equation 6) and (Equation 7). it can. Therefore, based on the blur amounts ΔX and ΔY calculated by these equations, the pixel data of the image data corresponding to each image is associated with each other so that the blur of each image is corrected. By performing the synthesizing process, it is possible to generate one image data in which blurring is corrected.

  FIG. 5 is a flowchart showing a flow of processing for calculating the movement amounts ΔX, ΔY and the like by the sub CPU 23. As described above, the process of FIG. 5 is executed as a process independent of the process shown in FIG. 3 from when the second release switch is closed by half-pressing the release button 21a until the exposure is completed. It is like that.

  In the process of FIG. 5, the sub CPU 23 determines whether or not the second release switch is closed (step S401). Then, the determination in step S401 is repeated until the second release switch is closed.

  If it is determined in step S401 that the second release switch is closed, the sub CPU 23 acquires the focal length f and the subject distance L of the imaging lens 1 (step S402).

These focal distance f and subject distance L may be calculated in the process shown in FIG. Further, in order to calculate the shake amount at a faster cycle, the focal distance f and the subject distance L are calculated using a separate processor or the like, and the sub CPU 23 acquires the calculated data in step S402. It is good to do so. As a result, it is possible to increase the processing speed and to follow the actual shake in real time.

  Next, the sub CPU 23 acquires the angular velocities ωx and ωy by reading the outputs of the angular velocity sensors 25 and 26 via the A / D converter 24 (step S403). Thereafter, the sub CPU 23 calculates the cumulative addition value of the angular velocity up to the value detected this time by adding the acquired angular velocity ωx, ωy to the cumulative addition value of the angular velocity up to the previously acquired value (step S404).

  Thereafter, the sub CPU 23 substitutes the cumulative addition value calculated in step S404 into (Expression 6) and (Expression 7), respectively, so that the time from the first image capturing end point in the multiple frame image capturing in the time-division image capturing is calculated. The movement amounts ΔY and ΔX of the image position are calculated (step S405).

  Next, the sub CPU 23 calculates Px = “ΔX / Lx” and Py = “ΔY / Ly” (step S406). Here, Lx and Ly represent the size of the pixel in the X direction and the Y direction, respectively, and “” means an integer value obtained by rounding off fractions. Therefore, Px and Py represent the movement amounts ΔX and ΔY of the image position from the end of imaging of the first time-division image in units of pixels.

  Subsequently, the sub CPU 23 stores Px and Py in the corresponding memories [Px] and [Py], respectively (step S407). Thereafter, the sub CPU 23 determines whether or not the exposure for the exposure time Texp has been completed (step S408). If it is determined in step S408 that the exposure has not ended, the process returns to step 403. In this case, the sub CPU 23 repeats the processes after obtaining the angular velocities ωx and ωy. On the other hand, when the exposure is completed in step S408, the sub CPU 23 ends the process shown in FIG. Therefore, the angular velocity sensor 25, the angular velocity sensor 26, the A / D conversion unit 24, and the sub CPU 23 constitute a shake amount detection unit.

  Returning to the description of FIG. In step S106, after the exposure of the image sensor 5 is started, the main CPU 20 starts counting of the timer counter 20a for measuring the exposure time (step S107). Next, the main CPU 20 performs initial setting of variables. Here, 0 is stored in each of the memories [Px0] and [Py0] for storing variables Px0 and Py0 necessary for the calculation described later. In the present embodiment, by calculating the cumulative blur amount from the start of exposure of the first time-division imaging, and subtracting the cumulative blur amount until reading the image data by the previous time-division imaging from this cumulative blur amount, respectively. The amount of blur within the exposure time of the time-division imaging is obtained. Variables Px0 and Py0 are cumulative blur amounts in the X and Y directions until image data obtained by the previous time-division imaging is read. Further, in time-division imaging, 0 is stored in a memory [Tm0] for storing an exposure time Tm0 until image data captured last time is read from the image sensor 5. Further, 0 is stored in the memory [n] for storing the number n of time-division imaging. Further, 1 is stored in the memory [A] for storing the amplification factor A of the amplifier circuit 9.

  Next, the main CPU 20 calculates Px−Px0 and Py−Py0 and stores the calculation results in the memories [Sx] and [Sy], respectively (step S109).

  Thereafter, the main CPU 20 amplifies from the count value Tm of the timer counter 20a (Tm is equal to the cumulative exposure time of the latest time-division image data from the first time-division imaging start time (substantially the same as the time of step S107)). The value Tm0 of the timer counter 20a before reading the previous image data set in the rate setting process (details will be described later) (Tm0 is the accumulation from the first time-division imaging start time to the time of the previous time-division image data reading) (Tm−Tm0) / Texp) obtained by dividing the time Tm−Tm0 (equal to the exposure time in the latest time-division imaging) by the standard exposure time Texp in normal imaging calculated in step S103. Is smaller than a constant k1 stored in advance in the nonvolatile memory 19 (step S110). Here, (Tm−Tm0) / Texp gives an indication of the level (exposure amount) of image data in the latest time-division imaging. When (Tm−Tm0) / Texp is smaller than k, it means that the S / N of the image data is at an unacceptable level. In this case, the image data is not read and the process branches to step S111. Then, the main CPU 20 determines whether or not Tm is equal to or higher than Texp (step S111). If Tm is greater than or equal to Texp in the determination in step S111, image data is read (described later) after step S116. As will be apparent from the description below, in the processing from step 113 onward, image data is not read out unless the amount of blur is greater than a predetermined amount. However, it is necessary to read out the image data even when there is no blurring. For this reason, the determination in step S111 is provided, and image data is read out when the exposure time Tm is equal to or greater than Texp even if no blurring occurs. Therefore, in the present embodiment, when no blur occurs at all, imaging is performed in the same manner as normal imaging. In step S111, if Tm ≧ Texp is not satisfied, the exposure amount is insufficient, so the process returns to step S109 and the process is repeated.

  If (Tm−Tm0) / Texp is not smaller than k1 in the determination in step S110, the S / N of the image data is at the allowable level. In this case, the main CPU 20 determines whether Sx is larger than the allowable shake amount Sx0 or whether Sy is larger than the allowable shake amount Sy0 (step S112).

  Here, the allowable shake amount in step S112 will be described. First, the allowable blur amounts in the X direction and the Y direction represented by the length on the imaging surface are Lx0 and Ly0, respectively. For example, an image obtained by an image sensor having a size of 24 × 36 [mm] is stretched at a magnification of 6 times on a print paper of 18 × 24 [cm] (so-called 6-cut) from a distance of 40 [cm]. Suppose you observe. Since the visual two-point discrimination resolution of a person with a visual acuity of 1.0 is about 1 [min], the allowable circle of confusion of the image stretched on the print paper is 40 [cm] × 1 [min] × 2π / (360 × 60 [min]) = 116 [μm]. Therefore, the allowable circle of confusion on the imaging surface of the imaging device is 116 [μm] / 6 = 19 [μm]. In the present embodiment, the allowable blur amounts Sx0 and Sy0 are set based on the size of the allowable circle of confusion.

  Note that the image quality mode of the digital camera in this embodiment can be selected by selecting an image size and a compression rate, as shown in FIG. 6 as an example. The allowable blur amount depends on the image size, compression ratio, image enlargement magnification, observation distance, and the like. In the present embodiment, for example, the allowable blur amount when the image size is 1600 pixels × 1200 pixels and the compression rate is normal is set to 19 [μm] equal to the size of the allowable circle of confusion described above. The image size of 1600 pixels × 1200 pixels is an image size at a level that does not impair the resolution of the original image sensor even when a blur of an allowable blur amount of 19 [μm] occurs. Although what allowable blur amount is set for other image sizes depends on the enlargement magnification and the observation distance of the image, in this embodiment, the allowable blur amount is set in inverse proportion to the image size. In addition, even if the image size is the same, the image quality deteriorates if the compression rate is high. Therefore, the higher the compression ratio, the larger the allowable blur amount is set.

  By dividing the allowable blur amount described above by the horizontal length (Lx) of the unit pixel and the vertical length (Ly), the allowable blur amounts Sx0, Sy0 in the X direction and the Y direction expressed in pixel units. Calculate The allowable blur amounts Sx0 and Sy0 are stored in advance in the nonvolatile memory 19 for each image size and compression rate.

  If it is determined in step S112 that Sx is greater than the allowable shake amount Sx0 or Sy is greater than the allowable shake amount Sy0, the main CPU 20 determines whether Sx / Sx0 is greater than the constant k2 stored in the nonvolatile memory 19 Or whether Sy / Sy0 is larger than k2 (step S113). If it is determined in step S113 that Sx / Sx0 is greater than k2 or Sy / Sy0 is greater than k2, the main CPU 20 displays a warning on the display unit 18 assuming that the shake is greater than the allowable level (step S114). . A warning sound may be used instead of the warning display. The determination process in step S113 is provided for the following reason. The limit at which blurring can be corrected depends on the period of reading out image data of a plurality of frames from the image sensor in time-division imaging. If the readout cycle is long, the amount of blur in time-division imaging tends to be larger than the allowable amount. It is a design problem whether the warning amount is displayed with respect to the permissible amount, but it is preferable to set k2 = 2, for example. This is because k2 = 1 is troublesome because warnings are frequently displayed, and if k2 is too large, the reliability of the product is questioned.

  In the determination at step S113, when both Sx / Sx0 and Sy / Sy0 are equal to or less than k2, the main CPU 20 executes a subroutine for setting the amplification factor of the amplifier circuit 9 (step S115). Since the exposure amount in the time division imaging is smaller than the exposure amount in the normal imaging, the image data in the time division imaging is amplified by the amplification circuit 9. As a result, the input signal level to the A / D conversion unit 10 during time-division imaging is substantially equal to the input signal level during normal imaging, and the quantization error associated with A / D conversion of the A / D conversion unit 10 is reduced. can do.

  Here, the subroutine for setting the amplification factor will be further described. FIG. 7 is a flowchart showing a subroutine for setting the amplification factor. The main CPU 20 subtracts the exposure time Tm0 until the image data in the previous time-division imaging is read from the cumulative exposure time Tm from the start of exposure in the first time-division imaging, and this value (this value is as described above). Is stored in a memory [ΔTexp] for storing a variable ΔTexp (step S501). Next, the main CPU 20 stores Tm in the memory [Tm0] as a value of a new variable Tm0 (step S502). Next, the main CPU 20 stores Texp / ΔTexp in the memory [A] for storing the amplification factor A (step S502). Thereafter, the main CPU 20 sets the amplification factor of the amplifier circuit 9 to A (step S504). By setting the amplification factor in this way, the image data from the image sensor 5 is multiplied by Texp / ΔTexp when reading the image data. As a result, the level of the image data input to the A / D conversion unit 10 becomes substantially equal to the level during normal imaging.

  Returning to FIG. 3 again, the description will be continued. After setting the amplification factor in step S115, the main CPU 20 stores the blur amounts Px and Py from the start of exposure in the memories [Px0] and [Py0] that store the cumulative blur amounts Px0 and Py0 until the previous image data is read. By doing so, Px0 and Py0 are updated (step S116). Next, the main CPU 20 controls the TG circuit 7 to start reading image data from the image sensor 5 (step S117). Next, the main CPU 20 performs image composition of the image data read this time and image data that has already been corrected and combined (step S118). Therefore, the main CPU 20, the image sensor driver 6, and the TG circuit 7 constitute a pixel readout control unit.

  The image composition subroutine will be further described with reference to FIG. FIG. 8 is a flowchart showing an image composition subroutine. First, the main CPU 20 determines whether or not a variable n indicating the number of time-division imaging is 0 (step S601). In the determination in step S601, when n = 0, the image data obtained by the first time-division imaging is read out. At this time, the main CPU 20 stores the image data A read out via the image sensor 5 and obtained by the A / D converter 10 in the built-in memory 12 (step S602). An image represented by the image data stored in the built-in memory 12 is referred to as an image A. On the other hand, if it is not n = 0 in the determination in step S601, it is a case where image data by the second time division imaging is read out. In this case, the main CPU 20 associates the image data A and the image data B so that the image B represented by the image data B newly read from the image sensor 5 and the same portion of the image A overlap each other. Synthesize. Specifically, when the blur amounts in the X direction and the Y direction stored in the memories [Sx] and [Sy] in step S109 in FIG. The read address of each pixel of the image data B is controlled so as to overlap by shifting by −Sx in the X direction and −Sy in the Y direction, and the correspondence is made after associating with the data of each pixel of the image data A The pixel data to be combined is synthesized. The synthesized image data is stored again in the built-in memory 12 (step S603). Next, the main CPU 20 stores n + 1 in the memory [n] for storing the variable n, and returns from the processing of FIG. Therefore, the main CPU 20, the built-in memory 12, and the image processing unit 13 constitute an image composition unit.

  Returning to FIG. 3 again, the description will be continued. When image composition ends in step S118, the main CPU 20 determines whether or not the value Tm of the timer counter 20a is equal to or longer than the exposure time Texp (step S119). If it is determined in step S119 that Tm <Texp, the process returns to step S109. Then, the main CPU 20 repeats the processing after step S109. On the other hand, in the determination of step S119, when Tm ≧ Texp, the main CPU 20 ends the processing of FIG.

  As described above, according to the present embodiment, the allowable amounts Sx0 and Sy0 of the blur amount are set according to the image quality mode (image size and compression rate) of the recorded image, and the blur amounts Sx and Sy are the blur amount. The image data is not read until the allowable amounts Sx0 and Sy0 are exceeded. As a result, the number of times of time-division imaging is not increased more than necessary, so that it is possible to perform blur correction at high speed and with low power consumption, and perform accurate blur correction according to the required image quality mode. Is possible.

  The present invention is not limited to the above-described embodiment, and includes the following modifications. Below, the modification contained in this invention is illustrated. The contents described in each modification can be arbitrarily combined as long as there is no contradiction.

[Modification 1]
In the present embodiment, image composition is performed based on image data read out from the image sensor 5 to the outside, but the image sensor 5 may be configured to have an image composition function. In addition, when the image sensor 5 is provided with an image composition function, the image data may be composed of an analog signal or may be composed after the analog signal is converted into a digital signal. Good.

[Modification 2]
In this embodiment, every time image data is read out from the image sensor 5, blur correction and image synthesis are performed between the read image data and image data that has already been read and subjected to blur correction and image synthesis processing. Repeatedly, high-speed image composition processing was made possible. However, after storing all image data in time-division shooting in the built-in memory 12, each image represented by a plurality of frames of image data is stored. The image data of the plurality of frames may be associated and combined so that the same portion overlaps.

[Modification 3]
In the present embodiment, the number of times of imaging (m) is controlled so that the total exposure time from the start of time-division imaging to the end of time-division imaging is substantially equal to the standard exposure time Texp, but is not necessarily limited to this. The total exposure time from the start of time-division imaging to the end of time-division imaging may be longer or shorter than the standard exposure time Texp as necessary. . For example, in order to reduce random noise in the combined image data, the total exposure time may be set longer than the standard exposure time to increase the number of times of time-division imaging, or the time-division imaging time may be shortened. In order to do this, the total exposure time may be made shorter than the standard exposure time to reduce the number of imaging in time-division imaging.

[Modification 4]
Although the image quality parameter is input by the operator through the image quality input unit 21b, the image quality parameter may be automatically set by the main CPU 20 according to the remaining memory capacity of the removable memory 17, for example. In this case, the main CPU 20 constitutes an image quality parameter setting unit.

1 is a block diagram illustrating a configuration of a digital camera as an example of an imaging apparatus according to an embodiment of the present invention. It is a figure which shows the relationship between the coordinate axis set to the digital camera, and arrangement | positioning of the two angular velocity sensors 25 and 26. FIG. It is a flowchart which shows the flow of the imaging operation | movement of a digital camera including the imaging method of one Embodiment of this invention. It is a figure which shows the movement state of the image of a to-be-photographed object on an imaging surface when a digital camera shakes only rotation angle (theta) x. It is a flowchart which shows the flow of a process which calculates movement amount (DELTA) X, (DELTA) Y etc. by sub CPU. It is a figure which shows the image quality mode of a digital camera. It is a flowchart which shows the subroutine of an amplification factor setting. It is a flowchart which shows the subroutine of an image composition.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Imaging lens, 2 ... Lens drive system, 3 ... Diaphragm, 4 ... Diaphragm drive system, 5 ... Image sensor, 6 ... Image sensor driver, 7 ... Timing generator (TG) circuit, 8 ... Correlated double sampling (CDS) Circuit 9, amplification circuit 10, 24, analog / digital (A / D) conversion unit 11 data bus 12 built-in memory 13 image processing unit 14 AE processing unit 15 AF processing unit DESCRIPTION OF SYMBOLS 16 ... Compression / decompression part, 17 ... Detachable memory, 18 ... Display part, 19 ... Nonvolatile memory, 20 ... Main CPU, 20a ... Timer counter, 21 ... Input part, 21a ... Release button, 22b ... Image quality input part, 23 ... Sub CPU, 25, 26 ... Angular velocity sensor

Claims (4)

An imaging device that synthesizes the image data of a plurality of frames so as to reduce mutual blurring of images represented by each of the image data of the plurality of frames obtained by time-division imaging,
An imaging device having an imaging surface in which a plurality of pixels are arranged two-dimensionally;
An imaging lens that forms a subject image on the imaging surface of the imaging element;
A blur amount detection unit for detecting a blur amount of the imaged subject image from a reference position on the imaging surface;
An image quality parameter setting unit for setting parameters related to the image quality of the images of the plurality of frames,
Pixel readout for performing multiple operations of reading image data from the pixels of the image sensor every time the amount of blur from the start of exposure reaches a predetermined amount that depends on the parameter relating to the image quality and the focal length of the imaging lens A control unit;
An image composition unit that associates and composes the image data of the plurality of frames so that the same portion of the image represented by each of the image data of the plurality of frames read out from the pixel of the image sensor a plurality of times overlaps;
An imaging apparatus comprising:   The imaging apparatus according to claim 1, wherein the parameter related to the image quality includes at least one of an image size and a compression rate. An imaging method for combining the image data of the plurality of frames so as to reduce the mutual blurring of images represented by each of the image data of the plurality of frames obtained by time-division imaging,
Detecting a blur amount of a subject image formed on an imaging surface of an imaging element from a reference position on the imaging surface;
Setting parameters related to the image quality of the images of the plurality of frames,
Each time the amount of blurring from the reference position on the imaging surface from the start of exposure reaches a predetermined amount that depends on the parameter relating to the image quality and the focal length of the imaging lens, image data from the pixels of the imaging element is acquired. A step of performing the reading operation multiple times;
Combining the image data of the plurality of frames in association with each other so that the same portion of the image represented by each of the image data of the plurality of frames read out from the pixel of the image sensor a plurality of times overlaps;
An imaging method characterized by comprising: An imaging method for combining the image data of the plurality of frames so as to reduce the mutual blurring of images represented by each of the image data of the plurality of frames obtained by time-division imaging,
Each time the amount of blurring from the start of exposure reaches a predetermined amount that depends on a parameter related to image quality and the focal length of the imaging lens, repeatedly performing an operation of reading image data from the pixels of the imaging element;
Combining the image data of the plurality of frames in association with each other so that the same portions of the images of the plurality of frames represented by the image data read a plurality of times overlap;
An imaging method characterized by comprising:

Images