JP2015233211A - Imaging apparatus, control method thereof, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of effectively supporting panning even in the case where a difference of a moving speed between the imaging apparatus and a subject exceeds a range within which the difference is correctable by correcting an optical system.SOLUTION: The imaging apparatus includes an imaging mode for imaging a moving subject while moving the imaging apparatus. The imaging apparatus includes: detection means for detecting a speed difference between the imaging apparatus and the subject under the imaging mode; setting means by which, in the case where it is determined that the detected speed difference exceeds a range within which the speed difference is correctable by correcting the optical system of the imaging apparatus, an imaging condition is set for performing imaging multiple times within the correctable range; and synthesizing means which generates a synthetic image using a plurality of images obtained by performing imaging multiple times in accordance with the imaging condition.

Description

  The present invention relates to an imaging apparatus, a control method therefor, and a program, and more particularly to an image generation technique for panning shooting.

  There is panning as a shooting technique that expresses the speed of moving subjects. In this photographing technique, when the photographer moves (panning) the camera in accordance with the movement of the subject, the moving subject is stopped and the background flows. In general panning, photographing is performed by adjusting the exposure time longer than usual in accordance with the moving speed of the subject to be photographed. Patent Document 1 discloses a configuration in which shooting is performed with an exposure time that can ensure a sufficient amount of background flow during panning shooting.

  In panning shots, the photographer needs to pan in time with the movement of the subject, but if the panning speed is too fast or too slow, the subject will move to the subject if there is a difference between the moving speed and the panning speed. It becomes an image. To solve such a problem, a technique is known in which a shift lens for optically correcting camera shake is moved to absorb the difference between the moving speed of the subject and the panning speed. Patent Document 2 discloses a configuration in which a panning shot is performed by correcting the position of a subject by moving a shift lens in order to position the detected subject at the center of the image.

JP 2010-245774 A JP 2006-317848 A

  In the correction by the shift lens disclosed in Patent Document 2, since the operating range of the shift lens is limited, the amount (maximum correctable angle) that can be corrected by the shift lens when the difference between the moving speed of the subject and the panning speed is large. May exceed.

  An example of such a situation is as shown in FIG. FIG. 7 shows the angular velocity at which the camera pans and the angular velocity of the subject with the rotation center at the panning center of the camera. For example, when the angular velocity of the subject is 30 deg / sec and the panning angular velocity of the camera is 24 deg / sec, the difference of 6 deg / sec needs to be corrected inside the camera. The correction angle finally required inside the camera is the angular velocity that needs to be corrected inside the camera multiplied by the exposure time. If the maximum correctionable angle of the shift lens is 0.4 degrees and the exposure time is 1/16 second, the correction angle that is finally required is given by Equation 1.

  In this case, since the correction angle is within the maximum correctable angle, correction by the shift lens is possible. On the other hand, when the image is taken with an exposure time as long as 1/8 second, the finally required correction angle is expressed by Equation 2.

  In this case, since the finally required correction angle exceeds the maximum correctable angle by the shift lens, it cannot be corrected by the shift lens. As described above, since the maximum correctable angle is exceeded depending on the exposure time, there is a case where the image of the photographed subject is blurred without being corrected. As described above, since it is necessary to take a long exposure time in panning shot shooting, it is assumed that inconveniences that cannot be sufficiently corrected become more conspicuous.

  The present invention has been made in view of the above-described problems of the prior art, and can effectively support panning even when the difference in moving speed between the imaging device and the subject exceeds the range that can be corrected by correction of the optical system. An object of the present invention is to provide an imaging device, a control method, and a program.

  In order to solve this problem, for example, an imaging apparatus of the present invention has the following configuration. That is, an imaging apparatus having a shooting mode for shooting a moving subject while moving the imaging apparatus, the detection means for detecting a speed difference between the imaging apparatus and the subject in the shooting mode, and the detected speed difference Is determined to exceed the range that can be corrected by the correction of the optical system of the imaging apparatus, setting means for setting shooting conditions for performing multiple shootings within the correctable range, and shooting conditions And a compositing means for generating a composite image using a plurality of images obtained by a plurality of photographing operations according to the above.

  According to the present invention, even when the difference between the moving speeds of the imaging device and the subject exceeds the range that can be corrected by the correction of the optical system, it is possible to effectively support panning.

1 is a block diagram illustrating an example of a functional configuration of a digital camera as an example of an imaging apparatus according to an embodiment of the present invention. 6 is a flowchart showing a series of operations of a panning assist process (a), an exposure time limit process (b), and an image composition process (c) according to the first embodiment. FIG. 4 is a diagram illustrating a method for calculating the angular velocity of a subject on the image plane according to the first embodiment. FIG. 5 is a diagram illustrating an example of setting shooting conditions according to the first embodiment. FIG. 4 is a diagram schematically illustrating a region of a subject that is photographed by moving the shift lens according to the first embodiment. The figure which shows the synthetic | combination process which concerns on Embodiment 1 typically. Diagram showing the amount of correction required for panning shots 7 is a flowchart showing a series of operations of a panning assist process (a) and an exposure time limit process (b) according to the second embodiment. 8 is a flowchart showing a series of operations of the panning composition processing according to the second embodiment. The figure which shows the subject in the case of producing | generating the panning shot image which concerns on Embodiment 2 by a synthesis | combination. The figure explaining calculation of the tap number of LPF which concerns on Embodiment 2. The figure which shows the calculation example of the movement vector which concerns on Embodiment 2. FIG. 6 is a diagram for explaining calculation of a pixel movement amount using an angular velocity of a digital camera according to a second embodiment. The figure explaining the synthetic | combination process which concerns on Embodiment 2. FIG.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following, an example in which the present invention is applied to an arbitrary digital camera capable of panning shooting will be described as an example of an imaging apparatus. However, the present invention is not limited to a digital camera and can be applied to any device capable of panning shooting. These devices may include, for example, a mobile phone, a game machine, a tablet terminal, a personal computer, a clock-type or glasses-type information terminal.

(1 Configuration of digital camera 100)
FIG. 1 is a block diagram illustrating a functional configuration example of a digital camera 100 as an example of an imaging apparatus according to the present embodiment. One or more of the functional blocks shown in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or MPU executing software. May be. Further, it may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as the operation subject, the same hardware can be realized as the subject.

  The optical system 101 includes a lens, a shutter, a diaphragm, and a shift lens, and forms an optical image from a subject on the image sensor 102 under the control of the control unit 103. The shift lens realizes a camera shake correction mechanism, and changes the relative position between the optical axis of the lens and the image sensor 102, and corrects the imaging position of the subject with respect to the imaging surface.

  The image sensor 102 photoelectrically converts an optical image from the imaged subject at each pixel, and further performs analog / digital conversion by an A / D conversion circuit to output a digital image signal (image data) in units of pixels. The image sensor 102 may be an image sensor such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

  The control unit 103 is, for example, a CPU or MPU, and is a control unit that controls the entire digital camera 100 by developing and executing a program stored in the secondary storage device 108 in the work area of the primary storage device 104. . It controls the entire panning assist process, exposure time limit processing, image composition processing, and the like in panning shooting to be described later.

  The primary storage device 104 is a volatile storage medium such as a RAM, for example, stores temporary data, and is used for the operation of the control unit 103. In addition, the information stored in the primary storage device 104 is used by the image processing unit 106 as necessary, and is recorded on the recording medium via the recording medium control unit 107.

  The secondary storage device 108 is a nonvolatile storage medium such as an EEPROM, for example, stores a program (firmware) for controlling the digital camera 100 and various setting information, and is used by the control unit 103.

  The angular velocity sensor 105 is composed of a gyro sensor or the like, detects an angular velocity representing the amount of movement of the digital camera 100, converts it into an electrical signal, and notifies the control unit 103 of it.

  The image processing unit 106 performs image processing called so-called development processing, and also adjusts color tone according to the shooting mode. Note that at least part of the functions of the image processing unit 106 may be realized by the control unit 103 in software.

  The recording medium control unit 107 records image data or the like obtained by photographing stored in the primary storage device 104 on a recording medium. The recording medium can be removed from the digital camera 100 such as a semiconductor memory card, for example, and the recorded data can be read by being attached to a personal computer or the like. The recording medium control unit 107 has a recording medium attaching / detaching mechanism and a read / write function.

  The display unit 109 displays a viewfinder image at the time of photographing, a photographed image, a GUI image for interactive operation, and the like.

  The operation unit 110 is an input device group that receives user operations and notifies the control unit 103 of input information. For example, the operation unit 110 may be a button, a lever, a touch panel, or the like, or may be an input device that uses voice or line of sight. . The user can set a shooting mode for shooting, and can select and select the setting for shooting one by one or the panning shooting mode of the present embodiment. Also, there are a plurality of image processing patterns to be applied to the photographed image, and these patterns can also be set from the operation unit 110 as a photographing mode.

(2 series of actions related to panning assist processing)
Next, a series of operations related to the panning assist process will be described with reference to FIG. The panning assist process in the present embodiment captures a plurality of images in panning shooting and generates a panning image by combining the captured image data. This process is started when the shutter button included in the operation unit 110 is half-pressed while the digital camera 100 is panned with respect to a moving subject, for example. This process is realized by the control unit 103 developing and executing a program stored in the secondary storage device 108 in the work area of the primary storage device 104.

  In step S <b> 201, the control unit 103 extracts a subject area from the image data input from the image sensor 102. The subject area is extracted by a known method described in Patent Document 2, for example, a motion vector is obtained from two input image data, and an area having a motion vector smaller than the surrounding area can be extracted as the subject area. The control unit 103 identifies a subject area for each rectangular area or pixel having a predetermined size, and stores it in the primary storage device 104 as information representing the subject area.

  In S202, the control unit 103 acquires the angular velocity information of the digital camera 100 detected by the angular velocity sensor 105 as the panning angular velocity, and the process proceeds to S203.

  In S203, the control unit 103 calculates the angular velocity of the subject. In this embodiment, since the angular velocity of the subject is calculated using the panning angular velocity, the angular velocity centered on the principal point is calculated as shown in FIG. FIG. 3 shows that the subject moved from point A to point B during t [sec], and the subject image formed on the sensor accordingly moved from point C to point D. Here, when the moving amount of the image plane from the point C to the point D is v [pixel], the focal length is f [mm], and the pixel pitch of the sensor is p [μm / pixel], the angular velocity of the subject on the image plane ω [rad / sec] is expressed by Equation 3.

Here, when the digital camera 100 is panned, the angular velocity ω of the subject on the image plane is a difference between the angular velocity (subject angular velocity) ω _s of the subject itself and the panning angular velocity ω _p acquired in S202. (Formula 4)

Therefore, in obtaining the subject angular velocity ω _s , the subject angular velocity ω _s is calculated from the panning angular velocity ω _p of the digital camera 100 detected by the angular velocity sensor 105 using Equation 5.

In the present embodiment, the subject angular velocity ω _s is calculated from the angular velocity ω of the subject on the image plane and the panning angular velocity ω _p , but a value previously designated or selected by the user via the operation unit 110 is used. May be. When the subject angular velocity ω _s is calculated, the control unit 103 advances the process to S204.

In step S204, the control unit 103 performs automatic exposure setting. In the automatic exposure setting, for example, a predetermined exposure time for panning shots that is longer than the exposure time set for normal shooting is set. As will be described later with reference to FIG. 4, if the exposure time set for normal shooting is 1/80, for example, in this step, a long exposure time such as a predetermined 1/10 determined by a setting value is set. The exposure time may be calculated and set according to the subject angular velocity ω _s calculated in S204.

  In S205, the control unit 103 further calculates the longest exposure time. If the exposure time set in S204 is used, the maximum correctable angle by the shift lens may be exceeded. For this reason, a plurality of shootings are performed, and a panning shooting that satisfies the exposure time set in S204 is performed, and each shooting is performed at an exposure time that is less than the maximum exposure time calculated below. The details of the longest exposure time calculation process will be described with reference to FIG. 2B.

First, the control unit 103 in S231 performs the calculation of the corrected angular velocity omega _r. Here, the corrected angular velocity ω _r is calculated by Equation 6 using the subject angular velocity ω _s and the panning angular velocity ω _p .

  In step S232, the control unit 103 acquires a maximum correctable angle. As described above, this maximum correctable angle represents the maximum angle that can be corrected by the optical system 101 having the shift lens. When the optical system 101 is fixed to the main body of the digital camera 100 in advance, the control unit 103 reads the maximum correctable angle of the optical system 101 recorded in advance in the secondary storage device 108. On the other hand, when the optical system 101 is removable, the control unit 103 acquires the maximum correctable angle of the attached optical system 101 by communication with the optical system 101 or is recorded in advance in the secondary storage device 108 or the like. Reads the maximum correctable angle corresponding to each optical system.

In S233, the control unit 103 calculates the longest exposure time when the maximum correctable angle is acquired. For example, when the maximum correctable angle is θ _max [deg] and the correction angular velocity calculated in S231 is ω _r , the longest exposure time t _max is obtained as shown in Equation 7.

In S234, the control unit 103 updates the shooting conditions based on the longest exposure time _tmax calculated in this way. An example of the imaging condition update process will be described with reference to FIG. FIG. 4 shows an example of imaging conditions determined in S204 and S205, where the time for exposing the subject image to the image sensor 102 is expressed as Tv, and the electrical gain of the image sensor 102 is expressed as ISO. For example, assume that shooting conditions for normal shooting (Tv = 1/80, ISO = 800) are set to perform panning shooting. On the other hand, the shooting conditions set in S204 are set so that shooting can be performed with the same brightness with an exposure time longer than normal (Tv = 1/10, ISO = 100). When the shooting conditions are set in this way, for example, when the correction angular velocity ω _r is 6 deg / sec, the angle to be finally corrected is expressed by Equation 8 from the relationship with the set exposure time.

  On the other hand, the control unit 103 compares the finally corrected angle (first correction amount) with the maximum correctable angle (second correction amount), and the maximum correctable angle is the final correction angle. If the angle is equal to or greater than the angle to be corrected, it is determined that correction is possible. On the other hand, for example, when the maximum correctable angle is 0.4 deg, it is determined that correction cannot be performed.

When the maximum correctable angle is less than the angle to be finally corrected, for example, when the maximum correctable angle 0.4 deg and the corrected angular velocity 6 deg / sec are substituted into Equation 7, the longest exposure time t _max is obtained as shown in Equation 9. .

That is, when the maximum correctable angle is 0.4 deg, 1/15 [sec] is the longest exposure time that can be corrected.

The control unit 103 sets shooting conditions for each shooting as shown in Expression 10, in order to perform shooting by shooting a plurality of images. Here, Tv _max is the exposure time determined in S204, and is referred to as a cumulative exposure time setting as a setting of the exposure time over a plurality of shots. By using Expression 10, the exposure time in each shooting with respect to the cumulative exposure time setting can be obtained, and the shooting conditions in each shooting can be set as shown in FIG. For example, assuming that the shooting conditions for the first shooting are set values obtained in S233 (Tv = 1/15, ISO = 100), the setting values (Tv = 1/1 /) separately set for S216 for the second shooting. 30 and ISO = 100) can be set. Note that the exposure time t _n at the n-th shooting is divided exposure time, and the total of the exposure times at the first to n-th shooting is referred to as a cumulative exposure time. When updating the shooting condition for the first shot, the control unit 103 ends the exposure time limiting process and advances the process to S206.

  In step S206, the control unit 103 determines whether to perform an exposure operation. The control unit 103 determines whether a shutter button (not shown) included in the operation unit 110 is fully pressed. If it is determined by the notification from the operation unit 110 that the shutter button has not been fully pressed, the control unit 103 returns the processing to S201 again and repeats the operations from S201 to S205. On the other hand, if it is determined that the shutter button is fully pressed, the process proceeds to S207 to start shooting.

  In step S207, the control unit 103 controls the shutter included in the optical system 101 to start exposure, and exposure is performed until the elapse of the divided exposure time set in step S234 or S216 is determined in step S209. In S208, the control unit 103 operates the shift lens of the optical system 101.

  If the control unit 103 determines in S209 that the divided exposure time has elapsed, in S210, the control unit 103 controls the shutter included in the optical system 101 and ends the exposure. The control unit 103 adds the elapsed divided exposure time to the total exposure time to update the total exposure time, and stores the image data output from the image sensor 102 in the primary storage device 104. On the other hand, if the control unit 103 determines in S209 that the divided exposure time has not elapsed, the process returns to S208.

  In step S211, the control unit 103 determines whether the cumulative exposure time has reached the cumulative exposure time setting, that is, whether all the continuous shooting has been completed. If it is determined that the continuous shooting has ended, the control unit 103 proceeds to step S212 and ends. If not, the process proceeds to S215.

  In step S215, the control unit 103 returns the position of the shift lens in response to an instruction to the optical system 101, and performs imaging again. The position where the shift lens is returned may be anywhere as long as it is within a range that can be corrected by the shift lens during continuous shooting.

  An area of a subject photographed when the shift lens is operated will be described with reference to FIG. In order to simplify the explanation, the change of the object region due to the driving of the shift lens is shown, and the case where there is no change of the region due to panning is shown.

  A region 501 illustrated in FIG. 5A indicates a region of a subject obtained as image data by the image sensor 102 before operating the shift lens, that is, at the time of S207. On the other hand, a region 502 shown in FIG. 5B shows a subject region obtained as image data by the image sensor 102 when the lens is shifted by operating the shift lens in the moving direction of the main subject in S208. Yes. An area 503 shown in FIG. 5C shows the area of the subject when the position of the shift lens moved in the moving direction of the main subject is moved in the reverse direction and the position is returned in S215. Further, an area 504 shown in FIG. 5D shows an area of the subject in which the shift lens is operated again in the moving direction of the main subject in S208 when the second image is taken. An area 505 shown in FIG. 5E is an image taken on the first image and an image taken on the second image, that is, the object area 502 shown in FIG. 5B and the object shown in FIG. The region 504 is aligned and overlapped so that the main subject overlaps. A region that overlaps the region 502 and the region 503, that is, a portion from the left end region of the region 504 to the right end region of the region 502 is a common portion 506. A region 507 on the left side of the common portion in the region 505 is a shift region caused by superposition, and indicates a region that exists in the first image but does not exist in the second image. Similarly, the area 508 is an area that does not exist in the first image but exists in the second image. The common portion 506 is an area that can be combined in an image combining process described later.

  In S216, the control unit 103 returns the process to S207 when the divided exposure time for exposure in the next shooting is set based on the above-described Expression 10.

  In S212, the image processing unit 106 performs an image composition process. Details of the image composition processing will be described with reference to FIG.

  In step S <b> 251, the image processing unit 106 performs a color interpolation process on the image data output from the image sensor 102 to form an RGB plane. In addition, since a plurality of shootings are performed according to the above-described cumulative exposure time, when there is image data stored in the primary storage device 104, the image data is referred to.

  In S <b> 252, the image processing unit 106 performs alignment between images so that the areas of the moving main subject overlap with each other for a plurality of image data obtained by a plurality of shootings. A known alignment technique can also be used for the alignment process. In the alignment process, for example, information representing the subject area corresponding to the first image data identified in S201 is read from the primary storage device 104, and an area that matches the main subject area is identified in the second and subsequent image data. To do.

  In S253, the image processing unit 106 performs filter processing for smoothing the region other than the main subject, that is, the background region, and in S254, the image processing unit 106 adds and combines the images obtained in S253. . Here, the details of the filtering process in S253 and the synthesizing process in S254 will be described with reference to FIG.

  In FIG. 6, an image 601 is an image when exposure is started in the first shooting among a plurality of continuous shootings, and an image 602 is an image when exposure is started in the second shooting. Show. The images exposed over the divided exposure time for these images are an image 603 and an image 604, respectively. Since the car, which is the main subject, is panned in an exposed state, the background subject flows continuously. When the image 603 obtained by the first photographing and the image 604 obtained by the second photographing are subjected to the alignment process in S252, an image with a broken background is obtained as shown in the image 605. This is because a non-exposure period exists because a plurality of shootings are performed, and the main subject moves during the non-exposure period. For this reason, it is different from the image 606 obtained by photographing only one sheet by one exposure. The interval at which the background is interrupted corresponds to the amount of movement of the main subject relative to the image.

  The image processing unit 106 performs processing to compensate for a pixel in a moving part of the background during the non-exposure period, that is, a part where the background is interrupted. Specifically, a low-pass filter (LPF) process using the amount of movement of the main subject image plane during the non-exposure period as the number of taps is applied to the image 603 obtained in S252 to express the background during the non-exposure period. An image 607 (background interpolation image) is generated. Then, the background interpolation image and the continuously captured image (for example, the image 604) are combined to generate a panning image in which the background is not interrupted.

  In step S254, the image processing unit 106 adds and combines the background interpolation image and the continuously captured image to generate a composite image that is an output image. Note that the image to which the LPF is applied may be any image taken continuously, and may be generated by mixing a plurality of background interpolated images when combining. The image processing unit 106 ends the image composition process and returns the process to the caller.

  In step S <b> 213, the image processing unit 106 performs development processing such as color tone adjustment on the composite image under the instruction of the control unit 103 and outputs image data. In step S <b> 214, the image processing unit 106 records the captured image generated by cropping only the common area when the continuously captured images are overlapped, via the secondary storage device 108 or the recording medium control unit 107. Record on media. When the image processing unit 106 finishes writing to the secondary storage device or the recording medium, the control unit 103 ends the series of panning assist processing.

  In the present embodiment, the shooting condition set in the automatic exposure setting in S204 is updated in S234. However, for example, even if only the shooting conditions are set in S234 without performing the automatic exposure setting in S204, an image by the same panning shot can be obtained. Although the exposure time is set as the shooting condition, an image by the same panning shot can be obtained even when the shutter speed and the exposure amount are applied. In addition, the process of applying the LPF to one of the continuously shot images in S253 is performed. However, even if the continuously shot images are synthesized and the LPF is applied to the background area, the above-described synthesized image is also obtained. A composite image similar to is obtained.

  In the present embodiment, an example in which the imaging position of the subject with respect to the imaging surface is corrected using the shift lens has been described. However, for example, the above-described panning image can be obtained by shifting the imaging element 102 itself.

  As described above, in the present embodiment, when the correction angle with respect to the movement of the subject exceeds the range that can be corrected by the shift lens, the exposure time is divided and a plurality of shootings are performed, and then a smoothing process is performed on the shot image To generate an image composed of these images. In this way, even when the difference in moving speed between the imaging apparatus and the subject exceeds the range that can be corrected by the correction of the optical system, it is possible to effectively support the panning.

(Embodiment 2)
Next, Embodiment 2 will be described. In the first embodiment, when it is determined that correction exceeding the maximum correctable angle is necessary, the exposure time is divided within a range that can be corrected by the shift lens, and photographing is performed a plurality of times. On the other hand, in the second embodiment, when correction exceeding the maximum correctable angle is required, a panning image is generated by shooting and combining a plurality of images without using optical correction such as a shift lens. To do. For this reason, the panning assist process of the present embodiment has the same configuration for performing a plurality of shootings in accordance with a comparison between an angle that needs to be corrected and a maximum correctable angle. Further, the processing after performing a plurality of shootings, that is, the application of a filter for generating a panning image and the synthesizing processing can have the same configuration as that of the first embodiment. However, in the description of the present embodiment, an example in which a modification that can be applied to the first embodiment is applied to the image composition processing will be described. In addition, about the same structure, the same code | symbol is attached | subjected and the overlapping description is abbreviate | omitted and it demonstrates focusing on a different point.

  A series of operations of the panning assist process in the second embodiment will be described with reference to FIG.

  In S801, when the calculated longest exposure time is shorter than the exposure time calculated in S204, the control unit 103 sets the exposure time calculated in S204 (for example, 1/10 second) as an equivalent shutter speed of 1/10 second. Set. Note that the equivalent shutter speed means a virtual shutter speed for the composite image, and corresponds to setting a predetermined shutter speed that represents the degree of movement when performing panning shot shooting with one shot. In the present embodiment, the shooting for each frame is performed according to a predetermined frame rate (for example, 60 fps) set in advance.

  In step S802, the control unit 103 determines whether the shutter button of the operation unit 110 is fully pressed. If it is determined that the shutter button is fully pressed, the process proceeds to step S803. If it is determined that the shutter button is not fully pressed, the process proceeds to step S201. To return.

  In step S803, the control unit 103 measures the exposure of the subject using a photometric sensor (not shown), determines an appropriate exposure using the shutter speed, the aperture value, and the ISO sensitivity, and adjusts the aperture included in the optical system 101. .

  In step S <b> 804, the control unit 103 calculates the lens driving amount of the optical system 101 based on, for example, the phase difference output from the image sensor 102, and specifies the in-focus position.

  In step S <b> 805, the control unit 103 acquires the angular velocity information of the digital camera 100 detected by the angular velocity sensor 105 as the panning angular velocity. In addition, since there is no dependency in the process order of S803-S805, you may change an order according to a process.

  In step S <b> 806, when the control unit 103 performs exposure for the exposure time updated in step S <b> 234, the control unit 103 stores the image data output from the image sensor 102 in the primary storage device 104.

  In step S <b> 807, the control unit 103 performs recognition processing for an important area of the subject area. The subject area recognition process can be realized by using a known recognition technique such as a face detection or human body detection technique or a focus position, and thus detailed description thereof is omitted. The control unit 103 causes the primary storage device 104 to store, for example, an area detected by human detection in the subject as an important area.

  In step S <b> 808, the image processing unit 106 performs development processing according to an instruction from the control unit 103. The image processing unit 106 performs development processing on the image data exposed in S806 and stored in the primary storage device 104 to generate post-development image data. In addition, the color, brightness, sharpness, etc. may be adjusted to preset values so that the important area recognized in S807 is optimized in the composition processing.

  In step S809, it is determined whether the shutter button of the operation unit 110 is fully pressed, and the control unit 103 repeats the operations in steps S803 to S808 as long as it is determined that the button is continuously pressed. If there is a free capacity for storing new data in the primary storage device 104, the process may proceed to S803 in parallel with the processes of S807 and S808. On the other hand, if it is determined that the shutter button has been fully pressed, the process proceeds to S810.

  In step S810, the control unit 103 determines whether all image data stored in the primary storage device 104 has been developed and image data after development has been generated. If all the image data has not been generated, the subject recognition process in S811 and the development process in S812 are repeated. Here, the processes in S812 and S813 are the same processes as the subject recognition process in S807 and the development process in S808.

  In step S813, the control unit 103 performs main subject area recognition processing on the developed image. For example, the control unit 103 reads out the important area acquired in S807 or S811 from the primary storage device 104, and recognizes the largest area from the center of the image as the main subject area.

  In step S814, the control unit 103 sets a search block for each image between successive images after development, and detects a movement vector in the search block. The movement vector is detected only for the main subject region in order to reduce the amount of calculation, but may be detected for the entire image.

  Note that the processing order of S813 and S814 may be switched. In the subject recognition processing in S813 when the order is changed, an area that is equal to or smaller than a predetermined movement amount is recognized as the main subject area based on the movement vector for the important subject area detected in S814. good. At this time, the moving body and the stationary body may be distinguished using the angular velocity and the movement vector of the digital camera 100 acquired in S805. A peripheral region having a movement vector equivalent to an important subject region may be extracted.

  In step S815, the control unit 103 uses the main subject area extracted in step S813 and the information on the movement vector detected in step S814 so as to align the main subject area for all developed images. At this time, the movement vector of an important area and the angular velocity of the digital camera 100 may be taken into consideration.

  In step S816, the image processing unit 106 performs a panning composition process using the registration information obtained in step S815, and superimposes the images to generate a panning image. The details of the panning composition processing will be described later with reference to FIG.

  In step S817, the control unit 103 displays the generated panning image on the display unit 109. In step S818, the image processing unit 106 performs panning on the recording medium via the recording medium control unit 107 in accordance with the control of the control unit 103. Record an image. At this time, the image processing unit 106 compresses the generated panning shot image into a predetermined compression format and outputs it to a recording medium. When the recording of the panning shot image on the recording medium is completed, the control unit 103 ends the series of panning shot assist processing operations.

  Next, a series of operations of the panning composition processing executed in S816 will be described with reference to FIG.

  In step S901, the image processing unit 106 performs addition synthesis on the plurality of images aligned in step S815 to create one synthesized image. In step S902, the image processing unit 106 calculates the number of taps used in the LPF process for the image combined in step S901. In step S903, the image processing unit 106 applies the calculated LPF process to the combined image, thereby changing the background flow. Create a representation of the image. Details of the tap count calculation process and the LPF process used for the LPF process will be described later.

  In step S904, the image processing unit 106 combines the main subject region of the image combined in step S901 and the region other than the main subject region of the image subjected to LPF in step S903, and generates an image having the same effect as the panning as a whole. Generate. When the image processing unit 106 completes image generation, the control unit 103 advances the process to step S817.

  Here, the calculation processing of the number of taps used for the LPF processing in S902 will be described in detail with reference to FIGS.

  Since general panning is performed with one exposure, the background changes so as to flow continuously. For this reason, it is desirable that the change of the background 1003 in the image 1001 and the image 1002 illustrated in FIG. 10 changes so as to continuously flow like the background of the image 1004. However, when the image 1001 and the image 1002 are combined, a background 1005 is obtained, and the image does not change so as to continuously flow. For this reason, it is necessary to compensate for the insufficient pixels of the background 1003 in the gap portion 1006 of the background so that the background of the image 1004 is expressed.

  In the present embodiment, the relationship between the frame rate and the number of combined images is used to generate an image with the equivalent shutter speed set in S801. For example, when an image is taken at 60 fps, the image is synthesized from one image when the corresponding shutter speed is set to 0 to 1/60 seconds, and from two images when the corresponding shutter speed is set to 1/30 seconds. This expresses the amount of sink. In the present embodiment, when the equivalent shutter speed is 1/10 seconds and the frame rate is 60 fps, six consecutive images are used for composition.

  FIG. 11A shows a state where the position of the background 1003 on the image changes from the pixels 1101, 1102, and 1103 in order for each shooting, with each block as one pixel on the image. Since the position of the background 1003 appears discretely for each captured image, it is necessary to interpolate the pixel group 1104 between the pixels 1101 and 1102 and the pixel group 1105 between the pixels 1102 and 1103. In addition, since the background flows between 1103 and 1106 when the image is actually shot at the set equivalent shutter speed, the pixel group 1107 between 1103 and 1106 must also be interpolated. FIGS. 11B to 11D show images captured at the timing when the background 1003 appears in 1101 to 1103 and sections to be interpolated.

  In this embodiment, after combining a plurality of captured images, an LPF process using the width of the pixel group 1104 (or 1105, 1107) as the number of taps is applied to the combined image, and the overall flow amount is determined as a pixel. A composite image having a width 1108 is generated. The flow amount of the generated composite image is as shown in FIG.

  In order to calculate the number of taps corresponding to the width of the pixel group 1104, the amount of pixel movement between the captured images is obtained. Hereinafter, a method for calculating the amount of movement of pixels between images using the angular velocity of the digital camera 100 will be described with reference to FIG.

  FIG. 13 shows a state in which the digital camera 100 pans in the direction indicated by reference numeral 1301 around the center point 1306 to photograph the background subject 1302 and movement of the subject image on the image plane when the center point 1306 is the main point. Displayed together.

  Expression 11 is an expression for calculating the number of taps of the LPF based on the amount of movement between images of the subject 1302 that changes when the digital camera 100 pans at an angle of 1303.

  First, the image processing unit 106 calculates a movement amount 1305 [pixel] on the image plane of the image sensor 102 from the rotation angle 1303 [rad] and the focal length 1304 [mm] between the images of the digital camera 100. At this time, the rotation angle 1303 between the images of the digital camera 100 is calculated from the average angular velocity [rad / sec] and the frame rate [frame / sec] of the digital camera 100. Then, using the pixel pitch [mm / pixel], the amount of movement 1305 on the surface of the image sensor 102 converted into the amount of movement on the image is handled as the number of taps used for LPF processing. Reproducing the flow amount corresponding to the equivalent shutter speed by further combining a plurality of images subjected to LPF processing with the number of taps obtained from the above formula according to the equivalent shutter speed arbitrarily set by the user Can do.

  Next, the details of the LPF processing applied in S903 will be described with reference to FIG.

  In FIG. 14, images 1401 and 1402 are images taken continuously, and are images taken while panning the moving main subject 1403. The subject area is separated by performing the main subject recognition in S813 described above. For example, the information 1404 representing the subject area represents the main subject area in white and the other areas in black. If the main subject recognition enables accurate separation at the boundary between the main subject and the background, in the image composition in S904, the pixels in the main subject region affect the image after applying the LPF (that is, the pixels in the background region). rare. For this reason, an image obtained by applying an LPF with a uniform coefficient in the tap to only the area excluding the main subject area is referred to as a background image 1405.

  Finally, in S904, the main subject area of the main subject image 1406 and the area excluding the main subject area of the background image 1405 are combined.

  By the way, in the above synthesis process, there is an advantage that almost no afterimage is generated around the main subject when the main subject can be accurately extracted, while when the main subject cannot be extracted accurately, the main subject and the background can be obtained. In some cases, an image with a sense of incongruity is generated at the boundary.

  In general, there are various cases where it is difficult to accurately separate the boundary between the main subject and the background. For example, when there is almost no difference in contrast between the boundary between the main subject and the background, the background may be extracted as the main subject region, or a part of the main subject may be classified as the background region.

  For this reason, when the main subject cannot be accurately extracted, a process of generating a composite image in which the boundary between the main subject image 1406 and the background image 1407 is smoothly changed is performed. First, a background image 1407 is generated. Specifically, using the panning angular velocity of the digital camera 100 acquired in S202 or S805, the panning direction and the amount of movement are detected for the entire image. Then, after generating an image excluding the main subject area, the main subject area is interpolated using background pixels. Specifically, the main subject region is interpolated by applying the above-described panning direction and size to pixels around the main subject region. In this way, it is possible to generate a background image 1407 in which the region that was the main subject region is filled with the pixels constituting the background. Further, smoothing processing is performed on the boundary between the main subject and the background in the main subject area, and the boundary between the main subject and the background in the main subject area is blurred as shown in the information 1408 indicating the subject area. Is added to the main subject image and the background image.

  In this way, the boundary between the main subject and the background gradually changes, so that even if the main subject has not been accurately extracted, it is possible to generate an image that is smoothly synthesized with little discomfort. Note that the boundary may be blurred by a process similar to a resolution reduction due to image reduction / enlargement instead of the smoothing process for the boundary between the main subject image 1406 and the background image 1407.

  Furthermore, it may be difficult to cope with the above-described synthesis method. For example, when the main subject itself has a low contrast, the recognition accuracy of the subject region is lowered, and a state such as a hollow 1409 may occur in the main subject region.

  For this reason, first, as in 1410, smoothing processing is performed on the boundary between the main subject and the background in the main subject region to reduce the influence of the void. Then, by applying an LPF with the pixel value of the main subject area as a coefficient as shown in Expression 12 to the entire image, a background image 1411 in which the pixels of the main subject area are difficult to spread around while blurring the entire image. create.

  At this time, in step S <b> 904, the control unit 103 combines the main subject image 1406 and the background image 1411 by weighted addition to the pixel value of the main subject area 1410 to generate a panning image 1412.

  In this way, not only can the pixels in the main subject area be prevented from spreading to the surroundings, but also the effects of the voids can be reduced when voids occur in the main subject. Since the surrounding pixels recognized as the main subject area are not used much, the image in which the pixels in the low contrast area are blurred is hardly noticeable. Furthermore, it becomes less noticeable by combining with the main subject image.

  In the present embodiment, the angular velocity of the digital camera 100 is used to determine the amount of pixel movement when calculating the number of taps of the LPF. However, the length of the movement vector obtained between images may be used as shown in FIG. More specifically, when two consecutive images 1201 and 1202 are captured, the background 1203 is moved between the two images by determining the correspondence between the search blocks provided in the images. A vector 1204 is calculated. The calculated length of the movement vector may be the number of LPF taps.

  As described above, in this embodiment, when the correction angle with respect to the movement of the subject exceeds the angle that can be corrected by the shift lens, a plurality of short-second images captured at the set frame rate are combined and smoothed against the background. I went and generated an image that represented the flow of the background. By doing so, even when the difference in moving speed between the imaging apparatus and the subject exceeds the range that can be corrected by the correction of the optical system as in the first embodiment, the panning can be effectively supported. Also, since the exposure time is short and it is combined using a blur-free image, a panoramic image representing the main subject with little blur is obtained in an expression equivalent to a low-speed shutter speed that is difficult to shoot with general panning. It is done. Furthermore, by synthesizing with the image by positioning with respect to the main subject, a panning image can be generated without requiring optical correction.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 101 ... Optical system, 102 ... Image sensor, 103 ... Control part, 105 ... Angular velocity sensor, 106 ... Image processing part

Claims (18)

An imaging apparatus having a shooting mode for shooting a moving subject while moving the imaging apparatus,
Detecting means for detecting a speed difference between the imaging device and the subject in the photographing mode;
When it is determined that the detected speed difference exceeds a range that can be corrected by correction of the optical system of the imaging device, shooting conditions for performing multiple shootings within the correctable range are set. Setting means to
Combining means for generating a combined image using a plurality of images obtained by shooting a plurality of times according to the shooting conditions;
An imaging device comprising:   When the speed obtained from the preset exposure time for the shooting mode and the detected speed difference exceeds a speed that can be corrected by correcting the optical system of the imaging apparatus, the setting means The imaging apparatus according to claim 1, wherein the determined speed difference exceeds a range that can be corrected by correcting an optical system of the imaging apparatus.   In the case where it is determined that the detected speed difference exceeds a range that can be corrected by correction of the optical system of the imaging apparatus, the setting means determines that the total exposure time of the plurality of shootings is the shooting mode. The imaging apparatus according to claim 1, wherein the imaging condition is set so as to be equal to an exposure time for the imaging.   The setting means is an exposure time for photographing within a range that can be corrected by correction of the optical system of the imaging device, obtained from the detected speed difference and a range that can be corrected by correction of the optical system of the imaging device. The imaging apparatus according to any one of claims 1 to 3, wherein the imaging condition is set based on the information.   5. The imaging apparatus according to claim 1, wherein the correction of the optical system is correction of a relative position between an optical axis of the optical system and an imaging element included in the imaging apparatus. .   The imaging apparatus according to claim 1, wherein the optical system is corrected using a camera shake correction mechanism. An imaging apparatus having a shooting mode for shooting a moving subject while moving the imaging apparatus,
Calculating means for calculating a first correction amount for correcting blurring of the subject in a captured image due to a difference in moving speed between the imaging device and the subject;
When the first correction amount is larger than a second correction amount that can be corrected by controlling the imaging position of the subject with respect to the imaging surface, a plurality of shootings are performed within the range of the second correction amount, and Setting means for setting shooting conditions for realizing the first correction amount;
Combining means for generating a combined image using a plurality of images shot under the shooting conditions;
An imaging device comprising:   When the first correction amount is larger than the second correction amount, the setting means determines an exposure time for each of the plurality of shootings based on an exposure time necessary to obtain the first correction amount. The imaging device according to claim 7, wherein the imaging device is set.   When the first correction amount is larger than the second correction amount, the setting means sets the exposure time of each of the plurality of shootings to be less than the time for obtaining the second correction amount. The imaging apparatus according to claim 7 or 8, wherein   When the first correction amount is larger than the second correction amount, the setting means is configured such that the sum of the exposure times of the plurality of photographings becomes a time for obtaining the first correction amount. The imaging apparatus according to claim 9, wherein an exposure time for each of the plurality of times of shooting is set.   11. The second correction amount is a maximum correction amount that can be corrected by driving an optical system to control an imaging position of a subject with respect to an imaging surface. The imaging device described in 1.   The imaging apparatus according to claim 1, wherein the synthesizing unit generates a synthesized image in which a background of the subject flows in a moving direction of the subject.   The synthesizing unit generates an image obtained by smoothing the background of the subject in the moving direction of the subject with respect to at least one of the plurality of images, and the synthesized image synthesized using the image is generated. The imaging device according to claim 1, wherein the imaging device is generated.   2. The composite image according to claim 1, wherein the composite unit generates the composite image by combining the plurality of images calculated from the exposure time necessary for the plurality of shootings and a frame rate for shooting. 8. The imaging device according to 7.   The synthesizing unit generates an image obtained by smoothing the background of the subject with the amount of movement on the background image of the subject obtained from the relationship between the moving speed of the imaging device and the focal length, and synthesizes the image using the image. The imaging apparatus according to claim 14, wherein the composite image is generated. A method of controlling an imaging apparatus having a shooting mode for shooting a moving subject while moving the imaging apparatus,
A detecting step for detecting a speed difference between the imaging device and the subject in the photographing mode;
When the setting means determines that the detected speed difference exceeds a range that can be corrected by correction of the optical system of the imaging apparatus, the setting unit is configured to perform shooting a plurality of times within the correctable range. A setting process for setting shooting conditions;
A combining step for generating a combined image using a plurality of images obtained by a plurality of shootings according to the shooting conditions;
A method for controlling an imaging apparatus, comprising: A method of controlling an imaging apparatus having a shooting mode for shooting a moving subject while moving the imaging apparatus,
A calculating step for calculating a first correction amount for correcting blurring of the subject in the captured image due to a difference in moving speed between the imaging device and the subject;
When the setting unit is larger than the second correction amount that can be corrected by controlling the imaging position of the subject with respect to the imaging surface, the first correction amount is shot a plurality of times within the range of the second correction amount. Performing a setting step for setting a photographing condition for realizing the first correction amount;
A combining step of generating a combined image using a plurality of images shot by the shooting condition;
A method for controlling an imaging apparatus, comprising:   The program for functioning a computer as each means of the imaging device of any one of Claims 1 thru | or 15.