JP2017079378A - Image processing apparatus and image processing method, and imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve camera shake correction where motion blur on a subject due to camera shake is not conspicuous.SOLUTION: A positional shift amount acquisition part 305 acquires the amount of camera shake when moving image data is photographed. A motion blur calculation part 306 calculates the amount of motion blur of a subject included in the moving image data due to the camera shake, from the amount of camera shake and a camera parameter of an imaging part that has photographed the moving image data. A limit value calculation part 307 calculates, as a correction limit value, the amount of shift of an image in which motion blur is not perceived on the basis of spatial frequency response characteristics of a visual sense related to the speed of the subject and the amount of motion blur. A correction amount setting part 308 sets the amount of correction to be applied to camera shake correction of the moving image data on the basis of the correction limit value.SELECTED DRAWING: Figure 3

Description

  The present invention relates to image processing for performing camera shake correction.

  Not only video cameras but also digital cameras and mobile phones are equipped with a moving image capturing function, and it is important to take measures against blurring caused by camera shake when capturing moving images with these image capturing apparatuses. In order to prevent blur caused by camera shake, various image processing techniques have been proposed as camera shake correction.

  If the camera shake correction is excessive, the subject is displayed stationary due to the correction, and motion blur on the subject caused by the camera shake at the time of shooting may be conspicuous. Patent Document 1 discloses a method of adjusting the correction strength according to the magnitude of camera shake. According to Patent Document 1, the strength of camera shake correction is specified by a user operation, the cut angle of view is increased or decreased according to the specified operation amount, and the strength of camera shake correction is changed. However, according to the technique of Patent Document 1, in order to manually set the strength of camera shake correction, it is necessary to set the correction strength every time shooting is performed.

  Patent Document 2 uses both switching between optical camera shake correction (hereinafter referred to as optical correction) and electronic camera shake correction (hereinafter referred to as electronic correction) according to the magnitude of camera shake, and a change in the correction intensity ratio. We propose a method to do this. According to the technique of Patent Document 2, it is possible to suppress motion blur for small camera shake that can be responded by optical correction. However, when a large amount of camera shake that cannot be suppressed by optical correction occurs, motion blur on the subject becomes noticeable when the electronic correction is increased.

JP 2007-274031 A JP 2013-126075 gazette

J. Laird, M. Rosen, J. Pelz, E. Montag, S. Daly `` Spatio-velocity CSF as a function of retinal velocity using unstabilized stimuli '' Proc. Of SPIE 6057, Human Vision and Electronic Imaging XI, 2006 D. H. Kelly "Motion and vision. II. Stabilized spatio-temporal threshold surface" JOSA, Vol. 69, 1340-1349, 1979

  An object of the present invention is to realize camera shake correction in which motion blur on a subject due to camera shake is inconspicuous.

  The present invention has the following configuration as one means for achieving the above object.

  The image processing according to the present invention acquires the amount of camera shake when shooting moving image data, and determines the subject included in the moving image data caused by camera shake from the camera shake amount and the camera parameters of the imaging unit that has shot the moving image data. The amount of motion blur is calculated, and based on the visual spatial frequency response characteristics related to the subject speed and the amount of motion blur, the amount of shift of the image where motion blur is not perceived is calculated as a correction limit value, and based on the correction limit value, A correction amount to be applied to shake correction of the moving image data is set.

  According to the present invention, it is possible to realize camera shake correction in which motion blur on a subject due to camera shake is not noticeable.

The figure which shows the one part frame image of the moving image image | photographed. 1 is a block diagram illustrating a configuration example of an imaging device. The block diagram which shows the structural example of an image process part. 7 is a flowchart for explaining camera shake correction processing by an image processing unit. The figure which shows the amount of correction | amendment set with respect to the positional offset amount of an image accompanying the camera shake at the time of imaging | photography, and a positional offset amount. The flowchart explaining the process of a motion blur calculation part. The figure which shows an example of camera shake speed and the amount of motion blur, and the amount of motion blur calculated with respect to the camera shake speed. The figure explaining the relationship between a blur filter, a blur image, and a Fourier-transform image. The figure explaining the spatial frequency characteristic of the profit calculated from a Fourier-transform image. The flowchart explaining the process of a limit value calculation part. The figure which shows the visual spatial frequency response characteristic according to the moving speed of a to-be-photographed object, and a blur perception speed. The figure which shows the relationship between motion blur perception speed and a correction limit value. The figure which shows the positional offset amount in the moving image data after the camera shake correction process of an Example. The figure which shows the one part frame image of the moving image after camera shake correction | amendment.

  Hereinafter, an image processing apparatus and an image processing method according to embodiments of the present invention will be described in detail with reference to the drawings. In addition, an Example does not limit this invention concerning a claim, and all the combinations of the structure demonstrated in an Example are not necessarily essential for the solution means of this invention.

[Shooting conditions in the description]
In the following, for ease of explanation, an example will be described in which a user holding an imaging device captures a stationary rectangular wave chart. As a hand shake, a period of 8 frames / cycle and a triangular wave shape, a movement corresponding to an amplitude x ₀ pixel in the horizontal direction on the image and a movement corresponding to x ₁ pixel (x ₀ <x ₁ ) in three cycles It is assumed that they are repeated alternately (see FIG. 5 (a)).

FIG. 1 shows a part of a frame image of a moving image shot under the above conditions. Figure 1 (a) (c) shows a frame image at the time of camera shake amplitude x _0, between each frame picture | 0.5 · x ₀ | positional deviation of results. Figure 1 (b) (d) shows a frame image at the time of camera shake amplitude x _1, between each frame picture | 0.5 · x ₁ | positional deviation of results. Further, when vibration larger hands amplitude x ₁ is motion blur occurs in the captured rectangular wave chart on.

[Device configuration]
A block diagram of FIG. 2 shows a configuration example of the imaging apparatus. A microprocessor (CPU) 205 executes a program stored in a read only memory (ROM) 203 or the like using a random access memory (RAM) 204 as a work memory, and controls each component to be described later via a system bus 215. To do. The programs executed by the CPU 205 include programs that realize various processes described later.

  The imaging unit 201 includes an imaging device such as a lens, an aperture, a shutter, an optical low-pass filter, a color filter, and a CMOS sensor and a charge coupled device (CCD). Output a signal. The flash 202 irradiates the subject with light according to the control of the CPU 205. The imaging control unit 206 controls the imaging system of the imaging unit 201 in accordance with instructions from the CPU 205, such as focusing, opening the shutter, and adjusting the aperture.

  The operation unit 207 corresponds to a button, a mode dial, and the like, and inputs a user instruction. The CG generation unit 208 generates characters and graphics. The display unit 209 is, for example, a liquid crystal display (LCD), and displays images, characters, and graphics input from the CG generation unit 208, the digital signal processing unit (DSP) 210, and the image processing unit 214. The display unit 209 may have a touch screen function. In that case, the display unit 209 functions as a part of the operation unit 207.

  The DSP 210 performs various processes such as demosaicing, development, white balance, gamma correction, and noise reduction on the digital signal input from the imaging unit 201 to generate image data. The compression / decompression unit 211 performs a process of encoding image data into a format such as JPEG (Joint Photographic Experts Group) or MPEG (Moving Picture Experts Group). In addition, JPEG data or MPEG data is decoded into image data.

  An image processing unit 214 which is an image processing apparatus according to the embodiment performs image processing described later on the digital signal output from the imaging unit 201 or the image data output from the DSP 210. The external memory control unit 212 is a media drive having an interface function for connecting the media 213 to the system bus 215. As the medium 213, for example, a hard disk, a memory card, a USB memory, or the like can be used.

  The network interface 215 is an interface for a wireless local area network. The CPU 205 communicates with a server device, a computer device, a printer, and the like via the network interface 215, and uploads and downloads image data and moving image data.

  The CPU 205 supplies the image data and moving image data output from the DSP 210 or the image processing unit 214 to the display unit 209 to display images and moving images, or supplies them to the external memory control unit 212 via the compression / decompression unit 211. Store in media 213. Further, the image data and moving image data downloaded from the server device or the like, or the image data and moving image data read from the medium 213 are supplied to the display unit 209 via the compression / decompression unit 211 to display the images and moving images.

Image Processing Unit A configuration example of the image processing unit 214 is shown in the block diagram of FIG. The image processing unit 214 may have a hardware configuration that includes a CPU and a memory independently, or may have a software configuration that is realized by the CPU 205 executing an image processing program stored in the ROM 203. Alternatively, the image processing unit 214 may be realized by supplying an image processing program to a computer device different from the imaging device.

  The data acquisition unit 303 acquires moving image data output from the DSP 210. The parameter acquisition unit 304 acquires camera parameters of the imaging unit 201 from the CPU 205 or the imaging control unit 206. The camera parameters include imaging condition parameters such as an exposure time, a frame rate, and an aperture, and device parameters representing resolution characteristics of the imaging device such as a lens focal length, a focus position, and a sensor pixel pitch.

  The positional deviation amount acquisition unit 305 acquires, as the amount of camera shake, the positional deviation amount of the image accompanying the camera shake at the time of shooting from the moving image data acquired by the data acquisition unit 303. The motion blur calculation unit 306 calculates the amount of motion blur that occurs on the image due to camera shake based on the camera parameters and the amount of displacement.

  The limit value calculation unit 307 calculates a correction limit value (hereinafter, correction limit value) based on the visual spatial frequency response characteristics related to the motion blur amount and the subject speed. The correction limit value is a positional deviation amount of an image in which motion blur of the subject due to camera shake is not perceived (at least inconspicuous). In other words, if the misregistration amount exceeding the correction limit value is suppressed (the misregistration amount is brought close to zero), motion blur accompanying camera shake becomes conspicuous.

  The correction amount setting unit 308 sets a correction amount to be applied to camera shake correction based on the correction limit value. The correction unit 309 outputs moving image data obtained by performing camera shake correction based on the correction amount on the moving image data acquired by the data acquisition unit 303.

  The camera shake correction processing by the image processing unit 214 will be described with reference to the flowchart of FIG. FIG. 4 shows processing for one frame, but the processing of FIG. 4 is executed for each frame image of the moving image data. In the following, it is assumed that the imaging unit 201 has ideal pinhole camera resolution characteristics, and motion blur due to camera shake depends only on the exposure time T, the frame rate FPS, and a camera shake amount x described later.

  The data acquisition unit 303 acquires the attention frame image of the moving image data (S401). The parameter acquisition unit 304 acquires camera parameters (S402). The misregistration amount acquisition unit 305 calculates a motion vector of the subject from the moving image data using a block matching method, a gradient method, etc., and acquires an image misregistration amount due to camera shake at the time of shooting in the target frame image (S403). . FIG. 5 (a) shows an example of the amount of image misregistration due to camera shake during shooting. In FIG. 5 (a), the vertical axis represents an image positional shift amount x (unit: pixels) due to camera shake, and the horizontal axis represents frame number i.

  Although described in detail later, the motion blur calculation unit 306 calculates the amount of motion blur that occurs on the image due to camera shake from the camera parameters and the positional deviation amount x (S404). Although the details will be described later, the limit value calculation unit 307 corrects the positional deviation amount of an image in which motion blur due to camera shake is not perceived based on the amount of motion blur and visual spatial frequency response characteristics related to the subject speed. Is calculated as (S405).

The correction amount setting unit 308 sets a correction amount based on the correction limit value (S406). That is, according to the following expression, a correction amount for canceling the positional deviation amount x equal to or less than the correction limit value is set. For the positional deviation amount x exceeding the correction limit value, a correction amount equal to the correction limit value is set.
if (x (i) ≦ x _VTF (i))
x '(i) = -x (i);
else
x '(i) = -x _VTF (i);… (1)
Here, x ′ (i) is the correction amount of the frame image i,
x (i) is the amount of displacement of the frame image i,
x _VTF (i) is the correction limit value.

  Based on the set correction amount x ′, the correction unit 309 performs camera shake correction using affine transformation or projective transformation on the frame image of interest, and outputs a frame image after the camera shake correction (S407).

FIG. 5 (b) shows the correction amount x ′ set for the positional deviation amount x in FIG. 5 (a). In FIG. 5 (b), the vertical axis indicates the correction amount x ′ (unit is pixel), and the horizontal axis indicates the frame number i. As shown in FIG. 5 (b), positional deviation of the amplitude x ₀ is canceled by the camera shake correction, the correction object is stationary. On the other hand, for the displacement of amplitude x _{1 with} a large amount of camera shake, motion blur is noticeable due to the stillness of the subject, so motion blur is not perceived (at least not noticeable) amplitude | xx _VTF | .

Motion Blur Calculation Unit The process (S404) of the motion blur calculation unit 306 will be described with reference to the flowchart of FIG. The motion blur calculation unit 306 calculates the difference in the amount of positional deviation between frame images due to camera shake as the camera shake speed using the following equation (S601). However, the camera shake speed v (0) of frame number 0 is 0.
v (i) = x (i)-x (i-1);… (2)
Where v (i) is the camera shake speed,
x (i) is the amount of displacement of the frame image with frame number i,
x (i-1) is the amount of displacement of the frame image with frame number i-1.

FIG. 7 shows an example of the camera shake speed v (i) and the amount of motion blur. In FIG. 7 (a), the vertical axis represents camera shake speed v (i) (unit: pixel / frame), and the horizontal axis represents frame number i. The camera shake speed in the hand shake with the amplitude x ₀ is v = 0.5 · x ₀ , and the hand shake speed in the hand shake with the amplitude x ₁ is v = 0.5 · x ₁ .

Next, the motion blur calculation unit 306 sets a blur filter that reproduces the motion blur that occurs on the target frame image due to camera shake from the camera parameters and the camera shake speed v (i) (S602). First, the blur filter size F (i) is determined by the following equation.
F (i) = T · FPS · v (i);… (3)
Where T is the exposure time,
FPS is the frame rate.

In the following description, it is assumed that the blur filter size at the time of camera shake with amplitude x ₀ is F ₀ pixel, and the blur filter size at the time of camera shake with amplitude x ₁ is F ₁ pixel (F ₀ <F ₁ ).

The relationship between the blur filter and the blur image will be described with reference to FIG. Based on the blur filter size F (i) of the frame image of interest, the motion blur calculation unit 306 is based on the F ₀ × 1 averaging filter (Fig. 8 (b)) when shaking with amplitude x ₀ , and when shaking with amplitude x ₁ The F ₁ × 1 averaging filter (FIG. 8 (c)) is calculated as a gain filter.

Next, the motion blur calculation unit 306 performs a convolution operation of the point light source image (FIG. 8 (a)) that is a reference of the spatial frequency characteristics of the blur and the calculated blur filter, and generates a point light source that is gained by camera shake. An image (for example, FIGS. 8D and 8E) is generated (S603). Fig. 8 (d) shows a blurred image obtained by convolution calculation of a point light source image and a blur filter F ₀ × 1, and Fig. 8 (e) shows a result of convolution calculation of a point light source image and a blur filter F ₁ × 1. The obtained profit image is shown.

  Next, the motion blur calculation unit 306 generates an image obtained by Fourier transforming the blurred image (S604). FIG. 8 (f) shows the Fourier transform image of the blurred image shown in FIG. 8 (d), and FIG. 8 (g) shows the Fourier transform image of the blurred image shown in FIG. 8 (e). Then, the motion blur calculation unit 306 calculates the spatial frequency characteristics of the motion blur that occurs on the image due to hand shake from the Fourier transform image (S605).

  The gain spatial frequency characteristics calculated from the Fourier transform image will be described with reference to FIG. In FIG. 9, the vertical axis indicates the intensity of the Fourier transform image, and the horizontal axis indicates the spatial frequency (cycle / pixel). FIG. 9 (a) shows the spatial frequency characteristics of the Fourier transform image shown in FIG. 8 (f), and FIG. 9 (b) shows the spatial frequency characteristics of the Fourier transform image shown in FIG. 8 (g).

The motion blur calculation unit 306 calculates the motion blur amount fc (f ₀ in the case of FIG. 9 (a) and f _{1 in} the case of FIG. 9 (b) in the calculated spatial frequency characteristic, where the intensity is 0.5. i) is determined (S606). FIG. 7 (b) shows the motion blur amount fc calculated for the camera shake speed v in FIG. 7 (a). In FIG. 7 (b), the vertical axis indicates the motion blur amount fc (unit: cycle / pixel), and the horizontal axis indicates the frame number i.

Limit Value Calculation Unit The process (S405) of the limit value calculation unit 307 will be described with reference to the flowchart of FIG. The limit value calculation unit 307 acquires, from the ROM 203, for example, data indicating the subject speed at which blur is perceived (hereinafter referred to as blur perception speed) with respect to the spatial frequency of the subject (S701). The acquisition in step S701 may be performed once for a series of moving image data processing.

FIG. 11 shows visual spatial frequency response characteristics and blur perception speed according to the moving speed of the subject. In FIG. 11 (a), the vertical axis indicates the intensity of the spatial frequency response, and the horizontal axis indicates the spatial frequency (unit: cycle / pixel). Here, the visual spatial frequency response characteristic shown in Non-Patent Document 1 is used. The visual spatial frequency response is given by
vtf (ρ, v _R ) = k ・ c ₀・ c ₁・ c ₂・ v _R・ (c ₁ 2πρ) ² exp (-c ₁ 4πρ / ρ _max ) (4)
Where k = s ₁ + s ₂ | log (c ₂ v _R / 3) | ³
ρ _max = p ₁ / (c ₂ v _R +2)
ρ is the spatial frequency (cycle / degree) of the subject when watching the video,
v _R is the subject speed (degrees / second)
s ₁ = 6.1, s ₂ = 7.3, p ₁ = 45.9,
c ₀ = 0.6329, c ₁ = 0.8404, c ₂ = 0.7986.

Unit of ρ (cycle / degree) and unit of motion blur amount fc calculated in step S606 (cycle / pixel), unit of v _R (degree / second) and unit of camera shake speed v calculated in step S601 (pixel / frame) ) Resampling to match. The resampled visual spatial frequency response VTF (f, v) follows the following equation.
VTF (f, v) = vtf [{Rπ / (Nx ・ p ・ 180)} f, {(Nx ・ p ・ 180) / Rπ} fps ・ v]… (5)
Where p is the pixel pitch (mm) of the display for viewing the video,
R is the viewing distance (mm),
Nx is the number of pixels in the x direction of the video,
fps is the viewing frame rate (frames / second),
The values of p, R, Nx, and fps are determined based on a general user viewing environment.

The two curves shown in Fig. 11 (a) show the visual spatial frequency response characteristics VTF (f, v) when the subject speed is V ₀ and V ₁ , and the responses are at spatial frequencies f ₀ and f ₁ respectively. Strength becomes 0.5. If the subject velocity v that satisfies VTF = 0.5 when the subject's spatial frequency f is changed is calculated with respect to VTF (f, v) in Equation (5), the subject's spatial frequency f and the perception of profit The speed relationship can be obtained. FIG. 11 (b) shows the relationship between the spatial frequency f of the subject and the perceived speed of blur. In FIG. 11 (b), the vertical axis represents the blur perception speed (unit: pixels / frame), and the horizontal axis represents the spatial frequency f of the subject (unit: cycles / pixel).

Next, the limit value calculation unit 307 acquires the speed v _VTF for perceiving motion blur associated with camera shake by referring to the data of the blur perception speed (S702). That is, referring to the curve of VTF = 0.5 shown in FIG. 11 (b), the blur perception speed corresponding to the motion blur amount fc calculated in step S606 is expressed as the motion blur perception speed | v _VTF | get.

Next, the limit value calculation unit 307 sets (adjusts) the motion blur perception speed v _VTF (i) of the frame image i by the following equation (S703).
if (| v _VTF (i) | ＞ Vth)
v _VTF (i) = {v (i) / | v (i) |} | v _VTF (i) |;
else
v _VTF (i) = 0;… (6)
Here, Vth is a predetermined threshold value.

In other words, in the case of small camera shake below the threshold (| v _VTF (i) | ≦ Vth), it is assumed that the motion blur accompanying the camera shake is sufficiently small. Adjust _VTF (i) = 0. Note that the motion blur perception speed at the time of camera shake with amplitude x ₀ is Vm ₀ (unit: pixels / frame), and the motion blur perception speed at the time of camera shake with amplitude x ₁ is Vm ₁ .

Next, the limit value calculation unit 307 calculates a correction limit value x _VTF (i) in the frame image i by the following equation (S704). That is, the sum of the correction limit value x _VTF (i-1) in the previous frame and the motion blur perception speed v _VTF (i-1) of the previous frame is set as the limit value x _VTF (i) of the frame of interest i.
x _VTF (i) = x _VTF (i-1) + v _VTF (i-1)… (7)
However, the correction limit value x _VTF (0) = 0 for frame number 0.

FIG. 12 shows the relationship between the motion blur perception speed v _VTF and the correction limit value x _VTF . Fig. 12 (a) is the motion blur perception speed v _VTF calculated according to equation (6), and the vertical axis is the motion blur perception speed v _VTF (unit: pixel / frame, equivalent to the amount of displacement between frame images) The horizontal axis indicates the frame number i. At the time of camera shake of amplitude x ₀ , even if the subject is stationary, motion blur is not perceived, so the speed Vm ₀ = 0. Further, at the time of hand shake of the amplitude x _1, the speed Vm ₁ > 0 at which no motion blur is perceived.

FIG. 12B shows the correction limit value x _VTF calculated according to the equation (7), the vertical axis indicates the correction limit value x _VTF (unit is pixel), and the horizontal axis indicates the frame number i. At the time of camera shake of amplitude x ₀ , even if the subject is stationary, motion blur is not perceived, so the correction limit value is Lx ₀ = 0. In addition, the correction limit value Lx ₁ > 0 at which motion blur is not perceived at the time of camera shake of the amplitude x ₁ is satisfied.

  By applying the above-described camera shake correction processing by the image processing unit 214 to the moving image data photographed by the imaging device, it is possible to observe an image in which motion blur due to camera shake is not noticeable. FIG. 13 shows a positional deviation amount in the moving image data after the camera shake correction process of the embodiment. In FIG. 13, the vertical axis indicates the amount of positional deviation (unit is pixel) of the image after camera shake correction, and the horizontal axis indicates the frame number i.

  The positional deviation amount before the camera shake correction indicated by the dotted line in FIG. 13 is suppressed to the positional deviation amount after the camera shake correction indicated by the solid line. In the embodiment, the positional deviation amount of the moving image data after the camera shake correction process is suppressed to a positional deviation amount (correction limit value) in which motion blur due to camera shake is not perceived (at least inconspicuous).

FIG. 14 shows a partial frame image of a moving image after camera shake correction. FIG. 14 shows a frame image obtained by performing the camera shake correction of the embodiment on the frame image shown in FIG. At the time of camera shake of the amplitude x ₀ shown in FIGS. 14 (a) and 14 (c), the correction limit value Lx ₀ = 0 is set and moving image data with the subject stationary is generated. On the other hand, at the time of hand movement of the amplitude x ₁ shown in FIGS. 14 (b) and 14 (d), the correction limit value Lx ₁ is left as the correction limit value Lx ₁ , and the displacement is suppressed to the amplitude x ₁ −x ₁ ′ = Lx _{1 so} that the motion blur is not noticeable. Generate video data.

  As described above, by controlling the intensity of camera shake correction based on the spatial frequency response characteristics of human vision related to the subject speed, it is possible to perform a camera shake correction process in which motion blur due to camera shake is not noticeable. Accordingly, it is possible to perform a camera shake correction process in which the motion blur on the subject is not noticeable even when a large camera shake occurs without setting the correction strength by the user.

[Modification]
In the above description, the example in which the positional deviation amount x of the image corresponding to the amount of camera shake is acquired on the assumption that the user holding the imaging device captures a stationary rectangular wave chart. The amount of camera shake can be acquired even when the user or the subject is moving and shooting is performed while tracking the subject. In such a case, the trajectory in the three-dimensional space of the imaging device at the time of shooting is calculated using Structure From Motion (SFM) or the like. Then, the trajectory obtained by low-pass filtering the trajectory of the imaging apparatus is acquired as camera work, and the difference between the trajectory of the imaging apparatus and camera work is acquired as the amount of camera shake. Note that SFM is a technique for simultaneously restoring the three-dimensional shape of a scene and the three-dimensional position of the imaging apparatus from a plurality of images obtained by changing a viewpoint of the imaging apparatus, for example.

  In the above description, an example in which the positional deviation amount x indicating the amount of camera shake is acquired from the moving image data using an alignment algorithm such as the block matching method or the gradient method has been described. However, the amount of camera shake using an external device such as a gyro sensor is described. May be obtained.

  In the above description, an example has been described in which a correction amount to be applied to camera shake correction is calculated using a position shift amount of an image in which motion blur caused by camera shake is not perceived (at least inconspicuous) as a correction limit value. Furthermore, the correction amount may be calculated by weighting processing that adds conditions such as maintaining the angle of view for the corrected moving image data. Further, the amount of correction may be set using a mode dial or the like.

  In the above description, an example of calculating the visual spatial frequency response characteristic based on the equation (4) has been described. However, the present invention is not limited to the spatial frequency response characteristic of a specific model. For example, other visual spatial frequency response characteristics and actual measurement data shown in Non-Patent Document 2 may be used.

  In general, it is known that the visual spatial frequency response characteristic has a bandpass characteristic. For example, when observing a stationary subject, the visual spatial frequency response characteristic is a bandpass characteristic having a peak in the vicinity of 4-5 cycles / degree. On the other hand, when the line of sight follows the subject, it is known that the sensitivity on the high frequency side decreases and the sensitivity on the low frequency side increases. For this reason, the visual spatial frequency response characteristic in the follow-up vision applied to the present invention may be a bandpass characteristic in which the response peak shifts to the low frequency side as the movement of the subject increases.

  In the above description, an example is described in which an averaging filter having a size proportional to the camera shake speed is used as the blur filter. However, the weighting coefficients in the filter do not have to be the same value, and arbitrary weighting coefficients are used. Good. In addition, although the case where the direction of camera shake is only the x direction has been described, a two-dimensional blur filter may be used when it corresponds to camera shake of two-dimensional motion.

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program It can also be realized by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  305 ... Position displacement amount acquisition unit, 306 ... Motion blur calculation unit, 307 ... Limit value calculation unit, 308 ... Correction amount setting unit

Claims (12)

Acquisition means for acquiring the amount of camera shake when shooting video data;
First calculation means for calculating the amount of motion blur of the subject included in the moving image data caused by camera shake from the camera parameters of the image capturing unit that has captured the camera shake amount and the moving image data;
Second calculation means for calculating a positional deviation amount of an image in which motion blur is not perceived as a correction limit value based on a visual spatial frequency response characteristic related to a subject speed and the motion blur amount;
An image processing apparatus comprising: setting means for setting a correction amount to be applied to camera shake correction of the moving image data based on the correction limit value.   2. The image processing apparatus according to claim 1, further comprising a correction unit that performs camera shake correction based on the correction amount on the moving image data.   3. The image processing according to claim 1, wherein the acquisition unit acquires, as the camera shake amount, a positional deviation amount of an image indicated by the motion vector of the subject calculated from the moving image data. apparatus.   3. The image processing apparatus according to claim 1, wherein the acquisition unit acquires the camera shake amount based on a three-dimensional position of the imaging unit. The first calculation means includes
Calculate the difference in the amount of camera shake between the frame images as the speed of the camera shake,
Set a blur filter from the camera parameters and the camera shake speed,
Generate a blurred image of a point light source image by the blur filter,
Calculate the spatial frequency characteristics of the motion blur from the Fourier transform image of the blur image,
5. The image processing apparatus according to claim 1, wherein the motion blur amount is determined based on the spatial frequency characteristics. The second calculating means includes
The blur perception speed data corresponding to the amount of motion blur is obtained as the motion blur perception speed with reference to the data of the blur perception speed obtained from the visual spatial frequency response characteristics,
Adjusting the motion blur perception speed based on a predetermined threshold;
6. The image processing apparatus according to claim 1, wherein a sum of the correction limit value of the previous frame and the motion blur perception speed is calculated as the correction limit value of the frame of interest.   7. The image processing apparatus according to claim 6, wherein when the motion blur perception speed is equal to or less than the threshold value, the motion blur perception speed is adjusted to zero.   8. The image processing according to claim 1, wherein the visual spatial frequency response characteristic has a bandpass characteristic in which a response peak shifts to a low frequency side as the subject speed increases. apparatus.   The setting means sets a correction amount for canceling the camera shake amount for the camera shake amount equal to or less than the correction limit value, and sets a correction amount equal to the correction limit value for the camera shake amount exceeding the correction limit value. 9. The image processing device according to claim 1, wherein the image processing device is an image processing device. An imaging unit for capturing video data;
10. An imaging apparatus comprising: the image processing apparatus according to claim 1. Get the amount of camera shake when shooting video data,
From the camera parameters of the imaging unit that captured the camera shake amount and the moving image data, calculate the amount of motion blur of the subject included in the moving image data due to camera shake,
Based on the visual spatial frequency response characteristics related to the subject speed and the amount of motion blur, the amount of shift of the image where motion blur is not perceived is calculated as a correction limit value,
An image processing method for setting a correction amount to be applied to shake correction of the moving image data based on the correction limit value.   11. A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.