JP2007104516A - Image processor, image processing method, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To separate motion vectors of a picked-up image into motion vectors caused by movement of an apparatus and motion vectors caused by its rotation. <P>SOLUTION: A motion vector detection section detects motion vectors of (t) and (t+1) vectors, and a camera movement information calculation section detects motion vectors of (t) and (t-1) frames on the basis of image data of the (t-1) frame for which the rotating motion of a camera is corrected by processing of a camera rotating motion correction section. A camera motion separating section separates motion vectors into motion vectors caused by movement of an apparatus and motion vectors caused by rotation. The present invention can be applied to an imaging apparatus. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing device, an image processing method, a program, and a recording medium, and more particularly to an image processing device and an image processing that are suitable for use in correcting blurring of a captured moving image. The present invention relates to a method, a program, and a recording medium.

  Conventionally, many devices for correcting blurring have been proposed in a silver salt camera and a video camera. Conventional blur correction techniques detect camera vibration (camera shake) using an angular velocity sensor, etc., and drive the photographic optical system based on the detected blur information to prevent the subject image from blurring. It is made to prevent.

  In video cameras, instead of driving the photographic optical system, a method of shifting the image reading range of the imaging sensor in accordance with the detected blur direction and speed has been proposed.

  On the other hand, there is a technique for distinguishing movements between camera shake and pan / tilt and correcting a blurred image by image processing at the stage of processing an image obtained by imaging (for example, Patent Document 1).

Japanese Patent No. 3218612

  For example, in the case of detecting camera vibration using an angular velocity sensor or the like, there is a possibility that camera work such as panning and tilting may be mistakenly detected as a movement of a camera shake, driving a photographing optical system, When correction in the image capturing stage that shifts the reading range of the image sensor is used, there is a possibility that an image that should originally be obtained cannot be obtained due to erroneous detection. On the other hand, a technique for correcting an image that has been blurred using image processing can display or output the image while leaving an uncorrected image even if a blur is erroneously detected. In addition, when processing in real time, there is a great advantage compared to correction at the image capturing stage in that a large-capacity memory is not required.

  In a technique for correcting a blurred image by image processing represented by the technique described in Patent Document 1 described above, it is important to detect a camera motion for correcting camera shake from an image.

  As shown in FIG. 1, the camera motion is originally composed of rotation around the X axis, Y axis, and Z axis, and movement in the X axis, Y axis, and Z axis directions. The apparent change of is different, and the detected motion vector is also different. For example, when there is rotation around the X axis and the Y axis (so-called pan / tilt), as shown in FIGS. 2 and 3, for example, the vertical direction, the horizontal direction, or not shown, As shown in FIG. 4, a motion vector indicating a uniform motion, such as an oblique motion in which these are combined, is detected on the entire screen and rotation around the Z axis (so-called rotation around the optical axis) is performed. A motion vector corresponding to the distance from the image center is detected.

  On the other hand, as shown in FIG. 5, when there is a movement in the X-axis direction at the camera position, the magnitude of the movement of each pixel due to the movement of the camera is not uniform and depends on the distance between the camera and the subject. It is inversely proportional and proportional to the actual camera movement. That is, the movement of each pixel is larger when the camera moves more than when it is small, but in either case, the movement of the subject close to the camera is larger than the movement of the subject far from the camera. . The same applies to camera movement in the Y-axis direction and the Z-axis direction.

  In this way, various movements occur in the image by changing the position and angle of the camera. However, in the conventional technique for correcting a blurred image by image processing, represented by the technique described in Patent Document 1 described above, changes in the image are caused by various camera movements. The movement of the camera is not taken into consideration, and the amount of translation in the image is estimated as pan and tilt, and camera shake correction has been performed.

  Therefore, in the conventional blur correction, it is judged that the camera is moving (false detection) even though the camera is actually moving, and reverse correction is applied. There is a risk of correcting itself by mistake.

  Furthermore, even if the camera motion can be estimated correctly and reverse correction is applied to cancel the camera motion, it is difficult to present the video originally intended by the photographer unless the viewpoint shift due to camera movement is properly taken into account. is there.

  The present invention has been made in view of such a situation, and makes it possible to correct an image in consideration of the movement of a camera.

  An image processing apparatus according to one aspect of the present invention is an image processing apparatus that detects a motion of an imaging unit that captures a moving image based on a plurality of frames in the moving image, and acquires the moving image. A motion vector detecting means for detecting a motion vector at an arbitrary interval in the moving image acquired by the acquiring means, and the motion vector detected by the motion vector detecting means by moving the imaging means. Separating means for separating a first motion vector and a second motion vector by rotation of the imaging means, the separating means statistically separating the first motion vector and the second motion vector; To calculate.

  A movement information detecting unit that detects movement of the imaging unit to obtain movement information of the imaging unit can be further provided, and the separation unit captures a sufficiently distant subject in the frame in the moving image. A distant area estimating means for estimating a region obtained, a removing means for removing a third motion vector due to rotation around the optical axis of the imaging means from the motion vector detected by the motion vector detecting means, and the removing Based on the motion vector from which the third motion vector has been removed by the means, the movement relationship estimation means for estimating the relationship between the movement information detected by the movement information detection means and the actual movement of the imaging means And a rotation estimating means for estimating the rotation of the imaging means around the optical axis, and a movement estimating means for estimating the movement of the imaging means.

  The frame may further include a rotational motion correcting unit that corrects the rotational movement of the imaging unit using affine transformation and converts it into a past frame. The movement information detecting unit includes the rotation The movement information can be acquired by detecting a motion vector between the past frame converted by the motion correction means and the frame, and inverting the detected motion vector.

  The removal means calculates, for each pixel of the frame, a point-symmetric position with the center point of the frame as a rotation center point, and adds the motion vectors at the point-symmetric positions to thereby add the optical axis. The rotation component around can be removed.

  The far region estimation means calculates at least one of a norm, an angle, or a rate of change with a neighboring sample point of the motion vector, and statistically processes the calculated value, so that it is sufficiently far away. It is possible to estimate the area where the subject is imaged.

  The far region estimation means can be made to estimate a region where a sufficiently far subject is imaged by obtaining a position where the norm value of the motion vector is minimum.

  The far region estimation means has a difference between a minimum value of the norm of the motion vector and a norm of the motion vector at each position to a value equal to or less than a change rate at a point where the minimum value of the norm of the motion vector is taken. Can be estimated as a region where a sufficiently distant subject is imaged.

  The distant area estimation means is configured to estimate an area where a sufficiently distant subject is imaged by using a value obtained by normalizing the norm value of the motion vector in all pixels in the frame as a weight. can do.

  The movement relation estimation unit estimates a speed ratio and a pan / tilt component of the rotational movement of the imaging unit in a region where a sufficiently far subject in the frame estimated by the far region estimation unit is imaged. Can be.

  The rotation estimation unit removes the movement of the imaging unit estimated by the movement relationship estimation unit and the pan / tilt component of the rotational movement of the imaging unit from the motion vector detected by the motion vector detection unit. Thereby, it is possible to estimate the rotation of the imaging means around the optical axis.

  Further, a correction signal generation unit that generates a correction signal for correcting the image of the frame based on the first motion vector and the second motion vector obtained by the separation unit is further provided. be able to.

  Image correction means for correcting the image data of the frame can be further provided based on the correction signal generated by the correction signal generation means.

  Recording means for recording the correction signal generated by the correction signal generation means can be further provided.

  A recording unit that records the corrected image data generated by the correction by the image correction unit may be further provided.

  Display control means for controlling display of corrected image data generated by correction by the image correction means can be further provided.

  The image processing method and program according to the first aspect of the present invention obtains the moving image, detects a motion vector at an arbitrary interval in the moving image, and statistically calculates the motion vector. Separating the first motion vector due to movement of the imaging means and the second motion vector due to rotation of the imaging means.

  In one aspect of the present invention, the inside of a moving image is acquired, a motion vector at an arbitrary interval in the moving image is detected, and the motion vector is obtained by a statistical calculation as a first motion vector due to movement of the imaging unit. And a second motion vector due to rotation of the imaging means.

  Note that the image processing apparatus may be an independent apparatus or a block that performs image processing of the imaging apparatus. The recording device may be an independent device or a block that performs recording processing of the imaging device. The display device may also be an independent device or a block that performs display processing of the imaging device.

  As described above, according to one aspect of the present invention, it is possible to detect a motion vector of an image. In particular, the motion vector includes a first motion vector due to movement of the imaging means and a second motion vector due to rotation of the imaging means. It is possible to correct the image with high accuracy by separating the motion vector.

  Embodiments of the present invention will be described below. Correspondences between constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  An image processing apparatus according to one aspect of the present invention is an image processing apparatus that detects a motion of an imaging unit that captures a moving image based on a plurality of frames in the moving image, and acquires the moving image. (For example, # 1 frame memory 81 in FIG. 7) and motion vector detecting means for detecting a motion vector at any interval in the moving image acquired by the acquiring means (for example, a motion vector detecting unit in FIG. 7) 83) and separation means (for example, separating the motion vector detected by the motion vector detection means into a first motion vector due to movement of the imaging means and a second motion vector due to rotation of the imaging means) , And the separation means statistically calculates the first motion vector and the second motion vector.

  A movement information detection unit (for example, a camera movement information calculation unit 88 in FIG. 7) that obtains movement information of the imaging unit by detecting movement of the imaging unit may be further provided, and the separation unit may include the image moving image. From the motion vector detected by the motion vector detection means, the distance region estimation means (for example, the distance area estimation means 111 in FIG. 12) that estimates the area where a sufficiently distant subject in the frame is imaged, Removal means for removing the third motion vector due to rotation around the optical axis of the imaging means (for example, the rotation component removal processing unit 112 around the optical axis in FIG. 12), and removal of the third motion vector by the removal means. Based on the obtained motion vector, the movement relation estimation for estimating the relation between the movement information detected by the movement information detection means and the actual movement of the imaging means 12 (for example, the pan / tilt component and translational separation processing unit 113 in FIG. 12) and rotation estimation means for estimating the rotation of the imaging means around the optical axis (for example, the extraction process of the rotational component around the optical axis in FIG. 12). Unit 116) and movement estimation means for estimating the movement of the imaging means (for example, the separation processing unit 117 for linear motion and translational motion in FIG. 12).

  Rotational motion correction means (for example, the camera rotational motion correction unit 86 in FIG. 7) that corrects the rotational movement of the imaging unit using affine transformation and converts the frame into a past frame. The movement information detection unit can detect the motion vector between the past frame converted by the rotational motion correction unit and the frame, and obtain the movement information by inverting the detected motion vector. be able to.

  Correction signal generation means for generating a correction signal for correcting the image of the frame based on the first motion vector and the second motion vector obtained by the separation means (for example, the correction in FIG. 7). An information generator 89) can be further provided.

  Image correction means for correcting the image data of the frame based on the correction signal generated by the correction signal generation means (for example, a correction image generation unit in FIG. 7) can be further provided.

  The image forming apparatus may further include recording means (for example, the image data recording unit 62 in FIG. 6) for recording the correction signal generated by the correction signal generating means.

  A recording unit (for example, the image data recording unit 62 in FIG. 6) for recording the corrected image data generated by the correction by the image correcting unit may be further provided.

  A display control unit (for example, the display control unit 64 in FIG. 6) for controlling the display of the corrected image data generated by the correction by the image correction unit may be further provided.

  The image processing method and program according to one aspect of the present invention acquires the moving image (for example, when it is determined Yes in step S1 in FIG. 16), and obtains a motion vector at an arbitrary interval in the moving image. By detecting (for example, the process of step S2 in FIG. 16) and calculating statistically, the motion vector is converted into the first motion vector by the movement of the imaging unit and the second motion vector by the rotation of the imaging unit. (For example, the process of step S3 in FIG. 16).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 6 is a block diagram showing a configuration of the imaging apparatus 51 to which the present invention is applied.

  The imaging device 51 is provided with an imaging processing unit 61, an image data recording unit 62, an image processing unit 63, a display control unit 64, and a display unit 65 as units for processing images. An operation input unit (not shown) that receives the operation input and an overall control unit (not shown) that controls the entire imaging device 51 based on the user operation input are provided.

  The imaging processing unit 61 includes, for example, a solid-state imaging device such as a lens, CMOS, or CCD (Charge-Coupled Device). The imaging processing unit 61 images a subject and supplies the obtained captured image data to the image data storage unit 62. To do.

  The image data storage unit 62 is configured by an internal or removable recording medium such as a hard disk drive (HDD), a semiconductor memory, an optical disk, a magneto-optical disk, or a magnetic disk, and an imaging processing unit. The pre-correction captured image data supplied from 61 is stored and supplied to the image processing unit 63, and the corrected image data obtained by correction in the image processing unit 63 is recorded. The image data storage unit 63 supplies the recorded captured image data or the corrected image data to the display control unit 64 based on the control of the overall control unit (not shown).

  The image processing unit 63 receives supply of captured image data recorded in the image data storage unit 62 and executes processing for correcting this.

  In the conventional image blur correction processing, pan / tilt (rotation of the camera shown in FIG. 1 around the X axis and rotation around the Y axis) and rotation around the optical axis (camera Z shown in FIG. 1) are performed. The camera shake correction for detecting and correcting the camera motion is performed only for the rotational motion of an imaging device such as a camera that captures an image, such as rotation around an axis. However, with such a conventional method, for example, when an imaging device such as a camera that captures an image such as a video from a car window is moving or when a large object enters the imaging range of the camera, etc. However, since these movements are not assumed, there is a possibility that it is erroneously detected as blurring.

  Therefore, the image processing unit 63 formulates and separates the change in the positional relationship of the subject in the screen that occurs when the imaging device 51 moves and the movement in the screen that occurs when the imaging device 51 rotates, thereby separating the images. The image is corrected, and the corrected image data is supplied to the display control unit 64 or the image data recording unit 62. Details of the processing of the image processing unit 63 will be described later.

  The display control unit 64 controls display on the display unit 65 of an image corresponding to captured image data or corrected image data supplied from the image processing unit 63 or the image data recording unit 62.

  The display unit 65 is configured by, for example, an LCD (Liquid Crystal Display) or the like, and displays an image based on the control of the display control unit 64.

  Needless to say, the display unit 65 may be provided as an independent display device outside the imaging device 51. Furthermore, it goes without saying that the imaging processing unit 61 and the image processing unit 63 may be provided in different apparatuses.

  FIG. 7 is a block diagram showing a more detailed configuration of the image processing unit 63.

  The image processing unit 63 includes a # 1 frame memory 81, a # 2 frame memory 82, a motion vector detection unit 83, a camera motion separation unit 84, a frame delay 85, a camera rotational motion correction unit 86, a # 3 frame memory 87, camera movement information. The calculation unit 88, the correction information generation unit 89, and the correction image generation unit 90 are configured.

  The # 1 frame memory 81 acquires captured image data from the image data recording unit 62 and supplies it to the # 2 frame memory 82 and the motion vector detection unit 83.

  The # 2 frame memory 82 holds one frame of the captured image data supplied from the # 1 frame memory 81.

  Here, the description will be made assuming that the frame image data (t frame) is captured at the timing t with reference to the frame held in the # 2 frame memory 82. That is, the frame stored in the # 1 frame memory 81 is the next frame (t + 1 frame) after the frame held in the # 2 frame memory 82.

  The motion vector detection unit 83 detects a motion vector between frames based on the t frame of the t + 1 frame held in the # 1 frame memory 81 and the t frame held in the # 2 frame memory 82. . The motion vector detection unit 83 acquires a motion vector by using, for example, a block matching method described in Japanese Patent Application Laid-Open No. 2004-134992. In addition, as a method of detecting a motion vector from a moving image sequence in the motion vector detection unit 83, if a motion vector can be detected for each pixel or every certain interval, for example, general block matching, gradient method, and other motion Any of the vector detection methods may be used.

  The camera motion separation unit 84 is based on the motion vector of the t frame and the t + 1 frame detected by the motion vector detection unit 83 and the camera movement information of the past frame calculated by the camera movement information calculation unit 88 described later. The camera rotation information and the camera movement information are calculated, the camera rotation information and the camera movement information are supplied to the correction information generation unit 89, and the camera rotation information is corrected via the frame delay 85. To the unit 86. Details of the camera motion separation unit 84 will be described later.

  The frame delay 85 delays the camera rotation information supplied from the camera motion separation unit 84 by a processing time of one frame and supplies the delayed information to the camera rotation motion correction unit 86. That is, when the frame held in the # 2 frame memory 82 is the t frame, the camera rotation information supplied from the frame delay 85 to the camera rotation motion correction unit 86 is the camera rotation information in the t−1 frame. is there.

  The camera rotation motion correction unit 86 acquires frame image data of t frames from the # 2 frame memory 82, and, for example, based on the camera rotation information in the t-1 frame supplied from the frame delay 85, for example, an affine described later. The rotational movement of the camera is corrected using conversion or the like, and the frame image data of the corrected t−1 frame is supplied to the # 3 frame memory 87.

  The # 3 frame memory 87 holds the frame image data of the t−1 frame whose rotational motion is corrected.

  The camera movement information calculation unit 88 acquires frame image data of t frame from the # 2 frame memory 82 and also acquires frame image data of t−1 frame whose rotational motion is corrected from the # 3 frame memory 87. Thus, a motion vector from the current frame to the past frame is obtained, camera movement information is calculated, and supplied to the camera motion separation unit 84. The motion vector obtained here is a motion vector generated by the movement of the imaging device 51 except for the occlusion portion since the rotation component of the imaging device 51 is completely removed.

  The correction information generation unit 89 generates correction information for correcting blurring of frame image data of t frames based on the camera rotation information and camera movement information supplied from the camera motion separation unit 84, and a correction image generation unit 90.

  The corrected image generation unit 90 acquires the frame image data of t frame from the # 2 frame memory 82, corrects the frame image data of t frame based on the correction information supplied from the correction information generation unit 89, and The data is output to the image data recording unit 62 or the display control unit 64.

  The correction information generated by the correction information generation unit 89 may be supplied to the image data recording unit 62 alone and recorded. Since the captured image data before correction is also stored in the image data recording unit 62, for example, correction processing may be performed in a different apparatus based on the correction information and the captured image data before correction.

  Next, the principle of correction in the image processing unit 63 described with reference to FIG. 7 will be described.

  The movement of the camera (the imaging device 51 or the imaging processing unit 61 thereof) generally includes rotational movement of the X axis, Y axis, and Z axis, and camera movement in the X axis direction, Y axis direction, and Z axis direction. There are six degrees of freedom movement with movement. Here, the movement of the camera in the Z-axis direction (optical axis direction) in FIG. 1 is referred to as a straight-ahead movement, and the movement of the camera in the X-axis direction is proved as a translational movement.

  For example, here, as shown in FIG. 8, a camera movement model including a movement component in each direction of the camera is rotated (rotated around the optical axis and pan / tilt) and then moved (straightly). Motion and translational motion).

  In this case, a motion vector mv (t) generated in the captured image (because it is a motion vector, originally, an arrow indicating the vector should be described above mv, but it is omitted here. , Shown in the formula below) is R (x), a function that defines where a point moves with camera rotation, and T (x), a function that defines where a point moves with camera movement, It is obtained by the equation (1).

・・・・（１）

・ ・ ・ ・ (1)

  When there are only three camera movements, pan / tilt and rotation around the optical axis, the viewpoint of the camera does not move, so the distance relationship between the camera and the object (subject) does not change at all. Such camera movement can be generally defined by affine transformation. This is also described in, for example, [A Computational Vision Approach to Image Registration, IEEE TRANSACTIONS ON IMAGE PROCESSING, Vol2, No3, JULY 1993].

  That is, when considering the rotational movement of the camera without considering the zoom, the movement due to the rotation of the pan / tilt in the captured image frame is (P (t), Q (t)), and the rotation around the optical axis is θ (t ), The coordinates (x, y) in the frame when the center of the image frame is the origin (0, 0) move according to the following equation (2).

・・・（２）

... (2)

  On the other hand, when the position of the camera moves, the amount of movement of each pixel between frames obtained by imaging varies depending on the distance between the subject and the camera.

  That is, the movement of the camera is represented by a vector V (because it is a motion vector, originally, an arrow indicating the vector should be described above V, but it is omitted here). When expressed as (Vx (t), Vy (t), Vz (t)), it is possible to model how the subject moves on the frame with respect to the camera movement.

  Here, the movement of the camera position is decomposed in two directions, and after moving straight in the direction perpendicular to the imaging surface of the camera based on the model described with reference to FIG. Think of it as a process.

  First, FIG. 9 shows a model of the movement of each pixel when the camera moves only in a straight direction moving in the direction perpendicular to the imaging surface, that is, when the vector V has only the Vz component.

  That is, the camera moves a distance Vz from the position indicated as Camera Position (t-1) in FIG. 9 to the position indicated as Camera Position (t), and from the position of the camera to the image plane (image plane). The distance is kw, and the vertical distance between the far object and Camera Position (t−1) is da.

  From the model shown in FIG. 9, the movement of each point in the frame is expressed by equation (3).

・・・（３）

... (3)

  Next, FIG. 10 shows a model of the movement of each pixel when the camera moves in the direction of translation of the imaging surface, that is, when the vector V has only the Vx component.

  In the model shown in FIG. 10, the camera position moves a distance Vx from the position indicated by (t-1) to the position indicated by (t), and from the camera position to the image plane (image plane). The distance is kw, and the distance in the vertical direction between the far object and the camera is da−Vz (that is, the movement is in the translation direction after the vertical movement described with reference to FIG. 9).

  At this time, as shown in FIG. 11, a Vam vector indicating the amount of movement of the subject a in the frame (because it is a motion vector, an arrow indicating the vector should be described above Vam originally. However, it is omitted here and can be obtained in the formulas described later), and based on this, the movement of each pixel point with the center of the imaging surface as the origin can be obtained by Equation (4). .

・・・（４）

... (4)

  That is, the movement in the three-dimensional space due to camera movement is expressed by the following equation (5) based on the equations (3) and (4).

・・・（５）

... (5)

  That is, each pixel in the case where the movement of the camera with 6 degrees of freedom is considered in the model shown in FIG. 8 (when the movement of the camera with 6 degrees of freedom is moved in the order shown in FIG. 8). Can be expressed as the following equation (6) by substituting equation (2) described above into equation (5).

・・・（６）

... (6)

  Thus, since the movement of each point in the captured image according to FIG. 8 is obtained from the equation (6), x (t−1) to x (t) and y (t−1) ) To y (t) are expressed by the following equation (7).

・・・（７）

... (7)

  The operation of each unit described with reference to FIG. 7 will be described using the contents described above.

  The camera rotation motion correction unit 86 corrects the camera rotation motion based on the affine transformation of Expression (2). That is, the camera rotational motion correction unit 86 is based on the camera rotational motion information (P (t), Q (t), θ (t)) supplied from the camera motion separation unit 84 via the frame delay 85. The image acquired from the # 2 frame memory 82 is converted based on Expression (2).

  Then, the camera movement information calculation unit 88 uses the past frame (t-1 frame) image obtained by the affine transformation and the current frame (t frame) image to obtain the current frame and the past frame. A motion vector is obtained and the direction of the motion vector is reversed.

  Since the motion vector obtained here completely removes the rotation component of the camera, it is a motion vector generated by moving the camera except for the occlusion. The motion vector dv of this camera movement information (because it is a motion vector, originally, an arrow indicating the vector should be described above dv, but it is omitted here and is shown in a mathematical expression described later) Based on the equation (5), it can be expressed by the following equation (8).

・・・（８）

... (8)

  The motion vector represented by Expression (8) can be expressed by Expression (9) particularly when there is a sufficiently far region or when there is almost no linear component, that is, when da >> Vz.

・・・（９）

... (9)

  From this equation (9), it can be said that, in a sufficiently far region, the camera movement information of Vx, Vy is proportional to dvx (t), dvy (t) and the object distance.

  The camera motion separation unit 84 is obtained between frames using the camera movement information of the previous frame (t−1 frame) and the current frame (t frame) calculated by the camera movement information calculation unit 88. The movement due to camera movement and the movement due to camera rotation are separated from the obtained motion vector. That is, the camera motion separation unit 84 solves the above-described equation (7).

  However, since the equation (7) described above includes unknowns P (t), Q (t), θ (t), da, Vx, Vy, and Vz, it is not easy to solve this.

  Therefore, the camera motion separation unit 84 statistically obtains a parameter from an image including various unknowns.

  FIG. 12 is a functional block diagram for explaining the functions of the camera motion separation unit 84. With reference to FIG. 12, an example of a method in which the camera motion separation unit 84 statistically obtains parameters from an image including various unknowns will be described.

  The camera motion separation unit 84 includes a far region detection unit 111, a rotation component removal processing unit 112 around the optical axis, a pan / tilt component / translational motion separation processing unit 113, a pan / tilt component acquisition unit 114, a translational motion vector acquisition unit 115, It has the functions of an extraction processing unit 116 for rotational components around the optical axis and a separation processing unit 117 for linear motion and translational motion.

  The far region detection unit 111 detects a far region in the frame based on the motion vectors of the t frame and the t + 1 frame supplied from the motion vector detection unit 82 as preprocessing for performing statistical processing.

  The far region detection unit 111 calculates a norm, an angle, and a change rate of the motion vector in order to detect the far region.

  First, the motion vector norm vnorm (x, y) is calculated by the following equation (10) using the x and y components of the motion vector for each pixel.

・・・（１０）

... (10)

  Next, the angle θ of the motion vector is calculated by the following equation (11) using the x and y components of the motion vector for each pixel.

・・・（１１）

(11)

  The motion vector change rate AveGrad is obtained by calculating an average value Grad of vector norm differences from eight adjacent pixels (spacing = m) (m and n are integers) in an n × n region by the equation (12). Using the average value Grad, calculation is made according to the equation (13).

・・・（１２）

(12)

・・・（１３）

... (13)

  In Expression (12), (i, j are integers), and in Expression (13), (k, l are integers).

  Since the detected motion vector includes an error and the value may be violated, in order to obtain a stable result, for example, the value of the motion vector for each pixel may be an average value or a median for each region. The process of replacing with a value may be performed in advance.

  Next, two examples of a method in which the far region detection unit 111 extracts a sufficiently far region in the frame by statistical processing of the motion vector detection result are shown.

  First, the first example will be described.

  FIG. 13 is an example of the distribution of the vector norm in the frame when the vector norm is actually calculated. The far region detection unit 111 uses the property that the far region has a smaller norm of the motion vector than the near region, and can estimate the position indicating the minimum value of the motion vector norm as the farthest point. Then, after the farthest point is estimated, the far region detection unit 111 searches the image again, and the difference between the vector norm of each pixel and the vector norm of the farthest point is a change in the norm of the vector of the farthest point. A position that falls below the rate is estimated as a sufficiently distant area.

  Next, the second method will be described.

  The far region detection unit 111 can estimate the farthest point by regarding the distribution itself of FIG. 13 as a weighting coefficient. For example, the far region detection unit 111 calculates the weighting coefficient coef (x, y) as shown in the following equation (14) by normalizing the norm of the motion vector of each pixel with the following equation. can do.

・・・（１４）

(14)

  In equation (14), vnormmax (x, y) and vnormmin (x, y) are the maximum and minimum vector norms, respectively. As a result, when each pixel position is p (x, y), the farthest point P (x, y) can be calculated using the following equation (15).

・・・（１５）

... (15)

  In Expression (15), w and h are the horizontal width and the vertical width of the image.

  In the same manner as in the first method, the far region detection unit 111 estimates a position that is not more than the rate of change of the norm of the vector of the farthest point as a sufficiently far region. As another normalization method, for example, a method using an average value vnormave (x, y), a minimum value vnormmin (x, y), or the like can be considered.

  By these methods, the far region detection unit 111 can detect a sufficiently far region. In a sufficiently distant area, a condition that no motion vector is generated due to camera movement can be used. Therefore, only a motion vector due to camera rotation can be estimated based on this condition. Therefore, it is possible to estimate the camera motion by applying a weight to a sufficiently distant area in the subsequent statistical processing under the same conditions as described above.

  Then, the rotation component removal processing unit 112 around the optical axis removes the rotation component around the optical axis under the condition that the region is sufficiently far away.

  That is, when a movement model as shown in FIG. 8 is assumed, a symmetric point (x ′, y ′) with respect to a stop point as shown in Expression (16) and Expression (17) with respect to a certain point (x, y). )think of. If the point symmetry point from the center of rotation is added to each pixel, that is, the result of substituting x (t-1) and y (t-1) into equation (7) based on equation (7) Then, the following equation (18) can be obtained by adding together x ′ (t−1) and y ′ (t−1) into equation (7).

・・・（１６）

... (16)

・・・（１７）

... (17)

・・・（１８）

... (18)

  Here, in a sufficiently far region or a region having almost no straight component, it can be regarded as Vz << da, so the following equation (19) holds.

・・・（１９）

... (19)

  That is, when considering the case where Vz is sufficiently small with respect to da, this equation (19) is obtained by removing the rotational component using point symmetry and Vx, da (x, y), P (t) and , Vy, da (x, y), Q (t) can be expressed by a function of three variables.

  Next, the pan / tilt component and translational motion separation processing unit 113 acquires camera movement information with respect to the past frame supplied from the camera movement information calculation unit 88, and based on Expression (19), Vx, da (x, The three variables y) and P (t) are obtained, and the respective components of pan tilt and translational motion are separated.

  That is, in the case of a sufficiently far region or when there is almost no straight movement, the above-described equation (9) is established, and therefore the motion vector dv (t) (the motion vector of the camera movement information obtained from the equation (9) Therefore, originally, an arrow indicating a vector should be described at the top of dv with respect to dv, but it is omitted here and shown in the mathematical formula described later). Can be represented.

・・・（２０）

... (20)

  This can be regarded as an equation of the speed ratio Vx (t) / Vx (t−1) and the pan / tilt components P (t) and Q (t). Three examples of specific methods for solving this equation are shown below.

  First, as a first method, the relationship between mv (t) + mv ′ (t) and dv (t) + dv ′ (t) (because it is a motion vector, originally mv, mv ′, dv, dv ′ An arrow indicating a vector should be described on the upper part of the image, but here it will be omitted. The same applies hereinafter) on the coordinates of the speed ratio and pan / tilt will be described.

  As shown in FIG. 14, a coordinate system having two axes of a speed ratio and a pan / tilt component is considered. Mv (t) and dv (t) given at each point can be expressed as straight lines on the coordinates. That is, in a sufficiently far region or when there is almost no straight movement, an intersection (for example, a coordinate indicated by a point α in the figure) where these straight lines intersect most frequently is obtained in the coordinate system of the speed ratio and the pan / tilt component. Thus, the speed ratio and the pan / tilt component can be estimated.

  Next, as a second method, a method using statistical values of mv (t) + mv ′ (t) and dv (t) + dv ′ (t) will be described.

  The histogram of mv (t) and dv (t) is as shown in FIG. 15 from the equation (20). That is, in FIG. 15, the difference between the average values of mv (t) + mv ′ (t) and dv (t) + dv ′ (t) represents the pan / tilt component, and mv (t) + mv ′ (t) The ratio of the standard deviations of dv (t) + dv ′ (t) represents the speed ratio. This is shown by Formula (21) and Formula (22).

・・・（２１）

(21)

・・・（２２）

(22)

  In the second method, the speed ratio and the pan / tilt component can be estimated in this way.

  As a third method, a method of obtaining a pan / tilt component using Vx (t) / Vx (t−1) = 1 as an initial value will be described.

  The camera speed rarely changes abruptly during one frame. Taking this into consideration, it can be considered that Vx (t) / Vx (t−1) = 1, assuming that the camera is moving at a constant speed between frames. By utilizing this, the average value or the mode value of the histogram can be considered as the pan / tilt component.

  The pan / tilt component and translational motion separation processing unit 113 can obtain the speed ratio and the pan / tilt component by using, for example, any one of these three methods.

  The pan / tilt component acquisition unit 114 acquires the pan / tilt component separated by the pan / tilt component and translational separation processing unit 113.

  The translation vector motion vector acquisition unit 115 acquires a translation vector motion vector separated by the pan / tilt component and translation motion separation processing unit 113.

  Then, the rotation component extraction processing unit 116 around the optical axis is based on the pan / tilt component supplied from the pan / tilt component acquisition unit 114 and the motion vector by translation motion supplied from the motion vector acquisition unit 115 by translation motion. A rotation component around the optical axis is extracted from the motion vectors of the t frame and the t + 1 frame supplied from the motion vector detection unit 82.

  Since the pan / tilt component and the speed ratio are obtained by the above-described calculation, it can be regarded as da (x, y) >> Vz (t) from the equation (7) when there is almost no far enough area or straight movement. The following equation (23) holds.

・・・（２３）

... (23)

  The rotation component extraction processing unit 116 around the optical axis solves the equation (23) to obtain the rotation component θ (t) around the optical axis, and obtains the camera rotation information from the correction information generation unit 89 and the frame delay. 85 can be supplied.

  The straight motion / translation motion separation processing unit 117 separates the straight motion and the translation motion from the motion vectors of the t frame and the t + 1 frame supplied from the motion vector detection unit 82.

  Since the pan / tilt component and the rotation component around the optical axis are obtained by the above calculation, if the pan / tilt component and the rotation component around the optical axis are substituted into the above equation (6) and transformed, the following equation (24) Can be obtained.

・・・（２４）

... (24)

  Then, when the formula (24) is transformed using the formula (9), the following formula (25) can be obtained.

・・・（２５）

... (25)

  This equation (25) can be regarded as an equation of Vx (t) and Vz (t) / kw. The separation unit 117 for linear motion and translational motion performs mapping in the same manner as described with reference to FIG. 14 or uses statistical values in the same manner as described with reference to FIG. Calculations for obtaining Vx (t) and Vz (t) / kw from Expression (25) can be executed, and the obtained camera movement information is supplied to the correction information generation unit 89.

  Through the processing described above, the camera motion separation unit 84 described with reference to FIG. 12 can generate rotation information and camera movement information by separating rotation and movement of the imaging device 51.

  In the image processing 63 shown in FIG. 7, based on the camera rotation information and the camera movement information obtained in this way, the correction information generation unit 89 considers not only the rotation of the imaging device 51 but also the movement. Correction information is generated, and the corrected image generation unit 90 corrects and outputs the frame image.

  Thereby, the motion detection accuracy of the imaging device 51 can be improved, and an image can be corrected by effectively removing camera shake. Further, since the corrected image is generated based on the motion estimation result, the corrected image can be presented in the way that the photographer originally intended by the processing of the image processing 63.

  Next, with reference to the flowchart of FIG. 16, processing when the imaging device 51 corrects captured image data using the image processing device 63 will be described.

  In step S <b> 1, the # 1 frame memory 81 of the image processing device 63 determines whether an image is captured by the imaging processing unit 61 and the captured image data is supplied via the image data recording unit 62. If it is determined in step S1 that an image has not been captured, the process of step S1 is repeated until it is determined that an image has been captured.

  If it is determined in step S1 that an image has been captured, in step S2, the motion vector detection unit 83 and the camera movement information calculation unit 88 detect a motion vector.

  Specifically, the motion vector detecting unit 83 detects the motion vectors of the frames t and t + 1 shown in FIG. 17 as described above, and the camera movement information calculating unit 88 is a camera rotational motion correcting unit 86. On the basis of the image data of the t-1 frame in which the rotational movement of the camera is corrected by the processing of, the motion vectors of the frames of t and t-1 shown in FIG. 17 are detected as described above. This is supplied to the camera motion separation unit 84.

  In step S3, the camera motion separation unit 84 separates and estimates camera movement and camera rotation as described with reference to FIGS. 12 to 15, and obtains camera movement information and camera rotation information obtained as a result. This is supplied to the correction information generation unit 89.

  In step S4, the # 1 frame memory 81 of the image processing apparatus 63 determines whether there is a next frame.

  If it is determined in step S4 that there is a next frame, in step S5, the correction information generation unit 89 generates correction information for correcting the captured image data based on the camera movement information and the camera rotation information. , And supplied to the corrected image generation unit 90. The corrected image generation unit 90 corrects the captured image data based on the correction information supplied from the correction information generation unit 89.

  In step S6, the normal image generation unit 90 outputs the corrected image to the image data recording unit 62 or the display control unit 64, the process returns to step S2, and the subsequent processes are repeated.

  If it is determined in step S4 that there is no next frame, in step S7, the correction information generation unit 89 generates correction information for correcting the captured image data based on the camera movement information and the camera rotation information. , And supplied to the corrected image generation unit 90. The corrected image generation unit 90 corrects the captured image data based on the correction information supplied from the correction information generation unit 89.

  In step S8, the normal image generation unit 90 outputs the corrected image to the image data recording unit 62 or the display control unit 64, and the process ends.

  By such processing, the motion of the imaging device 51 can be estimated by distinguishing between the camera rotational motion and the camera translational motion, so that the motion detection accuracy of the imaging device 51 is improved, and only camera shake is purely ( That is, the image can be effectively removed and corrected without erroneously detecting the movement of the imaging device 51.

  Further, since the corrected image is generated based on the motion estimation result of the imaging device 51, the corrected image can be presented in the way the photographer originally intended by the processing in the image processing 63. Furthermore, since statistical processing is used in the image processing 63, the robustness of camera motion estimation is high.

  In this way, in the imaging apparatus 51 to which the present invention is applied, blurring is erroneously detected as compared with correction at the imaging stage in which the imaging optical system is driven or the readout range of the imaging sensor is shifted. However, it has the advantage that an uncorrected image can be left and displayed or output, and that a large amount of memory is not required when processing in real time. At the same time, even though the camera is actually moving, it is judged that the camera is moving (false detection), and reverse correction is performed. Therefore, it is possible to correct blur with high accuracy.

  In the present invention, it is needless to say that not only camera shake but also blur that causes the subject to move within the shutter time can be corrected basically in the same manner.

  The series of processes described above can be executed by hardware or can be executed by software. In this case, the processing described above is executed by a personal computer 500 as shown in FIG.

  In FIG. 18, a CPU (Central Processing Unit) 501 performs various processes according to a program stored in a ROM (Read Only Memory) 502 or a program loaded from a storage unit 508 to a RAM (Random Access Memory) 503. Execute. The RAM 503 also appropriately stores data necessary for the CPU 501 to execute various processes.

  The CPU 501, ROM 502, and RAM 503 are connected to each other via an internal bus 504. An input / output interface 505 is also connected to the internal bus 504.

  The input / output interface 505 includes an input unit 506 including a keyboard and a mouse, a display including CRT and LCD, an output unit 507 including a speaker, a storage unit 508 including a hard disk, a modem, a terminal adapter, and the like. A communicator 509 is connected. A communication unit 509 performs communication processing via various networks including a telephone line and CATV.

  A drive 510 is connected to the input / output interface 505 as necessary, and a removable medium 521 made up of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted, and a computer program read therefrom is It is installed in the storage unit 508 as necessary.

  When a series of processing is executed by software, a program constituting the software is installed from a network or a program storage medium.

  As shown in FIG. 18, this program storage medium is configured only by a package medium including a removable medium 521 on which a program is recorded, which is distributed to provide a program to a user, separately from a computer. Rather, it is provided with a ROM 502 in which a program is recorded and a hard disk including a storage unit 508 provided to the user in a state of being pre-installed in the apparatus main body.

  Further, in the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but may be performed in parallel or It also includes processes that are executed individually.

  In the present specification, the term “system” represents the entire apparatus constituted by a plurality of apparatuses.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a figure for demonstrating a motion of a camera. It is a figure for demonstrating rotation around an X-axis. It is a figure for demonstrating rotation around a Y-axis. It is a figure for demonstrating rotation around a Z-axis. It is a figure for demonstrating the movement of the X-axis direction of a camera. It is a block diagram which shows the structure of the imaging device to which this invention is applied. It is a block diagram which shows the structure of the image process part of FIG. It is a figure for demonstrating a motion vector. It is a figure for demonstrating the case where a camera moves to the orthogonal | vertical direction of an imaging surface. It is a figure for demonstrating the case where a camera moves to the direction of translation with respect to an imaging surface. It is a figure for demonstrating the Vam vector which shows the moving amount | distance of the to-be-photographed object a in a flame | frame. It is a functional block diagram for demonstrating the function of the camera motion separation part of FIG. It is a figure for demonstrating distribution of a vector norm. It is a figure for demonstrating the coordinate system which makes a speed ratio and a pan-tilt component 2 axes. It is a figure for demonstrating the histogram of v (t) and dv (t). It is a flowchart for demonstrating the process of an imaging device. It is a figure for demonstrating a motion vector and its decomposition | disassembly. It is a block diagram which shows the structure of a personal computer.

Explanation of symbols

  51 imaging device, 63 image processing unit, 83 motion vector detection unit, 84 camera motion separation unit, 86 camera rotation motion correction unit, 88 camera movement information calculation unit, 89 correction information generation unit, 90 correction image generation unit, 111 far region Detection unit, 112 Rotational component removal processing unit around optical axis, 113 Pan / tilt component and translational separation processing unit, 116 Extraction processing unit of rotational component around optical axis, 117 Linear processing and translational separation processing unit

Claims (18)

In an image processing apparatus that detects the movement of an imaging unit that captured the moving image based on a plurality of frames in the moving image,
Obtaining means for obtaining the moving image;
Motion vector detection means for detecting a motion vector at any interval in the moving image acquired by the acquisition means;
Separating means for separating the motion vector detected by the motion vector detection means into a first motion vector due to movement of the imaging means and a second motion vector due to rotation of the imaging means;
The image processing apparatus, wherein the separation unit statistically calculates the first motion vector and the second motion vector. A movement information detecting means for detecting movement of the imaging means to obtain movement information of the imaging means;
The separating means includes
A distant area estimating means for estimating an area where a sufficiently distant subject in the frame in the image is captured;
Removing means for removing, from the motion vector detected by the motion vector detecting means, a third motion vector due to rotation around the optical axis of the imaging means;
A movement relationship for estimating a relationship between the movement information detected by the movement information detection unit and the actual movement of the imaging unit based on the motion vector from which the third motion vector has been removed by the removal unit. An estimation means;
Rotation estimation means for estimating rotation of the imaging means around the optical axis;
The image processing apparatus according to claim 1, further comprising: a movement estimation unit that estimates a movement of the imaging unit. The frame further comprises rotational motion correction means for correcting the rotational movement of the imaging means using affine transformation and converting it to a past frame,
The movement information detection unit detects the motion vector between the past frame converted by the rotational motion correction unit and the frame, and obtains the movement information by inverting the detected motion vector. 2. The image processing apparatus according to 2. The removing means calculates a point-symmetrical position with the center point of the frame as a rotation center point for each pixel of the frame, and adds the motion vectors at the point-symmetrical positions to add a motion around the optical axis. The image processing apparatus according to claim 2, wherein a rotation component is removed. The far region estimation means calculates at least one of a norm, an angle, or a rate of change with a nearby sample point of the motion vector, and statistically processes the calculated value to subject a sufficiently far subject The image processing device according to claim 2, wherein the imaged area is estimated. The image processing apparatus according to claim 5, wherein the far region estimation unit estimates a region in which a sufficiently far subject is imaged by obtaining a position where a norm value of the motion vector is minimum. The far region estimation means has a difference between a minimum value of the norm of the motion vector and a norm of the motion vector at each position equal to or less than a change rate at a point where the minimum value of the norm of the motion vector is taken. The image processing apparatus according to claim 6, wherein the area is estimated as an area where a sufficiently distant subject is imaged. 6. The distant area estimation means estimates an area where a sufficiently distant subject is imaged by using a value obtained by normalizing the norm value of the motion vector in all pixels in the frame as a weight. An image processing apparatus according to 1. The movement relationship estimation unit estimates a speed ratio and a pan / tilt component of a rotational movement of the imaging unit in a region where a sufficiently far subject in the frame estimated by the far region estimation unit is imaged. Item 3. The image processing apparatus according to Item 2. The rotation estimation means removes the movement of the imaging means estimated by the movement relationship estimation means and the pan / tilt component of the rotational movement of the imaging means from the motion vector detected by the motion vector detection means. The image processing apparatus according to claim 9, wherein rotation of the imaging unit around the optical axis is estimated. The correction signal generating means for generating a correction signal for correcting the image of the frame based on the first motion vector and the second motion vector obtained by the separating means. The image processing apparatus described. The image processing apparatus according to claim 11, further comprising an image correcting unit that corrects the image data of the frame based on the correction signal generated by the correction signal generating unit. The image processing apparatus according to claim 11, further comprising a recording unit that records the correction signal generated by the correction signal generation unit. The image processing apparatus according to claim 12, further comprising a recording unit that records the corrected image data generated by being corrected by the image correcting unit. The image processing apparatus according to claim 12, further comprising display control means for controlling display of corrected image data generated by correction by the image correction means. In an image processing method of an image processing apparatus for detecting a motion of an imaging unit that has captured the moving image based on a plurality of frames in the moving image,
Acquiring in the moving image,
Detecting a motion vector at an arbitrary interval in the moving image;
An image processing method including a step of separating the motion vector into a first motion vector due to movement of the imaging unit and a second motion vector due to rotation of the imaging unit by statistical calculation. A program for causing a computer to execute processing for detecting camera movement from a plurality of frames in a moving image captured by an imaging unit,
Acquiring in the moving image,
Detecting a motion vector at an arbitrary interval in the moving image;
A program that causes a computer to execute a process including a step of separating the motion vector into a first motion vector due to movement of the imaging unit and a second motion vector due to rotation of the imaging unit by statistical calculation.   A recording medium on which the program according to claim 17 is recorded.

Images