JP4622889B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for processing an image data to calculate an optical flow.

  Japanese Patent Application Laid-Open No. 2004-228561 describes an apparatus that can recognize the movement of the same point at a predetermined time as an optical flow at high speed in an image obtained by photographing with a camera. This device considers the lower area in the image as a road area, divides the area in the image into a road area and other areas in advance, and detects obstacles based on the optical flow of the road area. Yes.

  Further, in the technique described in Non-Patent Document 2, the optical flow on the horizontal plane is acquired in advance as a reference value, the optical flow calculated from the image photographed by the camera, and the reference value of the optical flow on the lateral surface Are compared to detect a lateral surface area included in the image. In this process, the horizontal plane area included in the image and the other areas are simply discriminated.

JP 2000-123183 A Proceedings of the 11th Image Sensing Symposium, pages 1 to 4, “Autonomous mobile robot navigation using vision systems”

  There are objects to be photographed in various postures in the image. There is a demand for discriminating the posture of such an object in more detail.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus and an image processing method capable of discriminating various postures of an object using an optical flow.

  To achieve the above-described object, an image processing apparatus according to the present invention includes an image acquisition unit that acquires a plurality of image data obtained at different times, and an optical flow calculation unit that calculates a movement vector based on the image data. And a partial differentiation means for partial differentiation of the movement vector calculated by the optical flow calculation means in a predetermined direction to calculate a partial differentiation value of the movement vector, and based on the partial differentiation value calculated by the partial differentiation means. Posture determining means for determining the posture of the surface constituting the photographed object.

  According to this configuration, the image processing apparatus performs partial differentiation of the movement vector in a predetermined direction, calculates a partial differentiation value of the movement vector, and further determines the posture of the surface constituting the subject based on the partial differentiation value. To do. Here, the posture of the surface constituting the object to be photographed can be determined in detail by determining the posture of the surface constituting the object to be photographed using the partial differential value of the movement vector. For example, the surface constituting the object to be photographed can take a plurality of postures such as upward, rightward, and leftward, but the actual posture of the surface constituting the object to be photographed can be determined.

  Here, the partial differential value of the movement vector is, for example, a partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the vertical direction, or a partial differentiation obtained by partial differentiation of the horizontal component of the movement vector in the horizontal direction. A partial differential value obtained by partial differentiation of the vertical component of the value and the movement vector in the vertical direction, and a partial differential value obtained by partial differentiation of the vertical component of the movement vector in the horizontal direction. The posture determination means may determine the posture of the subject using only one of these partial differential values, or may determine the posture of the subject using a plurality of partial differential values. Good.

  In particular, in the above-described image processing apparatus, the posture determination means includes a partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the vertical direction, and a partial differentiation obtained by partial differentiation of the horizontal component of the movement vector in the horizontal direction. Based on the combination of the value, the partial differential value obtained by partial differentiation of the vertical component of the movement vector in the vertical direction, and the partial differentiation value obtained by partial differentiation of the vertical component of the movement vector in the horizontal direction, It is preferable to determine the posture of the constituting surface. According to this configuration, since the posture of the surface that constitutes the object uniquely corresponds to the combination of the four partial differential values, the orientation of the surface that constitutes the object is determined based on the combination of the four partial differential values. The posture can be determined.

  Further, in the above-described image processing apparatus, the posture determination unit has a partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the vertical direction is substantially 0, and the partial differentiation of the vertical component of the movement vector in the horizontal direction. The partial differential value obtained from the above is approximately 0, the partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the horizontal direction, and the partial differential value obtained by partial differentiation of the vertical component of the movement vector in the vertical direction. Are substantially the same, it is preferable that the surfaces constituting the object to be photographed be perpendicular to the optical axis. According to this configuration, it is possible to determine that the surfaces constituting the object to be photographed are perpendicular to the optical axis of the image data.

  In the above-described image processing apparatus, the posture determination means has a partial differential value obtained by partially differentiating the horizontal component of the movement vector in the vertical direction is substantially zero, and partially differentiates the horizontal component of the movement vector in the horizontal direction. When the partial differential value obtained in this way is approximately twice the partial differential value obtained by partial differentiation of the vertical component of the movement vector in the vertical direction, the surface constituting the object to be imaged is the optical axis of the image data and It is preferable to determine that it is parallel to the vertical direction. According to this configuration, it is possible to determine that the surfaces constituting the object to be photographed are parallel to the optical axis and the vertical direction of the image data.

  In the above-described image processing apparatus, the posture determination means has a partial differential value obtained by partial differentiation of the vertical component of the movement vector in the horizontal direction is substantially 0, and partial differentiation of the vertical component of the movement vector in the vertical direction. When the partial differential value obtained in this way is approximately twice the partial differential value obtained by partial differentiation of the lateral component of the movement vector in the lateral direction, the surface constituting the object to be imaged is the optical axis of the image data and It is preferable to determine that it is parallel to the lateral direction. According to this configuration, it is possible to determine that the surfaces constituting the object to be photographed are parallel to the optical axis and the horizontal direction of the image data.

  In the determination conditions of the posture determination means described above, the partial differential value of approximately 0 means that the partial differential value is a value close to 0. In the specific processing of the posture determination means, it is determined that the partial differential value is substantially 0 when the partial differential value is smaller than a threshold value for substantially 0 determination.

  Further, “substantially identical partial differential values” means that two partial differential values are close to each other. In the specific processing of the posture determination means, it is determined that the partial differential values are substantially the same when the difference between the two partial differential values is smaller than the substantially identical determination threshold.

  Also, “substantially twice the partial differential value” means that one of the two partial differential values is close to twice the other. In the specific processing of the posture determination means, when the difference obtained by subtracting twice the partial differential value from one partial differential value is smaller than the threshold for determination of about double, the partial differential value is about double. It is determined that

  Further, the image processing apparatus described above stores the height data of the shooting position of the image data, and based on the height of the shooting position and the movement vector, the relative position between the shooting position of the image data and the object to be captured is stored. It is preferable to further include speed calculating means for calculating the speed. According to this configuration, it is possible to calculate the relative speed between the photographing position and the subject.

  Further, the above-described image processing apparatus further includes distance calculation means for calculating the distance from the shooting position of the image data to the subject based on the relative speed calculated by the speed calculation means and the movement vector. Is preferred. According to this configuration, the distance from the shooting position of the image data to the subject can be calculated.

Further, the above-described image processing apparatus is a preprocessing unit that processes the image data before the optical flow calculation unit calculates the movement vector based on the image data. Velocity eigenvalues V _X / V _Z and V _Y / V _Z (where V _X is the lateral velocity of the moving object, V _Y is the vertical velocity of the moving object, and V _Z is the optical axis velocity of the moving object. It is preferable to further include pre-processing means for performing parallel movement using (). According to this configuration, the image region corresponding to the moving object is translated using the velocity eigenvalue of the moving object, so that the processing by the image processing apparatus is suitably performed even when the subject is moving. Can do.

Further, in the above-described image processing apparatus, the pre-processing unit divides the image into the image areas where the speed eigenvalues V _X / V _Z and V _Y / V _Z are common, and the speed eigenvalue V _X / V for each divided image area. It is preferable to translate using _Z and V _Y / V _Z. Here, in order to classify each image region where the speed eigenvalues V _X / V _Z and V _Y / V _Z are common, a Hough transform process, a random sample consensus (RANdom SAmple Consensus: RANSAC) process, or the like may be performed.

  In order to achieve the above-described object, an image processing method according to the present invention is an image processing method for determining the posture of a surface constituting an object to be captured, and a plurality of images obtained at different times. An image acquisition step for acquiring data; an optical flow calculation step for calculating a movement vector based on the image data; a partial differentiation step for calculating a partial differential value of the movement vector by partial differentiation of the movement vector in a predetermined direction; A posture determination step of determining the posture of the surface constituting the object to be photographed based on the partial differential value.

  According to the image processing apparatus and the image processing method according to the present invention, it is possible to discriminate various postures of the subject using the optical flow.

  Hereinafter, preferred embodiments of an image processing apparatus of the present invention will be described with reference to the drawings. In the present embodiment, the image processing apparatus is mounted on a vehicle.

  As shown in FIG. 1, a camera 10 is connected to the image processing apparatus 20 as an imaging unit. The camera 10 is, for example, a CCD [Charge Coupled Device] camera 10 and is attached to the front and center of an automobile on which the image processing apparatus 20 is mounted. The camera 10 captures an image of the front of the automobile every time a predetermined time elapses and generates image data composed of a plurality of pixel data. The camera 10 sequentially transmits the generated image data to the image processing device 20.

  The three-dimensional coordinate system of the surrounding environment photographed by the camera 10 has the position of the camera 10 as the origin, the horizontal direction (horizontal direction) of the camera 10 as the X axis, and the vertical direction (vertical direction) of the camera 10 as the Y axis. The world coordinate system with the camera optical axis as the Z axis. In the three-dimensional coordinate system, the camera 10 moves in the optical axis direction, and the road and other objects around it are assumed to be stationary. In the two-dimensional coordinate system of an image taken by the camera 10, the position corresponding to the camera optical axis is the origin, the horizontal direction of the image plane is the x axis, and the vertical direction of the image plane is the y axis. Here, the x-axis is parallel to the X-axis, and the y-axis is parallel to the Y-axis.

  The image processing device 20 acquires image data from the camera 10 and processes the image data to obtain information useful for vehicle control. The image processing apparatus 20 is physically configured with a CPU, a ROM, a RAM, and the like. Functionally, the image processing apparatus 20 includes an image acquisition unit 22, an optical flow calculation unit 24, a partial differentiation unit 26, and an attitude determination unit 28. Furthermore, the image processing apparatus 20 may include an actual speed calculation unit 30 and an actual distance calculation unit 32 as described in a first modification described later. In addition, the image processing apparatus 20 may include a preprocessing unit 34 as described in a second modification described later.

  The image acquisition unit 22 takes in the image data sent from the camera 10 and puts the image data into a state suitable for later processing. The optical flow calculation unit 24 calculates a movement vector of the optical flow from image data around a predetermined time by a gradient method. More specifically, the optical flow calculation unit 24 calculates the following formula derived from the assumption (precondition) that the optical flow is constant because pixels in the vicinity of the pixel of interest in the image perform the same motion. Using (1), raster scanning is performed for all pixels to calculate the horizontal component u and the vertical component v of the movement vector.

  FIG. 2 shows an example in which the movement vector of the optical flow is superimposed and displayed on the image data photographed by the camera 10. The optical flow calculation unit 24 may calculate the movement vector by another method. For example, the optical flow calculation unit 24 may calculate a movement vector by extracting a feature point from an image and tracking the feature point.

  The partial differentiation unit 26 partially differentiates the movement vector calculated by the optical flow calculation unit 24 in a predetermined direction to calculate a partial differentiation value of the movement vector. More specifically, the partial differentiator 26 (I) a partial differential value obtained by partially differentiating the horizontal component u of the movement vector in the vertical direction, and (II) a partial differentiation of the horizontal component u of the movement vector in the horizontal direction. (III) partial differential value obtained by partial differentiation of the vertical component v of the movement vector in the vertical direction, and (IV) partial deviation obtained by partial differentiation of the vertical component v of the movement vector in the horizontal direction. Calculate the differential value. Specifically, as shown in FIG. 3, the horizontal component u and the vertical component of the movement vector at the pixel positions (x−1, y), (x + 1, y), (x, y−1) (x, y + 1). From the component v, the partial differential values of (I) to (IV) at the pixel position (x, y) are calculated by the following mathematical formula (2).

  The posture determination unit 28 determines the posture of the surface constituting the object to be photographed based on the partial differential value calculated by the partial differential unit 26. More specifically, the posture determination unit 28 stores map data in which the posture of the surface constituting the object to be photographed is uniquely associated with the combination of the four partial differential values (I) to (IV) described above. When four partial differential values are calculated, the orientation of the surface constituting the object to be photographed is obtained with reference to this map data. Then, the posture determination unit 28 performs a labeling process for associating information specifying the posture of the surface constituting the object with the pixel position corresponding to the object.

  The map data described above is obtained by theoretical calculation. That is, assuming that objects of various postures appear in each pixel of the image data, the optical flow movement vector is calculated for each object of various postures, and the four partial differential values of the movement vector are calculated. To do. Then, the calculated four partial differential values are stored in a ROM or the like in association with the posture of the object. Through the series of calculations described above, map data can be obtained in which the orientation of the surface constituting the object to be captured is stored in association with the four partial differential values for each pixel of the image data. In the map data, the posture of the surface constituting the object to be imaged may be specified by the inclination angle of the surface, the perpendicular direction of the surface, or the like. Although the map data is theoretically obtained by the above-described calculation, the map data may be obtained experimentally by actually photographing objects of various postures with the camera 10.

  The processing of the image processing apparatus 20 described above will be described with reference to the flowchart of FIG. First, the image processing apparatus 20 takes in image data from the camera 10 and calculates an optical flow movement vector for each area of the image data (S401). Next, the image processing apparatus 20 calculates a partial differential value of the movement vector by performing partial differentiation on the horizontal component and the vertical component of the movement vector in the horizontal direction and the vertical direction, respectively (S402). Next, the image processing device 20 determines the orientation of the surface constituting the subject to be labeled for each region of the image data based on the calculated partial differential value (S403).

  According to the above-described embodiment, the image processing device 20 performs partial differentiation of the movement vector in a predetermined direction to calculate a partial differentiation value of the movement vector, and further, the surface of the surface constituting the object to be photographed based on the partial differentiation value. Determine posture. Here, the posture of the surface constituting the object to be photographed can be determined in detail by determining the posture of the surface constituting the object to be photographed using the partial differential value of the movement vector. For example, the surface constituting the object to be photographed can take a plurality of postures such as upward, rightward, leftward, and oblique directions, but the actual posture of the surface constituting the object can be determined.

  In the above-described processing, the posture determination unit 28 determines the posture of the surface constituting the subject based on the combination of the four partial differential values. In another processing, the posture determination unit 28 The posture of the object to be photographed may be determined using only one of the four partial differential values, or the posture of the object to be photographed may be determined using two or three partial differential values. A process in which the posture determination unit 28 determines the posture of the object to be photographed using the three partial differential values will be described with reference to the flowchart of FIG.

  In the determination process shown in FIG. 5, first, the posture determination unit 28 has a partial differential value obtained by partially differentiating the horizontal component of the movement vector in the vertical direction, and the horizontal component of the movement vector is horizontal. It is determined whether or not the determination condition that the partial differential value obtained by partial differentiation in the direction is substantially 0 is satisfied (S501). Here, when the determination condition is satisfied, the posture determination unit 28 next converts the partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the horizontal direction and the vertical component of the movement vector. It is determined whether or not the determination condition that the partial differential value obtained by partial differentiation in the direction is substantially the same is satisfied (S502). Here, when the determination condition is satisfied, the posture determination unit 28 determines that the surface constituting the object to be photographed is a surface perpendicular to the camera optical axis of the image data (hereinafter referred to as a front vertical surface). And labeling is performed (S503). On the other hand, if the determination condition is not satisfied, the posture determination unit 28 determines that the surface constituting the object to be photographed has another posture and performs labeling (S510).

  When the determination condition of step 501 is not satisfied, the posture determination unit 28 satisfies the determination condition that the partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the vertical direction is substantially zero. It is determined whether or not (S504). Here, when the determination condition is satisfied, the posture determination unit 28 then determines that the partial differential value obtained by partial differential of the horizontal component of the movement vector in the horizontal direction is the vertical component of the movement vector. It is determined whether or not the determination condition that the partial differential value obtained by partial differentiation in the direction is approximately double is satisfied (S505). Here, when the determination condition is satisfied, the posture determination unit 28 determines that the surface constituting the object to be photographed is a surface parallel to the camera optical axis and the vertical direction X of the image data (hereinafter, referred to as a right / left vertical surface). ) And labeling is performed (S506). On the other hand, if the determination condition is not satisfied, the posture determination unit 28 determines that the surface constituting the object to be photographed has another posture and performs labeling (S510).

  If the determination condition of step 504 is not satisfied, the posture determination unit 28 satisfies the determination condition that the partial differential value obtained by partial differential of the vertical component of the movement vector in the horizontal direction is substantially zero. It is determined whether or not (S507). Here, when the determination condition is satisfied, the posture determination unit 28 next converts the horizontal component of the movement vector by the partial differential value obtained by partial differentiation of the vertical component of the movement vector in the vertical direction. It is determined whether or not the determination condition that the partial differential value obtained by partial differentiation in the direction is approximately double is satisfied (S508). Here, when the determination condition is satisfied, the posture determination unit 28 is a surface (hereinafter, referred to as a horizontal plane) in which the surface constituting the object is parallel to the camera optical axis and the lateral direction Y of the image data. It is determined that there is a label and labeling is performed (S509). On the other hand, if the determination condition is not satisfied, the posture determination unit 28 determines that the surface constituting the object to be photographed has another posture and performs labeling (S510).

The principle of the posture determination of the subject used in the above-described determination process will be described. An object to be photographed that exists at a position (X, Y, Z) in the world coordinate system is projected to a pixel position (x, y) represented by the following mathematical formula (3).

The movement vector (u, v) can be converted as represented by the following equation (4) in consideration of the above equation (3).

As shown in FIG. 6, since the position of the front vertical plane in the Z-axis direction is constant and the moving speed Vz of the camera 10 in the Z-axis direction can be regarded as constant, the movement vector (u, v) of the front vertical plane is , Vz / Z is a constant, and is expressed by the following formula (5).

Similarly, since the position in the X-axis direction of the left and right vertical plane is constant and the moving speed Vz of the camera 10 in the Z-axis direction can be regarded as constant, the movement vector (u, v) of the left and right vertical plane is Vz / X. As a constant, it is expressed by the following formula (6).

Similarly, since the position of the horizontal plane in the Y-axis direction is constant and the moving speed Vz of the camera 10 in the Z-axis direction can be regarded as constant, the horizontal plane movement vector (u, v) (7)

Furthermore, from the above formulas (5), (6), and (7), the four partial differential values of the front vertical plane, the left and right vertical plane, and the horizontal plane are expressed by the following formula (8).

For this reason, it is possible to make a determination such as the following formula (9).

  That is, the partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the horizontal direction is a constant a, the partial differentiation value obtained by partial differentiation of the horizontal component of the movement vector in the vertical direction is a constant 0, The partial differential value obtained by partial differentiation of the vertical component of the movement vector in the horizontal direction is 0, and the partial differentiation value obtained by partial differentiation of the vertical component of the movement vector in the vertical direction is a constant a. It can be determined that the surface constituting the photographed object is the front vertical surface.

  The partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the horizontal direction is twice the constant a, and the partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the vertical direction is the constant 0. The partial differential value obtained by partial differentiation of the vertical component of the movement vector in the horizontal direction is another constant b, and the partial differential value obtained by partial differentiation of the vertical component of the movement vector in the vertical direction is the constant a. In this case, it can be determined that the surface constituting the object to be photographed is a left-right vertical surface.

  The partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the horizontal direction is a constant a, and the partial differentiation value obtained by partial differentiation of the horizontal component of the movement vector in the vertical direction is another constant b. Yes, the partial differential value obtained by partially differentiating the vertical component of the movement vector in the horizontal direction is 0, and the partial differential value obtained by partial differentiation of the vertical component of the movement vector in the vertical direction is twice the constant a. In some cases, it can be determined that the surface constituting the object to be photographed is a horizontal plane.

  Next, a first modification of the image processing device 20 according to the above-described embodiment will be described. As described above, in the first modification, the image processing apparatus 20 includes the actual speed calculation unit 30 and the actual distance calculation unit 32 (see FIG. 1).

The actual speed calculation unit 30 performs a process of calculating a relative speed between the shooting position of the image data and the subject. More specifically, when the actual speed calculation unit 30 determines a horizontal plane extending below the vehicle in the image, the actual speed calculation unit 30 determines that the horizontal plane is a road surface. Then, the actual speed calculation unit 30 calculates the horizontal plane constant Vz / Y from the optical flow of the horizontal plane image area by using the above mathematical formula (7). Specifically, the horizontal component u of the optical flow calculated for the pixel coordinate (x, y) may be divided by x · y to calculate the constant Vz / Y, or the vertical component v of the optical flow may be calculated as y. ^The constant Vz / Y may be calculated by dividing by ² . Furthermore, the constant Vz / Y may be calculated for all the pixels in the horizontal plane area, and the average value thereof may be obtained. Here, the actual speed calculation unit 30 stores and calculates the mounting height Y of the camera 10 (for example, 1.5 m), in other words, data of the height Y from the road surface at the shooting position of the image data. By multiplying the constant Vz / Y by the height Y, the relative velocity Vz in the camera optical axis direction between the imaging position of the image data and the object to be imaged is calculated.

  The actual distance calculation unit 32 performs a process of calculating the distance from the shooting position of the image data to the subject. More specifically, when the real distance calculation unit 32 determines the front vertical plane in the image, the front vertical plane constant Vz / is calculated from the optical flow in the image area of the front vertical plane using the above formula (5). Z is calculated. Specifically, the horizontal component u of the optical flow calculated for the pixel coordinate (x, y) may be divided by x to calculate a constant Vz / Z, or the vertical component v of the optical flow is divided by y. Then, the constant Vz / Z may be calculated. Further, the constant Vz / Z may be calculated for all the pixels in the front vertical plane region, and the average value thereof may be obtained. Then, the actual distance calculation unit 32 divides the relative velocity Vz in the camera optical axis direction calculated by the actual velocity calculation unit 30 by a constant Vz / Z, so that the camera light from the imaging position of the image data to the front vertical plane is obtained. A distance Z in the axial direction is calculated.

Further, when the real distance calculation unit 32 determines the left and right vertical planes in the image, it calculates the left and right vertical plane constants Vz / X from the optical flow in the image area of the left and right vertical planes using the above formula (6). To do. Specifically, the horizontal component u of the optical flow calculated for the pixel coordinate (x, y) may be divided by x ² to calculate the constant Vz / X, or the vertical component v of the optical flow may be calculated as x · The constant Vz / X may be calculated by dividing by y. Furthermore, the constant Vz / X may be calculated for all the pixels in the left and right vertical plane region, and the average value thereof may be obtained. Then, the actual distance calculation unit 32 divides the relative velocity Vz in the camera optical axis direction calculated by the actual velocity calculation unit 30 by a constant Vz / X, so that the horizontal axis from the imaging position of the image data to the left and right vertical planes. The direction distance X is calculated.

  Processing of the image processing apparatus 20 according to the first modification will be described with reference to the flowchart of FIG. First, similar to the processing described above, when the image processing apparatus 20 captures image data, it calculates an optical flow and then calculates a partial differential value thereof, and determines the posture of the object to be photographed based on the partial differential value. And labeling (S701 to S703). Next, the image processing apparatus 20 calculates a constant represented by the following mathematical formula (10) based on the optical flow.

  That is, a constant corresponding to Vz / Y is calculated for the horizontal plane area of the image data, a constant corresponding to the constant Vz / Z is calculated for the front vertical plane area, and a constant corresponding to Vz / X is calculated for the left and right vertical plane areas. (S704). Then, the image processing apparatus 20 performs processing for calculating the relative speed Vz in the camera optical axis direction between the imaging position of the image data and the object to be captured, and then from the imaging position of the image data to the front vertical plane and the left and right vertical planes. The distances Z and X are calculated (S705).

  In the prior art, for example, the distance to the subject is calculated using a technique such as a motion stereo method. However, methods such as the motion stereo method cannot determine the actual distances from the imaging position of the image data to the front vertical plane and the left and right vertical planes because the scale factor is unknown. On the other hand, according to the first modification described above, since the data of the height Y from the road surface of the imaging position of the image data is stored, the imaging position of the image data is used using this height data. And the relative speed Vz between the object and the object and the accurate distances Z and X from the image data capturing position to the object can be calculated.

  Next, a second modification of the image processing device 20 according to the above-described embodiment will be described. As described above, in the second modification, the image processing apparatus 20 includes the preprocessing unit 34 that processes the image data before the movement vector is calculated (see FIG. 1).

First, the reason why the preprocessing unit 34 is provided will be described. A moving object to be photographed object is moving, the moving velocity component in the X-axis direction is V _X, the movement velocity component of the Y-axis direction is V _Y, the movement speed component of the Z-axis direction is V _Z situation Then, the movement vector (u, v) in the image area of the moving object is expressed by the following equation (11). However, since the moving object does not move in the direction of the camera optical axis, the movement vector (u, v) represented by the equation (11) is the movement vector (u, v) represented by the equation (5). And the form of the equations are different and are not suitable for the processing described above by the image processing apparatus 20. The preprocessing unit 34 is provided to solve such a problem.

  Next, processing by the preprocessing unit 34 will be described. The preprocessing unit 34 performs a process of dividing an image for each different moving object by performing a process using the following formula (12) for all pixels.

  In Equation (12) above, as described above, x is the horizontal position of the pixel, and y is the vertical position of the pixel. U is a horizontal component of the movement vector, and v is a vertical component of the movement vector. These values x, y, u, v are known by being calculated for each pixel. Therefore, the above formula (12) is a straight line with v / u as the slope and v / u · xy as the intercept in the coordinate system composed of the horizontal axis S and the vertical axis T for each pixel of the image data. Represents. Here, as shown in FIG. 8, when the straight lines of Expression (12) are obtained for each of the pixels that photograph the same moving object, the straight lines intersect at the same point, and the number of moving objects is the number of intersections. Exists. That is, the intersected coordinate values S and T are velocity eigenvalues of the moving object.

  The pre-processing unit 34 obtains the pixel groups having the common speed eigenvalues S and T by using a method such as Hough transform and random sample consensus (RANdom SAmple Consensus: RANSAC). Divide each image area. More specifically, the preprocessing unit 34 finely divides the coordinate system composed of the horizontal axis S and the vertical axis T in the horizontal axis direction and the vertical axis direction. Then, the preprocessing unit 34 performs a process of voting on a mesh region through which a straight line passes for all the straight lines of pixels. As a result, a mesh region having a large number of votes can be determined as the speed eigenvalues S and T, and a straight line pixel group passing through the speed eigenvalues S and T can be determined as a single moving object. In this way, the image is divided for each different moving object.

  Then, the preprocessing unit 34 performs a process of translating the pixel position for each image area of a different moving object. More specifically, the preprocessing unit 34 adds the speed eigenvalue S to the lateral position x of the pixel constituting the image region having the speed eigenvalues S and T as shown in the following formula (13). Thus, the speed eigenvalue T is added to the vertical position y of the pixel. Thereby, the horizontal position x ′ and the vertical position y ′ of the pixel after translation are calculated.

  Processing of the image processing apparatus 20 according to the second modification will be described with reference to the flowchart of FIG. First, when the image processing apparatus 20 captures image data, it performs preprocessing. In this pre-processing, the image processing apparatus 20 divides image regions with different motions and performs parallel movement processing for each region (S901, S902). After that, the image processing apparatus 20 calculates the partial differential value after calculating the optical flow movement vector, and determines the posture of the object to be photographed based on the partial differential value, similarly to the processing described above. (S903 to S905).

  Next, in order to explain the advantages of the second modified example, the principle of the preprocessing described above will be described. From the above equation (11), the following equation (14) is obtained. Further, when the formula (14) is modified, the following formula (15) is obtained.

As shown in FIG. 8, a space is assumed in which V _X / V _Z is the horizontal axis and V _Y / V _Z is the vertical axis. When there is a specific moving object, V _X / V _Z and V _Y / V _Z are common velocity eigenvalues for the pixel group corresponding to the moving object, and Expression (15) is satisfied. Here, Equation (15) corresponds to Equation (12) described above, and the speed eigenvalues V _X / V _Z and V _Y / V _Z correspond to the speed eigenvalues S and T. From this, the above-described equation (13) can be converted into the following equation (16).

In other words, in the second modified example, the pixel positions x and y are translated using the velocity eigenvalues V _X / V _Z and V _Y / V _Z for each image region with different motion. The movement vector (u ′, v ′) is expressed by the following formula (17).

  The movement vector (u ′, v ′) in the above equation (17) has the same form as the movement vector (u, v) represented by the above equation (5). Even in a moving situation, the processing by the image processing apparatus 20 can be suitably performed. Specifically, the posture determination process of the object to be photographed shown in FIG. 5 described above can be suitably performed.

In addition to the processing described above, the image processing apparatus 20 multiplies the velocity eigenvalue S (= V _X / V _Z ) by the velocity Vz in the camera optical axis direction, thereby causing the velocity V _{X in} the lateral direction of the moving object. May be calculated. Further, the image processing apparatus 20 may calculate the vertical velocity V _Y of the moving object by multiplying the velocity eigenvalue T (= V _Y / V _Z ) by the velocity Vz in the camera optical axis direction.

1 is a block configuration diagram of an image processing apparatus according to an embodiment. It is an image showing an example of an optical flow It is a figure for demonstrating the partial differentiation process of a movement vector. It is a flowchart of the process by an image processing apparatus. It is a flowchart of the attitude | position determination process of a to-be-photographed object. It is a figure for demonstrating the principle of the attitude | position determination process of a to-be-photographed object. It is a flowchart including the calculation process of an actual speed and an actual distance. It is a figure for demonstrating calculation of a speed eigenvalue. It is a flowchart including the pre-processing of image data.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Camera, 20 ... Image processing apparatus, 22 ... Image acquisition part, 24 ... Optical flow calculation part, 26 ... Partial differentiation part, 28 ... Attitude determination part, 30 ... Actual speed calculation part, 32 ... Real distance calculation part, 34 ... Pre-processing part.

Claims (10)

Image acquisition means for acquiring a plurality of image data obtained at different times;
Optical flow calculation means for calculating a movement vector based on the plurality of image data;
Partial differentiation means for partial differentiation of the movement vector calculated by the optical flow calculation means in a predetermined direction to calculate a partial differentiation value of the movement vector;
Posture determination means for determining the posture of the surface constituting the object to be photographed based on the partial differential value calculated by the partial differential means;
An image processing apparatus comprising:   The posture determination means includes a partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the vertical direction, a partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the horizontal direction, and the vertical component of the movement vector. Based on the combination of the partial differential value obtained by partial differentiation in the vertical direction and the partial differential value obtained by partial differentiation of the vertical component of the movement vector in the horizontal direction, the posture of the surface constituting the object is determined. The image processing apparatus according to claim 1, wherein: The posture determination means includes
The partial differential value obtained by partial differential of the horizontal component of the movement vector in the vertical direction is substantially 0,
The partial differential value obtained by partial differentiation of the vertical component of the movement vector in the horizontal direction is approximately 0,
When the partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the horizontal direction and the partial differentiation value obtained by partial differentiation of the vertical component of the movement vector in the vertical direction are substantially the same,
The image processing apparatus according to claim 1, wherein a surface constituting the object to be photographed is determined to be perpendicular to the optical axis. The posture determination means includes
The partial differential value obtained by partial differential of the horizontal component of the movement vector in the vertical direction is substantially 0,
When the partial differential value obtained by partial differentiation of the horizontal component of the movement vector in the horizontal direction is approximately twice the partial differentiation value obtained by partial differentiation of the vertical component of the movement vector in the vertical direction,
The image processing apparatus according to claim 1, wherein the image forming apparatus determines that the surface constituting the object to be photographed is parallel to the optical axis and the vertical direction of the image data. The posture determination means includes
The partial differential value obtained by partial differentiation of the vertical component of the movement vector in the horizontal direction is approximately 0,
When the partial differential value obtained by partial differentiation of the vertical component of the movement vector in the vertical direction is approximately twice the partial differentiation value obtained by partial differentiation of the horizontal component of the movement vector in the horizontal direction,
The image processing apparatus according to claim 1, wherein the surface constituting the object to be photographed is determined to be parallel to the optical axis and the horizontal direction of the image data.   Data on the height of the shooting position of the image data is stored, and a relative speed between the shooting position of the image data and the object to be shot is calculated based on the height of the shooting position and the movement vector. The image processing apparatus according to claim 1, further comprising a speed calculation unit.   The apparatus further comprises distance calculation means for calculating a distance from the shooting position of the image data to the object to be shot based on the relative speed calculated by the speed calculation means and the movement vector. Item 7. The image processing apparatus according to Item 6. Before the optical flow calculation means calculates the movement vector based on the image data, it is a preprocessing means for processing the image data, and the image region corresponding to the moving object is represented by the velocity eigenvalue V _X / Before translation using V _Z and V _Y / V _Z (where V _X is the lateral velocity of the moving object, V _Y is the vertical velocity of the moving object, and V _Z is the velocity of the moving object in the optical axis direction). The image processing apparatus according to claim 1, further comprising a processing unit. Said pre-processing means divides each image area rate eigenvalue _V X / _{V Z} and _V Y / _{V Z} in common, segmented for each image area rate eigenvalue _V X / _{V Z} and _V Y / _{V Z} The image processing apparatus according to claim 8, wherein the image processing apparatus is used for parallel translation. An image processing method for determining a posture of a surface constituting a subject,
An image acquisition step of acquiring a plurality of image data obtained at different times;
An optical flow calculating step of calculating a movement vector based on the plurality of image data;
A partial differentiation step of partial differentiation of the movement vector in a predetermined direction to calculate a partial differentiation value of the movement vector;
A posture determination step of determining a posture of a surface constituting the object based on the partial differential value.

Images