JP2006134035A - Moving object detecting device and moving object detecting method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving object detecting device which extracts a three-dimensional object existing in a monitoring area and computes the three-dimensional movement of the three-dimensional object by using an optical flow to accurately detect the moving object in the monitoring area, and also provide a moving object detecting method. <P>SOLUTION: The moving object detecting device 1 is provided with; a stereo camera 2 which picks up a monitoring area and outputs a pair of images in time series; a stereo image processing part 6 which computes distance data from a pair of images; a recognition part 10 which recognizes, from the distance data, a three-dimensional object On existing in the monitoring area; a detection part 11 which detects optical flows Δi, Δj and Δd from either a plurality of time-series images, of the plural pairs of images and from the distance data; a computation part 12 which computes the three-dimensional movement component flows Fmv_i, Fmv_j and Fmv_d of the three-dimensional object On; and a determination part 13 which determines a moving object and a static object based on the three-dimensional movement component flows. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a moving object detection apparatus and a moving object detection method, and more particularly to a moving object detection apparatus and a moving object detection method for detecting a three-dimensional object from a monitoring area and discriminating between a moving object and a stationary object.

  Conventionally, a stereo camera has been installed in a vehicle, a movable robot, etc. to capture the scenery in the monitoring area, and based on the information obtained, the outside world recognizes the situation in the monitoring area three-dimensionally A monitoring device is attracting attention (see, for example, Patent Document 1).

  By using such an external environment monitoring device, for example, in a vehicle such as an automobile, in order to detect a vehicle or an obstacle from the front or side monitoring area of the own vehicle and avoid a collision with them, It is possible to issue an alarm or automatically operate the brake of the host vehicle to decelerate and stop (see, for example, Patent Documents 2 to 8).

  In addition, for example, if such an external monitoring device is mounted on a robot, it is possible to determine that contact with an object detected in the monitoring region is avoided or that the object is closely observed while facing the object. . Furthermore, for example, when such an external environment monitoring device is mounted on a pan / tilt type surveillance camera, an object moving within the monitoring area is detected and the object is tracked or zoomed up and recorded. Is also possible.

As a method for recognizing the outside world by the outside world monitoring device, a method for obtaining a three-dimensional optical flow of an object as described in Patent Document 1 is known. In this method, a distance image corresponding to a three-dimensional vector of each point in the input image is generated based on a pair of input images captured by a plurality of cameras arranged at predetermined intervals, that is, a stereo camera. , Generate a two-dimensional optical flow based on input images captured at different times by the same camera, generate a three-dimensional optical flow based on the distance image and the two-dimensional optical flow, and generate each point in the input image. 3D movement is detected.
JP 2001-84383 A Japanese Patent Laid-Open No. 5-114099 JP-A-5-265547 JP-A-6-266828 JP-A-10-283461 Japanese Patent Laid-Open No. 10-283477 Japanese Patent Laid-Open No. 11-213138 JP 2001-92970 A

  However, in the method described in the above-mentioned patent document, since the three-dimensional optical flow is only calculated for each point in the input image from the stereo camera, a three-dimensional object is extracted from the scenery in the monitoring area. It has not been extracted, and the optical flow and the movement of the object are not necessarily clearly associated with each other.

  In the above method, only the three-dimensional optical flow based on the relative three-dimensional motion between the robot on which the stereo camera is mounted and the object is detected, so that when the robot itself is moving, for example, on the ground On the other hand, an object that is stationary is observed if it is moving relative to the robot, and a moving object and a stationary object cannot be distinguished.

  Recognizing a moving object and a stationary object by distinguishing them is, for example, when a traveling vehicle suddenly decelerates when a traveling vehicle suddenly decelerates, or a vehicle that has changed lanes or suddenly moved forward. This is an indispensable technique for avoiding a collision with a pedestrian who has jumped out. Further, for example, it is a technique that is also required for a self-propelled robot or the like.

  Further, in the pan / tilt type surveillance camera described above, it is observed that both the stationary object and the moving object are moved by the pan / tilt operation of the camera itself. Therefore, if the moving object and the stationary object are not distinguished and recognized, all objects in the monitoring area including the stationary object must be monitored, and the monitoring efficiency is reduced. It becomes impossible to perform more advanced monitoring.

  Therefore, the present invention extracts a three-dimensional object existing in the monitoring area, calculates a three-dimensional movement of the three-dimensional object using an optical flow, and can accurately detect a moving object in the monitoring area. An object of the present invention is to provide a detection device and a moving object detection method.

In order to solve the above problem, the moving object detection device according to claim 1,
A stereo camera that captures a landscape including a monitoring area and outputs a pair of images in time series;
A stereo image processing unit that calculates distance data by stereo matching based on the pair of images;
A recognition unit for recognizing a three-dimensional object existing in the monitoring area based on the distance data;
A detection unit that detects an optical flow based on a plurality of images arranged in time series and the distance data, with one image of the pair of images as a processing target;
A calculation unit that calculates a three-dimensional moving component flow of the three-dimensional object based on the distance data and the optical flow;
And a determination unit configured to determine a moving object and a stationary object based on the three-dimensional movement component flow.

  According to the first aspect of the present invention, the stereo image processing unit calculates the distance data based on the pair of images obtained in time series by imaging the monitoring area with the stereo camera, and based on the obtained distance data. The recognition unit recognizes a three-dimensional object existing in the monitoring area, while the detection unit detects an optical flow from one of a pair of images arranged in a time series and distance data, and the calculation unit A three-dimensional movement component flow that is a movement component of the three-dimensional object with respect to the ground is calculated using the optical flow and distance data for the recognized three-dimensional object, and the three-dimensional object is determined as a moving object by the determination unit based on the three-dimensional movement component flow. Determine a stationary object.

  According to a second aspect of the present invention, in the moving object detection device according to the first aspect, the calculation unit includes the distance data of the three-dimensional object specified by the distance data and the position of the three-dimensional object on the image. And a three-dimensional moving component flow of the three-dimensional object is calculated based on a change amount of the position of the three-dimensional object specified by the optical flow on the image and a change amount of the distance data with time. And

  According to the invention described in claim 2, the calculation unit assigns the distance data and the optical flow that is the positional and temporal change amount on the image to the solid object recognized by the recognition unit, and based on them The three-dimensional moving component flow of the three-dimensional object is calculated.

The invention according to claim 3 is the moving object detection device according to claim 1 or 2,
Speed detecting means for detecting the speed of the own machine equipped with the stereo camera;
Yaw rate detecting means for detecting the yaw rate of the own machine,
The calculation unit calculates an apparent component flow from the speed of the own device, the yaw rate of the own device, the distance data of the three-dimensional object, and the position on the image of the three-dimensional object, and the optical flow and the apparent A three-dimensional moving component flow is calculated from the difference between the component flows.

  According to the third aspect of the present invention, the calculation unit uses the speed, yaw rate, and distance data detected by the speed detection unit and the yaw rate detection unit, and is observed as the mobile unit moves. That is, the apparent component flow is calculated, and the apparent component flow is subtracted from the optical flow to calculate a three-dimensional moving component flow that is a substantial moving component of the three-dimensional object with respect to the ground.

According to a fourth aspect of the present invention, in the moving object detection device according to any one of the first to third aspects,
The detection unit is configured to detect the optical flow for each pixel block having a predetermined area constituting the one image.
The calculation unit includes:
A primary calculation unit for calculating the three-dimensional movement component flow for each pixel block belonging to the three-dimensional object;
A map generation unit that generates a movement component flow map indicating a distribution of each movement component flow of the three-dimensional movement component flow calculated for the pixel block belonging to the three-dimensional object;
Based on the generated moving component flow map, a separation unit that separates pixel block groups in which the difference in the moving component flow is larger than a predetermined threshold among the pixel block groups belonging to the same three-dimensional object as different three-dimensional objects. It is characterized by having.

  According to the fourth aspect of the present invention, the calculation unit uses the distribution of the three-dimensional movement component flow of the pixel block belonging to the three-dimensional object, and the other part of the three-dimensional object and the value of the movement component flow are more than a predetermined threshold A greatly different pixel block group is separated as a three-dimensional object different from the original three-dimensional object.

The invention according to claim 5 is the moving object detection device according to any one of claims 1 to 4,
The detection unit is configured to detect the optical flow for each pixel block having a predetermined area constituting the one image.
The calculation unit includes:
A primary calculation unit for calculating the three-dimensional movement component flow for each pixel block belonging to the three-dimensional object;
A map generation unit that generates a movement component flow map indicating a distribution of each movement component flow of the three-dimensional movement component flow calculated for the pixel block belonging to the three-dimensional object;
Based on the distance data and the generated moving component flow map, a difference in moving component flow between pixel blocks belonging to each of the three-dimensional object and another three-dimensional object adjacent to the three-dimensional object is equal to or less than a predetermined threshold value; When the difference between the respective parallaxes is minimum and not more than a predetermined threshold value, an integration unit is provided that integrates the three-dimensional object and the other three-dimensional object into one solid object. .

  According to the fifth aspect of the present invention, the calculation unit calculates values of adjacent three-dimensional objects whose values are close to each other, that is, their parallax, that is, the distance from the stereo camera and the three-dimensional moving component flow are equal to or less than a set threshold. If it has, it judges that it is the same solid object, and integrates it into one solid object.

A sixth aspect of the present invention is the moving object detection device according to any one of the first to fifth aspects,
The calculation unit is configured to calculate a three-dimensional representative movement component flow of a pixel block group belonging to the three-dimensional object,
The determination unit is configured to determine, as a moving object, the three-dimensional object in which at least one of the movement component flows of the three-dimensional representative movement component flow exceeds a predetermined threshold.

  According to the invention described in claim 6, the calculation unit obtains a three-dimensional representative movement component flow that is one three-dimensional movement component flow representing the three-dimensional object from the three-dimensional movement component flow of each pixel block belonging to the three-dimensional object. The determination unit uses the three-dimensional representative moving component flow to calculate that the three-dimensional object is a moving object when any of the three moving component flows exceeds a predetermined threshold value. judge.

  The invention according to claim 7 is the moving object detection device according to claim 6, wherein the three-dimensional representative moving component flow is calculated for each pixel block in a pixel block group belonging to the three-dimensional object. It is configured to be calculated by statistical processing of the moving component flow.

  According to the seventh aspect of the present invention, the calculation of the three-dimensional representative moving component flow according to the sixth aspect is performed by calculating a mode value or an average value for the three-dimensional moving component flow of each pixel block belonging to the three-dimensional object. It is calculated by processing.

The invention according to claim 8 is the moving object detection device according to claim 3,
A speed calculator that calculates the speed of the aircraft based on the optical flow and the distance data related to the stationary object determined based on the three-dimensional moving component flow by the determination unit;
And a speed correction unit that corrects the speed detected by the speed detection unit based on the speed of the own device calculated by the speed calculation unit.

  According to the invention described in claim 8, the speed calculation unit calculates the speed of the own device from the optical flow of the stationary object, and the speed correction unit uses the calculated speed of the own device to detect the speed. Correct the speed detected in.

The invention according to claim 9 is a moving object detection method,
Distance data calculation processing for calculating distance data by stereo matching based on a pair of images obtained by capturing a scene including a monitoring area with a stereo camera capable of capturing images in time series;
A three-dimensional object recognition process for recognizing a three-dimensional object existing in the monitoring area based on the distance data;
Optical flow detection processing for detecting an optical flow based on a plurality of images arranged in time series and the distance data, with one image of the pair of images as a processing target;
A three-dimensional movement component flow calculation process for calculating a three-dimensional movement component flow of the three-dimensional object based on the distance data and the optical flow;
A determination process for determining a moving object and a stationary object based on the three-dimensional moving component flow is performed.

  According to the ninth aspect of the present invention, distance data is calculated based on a pair of images obtained in time series by imaging a monitoring area with a stereo camera, and within the monitoring area based on the obtained distance data. Recognize existing three-dimensional object, detect optical flow from distance data using multiple images of one of a pair of images in time series, and use optical flow and distance data for recognized three-dimensional object A three-dimensional movement component flow that is a movement component of the three-dimensional object with respect to the ground is calculated, and the three-dimensional object determines a moving object and a stationary object based on the three-dimensional movement component flow.

  According to the first aspect of the present invention, the stereo image processing unit calculates the distance data based on the pair of images obtained in time series by imaging the monitoring area with the stereo camera, and based on the obtained distance data. The recognition unit recognizes a three-dimensional object existing in the monitoring area, while the detection unit detects an optical flow from one of a pair of images arranged in a time series and distance data, and the calculation unit A three-dimensional movement component flow that is a movement component of the three-dimensional object with respect to the ground is calculated using the optical flow and distance data for the recognized three-dimensional object, and the three-dimensional object is determined as a moving object by the determination unit based on the three-dimensional movement component flow. Determine a stationary object.

  Therefore, the optical flow can be accurately associated with the three-dimensional object, and the three-dimensional movement component flow of the three-dimensional object indicating whether or not the three-dimensional object is moving with respect to the ground can be accurately calculated. It is possible to accurately detect a moving object from within. In addition, since the three-dimensional object is recognized using the distance data, it is possible to recognize the three-dimensional object above the road surface and accurately recognize even if the area of the three-dimensional object displayed on the image is small. Become. Furthermore, it is possible to efficiently recognize a three-dimensional object without erroneously recognizing a shadow or the like on the road as a three-dimensional object.

  According to the invention described in claim 2, the calculation unit assigns the distance data and the optical flow that is the positional and temporal change amount on the image to the solid object recognized by the recognition unit, and based on them The three-dimensional moving component flow of the three-dimensional object is calculated. Therefore, in addition to the effect of the invention described in claim 1, it is possible to calculate a three-dimensional movement component flow that is a movement component of the three-dimensional object with respect to the ground more accurately, and whether or not the three-dimensional object is moving. It is possible to perform the determination more accurately.

  According to the third aspect of the present invention, the calculation unit uses the speed, yaw rate, and distance data detected by the speed detection unit and the yaw rate detection unit, and is observed as the mobile unit moves. By calculating the component flow and subtracting the apparent component flow from the optical flow, a three-dimensional moving component flow that is a moving component of the three-dimensional object with respect to the ground is calculated. Therefore, in addition to the effects of the inventions described in the above claims, the three-dimensional moving component flow that is a moving component of the three-dimensional object with respect to the ground without being affected by the movement of the own device on which the moving object detecting device is mounted Therefore, it is possible to more accurately determine whether or not the three-dimensional object is moving.

  According to the fourth aspect of the present invention, the calculation unit uses the distribution of the three-dimensional movement component flow of the pixel block belonging to the three-dimensional object, and the other part of the three-dimensional object and the value of the movement component flow are more than a predetermined threshold A greatly different pixel block group is separated as a three-dimensional object different from the original three-dimensional object. Therefore, in addition to the effects of the inventions described in the above claims, the three-dimensional object recognition using the distance data that are close to each other, such as pedestrians walking in front of a stopped vehicle, is recognized as the same three-dimensional object. The plurality of three-dimensional objects can be accurately separated based on the moving component flow. In addition, since the separation is performed using the moving component flow, in particular, it is possible to effectively separate the moving objects or the moving object and the stationary object, and more accurately detect the moving object in the monitoring region. This makes it possible to detect a moving object with higher accuracy.

  According to the fifth aspect of the present invention, the calculation unit calculates values of adjacent three-dimensional objects whose values are close to each other, that is, their parallax, that is, the distance from the stereo camera and the three-dimensional moving component flow are equal to or less than a set threshold. If it has, it judges that it is the same solid object, and integrates it into one solid object. For example, when a three-dimensional object is close to a stereo camera, a large three-dimensional object may be detected in the monitoring area, and a single three-dimensional object may be recognized as a plurality of three-dimensional objects. In addition to the effects of the inventions described in the above claims, the three-dimensional object can be more accurately recognized by integrating the adjacent three-dimensional objects in which the distance and the three-dimensional moving component flow are very close to each other. A moving object can be detected with high accuracy, and a moving object in the monitoring area can be detected more accurately. Further, when the moving object is to be monitored, the number of monitoring targets can be further limited.

  According to the invention described in claim 6, the calculation unit obtains a three-dimensional representative movement component flow that is one three-dimensional movement component flow representing the three-dimensional object from the three-dimensional movement component flow of each pixel block belonging to the three-dimensional object. The determination unit uses the three-dimensional representative moving component flow to calculate that the three-dimensional object is a moving object when any of the three moving component flows exceeds a predetermined threshold value. judge. Therefore, in addition to the effects of the inventions described in the above claims, the determination unit performs determination based on one representative moving component flow without determining whether each pixel block belonging to the three-dimensional object moves or stops. And the processing speed can be further improved.

  According to the seventh aspect of the present invention, the calculation of the three-dimensional representative moving component flow according to the sixth aspect is performed by calculating a mode value or an average value for the three-dimensional moving component flow of each pixel block belonging to the three-dimensional object. In order to calculate by performing processing, in addition to the effect of the invention of claim 6, in addition to the effect of the invention, the influence of errors caused by measurement errors included in each three-dimensional moving component flow of each pixel block belonging to a three-dimensional object is statistically calculated. It can be removed by the process, and it is possible to more accurately determine whether or not the three-dimensional object is moving.

  According to the invention described in claim 8, the speed calculation unit calculates the speed of the own device from the optical flow of the stationary object, and the speed correction unit uses the calculated speed of the own device to detect the speed. Correct the speed detected in. At this time, if the three-dimensional moving component flow for a stationary object is regarded as 0, the optical flow matches the apparent component flow observed with the movement of the own device. The speed of the aircraft is calculated from the optical flow, and the speed detected by the speed detecting means is corrected using the calculated speed. Therefore, in addition to the effect of the invention described in claim 3, it is possible to more accurately calculate the three-dimensional moving component flow of the three-dimensional object that is a criterion for determining the movement and stationary of the three-dimensional object. It is possible to further improve the accuracy of discrimination between a moving object and a stationary object in the detection apparatus or the moving object detection method, and to make the apparatus or method more reliable.

  According to the ninth aspect of the present invention, distance data is calculated based on a pair of images obtained in time series by imaging a monitoring area with a stereo camera, and within the monitoring area based on the obtained distance data. Recognize existing 3D object, detect optical flow from multiple images and distance data of one of a pair of images arranged in time series, and use optical flow and distance data for recognized 3D object A three-dimensional movement component flow that is a movement component of the three-dimensional object with respect to the ground is calculated, and the three-dimensional object determines a moving object and a stationary object based on the three-dimensional movement component flow.

  Therefore, the optical flow can be accurately associated with the three-dimensional object, and the three-dimensional movement component flow indicating whether or not the three-dimensional object is moving with respect to the ground is accurate without being influenced by the movement of the own aircraft. It is possible to calculate well, and a moving object can be accurately detected from the monitoring area. In addition, since the three-dimensional object is recognized using the distance data, it is possible to recognize the three-dimensional object above the road surface and accurately recognize even if the area of the three-dimensional object displayed on the image is small. Become. Furthermore, it is possible to efficiently recognize a three-dimensional object without erroneously recognizing a shadow or the like on the road as a three-dimensional object.

  Hereinafter, embodiments of the moving object detection device and the moving object detection method of the present invention will be described with reference to the drawings.

  The moving object detection device and the moving object detection method according to the present invention are mounted on a vehicle, a robot, a pan / tilt type monitoring camera, and the like, and are applied to monitoring a traveling environment in front, monitoring for security, and the like. In each of the following embodiments, a case where the moving object detection device is mounted on a vehicle may be exemplified as a specific example, but the present invention is not limited thereto. In order to distinguish between a moving object detection device and a vehicle, robot, surveillance camera, or the like on which the moving object detection device is mounted, the vehicle, robot, monitoring camera, etc. on which the moving object detection device is mounted will be referred to as the own device.

[First Embodiment]
FIG. 1 is a block diagram of the moving object detection device according to the first embodiment. The moving object detection device 1 mainly includes a stereo camera 2, a stereo image processing unit 6, a recognition unit 10, and a detection unit 11. , An image analysis unit 9 having a calculation unit 12 and a determination unit 13, a yaw rate detection unit 14, and a speed detection unit 15.

  The stereo camera 2 is composed of a main camera 3a and a sub camera 3b each incorporating a synchronized image sensor such as a CCD or CMOS sensor, and each of the main camera 3a and the sub camera 3b is a front monitor. A scene including a region is imaged, and a pair of images is output to the A / D converters 4a and 4b in a time series at regular time intervals. At this time, the image captured by the main camera 3a is used as a reference image that serves as a reference for performing stereo image processing, and the image captured by the sub camera 3b is used as a comparative image.

  The A / D converters 4a and 4b convert the analog reference image and the comparison image output from the main camera 3a and the sub camera 3b into digital data with predetermined luminance gradations such as 256 gray scales, respectively. It is configured.

  The image correction unit 5 performs affine transformation or the like on the pair of images output from the A / D converters 4a and 4b, such as displacement due to an error in the mounting position of the pair of cameras 3a and 3b and correction of luminance. It is configured to be used. The pair of image data subjected to the image correction in this manner is stored in the image data memory 8. As described above, images are taken at regular time intervals by the main camera 3a and the sub-camera 3b, but the image data memory 8 contains images of images formed at regular time intervals and subjected to correction or the like. Digital data is stored in time series.

  Here, digital data of a pair of images, that is, digital data of a reference image (hereinafter referred to as reference image data) obtained by the main camera 3a and obtained through each of the processes, and each of the images taken by the sub camera 3b. Digital data (hereinafter referred to as comparative image data) of the comparative image obtained through the processing is a set of luminance values of 0 to 255 for each pixel. Specifically, when the lower left corner of the image plane defined by the reference image data and the comparison image data is the origin, the horizontal direction is the i coordinate axis, and the vertical direction is the j coordinate axis, the image data is each pixel of the image plane. It is given as a set of luminance values pnij obtained for each (i, j). In the case of reference image data, the luminance value is represented by p1ij, and in the case of comparison image data, the luminance value is represented by p2ij.

  The stereo image processing unit 6 is configured to calculate distance data by stereo matching based on the reference image data and the comparison image data output from the A / D converters 4a and 4b.

  Here, the distance data is a set of parallax d calculated for each pixel block having a predetermined area of 4 × 4 pixels, which constitutes a part of an image defined by the reference image data. Further, the parallax refers to a relative displacement amount in the horizontal direction with respect to the mapping position of the same object in the reference image and the comparison image, which are derived from a certain distance between the main camera 3a and the sub camera 3b. The main camera 3a and the sub camera The distance from the central position 3b to the object and the parallax d are uniquely associated based on the principle of triangulation.

  In this embodiment, stereo matching in the stereo image processing unit is performed by first dividing a reference image composed of 512 × 200 pixels into 128 × 50 pixels by a 4 × 4 pixel block PBij as described above, and comparing them. The image is divided into strip-shaped portions having a width of 4 pixels extending in the i direction, and then the pixel block PBij on the reference image is formed on the corresponding strip-shaped portion on the comparison image, that is, on the epipolar line. The pixel block of 4 × 4 pixels on the comparison image that is judged to have the highest correlation while searching the i-direction pixel by pixel within the search range is searched.

  The correlation between the pixel block PBij on the reference image and the pixel block on the comparison image is evaluated, for example, by calculating a city block distance CB obtained by the following equation (1).

  Here, as described above, p1ij and p2ij are the luminance values of each pixel (i, j) of the reference image data and the comparison image data, and CB is the absolute value of the difference in luminance values for 16 pixels in the pixel block. Is the total. It is judged that the correlation between the pixels is higher as the difference between the luminance values of the pixels is smaller, and that the correlation between the two pixel blocks is higher as the sum of the differences is smaller.

  The stereo image processing unit 6 searches the epipolar line of the comparison image, determines that the pixel block having the calculated value of the city block distance CB is the smallest correlation destination, and the deviation from the pixel block PBij on the reference image The amount is configured to be calculated as the parallax d. In addition, about the structure for calculating | requiring the city block distance CB etc., it is explained in full detail in the said patent document 2 submitted previously by this applicant.

  The distance data calculated by the stereo image processing unit 6, that is, the set of parallax d calculated for each pixel block PBij on the reference image is stored in the distance data memory 7. As in the case of the image data memory 8 described above, distance data is transmitted to the distance data memory 7 from the stereo image processing unit 6 at a constant time interval, but is transmitted to the distance data memory 7. Coming distance data is stored in time series.

  The image analysis unit 9 is composed of a microcomputer in which a CPU, a ROM, a RAM, an input / output interface and the like (not shown) are connected to a bus, and when viewed functionally, a recognition unit 10, a detection unit 11, and a calculation unit 12 And a determination unit 13. The image analysis unit 9 is connected to a yaw rate detection unit 14 and a speed detection unit 15, and the yaw rate detection unit 14 and the speed detection unit 15 send the detected yaw rate ω and speed V to the image analysis unit 9. It is configured to output.

  The recognition unit 10 is configured to read the distance data calculated by the stereo image processing unit 6 and stored in the distance data memory 7, and to recognize the three-dimensional object On existing in the monitoring area based on the distance data.

  Specifically, the recognition unit 10 calculates a position in the real space for each pixel block from the value of the parallax d calculated for each pixel block of the distance data Dt read from the distance data memory 7, and Based on the model, if the position is above the road surface compared to the height of the road surface corresponding in position in the lateral direction and the parallax direction, the data of the position is extracted as data representing a three-dimensional object It is supposed to be. The parallax direction has the same meaning as a direction in which an object moves away from the center position of the main camera 3a and the sub camera 3b, that is, a so-called distance direction.

  The recognizing unit 10 collects the three-dimensional object data in which the positions of the data in the horizontal direction and the parallax direction are close to each other in the obtained three-dimensional object data, and the directionality of the arrangement of the group of data greatly changes. The group is appropriately divided into parts, and the three-dimensional object data group arranged mainly in the horizontal direction is grouped as the “front” of the three-dimensional object On, and the three-dimensional object data group arranged mainly in the parallax direction is used as the “side surface” of the three-dimensional object On. By grouping, the three-dimensional object On is recognized three-dimensionally in the real space. The recognition of the three-dimensional object On by the recognition unit 10 is described in detail in Patent Documents 3 to 8 previously submitted by the present applicant. In Patent Documents 3 to 8, “front” of the three-dimensional object On is expressed as “object”, and “side” is expressed as “side wall”.

  In the present embodiment, the recognizing unit 10 defines the portion corresponding to the “front” of the three-dimensional object On that is three-dimensionally recognized in the real space as shown in FIG. The image is converted into a region on the image plane, and the region is recognized as a three-dimensional object On. The recognizing unit 10 also includes a parallax dn representing a group represented by an i-coordinate inL at the left end, an i-coordinate inR at the right end, and an average value or a mode value of parallax of each three-dimensional object data in the group. Are associated with the three-dimensional object On and sent to the calculation unit 12 as data of the three-dimensional object On. Note that the three-dimensional object data related to the “side surface” of the three-dimensional object On can be used for the following analysis.

  The detection unit 11 uses the reference image as a processing target, and based on a plurality of reference images moving back and forth in time series and distance data corresponding thereto, the pixel block PBij on the previous reference image in time series is time-series. The optical flow is detected by searching for a position on the later reference image. In the present embodiment, reference images for two frames are used as a plurality of reference images.

  Specifically, the detection unit 11 includes three sub-blocks for imaging, and the reference image data It for the reference image stored in the image data memory 8 and the reference image data It in time series. The reference image data It-1 formed from the image captured at the imaging timing is read out, and based on them, the positional change amount on the reference image, that is, the i direction and the j direction, in two sub-blocks The optical flows Δi and Δj are respectively detected.

  In the third sub-block of the detection unit 11, distance data Dt and Dt-1 corresponding to the reference image data It and It-1 are read from the distance data memory 7 and correspond to the pixel block PBij on the reference image. The position parallax dt is read from the distance data Dt, the position parallax dt-1 corresponding to the previous pixel block in time series is read from the distance data Dt-1, and the amount of change in the distance data with time is calculated from the difference. That is, the optical flow Δd in the parallax direction is detected.

  The detected optical flows Δi, Δj, Δd are sent to the calculation unit 12 in association with each pixel (i, j) on the image plane. Note that “previous in time series” is not necessarily the immediately preceding frame, and may be a reference image several frames before.

  The calculation unit 12 identifies the pixel block PBij on the reference image belonging to each solid object On based on the data inL, inR, dn of each solid object On transmitted from the recognition unit 10 and transmits from the detection unit 11 The three-dimensional moving component flow of each three-dimensional object On is calculated on the basis of the optical flows Δi, Δj, Δd of the pixel blocks PBij thus obtained.

  Here, the optical flows Δi, Δj, and Δd are apparent component flows Fst_i, Fst_j, and Fst_d that are observed with respect to a stationary object during the movement of the aircraft itself, and substantial moving components of the three-dimensional object with respect to the ground. Since it is represented by the sum of the three-dimensional moving component flows Fmv_i, Fmv_j, and Fmv_d, the three-dimensional moving component flow is expressed by the following equations (2) to (4).

  The apparent component flows Fst_i, Fst_j, and Fst_d are obtained from the following formulas (5) to (7) using the yaw rate ω and the speed V of the own machine input from the yaw rate detection means 14 and the speed detection means 15, respectively. It is done.

  In the above equations, f represents the focal length of the main camera 3a and the sub camera 3b, L represents the distance between the main camera 3a and the sub camera 3b, and Wi represents the imaging element pitch in the i direction. In the above equations, i and j are values converted to i and j coordinates when the origin is an infinite point in the reference image.

  The calculation unit 12 is configured to calculate the three-dimensional movement component flows Fmv_i, Fmv_j, and Fmv_d for all the pixel blocks PBij belonging to the three-dimensional object On. In the present embodiment, the calculation unit 12 includes the three-dimensional movement component flow. The mode value is obtained for each directional component flow of the flow, and the three-dimensional representative moving component flows Fni, Fnj, Fnd of the pixel block group belonging to the three-dimensional object On are obtained. The three-dimensional representative moving component flows Fni, Fnj, and Fnd are obtained for each solid object On, and the determination unit 13 together with the i coordinates inL and inR of the left and right ends of the solid object On and the parallax dn representing the solid object On. It is supposed to be sent. The three-dimensional representative movement component flows Fni, Fnj, and Fnd are calculated not only by the mode value of the three-dimensional movement component flow but also by other statistical processing such as, for example, averaging the three-dimensional movement component flow. It is also possible to configure.

  Based on the three-dimensional representative movement component flows Fni, Fnj, and Fnd of the pixel block group belonging to each solid object On transmitted from the calculation unit 12, the determination unit 13 moves or stops each solid object On. That is, it is configured to determine a moving object and a stationary object.

  Specifically, the determination unit 13 compares the directional component flows Fni, Fnj, and Fnd of the three-dimensional representative moving component flow of the pixel block group belonging to each three-dimensional object On with a predetermined threshold value, and determines the three directional components. If even one of the flows exceeds the threshold, the three-dimensional object On is determined to be a moving object. Conversely, when all of the three directional component flows Fni, Fnj, and Fnd are equal to or less than the threshold value, the three-dimensional object On is determined to be a stationary object.

  Next, a moving object detection method using the moving object detection device of this embodiment will be described.

  FIG. 3 is a flowchart showing a processing procedure in the moving object detection method of the present embodiment. In the moving object detection method of the present embodiment, first, distance data calculation processing is performed, and in the distance data calculation processing, a pair of images obtained by capturing a scene including a monitoring area with a stereo camera capable of capturing images in time series. Based on the image, distance data is calculated by stereo matching (step S1). Subsequently, in the three-dimensional object recognition process, the three-dimensional object On existing in the monitoring area is recognized based on the distance data obtained in the distance data calculation process (step S2).

  The distance data calculation process and the three-dimensional object recognition process are as described in the description of the stereo image processing unit 6 of the moving object detection device 1 and the recognition unit 10 of the image analysis unit 9, and these processes are described above. As described in detail in Patent Documents 2 to 8, the description is omitted here.

  Next, an optical flow detection process is performed in parallel with the three-dimensional object recognition process (step S3). In the optical flow detection process, the detection unit 11 of the image analysis unit 9 converts a plurality of images and distance data arranged in time series using a reference image as a processing target among a pair of images obtained by imaging with a stereo camera. Based on this, optical flows Δi, Δj, Δd are detected for each pixel block PBij.

  In this optical flow detection process, as described above, first, the reference image data It and the reference image data It-1 formed from the image captured at the imaging timing before the reference image data It in time series are set to i. The optical flows Δi and Δj in the direction and the j direction are detected, respectively, and the optical flow Δd in the parallax direction is detected from the distance data Dt and Dt-1 corresponding to the reference image data It and It-1.

  Specifically, in the detection of the optical flows Δi and Δj, unlike the stereo matching process in the distance data calculation process described above, for example, a 4 × 4 pixel block PBij on the reference image It is time-sequentially. Then, a pixel block of 4 × 4 pixels on the reference image It-1 that is judged to have the highest correlation is searched while shifting in the i direction or j direction pixel by pixel over the entire area of the previous reference image It-1.

  In this case, the correlation can be evaluated by, for example, calculating the city block distance CB obtained by the equation (1). In this way, the pixel block PBij on the reference image It is associated with the pixel block on the reference image It-1, and the optical flows Δi and Δj are obtained from the difference between the i coordinate and the j coordinate.

  Further, as described above, the optical flow Δd is obtained by reading the parallax dt at the position corresponding to the pixel block PBij on the reference image It from the distance data Dt, and the most correlated on the previous reference image It-1 in time series. The parallax dt-1 at the position corresponding to the pixel block determined to be high is read from the distance data Dt-1, and the difference is obtained and detected.

  At that time, as shown in FIG. 4, the pixel block PBij on the reference image It-1 that is the correlation destination of the pixel block PBij on the reference image It is divided into the maximum four pixel blocks PBij. It may come to exist in the position that straddles. In that case, the parallax dprev of the pixel block in the previous reference image It-1 is calculated in time series by the product-sum operation of the parallax of the pixel block PBij included in the correlation destination based on the following equation (8). The difference between the parallax d of the pixel block PBij of the image It and the parallax dprev is defined as an optical flow Δd.

  Here, A is the area of the pixel block ij, which is 4 × 4 pixels in this embodiment. Dpn (n = 1 to m: m is 4 at maximum) is the parallax of each pixel block PBij included in the correlation destination, and An (n = 1 to m: m is 4 at maximum) is included in the correlation destination. This is the area of the pixel block PBij.

  Next, the process proceeds to a three-dimensional movement component flow calculation process (step S4). In the three-dimensional movement component flow calculation process, the distance data obtained by the distance data calculation process and the reference image obtained by the three-dimensional object recognition process are displayed. 3D of the three-dimensional object On based on the data inL, inR, dn of the three-dimensional object On and the optical flows Δi, Δj, Δd for each pixel block PBij on the reference image obtained by the optical flow detection process The moving component flow is calculated.

  FIG. 5 is a flowchart showing a calculation procedure in the three-dimensional moving component flow calculation process according to the present embodiment.

  First, the first pixel block PBij to be processed is selected from each pixel block PBij constituting the reference image data (step S40). As the pixel block selected first, for example, the pixel block PB11 at the lower left corner in the image plane can be used.

  Next, the three-dimensional object to which the pixel block PBij being processed belongs is specified from the three-dimensional object On obtained by the three-dimensional object recognition process (step S41). The i-coordinate of the pixel block PBij, that is, the i-coordinate ibh of the lower left corner of the pixel block PBij is larger than the i-coordinate inL of the left end of the three-dimensional object On when the pixel block PBij belongs to the three-dimensional object On, and It becomes smaller than the i coordinate inR at the right end. That is, the relationship inL <ibh <inR is established. Therefore, the three-dimensional object On to which the pixel block PBij belongs is specified by comparing the i-coordinates inL and inR at the left and right ends of the recognized three-dimensional object On with the i-coordinate ibh of the pixel block PBij.

  However, in such a method of comparing the i coordinates in the horizontal direction, when a plurality of three-dimensional objects On having different positions in the j direction are recognized, there are a plurality of specified three-dimensional objects On. Can occur. For example, in the case as shown in FIG. 2, the three-dimensional object to which the pixel block PBij shown by hatching belongs is specified as the three-dimensional object O1 and the three-dimensional object O2. Therefore, in such a case, the value of the parallax dn representing the three-dimensional object On among the plurality of specified three-dimensional objects On is closest to the value of the parallax d of the pixel block PBij and the difference between them is A solid object On having an absolute value equal to or less than a predetermined threshold is specified as a solid object to which the pixel block PBij belongs.

  Subsequently, it is determined whether or not there is a three-dimensional object On to which the pixel block PBij belongs (step S42). If the three-dimensional object On to which the pixel block PBij belongs is not specified in step S41, the next steps S43 and S44 are skipped. The process proceeds to step S45.

  If there is a three-dimensional object On to which the pixel block PBij belongs, the three-dimensional movement component flow of the pixel block PBij is calculated (step S43). Specifically, as described above, first, the apparent component flow is calculated from the yaw rate ω and the speed V of the own machine obtained by the yaw rate detection means 14 and the speed detection means 15 based on the expressions (5) to (7). Fst_i, Fst_j, and Fst_d are calculated. Then, based on the equations (2) to (4), the apparent components are obtained from the optical flows Δi, Δj, and Δd of the pixel block PBij obtained by the optical flow detection process. The three-dimensional moving component flows Fmv_i, Fmv_j, and Fmv_d are calculated by subtracting the flows Fst_i, Fst_j, and Fst_d.

  Next, for the three-dimensional object On to which the pixel block PBij to be processed belongs, a histogram is created with a predetermined class width for each moving component flow in each direction, and the calculated moving component flows Fmv_i, Fmv_j, The frequency of the class to which Fmv_d corresponds is increased by 1 (step S44). A histogram is created for each recognized three-dimensional object On, and if a histogram of the three-dimensional object On to which the pixel block PBij belongs has already been created, the histogram is updated by increasing the frequency of the corresponding class by one. To go.

  Subsequently, it is determined whether or not all pixel blocks PBij in the reference image have been selected (step S45). If there is a pixel block PBij that has not been selected, the next pixel block PBij is selected (step S45). S46) The above processing is performed on the next image block PBij.

  When all the pixel blocks PBij of the reference image are selected and the processing on the pixel blocks is completed, the three-dimensional representative movement component flows Fni, Fnj, and Fnd of the pixel block group belonging to the three-dimensional object On are calculated based on the obtained histogram. (Step S47). That is, the mode value is calculated from the histogram for the moving component flows Fmv_i, Fmv_j, and Fmv_d created and updated in the horizontal direction, the vertical direction, and the parallax direction for the three-dimensional object On, and the pixel block group belonging to the three-dimensional object On is calculated. The three-dimensional representative moving component flows are Fni, Fnj, and Fnd.

  This completes the three-dimensional moving component flow calculation process (step S4) of the moving object detection method of the present embodiment shown in FIG.

  Finally, in the determination process, a moving object and a stationary object are determined based on the three-dimensional representative movement component flows Fni, Fnj, and Fnd obtained in the three-dimensional movement component flow calculation process (step S5). Specifically, for each recognized three-dimensional object On, whether each directional component flow Fni, Fnj, Fnd of the three-dimensional representative moving component flow exceeds a predetermined threshold set to 1 pix / Frame, for example. If any one of the three directional component flows Fni, Fnj, Fnd exceeds the threshold, it is determined that the three-dimensional object On is a moving object. Further, when all of the three direction component flows Fni, Fnj, and Fnd are equal to or less than the threshold value, it is determined that the three-dimensional object On is a stationary object. This process is performed for all recognized three-dimensional objects On.

  As described above, according to the moving object detection device and the moving object detection method of the present embodiment, the three-dimensional object On is recognized and extracted from the scenery of the monitoring area captured by the stereo camera, and each three-dimensional object On is detected. Next, the three-dimensional moving component flows Fmv_i, Fmv_j, and Fmv_d, which are substantial moving components with respect to the ground, are obtained from the optical flows Δi, Δj, Δd of the pixel block PBij belonging to the three-dimensional object On, and the three-dimensional object On A motion is calculated, and it is determined whether it is a moving object based on it.

  Therefore, since the optical flows Δi, Δj, and Δd can be accurately associated with the three-dimensional object On, the three-dimensional movement component flow of the three-dimensional object On, that is, whether or not the three-dimensional object On is moving with respect to the ground. Can be accurately calculated without being influenced by the movement of the own device, and a moving object can be accurately detected from the monitoring area.

  In addition, since the three-dimensional object On is recognized using the distance data and the three-dimensional object On above the road surface is recognized, it can be accurately recognized even if the area of the three-dimensional object On displayed in the image is small. On the contrary, it is possible to efficiently recognize a three-dimensional object without erroneously recognizing a shadow on the road as a three-dimensional object.

  In addition, the actual i of each recognized three-dimensional object On using each direction component flow Fni, Fnj, Fnd of the three-dimensional representative movement component flow calculated as described above and the parallax dn representing the three-dimensional object On. It is also possible to calculate moving speeds Vi, Vj, Vd (unit: [m / sec]) in the direction, j direction, and d direction, and the speeds Vi, Vj, Vd are expressed by the following equations (9) to (11). Is calculated based on

  In the above equations, L is the distance [m] between the main camera 3a and the sub camera 3b, Wi and Wj are the imaging element pitches [m] in the i direction and the j direction, and f is the main camera 3a and the sub camera 3b. Are the frame intervals [sec], the units of Fni, Fnj, and Fnd are [pix / Frame] and the unit of dn is [pix], respectively.

[Second Embodiment]
The moving object detection device and the moving object detection method according to the second embodiment of the present invention are, for example, a plurality of three-dimensional objects On recognized as one three-dimensional object On, such as a pedestrian walking in front of a stopped vehicle. , Om,... Are separated into individual three-dimensional objects On, Om,... Using the moving component flow distribution, and the moving component flow is calculated for each separated three-dimensional object On, Om,. The three-dimensional object is specified with higher accuracy and the moving object is detected with higher accuracy.

  Note that in the moving object detection device of the present embodiment, the same reference numerals are used for the devices and means that are functionally the same as those of the moving object detection device according to the first embodiment, and a description thereof is omitted.

  FIG. 6 is a block diagram of a calculation unit of the moving object detection device according to the second embodiment. The calculation unit 12 includes a primary calculation unit 16, a map generation unit 17, a separation unit 18, and a secondary calculation unit 19.

  Similar to the calculation unit 12 in the first embodiment, the primary calculation unit 16 belongs to each solid object On based on the data inL, inR, dn of each solid object On transmitted from the recognition unit 10. The pixel block PBij on the reference image is specified, and the optical flow Δi, Δj, Δd of the pixel block PBij transmitted from the detection unit 11 and the yaw rate of the own device input from the yaw rate detection unit 14 and the speed detection unit 15 The three-dimensional moving component flows Fmv_i, Fmv_j, and Fmv_d of the pixel block PBij belonging to each three-dimensional object On are calculated based on the equations (2) to (7) using ω and the velocity V.

  The map generation unit 17 performs the three-dimensional movement calculated for the three-dimensional object number map Mo indicating the distribution of the number n of the three-dimensional object On to which the pixel block PBij belongs as shown in FIG. 7, and the pixel block PBij as shown in FIG. It is configured to generate moving component flow maps Mi, Mj, and Md indicating respective distributions of the moving component flows Fmv_i, Fmv_j, and Fmv_d of the component flows. FIG. 8 shows a moving component flow map Mi representing the distribution of the moving component flow Fmv_i on behalf of each moving component flow.

  Here, the three-dimensional object number map Mo has memory areas represented by the same number of blocks Bij as the pixel blocks PBij constituting the reference image, and each block Bij corresponds to the pixel block PBij in a positional manner. For example, they are arranged in a 128 × 50 matrix. In each block Bij, the number of the three-dimensional object On to which the pixel block PBij corresponding to the position belongs is written, and the block Bij corresponding to the pixel block PBij not belonging to any of the recognized three-dimensional object On. Is written with 0.

  Similarly to the three-dimensional object number map Mo, the moving component flow maps Mi, Mj, and Md have memory areas represented by the same number of blocks Bij as the pixel blocks PBij constituting the reference image, and each block Bij is a matrix. Are arranged in a shape. The movement component blow Fmv_i calculated for the positionally corresponding pixel block PBij is written in each block Bij of the movement component flow map Mi, and the movement component blow Fmv_j moves to each block Bij of the movement component flow map Mj. The moving component blow Fmv_d is written in each block Bij of the component flow map Md. Note that “f1” and “f2” in FIG. 8 indicate the number of the three-dimensional object On to which the block Bij belongs as a subscript, and each block Bij has a moving component calculated for the corresponding pixel block PBij. The flow value is written.

  Based on the generated moving component flow maps Mi, Mj, and Md, the separation unit 18 has a difference between the moving component flows Fmv_i, Fmv_j, and Fmv_d larger than a predetermined threshold among the pixel block groups belonging to the same three-dimensional object On. The pixel block group is separated as different three-dimensional objects Onew.

  Specifically, the case of the three-dimensional object O2 shown in FIG. 2 will be described as an example. In the three-dimensional object number map Mo, the separation unit 18 extends the three-dimensional object O2 in the j direction as shown in FIG. The average values Fr1 to Fr8 of the moving component flows Fmv_i of the blocks Bij belonging to the block columns are divided for each of the block columns R1 to R8 with reference to the moving component flow map Mi. As shown in FIG. 10, block sequences that have similar average values of the moving component flows Fmv_i of adjacent block sequences are grouped into groups G1 to G3, and the average value Fg1 of the moving component flows Fmv_i for each group. ~ Fg3 is calculated.

  For example, like the group G1 and the group G3 in FIG. 10, the separation unit 18 determines that groups having a smaller difference between the average values of the moving component flows Fmv_i than the predetermined threshold are the same three-dimensional object O2. Are grouped into one group, and a group having a difference in the average value of the moving component flows Fmv_i larger than the predetermined threshold is separated from the three-dimensional object O2 as another new three-dimensional object Onew, and as shown in FIG. A new number is assigned to Onew. Since there are three solid objects shown in FIG. 2, 4 is added as a new number in FIG.

  The separation unit 18 performs the same processing on the other moving component flows Fmv_j and Fmv_d, and separates the three-dimensional object On when there is a new three-dimensional object Onnew. The separation unit 18 is configured to associate the left and right i-coordinates inL and inR and the parallax dn representing the three-dimensional object Onnew with respect to the new three-dimensional object Onnew, and the original three-dimensional object On. The parallax dn representing the three-dimensional object On is again calculated, and the data is sent to the secondary calculation unit 19.

  Similar to the calculation unit 12 of the first embodiment, the secondary calculation unit 19 performs the three-dimensional movement component of all the pixel blocks PBij belonging to the three-dimensional object On for all three-dimensional objects On including the new three-dimensional object Onnew. The mode value is obtained for each direction component flow of the flow, and the three-dimensional representative moving component flows Fni, Fnj, and Fnd of the pixel block group belonging to the three-dimensional object On are calculated.

  Next, a moving object detection method using the moving object detection device according to the second embodiment will be described.

  In the moving object detection method of the present embodiment, the process is performed in the same procedure as the processing procedure of the moving object detection method according to the first embodiment shown in FIG. 3, but the three-dimensional moving component flow calculation process (step S4). ) Is added to the processing performed by the map generation unit 17, the separation unit 18, and the secondary calculation unit 19 of the calculation unit 12 of the moving object detection device.

  FIG. 12 is a flowchart showing a processing procedure in the three-dimensional moving component flow calculation processing according to the second embodiment. FIG. 13 is a flowchart showing a processing procedure in the primary calculation process and the map generation process of FIG.

  In the present embodiment, primary calculation processing (step S48) for calculating the three-dimensional movement component flows Fmv_i, Fmv_j, and Fmv_d for each pixel block PBij belonging to the three-dimensional object On, and each movement of the calculated three-dimensional movement component flow A map generation process (step S49) for generating a moving component flow map indicating the distribution of the component flows Fmv_i, Fmv_j, and Fmv_d is simultaneously performed.

  Specifically, as shown in FIG. 13, first, the first pixel block PBij to be processed is selected from the pixel blocks PBij constituting the reference image data (step S52). As the pixel block selected first, for example, the pixel block PB11 at the lower left corner in the image plane can be used.

  Next, the i-coordinate ibh of the lower left corner of the pixel block PBij being processed is compared with the i-coordinates inL and inR of the left and right ends of the three-dimensional object On to identify the three-dimensional object On to which the pixel block PBij belongs (step). S53). When a plurality of three-dimensional objects On having different positions in the j direction are recognized, it is first to identify the three-dimensional object On by comparing the value of the parallax d of the pixel block PBij with the parallax dn representing the three-dimensional object On. This is the same as in the first embodiment.

  Subsequently, it is determined whether there is a three-dimensional object On to which the pixel block PBij belongs (step S54). If there is a three-dimensional object On to which the pixel block PBij belongs, the three-dimensional movement component flows Fmv_i, Fmv_j, and Fmv_d of the pixel block PBij are calculated (step S55). A specific calculation method is the same as that in the first embodiment.

  Next, data is written to each moving component flow map Mi, Mj, Md and three-dimensional object number map Mo for each moving component flow Fmv_i, Fmv_j, Fmv_d of the three-dimensional moving component flow (step S56). As described above, each of the movement component flow maps Mi, Mj, Md and the three-dimensional object number map Mo are configured by arranging blocks Bij in a matrix, and the three-dimensional movement component flows Fmv_i, Fmv_j of the pixel block PBij. , Fmv_d are calculated, the values of the movement component flows Fmv_i, Fmv_j, Fmv_d are written in the blocks Bij of the movement component flow maps Mi, Mj, Md that correspond to the pixel block PBij, respectively. The number of the three-dimensional object On to which the pixel block PBij belongs is written in the block Bij of the three-dimensional object number map Mo corresponding to the position.

  If it is determined in step S54 that there is no three-dimensional object On to which the pixel block PBij belongs, 0 is written in the block Bij of the three-dimensional object number map Mo corresponding to the pixel block PBij, Proceed to step S57

  Subsequently, it is determined whether or not all pixel blocks PBij in the reference image have been selected (step S57). If there is an unselected pixel block PBij, the next pixel block PBij is selected (step S58). ), The above processing is performed on the next image block PBij.

  When the above processing is performed for all the pixel blocks PBij of the reference image, the moving component flow maps Mi, Mj, Md and the three-dimensional object number map Mo are generated (step S49 in FIG. 12).

  Next, in the separation process, among the pixel block groups belonging to the same three-dimensional object, the pixel block group in which the difference in the movement component flow is larger than a predetermined threshold based on the generated movement component flow maps Mi, Mj, and Md. Are separated as different three-dimensional objects (step S50). Specifically, as described in the description of the separation unit 18 of the moving object detection device, the three-dimensional object On on the three-dimensional object number map Mo is divided into block rows R extending in the j direction (see FIG. 9). The average values Fri, Frj, Frd of the movement component flows Fmv_i, Fmv_j, Fmv_d are calculated for each block row R with reference to the movement component flow maps Mi, Mj, Md (see FIG. 10).

  Then, block sequences in which the average values of the moving component flows of adjacent block sequences are approximated are grouped, and the average values Fgni, Fgnj, and Fgnd of the respective moving component flows Fmv_i, Fmv_j, and Fmv_d are calculated for each group. Groups having a difference in value smaller than a predetermined threshold are determined as the same three-dimensional object On and grouped into one group, and groups having a difference in average value of moving component flows larger than the predetermined threshold are separated. Is separated from the three-dimensional object On as a new three-dimensional object Onnew, and a new number is assigned to the three-dimensional object Onnew as shown in FIG.

  Finally, in the second calculation process (step S51), for all three-dimensional objects On including the new three-dimensional object Onnew, the three-dimensional movement component flow of all pixel blocks PBij belonging to the three-dimensional object On for each three-dimensional object On. For each direction component flow Fmv_i, Fmv_j, Fmv_d, a histogram is created for each movement component flow with a predetermined class width, and a mode value is calculated from each histogram, and the three-dimensional representative movement component of the pixel block group belonging to the three-dimensional object On. The flows are Fni, Fnj, and Fnd.

  As described above, according to the moving object detection device and the moving object detection method of the present embodiment, in addition to the effects of the moving object detection device and the moving object detection method according to the first embodiment, for example, immediately before a stopped vehicle. Like a walking pedestrian, the three-dimensional objects On, Om,... That are recognized as the same three-dimensional object On in the three-dimensional object recognition using the distance data are accurately separated based on the moving component flow. It becomes possible to do.

  In addition, since the separation is performed using the moving component flow, in particular, it is possible to effectively separate the moving objects or the moving object and the stationary object, and more accurately detect the moving object in the monitoring region. This makes it possible to detect a moving object with higher accuracy.

[Third Embodiment]
In the moving object detection device and the moving object detection method according to the third embodiment of the present invention, when the three-dimensional object On is at a relatively short distance from the own apparatus, the three-dimensional object On in the monitoring region, that is, in the reference image. In some cases, a solid object On is recognized as a plurality of solid objects On, Om,... In such a case, by integrating a plurality of three-dimensional objects On, Om,... Based on the moving component flow and the parallax, the three-dimensional object is recognized more accurately and the moving object is detected with higher accuracy.

  In the moving object detection device of the present embodiment, the same reference numerals are used for the devices and means that are functionally the same as those of the moving object detection device according to the first and second embodiments, and a description thereof is omitted.

  FIG. 14 is a block diagram of a calculation unit of the moving object detection device according to the third embodiment. The calculation unit 12 includes a primary calculation unit 16, a map generation unit 17, a separation unit 18, a secondary calculation unit 19, and an integration unit 20. The configuration from the primary calculation unit 16 to the secondary calculation unit 19 is as described in the second embodiment.

  For example, when the three-dimensional object On and the three-dimensional object Om are adjacent to each other as illustrated in FIG. 15A, the integrating unit 20 calculates the three-dimensional object On and the three-dimensional object calculated based on the moving component flow maps Mi, Mj, and Md. The difference between the three-dimensional representative moving component flows Fni, Fnj, Fnd, Fmi, Fmj, and Fmd of the pixel block group belonging to each of the objects Om is equal to or less than a predetermined threshold, and the parallax dn representing the three-dimensional object On and the three-dimensional object If the absolute value of the difference from the parallax dm representing Om is smaller than the absolute value of the difference from the parallax representing the other three-dimensional object and is equal to or less than a predetermined threshold, the three-dimensional object On The three-dimensional object Om is integrated into a single three-dimensional object On as shown in FIG.

  The integrating unit 20 calculates i-coordinates inL and inR and parallax dn representing the three-dimensional object On of the three-dimensional object On that are integrated into one solid object, and determines the corresponding one to the three-dimensional object On. Configured to send to.

  Next, a moving object detection method using the moving object detection device according to the third embodiment will be described.

  FIG. 16 is a flowchart illustrating a processing procedure in the three-dimensional moving component flow calculation processing according to the third embodiment. In the moving object detection method of the present embodiment, processing is performed in the same procedure as the processing procedure of the moving object detection method according to the first embodiment shown in FIG. 3, and three-dimensional moving component flow calculation processing (step S4). Also for the above, the same processing as in the second embodiment is performed until the secondary calculation processing (step S51) shown in FIG.

  In the integration process, as described in the description of the integration unit 20 of the moving object detection device of the present embodiment, for example, when the three-dimensional object On and the three-dimensional object Om are adjacent to each other as illustrated in FIG. The difference between the three-dimensional representative moving component flows Fni, Fnj, Fnd, Fmi, Fmj, and Fmd of the pixel block group belonging to each of the three-dimensional objects On and Om calculated based on the moving component flow maps Mi, Mj, and Md is predetermined. And the absolute value of the difference between the parallax dn representing the three-dimensional object On and the parallax dm representing the three-dimensional object Om is smaller than the absolute value of the difference from the parallax representing the other three-dimensional object and set. If it is equal to or less than the predetermined threshold value, the three-dimensional object On and the three-dimensional object Om are integrated into one solid object On as shown in FIG. The same processing is performed for other three-dimensional objects. Then, the i coordinates inL and inR of the left and right ends of the integrated three-dimensional object On and the parallax dn representing the three-dimensional object On are calculated and sent to the determination process (step S5) in association with the three-dimensional object On.

  As described above, according to the moving object detection device and the moving object detection method of the present embodiment, the three-dimensional movement component flow and the parallax are obtained for a plurality of adjacent three-dimensional objects that may actually be the same three-dimensional object. Whether or not they are the same three-dimensional object. If they are determined to be the same three-dimensional object, they can be integrated into one solid object.

  Therefore, in addition to the effects of the moving object detection device and the moving object detection method according to the above-described embodiment, it is possible to more accurately recognize a three-dimensional object in the monitoring area, and detect a moving object in the monitoring area. Can be performed more accurately. Further, when the moving object is to be monitored, the number of monitoring targets can be further limited.

  In the present embodiment, the calculation unit 12 and the three-dimensional moving component flow calculation process (step S4) are combined with the separation unit 18 and the separation process (step S50) of the second embodiment. A configuration in which the separation unit 18 and the separation process are not provided is also possible. In this embodiment, the case where the integration process is performed after the secondary calculation process has been described. However, the integration process may be performed before the secondary calculation process.

[Fourth Embodiment]
In the moving object detection device and the moving object detection method according to the fourth embodiment of the present invention, it can be said that the speed V of the own device detected by the speed detection means 15 is not necessarily accurate because of detection accuracy, resolution, communication delay, and the like. First, the velocity V of the own device is corrected using the optical flow with reference to a stationary object determined to be stationary by the determination unit 13, thereby improving the accuracy and reliability of the apparatus and method.

  The basic idea here is that if the three-dimensional moving component flows Fmv_i, Fmv_j, and Fmv_d of a stationary object that is determined to be stationary are regarded as 0, from the equations (2) to (4), the optical The flows Δi, Δj, Δd are equal to the apparent component flows Fst_i, Fst_j, Fst_d that are observed with the movement of the own device. Therefore, apart from the speed V detected by the speed detecting means, paying attention to a stationary object, the speed of the own device is calculated from the optical flow of the pixel block PBij belonging to it, and is detected by the speed detecting means using the value. The speed V is corrected.

  Note that, in the moving object detection device of the present embodiment, the same reference numerals are used for the devices and means that are functionally the same as those of the moving object detection device according to the first to third embodiments, and description thereof is omitted.

  FIG. 17 is a block diagram of a moving object detection device according to the fourth embodiment. In this embodiment, the image analysis unit 9 of the moving object detection device includes a recognition unit, a detection unit 11, a calculation unit 12, and a determination unit. 13, a speed map generation unit 21, a speed calculation unit 22, and a speed correction unit 23 are provided.

  The speed map generation unit 21 is configured to generate a speed map Mv indicating the distribution of the calculated speed Vc of the own device in a stationary object as shown in FIG. The velocity map Mv has a memory area represented by the same number of blocks Bij as the pixel blocks PBij constituting the reference image, and each block Bij is, for example, 128 × 50 so as to correspond to the pixel block PBij. Are arranged in a matrix.

  The velocity map generation unit 21 is detected by the coordinates (i, j) of the pixel block PBij belonging to the three-dimensional object On determined as a stationary object by the determination unit 13 and the optical flow Δi in the i direction, distance data, and yaw rate detection means. Based on the following yaw rate ω of the own device, the velocity Vc of the own device with respect to the stationary object in the pixel block PBij is calculated based on the following equation (12), and the block Bij of the velocity map Mv corresponding to the pixel block PBij is positioned. The speed Vc value is written.

  Here, x and z are pixel blocks PBij when the road surface directly below the center of the main camera 3a and the sub camera 3b is the origin, the horizontal direction is the X axis, the height direction is the Y axis, and the distance direction is the Z axis. X coordinates and z coordinates of the object calculated from the coordinates (i, j) and the distance data. Δt is a frame interval, f is a focal length of the main camera 3a and the sub camera 3b, and Wi is an imaging element pitch in the i direction.

  Note that “Vc” in FIG. 18 indicates the block Bij for which the speed Vc has been calculated, and the value of the own speed Vc calculated for the corresponding pixel block PBij is written in each block Bij. It is.

  The speed calculation unit 22 calculates an average value Vb of the speed Vc calculated for each pixel block PBij for all the pixel blocks PBij in which the speed Vc of the own apparatus is written in the speed map Mv, and calculates the average value Vb of the own apparatus. The speed is output to the speed correction unit 23 as a speed.

  The speed correction unit 23 is configured to correct the speed V detected by the speed detection unit 15 using the own speed Vb transmitted from the speed calculation unit 22. Specifically, the speed correction unit 23 calculates a difference value between the speed V detected by the speed detection unit 15 and the own speed Vb transmitted from the speed calculation unit, and the calculated difference value is calculated. The limiter process, filter process, etc. are applied to remove the influence of disturbance, etc., the difference value from which the disturbance is removed is added to the speed V detected by the speed detection means 15, and the corrected speed Vad of the own machine is obtained. The data is output to the calculation unit 12.

  Next, a moving object detection method using the moving object detection device according to the fourth embodiment will be described.

  In the moving object detection method of this embodiment, the process up to the determination process (step S5) is performed in the same procedure as that of the moving object detection method according to the first embodiment shown in FIG.

  FIG. 19 is a flowchart illustrating a processing procedure after the determination processing according to the fourth embodiment. FIG. 20 is a flowchart showing a processing procedure in the speed map generation processing of FIG.

  In the speed map generation process (step S6), as shown in FIG. 20, each component of the reference image data is configured in the same manner as in the three-dimensional moving component flow calculation process in the first embodiment shown in FIG. The first pixel block PBij to be processed is selected from the pixel blocks PBij (step S60), and the three-dimensional object to which the pixel block PBij being processed belongs is determined among the three-dimensional objects On obtained up to the determination process. (Step S61).

  Next, it is determined whether or not there is a three-dimensional object On to which the pixel block PBij belongs (step S62). If the three-dimensional object On to which the pixel block PBij belongs is not specified in step S61, the next steps S63 to S65 are skipped. The process proceeds to step S66.

  If there is a three-dimensional object On to which the pixel block PBij belongs, it is determined whether or not the three-dimensional object On is a stationary object (step S63). If it is not a stationary object, steps S64 and S65 are skipped and the process proceeds to step S66. If the three-dimensional object On is a stationary object, the speed Vc of the own device relative to the stationary object in the pixel block PBij is calculated based on the equation (12) as described above (step S64), as shown in FIG. The calculated speed Vc of the own machine is written in the speed map Mv (step S65).

  Subsequently, it is determined whether or not all pixel blocks PBij in the reference image have been selected (step S66). If there is an unselected pixel block PBij, the next pixel block PBij is selected (step S67). ), The above processing is performed on the next image block PBij.

  When all the pixel blocks PBij of the reference image are selected and the processing for the pixel blocks is completed, the process proceeds to the next speed calculation process (step S7, see FIG. 19).

  In the speed calculation process, the average value Vb of the speed Vc calculated for each pixel block PBij is calculated for all the pixel blocks PBij in which the speed Vc of the own device is written in the speed map Mv (step S7).

  In the speed correction process, the speed V detected by the speed detection means 15 is corrected using the calculated speed Vb of the own machine, and the corrected speed Vad of the own machine is sent to the three-dimensional movement component flow calculation process ( Step S8). Specifically, a difference value between the speed V detected by the speed detection unit 15 and the calculated speed Vb of the own device is calculated, and the calculated difference value is subjected to a limiter process, a filter process, or the like, thereby causing a disturbance or the like. Is added to the speed V detected by the speed detection means 15 and the corrected speed Vad of the own machine is output to calculate the three-dimensional movement component flow. send.

  As described above, according to the moving object detection device and the moving object detection method of the present embodiment, the optical flow for the stationary object, that is, the own apparatus for the stationary object, with the stationary object determined to be stationary as a reference. The speed of the own machine itself is calculated using the apparent flow observed along with the movement of the machine, and the speed V of the own machine detected by the speed detecting means 15 is corrected using the calculated flow.

  Therefore, in addition to the effects of the moving object detection device and the moving object detection method according to the above-described embodiment, it becomes possible to calculate the three-dimensional movement component flow based on the more accurate speed Vad of the own device, thereby detecting the moving object. It becomes possible to further improve the accuracy of discrimination between a moving object and a stationary object in the apparatus and the moving object detection method, and to make the apparatus and method more reliable.

  In the present embodiment, the case where only the optical flow Δi in the i direction, that is, the lateral direction is used in the calculation of the speed of the own aircraft has been described from the viewpoint of quick discovery of a pedestrian or the like crossing the road. It is possible to perform speed correction that takes into account the movement in the vertical direction and the distance direction by using the optical flow Δj in the j direction and the optical flow Δd in the d direction.

1 is a block diagram of a moving object detection device according to a first embodiment. It is a figure explaining the solid object On recognized by the recognition part. It is a flowchart which shows the procedure of the process in the moving object detection method of 1st Embodiment. It is a figure which shows the relationship between the pixel block on a reference | standard image, and the image block of a correlation destination. It is a flowchart which shows the calculation procedure in the three-dimensional movement component flow calculation process which concerns on 1st Embodiment. It is a block diagram of the calculation part of the moving object detection apparatus which concerns on 2nd Embodiment. It is a figure explaining a solid object number map. It is a figure explaining a movement component flow map. It is a figure which shows the state which divided | segmented the solid object into the block row | line | column. It is a graph of the average value of the movement component flow for every block row. It is a figure which shows the state which isolate | separated the new solid object from the solid object. It is a flowchart which shows the process sequence in the three-dimensional movement component flow calculation process which concerns on 2nd Embodiment. It is a flowchart which shows the process sequence in the primary calculation process and map generation process of FIG. It is a block diagram of the calculation part of the moving object detection apparatus which concerns on 3rd Embodiment. (A) is a figure which shows two adjacent solid objects, (B) is a figure which shows the integrated solid object. It is a flowchart which shows the process sequence in the three-dimensional movement component flow calculation process which concerns on 3rd Embodiment. It is a block diagram of the moving object detection apparatus which concerns on 4th Embodiment. It is a figure explaining a speed map. It is a flowchart which shows the process sequence after the determination process which concerns on 4th Embodiment. It is a flowchart which shows the process sequence in the speed map production | generation process of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Moving object detection apparatus 2 Stereo camera 6 Stereo image processing part 10 Recognition part 11 Detection part 12 Calculation part 13 Determination part 13 Yaw rate detection means 15 Speed detection means 16 Primary calculation part 17 Map generation part 18 Separation part 20 Integration part 22 Speed calculation unit 23 Speed correction unit Fmv_i, Fmv_j, Fmv_d Movement component flow Fni, Fnj, Fnd Representative movement component flow Fst_i, Fst_j, Fst_d Apparent component flow Mi, Mj, Md Movement component flow map On Solid object PBij Pixel block V Speed Vc Calculated speeds Δi, Δj, Δd Optical flow ω Yaw rate

Claims (9)

A stereo camera that captures a landscape including a monitoring area and outputs a pair of images in time series;
A stereo image processing unit that calculates distance data by stereo matching based on the pair of images;
A recognition unit for recognizing a three-dimensional object existing in the monitoring area based on the distance data;
A detection unit that detects an optical flow based on a plurality of images arranged in time series and the distance data, with one image of the pair of images as a processing target;
A calculation unit that calculates a three-dimensional moving component flow of the three-dimensional object based on the distance data and the optical flow;
A moving object detection apparatus comprising: a determination unit that determines a moving object and a stationary object based on the three-dimensional moving component flow.   The calculation unit includes a distance data of the three-dimensional object specified by the distance data, a position of the three-dimensional object on the image, and a change amount of the position of the three-dimensional object specified by the optical flow on the image. The moving object detection device according to claim 1, wherein a three-dimensional moving component flow of the three-dimensional object is calculated based on an amount of change of the distance data with time. Speed detecting means for detecting the speed of the own machine equipped with the stereo camera;
Yaw rate detecting means for detecting the yaw rate of the own machine,
The calculation unit calculates an apparent component flow from the speed of the own device, the yaw rate of the own device, the distance data of the three-dimensional object, and the position on the image of the three-dimensional object, and the optical flow and the apparent The moving object detection device according to claim 1, wherein a three-dimensional moving component flow is calculated from a difference in component flows. The detection unit is configured to detect the optical flow for each pixel block having a predetermined area constituting the one image.
The calculation unit includes:
A primary calculation unit for calculating the three-dimensional movement component flow for each pixel block belonging to the three-dimensional object;
A map generation unit that generates a movement component flow map indicating a distribution of each movement component flow of the three-dimensional movement component flow calculated for the pixel block belonging to the three-dimensional object;
Based on the generated moving component flow map, a separation unit that separates pixel block groups in which the difference in the moving component flow is larger than a predetermined threshold among the pixel block groups belonging to the same three-dimensional object as different three-dimensional objects. The moving object detection apparatus according to claim 1, wherein the moving object detection apparatus has a moving object detection apparatus. The detection unit is configured to detect the optical flow for each pixel block having a predetermined area constituting the one image.
The calculation unit includes:
A primary calculation unit for calculating the three-dimensional movement component flow for each pixel block belonging to the three-dimensional object;
A map generation unit that generates a movement component flow map indicating a distribution of each movement component flow of the three-dimensional movement component flow calculated for the pixel block belonging to the three-dimensional object;
Based on the distance data and the generated moving component flow map, a difference in moving component flow between pixel blocks belonging to each of the three-dimensional object and another three-dimensional object adjacent to the three-dimensional object is equal to or less than a predetermined threshold value; When the difference between the respective parallaxes is minimum and not more than a predetermined threshold value, an integration unit is provided that integrates the three-dimensional object and the other three-dimensional object into one solid object. The moving object detection device according to any one of claims 1 to 4. The calculation unit is configured to calculate a three-dimensional representative movement component flow of a pixel block group belonging to the three-dimensional object,
The determination unit is configured to determine, as a moving object, the three-dimensional object in which at least one of the movement component flows of the three-dimensional representative movement component flow exceeds a predetermined threshold. The moving object detection device according to any one of claims 1 to 5.   The three-dimensional representative movement component flow is configured to be calculated by statistical processing of the three-dimensional movement component flow calculated for each pixel block in a pixel block group belonging to the three-dimensional object. The moving object detection apparatus according to claim 6. A speed calculator that calculates the speed of the aircraft based on the optical flow and the distance data related to the stationary object determined based on the three-dimensional moving component flow by the determination unit;
The moving object detection apparatus according to claim 3, further comprising: a speed correction unit that corrects the speed detected by the speed detection unit based on the speed of the own device calculated by the speed calculation unit. Distance data calculation processing for calculating distance data by stereo matching based on a pair of images obtained by capturing a scene including a monitoring area with a stereo camera capable of capturing images in time series;
A three-dimensional object recognition process for recognizing a three-dimensional object existing in the monitoring area based on the distance data;
Optical flow detection processing for detecting an optical flow based on a plurality of images arranged in time series and the distance data, with one image of the pair of images as a processing target;
A three-dimensional movement component flow calculation process for calculating a three-dimensional movement component flow of the three-dimensional object based on the distance data and the optical flow;
A moving object detection method comprising: a determination process for determining a moving object and a stationary object based on the three-dimensional moving component flow.

Images