JP6733302B2 - Image processing device, imaging device, mobile device control system, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, an imaging device, a mobile device control system, an image processing method, and a program.

Regarding the safety of automobiles, conventionally, the body structure of automobiles and the like have been developed from the viewpoints of how to protect pedestrians and occupants when they collide with pedestrians and automobiles. However, in recent years, with the development of information processing technology and image processing technology, a technology for detecting a person, an automobile, etc. at high speed has been developed. By applying these technologies, automobiles that automatically brake before a collision to prevent a collision have already been released.

In order to automatically apply the brake, it is necessary to measure the distance to an object such as a person or another vehicle, and for that reason, measurement using an image of a stereo camera has been put into practical use.

In the measurement using the image of this stereo camera, there is known a technique of recognizing an object such as a person or another vehicle by performing the following image processing (for example, refer to Patent Document 1).

First, from a plurality of images captured by a stereo camera, the vertical coordinate of the image is one axis, the parallax (disparity) of the image is the other axis, the value of the frequency of parallax is a V-Disparity image To generate. Next, the road surface is detected from the generated V-Disparity image. After removing noise using the detected road surface, a U-Disparity image is generated in which the horizontal coordinate of the image is the vertical axis, the parallax of the image is the horizontal axis, and the parallax frequency is the pixel value. Then, an object such as a person or another vehicle is recognized based on the generated U-Disparity image.

However, in the related art, when there is a side wall or guardrail of a highway in front of the stereo camera due to, for example, a curved road surface, it is affected by parallax of an obstacle such as the side wall or guardrail. However, there is a problem that the road surface may not be accurately measured.

Then, it aims at providing the technique which improves the precision which detects a road surface.

In the image processing device, from a distance image having a distance value corresponding to a distance between a road surface and an obstacle blocking the road surface in a plurality of captured images respectively captured by a plurality of image capturing units, a distance value in a vertical direction of the distance image A generation unit that generates vertical distribution data indicating a distribution of frequency, an estimation unit that estimates the height of the road surface according to the distance based on the vertical distribution data, and the estimation unit that estimates the estimation unit. A correction unit that corrects the height of the road surface at a distance of the obstacle to be low , wherein the correction unit has a distribution of frequency of distance values in the vertical direction of the distance image in a plurality of segments according to the distance values. And the height of the road surface in the second segment whose distance value is smaller than that of the first segment whose value according to the divided distribution is less than a predetermined threshold value is estimated by the estimation unit. Fix lower than that .

According to the disclosed technology, it is possible to improve the accuracy of detecting a road surface.

It is a figure which shows the structure of the vehicle equipment control system as a mobile device control system which concerns on embodiment of this invention. It is a schematic diagram which shows the structure of an imaging unit and an image analysis unit. It is a functional block diagram of a mobile device control system. It is a figure for explaining parallax image data and a V map generated from the parallax image data. It is a figure which shows the image example of the picked-up image as a reference|standard image picked up by one imaging part, and the V map corresponding to the picked-up image. It is a figure which shows the image example of the picked-up image as a reference|standard image picked up by one imaging part, and the V map of the predetermined area|region corresponding to the picked-up image. It is a flowchart which shows an example of a road surface estimation process. It is a figure explaining the example in case the road surface which has a side wall is curved. 6 is a flowchart illustrating an example of a terminating segment correction process according to the first embodiment. It is a figure explaining the example of the process which corrects the road surface of a segment farther than an end segment. It is a flow chart which shows an example of end segment amendment processing concerning a 2nd embodiment. It is a figure explaining the process which searches the terminal segment of 2nd Embodiment. It is a flow chart which shows an example of end segment amendment processing concerning a 3rd embodiment. It is a figure explaining the correspondence of a picked-up image and a V map. It is a figure explaining the correspondence of the captured image of an obstacle, such as a side wall, and a V map. It is a flowchart which shows an example of the road surface estimation process which concerns on 4th Embodiment. It is a figure explaining an example of the sample point extraction processing concerning a 4th embodiment. It is a figure explaining the end segment correction processing in an uphill. It is a flow chart which shows an example of end segment amendment processing concerning a 5th embodiment.

Hereinafter, a mobile device control system including the image processing apparatus according to the embodiment will be described.

[First Embodiment]
<Configuration of in-vehicle device control system>
FIG. 1 is a diagram showing a configuration of an in-vehicle device control system as a mobile device control system according to an embodiment of the present invention.

The in-vehicle device control system 1 is mounted on a vehicle 100 such as a car, which is a moving body, and includes an imaging unit 500, an image analysis unit 600, a display monitor 103, and a vehicle travel control unit 104. Then, the relative height information (relative height of the road surface (moving surface) in front of the own vehicle is obtained from the imaged image data of the front area (imaging area) of the own vehicle traveling direction in which the image capturing unit 500 images the front of the moving body. The information indicating the inclination situation) is detected, and the three-dimensional shape of the road surface ahead of the vehicle is detected from the detection result, and the detection result is used to control the moving body and various in-vehicle devices. The control of the moving body includes, for example, notification of a warning, control of the steering wheel of the own vehicle 100 (self moving body), or braking of the own vehicle 100 (self moving body).

The imaging unit 500 is installed, for example, near the rearview mirror (not shown) of the windshield 105 of the vehicle 100. Various data such as picked-up image data obtained by the image pickup of the image pickup unit 500 is input to the image analysis unit 600 as an image processing unit.

The image analysis unit 600 analyzes the data transmitted from the imaging unit 500, and on the traveling road surface in front of the own vehicle 100 with respect to the road surface portion on which the own vehicle 100 is traveling (the road surface portion located directly below the own vehicle). The relative height (positional information) at each point is detected, and the three-dimensional shape of the traveling road surface in front of the vehicle is grasped. In addition, other recognition objects such as other vehicles ahead of the own vehicle, pedestrians, and various obstacles are recognized.

The analysis result of the image analysis unit 600 is sent to the display monitor 103 and the vehicle traveling control unit 104. The display monitor 103 displays the captured image data obtained by the image capturing unit 500 and the analysis result. The vehicle traveling control unit 104 notifies the driver of the own vehicle 100 of a warning, for example, based on the recognition result of the recognition target objects such as other vehicles in front of the own vehicle, pedestrians, and various obstacles by the image analysis unit 600. And driving support control such as controlling the steering wheel and brake of the own vehicle.

<Structures of Imaging Unit 500 and Image Analysis Unit 600>
FIG. 2 is a diagram showing the configurations of the imaging unit 500 and the image analysis unit 600.

The image pickup unit 500 is composed of a stereo camera having two image pickup units 510a and 510b as image pickup means, and the two image pickup units 510a and 510b are the same. The image pickup units 510a and 510b respectively include image pickup lenses 511a and 511b, sensor substrates 514a and 514b including image sensors 513a and 513b in which light receiving elements are two-dimensionally arranged, and analog signals output from the sensor substrates 514a and 514b, respectively. And signal processing units 515a and 515b for generating and outputting picked-up image data obtained by converting an electric signal (an electric signal corresponding to the amount of light received by each light receiving element on the image sensors 513a and 513b) into a digital electric signal. ing. Luminance image data and parallax image data are output from the imaging unit 500.

The image pickup unit 500 also includes a processing hardware unit 510 including an FPGA (Field-Programmable Gate Array). The processing hardware unit 510 obtains the parallax value of the corresponding image portion between the captured images captured by the image capturing units 510a and 510b in order to obtain the parallax image from the luminance image data output from the image capturing units 510a and 510b. A parallax calculation unit 511 is provided as a parallax image information generating unit for calculating.

The parallax value referred to here is a comparison image with respect to an image portion on the reference image corresponding to the same point in the imaging region, with one of the captured images captured by each of the imaging units 510a and 510b as a reference image and the other as a comparison image. The positional shift amount of the upper image portion is calculated as the parallax value of the image portion. By utilizing the principle of triangulation, it is possible to calculate the distance from the parallax value to the same point in the imaging region corresponding to the image portion.

The image analysis unit 600 is composed of an image processing board and the like, and a storage unit 601 configured of a RAM, a ROM and the like for storing the brightness image data and the parallax image data output from the image pickup unit 500, a recognition process of an identification object, A CPU (Central Processing Unit) 602 that executes a computer program for performing parallax calculation control, a data I/F (interface) 603, and a serial I/F 604 are provided.

The FPGA configuring the processing hardware unit 510 performs processing requiring real-time processing on image data, such as gamma correction, distortion correction (parallelization of left and right captured images), and parallax calculation by block matching to perform parallax image processing. Is generated and written in the RAM of the image analysis unit 600. The CPU 602 of the image analysis unit 600 is responsible for controlling the image sensor controller of each of the image pickup units 510a and 510b and the overall control of the image processing board, and also for detecting the three-dimensional shape of the road surface, guardrails, and various other objects (identification target). Object) detection program, etc. is loaded from the ROM, and various processing is executed by using the luminance image data and the parallax image data stored in the RAM as input, and the processing result is a data I/F 603 or serial I/F 603. Output from F604 to the outside. At the time of executing such processing, the data I/F 603 is used to input vehicle operation information such as the vehicle speed, acceleration (mainly in the longitudinal direction of the vehicle), steering angle, yaw rate, etc. It can also be used as a processing parameter. The data output to the outside is used as input data for controlling various devices of the vehicle 100 (brake control, vehicle speed control, warning control, etc.).

The image pickup unit 500 and the image analysis unit 600 may be configured as the image pickup device 2 which is an integrated device.

FIG. 3 is a functional block diagram of the in-vehicle device control system 1 realized by the processing hardware unit 510, the image analysis unit 600, and the vehicle traveling control unit 104 in FIG. Note that the functional unit realized by the image analysis unit 600 is realized by the process of causing the CPU 602 of the image analysis unit 600 to execute one or more programs installed in the image analysis unit 600.

The process in this embodiment will be described below.

<Parallax image generation processing>
The parallax image generation unit 11 performs parallax image generation processing for generating parallax image data (parallax image information). The parallax image generation unit 11 is configured by, for example, the parallax calculation unit 511 (FIG. 2).

In the parallax image generation process, first, the luminance image data of one of the two image pickup units 510a and 510b is set as the reference image data, and the luminance image data of the other image pickup unit 510b is set as the comparison image data. The parallax between the two is calculated using the generated parallax image data and output. The parallax image data represents a parallax image in which the pixel value corresponding to the parallax value d calculated for each image portion on the reference image data is represented as the pixel value of each image portion.

Specifically, the parallax image generation unit 11 defines a block including a plurality of pixels (for example, 16 pixels×1 pixel) centering on one pixel of interest in a row of the reference image data. On the other hand, in the same row in the comparison image data, a block having the same size as the defined block of the reference image data is shifted pixel by pixel in the horizontal line direction (x direction) to show the feature of the pixel value of the block defined in the reference image data. A correlation value indicating the correlation between the feature amount and the feature amount indicating the feature of the pixel value of each block in the comparison image data is calculated. Then, based on the calculated correlation value, a matching process is performed to select the block of the comparative image data that has the highest correlation with the block of the reference image data among the blocks in the comparative image data. After that, the amount of positional deviation between the pixel of interest in the block of the reference image data and the corresponding pixel of the block of the comparative image data selected in the matching process is calculated as the parallax value d. Parallax image data can be obtained by performing such a process of calculating the parallax value d for the entire area of the reference image data or a specific area.

For example, the value (luminance value) of each pixel in the block can be used as the feature amount of the block used for the matching process, and the value (luminance value) of each pixel in the block of the reference image data can be used as the correlation value. Value) and the sum of the absolute values of the differences between the value (luminance value) of each pixel in the block of the comparative image data corresponding to these pixels can be used. In this case, it can be said that the block having the smallest total sum has the highest correlation.

When the matching process in the parallax image generation unit 11 is realized by hardware processing, for example, SSD (Sum of Squared Difference), ZSSD (Zero-mean Sum of Squared Difference), SAD (Sum of Absolute Difference), ZSAD ( A method such as Zero-mean Sum of Absolute Difference) or NCC (Normalized cross correlation) can be used. Since only the parallax value in pixel units can be calculated in the matching process, the estimated value needs to be used when the sub-pixel level parallax value of less than one pixel is required. As the estimation method, for example, an equiangular straight line method, a quadratic curve method, or the like can be used.

<V map generation processing>
The V map generation unit 12 executes a V map generation process for generating a V map (V-Disparity Map, an example of “vertical distribution data”) based on the parallax pixel data. Each piece of parallax pixel data included in the parallax image data is represented by a set (x, y, d) of an x-direction position, a y-direction position, and a parallax value d. This is converted into three-dimensional coordinate information (d, y, f) having d on the X axis, y on the Y axis, and frequency f on the Z axis, or this three-dimensional coordinate information (d, y, f). 3D coordinate information (d, y, f) limited to information exceeding a predetermined frequency threshold is generated as parallax histogram information. The parallax histogram information of this embodiment is composed of three-dimensional coordinate information (d, y, f), and the three-dimensional histogram information distributed in the two-dimensional XY coordinate system is called a V map.

Specifically, the V map generation unit 12 calculates the parallax value frequency distribution for each row region obtained by dividing the parallax image into a plurality of vertically divided regions. The information indicating the parallax value frequency distribution is parallax histogram information.

FIG. 4 is a diagram for explaining parallax image data and a V map generated from the parallax image data. Here, FIG. 4A is a diagram showing an example of the parallax value distribution of the parallax image, and FIG. 5B is a diagram showing a V map showing the parallax value frequency distribution for each row of the parallax image of FIG. 4A.

When parallax image data having a parallax value distribution as shown in FIG. 4A is input, the V map generation unit 12 calculates a parallax value frequency distribution, which is a distribution of the number of pieces of parallax value data for each row, This is output as parallax histogram information. The information of the parallax value frequency distribution of each row obtained in this manner is used as a two-dimensional orthogonal coordinate system in which the Y-axis is the y-direction position on the parallax image (the vertical position of the captured image) and the X-axis is the parallax value d. By expressing the above, the V map as shown in FIG. 4B can be obtained. This V map can also be expressed as an image in which pixels having pixel values according to the frequency f are distributed on the two-dimensional orthogonal coordinate system.

FIG. 5 is a diagram showing an image example of a captured image as a reference image captured by one of the image capturing units and a V map corresponding to the captured image. Here, FIG. 5A is a photographed image, and FIG. 5B is a V map. That is, the V map shown in FIG. 5B is generated from the captured image shown in FIG. 5A. In the V map, no parallax is detected in the area below the road surface, and therefore no parallax is counted in the shaded area A.

In the image example shown in FIG. 5A, a road surface 401 on which the own vehicle is traveling, a preceding vehicle 402 existing in front of the own vehicle, and a utility pole 403 existing outside the road are projected. Further, in the V map shown in FIG. 5B, there are a road surface 501, a preceding vehicle 502, and a telephone pole 503 corresponding to the image example.

In this image example, a front road surface of the own vehicle is a relatively flat road surface, that is, a virtual road surface obtained by extending the front road surface of the own vehicle parallel to the road surface portion directly below the own vehicle to the front of the own vehicle. This is a case where it matches the reference road surface (virtual reference moving surface). In this case, in the lower part of the V map corresponding to the lower part of the image, the high-frequency points are distributed in a substantially straight line shape with an inclination such that the parallax value d becomes smaller toward the upper part of the image. Pixels showing such a distribution are pixels that are present at almost the same distance in each row on the parallax image, have the highest occupancy ratio, and further show the identification object whose distance increases continuously toward the top of the image. It can be said that

Since the image pickup unit 510a picks up an image of the front area of the host vehicle, the content of the picked-up image is such that the parallax value d of the road surface becomes smaller as it goes upward in the image, as shown in FIG. 5A. In addition, in the same row (horizontal line), pixels that project the road surface have substantially the same parallax value d. Therefore, the high-frequency points distributed in the above-described substantially linear shape on the V map correspond to the characteristics of the pixels showing the road surface (moving surface). Therefore, the pixels of the points distributed on or near the approximate straight line obtained by linearly approximating the high-frequency points on the V map can be estimated with high accuracy as the pixels showing the road surface. Further, the distance to the road surface portion displayed in each pixel can be obtained with high accuracy from the parallax value d of the corresponding points on the approximate straight line.

FIG. 6 is a diagram showing an image example of a captured image as a reference image captured by one of the image capturing units and a V map of a predetermined area corresponding to the captured image.

The V map generation unit 12 may generate the V map using all the pixels in the parallax image, or may generate a V map in a predetermined area (for example, in FIG. 6A, an example of a captured image that is the basis of the parallax image). The V map may be generated using only pixels in the area where the road surface can be seen. For example, since the road surface becomes narrower toward the vanishing point as the distance increases, an area may be set according to the width of the road surface as shown in FIG. 6A. As a result, it is possible to prevent noise due to an object (such as the telephone pole 403) located in a region other than the region where the road surface can be seen from entering the V map.

<Road surface estimation>
The road surface estimation unit 13 estimates (detects) the road surface based on the parallax image generated by the parallax image generation unit 11. The processing of the road surface estimation unit 13 will be described below. FIG. 7 is a flowchart showing an example of the road surface estimation processing.

The road surface estimation unit 13 divides the V map into a plurality of segments according to the value of the parallax d that is the horizontal axis of the V map (step S11). By dividing into a plurality of segments for processing, it is possible to capture a road surface having a complicated shape such as a slope shape that goes up and down from a flat road on the way.

Then, the road surface estimation unit 13 extracts sample points from the pixels of the V map (step S12). The road surface estimation unit 13 may extract one or more sample points for each coordinate position of each parallax value d, for example. Alternatively, the road surface estimation unit 13 may extract the most frequent frequency point as a sample point at the coordinate position of each parallax value d. Alternatively, the road surface estimation unit 13 determines the frequency at a plurality of d coordinates including each parallax value d (for example, the coordinate position of each parallax value d and one or more coordinate positions of at least one of the left and right of each parallax value d). The most frequent points with the most may be extracted as sample points.

Then, the road surface estimation unit 13 excludes inappropriate points from the extracted sample points (step S13). The road surface estimation unit 13 calculates an approximate straight line corresponding to each extracted sample point by using, for example, the least square method. Then, sample points whose Euclidean distance is more than a predetermined threshold value with respect to the calculated approximate straight line are removed.

Then, the road surface estimation unit 13 detects the shape (position, height) of the road surface based on the sample points not excluded in step S13 (step S14). The road surface estimation unit 13 calculates an approximate straight line from the sample points of each segment by, for example, the method of least squares, and detects (estimates) the calculated approximate straight line as a road surface. Alternatively, the road surface shape detection unit 133 may detect (estimate) the curve shape calculated from the sample points of each segment using polynomial approximation or the like as the road surface.

Subsequently, when the detected (estimated) road surface is inappropriate, the road surface estimation unit 13 supplements the road surface (step S15). When the road surface estimation unit 13 estimates an inappropriate road surface that cannot be captured by a stereo camera due to noise, for example, based on the data of the default road surface or the road surface estimated in the previous frame. And supplement (interpolate) an inappropriate road surface.

Then, the road surface estimation unit 13 corrects each road surface estimated in each segment so that each road surface is continuous (smoothing processing) (step S16). The road surface estimation unit 13 tilts each road surface so that the end point (end point) of one road surface and the start point (end point) of the other road surface match among the road surfaces estimated in two adjacent segments, for example. And change the intercept.

<Correction of end segment>
The terminal segment correction unit 14 corrects the height of the road surface at the obstacle distance estimated by the road surface estimation unit 13 to be low.

In the first embodiment, the terminating segment correction unit 14 divides the V map into a plurality of segments according to the parallax value, and a pixel whose parallax frequency value included in each divided segment is a predetermined value or more (hereinafter Are referred to as “parallax points.” Note that “parallax points” are an example of “distance points”.

Then, the end segment correction unit 14 determines the segment (second segment) that is the farthest from the host vehicle among the segments (end candidates) in which the number of parallax points is equal to or greater than a predetermined threshold.

Then, if there is a first segment that is farther from the host vehicle than the second segment, the end segment correction unit 14 corrects the height of the road surface in the second segment to be lower.

Next, a detailed example of the processing of the end segment correction unit 14 will be described with reference to FIGS. 8 and 9.

FIG. 8: is a figure explaining the example in case the road surface which has a side wall is curved. FIG. 8A is a diagram illustrating an image example of a captured image as a reference image captured by one of the image capturing units when the road surface having the side wall is curved. FIG. 8B is a diagram showing an example of a V map of a predetermined area corresponding to the captured image of FIG.

As shown in FIG. 8(A), when there is an obstacle such as a side wall far from the own vehicle and the far distance cannot be seen, as shown in FIG. 8(B), it is farther than the segment where the parallax of the obstacle exists. In the segment, there is no parallax distribution (pixel).

In this case, in the segment where the obstacle parallax exists, the road surface estimation unit 13 may erroneously detect the height of the road surface as high as 521 in FIG. 8B due to the obstacle parallax.

Further, in the segment farther than the segment in which the parallax of the obstacle exists, the road surface estimation unit 13 detects the road surface as indicated by 522 in FIG. 8B according to an inappropriate pixel due to noise or the like. There are cases.

Therefore, the end segment correction unit 14 performs the following processing. FIG. 9 is a flowchart showing an example of the end segment correction processing according to the first embodiment.

The end segment correction unit 14 divides the V map into a plurality of segments (step S101). The size, shape, and number of each segment to be divided may be arbitrary.

Subsequently, the end segment correction unit 14 counts parallax points having a value equal to or larger than a predetermined value (for example, 1) in each of the divided segments (step S102). The predetermined value may be two or more for removing noise, or may be an average value of parallax frequencies in each segment.

Then, the end segment correction unit 14 sets a segment whose count number is equal to or more than a predetermined threshold value as a segment A (end candidate) and a segment whose count number is less than the predetermined threshold value as a segment B (non-end candidate) (step S103). ). As a result, in the example of FIG. 8B, the first segment to the fourth segment, which are relatively close to the own vehicle, are the segments A, and the fifth segment to the seventh segment, which are relatively far from the own vehicle, are the segments. It is determined to be B.

Subsequently, the end segment correction unit 14 sequentially evaluates the segments that are far from the host vehicle, and determines the segment A when the segment B is changed to the segment A as the end segment (step S104). By searching in order from the segment farther from the own vehicle, the end segment is correctly detected even if the segment closer to the own vehicle than the end segment is determined to be the segment B due to noise or the like. it can.

Subsequently, the end segment correction unit 14 corrects the road surface so that the road surface of the end segment becomes low (step S105). The terminal segment correction unit 14 sets the lower road surface of the road surface estimated by the road surface estimation unit 13 for the terminal segment and the road surface estimated by a method different from that of the road surface estimation unit 13 as the road surface of the terminal segment. Here, the road surface existing at the position of low height may be, for example, a road surface having a low y coordinate at the end point (left end) of the end segment or a road surface having a low y coordinate of the middle point.

The terminal segment correction unit 14 may use, for example, the road surface calculated by the following method as a road surface estimated by a method different from that of the road surface estimation unit 13.

-Road surface that is the road surface estimated for the segment immediately before the end segment extended to the end segment-Road surface (historical road surface) estimated for the last captured image (frame) for the end segment
If the segment becomes discontinuous as a result of modifying the end segment, the smoothing process described above may be performed to modify each segment to be continuous.

As a result, the road surface at the end segment is corrected as indicated by 523 in FIG. 8B.

Subsequently, the end segment correction unit 14 corrects the road surface of the segment that is farther from the host vehicle than the end segment (step S106). The terminal segment correction unit 14 performs, for example, the following processing on the road surface of the segment that is farther from the host vehicle than the terminal segment. FIG. 10 is a diagram illustrating an example of processing for correcting the road surface of a segment farther than the end segment.

As shown in FIG. 10A, the road surface is cut off at the end segment (the road surface of the segment farther than the end segment is discarded). This makes it possible to prevent the U map used for the subsequent clustering from using the parallax of a segment that is farther from the host vehicle than the end segment.

As shown in FIG. 10(B), the road surface of the modified end segment is extended in the segment farther from the vehicle than the end segment.

As shown in FIG. 10(C), the road surface estimated in the previous frame for a segment that is farther from the vehicle than the end segment is based on the corrected end surface (left end) of the end segment. , Connect in series.

<Road height table calculation processing>
The road surface height table calculation unit 15 calculates the road surface height (the height relative to the road surface portion directly below the own vehicle) based on the road surface in each segment corrected by the smoothing processing unit 135 and creates a table. Road surface height table calculation processing is performed.

The road surface height table calculation unit 15 calculates the distance from the road surface information in each segment to each road surface portion displayed in each row region (each vertical position of the image) on the captured image. It should be noted that each surface portion in the traveling direction of the own vehicle of a virtual plane that extends forward in the traveling direction of the own vehicle so that the road surface portion located directly below the own vehicle is parallel to that surface is displayed in which row area in the captured image. It is determined in advance whether this will be done, and this virtual plane (reference road surface) is represented by a straight line (reference straight line) on the V map. The road surface height table calculation unit 15 can obtain the height of each road surface portion in front of the host vehicle by comparing the road surface in each segment with the reference straight line. Simply, from the Y-axis position on the road surface in each segment, the height of the road surface portion existing in front of the host vehicle by the distance obtained from the corresponding parallax value can be calculated. The road surface height table calculation unit 15 tabulates the heights of the road surface portions obtained from the approximate straight line for the necessary parallax range.

The height from the road surface of the object displayed in the captured image portion corresponding to the point where the Y-axis position is y′ at a certain parallax value d is the Y-axis position on the road surface at the parallax value d being y0. Then, it can be calculated from (y'-y0). Generally, the height H from the road surface of the object corresponding to the coordinates (d, y') on the V map can be calculated by the following formula. However, in the following formula, “z” is a distance (z=BF/(d-offset)) calculated from the parallax value d, and “f” is the focal length of the camera (y′−y0). It is a value converted into the same unit as the unit. Here, “BF” is a value obtained by multiplying the base length of the stereo camera by the focal length, and “offset” is a parallax value when an object at infinity is photographed.

H=z×(y′−y0)/f
<Clustering, rejection, tracking>
The clustering unit 16 sets a set (x, y, d) of the x-direction position, the y-direction position, and the parallax value d in each parallax pixel data included in the parallax image data as x on the X axis and d and Z on the Y axis. Frequency is set on the axis, and XY two-dimensional histogram information (frequency U map) is created.

The clustering unit 16 calculates the parallax images in which the height H from the road surface is within a predetermined height range (for example, 20 cm to 3 m) based on the height of each road surface portion tabulated by the road surface height table calculation unit 15. Create a frequency U map only for point (x,y,d). In this case, it is possible to appropriately extract the object existing in the predetermined height range from the road surface.

The clustering unit 16 detects, in the frequency U map, a region where the frequency is higher than a predetermined value and the parallax is dense, as the region of the object, and based on the coordinates on the parallax image and the actual size (size) of the object. Individual information such as the predicted object type (human or pedestrian) is added.

The rejection unit 17 rejects the information of the object that is not the recognition target based on the parallax image, the frequency U map, and the individual information of the object.

The tracking unit 18 determines whether or not the detected object is a tracking target when the detected objects continuously appear in frames of a plurality of parallax images.

<Driving support control>
Based on the detection result of the object by the clustering unit 16, the control unit 19 performs traveling support control such as issuing a warning to the driver of the own vehicle 100 or controlling the steering wheel or the brake of the own vehicle. To do.

[Second Embodiment]
In the first embodiment, an example in which the end segment is detected based on the number of parallax points in each segment has been described. In the second embodiment, an example will be described in which the end segment correction unit 14 detects the end segment based on the number of parallax points for each parallax d of each segment. The second embodiment is the same as the first embodiment except for some parts, and therefore the description thereof will be appropriately omitted.

Differences between the processing of the in-vehicle device control system 1 according to the second embodiment and the first embodiment will be described below.

Similarly to the first embodiment, the terminal segment correction unit 14 according to the second embodiment first divides the V map into a plurality of segments according to the parallax value, and the parallax frequency included in each divided segment. Pixels (parallax points) whose value is equal to or greater than a predetermined value are counted.

Then, the terminating segment correction unit 14 selects the segment with the longest distance from the host vehicle among the segments (termination candidates) in which the number of parallax points d is equal to or greater than a predetermined threshold and the number of parallax values d is equal to or greater than a predetermined threshold. Second segment).

Then, as in the case of the first embodiment, the terminal segment correction unit 14 lowers the height of the road surface in the second segment if there is a first segment that is farther from the host vehicle than the second segment. Fix it.

FIG. 11 is a flowchart showing an example of the end segment correction processing according to the second embodiment.

The end segment correction unit 14 divides the V map into a plurality of segments (step S201).

Subsequently, the end segment correction unit 14 counts the d-coordinates in which the number of pixels (parallax points) having a value of a predetermined value (for example, 1) or more is a predetermined number (for example, 1) or more for each d-coordinate of each divided segment. Yes (step S202). The predetermined value may be two or more for removing noise, or may be an average value of parallax frequencies in each segment.

The predetermined number may be set in consideration of road surface dispersion, or some predetermined value may be set in advance. For example, when it is known that the normal height when the parallax of the road surface is dispersed in the vertical direction (y direction) is 20 cm, the value obtained by converting the actual distance 20 cm into the number of pixels may be set as the predetermined value. This makes it possible to remove noise.

Subsequently, the end segment correction unit 14 sets a segment whose d-coordinate count number is equal to or greater than a predetermined threshold value as a segment A and a segment whose d-coordinate count number is less than a predetermined threshold value as a segment B (step S203).

Subsequently, the end segment correction unit 14 sequentially evaluates the segments that are far from the host vehicle, and determines the segment A when the segment B is changed to the segment A as the end segment (step S204).

Subsequently, the end segment correction unit 14 corrects the road surface so that the road surface of the end segment becomes low (step S205).

Subsequently, the end segment correction unit 14 corrects the road surface of the segment that is farther from the vehicle than the end segment (step S206).

The processing of steps S201 and S204 to S206 is the same as the processing of steps S101 and S104 to S106 of FIG. 9 according to the first embodiment.

Next, the processing of steps S202 and S203 will be described with reference to FIG. FIG. 12 is a diagram illustrating a process of searching for an end segment according to the second embodiment.

When the V map in a certain segment has a shape as shown in FIG. 12A, the number of pixels having a value equal to or larger than a predetermined value is counted in the vertical direction for each d coordinate as shown in FIG. 12B. Then, the d-coordinates whose number is less than the predetermined number are determined to be noise, and the number of d-coordinates not determined to be noise is counted. Then, a segment in which the number of d-coordinates not determined as noise is equal to or larger than a predetermined threshold value is determined as a segment A (end candidate).

Note that, in the example of FIG. 12B, an example in which the number of parallax points at each d coordinate is converted into a histogram for determination is shown, but it is not necessary to make a histogram. For example, the following processing may be performed.

The terminating segment correction unit 14 sequentially determines from the d coordinate of the end point (left end) of the target segment, and when the count number of each d coordinate (bin) exceeds a predetermined number 524, increments the d coordinate counter. Then, the next d coordinate is determined. Then, when the counter of the d coordinate reaches or exceeds a predetermined threshold value, the target segment is set as the segment A (end candidate).

According to the second embodiment, it is possible to capture a parallax point group of obstacles such as sidewalls that appear in a vertical direction (y direction) of the V map. Therefore, the accuracy of searching the end segment is improved.

[Third Embodiment]
In the third embodiment, an example will be described in which a predetermined area in a luminance image (reference image data) corresponding to each segment of a V map is image-recognized and the end segment is determined based on the recognition result.

In the first actual form and the second actual form, since the end segment is determined based on the parallax point on the V map, for example, when the parallax point cannot be detected because there is no unevenness or white line on the road surface, the end segment is detected. There is a possibility of misjudging that there is.

According to the third embodiment, even in the case of a segment for which a parallax point cannot be detected, it can be correctly determined whether or not it is the end segment.

The third embodiment is the same as the first embodiment except for some parts, and thus the description thereof will be appropriately omitted. The details of the processing of the in-vehicle device control system 1 according to the third embodiment will be described below with respect to the differences from the first embodiment.

The end segment correction processing by the end segment correction unit 14 of the third embodiment will be described with reference to FIGS. 13 to 15.

FIG. 13 is a flowchart showing an example of the end segment correction processing according to the third embodiment.

First, the end segment correction unit 14 sets each area in the captured image (step S301). FIG. 14 is a diagram illustrating a correspondence relationship between a captured image and a V map. FIG. 14 shows an example in which a region where a straight road surface can be seen is divided into three regions 591, 592, and 593.

Subsequently, the end segment correction unit 14 image-recognizes each area of the captured image (step S302).

Any method may be used as the image recognition method in step S302. For example, a template for an object such as a vehicle, a person, a road surface, and raindrops may be prepared in advance, and the object may be recognized by performing template matching.

Also, for example, "Csurka, G., Bray, C., Dance, C. and Fan, L, "Visual categorization with bags of keypoints," Proc. ECCV Workshop on Statistical Learning in Computer Vision, pp.1-22, 2004.”, a Bag-of-Visual-Words method used for recognition by using a vector called visual words in which a local feature quantity such as SIFT (Scale-Invariant Feature Transform) is quantized. May be used. Alternatively, as described in "Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton., "Imagenet classification with deep convolutional neural networks," In Advances in neural information processing systems, pp.1097-1105, 2012., A method using deep learning in which neural networks are layered may be used.

Further, when recognizing an image, a predetermined area of the reference image may be divided into smaller areas, and the recognition processing may be sequentially performed while raster scanning. In this case, since the area including noise can be removed rather than the entire segment being subjected to the recognition processing, the recognition accuracy can be improved.

Then, the end segment correction unit 14 determines the end segment based on the result of the image recognition in step S302 (step S303).

The parallax point of each segment of the V map is due to the object shown in the region corresponding to each segment in the luminance image. Therefore, if it is determined by image recognition whether or not an obstacle such as a side wall is reflected in each region in the brightness image, it can be determined whether or not each segment of the V map corresponding to each region is the end segment.

For example, when an obstacle such as a side wall is seen up to the upper end of the highest position area in each area in the luminance image, the end segment correction unit 14 determines that the segment including the lower end of the obstacle is the end segment. May be determined.

FIG. 15 is a diagram illustrating a correspondence relationship between a captured image of an obstacle such as a side wall and a V map.

For example, when an obstacle is shown (existing) in a part of the area 592 of the captured image and the entire area 593, the parallax extends vertically in the segment 592a on the V map. The parallax points do not appear in the segment 593a that is distributed and is farther than that. In this case, the segment 592a is determined to be the end segment.

According to the third embodiment, the accuracy of searching for the end segment is improved even in a situation where parallax-dispersed noise occurs due to the reflection of rain, for example, in bad weather.

The image recognition described above may be performed using a parallax image instead of the captured image (luminance image). Alternatively, both the captured image and the parallax image may be used. In this case, for example, the recognition result may be weighted to recognize the obstacle, and it may be determined that the obstacle exists when the average value of the scores of the recognition results is larger than a predetermined value.

The end segment may be determined based on the value obtained by weighting and adding the end segment determination result of the third embodiment and the end segment determination result of the first and second embodiments.

[Fourth Embodiment]
In the fourth embodiment, when estimating the road surface of each segment, based on the data of the end segment determined in the captured image (past frame) captured in the past, the termination determined in the captured image captured this time. An example of correcting the road surface of the segment will be described.

The fourth embodiment is the same as the first embodiment except for some parts, and therefore the description thereof will be appropriately omitted. Differences from the first embodiment will be described below in details of the processing of the in-vehicle device control system 1 according to the fourth embodiment.

FIG. 16 is a flowchart showing an example of the road surface estimation processing according to the fourth embodiment.

The road surface estimation unit 13 divides the V map into a plurality of segments according to the value of the parallax d that is the horizontal axis of the V map (step S21).

Subsequently, the road surface estimation unit 13 sets the segment having the shortest distance from the vehicle in the V map as the segment to be processed (step S22).

Then, the road surface estimation unit 13 extracts sample points from the pixels included in the segment to be processed (step S23).

FIG. 17 is a diagram illustrating an example of sample point extraction processing according to the fourth embodiment. For example, in the second segment, the road surface estimation unit 13 determines the end point position of the road surface 561 of the previous segment (the segment that has already detected the road surface and is adjacent to the second segment) (the road surface 561 of the previous segment is the second segment). (A position in contact with the segment) is set as a reference point 562. Then, the road surface estimation unit 13 sets a predetermined range corresponding to the reference point 562 as a search area 563 for sample points.

For example, as shown in FIG. 17A, the road surface estimation unit 13 searches for two straight lines 564 and 565 having a reference point as a starting point and extending between the two straight lines 564 and 565 extending to an adjacent segment at a predetermined angle. It may be the area 563. In this case, the road surface estimation unit 13 may determine the two straight lines 564 and 565 in correspondence with the angle at which the road surface can tilt. Alternatively, the road surface estimation unit 13 may set the search area 563 to a rectangular area having a height corresponding to the y coordinate of the reference point 562, as shown in FIG.

Then, the road surface estimation unit 13 excludes inappropriate points from the extracted sample points (step S24).

Then, the road surface estimation unit 13 detects the shape (position, height) of the road surface in the segment to be processed based on the sample points not excluded in step S24 (step S25).

Subsequently, when the detected (estimated) road surface is inappropriate, the road surface estimation unit 13 supplements the road surface in the segment to be processed (step S26).

Then, the end segment correction unit 14 determines whether the segment to be processed is the end segment (step S27). In step S27, the end segment correction unit 14 acquires the vehicle speed of the own vehicle 100 using the data I/F 603, calculates the moving amount of the obstacle according to the vehicle speed, and calculates the moving amount of the calculated obstacle. The end segment in the current frame may be determined according to

When the segment to be processed is the end segment (YES in step S27), the end segment correction unit 14 corrects the road surface of the end segment based on the history road surface of the end segment (step S28), and in step S31 described later. Go to processing. In step S28, the end segment correction unit 14 may set the road surface of the end segment calculated in the previous frame as the road surface of the end segment calculated in the current frame. Alternatively, the end segment correction unit 14 acquires the vehicle speed of the host vehicle 100 by using the data I/F 603, calculates the amount of movement of the road surface according to the vehicle speed, and ends the end segment according to the calculated amount of movement of the road surface. The road surface of the segment may be modified.

In addition, when the history road surface is not stored due to start-up or the like, the end segment correction unit 14 corrects and corrects the road surface of the end segment by the processing described in the above-described first to third embodiments. The road surface that has been subjected may be stored as a history road surface.

If the segment to be processed is not the end segment (NO in step S27), the road surface estimation unit 13 determines whether or not there is a next segment in the V map that is farther from the vehicle than the segment to be processed. (Step S29).

If there is a next segment (YES in step S29), the next segment is selected as a processing target (step S30), and the process proceeds to step S23.

If there is no next segment (NO in step S29), the road surface estimation unit 13 corrects each road surface estimated in each segment so that each road surface is continuous (smoothing processing) (step S31).

In step S23, as shown in FIG. 17, the road surface estimation unit 13 determines the search area of the sample point in the segment to be processed in order from the segment having a shorter distance from the host vehicle, based on the road surface of the previous segment. I shall. In this way, when the road surface estimated in the previous segment affects the road surface estimated in the next segment, the road surface estimation process and the end segment correction process are sequentially executed, and when processing the next segment, , The road surface of the previous segment must be modified.

According to the fourth embodiment, since the road surface estimation processing and the end segment correction processing are sequentially executed for each segment, for example, the search area of the sample point in the segment to be processed is set based on the road surface of the preceding segment. You can decide.

[Fifth Embodiment]
In the fifth embodiment, an example will be described in which when the uphill is erroneously determined to be an obstacle such as a side wall, the end segment correction process is not performed.

The fifth embodiment is the same as the first embodiment except for some parts, and therefore the description thereof will be appropriately omitted. The details of the processing of the in-vehicle device control system 1 according to the fifth embodiment will be described below with respect to the differences from the first embodiment.

In the first embodiment and the second embodiment, when the front of the host vehicle is an uphill, the height of the road surface estimated in a certain segment on the V map penetrates upward, and the parallax point in the farther segment. There may not be.

FIG. 18 is a diagram illustrating an end segment correction process on an uphill. As shown in FIG. 18(A), on the V map, the vehicle is divided into first to seventh segments in order of increasing distance from the host vehicle, and an uphill scene in which the road surface height increases in the sixth segment. Suppose there is. In this case, since the parallax point is small in the seventh segment, in the processing described in the first and second embodiments, the sixth segment is determined to be the end segment, and the road surface 541 in the end segment has a high height. There is a problem of being corrected to a low road surface 542.

Therefore, the end segment correction unit 14 according to the fifth embodiment performs the following processing.

FIG. 19 is a flowchart showing an example of the end segment correction processing according to the fifth embodiment.

The end segment correction unit 14 divides the V map into a plurality of segments (step S501).

The end segment correction unit 14 determines whether or not there is an uphill segment in each of the divided segments (step S502).

As shown in FIG. 18(B), when the predetermined range 551 of a certain segment includes a predetermined number (for example, 1) or more of parallax points, the end segment correction unit 14 determines this segment to be an uphill segment. You may. The predetermined range 551 may be a range in which the position is higher than the reference, for example, with the end (left end) of the road surface of the previous segment as the reference.

If there is an uphill segment (YES in step S502), the process ends.

When there is no uphill segment (NO in step S502), the end segment correction unit 14 searches for the end segment (step S503). The process of searching for the end segment may be the same as that of the above-described first embodiment or second embodiment, for example.

Subsequently, the end segment correction unit 14 corrects the road surface so that the road surface of the end segment becomes low (step S504).

Subsequently, the end segment correction unit 14 corrects the road surface of the segment that is farther from the host vehicle than the end segment (step S505).

According to the fifth embodiment, when the road surface is an uphill road, control is performed so that the end segment correction processing is not performed, so that the road surface can be estimated more appropriately.

<Summary>
According to each of the above-described embodiments, among the segments in the V map, the segment before the segment whose value corresponding to the number of parallax points is less than a predetermined threshold is determined to be the end segment, and the height of the road surface in the end segment is determined. Modify the height to be low. This makes it possible to improve the accuracy of detecting the road surface even when the parallax of an obstacle such as a side wall or a guard rail is detected.

Since the distance value (distance value) and the parallax value can be treated equivalently, the parallax image is used as an example of the distance image in the present embodiment, but the present invention is not limited to this. For example, a distance image may be generated by integrating distance information generated by using a detection device such as a millimeter wave radar or a laser radar with a parallax image generated by using a stereo camera. Further, a stereo camera may be used in combination with a detection device such as a millimeter wave radar or a laser radar, and the detection result of an object by the stereo camera described above may be combined to further improve the detection accuracy.

It goes without saying that the system configuration in the above-described embodiment is an example, and there are various system configuration examples according to the use and purpose. Further, it is possible to combine some or all of the above-described embodiments.

For example, in the road surface estimation processing, processing such as outlier removal, road surface supplementation, and smoothing processing is not essential, so these processings may be omitted.

The processing hardware unit 510, the image analysis unit 600, and the functional units of the vehicle travel control unit 104 may be implemented by hardware, or the CPU may execute a program stored in a storage device. The configuration may be realized by. The program may be recorded in a computer-readable recording medium and distributed by a file in an installable format or an executable format. Further, examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), and a Blu-ray disc. A recording medium such as a CD-ROM in which each program is stored, and the HD 504 in which these programs are stored can be provided as a program product (Program Product) domestically or overseas.

1 In-vehicle device control system (an example of "device control system")
100 own vehicle 101 image pickup unit 103 display monitor 106 vehicle traveling control unit (an example of "control section")
11 Parallax image generation unit (an example of “distance image generation unit”)
12 V map generation unit (an example of “generation unit”)
13 Road surface estimation unit (an example of "estimation unit")
14 End segment correction unit (an example of "correction unit")
15 Road Surface Height Table Calculation Unit 16 Clustering Unit (Example of “Object Detection Unit”)
17 Rejection Unit 18 Tracking Unit 19 Control Unit 2 Imaging Device 510a, 510b Imaging Unit 510 Processing Hardware Unit 600 Image Analysis Unit (an example of “image processing device”)

JP, 2005-075800, A

Claims (18)

The distribution of the frequency of distance values in the vertical direction of the distance image is shown from the distance image having the distance value according to the distance between the road surface and the obstacle obstructing the road surface in the plurality of captured images respectively captured by the plurality of imaging units. A generator for generating vertical distribution data,
Based on the vertical distribution data, an estimation unit that estimates the height of the road surface according to the distance,
The height of the road surface at the distance of the obstacle estimated by the estimation unit, a correction unit for correcting a low ,
The correction unit divides the distribution of the frequency of the distance value in the vertical direction of the distance image into a plurality of segments according to the distance value, and the value according to the divided distribution is less than a predetermined threshold value. Correcting the height of the road surface in the second segment having the distance value smaller than that of the first segment to be lower than the height estimated by the estimation unit,
An image processing device characterized by the above. In the second segment, the number of distance values in which the frequency value of each of the distance values included in each segment is equal to or more than a predetermined value and the number of distance points in the vertical direction is equal to or more than a predetermined threshold value is predetermined. Among the segments that are equal to or greater than the threshold of, the distance value included in the segment is the largest segment,
The image processing apparatus according to claim 1, wherein In the second segment, among the segments in which the number of the distance points in the vertical direction in which the frequency value of the distance value included in each segment is a predetermined value or more, is the distance included in the segment The segment with the highest value,
The image processing apparatus according to claim 1, wherein The distribution of the frequency of the distance value in the vertical direction of the distance image is shown from the distance image having the distance value according to the distance between the road surface and the obstacle obstructing the road surface in the plurality of captured images respectively captured by the plurality of imaging units. A generator for generating vertical distribution data,
Based on the vertical distribution data, an estimation unit that estimates the height of the road surface according to the distance,
The height of the road surface at the distance of the obstacle estimated by the estimation unit, a correction unit for correcting a low,
The correction unit image-recognizes the obstacle in the captured image, and corrects the height of the road surface at the distance of the obstacle estimated by the estimation unit to be low.
An image processing device characterized by the above. The distribution of the frequency of the distance value in the vertical direction of the distance image is shown from the distance image having the distance value according to the distance between the road surface and the obstacle obstructing the road surface in the plurality of captured images respectively captured by the plurality of imaging units. A generator for generating vertical distribution data,
Based on the vertical distribution data, an estimation unit that estimates the height of the road surface according to the distance,
The height of the road surface at the distance of the obstacle estimated by the estimation unit, a correction unit for correcting a low,
The correction unit determines whether the road surface is an uphill slope, and if the road surface is an uphill slope, does not correct the height of the road surface at the distance of the obstacle estimated by the estimation unit.
An image processing device characterized by the above. The distribution of the frequency of the distance value in the vertical direction of the distance image is shown from the distance image having the distance value according to the distance between the road surface and the obstacle obstructing the road surface in the plurality of captured images respectively captured by the plurality of imaging units. A generator for generating vertical distribution data,
Based on the vertical distribution data, an estimation unit that estimates the height of the road surface according to the distance,
The height of the road surface at the distance of the obstacle estimated by the estimation unit, a correction unit for correcting a low,
The correction unit discards the data of the height of the road surface at a distance having a distance value larger than the distance of the obstacle,
An image processing device characterized by the above. The correction unit corrects the height of the road surface at the distance of the obstacle, based on the height of the road surface estimated by the estimation unit according to the plurality of captured images previously captured.
The image processing apparatus according to any one of claims 1 to 6, characterized in that. The correction unit corrects the height of the road surface at the distance of the obstacle estimated by the estimation unit to the height of the road surface at a distance whose distance value is smaller than the distance of the obstacle estimated by the estimation unit. To do
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A plurality of imaging units,
From a plurality of captured images captured by each of the plurality of imaging units, a distance image generation unit that generates a distance image having a distance value according to the parallax of the road surface and the road surface in the plurality of captured images,
A generation unit that generates vertical distribution data indicating a distribution of frequency of distance values in the vertical direction of the distance image,
An estimation unit that estimates the height of the road surface according to the distance value based on the vertical direction distribution data,
The height of the road surface at the distance value of the obstacle estimated by the estimation unit, a correction unit for correcting a low ,
The correction unit divides the distribution of the frequency of the distance value in the vertical direction of the distance image into a plurality of segments according to the distance value, and the value according to the divided distribution is less than a predetermined threshold value. Correcting the height of the road surface in the second segment having the distance value smaller than that of the first segment to be lower than the height estimated by the estimation unit,
Imaging device. A plurality of imaging units,
From a plurality of captured images respectively captured by the plurality of imaging units, a distance image generation unit that generates a distance image having a distance value according to the parallax of the road surface and the road surface in the plurality of captured images,
A generation unit that generates vertical distribution data indicating a distribution of frequency of distance values in the vertical direction of the distance image,
Based on the vertical distribution data, an estimation unit that estimates the height of the road surface according to the distance,
The height of the road surface at the distance of the obstacle estimated by the estimation unit, a correction unit for correcting a low,
The correction unit image-recognizes the obstacle in the captured image, and corrects the height of the road surface at the distance of the obstacle estimated by the estimation unit to be low.
Imaging device. A plurality of imaging units,
From a plurality of captured images respectively captured by the plurality of imaging units, a distance image generation unit that generates a distance image having a distance value according to the parallax of the road surface and the road surface in the plurality of captured images,
A generation unit that generates vertical distribution data indicating a distribution of frequency of distance values in the vertical direction of the distance image,
Based on the vertical distribution data, an estimation unit that estimates the height of the road surface according to the distance,
The height of the road surface at the distance of the obstacle estimated by the estimation unit, a correction unit for correcting a low,
The correction unit determines whether the road surface is an uphill slope, and if the road surface is an uphill slope, does not correct the height of the road surface at the distance of the obstacle estimated by the estimation unit.
Imaging device. A plurality of imaging units,
From a plurality of captured images respectively captured by the plurality of imaging units, a distance image generation unit that generates a distance image having a distance value according to the parallax of the road surface and the road surface in the plurality of captured images,
A generation unit that generates vertical distribution data indicating a distribution of frequency of distance values in the vertical direction of the distance image,
Based on the vertical distribution data, an estimation unit that estimates the height of the road surface according to the distance,
The height of the road surface at the distance of the obstacle estimated by the estimation unit, a correction unit for correcting a low,
The correction unit discards the data of the height of the road surface at a distance having a distance value larger than the distance of the obstacle,
Imaging device. An imaging device mounted on a moving body, for imaging the front of the moving body, the imaging device according to any one of claims 9 to 12,
Moving surface modified by the imaging device, and based on the distance image, an object detecting unit that detects an object,
Based on the data of the object detected by the object detection unit, a control unit for controlling the moving body,
A mobile device control system including: Computer
The distribution of the frequency of the distance value in the vertical direction of the distance image is shown from the distance image having the distance value according to the distance between the road surface and the obstacle obstructing the road surface in the plurality of captured images respectively captured by the plurality of imaging units. Generating vertical distribution data,
Estimating the height of the road surface according to the distance based on the vertical direction distribution data,
A distribution of frequency of distance values in the vertical direction of the distance image is divided into a plurality of segments according to the distance value, and a value according to the divided distribution is smaller than a first segment whose value is less than a predetermined threshold value. Modifying the height of the road surface in the second segment having a small distance value to be lower than the height estimated by the estimating step;
Image processing method for executing. Computer
The distribution of the frequency of the distance value in the vertical direction of the distance image is shown from the distance image having the distance value according to the distance between the road surface and the obstacle obstructing the road surface in the plurality of captured images respectively captured by the plurality of imaging units. Generating vertical distribution data,
Estimating the height of the road surface according to the distance based on the vertical direction distribution data,
Image-recognizing the obstacle in the captured image, and correcting the height of the road surface at the distance of the obstacle estimated by the estimating step to be low,
Image processing method for executing. Computer
The distribution of the frequency of the distance value in the vertical direction of the distance image is shown from the distance image having the distance value according to the distance between the road surface and the obstacle obstructing the road surface in the plurality of captured images respectively captured by the plurality of imaging units. Generating vertical distribution data,
Estimating the height of the road surface according to the distance based on the vertical direction distribution data,
A step of correcting the height of the road surface at the estimated distance of the obstacle to be low, determining whether or not the road surface is an uphill, and if the road is an uphill, it is estimated by the estimating step. Not correcting the height of the road surface at the distance of the obstacle
Image processing method for executing. Computer
The distribution of the frequency of the distance value in the vertical direction of the distance image is shown from the distance image having the distance value according to the distance between the road surface and the obstacle obstructing the road surface in the plurality of captured images respectively captured by the plurality of imaging units. Generating vertical distribution data,
Estimating the height of the road surface according to the distance based on the vertical direction distribution data,
A step of correcting the height of the road surface at the estimated distance of the obstacle to be low, discarding data of the height of the road surface at a distance having a distance value larger than the distance of the obstacle,
Image processing method for executing. On the computer,
A program for executing the image processing method according to claim 14 .

Images