JP5579297B2 - Parallax calculation method and parallax calculation device - Google Patents

Description

  The present invention relates to a method for calculating parallax by stereo matching using a vehicle-mounted stereo camera for measuring and detecting a vehicle in front of the host vehicle or a road environment such as a pedestrian, a wall, a fence, and an implant. And in the field of equipment.

  Stereo matching refers to the corresponding points corresponding to each point of the reference image data, which is stereo image data shot with one lens, among the stereo image data shot with a stereo camera equipped with two lenses. The search is performed from a search area of reference image data which is stereo image data photographed by the other lens. As a method for determining the corresponding points, a small area is extracted around the reference point that is each point of the reference image data, and the extracted small area is compared with the small area in the search area. Of evaluation values such as the sum of absolute differences (SAD), sum of squared differences (SSD), normalized cross correlation (NCC), and the like. A method of determining using a standard is generally used.

  However, if the object to be measured includes an object including a continuous similar pattern such as a pedestrian crossing or a fence, in which a bar-like or rectangular pattern is repeated, a corresponding point candidate in SAD, SSD, or NCC Are calculated in plural, and it is difficult to calculate the corresponding points in principle.

  As a conventional countermeasure when a plurality of corresponding point candidates are calculated, there is a method of not outputting the corresponding point of the reference point as unknown (see, for example, Patent Document 1). Further, there is a method of selecting corresponding points according to application control modes such as pre-crash control, inter-vehicle control with a preceding vehicle, and parking support control (for example, see Patent Document 2). FIG. 12 shows a conventional stereo matching method described in Patent Document 2.

  In FIG. 12, the stereo image data acquisition unit 1202 is a pair of image data photographed simultaneously with a stereo camera equipped with two lenses, as standard image data photographed with one lens, and a reference image photographed with the other lens. Get data and.

  The stereo matching unit 1203 calculates the difference in image luminance between each point of the standard image data and the search point in the search area of the reference image data by SAD, and sets a set of similarities in the search area as an evaluation value distribution. calculate. Corresponding point candidate multiple existence determination section 1204 determines whether or not there are a plurality of corresponding point candidates from the evaluation value distribution.

  The evaluation value minimum corresponding point calculation unit 1205 calculates, as a corresponding point candidate, a search point at which the evaluation value that is the degree of difference in image luminance is minimum with respect to a reference point that is determined to have no plurality of corresponding point candidates. . The control mode data acquisition unit 1206 acquires control mode data representing the mode of the control mode.

  If the control mode is pre-crash control, the control mode corresponding point calculation unit 1207 selects the farthest corresponding point candidate with respect to the reference point determined to have a plurality of corresponding point candidates, and the control mode is the inter-vehicle distance. In the case of control, the closest corresponding point candidate is selected, and in the case where the control mode is parking assist control, the closest corresponding point candidate is selected.

The disparity data output unit 1208 substitutes the corresponding point with the smallest evaluation value for the reference point determined to have no plurality of corresponding point candidates, and for the reference point determined to have a plurality of corresponding transfer candidates Thus, the corresponding points selected by the control mode are substituted to calculate the parallax data for the reference image.

JP 2001-351200 A JP 2007-85773 A

  However, in the conventional configuration, since the position of the corresponding point is determined according to the control mode, it is difficult to measure the correct distance. For example, in practice, when the distance between successive similar patterns included in an object existing at a distance of 5.7 m from the host vehicle is 10 cm, it is assumed that the distance between two lenses constituting the stereo camera ( When the base line length is 0.12 m and the focal length of the camera is 1000 pixels, a plurality of corresponding point candidates with parallax of 5, 21, and 37 are calculated because they are minimal at similar pattern positions in the search region. . In such a case, since the relationship of (parallax) = (baseline length) × (focal length) / (distance) is established from the principle of triangulation, the corresponding point candidate with the parallax of 0.12 × 1000 / 5.7 = 21 Is the correct corresponding point.

  However, if the closest corresponding point candidate is selected, the corresponding point candidate with parallax of 37 is selected, so the distance is miscalculated as 0.12 × 1000/37 = 3.2 m, while the farthest distance candidate is selected. When a corresponding point candidate is selected, a corresponding point candidate with a parallax of 5 is selected, so that it is erroneously calculated as 0.12 × 1000/5 = 24 m.

  Therefore, when rod-like objects are arranged at intervals of 10 cm like a fence, and the similar image pattern is recognized as a continuous object in the parallax detection device, even if there is pre-crash control of the control mode, the brake control May not work well. Further, when the control mode is the inter-vehicle control, there is a possibility that the vehicle is not sufficiently decelerated, and when the control mode is the parking assist control, it may be difficult to park at an appropriate place.

  Furthermore, when there is an object including a continuous similar pattern such as a fence, it is difficult to detect the fence, and thus there is a problem that the determination of the travel path may be insufficient.

  An object of the present invention is to provide a parallax calculation method and apparatus for calculating a correct parallax even when there is an object including a continuous similar pattern.

  A disparity calculating apparatus according to one embodiment of the present invention includes a first imaging system and a second imaging system, and includes reference image data obtained by capturing an object using the first imaging system, and an object. Difference in image brightness between a stereo image data acquisition unit that acquires data of a reference image captured using the second imaging system, a reference point that the reference image has, and a plurality of search points that the reference image has Detected when a search point that has a minimum evaluation value is detected from the stereo matching unit that calculates the evaluation value distribution indicating the degree and the search points included in the evaluation value distribution, and the number of detected search points is plural A corresponding point candidate number determination unit that outputs a plurality of search points as corresponding point candidates, and a minimum value distribution indicating a coordinate distribution of the corresponding point candidates from an evaluation value distribution including the corresponding point candidates. Includes a first local minimum distribution corresponding to a point and a first reference point A minimum value distribution calculating unit that calculates a second minimum value distribution corresponding to one or a plurality of second reference points existing in the peripheral region of the quasi-image; a first minimum value distribution and a second minimum value; In the evaluation value map, an evaluation value map calculation unit that calculates an evaluation value map indicating a relationship between coordinate fluctuation values of the first reference point and the second reference point and a plurality of corresponding point candidates based on the value distribution, The difference between the corresponding point determination unit that determines the corresponding point candidate having the smallest coordinate variation value as the corresponding point, the coordinate point in the reference image of the corresponding point, and the coordinate point in the reference image of the first reference point And a parallax data output unit that outputs a parallax value.

  Thereby, in the reference image, by superimposing information on another reference point existing in the peripheral region of the reference point, it is possible to calculate a correct parallax even for an object including a continuous similar pattern. It has the effect.

  In the parallax calculation device according to an aspect of the present invention, the range of the peripheral region is determined based on the number of search points at which the evaluation value included in the minimum value distribution of the first reference point is minimum, and the interval. It is.

  Accordingly, there is an effect that the size of each object can be calculated with higher accuracy when there are a plurality of objects including continuous similar patterns in the reference image and the distances between the objects are different.

  In the parallax calculation device according to one aspect of the present invention, the corresponding point determination unit has a different number of corresponding point candidates related to the first reference point and a corresponding point candidate number related to the second reference point. In addition, the corresponding points are determined by excluding the corresponding point candidates included only in either the first reference point or the minimum value distribution related to the second reference point.

  This has the effect of reducing the information processing load applied to the parallax calculation device and calculating the correct parallax.

  In the parallax calculation device according to one aspect of the present invention, the corresponding point determination unit extracts a corresponding point candidate having the smallest coordinate variation value as a corresponding point by performing Hough transform on the evaluation value map. Is.

  Thereby, there is an effect that the distance can be calculated with higher accuracy in the case where the objects including continuous similar patterns exist in a straight line.

  In the parallax calculation device according to one embodiment of the present invention, the first imaging system includes the first lens, the second imaging system includes the second lens, and the peripheral region includes the first lens. And the second lens, the evaluation value map calculation unit includes the first evaluation value distribution, the second evaluation value distribution, and a reference existing in the peripheral region. An evaluation value map is calculated based on a local minimum value distribution corresponding to a third reference point existing in a direction perpendicular to the arrangement direction of the first lens and the second lens with respect to the point. is there.

  Thereby, after superimposing the local minimum value distribution of another reference point that exists in the direction perpendicular to the arrangement direction of the first lens and the second lens, the local minimum value of each reference point existing in the peripheral region By calculating the evaluation value map based on the distribution, there is an effect that the accuracy of corresponding point extraction is improved.

  In the parallax calculation device according to an aspect of the present invention, the range of the peripheral region includes a plurality of searches of an evaluation value distribution corresponding to the first reference point and an evaluation value distribution corresponding to the second reference point. It is determined based on the difference sum of the differences in image luminance corresponding to the points.

  Thus, by including a reference point of the same parallax (distance) as a peripheral region, there is an effect that the accuracy of estimating a correct parallax value is increased.

  In addition, in the parallax calculation method according to one aspect of the present invention, the reference image data obtained by imaging the target using the second imaging system and the data of the standard image obtained by imaging the target using the first imaging system. And calculating an evaluation value distribution indicating a degree of difference in image brightness between a reference point included in the reference image and a plurality of search points included in the reference image, and from the search points included in the evaluation value distribution When a search point having a minimum evaluation value is detected, and the number of detected search points is plural, the plurality of detected search points are output as corresponding point candidates, and the evaluation value distribution includes the corresponding point candidates Is a minimum value distribution indicating the coordinate distribution of the corresponding point candidate, and is present in the peripheral region of the reference image including the first minimum value distribution corresponding to the first reference point and the first reference point, A second minimum value distribution corresponding to a plurality of second reference points, and a first Based on the small value distribution and the second minimum value distribution, an evaluation value map indicating the relationship between the first reference point and the second reference point and a plurality of corresponding point candidate coordinate variation values is calculated, and the evaluation value In the map, a corresponding point candidate having the smallest coordinate variation value is determined as a corresponding point, and a parallax value that is a difference between the coordinate point in the reference image of the corresponding point and the coordinate point in the reference image of the first reference point is determined. Output.

  Thereby, in the reference image, by superimposing information on another reference point existing in the peripheral region of the reference point, it is possible to calculate a correct parallax even for an object including a continuous similar pattern. It has the effect.

  According to the present invention, in stereo matching, instead of searching for corresponding points in a comparison in a small region, it is difficult to calculate in stereo matching by determining corresponding points by adding comparison information of peripheral regions. It can calculate correct parallax for objects including image patterns, and can stably measure distances such as guardrails that are often present in traffic environments, fences that are often present in parking lots, etc. Can also work correctly.

The figure which shows the block configuration of the parallax calculation apparatus in Embodiment 1 of this invention. The figure which shows the processing flow of the parallax calculation method in Embodiment 1 of this invention. (A) The figure which shows the position of the target object in a reference | standard image among the stereo images in Embodiment 1 of this invention (b) The position and parallax of the target object in a reference image among the stereo images in Embodiment 1 of this invention are shown. Illustration (A) The figure which shows the position of the reference point to scan in the reference | standard image in Embodiment 1 of this invention (b) The figure which shows the search area | region and corresponding point candidate in the reference image in Embodiment 1 of this invention (c) This The figure which shows evaluation value distribution of the difference in Embodiment 1 of invention (A) The figure which shows the position of the reference point to scan in the reference | standard image in Embodiment 1 of this invention (b) The figure which shows the search area | region and corresponding point candidate in the reference image in Embodiment 1 of this invention (c) This The figure which shows evaluation value distribution of the difference in Embodiment 1 of invention (A) The figure which shows the position of the reference point to scan in the reference | standard image in the case of the object containing the continuous similar pattern in Embodiment 1 of this invention (b) The continuous similar pattern in Embodiment 1 of this invention is included FIG. 7C shows search areas and corresponding point candidates in a reference image in the case of an object. FIG. 8D shows an evaluation value distribution of the degree of difference in the case of an object including continuous similar patterns in the first embodiment of the present invention. The figure which shows calculation of evaluation value distribution of minimum value from the difference degree in the case of the object containing the continuous similar pattern in Embodiment 1 of invention (A) The figure which shows the position of two reference points in the reference | standard image in Embodiment 1 of this invention (b) The figure which shows the candidate point candidate with respect to each reference | standard point in the reference image in Embodiment 1 of this invention (c) FIG. 4D is a diagram showing an evaluation value distribution corresponding to a predetermined reference point in the first embodiment of the present invention. FIG. 4D is a diagram showing an evaluation value distribution corresponding to a predetermined reference point in the first embodiment of the present invention. The figure which shows the corresponding point candidate in Embodiment 1 of (A) The figure which shows the position of the some reference point in the reference | standard image in Embodiment 1 of this invention (b) The figure which shows the search area | region with respect to each reference | standard point in the reference image in Embodiment 1 of this invention (c) This The figure which shows evaluation value distribution corresponding to each reference point in Embodiment 1 of invention (d) The figure which shows the evaluation value map in Embodiment 1 of this invention (A) The figure which shows the position of the some reference point in the reference | standard image in Embodiment 1 of this invention (b) The figure which shows the search area | region with respect to each reference | standard point in the reference image in Embodiment 1 of this invention (c) This The figure which shows evaluation value distribution corresponding to each reference point in Embodiment 1 of invention (d) The figure which shows the evaluation value map in Embodiment 1 of this invention (A) The figure which shows the evaluation value map in the relationship between the captured image of the object containing the continuous similar pattern in Embodiment 1 of this invention, and an evaluation value map (b) The continuous similar pattern in Embodiment 1 of this invention FIG. 8C is a diagram showing a reference image in the relationship between the captured image of the object including the evaluation value map and one evaluation value map in the relationship between the captured image of the object including the continuous similar pattern and the evaluation value map in the first embodiment of the present invention. Of the local minimum distribution from which a part is extracted (A) The figure which shows the position of the some reference point in the reference | standard image in Embodiment 1 of this invention (b) The figure which shows the search area | region with respect to each reference | standard point in the reference image in Embodiment 1 of this invention (c) This The figure which shows the evaluation value distribution with respect to each reference point in Embodiment 1 of invention (d) The figure which shows the result of having overlapped the evaluation value distribution in Embodiment 1 of this invention The figure which shows the block structure of the conventional parallax calculation apparatus.

  Embodiments of the present invention will be described below with reference to the drawings.

(Embodiment 1)
FIG. 1 is a block diagram of a disparity calculating apparatus 100 according to Embodiment 1 of the present invention. A parallax calculation apparatus 100 illustrated in FIG. 1 includes a parallax calculation unit 101, a stereo image data acquisition unit 102, and a parallax data output unit 103.

  Furthermore, the parallax calculation unit 101 includes a stereo matching unit 104, a corresponding point candidate number determination unit 105, a local minimum value distribution calculation unit 106, an evaluation value map calculation unit 107, and a corresponding point determination unit 108. is there.

  In the parallax calculation apparatus 100, the parallax calculation unit 101 performs parallax calculation based on the stereo image acquired by the stereo image data acquisition unit 102, and the parallax data output unit 103 outputs the parallax data.

  FIG. 2 is a process flow diagram of the parallax calculation method of the parallax calculation apparatus 100 illustrated in FIG. 1. Hereinafter, the parallax calculation method and the parallax calculation apparatus according to the first embodiment will be described.

  The stereo image data acquisition unit 102 acquires a pair of stereo image data captured simultaneously by a stereo camera equipped with two lenses arranged side by side (S201). The stereo image data includes standard image data that is data of a standard image captured by one lens and reference image data that is data of a reference image captured by the other lens. The stereo camera has been described as a camera equipped with two lenses arranged side by side. However, the present invention is not limited to this, and two cameras can be substituted.

  FIGS. 3A and 3B are stereo images in the case where the object to be measured is a front vehicle body. FIG. 3A shows a standard image and FIG. 3B shows a reference image. Which of the two lenses is used as the reference image is arbitrary, but hereinafter, the image taken with the right lens toward the object to be measured is used as the reference image and the left lens. The captured image will be described as a reference image.

  The position of the ranging object photographed in the reference image is shifted to the right as compared with the position of the ranging object photographed in the standard image. This deviation is parallax and changes depending on the distance of the object to be measured. Specifically, when the coordinate of the left end of the distance measurement object in the standard image is xb and the coordinate of the left edge of the distance measurement object in the reference image is xr, the parallax d at the left end of the distance measurement object is the difference in coordinate position. Xr-xb.

  The acquired stereo image data is subjected to lens distortion correction and optical axis parallelization correction, and converted to corrected stereo image data. Lens distortion correction methods include distortion correction using a correction conversion table using lens design values, and correction using parameter estimation using a radial distortion aberration model. The present invention can be realized by any method for correcting the above, and the present invention is not limited thereto.

  Further, the optical axis parallelization correction can be realized by any method for performing the optical axis parallelization of a stereo camera, and the present invention is not limited thereto. For example, parallelization correction can also be performed by a method in which a lattice pattern is set in a common field of view of a stereo camera, the relative relationship of the stereo camera is calculated from the associated lattice point position, and the optical axis parallelization correction is performed. it can.

  In the stereo matching unit 104, the parallax calculation unit 101 performs stereo matching between the reference image and the reference image acquired by the stereo image data acquisition unit 102 (S202).

  The stereo matching unit 104 determines the image luminance of each reference point included in an arbitrary range of each reference image and a search point included in a search area including a coordinate point corresponding to the reference point of the reference image in the reference image. An evaluation value distribution indicating the degree of difference is calculated. Here, the search area is an area having an arbitrary range.

  The calculated evaluation value distribution for each reference point is recorded in a memory built in the stereo matching unit 104.

  Corresponding point candidate number determination section 105 determines whether or not there are a plurality of corresponding point candidates for which the evaluation value is minimal in the evaluation value distribution of each reference point (S203).

  For each reference point in the reference image that is determined to have only one corresponding point candidate, the disparity data output unit 103 uses the corresponding point candidate position and the coordinate point that is the same as the reference value in the reference image. The difference from the coordinate point located at is recorded as the parallax value of the reference point. Then, the parallax data output unit 103 outputs the parallax data (S208).

  On the other hand, for each reference point in the reference image that is determined to have a plurality of corresponding point candidates, the local minimum value distribution calculation unit 106 calculates a corresponding point candidate whose evaluation value is minimum in the evaluation value distribution. Then, the local minimum value distribution is calculated (S204).

  The evaluation value map calculation unit 107 calculates an evaluation value map (S205). The evaluation value map is based on the minimum value distribution of the reference point determined to have a plurality of corresponding point candidates and the minimum value distribution of another reference point located in the peripheral region including the reference point in the reference image. FIG. 10 is a map showing a variation value for each corresponding point candidate of a disparity value of each corresponding point candidate with respect to each reference point.

  The corresponding point determination unit 108 extracts the most straight line segment in the evaluation value map (S206), and determines the extracted coordinate point of the corresponding point candidate as a corresponding point (S207).

  Hereinafter, the function and effect of each component of the parallax calculation unit 101 will be described in detail.

  The stereo matching unit 104 performs stereo matching between the standard image and the reference image. That is, an evaluation value indicating the degree of difference in image brightness between each reference point included in an arbitrary range of the reference image and each search point in the search area including the same coordinate point as each reference point in the reference image A distribution is calculated (S202).

  Here, the range of the search area is arbitrary, and is determined by geometric parameters including the range of the distance of the measured object, the base line length of the stereo camera, and the focal length of the camera.

  FIG. 4 shows (a) a reference image for calculating an evaluation value distribution, (b) a reference image, and (c) an evaluation value distribution for each reference point in an arbitrary range of the reference image. It is shown. Hereinafter, the procedure for calculating the evaluation value distribution for the reference point will be described.

  In the description, the arbitrary range of the reference image will be described as a partial range as illustrated, but may be the entire range. Further, the coordinate of the reference point for calculating the evaluation value distribution is assumed to be (xb1, yb1), and the stereo image data is assumed to be parallelized and corrected. When the stereo image data is corrected for parallelization, the Y coordinate of the object in the standard image is the same as the Y coordinate of the object in the reference image.

  In FIG. 4, the area indicated by the square is one pixel, and one pixel is a reference point. The same applies to the description of the following drawings.

  The stereo matching unit 104 uses the coordinates located at the same coordinates (xr1 = xb1, yr1 = yb1) as the reference points (xb1, yb1) in the reference image as search points corresponding to the reference points (xb1, yb1) in the reference image. From the point, a certain range (search width) in the X coordinate direction is set as a search region shown in FIG.

  The search area is an area having a certain width on the Yr axis, that is, in the horizontal direction in the reference image. The reason why the search area is set in the horizontal direction of the reference image is that the stereo camera has lenses arranged in the horizontal direction.

  Specifically, with respect to the coordinates of the reference point on the reference image (xb1, yb1), the range of the search point on the reference image is the minimum / maximum distance of the distance measurement target necessary for the application. When the search width determined based on R is (xr1, yr1), (xr1 + 1, yr1), (xr1 + 2, yr1), ..., (xr1 + R, yr1). Note that the coordinates (xb1, yb1) on the standard image and the coordinates (xr1, yr1) on the reference image have the same coordinate position on the image.

  Then, the degree of difference in image luminance between each search point in the search area and the reference point xb1 is calculated, and an evaluation value distribution H (xb1) based on the degree of difference is calculated (S202).

  Here, the relationship between the D axis (Depth axis) of the evaluation value distribution H (xb1), the Xb axis, and the Xr axis is the D axis obtained by shifting the origin of the Xr axis to the position of xb1, and the D axis, Xb The dimensions of the axis and the Xr axis are both pixels.

  In the above description, the stereo image data has been described as being subjected to parallelization correction. However, if the stereo image data is not subjected to parallelization correction, the stereo image cannot be aligned in the vertical direction. Since the point of the distance measuring object photographed at the reference point (xb1, yb1) of the reference image is not photographed at the same Y coordinate as the Y coordinate of the reference point in the reference image, the Y axis of the reference point It is set as a search area including alignment of directions.

  The evaluation value distribution H (xb1) obtained by stereo matching is a one-dimensional distribution in which the horizontal axis is the D axis and the vertical axis indicates the difference in image luminance between the reference point and the search point corresponding thereto. The difference in image luminance between the reference point and the search point is the absolute sum of image luminance differences between the 8 × 8 small region centered on the reference point and the 8 × 8 small region centered on each search point ( SAD) is used.

  Note that the degree of difference in image brightness used as the evaluation value can be replaced with any degree of difference that minimizes the value when there is a complete match, such as the sum of squared differences (SSD). In addition, the degree of difference in image luminance used as an evaluation value can be used with the similarity index having the maximum value, such as normalized correlation (NCC), in the case of a perfect match, being reversed.

  The above processing is performed for each reference point, and the calculated evaluation value distribution for each reference point is stored in a memory built in the stereo matching unit 104.

  Corresponding point candidate number determination section 105 searches for a minimum value in evaluation value distribution H (xb1).

  As a result of searching for the minimum value by the corresponding point candidate number determination unit 105, when one minimum value exists, the search point having the minimum evaluation value is determined as the corresponding point (S203: No), and the corresponding point candidate position is determined. Recording is performed as the parallax value of the reference point, and parallax data is output (S208).

  FIG. 5 shows a case where the object to be measured is a vehicle body positioned in front, (a) shows a reference image, (b) shows a reference image, and (c) shows an evaluation value distribution. When the object is not an object including a continuous similar pattern such as a vehicle body, the corresponding point candidate having the same image luminance as the small region including the reference point (xb1, yb1) is as shown in FIG. There is only one on the reference screen.

  On the other hand, if a plurality of minimum values are detected in the evaluation value distribution as a result of the search for the minimum value by the corresponding point candidate number determination unit 105, a plurality of search points having the minimum evaluation value are set as a plurality of corresponding point candidates. Determine (S203: Yes).

  FIG. 6 shows a case where the object to be measured is a fence located in front, (a) shows a reference image, (b) shows a reference image, (c) shows an evaluation value distribution, d) shows a minimum value distribution Hm (xb1) described later. In the case of an object including a continuous similar pattern such as a fence, the reference point (xb1, yb1) in the reference image and the corresponding point candidate having the same image brightness as shown in FIG. There are multiple reference screens.

  Note that the method for determining whether or not there are multiple corresponding point candidates is to calculate the minimum value of the evaluation value distribution and determine how many local minimum values exist. Other methods for determining whether or not are possible can also be realized.

  The local minimum value distribution calculation unit 106 finds the coordinates of the corresponding point candidates whose evaluation value is minimum when a plurality of local minimum values are detected in the evaluation value distribution as a result of the corresponding point candidate number determination unit 105 searching for the local minimum value. Extraction is performed to calculate a local minimum value distribution Hm (xb1) indicating only the local minimum value distribution (S204).

  Hereinafter, the calculation procedure of the minimum value distribution Hm (xb1) will be described with reference to FIG.

  As shown in FIG. 6 (a), since the distance measurement target is a fence including a continuous pattern, the evaluation value distribution H (xb1) is a minimum value at a plurality of positions as shown in FIG. 6 (c). The distribution has

  The minimum value distribution calculation unit 106 calculates a distribution obtained by extracting only the positions of a plurality of minimum values in the evaluation value distribution H (xb1), and as shown in FIG. 6D, the minimum value of the calculated evaluation value distribution is calculated. A minimum value distribution Hm (xb1) is calculated by substituting a value of -1 for the position of, and a value of 0 for the other positions.

  Here, the position of the minimum value of the minimum value distribution Hm (xb1) shown in FIG. 6D is the position of the corresponding point candidate in the reference image with the left end of the search area for the reference point (xb1, yb1) as the origin. Show.

  It should be noted that the minimum value position constant and other position constant values used when calculating the minimum value distribution are merely examples, and the present invention is not limited thereto.

  Note that a series of procedures for determining whether there are a plurality of corresponding point candidates for the evaluation value distribution for the reference point and calculating a minimum value distribution when there are a plurality of corresponding point candidates are stored in the memory. Performed for all each reference point.

  When determining a plurality of search points having a minimum evaluation value as corresponding point candidates, a predetermined threshold is provided on the evaluation value axis, and a search point having a predetermined threshold or less is set as a corresponding point candidate. It is good. Thereby, a search point that is a search point with a high degree of difference and is a minimum point can be excluded from the corresponding point candidates, and the determination accuracy of the corresponding point candidates can be improved.

  FIG. 7 shows a local minimum distribution Hm (xb1) of a reference point (xb1, yb1) where a plurality of corresponding point candidates exist, and another reference point (xb2, yb1) adjacent to the reference point (xb1, yb1) in the reference image. It is a schematic diagram which shows the relationship with minimum value distribution Hm (xb2) of yb2). (A) shows a reference image, (b) shows a reference image, (c) shows a local minimum distribution Hm (xb1) of the reference point (xb1, yb1), and (d) shows a reference point (xb2, yb2). ) Shows a minimum value distribution Hm (xb2), and (e) shows corresponding point candidates for the reference point (xb1, yb1) and corresponding point candidates for the reference point (xb2, yb2) in the reference image shown in (b). .

  Hereinafter, the relationship of the minimum value distribution between the reference point (xb1, yb1) and the adjacent reference point (xb2, yb2) will be described. The corresponding point candidates of the reference points (xb1, yb1) shown in FIGS. 7A and 7E are the corresponding point candidates 1 to 4 shown in FIG. 7B, and FIG. The corresponding point candidates of the reference points (xb2, yb2) shown in (e) and (e) will be described as the corresponding point candidate 8 from the corresponding point candidate 5 shown in FIG. 7 (b). Corresponding point candidate 1 to corresponding point candidate 4 are indicated by dotted lines.

  In the reference image shown in FIG. 7A, the reference point (xb2, yb2) is selected as a reference point adjacent to the reference point (xb1, yb1). The search area for the reference point (xb2, yb2) shown in FIG. 7B is an area shifted from the search area for the reference point (xb1, yb1) to the Xr axis by the difference between xb2 and xb1.

  Here, the position of each local minimum value on the D-axis of the local minimum distribution Hm (xb1) of the reference point (xb1, yb1) shown in FIG. 7C, and the state shown in FIG. Comparing the position of each local minimum value on the D axis of the local minimum distribution Hm (xb2) of the reference point (xb2, yb2), there is a local minimum that is shifted on the D axis by the difference between xb2 and xb1. , There is also a local minimum that is shifted larger or smaller than the difference between xb2 and xb1.

  Referring to FIG. 7E, the reference point (xb2, yb2) is shifted from the coordinates of the corresponding point candidate 3 to the reference point (xb1, yb1) by the difference between xb2 and xb1 on the Xr axis. Since there is a corresponding point candidate 7 for, and the distances between the points captured by the adjacent reference points are considered to be approximately equal, it means that each corresponding point candidate is likely to be a correct corresponding point.

  On the other hand, the corresponding point for the reference point (xb2, yb2) is shifted to the position on the Xr axis by the difference between xb2 and xb1 from the coordinates of the corresponding point candidate 1 for the reference point (xb1, yb1) in FIG. Since candidate 5 does not exist, it means that the corresponding point candidate is not a correct corresponding point. In this case, the areas of the corresponding point candidate 3 and the corresponding point candidate 7 are the correct corresponding point positions and correct parallax, and the corresponding point candidate 1 and corresponding point candidate 5 areas are the correct corresponding point positions. That's not true.

  The evaluation value map calculation unit 107 includes a reference point (xb1, yb1) for which the minimum value distribution is calculated by the minimum value distribution calculation unit 106, and a position existing in the peripheral region of the reference point xb1 in the reference image, or a plurality of points An evaluation value map is calculated based on the local minimum value distribution with another reference point xbn (n: natural number) (S205).

  Here, the peripheral region is a region including a reference point (xb1, yb1) in the reference image and having an arbitrary width in the Yb axis direction. In principle, each reference point included in the peripheral area is an area having the same Yb coordinate. The reason why the peripheral area is set in the horizontal direction in the standard image is that, by setting the search area in the horizontal direction of the reference image with respect to each standard point, there is a probability that a similar pattern is erroneously detected in the horizontal direction. .

  The search area is the horizontal direction of the reference image because the stereo cameras are arranged in the horizontal direction, and when the stereo cameras are arranged in the vertical direction, the search area is the vertical direction of the reference image, The peripheral area in the reference image is also in the vertical direction.

  FIG. 8 shows the minimum value distribution Hm (xb1) of the reference point (xb1, yb1) and the minimum value of each reference point (xbn, ybn) (n: natural number) located in the peripheral region of the reference point (xb1, yb1). It is a schematic diagram which shows the procedure which calculates an evaluation value map based on distribution Hm (xbn). (A) shows the standard image, (b) shows the reference image, (c) shows the relationship of the minimum value distribution Hm (xbn) of each standard point (xbn, ybn), and (d) shows the standard point ( The evaluation value map calculated by superimposing the minimum value distribution of xbn, ybn) is shown. Hereinafter, a method for calculating the evaluation value map shown in FIG. 8 will be described.

  In the reference image shown in FIG. 8A, calculation is performed for each of the other reference points (xb2, yb2), (xb3, yb3), (xb4, yb4) that exist in the peripheral region of the reference point (xb1, yb1). The evaluated evaluation value distribution Hm (xbn) is read from the memory. Here, description will be made assuming that there are four corresponding point candidates for each reference point, but the present invention is not limited thereto.

  As shown in FIG. 8C, among the four corresponding point candidates distributed in each local minimum value distribution Hm (xbn) (n: an integer from 1 to 4), the distance from the origin is different for each local minimum value distribution. Point candidates exist. In FIG. 8C, the distance from the origin of the corresponding point candidate on the line drawn by the lines a, b, and d is different for each minimum value distribution. This means that the parallax value for each reference point varies.

  The evaluation value map M (D, Xb) aligns the D axes of the respective minimum value distributions Hm (xbn) (n: an integer from 1 to 4) shown in FIG. 8C and corresponds to the Xb axis on the D axis. The variation value of the point candidate, that is, the variation value of the parallax value is indicated by a line with respect to the Xb axis. As information processing, the dimension of the Xb axis is increased and the position of the corresponding point candidate is processed as a three-dimensional distribution.

  That is, the evaluation value map M (D, Xb) shows the relationship between the coordinate fluctuation values of each reference point and a plurality of corresponding point candidates based on each local minimum value distribution shown in FIG. .

  In the evaluation value map M (D, Xb), the corresponding point candidates on the line c are not changed in the disparity values of the corresponding point candidates with respect to the reference points, that is, the reference points exist in the same disparity, that is, the same distance. And the correct parallax value.

  Here, xb1 represents the X coordinate of the reference point, and xb2,..., Xbn represent the evaluation value map M (D, Xb) when the X coordinate of the reference point adjacent to the reference point in the horizontal direction is represented. The mathematical formula to be calculated is as shown in the following mathematical formula 1.

  In the above description, an arbitrary region in the horizontal direction of the image of the Y coordinate (yb1) is set as the peripheral region of the reference point (xb1, yb1), but the Xb axis of the Y coordinate (yb1) in the reference image All of the above reference points may be included.

  Note that the peripheral region setting method includes, for example, the feature value of the evaluation value distribution of the reference point, that is, the number of minimum values and the interval between the minimum values, and the feature value of the evaluation value distribution of another reference point existing in the peripheral region. The selected range can be up to a peripheral reference point where the difference between and a certain range. By limiting the selection range, the correct parallax of each object can be calculated when there are a plurality of objects including similar patterns that are continuous and the positions are not consecutive to each other.

  Corresponding point determination unit 108 has the most linear line among the lines a to d shown in evaluation value map M (D, Xb), that is, the coordinate variation value of the parallax value among the corresponding point candidates. Corresponding point candidates with small values are calculated as corresponding points. In the evaluation value map M (D, Xb) shown in FIG. 8D, the line c is the straight line having the strongest linear component among the lines a to d.

  Specifically, in order to extract an object that continuously exists on a straight line such as a fence, the most linear component in which points with small evaluation values are continuous on a straight line in the evaluation value map M (D, Xb). A strong line is extracted by the Hough transform (S206). The Hough transform is one basic image processing that extracts a linear component on an image. Here, as a result of the Hough transform performed on the entire evaluation value map, a line having the strongest linear component is extracted.

  In the physical sense, when objects containing continuous similar patterns are arranged in a straight line in real space, the corresponding points are distributed on the most linear line on the evaluation value map. .

  A point with a small evaluation value in the evaluation value map is a point that represents the position where the object exists, and an object including a continuous similar pattern such as a fence cannot always be associated with a pillar portion at the same position in the reference image and the reference image. Therefore, a plurality of object location candidate candidates are generated.

  When there are multiple corresponding point candidates, a plurality of lines are obtained on the evaluation value map. However, in a straight line that is not a true parallax position, there is a portion where the linearity is partially disturbed on the evaluation value map. Places that are not distributed.

  The said part is a part of a support | pillar from which the thickness of a fence differs a little in real space, and is a light-colored part in a pedestrian crossing sign. That is, an object including a continuous similar pattern, such as a fence, is actually disturbed in periodicity, and the position of a plurality of corresponding point candidates in the part and a plurality of corresponding points in a part without periodic disturbance. The position of the candidate is different. For this reason, an object including a continuous similar pattern, such as a fence, has corresponding points that match only with true parallax when the positions of multiple corresponding point candidates are compared between a part with periodicity disorder and a part without periodicity. No match except true parallax.

  When expressed as an evaluation value map, linearity is maintained only at the position of true parallax, and the result of disturbance of linearity is obtained at other parallax positions.

  Here, with respect to the reference point position (xb1, yb1) in FIG. 8A, that is, the position of xb = xb1 in the evaluation value map, the true corresponding point position to be obtained is DCcorrect (xb1), the evaluation value minimum When the position is DM (xb1) and the calculated position of the straight line is Dline (xb1), the true corresponding point can be expressed by Equation 2 below. The calculated straight line position Dline (xb1) represents the value of the D axis where xb = xb1 of the line c in FIG. 8D.

  The processing expressed by Equation 2 is executed at each reference point position. In this process, if the position of the line c in FIG. 8D is set to the true parallax of xb = xb1, the error in estimating the line c is superimposed, so the error in estimating the line c. This process is necessary to avoid being affected by

  In the above description, the number of minimum values in the minimum value distribution Hm (xbn) of another reference point (xbn, ybn) located in the peripheral region of the reference point (xb1, yb1) and the reference point (xb1, yb1) The case where the number of local minimum values in the local minimum value distribution Hm (xb1) is the same is described, but the case where the number of local minimum values is different will be described below. In such a case, a case where a corner of an object including a continuous similar pattern such as a fence is a distance measurement target is assumed.

  FIG. 9 is a schematic diagram showing the relationship of the local minimum distribution when the number of local minimum values in Hm (xbn) is different from the number of local minimum values in Hm (xb1). (A) shows a reference image, (b) shows a reference image, (c) shows the relationship of the local minimum distribution of each reference point (xbn, ybn), and (d) shows the reference point (xbn, ybn). The evaluation value map calculated by integrating the local minimum value distribution is shown.

  In FIG. 9A, the reference point (xb1, yb1) is located at the rightmost end of the fence shown in the reference image. Therefore, when the search area is limited to a certain width, as shown in FIG. 9B, there are search areas including three fences, while there are search areas including two fences. There is a case.

  In such a case, as shown in FIG. 9C, the number of local minimum values included in the local minimum value distribution Hm (xbn) between the reference point (xb1, yb1) and another reference point located in the peripheral region is Will be different. Therefore, as shown in FIG. 9 (d), a portion where a line segment is missing occurs in the straight line a.

  The method for extracting the corresponding points from the evaluation value map shown in FIG. 9D may be the same as the method described with reference to FIG.

  Further, the method for extracting the corresponding points from the evaluation value map shown in FIG. 9D is different from the content described with reference to FIG. It is also possible to perform information processing such as removing from corresponding point candidates. Thereby, it is possible to reduce a load required for the parallax calculation apparatus 100 to extract a line with the strongest linearity from the evaluation value map.

  In FIG. 9D, it has been described that the line a in which a part of the line segment is missing is not a straight line, but the corresponding point candidate is similarly applied even if the line in which a part of the line segment is missing is a straight line. It is also possible to perform information processing that is excluded from the above.

  In the above description, the case where the continuous direction of the fence is perpendicular to the optical axis of the lens has been described, but a correct parallax value can be calculated even when it is not perpendicular.

  FIG. 10 is a schematic diagram illustrating an example of a fence object reflected in a reference image and an obtained evaluation value map. (A) shows an evaluation value map, (b) shows a reference image rotated 90 degrees counterclockwise, and (c) shows a minimum value distribution obtained by extracting a part of the evaluation value map. Even if the continuous direction of the fence is not perpendicular to the optical axis of the lens, the evaluation value map is generated as a diagonal line that is not parallel to the Xb axis on the evaluation value map. It is possible to extract a line indicating parallax.

  Therefore, as shown in FIG. 10, it is possible to calculate the correct distance even when the continuous direction of the fence is not perpendicular to the optical axis of the lens.

  The parallax data output unit 103 calculates a difference d between the Xr coordinate point in the reference image of the corresponding point calculated by the corresponding point determination unit 108 and the Xb coordinate point in the reference image of the predetermined reference point, and the predetermined reference point And the result is output as parallax data.

  As described above, according to such a configuration, in the reference image, by superimposing information on another reference point existing in the peripheral area of the reference point, even in the case of an object including a continuous similar pattern, Correct parallax can be calculated.

  Here, the case where the optical axis of the stereo camera and the object are perpendicular to each other has been described. However, the present invention can be applied even when the object is not perpendicular and does not depend on the position of an object including a continuous similar pattern. Specifically, the present invention can be applied not only to a fence or guard rail positioned in front of the stereo camera but also to a fence or guard rail positioned obliquely.

  Although the Hough transform is used here, other straight line extraction methods such as Canny edge extraction may be used, and the present invention is not limited thereto.

  Here, in order to calculate an arbitrary line continuous in the direction of the Xb axis, an arbitrary line may be calculated using dynamic programming. In addition, calculating an arbitrary continuous line having a minimum evaluation value using dynamic programming is an example, and an arbitrary continuous line may be derived using a method other than dynamic programming. If an arbitrary line is calculated using dynamic programming, the correct parallax is calculated even when an object with a similar pattern continues not only in a straight line but also along a road curve. be able to.

  Here, the minimum value of the evaluation value distribution shown in FIG. 6C is extracted to generate the minimum value distribution shown in FIG. 6D, and the evaluation value map shown in FIG. 8D is calculated. However, instead of the local minimum value distribution, the evaluation value map itself may be calculated using the evaluation value distribution itself, which is the difference value.

  In this case, the evaluation value map shown in FIG. 8 (d) is different in that the evaluation value map is not a binary value of 0 and −1 but a multivalue indicating the degree of difference. The same is true in that small points calculate straight lines that are continuous on a straight line. In the evaluation value map when the evaluation value distribution itself, which is the value of the dissimilarity, is used, connecting the valleys with low evaluation values results in the same result as in FIG.

(Embodiment 2)
In the first embodiment, the peripheral region of the reference point (xb1, yb1) is provided in the horizontal direction on the Xb axis, and the evaluation value map is calculated based on the minimum value distribution related to each reference point existing in the peripheral region. As shown in FIG. 11, the minimum value distribution of one or more reference points positioned in the vertical direction on the Yb axis of the reference point (xb1, yb1) is superimposed and then present in the peripheral region. The evaluation value map may be calculated based on the minimum value distribution related to each reference point.

  FIG. 11 is a schematic diagram showing superposition of evaluation value distributions. (A) is a figure which shows the position of the some reference point in a reference | standard image, (b) is a figure which shows the search area | region with respect to each reference | standard point in a reference image, (c) is evaluation value distribution with respect to each reference | standard point. (D) is a figure which shows the result of having superimposed evaluation value distribution.

  As shown in FIG. 11D, when the evaluation value map generated by superimposing the minimum values is used, the accuracy of the straight line is improved because the image becomes more robust against the noise of the image.

  A superposition of the components of the minimum value distribution Hm (Xb) for each peripheral reference point is used as the evaluation value distribution.

  Here, when R is the number of components of the evaluation value distribution, the minimum value distribution Hm1 (Xb) at the reference point and the minimum value distribution HmR (Xb) at the reference point in the vertical direction in the reference image are superimposed. Formula 3 shows the calculation formula for the minimum value distribution H (Xb).

  As described above, in the reference image, the minimum value distribution of another reference point existing in the vertical direction of the reference point (xb1, yb1) is superimposed, and then the minimum value distribution related to each reference point existing in the peripheral region is overlapped. By calculating the evaluation value map based on this, there is an effect that the accuracy of corresponding point extraction is improved.

(Embodiment 3)
In the first embodiment, the peripheral region of the reference point (xb1, yb1) is provided in an arbitrary range in the horizontal direction on the Xb axis, and the evaluation value map is based on the minimum value distribution relating to each reference point existing in the peripheral region. However, the horizontal width of the peripheral area of the reference point (xb1, yb1) may be set to a predetermined width.

  In the present embodiment, the width in the horizontal direction of the peripheral region is determined as a peripheral region up to another reference point having an evaluation value distribution pattern similar to the evaluation value distribution pattern at the reference point (xb1, yb1).

  When the evaluation value map calculation unit 107 determines the lateral width of the peripheral region, the evaluation value map calculation unit 107 determines the minimum value distribution related to each reference point in the lateral direction of the reference point (xb1, yb1) and the reference point (xb1, yb1). The similarity of the local minimum value distribution is determined. Similarity is determined by the difference sum for each element of the local minimum distribution, and is determined to be similar when the difference sum is smaller than a predetermined threshold.

  Then, the evaluation value map calculation unit 107 determines a region including the minimum value distribution determined to be similar as a peripheral region, and calculates an evaluation value map.

  By setting the peripheral area of the reference point to the entire distance measuring object having the same distance (parallax) including the reference point, the accuracy of the corresponding point extraction is increased. This is because the accuracy of estimating the correct parallax value is increased by including the same parallax reference point as the peripheral region, and the accuracy of estimating the correct parallax value is decreased when the reference point having a different parallax value is included.

  Similarly, the present invention can be applied to the case where the peripheral area described in the second embodiment is enlarged in the vertical direction of the reference points (xb1, yb1), and the accuracy of corresponding point extraction can be improved.

  The parallax calculation apparatus according to the present invention has a function of calculating a correct distance even in a repetitive pattern in which it is theoretically difficult to calculate a distance in stereo matching, and is useful for pre-crash safety, parking assistance in a parking lot, and the like.

DESCRIPTION OF SYMBOLS 100 Disparity calculation apparatus 101 Disparity calculation part 102 Stereo image data acquisition part 103 Disparity data output part 104 Stereo matching part 105 Corresponding point candidate number determination part 106 Minimal value distribution calculation part 107 Evaluation value map calculation part 108 Corresponding point determination part 1201 Parallax calculation Unit 1202 Stereo image data acquisition unit 1203 Stereo matching unit 1204 Corresponding point candidate plural existence determination unit 1205 Evaluation value minimum corresponding point calculation 1206 Control mode data acquisition unit 1207 Control mode corresponding point calculation unit 1208 Parallax data output unit

Claims (2)

An image data acquisition unit for acquiring a reference image and a reference image;
A stereo matching unit that calculates a difference in image luminance between a reference point included in the reference image and a plurality of search points included in the reference image corresponding to the reference point;
Corresponding points for detecting, as a corresponding point candidate, a search point having a minimum difference in image brightness with respect to the coordinates of the plurality of search points from the plurality of search points, and determining whether or not there are a plurality of the corresponding point candidates. A candidate number determination unit;
The number n of first corresponding point candidates (n is an integer equal to or greater than 2) corresponding to the first reference point determined to have a plurality of corresponding point candidates , and exists in a peripheral region including the first reference point. When the number m of second corresponding point candidates corresponding to the second reference point is different from m (m is an integer smaller than n), among the first corresponding point candidates, the disparity value corresponding to the reference point corresponds to the first exclude corresponding point candidate second corresponding point candidate does not exist, among the first corresponding point candidates other than the corresponding point candidates the excluded, the disparity value and the second reference point with respect to the first reference point A corresponding point determination unit that determines a corresponding point candidate having a minimum variation value between the parallax value and the corresponding point as a corresponding point;
A parallax data output unit that outputs parallax data calculated based on the coordinate point of the corresponding point corresponding to the first reference point and the coordinate point of the first reference point;
A parallax calculation device comprising: The corresponding point candidate number determination unit detects, as the corresponding point candidate, a search point having a difference in image luminance that is equal to or less than a predetermined threshold among the detected search points.
The parallax calculation apparatus according to claim 1.

Images