JP3265171B2 - Stereo image correction condition detection device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a device for detecting a correction condition of a stereo image required for aligning an epipolar line in a stereo image, and more particularly to a correction device for detecting a correction condition using a detection rate of a phase difference known point. Related to the device.

[0002]

2. Description of the Related Art Conventionally, a twin-lens stereo image method has been used as a general-purpose method for obtaining distance information of a three-dimensional image. In the twin-lens stereo image method, the left and right images obtained by the left and right cameras are scanned, and corresponding pixels in the images, that is, corresponding points are detected. Distance information can be obtained from the phase difference between corresponding points. In order to reduce erroneous correspondences and improve the detection accuracy of corresponding points, it is important to match epipolar lines of the left and right images.

As a method of aligning epipolar lines, for example, a method of photographing a test pattern with left and right cameras and using an image recognition technique (document: Information Processing vol.31)
No.5 May 1990 pp. 650-659).

[0004]

In a conventional method using an image recognition technique, epipolar lines of right and left images are matched by recognizing a test pattern having a predetermined shape. Therefore, if the epipolar lines of the left and right images are shifted for some reason while the corresponding points are being detected, the shift can be corrected only when the same pattern as the test pattern exists in the left and right images. That is, when there is no pattern identical to the test pattern, the pattern cannot be corrected.

Therefore, it has been desired to realize an apparatus which can detect a correction condition necessary for matching epipolar lines of left and right images even if the same pattern as a test pattern does not exist.

More preferably, even if the same pattern as the test pattern does not exist, it is possible to detect a correction condition necessary for matching the epipolar lines of the left and right images and realize an apparatus capable of quickly detecting the correction condition. Was desired.

The correction condition detected here is used as, for example, an image conversion condition used when one of the left and right images is converted to match the epipolar line of the left and right images, or the right and left correction is performed to match the epipolar lines of the left and right images. It is used as the optical axis adjustment condition used when mechanically adjusting the optical axes of the lenses of the left and right cameras capturing the image.

[0008]

According to a first aspect of the present invention, there is provided a correction condition detecting apparatus for detecting a correction condition for matching epipolar lines of left and right images. , An image conversion unit, a corresponding point search unit, an evaluation value calculation unit, and a maximum value detection unit.

A condition setting unit sequentially sets a plurality of types of candidate conditions for detecting a correction condition, and an image conversion unit converts one of the left and right images according to the candidate condition. Examples of image conversion performed by the image conversion unit include translation, rotation, and magnification conversion of an image, and candidate conditions are used for image conversion performed by the image conversion unit. The image magnification conversion indicates that the image is enlarged or reduced.

[0010] The candidate conditions relating to the parallel translation of the image include x
An arbitrary suitable number of conditions Δx ₁ , Δx ₂ ,..., In which the amount of translation in the axial direction (horizontal direction) is determined stepwise, and the amount of translation in the y-axis direction (height direction) is stepwise. , A plurality of conditions Δy ₁ , Δy ₂ ,... The amount of parallel movement Δx ₁ , Δx ₂ ,
The one having the largest evaluation value, described later, is the translation amount in the x-axis direction suitable for adjusting the epipolar line, and the translation amounts Δy ₁ , Δy ₂ , in the y-axis direction.
.., The evaluation value of which will be described later is the translation amount in the y-axis direction suitable for adjusting the epipolar line.

The candidate conditions relating to the rotation of the image include an arbitrary suitable number of conditions Δα ₁ °, Δα ₂ °... In which the rotation angle around the x-axis is determined stepwise, and the rotation angle around the y-axis. Any number of conditions Δβ ₁ °, Δβ determined in stages
₂ °,..., Or any suitable number of conditions Δγ ₁ °, Δγ ₂ °,. Rotation angle around the x-axis Δα ₁ °, Δα
Of the ₂ °, the one having the largest evaluation value described later is the rotation angle about the x-axis suitable for adjusting the epipolar line, and the rotation angles Δβ ₁ °, Δβ ₂ ° about the y-axis.
Among them, the one having the largest evaluation value described later is the rotation angle around the y-axis suitable for adjusting the epipolar line, and the rotation value Δγ ₁ °, Δγ ₂ ° about the z-axis is the evaluation value described later. Is the rotation angle about the z-axis suitable for matching the epipolar line.

The candidate conditions relating to the magnification conversion of the image include an arbitrary suitable number of conditions K ₁ , in which the magnification is determined stepwise,
K ₂ ,... And any suitable number of conditions k ₁ , k ₂ ,.
Among the enlargement ratios K ₁ , K ₂ ,... And the reduction ratios k ₁ , k ₂ ,..., The one having the largest evaluation value, which will be described later, is the conversion magnification suitable for matching the epipolar line.

In order to accurately match the epipolar lines of the left and right images, the amount of parallel movement in the x-axis direction, the amount of parallel movement in the y-axis direction, the rotation angle around the x-axis, the rotation angle around the y-axis, z For all types of candidate conditions of the rotation angle around the axis and the conversion magnification, it is preferable to detect the candidate condition having the maximum evaluation value for each type of the candidate condition. Only for some types of candidate conditions of the amount of translation in the direction, the amount of translation in the y-axis direction, the rotation angle around the x-axis, the rotation angle around the y-axis, the rotation angle around the z-axis, and the conversion magnification, The candidate condition that maximizes the evaluation value for each type of candidate condition may be detected. still,
Parallel movement of the image in the z-axis direction (depth direction) is equivalent to image magnification conversion.

The condition setting section may set the candidate condition and the image conversion section may perform the image conversion only for the image conversion in which the candidate condition having the maximum evaluation value is to be detected.

The corresponding point search unit sets a plurality of types of search conditions by changing the scanning direction, the gradation expression of the pixel to be scanned, or the color of the pixel to be scanned, and sets the left and right images after image conversion. Is a left and right search image.

The corresponding point search unit scans the left and right search images according to the search conditions, compares the pixel characteristics of the left search image and the pixel characteristics of the right search image on the corresponding horizontal scanning lines, and determines the difference between the pixel characteristics. Are detected as corresponding candidate points, and corresponding candidate points that become a common set in a set of corresponding candidate points detected for each search condition are detected as corresponding points. After that, the corresponding point searching unit detects a known point and an unknown point of the phase difference in the left and right search images using the corresponding point detection result.

When a plurality of types of search conditions are set with different scanning directions, corresponding candidate points are detected as follows.

That is, pixels which are present on the corresponding horizontal scanning lines in the left and right search images and which are pixels of one of the left and right search images are considered as points of interest, and pixels of the other image of the left and right search images are defined as control points. I do. A condition a that the scanning direction of the point of interest is the right direction (direction from left to right on the horizontal scanning line) and the scanning direction of the control point is the right direction, the scanning direction of the focus point is the right direction, and the scanning direction of the control point is the left direction (The direction from right to left on the horizontal scanning line)
Select from among four conditions: condition c that sets the target point scanning direction to the left direction and sets the control point scanning direction to the right direction, and condition d that sets the target point scanning direction to the left direction and sets the control point scanning direction to the left direction. At least two types of conditions are set as search conditions, and corresponding candidate points are detected.

In any of the conditions a to d, a control point on the horizontal scan is sequentially scanned in the control point scanning direction with respect to the i-th attention point counted sequentially in the direction of interest scanning on the horizontal scanning line. Go on. When scanning of the control point is completed without detecting a control point having a pixel feature difference equal to or smaller than the threshold value for the i-th point of interest, scanning of the control point is performed on the (i + 1) -th point of interest. Further, when a control point at which the difference of the pixel feature with respect to the i-th point of interest is equal to or smaller than the threshold value is detected,
The second point of interest and the control point are detected as corresponding candidate points, and then the control point is scanned for the (i + 1) th point of interest.

As described above, a plurality of types of search conditions for scanning the left and right search images with different target point scanning directions or target point scanning directions are set, and a common set of corresponding candidate points obtained for each search condition is set. Erroneous correspondence can be reduced by detecting the correspondence candidate point which becomes as a candidate point.

When a plurality of types of search conditions are set by changing the gradation expression of pixels to be scanned, corresponding candidate points are detected as follows.

That is, the corresponding point search section converts the gradation of the left and right search images to prepare a plurality of types of left and right search images having different gradation expressions. Then, corresponding candidate points are detected for each of the left and right search images having different gradation expressions, and corresponding candidate points that are a common set in a set of corresponding candidate points obtained for each gradation expression are detected as corresponding points. For example, 256 cameras with left and right cameras
When a grayscale black and white image is obtained as a left and right image and the left and right images are obtained as left and right search images, first, no grayscale conversion is performed.
The left and right search images of the gray scale are scanned to detect the corresponding candidate points, and the left and right search images of the 256 gray scales are converted to obtain a left and right search image of 32 gray scales. Scanning is performed to detect corresponding candidate points. Then, a common set of corresponding candidate points detected from the left and right search images of 256 and 32 tones is detected as corresponding points. Alternatively, when a black-and-white image of 256 tones is obtained as a left and right image by the left and right cameras and the left and right images are obtained as left and right search images, the left and right search image of 256 tones is subjected to gradation conversion so that the gradation expression is different. A plurality of types of left and right search images of 16 tones are generated. In this case, the number of gradation steps is the same 16 gradations, and a left and right search image of 16 gradations in which the first and last gradation widths, here the first and 16th gradation widths, are different is generated. .
Then, the left and right search images of 16 gradations with different gradation expressions are respectively scanned to detect corresponding candidate points, and a common set of corresponding candidate points detected for each of the 16 left and right search images is detected as corresponding points. I do.

As described above, a plurality of types of search conditions are set such that a plurality of types of images having different levels of gradation are used as left and right search images to detect corresponding candidate points, or the number of levels of gradation is the same. As a plurality of types of search conditions for detecting corresponding candidate points as left and right search images using a plurality of types of images having different first and last gradation widths, a set of corresponding candidate points obtained for each search condition is set. By detecting corresponding candidate points that become a common set as candidate points, erroneous correspondence can be reduced.

When a plurality of types of search conditions are set by changing the colors of pixels to be scanned, corresponding candidate points are detected as follows.

That is, when a color image is obtained as a left and right image by the left and right cameras, the corresponding point searching unit determines two or three colors selected from the red, green and blue images of the left and right images which are color images. Let the image be a left and right search image. Then, corresponding candidate points are detected for each color image in the left and right search images, and corresponding candidate points that become a common set in a set of corresponding candidate points obtained for each color image are detected as corresponding points.
For example, a red image, a green image, and a blue image of the left and right images, which are color images, are set as left and right search images, and the left and right search images are scanned for each color to detect corresponding candidate points. Then, in the set of corresponding candidate points obtained for the red image, the set of corresponding candidate points obtained for the green image, and the set of corresponding candidate points obtained for the blue image, corresponding candidate points that are a common set are detected as corresponding points.

When the left and right cameras obtain color images as left and right images as described above, a plurality of types of color images selected from the red, green and blue images of the left and right images are respectively used as left and right search images. By setting search conditions and detecting corresponding candidate points that become a common set in a set of corresponding candidate points obtained for each search condition,
False responses can be reduced.

The corresponding point search section detects a known point and an unknown point of the phase difference in the left and right search images using the corresponding point detection results obtained by scanning the left and right search images.

The evaluation value calculation section obtains the evaluation value of the candidate condition for each type of candidate condition using the detection rate of the phase difference found point in the left and right search images, and the maximum value detection section obtains the maximum evaluation value for each type of candidate condition. The candidate condition for which the value has been obtained is detected as a correction condition.

As described above, the amount of parallel movement in the x-axis direction, the amount of parallel movement in the y-axis direction, the rotation angle around the x-axis, y
For all types of candidate conditions of the rotation angle around the axis, the rotation angle around the z-axis, and the conversion magnification, the evaluation value of the candidate condition is obtained for each type of the candidate condition, and the evaluation value becomes maximum for each type of the candidate condition. Candidate conditions may be detected, and if they are within a range in which there is no practical problem, the amount of translation in the x-axis direction, the amount of translation in the y-axis direction, the rotation angle around the x-axis, and the rotation angle around the y-axis , An evaluation value of the candidate condition is obtained for each type of the candidate condition only for some types of the rotation angle around the z-axis and the conversion magnification, and the candidate condition having the maximum evaluation value is detected. good.

According to a fifth aspect of the present invention, there is provided a correction condition detecting apparatus for detecting a correction condition for matching epipolar lines of left and right images, wherein a condition setting unit, an image conversion unit, a gradation conversion unit, a level cutout unit, a corresponding point A search unit, an evaluation value calculation unit, and a maximum value detection unit are provided.

The condition setting unit sequentially sets a plurality of types of candidate conditions for detecting the correction condition, and the image conversion unit converts one of the left and right images of m gradations according to the candidate condition. The image conversion performed by the image conversion unit includes translation of an image,
Rotation and magnification conversion can be mentioned, and candidate conditions are used for image conversion performed by the image conversion unit. The magnification conversion of an image indicates that the image is enlarged or reduced. The image conversion unit converts the left and right images of m gradations obtained by the left and right cameras.

The candidate conditions relating to the parallel movement of the image include x
An arbitrary preferred number of conditions Δx ₁ , Δx ₂ ,..., In which the amount of translation in the axial direction is determined stepwise, and an arbitrary preferred number of conditions, in which the amount of translation in the y-axis direction is determined stepwise Condition Δy
₁ , Δy ₂ ,... Can be used. Among the parallel movement amounts Δx ₁ , Δx ₂ ,... in the x-axis direction, the _one having the largest evaluation value described later is the parallel movement amount in the x-axis direction suitable for adjusting the epipolar line, and is in the y-axis direction. Among the parallel movement amounts Δy ₁ , Δy ₂ ,... Having the largest evaluation value, which will be described later, are the parallel movement amounts in the y-axis direction suitable for matching the epipolar lines.

The candidate conditions relating to the rotation of the image include an arbitrary suitable number of conditions Δα ₁ °, Δα ₂ °... In which the rotation angle around the x-axis is determined stepwise, and the rotation angle around the y-axis. Any number of conditions Δβ ₁ °, Δβ determined in stages
₂ °,..., Or any suitable number of conditions Δγ ₁ °, Δγ ₂ °,. Rotation angle around the x-axis Δα ₁ °, Δα
Of the ₂ °, the one having the largest evaluation value described later is the rotation angle about the x-axis suitable for adjusting the epipolar line, and the rotation angles Δβ ₁ °, Δβ ₂ ° about the y-axis.
Among them, the one having the largest evaluation value described later is the rotation angle around the y-axis suitable for adjusting the epipolar line, and the rotation value Δγ ₁ °, Δγ ₂ ° about the z-axis is the evaluation value described later. Is the rotation angle about the z-axis suitable for matching the epipolar line.

The candidate conditions relating to the image magnification conversion include an arbitrary suitable number of conditions K ₁ , in which the magnification is determined stepwise,
K ₂ ,... And any suitable number of conditions k ₁ , k ₂ ,. Among the enlargement ratios K ₁ , K ₂ ,... And the reduction ratios k ₁ , k ₂ ,..., The one having the largest evaluation value, which will be described later, is the conversion magnification suitable for matching the epipolar line.

In order to accurately match the epipolar lines of the left and right images, the amount of parallel movement in the x-axis direction, the amount of parallel movement in the y-axis direction, the rotation angle around the x-axis, the rotation angle around the y-axis, z For all types of candidate conditions of the rotation angle around the axis and the conversion magnification, it is preferable to detect the candidate condition having the maximum evaluation value for each type of the candidate condition. Only for some types of candidate conditions, such as the amount of translation in the direction, the amount of translation in the z-axis, the rotation angle around the x-axis, the rotation angle around the y-axis, the rotation angle around the z-axis, and the conversion magnification, The candidate condition that maximizes the evaluation value for each type of candidate condition may be detected. still,
Parallel movement of the image in the z-axis direction (depth direction) is equivalent to image magnification conversion.

The condition setting section may set the candidate conditions and the image conversion section may perform the image conversion only for the image conversion in which the candidate condition with the maximum evaluation value is to be detected.

The gradation conversion unit converts the left and right images after image conversion into left and right images of n gradations (m> n), and the level cutout unit extracts one or more types of attention from the left and right images after gradation conversion. The left and right images of the grayscale level are cut out as left and right attention level images.

The gradation conversion unit converts the left and right images of m gradations (for example, m = 256) obtained by the left and right cameras into n gradations (for example, n
= 32). Then, the level cutout unit focuses on one or more types of grayscale levels in the left and right images of n grayscales after the grayscale conversion, and cuts out the left and right images of the focused grayscale levels as left and right focused level images.

The left and right images of m gradations obtained by the left and right cameras are converted into left and right images of n gradations where m> n, and only the left and right images of the gradation level focused on from the left and right images of n gradations Are cut out as left and right attention level images. As will be described later, the detection speed of the corresponding candidate points can be improved by detecting the corresponding candidate points using the left and right attention level images as the left and right search images.

The corresponding point searching section sets a plurality of types of search conditions by changing the scanning direction, the gradation expression of the pixel to be scanned, or the color of the pixel to be scanned, Let it be a search image.

The corresponding point search unit scans the left and right search images according to the search conditions, compares the pixel characteristics of the left search image and the pixel characteristics of the right search image on the corresponding horizontal scanning lines, and determines the difference between the pixel characteristics. Are detected as corresponding candidate points, and corresponding candidate points that become a common set in a set of corresponding candidate points detected for each search condition are detected as corresponding points. After that, the corresponding point searching unit detects a known point and an unknown point of the phase difference in the left and right search images using the corresponding point detection result.

When a plurality of types of search conditions are set with different scanning directions, corresponding candidate points are detected as follows.

That is, the pixels existing on the corresponding horizontal scanning lines in the left and right search images, and the pixels of one of the left and right search images are regarded as the point of interest, and the pixels of the other image of the left and right search image are regarded as the control point. I do. A condition a that the scanning direction of the point of interest is the right direction (direction from left to right on the horizontal scanning line) and the scanning direction of the control point is the right direction, the scanning direction of the focus point is the right direction, and the scanning direction of the control point is the left direction (The direction from right to left on the horizontal scanning line)
Select from among four conditions: condition c that sets the target point scanning direction to the left direction and sets the control point scanning direction to the right direction, and condition d that sets the target point scanning direction to the left direction and sets the control point scanning direction to the left direction. At least two types of conditions are set as search conditions, and corresponding candidate points are detected.

In any of the conditions a to d, a reference point on the horizontal scan is sequentially scanned in the control point scanning direction with respect to the i-th point of interest counted sequentially in the direction of interest scanning on the horizontal scanning line. Go on. When scanning of the control point is completed without detecting a control point having a pixel feature difference equal to or smaller than the threshold value for the i-th point of interest, scanning of the control point is performed on the (i + 1) -th point of interest. Further, when a control point at which the difference of the pixel feature with respect to the i-th point of interest is equal to or smaller than the threshold value is detected,
The second point of interest and the control point are detected as corresponding candidate points, and then the control point is scanned for the (i + 1) th point of interest.

As described above, a plurality of types of search conditions for scanning the left and right search images with different target point scanning directions or target point scanning directions are set, and a common set of corresponding candidate points obtained for each search condition is set. Erroneous correspondence can be reduced by detecting the correspondence candidate point which becomes as a candidate point.

When a plurality of types of search conditions are set by changing the gradation expression of pixels to be scanned, corresponding candidate points are detected as follows.

That is, the corresponding point search section converts the gradation of the left and right search images to prepare a plurality of types of left and right search images having different gradation expressions. Then, corresponding candidate points are detected for each of the left and right search images having different gradation expressions, and corresponding candidate points that are a common set in a set of corresponding candidate points obtained for each gradation expression are detected as corresponding points. For example, when a 32-tone black-and-white image is obtained as a left-right focused level image and the 32-graded left-right focused level image is obtained as a left-right search image, first, a 32-gradation left-right search image that is not subjected to tone conversion is scanned. Corresponding candidate points are detected, and the left and right search images of 32 tones are subjected to gradation conversion to obtain a left and right search image of 16 tones. The left and right search images of 16 tones are scanned to detect corresponding candidate points. And these three
A common set of corresponding candidate points detected from the left and right search images of 2 and 16 tones is detected as corresponding points. Alternatively, a black-and-white image of 32 gradations is obtained as the left and right level-of-interest level image.
When the left and right target level images of the gradation are set as the left and right search images,
The left and right search image of 32 tones is subjected to gradation conversion to generate a plurality of types of left and right search images of 16 tones with different gradation expressions. In this case, the number of gradation steps is the same 16 gradations, and a left and right search image of 16 gradations in which the first and last gradation widths, here the first and 16th gradation widths, are different is generated. . Then, by scanning each of the left and right search images of 16 gradations with different gradation expressions, corresponding candidate points are detected.
A common set of corresponding candidate points detected for each of the left and right search images of 6 tones is detected as a corresponding point.

As described above, a plurality of types of search conditions are set such that a plurality of types of images having different levels of gradation are used as left and right search images to detect corresponding candidate points, or the number of levels of gradation is the same. As a plurality of types of search conditions for detecting corresponding candidate points as left and right search images using a plurality of types of images having different first and last gradation widths, a set of corresponding candidate points obtained for each search condition is set. By detecting corresponding candidate points that become a common set as candidate points, erroneous correspondence can be reduced.

When a plurality of types of search conditions are set by changing the color of a pixel to be scanned, corresponding candidate points are detected as follows.

That is, when a color image is obtained as a left and right image by the left and right cameras, the corresponding point searching unit determines two or three types of colors selected from the red, green and blue images of the left and right images which are color images. Let the image be a left and right search image. Then, corresponding candidate points are detected for each color image in the left and right search images, and corresponding candidate points that become a common set in a set of corresponding candidate points obtained for each color image are detected as corresponding points.
For example, a red image, a green image, and a blue image of the left and right images, which are color images, are set as left and right search images, and the left and right search images are scanned for each color to detect corresponding candidate points. Then, in the set of corresponding candidate points obtained for the red image, the set of corresponding candidate points obtained for the green image, and the set of corresponding candidate points obtained for the blue image, corresponding candidate points that are a common set are detected as corresponding points.

As described above, when the left and right cameras obtain color images as left and right images, a plurality of types of color images selected from the red, green and blue images of the left and right images are respectively used as left and right search images. By setting search conditions and detecting corresponding candidate points that become a common set in a set of corresponding candidate points obtained for each search condition,
False responses can be reduced.

The corresponding point search section detects a known point and an unknown point of the phase difference in the left and right search images using the corresponding point detection results obtained by scanning the left and right search images.

The evaluation value calculation unit obtains the evaluation value of the candidate condition for each type of candidate condition using the detection rate of the phase difference found point in the left and right search images, and the maximum value detection unit calculates the maximum value for each type of candidate condition. The candidate condition for which the evaluation value has been obtained is detected as a correction condition.

As described above, the translation amount in the x-axis direction, the translation amount in the y-axis direction, the rotation angle around the x-axis,
For all types of candidate conditions of the rotation angle around the axis, the rotation angle around the z-axis, and the conversion magnification, the evaluation value of the candidate condition is obtained for each type of the candidate condition, and the evaluation value becomes maximum for each type of the candidate condition. Candidate conditions may be detected, and if they are within a range in which there is no practical problem, the amount of translation in the x-axis direction, the amount of translation in the y-axis direction, the rotation angle around the x-axis, the rotation angle around the y-axis , An evaluation value of the candidate condition is obtained for each type of the candidate condition only for some types of the rotation angle around the z-axis and the conversion magnification, and the candidate condition having the maximum evaluation value is detected. good.

According to a ninth aspect of the present invention, there is provided a correction condition detecting apparatus for detecting a correction condition for matching epipolar lines of left and right images, comprising: a condition setting unit, an image conversion unit, a corresponding point search unit, an evaluation value calculation unit, It is characterized by comprising a value detecting section.

The condition setting unit sequentially sets candidate conditions for detecting the height direction correction condition, and the image conversion unit converts one of the left and right images according to the candidate condition. The image conversion unit sets the x-axis direction as the horizontal direction, the y-axis direction as the height direction, and the z-axis direction as the depth direction, and performs image conversion in the y-axis direction (height direction) as image conversion for detecting a height direction correction condition. The translation is performed, and the candidate condition is used for the image conversion.

The candidate conditions include an arbitrary suitable number of conditions Δy ₁ and Δy in which the amount of parallel movement in the y-axis direction is determined stepwise.
₂ , ... can be used. Among the parallel movement amounts Δy ₁ , Δy ₂ ,... in the z-axis direction, the _one having the largest evaluation value described later is suitable for adjusting the epipolar line.
The amount of translation in the axial direction, that is, the height direction correction condition.

The corresponding point search unit sets a plurality of types of search conditions by changing the scanning direction, the gradation expression of the pixel to be scanned, or the color of the pixel to be scanned, and sets the left and right images after image conversion. Is a left and right search image.

The corresponding point search unit scans the left and right search images according to the search conditions, compares the pixel characteristics of the left search image and the pixel characteristics of the right search image on the corresponding horizontal scanning lines, and determines the difference between the pixel characteristics. Pixels of the left and right search images that are equal to or less than the threshold are detected as corresponding candidate points, and corresponding candidate points that are a common set in a set of corresponding candidate points detected for each search condition are detected as corresponding points. After that, the corresponding point searching unit detects a known point and an unknown point of the phase difference in the left and right search images using the corresponding point detection result.

When a plurality of types of search conditions are set with different scanning directions, corresponding candidate points are detected as follows.

That is, the pixels existing on the corresponding horizontal scanning lines in the left and right search images, and the pixels of one of the left and right search images are regarded as the point of interest, and the pixels of the other image of the left and right search image are regarded as the control point. I do. A condition a that the scanning direction of the point of interest is the right direction (direction from left to right on the horizontal scanning line) and the scanning direction of the control point is the right direction, the scanning direction of the focus point is the right direction, and the scanning direction of the control point is the left direction (The direction from right to left on the horizontal scanning line)
Select from among four conditions: condition c that sets the target point scanning direction to the left direction and sets the control point scanning direction to the right direction, and condition d that sets the target point scanning direction to the left direction and sets the control point scanning direction to the left direction. At least two types of conditions are set as search conditions, and corresponding candidate points are detected.

In any of the conditions a to d, a control point on the horizontal scan is sequentially scanned in the control point scanning direction with respect to the i-th attention point counted sequentially in the direction of interest scanning on the horizontal scanning line. Go on. When scanning of the control point is completed without detecting a control point having a pixel feature difference equal to or smaller than the threshold value for the i-th point of interest, scanning of the control point is performed on the (i + 1) -th point of interest. Further, when a control point at which the difference of the pixel feature with respect to the i-th point of interest is equal to or smaller than the threshold value is detected,
The second point of interest and the control point are detected as corresponding candidate points, and then the control point is scanned for the (i + 1) th point of interest.

As described above, a plurality of types of search conditions for scanning the left and right search images with different target point scanning directions or target point scanning directions are set, and a common set of corresponding candidate points obtained for each search condition is set. Erroneous correspondence can be reduced by detecting the correspondence candidate point which becomes as a candidate point.

When a plurality of types of search conditions are set by changing the gradation expression of pixels to be scanned, corresponding candidate points are detected as follows.

That is, the corresponding point search unit converts the gradation of the left and right search image to prepare a plurality of types of left and right search images having different gradation expressions. Then, corresponding candidate points are detected for each of the left and right search images having different gradation expressions, and corresponding candidate points that are a common set in a set of corresponding candidate points obtained for each gradation expression are detected as corresponding points. For example, 256 cameras with left and right cameras
When a grayscale black and white image is obtained as a left and right image and the left and right images are obtained as left and right search images, first, no grayscale conversion is performed.
The left and right search images of the gray scale are scanned to detect the corresponding candidate points, and the left and right search images of the 256 gray scales are converted to obtain a left and right search image of 32 gray scales. Scanning is performed to detect corresponding candidate points. Then, a common set of corresponding candidate points detected from the left and right search images of 256 and 32 tones is detected as corresponding points. Alternatively, when a black-and-white image of 256 tones is obtained as a left and right image by the left and right cameras and the left and right images are obtained as left and right search images, the left and right search image of 256 tones is subjected to gradation conversion so that the gradation expression is different. A plurality of types of left and right search images of 16 tones are generated. In this case, the number of gradation steps is the same 16 gradations, and a left and right search image of 16 gradations in which the first and last gradation widths, here the first and 16th gradation widths, are different is generated. .
Then, the left and right search images of 16 gradations with different gradation expressions are respectively scanned to detect corresponding candidate points, and a common set of corresponding candidate points detected for each of the 16 left and right search images is detected as corresponding points. I do.

As described above, a plurality of types of search conditions for detecting corresponding candidate points by using a plurality of types of images having different levels of gradation as left and right search images are set, or the number of levels of gradation is the same. As a plurality of types of search conditions for detecting corresponding candidate points as left and right search images using a plurality of types of images having different first and last gradation widths, a set of corresponding candidate points obtained for each search condition is set. By detecting corresponding candidate points that become a common set as candidate points, erroneous correspondence can be reduced.

When a plurality of types of search conditions are set by changing the color of a pixel to be scanned, corresponding candidate points are detected as follows.

That is, when a color image is obtained as a left and right image by the left and right cameras, the corresponding point searching unit determines two or three colors selected from the red, green and blue images of the left and right images which are color images. Let the image be a left and right search image. Then, corresponding candidate points are detected for each color image in the left and right search images, and corresponding candidate points that become a common set in a set of corresponding candidate points obtained for each color image are detected as corresponding points.
For example, a red image, a green image, and a blue image of the left and right images, which are color images, are set as left and right search images, and the left and right search images are scanned for each color to detect corresponding candidate points. Then, in the set of corresponding candidate points obtained for the red image, the set of corresponding candidate points obtained for the green image, and the set of corresponding candidate points obtained for the blue image, corresponding candidate points that are a common set are detected as corresponding points.

As described above, when the left and right cameras obtain color images as left and right images, a plurality of types of color images selected from the red, green and blue images of the left and right images are respectively used as left and right search images. By setting search conditions and detecting corresponding candidate points that become a common set in a set of corresponding candidate points obtained for each search condition,
False responses can be reduced.

The corresponding point search section uses the corresponding point detection results obtained by scanning the left and right search images to determine the known and unknown points of the short-distance phase difference where the amount of displacement in the height direction is large in the left and right search images. Is detected.

The evaluation value calculation unit calculates the detection rate of the phase difference known point for the short-distance phase difference in which the amount of displacement in the height direction is large in the left and right search images, and uses the detection rate to evaluate the evaluation value of the height direction candidate condition. Is obtained, and the maximum value detection unit detects the height direction candidate condition having the maximum evaluation value as the height direction correction condition.

According to a thirteenth aspect of the present invention, there is provided a correction condition detecting apparatus for detecting a correction condition for matching epipolar lines of left and right images, wherein a condition setting unit, an image conversion unit, a corresponding point search unit, an evaluation value calculation unit, and a maximum value It is characterized by comprising a detection unit.

The condition setting unit sequentially sets candidate conditions for detecting the tilt angle correction condition, and the image conversion unit converts one of the left and right images according to the candidate condition. The image conversion unit sets the x-axis direction as the horizontal direction, the y-axis direction as the height direction, and the z-axis direction as the depth direction, and performs rotation about the z-axis as image conversion for detecting a tilt angle correction condition. Is used as the candidate condition.

The candidate conditions include an arbitrary preferred number of conditions Δγ ₁ , Δγ ₂ , and a rotation angle about the z-axis which are determined stepwise.
... can be used. Rotation angle Δγ around z axis
_{Of the 1} , Δγ ₂ ,..., The evaluation value described later is the maximum, which is the rotation angle around the z-axis, that is, the tilt angle correction condition suitable for adjusting the epipolar line.

The corresponding point searching section sets a plurality of types of search conditions by changing the scanning direction, the gradation expression of the pixel to be scanned, or the color of the pixel to be scanned, and sets the left and right images after image conversion. Is a left and right search image.

The corresponding point search unit scans the left and right search images according to the search conditions, compares the pixel characteristics of the left search image and the pixel characteristics of the right search image on the corresponding horizontal scanning lines, and determines the difference between the pixel characteristics. Pixels of the left and right search images that are equal to or less than the threshold are detected as corresponding candidate points, and corresponding candidate points that are a common set in a set of corresponding candidate points detected for each search condition are detected as corresponding points. After that, the corresponding point searching unit detects a known point and an unknown point of the phase difference in the left and right search images using the corresponding point detection result.

When a plurality of types of search conditions are set with different scanning directions, corresponding candidate points are detected as follows.

That is, the pixels existing on the corresponding horizontal scanning lines in the left and right search images, and the pixels of one of the left and right search images are taken as the point of interest, and the pixels of the other image of the left and right search image are taken as the control point. I do. A condition a that the scanning direction of the point of interest is the right direction (direction from left to right on the horizontal scanning line) and the scanning direction of the control point is the right direction, the scanning direction of the focus point is the right direction, and the scanning direction of the control point is the left direction (The direction from right to left on the horizontal scanning line)
Select from among four conditions: condition c that sets the target point scanning direction to the left direction and sets the control point scanning direction to the right direction, and condition d that sets the target point scanning direction to the left direction and sets the control point scanning direction to the left direction. At least two types of conditions are set as search conditions, and corresponding candidate points are detected.

In any of the conditions a to d, the control point on the horizontal scan is sequentially scanned in the control point scanning direction with respect to the i-th attention point counted sequentially in the direction of interest scanning on the horizontal scanning line. Go on. When scanning of the control point is completed without detecting a control point having a pixel feature difference equal to or smaller than the threshold value for the i-th point of interest, scanning of the control point is performed on the (i + 1) -th point of interest. Further, when a control point at which the difference of the pixel feature with respect to the i-th point of interest is equal to or smaller than the threshold value is detected,
The second point of interest and the control point are detected as corresponding candidate points, and then the control point is scanned for the (i + 1) th point of interest.

As described above, a plurality of types of search conditions for scanning the left and right search images with different target point scan directions or target point scan directions are set, and a common set of corresponding candidate points obtained for each search condition is set. Erroneous correspondence can be reduced by detecting the correspondence candidate point which becomes as a candidate point.

When a plurality of types of search conditions are set by changing the gradation expression of pixels to be scanned, corresponding candidate points are detected as follows.

That is, the corresponding point search section converts the gradation of the left and right search images to prepare a plurality of types of left and right search images having different gradation expressions. Then, corresponding candidate points are detected for each of the left and right search images having different gradation expressions, and corresponding candidate points that are a common set in a set of corresponding candidate points obtained for each gradation expression are detected as corresponding points. For example, 256 cameras with left and right cameras
When a grayscale black and white image is obtained as a left and right image and the left and right images are obtained as left and right search images, first, no grayscale conversion is performed.
The left and right search images of the gray scale are scanned to detect the corresponding candidate points, and the left and right search images of the 256 gray scales are converted to obtain a left and right search image of 32 gray scales. Scanning is performed to detect corresponding candidate points. Then, a common set of corresponding candidate points detected from the left and right search images of 256 and 32 tones is detected as corresponding points. Alternatively, when a black-and-white image of 256 tones is obtained as a left and right image by the left and right cameras and the left and right images are obtained as left and right search images, the left and right search image of 256 tones is subjected to gradation conversion so that the gradation expression is different. A plurality of types of left and right search images of 16 tones are generated. In this case, the number of gradation steps is the same 16 gradations, and a left and right search image of 16 gradations in which the first and last gradation widths, here the first and 16th gradation widths, are different is generated. .
Then, the left and right search images of 16 gradations with different gradation expressions are respectively scanned to detect corresponding candidate points, and a common set of corresponding candidate points detected for each of the 16 left and right search images is detected as corresponding points. I do.

A plurality of types of search conditions for detecting corresponding candidate points using a plurality of types of images having different numbers of gray levels as left and right search images are set, or the number of gray levels is the same. As a plurality of types of search conditions for detecting corresponding candidate points as left and right search images using a plurality of types of images having different first and last gradation widths, a set of corresponding candidate points obtained for each search condition is set. By detecting corresponding candidate points that become a common set as candidate points, erroneous correspondence can be reduced.

When a plurality of types of search conditions are set by changing the color of a pixel to be scanned, corresponding candidate points are detected as follows.

That is, when a color image is obtained as a left and right image by the left and right cameras, the corresponding point searching section determines two or three colors selected from the red, green and blue images of the left and right images which are color images. Let the image be a left and right search image. Then, corresponding candidate points are detected for each color image in the left and right search images, and corresponding candidate points that become a common set in a set of corresponding candidate points obtained for each color image are detected as corresponding points.
For example, a red image, a green image, and a blue image of the left and right images, which are color images, are set as left and right search images, and the left and right search images are scanned for each color to detect corresponding candidate points. Then, in the set of corresponding candidate points obtained for the red image, the set of corresponding candidate points obtained for the green image, and the set of corresponding candidate points obtained for the blue image, corresponding candidate points that are a common set are detected as corresponding points.

As described above, when the left and right cameras obtain color images as left and right images, a plurality of types of color images selected from the red, green and blue images of the left and right images are respectively used as left and right search images. By setting search conditions and detecting corresponding candidate points that become a common set in a set of corresponding candidate points obtained for each search condition,
False responses can be reduced.

The corresponding point searching section uses the corresponding point detection results obtained by scanning the left and right search images to find out the clear and unknown points of the long-distance phase difference where the tilt angle shift amount becomes large in the left and right search images. To detect.

The evaluation value calculation unit obtains a detection rate of a phase difference known point for a long-distance phase difference in which the tilt angle shift amount is large in the left and right search images, and obtains an evaluation value of a tilt angle candidate condition using the detection rate. The maximum value detection unit detects a tilt angle candidate condition that has obtained the maximum evaluation value as a tilt angle correction condition.

[0089]

BEST MODE FOR CARRYING OUT THE INVENTION

<First Embodiment of the Invention of Claim 1> FIG. 1 is a functional block diagram for explaining a first embodiment of the invention of claim 1. In the figure, reference numerals 10 and 12 denote a left camera and a right camera used for the two-lens stereo image method. The left image captured by the left camera 10 is stored in the first left memory 14 and the right camera 12
Is stored in the first right memory 16.

Reference numeral 18 denotes a correction condition detecting device. The correction condition detecting device 18 of this embodiment detects a correction condition used for epipolar line alignment of the left image and the right image taken by the left camera 10 and the right camera 12. . Further, in addition to this, the correction condition detecting device 18 of this embodiment performs image conversion of one of the left image and the right image captured by the left camera 10 and the right camera 12 based on the detected correction condition, and outputs the left image after the image conversion. The image and the right image are stored in the second left memory 20 and the second right memory 22. In the left image and the right image after the image conversion based on the correction condition, the epipolar lines match or almost match.

Reference numeral 24 denotes a corresponding point searching device. The corresponding point searching device 24 scans the left image and the right image after image conversion based on the correction condition, detects corresponding pixels (corresponding points) in the left image and the right image, and further detects the phase difference between the corresponding points. , And stores it in the distance information memory 26. The phase difference of the corresponding point is used as distance information of the three-dimensional image.

The correction condition detecting device 1 of this embodiment
8 includes a condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32, an evaluation value calculation unit 34 and a maximum value detection unit 36, and a third left memory 38 and a third right memory 40.

The condition setting unit 28 sequentially sets a plurality of types of candidate conditions for detecting a correction condition,
Converts one of the left image and the right image obtained by the left camera 10 and the right camera 12 according to the candidate condition.

Here, the condition setting section 28 stores a plurality of types of candidate conditions for detecting a correction condition and a plurality of types of correction conditions for matching epipolar lines. The correction condition to be stored is the correction condition detected by the maximum value detection unit 36.

When the correction condition is detected, the condition setting unit 2
8 outputs the candidate condition to the image conversion unit 30, and the image conversion unit 30 performs image conversion of one of the left image and the right image obtained by the left camera 10 and the right camera 12 according to the candidate condition. The image conversion unit 30 stores the left image and the right image after the image conversion based on the candidate condition in the third left memory 38 and the third right memory 40. One of the left image and the right image stored in the memories 38 and 40 is an image that has been subjected to image conversion, and the other is an image that has not been subjected to image conversion.

When the epipolar lines are matched,
The condition setting unit 28 outputs the correction condition to the image conversion unit 30, and the image conversion unit 30 outputs the correction condition to the left camera 10 and the right camera 12
One of the left image and the right image obtained by the above is image-converted according to the correction condition. The image conversion unit 30 stores the left image and the right image after the image conversion based on the correction condition in the second left memory 20.
And the second right memory 22. One of the left image and the right image stored in the memories 20 and 22 is an image converted image, and the other is an image not converted.

The condition setting section 28 stores candidate conditions relating to image parallel movement, rotation and magnification conversion. Therefore, the correction conditions detected by the maximum value detection unit 36 are correction conditions relating to the parallel movement, rotation, and magnification conversion of the image.
The image conversion performed by 0 according to the candidate condition and the correction condition is translation, rotation and magnification conversion of the image. Known methods can be used for the translation, rotation, and magnification conversion.

[0098] The candidate condition relating to translation of the image, the x-axis direction plurality example five moving amount Δx ₁ ~Δx ₅ that defines the parallel movement amount in the (horizontal) stepwise, y-axis direction two candidates as candidate condition storing a plurality example five variable Δy ₁ ~Δy ₅ which defines stepwise translation of the (vertical or height direction), thus involving translation of the image Conditions Δx _{1 to} Δx ₅ and Δy _{1 to} Δy ₅
Is stored.

Of the parallel movement amounts Δx _{1 to} Δx _{5 in} the x-axis direction, the one having the largest evaluation value, which will be described later, is the correction condition relating to the parallel movement in the x-axis direction, and the parallel movement in the y-axis direction. Among the quantities Δy _{1 to} Δy _5, the one having the largest evaluation value described later is the correction condition relating to the parallel movement in the y-axis direction.

As candidate conditions relating to image rotation, x
A plurality example five angles Δα ₁ ° ~Δα ₅ ° which defines the rotation angle around the axis in stages, y axis rotation angle stepwise-determined plurality example five angles [Delta] [beta] ₁ ° ~ Δβ ₅ °
And a plurality of, for example, five angles Δγ ₁ ° to Δγ _{5 in} which the rotation angle about the z-axis is determined in a stepwise manner. Therefore, three types of candidate conditions Δα ₁ ° are set as candidate conditions related to image rotation.
~ Δα ₅ °, Δβ ₁ ° ~ Δβ ₅ ° and Δγ ₁ ° ~ Δγ ₅
Remember °.

Among the rotation angles Δα ₁ ° to Δα ₅ ° around the x-axis, the one having the largest evaluation value, which will be described later, is the correction condition relating to the rotation around the x-axis, and the rotation angle Δβ ₁ around the y-axis.
Of the ° to Δβ ₅ °, the one having the largest evaluation value described later is the correction condition relating to rotation about the y-axis, and
Among the rotation angles Δγ ₁ ° to Δγ ₅ around the axis, the one having the largest evaluation value described later is the correction condition related to rotation about the z-axis.

The candidate conditions for the image magnification conversion include a plurality of image magnifications, for example, five conversion magnifications K _{1 to} K _5, and a plurality of image reduction ratios, stepwise determined. For example, five conversion magnifications k _{1 to} k ₅ are stored, and thus one type of conversion magnification is stored as a candidate condition relating to the magnification conversion of an image.

Among the conversion magnifications K _{1 to} K ₅ and k _{1 to} k ₅ , the one having the largest evaluation value described later is the correction condition relating to the magnification conversion. Note that the image magnification conversion is equivalent to a parallel movement in the z-axis direction (depth direction).

The corresponding point search section 32 sets a plurality of types of search conditions by changing the scanning direction, the gradation expression of the pixel to be scanned, or the color of the pixel to be scanned, and sets the left and right after image conversion. Let the image be a left and right search image. Here, a plurality of types of search conditions are set by changing the gradation expression of pixels to be scanned, and the left and right images after image conversion are set as left and right search images over the entire surface.

The corresponding point search unit 32 scans the left and right search images according to the search conditions, compares the pixel characteristics of the left search image and the pixel characteristics of the right search image on the corresponding horizontal scanning lines, and determines the difference between the pixel characteristics. The pixels of the left and right search images for which is less than or equal to the threshold are detected as corresponding candidate points. Further, the corresponding point search unit 32 detects, as corresponding points, corresponding candidate points that become a common set in the set of corresponding candidate points detected for each search condition. Here, the pixel feature is a pixel density.

For example, the converted left and right images stored in the third left memory 38 and the third right memory 40 are 256
In the case of a monochrome black and white image, the left and right images of 256 tones are subjected to grayscale conversion over the entire surface to obtain a left and right image of 32 tones. Set search conditions. Further, the left and right images of 256 gradations are subjected to gradation conversion over the entire surface to obtain a left and right image of 16 gradations, and a second search condition for setting the left and right images of 16 gradations as a left and right search image over the entire surface is set. Then, the left-right search image of 32 tones according to the first search condition is scanned over the entire surface, and pixels of the left-right search image in which the difference in pixel characteristics is equal to or less than the threshold on the corresponding horizontal scanning line are determined by the first search condition. It is detected as a correspondence candidate point. Further, the left and right search images of 16 gradations according to the second search condition are scanned over the entire surface, and the pixels of the left and right search image in which the difference of the pixel characteristics is equal to or less than the threshold value on the corresponding horizontal scanning line are determined by the second search condition. It is detected as a correspondence candidate point. Thereafter, a corresponding candidate point that is a common set between the set of corresponding candidate points detected under the first search condition and the set of corresponding candidate points detected under the second search condition is detected as a corresponding point.

As the pixel characteristics, various types other than the pixel density can be used. For example, if pixels are grouped according to density and a group of pixels of one group is represented as a pattern, the corresponding point searching unit 32 determines the width and length of the pattern to which the pixel belongs for each pixel of the left and right search images. The width and length may be detected and used as the pixel feature. In this case, the left search image is scanned when detecting the pixel characteristics of the left search image, and the right search image is scanned when detecting the pixel characteristics of the right search image. Then, the width of a pattern to which the pixel belongs is detected on a horizontal scanning line passing through the pixel whose pixel characteristic is to be detected. Further, the length of a pattern to which the pixel belongs is detected on a vertical scanning line passing through the pixel whose pixel characteristic is to be detected.

Next, the corresponding point search section 32 detects a known point and an unknown point of the phase difference in the left and right search images using the corresponding point detection result. Here, based on the phase difference between the corresponding points, the phase difference is estimated and applied to an unknown correspondence point (pixel not detected as the corresponding point). The corresponding unknown points to which the phase difference could not be applied are detected as the phase difference unknown points. Then, regarding the phase difference applied to the unknown point,
Test whether the phase difference is correct. Then, the unknown point of the correspondence of the phase difference determined to be correct by this test is detected as a known point of the phase difference, and the unknown point of the correspondence of the phase difference determined to be incorrect is detected as the unknown point of the phase difference.

The detection of the phase difference known point and the phase difference unknown point can be performed by various methods other than those described here. For example, without performing the fitting of the phase difference to the unknown correspondence point, the pixel serving as the corresponding point in the left and right search images may be detected as the known phase difference point and the pixel serving as the unknown correspondence point may be detected as the unknown phase difference point. .

The evaluation value calculation unit 34 obtains the evaluation value of the candidate condition for each type of the candidate condition using the detection rate of the phase difference found point in the left and right search images. Here, the detection rate of the phase difference found point in the left and right search images is obtained for each candidate condition, and the detection rate is used as the evaluation value of the candidate condition.

The maximum value detecting section 36 detects, for each type of candidate condition, a candidate condition having the highest evaluation value as a correction condition. Here, the maximum value detecting unit 36 causes the condition setting unit 28 to store the detected correction condition.

FIG. 2 is a diagram showing a flow of the operation of the first embodiment of the present invention. FIG. 7 shows a flow of operation in a case where attention is paid to one of a plurality of types of candidate conditions and correction conditions, and a correction condition is detected using the candidate conditions for the type of interest.

When the operator inputs an operation start signal to the correction condition detecting device 18 (start), first, the condition setting section 28 sets one of a plurality of types of candidate conditions as a candidate condition of a target type (S1). Thereafter, the condition setting unit 28 outputs one candidate condition (selected candidate condition) selected from the plurality of candidate conditions of the target species (S2). For example, the candidate condition of the target species is changed to a candidate condition Δ related to the parallel movement in the x-axis direction.
and x ₁ ~Δx _5, focusing species of a plurality of candidate conditions Δx ₁ ~
One candidate condition Δx ₁ selected from Δx ₅ is output.

Next, the image conversion unit 30 converts one of the left image and the right image according to the selection candidate condition input from the condition setting unit 28 (S3).

FIG. 3 is a diagram for explaining a first left and right memory for storing left and right images before image conversion (before the image conversion unit finishes image conversion). FIG. 4 is a diagram for use after image conversion (image conversion unit). FIG. 11 is a diagram provided for explanation of a third left and right memory for storing left and right images after image conversion is completed.

As shown in FIGS. 3A and 3B, the storage area 14a of the first left memory 14 and the storage area 16a of the first right memory 16 respectively have x, The y-coordinate is set, and the left image 42 before image conversion captured by the left camera 10 is stored in the first left memory 14, and the right image 44 before image conversion captured by the right camera 12 is stored in the first right memory 16. Store.

As shown in FIGS. 4A and 4B, x and x are also stored in the storage area 38a of the third left memory 38 and the storage area 40a of the third right memory 40 in pixel units. , Y coordinates are set, and the left image 42 after image conversion is set.
In the third memory 38, and the image conversion unit 30 stores the right image 44 after the image conversion in the third right memory 40.

Here, the left image 42 is converted to the right image 4 without image conversion.
Assuming that image conversion 4 is for image conversion, the image conversion unit 30 reads the left image 42 from the left camera 10 from the first left memory 14 and stores it in the third left memory 38 without image conversion. At the same time, the image conversion unit 30 outputs the right image 4 from the right camera 12.
4 is read from the first right memory 16 and stored in the third right memory 40 without image conversion. Image conversion is performed by the condition setting unit 28
This is performed according to the candidate condition input from. For example, when the candidate condition Δx ₁ is input, image conversion is performed to translate each pixel of the right image 44 in the x-axis direction by Δx ₁ , and the converted right image 44 is stored in the third right memory 40.

When a candidate condition relating to rotation or magnification conversion of an image is set as a candidate condition and the left image is to be image-converted, as shown in FIG. The image conversion is performed using the pixel at the center of the left image corresponding to the lens optical axis as the center of rotation or magnification conversion. Similarly, when performing image conversion on the right image, as shown in FIG. 3B, image conversion is performed by using the pixel at the center of the right image corresponding to the lens optical axis of the right camera 12 as the center of rotation or magnification conversion. Perform

Next, the corresponding point search unit 32 sets a plurality of types of search conditions by changing the gradation expression of the pixel to be scanned, and sets the entire left and right image after image conversion as a left and right search image. Corresponding candidate points in the left and right search images are detected for each condition (S4).

For example, the converted left image 42 and right image 44 stored in the third left memory 38 and the third right memory 40, respectively.
Is a black and white image of 256 gradations, the left image 42 and the right image 44 of 256 gradations are gradation-converted to obtain a left image 42 and a right image 44 of 32 gradations. 4
A first search condition for setting the second and right images 44 as a left search image and a right search image is set. Further, the left image 42 and the right image 44 of 256 gradations are subjected to gradation conversion to obtain a left image 42 and a right image 44 of 16 gradations. A second search condition for a right search image is set.

The corresponding point search section 32 is a horizontal scanning line L (horizontal scanning line L having the same vertical coordinate y; see FIGS. 4A and 4B) corresponding to the left and right search images. It has left and right line memories for storing the image of the line, and when detecting the corresponding candidate point under the first search condition, the horizontal scanning line L
The upper left image 42 of 256 tones is read from the third left memory 38, converted to 32 tones, and the left image 42 for one line converted to 32 tones is stored in the left line memory. At the same time, the right image 44 of 256 gradations on the horizontal scanning line L is read from the third right memory 40, converted to 32 gradations, and the right image 44 for one line converted to 32 gradations is stored in the right line memory. .

Then, the corresponding point search section 32 scans the left image 42 and the right image 44 for one line according to the first search condition. Here, one pixel of the left image 42 and the right image 44 for one line for which gradation conversion has been performed is set as a point of interest, and the other pixel is set as a control point. For the i-th point of interest counted sequentially,
The control point on the horizontal scanning line L is sequentially scanned in a predetermined main scanning direction, for example, rightward. When the scanning of the control point is completed without detecting a control point having a pixel characteristic, for example, a difference in pixel density equal to or smaller than a threshold value, with respect to the i-th point of interest, the (i + 1) th point
The control point is scanned with respect to the third point of interest. When a control point at which the difference of the pixel feature with respect to the i-th point of interest is smaller than or equal to the threshold is detected, the i-th point of interest and the control point are detected as corresponding candidate points, and then the (i + 1) -th point of interest is detected. Is scanned for the point of interest.

As described above, the left and right search images on the corresponding horizontal scanning line L are scanned, and when the difference between the pixel characteristics becomes equal to or smaller than the threshold value, the next point of interest is scanned to detect a corresponding candidate point. When the detection of the corresponding candidate points on the horizontal scanning line L is completed over the entire left and right search image, the detection of the corresponding candidate points based on the first search condition is ended.

When the corresponding candidate point is detected under the second search condition, the left and right search images on the corresponding horizontal scanning line L are scanned in the same manner as in the first search condition, and the difference between the pixel characteristics is detected. When the value becomes equal to or smaller than the threshold value, the next point of interest is scanned to detect a corresponding candidate point. When the detection of the corresponding candidate points on the horizontal scanning line L is completed over the entire left and right search image, the detection of the corresponding candidate points based on the second search condition is ended.

After the detection of the corresponding candidate points on the horizontal scanning line L over the entire left and right search images for all types of search conditions, the corresponding point search unit 32 next calculates the corresponding candidate points obtained for each search condition. Corresponding candidate points that become a common set in the set are detected as corresponding points (S5).

As described above, the left and right search images on the corresponding horizontal scanning line L are scanned, and when the difference between the pixel characteristics becomes equal to or less than the threshold value, the next point of interest is scanned to detect a corresponding candidate point. The corresponding candidate points that become a common set in the set of corresponding candidate points obtained for each search condition are a group of corresponding candidate points that are likely to be stochastic.

Next, the corresponding point search section 32 detects the phase difference found point and the phase difference unknown point using the corresponding point detection result, and evaluates the phase difference found point or the number of detected phase difference unknown points. Output to the value calculation unit 34 (S6).

Here, the detected phase difference points and the unknown phase difference points relate to the entire left and right images that are the current left and right search images.

Here, based on the phase difference between corresponding points, the phase difference is estimated and applied to an unknown correspondence point (pixel not detected as a corresponding point). The corresponding unknown points to which the phase difference could not be applied are detected as the phase difference unknown points. Thereafter, it is checked whether the phase difference applied to the correspondence unknown point is correct. Then, the unknown point of the correspondence of the phase difference determined to be correct by this test is determined to be a known point of the phase difference, and the unknown point of correspondence of the phase difference determined to be incorrect is detected as the unknown point of the phase difference.

Next, the evaluation value calculation unit 32 obtains the detection rate of the phase difference found point using the number of detected phase difference found points, and calculates (S
The detection rate of the phase difference found point is stored in the maximum value detection unit 36 as the evaluation value of the selection candidate condition output by the condition setting unit 28 in 2) (S7).

The detection rate of the phase difference found point obtained here relates to the entire left and right image which is the current left and right search image. In this case, the detection rate of the phase difference found point is, for example, (detection rate of the phase difference found point) = (the number of detected phase difference found points in the left search image) / (one left image captured by the left camera) (The number of all pixels), (detection rate of phase difference found point) = (number of detected phase difference point in left search image) / (sum of phase difference found point and phase difference unknown point in left search image), Alternatively, (detection rate of phase difference found point) = (detected number of phase difference found point in right search image) / (1 captured by right camera)
(Total number of pixels of the right image), (Detection rate of phase difference found point) = (Number of detected phase difference point in right search image) / (Detection of phase difference found point and phase difference unknown point in right search image) (Sum of the numbers).

Next, the condition setting section 28 determines whether or not all the candidate conditions of the target species have been selected (S8).

When the condition setting unit 28 determines that all the candidate conditions of the target species have not been selected, the condition setting unit 28 selects the next candidate condition of the target species (S9), and then selects the next candidate condition. The candidate condition is output (S2).

For example, only the selection of Δx ₁ among the candidate conditions Δx _{1 to} Δx ₅ of the target species has been completed, and Δx _{2 to} Δx ₂ have been selected.
If not finished the selection for x _5, one of the candidates conditions selected from among the remaining candidate condition Δx ₂ ~Δx ₅ Δ
Select x _2, and outputs the candidate condition [Delta] x ₂ selected.

When the condition setting unit 28 determines that all the candidate conditions of the target species have been selected, the maximum value detecting unit 36 determines the maximum evaluation value among the evaluation values obtained for each candidate condition of the target species. The candidate condition that has been detected and obtained the maximum evaluation value is detected as a correction condition. Then, the maximum value detecting unit 28 stores the correction condition of the target type in the condition setting unit 28 (S10). Thereafter, the correction condition detection device 18 ends the processing for detecting the correction condition of the target type (end).

For example, all the candidate conditions Δx ₁ -Δ
When you are finished selecting the x _5, focusing species of candidate conditions Δx ₁ ~
Detecting the maximum evaluation value from among the evaluation values obtained for each candidate condition of [Delta] x _5, candidate condition for example [Delta] x to give a maximum evaluation value
₄ is detected as the correction condition of the target species.

The correction condition detecting device 18 calculates the parallel movement amount in the x-axis direction, the parallel movement amount in the y-axis direction, the rotation angle around the x-axis, the rotation angle around the y-axis, the rotation angle around the z-axis, and the conversion angle. The processing of S1 to S10 described above is performed for each type of candidate condition of the magnification, and a correction condition is detected for each type of candidate condition.

FIG. 5 is a diagram for explaining the detection of the correction condition. In the figure, the vertical axis represents the evaluation value of the candidate condition, and the horizontal axis represents the candidate condition relating to the amount of translation in the x-axis direction.

The evaluation value obtained by performing image conversion individually for each of the candidate conditions Δx _{1 to} Δx _{5 of the} kind of interest is obtained using the detection rate of the phase difference known point. Therefore, the evaluation value increases if the epipolar line shift amount is small in the left and right images after image conversion and the epipolar line shift amount is small, and the evaluation value is large if the epipolar line shift amount in the left and right images after image conversion is large. Become smaller. The detection of the correction condition utilizes such a phenomenon. Therefore, by detecting, as a correction condition, a candidate condition having a maximum evaluation value among a plurality of candidate conditions of a target type, it is suitable for epipolar line alignment. Corrected condition can be detected.

In a plurality of candidate conditions prepared for each type of candidate condition, if the width of change of the candidate condition is too large, it becomes difficult to detect a correction condition suitable for epipolar line alignment. In consideration of this point, in a plurality of candidate conditions prepared for each type of the candidate condition, a plurality of candidate conditions having different values in a stepwise manner with a width of change suitable for epipolar line alignment, prepare.

For example, the candidate conditions Δx _{1 to} Δx ₅ relating to the parallel movement in the x-axis direction are changed stepwise at intervals of 1/3 pixel, and the candidate conditions Δα _{1 to} Δα ₅ relating to the rotation around the x-axis.
The stepwise varied at 1 ° intervals, varying the candidate condition ΔK ₁ ~ΔK ₅ and Δk ₁ ~Δk ₅ related to magnification conversion in steps of 1% intervals.

Next, detection of the phase difference at the corresponding point and fitting of the phase difference at the unknown point will be described. The corresponding point search unit 32
Since corresponding candidate points that become a common set in the set of corresponding candidate points obtained for each search condition are detected as corresponding points, FIG.
4A and 4B, the corresponding points a and b detected in the left and right search images are pixels on the horizontal scanning line L.

The phase difference is | (x coordinate of corresponding point a in left search image) − (x coordinate of corresponding point in right search image)
When the corresponding point a is located at the coordinates (5, 4) and the corresponding point b is located at the coordinates (8, 4), as shown in FIGS. Will detect | 5-8 | = 3 as the phase difference between the corresponding points a and b, respectively.

FIG. 6 is a diagram for explaining the fitting of the phase difference at the unknown point. The corresponding point search unit 32 estimates and fits the phase difference to an unknown correspondence point (pixel not detected as a corresponding point) in the left and right search images using the detected phase difference of the corresponding point.

Various methods can be used as a method of estimating and fitting the phase difference. In this case, if there is an unknown correspondence point that is in contact with or near the corresponding point, the position of the unknown correspondence point is determined. The phase difference is estimated to be equal to the phase difference of the corresponding point, and the phase difference of the corresponding point is applied to the corresponding unknown point. Specifically, the coordinates of the corresponding point in the left search image or the right search image are (x ₁ , y ₁ ), and the coordinates of the unknown point in the left search image or the right search image are (x ₂ ,
y ₂ ), the coordinates of the unknown point (x ₂ , y ₂ )
Satisfies x ₂ = x ₁ and y ₁ −α ≦ y ₂ ≦ y ₁ + α, or x ₁ −β ≦ x ₂ ≦ x ₁ + β and y ₂
= A is satisfied y _1, the phase difference between corresponding points of the coordinates (x _1, y _1), the phase difference of the corresponding point of uncertainty in the coordinates (x _2, y _2). α and β are natural numbers, for example, α = β = 2. When there are a plurality of corresponding points around the unknown point, a plurality of phase differences are applied to the unknown point.

For example, in FIG. 6, when α = β = 2, a pixel at coordinates (x ₂ , y ₂ ) satisfying x ₂ = x ₁ and y ₁ −α ≦ y ₂ ≦ y ₁ + α, x ₁ -β ≦ x ₂ ≦ x
Pixels having coordinates (x ₂ , y ₂ ) satisfying ₁ + β and y ₂ = y ₁ are indicated by hatching. If the hatched pixels are unknown correspondence points, the phase difference between the corresponding points at the coordinates (x ₁ , y ₁ ) is applied to the unknown correspondence points. In the figure, the storage area 38a for one pixel in the third left memory 38 is represented by a rectangle of one square.

Next, a description will be given of a test of the phase difference applied to the unknown correspondence point. Various methods can be used to test the phase difference. Here, the test is performed as described below.

When testing the phase difference applied to the unknown correspondence point in the left search image, the phase difference found point in the left search image is used as the test point, and the test is performed from the position corresponding to the test point in the right search image. A pixel in the right search image at a position shifted by the phase difference applied to the point is set as a reference point. When the difference between the density of the test point in the left search image and the density of the reference point in the right search image is equal to or smaller than the threshold, it is determined that the phase difference applied to the test point is correct. When the difference between the density of the test point in the left search image and the density of the reference point in the right search image exceeds a threshold value, it is determined that the phase difference applied to the test point is not correct.

When testing the phase difference applied to the unknown correspondence point in the right search image, the phase difference found point in the right search image is used as the test point, and the test is performed from the position corresponding to the test point in the left search image. A pixel in the left search image at a position shifted by the phase difference applied to the point is set as a reference point. When the difference between the density of the test point in the right search image and the density of the reference point in the left search image is equal to or smaller than the threshold value, it is determined that the phase difference applied to the test point is correct. When the difference between the density of the test point in the right search image and the density of the reference point in the left search image exceeds a threshold, it is determined that the phase difference of the test point is not correct.

Next, with reference to FIG. 7 and FIG. 8, a description will be given of a process of testing a phase difference applied to an unknown correspondence point in the left search image. FIG. 7 is a diagram showing the flow of the phase difference test process performed by the corresponding point search unit. FIG. 8 is a diagram for explaining a positional relationship between a test point and a reference point when performing a phase difference test on the left search image.

When the fitting of the phase difference to the unknown point of the left search image is completed, the corresponding point search unit 32 starts a test of the fitted phase difference (start). It is determined whether or not scanning has been completed for all of the unknown correspondence points to which is applied (S1).

(1-A) If it is determined in S1 that all the unknown points to which the phase difference has been applied have not been scanned, the corresponding point search unit 32 applies the phase difference in the left search image. The uncertain points that have been determined are used as test points, and the density d of the test points is read from the third left memory 38 (S2).

Next, the corresponding point search section 32 sets a pixel in the right search image at a position shifted from the position corresponding to the test point by the phase difference L applied to the test point in the right search image as a reference point, The density D of the point is read from the third right memory 40 (S3).

The coordinates (x ₀ ,
y ₀ ) is the coordinates (x ₀ −L, y ₀ ) of the reference point in the right search image. For example, as shown in FIG. 8A, when the coordinates (6, 6) of the test point in the left search image and the phase difference L of the test point are L = 2, FIG.
As shown in (B), the coordinates of the reference point in the right search image are (4, 6).

Next, the corresponding point searching section 32 determines whether or not the difference | d−D | between the read densities d and D is equal to or smaller than the threshold value THL (S4).

When it is determined in step S4 that the density difference | d−D | exceeds the threshold value THL, the corresponding point searching unit 32 determines that the phase difference applied to the test point is not correct, and It is detected as an unknown point (S5), and then the process of S1 is performed.

If it is determined in step S4 that the density difference | d−D | is equal to or smaller than the threshold value THL, the corresponding point searching unit 32 determines that the phase difference applied to the test point is correct, and It is detected as a known point (S6), and then the process of S1 is performed. In the above-described phase difference test, the density difference | d−D
The combination of the test point and the reference point where | is equal to or less than the threshold value THL is a combination of pixels having a high probability of becoming a corresponding point, and therefore, it can be determined that a stochastically reliable phase difference is correct.

When a plurality of phase differences are applied to the test point, a reference point is obtained for each phase difference and the density difference |
It is determined whether d−D | is equal to or less than the threshold value THL. For example, two phase differences 2, with respect to the test point of coordinates (6, 6),
1 is applied, a reference point at coordinates (4, 6) and a reference point at coordinates (5, 6) are obtained, and the density difference | d−D | It is determined whether the difference is equal to or less than the threshold value THL.

If the density difference | d−D | exceeds the threshold value THL for all phase differences, it is determined that the phase difference applied to the test point is not correct, and the test point is detected as an unknown phase difference point. I do. If there is one or more phase differences that are less than or equal to the threshold value THL, it is determined that the phase difference applied to the test point is correct, and the test point is detected as a phase difference clear point.

Even when a plurality of phase differences are applied to the test point, a reference point is obtained for each phase difference to determine whether or not the density difference | d−D | is less than or equal to the threshold value THL. So, the test point which fit the stochastic phase difference is
It can be detected as a phase difference clear point.

(1-B) If it is determined in S1 that all the unknown points to which the phase difference has been applied have been scanned, the phase difference test is completed for the unknown points in the left search image (end).

The threshold value TH used for the above-described phase difference test
L is preferably set to a value larger than a threshold value used for searching for a corresponding point in the corresponding point search unit 32. However, if the threshold value THL is too large, the stochastic probability of the phase difference determined to be correct decreases, so that a practically sufficient stochastic certainty is obtained.
Set the threshold value THL.

Next, with reference to FIGS. 7 and 9, a description will be given of a process of testing a phase difference applied to an unknown correspondence point in the right search image. FIG. 9 is a diagram for explaining a positional relationship between a test point and a reference point when a phase difference test is performed on the right search image.

When the fitting of the phase difference to the unknown point of the right search image is completed, the corresponding point search unit 32 starts the test of the fitted phase difference (start). It is determined whether or not scanning has been completed for all of the unknown correspondence points to which is applied (S1).

(2-A) If it is determined in S1 that all the unknown points to which the phase difference has been applied have not been scanned, the corresponding point search unit 32 applies the phase difference in the right search image. The corresponding unknown point is set as a test point, and the density d of the test point is read from the third right memory 40 (S2).

Next, the corresponding point searching section 32 sets a pixel in the left search image at a position shifted from the position corresponding to the test point in the left search image by the phase difference L applied to the test point as a reference point, and The density D of the point is read from the third left memory 38 (S3).

The coordinates of the test point (x ₀ ,
y ₀ ) is the coordinates (x ₀ + L, y ₀ ) of the reference point in the left search image. For example, if the pixel at the coordinates (4, 6) in FIG. 9B is a test point in the right search image and the phase difference L of the test point is L = 3, FIG.
In (A), the pixel at coordinates (7, 6) is a reference point in the left search image.

Next, the corresponding point searching section 32 determines whether or not the difference | d−D | between the read densities d and D is equal to or smaller than the threshold value THL (S4).

If it is determined in S4 that the density difference | d−D | exceeds the threshold value THL, the corresponding point searching unit 32 determines that the phase difference applied to the test point is not correct, and determines that the test point is the phase difference. It is detected as an unknown point (S5), and then the process of S1 is performed.

If it is determined in step S4 that the density difference | d−D | is equal to or smaller than the threshold value THL, the corresponding point searching unit 32 determines that the phase difference applied to the test point is correct, and determines that the test point is the phase difference. It is detected as a known point (S6), and then the process of S1 is performed.

(2-B) If it is determined in S1 that the entire surface has been scanned, the phase difference test is terminated for unknown points of the right search image (end).

When the number of detected phase difference points in the left search image is used to obtain the detection rate of the phase difference found points, a phase difference test is performed on the left search image, and after the phase difference test, the left search image is detected. The detection rate of the phase difference found point is determined using the number of detected phase difference points.

When the number of detected phase difference found points in the right search image is used to determine the detection rate of the phase difference found point, a phase difference test is performed on the right search image, and after the phase difference test is completed, the right search image is obtained. , The detection rate of the detected phase difference point is obtained using the detected number of the detected phase difference point.

When the number of detected phase difference points in the left search image or the right search image is not used for obtaining the detection rate of the phase difference determined points, the phase difference test of the left search image or the right search image is not performed. Good.

The detection rate of the phase difference found point is obtained using the detected number of phase difference found points for which the phase difference test has been completed, and the evaluation value of the candidate condition is obtained from the phase difference found point detection rate, thereby obtaining an epipolar signal. It is possible to improve the detection accuracy of the correction condition necessary for line alignment.

The left camera 1 stored in the first left memory 14
The left image from 0 and the right image from the right camera 12 stored in the first right memory 16 are rewritten at a rate of one screen every t seconds.

The correction condition may be detected from the left and right images of the memories 14 and 16 one screen at a time, or may be detected from the left and right images of the memories 14 and 16 every other screen. .

If the left and right images taken by the left and right cameras 10 and 12 are still images, the correction condition is detected every plural screens for each type of correction condition to obtain a plurality of correction conditions. An average of a plurality of correction conditions for each type of correction condition may be obtained, and the epipolar lines of the left and right images captured by the left and right cameras 10 and 12 may be matched using the averaged correction conditions. Even in the case of a still image, the noise situation changes for each screen. Therefore, by using an average of the correction conditions obtained every several screens, the accuracy of epipolar line alignment can be improved.

<Second Embodiment of the Invention of Claim 1> FIG.
FIG. 3 is a functional block diagram for explaining a second embodiment of the first aspect of the present invention. Hereinafter, points different from the first embodiment of the first aspect of the invention will be mainly described, and detailed description of the same points as those of the first embodiment of the first aspect of the invention will be omitted.

The modification condition detecting device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
An evaluation value calculation unit 34 and a maximum value detection unit 36; a third left memory 38 and a third right memory 40;
6 is provided.

The area extracting section 46 sequentially extracts left and right attention area images from a plurality of attention areas dispersed in the vertical direction on the left and right images after image conversion, and the corresponding point search section 32 separates the left and right attention area images into right and left images. As search images, corresponding candidate points, corresponding points, known and unknown points of the phase difference are detected for each of the left and right search images having the same candidate condition.

The evaluation value calculation unit calculates the detection rate of the phase difference found point for each of the left and right search images having the same candidate condition, and calculates the average value of the detection rates obtained for the left and right search images having the same candidate condition as the evaluation value of the candidate condition. And

FIG. 11 is a diagram showing a flow of the operation of the second embodiment of the present invention. FIG. 7 shows a flow of operation in a case where attention is paid to one of a plurality of types of candidate conditions and correction conditions, and a correction condition is detected using the candidate conditions for the type of interest.

When the operator inputs an operation start signal to the correction condition detecting device 18 (start), first, the condition setting unit 28 sets one of a plurality of types of candidate conditions as a candidate condition of a target type (S1). Thereafter, the condition setting unit 28 outputs one candidate condition (selected candidate condition) selected from the plurality of candidate conditions of the target species (S2).

[0186] For example, the candidate condition Δx ₁ ~Δx ₅ involved the interest species candidate condition parallel movement in the x-axis direction was selected from among the focus species plurality of candidate condition Δx ₁ ~Δx _{5 1}
The number of candidate conditions Δx ₁ are output.

Next, the image conversion unit 30 converts one of the left image and the right image according to the selection candidate condition input from the condition setting unit 28, and stores the left image and the right image after the image conversion in the third left memory 38. Stored in the third right memory 40 (S
3).

Here, the left image 42 is not converted and the right image 4 is not converted.
4A and 4B, if image conversion is performed, FIG.
As shown in FIG.
Is read from the first left memory 14 and stored in the third left memory 38 without image conversion. At the same time, as shown in FIGS. 3B and 4B, the image conversion unit 30 reads the right image 44 from the right camera 12 from the first right memory 16 and performs third image conversion without image conversion. It is stored in the memory 40. The image conversion is performed according to the candidate conditions input from the condition setting unit 28. For example, when the candidate condition Δx ₁ is input, image conversion is performed to translate each pixel of the right image 44 in the x-axis direction by Δx ₁ , and the right image 44 after the image conversion is performed.
Is stored in the third right memory 40.

Next, the region cutout section 46 cuts out the left and right region of interest images from the plurality of regions of interest dispersed in the vertical direction (y direction) on the left and right images after image conversion (S4).

FIG. 12 is a diagram for explaining a region of interest and a region of interest image. In FIG. 12A, the region of interest 48 and the region of interest 50 related to the left image 42 stored in the third left memory 38 are shown. In FIG. 12B, a region of interest 52 related to the right image 44 stored in the third right memory 40,
5 shows a region of interest image 54. In the figure, attention area images 50, 5
4 is indicated by hatching.

As shown in FIGS. 12A and 12B, the left image 42 and the right image 44 after image conversion are stored in the third left memory 38 and the third right memory 40.
A plurality of regions of interest 48 are set in the vertical direction on the left image 42 of the third left memory 38, and a plurality of regions of interest are set in the vertical direction on the right image 44 of the third right memory 40. An area 52 is set.

The number of regions of interest 48 and 50 in these left and right images 42 and 44 and the width in the vertical direction are the same. For example, the number of regions of interest 48 and 52 is three and the width in the vertical direction is horizontal. The width is one scanning line. Note that the vertical width of the regions of interest 48 and 52 may be the width of a plurality of horizontal scanning lines.

Further, the attention area 52 on the right image 44 is set at the same vertical position (y coordinate position) as the plurality of attention areas 48 set on the left image 42. In the illustrated example, the region of interest 4 is located at the position of y = 3, 6, and 9, respectively.
8, 50 are set.

Then, the area extracting section 46 outputs the left and right images 42,
One line of left and right region of interest images 50 and 54 are cut out from corresponding regions of interest 48 and 52 of the y-coordinate 44, for example, regions of interest 48 and 52 of y = 3, and stored.

Next, the corresponding point search unit 32 sets a plurality of types of search conditions by changing the gradation expression of the pixel to be scanned, sets the left and right region of interest images as the left and right search images, and Then, corresponding candidate points in the left and right search images are detected (S5).

For example, when the left and right region-of-interest images 50 and 54 stored in the region clipping section 46 are black and white images of 256 tones, the left and right region-of-interest images 50 and 54 of 256 tones are converted to 32 tones. The left and right region of interest images 50 and 54 are obtained, and the first search condition for setting the left and right region of interest images 50 and 54 of 32 tones as the left and right search images is set. This 2
The left and right region of interest images 50 and 54 of 56 gradations are subjected to gradation conversion to obtain the left and right region of interest images 50 and 54 of 16 gradations.
A second search condition is set in which the left and right focused area images 50 and 54 of the gradation are used as the left and right search images.

The corresponding point search unit 32 has left and right line memories for storing one line of the image on the horizontal scanning line L corresponding to the left and right search images, and detects corresponding candidate points under the first search condition. Read out the left focused area image 50 of 256 gradations on the horizontal scanning line L from the area cutout unit 46 and
The left region of interest image 50 for one line that has been converted to two gradations and converted to 32 gradations is stored in the left line memory. At the same time, the right focused area image 54 of 256 tones on the horizontal scanning line L is read out from the area extracting unit 46, converted to 32 tones, and converted to 32 tones. To be stored.

Then, the corresponding point searching section 32 scans the left focused area image 50 and the right focused area image 54 for one line according to the first search condition. Here, one pixel of the left region of interest image 50 and the right region of interest image 54 for one line subjected to gradation conversion is set as a point of interest, and the other pixel is set as a control point. To the i-th
A control point on the horizontal scanning line L is sequentially scanned in a predetermined main scanning direction with respect to the point of interest. When scanning of the control point is completed without detecting a control point having a pixel feature difference equal to or smaller than the threshold value for the i-th point of interest, scanning of the control point is performed on the (i + 1) -th point of interest. When a control point at which the difference of the pixel feature with respect to the i-th point of interest is smaller than or equal to the threshold is detected, the i-th point of interest and the control point are detected as corresponding candidate points, and then the (i + 1) -th point of interest is detected. Is scanned for the point of interest.

As described above, the left and right search images on the corresponding horizontal scanning line L are scanned here, and the left and right region of interest images 52 and 54 are scanned. A corresponding candidate point is detected. When the detection of the corresponding candidate points on the horizontal scanning line L is completed over the entire left and right search image, the detection of the corresponding candidate points based on the first search condition is ended.

When the corresponding candidate point is detected under the second search condition, the left and right search images on the corresponding horizontal scanning line L are scanned in the same manner as in the first search condition, and the difference between the pixel characteristics is detected. When the value becomes equal to or smaller than the threshold value, the next point of interest is scanned to detect a corresponding candidate point. When the detection of the corresponding candidate points on the horizontal scanning line L is completed over the entire left and right search image, the detection of the corresponding candidate points based on the second search condition is ended.

After the detection of the corresponding candidate points on the horizontal scanning line L over the entire left and right search images for all types of search conditions, the corresponding point search unit 32 next calculates the corresponding candidate points obtained for each search condition. Corresponding candidate points that become a common set in the set are detected as corresponding points (S6).

For example, a corresponding candidate point that is a common set between the set of corresponding candidate points detected under the first search condition and the set of corresponding candidate points detected under the second search condition is detected as a corresponding point.

Next, the corresponding point search section 32 detects the phase difference found point and the phase difference unknown point using the corresponding point detection result, and evaluates the phase difference found point or the number of detected phase difference unknown points. Output to the value calculation unit 34 (S7).

Here, the detected phase difference points and the unknown phase difference points relate to one of the regions of interest 48 and 52 from which the current left and right search images have been read.

In this case, the phase difference is estimated and applied to an unknown correspondence point (pixel not detected as the corresponding point) based on the phase difference between the corresponding points. The corresponding unknown points to which the phase difference could not be applied are detected as the phase difference unknown points. Thereafter, it is checked whether the phase difference applied to the correspondence unknown point is correct. Then, the unknown point of the correspondence of the phase difference determined to be correct by this test is determined to be a known point of the phase difference, and the unknown point of correspondence of the phase difference determined to be incorrect is detected as the unknown point of the phase difference.

Next, the evaluation value calculation unit 32 obtains the detection rate of the phase difference found point using the number of detected phase difference found points, and stores the detection rate of the phase difference found point in the maximum value detection unit 36 ( S
8).

The detection rate of the phase difference found point obtained here is determined by the one attention area 48, 5
2 In this case, the detection rate of the phase difference found point is, for example, (detection rate of the phase difference found point) = (the number of detected phase difference found points in the left search image) / (one left image captured by the left camera) (The number of all pixels), (detection rate of phase difference found point) = (number of detected phase difference point in left search image) / (sum of phase difference found point and phase difference unknown point in left search image), Alternatively, (detection rate of phase difference found point) = (detected number of phase difference found point in right search image) / (1 captured by right camera)
(Total number of pixels of the right image), (Detection rate of phase difference found point) = (Number of detected phase difference point in right search image) / (Detection of phase difference found point and phase difference unknown point in right search image) (Sum of the numbers).

Next, the area extracting section 46 extracts the right and left attention area images 50 and 54 from all the attention areas 48 and 52 with respect to the converted left and right images 42 and 44 currently stored in the third left and right memories 38 and 40. It is determined whether or not the data has been cut out (reading has been completed) (S9).

If the extraction of all the regions of interest 48 and 52 has not been completed for the left and right images 42 and 44 after the current image conversion, the region extracting section 46 selects a pair of regions of interest 48 and 52 that have not been extracted yet. (S10) Then, from the selected regions of interest 48 and 52, the next left and right region of interest images 50,
54 is cut out (S4).

For example, in the example of FIG. 12, the extraction from the regions of interest 48 and 52 with y = 3 is completed, and the extraction from the regions of interest 48 and 52 with y = 6 and 9 is not completed. As the set of target areas 48 and 52, the target areas 48 and 52 of y = 6 are selected.

When the extraction of all the regions of interest 48 and 52 is completed for the left and right images 42 and 44 after the current image conversion, the evaluation value calculation unit 34 obtains the right and left images 42 and 44 after the current image conversion. Regarding the candidate condition, each region of interest 48,
An average value of the detection rates obtained for every 50 is obtained, and the average value is stored in the maximum value detection unit 36 as an evaluation value of the candidate condition (S11).

For example, the left and right images 42 after the current image conversion,
If 44 is obtained according to the candidate condition Δx ₁ , y
= The average value of the detection rate was obtained for each region of interest 48 and 52 of the 3, 6, 9 and thus to obtain an evaluation value of the candidate condition [Delta] x _1.

Next, the condition setting unit 28 determines whether or not all the candidate conditions of the target species have been selected (S12).

If all the candidate conditions of the target species have not been selected, the condition setting unit 28 selects the next candidate condition of the target species (S13), and then outputs the selected candidate conditions (S2). ).

For example, out of the candidate conditions Δx _{1 to} Δx ₅ of the target species, only Δx ₁ has been selected, and Δx ₂ to Δx ₅ have been selected.
If not finished the selection for x _5, one of the candidates conditions selected from among the remaining candidate condition Δx ₂ ~Δx ₅ Δ
Select x _2, and outputs the candidate condition [Delta] x ₂ selected.

When all the candidate conditions of the target species have been selected, the maximum value detector 36 detects the maximum evaluation value from among the evaluation values obtained for each candidate condition of the target species, and The candidate condition for which the evaluation value has been obtained is detected as a correction condition. Then, the maximum value detecting unit 28 stores the correction condition of the target type in the condition setting unit 28 (S14). Thereafter, the correction condition detection device 18 ends the processing for detecting the correction condition of the target type (end).

For example, all candidate conditions Δx _{1 to} Δ
When you are finished selecting the x _5, focusing species of candidate conditions Δx ₁ ~
Detecting the maximum evaluation value from among the evaluation values obtained for each candidate condition of [Delta] x _5, candidate condition for example [Delta] x to give a maximum evaluation value
₄ is detected as the correction condition of the target species.

The correction condition detecting device 18 calculates the parallel movement amount in the x-axis direction, the parallel movement amount in the y-axis direction, the rotation angle around the x-axis, the rotation angle around the y-axis, the rotation angle around the z-axis, and the conversion angle. The processing of S1 to S14 described above is performed for each type of candidate condition of magnification, and a correction condition is detected for each type of candidate condition.

In this embodiment, the left and right search images are converted to the attention area images 50 and 5 which are a part of the left and right images 42 and 44.
4, the amount of scanning of the image required to detect the correction condition can be reduced, so that the detection time of the correction condition can be shortened.

<Third Embodiment of the Invention of Claim 1> Next, a third embodiment of the invention of claim 1 will be described with reference to FIG. Hereinafter, points different from the second embodiment of the first aspect of the present invention will be mainly described, and detailed description of the same points as those of the second embodiment of the first aspect of the present invention will be omitted.

The correction condition detecting device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
An evaluation value calculation unit 34 and a maximum value detection unit 36; a third left memory 38 and a third right memory 40;
6 is provided.

The evaluation value calculation unit 34 calculates the detection rate of the phase difference found point for each of the left and right search images having the same candidate condition, and sequentially determines the detection rates of the left and right search images having the same candidate condition from the top in order. The number of detection rates is selected, and the average value of the detection rates obtained by the selection is used as the evaluation value of the candidate condition.

FIG. 13 is a diagram showing a flow of the operation of the third embodiment of the present invention. FIG. 7 shows a flow of operation in a case where attention is paid to one of a plurality of types of candidate conditions and correction conditions, and a correction condition is detected using the candidate conditions for the type of interest.

When the operator inputs an operation start signal to the correction condition detecting device 18 (start), the condition setting section 28 sets one of a plurality of types of candidate conditions as a candidate condition of a target type (S1). Thereafter, the condition setting unit 28 outputs one candidate condition (selected candidate condition) selected from the plurality of candidate conditions of the target species (S2).

Next, the image conversion unit 30 converts one of the left image and the right image according to the selection candidate condition input from the condition setting unit 28, and stores the left image and the right image after the image conversion in the third left memory 38 Stored in the third right memory 40 (S
3).

Next, the area cutout section 46 cuts out the left and right attention area images from the plurality of attention areas dispersed in the vertical direction (y direction) on the left and right images after image conversion (S4).

Next, the corresponding point search section 32 sets a plurality of types of search conditions by changing the gradation expression of the pixel to be scanned, sets the left and right region of interest images as left and right search images, Then, corresponding candidate points in the left and right search images are detected (S5).

When the detection of the corresponding candidate points on the horizontal scanning line L is completed over the entire left and right search images for all types of search conditions, the corresponding point search unit 32 next calculates the corresponding candidate points obtained for each search condition. Corresponding candidate points that become a common set in the set are detected as corresponding points (S6).

Next, the corresponding point searching section 32 detects the phase difference found point and the phase difference unknown point using the corresponding point detection result, and evaluates the phase difference found point or the number of detected phase difference unknown points in addition thereto. Output to the value calculation unit 34 (S7).

Next, the evaluation value calculation unit 32 obtains the detection rate of the phase difference found point using the number of detected phase difference found points, and stores the detection rate of the phase difference found point in the maximum value detection unit 36 ( S
8).

Next, the area extracting section 46 extracts the left and right attention area images 50 and 54 from all the attention areas 48 and 52 with respect to the converted left and right images 42 and 44 currently stored in the third left and right memories 38 and 40. It is determined whether or not the data has been cut out (reading has been completed) (S9).

If the extraction of all the target areas 48 and 52 has not been completed for the left and right images 42 and 44 after the current image conversion, the area extracting section 46 selects a set of target areas 48 and 52 that have not been extracted yet. (S10) Then, from the selected regions of interest 48 and 52, the next left and right region of interest images 50,
54 is cut out (S4).

When the extraction of all the regions of interest 48 and 52 is completed with respect to the left and right images 42 and 44 after the current image conversion, the evaluation value calculation unit 34 obtains the right and left images 42 and 44 after the current image conversion. Regarding the candidate condition, each region of interest 48,
A higher detection rate is selected from among the detection rates obtained for every 50 (S
11).

Next, the evaluation value calculation unit 34 calculates the average value of the selected higher detection rates, and uses this average value as the evaluation value for the candidate conditions for obtaining the current image-converted left and right images 42 and 44. It is stored in the value detection unit 36 (S12).

Here, the left and right images 42 after the current image conversion,
Since the total number of regions of interest 48 and 52 set on 44 is the same, assuming that the total number is R, it is possible to obtain a detection rate of the total number R from the regions of interest 48 and 52 having the total number R. Becomes Then, among the detection rates of the total number R, the detection rates of the top r1 (r1 is a natural number satisfying R>r1> 1) are selected,
The average value of the selected r1 detection rates is used as the evaluation value.
The lower (R-r1) detection rates where (R-r1)> 0 are not used for calculating the evaluation value.

For example, a total number R = 30 attention areas 48,
If 52 are set and the detection rate of the total number R = 30 is obtained from these regions of interest, the evaluation value is obtained using the detection rates of the upper r1 = 10 to 20 and the lower (R-r1) = 20-
The ten detection rates are not used for calculating the evaluation value.

Next, the condition setting section 28 determines whether or not all the candidate conditions of the target species have been selected (S13).

If all the candidate conditions of the target species have not been selected, the condition setting unit 28 selects the next candidate condition of the target species (S13), and then outputs the selected candidate conditions (S2). ).

When all candidate conditions of the target species have been selected, the maximum value detecting section 36 detects the maximum evaluation value from among the evaluation values obtained for each candidate condition of the target species, and The candidate condition for which the evaluation value has been obtained is detected as a correction condition. Then, the maximum value detecting unit 28 stores the correction condition of the target type in the condition setting unit 28 (S15). Thereafter, the correction condition detection device 18 ends the processing for detecting the correction condition of the target type (end).

The correction condition detecting device 18 calculates the parallel movement amount in the x-axis direction, the parallel movement amount in the y-axis direction, the rotation angle around the x-axis, the rotation angle around the y-axis, the rotation angle around the z-axis, and the conversion. The processing of S1 to S15 described above is performed for each type of candidate condition of the magnification, and a correction condition is detected for each type of candidate condition.

In this embodiment, the left and right search images are converted into the left and right images 42 and 44 in the same manner as in the second embodiment.
Are the target area images 50 and 54 which are a part of the above. Therefore, the scanning amount of the image necessary for detecting the correction condition can be reduced, so that the detection time of the correction condition can be shortened.

Further, according to this embodiment, when an evaluation value is obtained for each type of candidate condition, the evaluation value is obtained by using the average value of the detection rates of the upper plurality, and the lower detection rate is obtained by calculating the lower detection rate. Since it is not used for calculation, detection accuracy of the correction condition can be improved.

FIG. 14 is a diagram for explaining an improvement in the correction condition detection accuracy in this embodiment. The vertical axis of the drawing on the evaluation value and the horizontal axis of the candidate condition Yes taking candidate condition relating to parallel movement amount in the x-axis direction, total In the FIG. 14 (A) determined for candidate condition [Delta] x ₄ number the R relates detection rate showed a variation range and the average value of the detection rate for all the characters, FIG. 14 (B) the total number was determined for the candidate condition [Delta] x ₄ in the R number of detection rate relates upper r1 amino detection The variation range and average value of the rate are shown.

When the left and right cameras 12, the right camera 12, and the image conversion unit 30 have different quantization noises in the left and right search images of the target area due to the quantization error of the image, the detection rate is extremely high in such a target area. descend.

Therefore, as shown in FIG. 14A, when the detection rate of the total number R obtained for the candidate condition Δx ₄ includes such an extremely small detection rate, the total number When the average value of the R detection rates is used as the evaluation value of the candidate condition Δx ₄ , the value of the evaluation value decreases due to noise, and it becomes difficult to detect the correction condition with high accuracy.

Therefore, as shown in FIG. 14B, the top r1 detection rates are selected from the total R detection rates, and
The average value of the detection rates of the top r1 items is used as the evaluation value. It is highly likely that the lower detection rate not used for calculating the evaluation value includes a detection rate whose value is extremely low due to noise, and thus the lower detection rate is used by using the higher detection rate. By determining the evaluation value without using the detection rate, the detection accuracy of the correction condition can be improved.

<Fourth Embodiment of the Invention of Claim 1>
FIG. 7 is a functional block diagram for explaining a fourth embodiment of the first aspect of the present invention. Hereinafter, points different from the first embodiment of the first aspect of the invention will be mainly described, and detailed description of the same points as the first embodiment of the first aspect of the invention will be omitted.

The modified condition detecting device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
An evaluation value calculation unit 34 and a maximum value detection unit 36; a third left memory 38 and a third right memory 40;
And

The left camera 1 stored in the first left memory 14
The left image from 0 and the right image from the right camera 12 stored in the first right memory 16 are rewritten at a rate of one screen every t seconds.

In this embodiment, the correction condition is not detected for each screen, but is detected for every several screens. The screen selection unit 56 sets the first left memory 1 every several screens.
4. The left image and the right image are read from the first right memory 16 and input to the image conversion unit 30.

The image conversion unit 30 converts the left image and the right image input from the screen selection unit 56 according to the candidate conditions, and stores them in the third left memory 38 and the third right memory 40.
As a result, the correction condition detecting device 18 detects the correction conditions for the left image and the right image read every several screens by the screen selection unit 56.

The image conversion unit 30 converts the left image and the right image input from the screen selection unit 56 according to the correction conditions, and stores them in the second left memory 20 and the second right memory 22. As a result, the corresponding point searching device 24 sets the screen selection unit 56
For each of the left and right images read out every few screens,
The detection of the corresponding point and the phase difference are detected.

Normally, the left camera 10 and the right camera 12 are installed on an anti-vibration structure so that the epipolar lines of the left image and the right image taken by the left camera 10 and the right camera 12 cannot be displaced by a slight vibration. I have. Therefore, in the detection of the corresponding point performed by the corresponding point search device 24, even if the corresponding point is detected by scanning the left image and the right image with the epipolar line aligned based on the correction condition detected every several screens, The detection rate can be sufficiently improved for practical use.

The corresponding point search unit 32 is configured using a computer so that processing for detecting a correction condition, for example, processing for obtaining an evaluation value for each candidate condition can be performed in parallel. By detecting the correction condition, the number of parallel processes is reduced to reduce the memory capacity required for the parallel processes, or the corresponding point searching unit 32 is configured using a computer having a lower processing speed. Or you can. As a result, the device scale of the corresponding point searching unit 32 can be reduced, or the device cost can be reduced.

<Fifth Embodiment of Claim 1> FIG.
FIG. 7 is a functional block diagram for explaining a fifth embodiment of the first aspect of the present invention. Hereinafter, points different from the first embodiment of the first aspect of the invention will be mainly described, and detailed description of the same points as those of the first embodiment of the first aspect of the invention will be omitted.

The modification condition detecting device 58 of this embodiment
A correction condition detection unit 60, a detection rate calculation unit 62, and a comparison determination unit 64 are provided.

The correction condition detecting section 60 includes the condition setting section 28 and the image converting section 3 as in the first embodiment of the present invention.
0, a corresponding point search unit 32, an evaluation value calculation unit 34, a maximum value detection unit 36, a third left memory 38 and a third right memory 40.

The detection rate calculating section 62 calculates the corresponding point searching device 24
The detection rate of the corresponding point in is calculated. For example, using the number of detected corresponding points in the corresponding point search device 24 or the number of detected unknown points in addition thereto, (corresponding point detection rate) = (corresponding point detection in the left image of the second left memory 20) (Number) / (sum of detected numbers of corresponding points and unknown points in left image of second left memory 20), (detection rate of corresponding points) = (number of detected corresponding points in left image of second left memory 20) ) / (Sum of the number of detected corresponding points and unknown points in the left image in the second left memory 20).

The comparison / determination section 64 receives the detection rate of the corresponding point from the detection rate calculation section 62 and compares the detection rate of the corresponding point with a threshold.

The correction condition detecting section 60 receives the comparison result between the corresponding point detection rate and the threshold value from the comparison / determination section 64, and detects and detects a new correction condition if the corresponding point detection rate is equal to or less than the threshold value. The left image and the right image that have undergone image conversion according to the new correction condition are stored in the second left memory 20 and the second left and right memory 22.

If the detection rate of the corresponding point exceeds the threshold value, the correction condition detection section 60 does not perform the process of detecting a new correction condition, and converts the left image and the right image, which have been converted in accordance with the already detected correction conditions, into the second image. The data is stored in the left memory 20 and the second right memory 22.

Before operating the corresponding point searching device 24 and the correction condition detecting device 58, the lens optical axes of the left camera 10 and the right camera 12 are mechanically moved and adjusted as usual, and The epipolar lines of the left image and the right image photographed by 10 and 12 are roughly adjusted, and thereafter, image conversion based on the correction conditions is performed by the correction condition detection device 58 of this embodiment, and the left image and the right image are converted. It is good to match the epipolar line.

According to this embodiment, the correction condition is detected only when the detection rate of the corresponding point is equal to or less than the threshold value, so that a long processing time for detecting the correction condition can be secured. The corresponding point search unit 32 is usually configured using a computer that can perform processing for detecting the correction condition in parallel, but only when the detection rate of the corresponding point falls below the threshold, the correction is performed. By detecting the condition, the number of parallel processes is reduced to reduce the memory capacity required for the parallel processes, or the corresponding point search unit 32 is configured using a computer having a lower processing speed. be able to. As a result, the device scale of the corresponding point searching unit 32 can be reduced, or the device cost can be reduced.

If the epipolar lines of the left image and the right image deviate for some reason, the detection rate of the corresponding point in the corresponding point searching device 24 decreases. Therefore, when the epipolar line is deviated, the correction condition is detected and the epipolar line is detected. You can adjust the line.

<First Embodiment of Claim 5> FIG.
FIG. 7 is a functional block diagram for explaining a first embodiment of the invention of claim 5. Hereinafter, points different from the first embodiment of the first aspect of the invention will be mainly described, and detailed description of the same points as those of the first embodiment of the first aspect of the invention will be omitted.

The modified condition detecting device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
An evaluation value calculation unit 34 and a maximum value detection unit 36, a third left memory 38 and a third right memory 40 are provided.

Further, the correction condition detecting device 1 of this embodiment
8 includes a gradation conversion unit 66 and a level cutout unit 68, a fourth left memory 70 and a fourth right memory 72.

The condition setting unit 28 sequentially sets a plurality of types of candidate conditions for detecting a correction condition,
Performs image conversion of one of an m-gradation left image and an m-gradation right image obtained by the left camera 10 and the right camera 12 according to candidate conditions.

Here, the condition setting section 28 stores a plurality of types of candidate conditions for detecting a correction condition and a plurality of types of correction conditions for matching epipolar lines. The correction condition to be stored is the correction condition detected by the maximum value detection unit 36.

The condition setting section 28 stores candidate conditions relating to image parallel movement, rotation, and magnification conversion. Therefore, the correction conditions detected by the maximum value detection unit 36 are correction conditions relating to the parallel movement, rotation, and magnification conversion of the image.
The image conversion performed by 0 according to the candidate condition and the correction condition is translation, rotation and magnification conversion of the image.

The gradation conversion unit 66 converts the right and left images after image conversion into left and right images of n gradations (m> n), and the level cutout unit 68 converts the left and right images after gradation conversion into one or more focused images. Plural types of gradation levels Here, the left and right images of one type of focused gradation level are cut out as left and right level images of interest.

For example, m obtained by the left and right cameras 10 and 12
= 256 gradation left and right images are obtained, the image conversion unit 30
Performs image conversion of one of the left and right images of m = 256 gradations according to the candidate condition, and the gradation conversion unit 66 sets m = 256 after the image conversion.
The left and right images of the gradation are converted into left and right images of n = 16 gradations, and the level cutout unit 68 extracts one type of gradation level of interest, for example, the eighth floor, from the 16 gradation left and right images after the gradation conversion. The left and right images at the tonal level are cut out as left and right attention level images.

The corresponding point search unit 32 sets a plurality of types of search conditions by changing the scanning direction, the gradation expression of the pixel to be scanned, or the color of the pixel to be scanned, and sets the left and right level image of interest. Let it be a left and right search image.

The corresponding point search unit 32 scans the left and right search images according to the search conditions, compares the pixel characteristics of the left search image and the pixel characteristics of the right search image on the corresponding horizontal scanning lines, and determines the difference between the pixel characteristics. The pixels of the left and right search images for which is less than or equal to the threshold are detected as corresponding candidate points. Further, the corresponding point search unit 32 detects, as corresponding points, corresponding candidate points that become a common set in the set of corresponding candidate points detected for each search condition.

Here, the left and right focused level images of the black and white image are assumed to be left and right search images. In addition, one pixel of the left and right search image, which is a pixel existing on the corresponding horizontal scanning line in the left and right search image, is set as a point of interest, and a pixel of the other image of the left and right search image is set as a control point. Then, a plurality of types of search conditions having different scanning directions, that is, a plurality of types of search conditions having different target point scanning directions and / or different control point scanning directions are set.

In accordance with the search conditions, the i-th point of interest sequentially counted in the scanning direction of interest on the corresponding horizontal scanning line L of the left and right search image is compared with the control point on the corresponding horizontal scanning line L. Are sequentially scanned in the control point scanning direction. After scanning all the control points on the horizontal scanning line L without detecting a control point whose pixel feature difference is equal to or smaller than the threshold value with respect to the i-th point of interest, the control is then performed on the (i + 1) -th point of interest. Perform a point scan. When a control point at which the difference of the pixel feature with respect to the i-th point of interest is smaller than or equal to the threshold is detected, the i-th point of interest and the control point are detected as corresponding candidate points, and then the (i + 1) -th point of interest is detected. Is scanned for the point of interest.

When the corresponding candidate points are detected for each search condition in this way, the corresponding candidate points that are a common set in the set of corresponding candidate points detected for each search condition are detected as corresponding points.

For example, a first search condition in which the scanning direction of the point of interest is the right direction (a direction in which the x coordinate gradually increases) and the scanning direction of the control point is the right direction, A second search condition for setting the scanning direction to the left (a direction in which the y coordinate becomes gradually smaller) is set. Then, scanning the left and right search images according to the first search condition to detect corresponding candidate points, further scanning the left and right search images according to the second search condition to detect corresponding candidate points, and detecting the corresponding candidate points according to the first search condition Corresponding candidate points that are a common set between the set of corresponding candidate points and the set of corresponding candidate points detected according to the second search condition are detected as corresponding points.

Next, the corresponding point search section 32 detects a known point and an unknown point of the phase difference in the left and right search images using the corresponding point detection result. Here, based on the phase difference between the corresponding points, the phase difference is estimated and applied to an unknown correspondence point (pixel not detected as the corresponding point). The corresponding unknown points to which the phase difference could not be applied are detected as the phase difference unknown points. Then, regarding the phase difference applied to the unknown point,
Test whether the phase difference is correct. Then, the unknown point of the correspondence of the phase difference determined to be correct by this test is detected as a known point of the phase difference, and the unknown point of the correspondence of the phase difference determined to be incorrect is detected as the unknown point of the phase difference.

The evaluation value producing section 34 obtains the evaluation value of the candidate condition for each type of the candidate condition by using the detection rate of the phase difference found point in the left and right search images. Here, the detection rate of the phase difference found point in the left and right search images is obtained for each candidate condition, and the detection rate is used as the evaluation value of the candidate condition.

The maximum value detector 36 detects a candidate condition that has obtained the maximum evaluation value for each type of candidate condition as a correction condition.
Here, the maximum value detecting unit 36 causes the condition setting unit 28 to store the detected correction condition.

FIG. 18 is a diagram showing a flow of the operation of the first embodiment of the present invention. FIG. 7 shows a flow of operation in a case where attention is paid to one of a plurality of types of candidate conditions and correction conditions, and a correction condition is detected using the candidate conditions for the type of interest.

When the operator inputs an operation start signal to the correction condition detecting device 18 (start), the condition setting section 28 first sets one of a plurality of types of candidate conditions as a candidate condition of a target type (S1). Thereafter, the condition setting unit 28 outputs one candidate condition (selected candidate condition) selected from the plurality of candidate conditions of the target species (S2).

[0284] Next, the image converter 30 includes the left and right cameras 10,
One of the left and right images of m gradation captured by the step 12 is subjected to image conversion in accordance with the selection candidate conditions input from the condition setting unit 28, and the left and right images after image conversion are stored in the third left and right memories 38 and 40.
(S3). The left and right images captured by the left and right cameras 10 and 12 are stored in first left and right memories 14 and 16, and the image conversion unit 30 reads the left and right images of the first left and right memories 14 and 16, and one of the read left and right images is an image. The left and right images after image conversion are stored in the third left and right memories 38 and 40 without conversion.

Next, the gradation conversion section 66 reads the left and right images after image conversion from the third left and right memories 38 and 40 and converts them into n gradation (m> n) left and right images. Is stored in the fourth left and right memories 70 and 72 (S4). For example, m = 256 gradations and n = 16 gradations.

Next, the level cutout section 68 scans the left and right images after gradation conversion stored in the fourth left and right memories 70 and 72 to detect a pixel of one type of gradation level of interest. The left and right images composed of pixels of the target gradation level are cut out as left and right target level images (S5). For example, n =
The eighth gray level of the 16 gray levels is set as the target gray level.

Next, the corresponding point search unit 32 sets a plurality of types of search conditions by changing the scanning direction, sets the left and right attention level images as left and right search images, and sets a corresponding candidate in the left and right search images for each search condition. A point is detected (S6). Here, the left and right focused level images corresponding to the entire left and right images after gradation conversion are set as left and right search images.

For example, it is assumed that the left and right images taken by the left and right cameras 10 and 12 are black and white images, and therefore, a black and white image can be obtained as the left and right search image. Further, the pixel of the left search image is a point of interest, and the pixel of the right search image is a control point,
A first search condition that sets the target point scanning direction to the right and a control point scanning direction to the right is set, and a second search condition that sets the target point scanning direction to the right and the control point scanning direction to the left is set. .

In accordance with the first search condition, the i-th point of interest is sequentially counted in the scanning direction of the point of interest on the corresponding horizontal scanning line L of the left and right search image, and the corresponding point on the corresponding horizontal scanning line L Are sequentially scanned in the scanning direction of the reference point. When all the control points on the horizontal scanning line L have been scanned without detecting a control point having a pixel characteristic, for example, a difference in pixel density equal to or smaller than a threshold value, with respect to the i-th point of interest,
The control point is scanned for the (+1) th point of interest. When a control point at which the difference of the pixel feature with respect to the i-th point of interest is smaller than or equal to the threshold is detected, the i-th point of interest and the control point are detected as corresponding candidate points, and then the (i + 1) -th point of interest is detected. Is scanned for the point of interest. When the scanning of the target point and the reference point on the horizontal scanning line L is completed over the entire left and right search image, the detection of the corresponding candidate point based on the first search condition is completed. Similarly, the left and right search images are scanned under the second search condition, and the detection of the corresponding candidate points under the second search condition ends.

When the detection of the corresponding candidate points is completed for all types of search conditions, the corresponding point search unit 32 next associates the corresponding candidate points that become a common set in the set of corresponding candidate points obtained for each search condition. It is detected as a point (S7).

As described above, the left and right search images on the corresponding horizontal scanning line L are scanned, and when the difference between the pixel characteristics is equal to or less than the threshold value, the next point of interest is scanned to detect a corresponding candidate point. By detecting as a corresponding point a corresponding candidate point that is a common set in a set of corresponding candidate points obtained for each search condition, it is possible to detect a corresponding point that is likely to be stochastic.

Next, the corresponding point search section 32 detects the phase difference found point and the phase difference unknown point using the corresponding point detection result, and evaluates the phase difference found point or the number of detected phase difference unknown points in addition thereto. Output to the value calculation unit 34 (S8).

Here, the detected phase difference points and the unknown phase difference points relate to the entire left and right search image.

Next, the evaluation value calculation section 32 obtains the detection rate of the phase difference found point using the number of detected phase difference found points, and calculates (S
The detection rate of the phase difference found point is stored in the maximum value detection unit 36 as the evaluation value of the selection candidate condition output by the condition setting unit 28 in 2) (S9).

[0295] The detection rate of the phase difference found point obtained here relates to the entire left and right images that are the current left and right search images.

Next, the condition setting section 28 determines whether or not all the candidate conditions of the target species have been selected (S10).

When the condition setting unit 28 determines that all the candidate conditions of the target species have not been selected, the condition setting unit 28 selects the next candidate condition of the target species (S11), and then selects the next candidate condition. The candidate condition is output (S2).

When the condition setting unit 28 determines that all the candidate conditions of the target species have been selected, the maximum value detection unit 36 determines the maximum evaluation value from among the evaluation values obtained for each candidate condition of the target species. The candidate condition that has been detected and obtained the maximum evaluation value is detected as a correction condition. Then, the maximum value detection unit 28 stores the correction condition of the target type in the condition setting unit 28 (S12). Thereafter, the correction condition detection device 18 ends the processing for detecting the correction condition of the target type (end).

According to this embodiment, the left and right cameras 10,
The left and right images obtained by photographing by the step 12 are subjected to the gradation conversion, and the left and right images after the gradation conversion are cut out from the left and right images, and the left and right attention level images composed only of the pixels of the attention gradation level are cut out. Is detected, the time required to detect the correction condition used for epipolar line alignment can be reduced.

In the above description of the present embodiment, the flow of operation of the gradation conversion unit 66, the level cutout unit 68, and the corresponding point search unit 32 when setting a plurality of types of search conditions with different scanning directions. However, it is also possible to set a plurality of types of search conditions by changing the gradation expression of the pixel to be scanned or the color of the pixel to be scanned.

In order to make the gradation expression of the pixel to be scanned different, for example, the following may be performed. That is, the left and right images are converted into black and white images, the gradation conversion unit 66 converts the m gradation left and right images after image conversion into n ₁ (m> n ₁ ) gradation left and right images, and the level cutout unit 68 performs gradation conversion. The left and right attention level images are cut out from the subsequent left and right images of n ₁ gradation, and the corresponding point search unit 32 detects corresponding candidate points for the left and right attention level images. When the detection of the corresponding candidate points for the left and right target level image of n ₁ gradation is completed, the gradation conversion unit 66 converts the left and right image of m gradation after the image conversion into the left and right image of n ₂ gradation, The cutout unit 68 outputs n ₂ (m>
From the left and right images of n ₂ ) gradation, the left and right attention level images are cut out, and the corresponding point search unit 32 detects corresponding candidate points for the left and right attention level images. At this time, assuming that n ₁ = n ₂ , the first and last gradation widths of the left and right images of n ₁ gradation and the first and last gradation widths of the left and right images of n ₂ gradation are made different,
Different gradation expression. Alternatively, the gradation expression is made different by setting n ₁ ≠ n ₂ . In this manner, a plurality of types of search conditions are set for detecting corresponding candidate points for the left and right attention level images cut out from the left and right images of n ₁ and n ₂ gradations in which the gradation expression of the pixel to be scanned is different. Then, corresponding candidate points may be detected, and corresponding candidate points that become a common set in a set of corresponding candidate points detected for each search condition may be detected as corresponding points.

When the colors of the pixels to be scanned are made different, for example, the following may be performed. That is, with the left and right images as color images, the gradation conversion unit 66
The left and right images of the gradation are converted into left and right images of the n (m> n) gradations, and the level cutout unit 68 is composed of only the red pixels of the focused gradation level from the left and right images of the n gradations after the gradation conversion. The left and right attention level images are cut out, and the corresponding point search unit 32 detects corresponding candidate points for the left and right attention level images of the red pixels. After the detection of the corresponding candidate points for the right and left focused level images of the red pixels is completed, the level cutout unit 68 next extracts the left and right focused images including only the green pixels of the focused gradation level from the n-graded left and right images after the gradation conversion. The level image is cut out, and the corresponding point search unit 32 detects a corresponding candidate point for the left and right focused level images of the green pixels. After the detection of the corresponding candidate points for the left and right focused level images of the green pixels is completed, the level cutout unit 68 next extracts the left and right focused images including only the blue pixels of the focused tone level from the n-graded left and right images after the tone conversion. The level image is cut out, and the corresponding point searching unit 32 detects a corresponding candidate point for the left and right focused level images of the blue pixel. In this way, a plurality of types of search conditions for detecting corresponding candidate points are set for the left and right target level images in which the colors of the pixels to be scanned are different, the corresponding candidate points are detected, and each of the search conditions is detected. It is only necessary to detect the corresponding candidate points that become a common set in the set of the corresponding candidate points as the corresponding points.

<Second Embodiment of the Invention of Claim 5> Next, a second embodiment of the invention of claim 5 will be described with reference to FIG. Hereinafter, points different from the first embodiment of the invention of claim 5 will be mainly described, and detailed description of the same points as those of the first embodiment of the invention of claim 5 will be omitted.

The modification condition detection device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
An evaluation value calculation unit 34 and a maximum value detection unit 36, a third left memory 38 and a third right memory 40 are provided.

Further, the correction condition detecting device 1 of this embodiment
8 includes a gradation conversion unit 66 and a level cutout unit 68, a fourth left memory 70 and a fourth right memory 72.

In this embodiment, the level extracting section 68 sequentially extracts left and right images of a plurality of types of gradation levels of interest from the left and right images after gradation conversion as left and right attention level images. The detection rate of the phase difference finding point is obtained for each of the same left and right search images, and the average value of the detection rates obtained for the left and right search images having the same candidate condition is set as the evaluation value of the candidate condition.

FIG. 19 is a diagram showing the flow of the operation of the second embodiment of the present invention. FIG. 7 shows a flow of operation in a case where attention is paid to one of a plurality of types of candidate conditions and correction conditions, and a correction condition is detected using the candidate conditions for the type of interest.

When the operator inputs an operation start signal to the correction condition detecting device 18 (start), the condition setting section 28 sets one of a plurality of types of candidate conditions as a candidate condition of a target type (S1). Thereafter, the condition setting unit 28 outputs one candidate condition (selected candidate condition) selected from the plurality of candidate conditions of the target species (S2).

[0309] Next, the image conversion unit 30 includes the left and right cameras 10,
One of the left and right images of m gradation captured by the step 12 is subjected to image conversion in accordance with the selection candidate conditions input from the condition setting unit 28, and the left and right images after image conversion are stored in the third left and right memories 38 and 40.
(S3). The left and right images captured by the left and right cameras 10 and 12 are stored in first left and right memories 14 and 16, and the image conversion unit 30 reads the left and right images of the first left and right memories 14 and 16, and one of the read left and right images is an image. The left and right images after image conversion are stored in the third left and right memories 38 and 40 without conversion.

Next, the gradation conversion section 66 reads the left and right images after image conversion from the third left and right memories 38 and 40 and converts them into n right and left images (m> n). Is stored in the fourth left and right memories 70 and 72 (S4).

Next, the level extracting section 68 sequentially extracts left and right images of a plurality of types of focused gradation levels from the left and right images after gradation conversion as left and right focused level images. The level cutout unit 68 scans the left and right images after gradation conversion stored in the fourth left and right memories 70 and 72, and detects a pixel of one kind of gradation level among a plurality of kinds of gradation levels of interest. Then, the left and right images composed of pixels of the target gradation level are
The image is cut out as the left and right attention level images (S5). For example, n
Of the 16th gray level, focus on the fifth, eighth, and eleventh gray levels. Cut out as an attention level image.

Next, the corresponding point search unit 32 sets a plurality of types of search conditions by changing the scanning direction, sets the left and right attention level images as left and right search images, and sets the corresponding candidate in the left and right search images for each search condition. A point is detected (S6). Here, the left and right focused level images corresponding to the entire left and right images after gradation conversion are set as left and right search images.

When the detection of the corresponding candidate points has been completed for all types of search conditions, the corresponding point search unit 32 next associates the corresponding candidate points that become a common set in the set of corresponding candidate points obtained for each search condition. It is detected as a point (S7).

Next, the corresponding point search section 32 detects the phase difference found point and the phase difference unknown point by using the corresponding point detection result, and evaluates the phase difference found point or the number of detected phase difference unknown points in addition thereto. Output to the value calculation unit 34 (S8).

Here, the detected phase difference points and the unknown phase difference points relate to the entire left and right search image.

Next, the evaluation value calculation section 32 obtains the detection rate of the phase difference found point using the number of detected phase difference found points, and calculates (S
The detection rate of the phase difference found point is stored in the maximum value detection unit 36 as the evaluation value of the selection candidate condition output by the condition setting unit 28 in 2) (S9).

[0317] The detection rate of the phase difference found point obtained here relates to the entire left and right images that are the current left and right search images.

Next, the level cutout section 68 determines whether or not the cutout of the left and right focused level images has been completed for all types of focused tone levels (S10).

If the processing has not been completed for all the target gradation levels, the level cutout section 68 selects the remaining target gradation levels (S11), and selects the left and right target level images for the selected target gradation levels. (S
5). For example, if attention is paid to the fifth, eighth, and eleventh gray levels among n = 16 gray levels, and the extraction of the left and right target level images is completed only for the fifth gray level, The eighth gray level is selected as the next target gray level, and the left and right target level images are cut out for the eighth target selected gray level.

If the processing has been completed for all the types of gradation levels of interest, the evaluation value calculation unit 34 next calculates the average value of the phase difference determination point detection rates obtained for each of the target gradation levels. The average value is stored in the maximum value detection unit 36 as the evaluation value of the candidate condition output from the condition selection unit 28 in (S2) (S1).
2). For example, focusing on the fifth, eighth, and eleventh gray levels among n = 16 gray levels, for example,
If the extraction of the left and right target level images has been completed for all the target target tone levels, the average value of the phase difference determination point detection rates obtained for each of the three target target tone levels is obtained.

Next, the condition setting section 28 judges whether or not all the candidate conditions of the target species have been selected (S13).

If all the candidate conditions of the target species have not been selected, the condition setting unit 28 selects the next candidate condition of the target species (S14), and then outputs the selected candidate conditions (S2). ).

When all the candidate conditions of the target species have been selected, the maximum value detecting unit 36 detects the maximum evaluation value from among the evaluation values obtained for each candidate condition of the target species, and Are detected as correction conditions. Then, the maximum value detecting unit 28 stores the correction condition of the target type in the condition setting unit 28 (S15). Thereafter, the correction condition detection device 18 ends the processing for detecting the correction condition of the target type (end).

<Third Embodiment of the Invention of Claim 5> Next, a third embodiment of the invention of claim 5 will be described with reference to FIG. Hereinafter, points different from the first embodiment of the invention of claim 5 will be mainly described, and detailed description of the same points as those of the first embodiment of the invention of claim 5 will be omitted.

The correction condition detecting device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
An evaluation value calculation unit 34 and a maximum value detection unit 36, a third left memory 38 and a third right memory 40 are provided.

Further, the correction condition detecting device 1 of this embodiment
8 includes a gradation conversion unit 66 and a level cutout unit 68, a fourth left memory 70 and a fourth right memory 72.

In this embodiment, the level extracting section 68 sequentially extracts left and right images of a plurality of types of focused gradation levels as left and right focused level images from the left and right images after gradation conversion. For each of the left and right search images having the same condition, the detection rate of the phase difference found point is obtained, and among the detection rates obtained for the left and right search images having the same candidate condition, a predetermined number of detection rates are sequentially selected from the top, and by this selection The average value of the obtained detection rates is used as the evaluation value of the candidate condition.

FIG. 20 is a diagram showing the flow of the operation of the third embodiment of the present invention. FIG. 7 shows a flow of operation in a case where attention is paid to one of a plurality of types of candidate conditions and correction conditions, and a correction condition is detected using the candidate conditions for the type of interest.

When the operator inputs an operation start signal to the correction condition detecting device 18 (start), first, the condition setting section 28 sets one of a plurality of types of candidate conditions as a candidate condition of a target type (S1). Thereafter, the condition setting unit 28 outputs one candidate condition (selected candidate condition) selected from the plurality of candidate conditions of the target species (S2).

Next, the image conversion unit 30 includes the left and right cameras 10,
One of the left and right images of m gradation captured by the step 12 is subjected to image conversion in accordance with the selection candidate conditions input from the condition setting unit 28, and the left and right images after image conversion are stored in the third left and right memories 38 and 40.
(S3). The left and right images captured by the left and right cameras 10 and 12 are stored in first left and right memories 14 and 16, and the image conversion unit 30 reads the left and right images of the first left and right memories 14 and 16, and one of the read left and right images is an image. The left and right images after image conversion are stored in the third left and right memories 38 and 40 without conversion.

Next, the gradation conversion section 66 reads the left and right images after image conversion from the third left and right memories 38 and 40 and converts them into left and right images of n gradations (m> n). Is stored in the fourth left and right memories 70 and 72 (S4).

Next, the level extracting section 68 sequentially extracts left and right images of a plurality of kinds of focused gradation levels as left and right focused level images from the left and right images after gradation conversion. The level cutout unit 68 scans the left and right images after gradation conversion stored in the fourth left and right memories 70 and 72, and detects a pixel of one kind of gradation level among a plurality of kinds of gradation levels of interest. Then, the left and right images composed of pixels of the target gradation level are
The image is cut out as the left and right attention level images (S5). For example, n
Of the 16th gray level, focus on the fifth, eighth, and eleventh gray levels. Cut out as an attention level image.

Next, the corresponding point search section 32 sets a plurality of types of search conditions by changing the scanning direction, sets the left and right attention level images as left and right search images, and sets the corresponding candidate in the left and right search images for each search condition. A point is detected (S6). Here, the left and right focused level images corresponding to the entire left and right images after gradation conversion are set as left and right search images.

When the detection of corresponding candidate points is completed for all types of search conditions, the corresponding point search unit 32 next associates the corresponding candidate points that are a common set in the set of corresponding candidate points obtained for each search condition. It is detected as a point (S7).

Next, the corresponding point searching section 32 detects the phase difference found point and the phase difference unknown point using the corresponding point detection result, and evaluates the phase difference found point or the number of detected phase difference unknown points in addition thereto. Output to the value calculation unit 34 (S8).

Here, the detected phase difference points and the unknown phase difference points relate to the entire left and right search image.

Next, the evaluation value calculation unit 32 obtains the detection rate of the phase difference found point by using the detected number of the phase difference found point, and calculates (S
The detection rate of the phase difference found point is stored in the maximum value detection unit 36 as the evaluation value of the selection candidate condition output by the condition setting unit 28 in 2) (S9).

[0338] The detection rate of the phase difference found point obtained here relates to the entire left and right images that are the current left and right search images.

Next, the level cutout section 68 determines whether or not the cutout of the left and right focused level images has been completed for all types of focused tone levels (S10).

If the processing has not been completed for all the target gradation levels, the level cutout unit 68 selects the remaining target gradation levels (S11), and selects the left and right target level images for the selected target gradation levels. (S
5). For example, if attention is paid to the fifth, eighth, and eleventh gray levels among n = 16 gray levels, and the extraction of the left and right target level images is completed only for the fifth gray level, The eighth gray level is selected as the next target gray level, and the left and right target level images are cut out for the eighth target selected gray level.

If the processing has been completed for all types of gray levels of interest, the evaluation value calculation unit 34 next calculates the detection rate of the phase difference determination point obtained for each gray level of interest with respect to the candidate condition. Among them, a higher detection rate is selected (S12).

Next, the evaluation value calculation section 34 calculates an average value of the selected higher detection rates, and stores the average value in the maximum value detection section 36 as an evaluation value relating to the candidate condition (S1).
3).

Here, assuming that, for one candidate condition, R gradation levels of different types are set as target gradation levels, the detection rate of the phase difference determination point is determined for each target gradation level. Therefore, the detection rate of the total number R is obtained. Then, a detection rate of the top r1 detection rates (r1 is a natural number satisfying R>r1> 1) is selected from among the detection rates of the total number R, and an average value of the selected top r1 detection rates is used as an evaluation value. I do. (R-r1)> 0
1) The detection rate is not used for calculating the evaluation value.

Next, the condition setting section 28 judges whether or not all the candidate conditions of the target type have been selected (S14).

If all the candidate conditions of the target species have not been selected, the condition setting section 28 selects the next candidate condition of the target species (S15), and then outputs the selected candidate conditions (S2). ).

When all the candidate conditions of the target species have been selected, the maximum value detecting unit 36 detects the maximum evaluation value from among the evaluation values obtained for each candidate condition of the target species, and Are detected as correction conditions. Then, the maximum value detection unit 28 stores the correction condition of the target type in the condition setting unit 28 (S16). Thereafter, the correction condition detection device 18 ends the processing for detecting the correction condition of the target type (end).

<First Embodiment of Claim 9> FIG.
FIG. 14 is a functional block diagram for explaining a first embodiment of the ninth aspect of the present invention. Hereinafter, points different from the first embodiment of the first aspect of the present invention will be described, and detailed description of the same points as those of the first embodiment of the first aspect of the present invention will be omitted.

The correction condition detecting device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
An evaluation value calculation unit 34 and a maximum value detection unit 36, a third left memory 38 and a third right memory 40, and a phase difference range input unit 74 are further provided.

The condition setting unit 28 sequentially sets candidate conditions for detecting the height direction correction condition, and the image conversion unit 30 converts one of the left image and the right image obtained by the left camera 10 and the right camera 12 Image conversion is performed according to the candidate condition.

In this case, other types of correction conditions including the height direction correction conditions are also detected. The condition setting section 28 sets a plurality of types of candidate conditions for detecting the correction conditions and epipolar lines. A plurality of types of correction conditions for matching are stored. The correction condition to be stored is the correction condition detected by the maximum value detection unit 36. One of the plurality of types of candidate conditions and correction conditions is a height-direction candidate condition and correction condition.

The condition setting section 28 stores candidate conditions relating to image parallel movement, rotation, and magnification conversion. Therefore, the correction conditions detected by the maximum value detection unit 36 are correction conditions relating to the parallel movement, rotation, and magnification conversion of the image.
The image conversion performed by 0 according to the candidate condition and the correction condition is translation, rotation and magnification conversion of the image. Known methods can be used for the translation, rotation, and magnification conversion.

The corresponding point search unit 32 sets a plurality of types of search conditions by changing the scanning direction, the gradation expression of the pixel to be scanned, or the color of the pixel to be scanned, and sets the left and right after image conversion. Let the image be a left and right search image. Here, left and right images of a color image are obtained by the left and right cameras 10 and 12, and a plurality of types of search conditions are set by changing colors of pixels to be scanned. The left and right images after the image conversion are set as the left and right search images over the entire surface.

The corresponding point search section 32 scans the left and right search images according to the search conditions, compares the pixel characteristics of the left search image and the pixel characteristics of the right search image on the corresponding horizontal scanning lines, and determines the difference between the pixel characteristics. The pixels of the left and right search images for which is less than or equal to the threshold are detected as corresponding candidate points. Further, the corresponding point search unit 32 detects, as corresponding points, corresponding candidate points that become a common set in the set of corresponding candidate points detected for each search condition. Here, the pixel feature is a pixel density.

For example, a color image is stored in the third left memory 38 and the third right memory 40 as left and right images after image conversion, and a red image among the left and right images after image conversion is used as a left and right search image. A search condition and a second search condition for setting the blue image among the left and right images after the image conversion as the left and right search image are set. Then, the left and right search images of the red image are scanned according to the first search condition, and the pixels of the left and right search image in which the difference of the pixel characteristics is equal to or less than the threshold on the corresponding horizontal scanning line are identified as the corresponding candidate points according to the first search condition Detected as
Further, the left and right search images of the blue image are scanned according to the second search condition, and the pixels of the left and right search images whose pixel feature difference is equal to or smaller than the threshold on the corresponding horizontal scanning line are identified as corresponding candidate points according to the second search condition. Detected as Thereafter, a corresponding candidate point that is a common set between the set of corresponding candidate points detected under the first search condition and the set of corresponding candidate points detected under the second search condition is detected as a corresponding point.

Next, the corresponding point search section 32 detects, using the corresponding point detection result, a known point and an unknown point of the short-distance phase difference in which the shift amount in the height direction becomes large in the left and right search images. The operator sets the range of the short-distance phase difference where the amount of displacement in the height direction is large via the phase difference range input unit 74 via the corresponding point search unit 3.
Enter 2 For example, the range where the phase difference is 80 or more is
It is input as a range of the short-distance phase difference in which the amount of displacement in the height direction is large.

Here, based on the phase difference between the corresponding points, the phase difference is estimated and applied to an unknown correspondence point (a pixel not detected as a corresponding point). The corresponding unknown points to which the phase difference could not be applied are detected as the phase difference unknown points. Then, regarding the phase difference applied to the unknown point,
Test whether the phase difference is correct. Then, the unknown point of the correspondence of the phase difference determined to be correct by this test is detected as a known point of the phase difference, and the unknown point of the correspondence of the phase difference determined to be incorrect is detected as the unknown point of the phase difference.

Since a plurality of types of search conditions are set by making the color of the pixel to be scanned different, the fitting of the phase difference and the test of the phase difference are performed for each color of the pixel to be scanned. Only pixels for which the phase difference has been determined to be correct for all colors of the pixel to be scanned are regarded as phase difference determination points, and pixels for which the phase difference has been determined to be incorrect for any of the colors of the pixel to be scanned. Is an unknown phase difference point.

Thereafter, from the phase difference determination points, the short distance phase difference determination point where the amount of displacement in the height direction becomes large is detected.

The evaluation value calculation unit calculates the detection rate of the phase difference known point for the short-distance phase difference in which the shift amount in the height direction is large in the left and right search images, and uses the detection rate to determine the height direction candidate. Find the evaluation value of the condition. Here, for the short-distance phase difference in which the amount of displacement in the height direction is large in the left and right search images, the detection rate of the phase difference known point is obtained for each height direction candidate condition, and the detection rate is the evaluation value of the height direction candidate condition. And

The maximum value detecting section 36 detects the height direction candidate condition that has obtained the maximum evaluation value as the height direction correction condition.
Here, the maximum value detection unit 36 stores the detected height direction correction condition in the condition setting unit 28.

FIG. 22 is a diagram showing the flow of the operation of the second embodiment of the ninth aspect of the present invention. In the same figure, the flow of operation when detecting the height direction correction condition using the height direction candidate condition by focusing on the height direction candidate condition and the correction condition among the plurality of types of candidate conditions and correction conditions. Is shown.

When the operator inputs an operation start signal to the correction condition detection device 18 (start), the condition setting section 28 first selects and sets a candidate condition in the height direction from a plurality of types of candidate conditions (S1). Thereafter, the condition setting unit 28 outputs one height direction candidate condition (selected height direction candidate condition) selected from the plurality of height direction candidate conditions (S).
2). The height direction candidate conditions are candidate conditions relating to parallel movement in the y-axis direction, and for example, five height direction candidate conditions Δ
outputs one height direction candidate condition [Delta] y ₁ selected from among y ₁ ~Δy _5.

Next, the image conversion unit 30 includes the left and right cameras 10,
One of the left and right images taken by the
The left and right images after the image conversion are converted according to the selected height direction candidate condition input from the third left and right memories 38 and 40.
(S3). The left and right images captured by the left and right cameras 10 and 12 are stored in first left and right memories 14 and 16, and the image conversion unit 30 reads the left and right images of the first left and right memories 14 and 16, and one of the read left and right images is The left and right images after the image conversion are stored in the third left and right memories 38 and 40 without performing the image conversion.

Next, the corresponding point search section 32 sets a plurality of types of search conditions by changing the colors of the pixels to be scanned, and sets the entire left and right images after image conversion as left and right search images. Next, corresponding candidate points in the left and right search images are detected (S4). Here, since a plurality of types of search conditions are set by changing the colors of the pixels, the left and right images taken by the left and right cameras 10 and 12 are set as color images.

For example, among the left and right images of the converted color image stored in the third left and right memories 38 and 40, a first search condition for scanning a red right and left image as a left and right search image;
A second search condition for scanning the left and right blue images as the left and right search images is set. Then, the corresponding point search unit 32 scans the red left and right images according to the first search condition, cuts out the corresponding red left and right images on the horizontal scanning line L from the third left and right memories 38 and 40, and The data is stored in the left and right line memories provided in the point search unit 32. Here, in the left and right red images, one pixel is set as a point of interest and the other pixel is set as a reference point, and the i-th point of interest is sequentially counted in a predetermined main scanning direction, for example, rightward on the corresponding horizontal scanning line L. With respect to the point, a control point on the horizontal scanning line L is sequentially scanned in a predetermined main scanning direction, for example, rightward. When the scanning of the control point is completed without detecting the control point at which the pixel characteristic, for example, the difference of the pixel density is equal to or less than the threshold value, with respect to the i-th point of interest,
The control point is scanned for the (+1) th point of interest. When a control point at which the difference of the pixel feature with respect to the i-th point of interest is smaller than or equal to the threshold is detected, the i-th point of interest and the control point are detected as corresponding candidate points, and then the (i + 1) -th point of interest is detected. Is scanned for the point of interest. Third left and right memory 38,
When the detection of the corresponding candidate points by the first search condition is completed for the entire 40 right and left images, the blue left and right images are scanned by the second search condition in the same manner as the first search condition.
A corresponding candidate point is detected based on the second search condition.

[0366] When the detection of the corresponding candidate points is completed for all types of search conditions, the corresponding point search unit 32 next associates the corresponding candidate points that become a common set in the set of corresponding candidate points obtained for each search condition. It is detected as a point (S5).

As described above, the left and right search images on the corresponding horizontal scanning line L are scanned, and when the difference between the pixel characteristics becomes equal to or less than the threshold value, the next point of interest is scanned to detect a corresponding candidate point. The corresponding candidate points that become a common set in the set of corresponding candidate points obtained for each search condition are a set of corresponding candidate points that are likely to be stochastic.

Next, the corresponding point searching section 32 detects a phase difference known point and a phase difference unknown point in the short distance phase difference range using the corresponding point detection result (S6).

The detected phase difference point and the phase difference unknown point detected here relate to the entire left and right image which is the current left and right search image.

In this case, the phase difference is estimated and applied to an unknown correspondence point (pixel not detected as a corresponding point) based on the phase difference between the corresponding points. The corresponding unknown points to which the phase difference could not be applied are detected as the phase difference unknown points. Thereafter, it is checked whether the phase difference applied to the correspondence unknown point is correct. Then, the unknown point of the correspondence of the phase difference determined to be correct by this test is determined to be a known point of the phase difference, and the unknown point of correspondence of the phase difference determined to be incorrect is detected as the unknown point of the phase difference.

Since a plurality of types of search conditions are set by making the color of the pixel to be scanned different, the fitting of the phase difference and the test of the phase difference are performed for each color of the pixel to be scanned. Only pixels for which the phase difference has been determined to be correct for all colors of the pixel to be scanned are regarded as phase difference determination points, and pixels for which the phase difference has been determined to be incorrect for any of the colors of the pixel to be scanned. Is an unknown phase difference point. Alternatively, a pixel for which the phase difference is determined to be incorrect for all colors of the pixel to be scanned is determined to be a phase difference unknown point. Then, in a case where a color whose phase difference is determined to be correct and a color whose phase difference is determined to be incorrect are mixed in a pixel to be scanned, it is determined whether or not the phase difference is correct according to the principle of majority decision. Whether the phase difference is known or unknown may be determined according to the determination result.

Next, the corresponding point search section 32 calculates the number of detected phase difference found points in the short distance phase difference range by the evaluation value calculation section 34.
(S7). For example, a range having a phase difference of 80 or more is set as a short-range phase difference range.

Next, the evaluation value calculation unit 32 obtains the detection rate of the phase difference found point using the number of detected phase difference found points in the short distance phase difference range, and selects the selection rate output by the condition setting unit 28 in (S2). As the evaluation value of the height direction candidate condition, the detection rate of the phase difference found point in the short distance phase difference range is set to the maximum value detection unit 3.
6 (S8).

Next, the condition setting section 28 determines whether or not all the height direction candidate conditions have been selected (S9).

If all the height direction candidate conditions have not been selected, the condition setting unit 28 selects the next height direction candidate condition (S10), and then outputs the selected height direction candidate condition. (S2).

For example, only the selection of Δy ₁ among the height direction candidate conditions Δy _{1 to} Δy ₅ has been completed, and Δy ₂ to Δy ₅ have been selected.
If not completed selection for y _5, one of the candidate condition selected from among the remaining candidate condition Δy ₂ ~Δy ₅ Δ
Select y _2, and outputs the height direction candidate condition [Delta] y ₂ selected.

If all the height direction candidate conditions have not been selected, the maximum value detector 36 detects the maximum evaluation value from among the evaluation values obtained for each height direction candidate condition, and The height direction candidate condition for which the evaluation value has been obtained is detected as a height direction correction condition. Then, the maximum value detection unit 28 stores the height direction correction condition in the condition setting unit 28 (S11). After that, the correction condition detection device 18 ends the processing for detecting the height direction correction condition (end).

According to this embodiment, the detection rate of the phase difference found point in the short-distance phase difference range is set as the evaluation value, and the height direction candidate condition having the maximum evaluation value is set as the height direction correction condition. Since the amount of displacement in the height direction is large in the short-range phase difference range, it is more suitable for epipolar line alignment by detecting the height direction correction condition using the detection rate of the phase difference found point in the short-range phase difference range as the evaluation value. Height correction condition can be detected.

<Second Embodiment of the Claim 9> FIG.
FIG. 14 is a functional block diagram used for describing a second embodiment of the ninth aspect of the present invention. Hereinafter, points different from the first embodiment of the ninth aspect of the present invention will be mainly described, and detailed description of the same points as those of the first embodiment of the ninth aspect of the present invention will be omitted.

The modification condition detecting device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
Evaluation value calculation unit 34 and maximum value detection unit 36, third left memory 38 and third right memory 40, and phase difference range input unit 74
And a region cutout section 46.

The region extracting section 46 sequentially extracts left and right region images from a plurality of regions of interest vertically dispersed on the left and right images after image conversion. As search images, corresponding candidate points, corresponding points, known and unknown points of the phase difference are detected for each of the left and right search images having the same candidate condition.

The evaluation value calculation unit 34 calculates the detection rate of the phase difference known point for each of the left and right search images having the same height direction candidate condition for the short distance phase difference in which the shift amount in the height direction is large in the left and right search images. The average value of the detection rates obtained for the left and right search images having the same height direction candidate condition is used as the evaluation value of the height direction candidate condition.

FIG. 24 is a diagram showing a flow of the operation of the second embodiment of the ninth aspect of the present invention. In the figure, focusing on the height direction candidate condition and the correction condition among a plurality of types of candidate conditions and correction conditions, the flow of operation when detecting the height direction correction condition using the height direction candidate condition is shown. Show.

When the operator inputs an operation start signal to the correction condition detection device 18 (start), the condition setting section 28 first selects and sets a height direction candidate condition from a plurality of types of candidate conditions (S1). Thereafter, the condition setting unit 28 outputs one candidate condition (selected height direction candidate condition) selected from the plurality of height direction candidate conditions (S2).

Next, the image conversion unit 30 converts one of the left image and the right image according to the selected height direction candidate condition input from the condition setting unit 28, and converts the left image and the right image after the image conversion into the third left image. The data is stored in the memory 38 and the third right memory 40 (S3). The left and right images captured by the left and right cameras 10 and 12 are stored in first left and right memories 14 and 16, and the image conversion unit 30 reads the left and right images of the first left and right memories 14 and 16, and one of the read left and right images is an image. Without conversion, the other is image-converted, and the left and right images after the image conversion are stored in the third left and right memories 38 and 40.

Next, the region cutout section cuts out the left and right region of interest images from the plurality of regions of interest dispersed in the vertical direction (y direction) on the left and right images after image conversion (S4).

For example, as shown in FIG. 12, the width of the left and right region of interest image in the vertical direction is the width of one horizontal scanning line or the width of a plurality of lines, here the width of one line. Cuts out the left and right region of interest images on the corresponding horizontal scanning lines L from the converted left and right images stored in the third left and right memories 38 and 40, and outputs the extracted horizontal scanning lines L
The upper left and right region of interest images are stored in the left and right line memories of the corresponding point search unit 32.

Next, the corresponding point search unit 32 sets a plurality of types of search conditions by changing the color of the pixel to be scanned, sets the left and right region of interest images as left and right search images, and sets the left and right search images for each search condition. A corresponding candidate point in the search image is detected (S
5).

When the detection of the corresponding candidate points on the horizontal scanning line L over the entire left and right search image has been completed for all types of search conditions, the corresponding point search unit 32 next calculates the corresponding candidate points obtained for each search condition. Corresponding candidate points that become a common set in the set are detected as corresponding points (S6).

Next, the corresponding point search section 32 detects a phase difference found point and a phase difference unknown point using the corresponding point detection result (S
7).

Here, the detected phase difference points and the unknown phase difference points relate to the one region of interest 48, 52 from which the current left and right search image has been read. Also in this case, fitting of the phase difference to the unknown point and verification of the fitted phase difference are performed.

Next, the corresponding point search section 32 detects a phase difference found point in the short distance phase difference range, and outputs the number of detected phase difference found points in the short distance phase difference range to the evaluation value calculation section 34 ( S8).

Next, the evaluation value calculation section 32 obtains the detection rate of the phase difference found point using the number of detected phase difference found points in the short distance phase difference range, and determines the detection rate of the phase difference found point as the maximum value. It is stored in the unit 36 (S9).

Next, the area extracting section 46 determines whether or not the left and right attention area images have been cut out from all the attention areas with respect to the converted left and right images currently stored in the third left and right memories 38 and 40 (whether or not the reading has been completed). Is determined (S
10).

If the extraction of all the regions of interest has not been completed for the left and right images after the current image conversion, the region extracting section 46 selects a set of unextracted regions of interest of the left and right region of interest images (S11). Thereafter, the next left and right region of interest images are cut out from the selected region of interest (S4).

For example, in the example of FIG. 12, the extraction from the regions of interest 48 and 52 with y = 3 is completed, and the extraction from the regions of interest 48 and 52 with y = 6 and 9 is not completed. As the set of target areas 48 and 52, the target areas 48 and 52 of y = 6 are selected.

When the extraction of all the regions of interest has been completed for the left and right images 42 and 44 after the current image conversion, the evaluation value calculation unit 34 determines the height direction candidate conditions from which the left and right images after the current image conversion have been obtained. Then, an average value of the detection rates obtained for each region of interest is obtained, and this average value is stored in the maximum value detection unit 36 as an evaluation value of the height direction candidate condition (S12).

For example, the left and right images 42 after the current image conversion,
In a case where 44 is obtained according to the candidate condition Δy ₁ , in the example of FIG.
The average value of the detection rates obtained every 2 is obtained as the evaluation value of the candidate condition Δy ₁ .

Next, the condition setting section judges whether or not all the height direction candidate conditions have been selected (S13).

If all the height direction candidate conditions have not been selected, the condition setting unit 28 selects the next height direction candidate condition (S14), and then outputs the selected height direction candidate condition. (S2).

For example, only the selection of Δy ₁ among the height direction candidate conditions Δy _{1 to} Δy ₅ has been completed, and Δy ₂ to Δy ₅ have been selected.
If not completed selection for y _5, one of the candidate condition selected from among the remaining candidate condition Δy ₂ ~Δy ₅ Δ
Select y _2, and outputs the candidate condition [Delta] y ₂ selected.

When all the height direction candidate conditions have been selected, the maximum value detector 36 detects the maximum evaluation value from among the evaluation values obtained for each height direction candidate condition, and determines the maximum value. The height direction candidate condition for which the evaluation value has been obtained is detected as a height direction correction condition. Then, the maximum value detecting unit 28 stores the height direction correction condition in the condition setting unit 28 (S15). After that,
The correction condition detecting device 18 ends the processing for detecting the height direction correction condition (end).

In this embodiment, the left and right search images are used as region of interest images which are a part of the left and right images after image conversion. Therefore, the amount of scanning of the image necessary to detect the height direction correction condition is reduced. Therefore, the detection time of the height direction correction condition can be reduced.

<Third Embodiment of the Invention of Claim 9> Next, a third embodiment of the invention of claim 9 will be described with reference to FIG. Hereinafter, points different from the second embodiment of the ninth aspect of the present invention will be mainly described, and detailed description of the same points as those of the second embodiment of the ninth aspect of the present invention will be omitted.

The correction condition detecting device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
An evaluation value calculation unit 34 and a maximum value detection unit 36; a third left memory 38 and a third right memory 40;
6 is provided.

[0406] The evaluation value calculation unit 34 calculates the detection rate of the phase difference known point for each of the left and right search images having the same height direction candidate condition for the short distance phase difference in which the shift amount in the height direction is large in the left and right search images. The desired height direction candidate condition selects a predetermined number of detection rates sequentially from the top among the detection rates obtained for the same left and right search images, and calculates the average value of the detection rates obtained by the selection as the evaluation value of the height direction candidate condition and I do.

FIG. 25 is a diagram showing a flow of the operation of the second embodiment of the present invention. In the figure, focusing on the height direction candidate condition and the correction condition among a plurality of types of candidate conditions and correction conditions, the flow of operation when detecting the height direction correction condition using the height direction candidate condition is shown. Show.

When the operator inputs an operation start signal to the correction condition detecting device 18 (start), the condition setting section 28 first selects and sets a height direction candidate condition from a plurality of types of candidate conditions (S1). Thereafter, the condition setting unit 28 outputs one candidate condition (selected height direction candidate condition) selected from the plurality of height direction candidate conditions (S2).

Next, the image conversion unit 30 converts one of the left image and the right image according to the selected height direction candidate condition input from the condition setting unit 28, and converts the left image and the right image after the image conversion into the third left image. The data is stored in the memory 38 and the third right memory 40 (S3). The left and right images captured by the left and right cameras 10 and 12 are stored in first left and right memories 14 and 16, and the image conversion unit 30 reads the left and right images of the first left and right memories 14 and 16, and one of the read left and right images is an image. Without conversion, the other is image-converted, and the left and right images after the image conversion are stored in the third left and right memories 38 and 40.

[0410] Next, the area cutout section 46 cuts out the left and right attention area images from the plurality of attention areas dispersed in the vertical direction (y direction) on the left and right images after image conversion (S4).

Next, the corresponding point search section 32 sets a plurality of types of search conditions by changing the color of the pixel to be scanned, sets the left and right region of interest images as left and right search images, and sets the left and right search images for each search condition. A corresponding candidate point in the search image is detected (S
5).

[0412] When the detection of the corresponding candidate points on the horizontal scanning line L is completed for the entire left and right search images for all types of search conditions, the corresponding point search unit 32 next calculates the corresponding candidate points obtained for each search condition. Corresponding candidate points that become a common set in the set are detected as corresponding points (S6).

[0413] Next, the corresponding point search section 32 detects a phase difference found point and a phase difference unknown point using the corresponding point detection result (S
7).

Next, the corresponding point searching section 32 detects a phase difference found point in the short distance phase difference range, and outputs the number of detected phase difference found points in the short distance phase difference range to the evaluation value calculation section 34 ( S8).

Next, the evaluation value calculation unit 32 obtains the detection rate of the phase difference found point using the number of detected phase difference found points in the short-distance phase difference range, and determines the detection rate of the phase difference found point as the maximum value. It is stored in the unit 36 (S9).

[0416] Next, the area cutout unit 46 determines whether the left and right attention area images have been cut out from all the attention areas (readout has been completed) for the converted left and right images currently stored in the third left and right memories 38 and 40. Is determined (S
10).

If the extraction of all the target areas has not been completed with respect to the current left and right images after the image conversion, the area extracting section 46 selects a set of unextracted target areas of the left and right target area images (S11). Thereafter, the next left and right region of interest images are cut out from the selected region of interest (S4).

[0418] When the extraction of all the regions of interest has been completed for the left and right images after the current image conversion, the evaluation value calculation unit 3
4 selects the higher detection rate among the detection rates obtained for each region of interest with respect to the height direction candidate conditions for obtaining the current left and right images after image conversion (S12).

[0419] Next, the evaluation value calculation unit 34 calculates the average value of the selected higher detection rates, and uses this average value as the evaluation value for the height direction candidate condition for obtaining the current left and right images after image conversion. It is stored in the value detection unit 36 (S13).

Here, as described with reference to FIG. 12, the attention area 4 to be set on the current image-converted left and right images 42 and 44.
Since the total number of 8, 52 is the same, assuming that the total number is R, the detection rate of the total number R is obtained from the total number R of the regions of interest 48, 52. Then, among the detection rates of the total number R, the detection rates of the top r1 (r1 is a natural number satisfying R>r1> 1) are selected, and the average value of the detection rates of the selected top r1 is used as the evaluation value. The lower (R-r1) detection rates where (R-r1)> 0 are not used for calculating the evaluation value.

For example, a total number R = 30 regions of interest 48,
If 52 are set and the detection rate of the total number R = 30 is obtained from these regions of interest, the evaluation value is obtained using the detection rates of the upper r1 = 10 to 20 and the lower (R-r1) = 20-
The ten detection rates are not used for calculating the evaluation value.

Next, the condition setting section 28 determines whether or not all the height direction candidate conditions have been selected (S14).

If all the height direction candidate conditions have not been selected, the condition setting unit 28 selects the next height direction candidate condition (S15), and then outputs the selected height direction candidate condition. (S2).

When all the height direction candidate conditions have been selected, the maximum value detecting section 36 detects the maximum evaluation value from the evaluation values obtained for each height direction candidate condition, and The height direction candidate condition for which the evaluation value has been obtained is detected as a height direction correction condition. Then, the maximum value detection unit 28 stores the height direction correction condition in the condition setting unit 28 (S16). After that,
The correction condition detecting device 18 ends the processing for detecting the height direction correction condition (end).

In this embodiment, as in the second embodiment of the present invention, the left and right search images are used as region-of-interest images which are a part of the left and right images after image conversion. Since it is possible to reduce the amount of scanning of the image necessary to perform the operation, the time required for detecting the height direction correction condition can be reduced.

Further, according to this embodiment, when the evaluation value of the height direction candidate condition is obtained, the evaluation value is obtained by using the average value of the upper plurality of detection rates, and the lower detection rate is calculated. , The detection accuracy of the height direction correction condition can be improved.

<First Embodiment of the Invention of Claim 13> Next, a first embodiment of the invention of claim 13 will be described with reference to FIG. Hereinafter, points different from the first embodiment of the first aspect of the present invention will be described, and detailed description of the same points as those of the first embodiment of the first aspect of the present invention will be omitted.

The correction condition detecting device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
An evaluation value calculation unit 34 and a maximum value detection unit 36, a third left memory 38 and a third right memory 40, and a phase difference range input unit 74 are further provided.

The condition setting section 28 sequentially sets candidate conditions for detecting the tilt angle correction condition, and the image conversion section 30 sets one of the left image and the right image obtained by the left camera 10 and the right camera 12 as a candidate. Image conversion is performed according to the conditions. The tilt angle is an image rotation angle around the z-axis when the x-axis direction is a horizontal direction, the y-axis direction is a vertical direction (height direction), and the z-axis direction is a depth direction.

In this case, other types of correction conditions including the tilt angle correction condition are also detected, and the condition setting unit 28 matches the epipolar line with a plurality of types of candidate conditions for detecting the correction condition. And a plurality of types of correction conditions. The correction condition to be stored is the correction condition detected by the maximum value detection unit 36. One of these plural types of candidate conditions and correction conditions is a tilt angle candidate condition and a correction condition.

[0431] The condition setting section 28 stores candidate conditions relating to parallel translation, rotation, and magnification conversion of an image. Therefore, the correction conditions detected by the maximum value detection unit 36 are correction conditions relating to the parallel movement, rotation, and magnification conversion of the image.
The image conversion performed by 0 according to the candidate condition and the correction condition is translation, rotation and magnification conversion of the image. Known methods can be used for the translation, rotation, and magnification conversion.

The corresponding point search unit 32 sets a plurality of types of search conditions by changing the scanning direction, the gradation expression of the pixel to be scanned, or the color of the pixel to be scanned, and sets the left and right after image conversion. Let the image be a left and right search image. Here, left and right images of a color image are obtained by the left and right cameras 10 and 12, and a plurality of types of search conditions are set by changing colors of pixels to be scanned. The left and right images after the image conversion are set as the left and right search images over the entire surface.

The corresponding point search section 32 scans the left and right search images according to the search conditions, compares the pixel characteristics of the left search image and the pixel characteristics of the right search image on the corresponding horizontal scanning lines, and determines the difference between the pixel characteristics. The pixels of the left and right search images for which is less than or equal to the threshold are detected as corresponding candidate points. Further, the corresponding point search unit 32 detects, as corresponding points, corresponding candidate points that become a common set in the set of corresponding candidate points detected for each search condition. Here, the pixel feature is a pixel density.

For example, a color image is stored in the third left memory 38 and the third right memory 40 as the left and right images after the image conversion, and the red image among the left and right images after the image conversion is used as the left and right search image. A search condition and a second search condition for setting the blue image among the left and right images after the image conversion as the left and right search image are set. Then, the left and right search images of the red image are scanned according to the first search condition, and the pixels of the left and right search image in which the difference of the pixel characteristics is equal to or less than the threshold on the corresponding horizontal scanning line are identified as the corresponding candidate points according to the first search condition Detected as
Further, the left and right search images of the blue image are scanned according to the second search condition, and the pixels of the left and right search images whose pixel feature difference is equal to or smaller than the threshold on the corresponding horizontal scanning line are identified as corresponding candidate points according to the second search condition. Detected as Thereafter, a corresponding candidate point that is a common set between the set of corresponding candidate points detected under the first search condition and the set of corresponding candidate points detected under the second search condition is detected as a corresponding point.

Next, the corresponding point search section 32 detects the clear and unknown points of the long-distance phase difference where the amount of deviation of the tilt angle becomes large in the left and right search images using the corresponding point detection results. The operator sets the range of the long-distance phase difference in which the amount of displacement in the height direction is large via the phase difference range input unit 74 to the corresponding point search unit 3.
Enter 2 For example, the range where the phase difference is less than 80 is
It is input as a range of a long-distance phase difference in which the amount of tilt angle deviation is large.

Here, based on the phase difference between the corresponding points, the phase difference is estimated and applied to the unknown correspondence point (the pixel not detected as the corresponding point). The corresponding unknown points to which the phase difference could not be applied are detected as the phase difference unknown points. Then, regarding the phase difference applied to the unknown point,
Test whether the phase difference is correct. Then, the unknown point of the correspondence of the phase difference determined to be correct by this test is detected as a known point of the phase difference, and the unknown point of the correspondence of the phase difference determined to be incorrect is detected as the unknown point of the phase difference.

Since a plurality of types of search conditions are set by making the color of the pixel to be scanned different, the fitting of the phase difference and the test of the phase difference are performed for each color of the pixel to be scanned. Only pixels for which the phase difference has been determined to be correct for all colors of the pixel to be scanned are regarded as phase difference determination points, and pixels for which the phase difference has been determined to be incorrect for any of the colors of the pixel to be scanned. Is an unknown phase difference point. Alternatively, a pixel whose phase difference is determined to be incorrect for all colors of the pixel to be scanned is determined to be a phase difference unknown point. Then, in a case where a color whose phase difference is determined to be correct and a color whose phase difference is determined to be incorrect are mixed in a pixel to be scanned, it is determined whether or not the phase difference is correct according to the principle of majority decision. Whether the phase difference is known or unknown may be determined according to the determination result.

Thereafter, from the phase difference known points, the long distance phase difference known point where the tilt angle shift amount becomes large is detected.

[0439] The evaluation value calculation unit 34 calculates the detection rate of the phase difference known point for the long-distance phase difference in which the tilt angle shift amount is large in the left and right search images, and uses the detection rate to determine the tilt angle candidate condition. Find the evaluation value. Here, the detection rate of the phase difference known point is obtained for each of the tilt angle candidate conditions for the long-distance phase difference in which the tilt angle shift amount is large in the left and right search images, and the detection rate is used as the evaluation value of the tilt angle candidate condition.

The maximum value detection section 36 detects the tilt angle candidate condition that has obtained the maximum evaluation value as the tilt angle correction condition.
Here, the maximum value detection unit 36 stores the detected tilt angle correction condition in the condition setting unit 28.

FIG. 26 is a diagram showing a flow of the operation of the second embodiment of the present invention. FIG. 5 shows a flow of an operation when the tilt angle candidate condition and the correction condition are detected from the plurality of types of candidate condition and correction condition, and the tilt angle correction condition is detected using the tilt angle candidate condition.

When the operator inputs an operation start signal to the correction condition detecting device 18 (start), the condition setting section 28 first selects and sets a tilt angle candidate condition from among a plurality of types of candidate conditions (S1). Thereafter, the condition setting unit 28 outputs one tilt angle candidate condition (selected tilt angle candidate condition) selected from the plurality of tilt angle candidate conditions (S).
2). The tilt angle candidate conditions are candidate conditions relating to rotation about the z-axis. For example, five tilt angle candidate conditions Δγ ₁ to
One tilt angle candidate condition Δγ selected from Δγ ₅
Outputs ₁ .

[0443] Next, the image conversion unit 30 includes the left and right cameras 10,
One of the left and right images taken by the
The left and right images after the image conversion are converted according to the selected tilt angle candidate conditions input from the third left and right memories 38 and 40.
(S3). The left and right images captured by the left and right cameras 10 and 12 are stored in first left and right memories 14 and 16, and the image conversion unit 30 reads the left and right images of the first left and right memories 14 and 16, and one of the read left and right images is The left and right images after the image conversion are stored in the third left and right memories 38 and 40 without performing the image conversion.

Next, the corresponding point search section 32 sets a plurality of types of search conditions by changing the color of the pixel to be scanned, and sets the entire left and right image after image conversion as a left and right search image. Next, corresponding candidate points in the left and right search images are detected (S4). Here, since a plurality of types of search conditions are set by changing the colors of the pixels, the left and right images taken by the left and right cameras 10 and 12 are set as color images.

For example, of the left and right images of the converted color image stored in the third left and right memories 38 and 40, a first search condition for scanning a red right and left image as a left and right search image;
A second search condition for scanning the left and right blue images as the left and right search images is set. Then, the corresponding point search unit 32 scans the red left and right images according to the first search condition, cuts out the corresponding red left and right images on the horizontal scanning line L from the third left and right memories 38 and 40, and The data is stored in the left and right line memories provided in the point search unit 32. Here, in the left and right red images, one pixel is set as a point of interest and the other pixel is set as a reference point, and the i-th point of interest is sequentially counted in a predetermined main scanning direction, for example, rightward on the corresponding horizontal scanning line L. With respect to the point, a control point on the horizontal scanning line L is sequentially scanned in a predetermined main scanning direction, for example, rightward. When the scanning of the control point is completed without detecting the control point at which the pixel characteristic, for example, the difference of the pixel density is equal to or less than the threshold value, with respect to the i-th point of interest,
The control point is scanned for the (+1) th point of interest. When a control point at which the difference of the pixel feature with respect to the i-th point of interest is smaller than or equal to the threshold is detected, the i-th point of interest and the control point are detected as corresponding candidate points, and then the (i + 1) -th point of interest is detected. Is scanned for the point of interest. Third left and right memory 38,
When the detection of the corresponding candidate points by the first search condition is completed for the entire 40 right and left images, the blue left and right images are scanned by the second search condition in the same manner as the first search condition.
A corresponding candidate point is detected based on the second search condition.

[0446] When the detection of the corresponding candidate points is completed for all types of search conditions, the corresponding point search unit 32 next associates the corresponding candidate points that become a common set in the set of corresponding candidate points obtained for each search condition. It is detected as a point (S5).

As described above, the left and right search images on the corresponding horizontal scanning line L are scanned, and when the difference between the pixel features becomes equal to or smaller than the threshold value, the next point of interest is scanned to detect a corresponding candidate point. The corresponding candidate points that become a common set in the set of corresponding candidate points obtained for each search condition are a set of corresponding candidate points that are likely to be stochastic.

Next, the corresponding point search section 32 detects a phase difference found point and a phase difference unknown point in the long-distance phase difference range using the corresponding point detection result (S6).

[0449] The detected phase difference points and the unknown phase difference points detected here relate to the entire left and right images that are the current left and right search images.

Here, based on the phase difference between corresponding points, the phase difference is estimated and applied to an unknown correspondence point (a pixel not detected as a corresponding point). The corresponding unknown points to which the phase difference could not be applied are detected as the phase difference unknown points. Thereafter, it is checked whether the phase difference applied to the correspondence unknown point is correct. Then, the unknown point of the correspondence of the phase difference determined to be correct by this test is determined to be a known point of the phase difference, and the unknown point of correspondence of the phase difference determined to be incorrect is detected as the unknown point of the phase difference.

Since a plurality of types of search conditions are set by making the color of the pixel to be scanned different, the fitting of the phase difference and the test of the phase difference are performed for each color of the pixel to be scanned. Only pixels for which the phase difference has been determined to be correct for all colors of the pixel to be scanned are regarded as phase difference determination points, and pixels for which the phase difference has been determined to be incorrect for any of the colors of the pixel to be scanned. Is an unknown phase difference point.

Next, the corresponding point searching section 32 calculates the number of detected phase difference points in the long-distance phase difference range as an evaluation value calculating section 34.
(S7). For example, a range having a phase difference of less than 80 is defined as a long-range phase difference range.

Next, the evaluation value calculation unit 32 obtains the detection rate of the phase difference found point using the number of detected phase difference points in the long-distance phase difference range, and selects the selection rate output by the condition setting unit 28 in (S2). As the evaluation value of the tilt angle candidate condition, the detection rate of the phase difference found point in the long distance phase difference range is set to the maximum value detection unit 3.
6 (S8).

Next, the condition setting section judges whether or not all the tilt angle candidate conditions have been selected (S9).

If all the tilt angle candidate conditions have not been selected, the condition setting section 28 selects the next tilt angle candidate condition (S10), and then outputs the selected tilt angle candidate condition (S2). ).

For example, of the tilt angle candidate conditions Δγ _{1 to} Δγ ₅ , only Δγ ₁ has been selected, and Δγ _{2 to} Δγ ₂ have been selected.
If not completed selection for gamma _5, one of the candidate condition selected from among the remaining candidate condition Δγ ₂ ~Δγ ₅ Δ
Select gamma _2, and outputs the tilt angle candidate condition [Delta] [gamma] ₂ was selected.

If all the tilt angle candidate conditions have not been selected, the maximum value detector 36 detects the maximum evaluation value from among the evaluation values obtained for each of the tilt angle candidate conditions, and determines the maximum evaluation value. The obtained tilt angle candidate condition is detected as a tilt angle correction condition. Then, the maximum value detection unit 28 stores the tilt angle correction condition in the condition setting unit 28 (S11). Thereafter, the correction condition detecting device 18 ends the process for detecting the tilt angle correction condition (end).

According to this embodiment, the detection rate of the phase difference found point in the long-distance phase difference range is used as the evaluation value, and the tilt angle candidate condition that has obtained the maximum evaluation value is used as the tilt angle correction condition. Since the amount of deviation of the tilt angle becomes large in the long-distance phase difference range, the tilt angle correction condition is detected using the detection rate of the phase difference known point in the long-distance phase difference range as an evaluation value, so that the tilt more suitable for epipolar line alignment Angle correction conditions can be detected.

<Second Embodiment of the Invention of Claim 13> Next, a second embodiment of the invention of claim 13 will be described with reference to FIG. Hereinafter, points different from the first embodiment of the invention of claim 13 will be mainly described, and detailed description of the same points as those of the first embodiment of the invention of claim 13 will be omitted.

The correction condition detecting device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
Evaluation value calculation unit 34 and maximum value detection unit 36, third left memory 38 and third right memory 40, and phase difference range input unit 74
And a region cutout section 46.

The area extracting section 46 sequentially extracts left and right attention area images from a plurality of attention areas dispersed in the vertical direction on the left and right images after image conversion, and the corresponding point search section 32 separates the left and right attention area images into right and left images. As search images, corresponding candidate points, corresponding points, known and unknown points of the phase difference are detected for each of the left and right search images having the same candidate condition.

The evaluation value calculation unit calculates the detection rate of the phase difference known point for each of the left and right search images having the same tilt angle candidate condition with respect to the long distance phase difference in which the tilt angle shift amount is large in the left and right search images. The average value of the detection rates obtained for the left and right search images having the same angle candidate condition is used as the evaluation value of the tilt angle candidate condition.

FIG. 27 is a diagram showing a flow of the operation of the second embodiment of the present invention. FIG. 7 shows a flow of operation in the case where the tilt angle candidate condition and the correction condition are detected from the plurality of types of candidate condition and correction condition, and the tilt angle correction condition is detected using the tilt angle candidate condition.

When the operator inputs an operation start signal to the correction condition detecting device 18 (start), the condition setting section 28 first selects and sets a tilt angle candidate condition from a plurality of types of candidate conditions (S1). After that, the condition setting unit 28 outputs one candidate condition (selected tilt angle candidate condition) selected from the plurality of tilt angle candidate conditions (S2).

Next, the image conversion unit 30 converts one of the left image and the right image according to the selected tilt angle candidate condition input from the condition setting unit 28, and stores the converted left image and right image in the third left memory. 38 and the third right memory 40 (S3). The left and right images captured by the left and right cameras 10 and 12 are stored in first left and right memories 14 and 16, and the image conversion unit 30 reads the left and right images of the first left and right memories 14 and 16, and one of the read left and right images is an image. Without conversion, the other is image-converted, and the left and right images after the image conversion are stored in the third left and right memories 38 and 40.

[0466] Next, the area cutout section 46 cuts out the left and right attention area images from the plurality of attention areas dispersed in the vertical direction (y direction) on the left and right images after image conversion (S4).

For example, as shown in FIG. 12, the width in the vertical direction of the left and right region of interest image is the width of one horizontal scanning line or the width of a plurality of lines, here the width of one line. Cuts out the left and right region of interest images on the corresponding horizontal scanning lines L from the converted left and right images stored in the third left and right memories 38 and 40, and outputs the extracted horizontal scanning lines L
The upper left and right region of interest images are stored in the left and right line memories of the corresponding point search unit 32.

Next, the corresponding point search section 32 sets a plurality of types of search conditions by changing the color of the pixel to be scanned, sets the left and right region of interest images as left and right search images, and sets the left and right search images for each search condition. A corresponding candidate point in the search image is detected (S
5).

When the detection of the corresponding candidate points on the horizontal scanning line L over the entire left and right search image has been completed for all types of search conditions, the corresponding point search unit 32 next calculates the corresponding candidate points obtained for each search condition. Corresponding candidate points that become a common set in the set are detected as corresponding points (S6).

Next, the corresponding point search section 32 detects a phase difference found point and a phase difference unknown point using the corresponding point detection result (S
7).

[0471] The detected phase difference points and the unknown phase difference points detected here relate to one attention area 48, 52 from which the current left and right search image has been read. Also in this case, fitting of the phase difference to the unknown point and verification of the fitted phase difference are performed.

[0472] Next, the corresponding point searching section 32 detects a phase difference found point in the long distance phase difference range, and outputs the number of detected phase difference found points in the long distance phase difference range to the evaluation value calculation section 34 ( S8).

[0473] Next, the evaluation value calculation section 32 obtains the detection rate of the phase difference found point using the number of detected phase difference found points in the long-distance phase difference range, and determines the detection rate of the phase difference found point to the maximum value. It is stored in the unit 36 (S9).

[0474] Next, the area cutout unit 46 determines whether the left and right attention area images have been cut out from all the attention areas for the currently converted left and right images currently stored in the third left and right memories 38 and 40 (whether or not reading has been completed). Is determined (S
10).

If the extraction of all the target areas has not been completed with respect to the current left and right images after the image conversion, the area cutout unit 46 selects a set of not-yet-completed target areas of the left and right target area images (S11). Thereafter, the next left and right region of interest images are cut out from the selected region of interest (S4).

For example, in the example of FIG. 12, the extraction from the regions of interest 48 and 52 with y = 3 has been completed, and the extraction from the regions of interest 48 and 52 with y = 6 and 9 has not been completed. As the set of target areas 48 and 52, the target areas 48 and 52 of y = 6 are selected.

When the extraction of all the regions of interest has been completed for the left and right images 42 and 44 after the current image conversion, the evaluation value calculation unit 34 calculates the tilt angle candidate conditions for obtaining the current left and right images after the image conversion. An average value of the detection rates obtained for each region of interest is obtained, and this average value is stored in the maximum value detection unit 36 as an evaluation value of the tilt angle candidate condition (S12).

For example, the left and right images 42 after the current image conversion,
In a case where 44 is obtained according to the candidate condition Δy ₁ , in the example of FIG.
The average value of the detection rates obtained every 2 is obtained as the evaluation value of the candidate condition Δy ₁ .

Next, the condition setting section judges whether or not all the tilt angle candidate conditions have been selected (S13).

If all the tilt angle candidate conditions have not been selected, the condition setting section 28 selects the next tilt angle candidate condition (S14), and then outputs the selected tilt angle candidate condition (S2). ).

For example, only Δγ ₁ among the tilt angle candidate conditions Δγ _{1 to} Δγ ₅ has been selected, and Δγ _{2 to} Δγ ₂ have been selected.
If not completed selection for gamma _5, one of the candidate condition selected from among the remaining candidate condition Δγ ₂ ~Δγ ₅ Δ
γ ₂ is selected, and the selected candidate condition Δγ ₂ is output.

When all the tilt angle candidate conditions have been selected, the maximum value detector 36 detects the maximum evaluation value from among the evaluation values obtained for each tilt angle candidate condition, and determines the maximum evaluation value. The obtained tilt angle candidate condition is detected as a tilt angle correction condition. Then, the maximum value detection unit 28 stores the tilt angle correction condition in the condition setting unit 28 (S15). After that,
The correction condition detection device 18 ends the process for detecting the tilt angle correction condition (end).

In this embodiment, the left and right search images are used as region of interest images which are a part of the left and right images after image conversion. Therefore, the amount of scanning of the image necessary to detect the tilt angle correction condition is reduced. Therefore, the detection time of the tilt angle correction condition can be reduced.

<Third Embodiment of the 13th Invention> Next, a third embodiment of the 13th invention will be described with reference to FIG. Hereinafter, points different from the second embodiment of the thirteenth aspect of the invention will be mainly described, and detailed description of the same points as those of the second embodiment of the thirteenth aspect will be omitted.

The correction condition detecting device 18 of this embodiment
A condition setting unit 28, an image conversion unit 30, a corresponding point search unit 32,
An evaluation value calculation unit 34 and a maximum value detection unit 36; a third left memory 38 and a third right memory 40;
6 is provided.

The evaluation value calculation unit calculates the detection rate of the phase difference clear point for each of the left and right search images having the same tilt angle candidate condition with respect to the long distance phase difference in which the tilt angle shift amount is large in the left and right search images. A predetermined number of detection rates are sequentially selected from the top in the detection rates obtained for the left and right search images having the same corner candidate condition, and the average value of the detection rates obtained by the selection is used as the evaluation value of the tilt angle candidate condition.

FIG. 28 is a diagram showing a flow of the operation of the second embodiment of the present invention. FIG. 7 shows a flow of operation in the case where the tilt angle candidate condition and the correction condition are detected from the plurality of types of candidate condition and correction condition, and the tilt angle correction condition is detected using the tilt angle candidate condition.

When the operator inputs an operation start signal to the correction condition detecting device 18 (start), the condition setting section 28 first selects and sets a tilt angle candidate condition from a plurality of types of candidate conditions (S1). After that, the condition setting unit 28 outputs one candidate condition (selected tilt angle candidate condition) selected from the plurality of tilt angle candidate conditions (S2).

[0489] Next, the image conversion section 30 converts one of the left image and the right image according to the selected tilt angle candidate condition input from the condition setting section 28, and stores the converted left image and right image in the third left memory. 38 and the third right memory 40 (S3). The left and right images captured by the left and right cameras 10 and 12 are stored in first left and right memories 14 and 16, and the image conversion unit 30 reads the left and right images of the first left and right memories 14 and 16, and one of the read left and right images is an image. Without conversion, the other is image-converted, and the left and right images after the image conversion are stored in the third left and right memories 38 and 40.

[0490] Next, the region cutout unit 46 cuts out the left and right region of interest images from a plurality of regions of interest dispersed in the vertical direction (y direction) on the left and right images after image conversion (S4).

Next, the corresponding point search unit 32 sets a plurality of types of search conditions by changing the color of the pixel to be scanned, sets the left and right region of interest images as left and right search images, and sets the left and right search images for each search condition. A corresponding candidate point in the search image is detected (S
5).

After the detection of the corresponding candidate points on the horizontal scanning line L over the entire left and right search images for all types of search conditions, the corresponding point search unit 32 next calculates the corresponding candidate points obtained for each search condition. Corresponding candidate points that become a common set in the set are detected as corresponding points (S6).

Next, the corresponding point search unit 32 detects a phase difference found point and a phase difference unknown point using the corresponding point detection result (S
7).

Next, the corresponding point search section 32 detects a phase difference found point in the long distance phase difference range, and outputs the number of detected phase difference found points in the long distance phase difference range to the evaluation value calculation section 34 ( S8).

Next, the evaluation value calculation section 32 obtains the detection rate of the phase difference found point using the number of detected phase difference found points in the long distance phase difference range, and determines the detection rate of the phase difference found point as the maximum value. It is stored in the unit 36 (S9).

[0496] Next, the area cutout unit 46 determines whether or not the left and right attention area images have been cut out from all the attention areas (readout has been completed) for the converted left and right images currently stored in the third left and right memories 38 and 40. Is determined (S
10).

If the extraction of all the regions of interest has not been completed with respect to the current left and right images after image conversion, the region extracting section 46 selects a set of unextracted regions of interest of the left and right region of interest images (S11). Thereafter, the next left and right region of interest images are cut out from the selected region of interest (S4).

When the extraction of all the regions of interest has been completed for the left and right images after the current image conversion, the evaluation value calculation unit 3
4 selects a higher detection rate from among the detection rates obtained for each region of interest with respect to the tilt angle candidate conditions for obtaining the current left and right images after image conversion (S12).

[0499] Next, the evaluation value calculation unit 34 calculates the average value of the selected higher detection rates, and uses this average value as the evaluation value for the tilt angle candidate condition that obtained the current left and right images after image conversion. It is stored in the detection unit 36 (S13).

Here, as described with reference to FIG. 12, the region of interest 4 to be set on the current image-converted left and right images 42 and 44.
Since the total number of 8, 52 is the same, assuming that the total number is R, the detection rate of the total number R is obtained from the total number R of the regions of interest 48, 52. Then, among the detection rates of the total number R, the detection rates of the top r1 (r1 is a natural number satisfying R>r1> 1) are selected, and the average value of the detection rates of the selected top r1 is used as the evaluation value. The lower (R-r1) detection rates where (R-r1)> 0 are not used for calculating the evaluation value.

For example, a total number R = 30 attention areas 48,
If 52 are set and the detection rate of the total number R = 30 is obtained from these regions of interest, the evaluation value is obtained using the detection rates of the upper r1 = 10 to 20 and the lower (R-r1) = 20-
The ten detection rates are not used for calculating the evaluation value.

Next, the condition setting section judges whether or not all the tilt angle candidate conditions have been selected (S14).

If all the tilt angle candidate conditions have not been selected, the condition setting section 28 selects the next tilt angle candidate condition (S15), and then outputs the selected tilt angle candidate condition (S2). ).

When all the tilt angle candidate conditions have been selected, the maximum value detector 36 detects the maximum evaluation value from among the evaluation values obtained for each tilt angle candidate condition, and determines the maximum evaluation value. The obtained tilt angle candidate condition is detected as a tilt angle correction condition. Then, the maximum value detection unit 28 stores the tilt angle correction condition in the condition setting unit 28 (S16). After that,
The correction condition detection device 18 ends the process for detecting the tilt angle correction condition (end).

Also in this embodiment, similarly to the second embodiment of the present invention, the left and right search images are used as region of interest images which are a part of the left and right images after image conversion, and therefore, the tilt angle correction condition is detected. Therefore, the amount of scanning of the image required for this can be reduced, so that the detection time of the tilt angle correction condition can be reduced.

Further, according to this embodiment, when the evaluation value of the tilt angle candidate condition is obtained, the evaluation value is obtained by using the average value of the upper plurality of detection rates, and the lower detection rate is used for calculating the evaluation value. Since it is not used, the detection accuracy of the tilt angle correction condition can be improved.

[0507]

As is apparent from the above description, according to the first aspect of the present invention, a plurality of candidate conditions are prepared for each type, a plurality of types of candidate conditions are prepared, and obtained by the left and right cameras. One of the left and right images is image-converted in accordance with the candidate condition, and a known point and an unknown point of the phase difference are detected from the left and right images after the image conversion. Then, for each candidate condition, an evaluation value of the candidate condition is obtained using the detection rate of the phase difference found point in the left and right images after the image conversion, and the candidate condition having the maximum evaluation value is used to match the epipolar line. Detected as a correction condition.

Further, when detecting a point where the phase difference is known or unknown, a plurality of types of search conditions are set, and the left and right images after image conversion are scanned in accordance with the search conditions. Are detected as corresponding candidate points. Then, a corresponding candidate point that is a common set in a set of corresponding candidate points detected for each search condition is detected as a corresponding point, and a known point and an unknown point of the phase difference are detected using the corresponding point detection result.

Therefore, according to the first aspect of the present invention, even if a specific test pattern does not exist in the left and right images photographed by the left and right cameras, it is possible to detect the correction condition used for adjusting the epipolar line.

According to the fifth aspect of the present invention, a plurality of candidate conditions are prepared for each type, a plurality of types of candidate conditions are prepared, and one of the left and right images obtained by the left and right cameras is converted into an image according to the candidate conditions. Then, the known and unknown points of the phase difference are detected from the left and right images after the image conversion. Then, for each candidate condition, an evaluation value of the candidate condition is obtained using the detection rate of the phase difference found point in the left and right images after the image conversion, and the candidate condition having the maximum evaluation value is used to match the epipolar line. Detected as a correction condition.

[0511] Further, when detecting a point where the phase difference is known or unknown, a plurality of types of search conditions are set, and the left and right images after image conversion are scanned in accordance with the search conditions. Are detected as corresponding candidate points. Then, a corresponding candidate point that is a common set in a set of corresponding candidate points detected for each search condition is detected as a corresponding point, and a known point and an unknown point of the phase difference are detected using the corresponding point detection result.

[0512] Therefore, according to the invention of claim 5, even if a specific test pattern does not exist in the left and right images photographed by the left and right cameras, it is possible to detect the correction condition used for adjusting the epipolar line.

According to the invention of claim 5, the left and right images after image conversion are converted from m gradations to n gradations, and one or more of the left and right images of n gradations after the image conversion are focused on. Since the correction condition is detected by scanning only the image of the various gradation levels, there is an advantage that the time required for detecting the correction condition can be reduced.

According to the ninth aspect of the present invention, a plurality of height direction candidate conditions are prepared, and one of the left and right images obtained by the left and right cameras is image-converted in accordance with the height direction candidate condition. From the later left and right images, the known and unknown points of the phase difference are detected. Then, for each candidate condition, the evaluation value of the height direction candidate condition is obtained using the detection rate of the phase difference known point in the short distance phase difference range where the amount of displacement in the height direction is large, and the maximum evaluation value is obtained. The height direction candidate condition is detected as a height direction correction condition used for matching epipolar lines.

[0515] Further, when detecting a point where the phase difference is known or unknown, a plurality of types of search conditions are set, and the left and right images after image conversion are scanned in accordance with the search conditions. Are detected as corresponding candidate points. Then, a corresponding candidate point that is a common set in a set of corresponding candidate points detected for each search condition is detected as a corresponding point, and a known point and an unknown point of the phase difference are detected using the corresponding point detection result.

Therefore, according to the ninth aspect of the present invention, even if a specific test pattern does not exist in the left and right images captured by the left and right cameras, the height direction correction condition used for adjusting the epipolar line can be detected. .

According to the ninth aspect of the present invention, the evaluation value of the candidate condition in the height direction is obtained by using the detection rate of the phase difference known point in the short distance phase difference range where the amount of displacement in the height direction is large. Then, the height direction candidate condition for which the maximum evaluation value is obtained is detected as the height direction correction condition, so that a height direction correction condition more suitable for epipolar line alignment can be detected.

According to the thirteenth aspect of the present invention, a plurality of tilt angle candidate conditions are prepared, and one of the left and right images obtained by the left and right cameras is image-converted in accordance with the tilt angle candidate conditions. Detected points and unknown points of the phase difference are detected from the left and right images. Then, for each candidate condition, an evaluation value of the tilt angle candidate condition is obtained using the detection rate of the phase difference known point in the short-distance phase difference range where the amount of tilt angle deviation is large, and the tilt that obtains the maximum evaluation value is obtained. The angle candidate condition is detected as a tilt angle correction condition used for matching epipolar lines.

[0519] In addition, when detecting a known point or an unknown point of the phase difference, a plurality of types of search conditions are set, and the left and right images after image conversion are scanned in accordance with the search conditions. Are detected as corresponding candidate points. Then, a corresponding candidate point that is a common set in a set of corresponding candidate points detected for each search condition is detected as a corresponding point, and a known point and an unknown point of the phase difference are detected using the corresponding point detection result.

Therefore, according to the thirteenth aspect, the tilt angle correction condition used for adjusting the epipolar line can be detected even if a specific test pattern does not exist in the left and right images taken by the left and right cameras.

According to the thirteenth aspect of the present invention, the evaluation value of the candidate condition of the tilt angle is obtained by using the detection rate of the phase difference known point in the short-distance phase difference range in which the tilt angle shift amount is large, Since the tilt angle candidate condition that has obtained the maximum evaluation value is detected as the tilt angle correction condition, the tilt angle correction condition more suitable for epipolar line alignment can be detected.

[Brief description of the drawings]

FIG. 1 is a functional block diagram for explaining an embodiment;

FIG. 2 is a diagram showing a flow of an operation focusing on one type of candidate condition.

FIGS. 3A and 3B are diagrams for explaining left and right images stored in a first left and right memory;

FIGS. 4A and 4B are diagrams for explaining left and right images stored in a third left and right memory;

FIG. 5 is a diagram provided for describing correction condition detection.

FIG. 6 is a diagram provided for explaining a phase difference fitting;

FIG. 7 is a diagram showing a flow of a phase difference test process.

FIGS. 8A and 8B are diagrams for explaining test points and reference points.

FIGS. 9A and 9B are diagrams for explaining test points and reference points.

FIG. 10 is a functional block diagram for explaining the embodiment;

FIG. 11 is a diagram showing a flow of operation focusing on one type of candidate condition.

FIGS. 12A and 12B are diagrams for explaining a target area and a target area image;

FIG. 13 is a diagram showing a flow of operation focusing on one type of candidate condition.

FIGS. 14A and 14B are diagrams for describing improvement in detection accuracy of correction conditions.

FIG. 15 is a functional block diagram for explaining the embodiment;

FIG. 16 is a functional block diagram for explaining the embodiment;

FIG. 17 is a functional block diagram for explaining the embodiment;

FIG. 18 is a diagram showing a flow of operation focusing on one type of candidate condition.

FIG. 19 is a diagram showing a flow of operation focusing on one type of candidate condition.

FIG. 20 is a diagram showing a flow of operation focusing on one type of candidate condition.

FIG. 21 is a functional block diagram for explaining the embodiment;

FIG. 22 is a diagram showing a flow of an operation focusing on height direction candidate conditions.

FIG. 23 is a functional block diagram for explaining the embodiment;

FIG. 24 is a diagram showing a flow of operation focusing on height direction candidate conditions.

FIG. 25 is a diagram showing a flow of operation focusing on height direction candidate conditions.

FIG. 26 is a diagram showing a flow of operation focusing on tilt angle candidate conditions.

FIG. 27 is a diagram showing a flow of operation focusing on a tilt angle candidate condition.

FIG. 28 is a diagram showing a flow of operation focusing on a tilt angle candidate condition.

[Explanation of symbols]

 18: Correction condition detection device 28: Condition setting unit 30: Image conversion unit 32: Corresponding point search unit 34: Evaluation value calculation unit 36: Maximum value detection unit 46: Area extraction unit

Continuation of the front page (56) References JP-A-6-195446 (JP, A) JP-A-7-218251 (JP, A) JP-A-60-213192 (JP, A) JP-A-64-62777 (JP, A) , A) JP-A-7-27540 (JP, A) "Camera calibration for robot vision", Journal of the Robotics Society of Japan, April 15, 1992, Vol. 10, No. 2, p. 39-46 (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T 17/40 H04N 13/02 G01B 11/00-11/30 CSDB (Japan Patent Office) JICST file (JOIS)

Claims (16)

(57) [Claims] 1. A correction condition detecting device for detecting correction conditions for matching epipolar lines of left and right images, a condition setting unit for sequentially setting a plurality of types of candidate conditions for detecting correction conditions, An image conversion unit that converts one of the images according to the candidate condition; and a scanning direction, a gradation expression of a pixel to be scanned, or
A plurality of types of search conditions are set by changing the color of the pixel to be scanned, and the left and right images after image conversion are used as left and right search images,
The left and right search images are scanned according to the search condition, and the pixel characteristics of the left and right search images are compared with the pixel characteristics of the left search image and the pixel characteristics of the right search image on the corresponding horizontal scanning lines. Are detected as corresponding candidate points, corresponding candidate points that become a common set in the set of corresponding candidate points detected for each search condition are detected as corresponding points, and the phase difference of the left and right search images is detected using the corresponding point detection results. Using a corresponding point search unit that detects a known point and an unknown point, and a detection rate of a phase difference known point in the left and right search images,
An evaluation value calculation unit that obtains an evaluation value of a candidate condition for each type of candidate condition, and a maximum value detection unit that detects, as a correction condition, a candidate condition that has obtained the maximum evaluation value for each type of candidate condition A stereo image correction condition detecting device characterized by the following. 2. The stereo image correction condition detecting apparatus according to claim 1, wherein the corresponding point searching unit sets the entire left and right image after image conversion as a left and right search image, and the evaluation value calculating unit determines a phase difference in the left and right search image. A stereo image correction condition detection apparatus, wherein a detection rate of a known point is obtained for each candidate condition, and the detection rate is used as an evaluation value of the candidate condition. 3. The stereo image correction condition detecting apparatus according to claim 1, wherein the left and right attention area images are sequentially cut out from a plurality of attention areas vertically dispersed on the left and right images after image conversion. The corresponding point search unit sets the left and right region of interest images as left and right search images, and the evaluation value calculation unit obtains the detection rate of the phase difference found point for each of the left and right search images having the same candidate condition. A stereo image correction condition detecting device, wherein an average value of detection rates obtained for the same left and right search images is used as an evaluation value of the candidate condition. 4. The stereo image correction condition detecting apparatus according to claim 1, wherein left and right regions of interest images are sequentially cut out from a plurality of regions of interest vertically dispersed on the left and right images after image conversion. The corresponding point search unit sets the left and right region of interest images as left and right search images, and the evaluation value calculation unit obtains the detection rate of the phase difference found point for each of the left and right search images having the same candidate condition. A stereo image characterized by selecting a predetermined number of detection rates in order from the top among detection rates obtained for the same left and right search images, and using an average value of the detection rates obtained by the selection as an evaluation value of a candidate condition. Correction condition detection device. 5. A correction condition detecting device for detecting a correction condition for matching epipolar lines of left and right images, a condition setting unit for sequentially setting a plurality of types of candidate conditions for detecting the correction condition, and m gradations An image conversion unit for converting one of the left and right images according to the candidate condition, a grayscale conversion unit for converting the left and right image after the image conversion into a left and right image of n gradations (m> n), A level cutout unit that cuts out one or more types of focused left and right images as left and right focused level images from the left and right images as a left and right focused level image;
A plurality of types of search conditions are set by changing the color of the pixel to be scanned, the left and right focus level images are set as left and right search images, the left and right search images are scanned in accordance with the search conditions, and a left search is performed on a corresponding horizontal scanning line. A set of corresponding candidate points detected for each search condition by comparing the pixel features of the image with the pixel features of the right search image to detect pixels of the left and right search images for which the difference between the pixel features is equal to or less than a threshold as corresponding candidate points. A corresponding point search unit that detects a candidate point corresponding to a common set as a corresponding point in the left and right search images and detects a known point and an unknown point of the phase difference in the left and right search images using the corresponding point detection result; Using the detection rate of known points,
An evaluation value calculation unit that obtains an evaluation value of a candidate condition for each type of candidate condition, and a maximum value detection unit that detects, as a correction condition, a candidate condition that has obtained the maximum evaluation value for each type of candidate condition A stereo image correction condition detecting device characterized by the following. 6. The stereo image correction condition detecting apparatus according to claim 5, wherein the level cutout unit extracts only the left and right images of the one type of gradation level of interest from the left and right images after the gradation conversion. The evaluation value calculation unit obtains, for each candidate condition, a detection rate of a phase difference found point in the left and right search images, and uses the detection rate as an evaluation value of the candidate condition. Detection device. 7. The stereo image correction condition detecting apparatus according to claim 5, wherein the level cutout unit sequentially reads left and right images of a plurality of types of gradation levels of interest from the left and right images after gradation conversion. The image is cut out as an image, and the evaluation value calculation unit obtains the detection rate of the phase difference found point for each of the left and right search images having the same candidate condition, and calculates the average value of the detection rates obtained for the left and right search images having the same candidate condition as the candidate condition. A stereo image correction condition detection device characterized by an evaluation value. 8. The stereo image correction condition detecting apparatus according to claim 5, wherein the level cutout unit sequentially reads left and right images of a plurality of types of focused gradation levels from the left and right images after gradation conversion. The image is cut out as an image, and the evaluation value calculation unit obtains the detection rate of the phase difference found point for each of the left and right search images having the same candidate condition, and sequentially determines the detection rates obtained for the left and right search images having the same candidate condition from the top in order. A stereo image correction condition detecting device, wherein a number of detection rates are selected, and an average value of the detection rates obtained by the selection is used as an evaluation value of a candidate condition. 9. A correction condition detection device for detecting correction conditions for matching epipolar lines of left and right images, a condition setting unit for sequentially setting candidate conditions for detecting height direction correction conditions, An image conversion unit that converts one of the images according to the candidate condition; and a scanning direction, a gradation expression of a pixel to be scanned, or
A plurality of types of search conditions are set by changing the color of the pixel to be scanned, and the left and right images after image conversion are used as left and right search images,
The left and right search images are scanned according to the search condition, and the pixel characteristics of the left and right search images are compared with the pixel characteristics of the left search image and the pixel characteristics of the right search image on the corresponding horizontal scanning lines. Are detected as corresponding candidate points, corresponding candidate points that become a common set in the set of corresponding candidate points detected for each search condition are detected as corresponding points, and the corresponding point detection results are used to detect height directions in the left and right search images. A corresponding point search unit that detects a known point and an unknown point of a short-distance phase difference in which the amount of deviation of the phase difference is large; An evaluation value calculation unit that obtains a detection rate and calculates an evaluation value of a height direction candidate condition using the detection rate, and a maximum value detection that detects a height direction candidate condition that obtains a maximum evaluation value as a height direction correction condition. Comprising a part Fixed condition detection apparatus of the stereo image, wherein the door. 10. The stereo image correction condition detecting apparatus according to claim 9, wherein the corresponding point search unit sets the entire left and right image after image conversion as a left and right search image, and the evaluation value calculation unit sets the height in the left and right search image. A stereo, characterized in that a detection rate of a phase difference known point is obtained for each height direction candidate condition for a short-distance phase difference in which a direction shift amount is large, and the detection rate is used as an evaluation value of the height direction candidate condition. Image correction condition detection device. 11. A stereo image correction condition detecting apparatus according to claim 9, wherein left and right regions of interest images are sequentially cut out from a plurality of regions of interest vertically dispersed on the left and right images after image conversion. The corresponding point searching unit sets the left and right region of interest images as the left and right search images, and the evaluation value calculating unit sets the height direction candidate condition for the short-distance phase difference in which the shift amount in the height direction in the left and right search images becomes large. Is used to determine the detection rate of the phase difference found point for each of the same left and right search images, and the average value of the detection rates obtained for the left and right search images having the same height direction candidate condition is used as the evaluation value of the height direction candidate condition. Correction condition detecting device for stereo images. 12. The stereo image correction condition detecting apparatus according to claim 9, wherein left and right regions of interest images are sequentially cut out from a plurality of regions of interest vertically dispersed on the left and right images after image conversion. The corresponding point searching unit sets the left and right region of interest images as the left and right search images, and the evaluation value calculating unit sets the height direction candidate condition for the short-distance phase difference in which the shift amount in the height direction in the left and right search images becomes large. Is determined for each of the left and right search images with the same phase difference detection point, and a predetermined number of detection rates are selected in order from the highest among the detection rates obtained for the left and right search images having the same height direction candidate condition, and the selection rate is obtained. A correction condition detection apparatus for stereo images, wherein an average value of the detected rates is used as an evaluation value of the height direction candidate condition. 13. A correction condition detecting device for detecting a correction condition for matching epipolar lines of left and right images, a condition setting unit for sequentially setting candidate conditions for detecting a tilt angle correction condition, and one of the left and right images. An image conversion unit that performs image conversion in accordance with the candidate condition;
A plurality of types of search conditions are set by changing the color of the pixel to be scanned, and the left and right images after image conversion are used as left and right search images,
The left and right search images are scanned according to the search condition, and the pixel characteristics of the left and right search images are compared with the pixel characteristics of the left search image and the pixel characteristics of the right search image on the corresponding horizontal scanning lines. Is detected as a corresponding candidate point, a corresponding candidate point that is a common set in a set of corresponding candidate points detected for each search condition is detected as a corresponding point, and the tilt angle of the tilt angle in the left and right search images is detected using the corresponding point detection result. A corresponding point search unit that detects a known point and an unknown point of a long-distance phase difference where the amount of deviation is large, and a detection rate of a phase difference known point for a long-distance phase difference where the deviation amount of the tilt angle is large in the left and right search images An evaluation value calculation unit that obtains an evaluation value of a tilt angle candidate condition using the detection rate, and a maximum value detection unit that detects a tilt angle candidate condition that obtains a maximum evaluation value as a tilt angle correction condition. Become Fixed condition detection apparatus of the stereo image, wherein the door. 14. The stereo image correction condition detecting apparatus according to claim 13, wherein the corresponding point search unit sets the entire left and right image after image conversion as a left and right search image, and the evaluation value calculation unit sets the tilt angle in the left and right search image. A stereo image correction, wherein a detection rate of a phase difference known point is obtained for each of the tilt angle candidate conditions for a long-distance phase difference in which the amount of deviation is large, and the detection rate is used as an evaluation value of the tilt angle candidate condition. Condition detector. 15. The stereo image correction condition detecting apparatus according to claim 13, wherein left and right target area images are sequentially cut out from a plurality of target areas dispersed in the vertical direction on the left and right images after image conversion. The corresponding point search unit sets the left and right region of interest images as the left and right search images, and the evaluation value calculation unit sets the same tilt angle candidate condition for the long-distance phase difference in which the tilt angle shift amount is large in the left and right search images. A stereo image characterized in that a detection rate of a phase difference known point is obtained for each of the left and right search images, and an average value of the detection rates obtained for the left and right search images having the same tilt angle candidate condition is used as an evaluation value of the tilt angle candidate condition. Correction condition detection device. 16. The stereo image correction condition detecting apparatus according to claim 13, wherein a left and right region of interest image is sequentially cut out from a plurality of regions of interest vertically dispersed on the left and right images after image conversion. The corresponding point search unit sets the left and right region of interest images as the left and right search images, and the evaluation value calculation unit sets the same tilt angle candidate condition for the long-distance phase difference in which the tilt angle shift amount is large in the left and right search images. The detection rate of the phase difference found point is determined for each of the left and right search images, and a predetermined number of the detection rates are sequentially selected from the top in the detection rates obtained for the left and right search images having the same tilt angle candidate condition, and the detection rates obtained by the selection are determined. The correction condition detecting apparatus for stereo images, wherein an average value of the correction conditions is used as an evaluation value of the tilt angle candidate condition.