JP2007037011A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of accurately determining abnormality of imaging means. <P>SOLUTION: An object detection apparatus is to calculate a corresponding point on the basis of a plurality of images by a plurality of imaging means 2R, 2L and detect an object on the basis of the calculated corresponding point, and comprises feature point extraction means for extracting a feature point of each image; feature point number measurement means for measuring the number of feature points; corresponding point detection means for detecting feature points existent in common in a plurality of images among feature points as the corresponding points; and corresponding point number measurement means for measuring the number of the corresponding points. The corresponding point detection means retrieves the corresponding points only for a predetermined region of an image, and judges that an optical axis of the imaging means is abnormal when the number of the corresponding points to that of the feature points is less than a predetermined reference. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention calculates corresponding points based on a plurality of images from a plurality of imaging means, detects an object based on the calculated corresponding points, and can detect an abnormality of the imaging means accurately. About.

An image processing apparatus that detects the position of an object using parallax based on a plurality of input images, that is, a pair of images that are usually called stereo images, is known, and is also disclosed in [Patent Document 1] below. . In the one described in [Patent Document 1] below, when a corresponding point (a point indicating the same part in the right image and the left image of the stereo image) is searched by paying attention to a small area on the stereo image acquired by the stereo camera. When the minimum value of the difference value S is equal to or greater than a predetermined value, the small area is removed as noise (referred to as noise in the above publication), and the number of removed noise exceeds a predetermined amount Alarm signal is output as an abnormality of the optical system (stereo camera).
Japanese Patent Laid-Open No. 11-108660

  However, in the one described in [Patent Document 1], when searching for corresponding points of the extracted feature points (points that are characteristic in image processing in the input image), there are few feature portions in the image in an indoor parking lot or the like. Since the number of feature points is reduced and the number of corresponding points is also reduced, there is a possibility that it is determined to be abnormal although it is not abnormal. Further, it cannot be determined whether the reason why the number of corresponding points is small is due to the imaging function of the optical system (for example, disconnection of the video signal line) or the optical axis shift. Accordingly, an object of the present invention is to provide an object detection apparatus capable of accurately determining an abnormality of an imaging unit.

  The image processing apparatus according to claim 1 calculates corresponding points based on a plurality of images from a plurality of imaging units, detects an object based on the calculated corresponding points, and extracts feature points of each image. A feature point extracting unit; a feature point number measuring unit for measuring the number of feature points; a corresponding point detecting unit for detecting a feature point that is common to a plurality of images among the feature points as a corresponding point; Corresponding point number measuring means for measuring the number of points, the corresponding point detecting means searches for a corresponding point only in a predetermined area of the image, and when the number of corresponding points with respect to the number of feature points is less than a predetermined reference, It is characterized by determining that the optical axis is abnormal.

  According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the plurality of imaging units are installed so that the optical axes thereof are parallel, and the corresponding point detection unit is configured as the predetermined region in the parallel direction described above. Is longer than the length in the direction orthogonal to the parallel direction.

  According to a third aspect of the present invention, in the image processing apparatus according to the second aspect, when it is determined that there is an abnormality, the optical axes of the plurality of imaging means are not in a normal positional relationship in the vertical direction in the image. It is characterized by judging.

  The image processing apparatus according to claim 4 calculates corresponding points based on a plurality of images by a plurality of imaging means, detects an object based on the calculated corresponding points, and extracts feature points of each image. Feature point extraction means, corresponding point detection means for detecting feature points that exist in common among a plurality of images as corresponding points, and the detected corresponding points based on the information on the corresponding points are abnormal corresponding points Abnormality corresponding point determining means for determining whether or not, abnormality corresponding point number measuring means for measuring the number of abnormality corresponding points, feature point number measuring means for measuring the number of feature points, or corresponding point number measurement for measuring the number of corresponding points And an optical axis of the imaging means is determined to be abnormal when the number of characteristic points or the number of abnormal corresponding points with respect to the number of corresponding points is equal to or greater than a predetermined reference.

  According to a fifth aspect of the present invention, in the image processing apparatus according to the fourth aspect of the invention, the abnormality-corresponding point number measuring means sets the number of corresponding points having a negative parallax value as the abnormality-corresponding point number. Yes.

  According to a sixth aspect of the present invention, in the image processing apparatus according to the fifth aspect, when it is determined that there is an abnormality, the optical axes of the plurality of imaging means are not in a normal positional relationship in the horizontal direction in the image. It is characterized by judging.

  According to a seventh aspect of the present invention, in the image processing apparatus according to the fourth aspect of the present invention, the image processing apparatus further comprises a height calculating means for calculating the height of the detected corresponding point in the three-dimensional coordinate system, The feature is that the number of corresponding points having a negative value is set as the number of abnormal corresponding points.

  According to an eighth aspect of the present invention, in the image processing apparatus according to the seventh aspect, when it is determined that there is an abnormality, the optical axes of the plurality of imaging means are not in a normal positional relationship in the vertical direction in the image. It is characterized by judging.

  According to a ninth aspect of the present invention, in the image processing apparatus according to any one of the first to eighth aspects, the feature point is an edge point. An edge point is a point (location: position) where the rate of change in brightness of an image is locally maximum (maximum value).

  Considering that “feature point detection = normal function of imaging means (feature point cannot be detected due to broken signal line)” and “detection of corresponding point = normal camera optical axis position”, Since the state of the optical axis position of the imaging means is determined only when it can be determined that the function of the imaging means is functioning normally, the image processing apparatus according to claim 1, It is possible to reliably detect the positional deviation (direction deviation) of the optical axis. Since the corresponding point is detected only for the predetermined region, the corresponding point cannot be determined if the optical axis is shifted (because the position of the predetermined region corresponding to the plurality of input images is shifted).

  Here, a plurality of imaging means are installed so that their optical axes are parallel, and a predetermined area in which the length of the predetermined area in the parallel direction is longer than the length in the direction orthogonal to the parallel direction is detected as a corresponding point. Preferably the means searches. Since parallax occurs in the parallel direction, the search range in the parallel direction is lengthened, and a rectangular search range in which the length in the direction orthogonal to the parallel direction that is not deviated theoretically is shortened. Here, when it is determined that there is an abnormality when searching for corresponding points only in the horizontal direction, it is determined that there is no normal positional relationship between the plurality of imaging units in the vertical direction. It is possible to accurately detect the positional deviation of the means in the vertical direction.

  Further, according to the image processing apparatus of claim 4, the function of the imaging means is considered as “detection of feature points = normal function of the imaging means” and “detection of corresponding points = normal camera optical axis position”. Only when it can be determined that the optical device is functioning normally, an abnormality in the optical axis position of the imaging means can be determined. However, since there is an error in detecting the corresponding points, the corresponding point is set to be equal to or greater than a predetermined reference greater than zero.

  Here, when the direction deviation occurs so that the optical axis directions of a plurality of imaging means intersect, since the parallax becomes a negative value that cannot normally be taken, imaging is performed according to the ratio of corresponding points having negative parallax. An abnormality of the means can be determined. The parallax is zero when the left and right images are at the same position, and the possible parallax direction is determined to be positive according to the positional relationship of the cameras. Since the positive / negative of the parallax is determined in this way, minus parallax usually cannot occur if the camera position is normal. Further, when the optical axis direction of the plurality of imaging units is directed upward from the normal time, the ratio of the corresponding points in the three-dimensional coordinate system (real space) to take a negative value increases. (If the front of the image pickup means is flat, a negative value is usually not possible). For this reason, it is possible to determine the abnormality of the imaging means based on the ratio of corresponding points having a negative height value. The height is determined to be zero at a predetermined level (usually the ground level), and positive above that (that is, not on the ground but on the ground). Since the height is determined in this way, if the camera position is normal, a minus height cannot normally be generated.

  Hereinafter, an embodiment of an image processing apparatus of the present invention will be described with reference to the drawings. The object detection apparatus of this embodiment is mounted on a vehicle 1 as shown in FIG. The object detection apparatus includes an image acquisition unit (imaging unit) 2R and 2L and an image processing unit 3 (feature point extraction unit / feature point measurement unit / corresponding point detection unit / corresponding unit) that processes the acquired image by applying various filters. Point measuring means, abnormality corresponding point determining means, abnormality corresponding point measuring means). The image acquisition unit is a pair of CCD cameras 2R and 2L arranged at a constant interval in the horizontal direction. The image processing unit performs various calculations based on a pair of input images acquired by a CCD camera, and is an ECU 3 that includes a CPU, GPU, ROM / RAM, and the like.

  The pair of CCD cameras 2 </ b> R and 2 </ b> L are embedded in front of the room mirror in the vehicle 1. The pair of CCD cameras 2R and 2L have the same performance and specifications, and their installation interval, focal length, and the like are stored in advance in a ROM or the like in the ECU 3. If the optical axes of the pair of CCD cameras 2R and 2L are normal, they are arranged parallel to the road surface when the vehicle 1 is placed on a flat road. Further, the optical axes of the pair of CCD cameras 2R and 2L are parallel to each other in a normal state and also to the center line in the front-rear direction of the vehicle 1.

  When detecting an object, first, a front image is acquired by the pair of cameras 2R and 2L. Since the pair of cameras 2R and 2L are installed at a predetermined interval, the pair of captured images are not exactly the same image, and a shift corresponding to a so-called parallax with the naked eye (see FIG. Hereinafter, this deviation is also referred to as parallax). That is, the parallax regarding the points indicating the same in the two images (hereinafter, this pair of points is referred to as the corresponding points) differs depending on the direction and distance from the cameras 2R and 2L. Therefore, the coordinates in the actual three-dimensional space (the corresponding three-dimensional coordinate axis) can be calculated from the position on the image (coordinates on the two-dimensional coordinate axis: usually one of the images as a reference) and the parallax. It becomes.

  A pair of corresponding points are searched from the input image. In this case, a pair of corresponding points is searched using a point (feature point) having a feature in the input image. In this embodiment, an edge point is used as this feature point. An edge point is a point (location: position) where the rate of change in brightness of an image is locally maximum (maximum value). Edge points are easy to detect in image processing, which is convenient for searching for corresponding points.

  FIG. 2 shows a flowchart of abnormality determination control of the object detection apparatus of the present embodiment. Hereinafter, the abnormality determination control will be described with reference to FIG. First, a stereo image is acquired using the CCD cameras 2R and 2L (step 200). Next, edge points (feature points) are extracted from the pair of stereo images, corresponding points are searched for each edge point, and parallax is calculated for each corresponding point (step 205). Normally, about 50% of the corresponding points are searched for the edge points in the image. However, if the CCD cameras 2R and 2L are not set correctly (the optical axis position is deviated), the corresponding points are difficult to calculate, and this ratio decreases.

  When the optical axes of the two cameras 2L and 2R are set parallel to each other as in the present invention, the corresponding point is perpendicular to the image (= Y axis: orthogonal to the parallel direction) if there is no abnormality in the optical axis. In the horizontal direction of the image (= X axis: a direction parallel to the parallel direction) with a shift (parallax). Therefore, in this embodiment, the length in the vertical direction of the predetermined area for searching for the corresponding point is set sufficiently smaller than the length in the horizontal direction. Specifically, a rectangle having a long side in the horizontal direction is set as the search direction. At this time, if the two cameras 2L and 2R are not in a normal position in the vertical direction, the search area in the vertical direction is small, and corresponding points are difficult to detect. In the present invention, when this is utilized, the vertical length of a predetermined area for searching for corresponding points of a plurality of imaging units installed in parallel is set smaller than the horizontal direction and sufficient corresponding points cannot be detected. It is determined that the optical axes of the two cameras 2L and 2R are not in a normal positional relationship in the vertical direction of the image.

  At this time, for the abnormality determination control, the search for the corresponding point (edge point) is performed only for a predetermined search region (predetermined region), for example. If the horizontal optical axes of the pair of CCD cameras 2R and 2L are not parallel, searching for corresponding points in the entire input image can search for corresponding points even if the parallax is larger (or smaller) than usual. End up. Therefore, the area for searching for corresponding points (edge points) is limited to a predetermined area. Note that the position and area of the predetermined area may be changed according to the input image. For example, a predetermined range may be set aiming at a range with many edge points according to the input image.

  The parallax is calculated as follows. One image of the stereo sitting image is binarized and a first object image is extracted. Next, a predetermined area (search area) for searching for an image (corresponding image) corresponding to the first object image is set with respect to the other image (grayscale image). By performing a correlation operation with the object image, a corresponding image corresponding to the first target image is extracted as a second target image. Then, the parallax is determined by the number of pixels or the like from the difference between the centroid position of the first target image of one image and the centroid position of the second target image of the other image.

  FIG. 3 shows a diagram of parallax calculation. Here, a case where a corresponding point is searched for in the left image on the basis of the right image will be described as an example. A small region having a high correlation (high similarity) with a small region of interest in the right image is searched from a predetermined region of the left image. First, in the left image, a search is made in the right direction (X−2 to X + 128) with reference to the template coordinates (X) of the right image. Here, the Y coordinate direction is searched for only one pixel (coordinate Y). By searching in this way, as shown on the right side of FIG. 3, corresponding points can be searched for in the left image, and the parallax shown in the drawing at this time is obtained.

  At this time, if the X coordinate at which the corresponding point is searched is X-2 or X-1, it means negative parallax. In order to evaluate the deviation of the cameras 2R and 2L, minus parallax is also calculated. As described above, when searching for the left image with reference to the right image, the search range is set to be large in the direction (right direction) in which the normal parallax occurs. When searching for the right image with reference to the left image, the search range may be (X + 2 to X-128): left direction / Y coordinate may be one pixel.

  In the case described above, the search range is one pixel in the Y coordinate direction, but a search may be made for a plurality of pixels. For example, when the width of the search range in the Y-axis direction is 3 pixels, first, (X-2 to X + 128) is searched in the left image with reference to the template coordinates (X, Y-1) of the right image. Next, (X−2 to X + 128) is searched in the left image with reference to the template coordinates (X, Y) of the right image, and further, with reference to the template coordinates (X, Y + 1) of the right image, What is necessary is just to search (X-2 to X + 128) in the left image. At this time, a range having (X−2, Y−1), (X + 128, Y−1), (X−2, Y + 1), and (X + 128, Y + 1) as vertices is a predetermined region described in the claims.

  After step 205, the number of edge points is measured, and it is determined whether or not this value is smaller than a predetermined threshold value 0 (step 207). If step 207 is positive, it is determined that an abnormality has occurred in the CCD cameras 2R and 2L (step 225). When there is an abnormality in the camera function itself, such as when the signal lines of the CCD cameras 2R and 2L are disconnected, it is difficult to detect the edge point itself, and the number of edge points described above is smaller than the threshold value 0. An abnormality of the camera function itself can be detected (this process is not the essence of the invention and may be omitted).

  Next, if step 207 is negative, (corresponding point number / edge point number)... Is calculated, and it is determined whether or not this value is smaller than a predetermined first threshold value 1 (step 210). . If step 210 is positive, it is determined that an abnormality has occurred in the CCD cameras 2R and 2L (step 225). As described above, the optical axis positions of the pair of CCD cameras 2R and 2L are not normal positions, and corresponding points are difficult to detect. Therefore, since the value of ☆ described above is smaller than the threshold value 1, it is possible to detect abnormality of the optical axes of the CCD cameras 2R and 2L. Since the corresponding points are searched for at (almost) the same height in the image, it is determined here that there is a deviation in the vertical (vertical direction) in the left and right stereo images if abnormal.

  On the other hand, if step 210 is negative, then (corresponding point of minus parallax / number of edge points) is calculated, and it is determined whether this value is equal to or greater than a predetermined second threshold 2 (step 215). . As described above, normally, the parallax does not take a negative value. (However, in reality, a negative parallax of about 3% of the total parallax calculated may be calculated due to a stereo matching calculation error or the like, which is a normal range.) Step 215 seems to be affirmed. If there is, it is determined that an abnormality has occurred in the CCD cameras 2R and 2L (step 225). If step 215 is affirmed, it can be determined that the optical axes of the pair of CCD cameras 2R and 2L cross each other (the optical axis is not in a normal position in the horizontal direction), so an abnormality is detected. it can.

  If step 215 is negative, next, (corresponding point of minus height / number of edge points) is calculated, and it is determined whether this value is equal to or greater than a predetermined third threshold 3 (step 220). ). As described above, normally, the height of the corresponding point in the three-dimensional coordinate system (real space) does not take a negative value. (However, if the front of the vehicle 1 is a downhill, a negative height may be calculated. A sensor for measuring the level of the vehicle 1 is mounted, and step 220 is executed only when the vehicle 1 is level. (If step 220 is affirmed, it is determined that an abnormality has occurred in the CCD cameras 2R and 2L (step 225)). When step 220 is affirmed, it can be determined that the optical axes of the pair of CCD cameras 2R and 2L are shifted so as to face upward (the optical axis is not in a normal position in the vertical direction). Anomalies can be detected. If step 220 is negative, it is determined that there is no abnormality (step 230). Note that the order of comparison with each threshold value is not limited to the embodiment, and some comparisons may be omitted, and for example, abnormality detection using only threshold value 1 and only threshold value 2 may be performed, or the order may be changed.

  In addition, this invention is not limited to embodiment mentioned above. For example, in the above-described embodiment, the example in which the object detection device is mounted on the vehicle has been described. However, the object detection device may be used alone, or may be mounted on a robot or the like. Further, in the above-described embodiment, when calculating the ratios in step 215 and step 220, the denominator is the number of edge points, but the same determination can be made for the number of corresponding points (the threshold value is naturally changed). Further, although not included in the present invention, in step 215 and step 220, the ratio of the number of edge points and the number of corresponding points is not calculated, but the number of corresponding points of the negative parallax is equal to or greater than a predetermined threshold value, or It is also possible to determine whether the number of corresponding points is equal to or greater than a predetermined threshold. However, as described above, the determination accuracy is improved by performing determination as a ratio to the number of edge points or the number of corresponding points.

  In the present embodiment, edge points are used as feature points. However, the present invention is not limited to this, and any point having a feature capable of calculating corresponding points between a plurality of images may be used. For example, the feature points may be extracted using an arbitrary filter such as a Sobel filter, a differential filter, or a Gabor filter. In addition, the points in the present invention are not limited to pixels in an image, but may be regions having characteristics.

It is a vehicle block diagram carrying one Embodiment of the object detection apparatus of this invention. It is a flowchart of abnormality determination control. It is explanatory drawing explaining a corresponding point search.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Vehicle, 2R, 2L ... CCD camera (imaging means), 3 ... ECU (feature point extracting means / feature point measuring means / corresponding point detecting means / corresponding point measuring means / abnormal corresponding point determining means / abnormal corresponding point measuring means) ).

Claims (9)

In an object detection device for calculating corresponding points based on a plurality of images by a plurality of imaging means and detecting an object based on the calculated corresponding points,
Feature point extraction means for extracting feature points of each image;
A feature point measuring means for measuring the number of feature points;
Corresponding point detection means for detecting, as corresponding points, feature points that exist in common among a plurality of images among the feature points;
And corresponding point number measuring means for measuring the number of corresponding points,
The corresponding point detection means searches for a corresponding point only in a predetermined area of the image,
An image processing apparatus that determines that the optical axis of the imaging means is abnormal when the number of corresponding points with respect to the number of feature points is less than a predetermined reference.   The plurality of image pickup means are installed so that their optical axes are parallel, and the corresponding point detection means is such that the length of the predetermined region in the parallel direction is longer than the length in the direction orthogonal to the parallel direction. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 2, wherein when it is determined that there is an abnormality, it is determined that the optical axes of the plurality of imaging units are not in a normal positional relationship in a vertical direction in the image. In an object detection device for calculating corresponding points based on a plurality of images by a plurality of imaging means and detecting an object based on the calculated corresponding points,
Feature point extraction means for extracting feature points of each image;
Corresponding point detection means for detecting, as corresponding points, feature points that exist in common among a plurality of images among the feature points;
An abnormal corresponding point determination means for determining whether or not the detected corresponding point is an abnormal corresponding point based on the information of the corresponding point;
Anomaly corresponding point number measuring means for measuring the number of anomaly corresponding points;
A feature point measuring means for measuring the number of feature points or a corresponding point number measuring means for measuring the number of corresponding points;
An image processing apparatus that determines that the optical axis of the imaging means is abnormal when the number of characteristic points or the number of corresponding points corresponding to the number of corresponding points is equal to or greater than a predetermined reference.   The image processing apparatus according to claim 4, wherein the abnormality-corresponding point number measuring unit sets the number of corresponding points having a negative parallax value as the abnormality-corresponding point number.   The image processing apparatus according to claim 5, wherein when it is determined that there is an abnormality, it is determined that the optical axes of the plurality of imaging units are not in a normal positional relationship in the horizontal direction in the image. A height calculating means for calculating the height of the detected corresponding point in the three-dimensional coordinate system;
The image processing apparatus according to claim 4, wherein the abnormality corresponding point number measuring unit sets the number of corresponding points having a negative height value as the abnormality corresponding point number.   The image processing apparatus according to claim 7, wherein when it is determined that there is an abnormality, it is determined that the optical axes of the plurality of imaging units are not in a normal positional relationship in a vertical direction in the image. The image processing apparatus according to claim 1, wherein the feature point is an edge point.