JP7291594B2 - Image processing device - Google Patents

Description

The present disclosure relates to an image processing device that obtains parallax images based on stereo images.

Some vehicles, such as automobiles, generate parallax images including information about parallax based on stereo images obtained by a stereo camera. For example, Patent Literature 1 discloses a distance estimation method that generates a parallax image by calculating the amount of positional deviation between left and right image sensors using monocular motion stereo, and correcting the captured image based on the amount of positional deviation. An apparatus is disclosed.

JP 2015-175635 A

In the image processing apparatus, high accuracy of parallax images is desired, and further improvement in accuracy is expected.

It is desirable to provide an image processing apparatus that can improve the accuracy of parallax images.

An image processing device according to an embodiment of the present disclosure includes a rotation processing section, a parallax image generation section, and a rotation control section. The rotation processing unit is configured to generate a second left image and a second right image by performing rotation processing to rotate the first left image and the first right image included in the stereo image. The parallax image generator calculates a left image point in the second left image and a right image point in the second right image, which correspond to each other, based on the second left image and the second right image, configured to generate parallax images; The rotation control unit rotates a first position in the horizontal direction between a first position, which is the position of the left image point in the second left image, and a second position, which is the position of the right image point in the second right image. and a second positional difference in the vertical direction are obtained, and based on the coordinate points, the angular change amount of the rotation angle in the rotation processing is calculated.

According to the image processing device according to the embodiment of the present disclosure, it is possible to improve the accuracy of parallax images.

1 is a block diagram showing a configuration example of an image processing device according to an embodiment of the present disclosure; FIG. 2 is an explanatory diagram showing an example of a left image and a right image generated by the stereo camera shown in FIG. 1; FIG. 3 is an explanatory diagram showing another example of left and right images generated by the stereo camera shown in FIG. 1; FIG. FIG. 3 is an explanatory diagram showing an operation example of a rotation processing unit shown in FIG. 1; 2 is an explanatory diagram showing an example of a left image point and a right image point related to corresponding points calculated by a corresponding point calculation unit shown in FIG. 1; FIG. 2 is a flow chart showing an operation example of a rotation control unit shown in FIG. 1; 3 is an explanatory diagram showing an operation example of the rotation control unit shown in FIG. 1; FIG. FIG. 10 is another explanatory diagram showing an operation example of the rotation control unit shown in FIG. 1;

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

<Embodiment>
[Configuration example]
FIG. 1 shows a configuration example of an image processing apparatus (image processing apparatus 1) according to one embodiment. The image processing device 1 has a stereo camera 11 and a processing section 20 . The image processing apparatus 1 is mounted on a vehicle 10 such as an automobile.

Stereo camera 11 is configured to generate a pair of images (left image PL and right image PR) having parallax with each other by capturing an image in front of vehicle 10 . The stereo camera 11 has a left camera 11L and a right camera 11R. Each of left camera 11L and right camera 11R includes a lens and an image sensor. In this example, the left camera 11L and the right camera 11R are arranged inside the vehicle 10 at a predetermined distance in the width direction of the vehicle 10 near the top of the windshield of the vehicle 10 . The left camera 11L and the right camera 11R perform imaging operations in synchronization with each other. The left camera 11L generates a left image PL and the right camera 11R generates a right image PR. The left image PL and the right image PR constitute a stereo image PIC. The stereo camera 11 is adapted to generate a series of stereo images PIC by performing imaging operations at a predetermined frame rate (for example, 60 [fps]).

FIG. 2 shows an example of the stereo image PIC, FIG. 2(A) shows an example of the left image PL, and FIG. 2(B) shows an example of the right image PR. In this example, another vehicle (preceding vehicle 90) is traveling in front of the vehicle 10 on the road on which the vehicle 10 is traveling. The left camera 11L captures the preceding vehicle 90 to generate the left image PL, and the right camera 11R captures the preceding vehicle 90 to generate the right image PR. The stereo camera 11 is adapted to generate a stereo image PIC including such left image PL and right image PR.

The processing unit 20 is configured to recognize an object in front of the vehicle 10 based on the stereo image PIC supplied from the stereo camera 11 . In the vehicle 10, for example, based on the information about the object recognized by the processing unit 20, the running control of the vehicle 10 can be performed, or the information about the recognized object can be displayed on the console monitor. ing. The processing unit 20 includes, for example, a CPU (Central Processing Unit) that executes programs, a RAM (Random Access Memory) that temporarily stores processing data, and a ROM (Read Only Memory) that stores programs. The processing unit 20 has a rotation processing unit 21 , a parallax image generation unit 22 , a rotation control unit 24 and an object recognition unit 27 .

Based on the rotation control data CTL supplied from the rotation control unit 24, the rotation processing unit 21 rotates the left image PL1 and the right image PR included in the stereo image PIC, thereby rotating the left image PL1 and the right image PR. It is arranged to generate an image PR1.

That is, in the processing unit 20, as will be described later, the parallax image generation unit 22 detects corresponding points based on the left image PL (left image PL1) and the right image PR (right image PR1) to generate the parallax image PD. to generate As described above, when the parallax image generation unit 22 generates the parallax image PD, as shown in FIG. 2, the vertical positions of the images of the preceding vehicle 90 in the left image PL and the right image PR are substantially the same. desired. However, in the stereo camera 11, the left camera 11L and the right camera 11R may be arranged relatively shifted in the height direction, for example.

FIG. 3 shows an example of the stereo image PIC when the position of the left camera 11L is arranged at a position lower than the position of the right camera 11R, and FIG. 3A shows an example of the left image PL. 3B, and FIG. 3B shows an example of the right image PR. In this way, when the position of the left camera 11L is positioned lower than the position of the right camera 11R, as shown in FIG. 3, the position of the image of the preceding vehicle 90 in the left image PL is It is above the position of the image of the preceding vehicle 90 in the right image PR. On the other hand, when the position of the left camera 11L is higher than the position of the right camera 11R, contrary to the case of FIG. It is below the position of the image of the preceding vehicle 90 in the image PR. In this way, when the left camera 11L and the right camera 11R are arranged with relative displacement in the height direction, the vertical position of the image of the preceding vehicle 90 in the left image PL and the right image PR is also , deviate from each other.

In this way, when the position of the image of the preceding vehicle 90 is shifted in the vertical direction between the left image PL and the right image PR, the parallax image generator 22 generates the image of the preceding vehicle 90 based on the left image PL and the right image PR. cannot be detected, and as a result, the accuracy of the parallax image PD may deteriorate. Therefore, in the image processing device 1, the rotation processing unit 21 rotates the left image PL1 and the right image PR by the same rotation angle θ in the same rotation direction based on the rotation control data CTL. Generate an image PR1. Then, the parallax image generator 22 detects corresponding points based on the left image PL1 and the right image PR1.

4A shows the left image PL1 generated by the rotation processing unit 21, and FIG. 4B shows the right image PR1 generated by the rotation processing unit 21. FIG. In this example, as shown in FIG. 3, the position of the image of preceding vehicle 90 in left image PL is above the position of the image of preceding vehicle 90 in right image PR. In this case, as shown in FIG. 4, the rotation processing unit 21 rotates the left image PL and the right image PR clockwise by the rotation angle θ to generate the left image PL1 and the right image PR1. As a result of the rotation processing performed by the rotation processing unit 21 in this manner, the positions of the images of the preceding vehicle 90 in the left image PL1 and the right image PR1 in the vertical direction (Y direction) can be made substantially the same. Note that, for example, contrary to the case of FIG. 3, when the position of the image of the preceding vehicle 90 in the left image PL is below the position of the image of the preceding vehicle 90 in the right image PR, the rotation processing unit 21 By rotating the left image PL and the right image PR counterclockwise, the positions of the images of the preceding vehicle 90 in the left image PL1 and the right image PR1 in the vertical direction (Y direction) can be substantially the same. can. Thereby, the parallax image generator 22 can detect corresponding points based on the left image PL1 and the right image PR1, and can generate the parallax image PD based on the detection result.

The parallax image generation unit 22 (FIG. 1) performs predetermined image processing including stereo matching processing and filtering processing based on the left image PL1 and the right image PR1 generated by the rotation processing unit 21, thereby generating parallax images. configured to generate a PD; The parallax image PD has multiple pixel values. Each of the plurality of pixel values indicates a parallax value for each pixel. In other words, each of the plurality of pixel values corresponds to the distance to the point corresponding to each pixel in the three-dimensional real space.

The parallax image generator 22 has a corresponding point calculator 23 . Based on the left image PL1 and the right image PR1, the corresponding point calculation unit 23 performs matching processing on each of the plurality of divided sub-regions S as a unit, thereby obtaining corresponding points CP including two image points corresponding to each other. is configured to identify the Specifically, the corresponding point calculation unit 23 searches for sub-regions SL and SR having similar image patterns among the plurality of sub-regions SL in the left image PL1 and the plurality of sub-regions SR in the right image PR1 . , the left image point CPL and the right image point CPR that correspond to each other are specified as corresponding points CP. The corresponding point calculation unit 23 may specify the corresponding points CP by, for example, template matching, or may specify the corresponding points CP by feature amount matching based on local feature amounts.

FIG. 5 shows an example of a left image point CPL and a right image point CPR that constitute one corresponding point CP. In FIG. 5, the left image point CPL in the left image PL1 and the right image point CPR in the right image PR1 are drawn in an overlapping manner. In this example, as shown in FIG. 3, the position of the image of preceding vehicle 90 in left image PL is above the position of the image of preceding vehicle 90 in right image PR. Also, the rotation angle θ is 0 degree, which is not set to an appropriate angle. Therefore, the coordinate position of the left image point CPL in the left image PL1 and the coordinate position of the right image point CPR in the right image PR1 are shifted from each other. Specifically, in this example, the coordinate position of the left image point CPL is shifted from the coordinate position of the right image point CPR by a coordinate difference ΔX in the X direction and by a coordinate difference ΔY in the Y direction.

The coordinate difference ΔX corresponds to, for example, parallax between the left image PL1 and the right image PR1. That is, the coordinate difference ΔX is large when the distance to the subject related to the corresponding point CP is short, and the coordinate difference ΔX is small when the distance to the subject is long.

The coordinate difference ΔY corresponds to the position difference in the vertical direction (Y direction) of the subject image related to the corresponding point CP in the left image PL1 and the right image PR1. In this example, as shown in FIG. 3, the position of the subject image in the left image PL is higher than the position of the subject image in the right image PR, so the coordinate difference ΔY is positive. Contrary to case 3, the coordinate difference ΔY is negative when the position of the subject image in the left image PL is below the position of the subject image in the right image PR. If the rotation angle θ is not set to an appropriate angle, the closer the object is to the subject, the larger the absolute value of the coordinate difference ΔY. becomes smaller.

The rotation control unit 24 calculates the rotation angle θ in the rotation processing of the rotation processing unit 21 based on the data about the corresponding points CP calculated by the corresponding point calculation unit 23, thereby causing the rotation processing unit 21 to perform the rotation processing. is configured to generate rotation control data CTL that instructs to The rotation control section 24 has an approximate straight line calculation section 25 and a rotation angle calculation section 26 .

Based on the data about the corresponding points CP, the approximate straight line calculation unit 25 calculates the horizontal direction (X A coordinate point P indicating the relationship between the coordinate difference ΔX in the vertical direction (Y direction) and the coordinate difference ΔY in the vertical direction (Y direction) is obtained, and an approximate straight line LIN for the coordinate point P is calculated based on the coordinate point P. be.

The rotation angle calculation unit 26 calculates an angle change amount Δθ from the current rotation angle θ based on the approximate straight line LIN calculated by the approximate straight line calculation unit 25, and adds the angle change amount Δθ to the current rotation angle θ. is configured to calculate the rotation angle .theta. Then, the rotation angle calculator 26 generates rotation control data CTL including data about the calculated rotation angle θ.

Object recognition unit 27 is configured to recognize an object in front of vehicle 10 based on left image PL<b>1 and right image PR<b>1 and parallax image PD generated by parallax image generation unit 22 . Then, the object recognition section 27 outputs data about the recognition result.

With this configuration, in the image processing device 1, the rotation processing unit 21 performs rotation processing for rotating the left image PL and the right image PR based on the rotation control data CTL, thereby generating the left image PL1 and the right image PR1. Then, the parallax image generator 22 generates a parallax image PD based on the left image PL1 and the right image PR1. The rotation control unit 24 generates rotation control data CTL based on data about the corresponding points CP calculated by the corresponding point calculation unit 23 of the parallax image generation unit 22 . As a result, the image processing device 1 adjusts the rotation angle θ so that, for example, the position (Y coordinate) in the height direction of the images of the preceding vehicle 90 in the left image PL1 and the right image PR1 are substantially the same. be able to. The parallax image generator 22 can detect the corresponding points CP based on such left image PL1 and right image PR1. As a result, the image processing device 1 can improve the accuracy of the parallax image PD.

Here, the rotation processing unit 21 corresponds to a specific example of "rotation processing unit" in the present disclosure. The parallax image generator 22 corresponds to a specific example of the “parallax image generator” in the present disclosure. The rotation control unit 24 corresponds to a specific example of "rotation control unit" in the present disclosure. A stereo image PIC corresponds to a specific example of a “stereo image” in the present disclosure. The left image PL corresponds to a specific example of "first left image" in the present disclosure. The right image PR corresponds to a specific example of "first right image" in the present disclosure. The left image PL1 corresponds to a specific example of "second left image" in the present disclosure. The right image PR1 corresponds to a specific example of "second right image" in the present disclosure. The left image point CPL corresponds to one specific example of "left image point" in this disclosure. The right image point CPR corresponds to one specific example of "right image point" in this disclosure. The coordinate difference ΔX corresponds to a specific example of "first positional difference" in the present disclosure. The coordinate difference ΔY corresponds to a specific example of "second positional difference" in the present disclosure. The rotation angle θ corresponds to a specific example of "rotation angle" in the present disclosure. The angle change amount Δθ corresponds to a specific example of "angle change amount" in the present disclosure. Approximate straight line LIN corresponds to a specific example of "approximate straight line" in the present disclosure.

[Operation and action]
Next, the operation and action of the image processing apparatus 1 of this embodiment will be described.

(Outline of overall operation)
First, an overview of the overall operation of the image processing apparatus 1 will be described with reference to FIG. Stereo camera 11 generates a stereo image PIC including a left image PL and a right image PR by capturing an image in front of vehicle 10 . In the processing unit 20, the rotation processing unit 21 performs rotation processing for rotating the left image PL and the right image PR included in the stereo image PIC based on the rotation control data CTL, thereby rotating the left image PL1 and the right image PR1. Generate. The parallax image generation unit 22 performs predetermined image processing including stereo matching processing and filtering processing based on the left image PL1 and the right image PR1 generated by the rotation processing unit 21, thereby generating the parallax image PD. . The rotation control unit 24 calculates the rotation angle θ in the rotation processing of the rotation processing unit 21 based on the data about the corresponding points CP calculated by the corresponding point calculation unit 23 of the parallax image generation unit 22. 21 to generate rotation control data CTL for instructing rotation processing. Object recognition unit 27 recognizes an object in front of vehicle 10 based on left image PL<b>1 and right image PR<b>1 and parallax image PD generated by parallax image generation unit 22 .

(detailed operation)
The rotation control unit 24 calculates the rotation angle θ in the rotation processing of the rotation processing unit 21 based on the data about the corresponding points CP calculated by the corresponding point calculation unit 23 of the parallax image generation unit 22 . Specifically, the approximate straight line calculation unit 25 calculates the distance between the coordinate position of the left image point CPL in the left image PL1 and the coordinate position of the right image point CPR in the right image PR1 based on the data about the corresponding point CP. , the coordinate difference ΔX in the horizontal direction (X direction) and the coordinate difference ΔY in the vertical direction (Y direction) are obtained, and based on the coordinate point P, an approximate straight line LIN for the coordinate point P is calculated. do. Then, the rotation angle calculator 26 calculates the angle change amount Δθ from the current rotation angle θ based on the approximate straight line LIN calculated by the approximate straight line calculator 25, and adds the angle change amount Δθ to the current rotation angle θ. The rotation angle θ is calculated by the addition. The operation of the rotation control section 24 will be described in detail below.

FIG. 6 shows an operation example of the rotation control section 24 . In this example, the rotation control unit 24 calculates the approximate straight line LIN based on the data on the corresponding points CP each time the corresponding points CP are calculated by the corresponding point calculation unit 23 . Then, the rotation angle calculator 26 calculates the rotation angle θ based on the approximate straight line LIN calculated by the approximate straight line calculator 25 . This operation will be described in detail below.

First, the approximate straight line calculator 25 obtains the coordinate point P based on the data of the corresponding point CP, and updates the plot 100 for the coordinate point P based on the coordinate point P (step S101). Specifically, the approximate straight line calculation unit 25 calculates the coordinate difference in the horizontal direction (X direction) between the coordinate position of the left image point CPL in the left image PL1 and the coordinate position of the right image point CPR in the right image PR1. A coordinate point P representing the relationship between ΔX and the coordinate difference ΔY in the vertical direction (Y direction) is obtained. Then, the approximate straight line calculation unit 25 updates the plot 100 by recording the calculated coordinate points P in the plot 100 in which a plurality of coordinate points P calculated in the past are recorded.

FIG. 7 shows a configuration example of the plot diagram 100. As shown in FIG. In this plot diagram 100, the horizontal axis indicates the coordinate difference ΔX, and the vertical axis indicates the coordinate difference ΔY. In FIG. 7, the dot indicates the coordinate point P. As shown in FIG. These multiple coordinate points P are calculated, for example, based on multiple stereo images PIC generated by the stereo camera 11 after the engine of the vehicle 10 is started. In this example, the number of coordinate points P recorded in the plot diagram 100 has an upper limit, and when a new coordinate point P is calculated when the number of coordinate points P reaches the upper limit, the oldest The coordinate point P is deleted from the plot 100 and the newly calculated coordinate point P is added to this plot 100 . In this example, plot 100 is divided into a plurality of areas BIN (three areas BIN1 to BIN3 in this example) along the horizontal axis. Specifically, the area BIN1 is an area where the coordinate difference ΔX is 0 or more and less than 50, the area BIN2 is an area where the coordinate difference ΔX is 50 or more and less than 100, and the area BIN3 is an area where the coordinate difference ΔX is 100. It is a region that is greater than or equal to less than 150.

Next, the approximate straight line calculator 25 calculates an approximate straight line LIN based on the plurality of coordinate points P in the plot diagram 100 (step S102). Specifically, in this example, the approximate straight line calculator 25 calculates the center of gravity of the coordinate point P in each of the plurality of regions BIN. Specifically, the approximate straight line calculation unit 25 calculates the center of gravity of the plurality of coordinate points P belonging to the region BIN1, the center of gravity of the plurality of coordinate points P belonging to the region BIN2, and the center of gravity of the plurality of coordinate points P belonging to the region BIN3. A center of gravity of the coordinate point P is calculated. Then, the approximate straight line calculator 25 calculates the approximate straight line LIN by the least squares method based on the center of gravity of each of the plurality of areas BIN. Specifically, the approximate straight line calculator 25 calculates, for example, a slope value a and an intercept value b in a linear function represented by the following equation EQ1 by the method of least squares.
ΔY = a × ΔX + b (EQ1)
The approximate straight line calculator 25 thus calculates the approximate straight line LIN. In this example, the slope value a and the intercept value b are calculated by the method of least squares, but the present invention is not limited to this. Only the slope value a may be calculated by the square method.

In this example, the approximate straight line calculation unit 25 calculates the center of gravity in each of the plurality of areas BIN based on the plurality of coordinate points P in the plot diagram 100, and calculates the approximate straight line LIN based on these center of gravity. However, it is not limited to this. Alternatively, the approximate straight line calculator 25 may directly calculate the approximate straight line LIN by the least squares method based on the plurality of coordinate points P in the plot 100, for example. In this case, the approximate straight line calculator 25 can calculate the approximate straight line LIN by, for example, the weighted least squares method. Specifically, for example, the weight of the coordinate point P can be set according to the number of coordinate points P in the area BIN to which the coordinate point P belongs. For example, when the number of coordinate points P in a certain area BIN is large, the weight of the coordinate points P belonging to the area BIN is reduced. The weight of the coordinate point P to which it belongs can be increased.

An example of the calculated approximate straight line LIN is shown in FIG. In the example of FIG. 7, the slope value a of the approximate straight line LIN is positive, the larger the coordinate difference ΔX, the larger the coordinate difference ΔY, and the smaller the coordinate difference ΔX, the smaller the coordinate difference ΔY and approaching 0 (zero). . In other words, the shorter the distance to the object related to the corresponding point CP, the larger the absolute value of the coordinate difference ΔY, and the longer the distance to the object, the smaller the coordinate difference ΔY. This indicates that the positions of the subject images in the left image PL1 and the right image PR1 are shifted in the vertical direction, and that the rotation angle θ has not yet been adjusted to an appropriate angle. .

Next, the rotation angle calculator 26 adds data (inclination data) about the inclination value a of the approximate straight line LIN to the inclination list La (step S103). This tilt list La is configured to be able to store a predetermined number of tilt data. There is an upper limit to the number of tilt data that can be stored in the tilt list La. If a new approximate straight line LIN is calculated when the number of tilt data reaches the upper limit, the oldest tilt data will , and the inclination data for the newly calculated approximate straight line LIN is added to the inclination list La.

Next, the rotation angle calculator 26 calculates the variance A of the tilt values a indicated by the plurality of tilt data in the tilt list La (step S104). If the variance A is greater than or equal to the threshold Ath ("N" in step S105), this flow ends. That is, when the variance value A is equal to or greater than the threshold value Ath, the tilt value a varies and the accuracy is not sufficient , so the flow ends without calculating the rotation angle θ.

If the variance value A is smaller than the threshold value Ath (A<Ath) (“Y” in step S105), the rotation angle calculator 26 calculates the average value of the tilt values a indicated by the plurality of tilt data in the tilt list La. (Slope average value AVG) is calculated (step S106).

Then, the rotation angle calculator 26 calculates the rotation angle θ by calculating the angle change amount Δθ from the current rotation angle θ based on the inclination average value AVG (step S107). Specifically, the rotation angle calculator 26 sets the polarity of the angle change amount Δθ based on the polarity of the gradient average value AVG. Further, the rotation angle calculator 26 increases the absolute value of the angle change amount Δθ as the absolute value of the average inclination value AVG increases. Specifically, for example, when the absolute value of the gradient average value AVG is the first value, the rotation angle calculator 26 sets the absolute value of the angle change amount Δθ to the second value, When the absolute value of the value AVG is a third value larger than the first value, the absolute value of the angle change amount Δθ is set to a fourth value larger than the second value. Then, the rotation angle calculator 26 calculates the rotation angle θ by adding the angle change amount Δθ to the current rotation angle θ.

Thus, the rotation control unit 24 calculates the rotation angle θ based on the data about the corresponding point CP, and generates the rotation control data CTL including data about the calculated rotation angle θ. The rotation processing unit 21 rotates the left image PL and the right image PR at the rotation angle θ based on the rotation control data CTL to generate the left image PL1 and the right image PR1. The image processing apparatus 1 repeats such processing.

The rotation angle θ is controlled in a negative feedback manner by the image processing apparatus 1 repeating such processing. By this negative feedback operation, the slope value a of the approximate straight line LIN in the plot diagram 100 gradually decreases, and finally, as shown in FIG. 8, the slope value a of the approximate straight line LIN is almost 0 ( zero). As a result, as shown in FIG. 4, the rotation angle θ is adjusted to an angle such that the positions of the images of preceding vehicle 90 in the vertical direction (Y direction) are substantially the same in left image PL1 and right image PR1. be. The parallax image generator 22 can detect the corresponding points CP based on such left image PL1 and right image PR1. As a result, the image processing device 1 can improve the accuracy of the parallax image PD.

In this way, the image processing apparatus 1 generates a left image PL1 and a right image PR1 by performing rotation processing for rotating the left image PL and the right image PR included in the stereo image PIC, and generates the left image PL1 and the right image PR1. A left image point CPL in the left image PL1 and a right image point CPR in the right image PR1, which correspond to each other, are calculated based on the image PR1. As a result, even when the left camera 11L and the right camera 11R are arranged with a relative displacement in the height direction, for example, the image of the preceding vehicle 90 in the left image PL1 and the right image PR1 can be detected in the vertical direction (Y direction). can be made substantially the same, the corresponding point CP can be detected based on the left image PL1 and the right image PR1. As a result, the accuracy of the parallax image PD can be improved.

Further, in the image processing apparatus 1, the coordinate difference ΔX in the horizontal direction (X direction) and the vertical direction ( The coordinate point P indicating the relationship with the coordinate difference ΔY in the Y direction) is obtained, and the angle change amount Δθ of the rotation angle θ in the rotation processing is calculated based on the coordinate point P. θ can be calculated. That is, for example, based on the corresponding points CP, the epipolar geometry is used to obtain the relative displacement amount in the height direction of the left camera 11L and the right camera 11R, and based on this displacement amount, the left image PL and the right image PL are calculated. A method of correcting the PR is also possible. However, in this case, for example, the algorithm may become complicated and the processing may take a long time. On the other hand, in the image processing apparatus 1, the angle change amount Δθ of the rotation angle θ is calculated by performing calculations on a two-dimensional plane using the coordinate difference ΔX and the coordinate difference ΔY. A rotation angle θ in the process can be calculated.

Further, in the image processing apparatus 1, the angle change amount Δθ is calculated based on a predetermined number of tilt values a out of the tilt values a obtained each time the approximate straight line LIN is updated. In particular, the image processing apparatus 1 calculates the variance value A of the predetermined number of slope values a, and if the variance value A is smaller than the threshold value Ath, based on the slope average value AVG of the predetermined number of slope values a. Then, the angle change amount Δθ is calculated. As a result, the image processing apparatus 1 does not calculate the angle change amount Δθ when the tilt value a varies, so that the rotation angle θ can be calculated with high accuracy. can be enhanced.

[effect]
As described above, in the present embodiment, left image PL1 and right image PR1 are generated by performing rotation processing for rotating left image PL and right image PR included in a stereo image, and left image PL1 and right image PR1 are generated. Since the left image point in the left image PL1 and the right image point in the right image PR1, which correspond to each other, are calculated based on PR1, the accuracy of the parallax image can be improved.

In the present embodiment, a coordinate point indicating the relationship between the coordinate position of the left image point in the left image PL1 and the coordinate position of the right image point in the right image PR1 in the horizontal direction and the coordinate difference in the vertical direction. is obtained, and the amount of change in the rotation angle in the rotation process is calculated based on this coordinate point, so the rotation angle can be calculated by a simple method.

In the present embodiment, the angle change amount is calculated based on the predetermined number of tilt values among the tilt values obtained each time the approximation straight line is updated, so the accuracy of the parallax image can be improved.

Although the present technology has been described above with reference to the embodiments, the present technology is not limited to these embodiments and the like, and various modifications are possible.

For example, in the above-described embodiment, the stereo camera 11 images the front of the vehicle 10, but the present invention is not limited to this.

Note that the effects described in this specification are merely examples and are not limited, and other effects may be provided.

Reference Signs List 1 image processing device 11 stereo camera 11L left camera 11R right camera 20 processing unit 21 rotation processing unit 22 parallax image generation unit 23 corresponding point calculation unit 24 rotation control Section 25 Approximate line calculator 26 Rotation angle calculator 27 Object recognition section CP Corresponding point CPL Left image point CPR Right image point LIN Approximate line P Coordinate point PD Parallax image PL, PL1 Left image PR, PR1 Right image θ Rotation angle ΔX, ΔY Coordinate difference Δθ Angle change amount.

Claims (7)

a rotation processing unit that generates a second left image and a second right image by performing rotation processing for rotating the first left image and the first right image included in the stereo image;
disparity images by calculating corresponding left image points in the second left image and right image points in the second right image based on the second left image and the second right image; a parallax image generator that generates
A first lateral position between a first position, which is the position of the left image point in the second left image, and a second position, which is the position of the right image point in the second right image. a rotation control unit that obtains a coordinate point indicating a relationship between the positional difference and a second positional difference in the vertical direction, and calculates an angle change amount of the rotation angle in the rotation processing based on the coordinate point. . The image processing apparatus according to claim 1, wherein the rotation control section obtains the coordinate points based on each of the plurality of stereo images, and sets the angle change amount based on the plurality of coordinate points. The rotation control unit allocates the first positional difference to a first direction and assigns the second positional difference to a second direction, thereby arranging the plurality of coordinate points. The image processing apparatus according to claim 2, wherein an approximate straight line of points is calculated, and the angle change amount is calculated based on an inclination value of the approximate straight line. The image processing apparatus according to claim 3, wherein the rotation control section calculates the angle change amount based on a predetermined number of the tilt values among the tilt values obtained each time the approximate straight line is calculated. The rotation control unit calculates a variance value of the predetermined number of tilt values,
5. The image processing device according to claim 4, wherein when the variance value is smaller than a predetermined threshold value, the angle change amount is calculated based on a slope average value that is an average value of the predetermined number of slope values. The image processing apparatus according to claim 5, wherein the rotation control section sets the polarity of the angle change amount based on the polarity of the tilt average value. The rotation control unit is
setting the absolute value of the angle change amount to a second value when the absolute value of the tilt average value is a first value;
6. When the absolute value of the tilt average value is a third value larger than the first value, the absolute value of the angle change amount is set to a fourth value larger than the second value. 7. The image processing apparatus according to claim 6.

Images