JP7571843B2 - Vehicle image processing device and vehicle image processing method - Google Patents

Description

The present invention relates to a vehicle image processing device and a vehicle image processing method.

There is a known technology for checking the surroundings of a vehicle using image data captured around the vehicle (see, for example, Patent Document 1). The technology described in Patent Document 1 displays on a display an image captured by a camera at an intersection with poor visibility that is registered in a car navigation system.

Japanese Patent Application Publication No. 08-293098

For example, if the road on which the vehicle is traveling is covered in snow or flooded, it is necessary to display a larger area than usual for the video data captured by the camera installed in the vehicle.

The present invention has been made in consideration of the above, and aims to properly display an image of the surroundings of the vehicle in the display area.

In order to solve the above-mentioned problems and achieve the object, the vehicle image processing device of the present invention includes an image data acquisition unit that acquires image data captured by an image capture unit that captures the surroundings of the vehicle, a distortion correction unit that corrects distortion of the image data acquired by the image data acquisition unit using distortion parameters and outputs distortion-corrected image data, and a road surface area detection unit that detects the road surface area captured in the image data acquired by the image data acquisition unit, and is characterized in that, when the road surface area detection unit cannot detect the road surface, the distortion correction unit corrects the distortion by changing the distortion parameters so that the display area is wider than the display area when the distortion parameters are not changed.

The vehicle image processing method according to the present invention includes a video data acquisition step of acquiring video data captured by an image capture unit that captures the surroundings of the vehicle, a distortion correction step of correcting distortion of the video data acquired in the video data acquisition step using distortion parameters and outputting distortion-corrected video data, and a road surface area detection step of detecting a road surface area captured in the video data acquired in the video data acquisition step, and in the distortion correction step, if the road surface cannot be detected in the road surface area detection step, the distortion parameters are changed to correct the distortion so that the display area is wider than the display area when the distortion parameters are not changed.

The present invention has the effect of being able to properly display an image of the surroundings of the vehicle in the display area.

FIG. 1 is a block diagram showing an example of the configuration of a vehicle image processing device according to an embodiment. FIG. 2 is a diagram for explaining a state in which a vehicle is traveling on a flat road. FIG. 3 is a diagram for explaining a state in which a vehicle is traveling on a flat road. FIG. 4 is a diagram showing an example of video data captured by a camera when a vehicle is traveling on a flat road. FIG. 5 is a diagram showing an example of a display area on which distortion correction processing is performed, out of the video data shown in FIG. FIG. 6 is a diagram for explaining a state in which a vehicle is traveling on a road having a steep upward gradient. FIG. 7 is a diagram for explaining a state in which a vehicle is traveling on a road having a steep upward gradient. FIG. 8 is a diagram showing an example of video data captured by a camera when a vehicle is traveling on a road having a steep upward gradient. FIG. 9 is a diagram showing an example of a display area on which distortion correction processing is performed, out of the video data shown in FIG. FIG. 10 is a diagram for explaining a state in which a vehicle is traveling on a road having a steep downward gradient. FIG. 11 is a diagram for explaining a state in which a vehicle is traveling on a road having a steep downward gradient. FIG. 12 is a diagram showing an example of video data captured by a camera when a vehicle is traveling on a road having a steep downward slope. FIG. 13 is a diagram showing an example of a display area on which distortion correction processing is performed, out of the video data shown in FIG. FIG. 14 is a flowchart showing a process flow in the vehicle image processing device according to the embodiment. FIG. 15 is a diagram showing an example of a display area on which distortion correction processing is performed when the intersecting road is curved. FIG. 16 is a diagram illustrating a state in which a road intersecting the road on which the vehicle is traveling is curved. FIG. 17 is a diagram showing an example of a display area on which distortion correction processing is performed when a vehicle is traveling on a snowy road.

The following describes in detail the embodiments of the vehicle image processing device, vehicle image processing system, vehicle image processing method, and program according to the present invention with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiments.

FIG. 1 is a block diagram showing an example of the configuration of a vehicle image processing device according to an embodiment. The vehicle image processing system 1 displays an image for checking the surroundings of the vehicle V, for example, when driving at low speed or when parking. The vehicle image processing system 1 displays an image in an appropriate display area regardless of the inclination angle of the vehicle V. The vehicle image processing system 1 has a camera unit (photographing section) 10, a display section 21, a headlight switch 22, a vehicle speed sensor 23, a three-axis gyro sensor 24, and a vehicle image processing device 30.

The camera unit 10 has a camera that captures images of the surroundings of the vehicle V. A plurality of cameras may be provided. In this embodiment, the camera unit 10 has a front camera 11, a right side camera 12, a left side camera 13, and a rear camera 14.

The front camera 11 is disposed in front of the vehicle V and captures images of the surroundings, centered on the area in front of the vehicle V. The front camera 11 captures images with an ultra-wide angle, for example, a horizontal range of approximately 190°. The front camera 11 outputs the captured images to the image processing unit 34 of the vehicle image processing device 30.

The right side camera 12 is disposed on the right side of the vehicle V and captures images of the surroundings, mainly on the right side of the vehicle V. The right side camera 12 captures images with an ultra-wide angle, for example, a horizontal range of about 190°. The right side camera 12 outputs the captured images to the image processing unit 34 of the vehicle image processing device 30.

The left side camera 13 is disposed on the left side of the vehicle V and captures the surroundings centered on the left side of the vehicle V. The left side camera 13 captures an ultra-wide-angle image, for example, a horizontal image range of about 190°. The left side camera 13 outputs the captured image to the image processing unit 34 of the vehicle image processing device 30.

The rear camera 14 is disposed at the rear of the vehicle V and captures images of the surrounding area, mainly the rear of the vehicle V. The rear camera 14 captures images with an ultra-wide angle, for example, a horizontal range of about 190°. The rear camera 14 outputs the captured images to the image processing unit 34 of the vehicle image processing device 30.

The camera unit 10, which includes the front camera 11, the right side camera 12, the left side camera 13, and the rear camera 14, captures images in all directions of the vehicle V.

The display unit 21 is, for example, a display including a liquid crystal display (LCD) or an organic electroluminescence (EL) display. The display unit 21 displays, for example, an image of the periphery of the vehicle V based on a video signal output from the vehicle image processing device 30 of the vehicle image processing system 1. The display unit 21 may be dedicated to the vehicle image processing system 1 or may be used in conjunction with other systems including, for example, a navigation system. The display unit 21 is disposed in a position that is easily visible to the user. In this embodiment, the display unit 21 is disposed in the center console in the center of the dashboard in the vehicle width direction.

The headlight switch 22 outputs a signal capable of detecting whether the headlights are on or off to the headlight on/off detection unit 31 of the vehicle image processing device 30.

The vehicle speed sensor 23 detects the vehicle speed of the vehicle V. More specifically, the vehicle speed sensor 23 is disposed on the drive shaft or tires of the vehicle V. The vehicle speed sensor 23 detects a pulse signal corresponding to the rotation of the drive shaft or tires. The vehicle speed sensor 23 outputs the detected pulse signal to the vehicle speed detection unit 32 of the vehicle image processing device 30.

The three-axis gyro sensor 24 detects the gradient of the road on which the vehicle V runs. The three-axis gyro sensor 24 is a sensor that detects the inclination of the vehicle V. The three-axis gyro sensor 24 detects, for example, the angular velocity of the vehicle V. More specifically, the three-axis gyro sensor 24 has a roll rate gyro, a pitch rate gyro, and a yaw rate gyro. The roll rate gyro detects the roll angular velocity, which is the angular velocity of rotation around the axis of the front-rear direction of the vehicle V. The pitch rate gyro detects the pitch angular velocity, which is the angular velocity of rotation around the axis of the left-right direction of the vehicle V. The yaw rate gyro detects the yaw angular velocity, which is the angular velocity of rotation around the axis of the vertical direction of the vehicle V. The three-axis gyro sensor 24 outputs the roll angular velocity detected by the roll rate gyro, the pitch angular velocity detected by the pitch rate gyro, and the yaw angular velocity detected by the yaw rate gyro as angular velocity signals to the inclination angle detection unit 33 of the vehicle image processing device 30.

The vehicle image processing device 30 generates an image showing the surroundings of the vehicle V and displays it on the display unit 21. The vehicle image processing device 30 loads a program stored in the storage unit 39 into memory and executes the instructions contained in the program. The vehicle image processing device 30 has a headlight illumination detection unit 31, a vehicle speed detection unit 32, a tilt angle detection unit 33, an image processing unit 34, a determination unit 35, a photometry area control unit 36, a display control unit 37, and a storage unit 39.

The headlight on detection unit 31 detects whether the headlights are on or off from the signal output from the headlight switch 22. When the headlights are on, the road surface illuminated by the headlights is captured in the forward image data captured by the forward camera 11. The road surface illuminated by the headlights captured in the forward image data differs in hue, lightness, saturation, and luminance from the road surface not illuminated by the headlights. The headlight on detection unit 31 outputs the detection result to the determination unit 35.

The vehicle speed detection unit 32 detects the vehicle speed of the vehicle V from the pulse signal acquired from the vehicle speed sensor 23. The vehicle speed detection unit 32 outputs vehicle speed information indicating the calculated vehicle speed to the determination unit 35.

The inclination angle detection unit 33 detects the inclination angle of the road on which the vehicle V travels. In this embodiment, the inclination angle detection unit 33 acquires the angular velocity signal output from the three-axis gyro sensor 24. The inclination angle detection unit 33 calculates the roll angle, pitch angle, and yaw angle, which indicate the inclination of the vehicle V, from the roll angular velocity, pitch angular velocity, and yaw angular velocity contained in the angular velocity signal. The inclination angle detection unit 33 outputs the calculated roll angle, pitch angle, and yaw angle to the determination unit 35 as inclination angle information. Note that the inclination angle detection unit 33 may calculate only the pitch angle and output only the pitch angle to the determination unit 35 as inclination angle information.

The image processing unit 34 generates display image data for an image to be displayed on the display unit 21 based on the image data captured by the camera unit 10. The image processing unit 34 has an image data acquisition unit 341, a road surface area detection unit 342, a distortion correction unit 343, and an image generation unit 344.

The video data acquisition unit 341 acquires video data captured by the camera unit 10. The video data acquisition unit 341 acquires forward video data captured by the forward camera 11, right side video data captured by the right side camera 12, left side video data captured by the left side camera 13, and rearward video data captured by the rearward camera 14.

The road surface area detection unit 342 detects the road surface area shown in the video data acquired by the video data acquisition unit 341. The road surface area is the area of the road surface shown in the video data. The road surface area is defined by the shape, width, and depth of the road. In this embodiment, the road surface area detection unit 342 detects the road surface area shown in the forward video data. The road surface area detection unit 342 stores the road surface area detected when the vehicle V is traveling on a flat road as a reference road surface area. It is preferable that the reference road surface area is stored for each road width of the road on which the vehicle V is traveling. The method of detecting the road surface may be any known method and is not limited. When the detection result of the headlight illumination detection unit 31 indicates that the headlights are illuminated, the road surface shown in the video data is illuminated by the headlights, so the road surface area detection unit 342 corrects the hue, lightness, saturation, and brightness of the video data to detect the road surface area.

The distortion correction unit 343 uses distortion parameters to correct distortion in the display area, which is a part of the ultra-wide-angle image data acquired by the image data acquisition unit 341, and outputs rectangular distortion-corrected image data. The distortion-corrected image data includes front distortion-corrected image data obtained by correcting front image data, right side distortion-corrected image data obtained by correcting right side image data, left side distortion-corrected image data obtained by correcting left side image data, and rear distortion-corrected image data obtained by correcting rear image data. When it is not necessary to distinguish between the front distortion-corrected image data, right side distortion-corrected image data, left side distortion-corrected image data, and rear distortion-corrected image data, they will be described as distortion-corrected image data.

When the determination unit 35 determines that the tilt angle of the vehicle V is equal to or greater than a threshold, the vehicle speed of the vehicle V is equal to or less than a threshold, and the area of the road surface area has changed by equal to or greater than a threshold, the distortion correction unit 343 changes the distortion parameters to correct the distortion in accordance with the change in the area of the road surface area, and outputs the distortion-corrected image data. More specifically, when the area of the road surface area has increased compared to the reference road surface area, the distortion correction unit 343 changes the distortion parameters to correct the distortion so as to shift the display area for correcting the distortion upward, and outputs the distortion-corrected image data. When the area of the road surface area has decreased compared to the reference road surface area, the distortion correction unit 343 changes the distortion parameters to correct the distortion so as to shift the display area for correcting the distortion downward, and outputs the distortion-corrected image data.

The processing in the distortion correction unit 343 when the vehicle V is traveling on a flat road will be described with reference to Figs. 2 to 5. Fig. 2 is a diagram for explaining a state in which the vehicle is traveling on a flat road. Fig. 3 is a diagram for explaining a state in which the vehicle is traveling on a flat road. Fig. 4 is a diagram for explaining an example of image data captured by the camera when the vehicle is traveling on a flat road. Fig. 5 is a diagram for explaining an example of a display area for which distortion correction processing is performed among the image data shown in Fig. 4. As shown in Figs. 2 and 3, when the vehicle V is traveling on a flat road, the forward image data captured by the forward camera 11 is, for example, as shown in Fig. 4. Such forward image data is set as reference image data, and a display area indicating an area to be visually recognized by the user as an image and distortion parameters are set. In addition, the road surface area detected from this forward image data is the reference road surface area. For example, the display area is set to A1, which is a barrel-shaped area shown in the figure. The distortion parameters are set so that the distortion of such a display area A1 is corrected to generate rectangular forward distortion-corrected image data.

Here, the camera unit 10 including the forward camera 11 captures an image with an ultra-wide viewing angle. For applications in which a wide range is to be viewed, such as generating an image for viewing the surroundings of the vehicle V, it is desirable to correct the distortion so that the full viewing angle can be secured in the horizontal direction. In such a case, if the distortion is corrected while maintaining a natural sense of distance from the front, the display area in the vertical direction, as opposed to the horizontal direction, will appear to be a cut-out portion of the camera's shooting range. Conversely, if the display area in the vertical direction is secured as is and the distortion is corrected, the image will be unnatural in that the sense of distance in front will feel farther than it actually is. Therefore, depending on the inclination angle of the vehicle V, it is necessary to adjust the vertical display area to be displayed. Specifically, the distortion parameters are changed so that the display area for which distortion is corrected is shifted upward or downward from the center of the camera's shooting range according to the increase/decrease ratio of the road surface area.

Using FIG. 6 to FIG. 9, the processing in the distortion correction unit 343 when the vehicle V is traveling on a road with a steep upward slope will be described. FIG. 6 is a diagram illustrating a state in which the vehicle is traveling on a road with a steep upward slope. FIG. 7 is a diagram illustrating a state in which the vehicle is traveling on a road with a steep upward slope. FIG. 8 is a diagram illustrating an example of video data captured by the camera when the vehicle is traveling on a road with a steep upward slope. FIG. 9 is a diagram illustrating an example of a display area in which distortion correction processing is performed, among the video data shown in FIG. 8. As shown in FIG. 6 and FIG. 7, when the vehicle V is traveling on a road with a steep upward slope, the front video data captured by the front camera 11 is, for example, as shown in FIG. 8. Here, it is assumed that there is an intersection that intersects with the cross road R1 in the middle of the steep upward slope. Another vehicle V1 is traveling on the cross road R1. When the vehicle V rises in front, the shooting range of the front camera 11 faces upward. As a result, the road surface area in the display area of the front video data is reduced. Therefore, the distortion parameters are changed so that the position of the road surface area in the display area is appropriate, and the distortion parameters are changed so that the barrel-shaped area A2 shown in the figure becomes the display area. In this way, the distortion is corrected for the display area A2, and forward distortion-corrected image data is output. Note that the display area A1 shown by the dashed line in the figure is the display area when the distortion parameters are not changed. The same applies to the following figures.

Using Figures 10 to 13, the processing in the distortion correction unit 343 when the vehicle V is traveling on a road with a steep downward slope will be described. Figure 10 is a diagram explaining the state in which the vehicle is traveling on a road with a steep downward slope. Figure 11 is a diagram explaining the state in which the vehicle is traveling on a road with a steep downward slope. Figure 12 is a diagram showing an example of video data captured by the camera when the vehicle is traveling on a road with a steep downward slope. Figure 13 is a diagram showing an example of a display area in which distortion correction processing is performed among the video data shown in Figure 12. As shown in Figures 10 and 11, when the vehicle V is traveling on a road with a steep downward slope, the front video data captured by the front camera 11 is, for example, as shown in Figure 12. Here, it is assumed that there is an intersection that intersects with the cross road R1 in the middle of the steep downward slope. Another vehicle V1 is traveling on the cross road R1. When the vehicle V is slanting forward, the shooting range of the front camera 11 faces downward. This increases the road surface area in the display area of the front video data. Therefore, the distortion parameters are changed so that the position of the road surface area in the display area is appropriate, and the distortion parameters are changed so that the barrel-shaped area A3 shown in the figure becomes the display area. In this way, the distortion is corrected for the display area A3, and forward distortion-corrected image data is output.

The image generating unit 344 generates display image data for checking the surroundings of the vehicle V based on the distortion-corrected image data output by the distortion correcting unit 343. In this embodiment, the image generating unit 344 generates display image data that allows checking the area ahead of the vehicle V based on the forward distortion-corrected image data.

The image generating unit 344 may generate display image data of an overhead image including a virtual host vehicle image looking down on the vehicle V from above, based on the distortion-corrected image data output by the distortion correction unit 343. In this case, the image generating unit 344 performs a viewpoint conversion process on the front distortion-corrected image data, the right side distortion-corrected image data, the left side distortion-corrected image data, and the rear distortion-corrected image data, and a synthesis process to synthesize a host vehicle icon representing the vehicle V, and generates display image data of an overhead image showing the surroundings of the vehicle V. The method of generating the overhead image may be any known method, and is not limited.

The determination unit 35 determines whether the inclination angle of the vehicle V and the area of the road surface area have changed relative to a threshold value based on the inclination angle of the vehicle V detected by the inclination angle detection unit 33 and the road surface area detected by the road surface area detection unit 342. More specifically, the determination unit 35 determines whether the inclination angle of the vehicle V is equal to or greater than a threshold value, the vehicle speed of the vehicle V is equal to or less than a threshold value, and the area of the road surface area has changed by equal to or greater than a threshold value based on the inclination angle of the vehicle V detected by the inclination angle detection unit 33, the vehicle speed detected by the vehicle speed detection unit 32, and the road surface area detected by the road surface area detection unit 342. The threshold value of the inclination angle is, for example, about 5°. The threshold value of the vehicle speed is, for example, about 10 km/h. The threshold value of the area of the road surface area is, for example, about 20%.

Whether the area of the road surface area has changed by more than the threshold is determined based on whether the area of the road surface area detected by the road surface area detection unit 342 has increased or decreased by about 20% or more compared to the area of a reference road surface area of a road with the same road width. If the area of the road surface area detected by the road surface area detection unit 342 is determined to have increased or decreased by about 20% or more compared to the area of the reference road surface area, it is determined that the area of the road surface area has changed by more than the threshold. If the area of the road surface area detected by the road surface area detection unit 342 is determined to have not increased or decreased by more than 20% compared to the area of the reference road surface area, it is determined that the area of the road surface area has not changed by more than the threshold.

Alternatively, whether the area of the road surface area has changed by more than a threshold value may be determined based on whether the area of the road surface area detected by the road surface area detection unit 342 has increased or decreased by about 20% or more over a specified period or a specified travel distance. If it is determined that the area of the road surface area detected by the road surface area detection unit 342 has increased or decreased by about 20% or more over a specified period or a specified travel distance, it is determined that the area of the road surface area has changed by more than a threshold value. If it is determined that the area of the road surface area detected by the road surface area detection unit 342 has not increased or decreased by about 20% or more over a specified period or a specified travel distance, it is determined that the area of the road surface area has not changed by more than a threshold value.

When the distortion correction unit 343 changes the distortion parameters, the photometry area control unit 36 controls the position of the photometry area when the camera unit 10 captures images in accordance with the change in the distortion parameters. For example, the position of the photometry area when the camera unit 10 captures images under normal circumstances is the center of the display area. When the distortion correction unit 343 shifts the display area for which distortion is to be corrected upward, the photometry area control unit 36 outputs a control signal to the camera unit 10 to shift the photometry area upward so that the center of the display area shifted upward becomes the photometry area. When the distortion correction unit 343 shifts the display area for which distortion is to be corrected downward, the photometry area control unit 36 outputs a control signal to the camera unit 10 to shift the photometry area downward so that the center of the display area shifted downward becomes the photometry area.

The display control unit 37 controls the display of the display video data generated by the video generation unit 344 of the video processing unit 34 on the display unit 21.

The memory unit 39 stores data required for various processes and the results of various processes in the vehicle image processing device 30. The memory unit 39 is, for example, a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), or a flash memory, or a memory device such as a hard disk, an optical disk, or an external memory device connected via a communication network.

Next, the flow of processing in the vehicle image processing device 30 will be described with reference to FIG. 14. FIG. 14 is a flowchart showing the flow of processing in the vehicle image processing device 30 according to the embodiment. Here, a case will be described in which an image for confirming safety ahead of the vehicle V is displayed based on forward image data captured by the forward camera 11.

The vehicle image processing system 1 is assumed to be activated when the engine of the vehicle V starts. While the vehicle image processing system 1 is activated, the camera unit 10 captures images of the surroundings of the vehicle V. While the vehicle image processing system 1 is activated, the image data acquisition unit 341 acquires image data captured by the camera unit 10.

The vehicle image processing device 30 detects whether the headlights are on (step S101). The vehicle image processing device 30 detects whether the headlights are on or off from the signal output from the headlight switch 22 using the headlight on/off detection unit 31. The vehicle image processing device 30 proceeds to step S102.

The vehicle image processing device 30 detects the road surface area (step S102). The vehicle image processing device 30 detects the road surface area captured in the forward image data using a known road surface detection method by the road surface area detection unit 342. The vehicle image processing device 30 proceeds to step S103.

The vehicle image processing device 30 uses the determination unit 35 to determine whether the vehicle speed is equal to or less than the threshold (step S103). If the determination unit 35 determines that the vehicle speed detected by the vehicle speed detection unit 32 is equal to or less than the threshold (Yes in step S103), the vehicle image processing device 30 proceeds to step S104. If the determination unit 35 determines that the vehicle speed detected by the vehicle speed detection unit 32 is not equal to or less than the threshold (No in step S103), the vehicle image processing device 30 executes the process of step S101 again.

When the vehicle speed detection unit 32 determines that the vehicle speed detected is equal to or less than the threshold (Yes in step S103), the vehicle image processing device 30 determines whether the pitch angle is equal to or greater than the threshold (step S104) using the determination unit 35. When the determination unit 35 determines that the pitch angle detected by the tilt angle detection unit 33 is equal to or greater than the threshold (Yes in step S104), the vehicle image processing device 30 proceeds to step S105. When the determination unit 35 determines that the pitch angle detected by the tilt angle detection unit 33 is not equal to or greater than the threshold (No in step S104), the vehicle image processing device 30 proceeds to step S111.

When the pitch angle detected by the tilt angle detection unit 33 is determined to be equal to or greater than the threshold value (Yes in step S104), the vehicle image processing device 30 determines, by the determination unit 35, whether the area of the road surface area has increased or decreased based on the road surface area detected by the road surface area detection unit 342 (step S105). When the determination unit 35 determines that the area of the road surface area has increased or decreased (Yes in step S105), the vehicle image processing device 30 proceeds to step S106. When the determination unit 35 determines that the area of the road surface area has not increased or decreased (No in step S105), the vehicle image processing device 30 executes the process of step S101 again.

When it is determined that the area of the road surface area has increased or decreased (Yes in step S105), the vehicle image processing device 30 performs a display area correction process (step S106). The vehicle image processing device 30 performs a display area correction process in which the distortion parameters are changed and the distortion is corrected by the distortion correction unit 343 in accordance with the change in the area of the road surface area. When the area of the road surface area has increased, the vehicle image processing device 30 performs a display area correction process in which the distortion parameters are changed and the display area in which the distortion is corrected is shifted upward by the distortion correction unit 343. When the area of the road surface area has decreased, the vehicle image processing device 30 performs a display area correction process in which the distortion parameters are changed and the display area in which the distortion is corrected is shifted downward by the distortion correction unit 343. The vehicle image processing device 30 proceeds to step S107.

The vehicle image processing device 30 generates display image data (step S107). The vehicle image processing device 30 generates the display image data from the forward distortion corrected image data by the image generation unit 344. The vehicle image processing device 30 proceeds to step S108.

The vehicle image processing device 30 causes the display control unit 37 to display the generated display image data on the display unit 21 (step S108). The vehicle image processing device 30 proceeds to step S109.

The vehicle image processing device 30 determines whether the pitch angle is less than the threshold value by the determination unit 35 (step S109). If the determination unit 35 determines that the pitch angle detected by the tilt angle detection unit 33 is less than the threshold value (Yes in step S109), the vehicle image processing device 30 proceeds to step S110. If the determination unit 35 determines that the pitch angle detected by the tilt angle detection unit 33 is not less than the threshold value (No in step S109), the vehicle image processing device 30 executes the process of step S109 again.

When it is determined that the pitch angle detected by the tilt angle detection unit 33 is less than the threshold value (Yes in step S109), the vehicle image processing device 30 returns the display area to the initial setting (step S110). The vehicle image processing device 30 returns the distortion parameters changed by the distortion correction unit 343 in step S106 to the initial values. The vehicle image processing device 30 returns the distortion parameters to the distortion parameters when the vehicle V is traveling on a flat road by the distortion correction unit 343. The vehicle image processing device 30 proceeds to step S113.

When it is determined that the pitch angle detected by the tilt angle detection unit 33 is not equal to or greater than the threshold value (No in step S104), the vehicle image processing device 30 generates display image data (step S111). The vehicle image processing device 30 generates display image data from the image data by the image generation unit 344. The vehicle image processing device 30 proceeds to step S112.

The vehicle image processing device 30 causes the display control unit 37 to display the display image data on the display unit 21 (step S112). The vehicle image processing device 30 proceeds to step S113.

The vehicle image processing device 30 uses the determination unit 35 to determine whether the vehicle speed is greater than the threshold value (step S113). If the determination unit 35 determines that the vehicle speed detected by the vehicle speed detection unit 32 is greater than the threshold value (Yes in step S113), the vehicle image processing device 30 proceeds to step S114. If the determination unit 35 determines that the vehicle speed detected by the vehicle speed detection unit 32 is not greater than the threshold value (No in step S113), the vehicle image processing device 30 executes the process of step S113 again.

If the vehicle speed detection unit 32 determines that the vehicle speed detected is greater than the threshold value (Yes in step S113), the vehicle image processing device 30 stops displaying the display image data (step S114). The vehicle image processing device 30 proceeds to step S115.

The vehicle image processing device 30 determines whether to end the process (step S115). More specifically, the vehicle image processing device 30 determines whether to end the process based on the presence or absence of an end trigger. The end trigger may be, for example, a combination of turning off the engine, putting the gear in park, and applying the handbrake. The end trigger may also be a user operation. If there is an end trigger, the vehicle image processing device 30 determines to end the process (Yes in step S115) and ends the process. If there is no end trigger, the vehicle image processing device 30 determines not to end the process (No in step S115) and executes the process of step S101 again.

In this way, when the determination unit 35 determines that the inclination angle of the vehicle V is equal to or greater than the threshold, the vehicle speed of the vehicle V is equal to or less than the threshold, and the area of the road surface area has changed by equal to or greater than the threshold, the distortion correction unit 343 changes the distortion parameters to correct the distortion in accordance with the change in the area of the road surface area. When the area of the road surface area has increased, the distortion correction unit 343 changes the distortion parameters to correct the distortion so as to shift the display area for correcting the distortion upward. When the area of the road surface area has decreased, the distortion correction unit 343 changes the distortion parameters to correct the distortion so as to shift the display area for correcting the distortion downward.

As described above, in this embodiment, when it is determined that the inclination angle of the vehicle V is equal to or greater than a threshold value, the vehicle speed of the vehicle V is equal to or less than a threshold value, and the area of the road surface area has changed by equal to or greater than a threshold value, the distortion parameters can be changed to correct the distortion in accordance with the change in the area of the road surface area. In this embodiment, when the area of the road surface area increases, the distortion parameters can be changed to correct the distortion so as to shift the display area for correcting the distortion upward. In this embodiment, when the area of the road surface area decreases, the distortion parameters can be changed to correct the distortion so as to shift the display area for correcting the distortion downward. In this way, according to this embodiment, an image of an appropriate display area can be displayed regardless of the inclination angle of the vehicle V.

In this embodiment, when the distortion parameters are changed, the position of the photometry area when the camera unit 10 captures an image can be controlled in accordance with the change in the distortion parameters. According to this embodiment, an appropriate photometry area can be set and appropriate photometry can be performed to capture an image, regardless of the inclination angle of the vehicle V.

In this way, according to this embodiment, the surroundings of the vehicle V can be confirmed in an appropriate image regardless of the inclination angle of the vehicle V.

So far, we have explained the vehicle image processing system 1 according to the present invention, but it may be implemented in various different forms other than the embodiment described above.

Each of the components of the illustrated vehicle image processing system 1 is a functional concept, and does not necessarily have to be physically configured as illustrated. In other words, the specific form of each device is not limited to that shown in the figure, and all or part of the devices may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device.

The configuration of the vehicle image processing system 1 is realized, for example, as software, such as a program loaded into memory. In the above embodiment, these functional blocks are described as being realized by the cooperation of these hardware and software. In other words, these functional blocks can be realized in various forms, using only hardware, only software, or a combination of both.

The above-mentioned components include those that a person skilled in the art would easily imagine and those that are substantially the same. Furthermore, the above-mentioned configurations can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the configurations are possible without departing from the spirit of the present invention.

Using Figures 15 and 16, we will explain the case where the road on which the vehicle V is traveling and the intersecting road are curved. Figure 15 is a diagram showing an example of a display area in which distortion correction processing is performed when the intersecting road is curved. Figure 16 is a diagram explaining a state in which a road intersecting the road on which the vehicle is traveling is curved. When the road on which the vehicle V is traveling and the intersecting road are curved, the distortion correction unit 343 may change the distortion parameter to match the shape of the curve to correct the distortion. The vehicle image processing device 30 has a navigation information acquisition unit that acquires current position information of the current position of the vehicle V and map information around the vehicle V from a navigation system not shown. Furthermore, the vehicle image processing device 30 has a road shape detection unit that detects the shape of the road based on the information acquired by the navigation information acquisition unit. The distortion correction unit 343 changes the distortion parameter to correct the distortion according to the change in the area of the road surface area and the shape of the road detected by the road shape detection unit. The distortion correction unit 343 changes the distortion parameters to shift the right or left side of the display area to be corrected upward or downward so that the display surface area on the curved side can be displayed far away in accordance with the curve shape of the intersecting road, and changes the distortion parameters so that the display area becomes area A4, which is the area of the shape shown in the figure. This makes it possible to display an image of the surroundings of the vehicle V in a display area that matches the curve shape, which is difficult for the user to see directly, when the intersecting road is curved.

Using FIG. 17, we will explain the case where vehicle V is traveling on a snowy road or the like. FIG. 17 is a diagram showing an example of a display area where distortion correction processing is performed when the vehicle is traveling on a snowy road. For example, if the road surface area detection unit 342 cannot detect the road surface due to snow accumulation or flooding, the distortion parameters are changed so that a wider range is set as the display area, and the distortion parameters are changed so that A5, which is an area with the shape shown in the figure, becomes the display area. As a result, when it is difficult to detect the road surface area due to snow accumulation or flooding, it is possible to display an image of the surroundings of vehicle V in a display area where the area that is set as the road surface is wider than usual.

A case will be described where an image for checking the surroundings of the vehicle V is displayed when parking. In this case, when it is determined that there is a reverse trigger, the vehicle image processing device 30 executes the process of the flowchart shown in FIG. 14. A reverse trigger refers to, for example, the shift position being set to "reverse" or the traveling direction of the vehicle V becoming rearward in the fore-and-aft direction of the vehicle V. Then, in step S102, the road surface area captured in the rear image data is detected, and in step S105, it is determined whether the area of the road surface area in the rear image data has increased or decreased. Then, in step S106, the distortion parameters are changed to correct the distortion of the rear image data. In this way, when parking, an image of the rear of the vehicle V in an appropriate display area can be displayed regardless of the inclination angle of the vehicle V.

The vehicle image processing system 1 can also be applied to a so-called drive recorder that captures and records surrounding images while the vehicle V is traveling. In this case, regardless of the vehicle speed of the vehicle V, the determination unit 35 determines whether the inclination angle of the vehicle V is equal to or greater than a threshold and the area of the road surface area has changed by equal to or greater than a threshold based on the inclination angle of the vehicle V detected by the inclination angle detection unit 33 and the road surface area detected by the road surface area detection unit 342. In addition, when the determination unit 35 determines that the inclination angle of the vehicle V is equal to or greater than a threshold and the area of the road surface area has changed by equal to or greater than a threshold, the distortion correction unit 343 corrects the distortion by changing the distortion parameter according to the change in the area of the road surface area. The vehicle image processing device 30 executes processes other than steps S103, S113, and S114 of the flowchart shown in FIG. 14. Furthermore, when applied to a drive recorder, an external recording device including, for example, a card-type recording medium for storing image data may be provided outside the vehicle image processing device 30 in addition to the storage unit 39.

The vehicle image processing device 30 may have an optical axis control unit that controls the position of the optical axis of the headlights in accordance with the change in the distortion parameter when the distortion correction unit 343 changes the distortion parameter. This allows an appropriate image of the surroundings of the vehicle V to be displayed.

Instead of the above-mentioned three-axis gyro sensor 24, a six-axis sensor having the functions of a three-axis acceleration sensor capable of detecting acceleration in the X-, Y-, and Z-directions and a three-axis gyro sensor may be used.

REFERENCE SIGNS LIST 1 Vehicle image processing system 10 Camera unit 21 Display unit 22 Headlight switch 23 Vehicle speed sensor 24 Three-axis gyro sensor 30 Vehicle image processing device 31 Headlight illumination detection unit 32 Vehicle speed detection unit 33 Inclination angle detection unit 34 Image processing unit 341 Image data acquisition unit 342 Road surface area detection unit 343 Distortion correction unit 344 Image generation unit 35 Determination unit 36 Photometric area control unit 37 Display control unit 39 Memory unit

Claims (3)

a video data acquisition unit that acquires video data captured by an image capture unit that captures images of the surroundings of the vehicle;
a distortion correction unit that corrects distortion of the image data acquired by the image data acquisition unit using a distortion parameter and outputs distortion-corrected image data;
a road surface area detection unit that detects a road surface area captured in the video data acquired by the video data acquisition unit;
Equipped with
when the road surface area detection unit cannot detect a road surface, the distortion correction unit corrects the distortion by changing the distortion parameters so that a display area is wider than a display area in a case where the distortion parameters are not changed.
2. A vehicle image processing device comprising: when the road surface area detection unit cannot detect the road surface, the distortion correction unit corrects the distortion by changing the distortion parameters so as to expand the display range in which the distortion is corrected to the upper and lower sides;
The vehicle image processing device according to claim 1 . A video data acquisition step of acquiring video data captured by an image capturing unit that captures images of the periphery of the vehicle;
a distortion correcting step of correcting distortion of the image data acquired in the image data acquiring step by using a distortion parameter, and outputting distortion-corrected image data;
a road surface area detection step of detecting a road surface area captured in the image data acquired in the image data acquisition step;
Including,
In the distortion correction step, when a road surface cannot be detected in the road surface area detection step, the distortion parameters are changed to correct the distortion so that a display area is wider than a display area in a case where the distortion parameters are not changed.
Image processing method for vehicles.

Images