JP7426174B2 - Vehicle surrounding image display system and vehicle surrounding image display method - Google Patents

Description

The present invention relates to a vehicle surrounding image display system and display method for displaying images of the surroundings of a vehicle such as an automobile, and more specifically, the present invention relates to a vehicle surrounding image display system that displays images of the surroundings of a vehicle such as an automobile, and a method for displaying the same. The present invention relates to a vehicle surrounding image display system and a vehicle surrounding image display method that can display surrounding images including the area directly below the own vehicle without blind spots as the own vehicle moves when synthesizing and displaying overhead images.

The range that a driver who drives a vehicle can directly see is limited. In particular, there are many blind spots at the rear of the vehicle that cannot be seen from the driver's seat even if a general rearview mirror or door mirror is used. For example, a portion of the driver's field of vision when looking toward the rear of the vehicle is obstructed by the trunk or rear pillar. Therefore, conventionally, a device has been proposed that assists the driver in checking the surroundings by mounting a camera on a vehicle and displaying the captured image on a display.

For example, in the technology disclosed in Patent Document 1, captured images obtained by a plurality of cameras that capture images of the surroundings of a vehicle are processed, and a display screen of a display means disposed at a position where the driver can see through the rearview mirror is used. It is proposed to display a composite image that also includes images of blind spots around the vehicle. Furthermore, the technique disclosed in Patent Document 2 discloses a technique for superimposing and displaying a past image photographed by a photographing means in the past when a parking lot line detected from a photographed image is incomplete. ing.

However, the technology of Patent Document 1 cannot reflect the situation near the bottom or outer circumference of the vehicle, so it is necessary to check the position of tires and obstacles, and the positional relationship between tires and grooves using images. I couldn't. Furthermore, the technique disclosed in Patent Document 2 replaces parts where the image is not clear with images taken in the past, and therefore cannot reflect the current situation.

Japanese Patent Application Publication No. 2006-135797 Japanese Patent Application Publication No. 2006-311299

The present invention has been made in view of the above-mentioned conventional problems, and an object of the present invention is to display a vehicle surrounding image in which the situation near the lower part of the vehicle and the outer circumference, which is a blind spot caused by the own vehicle, is visualized. An object of the present invention is to provide a vehicle surrounding image display system that optimizes the display range of vehicle surrounding images without blind spots according to the movement of the vehicle.

A vehicle surrounding image display system according to one aspect of the present invention, which has been made to achieve the above object, is a vehicle surrounding image display system that displays a vehicle surrounding image synchronized with the progress of the vehicle, and wherein the image display system a photographing means disposed in the photographing means for photographing the surroundings of the vehicle; an image storage means for storing the image photographed by the photographing means; an image processing means for processing the photographed image; and an image processing means for processing the photographed image; and an image display means for displaying an image, wherein the image processing means complements a blind spot portion of the current image photographed by the photographing means with a history image generated from an image photographed before the current image. The present invention is characterized in that a vehicle surroundings image is generated, and the image display means displays a transparent image of the own vehicle stored in the image storage means superimposed on the generated vehicle surroundings image.

The photographing means includes one or more wide-angle cameras capable of photographing the vicinity of the rear or front of the own vehicle, and transmits the image photographed by the wide-angle camera to the image processing means in frame units, and the image processing means can convert the received image into an overhead image in continuous frame units, and synthesize the converted overhead images to generate a history image.
The image display system further includes a vehicle position detecting unit that calculates the orientation, moving direction, and moving distance of the own vehicle based on the captured image, and the image processing unit is configured to detect that the moving distance of the vehicle is a predetermined distance. Each time the vehicle reaches a travel distance of may be adjusted frame by frame to update the historical image.
The image processing means compares the captured current image with the generated history image as the vehicle progresses, and if the captured current image has a portion different from the historical image, , a vehicle surrounding image in which images of the different portions are highlighted can be generated.
The image processing means combines the photographed current image and the current image to be combined based on the orientation, movement direction, and movement distance of the own vehicle calculated by the own vehicle position estimation means in accordance with the progress of the vehicle. The image taken before the current time may be cropped, or an image compositing position may be set.

A vehicle surrounding image display method according to an aspect of the present invention, which has been made to achieve the above object, is a method for displaying a vehicle surrounding image synchronized with the progress of the vehicle , and is a method that is arranged in a vehicle and displays the surroundings of the vehicle. A vehicle comprising a photographing means for photographing, an image storage means for storing an image photographed by the photographing means, an image processing means for processing the photographed image, and an image display means for displaying the image processed. In the surrounding image display system, the image processing means receives an image photographed by the photographing means, converts the received image into an overhead image in continuous frame units, and synthesizes the converted overhead images. a step of generating a vehicle surroundings image by supplementing the blind spot portion of the current image photographed by the photographing means with the generated history image; The present invention is characterized by comprising the step of superimposing and displaying a transparent image of the own vehicle stored in the image storage means.

The step of generating the history image may include the step of extracting feature points from the continuous bird's-eye view images in units of frames and adjusting a synthesis position so that the extracted feature points match.
The vehicle surrounding image display method further includes a step in which the image processing means calculates an orientation, a moving direction, and a moving distance of the own vehicle based on the captured image, and the step of generating a history image includes , Every time the calculated moving distance of the vehicle reaches a predetermined moving distance, the captured current image and the current image are calculated based on the calculated orientation, moving direction, and moving distance of the own vehicle. The method may also include the step of updating the historical image by adjusting the position at which it is combined with a previously captured image for each frame.
The step of displaying the transparent images in an overlapping manner includes comparing the captured current image with the generated history image in accordance with the progress of the vehicle, and adding the historical image to the captured current image. If there is a different portion, the method may include a step of emphasizing and displaying an image of the different portion.
The step of generating the vehicle surrounding image includes combining the photographed current image and the current image, which are combined based on the calculated orientation, moving direction, and moving distance of the own vehicle according to the progress of the vehicle. The method may include a step of cropping an image taken earlier or setting an image compositing position.

According to the present invention, it is possible to display a vehicle surrounding image that visualizes the situation near the lower part of the vehicle and the outer periphery, which is a blind spot caused by the own vehicle, and furthermore, the blind spot can be displayed in which the display range is optimized in synchronization with the progress of the vehicle. It is possible to display images of the surroundings of the vehicle.
In addition, the current image taken and the image taken before the current one are calculated based on the orientation, direction of movement, and distance of the own vehicle, which are calculated each time the distance traveled by the vehicle reaches a predetermined distance. The historical image is updated by adjusting the compositing position for each frame, reducing the amount of processing and reducing the image processing load compared to generating a composite image using all acquired frames. At the same time, it is possible to reduce the accumulation of slight errors that occur during frame synthesis, that is, the cumulative error.

1 is a diagram showing the overall configuration of a vehicle surrounding image display system according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a camera shooting range of the vehicle surrounding image display system according to the present embodiment. FIG. 1 is a block diagram showing an example of the functional configuration of a vehicle surrounding image display system according to the present embodiment. 2 is a flowchart illustrating an image processing procedure in which the vehicle surrounding image display system according to the present embodiment acquires and displays a vehicle surrounding image. FIG. 3 is a diagram showing an example of processed images at each stage of image processing by the vehicle surrounding image display system of the present embodiment. FIG. 2 is a diagram showing an example of a vehicle surrounding image displayed by the vehicle surrounding image display system of the present embodiment. FIG. 7 is a diagram showing an example of an image generated by additional processing of image processing in the present embodiment. It is a figure which shows the example of an image produced|generated by the other additional process of the image processing in this embodiment. FIG. 7 is a conceptual diagram showing another example of image processing of the vehicle surrounding image display system according to the present embodiment. FIG. 7 is a conceptual diagram showing still another image processing example of the vehicle surrounding image display system according to the present embodiment.

Hereinafter, specific examples of modes for carrying out the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing the overall configuration of a vehicle surrounding image display system according to an embodiment of the present invention. As shown in FIG. 1, a vehicle surrounding image display system 1 according to the present embodiment includes a camera (photographing means) 3 installed at the rear of a vehicle 2, an in-vehicle monitor device (image display device) consisting of a liquid crystal display, etc. disposed inside the vehicle. 5, an image processing ECU (image processing means) 6, a vehicle information DB 7, a driving image DB (image storage means) 8, a vehicle ECU 9, and the like. Note that it is also possible to install a camera at the front of the vehicle 2 and take images of the surroundings in front and behind. It is also possible to install cameras (not shown) on the left and right sides of the vehicle 2, and use the four cameras to perform a display including a conventional around-view image. Below, in order to simplify the explanation, an example will be described in which the device is installed at the rear.

FIG. 2 is a diagram showing an example of a camera photographing range of the vehicle surrounding image display system according to the present embodiment. In this embodiment, the camera 3 uses a solid-state imaging device such as a CCD or CMOS, and is mounted near the rear center of the vehicle 2, as shown in FIG. It will be set up towards. Then, a photograph is taken of the rear of the vehicle along the traveling direction (that is, a moving direction) of the vehicle 2, and the photographed image (hereinafter referred to as a photographed image) is output as an image signal. The image signal output from the camera is converted into image data by an image processing means, which will be described later, and is displayed on the in-vehicle monitor device 5 as a vehicle surrounding image, for example, a bird's-eye view image, after undergoing predetermined image processing. In this embodiment, the camera 3 is equipped with a wide-angle lens and is capable of photographing a range of about 110 degrees to 140 degrees left and right, but the camera 3 is not limited thereto.

The in-vehicle monitor device 5 is installed on the center console or panel surface inside the vehicle 2, and displays a composite image generated from images taken by the camera 3. Note that the in-vehicle monitor device 5 may also be used as a navigation device.

An image processing ECU (Electronic Control Unit) 6 is an electronic control unit that performs various image processing on image data of an image photographed by the camera 3. In the present embodiment, the first image processing performed by the image processing ECU 6 is to perform coordinate transformation on the image data of the photographed image to generate an overhead image, and store it in the traveling image DB 8 in units of frames. The second image processing detects characteristic parts of the road, including marking lines and markings on the route on which the vehicle 2 is traveling, from image data for each frame, and utilizes the detected marking lines and markings. Performs time-series image synthesis processing by extracting feature points and searching for matching points. Furthermore, as a third image process, an overhead image is generated that complements the blind spot area caused by the own vehicle in synchronization with the progress (that is, movement) of the own vehicle, and image data in which a transparent image of the own vehicle is superimposed on this is generated. generate. Note that the detailed configuration of the image processing ECU 6 will be described later.

The vehicle information DB (database) 7 is a storage means in which various parameters regarding the vehicle 2 and the camera 3 are stored. The vehicle information DB 7 includes, for example, information regarding the vehicle model such as the minimum turning radius of the vehicle 2, vehicle width, tread length, and tire radius, as well as information about the camera 3 such as internal parameters of the camera 3, optical axis direction, angle of view, and installation position. Information regarding design values is stored.

In the traveling image DB 8, image data of an image photographed by the camera 3 is stored in frame units in association with additional information such as the camera position of the camera 3 at the time of photographing, the photographing direction angle, the vehicle position, and the direction of the vehicle. It is a storage means. In this embodiment, an SSD (solid state drive) is used as the storage device for the traveling image DB8, but a magnetic disk, a memory card, a DVD, etc. can also be used as an external storage device.

The vehicle ECU 9 is an electronic control unit of the vehicle 2 that controls operations of the engine, transmission, brakes, and the like. Various sensors such as a vehicle speed sensor 11, a gyro sensor 12, a steering angle sensor 13, and a shift position detection sensor 14 are connected to the vehicle ECU 9. The image processing ECU 6 can calculate the vehicle speed, acceleration, steering angle, direction of the vehicle, travel distance, etc. of the vehicle 2 from the output values of these sensors.

Next, the functional configuration of the vehicle surrounding image display system 1 according to this embodiment will be described with reference to FIG. 3. FIG. 3 is a block diagram showing an example of the functional configuration of the vehicle surrounding image display system according to this embodiment.

As shown in FIG. 3, the vehicle surrounding image display system 1 according to the present embodiment is configured based on functional means realized by a vehicle surrounding image generation program executed by the image processing ECU 6. Predetermined peripheral devices are connected.

The image processing ECU 6 includes an image input section 61, a vehicle signal input section 62, a calculation section 63, and an image output section 64.

The image input unit 61 is an input interface for acquiring image data of an image around the vehicle captured by the camera 3. The image input unit 61 includes an A/D converter, converts the image signal output from the camera 3 into a digital signal, and generates image data in units of frames. Note that if the camera 3 is of a type that outputs a digital image signal, the A/D converter can be omitted. The image input unit 61 can also acquire images as continuous images (videos), but in this embodiment, as the vehicle moves, the image input unit 61 acquires still images for each moving distance corresponding to the shooting range of the camera 3. Images are acquired in units of screens, that is, in units of frames. Note that the interval at which images are acquired is preferably equal to or less than the image length in the moving direction (the imaging range indicated by the symbol L in FIG. 2) so that a portion that will serve as a glue margin when pasting adjacent frames is obtained. .

The vehicle signal input unit 62 is an interface for acquiring data related to vehicle movement detected by various sensors (hereinafter referred to as movement amount data). The vehicle signal input unit 62 acquires movement amount data from the vehicle ECU 9, for example, via the in-vehicle LAN.

The calculation unit 63 is composed of a well-known microcontroller including a CPU, RAM, ROM, and the like. Note that the ROM stores an image processing program, which will be described later, and various other programs necessary for controlling the in-vehicle monitor device 5 and the like. Further, the RAM is a memory that temporarily stores various data calculated by the CPU.

Every time image data is acquired frame by frame from the image input unit 61, the calculation unit 63 operates each functional unit described below to combine the image data of the currently acquired captured image with the previously acquired image. Vehicle surrounding image data is generated by combining the image data of the historical image based on the image data, and the generated vehicle surrounding image data is output to the image output section 64.

The image output unit 64 sequentially outputs the vehicle surrounding image data output from the calculation unit 63 to the in-vehicle monitor device 5 in accordance with the movement of the vehicle. This allows occupants in the vehicle, such as the driver, to view images around the vehicle displayed on the in-vehicle monitor device 5.

The calculation unit 63 includes an image conversion unit 63a, a feature point extraction unit 63b, an own vehicle position estimation unit 63c, a history image storage unit 63d, a moving distance estimation unit 63e, as functional units realized by executing a vehicle surrounding image generation program. The in-vehicle monitor device includes a steering angle estimation section 63f, a vehicle movement amount calculation section 63g, an image composition section 63h, and an output image generation section 63i, and based on various information acquired from each peripheral device via the driving image DB 8 and the vehicle ECU 9. Control is performed so that the captured image (see FIG. 5) is displayed.

When the image conversion unit 63a receives image data in units of frames from the image input unit 61, the image conversion unit 63a corrects image distortion caused by wide-angle photography on the received image data. Thereafter, coordinate transformation is performed on the image data that has been subjected to distortion correction, and processing is performed to generate image data of an overhead image. Then, the image conversion unit 63a stores the image data of the generated bird's-eye view image in the running image DB8 in units of frames. As a result, image data of past captured images is accumulated. Here, as a technique for converting into an overhead image, for example, the conventional technique disclosed in Japanese Patent Application Laid-Open No. 10-211849 can be used.

The feature point extraction unit 63b extracts a target frame-by-frame image (hereinafter referred to as frame Extracts feature points that exist within the screen of an image (referred to as an image). A feature point is a point that can be followed when a frame screen moves, and generally a point that has a difference in brightness in both the vertical and horizontal directions within each frame image is extracted as a feature point. That is, feature points can be detected by image processing such as edge processing, and various techniques (SIFT, A-KAZE, SURF, etc.) have been proposed as known processing methods for detecting feature points. Feature points are extracted using these processing methods.

The own vehicle position estimating unit 63c compares the feature points of the current frame image converted to the bird's-eye view image with the feature points of the previous frame image (accumulated bird's-eye view image) stored in the traveling image DB 8. to estimate the vehicle's position. Specifically, image data corresponding to the immediately previous frame is read from the running image DB 8, and feature points are selected from among the feature points in the read immediately previous frame image and the feature points in the current frame image. Then, the position of the selected feature point within the frame image is estimated, and the selected feature point is searched within the image using the estimated value of the position within the frame image as the center. Through this search, a deviation between the position in the image assumed from the previous position and the position in the image actually detected is detected. After that, the own vehicle position estimating unit 63c corrects the estimated value of the own vehicle position based on the deviation between the feature point position in the frame image and the feature point position detected from the position immediately before the own vehicle position. Calculate. These processes utilize SLAM (Simultaneous Localization and Mapping) technology.

The history image storage unit 63d converts the image data of each frame converted into an overhead image into the feature points obtained by the feature point detection unit 31 and the own vehicle position estimation for each adjacent frame based on the vehicle movement amount information. The information regarding the own vehicle position estimated by the unit 63c is associated with the information and stored in the traveling image DB8 as historical image data.

When the vehicle signal input unit 62 acquires the vehicle travel distance data, the travel distance estimating unit 63e determines the next timing for the image input unit 61 to acquire a photographed image based on the travel distance included in the acquired travel distance data. presume. That is, it is determined whether the distance traveled by the vehicle has reached a distance corresponding to the frame size photographed by the camera 3 (the photographing range L shown in FIG. 2). When the moving distance estimating unit 63e determines that the moving distance of the vehicle has reached a preset value (for example, 90% of the length of the photographing range L), the moving distance estimating unit 63e sends an image to the image input unit 61 to determine whether or not to take a photograph at that time. A signal for acquiring a frame image (hereinafter referred to as an image acquisition signal) is transmitted. The image acquisition signal is also transmitted to the steering angle estimator 63f.

When the vehicle signal input section 62 acquires vehicle movement amount data, the steering angle estimating section 63f determines the steering angle data included in the acquired movement amount data, and upon receiving the image acquisition signal from the movement distance estimating section 63e, The steering angle of the vehicle at the time when the image input unit 61 acquires the photographed image is transmitted to the vehicle movement amount calculation unit 63g.

The vehicle movement amount calculating section 63g uses the image input section 61 to calculate the current state based on data on the movement distance and steering angle for a predetermined time period including the current period acquired by the movement distance estimating section 63e and the steering angle estimating section 63f. Calculating how the vehicle moved during a predetermined period of time including: Each of the calculated data is stored in the traveling image DB 8 in association with the frame-by-frame image data acquired by the image input unit 61, and is used when adjusting the range in which the frame image is cropped (or trimmed). be done.

When the image input section 61 acquires the image data of the photographed image, the image synthesis section 63h converts the latest photographed image into an actual image. Further, time-series frame image data acquired during a predetermined time period before the current time is read out from the traveling image DB 8 and arranged in accordance with the movement of the vehicle. Based on the calculation result of the movement content of the vehicle calculated by the vehicle movement amount calculation unit 63g, the position of the current photographed image of the vehicle and the position of the previous frame image read from the traveling image DB 8 are calculated for each frame. The synthesis position is adjusted so that the extracted feature points match, and historical image data is generated. The image data of the vehicle surroundings is synthesized to generate image data of a vehicle surrounding image.

The output image generation section 63i superimposes the transparent image data of the own vehicle on the image data of the generated vehicle surrounding image and sequentially outputs the superimposed image data to the image output section 64. Furthermore, when an object detected by another sensor or a difference from the historical image is detected in the currently captured image, the output image generation unit 63i performs a preset emphasis process and generates the supplemented image data. The image is transmitted to the image output section 64.

The image output section 64 causes the in-vehicle monitor device 5 to display the image data of the vehicle surrounding image received from the output image generation section 63i.

Next, with reference to FIGS. 4 and 5, the operation of the vehicle surrounding image display system according to an embodiment of the present invention will be described. FIG. 4 is a flowchart showing an image processing procedure in which the vehicle surrounding image display system according to the present embodiment acquires and displays a vehicle surrounding image. FIG. 5 is a diagram showing an example of processed images at each stage of image processing by the vehicle surrounding image display system of this embodiment. In this embodiment, a case will be described in which the vehicle is moved backward.

As shown in FIG. 4, when this system is activated, the camera 3 photographs the road surface behind the vehicle and its surroundings, and the image processing ECU 6 controls each functional means to perform a series of image processing as shown below. Execute.

First, the image processing ECU 6 acquires image data of the surrounding image behind the vehicle captured by the camera 3 in units of frames synchronized with the movement of the vehicle via the image input unit 61, and stores it in the running image DB 8. (Step S101). At this time, the image data of the frame image stored in the traveling image DB 8 is image data that has been converted into an overhead image in units of frames by the image conversion unit 63a as described above.

Specifically, the original image (camera image) output from the camera 3 is, for example, as shown in image (a) in FIG. Since this is an image taken of the drawn white line (department line or sign) and its surroundings, the white line etc. may be distorted more than the original state depending on the distance between the own vehicle (i.e. camera 3) and the position of the white line. is displayed. Therefore, the image conversion unit 63a performs image distortion correction due to wide-angle photography, as shown in image (b) of FIG. Image data is generated.

Next, the image data of the bird's-eye view image sequentially acquired frame by frame in synchronization with the movement of the vehicle is used by the feature point extraction unit 63b to search for matching points with the image data of the immediately previous frame. The feature points are extracted. For example, as shown in image (d) of FIG. 5, a diamond mark detected in the image of frame N acquired at time (t) and a frame (N+1) acquired at time (t+1) immediately after that. The diamond-shaped mark detected in the image is extracted as a feature point (step S102).

Thereafter, the vehicle position estimation unit 63c uses the extracted feature points to search for matching points between the two images, frame (N) and frame (N+1), as shown in image (e) in FIG. be done. Further, the own vehicle position estimating unit 63c compares the positions of the feature points selected by the matching point search in each frame image to detect a shift between the two images. The own vehicle position estimating unit 63c calculates the orientation, moving direction, and moving distance of the own vehicle from the deviation of the feature point positions (step S103). The SLAM technology is used for these processes as described above. Various algorithms have been announced regarding SLAM, including Mono-SLAM and PTAM.

Next, the image synthesis unit 63h sets an image boundary with fewer discontinuous points, as shown in image (f) in FIG. 5, based on the positions of the selected feature points, etc. The synthesis position is adjusted so that the feature points of the image data of the immediately previous frame N and the frame (N+1) of the image data of the current surroundings of the vehicle match, and the image data as shown in image (g) in Fig. 5 is obtained. , images are combined (step S104). In a similar procedure, the image data of the immediately previous frame N and the image data of the previous frame (N-1) are combined to generate image data of a history image in which a plurality of frames are combined. In addition, when composing these images, the image compositing unit 63h determines the frame range of the historical image based on the movement amount calculated by the vehicle movement amount calculation unit 63g, as shown in image (h) in FIG. The image to be combined may be shaped by performing processing such as appropriately cropping the image.

Thereafter, the output image generation unit 63i generates an image in which a transparent image of the own vehicle is superimposed on a vehicle surrounding image obtained by combining the current vehicle surrounding image and the history image, as shown in image (i) of FIG. is displayed on the in-vehicle monitor device 5 (step S105).

FIG. 6 is a diagram showing an example of a vehicle surrounding image displayed by the vehicle surrounding image display system of this embodiment. Through the above-described operation, after this system is activated, the in-vehicle monitor device 5 displays an image in which the blind spot portion of the currently captured image is supplemented with the history image generated in synchronization with the movement of the vehicle. That is, in contrast to the image (A) when the vehicle starts to move backward at time (t) in FIG. 6, the images (B) and (C) at subsequent times (t+i1) and (t+i2) respectively The blind spots of the currently captured image are complemented and displayed with the history image generated from previously captured images. Note that the image range displayed on the in-vehicle monitor device 5 is adjusted so that the current position of the own vehicle is placed in the center of the display screen after startup.

Other examples of image processing by the vehicle surrounding image display system of this embodiment will be described below.

FIG. 7 is a diagram showing an example of an image generated by additional processing of image processing in this embodiment. In the image example shown in FIG. 7, a predetermined functional unit (not shown) included in the calculation unit 63 compares the currently captured image and the historical image when combining the currently captured image and the historical image. If there is a different part in the currently captured image that does not exist in the historical image, it is determined whether the detected different part is an obstacle by image recognition using a predetermined method. If an obstacle exists in the currently captured image and there is no obstacle in the previously captured image, the output image generation unit 63i generates image data in which the obstacle is highlighted. As a result, the obstacle is highlighted on the in-vehicle monitor device 5, thereby attracting the driver's attention and making it possible to reduce driving mistakes.

FIG. 8 is a diagram showing an example of an image generated by further additional processing of the image processing in this embodiment. In the image example shown in FIG. 8, when an approaching moving object is detected by another sensor installed in the vehicle, the output image generation unit 63i adds a blind spot to the currently captured image based on the detected information. A vehicle surrounding image is generated in such a manner that the predicted course of the moving object calculated by a predetermined functional unit (not shown) provided in the image processing ECU 6 is superimposed and displayed together with the supplemented history image. This makes it possible to alert the driver to the moving object.

Above, the image processing procedure in the vehicle surrounding image display system according to an embodiment of the present invention has been described for the case where the vehicle is moved backward, but the present invention also applies to the case where the vehicle is moved forward. It is possible to implement.

FIG. 9 is a conceptual diagram showing another image processing example of the vehicle surrounding image display system according to this embodiment. The image processing example shown in FIG. 9 shows how a camera 3' is also installed at the front of the vehicle, and a history image of the surroundings of the vehicle is generated when the vehicle moves forward. In this case as well, according to the image processing procedure shown in Fig. 4, a history image is generated frame by frame as the vehicle progresses (that is, movement), and the current captured image around the vehicle is supplemented with the history image. By generating the image, it is possible to display a vehicle surrounding image in which the blind spot including the lower part of the vehicle is complemented.

FIG. 10 is a conceptual diagram showing still another image processing example of the vehicle surrounding image display system according to this embodiment. The image processing example shown in FIG. 10 is a method for generating a vehicle surrounding image displayed on an in-vehicle monitor device when generating a vehicle surrounding image using images taken by cameras installed at the front and rear of the vehicle. FIG. 6 is a diagram showing an example according to the passage of time from the start of the operation. Generation of the vehicle surrounding image is the same as the procedure described above, so a description thereof will be omitted.

As explained above, the vehicle surrounding image display system according to the present invention sequentially visualizes and updates images of blind spots in synchronization with the progress of the vehicle, thereby finally displaying vehicle surrounding images without blind spots. is possible. As a result, it is possible to comfortably check the normal vehicle surrounding image that displays real-time images and the blind spot image that is visualized and updated (the blind spot images are sequentially generated and combined). In other words, it is possible to check the image of the ground below the vehicle, which is normally invisible, and the position of obstacles. In particular, since images without blind spots are generated in synchronization with the direction of movement of the vehicle (that is, the direction of movement), there is no sense of discomfort.

Furthermore, by using camera images from the front and rear of the vehicle, a pseudo vehicle surrounding image without blind spots can be generated. An image of a blind spot can be generated regardless of whether the vehicle is moving forward or backward, and by combining the front and rear images, a pseudo vehicle surrounding image can be generated using two cameras.

Since the surrounding image is widely photographed and a display image is generated using a part of the photographed image, even when the steering angle of the steering wheel is large, the display image can be generated without missing any blind spots, which is highly convenient. Compared to processing the entire screen, processing is performed using the necessary partial images, so the processing load can be reduced.

In the vehicle surrounding image display system according to the present invention, when the steering angle of the steering wheel is greatly reduced, the image range required to generate a blind spot image becomes wider. After a slightly wider range is cropped to generate a composite image, a blind spot image is generated. As a result, even if the displayed range becomes partially wider than the previously displayed range due to a large steering angle, the image can be displayed without being missing.

The system compares images taken in the past with the current image in a parking lot, etc., and if there are any differences in the blind spot generated image, that location is highlighted. This can lead to distractions when driving in familiar locations, but by highlighting parts that are different from the past, it is possible to reduce driving mistakes.

Based on the moving object detection results (speed, direction, etc.) in the captured image, an estimated course map of objects that may fall within the blind spot is displayed. Since the blind spot display part uses past images, if a moving object enters the blind spot, the position of the moving object cannot be displayed in real time, but by displaying the predicted course, it is possible to alert people to the moving object. .

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the present invention is not limited to the above-described embodiments, and can be modified in various ways without departing from the technical scope of the present invention. It is possible to do so.

1 Vehicle surrounding image display system 2 Vehicle 3, 3' camera 5 In-vehicle monitor device 6 Image processing ECU
7 Vehicle information DB
8 Driving image DB
9 Vehicle ECU
11 Vehicle speed sensor 12 Gyro sensor 13 Steering angle sensor 14 Shift position detection sensor 61 Image input section 62 Vehicle signal input section 63 Arithmetic section 63a Image conversion section 63b Feature point extraction section 63c Own vehicle position estimation section 63d History image accumulation section 63e Travel distance Estimation unit 63f Rudder angle estimation unit 63g Vehicle movement amount calculation unit 63h Image composition unit 63i Output image generation unit 64 Image output unit

Claims (8)

A vehicle surrounding image display system that displays a vehicle surrounding image synchronized with the progress of the vehicle, the system comprising:
The vehicle surrounding image display system includes:
a photographing means disposed in a vehicle to photograph the surroundings of the vehicle;
an image storage means for storing images taken by the photographing means in frame units for each moving distance corresponding to a photographing range of the photographing means in chronological order in frame units in synchronization with the movement of the vehicle;
image processing means for image processing the image acquired in frame units for each moving distance corresponding to a photographing range of the photographing means ;
an image display means for displaying a vehicle surrounding image generated by combining the image-processed images in units of frames;
The image processing means includes:
Receiving the image taken by the photographing means and acquired in frame units for each moving distance corresponding to a photographing range of the photographing means , and converting the received image into a time-series continuous bird's-eye view image in frame units. Converted,
generating a history image by composing a plurality of frames of the converted frame-by-frame overhead image arranged in synchronization with the movement of the vehicle;
generating a vehicle surrounding image in which a blind spot portion of a current image photographed by the photographing means and acquired frame by frame for each moving distance corresponding to a photographing range of the photographing means is complemented with the generated history image;
The image display means sequentially displays a transparent image of the own vehicle stored in the image storage means superimposed on the generated vehicle surrounding image in synchronization with the progress of the vehicle ,
The vehicle surrounding image display system includes:
further comprising own vehicle position estimating means for calculating the orientation, moving direction, and moving distance of the own vehicle based on the image acquired in frame units for each moving distance corresponding to the photographing range of the photographing means,
The image processing means determines the current state of the vehicle based on the calculated orientation, direction of movement, and distance of the own vehicle each time the distance of movement of the vehicle reaches a distance of movement corresponding to the photographing range of the photographing means. The history image is updated by adjusting, for each frame, a compositing position of the current bird's-eye view image obtained by converting the image of Features a vehicle surrounding image display system. 2. The camera according to claim 1, wherein the photographing means includes one or more cameras capable of photographing the vicinity of the rear or front of the own vehicle, and transmits the image photographed by the camera to the image processing means. Vehicle surrounding image display system. The image processing means compares the current bird's-eye view image with the generated history image in accordance with the progress of the vehicle, and if the current bird's-eye view image has a different portion from the history image, the image processing means compares the current bird's-eye view image with the generated history image, The vehicle surrounding image display system according to claim 1 , wherein the vehicle surrounding image display system generates a vehicle surrounding image in which a portion of the image is highlighted. The image processing means combines the current bird's-eye view image and the current image based on the orientation, moving direction, and moving distance of the own vehicle calculated by the own vehicle position estimating means in accordance with the progress of the vehicle. 2. The vehicle surrounding image display system according to claim 1, further comprising: cropping a frame-by-frame bird's-eye view image photographed earlier than the above image, or setting an image compositing position. A method for displaying a vehicle surrounding image synchronized with the progress of the vehicle, the method comprising:
a photographing means disposed in a vehicle to photograph the surroundings of the vehicle; images taken by the photographing means and acquired frame by frame for each moving distance corresponding to a photographing range of the photographing means are synchronized with the movement of the vehicle; image storage means for chronologically storing the images in units of frames; image processing means for processing the images acquired in units of frames for each moving distance corresponding to the photographing range of the photographing means ; and the image processing means. A vehicle surrounding image display system comprising an image display means for displaying a vehicle surrounding image generated by synthesizing frame-by-frame images,
The image processing means,
Receiving the image taken by the photographing means and acquired in frame units for each moving distance corresponding to a photographing range of the photographing means , and converting the received image into a time-series continuous bird's-eye view image in frame units. generating a history image by synthesizing a plurality of frames in which the converted frame-by-frame overhead image is arranged in synchronization with the movement of the vehicle;
generating a vehicle surrounding image in which a blind spot portion of a current image photographed by the photographing means and acquired frame by frame for each moving distance corresponding to a photographing range of the photographing means is complemented with the generated history image; and,
a step of superimposing a transparent image of the own vehicle stored in the image storage means on the generated vehicle surrounding image and sequentially displaying the same in synchronization with the progress of the vehicle ;
The method for displaying the vehicle surrounding image is as follows:
The image processing means further includes a step of calculating an orientation, a moving direction, and a moving distance of the own vehicle based on the image acquired in frame units for each moving distance corresponding to a photographing range of the photographing means,
In the step of generating the history image, each time the calculated moving distance of the vehicle reaches a moving distance corresponding to the photographing range of the photographing means, the calculated orientation, moving direction, and moving distance of the own vehicle are determined. The synthesis position of the current bird's-eye image obtained by converting the current image into a frame-by-frame bird's-eye image and the frame-by-frame bird's-eye image photographed before the current image is adjusted for each frame based on the history. A method for displaying images around a vehicle , the method comprising the step of updating an image . The step of generating the history image includes the step of extracting feature points from the bird's-eye view image in units of continuous frames and adjusting the combining position of the plurality of frames so that the extracted feature points match. The vehicle surrounding image display method according to claim 5 . The step of displaying the transparent images in an overlapping manner includes comparing the current bird's-eye view image with the generated history image as the vehicle progresses, and determining whether the current bird's-eye image has any portions different from the history image. 6. The vehicle surrounding image display method according to claim 5 , further comprising the step of emphasizing and displaying images of the different portions in some cases. The step of generating the vehicle surrounding image includes combining the current bird's-eye view image and the current bird's-eye view image to be combined based on the calculated orientation, moving direction, and moving distance of the own vehicle according to the progress of the vehicle. 6. The vehicle surrounding image display method according to claim 5 , further comprising the step of cropping a previously photographed frame-by-frame bird's-eye view image or setting an image compositing position.

Images