JP7143728B2 - Superimposed image display device and computer program - Google Patents

Description

The present invention relates to a superimposed image display device and a computer program for assisting driving of a vehicle.

2. Description of the Related Art Conventionally, various means have been used as information providing means for providing various types of information for assisting vehicle travel, such as route guidance and warnings of obstacles, to occupants of the vehicle. For example, it is the display by the liquid crystal display installed in the vehicle, the sound output from the speaker, or the like. In recent years, as one of such information providing means, an image superimposed on the driver's surrounding environment (scenery) such as a head-up display (hereinafter referred to as HUD) or a window shield display (hereinafter referred to as WSD) has been introduced. There are devices that provide information by displaying.

For example, in Japanese Patent Laid-Open No. 2012-218505, an infrared camera installed in the vehicle captures an area in front of the vehicle, detects a pedestrian located in the captured image, and captures the captured image in addition to the captured image. A technique is disclosed for displaying a frame surrounding a pedestrian positioned inside on a liquid crystal display.

JP 2012-218505 A (pages 9-11, FIG. 2)

Here, in the technique described in Patent Document 1, the position of the warning target (pedestrian, etc.) is warned by surrounding the warning target with a two-dimensional frame parallel to the captured image. Although it is possible to grasp the position of the warning object in the captured image, there is a problem that the range in which the warning object exists cannot be specified accurately.

It is important to grasp the range in which the object to be warned exists not only in the vertical and horizontal directions (the direction that intersects the traveling direction of the vehicle) but also in the depth direction (the traveling direction of the vehicle), that is, in a three-dimensional space. . For example, for a warning target object that is long in one direction, such as a bicycle, the position of the warning target object is grasped only in the vertical and horizontal directions as in the above-mentioned Patent Document 1, and how the warning target object exists in the depth direction. If the driver does not know whether or not the driver is on the alert, there is a risk that the driver will not be able to properly perform the driving operation to avoid the warning target.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems in the prior art, and it is possible to three-dimensionally grasp the range in which an object to be warned exists, and to more reliably issue a warning to the object to be warned. It is an object of the present invention to provide a superimposed image display device and a computer program that make it possible.

In order to achieve the object, the superimposed image display device according to the present invention is mounted on a vehicle, and superimposes a warning image that warns of a warning object positioned around the vehicle on the scenery in front of the vehicle so that the superimposed image can be visually recognized. A display device, comprising: a three-dimensional space generating means for generating a three-dimensional space corresponding to the vicinity of a current position of a vehicle based on map information; virtual object arrangement means for arranging a virtual object imitating an object; image arrangement means for arranging the warning image, which is a three-dimensional object surrounding the virtual object, in a three-dimensional space; shape acquisition means for acquiring the shape of the warning image and the shape of the virtual object when both the warning image and the virtual object are visually recognized; and the shape acquisition together with the warning image of the shape acquired by the shape acquisition means and image display means for displaying a black image representing the shape of the virtual object acquired by said means on an image display surface, wherein said warning image and said black image displayed on said image display surface are transmitted from a vehicle occupant. A virtual image of the warning image displayed on the image display surface is superimposed on the scenery in front of the vehicle as a three-dimensional object surrounding the warning object by being reflected by a visible reflector and being visually recognized by a vehicle occupant. visible .
The "warning object" is an object whose presence should be warned to the occupants of the vehicle.
For example, there are moving objects such as pedestrians, bicycles, and other vehicles. Moreover, not only moving objects but also stationary objects such as traffic lights and road signs may be included in the warning objects.

Further, a computer program according to the present invention is a program for assisting driving of a vehicle. Specifically, a superimposed image display device, which is mounted on a vehicle and displays a warning image that warns of a warning target located in the vicinity of the vehicle, superimposed on the landscape in front of the vehicle, is displayed on the vehicle based on map information. A three-dimensional space generating means for generating a three-dimensional space corresponding to the vicinity of the current position, and a virtual object placement means for placing a virtual object simulating the warning object at the position where the warning object exists in the three-dimensional space. and image arrangement means for arranging the warning image, which is a three-dimensional object surrounding the virtual object, in a three-dimensional space; a shape obtaining means for obtaining a shape of a warning image and a shape of the virtual object; and a black color indicating the shape of the virtual object obtained by the shape obtaining means together with the warning image of the shape obtained by the shape obtaining means. and image display means for displaying an image on an image display surface, the computer program for displaying the warning image and the black image displayed on the image display surface on a reflector that is visible to the occupants of the vehicle. A virtual image of the warning image displayed on the image display surface is superimposed on the landscape in front of the vehicle and viewed as a three-dimensional object surrounding the warning object by being reflected and visually recognized by a vehicle occupant .

According to the superimposed image display device and the computer program according to the present invention having the above configuration, by displaying a three-dimensional object surrounding the object as a warning image, the range in which the object of warning exists can be displayed to the occupant of the vehicle in three dimensions. It is possible to grasp the As a result, it becomes possible to issue a warning to the warning object more reliably, and to appropriately perform the driving operation for avoiding the warning object. In the case of displaying the warning image as a virtual image, by displaying a black image in the position and range corresponding to the real image of the warning object, the virtual image of the warning image showing the anteroposterior relationship between the warning object and the warning image can be displayed on the vehicle. can be visually recognized by the occupant.

1 is a schematic configuration diagram of a superimposed image display device according to an embodiment; FIG. 1 is a block diagram showing a navigation device according to an embodiment; FIG. 4 is a flowchart of a driving support processing program according to the embodiment; FIG. 4 is a diagram showing a three-dimensional map generated based on three-dimensional map information; It is a figure explaining the warning target object located around self-vehicles. FIG. 10 is a diagram showing a three-dimensional map after arranging a virtual object and a warning image; It is a figure explaining the generation method of the warning image and black image which are displayed on HUD. FIG. 3 is a diagram showing an example of a virtual image of a warning image visually recognized by an occupant of a vehicle; 9 is a flowchart of a sub-processing program of warning target area determination processing; It is the figure which showed an example of the determination method of the warning object area|region. It is the figure which showed an example of the determination method of the warning object area|region. It is the figure which showed an example of the determination method of the warning object area|region. 9 is a flowchart of a sub-processing program of display range determination processing;

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of a superimposed image display device according to the present invention will be described in detail below with reference to the drawings. First, a schematic configuration of a superimposed image display device 1 according to this embodiment will be described with reference to FIG. FIG. 1 is a schematic configuration diagram of a superimposed image display device 1 according to this embodiment.

As shown in FIG. 1, the superimposed image display device 1 includes a navigation device 3 mounted on a vehicle 2 and a head-up display device (hereinafter referred to as HUD) 4 also mounted on the vehicle 2 and connected to the navigation device 3. and basically.

Here, the navigation device 3 searches for a recommended route to a destination, displays a map image around the current position of the vehicle 2 based on map data acquired from a server or stored in a memory, and detects obstacles. It has a function to issue a warning to a warning object such as the HUD 4, and to perform travel guidance along a set guide route together with the HUD 4. It should be noted that the navigation device 3 does not have to have all of the above functions, and the present invention can be configured as long as it has at least the function of giving a warning to an object to be warned. Details of the structure of the navigation device 3 will be described later.

On the other hand, the HUD 4 is installed inside the dashboard 5 of the vehicle 2, and has a liquid crystal display 6, which is an image display screen on which an image is displayed, inside. The image projected on the liquid crystal display 6 is reflected by the front window 8 in front of the driver's seat via a concave mirror 7 provided in the HUD 4 as will be described later, so that the occupant 9 of the vehicle 2 can see the image. ing. The images displayed on the liquid crystal display 6 include information on the vehicle 2 and various information used for assisting the driver 9 in driving. For example, a warning to the passenger 9 against objects to be warned (other vehicles and pedestrians), a guidance route set by the navigation device 3 and guidance information based on the guidance route (arrows indicating right and left turn directions, etc.), road surface Warnings to be displayed (caution for rear-end collision, speed limit, etc.), current vehicle speed, advertisement images, guide signs, map images, traffic information, news, weather forecasts, time, screens of connected smartphones, TV programs, etc.

In addition, in the HUD 4 of the present embodiment, when the occupant 9 visually recognizes the image displayed on the liquid crystal display 6 by reflecting the front window 8, the occupant 9 does not see the position of the front window 8, but the tip of the front window 8. An image displayed on the liquid crystal display 6 at a distant position is visually recognized as a virtual image 10. - 特許庁In addition, the virtual image 10 is superimposed on the scenery (real scene) in front of the vehicle and displayed. It is also possible to display it. The virtual image 10 visible to the occupant 9 is the image displayed on the liquid crystal display 6, but the vertical direction and the horizontal direction may be reversed through the concave mirror 7 or other mirrors. It is necessary to display the image on the liquid crystal display 6 by using the Also, the size is changed through the concave mirror 7 .

Here, the position where the virtual image 10 is generated, more specifically, the distance (hereinafter referred to as imaging distance) L from the occupant 9 to the virtual image 10, the curvature of the concave mirror 7 provided in the HUD 4, the liquid crystal display 6 and the concave mirror 7 can be appropriately set depending on the relative positions of the . For example, if the curvature of the concave mirror 7 is fixed, the imaging distance L is determined by the distance (optical path length) along the optical path from the position where the image is displayed on the liquid crystal display 6 to the concave mirror 7 . For example, the optical path length is set so that the imaging distance L is 2.5 m.

Also, in this embodiment, the HUD 4 is used as means for displaying an image superimposed on the scenery in front of the vehicle, but other means may be used. For example, a window shield display (WSD) that displays an image on the front window 8 may be used. In WSD, an image may be displayed from a projector using the front window 8 as a screen, or the front window 8 may be a transmissive liquid crystal display. The image displayed on the front window 8 by the WSD is an image superimposed on the scenery in front of the vehicle, like the HUD 4 .

Furthermore, it is possible to display the scenery in front of the vehicle captured by the front camera 11, which will be described later, on the liquid crystal display inside the vehicle, and to display an image superimposed on the scenery displayed in the same liquid crystal display. Even in that case, the image displayed on the liquid crystal display is an image superimposed on the scenery in front of the vehicle, similar to the HUD 4 .

A front camera 11 is installed above the front bumper of the vehicle, behind the rearview mirror, or the like. The front camera 11 is, for example, an imaging device having a camera using a solid-state imaging device such as a CCD, and is installed with the optical axis directed forward in the traveling direction of the vehicle. Then, by performing image processing on the captured image captured by the front camera 11, the situation of the front environment (that is, the environment where the virtual image 10 is superimposed) that is visually recognized by the passenger 9 through the front window 8 is detected. be done. A sensor such as a millimeter wave radar may be used instead of the front camera 11 .

In addition, an in-vehicle camera 12 is installed on the upper surface of the instrument panel of the vehicle. The in-vehicle camera 12 is, for example, an imaging device having a camera using a solid-state imaging device such as a CCD, and is installed with its optical axis directed toward the driver's seat. Then, the face of the passenger 9 sitting in the driver's seat is imaged. By performing image processing on the captured image captured by the in-vehicle camera 12, the eye position (line-of-sight starting point) and line-of-sight direction of the occupant 9 are detected.

Next, a schematic configuration of the navigation device 3 that constitutes the superimposed image display device 1 will be described with reference to FIG. FIG. 2 is a block diagram showing the navigation device 3 according to this embodiment.

As shown in FIG. 2, the navigation device 3 according to this embodiment includes a current position detection unit 13 for detecting the current position of the vehicle 2 on which the navigation device 3 is mounted, and a data recording unit 14 in which various data are recorded. , a navigation ECU 15 that performs various arithmetic processing based on the input information, an operation unit 16 that receives operations from the user, and a liquid crystal display 17 that displays a map of the vehicle's surroundings and facility information regarding facilities to the user. , a speaker 18 that outputs voice guidance regarding route guidance, a DVD drive 19 that reads a DVD as a storage medium, and an information center such as a VICS (registered trademark: Vehicle Information and Communication System) center. a module 20; Further, the navigation device 3 is connected to the HUD 4, the front camera 11, the in-vehicle camera 12, etc., via an in-vehicle network such as CAN.

Each component of the navigation device 3 will be described in order below.
The current position detection unit 13 includes a GPS 21, a vehicle speed sensor 22, a steering sensor 23, a gyro sensor 24, etc., and is capable of detecting the current vehicle position, direction, vehicle speed, current time, and the like. . In particular, the vehicle speed sensor 22 is a sensor for detecting the moving distance and speed of the vehicle, generates a pulse according to the rotation of the drive wheels of the vehicle, and outputs the pulse signal to the navigation ECU 15 . Then, the navigation ECU 15 calculates the rotational speed of the drive wheels and the movement distance by counting the generated pulses. It should be noted that the navigation device 3 does not need to have all of the above four types of sensors, and the navigation device 3 may have only one or a plurality of types of sensors among them.

The data recording unit 14 also includes a hard disk (not shown) as an external storage device and recording medium, and a driver for reading out the map information DB 31 and predetermined programs recorded in the hard disk and writing predetermined data to the hard disk. and a recording head (not shown). The data recording unit 14 may have a flash memory, a memory card, or an optical disk such as a CD or DVD instead of the hard disk. Further, the map information DB 31 may be stored in an external server, and may be acquired by the navigation device 3 through communication.

Here, the map information DB 31 stores two-dimensional map information 33 and three-dimensional map information 34, respectively. The two-dimensional map information 33 is map information used in a general navigation device 3, and includes, for example, link data related to roads (links), node data related to node points, facility data related to facilities, and search data used for route search processing. , map display data for displaying a map, intersection data for each intersection, search data for searching for points, and the like.

On the other hand, the 3D map information 34 is map information for representing a map in 3D, having not only plane information but also height information. In particular, in the present embodiment, the map information is used to express road contours, building shapes, road division lines, traffic lights, road signs, signboards, etc. in three dimensions. The three-dimensional map information 34 may include information other than the contours of roads, shapes of buildings, division lines of roads, traffic lights, road signs, and signboards. For example, information for representing roadside trees, road markings, etc. in three dimensions may be included. As the three-dimensional map information 34, a map itself may be stored in which objects such as road outlines, building shapes, road division lines, traffic lights, road signs, and signboards are arranged in a three-dimensional space. However, information necessary for representing a map in three dimensions (three-dimensional coordinate data of road contours, building shapes, road division lines, traffic lights, road signs, billboards, etc.) may also be stored. . When the information necessary for expressing the map in three dimensions is stored, the navigation device 3 uses the information stored as the three-dimensional map information 34 at the necessary timing to three-dimensionally display the target area. Generate a representational map.

The navigation device 3 uses the two-dimensional map information 33 for general functions such as displaying a map image on the liquid crystal display 17 and searching for a guidance route. Further, as will be described later, processing related to displaying a warning image is performed using not only the two-dimensional map information 33 but also the three-dimensional map information 34 .

On the other hand, a navigation ECU (electronic control unit) 15 is an electronic control unit that controls the entire navigation device 3, and includes a CPU 41 as an arithmetic device and a control device, and a working memory when the CPU 41 performs various arithmetic processing. In addition to the RAM 42 that stores the route data etc. when the route is searched, the ROM 43 that stores the driving support processing program (FIG. 3) etc. described later in addition to the control program, and the ROM 43 It has an internal storage device such as a flash memory 44 for storing programs. The navigation ECU 15 has various means as processing algorithms. For example, the captured image acquisition means acquires a captured image of the landscape in front of the vehicle. The map information acquisition means acquires three-dimensional map information. The collation means collates the captured image and the three-dimensional map information. The image arranging means arranges the warning image at the position where the warning object exists in the three-dimensional map information based on the collation result of the collating means. The shape obtaining means obtains the shape of the warning image when the warning image is viewed from the position of the vehicle in the three-dimensional map information.

The operation unit 16 is operated when inputting a departure point as a travel start point and a destination as a travel end point, and has a plurality of operation switches (not shown) such as various keys and buttons. Then, the navigation ECU 15 performs control to execute various corresponding operations based on switch signals output by pressing of each switch or the like. Note that the operation unit 16 may have a touch panel provided on the front surface of the liquid crystal display 17 . It may also have a microphone and a speech recognition device.

The liquid crystal display 17 also displays a map image including roads, traffic information, operation guidance, operation menu, key guidance, guidance route from the departure point to the destination, guidance information along the guidance route, news, weather forecast, Time, mail, TV program, etc. are displayed. Since the HUD 4 is provided as information display means in this embodiment, the liquid crystal display 17 may be omitted if the HUD 4 is configured to display the map image and the like.

In addition, the speaker 18 outputs voice guidance for guiding the vehicle along the guidance route based on instructions from the navigation ECU 15, and guidance for traffic information.

Also, the DVD drive 19 is a drive capable of reading data recorded on recording media such as DVDs and CDs. Then, based on the read data, music and video are reproduced, the map information DB 31 is updated, and so on. Incidentally, instead of the DVD drive 19, a card slot for reading and writing a memory card may be provided.

The communication module 20 is a communication device for receiving traffic information including congestion information, traffic regulation information, and traffic accident information transmitted from a traffic information center such as a VICS center or a probe center. For example, mobile phones and DCMs correspond.

Next, the driving support processing program executed especially in the navigation device 3 of the superimposed image display device 1 having the above configuration will be described with reference to FIG. FIG. 3 is a flow chart of a driving support processing program according to this embodiment. Here, the driving support processing program is executed after the vehicle's ACC power supply (accessory power supply) is turned on, and uses the HUD 4 to warn the occupant 9 of the vehicle 2 about warning objects located around the vehicle. is. 3 is stored in the RAM 42 and ROM 43 provided in the navigation device 3 and executed by the CPU 41. The program shown in FIG.

The "warning object" is an object whose presence should be warned to the occupants of the vehicle, and includes moving objects such as pedestrians, bicycles, and other vehicles. Moreover, not only moving objects but also stationary objects such as traffic lights and road signs may be included in the warning objects. However, in the following explanation, the warning target is a moving object that has an impact when the own vehicle travels along the traveling path. Make it a warning target.

First, in the driving support processing program, in step (hereinafter abbreviated as S) 1, the CPU 41 acquires a captured image of the scenery in front of the vehicle captured by the front camera 11 . Furthermore, the CPU 41 reads the three-dimensional map information 34 around the current position of the vehicle from the map information DB 31 . The 3D map information 34 includes information necessary for expressing a map in 3D (3D coordinate data such as outlines of roads, shapes of buildings, division lines of roads, traffic lights, road signs, signboards, etc.). etc.) are stored.

Next, in S2, the CPU 41 generates a three-dimensional map (a map representing buildings, roads, etc. in three dimensions) indicating the vicinity of the current position of the vehicle based on the three-dimensional map information 34 read out in S1. Specifically, a three-dimensional map is generated by executing the following processes.

First, the CPU 41 acquires objects (shape data) modeled on roads, buildings, road signs, signboards, etc. on the three-dimensional space from the three-dimensional map information 34 . Note that the CPU 41 may be configured to acquire a corresponding object from objects modeled in advance and stored in the DB, or may be configured to create a new object by performing modeling in S2. When performing modeling, information specifying the shapes and positions of roads and structures (buildings, road signs, billboards, etc.) in the vicinity of roads is acquired from the three-dimensional map information 34, and based on the acquired information Carry out the modeling process.

Here, modeling is the process of creating the shape of a model (object) in a three-dimensional space, and more specifically, determination of the coordinates of each vertex, determination of parameters of equations that express boundary lines and surfaces, etc. I do. Since the modeling is a well-known technique, the details will be omitted. The modeled object (shape data) is expressed in the form of a "wireframe model" that displays only sides, a "surface model" that displays surfaces, or the like, depending on the application. A three-dimensional space in which each object is formed is defined as a three-dimensional map.

In S2, the CPU 41 also identifies the current position and direction of the vehicle on the generated three-dimensional map based on the parameters detected by the current position detection section 13. FIG. In addition, in order to facilitate the process of collating the three-dimensional stereoscopic map and the captured image, which will be described later, the current position of the vehicle is the installation position of the front camera 11 installed in the vehicle, and the direction of the vehicle is the front camera. It is desirable to set it in the optical axis direction of the camera 11 .

Furthermore, the CPU 41 detects the positions and shapes of stationary objects and moving objects existing around the front of the vehicle by performing image processing on the captured image of the scenery in front of the vehicle acquired in S1. Stationary objects include, for example, artificial structures (buildings, road signs, billboards, etc.) and natural objects (trees, etc.), and moving objects include other vehicles, people, bicycles, and the like. When a stationary or moving object that is not included in the 3D map information 34 is detected, information specifying the position and shape of the detected stationary or moving object is included in the 3D map. . In S2, it is not necessary to model the detected stationary or moving object on the three-dimensional map, but modeling may be performed.

Here, FIG. 4 is a diagram showing an example of the three-dimensional map 51 generated in S2. As shown in FIG. 4, on a three-dimensional map 51, objects 52 representing roads and structures (buildings, road signs, signboards, etc.) are arranged in a three-dimensional space. In particular, objects 52 representing roads and structures located around the current position of the vehicle are arranged. A vehicle position mark 53 indicating the current position and direction of the vehicle is also arranged on the three-dimensional map 51 .

Subsequently, in S3, the CPU 41 collates the captured image acquired in S1 and the three-dimensional map generated in S2, and determines whether there is any deviation between them. Specifically, the CPU 41 uses the current position of the vehicle set in the three-dimensional map (more specifically, the installation position of the front camera 11 and also considers the height) as the viewpoint, and the direction of the vehicle as the line of sight. A direction is set, and an image when a three-dimensional map is visually recognized from the set viewpoint and line-of-sight direction is compared with a captured image. Since the three-dimensional map does not include moving objects such as pedestrians and other vehicles, and some fixed objects such as trees, deviations caused by these objects are basically ignored in the determination. Moreover, it is desirable to consider that there is no deviation not only when the two match completely, but also when there is a deviation within a certain allowable range.

If it is determined that there is no deviation between the captured image acquired in S1 and the three-dimensional map generated in S2 (S3: YES), the process proceeds to S5. On the other hand, if it is determined that there is a discrepancy between the captured image acquired in S1 and the three-dimensional map generated in S2 (S3: NO), the process proceeds to S4.

In S4, the CPU 41 calculates the current position and direction of the vehicle set in the three-dimensional map so as to minimize the deviation between the three-dimensional map generated in S2 and the captured image obtained in S1. to correct. Incidentally, the current position and direction of the own vehicle on the three-dimensional map may be fixed and corrected on the object side. As a result, it is possible to accurately identify the current position and orientation of the own vehicle (more specifically, the installation position and optical axis direction of the front camera 11) within the three-dimensional map. After that, the process returns to S3.

On the other hand, in S5, the CPU 41 detects a warning object positioned around the vehicle. In the present embodiment, a warning target located in a range that can be visually recognized by the occupant 9 in the front of the vehicle in the direction of travel is set as a detection target, and is detected by performing image processing on the captured image captured by the front camera 11 . As described above, the "warning object" is defined as a moving object that has an effect when the own vehicle travels along the travel route. A moving object is a warning target. Note that the future course of the moving object is specified based on the displacement of the position of the moving object in the captured image between frames.

For example, as shown in FIG. 5, a case where a pedestrian 55 and a bicycle 56 are detected in front of the vehicle in the traveling direction and the own vehicle turns left at an intersection 57 ahead will be described. In addition, the path (direction of travel) of the pedestrian 55 is identified as a path crossing the road connected to the left side of the guidance junction, and the path (direction of travel) of the bicycle 56 is specified to the left side of the intersection 57. Assume that the route is identified as moving away from intersection 57 without crossing the road.
In the example shown in FIG. 5, the pedestrian 55 is determined to be the object to be warned because the future path of the pedestrian 55 overlaps the path of the own vehicle. On the other hand, the future traveling route of the bicycle 56 and the traveling route of the own vehicle do not overlap. Therefore, it is determined that the bicycle 56 is not a warning object.

Further, in S5, the CPU 41 also identifies the position, type (only if it can be identified), and shape of the warning target based on the captured image.

After that, in S6, the CPU 41 performs warning target area determination processing (FIG. 9), which will be described later. In the warning target region determination process, a warning region (hereinafter referred to as a warning target region) for the warning object detected in S5 is determined. Although the details will be described later, the warning target area is specified in a three-dimensional space and is a three-dimensional object that surrounds the warning target, more specifically, the smallest rectangular parallelepiped that surrounds the entire warning target. The warning target area also corresponds to an area for displaying a warning image, which will be described later.

Subsequently, in S7, the CPU 41 arranges a warning image for warning the warning object detected in S5 on the three-dimensional solid map generated in S2. Note that the warning image is displayed for the warning target area determined in S6. That is, the warning image has a three-dimensional shape and is a three-dimensional object surrounding the warning target, more specifically, a minimum rectangular parallelepiped that surrounds the entire warning target. However, the warning image has a shape of only a frame without rectangular parallelepiped faces so that the surrounding warning object can be visually recognized by the vehicle occupants.

Next, in S8, the CPU 41 converts the model of the warning target detected around the host vehicle (virtual body of the warning target, hereinafter referred to as the target virtual body) to the three-dimensional solid generated in S2. Arrange on the map where the warning target is detected. When a plurality of warning objects are detected, object virtual objects are arranged on the three-dimensional solid map for all the detected warning objects. It should be noted that the virtual object placed on the three-dimensional map has a shape that imitates the shape of the actual warning object. For example, a pedestrian is assumed to be an object virtual object having the shape of a person, and a vehicle is assumed to be an object virtual object having the shape of a vehicle. The size is also made to correspond to the size of the actual warning object.

However, considering that the position of the warning target that can be visually recognized by the occupant may be blurred due to vibrations that occur in the vehicle, the size of the virtual target object is slightly larger than the actual size of the warning target (for example, 110%). It is desirable to In addition, it is desirable to increase the size significantly in the direction of relative movement (the direction in which the object to be warned moves from the vehicle occupant's point of view), instead of increasing the size uniformly overall. If the object to be warned has an irregular shape (its shape and type cannot be specified), the virtual object object is exceptionally the smallest rectangular parallelepiped that includes all of the object to be warned.

In addition, it is assumed that the object virtual object placed on the three-dimensional map is an object displayed entirely in black. On the other hand, the warning image arranged in S7 is an image displayed in a color other than black (for example, red or yellow).

Here, FIG. 6 is a diagram in which a virtual object 60 of a warning object and a warning image 61 are arranged on a three-dimensional stereoscopic map 51 . Incidentally, in the example shown in FIG. 6, the virtual object 60 indicates a pedestrian. That is, it indicates that a pedestrian is present as a warning object in the vicinity of the own vehicle. On the other hand, the warning image 61 has a minimum rectangular parallelepiped frame shape that surrounds the entire virtual object 60 .

After that, in S9, the CPU 41 first acquires an image (hereinafter referred to as a "visualized image") of the 3D 3D map in which the target virtual object and the warning image are arranged, viewed from the viewpoint of the vehicle (passenger) in the traveling direction of the vehicle. . In particular, the viewpoint of the vehicle is the eye position of the vehicle occupant. Incidentally, it is possible to detect the positions of the eyes of the occupant by the in-vehicle camera 12 . In other words, the acquired visible image is the traveling direction of the vehicle from the viewpoint of the vehicle (passenger), each object (road, building, road sign, signboard, warning image, target object virtual object, etc.) arranged on the 3D map. It is an image that can be visually recognized when visually recognizing the vehicle, and corresponds to the visual field of the vehicle occupant. Note that the visually recognized image does not necessarily have to be an image visually recognized in the traveling direction of the vehicle as long as it is an image visually recognized from the viewpoint of the vehicle (passenger). However, at least the visible image must include the warning image and the virtual object.

Subsequently, in S<b>9 , the CPU 41 stores the shape of the warning image and the shape of the virtual object included in the visual recognition image as the shape of the image to be displayed by the HUD 4 . Here, the shape of the warning image stored in S9 is the shape of each object (road, building, road sign, signboard, virtual object, warning image, etc.) placed on the 3D map. This is the shape of the warning image that can be visually recognized when viewed from the viewpoint of . On the other hand, the shape of the virtual object included in the visual image is determined by the vehicle (passenger) placing each object (road, building, road sign, signboard, virtual object, warning image, etc.) placed on the 3D map. It is the shape of the virtual object that can be visually recognized when viewed from the viewpoint of . Therefore, for example, when the shape of the warning image stored in S9 is superimposed on the object virtual object 60 located on the front side of the warning image 61 as shown in FIG. It becomes the removed shape. On the other hand, when the shape of the target virtual object stored in S9 is visually recognized as overlapping with the warning image 61 located on the front side of the target virtual object 60 as shown in FIG. , the shape is excluded.

Furthermore, in S10, the CPU 41 acquires the position of the warning image arranged on the three-dimensional map. Specifically, it corresponds to the position of the warning target area determined in the above S7, and is a position surrounding the warning target objects located around the own vehicle. Similarly, the CPU 41 also acquires the position of the virtual object placed on the three-dimensional map. Specifically, it is the position where the warning object exists.

Thereafter, in S11, the CPU 41 performs display range determination processing (FIG. 13), which will be described later. In the display range determination process, the range in which the warning image and the black image are displayed on the HUD 4 (the range in which the warning image and the black image are projected onto the front window 8, or the warning image and the black image are displayed on the liquid crystal display 6 of the HUD 4 range).

Subsequently, at S12, the CPU 41 transmits a control signal to the HUD 4, and displays the warning image having the shape stored at S9 on the liquid crystal display 6 of the HUD 4 in the display range determined at S11. Similarly, the CPU 41 displays the black image having the shape of the virtual object stored in S9 in the display range determined in S11. Note that the warning image and the black image are basically displayed as a single image. In addition, the virtual object placed on the three-dimensional map does not necessarily have to be black, but even in that case, the image displayed on the liquid crystal display 6 is a black image in accordance with the virtual object. and

Here, when the image displayed on the liquid crystal display 6 is viewed by the occupant of the vehicle, if the image displayed on the liquid crystal display 6 is a black image, the virtual image of the black image cannot be viewed (becomes in a transparent state). . Therefore, even if the black image is displayed at a position where the warning target is superimposed, the occupant of the vehicle does not see the virtual image of the black image (more precisely, it sees the black image in a transparent state), and only the warning target is displayed. will be visible.

As a result, as shown in FIG. 8, when the occupant of the vehicle visually recognizes the warning image 61 and the black image 66 displayed on the liquid crystal display 6, the portion corresponding to the black image 66 (that is, the warning object in the actual scene overlaps). The virtual image is not visually recognized (becomes a transparent state). Further, with respect to the virtual image 67 of the warning image 61, the part hidden behind the warning object 68 (broken line part in the right diagram of FIG. 8) cannot be visually recognized by the black image 66, while the other part (the right diagram of FIG. 8) is invisible. Solid line part) is visible. As a result, the occupant can recognize the anteroposterior relationship between the position of the warning object 68 and the position of the virtual image 67 of the warning image 61 from the shape of the virtual image 67 of the warning image 61. The object 68 can be accurately grasped.

When drawing the warning image and the black image on the liquid crystal display 6, they may be drawn on different layers. As an order, objects positioned on the near side are drawn on higher (upper) layers. Specifically, the frame portion of the warning image located on the far side of the virtual object is drawn on the lowest layer. Next, the black image corresponding to the target virtual object is drawn on the middle layer, and the frame portion of the warning image located in front of the target virtual object is drawn on the highest layer. In this case, even if the warning image 61 and the black image 66 are drawn in full, the virtual image 67 of the warning image 61 shown in FIG. 8 can be visually recognized by the occupant of the vehicle.

Next, a sub-process of the warning target area determination process executed in S6 will be described with reference to FIG. FIG. 9 is a flowchart of a sub-processing program of the warning target area determination process.

First, in S21, the CPU 41 determines whether or not at least one or more warning objects exist in the range visible to the passenger 9 ahead of the vehicle in the traveling direction as a result of the warning object detection processing in S5.

Then, when it is determined that at least one or more warning objects exist within the visible range of the passenger 9 ahead of the vehicle in the traveling direction (S21: YES), the process proceeds to S22. On the other hand, if it is determined that there is no warning object within the visible range of the occupant 9 ahead of the vehicle in the traveling direction (S21: NO), the process ends without setting the warning target area.

In S<b>22 , the CPU 41 determines whether or not there are a plurality of warning objects in a range that can be visually recognized by the occupant 9 particularly in front of the vehicle in the traveling direction.

Then, when it is determined that a plurality of warning objects exist within the visible range of the occupant 9 ahead of the vehicle in the traveling direction (S22: YES), the process proceeds to S26. On the other hand, if it is determined that there is only one warning object within the visible range of the passenger 9 ahead of the vehicle in the traveling direction (S22: NO), the process proceeds to S23.

In S23, the CPU 41 determines whether or not there is an accompanying object that is partly connected to the warning object and moves in the same direction and at the same speed as the warning object. For example, when a pedestrian is pulling luggage such as a trunk, the pedestrian is detected as a warning object, and the trunk is detected as an accompanying object. Alternatively, if the pedestrian is accompanied by a dog, the pedestrian is detected as a warning object and the dog is detected as an accompanying object. Incidentally, detection of the accompanying object is performed by performing image processing on the captured image captured by the front camera 11 in the same manner as the warning target object. As for the accompanying object, the model (virtual object) of the accompanying object is placed at the position detected on the three-dimensional map, similarly to the warning object.

If it is determined that there is an accompanying object that is partly connected to the warning object and moves in the same direction and at the same speed as the warning object (S23: YES), the process proceeds to S25. On the other hand, when it is determined that the accompanying object does not exist (S23: NO), the process proceeds to S24.

In S24, the CPU 41 determines the minimum rectangular parallelepiped surrounding the entire warning object 74 as the warning object area 75, as shown in FIG. A warning target area 75 is determined on the three-dimensional map by using the virtual body of the warning target object 74 actually arranged on the three-dimensional map. After that, the process proceeds to S8.

Further, in S25, the CPU 41 determines the minimum rectangular parallelepiped surrounding both the warning target object 74 and the accompanying target object 76 as the warning target area 75 as shown in FIG. In practice, a warning target area 75 is determined on the three-dimensional map by using virtual objects of the warning object 74 and accompanying object 76 placed on the three-dimensional map. After that, the process proceeds to S8.

On the other hand, in S26, the CPU 41 determines that there are multiple warning targets within a predetermined range (for example, 2 m square) among the multiple detected warning targets, and that each warning target located within the predetermined range is in the same direction. It is determined whether or not to move at the same speed. Note that the moving direction and moving speed of the warning object are specified based on the displacement of the position of the warning object in the captured image between frames.

Then, when it is determined that among the plurality of detected warning objects, there are a plurality of warning objects within a predetermined range, and each warning object positioned within the predetermined range moves in the same direction at the same speed. If (S26: YES), the process proceeds to S28. On the other hand, if it is determined that a plurality of warning objects are not located within the predetermined range, or that each warning target does not move in the same direction and at the same speed even if they are located within the predetermined range (S26: NO) moves to S27.

In S27, the CPU 41 determines, as the warning target area 75, the smallest rectangular parallelepiped surrounding the entire warning target 74 for each of the plurality of warning targets 74 detected as shown in FIG. Therefore, for example, if there are three warning objects 74, three warning regions 75 are also set. A warning target area 75 is determined on the three-dimensional map by using virtual bodies of a plurality of warning targets 74 actually arranged on the three-dimensional map. Among a plurality of warning objects, for a warning object that has an accompanying object, it is desirable to determine the smallest rectangular parallelepiped that surrounds both the warning object and the accompanying object as the warning object area, as in S25. . After that, the process proceeds to S8.

On the other hand, in S28, the CPU 41 determines, as the warning target area 75, the smallest rectangular parallelepiped that collectively encloses all of the plurality of warning targets 74 detected as shown in FIG. A warning target area 75 is determined on the three-dimensional map by using virtual bodies of a plurality of warning targets 74 actually arranged on the three-dimensional map. In addition, in the case where additional objects exist, it is desirable to determine the minimum rectangular parallelepiped surrounding the accompanying objects in addition to the plurality of warning objects as the warning target area. After that, the process proceeds to S8.

Next, sub-processing of the display range determination processing executed in S11 will be described with reference to FIG. FIG. 13 is a flowchart of a sub-processing program for display range determination processing.

First, in S<b>31 , the CPU 41 detects the positions of the eyes of the vehicle occupant based on the captured image captured by the in-vehicle camera 12 . The positions of the detected eyes are identified by three-dimensional position coordinates.

Next, in S32, the CPU 41 determines whether the display of the HUD 4 is ON. It should be noted that ON/OFF switching of the display of the HUD 4 can be performed by the operation of the vehicle occupant. Alternatively, ON or OFF may be automatically switched based on the surrounding conditions or the state of the vehicle.

Then, when it is determined that the display of the HUD 4 is ON (S32: YES), the process proceeds to S33. On the other hand, when it is determined that the display of the HUD 4 is OFF (S32: NO), the process ends without displaying the virtual image of the warning image on the HUD 4.

In S<b>33 , the CPU 41 acquires the position coordinates of the front window 8 on which the image is projected by the HUD 4 . The positional coordinates of the front window 8 are specified by three-dimensional positional coordinates.

Next, in S34, the CPU 41 acquires the coordinates specifying the positions of the warning image and the target virtual object placed on the three-dimensional map acquired in S10 as the position coordinates of the warning image and the target virtual object, respectively. do. Note that the positional coordinates of the warning image and the virtual object are also identified by three-dimensional positional coordinates.

Subsequently, at S35, the CPU 41 determines the projection range of the warning image and the black image on the front window 8 based on the position coordinates obtained at S31, S33 and S34. Furthermore, the display range of the warning image and the black image on the liquid crystal display 6 inside the HUD 4 is also determined from the determined projection range. The display range of the warning image is the range in which the warning image having the shape stored in S9 is superimposed on the warning image arranged on the three-dimensional map and visually recognized by the occupant of the vehicle. In addition, the display range of the black image is a black image having the shape of the virtual object stored in S9, which is viewed from the occupant of the vehicle by the virtual object arranged on the three-dimensional map (that is, the warning object in the real scene). ) to be superimposed and visually recognized. After that, the process proceeds to S12, and a virtual image is displayed using the HUD 4 based on the determined projection range and display range. The processing of S9 to S12 is repeated until the display of the HUD is turned off.

As described in detail above, according to the superimposed image display device 1 according to the present embodiment and the computer program executed by the superimposed image display device 1, the warning image 61 that warns of the warning object located around the vehicle is When the warning image 61 is superimposed on the scenery in front of the vehicle and visually recognized, the warning image 61 is displayed as a three-dimensional object surrounding the warning object (S7 to S12). It is possible to comprehend dimensionally. As a result, it is possible to more reliably issue a warning to the warning object, and to appropriately perform the driving operation for avoiding the warning object.

It should be noted that the present invention is not limited to the above-described embodiments, and of course various improvements and modifications are possible without departing from the gist of the present invention.
For example, in the present embodiment described above, the HUD 4 is configured to generate a virtual image in front of the front window 8 of the vehicle 2 , but a configuration in which a virtual image is generated in front of a window other than the front window 8 is also possible. Further, the object on which the image is reflected by the HUD 4 may be a visor (combiner) installed around the front window 8 instead of the front window 8 itself.

Further, in this embodiment, the HUD 4 is used as means for displaying an image superimposed on the surrounding environment, but a window shield display (WSD) for displaying an image on the front window 8 may be used. Alternatively, a display device that superimposes a warning image on the actual scene displayed on the liquid crystal display 17 of the navigation device 3 may be used.

Also, in the present embodiment, the warning image is the smallest rectangular parallelepiped that surrounds the warning target, but it may be of any other shape as long as it can surround the warning target. For example, it may have a cylindrical shape or a spherical shape. Also, the warning image may be surrounded by a certain amount of blank space around the object to be warned.

Further, in the present embodiment, the processing of the driving support processing program (FIG. 3) is configured to be executed by the navigation ECU 15 of the navigation device 3, but the executing body can be changed as appropriate. For example, it is good also as a structure which the control part of HUD4, vehicle control ECU, and other vehicle equipment perform. Incidentally, when the controller of the HUD 4 executes, the superimposed image display device according to the present invention can also be configured by the HUD 4 only.

Moreover, although the embodiments embodying the superimposed image display device according to the present invention have been described above, the superimposed image display device can also have the following configuration, and in that case, the following effects can be obtained.

For example, the first configuration is as follows.
A superimposed image display device (1) mounted on a vehicle (2) for superimposing a warning image (61) for warning a warning object (74) positioned around the vehicle on a landscape in front of the vehicle for visual recognition. a three-dimensional space generating means for generating a three-dimensional space corresponding to the vicinity of the current position of the vehicle based on map information; virtual object arrangement means for arranging a virtual object; image arrangement means for arranging the warning image, which is a three-dimensional object surrounding the virtual object, in a three-dimensional space; a shape acquisition unit for acquiring the shape of the warning image and the shape of the virtual object when both virtual objects are visually recognized; and an image display means for displaying a black image representing the shape of the virtual object on an image display surface, the warning image and the black image displayed on the image display surface being reflected so as to be visually recognizable by an occupant of the vehicle. A virtual image of the warning image displayed on the image display surface is superimposed on the scenery in front of the vehicle and visually recognized as a three-dimensional object surrounding the warning object by being reflected by the body and visually recognized by the occupant of the vehicle.
According to the superimposed image display device having the above configuration, by displaying a three-dimensional object surrounding the object as a warning image, it is possible for the vehicle occupant to three-dimensionally grasp the range in which the object to be warned exists. becomes. As a result, it becomes possible to issue a warning to the warning object more reliably, and to appropriately perform the driving operation for avoiding the warning object. In the case of displaying the warning image as a virtual image, by displaying a black image in the position and range corresponding to the real image of the warning object, the virtual image of the warning image showing the anteroposterior relationship between the warning object and the warning image can be displayed on the vehicle. can be visually recognized by the occupant.

A second configuration is as follows.
The warning image (61) is displayed with the smallest cuboid surrounding all of the warning objects (74).
According to the superimposed image display device having the above configuration, it is possible for the occupant to clearly grasp the range in which the object to be warned exists by means of the warning image having a simple shape.

A third configuration is as follows.
The object to be warned (61) is a moving object that has an effect when the vehicle travels along the travel route.
According to the superimposed image display device having the above configuration, it is possible to allow the occupant of the vehicle to three-dimensionally grasp the range in which the moving object exists, especially for the moving object that will affect the running of the vehicle in the future. becomes.

A fourth configuration is as follows.
When there is an accompanying object (76) partially connected to the warning object (61) and moving in the same direction and at the same speed as the warning object, the warning object is the same as the warning object. A three-dimensional object surrounding the accompanying object is displayed.
According to the superimposed image display device having the above configuration, when there is a warning target and an accompanying target, the warning target is displayed by displaying three-dimensional objects surrounding the warning target including the target. It is possible to three-dimensionally grasp the area affected by . As a result, it becomes possible to issue a warning to the warning object more reliably, and to appropriately perform the driving operation for avoiding the warning object.

Also, the fifth configuration is as follows.
When there are a plurality of warning objects (74) within a predetermined range and the plurality of warning objects move in the same direction at the same speed, the warning image (61) surrounds the plurality of warning objects. displayed in solid form.
According to the superimposed image display device having the above configuration, especially when the warning objects move together in groups, warning images are overlapped by displaying three-dimensional objects surrounding the warning objects collectively in group units. It is possible to prevent the warning object from being displayed as much as possible, and allow the vehicle occupant to clearly grasp the range in which the object to be warned exists three-dimensionally.

Also, the sixth configuration is as follows.
the three-dimensional space generation means generates a three-dimensional map space around the current position of the vehicle as the three-dimensional space, and a captured image acquisition means (41) for acquiring a captured image of the landscape in front of the vehicle ; collation means (41) for collating the captured image and the three-dimensional map space , and the image arrangement means (74) for the warning object in the three-dimensional map space based on the collation result of the collation means. , the warning image (61) is placed at the position where
According to the superimposed image display device having the above configuration, by using the captured image of the landscape in front of the vehicle and the three-dimensional map information, it is possible to easily generate a warning image that is a three-dimensional object surrounding the warning target. It becomes possible.

1 superimposed image display device 2 vehicle 3 navigation device 4 HUD
6 liquid crystal display 8 front window 9 passenger 41 CPU
42 RAMs
43 ROMs
51 Three-dimensional map 61 Warning image 66 Black image 67 Virtual image of warning image 68 Warning object

Claims (7)

A superimposed image display device that is mounted on a vehicle and superimposes a warning image that warns of a warning target located around the vehicle on the landscape in front of the vehicle,
a three-dimensional space generating means for generating a three-dimensional space corresponding to the vicinity of the current position of the vehicle based on the map information;
a virtual object arranging means for arranging a virtual object simulating the warning target at a position where the warning target exists in a three-dimensional space;
image arrangement means for arranging the warning image, which is a three-dimensional object surrounding the virtual object, in a three-dimensional space;
shape acquisition means for acquiring the shape of the warning image and the shape of the virtual object when both the warning image and the virtual object are viewed from the position of the vehicle in a three-dimensional space;
an image display means for displaying, on an image display surface, a black image showing the shape of the virtual object obtained by the shape obtaining means together with the warning image of the shape obtained by the shape obtaining means;
The warning image and the black image displayed on the image display surface are reflected by a reflector visible to the vehicle occupant so that the vehicle occupant can visually recognize the warning image displayed on the image display surface. A superimposed image display device in which a virtual image as a three-dimensional object surrounding the warning object is superimposed on the scenery in front of the vehicle and visually recognized . 2. The superimposed image display device according to claim 1, wherein the warning image is displayed as a minimum cuboid surrounding all of the warning objects. 3. The superimposed image display device according to claim 1, wherein the warning object is a moving object that has an effect when the vehicle travels along the travel route. When there is an accompanying object that is partly connected to the warning object and moves in the same direction and at the same speed as the warning object, the warning image is a three-dimensional image surrounding the warning object and the accompanying object. 4. The superimposed image display device according to any one of claims 1 to 3, which is displayed as an object. When a plurality of warning objects are present within a predetermined range and the plurality of warning objects move in the same direction at the same speed, the warning image is displayed as a three-dimensional object surrounding the plurality of warning objects. 4. The superimposed image display device according to claim 1. The three-dimensional space generating means generates a three-dimensional map space around the current position of the vehicle as the three-dimensional space,
a captured image acquiring means for acquiring a captured image of the landscape in front of the vehicle;
a matching means for matching the captured image with the three-dimensional map space ;
6. The warning image according to any one of claims 1 to 5, wherein said image placement means places said warning image at a position where said warning object exists in a three-dimensional map space based on the matching result of said matching means. A superimposed image display device. A superimposed image display device that is mounted on a vehicle and that superimposes a warning image that warns of warning objects located around the vehicle on the landscape in front of the vehicle for visual recognition,
a three-dimensional space generating means for generating a three-dimensional space corresponding to the vicinity of the current position of the vehicle based on the map information;
a virtual object arranging means for arranging a virtual object simulating the warning target at a position where the warning target exists in a three-dimensional space;
image arrangement means for arranging the warning image, which is a three-dimensional object surrounding the virtual object, in a three-dimensional space;
shape acquisition means for acquiring the shape of the warning image and the shape of the virtual object when both the warning image and the virtual object are viewed from the position of the vehicle in a three-dimensional space;
and image display means for displaying, on an image display surface, a black image representing the shape of the virtual object obtained by the shape obtaining means together with the warning image of the shape obtained by the shape obtaining means. a program,
The warning image and the black image displayed on the image display surface are reflected by a reflector visible to the vehicle occupant so that the vehicle occupant can visually recognize the warning image displayed on the image display surface. A computer program in which a virtual image is superimposed on the scenery in front of the vehicle as a three-dimensional object surrounding the warning object and visually recognized .

Images