JP5825713B2 - Dangerous scene reproduction device for vehicles - Google Patents

Description

  The present invention relates to a vehicle dangerous scene reproduction device that is installed in a vehicle and reproduces a virtual dangerous scene such as a pedestrian jumping out or a sudden deceleration of a preceding vehicle while the vehicle is running.

  Obtaining and analyzing the behavior of the driver when encountering a dangerous scene while driving a vehicle is an important and effective technique for elucidating the cause of a traffic accident.

  Therefore, in recent years, a driving simulator is often used as a method for reproducing a dangerous scene while the vehicle is traveling (Patent Document 1).

JP 2010-134101 A

  However, since the driving simulator disclosed in Patent Document 1 travels in a virtually created road environment, some drivers may be aware that it is not real. In addition, lack of reality increases the driver's sense of pride and causes a problem that it becomes impossible to correctly evaluate the driver's behavior when encountering a dangerous scene.

  The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide a vehicular danger scene reproduction device that can reproduce a virtual danger scene for a driver with a high sense of reality while the vehicle is running. And

  In particular, the present invention relates to a vehicle danger that can reproduce a realistic virtual danger scene for a driver using a vehicle that actually travels, regardless of the travel position or traveling direction of the vehicle. A scene reproduction device is provided.

  The vehicular danger scene reproduction device according to the present invention includes a still image or a moving image that constitutes a virtual danger scene on a video display unit placed at a position that obstructs a driver's direct field of view in an actually traveling vehicle. An image of the vehicle traveling direction superimposed (hereinafter referred to as superimposition) is displayed to reproduce a virtual danger scene for the driver while the vehicle is traveling.

That is, the vehicular danger scene reproduction device according to the present invention is installed in the traveling direction of the vehicle, and an image capturing unit that captures an image of the traveling direction of the vehicle, and a vehicle position that calculates the position and the traveling direction of the vehicle. A traveling direction calculation unit, a vehicle behavior calculation unit that calculates the behavior of the vehicle, an image information storage unit that stores in advance an image representing a virtual object to be superimposed on the video, and the vehicle position and travel in the video A superimposed video creation unit that superimposes an image on a position calculated based on the position and traveling direction of the vehicle calculated by the direction calculating unit and the behavior of the vehicle calculated by the vehicle behavior calculating unit, the vehicle position, and the traveling the position and the traveling direction of the vehicle calculated by the direction calculation unit, based on the behavior of the vehicle calculated by the vehicle behavior calculation unit, with respect to superimposed image creation unit, an instruction to perform the superimposition of the image superimposing timing The calculation unit and the driver of the vehicle It is arranged so as to block the view, an image created by superimposing image generator, to an image display unit for displaying substantially real time, comprising: a.

According to the vehicular danger scene reproduction device according to the present invention configured as described above, the image capturing unit captures the image in front of the vehicle, and the vehicle position, the traveling direction calculating unit, and the vehicle behavior calculating unit calculate the vehicle. An image representing a virtual object stored in advance in the image information storage unit at the timing calculated by the superimposition timing calculation unit based on the position and the traveling direction is included in the video captured by the video imaging unit in the superimposed video creation unit. , vehicle position, and the position and traveling direction of the vehicle calculated by the moving direction calculation unit, and the behavior of the vehicle calculated, virtual hazardous situations are superimposed at a position corresponding to that created in the vehicle behavior calculation unit, Since the image created in this way is displayed in a substantially real-time on the image display unit placed at a position that obstructs the driver's direct field of view and presented to the driving driver, the driving position of the vehicle Or depending on the direction of travel In, it is possible to reproduce the high realistic virtual dangerous scene.

  According to the danger scene reproduction device for a vehicle according to the present invention, a still image representing a virtual object at a position corresponding to the traveling position or traveling direction of the vehicle on the video display unit placed at a position that obstructs the driver's direct view. Since it is configured to display a video in the vehicle traveling direction on which a moving image is superimposed, a virtual danger scene can be reproduced with high realism to the driver while the vehicle is traveling.

1 is a block diagram showing a schematic configuration of a vehicular danger scene reproduction device according to an embodiment of the present invention. (1) It is a side view which shows the installation state to the vehicle of the dangerous scene reproduction apparatus for vehicles. (2) It is a top view which shows the installation state to the vehicle of the dangerous scene reproduction apparatus for vehicles. It is a figure which shows the reference | standard marker installed in the road environment which operates the dangerous scene reproduction apparatus for vehicles. It is a figure which shows the road model which operates the dangerous scene reproduction apparatus for vehicles. It is a figure which shows the example of the image | video image | photographed by the 1st image | video imaging | photography part. It is a figure which shows the method of calculating the present position of a vehicle. It is a figure which shows the method of calculating the direction of a vehicle. It is a figure explaining the example of reproduction of a dangerous scene. It is a figure which shows the operation | movement flowchart of embodiment of this invention. It is a figure which shows the flowchart of the interruption process at the time of a horn sounding. It is a figure which shows the flowchart of the interruption process at the time of the pedestrian signal blue blinking.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments of a vehicular dangerous scene reproduction device according to the present invention will be described with reference to the drawings.

  In this embodiment, the present invention is displayed by superimposing a still image or a moving image representing a virtual object on a video in front of the traveling direction on a video display unit placed at a position that blocks the driver's direct view. The present invention is applied to a vehicular danger scene reproduction device that can reproduce a virtual danger scene with high realism for a driver while the vehicle is traveling.

  As shown in FIG. 1, a vehicular danger scene reproduction device 1 according to the present embodiment is installed in a vehicle 10, and an image photographing unit 100 that photographs a traveling direction, and a virtual object such as a pedestrian or a vehicle created in advance. An image information storage unit 130 that stores an image representing the image as a still image or a moving image, and a superimposed video creation that superimposes the image information stored in the image information storage unit 130 on the video captured by the video imaging unit 100 Unit 120, vehicle position for calculating the position and traveling direction of the vehicle 10 based on the result of photographing the reference marker 142 installed on the road by the video photographing unit 100, the traveling direction calculating unit 150, the vehicle speed sensor 192, and the yaw rate sensor Based on the vehicle speed and the angular velocity calculated in 193, the vehicle behavior calculation unit 160 that calculates the behavior of the vehicle 10 and the superimposed video creation unit 120 calculate the timing for starting the superimposition. The tatami mat timing calculation unit 140, the video display unit 110 that displays a video on which still images and moving images are superimposed by the superimposed video creation unit 120, the vehicle information recording unit 190 that records the behavior of the vehicle, and the driver's behavior A driver information recording unit 200 for recording the vehicle, an operation detection unit 210 for detecting a driver's horn sounding operation, a horn switch 220, and a road in a direction perpendicular to the traveling direction of the vehicle 10 in front of the traveling direction of the vehicle 10. And a traffic environment detection unit 230 that detects the operating state of the traffic light for pedestrians.

  In detail, the video shooting unit 100 includes a first video shooting unit 102, a second video shooting unit 104, and a third video shooting unit 106, and specifically includes three video cameras.

  The video display unit 110 includes a first video display unit 112, a second video display unit 114, and a third video display unit 116, and specifically includes three rectangular liquid crystal monitors. .

  In detail, the superimposed image creating unit 120 calculates the position where the still image or the moving image is superimposed based on the vehicle position and the vehicle state calculated by the traveling direction calculating unit 150 and the vehicle behavior calculating unit 160 in detail. A calculating unit 122; a superimposing size calculating unit 124 that calculates a superimposing size of a still image or a moving image; a superimposing direction calculating unit 126 that calculates a direction in which the still image or moving image is superimposed; and a superimposing processing unit that actually performs superimposing. 128.

  Further, the superimposition timing calculation unit 140 is more specifically selected from among a plurality of reference markers 142 installed at predetermined positions on the shoulders of the road on which the vehicle 10 travels, and images in front of the road captured by the image capturing unit 100. A reference marker extraction unit 144 that extracts the position of the reference marker 142, and a still image that represents a virtual object stored in the image information storage unit 130 in a video imaged by the video imaging unit 100 based on the extraction result of the reference marker 142 The trigger signal generation unit 146 outputs a trigger signal for starting superimposition of an image or a moving image to the superimposed video creation unit 120.

  The vehicle information recording unit 190 includes a vehicle speed sensor 192 that measures the vehicle speed of the vehicle 10, a yaw rate sensor 193 that measures the angular velocity of the vehicle 10, a pitch angle sensor 194 that measures the pitch angle of the vehicle 10, A roll angle sensor 195 that measures the roll angle of the vehicle 10, an accelerator opening sensor 196 that detects the accelerator opening, a brake pedal force sensor 197 that detects the brake pedal force, and the values detected by these sensors are acquired and recorded. Vehicle information acquisition unit 191.

  In detail, the driver information recording unit 200 includes a driver face photographing camera 202 that photographs an image of the driver's face, and a driver foot photographing camera that photographs a driver's feet operating the accelerator and the brake. 203, a driver gaze measurement sensor 204 that measures the driver's gaze direction, and a driver information acquisition unit 201 that acquires and records data obtained by these cameras and sensors.

  Here, the first video shooting unit 102, the second video shooting unit 104, the third video shooting unit 106, the first video display unit 112, the second video display unit 114, and the third video display unit 116 are respectively shown in FIG. (1) As shown in (2), it is installed on the hood of the vehicle 10.

  Specifically, the first video shooting unit 102, the second video shooting unit 104, and the third video shooting unit 106 are arranged on the hood of the vehicle 10 with the first video shooting unit 102, the second video shooting unit 104, and the third video shooting unit 104. The optical axes of the video photographing unit 106 are arranged at a predetermined angle θ in the horizontal direction so that the respective photographing ranges do not overlap. In order to prevent the shooting ranges from overlapping, the images shot by the first video shooting unit 102, the second video shooting unit 104, and the third video shooting unit 106 are the first video display unit 112 and the second video display unit 114. This is to prevent the same area from being displayed repeatedly when displayed on the third video display unit 116.

  In addition, when it is difficult to arrange the shooting ranges of the first video shooting unit 102, the second video shooting unit 104, and the third video shooting unit 106 so as not to overlap, the actually shot video is displayed as the first video display unit. 112, the second video display unit 114, the third video display unit 116, and the first video shooting unit 102 and the second video shooting unit 104 so as not to cause a sense of incongruity while visually confirming the displayed video. The installation position of the third video display unit 116 may be adjusted.

  Alternatively, a plurality of videos whose shooting fields of view overlap each other are combined to generate panoramic videos, thereby generating non-overlapping videos, which are displayed as the first video display unit 112 and the second video display unit. 114 may be displayed on the third video display unit 116.

  The first video display unit 112, the second video display unit 114, and the third video display unit 116 are arranged on the hood of the vehicle 10 with the short side (vertical side) of the first video display unit 112 and the second video display unit. The short side (vertical side) of 114 is substantially in contact, and the short side (vertical side) of the second video display unit 114 and the short side (vertical side) of the third video display unit 116 are substantially in contact. Each video display surface is installed so as to be substantially vertical to the ground.

  Furthermore, the video display surface of the second video display unit 114 is installed so as to face the driver who has watched the front while traveling, and the long side (horizontal side) of the first video display unit 112 and the second video display. The long side (horizontal side) of the unit 114, the long side (horizontal side) of the second video display unit 114, and the long side (horizontal side) of the third video display unit 116 each form a predetermined angle θ. Is installed.

  Here, the angle θ formed by the long side of the first video display unit 112 and the long side of the second video display unit 114 is the angle θ formed by the optical axes of the first video shooting unit 102 and the second video shooting unit 104. The angle θ formed between the long side of the second video display unit 114 and the long side of the third video display unit 116 is an angle formed by the optical axes of the second video shooting unit 104 and the third video shooting unit 106. It is desirable to be approximately equal to θ.

  However, when the installation space of the video display unit 110 is insufficient due to a lack of space in the hood of the vehicle 10 or the shape of the hood, the long side of the first video display unit 112 and the second video display unit 114 In some cases, the angle between the long side and the long side of the second video display unit 114 and the long side of the third video display unit 116 cannot be set to θ. In such a case, while confirming the images displayed on the first image display unit 112, the second image display unit 114, and the third image display unit 116, at an appropriate angle so as not to cause a sense of incongruity, The one video display unit 112, the second video display unit 114, and the third video display unit 116 may be arranged.

  The first video display unit 112, the second video display unit 114, and the third video display unit 116 are preferably installed so as to display a range of viewing angles of 55 ° or more on the left and right sides as viewed from the driver. . This is because the video imaged by the video imaging unit 100 is displayed in the direction of the driver's line of sight even during a right or left turn when the driver's left and right line of sight moves greatly.

  The driver is photographed by the first video photographing unit 102, the second video photographing unit 104, and the third video photographing unit 106 arranged in this manner, and the first video display unit 112 and the second video display unit are provided in substantially real time. 114, the vehicle 10 can actually be run while watching the video displayed on the third video display unit 116.

  Next, the operation of the vehicular dangerous scene reproduction device 1 according to the present embodiment will be described based on the flowchart of FIG. 9 and FIGS. As shown in FIG. 8, the vehicular danger scene reproduction device 1 according to the present embodiment is an apparatus that reproduces a scene in which a pedestrian 803 and a moving vehicle 804 appear from the shadow of a wall 801 in front of an intersection. Is applied.

  It is assumed that the vehicle 10 is traveling on a test course prepared in advance. In this test course, the flow of other traffic is restricted, and a virtual danger scene can be reproduced as described below in a safe state.

  First, the first image capturing unit 102, the second image capturing unit 104, and the third image capturing unit 106 simultaneously capture images in front of the vehicle 10 (step S1 in FIG. 9).

  The reproduction of the virtual danger scene is performed at a predetermined point on the test course. In order to reliably reproduce a virtual danger scene at a predetermined point, as shown in FIG. 4, the reference markers 142 having the symbols shown in FIG. 3 are installed at a plurality of predetermined positions (M1 to M6) on the road. At least two reference markers 142 are extracted by the reference marker extraction unit 144 from the images captured by the one image capturing unit 102, the second image capturing unit 104, and the third image capturing unit 106 (see FIG. 9). Steps S2 and S3) When two reference markers 142 are extracted (when Step S4 in FIG. 9 is YES), the type of the extracted reference marker 142 and the size of the reference marker 142 in the video From the (lateral side length) and the center coordinates, the vehicle position / travel direction calculation unit 150 calculates the position and travel direction of the vehicle on the road (steps S7 and S9 in FIG. 9).

  The symbols of the reference markers (142-1 to 142-6) that are arranged are different from each other, and the symbol marker 142 of which symbol is installed at which position is determined in advance as a road model shown in FIG. 4. Since it is stored in the vehicle position and traveling direction calculation unit 150, the current position of the vehicle 10 can be calculated if the type of the extracted reference marker 142 is known.

  Here, a method of extracting the reference marker 142 will be described with reference to FIGS.

  Here, the vehicle 10 is traveling at the position shown in FIG. 4, and the reference marker 142-1 closest to the vehicle 10 installed on the left side of the road from the images captured by the first image capturing unit 102. And the reference marker 142-2 closest to the vehicle 10 installed on the right side of the road is extracted from the video imaged by the third video imaging unit 106.

  As shown in FIG. 3, the reference marker 142 has a black two-dimensional pattern (design described above) that is different for each reference marker 142 in a white square drawn in a black square.

  In addition, the video cameras constituting the first video shooting unit 102, the second video shooting unit 104, and the third video shooting unit 106 are respectively installed horizontally at the height H from the road surface. It is assumed that the marker 142 is installed at the position of the height H of the left and right shoulders of the road and facing the vehicle 10 side in a direction orthogonal to the traveling direction of the road.

  In the reference marker extraction unit 144, each template of the reference marker 142 and the surrounding white square and black square are used as one template, and this template is stored in advance for the type of the reference marker 142. Template matching is performed on the captured video, and the position where the reference marker 142 is reflected and the type of the reference marker 142 reflected there are detected.

  For example, the first video imaging unit 102 captures the video shown in FIG.

  Template matching is performed by applying the prepared templates to the video in FIG. 5 to detect the location where the reference marker 142 is shown.

  Note that the center of the reference marker 142 is located at a position corresponding to the height H from the road surface, that is, in the vicinity of the center in the vertical direction of the captured video (depending on the installation conditions of the video shooting unit 100 and the installation conditions of the reference marker 142. Therefore, template matching may be performed only near the center in the vertical direction of the video.

  Further, since the reference marker 142 appears smaller as the distance from the vehicle 10 decreases, templates having different sizes are prepared as templates prepared in advance. Then, template matching is performed while changing the type and size of the template.

  Note that this template matching may be performed on a video composed of grayscale images photographed by the video photographing unit 100, or a video composed of grayscale images photographed by the video photographing unit 100 is used as a reference marker. The binarized image may be binarized with a threshold value set so that 142 symbols can be distinguished from the surrounding white background. Note that the amount of calculation can be reduced by performing template matching on the binarized video, so that the processing speed can be increased.

  Thus, after the position and design of the reference marker 142 are detected, the reference marker closest to the vehicle 10, that is, the center coordinate (x0, y0) of the extracted largest reference marker 142-1, The lateral side length L0 of the external black square is calculated.

  Similarly, the reference marker 142 on the right side of the road is extracted by the reference marker extraction unit 144 for the video imaged by the third video imaging unit 106, and the reference marker closest to the vehicle 10 is That is, the center coordinate (x1, y1) of the extracted largest reference marker 142-2 and the lateral side length L1 of the outermost black square are calculated.

  At this time, it can be seen that the reference markers 142-1 and 142-2 are extracted from the detected pattern of the reference marker 142.

  Note that the method of extracting the reference marker 142 is not limited to the image processing method described above. That is, for example, the reference marker 142 may be extracted by applying signal processing used for reading by a two-dimensional barcode reader.

  Next, in the vehicle position / traveling direction calculation unit 150, the first video shooting unit 102 is calculated from the lateral side length L0 of the reference marker 142-1 on the left side of the road extracted from the video shot by the first video shooting unit 102. Is calculated from the horizontal length L1 of the reference marker 142-2 on the right side of the road extracted from the video captured by the third video imaging unit 106. A distance d1 between the imaging unit 106 and the reference marker 142-2 is calculated.

  That is, the image capturing unit 100 captures an image of the reference marker 142 in advance while changing the distance to the reference marker 142 by a predetermined amount, and the distance to the reference marker 142 and the horizontal length of the captured reference marker Measure the relationship.

  This measurement result is stored in the vehicle position / travel direction calculation unit 150, and based on the stored data, the calculated distance d0 from the lateral side length L0 of the reference marker 142-1 to the reference marker 142-1. And the distance d1 from the lateral side length L1 of the reference marker 142-2 to the reference marker 142-2 is calculated (step S5 in FIG. 9).

  Here, when at least one of the calculated distances d0 and d1 is smaller than the predetermined distance dth determined in advance (when step S6 in FIG. 9 is YES), the reference marker 142 is extracted. Based on the result, the current position (xc, yc) of the vehicle is calculated (step S7 in FIG. 9).

  Here, it is assumed that the installation position of the first video photographing unit 102, the installation position of the second video photographing unit 104, and the installation position of the third video photographing unit 106 are all equal. That is, it is assumed that the installation position is at point Q in FIG.

  At this time, as shown in FIG. 6, a circle P1 having a radius d0 centered on the reference marker 142-1 on the left side of the road and a distance d1 centered on the reference marker 142-2 on the right side of the road have a radius The position (xc, yc) of the vehicle 10 is uniquely determined as a point C0 on the side closer to the vehicle 10 among the intersection points of the circle P2. The point C0 coincides with the point Q.

  Next, the vehicle position and traveling direction calculation unit 150 calculates the direction ω of the vehicle 10 (step S9 in FIG. 9).

  Here, first, the amount of displacement xh0 in the horizontal direction from the center position of the video (on the line segment VV ′) at that point is calculated from the value of the center coordinates (x0, y0) of the reference marker 142-1.

That is, xh0 is calculated by (Equation 1).
xh0 = m / 2−x0 (Formula 1)
Here, m represents the number of pixels in the horizontal direction of the video shot by the first video shooting unit 102.

  Then, from the calculated displacement amount xh0, the perpendicular length (dx0 in FIG. 7) lowered from the center of the reference marker 142-1 to the optical axis of the first video photographing unit 102 is calculated.

  The perpendicular length dx0 is calculated using data obtained in advance by experiments.

  That is, the image capturing unit 100 captures the reference marker 142 placed at a plurality of distances from the image capturing unit 100 while changing the optical axis direction φ of the image capturing unit 100 by a predetermined amount in the horizontal direction, The distance d from the image capturing unit 100 to the center of the reference marker 142, the optical axis direction φ of the image capturing unit 100, the center coordinates (x0, y0) of the reference marker 142 captured at that time, or the displacement xh0. Measure the relationship with.

  The measurement result is stored in the vehicle position / traveling direction calculation unit 150, and based on the stored data, the reference marker 142- is calculated from the center coordinates (x0, y0) of the reference marker 142-1 or the displacement xh0. A perpendicular length dx0 lowered from the center of 1 to the optical axis of the first video photographing unit 102 is calculated.

  The perpendicular length dx0 calculated in this way, the distance d0 to the center of the reference marker 142-1 calculated earlier, the current position (xc, yc) of the vehicle 10, and the optical axis of the first video photographing unit 102 And the angle θ formed by the optical axis of the second video photographing unit 104, the direction ω of the vehicle 10 is calculated by (Equation 2).

ω = tan ⁻¹ {dx0 / (d0 ² −dx0 ² ) ^1/2 } −sin ⁻¹ (xc / d0) + θ (Formula 2)
On the other hand, when both the distance d0 to the reference marker 142-1 and the distance d1 to the reference marker 142-2 calculated in step S5 are greater than a predetermined distance dth, the vehicle behavior calculation unit 160 , The vehicle speed calculated by the vehicle speed sensor 192 is integrated, and the yaw rate calculated by the yaw rate sensor 193 is integrated.

  Then, the vehicle 10 calculated based on the integrated value of the vehicle speed and the integrated value of the yaw rate in the closest position (xc, yc) of the vehicle 10 calculated based on the extracted position of the reference marker 142 and the closest direction ω. Are added together to calculate the position (xc ′, yc ′) and the direction ω ′ of the vehicle 10 (step S8 in FIG. 9), and then the process proceeds to step S10.

  Note that the current position (xc ′, yc ′) and the direction ω ′ calculated by the vehicle behavior calculation unit 160 are the current position (xc, yc) of the vehicle 10 calculated by the vehicle position / travel direction calculation unit 150. In order to distinguish from the direction ω, they are represented by different symbols.

  Here, the calculation method of the position and direction of the vehicle 10 is changed in accordance with the distance to the reference marker 142 because the distance calculation error may increase if the distance to the reference marker 142 is large. It is.

  Therefore, in such a case, the movement trajectory of the vehicle 10 is calculated based on the integrated value of the vehicle speed and the yaw rate, and the movement trajectory of the vehicle 10 calculated in this way is calculated based on the position of the reference marker 142. The position and direction of the vehicle 10 are updated by adding to the position and direction.

  Then, when the reference marker 142 is detected within a predetermined distance, the position and direction of the vehicle 10 are calculated and corrected again based on the position of the reference marker 142.

  When the position and direction of the vehicle 10 are calculated in the vehicle position / traveling direction calculation unit 150 or the vehicle behavior calculation unit 160, the trigger signal generation unit 146 instructs the superimposed video creation unit 120 to execute superposition. A trigger signal is output.

  When the trigger signal is output (when step S10 in FIG. 9 is YES), the superimposed position calculation unit 122 captures a still image or a moving image representing a virtual object stored in the image information storage unit 130. It is calculated at which position of the video image captured by the unit 100 (step S11 in FIG. 9).

  Note that the still image or moving image to be superimposed is created in advance as follows.

  That is, after confirming safety in an actual road environment, images of a pedestrian 803 jumping out from an intersection with poor visibility and a moving vehicle 804 crossing in front of the user are captured, and the area of the pedestrian is determined from the captured images. A moving image in which only the extracted pedestrian 803 is captured by performing image processing for extracting only the region of the vehicle or image processing for extracting only the region of the vehicle, or a moving image in which only the extracted moving vehicle 804 is captured. Generated and stored in the image information storage unit 130.

  At this time, a predetermined specific value is stored in pixels other than the pedestrian 803 and the moving vehicle 804 in order to indicate that the information is not to be superimposed.

  The walls 801 and 802 arranged before the intersection are created as still images. This may be created by computer graphics (CG). A predetermined specific value is stored in pixels other than the walls 801 and 802 in order to indicate that the information is not to be superimposed.

  Furthermore, the still image and the moving image representing the virtual object are stored together with the information of the shooting direction, so that the images taken from various directions can be generated according to the direction of the vehicle 10. Keep it.

  In the superimposed position calculation unit 122, predetermined virtual objects (pedestrian 803, moving vehicle 804, walls 801 and 802) are arranged at a position where a virtual danger scene is reproduced in the road model of FIG. Based on the calculated position (xc, yc) or (xc ′, yc ′) of the vehicle 10 and the direction ω or ω ′, when the virtual object is photographed by the video photographing unit 100, the video display unit 110 It is calculated in which position of the image the image is displayed.

  The calculation for calculating the superposition position can be executed by simple geometric calculation because the positional relationship between the video photographing unit 100 and the object is clear.

  Next, the superimposition size calculation unit 124 calculates the size of the virtual object to be superimposed based on the positional relationship between the vehicle 10 and the virtual object (step S12 in FIG. 9).

  Specifically, a still image or a moving image stored in the image information storage unit 130 is multiplied by an appropriate magnification in accordance with the distance between the video shooting unit 100 and the virtual object, and is superimposed small when superposed far away. Then, the size of the virtual object is calculated so as to be largely superimposed when superimposed on a close position.

  Further, the superimposition direction calculation unit 126 calculates the superimposition direction of the virtual object (step S13 in FIG. 9).

  Specifically, the superimposition direction is based on the positional relationship between the vehicle position, the direction of the image capturing unit 100 calculated by the traveling direction calculation unit 150 or the vehicle behavior calculation unit 160, and the virtual object to be superimposed. It is calculated as the direction of the virtual object that appears in each video photographing unit (102, 104, 106).

  Next, in the superimposition processing unit 128, an image of the virtual object viewed from the direction calculated by the superimposition direction calculation unit 126 is generated, the image is transformed into the size calculated by the superimposition size calculation unit 124, and superimposed. It is superimposed on the position calculated by the position calculation unit 122 (step S14 in FIG. 9).

  Specifically, this superimposition processing is performed in such a manner that the image of the virtual object viewed from the direction calculated in step S13 at the position calculated in step S11 of the video input from the video photographing unit 100 is obtained in step S12. In the process of replacing the pixel value of a pixel having a value other than a specific value in the superimposed image with the pixel value of the superimposed image by transforming into the size calculated in (3) and superimposing Done.

  The video on which the image representing the virtual object is superimposed is displayed on the first video display unit 112, the second video display unit 114, and the third video display unit 116 in substantially real time (step S15 in FIG. 9).

  When the driver of the vehicle 10 seeing this display notices that the pedestrian 803 or the moving vehicle 804 has jumped out from behind the wall 801, he immediately performs avoidance actions such as braking and steering.

  Here, when it is detected in the operation detection unit 210 that the driver has performed a horn sounding operation by pressing the horn switch 220, an interruption process is started and the process shown in the flowchart of FIG. 10 is executed. Is done.

  That is, when the operation detecting unit 210 detects that the driver has pressed the horn switch 220, the superimposing processing unit 128 superimposes the pedestrian 803 or the moving vehicle 804 (step in FIG. 10). When S17 is YES, the superimposition processing unit 128 performs drawing control of the pedestrian 803 and the moving vehicle 804 corresponding to the horn sounding (step S18 in FIG. 10).

  Here, when the driver of the vehicle 10 notices the pedestrian 803 jumping out or approaching the moving vehicle 804 and makes the horn blow, a scene where the pedestrian 803 notices the horn and stops, or In the superimposition processing unit 128, superimposition is performed so that the movement of the pedestrian 803 and the mobile vehicle 804, which are virtual objects, is controlled so as to reproduce the scene where the moving vehicle 804 notices the sound of the horn and decelerates. Then, after step S18 is executed, the operation returns to the flowchart of FIG.

  Various methods for controlling the movement of the virtual object described above can be considered, and any method may be used.

  That is, by newly generating an image that interpolates between adjacent images in the moving image representing the virtual object, and reproducing the moving image including the interpolated image, the movement of the virtual object is reduced. Can be late.

  On the other hand, by deleting (thinning out) the image forming the moving image from the moving image representing the virtual object every predetermined number of frames, and reproducing the moving image thus obtained, The movement of the virtual object can be made faster.

  Alternatively, when a moving image representing a virtual object is created in advance and stored in the image information storage unit 130, a plurality of moving images in which the virtual object moves at various speeds are created and stored together with information representing the speed of movement. In addition, a moving image having a motion speed suitable for a scene may be selected and reproduced from the plurality of moving images.

  Further, the movement of the virtual object may be controlled by changing the frame rate for reproducing the moving image representing the virtual object.

  On the other hand, when the driver presses the horn switch 220, when the pedestrian 803 or the moving vehicle 804 is not superimposed (when step S17 in FIG. 10 is NO), no processing is performed. Returning to the flowchart of FIG. 9, the operation continues.

  Then, in FIG. 9, when an end instruction such as stopping of the engine of the vehicle 10 is detected, the process ends (when step S <b> 16 in FIG. 9 is YES).

  In this way, by controlling the virtual object superimposing method in accordance with the operation of the driver, a more natural driving scene can be reproduced.

  The operation detection unit 210 has been described as a configuration that detects pressing of the horn switch 220. However, an operation other than pressing of the horn switch 220 may be detected to perform drawing control.

  For example, it is possible to detect that the driver has passed by a headlight switch (not shown) and perform drawing control corresponding to the detected passing.

Further, the drawing control performed in the superimposition processing unit 128 can be performed based on the output other than the operation detection unit 210.
For example, a traffic environment detection unit 230 that detects an operating state of a pedestrian traffic light on a road on which the vehicle 10 is traveling is provided, and drawing control of the pedestrian 803 is performed based on the output of the traffic environment detection unit 230. It can also be done.

  That is, the traffic environment in which the vehicle 10 approaches a pedestrian traffic light provided on a road in a direction orthogonal to the traveling direction of the vehicle 10 within a predetermined distance and detects the operating state of the pedestrian traffic light. When the detection unit 230 detects that the pedestrian traffic light is in the blinking state of the green light, the interrupt process is started and the process shown in the flowchart of FIG. 11 is executed.

  That is, when the in-vehicle camera constituting the traffic environment detection unit 230 detects that the pedestrian traffic light is in the blinking state of the green light, when the pedestrian 803 is superimposed in the superimposition processing unit 128 (see FIG. In addition, when the step S19 of 11 is YES), the superimposition processing unit 128 performs drawing control so that the superimposed pedestrian 803 suddenly starts (step S20 in FIG. 10). The specific drawing control method is as described above.

  And after execution of step S20, it returns to the flowchart of FIG. 9 and operation | movement continues.

  Then, in FIG. 9, when an end instruction such as stopping of the engine of the vehicle 10 is detected, the process ends (when step S <b> 16 in FIG. 9 is YES).

  In this way, since the pedestrian 803 can recognize that the traffic light for the pedestrian is in the blinking state of the green light and reproduce the scene that suddenly started to rush across the road, A scene close to the actual road environment can be reproduced.

  The behavior of the driver of the vehicle 10 when such a scene is encountered can be analyzed in detail.

  In the above-described embodiment, the in-vehicle camera constituting the traffic environment detection unit 230 detects that the pedestrian traffic light is in the blinking state of the green light, but the embodiment is not limited thereto.

  That is, the traffic environment detection unit 230 has a communication function between the inside and the outside of the vehicle 10, and the pedestrian traffic light receives the communication signal output from the traffic environment detection unit 230 of the vehicle 10, When the approach of the vehicle 10 is detected, it may be configured to start blinking the green light.

  Further, the traffic environment detection unit 230 may be provided with a ranging function such as a laser radar in addition to the function of detecting the operating state of the pedestrian traffic light.

  Then, the distance between the pedestrian and the vehicle actually existing around the vehicle 10 is measured, and based on the measured distance, the superimposition processing unit 128 determines the pedestrian 803 and the moving vehicle 804 that are virtual objects. It is also possible to provide a function of performing drawing control with controlled movement.

  Thus, for example, the moving vehicle 804 is superimposed so as to overlap the preceding vehicle actually traveling in front of the vehicle 10, and the traffic environment detection unit 230 includes the distance between the vehicle 10 and the actual preceding vehicle. When the measured distance is less than or equal to a predetermined value, the superimposed moving vehicle 804 can be drawn and controlled as if it has been decelerated rapidly to reproduce a virtual danger scene.

  Although a specific operation flowchart of this process is not shown, it can be realized by performing an interrupt process in accordance with FIGS.

  That is, when the traffic environment detection unit 230 detects that the distance measured by the laser radar is below a predetermined value, the interrupt process is activated.

  When the moving vehicle 804 is superimposed when the interrupt process is started, the superimposition processing unit 128 can reproduce the above-mentioned dangerous scene by performing drawing control in which the moving vehicle 804 decelerates rapidly. it can.

  Thus, according to the function with which the traffic environment detection part 230 is equipped, the scene close | similar to an actual road environment can be reproduced with high reality.

  Furthermore, as an example of controlling the movement of the superimposed image, drawing control for controlling the movement of the pedestrian 803 or the moving vehicle 804, which are virtual objects, is performed based on the behavior of the vehicle 10 calculated by the vehicle behavior calculation unit 160. You may make it perform.

  For example, when the yaw rate sensor 193 detects that the vehicle 10 has started making a right turn at an intersection, the moving vehicle 804 traveling on the opposite lane toward the intersection may enter the intersection at the same time as the vehicle 10. In the superimposition processing unit 128, by performing drawing control for adjusting the speed of the moving vehicle 804, it is possible to reproduce a scene that may cause a right-handed accident. As a result, the behavior of the driver in the dangerous scene can be analyzed in detail in a safe state.

  Although a specific operation flowchart of this process is not shown, it can be realized by performing an interrupt process in accordance with FIGS.

  That is, the traffic environment detection unit 230 is provided with map data of the traveling environment of the vehicle 10 and detects that the vehicle 10 is approaching the intersection based on the map data.

  When the vehicle 10 is approaching the intersection, the vehicle behavior calculation unit 160 analyzes the yaw rate of the vehicle 10 measured by the yaw rate sensor 193 and detects that the vehicle 10 has started a right turn. Then, interrupt processing is started.

  When the interruption process is activated, when the moving vehicle 804 traveling in the opposite lane toward the intersection is being superimposed, the overlapping processing unit 128 causes the moving vehicle 804 to enter the intersection simultaneously with the vehicle 10. By performing the drawing control, the above-mentioned dangerous scene can be reproduced.

  Although not described in the flowchart of FIG. 9, when a trigger signal is output from the trigger signal generation unit 146, the vehicle information recording unit 190 displays the vehicle speed (measured by the vehicle speed sensor 192) representing the behavior of the vehicle, and the yaw Angle (measured with yaw rate sensor 193), pitch angle (measured with pitch angle sensor 194), roll angle (measured with roll angle sensor 195), accelerator opening (measured with accelerator opening sensor 196), brake pedal force (brake pedal force sensor) (Measured in 197) is sequentially acquired and recorded in the vehicle information acquisition unit 191.

  In addition, in the driver information recording unit 200, a face image (captured with the driver's face photographing camera 202), a foot image (captured with the driver's foot photographing camera 203) representing the driver's behavior, and the driver's gaze direction (Measured by the driver's line-of-sight measurement sensor 204) is sequentially acquired and recorded in the driver information acquisition unit 201.

  The vehicle information and the driver information recorded in this way are used for analysis of driving behavior after the series of processes in FIGS.

  Note that the information to be recorded is not limited to the information described here, and an apparatus for acquiring predetermined information is appropriately mounted according to the evaluation contents, and the predetermined information is acquired and recorded.

  Since the timing for reproducing the virtual danger scene is known in advance, recording of vehicle information and driver behavior may be started without waiting for the output of the trigger signal. As a result, vehicle information and driver information can be recorded before the virtual danger scene is reproduced, so the changes in the behavior of the vehicle and driver when encountering the danger scene are analyzed in more detail. can do.

  As described above, according to the vehicular danger scene reproduction device according to the present embodiment, the video display unit placed at a position that obstructs the driver's direct field of view is placed at a position corresponding to the traveling position and traveling direction of the vehicle. Because it is configured to display a video of the vehicle traveling direction with a still image or a moving image representing a virtual object superimposed, a virtual danger scene can be performed regardless of the traveling position or traveling direction of the vehicle in an actually traveling vehicle. Can be reproduced with a high sense of reality.

Further, according to the dangerous situation reproduction apparatus for a vehicle according to the present embodiment, the vehicle position, the position and traveling direction of the vehicle calculated by the moving direction calculation unit, and based on the behavior of the vehicle calculated by the vehicle behavior calculation unit Because it is configured to determine the position, size, and direction of the superimposed image, it reproduces virtual danger scenes with a high sense of reality when driving without any restrictions such as driving lanes or driving speed. can do. Therefore, it is possible to reproduce a virtual danger scene in a more natural driving state, and thereby, it is possible to evaluate a driver's natural reaction.

  As described below, various modifications having the same effect can be realized without changing the basic configuration of the present embodiment.

  In the above description, the vehicle behavior calculation unit 160 calculates the behavior of the vehicle 10 based on the vehicle speed measured by the vehicle speed sensor 192 and the yaw angle measured by the yaw rate sensor 193. The measured pitch angle and the roll angle measured by the roll angle sensor 195 may be used.

  Thus, since the position, size, and direction of the superimposed image can be controlled according to the finer behavior of the vehicle 10, a more natural superimposed image can be displayed.

  In the above description, the position and direction of the vehicle 10 are calculated based on the position of the photographed reference marker 142 when the distance is short according to the distance to the photographed reference marker 142, and when the distance is long. The movement of the vehicle 10 calculated based on the integrated value of the vehicle speed calculated by the vehicle behavior calculating unit 160 and the integrated value of the yaw rate in the nearest position and direction of the vehicle 10 calculated based on the position of the reference marker 142. The position and direction of the vehicle 10 are calculated by adding the trajectory. This is because the vehicle 10 calculated by the integrated value of the vehicle speed after adding the movement trajectory of the vehicle 10 calculated by the vehicle behavior calculation unit 160 is added. The position and direction of the vehicle 10 may be calculated based on the calculation result of the vehicle behavior calculation unit 160 only in a range where the movement distance of the vehicle reaches a predetermined value.

  This eliminates the need to determine the magnitude of the calculated distance by calculating the distance to the reference marker 142 while calculating the position and direction of the vehicle 10 based on the calculation result of the vehicle behavior calculation unit 160. The extraction process of the reference marker 142 can be interrupted for a predetermined period, and the computer load can be reduced.

  Further, the timing generation method for reproducing the virtual danger scene in the superimposition timing calculation unit 140 is not limited to the method using the reference marker 142 described in the present embodiment.

  That is, for example, the position and direction of the vehicle 10 is specified using the GPS (Global Positioning System) used in car navigation, the vehicle speed sensor 192, and the yaw rate sensor 193, and based on the specified information, the virtual Superimposition timing for reproducing a dangerous scene may be generated.

  Alternatively, information emitted from a radio beacon or an optical beacon transmitter installed on the road shoulder is received by a receiver installed in the vehicle 10, the current position of the vehicle 10 is specified, and based on the specified position information, Superimposition timing for reproducing a virtual danger scene may be generated.

  On the other hand, a superimposition timing for reproducing a virtual danger scene may be generated based on not only the position of the vehicle 10 but also the behavior of the driver and the behavior of the vehicle. At this time, sensors provided in the vehicle information recording unit 190 and the driver information recording unit 200 are used.

  For example, the driver's line-of-sight measurement sensor 204 continuously measures the driver's line-of-sight direction, and when the measured driver's line-of-sight direction is off the road, it is determined that the driver is looking aside. If superimposition is started, a virtual danger scene can be reproduced with the driver's side-arm as a trigger.

  Alternatively, when the vehicle speed detected by the vehicle speed sensor 192 is larger than a predetermined value, it is possible to set the superimposition start timing to reproduce a virtual dangerous scene.

  In this way, it is possible to generate the timing for superimposing by an appropriate method according to the content to be evaluated and the danger scene to be reproduced.

  The virtual danger scene that can be reproduced by the present invention is not limited to the pedestrian 803 or the moving vehicle 804 jumping out scene described in the embodiment.

  That is, according to the present invention, an appropriate superimposed object can be set according to the driving scene set based on the behavior of the driver to be evaluated, and the superimposed image creating unit 120 can perform superimposition with high realism. You can create a video.

  In addition, the specific component of this invention is not limited to what is shown in FIG. 1 if it has the same effect. For example, the video display unit 110 may be configured by a curved display, or may be projected from a projector installed on the vehicle 10 onto a screen installed on the hood.

  Further, it goes without saying that the present invention can be implemented in a mode in which modifications and improvements are made based on the knowledge of those skilled in the art without departing from the scope of the claims.

DESCRIPTION OF SYMBOLS 10 Vehicle 100 Image | video imaging | photography part 110 Image | video display part 120 Superimposition image | video production | generation part 122 Superimposition position calculation part 124 Superimposition size calculation part 126 Superimposition direction calculation part 128 Superimposition processing part 130 Image information storage part 140 Superimposition timing calculation part 142 Reference marker 144 Reference marker 144 Extraction unit 146 Trigger signal generation unit 150 Vehicle position / travel direction calculation unit 160 Vehicle behavior calculation unit 190 Vehicle information recording unit 200 Driver information recording unit 210 Operation detection unit 220 Horn switch 230 Traffic environment detection unit

Claims (12)

An image capturing unit that is installed toward the traveling direction of the vehicle and captures an image of the traveling direction of the vehicle;
A vehicle position for calculating a position and a traveling direction of the vehicle, a traveling direction calculation unit;
A vehicle behavior calculation unit for calculating the behavior of the vehicle;
An image information storage unit that stores in advance an image representing a virtual object to be superimposed on the video;
In the image, in the position calculated based on the vehicle position and the traveling direction calculated by the vehicle position and traveling direction calculation unit, and the behavior of the vehicle calculated by the vehicle behavior calculating unit, A superimposed video creation unit for superimposing the image;
The vehicle position, and the position and traveling direction of the vehicle calculated by the moving direction calculation unit, based on the behavior of the vehicle calculated by the vehicle behavior calculation unit, with respect to the superimposed image generator, the image A superimposition timing calculation unit that instructs to superimpose,
A vehicular dangerous scene reproduction device comprising: a video display unit that is arranged so as to block the direct field of view of the driver of the vehicle and displays the video created by the superimposed video creation unit in substantially real time. . The video photographing unit is composed of a plurality of video cameras, and the optical axes of adjacent video cameras form a predetermined angle in the horizontal direction so that the photographing fields of the plurality of video cameras do not overlap each other. And
The video display unit includes a plurality of rectangular displays, and the short sides of the adjacent displays are substantially in contact with each other on the hood of the vehicle so that the short sides are substantially perpendicular to the ground. The vehicular danger scene according to claim 1, wherein the long sides of the displays that are arranged and adjacent to each other form a predetermined angle so as to face the driver of the vehicle. Reproduction device.   3. The danger scene reproduction device for a vehicle according to claim 2, wherein an angle formed between the optical axes of the video cameras adjacent to each other in a horizontal direction is substantially equal to an angle formed between long sides of the displays adjacent to each other. The superimposed image creating unit is based on the vehicle position, the traveling direction calculated by the traveling direction calculating unit, and the traveling direction of the vehicle and the behavior of the vehicle calculated by the vehicle behavior calculating unit.
A superposition position calculation unit for calculating a position at which the image is superposed;
A superimposition size calculator for calculating a superimposition size of the image;
A superimposition direction calculation unit that calculates a direction in which the image is superimposed;
The vehicular dangerous scene reproduction device according to any one of claims 1 to 3, wherein A vehicle information recording unit that acquires and records vehicle information of the vehicle at least during a period in which a video on which the image is superimposed is displayed on the video display unit;
A driver information recording unit that acquires and records the behavior of the driver of the vehicle at least during a period in which the video on which the image is superimposed is displayed on the video display unit;
The vehicle dangerous scene reproduction device according to any one of claims 1 to 4, further comprising:   The position of the vehicle is at least two of the plurality of reference markers having different patterns installed at predetermined positions on the road in the vehicle position / traveling direction calculation unit. The vehicular dangerous scene reproduction device according to any one of claims 1 to 5, wherein the vehicular danger scene reproduction device is determined based on a position of the reference marker.   The direction of the vehicle is at least one of a plurality of reference markers having different symbols installed at predetermined positions on the road in the vehicle position / traveling direction calculation unit. The vehicular dangerous scene reproduction device according to any one of claims 1 to 6, wherein the vehicular danger scene reproduction device is determined based on a position of the reference marker.   After the vehicle position and travel direction calculation unit calculates the vehicle position and travel direction, the vehicle position and travel direction are determined based on the vehicle behavior calculated by the vehicle behavior calculation unit for a predetermined period. The vehicle dangerous scene reproduction device according to any one of claims 1 to 7, wherein a direction is calculated.   The vehicular dangerous scene reproduction device according to claim 8, wherein the predetermined period is a period in which a reference marker is detected within a predetermined distance from the vehicle.   9. The vehicular dangerous scene reproduction device according to claim 8, wherein the predetermined period is a time until the moving distance of the vehicle calculated by the vehicle behavior calculation unit reaches a predetermined value.   The vehicle behavior calculation unit calculates the behavior of the vehicle based on all or part of the vehicle speed, yaw angle, pitch angle, and roll angle of the vehicle. The vehicle dangerous scene reproduction device according to claim 1. An operation detection unit that detects a predetermined operation of the driver, and a traffic environment detection unit that detects traffic environment information related to the traveling of the vehicle,
When a predetermined operation is detected by the operation detection unit, when a predetermined traffic environment is detected by the traffic environment detection unit, or when a predetermined behavior of the vehicle is detected by the vehicle behavior calculation unit The superimposed image creation unit controls the movement of the image to be superimposed so as to correspond to the predetermined operation, the predetermined traffic environment information, or a predetermined behavior of the vehicle. The vehicle dangerous scene reproduction device according to any one of 1 to 11.