KR20130089139A - Augmented reality head-up display apparatus and method for vehicles - Google Patents

Abstract

The present invention provides a vehicle augmented reality head-up display device and method for adaptively shifting the position and direction around a driver's field of view to reduce the error of alignment of virtual object information and real-world information. The proposed device estimates the driver's line of sight and calculates the viewing angle based on the face direction and the pupil center position detected based on the driver's face image, and the driver's seat based on the viewing direction and the viewing angle from the viewing angle calculator. A matching unit for matching the position of the front real-world information and the position of the virtual object information corresponding to the front of the driver's seat, and a display unit for displaying the result matched in the matching unit, the rotation and position can be changed.

Description

Augmented reality head-up display apparatus and method for vehicles}

The present invention relates to an augmented reality head-up display device and method for a vehicle, and more particularly, to provide a driver with various convenience information and safety driving information to a driver in a vehicle in various driving environments such as day and night driving, fog and heavy rain. The present invention relates to a vehicle augmented reality head-up display device and method for providing augmented reality in accordance with the real-world information.

Conventionally developed vehicle information providing apparatus is roughly classified into three types.

First, there is a 'vehicle terminal' that is mainly installed in the lower right center of the driver's seat to provide various information to the driver. When the driver normally keeps looking ahead and the information is provided from the vehicle terminal, the driver turns his head or looks to the place where the terminal is located to view the provided information. At this time, the attention of the driver is dispersed, there is a risk of accident occurrence.

Second is the 'head-up display (HUD) device' mainly installed in luxury vehicles. The HUD device displays simple vehicle information such as arrow information for guiding a route change and text information indicating a speed at the top of the dashboard. Therefore, the driver can view the information provided from the driver's front position while watching the front without changing the line of sight or the head to view the provided information, thereby reducing the risk of distraction. However, since the location of the information provided is provided to the driver without matching the location of forward real-world information (eg, roads, buildings, vehicles, signs, etc.), the driver is burdened with recognizing the information provided, rather than driving. It can reduce safety and convenience. In addition, when there is no location matching with the real world, it is difficult to provide various information (eg, an obstacle such as a lane, a pedestrian or a vehicle ahead) through the HUD.

Third, there is an 'augmented reality HUD device' that is currently being researched and developed for incorporation into vehicles. The augmented reality HUD device provides the driver with safe driving information by matching the position of the driver's forward real world information with the information expressed in the HUD based on the driver's field of view. This augmented reality HUD device provides safe driving information in accordance with the location of the real-world information, the driver is more likely to recognize the information provided. However, it is not easy to provide augmented reality information to exactly match the location of real-world information. For this purpose, it is necessary to accurately detect and recognize the position and posture of the vehicle, and also to accurately detect and recognize the position and posture of the driver's head and gaze. However, due to technical limitations, errors in position and posture accuracy occur, and these errors reduce the accuracy of position matching between real world information and safe driving information.

An example of such an augmented reality HUD device is disclosed in Korean Patent Laid-Open Publication No. 10-2009-0076242 (vehicle head-up display device and its operation control method). The vehicular head-up display device disclosed in Korean Laid-Open Patent Publication No. 10-2009-0076242 adjusts the image information projection angle of the HUD unit by adjusting the position of the HUD unit up / down / left / right according to the driver's eye movement. The vehicle head-up display device disclosed in Korean Unexamined Patent Publication No. 10-2009-0076242 can only change the display position of the virtual safe driving information by adjusting the projection angle of the HUD unit according to the movement of the driver's eyeball. Can't do it at all.

The present invention has been proposed to solve the above-mentioned conventional problems, and is capable of adaptively shifting the position and direction around the driver's field of view to reduce the error of the alignment of the virtual object information and the real-world information. An object of the present invention is to provide an augmented reality head-up display device and method.

In order to achieve the above object, the augmented reality head-up display device for a vehicle according to a preferred embodiment of the present invention estimates the driver's gaze direction based on a face direction and a pupil center position detected based on a driver's face image. A viewing angle calculator configured to calculate a viewing angle; A matching unit for matching the position of the real world information in front of the driver's seat with the position of the virtual object information corresponding to the front of the driver's seat based on the viewing direction and the viewing angle from the viewing angle calculator; And a display unit configured to display the result matched by the matching unit, and to rotate and shift the position.

Rotation of the display portion includes movement in the pitch, roll, and yaw directions, and positional shifts of the display portion include movement of left, right, up, down, front, and back.

The display portion is rotated or shifted in position by manual or automatic adjustment by the driver.

Preferably, a matching error calculation unit for calculating a matching error in the matching unit; And an adjusting unit for rotating or shifting the display unit based on the matching error calculated by the matching error calculating unit. In this case, the matching unit realigns the position of the real world information and the position of the virtual object information according to the adjustment in the adjusting unit. The adjusting unit rotates or shifts the display unit until the calculated matching error is smaller than the error range.

On the other hand, the display unit is operated in a scroll type or a folder type at an initial position, and becomes a displayable state. In this case, the display portion is scrolled or folded in the upward or downward direction from the initial position by the driver's manual operation. Alternatively, the display portion is automatically scrolled or folded in an upward or downward direction from the initial position by the driver's command.

The virtual object information includes at least one of a static object in front of the driver's seat through the map information and a moving object in front of the driver's seat by image recognition.

In the augmented reality head-up display method for a vehicle according to a preferred embodiment of the present invention, the viewing angle calculator estimates the driver's gaze direction and calculates the viewing angle based on the face direction and the pupil center position detected based on the driver's face image. Making; Matching the position of the real world information in front of the driver's seat with the position of the virtual object information corresponding to the front of the driver's seat based on the eye's direction and the viewing angle at the step of estimating the eyeline direction and calculating the viewing angle; Displaying, by the display unit, the matched result in the matching step; And the adjusting unit performing one or more of rotation and position shift with respect to the display unit according to the result matched in the matching step.

In the step of performing one or more of rotation and position shifts, the rotation includes movement in the pitch, roll, and yaw directions, and in the step of performing one or more of rotation and position shifts, the position shift is left Includes right, up, down, front and back movements.

Preferably, the step of performing one or more of rotation and position shifting comprises: calculating a matching error by matching; And rotating or displacing the display unit based on the calculated matching error. In this case, the matching step realigns the position of the real world information and the position of the virtual object information according to the adjustment in the step of performing one or more of rotation and positional variation.

Performing one or more of rotation and position shifting rotates or position shifts the display until the calculated matching error is less than the error range.

The displaying step is performed after the display portion is operated in a scroll type or a folder type at an initial position and becomes a displayable state.

According to the present invention of such a configuration, according to the necessity of providing guidance information on the safety and convenience of the driver, the driver can use the augmented reality display when desired, providing driver-oriented information.

In addition, the driver may directly adjust the angle or position of the display in order to provide necessary guidance information more accurately matching the driver's field of view. Therefore, the present invention can adjust the rotation and horizontal, vertical, front and rear position shift for each axis.

The present invention capable of such an adaptive position / direction shift can reduce the position error between the virtual object information and the real world information provided to the driver in the head-up display. Accordingly, the driver's cognitive load on the provided information can be reduced, and the risk of accidents due to misunderstanding can be reduced.

The present invention can achieve the convenience and safety of the driver in all driving environments, including nighttime, bad weather difficult to secure visibility.

1 is a view showing an example of the use of the display unit employed in the augmented reality head-up display device for a vehicle according to an embodiment of the present invention.
2 is a view showing an example of mounting the display unit employed in the augmented reality head-up display device for vehicles according to an embodiment of the present invention.
3 is a view showing an example of adjusting the direction of the display unit employed in the augmented reality head-up display device for vehicles according to an embodiment of the present invention.
4 is a view showing an example of adjusting the position of the display unit employed in the augmented reality head-up display device for vehicles according to an embodiment of the present invention.
5 is a block diagram of an augmented reality head-up display device for a vehicle according to an embodiment of the present invention.
6 is a flowchart illustrating a method of displaying augmented reality head-up vehicle for a vehicle according to an embodiment of the present invention.
7 to 9 are diagrams employed in the description of the augmented reality head-up display method for a vehicle according to an embodiment of the present invention.

In order to solve the problems of the conventional augmented reality HUD device described above, there is a need for a method of reducing the matching error between the location of the real world information and the safety operation information. One of the causes of this error is that the display panel of the augmented reality head-up display device is fixed. Due to the problems mentioned above, the fixed augmented reality HUD device has an error in the process of matching the real world information and the safety operation information provided from the HUD.

Therefore, if the driver adjusts the direction or position of the HUD according to his or her posture or position, the real world information and safe driving information may be more accurately matched to the driver's field of view.

Accordingly, the present invention has been implemented to reduce the error caused by position matching with real-world information and to provide information according to the driver's view by adaptively changing the angle or position of the display according to the driver's view.

Hereinafter, with reference to the accompanying drawings, a vehicle augmented reality head up display device and method of the present invention will be described. Prior to the detailed description of the present invention, terms and words used in the present specification and claims should not be construed as limited to ordinary or dictionary terms. Therefore, the embodiments described in this specification and the configurations shown in the drawings are merely the most preferred embodiments of the present invention and do not represent all the technical ideas of the present invention. Therefore, It is to be understood that equivalents and modifications are possible.

1 is a view showing an example of the use of the display unit employed in the augmented reality head-up display device for vehicles according to an embodiment of the present invention.

Convenient information (driving route information, point of interest (POI) information, information board information, etc.) provided through the augmented reality head-up display device for a vehicle according to an embodiment of the present invention and safety driving information (lane information, vehicle information, pedestrian information) Etc.), the display unit 10 at the initial position should be moved to a displayable position by operating the scroll type or the folder type.

In FIG. 1, the scroll type refers to a form that descends from the top or ascends from the bottom, and the folder type refers to a form in which a folded state of a sunshade mounted on the upper end of the driver's seat is unfolded.

Operation in the scroll type or folder type is performed automatically or manually. Automatically, the display unit 10 moves to a displayable position by a motor (not shown) or the like provided therein. In the manual operation, the driver directly holds the display unit 10 by hand and scrolls upward or downward or directly unfolds the folded state.

2 is a view showing a mounting example of the display unit employed in the vehicle augmented reality head-up display device according to an embodiment of the present invention, Figure 3 is a display unit employed in a vehicle augmented reality head-up display device according to an embodiment of the present invention FIG. 4 is a view illustrating an example of adjusting a direction of the display unit, and FIG. 4 is a view illustrating an example of adjusting a position of a display unit employed in the augmented reality head-up display device for a vehicle according to an exemplary embodiment of the present invention.

The display unit 10 employed in the augmented reality head-up display device for a vehicle according to an embodiment of the present invention may be installed in a dashboard or a windshield in front of a driver's seat. The display unit 10 is preferably made of a transparent material. In FIG. 2, it is assumed that the display unit 10 is mounted near the sun shade in front of the driver's seat or on the windshield 12.

The display unit 10 can both rotate and change position. Referring to FIG. 3, the display unit 10 may rotate in a pitch (x-axis), roll (y-axis), and yaw (z-axis) direction. Referring to FIG. 4, the display unit 10 may move left, right, up, down, front and back. Here, the left or right positional movement is called pan movement, the up or down positional movement is called tilt movement, and the forward or backward positional movement is called dolly movement.

In this way, the display unit 10 adjusts the position of the up, down, left, right, front and back and / or rotates in the x, y, and z-axis directions according to the driver's eyes. By adjusting the position and direction of the display unit 10 to the driver's field of view, the position of the virtual object information can be precisely aligned with the position of the real world information. Accordingly, the driver may be provided with more accurate augmented reality information, so that the driver may be provided with safe driving information more accurately and easily.

As described above, the position and direction of the display unit 10 may be manually adjusted by the driver, or may be automatically adjusted according to the driver's gaze.

5 is a block diagram of an augmented reality head-up display device for a vehicle according to an embodiment of the present invention.

The augmented reality head-up display device for a vehicle according to an embodiment of the present invention, the display unit 10, the photographing unit 20, the face direction detection unit 22, the pupil center position detection unit 24, the viewing angle calculator 26, the virtual object The generation unit 28, the navigation unit 30, the matching unit 32, the matching error calculating unit 34, and the adjusting unit 36 are included.

The photographing unit 20 is installed on the dashboard in front of the driver's seat to face the driver. The photographing unit 20 includes at least one camera. The photographing unit 20 photographs a driver's face image and outputs a plurality of captured face images. For example, the photographing unit 20 first captures an image of the driver's front face, and then faces up, down, and left and right in consideration of a face direction and a gaze direction that can be moved to look out of the vehicle on a shield while driving. Capture images and images from various angles of view. At this time, the facial direction values of the top, bottom, left and right and various angle values in the gaze direction use experimental threshold values obtained by driving experiments for various experimenters.

The face direction detection unit 22 detects the driver's face direction from the plurality of captured face images from the photographing unit 20. For example, the face direction detection unit 22 determines facial part feature points such as eyebrows, eyes, nose, and mouth, respectively, from the captured multiple face images received and models the driver's face through these feature points. The face direction detection unit 22 detects the driver's face three-dimensional direction (position and posture) using the generated face model.

When the face direction is determined by the face direction detector 22, the pupil center position detector 24 detects the pupil center position of the driver using the estimated eye position values corresponding to the feature points of the eyes and the face direction. In other words, when the face direction detection unit 22 determines the face direction, the pupil center position detection unit 24 performs the pupil center position detection operation based on various kinds of information from the face direction detection unit 22. Of course, if necessary, when the face direction detection unit 22 starts the face direction detection operation, the pupil center position detection unit 24 performs the pupil center position detection operation based on the information from the photographing unit 20. It's okay.

Accordingly, in FIG. 5, the face direction detector 22 and the pupil center position detector 24 are shown as separate components, but the face direction detector 22 and the pupil center position detector 24 are collectively referred to as one block. It is also possible to detect the direction of the face and the center of the pupil.

The viewing angle calculator 26 calculates the viewing angle after estimating the driver's gaze direction based on the face direction by the face direction detector 22 and the pupil center position by the pupil center position detector 24. Preferably, the viewing angle calculator 26 estimates the head center coordinates and the eye center coordinates based on the human face morphology and structure from the input face direction and the pupil center position. Then, the viewing angle calculator 26 estimates (detects) the gaze direction by using the estimated coordinate values (ie, the head center coordinate value and the eyeball center coordinate value) and the eye center position. The viewing angle calculator 26 calculates the viewing angle of the driver in consideration of the driver's face direction and the gaze direction.

The virtual object generator 28 generates a virtual object through image recognition using a camera sensor or guide plate information recognition using 3D map information. In more detail, the virtual object generator 28 recognizes a moving object such as a pedestrian or a preceding vehicle in front of the driver's seat using a camera sensor, and uses static information such as a sign board, a road sign, a building, etc. using three-dimensional map information. Can recognize objects Accordingly, the virtual object generation unit 28 generates information about the virtual object including the static object in front of the driver's seat through the three-dimensional map information and the moving object in front of the driver's seat by image recognition. The information about the virtual object includes a position coordinate value for each virtual object.

The navigation unit 30 calculates and outputs a position value of the vehicle in real time.

The matching unit 32 matches the position of the real world information in front of the driver's seat with the position of the virtual object information corresponding to the driver's seat in front of the virtual object generator 28 based on the viewing direction and the viewing angle from the viewing angle calculator 26. do. That is, the matching unit 32 receives the virtual object information from the virtual object generating unit 28, and adjusts the coordinate values of the real world information and the coordinate values of the virtual object information according to the viewing direction and the viewing angle from the viewing angle calculator 26. Match. Here, the real-world information means actually seen through the windshield in front of the driver's seat.

More specifically, the matching unit 32 calculates by mapping the driver's head direction and the gaze position value to real world coordinate values in consideration of the relative positional relationship between the vehicle and the driver based on the vehicle position value from the navigation unit 30. . In addition, the matching unit 32 may calculate the initial values of the direction and the position of the display unit 10 by automatically mapping them to real world coordinate values based on the vehicle position values. Meanwhile, the matching unit 32 maps and matches the virtual object information coordinate values according to the driver viewing angle range and the previously calculated real world mapping position coordinate values through a camera calibration and registration method.

The matching error calculator 34 calculates a matching error generated in the matching process in the matching unit 32.

The adjusting unit 36 rotates or shifts the position of the display unit 10 based on the matching error calculated by the matching error calculating unit 34. The adjusting unit 36 rotates or shifts the display unit 10 until the matching error calculated by the matching error calculator 34 is smaller than the preset error range. Accordingly, the matching unit 32 realigns the position of the real world information and the position of the virtual object information when there is an adjustment in the adjusting unit 36. In this way, the coordinate values of the virtual object are precisely matched with the coordinate values of the real world information.

This time, the augmented reality head-up display method for a vehicle according to an embodiment of the present invention will be described with reference to the flowchart of FIG. 6. 7 to 9 are diagrams employed in the description of the augmented reality head-up display method for a vehicle according to an embodiment of the present invention.

First, the photographing unit 20 photographs a driver's face image and outputs a plurality of captured face images.

Subsequently, the face direction detection unit 22 detects the driver's face direction based on the plurality of captured face images from the photographing unit 20 (S10).

The pupil center position detection unit 24 uses the estimated eye position values according to the feature points of the eye and the face direction based on the multiple captured face images from the photographing unit 20 or the information from the face direction detection unit 22. The driver's pupil center position is detected (S20).

The information corresponding to the detected face direction and the pupil center position is transmitted to the viewing angle calculator 26.

Accordingly, the viewing angle calculator 26 calculates the viewing angle after estimating the driver's gaze direction based on the detected face direction and the pupil center position (S30).

Thereafter, the virtual object generator 28 generates a virtual object through image recognition using a camera sensor or guide plate information recognition using 3D map information (S40).

Subsequently, the matching unit 32 positions the position of the real world information in front of the driver's seat and the position of the virtual object information corresponding to the driver's seat in front of the virtual object generator 28 based on the viewing direction and the viewing angle from the viewing angle calculator 26. Match (S50).

When the matching of the matching unit 32 is completed as described above, the matching error calculator 34 calculates the matching error in the matching unit 32, and the adjusting unit 36 displays the display unit 10 according to the calculated matching error value. Determine whether the rotation and position shift for the corresponding to perform the corresponding operation.

That is, if the calculated matching error value is smaller than the preset error range (“Yes” in S60), the adjusting unit 36 determines that the rotation and positional shift of the display unit 10 are not necessary. Accordingly, the screen 14 as illustrated in FIG. 7 is displayed on the display unit 10 (S70). Here, the screen 14 of FIG. 7 indicates that the position of the real world information and the position of the virtual object information are properly matched and displayed in the driver's field of view.

On the contrary, when the calculated matching error value is out of the preset error range (“No” in S60), as shown in FIG. 8, the matching between the position of the real world information and the position of the virtual object information is not properly performed. Accordingly, the adjusting unit 36 rotates or shifts the display unit 10 as illustrated in FIG. 9 until the matching error value is smaller than the preset error range, and the matching unit 32 adjusts the adjusting unit 36. The position of the real-world information and the position of the virtual object information are realigned according to the adjustment of (S80).

Here, a method of adjusting the direction and position of the display unit 10 will be described once again.

First, the manual adjustment method is explained. In the first manual adjustment method, the driver directly adjusts the direction and position of the display unit 10 to the driver's viewing angle and gaze by hand. In the second manual adjustment method, the driver adjusts the direction and position of the display unit 10 through a motor (not shown) and a button (not shown) that can move the directions and positions of each x, y, and z by a predetermined interval. do. Motors and buttons not shown may be provided in the display unit 10, and even if not shown separately, those skilled in the same industry can fully understand the technology.

This time, the method to adjust automatically is demonstrated. The automatic adjustment method automatically extracts the driver's viewing angle and gaze direction through the face direction based on the driver's eye position and face model without the driver's intervention and automatically adjusts the position and direction of the display unit 10 according to the extracted value. Adjusted.

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and also implemented in the form of a carrier wave (for example, transmission over the Internet) . The computer readable recording medium may also be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. You must see.

10: display unit 20: photographing unit
22: face direction detection unit 24: pupil center position detection unit
26: viewing angle calculator 28: virtual object generation unit
30: navigation unit 32: matching unit
34: matching error calculation unit 36: adjustment unit

Claims (20)

A viewing angle calculator configured to estimate a gaze direction of a corresponding driver and calculate a viewing angle based on a face direction and a pupil center position detected based on the driver's face image;
A matching unit for matching the position of the real world information in front of the driver's seat with the position of the virtual object information corresponding to the front of the driver's seat based on the viewing direction and the viewing angle from the viewing angle calculator; And
And a display unit configured to display a result matched by the matching unit, the display unit being capable of rotation and shifting of position. The method according to claim 1,
Rotation of the display unit is a vehicle augmented reality head-up display device comprising a movement in the pitch (pitch), roll (roll), yaw direction. The method according to claim 1,
Position shift of the display unit is a vehicle augmented reality head-up display device, characterized in that it comprises the movement of the left, right, up, down, front, back. The method according to claim 1,
The display unit is augmented reality head up display device for the vehicle, characterized in that the rotation or position shift by the manual operation of the driver. The method according to claim 1,
The display unit is a vehicle augmented reality head up display device, characterized in that the rotation or position shift by automatic adjustment. The method according to claim 1,
A matching error calculating unit calculating a matching error in the matching unit; And
And an adjusting unit for rotating or shifting the display unit based on the matching error calculated by the matching error calculating unit.
And the matching unit realigns the position of the real world information and the position of the virtual object information according to the adjustment by the adjusting unit. The method of claim 6,
And the adjusting unit rotates or shifts the display unit until the calculated matching error is smaller than an error range. The method according to claim 1,
The display unit, the augmented reality head-up display device for a vehicle, characterized in that the display unit is operated in a scroll type or a folder type at the initial position. The method according to claim 8,
The display unit is augmented reality head-up display device for a vehicle, characterized in that the state in which the scrolled or folded in the upward or downward direction from the initial position by the manual operation of the driver. The method according to claim 8,
The display unit is augmented reality head-up display device for a vehicle, characterized in that the scrolled or folded state in the upward or downward direction from the initial position automatically by the command of the driver. The method according to claim 1,
The virtual object information includes at least one of a static object in front of the driver's seat through the map information and a moving object in front of the driver's seat by image recognition. Calculating, by the viewing angle calculator, a gaze direction of the corresponding driver based on the detected face direction and the pupil center position based on the face image of the driver;
Matching the position of the real-world information in front of the driver's seat with the position of the virtual object information corresponding to the front of the driver's seat based on the eye's direction and the viewing angle in the step of estimating the eyeline direction and calculating the viewing angle;
Displaying, by a display unit, a result matched in the matching step; And
And an adjusting unit, performing one or more of rotation and position shift with respect to the display unit according to the result matched in the matching step. The method of claim 12,
And in the step of performing at least one of the rotation and position shift, the rotation includes movement in a pitch, roll, and yaw direction. The method of claim 12,
And the position shift in the step of performing one or more of the rotation and position shift comprises moving left, right, up, down, front and back. The method of claim 12,
Performing at least one of the rotation and position shift,
Calculating a matching error by the matching step; And
Rotating or displacing the display unit based on the calculated matching error;
And the matching step realigns the position of the real world information and the position of the virtual object information according to the adjustment in the step of performing one or more of the rotation and position shift. 16. The method of claim 15,
And performing one or more of the rotation and position shifting comprises rotating or displacing the display unit until the calculated matching error is less than an error range. The method of claim 12,
And the displaying step is performed after the display unit is operated in a scroll type or a folder type at an initial position and becomes a displayable state. 18. The method of claim 17,
The display unit augmented reality head-up display method for the vehicle, characterized in that the state in which the scroll or folded in the upward or downward direction from the initial position by the manual operation of the driver is unfolded. 18. The method of claim 17,
The display unit augmented reality head-up display method for the vehicle, characterized in that the scrolled or folded state in the upward or downward direction from the initial position automatically unfolded by the command of the driver. The method of claim 12,
And the virtual object information comprises at least one of a static object in front of the driver's seat through map information and a moving object in front of the driver's seat by image recognition.

Images