JP2020142792A - AR display device - Google Patents

Abstract

To provide a technique capable of realizing the suitable AR presentation by increasing the visibility of a video including an object and an AR image by a user (driver or the like) for an AR display device.SOLUTION: A head up display (HUD) device 1 being an augmented reality (AR) display device mounted on an on-vehicle system 100 comprises: an image selection unit 11 which inputs an image captured by a camera 2, selects and extracts a region of an object from the image; a visibility determination unit 12 which determines the visibility of a user for the region of the object on the basis of the index value; an AR image generation unit 13 which generates an AR image for the object by performing image processing for increasing the visibility on the basis of the visibility of the region of the object; and an AR display unit 15 which displays the AR image so as to be overlapped on a screen 5, and performs control so as to change the presence/absence of the display of the AR image and the content of the image processing in accordance with the visibility.SELECTED DRAWING: Figure 1

Description

The present invention relates to a display device technology, and relates to a display technology for providing a user with augmented reality (AR) image information.

Examples of AR display devices and their systems having a function of providing AR image information include those that present AR image information to a driver for the purpose of driving support, navigation, provision of related information, etc. in an in-vehicle system of an automobile. Be done. For example, there is an in-vehicle head-up display (HUD: Head Up Display) device having an AR display function (AR-HUD). In the AR-HUD, the AR image is projected and displayed so as to be superimposed as a virtual image on the actual view of the driver's field of view, which is an outside world image transmitted through the screen of the front shield of the driver's seat. Examples of AR image information include vehicle information such as vehicle speed, vehicle traveling direction arrow images for driving support and navigation, caution information such as pedestrians, information such as signs and signs, and the like. As the original data of the AR image information, for example, information on an in-vehicle sensor, map information of a car navigation system, and the like are used.

As an example of the prior art relating to the AR display device, Japanese Patent Application Laid-Open No. 2016-95688 (Patent Document 1) can be mentioned. In Patent Document 1, when a signboard or the like outside the vehicle is photographed by an in-vehicle camera as an in-vehicle information display device, the image of the signboard or the like in the photographed image is accurately recognized, and the user understands the recognition result. It is described that the image is converted into a possible image and displayed. Patent Document 1 describes that a character string included in a signboard is translated and displayed.

Japanese Unexamined Patent Publication No. 2016-95688

In the conventional AR display device such as an in-vehicle HUD device, regarding the presentation of the image including the AR image on the screen, there is a problem regarding the visibility of the image including the object and the AR image by the user (driver, etc.) as follows. There is. In other words, there are problems regarding the ease of recognition and information processing by the user, the comfort of using AR, and the like.

For example, in a conventional in-vehicle HUD device, an AR image relating to an object is generated as it is based on an image including an object in the outside world taken by the camera, and the AR image is superimposed and displayed on the screen. At that time, for example, there is an object such as a signboard that the driver is interested in in the screen and the field of view, but the object may be difficult or invisible to the driver. Alternatively, even if an AR image relating to the object is presented, the AR image may be difficult to see or may not be visible. For example, depending on the driving situation, the driver may not be able to read the characters written on an object such as a signboard. Further, for example, although there is a signboard relatively far from the own vehicle, the information on the signboard may not be visible unless it is approached to some extent. In such a case, the driver cannot recognize the object or cannot obtain information about the object, which is inconvenient.

An object of the present invention is to provide a technique for improving the visibility of an image including an object and an AR image by a user (driver or the like) with respect to an AR display device and realizing suitable AR presentation.

A typical embodiment of the present invention is an AR display device, which has the following configuration.

The AR display device of one embodiment is an AR display device having an AR function that superimposes and displays augmented reality image information on a screen that transmits an actual view of the outside world, and inputs an image taken by a camera. An image selection unit that selects and extracts a predetermined object area from the image, and a visibility determination unit that determines the user's visibility with respect to the extracted area of the object based on an index value. AR image generation that generates an AR image that is image information of the augmented reality related to the object by performing image processing processing for enhancing the visibility of the area of the object based on the visibility of the region of the object. It has a unit and an AR display unit that superimposes and displays the AR image on a screen that transmits the actual view of the outside world, and depending on the visibility, whether or not to display the AR image and the image processing process. Control to change the content.

According to a typical embodiment of the present invention, it is possible to improve the visibility of an image including an object and an AR image by a user with respect to an AR display device and realize a suitable AR presentation.

It is a figure which shows the structure of the in-vehicle system including the HUD device which is the AR display device of Embodiment 1 of this invention. It is a figure which shows typically the arrangement composition example of each part which looked at the vicinity of the driver's seat of an automobile from the side in Embodiment 1. FIG. It is a figure which shows the flow of the main processing in Embodiment 1. FIG. It is a figure which shows the example of the screen seen from the driver in Embodiment 1. FIG. It is a figure which shows the example of the area of a screen in Embodiment 1. FIG. It is a figure which shows the example of AR display on the screen in Embodiment 1. FIG. FIG. 5 is a diagram showing a first example of AR display control according to visibility in the first embodiment. It is a figure which shows the 2nd example of AR display control according to visibility in Embodiment 1. FIG. It is a figure which shows the example of the user setting of the AR function in Embodiment 1. FIG. It is a figure which shows the HUD apparatus which is the AR display apparatus of Embodiment 2 of this invention. It is a figure which shows the example of the screen in Embodiment 2. FIG. It is a figure which shows the HUD apparatus which is the AR display apparatus of Embodiment 3 of this invention. It is a figure which shows the example of the screen in Embodiment 3. FIG. It is a figure which shows the HUD apparatus which is the AR display apparatus of Embodiment 4 of this invention. It is a figure which shows the example of the screen in Embodiment 4. FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In principle, the same parts are designated by the same reference numerals in all the drawings for explaining the embodiments, and the repeated description thereof will be omitted.

(Embodiment 1)
The AR display device and the like according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 9. The AR display device of the first embodiment shows a case where it is mounted as a HUD device of an in-vehicle system. The AR display method of the first embodiment is a method having a step executed by the AR display device of the first embodiment. In the AR display device of the first embodiment, an object such as a signboard is extracted from an image taken by an external camera, and an AR image that is easy for the driver to see is generated and presented by image processing. Has. Further, the AR display device of the first embodiment has a function of improving the visibility of the driver by changing the presentation method of the AR image according to the position, size, distance, posture, image quality, etc. of the object in the captured image. Has.

[AR display device and in-vehicle system (1)]
FIG. 1 shows the configuration of an in-vehicle system 100 including the HUD device 1 which is the AR display device of the first embodiment. The in-vehicle system 100 is a system mounted on an automobile. The driver, who is a user, operates and uses the in-vehicle system 100 and the HUD device 1. This HUD device 1 is an AR-HUD device having an AR function in particular.

The in-vehicle system 100 includes an ECU (Engine Control Unit) 101, a HUD device 1, an external photographing unit 102 including a camera 2, a video data storage unit 103, a communication unit 104, a GPS receiver 105, a car navigation unit 106, and a camera. It has a driver photographing unit 107, a sensor unit 108, a DB unit 109, and the like including 3, and they are connected to an in-vehicle bus and a CAN (Car Area Network) 110. The in-vehicle system 100 has other operation units, power supply units, and the like (not shown).

The HUD device 1 includes a control unit 10, a display unit 20, other storage units (not shown), an operation unit, an audio output unit, a power supply unit, and the like. The control unit 10 is composed of a microcomputer or the like and controls the entire HUD device 1. The display unit 20 is composed of a projection type display device, and projects image light for forming a virtual image of AR on the front shield 9. As a result, the screen 5 is configured on the front shield 9, and the virtual image of AR is superimposed and displayed on the transparent actual scene on the screen 5. Using the operation unit (including the operation panel, for example), the user can input the operation, for example, turn on / off the AR function. Using the audio output unit, it is also possible to output audio accompanying car navigation and AR display.

The display unit 20 includes a display drive circuit 21, a display element 22, a light source 23, an optical system 24, and the like, and they are connected to each other. The display drive circuit 21 generates a display signal for AR display control based on video data which is AR data from the AR display unit 15 of the control unit 10, and drives the display element 22 and the light source 23. The light source 23 emits light to the display element 22 based on the display signal. The light source 23 is composed of, for example, an LED element, a laser element, or the like. The display element 22 can be composed of, for example, a known SLM (Spatial Light Modulator), a MEMS mirror, a DMD (Digital Micromirror Device, registered trademark), an LCD, or the like. The display element 22 generates image light based on the display signal and the light from the light source 23, and emits the image light to the optical system 24.

The optical system 24 guides the image light from the display element 22 to the screen 5 of the front shield 9. The optical system 24 includes elements such as lenses and mirrors, and a driving unit such as a motor for driving them. The optical system 24 also has a function of driving the mirror to change the tilt angle of the mirror based on, for example, control of the display unit 20 or user operation. As a result, the position of the screen 5 (basic display area of the virtual image) of the front shield 9 can be basically adjusted so as to move up, down, left and right when viewed from the driver.

The ECU 101 performs vehicle control including engine operation control. The ECU 101 may have advanced functions (for example, an inter-vehicle distance control function) for driving support and automatic driving control. In that case, the ECU 101 outputs information for driving support or the like to the HUD device 1 to control the HUD device 1, thereby displaying AR for driving support or the like on the HUD device 1 (for example, inter-vehicle distance information output). May be done.

The vehicle exterior photographing unit 102 includes a camera 2 and captures an actual view of the outside world of the own vehicle by using one or more cameras 2 while the vehicle is stopped or running, and video data (including a time-series image frame). ) And vehicle surrounding information are acquired and stored in the video data storage unit 103 or output to the ECU 101. The camera 2 is an outside camera 2, and is installed at a predetermined position, for example, near the bumper at the front of the vehicle, near the four sides of the front shield 9, or near the rearview mirror at the side of the vehicle. The camera 2 captures a predetermined range in a predetermined direction. The camera 2 captures a range including the actual view in front of the vehicle corresponding to the driver's field of view.

The vehicle exterior photographing unit 102 may or may not be provided with a signal processing unit or the like obtained by processing images of one or more cameras 2 and calculating vehicle surrounding information. The signal processing unit and the like may be provided in the ECU 101 and the HUD device 1. The vehicle surrounding information is information including a state such as proximity of an object in a range including the front of the vehicle and the vicinity of the side surface of the vehicle.

The video data storage unit 103 stores video data and the like from the camera 2. The video data storage unit 103 may be provided in the HUD device 1. From the viewpoint of information security, the video data is not stored and left in the video data storage unit 103 or the like in the vehicle, but is stored in an external data center or the like through the communication unit 104 and the data is managed externally. May be.

The communication unit 104 is a part including a communication interface device that communicates with an external mobile network, the Internet, or the like. The communication unit 104 communicates with a server such as a data center on the Internet based on the control from the HUD device 1 or the like. As a result, the HUD device 1 can refer to and acquire the original data and related information for AR display from the server.

The GPS receiver 105 obtains position information (for example, represented by latitude and longitude) of its own vehicle based on communication with an external GPS satellite. Although not shown, it also has a known VICS receiver and the like.

The car navigation unit 106 is a part of an existing car navigation system mounted on a car, and holds map information, position information acquired by using the GPS receiver 105, and the like. The car navigation unit 106 grasps the position information of the own vehicle through the GPS receiver 105. The ECU 101 and the HUD device 1 acquire the position information of the own vehicle from the GPS receiver 105 and the car navigation unit 106. The HUD device 1 can also be used as original data for AR display by referring to map information, destination information, and the like from the car navigation unit 106. For example, the HUD device 1 may generate an arrow image or the like indicating the traveling direction on the road for navigation to the destination as an AR image example based on the original data.

The driver photographing unit 107 photographs a range including the driver's eyes and face by using one or more cameras 3 including the camera 3, and performs a predetermined process. The camera 3 is a driver camera and an in-vehicle camera. The driver photographing unit 107 may store the image data photographed by the camera 3 in the image data storage unit 103. The driver photographing unit 107 includes a signal processing unit and the like that perform predetermined processing from the image of the camera 3. The signal processing unit and the like may be provided in the ECU 101 and the HUD device 1. The driver photographing unit 107 can be configured by using a known technique. In the first embodiment, it is not necessary to use the driver photographing unit 107, and it is used in the third embodiment and the like.

The driver photographing unit 107 has, for example, the following as a predetermined processing function. The driver photographing unit 107 can detect a dozing driving or the like by, for example, determining a state such as blinking of the driver's eyes. The driver photographing unit 107 can detect, for example, the position of the driver's eyes in the space (inside the vehicle), and can determine the posture of the driver from the position of the eyes. The driver photographing unit 107 has, for example, a function of detecting the driver's line of sight (line-of-sight tracking function) by determining the pupil and the light reflection point in the driver's eyes. The ECU 101 and the HUD device 1 can acquire driver information such as a line of sight from the driver photographing unit 107 and perform control.

The sensor unit 108 has a known sensor group mounted on the automobile, and outputs vehicle information such as the detected vehicle speed. The ECU 101 and the HUD device 1 acquire the vehicle information and control the vehicle information. The sensor unit 108 includes a vehicle speedometer, an acceleration sensor, a gyro sensor, a geomagnetic sensor, an infrared sensor, and the like as a sensor group. The sensor unit 108 may include a distance sensor that measures the distance between the vehicle and an object. Acceleration sensors and gyro sensors detect acceleration, angular velocity, etc. as state information of the own vehicle.

The DB unit 109 is composed of storage and the like, and data and information to be used for AR display are stored in the DB. The data and information of this DB include original data for generating an AR image, setting information for AR processing, user setting information, and the like. The original data is, for example, data used for driving support and navigation. The setting information is, for example, feature information or definition information of an object, a reference image for image matching, or the like. The user setting information includes, for example, on / off setting information of a function provided by the AR display device, setting information of a category of AR display target information, and the like.

The DB unit 109 and its DB are provided inside the in-vehicle system 100 outside the HUD device 1, but the present invention is not limited to this, and the DB unit 109 and its DB may be provided inside the HUD device 1 or on a communication network outside the in-vehicle system 100. It may be provided in a data center or the like. This DB may be merged with the DB of the car navigation unit 106. Information acquired from an external server, DB, or the like may be stored in this DB via the communication unit 104. When the DB is on the external communication network, communication is required, but it is not necessary to have the DB in the in-vehicle system 100, and the memory capacity can be saved. When the DB is in the in-vehicle system 100 or the HUD device 1, it is necessary to secure the memory capacity, but communication is not required and high-speed search can be performed.

The front shield 9 is made of glass or the like having transparency and rigidity. When the AR function is not used, the entire area of the screen 5 of the front shield 9 is transparent, and the actual view of the outside world is transmitted. On the screen 5, when the AR function is used, the AR image is superimposed and displayed on the transparent actual scene. The front shield 9 may be made of a film or the like having a reflection and transmission function. The front shield 9 may be composed of an organic EL device or the like having a property of being transparent in a non-energized state and emitting light in an energized state. The front shield 9 may be composed of a flexible organic EL display device or the like. A display board (combiner or the like) dedicated to AR may be separately provided in front of the front shield 9.

[AR display device and in-vehicle system (2)]
FIG. 2 schematically shows an example of arrangement configuration of each part when the vicinity of the front driver's seat of the automobile is viewed from the side. Indicates that the driver is sitting in the driver's seat. A curved front shield 9 is provided in front of the driver (Z direction). For example, a display unit 20 of an AR display device, a car navigation unit 106 (not shown), and the like are installed at the position of the console in the center of the dashboard at the front. FIG. 2 shows the display element 22 and the mirror 24a of the optical system 24. The light emitted forward from the display element 22 is reflected toward the front shield 9 by the mirror 24a which is a concave mirror, and the reflected light is directed toward the driver's eyes (point p1) on the inner surface of the front shield 9. Be reflected. As a result, the screen 5 is formed on a part of the inner surface of the front shield 9. The position of the driver's eyes or viewpoint is indicated by point p1. The driver's line of sight L1 is indicated by a alternate long and short dash line. The point of gaze, which is the intersection of the line of sight L1 with the screen 5, is indicated by point p4.

A camera 2 is installed at a predetermined position on the front portion of the automobile, for example, near the rear-view mirror. The position of the camera 2 is indicated by a point p2. The camera 2 is photographing the front direction of the automobile. A camera 3 is installed at a predetermined position in the front part of the vehicle, for example, diagonally above the driver's seat. The position of the camera 3 is indicated by point p3. The camera 3 captures the direction of the driver's seat and the driver's face.

The positions (three-dimensional space coordinates) and directions of the display unit 20, the camera 2 and the camera 3 of the AR display device are preset, and they have a predetermined relationship. Moreover, they can be set variably by the adjustment operation of the user. The driver adjusts the driver's seat and adjusts the position of the screen 5 on the display unit 20.

Beyond the driver's line of sight L1, there is an object in the actual scene (for example, a road surface mark, a signboard, etc.). At point p4 on the screen 5, the object or an object as an AR virtual image associated with the object can be seen. The in-vehicle system 100 and the HUD device 1 have a function of measuring and calculating the distance (object distance) between the object in the actual scene and the camera 2 (point p2) of the car. Further, the object distance can be converted as the distance between the viewpoint and the object based on the positional relationship between the point p2 of the camera 2 and the point p1 of the viewpoint. That is, the in-vehicle system 100 and the HUD device 1 can measure the object distance between the object in the actual scene and the viewpoint of the car and the driver. In addition, the position of the object in the space can be calculated from the object distance based on the position information of the own vehicle. Further, in the case of a certain form of the camera 3, the position of the point p1 of the fluctuating viewpoint can also be measured. In that case, the distance between the fluctuating viewpoint and the object can be grasped. The HUD device 1 uses the distance information between the viewpoints of the camera 2, the car and the driver, and the object to perform visibility determination and AR image processing described later.

The above-mentioned object distance and position measurement functions can be realized by various known means, and are, for example, as follows. A dedicated distance sensor provided in the sensor unit 108 may be used. For example, it is possible to measure the time until the light emitted from the sensor hits the object and returns, and calculate the object distance from that time. Further, when one camera 2 is provided in the vehicle exterior photographing unit 102, the object distance or the like can be obtained by calculation using the position or the like of the object in the image of the image of the camera 2 and the information of another sensor. Further, when the outside vehicle photographing unit 102 is provided with two cameras 2, that is, stereo cameras, the object distance and the like can be obtained by calculation using the two images on the left and right of the two cameras 2. Calculation of the object distance and the like may be performed in the ECU 101 or the HUD device 1.

As the camera 2, the vehicle exterior photographing unit 102 may use, for example, a monocular camera including a set of camera lenses and a light receiving element. In this case, it can be realized at low cost. The vehicle exterior photographing unit 102 or the HUD device 1 measures the object distance by a known calculation process using the image of the monocular camera, the information of other sensors, and the like. The vehicle exterior photographing unit 102 may use a stereo camera including two sets of camera lenses and a light receiving element as the camera 2. The vehicle exterior photographing unit 102 or the HUD device 1 measures the object distance by a known calculation process using the images of the two cameras.

As the lens of the camera 2, a telephoto lens or a wide-angle lens may be used. When a telephoto lens is used, it is possible to take an image of an object far from the driver, and it is possible to increase the number of objects that are candidates for AR. When a wide-angle lens is used, it can be realized at low cost because a wide range can be photographed with one camera 2. A polarizing filter may be provided in front of the lens or the light receiving element of the camera 2. As a result, it is possible to prevent a decrease in detection accuracy due to unnecessary reflected light from the object.

[AR display device and in-vehicle system (3)]
The control unit 10 includes an image selection unit 11, a visibility determination unit 12, an AR image generation unit 13, an image processing unit 14, and an AR display unit 15 as units realized by software program processing of a microcomputer or the like. The AR image generation unit 13 includes an image processing unit 14. Each part may be realized by a dedicated circuit (for example, FPGA).

The image selection unit 11 inputs the video data captured by the camera 2 from the video data storage unit 103, and from each image frame of the video data, a predetermined image captured in the image based on an image analysis process or the like. A region of an object (may be described as an object region, an object image, etc.) is selected and extracted as image data. In other words, the processing function of the image selection unit 11 is a processing function that divides the entire area of the image into pixel areas corresponding to units that can be recognized as objects. The image selection unit 11 stores the image data of the selected and extracted object area in the memory or the video data storage unit 103 in the control unit 10.

The image selection unit 11 performs image analysis processing such as extraction and determination of a feature amount from an image, image matching, and the like. The image selection unit 11 compares the extracted feature amount with the preset feature information and definition information of the object, determines the object region from the similarity, and extracts the object region. Alternatively, as an image matching process, the image selection unit 11 compares a region in the image with a preset reference image, determines an object region from the similarity, and extracts it.

In the first embodiment, at least a signboard is included as the target object. In the HUD device 1, feature information and definition information of a predetermined object are set in advance. For example, as the feature information and definition information of the signboard, the shape and size range of a quadrangle or the like, the color of the signboard surface, the characters and images described on the signboard surface, and the like are defined. The feature information and definition information of the object may be a geometric figure such as a rectangle, a polygon, or an ellipse, may be a fine region in pixel units, or may be a reference image.

For example, when the target object is a signboard, the image selection unit 11 may extract the quadrangular area separated by the boundary line extracted from the image as the object area of the signboard. Alternatively, the image selection unit 11 may extract a simplified right-angled quadrangle that includes a quadrangle area corresponding to the signboard as an object area of the signboard. When the object area is, for example, a simple rectangle, high-speed processing is possible. When the object area is, for example, a fine pixel area in pixel units, it can be extracted finely, noise (the influence of other objects around the object, etc.) can be reduced, and more detailed AR can be presented. These viewpoints may be appropriately designed in a balanced manner.

It should be noted that the type of the target object and the like can be set by the user by using the user setting function described later. For example, it is possible to set the target object to be only a signboard.

The visibility determination unit 12 determines the degree of visibility (high or low) of the driver in displaying the AR image related to the target object based on the video data and the image data of the object area. This visibility includes the visibility of the object itself in the actual scene and the visibility of the AR image associated with the object. The visibility determination unit 12 calculates the visibility index value by a predetermined method for the determination. Based on the visibility index value, the visibility determination unit 12 determines, among the objects extracted from the image, an object that is considered to be difficult for the driver to see as an AR display target or an AR image processing target. The visibility determination unit 12 determines how to display the AR of the object according to the visibility index value.

The AR image generation unit 13 uses the image processing process of the image processing unit 14 to generate an AR image having higher visibility than the original object so that the target object can be easily seen by the driver. The AR image generation unit 13 generates an AR image based on the result of determining the visibility of the target object, the input video data or the image data of the object area, the original data for the AR image, and the like. At that time, the AR image generation unit 13 processes the image area of the target object by using the image processing unit 14 as necessary, and generates an AR image using the processed image. The AR image generation unit 13 uses the DB data of the DB unit 109 as reference data as the original data, if necessary. Alternatively, the AR image generation unit 13 uses the vehicle information from the ECU 101 and the sensor unit 108, the map information from the car navigation unit 106, the external information via the communication unit 104, and the like.

The image processing unit 14 performs image processing such as processing and conversion for improving the visibility of the target object and the AR image based on the visibility determination result, the input video data or the image data of the object area, the original data, and the like. Do. Examples of the processing process include enlargement processing, sharpening processing, contrast changing processing, color tone changing processing, posture changing processing, OCR processing, and the like. The image processing unit 14 has a function of performing each of these processes. In the processing, the object image may be processed into an AR image, or the original AR image may be further processed into an AR image. In terms of mounting, only some of them may have a processing function.

The AR display unit 15 generates AR data for AR display based on the AR image data generated by the AR image generation unit 13 and outputs the AR data to the display unit 20, thereby displaying the AR display on the display unit 20. Let me do it. The AR data includes information such as the display position and display size of the AR image on the screen 5. As a result, the AR image associated with the object is superimposed and displayed as a virtual image at the display position on the screen 5 in a state such as a display size of a predetermined enlargement ratio. It is visible to the driver's eyes as an AR image associated with the object. The AR display unit 15 also performs conversion processing for associating the coordinate system of a plane, which is a conceptual screen for information processing, with the coordinate system of a curved surface, which is the screen 5 of the front shield 9.

The HUD device 1 also has an optical character recognition (OCR) function. The image selection unit 11 recognizes characters (alphanumeric characters, kana-kanji, symbols, etc.) from the input image or the object image thereof by OCR processing, obtains character data, and stores the character data in the memory of the control unit 10 or the video data storage unit 103. Store. The image selection unit 11 may extract predetermined images, marks, etc., such as traffic signs and trademarks, from the input image.

Each unit such as the image selection unit 11 is realized by the processing of the control unit 10 of the HUD device 1, but the present invention is not limited to this. Each of these parts may be partially or wholly realized by hardware or software outside the HUD device 1.

[Main processing flow]
FIG. 3 shows a flow of main processing in the HUD device 1 which is the AR display device of the first embodiment. FIG. 3 has steps S1 to S7. Hereinafter, the steps will be described in order.

(S1) Normally, the control unit 10 of the HUD device 1 turns on the camera 2 of the vehicle exterior shooting unit 102 when the AR function is on, and outputs the video data of the outside world captured by the camera 2 of the vehicle exterior shooting unit 102. , Stored in the video data storage unit 103. The control unit 10 sequentially inputs images of the video data from the video data storage unit 103.

As a modification, the ECU 101 or the like of the in-vehicle system 100 may control the operation of the camera 2. Further, the HUD device 1 may input video data directly from the camera 2.

(S2) The image selection unit 11 selects and extracts a region of a predetermined target object (for example, a signboard) from the image of the input video data of S1. The image selection unit 11 stores the image data of the selected and extracted object area in the memory or the video data storage unit 103 in the control unit 10.

Further, in S2, when the image selection unit 11 extracts an object from the image, the feature amount of the object and the like are extracted based on the image analysis process. Examples of feature quantities include shape, size, color, brightness, contrast, blur (frequency component), and the like.

Further, in S2, the image selection unit 11 may perform OCR processing on the image or the extracted object area, and if the character can be recognized, extract the character information.

In the first embodiment, the image selection unit 11 extracts an object area for the entire AR displayable area of the screen 5 and the entire image of the camera 2. This is not limited to this, and a part of the screen 5 and the entire image may be set as the extraction target area. For example, FIG. 5 shows a setting example of a partial area R6.

Further, in the first embodiment, the image selection unit 11 simultaneously extracts up to a predetermined number of object regions from the image of the camera 2. In that case, for example, an object at a position close to the own vehicle may be preferentially extracted, or an object near the center point may be preferentially extracted on the screen 5. May be good.

(S3) The visibility determination unit 12 calculates an index value for determining the degree of visibility as seen by the driver for the object extracted in S2 by a predetermined method. For the sake of explanation, this index value is defined as a "visibility index value". This index value is calculated so that, for example, the easier it is to see, the higher the value, and the harder it is to see, the lower the value. The visibility determination unit 12 calculates this index value by using, for example, the feature information extracted from the image and the information of the sensor of the sensor unit 108 and the like. In the calculation method of the first embodiment, this index value is calculated using at least one of the object distance, the position and size of the object on the screen 5 and the image, and the like. The object distance is the distance between the camera 2 or the viewpoint of the own vehicle and the target object. For example, the larger the object distance, the lower the index value. Further, for example, the smaller the object size in the image, the lower the index value. For example, the index value becomes lower as the position of the object in the image becomes the position of the periphery relatively far from the center.

(S4) The visibility determination unit 12 determines whether the target object is easy to see or hard to see from the driver based on the visibility index value of S3, and classifies the target object. For example, the visibility determination unit 12 compares the visibility index value with the threshold value and classifies them in some cases. It has at least two classifications. The first embodiment has a first classification and a second classification (FIG. 7 described later). The first classification is a case where it is judged that it is easy to visually recognize. The second classification is a case where it is judged that it is difficult to visually recognize the classification. The HUD device 1 classifies using a range of threshold values related to an object distance, an object size, and the like.

For example, when the object distance is used, it can be estimated that the driver's visibility is low when the distance between the own vehicle and the object is larger than the threshold value. Further, for example, when the object size is used, if the size of the object area in the image is smaller than the threshold value, it can be estimated that the object is located far from the own vehicle, which makes the driver's visibility. It can be estimated to be low.

For example, the HUD device 1 sets in advance an image size (length in pixel units) that makes it difficult for the driver to see, as a threshold value, according to the driver's visual acuity, the angle of view characteristics of the lens of the camera 2, and the like. I will do it. This threshold value may be set by a magnification with respect to a standard image size for each object. The HUD device 1 compares the size of the object region extracted in S2 with the threshold value of the image size, determines whether or not it is visible, and classifies it.

The above-mentioned classification of visibility is not limited to two, and may be classified into three or more according to the stage of ease of visibility and difficulty in visibility (FIG. 8 described later). For example, it has a second classification and a third classification according to the first stage and the second stage of difficulty in visual recognition.

(S5) The visibility determination unit 12 determines whether or not AR display is executed, an AR image, whether or not processing is performed, and the content (described as a type, etc.) of the target object based on the classification of the visibility determination result in S4. To do. For example, when the classification is the first classification, AR display is not performed as the first type, and in the case of the second classification, AR display is performed as the second type with image processing.

At the time of S5 and S6, the basic rule regarding AR display (described later) is also taken into consideration. For example, as a basic rule, AR image display is avoided in the central area of the screen 5. Further, as a basic rule, the number of AR images displayed simultaneously on the screen 5 is limited to a predetermined maximum number.

(S6) The AR image generation unit 13 generates an AR image of an object to be AR-displayed based on the determination of the type of S5. At that time, the AR image generation unit 13 uses the image processing unit 14 to perform image processing processing for improving the visibility of the object and the AR image, which are classified as the second category and have low visibility. The image processing unit 14 applies enlargement processing or the like according to the type as image processing processing for the object region, and outputs the processed image data. The AR image generation unit 13 constructs an AR image using the processed image data.

The image processing unit 14 applies at least one of enlargement processing, sharpening processing, contrast changing processing, color tone changing processing, posture changing processing, and OCR processing as processing processing. For example, in the case of enlargement processing, the image processing unit 14 enlarges the image of the object area to obtain an AR image. For example, in the case of OCR processing (in other words, conversion processing to a character image), the image processing unit 14 extracts character information from an object area by OCR processing, and sets the font, size, color, etc. of the character to the desired visibility. Select to be a level to generate a character image and use it as an AR image. In this OCR process, the character information extracted by the image selection unit 11 in S2 may be used.

The AR image generation unit 13 may process the original data by the image processing unit 14 to generate an AR image. For example, it is assumed that the character information and the image described on the signboard surface are obtained as the original data. The image processing unit 14 creates an AR image with high visibility by enlarging the size of the character or the image.

Further, at the time of S6, the AR image generation unit 13 determines the display position, the display size, and the like in the screen 5 based on a predetermined method (described later) for the generated AR image. For example, in the case of the fixed position method, the position of a vacant portion in a predetermined fixed area (for example, the lower side area R3 of FIG. 6) in the screen 5 is selected as the display position. In the case of the neighborhood position method, the position of a vacant portion in the screen 5 within the vicinity of the target object and outside the central region is selected as the display position. Regarding the display size of the AR image, basically, the display size changes according to the change in the object distance.

Further, in S6, the AR image generation unit 13 may adjust the display position, display size, number of displays, and the like of the AR image in order to improve the visibility of one or more AR images in the screen 5. The visibility of the driver is related not only to the visibility of individual objects and AR images, but also to the overall visibility of a plurality of objects and a plurality of AR images when the screen 5 is viewed. Therefore, the HUD device 1 determines the AR image in consideration of the overall visibility on the screen 5. For example, when there are a plurality of AR images for a plurality of objects in the second category, their display positions, display sizes, display numbers, etc. are determined so that their overall visibility is high. For example, when the display positions of a plurality of AR images are close to each other, the display positions are adjusted so as to be separated from each other to some extent so as not to overlap as much as possible. For example, the number of displayed images is limited to a predetermined maximum number of AR images at the same time on the screen 5. In the case of limitation, it is determined in consideration of the object distance or the definition of the priority of the object and the AR image (traffic sign has higher priority than the signboard, etc.). As a result, the amount of information of the AR image on the screen 5 is suppressed, and the driver can easily recognize the object or the AR image.

(S7) The AR display unit 15 uses the AR image data generated in S6 and the control information thereof to create AR data for superimposing and displaying the AR image in association with the target object on the screen 5. , Output to the display unit 20. The AR display unit 15 converts the AR image data on the rectangular flat screen of S6 so as to correspond to the curved screen 5 of the front shield 9. The display unit 20 superimposes and displays the AR image on the screen 5 according to the AR data. The above processing is performed as a similar repeating loop.

[Screen, image]
FIG. 4 schematically shows a state in which the driver in the driver's seat in the vehicle looks at the screen 5 of the front shield 9 in front. Although the actual front shield 9 and screen 5 are curved surfaces that look trapezoidal, they are simplified and shown as rectangular planes in FIG. The screen 5 includes an AR displayable area. In this example, the entire screen 5 is set as an AR displayable area. On the screen 5, as an actual view corresponding to the driver's field of view, the road 31 extending forward, the buildings 32a, 32b, 32c on the side of the road, the sky 33 in the background, and the like are transparently seen. Further, in the actual view of the screen 5, the object includes a sign 34, a signboard 41 to 44, and the like.

For example, on the front of a right-angled quadrangular flat plate, characters and images such as store information and advertisements are drawn on the signboard. The shape of the signboard when viewed by the driver differs depending on the relationship between the position of the driver's viewpoint and the orientation and position of the signboard. For example, the signboard 44 is seen as a rectangle because the front surface is arranged substantially perpendicular to the line of sight and the traveling direction of the vehicle. For example, the front surface of the signboard 41 is similarly seen as a rectangle, but since the object distance is relatively long, the signboard 41 is seen in a small size, and the written characters and the like are difficult to see. For example, the signboard 42 is arranged so that the front surface faces the road 31 and the direction is close to 90 degrees with respect to the line of sight and the vehicle traveling direction. The front of the signboard 42 is in an oblique posture with respect to the line of sight. Therefore, the signboard 42 appears as a trapezoidal distorted region. If it has such distortion, it may be difficult for the driver to see it. Depending on the relationship between the position and direction of the vehicle and the object, the object may not be visible at all. The signboard 43 is arranged at a position near the left side with respect to the own vehicle, and is similarly in an oblique posture.

The signboard that is the target object includes a signboard installed independently, a signboard installed in a building, and the like. The types of information described on the surface of the signboard that is the target object include characters, images (including logos, etc.), and the like. Examples of the described information include general advertisements, advertisements for buildings such as stores, guides and maps, and the like. The signboard may be a digital advertising signboard whose content changes with time. The signboard is not limited to a signboard having a fixed installation position, but may be a moving signboard, for example, a signboard mounted on a moving body.

[Area, basic rule, display position method]
FIG. 5 shows an area of the screen 5 and the like. As explanatory areas, there are a central area R1 near the center of the screen 5, an upper side area R2 near the upper side, a lower side area R3 near the lower side, a right side area R4 near the right side, and a left side area R5 near the left side. ..

The central area R1 is an area for avoiding the display of an AR image as a basic rule regarding AR display in the first embodiment. The central region R1 is an ellipse, but the present invention is not limited to this and can be set to a rectangle or the like. The central region R1 of the screen 5 is an region that the driver often gazes at while driving. This central area R1 is basically defined as an area where AR display is not performed, or an important predetermined AR display area for driving assistance and the like. Examples of important AR displays include road surface marks and traffic light displays, traveling direction arrow displays, speed limit displays, inter-vehicle distance displays, caution displays for pedestrians and oncoming vehicles, warning displays, and the like. This region is not limited to the central region R1, and may be a line-of-sight center region when there is a line-of-sight tracking function described later. The basic rule setting is an example. As another setting example, the rate of displaying the AR image on the screen 5 may be set to a certain rate or less.

The HUD device 1 of the first embodiment performs AR display on an object such as a signboard, but avoids the central region R1 as much as possible when determining the display position of the AR image. That is, the display position of the AR image of the signboard is basically determined to be a position outside the central region R1. As a result, in the in-vehicle system 100, AR image information such as a signboard can be suitably presented in a state of good visibility while prioritizing driving support and the like for traffic safety. Depending on the situation (for example, when driving at low speed or when the vehicle is stopped), AR display such as a signboard may be displayed on R1 in the central region.

There are several possible methods for determining the display position of the AR image on the screen 5, and the following can be mentioned. In S6 described above, the AR image generation unit 13 determines the display position, display size, and the like of the AR image in the screen 5 to be displayed in association with the object by using any of the following methods.

As the display position method, there are (1) fixed position method, (2) neighborhood position method, and (3) same position method.

In the fixed position method of (1), a fixed area for displaying an AR image is provided in advance outside the central area R1 in the AR displayable area of the screen 5. The upper side region R2 and the lower side region R3 of FIG. 5 are setting examples of the fixed region corresponding to the fixed position method. The display position of the AR image is determined within the fixed area. More specifically, it may be set so that different types of AR images are displayed for each fixed area. For example, vehicle information is displayed in the lower side area R3, signboard information is displayed in the upper side area R4, and the like.

In the neighborhood position method of (2), the display position of the AR image is determined to be a position selected from the range in the vicinity of the target object. For example, a circular region having a predetermined radius distance is tentatively set with the position of the target object as the center point. Any vacant position within the circular region and outside the central region R1 is selected as the display position of the AR image. Further, in particular, when there is a vacant position in the candidate area such that there is no other object or AR image, that position is selected. For example, when the target object is a signboard, a vacant position with a background of sky 33 or the like is selected as a display position on the upper side or the side of the signboard.

Further, in this method, a line (leader line) connecting the object and the AR image may be displayed as a part of the AR image. If the display position of the AR image cannot be secured within a range of a relatively short distance from the position of the object in the image, the display position is set at a position separated from the object by a certain distance or more, and the object and AR are set. A line connecting the image may be displayed. By displaying the leader line, the driver can more easily recognize the relationship between the object and the AR image.

In the same position method of (3), the display position of the AR image is set to the same position as the position of the object in the image. For example, the AR image may be superimposed and displayed so as to cover the object, or the AR image such as a frame surrounding the outer shape of the object may be displayed. For example, in FIG. 6, the object is a sign 35, and in order to emphasize the existence of the sign 35, a frame surrounding the sign 35 is displayed as an AR image at the same position.

[AR display example]
FIG. 6 shows an example of display control of an AR image on the screen 5. In this example, a display example of an AR image relating to a signboard 41 or the like is shown. First, the signboard 44 is determined to be sufficiently large and easy to see, so that the AR image is not displayed.

The AR image 51a is a first display example relating to the signboard 41, and is an example in which the image of the signboard 41 is enlarged and displayed in a state where it is easy to see in the lower side region R3 by a fixed position method. Similarly, the AR image 51a may be displayed in the upper side region R3. Further, as shown in the figure, the AR image 51a may be displayed with a leader line 51a1 (for example, a broken line arrow) extending from the signboard 41. The leader line 51a1 is displayed so as to avoid the central region R1 and bypass the outside. Further, the display size of the AR image 51a may be at least larger than the object area of the original signboard 41, and may be a predetermined size according to the size (for example, the vertical width) of the lower side area R3.

The AR image 51b is a second display example relating to the signboard 41, and similarly enlarges the image of the signboard 41 to a vacant position outside the central region R1 within a predetermined radius distance in the vicinity of the signboard 41. This is an example of displaying in a state where it is easy to see. Further, as shown in the figure, the AR image 51b may be displayed with a leader line 51b1 (for example, a solid line) extending from the signboard 41. The leader line 51b1 is displayed so as to extend in the direction from the center of the screen 5 to the periphery. Further, the display size of the AR image 51b is at least larger than the object area of the original signboard 41. The AR image and the leader line may be an image in a balloon format or the like. By attaching a leader line or the like, it becomes easier for the driver to recognize the relationship between the object and the AR image.

The signboard 42 has an oblique posture when viewed from the driver's point of view, and looks distorted in a trapezoidal shape. Therefore, it is difficult to visually recognize the description on the front of the signboard 42. Similarly, the AR image 52 relating to the signboard 42 is an example of displaying a leader line at a position near the signboard 42. The AR image 52 is subjected to enlargement processing and posture change processing as image processing processing. The posture of the object area of the signboard 42 has been changed so that it is in the front state when viewed from the viewpoint. That is, the original trapezoid is converted into a right-angled quadrangle. This change in posture makes it easier to see the characters on the front of the signboard 42.

Similarly, the signboard 43 is located at the left end of the screen 5 and the field of view and is in an oblique posture, so that it is difficult to see the signboard 43. The AR image 53 relating to the signboard 43 is an example of displaying in the lower side region R3.

In the first embodiment, as one function, there is a function of displaying an AR image according to the posture state of an object in the image, as in the example of the signboard 42 and the like. The HUD device 1 determines the visibility according to the posture state of the object as seen from the driver's point of view, and selects and determines the AR display content. For example, in the case of a signboard 44 in which the front surface is in a direction generally along the direction of the driver's line of sight and the traveling direction of the own vehicle, in other words, in the case where the object area in the image is close to a right-angled quadrangle. It can be judged that the visibility is relatively easy. In that case, the visibility index value is calculated as a relatively high value. On the other hand, when the front is in an oblique posture state corresponding to different directions intersecting the direction of the driver's line of sight and the traveling direction of the own vehicle as in the signboard 42, in other words, the object area in the image is trapezoidal or the like. In the case of, it can be judged that it is relatively difficult to visually recognize. Alternatively, when the width of the object area (length in the horizontal direction of the screen) looks narrow as in the signboard 43, it can be determined that it is relatively difficult to see. In that case, the visibility index value is calculated as a relatively low value. Alternatively, if the front of the signboard is completely invisible, it can be determined that the signboard is invisible. As a method of calculating the visibility index value, the attitude state of the object area, for example, the angle of the front direction of the signboard, the angle of the oblique side of the trapezoid in the image, etc. are calculated, and the angle etc. is calculated with respect to the object distance etc. It may be calculated by a method of multiplying as a coefficient. Alternatively, such a calculation may be omitted, and when the object region is extracted from the image, the classification may be performed directly according to the determination of the shape such as a trapezoid.

[Image processing (1)]
The following is an example of image processing processing by the image processing unit 14. The image processing unit 14 has (1) enlargement, (2) sharpening, (3) contrast change, (4) color tone change, and (5) object as image processing functions for enhancing the visibility of objects and AR images. It has processing functions such as posture change and (6) OCR.

(1) The enlargement processing function enlarges the object area that appears in a small size in the image to a large size when an object that is relatively far from the camera 2 (corresponding car and driver) is photographed. It is a processing function to do. For example, in FIG. 6, when a signboard 41 located at a relatively distant position is extracted as an object area, the image processing unit 14 enlarges the image of the object area of the signboard 41 at a predetermined enlargement ratio or the like. For example, the written character "ABC" on the signboard 41 is enlarged from a small state to a large state.

(2) The sharpening processing function (in other words, sharpness change, edge enhancement) is a function of sharpening a blurred image by using a known technique. The sharpening method may be, for example, a method using a filter that emphasizes gradation changes, or a method in which an image is Fourier transformed and decomposed for each frequency component to remove low frequency components. The image processing unit 14 performs sharpening processing on, for example, the image of the signboard 41 (or the enlarged image thereof). As a result, the blurring of the written characters on the signboard 41 is reduced.

(3) The contrast change processing function is a function that uses a known technique to enhance the contrast in an image having no gradation due to strong light or the like. The image processing unit 14 performs contrast changing processing on, for example, the image of the signboard 41 (or the enlarged image thereof). As a result, the contrast between the written characters on the signboard 41 and the background is increased, and the written characters are easily visible.

(4) The color tone change processing function is a function that uses a known technique to change the color tone of an image of an object that is difficult to see because it has a color similar to that of the actual scene. The image processing unit 14 performs color tone changing processing on, for example, the image of the signboard 41 (or the enlarged image thereof). For example, the image processing unit 14 compares the color of the object area of the signboard 41 with the color of the background area (building, sky, etc.) around the signboard 41 based on the captured image, and determines whether the colors are similar (same color tone). Judge whether or not. When the colors are similar, the image processing unit 14 changes the color tone of the object area so as to have a sufficient difference from the color tone of the background area. This makes it easier to see the area of the signboard 41.

(5) The object posture change processing function uses known technology to convert the captured image so that the object region is in the front state when the target object is in an inclined posture state when viewed from the driver's viewpoint. It is a function to perform processing. The image processing unit 14 determines the posture state from the shape of the object region in the image, and if the posture is oblique (for example, the shape is trapezoidal), performs conversion. The image processing unit 14 uses a known projective transformation or the like to perform transformation so that the posture of the object region is in front, for example, the trapezoidal shape is a right-angled quadrangle. For example, in the signboard 42 of FIG. 6, the trapezoidal image is converted into a right-angled quadrangular image.

(6) The OCR processing function is a function of extracting characters from an object area in an image using a known technique and generating a character image using the character information. The extracted character information is used as the original data for generating the AR image. The image processing unit 14 may perform OCR processing on the object region in the image to extract characters. When the character "ABC" can be recognized by the OCR process from the object area of the signboard 41, for example, the image processing unit 14 may enlarge the character by the enlargement process to make it a sufficient size. Alternatively, the image processing unit 14 may perform OCR processing on the image of the object region after being enlarged by the enlargement processing to extract characters. For example, when a character can be recognized by the OCR process from the image corresponding to the enlarged AR image 51b, the image processing unit 14 may replace the image with the image by the character.

[Image processing (2)]
The following are examples of other processing related to image processing and the like.

(1) The image selection unit 11 of the HUD device 1 analyzes the frequency component of the object region in the image. In the analysis result, when there is only a low frequency component, it can be judged that the object image is blurred (in other words, the degree of blurring is large) and it is difficult to see. The visibility determination unit 12 can calculate the visibility index value based on the frequency component thereof. In that case, the HUD device 1 uses the image processing unit 14 to perform processing such as sharpening and contrast change on the object region as image processing processing. The details of this process may be as follows. The HUD device 1 acquires the object area of the image taken at a certain time during driving and the object area of the image taken at a position where the object approaches the own vehicle at a later time, and both of them. Align the size of the object images and compare the frequency components. From the comparison result, when the change of the frequency component is relatively small, it can be determined that there is only the low frequency component (blurred) in the object region.

(2) The HUD device 1 determines the brightness (corresponding pixel value, etc.) of the object region of the image and the region around it. The visibility determination unit 12 can determine that it is difficult to visually recognize when the brightness value of the object region is relatively low (dark) or relatively high (bright). In this case, as the image processing process, an offset value is added or subtracted from the brightness of the object area. As a result, the brightness is adjusted so as to be different between the object area and the surrounding area, and the visibility is improved.

(3) The HUD device 1 generates a histogram regarding the brightness of the object region of the image, and when the object region is replaced with the data signal handled in the AR image generation processing and the image processing processing, the contrast is increased again. Constitute. For example, the image selection unit 11 converts the range from the minimum value to the maximum value of the brightness of the object region into the range from the minimum value to the maximum value that can be expressed by the data signal at the time of image analysis. Interpolate the intermediate values. As a result, when the contrast is increased, it becomes easier to detect the feature information of the object, and it becomes easier to judge the visibility.

[AR display control according to visibility (1)]
FIG. 7 shows a table in which the classification of the visibility of S4 in FIG. 3 and the determination of the AR display content of S5 are arranged as the first example of the AR display control in the first embodiment. The first example is a control example in which the AR display contents are classified into two types according to the object distance and the like. In the table of FIG. 7, as rows, classification, an image near a signboard, driver visibility, object distance, object size, VI: visibility index value, AR display type, and AR image are shown in order from the top. The classifications are (A) first classification and (B) second classification from the left.

Regarding the first classification of (A), in the image, the rectangle in front of the signboard 41 is shown relatively large. Therefore, the driver can easily visually recognize the characters on the signboard 41. The object distance between the driver and the signboard 41, which is an object, is relatively close (small), and the object size is relatively large. The visibility index value VI is within the first range and is calculated as a relatively high value based on the object distance and the object size. The first range is VI ≧ H1 using the threshold H1. Since the visibility index value VI is within the first range, it is set to the AR display type (first type) of the first classification. In this AR display type, the object is left as it is and the AR image is not displayed. Alternatively, the AR image may be displayed in association with the object, in which case the image processing process is not performed. By not displaying the AR image, the amount of information presented to the driver on the screen 5 is reduced, and the driver can easily recognize other AR images and objects.

Regarding the second classification of (B), in the image, the rectangle in front of the signboard 41 is shown relatively small. Therefore, it is relatively difficult for the driver to visually recognize the characters on the signboard 41. The object distance and object size from the sign 41 are relatively medium. The visibility index value VI is calculated as a relatively medium value within the second range, based on distance and size. The second range is H1> VI ≧ H2 using the thresholds H1 and H2. Since the visibility index value VI is within the second range, it is set to the AR display type (second type) of the second classification. In this AR display type, an AR image of an object is displayed. At that time, the object region is subjected to the first processing process to generate an AR image. The first processing process is, for example, an enlargement process, and the AR image is an enlarged image of the area of the signboard. As another processing process, the above-mentioned processes (2) to (5) may be applied in combination.

[AR display control according to visibility (2)]
FIG. 8 similarly shows a second example of AR display control as a modified example of the first embodiment. The second example is a control example in which the AR display contents are classified into three types according to the object distance and the like. In the table of FIG. 8, the classifications are (A) first classification, (B) second classification, and (C) third classification from the left. The second classification in FIG. 7 is further divided into a second classification and a third classification in FIG. 8 according to the stage of difficulty in visual recognition.

The first classification of (A) is the same as the first classification of FIG. 7. The threshold value H1 may be a different value. The driver can clearly see the characters on the signboard 41, for example, and is easy to see.

Regarding the second classification of (B), in the image, the rectangle in front of the signboard 41 is shown in a relatively medium size. Therefore, the driver can visually recognize the characters on the signboard 41 by paying close attention to them, but it is relatively difficult to see and corresponds to the first stage of difficulty in viewing. Object distance and object size are relatively medium. The visibility index value VI is calculated as a relatively medium value within the second range based on the object distance and the object size. The second range is H1> VI ≧ H2 using the thresholds H1 and H2. Since the visibility index value VI is within the second range, it is set to the second classification AR display type (second type). In this AR display type, an AR image of an object is displayed. At that time, the object region is subjected to the first processing process to generate an AR image. The first processing process is, for example, an enlargement process, and the AR image is an enlarged image of the area of the signboard.

Regarding the third classification of (C), in the image, the rectangle in front of the signboard 41 is shown relatively small. Therefore, the driver cannot see or is difficult to see the characters on the signboard 41, which corresponds to the second stage of difficulty in seeing. The object distance is relatively long (large), and the object size is relatively small. The visibility index value VI is calculated as a relatively low value within the third range based on the object distance and the object size. The third range is H2> VI using the threshold H2. Since the visibility index value VI is within the third range, it is set to the AR display type (third type) of the third classification. In this AR display type, an AR image of an object is displayed. At that time, the object region is subjected to the second processing process to generate an AR image. As the second processing process, for example, an OCR process is used in addition to the enlargement process. The AR image is created using the character information recognized by OCR. As another processing process, the above-mentioned processes (2) to (5) may be applied in combination.

If the characters cannot be recognized by OCR, the AR image may not be displayed. Alternatively, when the character cannot be recognized by the OCR, a process of searching the DB information of the DB unit 109 may be performed as described later. In that case, for example, the object information of the target object (for example, the signboard 41) is searched from the DB using the position information of the object (the position relative to the own vehicle), and the AR image is created using the object information of the search result. Will be done.

When the visibility determination unit 12 calculates and determines the VI value in the above, it uses a predetermined maximum value or minimum value, and if the distance or size of the object is too large or too small, it is classified. It is out of range and out of processing. Further, the threshold value H1 and the like related to VI may be set as a fixed setting value on the mounting, or may be variably set by the user by the user setting function. For example, the threshold value H1 or the like may be determined and set according to the eyesight of the driver. For example, in the case of a person with good eyesight, the threshold value H1 is set to a relatively low value, and in the case of a person with relatively poor eyesight, the threshold value H1 is set to a relatively high value. As a result, even for an object at the same distance from the driver, in the case of the former person, the VI value becomes the threshold value H1 or more, which is classified as the first category, and in the case of the latter person, the VI value is the threshold value H1. If it is less than, it is classified as the second category.

[User setting function]
The AR display device of the first embodiment has a user setting function related to the AR function. With this user setting function, it is possible to set the type related to the AR function and the AR information category. The user can select and set the information that he / she wants to preferentially acquire and display by using the user setting function. The HUD device 1 displays the information of the selected category as an AR image by the AR function of the selected type based on the user setting. The limited AR information desired by the driver is presented in the screen 5. Therefore, the driver can easily visually recognize the desired AR information on the screen 5. Note that these are not limited to user settings, and may be in the form of fixed settings for mounting.

FIG. 9 shows an example of a user setting screen. For example, the HUD device 1 displays a user setting screen on a hardware operation panel so that the HUD device 1 can be set according to a user input operation. The screen of FIG. 9 has an [AR function type setting] item and an [AR information category setting] item. The [AR function type setting] item has items such as [driving support], [navigation], [traffic sign], [signboard], and [building], and can be turned on / off by the user. is there. For example, the [Signboard] item is set to the on state. As a result, the function of performing AR display with the signboard as the target object is enabled.

In the [AR information category setting] item, it is possible to set the category of AR information to be acquired and displayed by the AR function. This item has, for example, [People (Category A)], [Place (Category B)], [Situation (Category C)], [Time (Category D)], etc., and can be turned on / off by the user. It is possible. For example, the [Location (Category B)] item is set to the on state. As a result, information regarding the location of the AR function of the type (for example, [signboard]) selected above is provided as an AR image.

Category A is [person], and examples of the information content include a name, an address, a telephone number, and the like. From the viewpoint of privacy and the like, the target person can be limited to a person such as an acquaintance of the set driver or an information disclosure person. Category B is [location], and the information content includes convenience stores, gas stations, parking lots, hospitals, toilets, restaurants, homes, stores, tourist spots, stations, and the like. Category C is [Situation], and examples of the information content include weather, road surface conditions, traffic jams, vacant parking lots, accidents, construction, and store queues. Category D is [time], and the information content includes waiting time of a store or the like, opening time, closing time, required time, passing time, train arrival time, and the like. Information items in each category can be turned on / off individually.

In the first embodiment, the target object is mainly a signboard. By setting the location of category B, it is possible to set which signboard is particularly targeted in the signboard group. For example, when only the [Restaurant] item is set to the ON state, the AR image is preferentially displayed for the sign corresponding to the sign of the restaurant among the signs in the screen 5 and the image.

Further, when there is a passenger in the automobile, the passenger can operate the HUD device 1 or the like to select the above information category or the like. The HUD device 1 displays the AR image so as to switch according to the selection operation.

[Effects, etc.]
As described above, according to the AR display device of the first embodiment, it is possible to improve the visibility of the image including the object and the AR image by the driver and realize a suitable AR presentation. When presenting an AR image related to an object such as a signboard, even an object that is difficult to see can be presented as an AR image that is easy to see. The driver can easily recognize objects such as signboards and AR images. As a result, the driver can obtain information that he / she wants to know about an object such as a signboard. The driver can easily read the characters on the signboard. In the first embodiment, the presentation content of the AR image information is switched according to the degree of visibility of the object from the driver, that is, the visibility index value. This makes it possible to present AR suitable for the driver. The AR image is presented with different contents depending on the degree of visibility, and the visibility of the entire screen 5 can be made suitable.

[Modification example]
The configuration is not limited to the configuration of the HUD device 1 which is the AR display device of the first embodiment and the in-vehicle system 100, and the following are examples of modifications. A combination of each modification is also possible.

(A) In the first embodiment, the AR display is realized by using the HUD device 1 as the AR display device, but the AR display is not limited to this, and the AR display is performed by using the head mounted display (HMD) device. It may be realized. That is, the AR display device may be realized as a transmissive HMD device or a system thereof that the user wears on the head and uses. In the case of this form, the AR image is superimposed and displayed on the transparent actual scene on the screen of the HMD device in front of the user. Further, for example, a control that detects the vibration of the vehicle from the information of the acceleration sensor provided in the HMD device and vibrates the AR image so as to cancel the vibration of the vehicle as much as possible within a range that the user does not feel uncomfortable due to getting drunk. You may do. Further, in the in-vehicle system, AR display may be realized by a method other than HUD or HMD. Further, the same AR display may be realized in a system other than an automobile (for example, a game machine fixedly installed in a facility).

(B) The AR display device of the first embodiment can generate an AR image by using the information of the existing car navigation unit 106 in the vehicle as the source data of the external input, but the AR image is not limited to this. As a modification, an original car navigation unit may be provided in the HUD device 1 and an AR image may be generated by using the information. When the information on the building and the signboard on the map is prepared as the DB information of the car navigation unit, the information can be used. For example, the HUD device 1 refers to the map information of the car navigation unit based on the position information of the own vehicle by GPS, and searches for information such as a building or a signboard near the current position of the own vehicle or the position of the target object. When the HUD device 1 obtains information such as a signboard or a building associated with the signboard as the search result information, the HUD device 1 can construct an AR image using the information.

(C) In the first embodiment, the video data is externally input to the HUD device 1 by using the existing in-vehicle camera 2, and the AR display is realized. As a modification, each form in which elements such as the camera 2 and the camera 3 are included in the AR display device is possible. For example, the HUD device 1 may be provided with the camera 2 and the external photographing unit 102. The DB unit 109 may be provided in the HUD device 1.

(D) A portion corresponding to the control unit 10 (image selection unit 11 or the like) of the HUD device 1 is realized by a portion other than the HUD device 1 such as the ECU 101, and the display unit 20 of the HUD device 1 is transmitted from the control unit such as the ECU 101. AR display may be realized in cooperation with the projection type display device. In other words, the control unit 10 and the display unit 20 may be realized by different devices.

(E) Elements such as the car navigation unit 106 and the DB unit 109 include an independent device other than the in-vehicle system 100 and the HUD device 1, for example, a mobile terminal device (smartphone, tablet terminal, Web camera) brought into the vehicle by the user. It may be realized by a provided PC or the like, and the HUD device 1 or the like and the device may communicate and cooperate with each other.

(Embodiment 2)
The AR display device and the like according to the second embodiment will be described with reference to FIGS. 10 and 11. The basic configuration in the second embodiment and the like is the same as that in the first embodiment. Hereinafter, the components different from those of the first embodiment in the second embodiment and the like will be described. In the second embodiment, the AR function has an additional function. The second embodiment has a function of ensuring the visibility of the AR image of the object even when the object changes so as to move at a relatively high speed in the driver's field of view and the screen 5. When the HUD device 1 determines that the object is moving at a relatively high speed on the screen 5, the HUD device 1 controls the display position of the AR image to, for example, a stationary position so as to improve the visibility of the AR image.

FIG. 10 shows a functional block configuration of the HUD device 1 which is the AR display device of the second embodiment. The configuration of FIG. 10 is different from the configuration of FIG. 1, and the control unit 10 has a moving object determination unit 16. The moving object determination unit 16 cooperates with the image selection unit 11 and the visibility determination unit 12, processes to distinguish and extract a fixed object and a moving object from the image of the camera 2, and calculates the moving speed of the object in the image. Perform processing. Further, the DB of the DB unit 109 of FIG. 1 may include information indicating the type such as a fixed object or a moving object as the definition information of the object.

Objects in the actual scene include fixed objects such as signboards installed at fixed positions and moving objects such as other vehicles and people. Other examples of moving objects include mobile signboards (for example, in-vehicle advertisements). While the vehicle is running, all of these objects appear to change so as to move in the screen 5 and the image. In the second embodiment, as in the first embodiment, an object such as a signboard is set as an AR target. Further, in the second embodiment, moving objects such as other vehicles and people can also be set as AR targets. The AR image generation unit 13 also generates an AR image related to a moving object.

In the second embodiment, as the first function, it has a function of determining whether the object is a fixed object or a moving object, and displaying AR on the moving object as well. In the second embodiment, the image selection unit 11 of the HUD device 1 extracts not only a fixed object such as a signboard but also a moving object such as another vehicle or a person as an object from the image of the camera 2. The moving object determination unit 16 determines whether the extracted object area is a fixed object or a moving object. As a method of this determination, the moving speed described later may be used. The visibility determination unit 12 controls so as to change the AR display content as in the first embodiment according to the determination result of the moving object and the visibility index value.

Further, when the types of objects are different, for example, in the case of a signboard and a traffic sign, or in the case of a fixed object and a moving object, the HUD device 1 displays the display color of the AR image and the display color of the AR image. The display shape and the like may be different. This makes it easier for the driver to recognize the difference in the types of objects.

Further, in the second embodiment, as a second function, there is a function of changing the AR display method according to the moving speed of the screen 5 and the object in the image. In particular, it has a function of stopping the display position of the AR image when the moving speed of the object is high. The moving object determination unit 16 of the HUD device 1 calculates the moving speed and moving direction of a moving object (signboard, car, etc.) in the image of the screen 5 and the camera 2 corresponding to the driver's field of view. This moving speed is the speed at which the position of the object region moves in the screen 5 and the image, and is the traveling speed of the own vehicle and the relative speed with respect to the driver's viewpoint.

The visibility determination unit 12 uses the moving speed of the object as the visibility index value. The visibility determination unit 12 compares the moving speed of an object with, for example, a range of a predetermined threshold value. For example, when the moving speed of a signboard, which is a fixed object, is within a predetermined threshold value, the visibility determination unit 12 determines that it is easy to see and classifies the signboard (similar to the first embodiment). To do. If the visibility determination unit 12 is out of the range, the visibility determination unit 12 determines that it is difficult to visually recognize and classifies the unit into a predetermined category. That is, it is determined that the driver cannot visually follow the object moving at a relatively high speed. The visibility determination unit 12 determines a predetermined AR display content (type) in which the display position of the AR image of the moving object is a stationary position according to the latter classification. The AR image generation unit 13 determines the stationary position or the like, which is the display position of the AR image of the object, in consideration of the moving direction or the like according to the determination. Further, the AR image generation unit 13 may also maintain a constant size for the display size of the AR image of the moving object. This makes it easier for the driver to visually recognize the AR image of the moving object.

The moving speed of an object in an image can be calculated, for example, as follows. Simply, the moving speed may be calculated from the difference in the position of the object region (distance between pixels) between the image frames of the video data. Since it is only necessary to be able to classify, high accuracy is not required in the calculation of the moving speed. Alternatively, the moving speed can be calculated based on the relative positional relationship between the own vehicle and the object, for example, using the speed or angular velocity of the own vehicle when turning right or left. For example, information such as the speed, acceleration, angular velocity, and angle of the own vehicle can be obtained from the vehicle speedometer, acceleration sensor, gyro sensor, and the like of the own vehicle. Further, in the form of using the HMD device, information such as the angular velocity and the angle can be obtained by using the magnetic sensor, the gyro sensor, or the like mounted on the HMD device.

An example of the movement of the own vehicle is shown in FIG. In the vicinity of the intersection of the road 31 in the screen 5 and the field of view, the signboard 44 seen diagonally forward to the right from the viewpoint is a fixed object and is an AR target. It is assumed that the own vehicle turns right at the intersection in the direction indicated by the arrow 401 and passes in front of the signboard 44 at that time. In that case, on the screen 5, the signboard 44 appears to the driver to move at a relatively high speed.

FIG. 11 briefly shows the time-series state of the screen 5 corresponding to that case. FIG. 11A schematically shows how the object area corresponding to the signboard 44 appears to move in the field of view. For example, the object area of the signboard 44 is visible at the position of the area 91 at the first time point (for example, at the start of entering an intersection). Next, at the second time point, it moves to the area 92 as the right turn progresses. Similarly, the position is continuously changed by moving the area 93 at the third time point, the area 94 at the fourth time point, the area 95 at the fifth time point, and so on. Arrows indicate movement loci.

FIG. 11B shows an example of displaying the AR image 54 of the signboard 44 at the time corresponding to (A). The moving object determination unit 16 calculates the moving speed of the signboard 44 from, for example, the speed of the own vehicle or the angular velocity when turning right, compares it with the threshold value, and when it is equal to or higher than the threshold value, it is visually recognized because it is moving at a relatively high speed. Judging that it is difficult, we will classify and type it. The AR image generation unit 13 generates the AR image 54 of the signboard 44 accordingly. As an example of the AR image 54, the image of the front surface of the signboard 44 is enlarged to make the written character "XYZ" sufficiently large. The AR image generation unit 13 determines the display position of the AR image 54 so as to be a stationary position in the screen 5 with respect to the positions of the regions 91 to 95 that change in time series. In this example, the stationary position of the AR image 54 is a position above the signboard 44, and in particular, a position near the center in the range of regions 91 to 95.

The control unit 10 captures an object and detects an object area (or an image frame) in time series, and the position of the object area in the image at that time (for example, the position coordinates of the pixel at the center point of the object area). And, to grasp in relation. The control unit 10 may grasp the vehicle position or the object distance at each time point. The control unit 10 calculates and grasps the moving speed and the moving direction of the object in the screen 5 and the image by using the information grasped in the time series. The image selection unit 11 is, for example, based on the difference between the area 91 at the first time point and the area 92 at the second time point, the moving direction (example: substantially leftward as shown by the arrow shown) and the moving speed of the object area of the signboard 44. To calculate.

The AR image generation unit 13 determines the stationary position of the AR image 54 according to its moving direction and moving speed. Since it is known that the moving direction is substantially left at the second time point, the position on the left side of the area 92 is selected as the stationary position together with such a moving destination prediction.

As a result, the AR image 54 is displayed so as to stand still at the same position while the signboard 44 is moving on the screen 5. Therefore, the driver can easily see the AR image 54 associated with the signboard 44 while maintaining a clear state without obstructing the field of view (central region R1) during the time during driving. As a comparative example, in the form in which the AR image is displayed at a position following the movement of the object (signboard 44) on the screen 5, it becomes difficult for the driver to see when the moving speed is high.

Further, in this example, the AR image generation unit 13 is a leader line 541 which is a line segment connecting the object area of the signboard 44 moving in time series and the AR image 54 as a part of the AR image 54 or as a video effect. Is displayed. For example, in the state at the fifth time point, the leader line 541 is displayed so as to connect the area 95 and the AR image 54. The display of the leader line makes it easier for the driver to recognize the relationship between the object and the AR image.

The HUD device 1 may be used when the target object disappears from the screen 5, the distance between the object area and the AR image exceeds a predetermined value, or a predetermined time has elapsed after displaying the AR image. The display of the AR image is erased at the opportunity.

It is possible not only to determine the display position of the AR image and the like. For example, first, at the first time point, the AR image 54 is displayed in the vicinity of the area 91 of the signboard 44. After that, the AR image 54 is displayed still at the same position at the first time point with respect to the movement of the area. Moreover, at each time point, a leader line connecting the area and the AR image 54 is displayed. Further, for example, the display position of the AR image is not limited to the stationary position, and may be determined so that the movement of the AR image on the screen 5 is at least slower than the speed of movement of the target object.

As described above, according to the AR display device of the second embodiment, the AR image can be easily visually presented even for an object that moves at a relatively high speed within the driver's field of view and is difficult to follow visually.

(Embodiment 3)
The AR display device and the like according to the third embodiment will be described with reference to FIGS. 12 and 13. In the third embodiment, the line-of-sight tracking function is used to limit the range of objects to be extracted from the image of the camera 2 according to the state of the driver's line of sight. As a result, the information of the portion that is likely to be unnecessary for the driver is removed to reduce the amount of information, and the visibility of the object and the AR image in the portion that is likely to be necessary is improved.

FIG. 12 shows a functional block configuration of the HUD device 1 which is the AR display device of the third embodiment. The configuration of FIG. 12 is different from the configuration of FIG. 1, and the in-vehicle system 100 and the HUD device 1 have a line-of-sight tracking function for tracking the line of sight of the driver. The line-of-sight tracking function can be configured by a known technique. In the third embodiment, the line-of-sight tracking function is mainly realized by the driver photographing unit 107, but the present invention is not limited to this. The line-of-sight tracking function may be realized by any one of the driver photographing unit 107, the ECU 101, the control unit 10 of the HUD device 1, and the like.

The driver photographing unit 107 is a device (that is, an eye tracking device) having a known line-of-sight tracking function including a camera 3, and detects the position of the driver's eyes, the line of sight, and the like. The eye tracking device, which is the driver photographing unit 107, detects the driver's eyes and the like from the image of the camera 3 and calculates the direction of the line of sight. As a calculation method, for example, a method of calculating the line of sight from the position of a point of light reflected in the pupil is known. The driver photographing unit 107 outputs information (line-of-sight information) such as the grasped line of sight.

The control unit 10 of the HUD device 1 has a line-of-sight tracking unit 17 that cooperates with the driver imaging unit 107. The line-of-sight tracking unit 17 inputs line-of-sight information from the driver photographing unit 107 to grasp the state of the driver's line of sight. The line-of-sight tracking unit 17 grasps the gazing point (in other words, the line-of-sight intersection, the reference point), which is the intersection of the line of sight on the screen 5, from the state of the line of sight. The driver's photographing unit 107 may calculate the gazing point. The line-of-sight tracking unit 17 sets the line-of-sight center range in the screen 5 in real time with reference to the line of sight and the gazing point. The image selection unit 11 extracts an object only in the line-of-sight center range in the screen 5.

FIG. 13 shows an example of the line-of-sight center range R10 and AR image display on the screen 5. The gazing point P1 indicated by a cross indicates an example of the gazing point ahead of the line of sight. The line of sight and gaze point change from time to time. The line-of-sight tracking unit 17 sets, for example, a circular region having a predetermined radius distance as a line-of-sight center range R10 with the gazing point P1 as the center point. The line-of-sight center range R10 also fluctuates according to the fluctuation of the gazing point P1. The line-of-sight center range R10 is a range included in a predetermined angle of view when the screen 5 is viewed from the driver's point of view. There is a correspondence between the angle of view and the radius. For example, when the angle of view is about 30 degrees, it is equivalent to the effective viewing range of the human eye. When an object is included in the line-of-sight center range R10 corresponding to its effective viewing range, the object is almost visible to the driver. Further, when the angle of view is, for example, about 5 degrees, it is equivalent to the discriminative visual field of the human eye (the gaze range in which characters can be read). If the object is included in the line-of-sight center range R10 corresponding to its discriminative visual field, it is likely that the object is being gazed at. The line-of-sight center range R10 may be set by a rectangle or the like.

The image selection unit 11 performs selective extraction of the object region as in the first embodiment, with the line-of-sight center range R10 as a limited region for extracting an object from the image of the camera 2. When the object region is extracted from the line-of-sight center range R10, it is possible to extract only the objects that are likely to be intentionally gazed by the driver, and to acquire the image information of the object region.

The details regarding the setting of the line-of-sight center range R10 may be as follows. The line-of-sight tracking unit 17 adds a time determination when grasping the line of sight and the gazing point, and when the gazing point stays at the same position on the screen 5 for a certain period of time (for example, a predetermined second), the driver It may be determined that the person is gazing at the vicinity of the position. The line-of-sight tracking unit 17 may set the line-of-sight center range R10 for the first time based on the determination of the gaze.

The visibility of an object within the area of the line-of-sight center range R10 is on a case-by-case basis. For example, if the signboard is small, the visibility is low. The visibility determination unit 12 calculates a visibility index value for an object region within the line-of-sight center range R10, that is, an object region whose center point is included in the line-of-sight center range R10.

In the third embodiment, basically, an object within the region of the line-of-sight center range R10 is used as an AR image display candidate. In particular, when it is outside the central region R1 and within the region of the line-of-sight center range R10, it is used as a display candidate. Not limited to this, as a modification, it may be a display candidate as long as it is within the line-of-sight center range R10 regardless of the inside and outside of the central region R1.

The visibility determination unit 12 classifies and determines whether or not an AR image is displayed, processing details, and the like for an object within the line-of-sight center range R10 according to the visibility index value. This determination method is possible in the same manner as in the first embodiment. The AR image generation unit 13 generates an AR image according to the determination.

In the example of FIG. 13, objects such as signs 41 and 42 are included in the area of the line-of-sight center range R10. Therefore, for example, the signboard 41 is a display candidate for the AR image. Regarding the visibility of the signboard 41, for example, it is judged that it is difficult to see because the object distance is long, and it is classified as displaying an AR image. As the AR image of the signboard 41, for example, the AR image 51a in the fixed position method or the AR image 51b in the neighborhood position method may be displayed as described above. As an example of the content of this AR image, enlargement processing or the like is performed as described above. Further, when displaying the AR image, a leader line or the like connecting the object area and the AR image may be attached as described above. The leader line is arranged so as to avoid the vicinity of the gazing point P1 as much as possible. Similarly, the signboard 42 has an oblique posture and is judged to be difficult to see, and is classified as displaying an AR image. As an example of the content of the AR image 52 of the signboard 42, a posture change process or the like is performed.

The AR image 51b and the AR image 52 are displayed at a position on the circumference including a part of the line-of-sight center range R10, but are not limited to this, and are displayed at a position outside the line-of-sight center range R10. May be done.

An example in which the line-of-sight and the gazing point are moved and the line-of-sight center range R10 is set at another position is shown as the line-of-sight center range R10b. The area of the line-of-sight center range R10b includes, for example, a signboard 44. Within the line-of-sight center range R10, the signboard 44 is classified as the first category because it is judged to have high visibility, and the AR image is not displayed. The HUD device 1 may erase the AR image of the object when the object within the line-of-sight center range R10 goes out due to the movement of the line of sight and the gazing point.

As described above, according to the AR display device of the third embodiment, it is possible to present an AR image that is easy to see for an object that is difficult to see within the range of the center of the driver's line of sight. Since the processing target area is limited, the amount of information in the screen 5 can be suppressed and the visibility can be improved as a whole.

(Embodiment 4)
The AR display device and the like according to the fourth embodiment will be described with reference to FIGS. 14 and 15. In the fourth embodiment, the AR image including the related information about the object is displayed in a highly visible state. This provides useful information to the driver and enhances convenience. In the fourth embodiment, related information is searched and acquired from the DB of the DB unit 109 based on the extracted object. In the fourth embodiment, not only information such as characters described for an object visible to the driver but also character information based on related information can be provided. In the fourth embodiment, it is determined what is related to the object included in the captured image, and the related information is provided. In the fourth embodiment, the information obtained by combining the information of the actually visible object and the related information obtained from the object in the image is displayed as an AR image.

FIG. 14 shows a functional block configuration of the HUD device 1 which is the AR display device of the fourth embodiment. The configuration of FIG. 14 is different from the configuration of FIG. 1, and the control unit 10 has an information retrieval unit 18 that cooperates with the image selection unit 11 and the like. The information retrieval unit 18 searches the DB of the DB unit 109 for related information based on the information of the object extracted from the image.

The image selection unit 11 of the HUD device 1 extracts an object area of an object (for example, a signboard) from the image of the camera 2, and also extracts the character information from the object area when the character can be recognized by the OCR process. The information retrieval unit 18 uses the information of the object area extracted by the image selection unit 11 as a search condition from the DB of the DB unit 109 (or the DB of the car navigation unit 106, the DB server on the external communication network, etc.). Search for related information about the object. The information retrieval unit 18 obtains related information of the object from the search result. The related information is, for example, what signboard, which building has a signboard, and the like, and is arbitrary information registered in the DB.

The information retrieval unit 18 may search the DB using the character information extracted from the object area. Further, the information retrieval unit 18 may perform an image retrieval from the DB using the image of the object region extracted from the image. In this case, for example, related information can be searched from the appearance of the signboard. Alternatively, the information retrieval unit 18 may once acquire the basic information of the object from the DB using the information of the object area, and then search the related information from the DB or the like of the DB unit 109 using the basic information. Good.

The DB of the DB unit 109 stores information acquired from an external communication network (for example, a data center of a business operator) via, for example, the communication unit 104. As described above, various forms such as a form of searching an external DB are possible. This DB includes an image DB capable of image search. In this DB, for example, an image related to an object (object image) and information about the object (basic information and related information) are registered in association with each other.

The object image may be, for example, a high-quality reference image viewed from the front of the signboard which is the target object. In that case, the reference image can be used as the original data to generate an AR image with high visibility. In addition to the above-mentioned feature information and definition information, the basic information may include, for example, information indicating the type of a convenience store, a restaurant, or the like having a signboard (information such as items in the information category of FIG. 9). Related information includes, for example, detailed explanatory information about a store displaying a signboard. As the related information, the information of the characters written on the signboard may be included. In that case, when presenting the AR image, it is possible to present information about the store or the like displaying the signboard.

The visibility determination unit 12 determines and classifies the visibility of the object region in the same manner as described above. Based on the result, the AR image generation unit 13 generates an AR image with improved visibility in the same manner as described above, but at that time, the AR image is generated using the related information of the object. The AR image generation unit 13 generates an AR image with high visibility by combining the information directly obtained from the object area and the related information obtained by the search. The AR image generation unit 13 may use the information retrieval unit 18 to perform a search when generating an AR image.

For example, the information retrieval unit 18 searches the image DB of the DB unit 109 using the image of the object region as a search condition, and obtains an object image with high visibility from the search result. For example, an image in which the front of a signboard is a right-angled quadrangle and is photographed with high quality can be obtained. The AR image generation unit 13 generates an AR image using the image. The AR image generation unit 13 may use the image processing unit 14 to perform image processing processing on the image of the search result to further improve the visibility.

FIG. 15 shows an example of screen 5. In this example, the signboard 45 is installed at a position near the road 31 in the building 32a in the screen 5 and the image. It is assumed that the target object is, for example, a signboard 45. The front of the signboard 45 faces the driver's point of view, but it is located at a relatively distant position, and the written characters are difficult to see.

The AR image 55A shows a first display example regarding the AR image of the signboard 45. The AR image 55A is composed of an AR image 55A1 and an AR image 55A2. The AR image 55A is displayed at a position in the lower side region R3 by a fixed position method. The AR image 55A1 is an AR image based on information obtained directly from the object region. The AR image 55A1 is an AR image obtained by subjecting an object area to an enlargement process or the like as a processing process so that the characters on the signboard 45 can be easily seen. In this example, the AR image 55A1 is created using the character information extracted by the OCR process from the object area of the signboard 45. The characters on the signboard 45 are, for example, "OPEN from 10:00 to 21:00 as of OPEN", and include information such as opening time, closing time, and current opening state.

The AR image 55A2 is an AR image based on the related information of the object. The AR image 55A2 is created as an image with high visibility by using the character information "Restaurant XYZ", for example, as the related information of the signboard 45. This related information indicates that the type of store displaying the signboard 45 is a restaurant and the name of the store is "XYZ".

The AR image 55B shows a second display example regarding the AR image of the signboard 45. The AR image 55B is composed of an AR image 55B1 and an AR image 55B2. The AR image 55B is displayed with a leader line attached to a position (example: left side) near the signboard 45 in the neighborhood position method. The AR image 55B1 is an AR image based on information obtained directly from the object region. The AR image 55B1 is an AR image obtained by subjecting an object area to an enlargement process or the like as a processing process so that the characters on the signboard 45 can be easily seen. The AR image 55B2 is an AR image based on the related information of the object, like the AR image 55A2. In this example, the AR image 55B2 is displayed as an area of a broken line frame below the AR image 55B1. The background of each AR image is a transparent area.

As described above, according to the AR display device of the fourth embodiment, more detailed information including the related information of the object can be presented as an AR image that is easy to see, and the convenience and added value of the driver's AR use. Can be enhanced. Further, in the fourth embodiment, the image search is used to generate and display an AR image with high visibility by using an independent image with high visibility that is different from the original object with low visibility. Is also possible. Therefore, even when it is difficult to obtain an image with high visibility by image processing, a suitable AR image can be presented.

A general image search service on the Web may be used for image search, but if an object image with high visibility is prepared in advance in the image DB of the DB unit 109, the speed is high. Processing and suitable AR image presentation are possible. Further, as a modification, a plurality of images of the target object taken in advance under various conditions may be registered in the image DB of the DB unit 109. For example, the images have different conditions such as camera lens performance, shooting distance, and weather. In the image database, a low-quality image in a situation where the visibility of the target object is not good, a high-quality image in a situation where the visibility is good, and other object information are registered in association with each other. As a result, it is possible to realize a suitable AR display corresponding to various situations when using AR. Even a low-quality image when the shooting conditions are not good can be obtained by searching.

The image search function of the information search unit 18 may be used when the image selection unit 11 determines to extract an object from an image. That is, the information retrieval unit 18 performs image retrieval from the image DB (in other words, image matching with other image groups) using the image of the object region as a search condition. The image selection unit 11 can extract a predetermined object (for example, a signboard) from the image search result.

[Modification example (1)]
The following is possible as a modification of the fourth embodiment. The HUD device 1 uses the related information of the object in the image to determine the presence or absence of another object (for example, a building) in the space associated with the object (for example, a signboard) in the image. If the HUD device 1 can identify other objects in the space, the HUD device 1 may present an AR image by associating those objects (signboard and building). For example, as a building associated with the signboard 45 of FIG. 15, a building 32a or a building 32b in the vicinity thereof can be considered as candidates. For example, the information retrieval unit 18 may search the DB for a building within a predetermined radius distance centered on the position information of the signboard 45 in the space, and may use the search result building as a candidate.

Further, the information retrieval unit 18 may search the information of the building from the DB using the position information of the candidate building and determine whether the building is associated with the target signboard. For example, suppose that the signboard 45 is determined to be associated with the building 32a.

When the HUD device 1 identifies the building 32a associated with the signboard 45, the HUD device 1 generates and displays an AR image including information on the associated building 32a as an AR image of the signboard 45. Alternatively, the HUD device 1 may generate another AR image relating to the building 32a, and display the AR image of the signboard 45 and the AR image of the building 32a in association with each other by a leader line or the like.

[Modification example (2)]
The following is possible as a modification of the fourth embodiment. This modified example has a function of performing maintenance and updating regarding the DB information of the DB unit 109. It is assumed that the DB of the DB unit 109 is provided on an external communication network, for example, in a DB server of a data center. The business operator manages the DB. Users of a plurality of automobiles use the same service, and the in-vehicle system 100 accesses so as to share the same DB. The DB updates the image and information of the object so as to be in the latest state based on the information of the object obtained from the image taken by each user's automobile.

For example, the installation position of the signboard may change, and the description content may change. In addition, the appearance of the building may change, such as under construction. As for the map information and the like of the DB of the car navigation unit 106, if the registered contents are not updated, the information of the latest actual object cannot be obtained in real time. Therefore, this modification has a function of updating the information content of the DB of the DB unit 109 to the latest state so that the change in the appearance of the object can be dealt with as soon as possible. As a result, each user can obtain the latest information on the object as an AR image when using the AR function, which is highly convenient. At the same time, the maintenance work of the DB of the business operator becomes easy.

In the DB of the DB unit 109 of the data center, as object information and images, position and orientation information, information such as whether or not the appearance of the object is changed and the date and time of the change, and the latest object area when the appearance of the object is changed Images, etc. are registered in association with each other. Further, information such as the shooting direction of the camera 2 regarding the object image may be registered in the DB. A plurality of object images obtained by photographing the same object from each direction may be registered.

The HUD device 1 extracts an object region in the latest state from the image of the camera 2. The information retrieval unit 18 searches the DB of the DB unit 109 via the communication unit 104 using the image or information of the object area. The information retrieval unit 18 transmits the latest information on the object to the server of the data center. The server of the data center receives the information, confirms the registration information of the DB, and updates the registration information of the DB to the latest state when it is determined that the appearance of the object is changed.

At the time of AR presentation, the latest AR image with high visibility can be displayed by using the object information and the object image acquired from the DB. Further, when a plurality of object images having different shooting directions are used, a suitable AR image can be generated according to the direction from the own vehicle to the object.

[Modification example (3)]
The following is possible as a modification of the fourth embodiment. In this modification, the fluctuating position of the moving object in the real space is registered so as to be reflected in the map information of the DB of the DB unit 109, and can be shared by oneself or a plurality of users including oneself. As a moving object, for example, another automobile driven by a person such as a driver's family member or an acquaintance can be set as a target. The target car or person can be set in advance in an image format. Also, as an application example, it can be used for searching for cars and people. In addition, as an application example, it can also be used for registering conditions such as weather and traffic jams.

The HUD device 1 detects a position in space with respect to an object in an image taken by the camera 2. For example, the position of an object can be calculated with a certain degree of accuracy based on the relationship between the position information of the own vehicle and the position of the object in the image. Moreover, if the position of a known object on the map is used, the position of the target object can be detected with high accuracy.

The HUD device 1 accesses, for example, the map information of the DB of the DB unit 109 of the external data center. Then, the HUD device 1 registers information including the position of the own vehicle, the position of the detected object, the image of the appearance, the sensor information, and the like in the map information of the DB. The server of the data center registers and updates the information of the DB, aggregates the status, and the like. The in-vehicle system 100 of each user can access and use the updated map information of the DB. Each user can obtain the latest information such as another person's car as an AR image.

Although the present invention has been specifically described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist thereof. It is possible to add or delete the components of the embodiment, separate or merge, replace, combine, and the like. The numerical values and the like of the specific examples of the embodiment are examples. Some or all of the functions of the embodiment may be realized by hardware such as an integrated circuit, or may be realized by software program processing. Each software may be stored in the device in advance at the time of product shipment, or may be acquired from an external device via communication after the product is shipped. The present invention is not limited to in-vehicle systems, and can be applied to various uses.

1 ... HUD device (AR display device), 2 ... camera, 3 ... camera, 5 ... screen, 9 ... front shield, 10 ... control unit, 11 ... image selection unit, 12 ... visibility judgment unit, 13 ... AR image generation Unit, 14 ... Image processing unit, 15 ... AR display unit, 20 ... Display unit, 21 ... Display drive circuit, 22 ... Display element, 23 ... Light source, 24 ... Optical system, 100 ... In-vehicle system, 101 ... ECU, 102 ... Outside the vehicle shooting unit, 103 ... video data storage unit, 104 ... communication unit, 105 ... GPS receiver, 106 ... car navigation unit, 107 ... driver photography unit, 108 ... sensor unit, 109 ... DB unit.

Claims (5)

It is an AR display device that superimposes an AR image, which is image information of augmented reality, on a screen that is arranged in front of the user and transmits the actual view of the outside world.
An imaging unit that has a predetermined positional relationship with the user and captures at least a part of the actual scene of the outside world to acquire an image.
An AR image generation unit that generates the AR image of the target object selected and extracted from the captured images, and
In addition to being provided with an AR display unit for displaying the AR image on the screen,
The first distance between the photographing unit and the extracted target object is measured or calculated, and the first distance between the target object and the user is based on the predetermined positional relationship between the photographing unit and the user. Distance information acquisition means for calculating the distance of 2 and
Further provided with a visibility determination unit that compares the second distance with a predetermined threshold distance and determines the level of visibility of the user with respect to the target object.
As a result of the comparison, when the visibility determination unit determines that the visibility is low, the AR image generation unit generates the AR image, and the AR display unit generates the generated AR image and displays it on the screen. An AR display device that controls the operation. The AR display device according to claim 1, wherein the second distance is a distance between the target object and the position of the user's eyes. The AR display device according to claim 1 or 2, wherein the distance information acquisition means includes a distance sensor. The AR display device according to any one of claims 1 to 3, wherein the AR image generation unit generates the AR image by performing an image processing process on the target object to enhance the visibility. .. According to any one of claims 1 to 4, the line of sight of the user is tracked, and when the tracked line of sight stays at substantially the same position for a predetermined time or longer, the target object is extracted from the vicinity of that position. The AR display device described.

Images