JP2018090170A - Head-up display system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head-up display system which can display an image in a display mode that is more visible for a driver.SOLUTION: A HUD control unit 13 includes a camera image acquisition part F3, a driver view point image generation part F4, a visibility evaluation part F5 and a hue adjustment part F61. The camera image acquisition part F3 acquires image data imaged by a stereo camera unit which is arranged so as to image a region where a HUD image is displayed as a virtual image. The driver view point image generation part F4 generates an image (hereinafter, a driver view point image) being the image at the time when the region where the HUD image is displayed is viewed from the view point of the driver on the basis of the image data acquired by the camera image acquisition part F3. The visibility evaluation part F5 quantitatively evaluates the visibility of the HUD image on the basis of the driver view point image. The hue adjustment part F61 sequentially adjusts the hue of the HUD image so that the visibility becomes a prescribed level.SELECTED DRAWING: Figure 4

Description

  The present invention provides a head-up display system that forms an image as a virtual image in front of a vehicle by irradiating the windshield of the vehicle with display light from a display, and presents information to a driver's seat occupant using the virtual image. About.

  Conventionally, a head-up display for a vehicle (hereinafter referred to as HUD: Head) that presents a virtual image to an occupant (hereinafter referred to as a driver) seated in a driver's seat by irradiating the windshield with display light from a display device -Up Display) system. The visibility of an image (hereinafter referred to as a HUD image) displayed by such a HUD system varies depending on the illuminance of the environment outside the passenger compartment and the color of an object that appears around the HUD image as viewed from the driver.

  For example, in a situation where green text is displayed as a HUD image, the visibility of the HUD image deteriorates when a green preceding vehicle is present at a position overlapping the HUD image. Further, when the outside of the passenger compartment is bright, it is relatively difficult to see compared to when the outside of the passenger compartment is dark.

  In order to deal with such a problem, Patent Document 1 discloses that an image in front of the vehicle (hereinafter referred to as a front image) is acquired using a camera that images the front of the vehicle, and the average value of the brightness of each color component in the front image is obtained. Based on this, a configuration for adjusting the hue of the HUD image is disclosed. Specifically, the average value of the brightness of each of the three color components of red, green, and blue in the front image is calculated, and the color component having the highest average value of brightness is specified. Then, the HUD image color is set to a hue on the side opposite to the color component having the highest average value. For example, when the average value of the lightness of red is the highest, the lightness of green and blue is set to the maximum value.

JP2015-178297A

  Since there are guardrails, road surfaces, road fences, road signs, other vehicles, and the like in front of the vehicle, the color of the front image is not uniform. Various colors are mixed with various luminances in the field of view of the driver looking at the front of the vehicle. Therefore, just because the color of the HUD image is set to the color opposite to the color component having the highest average brightness value in the front image (hereinafter referred to as the average brightness maximum color), the visibility of the HUD image is not necessarily good. Not necessarily.

  The present invention has been made based on this situation, and an object of the present invention is to provide a head-up display system capable of displaying a HUD image in a display mode that is easier for the driver to see.

  In order to achieve the object, the present invention provides a display (1) for irradiating display light for displaying an image as a virtual image to a predetermined irradiation area of a windshield disposed on the front side of a vehicle, and a display A display mode adjustment unit (F61) that adjusts the display mode of a HUD image that is a virtual image displayed by irradiating the irradiation area with display light, and image data captured by a camera that captures the front of the vehicle Based on the driver viewpoint image acquisition unit (F4, which acquires an image when the vehicle front is viewed from the viewpoint of the driver who is a passenger seated in the driver's seat and includes an HUD image) F3) and the driver viewpoint image acquired by the driver viewpoint image acquisition unit are analyzed to sequentially determine the visibility index value indicating the visibility of the HUD image in the driver viewpoint image. A visibility evaluation unit (F5) to be output, and an appropriate range storage unit (M2) in which data indicating an appropriate range of the visibility index value calculated by the visibility evaluation unit is registered. The display mode of the HUD image is adjusted so that the visibility index value calculated by the visibility evaluation unit is a value within an appropriate range.

  With the above configuration, the visibility evaluation unit sequentially calculates the visibility index value indicating the visibility of the HUD image when viewed from the viewpoint of the driver, using the driver viewpoint image acquired by the driver viewpoint image acquisition unit. . Then, the display mode of the HUD image is adjusted by the display mode adjusting unit so that the visibility index value becomes a value within the appropriate range. That is, the visibility of the HUD image for the driver is quantitatively evaluated, and the display mode of the HUD image is controlled so that the visibility expected by the designer or the like is ensured. Therefore, the HUD image can be displayed in a display mode that is easier for the driver to see.

  In addition, the code | symbol in the parenthesis described in the claim shows the correspondence with the specific means as described in embodiment mentioned later as one aspect, Comprising: The technical scope of this invention is limited is not.

It is a figure which shows the example of mounting to the vehicle of the head up display system 100. FIG. 3 is a diagram illustrating an example of a HUD image 20. FIG. 1 is a block diagram showing a configuration of a head-up display system 100. FIG. 3 is a block diagram illustrating a schematic configuration of a HUD control unit 13. FIG. It is a flowchart for demonstrating a display control process. It is a flowchart for demonstrating visibility evaluation processing. It is a figure for demonstrating the relationship between the magnitude | size of a calculation object area | region, and a vehicle speed. It is a figure which shows an example of a calculation object area | region. It is a figure showing the visual saliency map calculated with respect to the image shown in FIG. It is a figure for demonstrating the color pattern which can be selected as a background representative color. It is a figure for demonstrating the action | operation of the hue adjustment part F61. It is a figure which shows the structure of the HUD system 100 in the modification 5. FIG. It is a figure which shows the structure of the HUD system 100 in the modification 6. FIG. It is a figure which shows the structure of the HUD control part 13 in the modification 6. FIG. It is a figure which shows the structure of the HUD system 100 in the modification 7. FIG. It is a figure which shows the structure of the HUD system 100 in the modification 7. FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an example of a schematic configuration of a head-up display (hereinafter, HUD) system 100 according to the present invention. As shown in FIG. 1, the HUD system 100 is used by being mounted on a vehicle such as a four-wheeled vehicle.

  In general, the HUD system 100 irradiates the irradiation area set on the windshield 10 of the vehicle with the display light, thereby connecting the eyes of the driver's seat occupant and the irradiation area as shown in FIG. The image 20 is displayed as a virtual image on the vehicle front extension line. The windshield 10 is a windshield on the front side of the vehicle. The windshield 10 may be realized using, for example, a laminated glass formed of two sheets of glass and an intermediate film provided in the middle.

  The display light here refers to light that forms a virtual image as the image 20. With this HUD system 100, an occupant (hereinafter referred to as a driver) sitting in the driver's seat can visually recognize the HUD image 20 and the foreground of the vehicle in a superimposed manner. For convenience, hereinafter, the image 20 perceived by the driver is referred to as a HUD image.

  Here, the information indicated by the HUD image 20 is, for example, the traveling speed of the vehicle. Of course, as another aspect, the HUD image 20 may be an image indicating any one or more of the engine speed, the engine coolant temperature, the battery voltage, and the like as vehicle information during vehicle travel. Moreover, it can be set as the map image in a navigation system for vehicles, the image (the image as what is called a turn-by-turn) which shows the advancing direction in an intersection, etc.

  As shown in FIG. 3, the HUD system 100 includes a HUD unit 1, a stereo camera unit 2, and a vehicle speed sensor 3. The HUD unit 1 is communicably connected to the stereo camera unit 2 and the vehicle speed sensor 3 via a local area network (hereinafter, LAN: Local Area Network) 30 constructed in the vehicle.

  The HUD unit 1 is a device that irradiates the windshield 10 with light (that is, display light) that forms the HUD image 20 and presents information to the driver using the HUD image 20. The HUD unit 1 is accommodated in an instrument panel 40 that extends from the lower end of the windshield 10 to the rear of the vehicle interior and further downward. Note that an opening 40 a through which display light emitted from the HUD unit 1 passes is provided on the upper surface of the instrument panel 40. The opening 40a is provided with a light-transmitting dustproof cover.

  The HUD unit 1 includes a communication driver 11, a display device 12, and a HUD control unit 13 as finer components. The communication driver 11 is configured to allow the HUD control unit 13 to perform mutual communication or unidirectional communication with other devices connected to the LAN 30 such as the stereo camera unit 2. The communication driver 11 is configured to communicate with the HUD control unit 13.

  The display 12 is configured to emit display light toward an irradiation region set on the windshield 10 based on a control signal input from the HUD control unit 13. In the present embodiment, as an example, the display device 12 is a projector of a type that outputs display light by transmitting light from the backlight 121 to the liquid crystal panel 122 (so-called liquid crystal projector). In addition to the backlight 121 and the liquid crystal panel 122, the display device 12 includes a reflection device 123 for guiding light output from the liquid crystal panel 122 to an irradiation area.

  Here, as an example, it is assumed that the backlight 121, the liquid crystal panel 122, and the reflection device 123 are arranged in this order in the vehicle front-rear direction. Of these three, it is assumed that the backlight 121 is located closest to the vehicle rear side and the reflection device 123 is located closest to the vehicle front side. Note that the arrangement of these members is a design matter that can be changed to form a predetermined optical path.

  The backlight 121 is a light source that emits white light to the liquid crystal panel 122 when energized. The backlight 121 is realized using, for example, a plurality of light emitting diodes (LEDs) and a prism that guides light emitted from each LED to the liquid crystal panel 122. The backlight 121 is arranged so that the output light is incident on the liquid crystal panel 122 perpendicularly. For example, the backlight 121 is disposed facing the liquid crystal panel 122 so that the optical axes of the backlights 121 overlap.

  The liquid crystal panel 122 includes a display surface formed in, for example, a rectangular flat plate shape. A large number of pixels are two-dimensionally arranged on the display surface. Each pixel is provided with red (R), green (G), and blue (B) sub-pixels. The liquid crystal panel 122 can display various images in color on the display surface by changing the light transmittance of the sub-pixels. The transmittance for each sub-pixel is controlled by a control signal input from the HUD control unit 13.

  The liquid crystal panel 122 emits display light forming the HUD image 20 toward the reflection device 123 disposed on the opposite side of the backlight 121 by the light incident from the backlight 121. The display surface is arranged so that the optical axis of the display light faces the front-rear direction of the vehicle, in other words, the emission direction faces the front side of the vehicle.

  The liquid crystal panel 122 includes, for example, a TFT liquid crystal panel using a thin film transistor (TFT), a dual scan type display (d-STN: Dual Scan Super Twisted Nematic), and a TN (Twisted Nematic) segment. It can be realized using a liquid crystal or the like.

  The reflection device 123 is a device that reflects display light incident from the liquid crystal panel 122 toward an irradiation region through an opening 40 a provided in the instrument panel 40. The reflection device 123 includes a mirror member for reflecting display light. The mirror member may be formed by depositing a metal such as aluminum or silver on a transparent glass plate. For convenience, the surface for reflecting the display light from the liquid crystal panel 122 in the mirror member is referred to as a reflective surface. The mirror member is, for example, a concave mirror in which the central portion of the reflecting surface is recessed on the back side (the side opposite to the reflecting surface) compared to the peripheral portion. By forming the mirror member as a concave mirror, the HUD image 20 can be enlarged and irradiated onto the irradiation region.

  The HUD control unit 13 is configured to control operations of the backlight 121 and the liquid crystal panel 122. That is, the brightness of the backlight 121 is adjusted and the display screen is controlled on the liquid crystal panel 122. The HUD control unit 13 is realized using a computer. That is, the HUD control unit 13 includes a CPU 131 as a central processing unit, a RAM 132 that is a volatile storage medium, a ROM 133 that is a nonvolatile storage medium, a flash memory 134 that is a rewritable nonvolatile storage medium, an I / O, and the like. Is provided. The ROM 133 stores a program (hereinafter referred to as a HUD control program) for causing a normal computer to function as the HUD control unit 13 in the present embodiment.

  Note that the above-described HUD control program only needs to be stored in a non-transitory tangible storage medium. Executing the HUD control program by the CPU 131 corresponds to executing a method corresponding to the HUD control program. The functions of the HUD control unit 13 will be described later separately.

  The stereo camera unit 2 is a device that includes two cameras, a first camera 21 and a second camera 22. Each of the first camera 21 and the second camera 22 is a color camera that outputs image data including color information of an imaging target. In the stereo camera unit 2, the first camera 21 and the second camera 22 are arranged in parallel at a predetermined interval so as to capture an image in the same direction. The stereo camera unit 2 is disposed on the upper surface of the instrument panel 40 so that both the first camera 21 and the second camera 22 image the front of the vehicle including the irradiation area.

  The stereo camera unit 2 is arranged as close to the driver's seat as possible on the line segment obtained by orthogonally projecting the line segment connecting the center of the iris set to the center of the irradiation area to the upper surface of the instrument panel 40. It is preferable. Here, as shown in FIG. 1, it is assumed that the instrument panel 40 is disposed at the end on the driver's seat side of the upper surface portion. The imaging data of the first camera 21 and the imaging data of the second camera 22 included in the stereo camera unit 2 are sequentially provided to the HUD control unit 13. The eyelips are set based on an eye range that statistically represents the eye point distribution of the driver's occupant (that is, the driver) (see JIS D0021: 1998 for details).

  Data indicating the mounting position and mounting posture of the stereo camera unit 2 in the vehicle is registered as a camera parameter in a flash memory 134 provided in the HUD control unit 13. The mounting position may be represented by coordinates in a three-dimensional coordinate system having a predetermined position in the vehicle as an origin, such as the center of the eyelips or the center of the vehicle body. The mounting posture of the stereo camera unit 2 is represented by, for example, angles formed by the optical axis of the stereo camera unit 2 with respect to the vehicle front-rear direction, the vehicle width direction, and the vertical direction (that is, roll angle, pitch angle, yaw angle). Just do it.

  If the mounting position and mounting posture of the stereo camera unit 2 are determined, the mounting positions and mounting postures of the first camera 21 and the second camera 22 are uniquely determined. Therefore, the camera parameters registered in the flash memory 134 function as data indicating the mounting positions and mounting postures of the first camera 21 and the second camera 22. The storage destination of the camera parameters may be the ROM 133. Further, the camera parameters may include an imaging field angle, a lens distortion coefficient, a focal length, a pixel size, a pixel ratio, and the like in addition to a value of six degrees of freedom that uniquely determine the installation position and orientation.

  The vehicle speed sensor 3 is a sensor that detects the traveling speed of the vehicle (hereinafter, vehicle speed). The vehicle speed sensor 3 sequentially provides vehicle speed data indicating the vehicle speed to the HUD control unit 13.

<About functions of HUD control unit>
The HUD control unit 13 provides various functions shown in FIG. 4 when the CPU 131 executes a HUD control program stored in the ROM 133. That is, the HUD control unit 13 includes a vehicle speed acquisition unit F1, a base image generation unit F2, a camera image acquisition unit F3, a driver viewpoint image generation unit F4, a visibility evaluation unit F5, and a display control unit F6 as functional blocks.

  Note that some or all of the functional blocks described above may be realized as hardware by one or a plurality of ICs. In addition, some or all of the functional blocks included in the HUD control unit 13 may be realized by a combination of software execution by the CPU 131 and hardware members.

  Further, the HUD control unit 13 includes a camera parameter storage unit M1 and an appropriate range storage unit M2 as a configuration realized using a storage area included in the flash memory 134. The camera parameter storage unit M1 is a storage area that stores the above-described camera parameters. The appropriate range storage unit M2 is a storage area in which an appropriate range of the visibility index value described later is registered. The appropriate range of the visibility index value is specified in advance by a test. The camera parameter storage unit M1 and the appropriate range storage unit M2 may be realized using a storage area provided in the ROM 133.

  The vehicle speed acquisition unit F1 sequentially acquires vehicle speed data output from the vehicle speed sensor 3. The vehicle speed data acquired by the vehicle speed acquisition unit F1 is stored in the RAM 132 for a certain period of time, for example. The plurality of vehicle speed data having different acquisition times may be sorted and stored in the RAM 132 in chronological order so that the latest data comes first. The vehicle speed data stored in the RAM 132 is referred to by the base image generation unit F2, the visibility evaluation unit F5, and the like.

  Based on the vehicle speed data acquired by the vehicle speed acquisition unit F1, the base image generation unit F2 generates an image (hereinafter referred to as a base image) that is the basis of the HUD image 20 to be displayed on the liquid crystal panel 122. The base image is an image that represents text representing the current vehicle speed in a preset design. The design here includes the arrangement of character strings, fonts of characters, colors, and the like. The base image generated by the base image generation unit F2 is provided to the display control unit F6.

  The camera image acquisition unit F <b> 3 acquires image data captured by the first camera 21 and the second camera 22 from the stereo camera unit 2. The captured image data of each camera acquired by the camera image acquisition unit F3 is provided to the driver viewpoint image generation unit F4.

  The driver viewpoint image generation unit F4 is an image when the vehicle front is viewed from the viewpoint of the driver based on the image data captured by each of the first camera 21 and the second camera 22, and is currently displayed on the HUD. A driver viewpoint image including the image 20 is generated. Specifically, based on the amount of parallax between the first camera 21 and the second camera 22 and the camera parameters of the stereo camera unit 2, a preceding vehicle existing in front of the vehicle, a pedestrian, a white line on the road, HUD, The coordinates in the three-dimensional space of an object that is imaged in common by two cameras, such as an image, are specified.

  Then, using these coordinates as a base point, an image assuming that a monocular color camera is present at the driver's eye position is obtained. Since a technique for converting an image at a certain viewpoint into an image viewed from another viewpoint is well known, detailed description thereof is omitted here. The position where the driver's eyes are assumed to be present may be, for example, the center of the eyelips. Here, as an example, the headrest is set to a position of 19 cm in front of the vehicle from the center of the front surface of the vehicle. The assumed position of the driver's eyes may be represented by coordinates in a three-dimensional coordinate system used to indicate the mounting position of the camera. Coordinate data indicating the assumed position of the driver's eyes (hereinafter referred to as viewpoint coordinate data) may also be stored in the camera parameter storage unit M1.

  The visibility evaluation unit F5 analyzes the luminance and color distribution of each pixel included in the driver viewpoint image generated by the driver viewpoint image generation unit F4, thereby indicating an index value indicating the visibility (that is, visibility) of the HUD image 20. (Hereinafter, visibility index value) is calculated. The specific operation of the visibility evaluation unit F5 when calculating the visibility index value will be described later separately. Then, the visibility evaluation unit F5 determines whether or not the calculated visibility index value is within the appropriate range (including the boundary). As will be described later, determining whether or not the visibility index value is within an appropriate range determines whether or not the visibility of the HUD image 20 is at an appropriate level desired by the designer. It corresponds to that. The visibility determination result by the visibility evaluation unit F5 is notified to the display control unit F6.

  The display control unit F6 is configured to control the display mode of the HUD 20 by controlling the display screen of the liquid crystal panel 122 based on the determination result of the visibility evaluation unit F5. The image displayed on the liquid crystal panel 122 (hereinafter referred to as a display image) is a base image generated by the base image generation unit F2 or an image obtained by adjusting the color of the base image. The display control unit F6 causes the windshield 10 to emit display light corresponding to the display image by causing the liquid crystal panel 122 to display the display image.

  The display light mounted on the windshield 10 forms a virtual image as the HUD image 20 as described above. That is, the HUD image 20 is a virtual image corresponding to the display image. Adjusting the hue of the display image corresponds to adjusting the appearance (that is, the display mode) of the HUD image 20. The display control unit F6 includes a hue adjustment unit F61 as a sub-function for controlling the hue of the HUD image 20 (in other words, a display image).

  The display control unit F6 generally operates as follows. First, when the base image provided from the base image generation unit F2 is directly used as a display image, the display control unit F6 has a visibility index value within an appropriate range by the visibility evaluation unit F5. Is determined, the current display mode of the display image is maintained. That is, the state in which the base image is displayed on the liquid crystal panel 122 as it is is maintained.

  Further, in the case where the base image is directly used as the display image, when the visibility evaluation unit F5 determines that the visibility index value is outside the appropriate range, the hue adjustment unit F61 Then, as the display image, an image in which the hue of the base image provided from the base image generation unit F2 is adjusted is generated. Then, the liquid crystal panel 122 is caused to display the image whose color is adjusted as a display image.

  Details of the operation of the hue adjustment unit F61 will be described later. The adjustment of the hue of the base image can be realized, for example, by changing the gradation value of each color (specifically, RGB) constituting the base image. Various colors that can be perceived by humans are represented by combinations of gradation values for each of the three color components of red (R), green (G), and blue (B). The gradation value ranges from 0 to 255. The closer the gradation value of a certain color component is to 0, the lighter that color component is. When all the gradation values of each color are 0, the color is black, and when all the gradation values of each color are 255, the color is white. Data (hereinafter referred to as adjustment setting data) indicating the amount of hue adjustment performed on the base image is stored in the RAM 132.

  Further, the display control unit F6 displays an image in which the color of the base image is adjusted, and when the visibility evaluation unit F5 determines that the visibility index value is outside the proper range. Causes the hue adjustment unit F61 to execute the hue adjustment again. In the case of an image in which the hue of the base image is adjusted, when the visibility evaluation unit F5 determines that the visibility index value is within the appropriate range, the current display mode of the display image is maintained. Maintaining the display mode corresponds to maintaining the adjustment amount of the hue (specifically, the RGB value). The hue adjustment unit F61 corresponds to the display mode adjustment unit described in the claims.

[Display control processing]
Next, display control processing performed by the HUD control unit 13 will be described using the flowchart shown in FIG. The display control process is a process for maintaining the visibility of the HUD image 20 at a predetermined level (that is, an appropriate state). The flowchart shown in FIG. 5 may be executed sequentially when the conditions for displaying the HUD image 20 are satisfied. Specifically, it may be executed at predetermined time intervals (for example, every 0.5 seconds) while the vehicle is running.

  First, in step S1, the camera image acquisition unit F3 acquires image data captured by the first camera 21 and the second camera 22 from the stereo camera unit 2, and proceeds to step S2. In step S2, the driver viewpoint image generation unit F4 generates a driver viewpoint image based on the image data acquired in step S1, and proceeds to step S3. In step S3, the visibility evaluation unit F5 executes a process of calculating a visibility index value using the driver viewpoint image (hereinafter referred to as visibility evaluation process), and proceeds to step S4.

  Here, the visibility evaluation process will be described with reference to the flowchart shown in FIG. In step S31 as the first step in the visibility evaluation process, the current vehicle speed stored in the RAM 132 is read, and the process proceeds to step S32.

  In step S32, an area (hereinafter referred to as a calculation target area) used when calculating the visibility index value in the driver viewpoint image is extracted. The calculation target area is defined by a horizontal angle range and a vertical angle range with respect to a half line (hereinafter referred to as a line-of-sight center line) from the center of the iris to the center of the irradiation area. Here, the horizontal angle range and the vertical angle range are set to a square shape using the same parameters (hereinafter, target region angles). Of course, as another aspect, the calculation target area may be set to be circular. Further, it may be set to a rectangular shape or an elliptical shape that is long in the vehicle width direction. The calculation target area may be set so that the HUD image 20 is included.

  In the present embodiment, the target area angle is set to a smaller value as the vehicle speed increases as shown in FIG. Specifically, when the vehicle speed is less than the predetermined first threshold value V1, the target area angle is set to 20 degrees. Further, when the vehicle speed is equal to or higher than the second threshold value V2 that is larger than the predetermined first threshold value V1, the target area angle is set to 10 degrees. When the vehicle speed is greater than or equal to the first threshold value V1 and less than the second threshold value V2, the target area angle is set to a daily value of 20 degrees to 10 degrees as the vehicle speed increases. According to such a setting, the target region angle decreases as the vehicle speed increases.

  Generally, the effective visual field when a human recognizes a character string is about 10 degrees to 20 degrees, and the human visual field tends to narrow as the vehicle speed increases. Therefore, according to the above setting, it is possible to extract a necessary and sufficient area for judging the visibility of the HUD display. Since the extra background area is excluded, the calculation load on the CPU 131 can be reduced. In addition, the visibility of the HUD image 20 can be evaluated with higher accuracy by not using a background region that does not affect the visibility of the HUD image 20 as a calculation target region in the driver viewpoint image. The visibility evaluation part F5 which implements this step S32 is corresponded to the object area | region extraction part as described in a claim.

  When the specification and extraction of the calculation target area in step S32 are completed, the process proceeds to step S33, and a value (hereinafter referred to as saliency) representing the visual saliency for each pixel constituting the calculation target area is calculated. The visual saliency here is, in other words, the degree of visual conspicuousness. The saliency of a certain pixel in the image is a numerical value of the degree of conspicuousness of the pixel in the image.

  The method for calculating the saliency per pixel is described in “L. Itti, C. Koch, and E. Niebur.“ A model of saliency-based visual attention for rapid scene analysis, ”IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 11, pp. 1254-1259, 1998. "and J. Harel, C. Koch, and P. Perona," Graph-based visual saliency, "Advances in Neural Information Processing Systems, pp. 545- 552, 2007. ”and the like, and detailed description thereof is omitted here. According to these methods, the saliency for each pixel can be calculated in consideration of the luminance distribution and color distribution of the background region. The visibility evaluation unit F5 that performs this step S33 corresponds to the saliency calculation unit described in the claims.

  In addition, the result (what is called a saliency map) which calculated the visual saliency about the image shown in FIG. 8 is shown in FIG. In FIG. 9, the whiter the color, the higher the visual saliency (that is, the more noticeable). FIG. 8 shows the calculation target area at a certain time. “37 km / h” in FIG. 8 represents the HUD image 20.

  When the calculation of the saliency for each pixel in step S33 is completed, the process proceeds to step S34. In step S34, an area where the HUD image 20 is shown in the calculation target area (hereinafter referred to as the HUD image area) is specified by pattern matching or the like. Then, the HUD image saliency indicating the visual saliency of the HUD image 20 is calculated using the saliency for each pixel constituting the HUD image region as a population. The HUD image saliency is, for example, an average value of saliency for each pixel constituting the HUD image region. As another aspect, the HUD image saliency may be a median value of saliency for each pixel constituting the HUD image region. The visibility evaluation part F5 which implements this step S34 corresponds to the HUD image saliency calculation part described in the claims. When the calculation process in step S34 is completed, the process proceeds to step S35.

  In step S35, the background area saliency indicating the visual saliency of the background area is calculated using the saliency for each pixel constituting the portion other than the HUD image area (hereinafter referred to as background area) in the calculation target area. Similar to the HUD image saliency, the background region saliency is an average value of saliency for each pixel constituting the background region. Of course, as another aspect, the background region saliency may be a median value of saliency for each pixel constituting the background region. The visibility evaluation unit F5 that executes step S35 corresponds to the background region saliency calculation unit described in the claims. When the calculation process in step S35 is completed, the process proceeds to step S36.

  In step S36, the background region saliency is subtracted from the HUD image saliency, and the value obtained by the calculation process is adopted as the visibility index value. When a series of processes from step S31 to S36 is completed, the process returns to the flowchart of FIG.

  In step S4 in FIG. 5, it is determined whether or not the visibility index value calculated by the visibility evaluation process in step S3 is within the appropriate range registered in the appropriate range storage unit M2. If the visibility index value is within the appropriate range, an affirmative determination is made in step S4 and the process proceeds to step S5. On the other hand, if the visibility index value is outside the appropriate range, a negative determination is made in step S4 and the process proceeds to step S6.

  The lower limit value and the upper limit value of the appropriate range are parameters that define a state in which the visibility of the HUD image 20 is at a level desired by a designer or the like. Each of the lower limit value and the upper limit value of the appropriate range may be appropriately designed by a test. For example, as a result of having a plurality of subjects show various images with different visibility index values and answering subjective visual goodness for each image, an answer that visibility is good is obtained. The range of the visibility index value may be adopted as an appropriate range. The appropriate range of visibility is set not only to exclude areas where answers such as invisible and difficult to see are obtained, but also exclude areas where answers such as too dazzling or too conspicuous are obtained. Here, as a result of the test, +0.4 to +0.7 is set as an appropriate range.

  In step S5, the display control unit F6 maintains the current display mode setting. In step S <b> 6, the hue adjustment unit F <b> 61 specifies the most hue among all the pixels constituting the background area, and adopts the hue as the representative color of the background area. The representative color of the background area may be specified based on the color distribution for each pixel constituting the background area.

  Here, the hue is obtained by dividing various colors that can be expressed by a combination of RGB into 12 patterns. The 12 hues are obtained by dividing the hue circle into 12 parts every 30 degrees so as to include red, green, and blue. FIG. 10 is a diagram showing RGB values for realizing each hue. The numbers in parentheses in FIG. 10 represent the gradation value of the red (R) component, the gradation value of the green (G) component, and the gradation value of the blue (B) component in order from the left. For a certain hue, the hue located in the opposite direction in the hue circle shown in FIG. 10 corresponds to the opposite hue (in other words, the complementary color).

  In step S7, a color on the opposite side to the representative color of the background area determined in step S6 (hereinafter, background opposite color) is specified. Next, the hue (specifically, RGB value) of the display image is adjusted so that the hue of the currently displayed HUD image 20 approaches the color opposite to the background. Then, the display image with the hue adjusted is displayed on the liquid crystal panel 122, whereby the HUD image 20 being displayed is brought close to the color opposite to the background.

  When such a display control process is repeatedly performed when the representative color of the background area and the hue of the base image are constant, as shown in FIG. 11, the hue of the HUD image 20 approaches the background opposite color. FIG. 11 shows the hue when the visibility index value is out of the appropriate range due to the representative color of the background area being cyan and the hue of the currently displayed HUD image 20 being green. It is the figure which represented notionally the operation | movement of the adjustment part F61.

  If the representative color of the background area is cyan, the background opposite color is set to red in step S7. Further, since the color of the currently generated display image is green, the parameter to be adjusted among RGB is set to R. Next, an image in which the R component gradation value of each pixel of the display image corresponding to the currently displayed HUD image 20 is increased by a certain amount (for example, 64) is newly generated as a display image. In this way, the display image in which the color of the entire image is adjusted is displayed on the liquid crystal panel 122. As a result, the hue of the HUD image 20 being displayed also approaches the opposite color.

  If such a display control process is repeatedly performed when the representative color of the background area and the hue of the base image are constant, the hue of the HUD image 20 gradually changes to the opposite color of the background area (in other words, step by step). ) Get closer. For example, when the RGB value of the display image is set from (0, 255, 0) to (64, 255, 0) but the visibility index value still does not reach the appropriate range, the R component is further strengthened. , A display image with RGB values set to (128, 255, 0) is generated and displayed on the liquid crystal panel 122. For this reason, the visibility of the HUD image 20 gradually increases, and it can be expected that the visibility index value will be within an appropriate range in the end. If the visibility index value falls within the appropriate range, the determination in step S4 is affirmative. At that time, the hue adjustment processing by the hue adjustment unit F61 is temporarily terminated. In the hue adjustment, not only the red component corresponding to the background opposite color is strengthened, but also the green component constituting the background representative color may be weakened.

  The RGB adjustment amount applied to the base image is stored in the RAM 132 as adjustment amount data. For example, when the red component is +128 with respect to the base image, the adjustment amount data is saved as (+128, 0, 0). The adjustment amount data once it is determined that the visibility is good may be maintained while the representative color of the background area and the hue of the base image are not changed. When the background representative color changes, the color of the display image may be brought closer to the background opposite color after the change from the currently displayed color so that the visibility index value falls within the appropriate range again.

  In this embodiment, a color component to be adjusted is determined among red, green, and blue based on the background representative color and the color of the displayed image, and the gradation value of the color component is adjusted by a certain amount. The mode of gradually approaching the opposite color of the background has been disclosed, but is not limited thereto. The tone of the HUD image 20 (in other words, the display image) may be corrected using the tone curve image correction technique so as to emphasize the color opposite to the background. The image may be corrected so as to emphasize the opposite color of the background, or the image may be corrected so as to weaken the background representative color.

<Effect of embodiment>
In the above configuration, a driver viewpoint image including the HUD image 20 is generated, and a visibility index value indicating the visibility of the HUD image 20 for the driver is calculated using the driver viewpoint image. Then, the hue adjustment unit F61 adjusts the hue of the HUD image 20 so that the visibility index value falls within an appropriate range set in advance. That is, the visibility of the HUD image 20 for the driver is quantitatively evaluated based on the driver viewpoint image, and the hue is adjusted based on the evaluation result. Therefore, the HUD image 20 can be displayed in a display mode that is much easier for the driver to see than the configuration in which the hue of the HUD image 20 is adjusted without using the driver viewpoint image as disclosed in Patent Document 1.

  Further, in the above-described embodiment, the appropriate range is set not only to exclude a state where it is difficult to see but also to exclude a state such as excessive glare and excessively conspicuous HUD image 20. According to such a setting, it is possible to reduce the risk of bothering the driver due to the HUD image 20 being too conspicuous. Moreover, since the HUD image 20 is too conspicuous, it is possible to reduce the risk of overlooking information on nearby moving objects and road guidance. That is, the visibility of the HUD image 20 and the visibility of other objects can be made compatible.

  Further, according to the above-described embodiment, when the visibility index value becomes a value within the proper range, the adjustment of the hue of the HUD image 20 is stopped, and the visibility index value becomes a value within the proper range. The display mode at the time is maintained. According to such a configuration, the hue of the HUD image 20 is not always the color opposite to the background. If the hue of the HUD image 20 is set to a color opposite to the background, it can be expected that the driver will be easily noticed by the driver. However, there is a concern that the display mode may be too conspicuous. In the above configuration, since the hue is adjusted based on the result of quantitatively evaluating the visibility, the possibility that the display mode of the HUD image 20 becomes a display mode that is too conspicuous to the driver can be further reduced.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the above-mentioned embodiment, The various modifications described below are also contained in the technical scope of this invention, and also in addition to the following However, various modifications can be made without departing from the scope of the invention.

  In addition, about the member which has the same function as the member described in the above-mentioned embodiment, the same code | symbol is attached | subjected and the description is abbreviate | omitted. In addition, when only a part of the configuration is mentioned, the configuration of the above-described embodiment can be applied to the other portions.

[Modification 1]
In the above-described embodiment, the aspect of evaluating the visibility of the HUD image 20 using visual saliency is disclosed, but the visibility evaluation method is not limited to this. For example, "W. Adrian." Visibility Level in Street Lighting: An Analysis of Different Experiments, "J. Illum. Engng. Soc., 22-2, pp. 49-52, 1993. The visibility index value of the HUD image 20 may be calculated by the following formula.
Vi = | Lt−Lb | / ΔLmin

  In the above formula, Vi represents a visibility index value. Lt represents the luminance (unit: cd / m 2) of the HUD image 20 as an evaluation target, Lb represents the background luminance, and Δmin represents a predetermined luminance threshold.

  The value obtained by the above expression represents how many times the luminance difference between the HUD image 20 and the background is the luminance threshold value. Naturally, the larger the value calculated by the above equation, the larger the luminance difference, indicating that the HUD image 20 is more conspicuous. The visibility evaluation by the above formula is useful when the luminance distribution and color distribution of the background are uniform to some extent.

  However, the visibility of the HUD image 20 for the driver is affected by the color and brightness of the preceding vehicle and various objects such as roads. The visual saliency is calculated using the luminance distribution in the image, the color distribution, the edge direction, and the like, as mentioned in various papers including those described above. Therefore, in view of the actual driving environment, the configuration for evaluating visibility using visual saliency as in the embodiment is more accurate than the configuration for evaluating visibility using the above formula. The visibility of the image 20 can be quantitatively evaluated. As a result, it is expected that the display mode of the HUD image 20 can be set to a display mode that is easier to see.

[Modification 2]
In the above, the configuration in which the HUD control unit 13 adjusts the display mode of the HUD image 20 by generating an image in which the hue of the base image is adjusted and using it as the liquid crystal panel 122 is disclosed, but the present invention is not limited thereto. . The adjustment of the display mode of the HUD image 20 may be realized by adjusting the gradation value of each color (specifically, RGB) on the liquid crystal panel 122. For example, the display mode of the HUD image 20 may be adjusted by changing the setting of the color temperature on the liquid crystal panel 122. Also in that case, control is performed so that the appearance of the HUD image 20 approaches the opposite color of the background.

[Modification 3]
In the above, the configuration in which the display mode of the HUD image 20 is adjusted by adjusting the RGB value (in other words, the brightness of each color) for each pixel of the display image has been disclosed, but the present invention is not limited thereto. The display mode of the HUD image 20 may be adjusted by adjusting the intensity of the output light from the backlight 121 (that is, the luminance of the backlight 121). In that case, the luminance of the backlight 121 is controlled so that the visibility index value approaches the appropriate range.

  Of course, the display mode of the HUD image 20 may be adjusted by using both the adjustment of the hue in the image data and the adjustment of the luminance in the backlight 121. Further, only when the visibility index value cannot be within the proper range even after adjusting the luminance of the backlight 121, the display color of the HUD image 20 is adjusted as disclosed in the embodiment and the second modification. You may employ | adopt the control system which implements.

[Modification 4]
In the above-described embodiment, a mode in which the display 12 is a liquid crystal projector is disclosed, but the present invention is not limited to this. The display device 12 may be a projector (so-called laser projector) that displays an image by scanning laser light in a two-dimensional direction using a MEMS (Micro Electro Mechanical Systems) scanner. In that case, the adjustment of the display mode of the HUD image 20 may be realized by adjusting the luminance of each color (that is, RGB) in the laser light source.

[Modification 5]
In the above-described embodiment, the configuration including the stereo camera unit 2 as a camera for imaging the front of the vehicle is disclosed, but the present invention is not limited thereto. As shown in FIG. 12, the HUD system 100 may be configured using a monocular camera 4 and a laser radar 5 instead of the stereo camera unit 2. Hereinafter, the configuration and operation of the HUD system 100 according to the fifth modification will be described using the above configuration as the fifth modification.

  The monocular camera 4 is a full-color camera, and for example, a CCD camera can be adopted. The monocular camera 4 is disposed at the driver seat side end of the upper surface portion of the instrument panel 40, similarly to the stereo camera unit 2 in the above-described embodiment. The image data captured by the monocular camera 4 is sequentially provided to the HUD control unit 13.

  The laser radar 5 is a device that detects a distance and direction (that is, a relative position) from an object existing in front of the vehicle by irradiating laser light toward a predetermined range in front of the vehicle. Further, the laser radar 5 identifies the detected object from the shape of the detected object. The laser radar 5 sequentially provides the HUD control unit 13 with data representing the relative position and contour shape of an object existing in front of the vehicle in a three-dimensional coordinate system (for example, every 100 milliseconds).

  The driver viewpoint image generation unit F4 in the modification 5 identifies the correspondence between the object included in the image captured by the monocular camera 4 and the object detected by the laser radar 5 using a known sensor fusion technique. . Thereby, the coordinates in the three-dimensional space of the object imaged by the monocular camera 4 are specified. Then, an image (that is, a driver viewpoint image) is generated on the assumption that a monocular color camera is present at the position of the driver's eye, using the coordinates of these objects as a base point.

  That is, even when the monocular camera 4 is provided instead of the stereo camera unit 2, the position of the object being imaged by the monocular camera 4 in the three-dimensional space is specified from the detection result of the laser radar 5. A driver viewpoint image can be generated. Since the operation after the generation of the driver viewpoint image can be applied to the above-described embodiment and various modifications, a description thereof will be omitted.

  In place of the laser radar 5, a millimeter wave radar or the like may be used. Any device that can acquire information indicating the position of the object captured by the monocular camera 4 and the contour shape of the object may be used.

[Modification 6]
In the above-described embodiment, the aspect in which the stereo camera unit 2 is disposed on the upper surface portion of the instrument panel 40 is disclosed, but the present invention is not limited thereto. As shown in FIG. 13, the stereo camera unit 2 may be arranged at the upper end of a rearview mirror or the windshield 10. Hereinafter, the configuration and operation of the HUD system 100 (particularly, the HUD control unit 13) in the modification 6 will be described with such a configuration as the modification 6.

  As shown in FIG. 14, the HUD control unit 13 in Modification 6 includes a HUD display coordinate storage unit M3 in addition to the various configurations disclosed in the embodiment. The HUD display coordinate storage unit M3 is a storage medium in which coordinates indicating the position in the three-dimensional space where the HUD image 20 appears to the driver (hereinafter referred to as HUD display coordinates) are registered. The HUD display coordinate storage unit M3 may be realized by using a part of the storage area provided in the flash memory 134, for example. The design value etc. should just register the HUD display coordinate.

  In addition, the driver viewpoint image generation unit F4 according to the modified example 6 uses a method similar to that of the above-described embodiment, based on the captured image of the stereo camera unit 2, a driver viewpoint image that does not include the HUD image 20 (hereinafter referred to as HUD non-display). Display image). Next, the currently output display image is acquired from the display control unit F6. Then, in the HUD non-display image, an image synthesized so that the display image exists at the HUD display coordinates is generated as a driver viewpoint image.

  Even with such a configuration, the same effects as those of the above-described embodiment and the like can be obtained. The configuration of Modification 6 can be implemented in combination with the configuration using the monocular camera 4 disclosed in Modification 5 as appropriate.

[Modification 7]
A camera that images the front of the vehicle may be attached to the driver's face. For example, as shown in FIGS. 15 and 16, the HUD system 100 uses a spectacle-type device 6 (hereinafter referred to as a spectacle-type device) 6 that is equipped with a camera 61 and a short-range communication unit 62 and that is worn on the face of the driver. It may be realized. Hereinafter, the configuration and operation of the HUD system 100 (particularly, the HUD control unit 13) in the modified example 7 will be described using such a configuration as the modified example 7.

  The HUD system 100 according to the modified example 7 includes an in-vehicle system including a HUD unit 1, a vehicle speed sensor 3, and a short-range communication unit 7 that are mounted on a vehicle, and a glasses-type device 6 that is mounted by a driver. The short-range communication unit 7 included in the in-vehicle system is a communication module for performing communication (hereinafter, short-range communication) compliant with a predetermined short-range wireless communication standard having a communicable range of about several tens of meters at the maximum. . As the short-range wireless communication standard, for example, Bluetooth Low Energy (Bluetooth is a registered trademark), Wi-Fi (registered trademark), or the like can be adopted. The HUD control unit 13 performs wireless communication with the glasses-type device 6 via the short-range communication unit 7.

  The eyeglass-type device 6 includes a camera 61 and a short-range communication unit 62. The camera 61 is a color camera that captures an image of a direction in which the driver's face is directed in a state where the camera 61 is attached to the driver. The near field communication unit 62 is a communication module for performing the above-described near field communication. When the communication connection with the short-range communication unit 7 is established, the short-range communication unit 62 sequentially transmits an image captured by the camera 61 to the HUD control unit 13 by short-range communication. The eyeglass device 6 corresponds to the wearable device described in the claims.

  The HUD control unit 13 in the modified example 7 sequentially acquires image data transmitted from the eyeglass-type device 6, and uses this as a driver viewpoint image to calculate a visibility index value. Even with such a configuration, the same effects as those of the above-described embodiment can be obtained.

  In the configuration of the modified example 7, since an image captured by the camera 61 can function as a driver viewpoint image, the camera image acquisition unit F3 corresponds to the driver viewpoint image acquisition unit described in the claims. Therefore, as a function of the HUD control unit 13, the driver viewpoint image generation unit F4 and the camera parameter storage unit M1 illustrated in FIG. 4 can be omitted. However, since the driver does not always face the front of the vehicle, the image captured by the camera 61 does not always function as a driver viewpoint image. The camera image acquisition unit F3 in the modified example 7 analyzes an image acquired from the glasses-type device 6 and adopts only an image in which the HUD image 20 is captured as a driver viewpoint image.

  In addition, although the structure which utilizes the camera 61 with which the spectacles type device 6 is provided as a camera which images the vehicle front above was disclosed, it is not restricted to this. The camera may be attached to the driver's forehead with a band or the like. Further, a wearable camera of a system other than the glasses type, such as a camera integrated with a headset worn on the head or a wearable camera worn on one ear, may be used.

100 HUD system (head-up display system), 1 HUD unit, 2 stereo camera unit, 4 monocular camera, 6 glasses-type device (wearable device), 12 display, 13 HUD control unit, 20 HUD image, 40 instrument panel 61 camera, F1 vehicle speed acquisition unit, F2 base image generation unit, F3 camera image acquisition unit, F4 driver viewpoint image generation unit, F5 visibility evaluation unit, F6 display control unit, F61 hue adjustment unit (display mode adjustment unit), M1 camera parameter storage unit, M2 appropriate range storage unit, M3 HUD display coordinate storage unit, S32 target region extraction unit, S33 saliency calculation unit, S34 HUD image saliency calculation unit, S35 background region saliency calculation unit

Claims (7)

A display (12) for irradiating display light for displaying an image as a virtual image on a predetermined irradiation area of a windshield disposed on the front side of the vehicle;
A display mode adjustment unit (F61) that adjusts a display mode of a HUD image that is an image as the virtual image displayed when the display unit irradiates the display area with the display light;
Based on image data captured by a camera that images the front of the vehicle, the driver is an image when the vehicle is viewed from the viewpoint of a driver who is a passenger seated in the driver's seat and includes the HUD image. A driver viewpoint image acquisition unit (F4, F3) for acquiring viewpoint images;
A visibility evaluation unit (F5) that sequentially calculates a visibility index value indicating the visibility of the HUD image in the driver viewpoint image by analyzing the driver viewpoint image acquired by the driver viewpoint image acquisition unit;
An appropriate range storage unit (M2) in which data indicating an appropriate range of the visibility index value calculated by the visibility evaluation unit is registered;
The display mode adjustment unit adjusts the display mode of the HUD image so that the visibility index value calculated by the visibility evaluation unit is a value within the appropriate range. system. In claim 1,
The head-up display system, wherein the display mode adjusting unit changes the visibility index value within the appropriate range by changing at least one of luminance and display color of the HUD image. In claim 1 or 2,
The visibility evaluation unit
A target area extraction unit (S32) that extracts a predetermined range including the HUD image in the driver viewpoint image as a calculation target area;
A saliency calculating unit (S33) that calculates a saliency indicating a degree of visual saliency in the calculation target area for each pixel constituting the calculation target area;
A HUD image saliency that calculates a HUD image saliency indicating the visual saliency of the HUD image based on the saliency of each pixel constituting the HUD image region that is a portion corresponding to the HUD image in the calculation target region. Degree calculation unit (S34);
Background area saliency that calculates a background area saliency that indicates the visual saliency of the background area based on the saliency for each pixel that forms a background area that is a part other than the HUD image area in the calculation target area A calculation unit (S35),
A value obtained by subtracting the background region saliency calculated by the background region saliency calculation unit from the HUD image saliency calculated by the HUD image saliency calculation unit is adopted as the visibility index value. Head-up display system. In claim 3,
A vehicle speed acquisition unit (F1) that sequentially acquires information indicating the travel speed from a vehicle speed sensor that detects the travel speed of the vehicle;
The head-up display system, wherein the visibility evaluation unit sets the calculation target region to be smaller as the current traveling speed acquired by the vehicle speed acquisition unit is higher. In claim 3 or 4,
The display mode adjustment unit
Based on the color distribution of the background area, which is determined by setting all pixels constituting the background area as a population, the most common hue among the pixels constituting the background area is used as the background representative color representing the hue of the background area. Identify,
Set the color that is the hue located opposite to the background representative color and the hue ring to the background opposite color,
A head-up display system, wherein the hue of the HUD image is gradually brought close to the opposite color of the background until the visibility index value becomes a value within the appropriate range. In any one of Claim 1 to 5,
A device that is mounted on the driver's face or head, a function that performs wireless communication in accordance with a predetermined short-range wireless communication standard, and a camera that captures the direction in which the driver's face is facing A wearable device (6) comprising a wearable camera;
A short-range communication unit (7) that is a communication module for performing wireless communication in conformity with the wearable device and the short-range wireless communication standard,
The wearable device transmits an image captured by the wearable camera to the short-range communication unit when a communication connection with the short-range communication unit is established,
The driver viewpoint image acquisition unit acquires an image captured by the wearable camera as the driver viewpoint image via the short-range communication unit. In any one of Claim 1 to 5,
The camera is a stereo camera arranged to include the irradiation area in an imaging range on an upper surface of an instrument panel of the vehicle,
A camera parameter storage unit (M1) for storing data indicating the position and mounting posture of the stereo camera in the vehicle as camera parameters;
In the camera parameter storage unit, viewpoint coordinate data indicating a position where the driver's eyes are assumed to be present in the passenger compartment is registered,
The driver viewpoint image acquisition unit
A head-up display system, wherein the driver viewpoint image is generated based on a captured image of the stereo camera, the camera parameter stored in the camera parameter storage unit, and the viewpoint coordinate data.