JP2024132300A - Display control device and display control program - Google Patents

Abstract

[Problem] To provide a display control device and a display control program that suppress misalignment of a superimposed display relative to a target. [Solution] The display control device 20 includes an acquisition unit 30 that acquires a captured image Im captured by an image capturing device 13, and an HCU 50 that displays a display image Di superimposed on a target T based on a correction region Rr calculated using a target region Rt that is a region in the captured image Im that includes the position of the target T shown in the captured image Im, and a prediction region Rp that is a region in the captured image Im that includes the predicted position of the target T shown in the captured image Im. [Selected Figure] Figure 2

Description

This disclosure relates to a display control device and a display control program.

As described in Patent Document 1, a head-up display device is known that projects an image onto the windshield or combiner of a vehicle, thereby displaying a virtual image to the driver by superimposing it on the scenery in front of the vehicle.

The head-up display device described in Patent Document 1 includes an image input unit, an information acquisition unit, an image generation unit, a conversion unit, a display control unit, and a display unit. The image input unit inputs an image captured by a camera and extracts a predetermined target from the image. The information acquisition unit acquires target information including the position of the target in the image and distance information including the distance to the target in space. The image generation unit generates a virtual image to be superimposed on the target. The conversion unit uses the acquired information to perform a conversion process to correct the position of the display area, which is the range in which an image can be displayed in the visual recognition area of the windshield or combiner, and the display position of the image in the display area. The display control unit uses the corrected data to control the superimposition of the image on the visual recognition area. The display unit superimposes the image on the visual recognition area according to the above control. The conversion unit also performs a conversion process to match the display position of the image in the display area to the position of the target when the target is viewed through the visual recognition area from the driver's viewpoint position in the basic setting.

International Publication No. 2018/193708

In the head-up display device described in Patent Document 1, even if the position of the display area and the display position of the image within the display area are corrected, if the recognition position of the target in the image shifts, the position of the superimposed display relative to the target shifts. This causes the driver to feel uncomfortable.

The present disclosure aims to provide a display control device and a display control program that suppresses misalignment of an overlay display relative to a target.

The invention described in claim 1 is a display control device that includes an acquisition unit (30) that acquires an image (Im) captured by an image capture device (13), and a display control unit (50) that superimposes a display image (Di) on a target based on a correction area (Rr) that is calculated using a target area (Rt) that is an area in the captured image that includes the position of a target (T) that appears in the captured image, and a prediction area (Rp) that is an area in the captured image that includes the predicted position of the target that appears in the captured image.

The invention described in claim 23 is a display control program that causes the display control device to function as an acquisition unit (30) that acquires an image (Im) captured by an image capture device (13), and a display control unit (50) that superimposes a display image (Di) on a target based on a correction area (Rr) calculated using a target area (Rt) that is an area in the captured image that includes the position of a target (T) that appears in the captured image, and a prediction area (Rp) that is an area in the captured image that includes a predicted position of the target that appears in the captured image.

As a result, even if the recognized position of the target is shifted, the display image is superimposed on the target using the corrected area calculated and corrected based on the target area and the predicted area. Therefore, the display image is superimposed on the target while the shift in the recognized position of the target is suppressed. Therefore, the position shift of the superimposed display on the target is suppressed.

The reference symbols in parentheses attached to each component indicate an example of the correspondence between the component and the specific components described in the embodiments described below.

1 is a configuration diagram of a vehicle in which a display control device according to a first embodiment is used. FIG. 2 is a diagram showing the configuration of a display system for a vehicle. FIG. 2 is a block diagram of a recognition unit of the display system. FIG. 13 is a diagram showing a display image superimposed on a target shown in a captured image. FIG. 2 is a block diagram of an HCU of the display system. 5 is a diagram showing calculation of a target area by an image processing unit of the recognition unit. FIG. 6 is a diagram showing calculation of a predicted area by a prediction unit of the recognition unit; 4 is a diagram for explaining tracking of a target by a tracking unit of the recognition unit; FIG. 13 is a diagram showing calculation of the maximum length and overlap length by the estimation unit of the HCU. FIG. 13 is a diagram showing the relationship between an overlap rate and a first reliability. FIG. 11 is a diagram showing the relationship between a recognition distance and a second reliability. FIG. 11 is a diagram showing the relationship between the DNN recognition score value and the third reliability level. FIG. 13 is a diagram showing the relationship between the tracking state and the fourth reliability level. FIG. 13 is a graph showing the relationship between the cumulative number of tracking attempts and the fifth reliability level. 11 is a diagram showing calculation of a correction area by a position correction unit of the HCU; FIG. 11 is a configuration diagram of a display system including a display control device according to a second embodiment. 13 is a diagram showing a case where a target region and a predicted region are unlikely to overlap each other; 6 is a diagram showing calculation of a time change in position in a target region by an estimation unit; FIG. 13 is a configuration diagram of a display system including a display control device according to a third embodiment. FIG. 2 is a block diagram of an HCU of the display control device. FIG. 13 is a diagram showing when the reliability of a target region becomes low. 6A and 6B are diagrams showing calculation of a correction area by a position correction unit.

The following embodiments will be described with reference to the drawings. Note that in the following embodiments, parts that are identical or equivalent to each other will be given the same reference numerals and their description will be omitted.

First Embodiment
The display control device of the first embodiment controls a HUD mounted on a vehicle to superimpose and display an image on the foreground of the vehicle. First, a vehicle in which this display control device is used will be described. HUD is an abbreviation for Head-Up Display.

As shown in Figures 1 and 2, the vehicle 1 has a windshield 4, an instrument panel 5, and a display system 10.

As shown in FIG. 1, the windshield 4 is disposed at the front of the vehicle 1. The windshield 4 also ensures forward visibility for the driver of the vehicle 1. The instrument panel 5 houses the HUD 15, which will be described later.

As shown in FIG. 2, the display system 10 has an image capturing device 13, a HUD 15, and a display control device 20.

The image capturing device 13 is a camera having an image sensor such as a CCD or CMOS. As shown in FIG. 1, the image capturing device 13 captures an image of the area ahead of the vehicle 1 at a frame rate of, for example, 60 fps. As shown in FIG. 2, the image capturing device 13 outputs the captured image Im to the display control device 20, which will be described later.

The HUD 15 forms a display image Di based on image data from the HCU of the display control device 20 described later. Specifically, as shown in FIG. 1, the HUD 15 includes a projector 151 such as a liquid crystal type or a scanning type, and an optical system 152 such as a concave mirror. Light from the projector 151 is reflected by the optical system 152. The light reflected by the optical system 152 is projected onto the projection surface of the windshield 4. The projected light is reflected by the windshield 4, and the display image Di from the projector 151 is perceived by the driver of the vehicle 1. The scenery ahead of the vehicle 1 is also perceived by the driver of the vehicle 1. As a result, a virtual image 153 of the display image Di formed in front of the windshield 4 and a part of the scenery ahead are superimposed and displayed. Therefore, the driver of the vehicle 1 can see the AR display. Note that AR is an abbreviation for Augmented Reality.

Returning to FIG. 2, the display control device 20 is mainly composed of a microcomputer and includes a CPU, ROM, RAM, I/O, and bus lines connecting these components. Specifically, the display control device 20 includes a recognition unit 30 and an HCU 50. HCU is an abbreviation for HMI Control Unit. HMI is also an abbreviation for Human Machine Interface.

The recognition unit 30 executes the program of the recognition unit 30 to process the captured image Im from the image capturing device 13. Specifically, as shown in FIG. 3, the recognition unit 30 includes an image processing unit 32, a tracking unit 34, and a prediction unit 36 as functional blocks.

The image processing unit 32 performs image recognition and machine learning such as deep learning to calculate first intermediate information obtained from the captured image Im. The image processing unit 32 also outputs this calculated first intermediate information to the tracking unit 34 and HCU 50 described below. Details of the first intermediate information will be described later.

The tracking unit 34 tracks the target T shown in the captured image Im as shown in FIG. 4, based on the first intermediate information calculated by the image processing unit 32 and information related to the predicted position of the target T calculated by the prediction unit 36 described below. As a result, the tracking unit 34 calculates information related to the tracking of the target T shown in the captured image Im as second intermediate information. Note that, in this case, the target T is a three-dimensional object such as a four-wheeled motor vehicle, a two-wheeled motor vehicle, a bicycle, or a person. Details of the second intermediate information will be described later.

The prediction unit 36 uses the first intermediate information calculated by the image processing unit 32 to calculate information about the predicted position of the target T that will appear in the captured image Im of the next frame, with respect to the target T being tracked by the tracking unit 34. Furthermore, the prediction unit 36 outputs the information about the predicted position of the target T to the tracking unit 34 and the HCU 50 described below.

The HCU 50 executes its own program to generate image data for superimposing a display image Di as shown in FIG. 4 on a target T shown in a captured image Im, based on information from the recognition unit 30. The HCU 50 also outputs the generated image data to the HCU 50. Specifically, the HCU 50 includes, as functional blocks, an estimation unit 52, a reliability calculation unit 54, an integration unit 56, a display brightness calculation unit 58, a position correction unit 60, and a filter processing unit 62, as shown in FIG. 5.

The estimation unit 52 acquires information on the captured image Im, the first intermediate information calculated by the image processing unit 32, and the predicted position of the target T calculated by the prediction unit 36. The estimation unit 52 also calculates third intermediate information based on this acquired information. The estimation unit 52 also outputs this calculated third intermediate information together with a portion of the first intermediate information and the second intermediate information to a reliability calculation unit 54 described below. The estimation unit 52 also outputs the acquired captured image Im, a portion of the first intermediate information, and information on the predicted position of the target T calculated by the prediction unit 36 to a position correction unit 60 described below. Details of the third intermediate information will be described later.

The reliability calculation unit 54 calculates the reliability of each of the part of the first intermediate information, the second intermediate information, and the third intermediate information from the estimation unit 52. In this way, the reliability calculation unit 54 evaluates the reliability of the part of the first intermediate information, the second intermediate information, and the third intermediate information from the estimation unit 52. The reliability calculation unit 54 also outputs these calculated reliability to the integration unit 56 described below. Note that the reliability is, for example, a likelihood, which is a degree indicating the likelihood of information. Therefore, a high likelihood corresponds to a high reliability. Also, a low likelihood corresponds to a low reliability.

The integration unit 56 calculates the integrated reliability Pi by integrating the respective reliabilities calculated by the reliability calculation unit 54 using Haze estimation or the like. In this way, the integration unit 56 comprehensively evaluates the reliability of the information related to the predicted position of the target T calculated by the prediction unit 36. The integration unit 56 also outputs the calculated integrated reliability Pi to the display brightness calculation unit 58 and the position correction unit 60, which will be described later.

The display brightness calculation unit 58 calculates the display brightness Ld based on the integrated reliability Pi calculated by the integration unit 56. Furthermore, the display brightness calculation unit 58 outputs the calculated display brightness Ld to the filter processing unit 62 described below. Note that the display brightness Ld is the brightness of the display image Di.

The position correction unit 60 corrects the position of the display image Di based on the captured image Im from the estimation unit 52, part of the first intermediate information, the target T predicted by the prediction unit 36, and the integration reliability Pi calculated by the integration unit 56. The position correction unit 60 also outputs the corrected position information of the display image Di to the filter processing unit 62 described below.

The filter processing unit 62 performs low-pass filtering such as moving average on each of the display luminance Ld calculated by the display luminance calculation unit 58 and the position information of the corrected display image Di calculated by the position correction unit 60. This removes noise contained in the display luminance Ld and the position information of the corrected display image Di. The filter processing unit 62 also generates image data for displaying the display image Di at the display luminance Ld and the position of the corrected display image Di from which the noise has been removed. Furthermore, the filter processing unit 62 outputs the generated image data to the HUD 15. Therefore, the HUD 15 displays the display image Di as shown in FIG. 4 at the display luminance Ld and the position of the corrected display image Di calculated in the HCU 50. This allows the driver to view the display image Di whose position has been corrected and whose display luminance Ld has been calculated in the HCU 50.

The vehicle 1 is configured as described above. Next, the details of the processing performed by the recognition unit 30 through program execution will be described with reference to Figures 3, 6, 7, and 8.

First, as shown in FIG. 3, the image processing unit 32 acquires the captured image Im from the image capturing device 13. The image processing unit 32 detects the target T from the acquired captured image Im by using image recognition or the like, and calculates the target region Rt as one of the first intermediate information. For example, as shown in FIG. 6, the image processing unit 32 detects the target T in the captured image Im(t) of the tth frame, and calculates the target region Rt(t) of the tth frame. The target region Rt is an area within the captured image Im that includes the position of the target T shown in the captured image Im. Here, the target region Rt is rectangular, but is not limited to this. The target region Rt may be polygonal or circular.

The image processing unit 32 also uses image recognition or the like to calculate the recognition distance Dr as one piece of the first intermediate information. Note that the recognition distance Dr is, for example, the distance from the front end of the vehicle 1 to the rear end of the target T shown in the captured image Im, and corresponds to the distance from the vehicle 1 to the target T.

Furthermore, the image processing unit 32 uses image recognition and deep learning to identify the type of the target T and calculates a DNN recognition score value Sc_DNN as one of the first intermediate information. The DNN recognition score value Sc_DNN is a value related to the identification of the type of the target T calculated using deep learning. As the DNN recognition score value Sc_DNN increases, the accuracy of the identified type of the target T increases. For example, assume that the type of the target T shown in the captured image Im is a four-wheeled vehicle. In this case, the image processing unit 32 determines that the type of the target T shown in the captured image Im is a four-wheeled vehicle. At this time, the image processing unit 32 also calculates the DNN recognition score value Sc_DNN. As the calculated DNN recognition score value Sc_DNN increases, the accuracy of the actual type of the target T being a four-wheeled vehicle increases. Note that DNN is an abbreviation for Deep Neural Network.

Returning to FIG. 3, the image processing unit 32 outputs the calculated target area Rt together with the acquired captured image Im to the tracking unit 34 described below. Furthermore, the image processing unit 32 outputs the calculated recognition distance Dr and DNN recognition score value Sc_DNN to the HCU 50 described below, in addition to the acquired captured image Im and the calculated target area Rt.

Next, the tracking unit 34 tracks the target T shown in the captured image Im based on the target area Rt calculated by the image processing unit 32 and information on the predicted position of the target T calculated by the prediction unit 36 described below. When performing this tracking, information on the predicted position of the target T calculated by the prediction unit 36 described below is used, so next, the processing of the prediction unit 36 will be described.

The prediction unit 36 acquires the captured image Im and the target area Rt via the tracking unit 34. The prediction unit 36 also calculates the predicted position of the target T in the captured image Im of the next frame by using the acquired target area Rt and a Kalman filter or the like. Furthermore, the prediction unit 36 calculates the predicted area Rp from the calculated predicted position. For example, as shown in FIG. 7, the prediction unit 36 calculates a predicted area Rp (t+1, t) for the target T in the captured image Im (t) of the t+1th frame from the target T in the captured image Im (t) of the tth frame. Note that the predicted area Rp is an area in the captured image Im that includes the predicted position of the target T in the captured image Im. Furthermore, the predicted area Rp has a shape corresponding to the target area Rt. For this reason, the shape of the predicted area Rp is a rectangle. Furthermore, the predicted area Rp is calculated using a Kalman filter, but is not limited to this calculation method. For example, the speed of the target T is calculated from the target T shown in the captured image Im(t-1) of the t-1th frame and the target T shown in the captured image Im(t) of the tth frame. The predicted region Rp(t+1, t) may then be calculated from the calculated speed of the target T and the position of the target T shown in the captured image Im(t) of the tth frame.

Also, returning to FIG. 3, the prediction unit 36 outputs this calculated prediction region Rp to the tracking unit 34 and the HCU 50 described below.

Then, the tracking unit 34 tracks the target T shown in the captured image Im based on the target area Rt calculated by the image processing unit 32 and the predicted area Rp calculated by the prediction unit 36.

For example, the tracking unit 34 calculates the inter-region distance Dtp as shown in FIG. 8. The inter-region distance Dtp(t+1) is the distance between corresponding positions from the target region Rt(t+1) in the captured image Im(t+1) of the t+1th frame to the predicted region Rp(t+1, t) of the tth frame. The tracking unit 34 also calculates the average brightness in the target region Rt(t+1) in the captured image Im(t+1) of the t+1th frame and the average brightness in the predicted region Rp(t+1, t) of the tth frame. Furthermore, the tracking unit 34 calculates the absolute value |ΔL| of the difference between these average brightnesses.

Then, the inter-region distance Dtp(t+1) is equal to or less than the inter-region distance threshold Dtp_th, and the absolute value of the average brightness difference |ΔL| is equal to or less than the brightness difference threshold ΔL_th. At this time, the tracking unit 34 determines that the target T shown in the captured image Im(t+1) of the t+1th frame coincides with the target T shown in the captured image Im(t) of the tth frame. Also, at this time, the tracking unit 34 determines that the target T shown in the captured image Im is being tracked. As a result, the tracking unit 34 calculates the tracking state St of the target T shown in the captured image Im as one of the second intermediate information. Furthermore, at this time, the tracking unit 34 adds 1 to the number of times that the target T has been tracked so far, thereby calculating the cumulative tracking count Nt_sum corresponding to the number of times that the target T in the target region Rt coincides with the target T in the prediction region Rp as one of the second intermediate information. The region distance threshold Dtp_th and brightness difference threshold ΔL_th are set through experiments, simulations, etc. so that it is determined that the target T shown in the captured image Im is being tracked.

Also, when the inter-region distance Dtp(t+1) is greater than the inter-region distance threshold Dtp_th, the tracking unit 34 determines that the target T shown in the captured image Im(t+1) of the t+1th frame does not match the target T shown in the captured image Im(t) of the tth frame. At this time, the tracking unit 34 determines that the target T shown in the captured image Im is in a ghost state, that is, is not tracked. Furthermore, when the absolute value |ΔL| of the average brightness difference is greater than the brightness difference threshold ΔL_th, the tracking unit 34 determines that the target T shown in the captured image Im(t+1) of the t+1th frame does not match the target T shown in the captured image Im(t). At this time, the tracking unit 34 determines that the target T shown in the captured image Im is in a ghost state. From these, the tracking unit 34 calculates the tracking state St of the target T shown in the captured image Im as one piece of second intermediate information.

Returning to FIG. 3, the tracking unit 34 calculates the tracking state St and the cumulative tracking count Nt_sum for the HCU 50, which will be described later. Furthermore, the tracking unit 34 outputs the tracking state St to the prediction unit 36 together with the captured image Im and the target region Rt.

As described above, the recognition unit 30 performs processing. Next, the details of the processing performed by the HCU 50 executing the program will be described with reference to Figures 4 to 15.

First, as shown in FIG. 5, the estimation unit 52 acquires the captured image Im, the target area Rt, the predicted area Rp, the recognition distance Dr, the DNN recognition score value Sc_DNN, the tracking state St, and the cumulative tracking count Nt_sum from the recognition unit 30.

The estimation unit 52 also calculates the maximum length Dmax and overlap length Dd from the acquired target region Rt and predicted region Rp. The maximum length Dmax is the maximum length from the end of the target region Rt to the end of the predicted region Rp in one direction. Furthermore, the overlap length Dd is the length of the portion where the target region Rt and the predicted region Rp overlap in that one direction.

For example, as shown in FIG. 9, the estimation unit 52 calculates a maximum length Dmax with one direction being the vertical direction of the captured image Im from the target region Rt(t+1) of the captured image Im(t+1) in the t+1th frame and the predicted region Rp(t+1, t) in the tth frame. The estimation unit 52 also calculates an overlap length Dd with one direction being the vertical direction of the captured image Im. Note that, although the one direction here is the vertical direction of the captured image Im, this is not limited to this. For example, the one direction may be the horizontal direction of the captured image Im.

The estimation unit 52 also divides these calculated overlap lengths Dd by the maximum length Dmax to calculate the overlap rate Ov as third intermediate information. Returning to FIG. 5, the estimation unit 52 outputs this calculated overlap rate Ov to the reliability calculation unit 54 described below, together with the recognition distance Dr, the DNN recognition score value Sc_DNN, the tracking state St, and the cumulative tracking count Nt_sum. The estimation unit 52 also outputs the captured image Im, the target area Rt, and the predicted area Rp to the position correction unit 60 described below.

Then, the reliability calculation unit 54 acquires the overlap rate Ov, the recognition distance Dr, the DNN recognition score value Sc_DNN, the tracking state St, and the cumulative tracking count Nt_sum from the estimation unit 52. The reliability calculation unit 54 also calculates the reliability for each of the acquired overlap rate Ov, the recognition distance Dr, the DNN recognition score value Sc_DNN, the tracking state St, and the cumulative tracking count Nt_sum. In this way, the reliability calculation unit 54 evaluates the reliability of the prediction region Rp.

Specifically, the reliability calculation unit 54 calculates the first reliability P1 by using the acquired overlap rate Ov and the map.

Here, when the overlap rate Ov is relatively small, the size of the overlapping portion between the target region Rt and the predicted region Rp is small, so the deviation between the target region Rt and the predicted region Rp is large, and the likelihood of the predicted region Rp is low. Therefore, at this time, the reliability of the predicted region Rp is low. Also, when the overlap rate Ov is relatively large, the size of the overlapping portion between the target region Rt and the predicted region Rp is large, so the deviation between the target region Rt and the predicted region Rp is small, and the likelihood of the predicted region Rp is high. Therefore, at this time, the reliability of the predicted region Rp is high. Therefore, in the map for calculating the first reliability P1, for example, as shown in FIG. 10, when the overlap rate Ov is relatively small, the first reliability P1 is set to a constant value at a relatively low value. Also, as the overlap rate Ov increases from a relatively small value, the first reliability P1 becomes higher. Furthermore, when the overlap rate Ov is relatively large, the first reliability P1 is set to a constant, relatively high value.

The reliability calculation unit 54 also calculates the second reliability P2 by using the acquired recognition distance Dr and the map.

Here, when the recognition distance Dr is relatively short, the resolution of the target T shown in the captured image Im is high, so it is easy to predict the position of the target T, and the likelihood of the predicted region Rp is high. Therefore, at this time, the reliability of the predicted region Rp is high. Also, when the recognition distance Dr is relatively long, the resolution of the target T shown in the captured image Im is low, so it is difficult to predict the position of the target T, and the likelihood of the predicted region Rp is low. Therefore, at this time, the reliability of the predicted region Rp is low. Therefore, in the map for calculating the second reliability P2, as shown in FIG. 11, for example, when the recognition distance Dr is relatively small, the second reliability P2 is set to a constant value at a relatively high value. Also, as the recognition distance Dr increases from a relatively short value, the second reliability P2 becomes lower. Furthermore, when the recognition distance Dr is relatively long, the second reliability P2 is set to a constant value at a relatively low value.

The reliability calculation unit 54 also calculates the third reliability P3 by using the acquired DNN recognition score value Sc_DNN and the map.

Here, when the DNN recognition score value Sc_DNN is relatively small, the accuracy of the type of target T shown in the captured image Im is low, and therefore the likelihood of the predicted area Rp is low. Therefore, at this time, the reliability of the predicted area Rp is low. Also, when the DNN recognition score value Sc_DNN is relatively large, the accuracy of the type of target T shown in the captured image Im is high, and therefore the likelihood of the predicted area Rp is high. Therefore, at this time, the reliability of the predicted area Rp is high. Therefore, in the map for calculating the third reliability P3, for example, as shown in FIG. 12, when the DNN recognition score value Sc_DNN is relatively small, the third reliability P3 is set to a constant value at a relatively low value. Also, as the DNN recognition score value Sc_DNN increases from a relatively small value, the third reliability P3 increases. Furthermore, when the DNN recognition score value Sc_DNN is relatively large, the third reliability P3 is set to a constant, relatively high value.

The reliability calculation unit 54 also calculates the fourth reliability P4 by using the acquired tracking state St and the map.

Here, when the tracking state St is a ghost state, that is, when the target T is not being tracked, it is difficult to predict the position of the target T, so the likelihood of the prediction region Rp is low. Therefore, at this time, the reliability of the prediction region Rp is low. Also, when the tracking state St is tracking, it is easy to predict the position of the target T, so the likelihood of the prediction region Rp is high. Therefore, at this time, the reliability of the prediction region Rp is high. Therefore, in the map for calculating the fourth reliability P4, for example, as shown in FIG. 13, when the tracking state St is a ghost state, the fourth reliability P4 is set to a relatively low value. Also, when the tracking state St is tracking, the fourth reliability P4 is set to a higher value compared to when the tracking state St is a ghost state.

Furthermore, the reliability calculation unit 54 calculates the fifth reliability P5 by using the acquired cumulative tracking count Nt_sum and the map.

Here, when the cumulative tracking count Nt_sum is relatively small, the accuracy of the target T tracked by the tracking unit 34 being the same is low, so the likelihood of the predicted region Rp is low. Therefore, at this time, the reliability of the predicted region Rp is low. Also, when the cumulative tracking count Nt_sum is relatively large, the accuracy of the target T tracked by the tracking unit 34 being the same is high, so the likelihood of the predicted region Rp is high. Therefore, at this time, the reliability of the predicted region Rp is high. Therefore, in the map for calculating the fifth reliability P5, for example, as shown in FIG. 14, when the cumulative tracking count Nt_sum is relatively small, the fifth reliability P5 is set to a relatively low value. Also, when the cumulative tracking count Nt_sum increases from a relatively small value, the fifth reliability P5 increases rapidly, and then as the cumulative tracking count Nt_sum increases, the fifth reliability P5 increases. Furthermore, when the cumulative number of tracking attempts Nt_sum is relatively large, the fifth reliability P5 is set to a constant, relatively high value.

Returning to FIG. 5, the reliability calculation unit 54 then outputs the calculated first reliability P1, second reliability P2, third reliability P3, fourth reliability P4, and fifth reliability P5 to the integration unit 56, which will be described later.

Then, the integration unit 56 acquires the first reliability P1, the second reliability P2, the third reliability P3, the fourth reliability P4, and the fifth reliability P5 from the reliability calculation unit 54. The integration unit 56 also integrates these reliabilities by using the acquired first reliability P1, the second reliability P2, the third reliability P3, the fourth reliability P4, and the fifth reliability P5, and the Haze estimation. As a result, the integration unit 56 calculates the integrated reliability Pi. Furthermore, the reliability of the prediction region Rp is evaluated comprehensively.

For example, the integration unit 56 substitutes the acquired first reliability P1, second reliability P2, third reliability P3, fourth reliability P4, and fifth reliability P5 into the following relational expression (1). As a result, the integration unit 56 calculates the integrated reliability Pi. When this calculated integrated reliability Pi is high, the reliability of the predicted region Rp is evaluated to be high. In addition, the integration unit 56 outputs this calculated integrated reliability Pi to the display brightness calculation unit 58 and position correction unit 60, which will be described later.

Then, the display brightness calculation unit 58 acquires the integrated reliability Pi. Furthermore, the display brightness calculation unit 58 calculates the display brightness Ld, for example, by multiplying the acquired integrated reliability Pi by a reference brightness. As a result, when the integrated reliability Pi is high, the reliability of the predicted region Rp is high, and therefore the reliability of the recognition position of the target T is high, and the display brightness Ld is increased. This improves the visibility of the display image Di by the driver. Furthermore, the display brightness calculation unit 58 outputs the calculated display brightness Ld to the filter processing unit 62 described later. Note that the reference brightness is set by experiments, simulations, etc. so that the display brightness Ld changes depending on the integrated reliability Pi.

The position correction unit 60 then acquires the integrated reliability Pi from the integration unit 56. The position correction unit 60 also acquires the captured image Im, the target region Rt, and the predicted region Rp from the estimation unit 52. The position correction unit 60 then uses the acquired integrated reliability Pi, the captured image Im, the target region Rt, and the predicted region Rp to calculate the corrected region Rr as the position of the corrected display image Di.

For example, the position correction unit 60 calculates the correction region Rr so that the correction region Rr is an area that divides the line segment connecting corresponding points in the target region Rt and the prediction region Rp into Pi:(1-Pi). Note that here, the integrated reliability Pi is in the range of 0 to 1.

As shown in FIG. 15, the position of one vertex of the target region Rt in the t+1th frame is defined as Vt. The X coordinate of Vt is defined as xt. The Y coordinate of Vt is defined as yt. The vertex of the prediction region Rp corresponding to Vt is defined as Vp. The X coordinate of Vp is defined as xp. The Y coordinate of Vp is defined as yp. The vertex of the correction region Rr corresponding to Vt is defined as Vr. The X coordinate of Vr is defined as xr. The Y coordinate of Vr is defined as yr. Note that the X coordinate is the horizontal coordinate of the captured image Im. Also, the Y coordinate is the vertical coordinate of the captured image Im.

In this case, xr is expressed as the following relational expression (2-1) using the integrated reliability Pi, xt, and xp. Furthermore, yr is expressed as the following relational expression (2-2) using the integrated reliability Pi, yt, and yp.

xr=xt×(1-Pi)+xp×Pi...(2-1)
yr=yt×(1-Pi)+yp×Pi...(2-2)

Returning to FIG. 5, the position correction unit 60 outputs this calculated correction region Rr to the filter processing unit 62, which will be described later.

Then, the filter processing unit 62 acquires the display luminance Ld from the display luminance calculation unit 58. Furthermore, the filter processing unit 62 acquires the correction region Rr from the position correction unit 60. Furthermore, the filter processing unit 62 performs low-pass filter processing such as moving average on the acquired display luminance Ld and correction region Rr. This removes noise contained in the display luminance Ld and correction region Rr. Furthermore, the filter processing unit 62 generates image data for displaying the display image Di at the lower end of the display luminance Ld and correction region Rr from which the noise has been removed. Furthermore, the filter processing unit 62 outputs this generated image data to the HUD 15. Therefore, the display image Di as shown in FIG. 4 is displayed. Therefore, the driver can view the display image Di.

The HCU 50 performs the processing as described above. Next, we will explain how the display control device 20 of this embodiment suppresses positional deviation of the superimposed display relative to the target T.

The recognition unit 30 of the display control device 20 serves as an acquisition unit that acquires the captured image Im captured by the image capturing device 13. The HCU 50 of the display control device 20 also serves as a display control unit that superimposes the display image Di on the target T based on the correction area Rr calculated using the target area Rt and the prediction area Rp.

As a result, even if the recognized position of the target T shifts, the display image Di is superimposed on the target T using the corrected area Rr calculated and corrected using the target area Rt and the predicted area Rp. Therefore, the display image Di is superimposed on the target T while the shift in the recognized position of the target T is suppressed. Therefore, the position shift of the superimposed display on the target T is suppressed.

The display control device 20 of this embodiment also provides the following advantages.

[1-1] Here, as described above, as the overlap length Dd increases, the overlap rate Ov increases. In addition, as the overlap length Dd increases, the reliability of the prediction region Rp increases, and the first reliability P1 and the integrated reliability Pi also increase.

For this reason, the position correction unit 60 of the HCU 50 changes the correction region Rr to move closer to the prediction region Rp as the overlap length Dd increases.

[1-2] As described above, as the recognition distance Dr becomes shorter, the resolution of the target T shown in the captured image Im becomes higher, and the reliability of the predicted area Rp increases, and the second reliability P2 and the integrated reliability Pi also increase.

For this reason, the position correction unit 60 changes the correction area Rr to be closer to the predicted area Rp as the recognition distance Dr becomes shorter.

[1-3] As described above, as the DNN recognition score value Sc_DNN increases, the accuracy of the type of target T shown in the captured image Im increases, and the likelihood of the predicted region Rp increases. As a result, the reliability of the predicted region Rp increases, and the third reliability P3 and the integrated reliability Pi also increase.

For this reason, the position correction unit 60 changes the correction region Rr to approach the predicted region Rp as the DNN recognition score value Sc_DNN increases.

[1-4] As described above, as the cumulative number of tracking times Nt_sum increases, the accuracy of the target T being tracked by the tracking unit 34 being the same increases, and the likelihood of the predicted region Rp increases. As a result, the reliability of the predicted region Rp increases, and the fifth reliability P5 and the integrated reliability Pi also increase.

For this reason, the position correction unit 60 changes the correction area Rr to approach the predicted area Rp as the cumulative tracking count Nt_sum increases.

As a result, the correction region Rr approaches the highly reliable predicted region Rp. Because the reliability of the correction region Rr is therefore increased, the display image Di is more likely to be superimposed on the target T while suppressing deviation in the recognized position of the target T. This suppresses deviation in the position of the superimposed display on the target T.

[1-5] Furthermore, the display brightness calculation unit 58 of the HCU 50 increases the display brightness Ld as the overlap length Dd increases.

[1-6] In addition, the display brightness calculation unit 58 increases the display brightness Ld as the recognition distance Dr becomes shorter.

[1-7] Furthermore, the display luminance calculation unit 58 increases the display luminance Ld as the DNN recognition score value Sc_DNN increases.

[1-8] In addition, the display brightness calculation unit 58 increases the display brightness Ld as the cumulative tracking count Nt_sum increases.

As a result, the visibility of the display image Di is improved by suppressing the positional deviation of the superimposed display relative to the target T.

(Modification of the first embodiment)
In the first embodiment, the display brightness calculation unit 58 calculates the display brightness Ld based on the integrated reliability Pi calculated by the integration unit 56. In contrast, the display brightness calculation unit 58 may calculate the display brightness Ld based on the situation and state of the target T reflected in the captured image Im. For example, it is assumed that the target T is a pedestrian. In this case, when the pedestrian is near a crosswalk, a traffic light, a stop line, or an intersection, the display brightness calculation unit 58 may increase the display brightness Ld. In addition, at this time, the display brightness calculation unit 58 may calculate the display brightness Ld for blinking the display image Di by calculating the time when the display brightness Ld is a value obtained by multiplying the integrated reliability Pi by the reference brightness and the time when the display brightness Ld is zero. In addition, for example, it is assumed that the target T is a traffic sign. In this case, when the traffic sign is a traffic sign that displays a regulation such as a stop sign, a no entry sign, and a speed display, the display brightness calculation unit 58 may increase the display brightness Ld. As a result, the visibility of the display image Di in which the positional deviation of the superimposed display with respect to the target T is suppressed is improved.

Second Embodiment
In the second embodiment, the display system 10 further includes an acceleration sensor 70. Also, the process of the estimation unit 52 differs from that of the first embodiment. Other than this, the second embodiment is similar to the first embodiment.

As shown in FIG. 16, the acceleration sensor 70 outputs a signal corresponding to the gravity direction acceleration Ag to the HCU 50. Note that the gravity direction acceleration Ag is the acceleration of the vehicle 1 in the gravity direction.

Here, vehicle 1 pitches due to unevenness or steps on the road on which vehicle 1 is traveling. At this time, the fluctuation of target area Rt increases, and the fluctuation of predicted area Rp also increases. As a result, as shown in FIG. 17, target area Rt and predicted area Rp are less likely to overlap. Therefore, at this time, overlap rate Ov decreases, and therefore first reliability P1 decreases. Therefore, the reliability of predicted area Rp decreases.

For this reason, the estimation unit 52 does not calculate the overlap rate Ov when the amount related to pitching of the vehicle 1 is large. Specifically, the estimation unit 52 acquires the gravity direction acceleration Ag from the acceleration sensor 70. Furthermore, when the acquired gravity direction acceleration Ag is equal to or greater than the acceleration threshold Ag_th, the estimation unit 52 determines that the amount related to pitching of the vehicle 1 is large. At this time, the estimation unit 52 does not calculate the overlap rate Ov. Note that the acceleration threshold Ag_th is set by experiments, simulations, etc. so that the amount related to pitching of the vehicle 1 is determined to be large.

In addition, since the overlap rate Ov is not calculated, the reliability calculation unit 54 does not calculate the first reliability P1. Furthermore, the integration unit 56 calculates the integrated reliability Pi without using the first reliability P1. Therefore, the position correction unit 60 calculates the correction region Rr without using the overlap rate Ov. Therefore, when the amount related to pitching of the vehicle 1 is large, even if the length of the overlap length Dd increases, the correction region Rr does not approach the predicted region Rp, and the position of the correction region Rr is maintained.

As described above, in the second embodiment, the display system 10 further includes an acceleration sensor 70, and the estimation unit 52 performs processing using the gravity direction acceleration Ag. In this second embodiment, the same effects as in the first embodiment are achieved. In addition, the second embodiment also achieves the effects described below.

[2] When the gravity direction acceleration Ag is equal to or greater than the acceleration threshold Ag_th, the position correction unit 60 maintains the position of the correction region Rr even as the overlap length Dd increases.

As a result, when the amount of pitching of the vehicle 1 is large, i.e., when the reliability of the prediction region Rp is low, the correction region Rr is not calculated using the prediction region Rp with low reliability. This reduces the position shift of the superimposed display relative to the target T caused by the prediction region Rp with low reliability.

(Modification 1 of the second embodiment)
In the second embodiment, the estimation unit 52 uses the gravity direction acceleration Ag to determine whether or not the amount related to pitching of the vehicle 1 is large. In contrast, the estimation unit 52 is not limited to using the gravity direction acceleration Ag when determining whether or not the amount related to pitching of the vehicle 1 is large. For example, the estimation unit 52 may use a time change in the position of the target region Rt to determine whether or not the amount related to pitching of the vehicle 1 is large.

For example, as shown in FIG. 18, the estimation unit 52 calculates a distance change amount ΔDt. The distance change amount ΔDt is the distance between corresponding positions from the position of the target region Rt(t) in the tth frame to the position of the target region Rt(t+1) in the t+1th frame, and is the vertical distance in the captured image Im. In this way, the estimation unit 52 calculates the change over time in the position of the target region Rt. Then, when the calculated distance change amount ΔDt is equal to or greater than the change amount threshold ΔDt_th, the estimation unit 52 determines that the amount related to the pitching of the vehicle 1 is large. Note that the change amount threshold ΔDt_th is calculated by experiments, simulations, etc. so that the amount related to the pitching of the vehicle 1 is determined to be large.

In addition, here, if the target T shown in the captured image Im is a four-wheeled motor vehicle or the like, when the four-wheeled motor vehicle shown in the captured image Im travels with abrupt lateral movement, the fluctuation of the target region Rt increases, and the fluctuation of the predicted region Rp also increases. As a result, the target region Rt and the predicted region Rp are less likely to overlap. Therefore, at this time, the overlap rate Ov decreases, and the first reliability P1 decreases. Therefore, at this time, the reliability of the predicted region Rp decreases.

Furthermore, if the four-wheeled motor vehicle shown in the captured image Im travels with a sudden lateral movement, the position of the four-wheeled motor vehicle shown in the captured image Im moves in the vertical direction of the captured image Im, and the distance change amount ΔDt becomes large.

For this reason, for example, when the target T shown in the captured image Im is a four-wheeled vehicle or the like, the estimation unit 52 determines that the four-wheeled vehicle shown in the captured image Im is making a sudden lateral movement when the distance change amount ΔDt is equal to or greater than the threshold value. In addition, at this time, the position correction unit 60 maintains the position of the correction region Rr even as the overlap length Dd becomes longer. This provides the same effect as that described in [2] above. The threshold value is set by experiment, simulation, or the like so that the four-wheeled vehicle shown in the captured image Im is determined to be making a sudden lateral movement.

(Modification 2 of the second embodiment)
Here, the reliability of the target region Rt may be higher than the reliability of the predicted region Rp. For example, when the variation in the time change of the position of the target region Rt is small, the accuracy of the target region Rt is high, and therefore the reliability of the target region Rt is high.

Therefore, when the variation in the time change of the position of the target region Rt is small, the estimation unit 52 does not calculate the overlap rate Ov. Specifically, the estimation unit 52 calculates a value related to the variation in the time change of the position of the target region Rt by calculating the variance, standard deviation, etc. of the target region Rt in multiple frames. Furthermore, when the calculated value related to the variation is equal to or less than a threshold, the estimation unit 52 determines that the variation in the time change of the position of the target region Rt is small. At this time, the estimation unit 52 does not calculate the overlap rate Ov. Furthermore, at this time, the position correction unit 60 changes the correction region Rr so as to bring it closer to a target region Rt with high reliability. This increases the reliability of the correction region Rr. Therefore, the position shift of the superimposed display with respect to the target T is suppressed. Note that the threshold value related to the variation is set by experiments, simulations, etc. so that the value related to the variation is determined to be small.

Also, for example, when the DNN recognition score value Sc_DNN is high, the accuracy of the type of target T shown in the captured image Im is high, and therefore the reliability of the target region Rt is high.

Therefore, when the DNN recognition score value Sc_DNN is high, the estimation unit 52 does not calculate the overlap rate Ov. Specifically, when the DNN recognition score value Sc_DNN is equal to or greater than the DNN threshold value Sc_th, the estimation unit 52 determines that the DNN recognition score value Sc_DNN is high. At this time, the estimation unit 52 does not calculate the overlap rate Ov. Also, at this time, the position correction unit 60 changes the correction area Rr so as to bring it closer to the target area Rt with high reliability. This increases the reliability of the correction area Rr. Therefore, the position shift of the superimposed display with respect to the target T is suppressed. Note that the DNN threshold value Sc_th is set by experiment, simulation, etc. so that the DNN recognition score value Sc_DNN is determined to be high.

Third Embodiment
In the third embodiment, the display system 10 further includes a solar radiation sensor 72 and a search wave transmitting/receiving unit 74. Also, the processing of the HCU 50 is different from that of the first embodiment. The rest is the same as that of the first embodiment.

As shown in FIG. 19, the solar radiation sensor 72 outputs a signal corresponding to the amount of solar radiation Ms from outside the vehicle 1 to the HCU 50.

The search wave transmitting/receiving unit 74 transmits search waves such as millimeter waves, sonar, and infrared rays to a target T in front of the vehicle 1. The search wave transmitting/receiving unit 74 further receives the search waves reflected by the target T. Based on the information obtained from the search waves, the search wave transmitting/receiving unit 74 then outputs a signal to the HCU 50 corresponding to the relative distance Dct of the target T to the vehicle 1 and the relative speed Vct of the target T to the vehicle 1.

As shown in FIG. 20, the HCU 50 does not include an estimation unit 52, a reliability calculation unit 54, an integration unit 56, and a display brightness calculation unit 58 as functional blocks, but includes an information acquisition unit 64, a position correction unit 60, and a filter processing unit 62 as functional blocks.

The information acquisition unit 64 acquires the exposure amount Me of the image capturing device 13 from the image capturing device 13. The information acquisition unit 64 also acquires the captured image Im, the target area Rt, and the predicted area Rp from the recognition unit 30. The information acquisition unit 64 also acquires the amount of solar radiation Ms from the solar radiation sensor 72. The information acquisition unit 64 also acquires the relative distance Dct of the target T to the vehicle 1 and the relative speed Vct of the target T to the vehicle 1 from the search wave transmitting/receiving unit 74. The information acquisition unit 64 also outputs the acquired amount of exposure Me, captured image Im, target area Rt, predicted area Rp, amount of solar radiation Ms, relative distance Dct of the target T to the vehicle 1, and relative speed Vct of the target T to the vehicle 1 to the position correction unit 60.

The position correction unit 60 calculates the correction area Rr based on the exposure amount Me, the captured image Im, the target area Rt, the predicted area Rp, the amount of solar radiation Ms, the relative distance Dct of the target T to the vehicle 1, and the relative speed Vct of the target T to the vehicle 1. The position correction unit 60 also outputs the calculated correction area Rr to the filter processing unit 62.

The filter processing unit 62 performs low-pass filtering such as moving average on the correction region Rr calculated by the position correction unit 60. This removes noise contained in the correction region Rr. The filter processing unit 62 also generates image data for displaying the display image Di in the correction region Rr from which the noise has been removed. The filter processing unit 62 also outputs the generated image data to the HUD 15. Therefore, when the display image Di is displayed, the driver can visually recognize the display image Di.

As described above, in the third embodiment, the display system 10 further includes a solar radiation sensor 72 and a search wave transmitting/receiving unit 74, and the HCU 50 includes an information acquisition unit 64, a position correction unit 60, and a filter processing unit 62 as functional blocks. Next, the processing of the position correction unit 60 will be described.

[3-1] For example, the position correction unit 60 detects vertical edge points of the captured image Im from the target region Rt and the prediction region Rp, and sets the region with the greater number of detected edge points as the correction region Rr. As a result, the correction region Rr is set as a region with a relatively high reliability, and the reliability of the correction region Rr is increased.

[3-2] Furthermore, the position correction unit 60 calculates the correction region Rr so that, for example, the correction region Rr is an area that divides the line segment connecting the corresponding points of the target region Rt and the prediction region Rp internally in the ratio m:n.

For example, if the target T shown in the captured image Im is a four-wheeled motor vehicle or the like, and the distance from the vehicle 1 to the target T is short and the background shown in the captured image Im is dark, such as at night, the tires of the four-wheeled motor vehicle and the like are difficult to image-recognize by the image processing unit 32. As a result, as shown in FIG. 21, when the image processing unit 32 performs image recognition, the pattern of the four-wheeled motor vehicle and the like may be erroneously recognized as the tires of the four-wheeled motor vehicle. Therefore, at this time, the reliability of the target region Rt is low.

For this reason, when the distance from the vehicle 1 to the target T is short and the background shown in the captured image Im is dark, the position correction unit 60 increases m. This causes the correction region Rr to approach the prediction region Rp, which has a higher reliability than the target region Rt. Therefore, the reliability of the correction region Rr increases.

Specifically, when the relative distance Dct of the target T to the vehicle 1 is equal to or less than the relative distance threshold Dct_th and the exposure amount Me is equal to or less than the exposure amount threshold Me_th, the position correction unit 60 determines that the distance from the vehicle 1 to the target T is short and the background shown in the captured image Im is dark. Alternatively, when the relative distance Dct of the target T to the vehicle 1 is equal to or less than the relative distance threshold Dct_th and the solar radiation Ms is equal to or less than the solar radiation threshold Ms_th, the position correction unit 60 determines that the distance from the vehicle 1 to the target T is short and the background shown in the captured image Im is dark. At this time, the position correction unit 60 increases m. The relative distance threshold Dct_th, the exposure amount threshold Me_th, and the solar radiation threshold Ms_th are set by experiments, simulations, etc. so that the distance from the vehicle 1 to the target T is short and the background shown in the captured image Im is determined to be dark. In addition, when determining whether the distance from the vehicle 1 to the target T is short, the recognition distance Dr obtained by the image processing unit 32 may be used instead of the relative distance Dct of the target T to the vehicle 1 obtained by the search wave transmitting/receiving unit 74.

[3-3] In addition, for example, when the distance from the vehicle 1 to the target T is short and the background shown in the captured image Im is dark, if the relative speed Vct of the target T with respect to the vehicle 1 is small, the resolution of the pattern of the target T becomes high, making it easy for erroneous recognition to occur due to image recognition.

Therefore, it is assumed that the distance from vehicle 1 to target T is short, the background shown in the captured image Im is dark, and the relative speed Vct of target T with respect to vehicle 1 is equal to or less than the speed threshold Vct_th. In this case, the position correction unit 60 determines that the distance from vehicle 1 to target T is short, the background shown in the captured image Im is dark, and the relative speed Vct of target T with respect to vehicle 1 is small.

At this time, the position correction unit 60 sets the correction region Rr to either the target region Rt or the prediction region Rp when the relative distance Dct of the target T to the vehicle 1 is smallest among the target region Rt and the prediction region Rp acquired up to the current time. As a result, the size of the correction region Rr becomes relatively large, so that the target T shown in the captured image Im is likely to fit within the correction region Rr, and the display image Di is prevented from overlapping with the target T. This prevents the driver from feeling uncomfortable. The speed threshold Vct_th is set by experiments, simulations, etc. so that it is determined whether the relative speed Vct of the target T to the vehicle 1 is small or not.

[3-4] In addition, at this time, the position correction unit 60 may set the region formed by the bottommost edge, the leftmost edge, and the rightmost edge of the captured image Im, which is the combined region of the target region Rt and the prediction region Rp, as the correction region Rr, as shown in FIG. 22. As a result, as in the above, the size of the correction region Rr becomes relatively large, and the target T shown in the captured image Im tends to fit within the correction region Rr. This prevents the display image Di from overlapping with the target T.

[3-5] Furthermore, at this time, the position correction unit 60 may set as the correction region Rr either the target region Rt or the prediction region Rp for which the DNN recognition score value Sc_DNN is the largest among the target region Rt and the prediction region Rp acquired up to this point. This increases the reliability of the correction region Rr.

As described above, in the third embodiment, the position correction unit 60 performs processing. The third embodiment also achieves the same effects as the first embodiment.

Other Embodiments
The present disclosure is not limited to the above-described embodiments, and appropriate modifications can be made to the above-described embodiments. Furthermore, in each of the above-described embodiments, it goes without saying that the elements constituting the embodiments are not necessarily essential, except in cases where they are particularly clearly stated as essential or where they are clearly considered essential in principle.

The acquisition unit, display control unit, etc., and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor and memory programmed to execute one or more functions embodied in a computer program. Alternatively, the acquisition unit, display control unit, etc., and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the acquisition unit, display control unit, etc., and the method thereof described in the present disclosure may be realized by one or more dedicated computers configured by combining a processor and memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transient tangible recording medium as instructions executed by the computer.

The above embodiments and modifications may be combined as appropriate.

1 Vehicle 10 Display system 13 Image capturing device 15 HUD
20 Display control device 30 Recognition unit 32 Image processing unit 34 Tracking unit 36 Prediction unit 50 HCU

Claims (23)

an acquisition unit (30) that acquires an image (Im) captured by an image capturing device (13);
a display control unit (50) that displays a display image (Di) superimposed on the target based on a correction area (Rr) calculated using a target area (Rt) that is an area within the captured image and includes a position of the target (T) reflected in the captured image and a prediction area (Rp) that is an area within the captured image and includes a predicted position of the target reflected in the captured image; and
A display control device comprising: The display control device according to claim 1, wherein the display control unit changes the correction area to approach the predicted area as the length (Dd) of the overlapping portion between the target area and the predicted area in one direction in the captured image increases. The image capturing device is mounted on a vehicle (1),
The display control device according to claim 1 , wherein the display control unit changes the correction area so as to approach the prediction area as a distance (Dr) from the vehicle to the target becomes shorter. The display control device according to claim 1, wherein the display control unit changes the correction area to approach the prediction area as a value (Sc_DNN) related to the identification of the type of target calculated using deep learning increases. The display control device according to claim 1, wherein the display control unit changes the correction area to approach the prediction area as the number of times (Nt_sum) that the target in the target area coincides with the target in the prediction area increases. The display control device according to claim 2, wherein the display control unit increases the brightness of the display image as the length (Dd) of the overlapping portion increases. The display control device according to claim 3, wherein the display control unit increases the brightness of the display image as the distance (Dr) from the vehicle to the target decreases. The display control device according to claim 4, wherein the display control unit increases the brightness of the display image as a value (Sc_DNN) related to the identification of the type of target calculated using deep learning increases. The display control device according to claim 5, wherein the display control unit increases the luminance of the display image as the number of times (Nt_sum) that the target in the target area coincides with the target in the prediction area increases. The display control device according to claim 2, wherein the display control unit maintains the position of the correction area even when the length (Dd) of the overlapping portion increases when the acceleration (Ag) acting on the vehicle (1) in the direction of gravity is equal to or greater than an acceleration threshold (Ag_th). The display control device according to claim 2, wherein the display control unit maintains the position of the correction area even when the length of the overlapping portion (Dd) increases when a value (ΔDt) relating to the change in the position of the target area over time is equal to or greater than a threshold value. The display control device according to claim 1, wherein the display control unit changes the correction area to approach the target area when a value relating to the variation in the time change of the position of the target area is equal to or less than a threshold value. The display control device according to claim 1, wherein the display control unit changes the correction area to be closer to the target area when a value (Sc_DNN) relating to the identification of the type of target calculated using deep learning is equal to or greater than a threshold (Sc_th). The display control device according to claim 1, wherein the display control unit sets the correction area to an area of the target area and the prediction area that has a large number of edge points in the vertical direction of the captured image. The image capturing device is mounted on a vehicle (1),
2. The display control device according to claim 1, wherein the display control unit changes the correction area to be closer to the prediction area when a relative distance (Dct) of the target to the vehicle is equal to or less than a relative distance threshold (Dct_th) and an exposure (Me) of the image capturing device is equal to or less than an exposure threshold (Me_th). The image capturing device is mounted on a vehicle (1),
2. The display control device according to claim 1, wherein the display control unit changes the correction area to approach the prediction area when the relative distance (Dct) of the target to the vehicle is equal to or less than a relative distance threshold (Dct_th) and the amount of solar radiation (Ms) from outside the vehicle is equal to or less than a solar radiation threshold (Ms_th). The image capturing device is mounted on a vehicle (1),
The display control unit is
When the relative distance (Dct) of the target with respect to the vehicle is equal to or less than a relative distance threshold (Dct_th) and the exposure amount (Me) of the image capturing device is equal to or less than an exposure amount threshold (Me_th),
When the relative speed (Vct) of the target with respect to the vehicle is equal to or less than a speed threshold (Vct_th),
2 . The display control device according to claim 1 , wherein the correction area is determined to be one of the target area and the predicted area when the relative distance of the target to the vehicle is smallest among the target area and the predicted area acquired up to the present time. The image capturing device is mounted on a vehicle (1),
The display control unit is
When the relative distance (Dct) of the target with respect to the vehicle is equal to or less than a relative distance threshold (Dct_th) and the amount of solar radiation (Ms) from outside the vehicle is equal to or less than a solar radiation threshold (Ms_th),
When the relative speed (Vct) of the target with respect to the vehicle is equal to or less than a speed threshold (Vct_th),
2 . The display control device according to claim 1 , wherein the correction area is determined to be one of the target area and the predicted area when the relative distance of the target to the vehicle is smallest among the target area and the predicted area acquired up to the present time. The image capturing device is mounted on a vehicle (1),
The display control unit is
When the relative distance (Dct) of the target with respect to the vehicle is equal to or less than a relative distance threshold (Dct_th) and the exposure amount (Me) of the image capturing device is equal to or less than an exposure amount threshold (Me_th),
When the relative speed (Vct) of the target with respect to the vehicle is equal to or less than a speed threshold (Vct_th),
The display control device according to claim 1 , wherein the correction area is an area formed by the lowermost edge of the captured image, the leftmost edge of the captured image, and the rightmost edge of the captured image, among the area obtained by combining the target area and the prediction area. The image capturing device is mounted on a vehicle (1),
The display control unit is
When the relative distance (Dct) of the target with respect to the vehicle is equal to or less than a relative distance threshold (Dct_th) and the amount of solar radiation (Ms) from outside the vehicle is equal to or less than a solar radiation threshold (Ms_th),
When the relative speed (Vct) of the target with respect to the vehicle is equal to or less than a speed threshold (Vct_th),
The display control device according to claim 1 , wherein the correction area is an area formed by the lowermost edge of the captured image, the leftmost edge of the captured image, and the rightmost edge of the captured image, among the area obtained by combining the target area and the prediction area. The image capturing device is mounted on a vehicle (1),
The display control unit is
When the relative distance (Dct) of the target with respect to the vehicle is equal to or less than a relative distance threshold (Dct_th) and the exposure amount (Me) of the image capturing device is equal to or less than an exposure amount threshold (Me_th),
When the relative speed (Vct) of the target with respect to the vehicle is equal to or less than a speed threshold (Vct_th),
The display control device according to claim 1, wherein the correction area is selected from the target area and the prediction area when a value (Sc_DNN) related to identification of the type of the target calculated using deep learning is the largest among the target area and the prediction area acquired up to the present time. The image capturing device is mounted on a vehicle (1),
The display control unit is
When the relative distance (Dct) of the target with respect to the vehicle is equal to or less than a relative distance threshold (Dct_th) and the amount of solar radiation (Ms) from outside the vehicle is equal to or less than a solar radiation threshold (Ms_th),
When the relative speed (Vct) of the target with respect to the vehicle is equal to or less than a speed threshold (Vct_th),
The display control device according to claim 1, wherein the correction area is selected from the target area and the prediction area when a value (Sc_DNN) related to identification of the type of the target calculated using deep learning is the largest among the target area and the prediction area acquired up to the present time. A display control device
An acquisition unit (30) that acquires an image (Im) captured by an image capturing device (13); and
A display control program that functions as a display control unit (50) that superimposes a display image (Di) on a target based on a correction area (Rr) calculated using a target area (Rt) that is an area within the captured image that includes the position of the target (T) shown in the captured image and a prediction area (Rp) that is an area within the captured image that includes the predicted position of the target shown in the captured image.

Images