JP2020161021A - Traffic light detection method and traffic light detection device - Google Patents

Abstract

To provide a traffic light detection method and a traffic light detection device that can prevent a high luminance object that moves relative to an own vehicle, such as a tail lamp of another vehicle, from being falsely detected as a traffic light, and improve traffic signal recognition performance.SOLUTION: A traffic light detection method and a traffic light detection device create a second image in which time change of a luminance value due to an object moving relative to a vehicle is suppressed based on a first image captured by an imaging unit mounted on the vehicle, extracts a pixel whose magnitude of the component of a predetermined frequency included in the time change of the luminance value among pixels of the second image exceeds a predetermined threshold value as a traffic light pixel showing a traffic light, and thereby detects the traffic light.SELECTED DRAWING: Figure 1

Description

The present invention relates to a traffic light detection method and a traffic light detection device.

Focusing on the fact that the brightness of the traffic light lamp changes depending on the frequency of the input AC power supply as a technology for recognizing the traffic light even when the entire shape of the traffic light cannot be seen at night or when the traffic light is partially shielded. Then, a traffic light detection device and a traffic light detection method using a synchronous image generator that extracts a region where the brightness changes at the same frequency as the lamp of the traffic light by frequency analysis and calculates a frequency component have been proposed (see Patent Document 1). ..

International Publication No. 2015/136594

However, according to the technique described in Patent Document 1, since a region having the same frequency component as the lamp of a traffic light is extracted, an object that is not a traffic light when a high-brightness object moves on a pixel. May be falsely detected as a traffic light.

For example, when a high-luminance object such as a tail lamp passes over a specific pixel, the brightness value of the pixel gradually increases as the high-luminance object moves, and after the brightness value reaches the maximum. Since the brightness value of the pixel gradually decreases, the brightness change in the pixel includes a frequency component. Therefore, the change in brightness due to the movement of a high-brightness object may include the same frequency component as the lamp of the traffic light, and there is a possibility that an object that is not a traffic light may be erroneously detected as a traffic light.

The present invention has been made in view of the above problems, and an object of the present invention is to erroneously detect a high-intensity object that moves relative to its own vehicle, such as a tail lamp of another vehicle, as a traffic light. It is an object of the present invention to provide a traffic light detection method and a traffic light detection device capable of preventing such a situation and improving signal recognition performance.

In order to solve the above-mentioned problems, the traffic light detection method and the traffic light detection device according to one aspect of the present invention move relative to the vehicle based on the first image acquired by the imaging unit mounted on the vehicle. A second image is created in which the time change of the brightness value due to the object is suppressed, and among the pixels of the second image, the traffic light selects the pixel in which the magnitude of the component of the predetermined frequency included in the time change of the brightness value exceeds the predetermined threshold value. The traffic light is detected by extracting it as a captured signal pixel.

According to the present invention, when the own vehicle approaches an intersection, it is possible to prevent a situation in which a high-intensity object that moves relative to the own vehicle, such as the tail lamp of another vehicle, is erroneously detected as a traffic light and a signal. The recognition performance can be improved.

FIG. 1 is a block diagram showing a configuration of a traffic light detection device according to an embodiment of the present invention. FIG. 2 is a block diagram showing a configuration of a frequency component calculation unit in the traffic signal detection device according to the embodiment of the present invention. FIG. 3 is a flowchart showing a processing procedure of the traffic light detection device according to the embodiment of the present invention. FIG. 4 is a diagram showing a state of time change of the brightness value of the lamp of the traffic light. FIG. 5 is a diagram showing how the luminance value changes in the pixels through which a high-luminance object passes.

Next, embodiments of the present invention will be described in detail with reference to the drawings. In the description, the same reference numerals are given and duplicate description is omitted.

[Structure of traffic light detector]
FIG. 1 is a block diagram showing a configuration of a traffic light detection device according to the present embodiment. As shown in FIG. 1, the traffic signal detection device according to the present embodiment includes an imaging unit 20 and a controller 100. The traffic light detection device is mounted on a vehicle (not shown).

The image pickup unit 20 is a digital camera provided with a solid-state image sensor such as a CCD or CMOS, and acquires an image capable of image processing as a first image. The imaging unit 20 includes a wide-angle lens having a wide angle of view, and the imaging range (angle of view) of the camera includes from the traveling direction of the vehicle to the left-right direction of the vehicle, and also includes road shoulders around the vehicle. .. The imaging unit 20 repeatedly images the surroundings of the vehicle at predetermined time intervals, and acquires a plurality of continuous first images (frames).

It is desirable that the sampling frequency (the reciprocal of the predetermined time interval at which the imaging is performed) when the imaging unit 20 performs imaging is higher than the frequency of the AC power supply supplied to the lamp of the traffic signal to be detected. More preferably, the sampling frequency is at least twice the frequency of the AC power supply.

The controller 100 (an example of a control unit and a processing unit) is a general-purpose microcomputer including a CPU (central processing unit), a memory, and an input / output unit. A computer program (traffic light detector program) for functioning as a traffic light detector is installed in the controller 100. By executing the computer program, the controller 100 functions as a plurality of information processing circuits (51, 53, 55, 57, 61, 63) included in the traffic signal detection device.

Here, an example is shown in which a plurality of information processing circuits (51, 53, 55, 57, 61, 63) included in the traffic signal detection device are realized by software. However, it is also possible to configure information processing circuits (51, 53, 55, 57, 61, 63) by preparing dedicated hardware for executing each of the following information processing. Further, a plurality of information processing circuits (51, 53, 55, 57, 61, 63) may be configured by individual hardware. Further, the information processing circuit (51, 53, 55, 57, 61, 63) may also be used as an electronic control unit (ECU) used for other control related to the vehicle.

The controller 100, as a plurality of information processing circuits (51, 53, 55, 57, 61, 63), includes a fluctuation suppression unit 51, a region determination unit 53, a frequency component calculation unit 55, a region extraction unit 57, and a color determination unit 61. The output unit 63 is provided. Further, as shown in FIG. 2, the frequency component calculation unit 55 includes a multiplication unit 81, a low-pass filter 83 (LPF), a reference signal generation unit 85, and a phase determination unit 87.

The area determination unit 53 extracts the target pixel (target pixel) for suppressing the time change of the brightness value based on the time-series change in the brightness value of each pixel constituting the first image. For example, the area determination unit 53 compares the brightness values between frames that are before and after the time, and sets the target pixel as a pixel in which the brightness value continuously increases or the brightness value continuously decreases for a predetermined time. Extract as. Further, the area determination unit 53 may be extracted as a target pixel based on pattern matching.

In addition, the area determination unit 53 is a pixel that compares the luminance values between frames that are before and after the time, and continuously increases or decreases the luminance value for a predetermined time. Moreover, the pixel whose luminance difference with the adjacent pixel is equal to or more than a predetermined value may be extracted as a target pixel.

The predetermined time used when extracting a pixel whose brightness value continuously increases or decreases continuously as a target pixel is a value equal to or greater than the reciprocal of the frequency of the AC power supply supplied to the traffic light. It is desirable to have.

The fluctuation suppression unit 51 is an object that moves relative to the vehicle with respect to the target pixel extracted by the area determination unit 53 based on the time-series change in the brightness value of each pixel constituting the first image. Performs a process of suppressing the time change of the brightness value due to. For example, the fluctuation suppression unit 51 takes a difference in brightness values between frames before and after the target pixel with respect to the target pixel, and creates a second image using the absolute value of the difference as a new brightness value. Taking the difference between the luminance values between the frames corresponds to performing the first-order time differentiation of the luminance values in the first image, that is, processing by the first derivative filter. The fluctuation suppression unit 51 may create a second image by setting the brightness value of the target pixel as a constant.

The fluctuation suppression unit 51 performs a process of suppressing the time change of the brightness value for the target pixel extracted by the region determination unit 53, and creates a second image. The brightness values of the pixels other than the target pixels of the second image are the same as the brightness values of the pixels of the first image. The fluctuation suppression unit 51 sequentially creates a second image by sequentially processing the first image repeatedly acquired by the imaging unit 20.

The frequency component calculation unit 55 calculates a component of a predetermined frequency included in the time change of the brightness value based on the time-series change in the brightness value of each pixel constituting the second image created by the fluctuation suppression unit 51. ..

For the calculation, the frequency component calculation unit 55 uses the reference signal generated by the reference signal generation unit 85. The reference signal is a sine wave. The reference signal generation unit 85 includes a normal signal having the same frequency as the AC power supply supplied to the signal, a delay signal having a predetermined delay phase with respect to the normal signal, and an advanced signal having a predetermined lead phase with respect to the normal signal. Generates three types of reference signals.

The multiplication unit 81 multiplies each of the three types of reference signals by a time-series luminance value, and calculates three types of detection signals corresponding to each of the three types of reference signals. The low-pass filter 83 removes harmonic components from the detection signal and outputs an amplitude detection signal. The frequency component calculation unit 55 outputs the amplitude detection signal having the largest amplitude among the three types of amplitude detection signals.

The reason for using the three types of reference signals is that the phase difference between the AC power supply actually supplied to the traffic light and the reference signal is generally unknown, and the amplitude of the amplitude detection signal changes due to the phase difference. This is to deal with the situation. The phase determination unit 87 controls the reference signal generation unit 85 so that the reference signal generation unit 85 generates a normal signal in synchronization with the phase of the amplitude detection signal having the largest amplitude among the three types of amplitude detection signals. To do.

In addition to the above, the frequency component calculation unit 55 may calculate a predetermined frequency component by performing a discrete-time Fourier transform based on the data of the frame having a finite time length. In this case, no reference signal is required.

Based on the output of the frequency component calculation unit 55, the region extraction unit 57 extracts pixels in which the magnitude (amplitude) of the component of the predetermined frequency exceeds the predetermined threshold value as the signal pixel in which the signal is captured. In addition, the area extraction unit 57 may extract traffic light pixels and group adjacent traffic light pixels. A set of pixels obtained by grouping may be extracted as an isolated point region having a set of pixels of a predetermined number or less. Further, the isolated point region may be determined as a traffic signal candidate region.

The color determination unit 61 determines the signal color of the traffic light by referring to the color information (for example, RGB value) of the pixel on the first image corresponding to the traffic light pixel.

The output unit 63 notifies the driver of the signal color of the traffic light determined by the color determination unit 61. For example, the output unit 63 may notify the driver of the signal color of the traffic light as visual information via an image display unit (not shown). Further, the output unit 63 may notify the driver of the signal color of the traffic light as auditory information by a speaker (not shown).

[Traffic light detector processing procedure]
Next, the processing procedure of the traffic light detection by the traffic light detection device according to the present embodiment will be described with reference to the flowchart of FIG. The traffic light detection process shown in FIG. 3 starts when the ignition of the vehicle is turned on, and is repeatedly executed while the ignition is turned on.

In step S101, the imaging unit 20 repeatedly images the surroundings of the vehicle to acquire a plurality of continuous first images (frames). The acquired first image is recorded in a memory or the like (not shown). The image pickup unit 20 performs a plurality of times of imaging during one cycle of the frequency of the AC power supply supplied to the lamp of the traffic light.

In step S103, the area determination unit 53 refers to the plurality of recorded first images and calculates a change in the luminance value between frames that are temporally back and forth for each pixel. Then, in the pixel for which the change in the brightness value is calculated in step S105, when the brightness value continuously increases or continuously decreases for a predetermined time (when YES in step S105). The area determination unit 53 extracts the pixel as a target pixel, and in step S107, performs a process of suppressing the time change of the brightness value of the extracted target pixel. If NO in step S105, the process of suppressing the time change of the luminance value is not performed. Information on whether or not the pixels constituting the first image are extracted as the target pixels is recorded in a memory or the like (not shown).

In step S109, the reference signal generation unit 85 included in the frequency component calculation unit 55 is a normal signal having the same frequency as the AC power supply supplied to the signal, a delay signal having a predetermined lag phase with respect to the normal signal, and a normal signal. Three types of reference signals, an advanced signal having a predetermined lead phase with respect to the signal, are generated.

In step S111, the multiplication unit 81 multiplies each of the three types of reference signals calculated in step S109 by a time-series luminance value, and three types of detection signals corresponding to each of the three types of reference signals. Is calculated. Then, the low-pass filter 83 removes the harmonic component from the detection signal and outputs the amplitude detection signal.

In step S113, the frequency component calculation unit 55 detects the amplitude detection signal having the largest amplitude among the three types of amplitude detection signals calculated in step S111. The frequency component calculation unit 55 outputs the detected amplitude detection signal.

In step S115, the area extraction unit 57 extracts pixels in which the size of the component of the predetermined frequency exceeds the predetermined threshold value as the signal pixel in which the signal is captured, based on the output of the frequency component calculation unit 55. The area extraction unit 57 extracts signal pixels, groups adjacent signal pixels, and extracts a set of pixels obtained by grouping as an isolated point region.

In step S117, the region extraction unit 57 performs image binarization based on the first image. Specifically, in the first image, pixels showing a predetermined luminance value or more are extracted as high-luminance pixels. In step S119, the area extraction unit 57 extracts the pixels extracted as high-luminance pixels and included in the isolated point region as the signal pixel in which the signal is captured. As a result, the isolated point region can be determined as a traffic light candidate region.

In step S121, the region extraction unit 57 performs tracking processing of an isolated point region determined as a traffic light candidate region. There are various methods for the tracking process, for example, the tracking process is performed based on the normalized correlation method.

In step S123, the area extraction unit 57 determines whether or not the number of traces in step S121 is equal to or greater than a predetermined threshold for each traffic signal candidate region, and if the number of traces is equal to or greater than a predetermined threshold (YES in step S123). In the case), in step S131, the color determination for the signal candidate region is performed. If NO in step S123, no color determination is performed.

In step S131, the color determination unit 61 determines the signal color of the traffic light by referring to the color information of the pixel on the first image corresponding to the traffic light pixel. More specifically, the signal color of the traffic light is determined for the traffic light candidate region where the number of tracking times is determined to be equal to or greater than a predetermined threshold value in step S123. For example, the hue is determined by referring to the RGB value as the color information of the pixel, and if it is determined to be "red" or "blue" ("green"), it is regarded as a signal, and the color is recorded as a signal color.

In step S133, the output unit 63 notifies the driver of the signal color of the traffic light determined by the color determination unit 61. For example, the output unit 63 notifies the driver of the signal color of the traffic light as visual information via an image display unit (not shown). Alternatively, the output unit 63 notifies the driver of the signal color of the traffic light as auditory information by a speaker (not shown).

[Effect of Embodiment]
As described in detail above, the signal detection method and the signal detection device according to the present embodiment are based on an object that moves relative to the vehicle based on the first image acquired by the imaging unit mounted on the vehicle. A second image in which the time change of the brightness value is suppressed is created, a component of a predetermined frequency included in the time change of the brightness value is calculated for each pixel of the second image, and the magnitude (amplitude) of the component exceeds a predetermined threshold. Pixels are extracted as signal pixel in which the signal appears.

As a result, it becomes difficult to observe from the second image the same frequency component as the time change of the brightness value of the lamp of the traffic light, which may occur in the pixel in which the moving high-intensity object is reflected. Therefore, it is possible to reduce the possibility of erroneously detecting a pixel in which a moving high-intensity object is reflected as an area where a traffic light exists. As a result, only the lamp portion of the traffic light can be detected with high accuracy. In addition, the detected portions can be grouped and extracted, and the accuracy of detecting the traffic light and recognizing the lighting state of the traffic light can be improved.

Further, in the traffic light detection method and the traffic light detection device according to the present embodiment, the second image is obtained by setting the first-order time derivative of the brightness value of the pixel of the first image as a new brightness value for each pixel of the first image. It may be the one to be created. As a result, it becomes difficult to observe the same frequency component as the time change of the brightness value of the lamp of the traffic light from the second image, and the possibility of erroneous detection in the region where the traffic light exists can be reduced.

The reason why the frequency component that can be generated in the pixel in which the moving high-luminance object is reflected becomes difficult to be observed due to the first-order time derivative of the brightness value will be described with reference to FIGS. 4 and 5.

FIG. 4 is a diagram showing a state of time change of the brightness value of the lamp of the traffic light. As shown in FIG. 4, in general, the lamp of the traffic light changes as shown by the curve F1 and the curve F2 depending on the voltage change of the AC power supply supplied to the lamp of the traffic light. In FIG. 4, the curve F1 shows the time change of the brightness value in a highly responsive lamp such as an LED, and the curve F2 shows the time change of the brightness value in a low responsive lamp such as an incandescent lamp. ing. Since the first-order time derivative of the curves F1 and F2 also shows the same change as in FIG. 4, the first-order time derivative does not suppress the time change of the brightness value of the lamp of the traffic light.

On the other hand, FIG. 5 is a diagram showing how the luminance value changes in the pixels through which a high-luminance object passes. FIG. 5 shows a case where the high-luminance object BR passes through the region R1 reflected in a certain pixel, and the high-luminance object BR starts to be reflected in the pixel at time t1 (state A), and is high at time t2. A state in which the luminance object BR is completely reflected in the pixel (state B) and a state in which the state in which the luminance object BR is reflected in the pixel at time t3 ends (state C) are shown. The temporal change of the luminance value due to the movement of the high-luminance object does not continue steadily as shown in FIG. It only occurs between time t2 and time t3) and does not continue. Therefore, the first-order time derivative suppresses the frequency component that can occur in the pixel in which the moving high-intensity object is reflected.

Further, in the signal detection method and the signal detection device according to the present embodiment, pixels whose brightness value continuously increases or decreases continuously for a predetermined time are extracted as target pixels and are targeted pixels. The second image may be created by suppressing the time change of the brightness value in.

As shown in FIG. 5, since pixels in which the luminance value continuously increases or decreases continuously can be extracted as target pixels, the pixels through which a high-luminance object passes can be captured more accurately. Therefore, it is possible to suppress the time change of the luminance value in the pixel. As a result, it becomes difficult to observe the same frequency component as the time change of the brightness value of the lamp of the traffic light from the second image, and the possibility of erroneous detection in the region where the traffic light exists can be reduced.

Further, in the signal detection method and the signal detection device according to the present embodiment, a pixel whose brightness value continuously increases or decreases continuously for a predetermined time, and whose brightness is that of an adjacent pixel. Pixels whose difference is equal to or greater than a predetermined value may be extracted as target pixels, and a second image may be created by suppressing the time change of the brightness value in the target pixels.

Since pixels whose brightness difference from adjacent pixels is equal to or greater than a predetermined value are extracted as target pixels, the target pixels are extracted using the edge information that can exist around the pixels in which a high-brightness object is reflected. Become. Therefore, it is possible to more accurately capture the pixel through which a high-luminance object passes and suppress the time change of the brightness value in the pixel. As a result, it becomes difficult to observe the same frequency component as the time change of the brightness value of the lamp of the traffic light from the second image, and the possibility of erroneous detection in the region where the traffic light exists can be reduced.

Further, in the traffic light detection method and the traffic light detection device according to the present embodiment, the second image may be created by setting a new luminance value of the pixel extracted as the target pixel as a constant. It is possible to more accurately capture the pixel through which a high-luminance object passes and remove the time change of the brightness value in the pixel.

Further, in the traffic light detection method and the traffic light detection device according to the present embodiment, the predetermined time may be a length equal to or greater than the reciprocal of the predetermined frequency. Pixels whose brightness value continuously increases or decreases continuously over a period longer than the time length of one cycle of the time change of the brightness value generated in the lamp of the traffic light are extracted as target pixels. Therefore, it is possible to more accurately capture the pixel through which the high-luminance object passes and suppress the time change of the brightness value in the pixel. As a result, it becomes difficult to observe the same frequency component as the time change of the brightness value of the lamp of the traffic light from the second image, and the possibility of erroneous detection in the region where the traffic light exists can be reduced.

Further, in the traffic light detection method and the traffic light detection device according to the present embodiment, the predetermined frequency may be twice the frequency of the AC power supply supplied to the traffic light. Since the frequency of the time change of the brightness value that occurs in the lamp of the traffic light is often twice the frequency of the AC power supply, it is possible to more accurately capture the time change of the brightness value that occurs in the lamp of the traffic light.

Further, in the traffic light detection method and the traffic light detection device according to the present embodiment, the signal color of the traffic light may be determined based on the color information of the traffic light pixel, and the signal color may be notified to the driver of the vehicle. Even in an environment where it is difficult for the driver to recognize the traffic light, the signal color of the traffic light detected based on the time change of the brightness value generated in the lamp of the traffic light can be notified to the driver, and as a result, the driver can promptly notify the driver of the traffic light. The lighting state can be recognized.

Each function shown in the above-described embodiment can be implemented by one or more processing circuits. Processing circuits include programmed processors, electrical circuits, etc., as well as devices such as application specific integrated circuits (ASICs) and circuit components arranged to perform the described functions. Etc. are also included.

Although the contents of the present invention have been described above with reference to the embodiments, it is obvious to those skilled in the art that the present invention is not limited to these descriptions and various modifications and improvements can be made. The discourses and drawings that form part of this disclosure should not be understood as limiting the invention. Various alternative embodiments, examples and operational techniques will be apparent to those skilled in the art from this disclosure.

It goes without saying that the present invention includes various embodiments not described here. Therefore, the technical scope of the present invention is defined only by the matters specifying the invention relating to the reasonable claims from the above description.

20 Imaging unit 51 Fluctuation suppression unit 53 Region determination unit 55 Frequency component calculation unit 57 Region extraction unit 61 Color determination unit 63 Output unit 81 Multiplication unit 83 Low-pass filter (LPF)
85 Reference signal generator 87 Phase determination unit 100 Controller

Claims (9)

It is a traffic light detection method that detects a traffic light based on a first image acquired by an imaging unit mounted on a vehicle.
Based on the first image, a second image in which the time change of the brightness value due to the object moving relative to the vehicle is suppressed is created.
A component of a predetermined frequency included in the time change of the luminance value is calculated for each pixel of the second image.
A traffic light detection method, characterized in that pixels in which the size of the component exceeds a predetermined threshold value are extracted as traffic light pixels in which the traffic light is captured. The traffic light detection method according to claim 1.
A traffic light detection method, characterized in that the second image is created by setting the first-order time derivative of the brightness value of the pixel of the first image as a new brightness value for each pixel of the first image. The traffic light detection method according to claim 1.
Pixels whose luminance value continuously increases or whose luminance value continuously decreases for a predetermined time are extracted as target pixels.
A traffic light detection method characterized in that the second image is created by suppressing a time change of a luminance value in the target pixel. The traffic light detection method according to claim 1.
Pixels whose luminance value continuously increases or decreases continuously for a predetermined time and whose luminance difference from adjacent pixels is equal to or greater than a predetermined value are extracted as target pixels.
A traffic light detection method characterized in that the second image is created by suppressing a time change of a luminance value in the target pixel. The traffic light detection method according to claim 3 or 4.
A traffic light detection method characterized in that the second image is created by setting a new luminance value of a pixel extracted as a target pixel as a constant. The traffic light detection method according to any one of claims 3 to 5.
A traffic signal detection method, wherein the predetermined time is a length equal to or greater than the reciprocal of the predetermined frequency. The traffic light detection method according to any one of claims 1 to 6.
A traffic light detection method, wherein the predetermined frequency is twice the frequency of an AC power supply supplied to the traffic light. The traffic light detection method according to any one of claims 1 to 7.
The signal color of the traffic light is determined based on the color information of the traffic light pixel,
A traffic light detection method comprising notifying the driver of the vehicle of the signal color. The image pickup unit mounted on the vehicle and
A controller that processes the first image acquired by the imaging unit, and
It is a traffic light detector equipped with
The controller
Based on the first image, a second image in which the time change of the brightness value due to the object moving relative to the vehicle is suppressed is created.
For each pixel of the second image, a component of a predetermined frequency included in the time change of the brightness value is calculated.
A traffic light detection device characterized in that pixels in which the size of the component exceeds a predetermined threshold value are extracted as traffic light pixels in which a traffic light is captured.

Images