WO2024161521A1

WO2024161521A1 - Traffic signal recognition method and traffic signal recognition device

Info

Publication number: WO2024161521A1
Application number: PCT/JP2023/003104
Authority: WO
Inventors: 誠信永瀬; 宏寿植田
Original assignee: 日産自動車株式会社
Priority date: 2023-01-31
Filing date: 2023-01-31
Publication date: 2024-08-08

Abstract

A traffic signal recognition method and traffic signal recognition device according to the present invention involve determining the present state of every traffic signal in each of a plurality of images captured in the advancement direction of a vehicle and acquiring a result series that comprises a plurality of determination results in time series order for every traffic signal. When a plurality of traffic signals include coordinated traffic signals, it is determined whether partial series that comprise at least a prescribed number of consecutive identical determination results that include the newest of the determination results of the result series for the coordinated traffic signals coincide for at least two traffic signals. When it has been determined that the partial series coincide, the present state as in the determination results of the coinciding partial series is outputted as a present state that is based on the traffic signals.

Description

Traffic light recognition method and traffic light recognition device

The present invention relates to a traffic light recognition method and a traffic light recognition device.

Patent Document 1 proposes a traffic light recognition technology that accumulates the traffic light color (current display state) for each frame across multiple frames, and then performs a majority vote on the traffic light color results for the multiple frames to determine the latest traffic light color.

International Publication No. 2019/177019

However, the technology described in Patent Document 1 involves majority voting on the results of the current signal status of a traffic light in multiple frames, which creates the problem of a delay between when the current signal status of a traffic light changes and when it is actually determined that the current signal status has changed.

The present invention was made in consideration of the above problems, and its purpose is to provide a traffic light recognition method and a traffic light recognition device that can reduce the delay between when the current signal state of a traffic light changes and when it is actually determined that the current signal state has changed.

In order to solve the above-mentioned problems, a traffic light recognition method and traffic light recognition device according to one embodiment of the present invention targets multiple images captured in the direction of travel of the vehicle, determines the current status of each traffic light for each image, and obtains a result sequence consisting of multiple judgment results in chronological order for each traffic light. If there are interlocking traffic lights among the multiple traffic lights, it is determined whether or not subsequences consisting of a predetermined number or more consecutive identical judgment results, including the most recent judgment result, among the judgment results that make up the result sequence related to the interlocking traffic lights match between at least two or more traffic lights. Then, if it is determined that the subsequences match, the current status related to the judgment results in the matching subsequences is output as the current status based on the traffic light.

The present invention makes it possible to reduce the delay between when the current signal state of a traffic light changes and when it is actually determined that the current signal state has changed.

FIG. 1 is a block diagram showing the configuration of a traffic light recognition device according to an embodiment of the present invention. FIG. 2 is a flowchart showing a processing procedure of the traffic light recognition device according to one embodiment of the present invention. FIG. 3 is a flowchart showing the output determination process. FIG. 4 is a diagram showing an example of a result sequence made up of a plurality of determination results in chronological order. FIG. 5 shows an example of a current status output based on a result sequence for one traffic light. FIG. 6 shows an example of a current state output based on a result sequence relating to two interlocking traffic signals. FIG. 7 is a diagram showing an example of the output of the current status when the image changes from a state in which an interlocking traffic signal is present to a state in which an interlocking traffic signal is not present. FIG. 8 shows an example of a current status output based on a result sequence relating to three interlocking traffic signals. FIG. 9 is a diagram showing an example of complementation of the determination results in the result string.

Next, an embodiment of the present invention will be described in detail with reference to the drawings. In the description, the same parts will be given the same reference numerals and duplicate explanations will be omitted.

[Configuration of traffic light recognition device]
Fig. 1 is a block diagram showing the configuration of a traffic light recognition device according to this embodiment. As shown in Fig. 1, the traffic light recognition device according to this embodiment includes an imaging unit 71 and a controller 100. The traffic light recognition device may include an on-board sensor 73, a map information acquisition unit 75, and a vehicle control device 400. The controller 100 is connected to the imaging unit 71, the on-board sensor 73, the map information acquisition unit 75, and the vehicle control device 400 via a wired or wireless communication path.

Here, the imaging unit 71, the on-board sensor 73, and the vehicle control device 400 are mounted on a vehicle (not shown), but the map information acquisition unit 75 and the controller 100 may be mounted on the vehicle or installed outside the vehicle.

The imaging unit 71 captures images in the direction in which the vehicle is traveling. For example, the imaging unit 71 is a digital camera equipped with a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and captures images of the surroundings of the vehicle to obtain digital images of the surrounding area. The imaging unit 71 captures an image of a specified range around the vehicle by setting the focal length, the lens angle of view, the vertical and horizontal angles of the camera, etc.

The captured images captured by the imaging unit 71 are output to the controller 100 and stored in a storage unit (not shown) for a predetermined period of time. For example, the imaging unit 71 captures captured images at predetermined time intervals, and the captured images captured at the predetermined time intervals are stored in the storage unit as past images. The past images may be deleted after a predetermined period of time has passed since the capture of the past images.

The on-board sensor 73 is composed of an object detection sensor, such as a laser radar, a millimeter wave radar, or a camera, mounted on the vehicle, that detects objects present around the vehicle. The on-board sensor 73 may be equipped with a plurality of different types of object detection sensors.

The on-board sensor 73 detects the environment around the vehicle. For example, the on-board sensor 73 may detect moving objects including other vehicles, motorcycles, bicycles, and pedestrians, and stationary objects including stopped vehicles, and detect the position, attitude, size, speed, acceleration, deceleration, yaw rate, etc. of the moving and stationary objects relative to the vehicle. The on-board sensor 73 may output, as a detection result, the behavior of two-dimensional objects in a zenith diagram (also called a plan view) viewed from the air above the vehicle. The on-board sensor 73 may also detect signs (road signs and signs marked on the road surface) and guide rails that exist around the vehicle. In addition, the on-board sensor 73 may detect the rotational speed or difference in rotational speed of the wheels of the vehicle to detect the slipperiness of the road surface on the lane on which the vehicle is traveling.

In addition to the environment around the vehicle, the on-board sensor 73 detects the state of the vehicle. For example, the on-board sensor 73 may detect the vehicle's moving speed (forward/backward and left/right moving speed, turning speed), the steering angle of the wheels of the vehicle, and the speed of change of the steering angle.

In addition, the on-board sensor 73 may include a sensor that measures the absolute position of the vehicle, that is, the position, attitude, and speed of the vehicle relative to a predetermined reference point, using a position detection sensor that measures the absolute position of the vehicle, such as a GPS (Global Positioning System) or odometry.

The map information acquisition unit 75 acquires map information that indicates the structure of the road on which the vehicle is traveling. The map information acquired by the map information acquisition unit 75 includes road structure information such as absolute lane positions, lane connection relationships, and relative position relationships. The map information acquired by the map information acquisition unit 75 may also include facility information such as parking lots and gas stations. In addition, the map information may include traffic light position information, traffic light type, and the position of the stop line corresponding to the traffic light. The map information acquisition unit 75 may own a map database that stores map information, or may acquire map information from an external map data server by cloud computing. The map information acquisition unit 75 may also acquire map information using vehicle-to-vehicle communication or road-to-vehicle communication.

The vehicle control device 400 controls a vehicle (not shown) based on the traffic light recognition results obtained by the controller 100. For example, the vehicle control device 400 may be a device that automatically drives the vehicle along a predetermined driving route, or a device that assists the vehicle occupant in driving operations. Additionally, the vehicle control device 400 may be a notification device that notifies the vehicle occupant of the traffic light recognition results.

The controller 100 (an example of a control unit or processing unit) is a general-purpose microcomputer equipped with a CPU (central processing unit), memory, and input/output units. A computer program (traffic light recognition program) for functioning as part of the traffic light recognition device is installed in the controller 100. By executing the computer program, the controller 100 functions as multiple information processing circuits (140, 150, 160, 170, 180) equipped in the traffic light recognition device.

Here, an example is shown in which the multiple information processing circuits (140, 150, 160, 170, 180) of the traffic light recognition device are realized by software. However, it is also possible to configure the information processing circuits (140, 150, 160, 170, 180) by preparing dedicated hardware for executing each of the information processes shown below. Also, the multiple information processing circuits (140, 150, 160, 170, 180) may be configured by individual hardware. Furthermore, the information processing circuits (140, 150, 160, 170, 180) may also be used as electronic control units (ECUs) used for other vehicle-related controls.

The controller 100 includes a detection area setting unit 140, a judgment unit 150, a judgment result storage unit 160, an output determination unit 170, and an output unit 180 as multiple information processing circuits (140, 150, 160, 170, 180).

The detection area setting unit 140 sets a detection area corresponding to a traffic light on the image captured by the imaging unit 71. Here, "detection area" means an area in the image where a traffic light is estimated to be present. The detection area setting unit 140 sets a detection area for each traffic light, for one or more traffic lights estimated to be present in the image.

The position of the traffic light captured within the imaging range in the image can be estimated based on the imaging direction of the imaging unit 71, the position and posture of the vehicle at the time of imaging, and the position of the traffic light. The determination unit 150 sets, for example, a part of the captured image that includes the estimated position of the traffic light in the image as the detection area.

The detection area setting unit 140 may extract, from among the traffic lights captured within the imaging range, those that correspond to the road on which the vehicle is traveling as target traffic lights, and set a detection area for each target traffic light, targeting only the extracted target traffic lights. For example, the detection area setting unit 140 extracts, as the target traffic light, a traffic light that corresponds to the road on which the vehicle is traveling, based on map information that indicates the position of the vehicle and the structure of the road on which the vehicle is traveling. This makes it possible to output the current indication status of the target traffic light based on the result sequence related to the target traffic light, through processing by the determination unit 150, determination result storage unit 160, output determination unit 170, and output unit 180, which will be described later.

The detection area setting unit 140 also determines whether or not there is an "interlocking traffic light" among the target traffic lights. Here, "interlocking traffic lights" refers to multiple traffic lights that correspond to the same stop line. The detection area setting unit 140 may directly obtain the position of the stop line corresponding to the traffic lights from the map information, and determine whether or not multiple traffic lights are "multiple traffic lights" based on whether or not the positions of the stop lines are the same. The detection area setting unit 140 may also determine whether or not multiple traffic lights are "multiple traffic lights" based on position information of the traffic lights. The detection area setting unit 140 may also determine whether or not multiple traffic lights are "multiple traffic lights" based on position information of the multiple traffic lights detected by the in-vehicle device.

The determination unit 150 performs image processing on the detection area set for each traffic light, detects the traffic light within the detection area, and determines the current status of the traffic light. The determination unit 150 detects traffic lights by, for example, template matching. Template matching uses an image of a standard traffic light as a template, scans the detection area while shifting it by one pixel at a time, and calculates the correlation of the brightness distribution, for example. Then, when the correlation reaches the highest value, it detects that a traffic light is present at the position on the image where the template is located.

The "color signals" shown by traffic lights include "green," "yellow," and "red." The meaning of a "color signal" is determined by traffic laws that vehicles must follow; for example, a "green" means "you may proceed," a "red" means "stop at the stop position," and a "yellow" means "stop at the stop position, unless you are too close to the stop position to stop safely."

Such discrimination between "green light," "yellow light," and "red light" may be performed by assuming that the "color signal" with the highest brightness level among the three "color signals" is illuminated.

In addition, traffic lights are not limited to "color signals," but may also display "arrow signals" that indicate the direction vehicles are permitted to go at the intersection where the traffic lights are installed. Examples include "right turn signals," "straight ahead signals," and "left turn signals."

"Arrow signals" are not limited to "right turn signals," "straight ahead signals," and "left turn signals," and there are many variations depending on the structure of the intersection where the traffic lights are installed. The meaning of an "arrow signal" is determined by the traffic laws that vehicles must follow.

The determination unit 150 performs image processing on the detection area to determine the current status of the traffic light, such as the illumination status of the "color signal" or "arrow signal."

In addition, the image processing for traffic light detection in the determination unit 150 may utilize machine learning such as a support vector machine or a neural network. When detecting traffic lights, a learning database in which templates of traffic lights of different sizes are stored in advance can be prepared, and the recognition rate can be improved by referring to different learning databases depending on the distance to the traffic light.

The determination result storage unit 160 stores the current state of one or more traffic lights determined by the determination unit 150 for each traffic light. Specifically, the determination unit 150 sequentially performs a determination for each traffic light on multiple images in chronological order acquired by the imaging unit 71, and the determination result storage unit 160 acquires the obtained determination results from the determination unit 150. The determination result storage unit 160 then stores the multiple determination results in chronological order for each traffic light as a result sequence associated with the traffic light.

The structure of the result string stored by the judgment result storage unit 160 will be explained using Figure 4. Figure 4 is a diagram showing an example of a result string for one traffic light, consisting of multiple judgment results in chronological order. For the sake of simplicity, the judgment results obtained by the judgment unit 150 are assumed to be of four types: "green", "yellow", "red", and "unknown" (a judgment result in which the current display status is unknown). In Figure 4, the judgment results of "green", "yellow", "red", and "unknown" are represented by a square filled with white, a square filled with a diagonal stripe pattern, a square filled with a checkerboard pattern, and a square marked with a "?" mark, respectively. The same notation is used in the subsequent figures.

The reason why the determination unit 150 may determine the status as "unknown" is, for example, when the traffic light flickers due to a flicker phenomenon (when the traffic light is an LED lamp that flashes periodically, the brightness in the captured image changes depending on the timing of the image capture by the image capture unit 71) or a rolling shutter phenomenon (when the image capture method used by the image capture unit 71 causes the image capture timing to differ for each position in the captured image).

The number N of judgment results that constitute one result sequence associated with one traffic light and are stored by the judgment result storage unit 160 is preset.

The N judgment results that make up one result string associated with one traffic light are each assigned a subscript ranging from 0 to N-1. Here, the judgment result with subscript 0 is the most recent judgment result obtained by the judgment unit 150, and as the subscript assigned to the judgment result increases, it indicates that the judgment result is older.

When a new judgment result is input to the judgment result storage unit 160, the judgment result storage unit 160 increments the subscript assigned to the judgment result by 1 and assigns the subscript 0 to the new judgment result input to the judgment result storage unit 160. The judgment result with the subscript N is erased.

In this way, the judgment results with subscripts from 0 to N-1 are stored in the judgment result storage unit 160 as a result string.

In addition, the judgment result storage unit 160 may have a function of supplementing the judgment results. Specifically, when "unknown" is included in the judgment results constituting the result sequence stored in the judgment result storage unit 160, the judgment result of "unknown" may be replaced based on the judgment result immediately before the "unknown" and the judgment result immediately after the "unknown".

For example, in the result string shown in FIG. 4, the judgment result for subscript 2 is "unknown." In this case, the judgment result storage unit 160 refers to the judgment result for subscript 3, which is immediately before the judgment result for subscript 2, and the judgment result for subscript 1, which is immediately after the judgment result for subscript 2. The judgment results for subscript 1 and subscript 3 are both "yellow light." Therefore, the judgment result for subscript 2 may be "yellow light," which is the judgment result for subscript 3, which is immediately before the judgment result for subscript 2, rather than "unknown."

In this way, the judgment result storage unit 160 may extract, from among the judgment results constituting the result string, a judgment result that indicates that the current display state is unknown as the first judgment result. The judgment result storage unit 160 may then extract, from among the judgment results constituting the result string, excluding judgment results that indicate that the current display state is unknown, the judgment result immediately before the first judgment result and the judgment result immediately after the first judgment result as the second judgment result and the third judgment result, respectively, and may replace the first judgment result in the result string with the second judgment result when the second judgment result and the third judgment result are identical.

In addition, the judgment result storage unit 160 may replace the first judgment result in the result string with the judgment result immediately following the first judgment result if there is no judgment result immediately preceding the first judgment result among the judgment results constituting the result string, excluding the first judgment result. This makes it possible to complement the judgment results even if there is a judgment result at the beginning of the result string that indicates that the current display state is unknown.

In addition, the judgment result storage unit 160 may replace the first judgment result in the result string with the judgment result immediately preceding the first judgment result if there is no judgment result immediately following the first judgment result among the judgment results constituting the result string, excluding the first judgment result. This makes it possible to complement the judgment results even if there is a judgment result at the end of the result string indicating that the current display state is unknown.

In situations where the determination unit 150 cannot determine the current state of a traffic light due to flickering of the traffic light, the function of complementing the determination result by the determination result storage unit 160 described above is effective in improving the accuracy of traffic light recognition.

In the above explanation, the judgment results stored in the judgment result storage unit 160 are described as N, but this is not limited to this.

For example, the judgment result storage unit 160 may determine whether or not to store a judgment result based on the capture time of the image on which the judgment result is based, instead of the number of judgment results. More specifically, the judgment result storage unit 160 may erase judgment results corresponding to captured images captured a predetermined time or more after capture, so that the judgment results stored in the judgment result storage unit 160 are only those corresponding to images captured up to a predetermined time in the past from the present.

When the imaging unit 71 captures images at a predetermined cycle, the result is the same whether the number of judgment results stored in the judgment result storage unit 160 is limited based on the number N or limited based on a predetermined time.

On the other hand, when the imaging unit 71 captures images at unspecified timing that is not a specified cycle, the results may differ depending on whether the number is limited based on the number N or based on a specified time. However, the configuration of the judgment result storage unit 160 can be adopted regardless of whether the number of judgment results to be stored is limited based on the number N or based on a specified time.

The output determination unit 170 determines the output value based on the judgment result included in the result sequence for each traffic light.

First, when there is no linked traffic light, the output determination unit 170 sets the current indication state associated with the most frequent judgment result among the judgment results that make up the result string related to the traffic light as the output value. In other words, the output value is set to the current indication state associated with the judgment result that is determined by majority vote among the judgment results that make up the result string.

In addition, if the most common judgment result is not determined to be one, the current status related to the judgment result to be used preferentially when setting the output value may be set in advance. For example, to give priority to ensuring that the vehicle stops, the output value may be set preferentially in the following order: "red light," "yellow light," "green light," and "unknown."

The operation of the output determination unit 170 when the number of traffic lights on the image is one will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of the output of the current display state based on the result sequence related to one traffic light. FIG. 5 shows how the result sequence related to one traffic light transitions from step "t" to step "t+13".

Between step "t" and step "t+4", the majority of the judgment results in the result sequence are related to "red light", so the output determination unit 170 sets the output value to "red".

In step "t+5", there are the same number of judgment results related to "red light" and "green light", and it is not possible to determine which is the most common judgment result. Therefore, the output determination unit 170 prioritizes "red light" over "green light", and sets the output value to "red".

Between step "t+6" and step "t+11", the majority of the judgment results in the result sequence are related to "green light", so the output determination unit 170 sets the output value to "green".

Between step "t+12" and step "t+13", the majority of the judgment results in the result sequence are related to "yellow light", so the output determination unit 170 sets the output value to "yellow".

In addition, when there are interlocking traffic lights, the output determination unit 170 may determine whether or not a subsequence consisting of a predetermined number or more of consecutive identical judgment results, including the most recent judgment result, among the judgment results that make up the result sequence related to the interlocking traffic lights, matches between at least two or more traffic lights. When it is determined that the subsequences match, the output determination unit 170 may set the current indication state related to the judgment results in the matching subsequence as the output value.

In addition, when there is an interlocking traffic light and it is determined that the subsequences do not match, the output determination unit 170 may continue to maintain the current indication state based on the traffic light a number of times not exceeding a first predetermined number of times, and set this as the output value. Note that the first predetermined number of times may be less than or equal to half the number of determination results that make up the result sequence. In order to count the number of times the current indication state is maintained, the output determination unit 170 manages a retention count variable.

The operation of the output determination unit 170 when there are interlocking traffic lights will be described with reference to Figure 6. Figure 6 is a diagram showing an example of the output of the current state based on the result sequence related to two interlocking traffic lights. Figure 6 shows how the result sequence related to the first traffic light transitions from step "t" to step "t+13", and how the result sequence related to the second traffic light transitions from step "t+1" to step "t+13".

In the following, it will be explained that the first two judgment results in the result string form a "subsequence." In other words, it will be explained that the predetermined number is two. It will also be explained that the first predetermined number is five (less than half the number of judgment results that form the result string).

In step "t", since only the result sequence related to the first traffic light exists, the output determination unit 170 sets the output value to "red" since the majority vote result in the result sequence related to the first traffic light is the most numerous (the majority vote result is "red").

In step "t+1", the majority result based on the result sequence related to the first traffic light is "red", and the majority result based on the result sequence related to the second traffic light is "unknown". When setting the output value, "red light" takes priority over "unknown", so the output determination unit 170 sets the output value to "red".

In step "t+2", the subsequence related to the first traffic light (the two judgment results with

subscripts

0 and 1 at the beginning of the result sequence) is "green, green", and the subsequence related to the second traffic light is "green, green", which matches. Therefore, the output determination unit 170 sets the output value to "green" in correspondence with the current status "green" related to the judgment result in the matching subsequence.

At step "t+3", the subsequence for the first traffic light is "red, green", and the subsequence for the second traffic light is "green, green", which do not match. Similarly, at step "t+4", the subsequences do not match. Therefore, the output determination unit 170 maintains "green" as the output value.

In steps "t+5" and "t+6", the subsequence related to the first traffic light and the subsequence related to the second traffic light match, and the output determination unit 170 sets the output value to "green".

Between step "t+7" and step "t+11", the subsequence related to the first traffic light and the subsequence related to the second traffic light do not match. Therefore, the output determination unit 170 maintains the output value as "green".

In steps "t+12" and "t+13", the subsequence related to the first traffic light and the subsequence related to the second traffic light do not match. However, the number of times that the output determination unit 170 maintains "green" as the output value does not exceed the first predetermined number of times. Here, the majority result based on the result sequence related to the first traffic light is "yellow", and the majority result based on the result sequence related to the second traffic light is "unknown". When setting the output value, "yellow" is given priority over "unknown", so the output determination unit 170 sets "yellow" as the output value.

If the output value is set using the majority decision result based on the result sequence related to the first traffic light and the majority decision result based on the result sequence related to the second traffic light, the output determination unit 170 can set the output value to "green" for the first time at step "t+6".

In contrast, when the output value is set by the output determination process using the above-mentioned subsequence, the output determination unit 170 can set the output value to "green" for the first time at step "t+2". Therefore, the delay from when the current signal state of the traffic light changes to when it is actually determined that the current signal state has changed is suppressed.

Furthermore, when the output value is set by the output determination process using the subsequence described above, the output value can be stably maintained from the time when "blue" is set as the output value in step "t+2" until the time when "yellow" is set as the output value in step "t+12".

The output determination unit 170 may also determine whether or not an interlocking traffic signal has changed from a state in which there is an interlocking traffic signal to a state in which there is no interlocking traffic signal in a plurality of chronologically-ordered images acquired by the imaging unit 71. If the output determination unit 170 determines that an interlocking traffic signal has changed from a state in which there is an interlocking traffic signal to a state in which there is no interlocking traffic signal, the output determination unit 170 may continue to maintain the current indication state based on the traffic signal for a number of times not exceeding a second predetermined number of times, and set the current indication state as the output value. Note that the second predetermined number of times may be less than or equal to half the number of determination results constituting the result string. To count the number of times the current indication state has been maintained, the output determination unit 170 manages a retention count variable.

With reference to Figure 7, the operation of the output determination unit 170 when an image changes from a state where there is an interlocking traffic signal to a state where there is no interlocking traffic signal will be described. Figure 7 is a diagram showing an example of the output of the current display state when an image changes from a state where there is an interlocking traffic signal to a state where there is no interlocking traffic signal. Figure 7 shows how the result sequence related to the first traffic signal transitions from step "t" to step "t+8", and how the result sequence related to the second traffic signal transitions from step "t" to step "t+3". That is, in Figure 7, at step "t+4", the image changes from a state where there is an interlocking traffic signal to a state where there is no interlocking traffic signal.

In the following, the second predetermined number of times will be described as 5 (less than half the number of judgment results that make up the result string).

In step "t", the subsequence related to the first traffic light is "red, red" and the subsequence related to the second traffic light is "red, red", which are a match. Therefore, the output determination unit 170 sets the output value to "red" in correspondence with the current status "red light" related to the judgment result in the matched subsequence.

At step "t+1", the subsequence for the first traffic light is "green, red", and the subsequence for the second traffic light is "green, red", which are not consecutive identical judgment results. Therefore, the output determination unit 170 maintains "red" as the output value.

In steps "t+2" and "t+3", the subsequence related to the first traffic light is "green, green", and the subsequence related to the second traffic light is "green, green", which matches. Therefore, the output determination unit 170 sets the output value to "green" in correspondence with the current status "green" related to the determination result in the matching subsequence.

From step "t+4" onwards, there is no result sequence relating to the second traffic light. Therefore, if the output value is set based on the majority vote result based on the result sequence relating to the first traffic light, the output value that was "green" in step "t+3" will change to "red" in step "t+4". Because the output value changes frequently over a very short number of steps, it is difficult to use the current state judgment result for vehicle control as it is.

Therefore, from step "t+4" to step "t+8", the output determination unit 170 maintains the output value as "blue".

This prevents the output value from changing too frequently in very short steps, making it easier to use the current state determination results for vehicle control.

In addition, when the number of interlocking traffic lights is three or more, the output determination unit 170 may set the current state related to the judgment result of the most common subsequence among the subsequences related to traffic lights as the output value.

The operation of the output determination unit 170 when the number of interlocking traffic lights is three will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of the output of the current state based on the result sequence related to three interlocking traffic lights. FIG. 8 shows how the result sequence related to the first traffic light, the result sequence related to the second traffic light, and the result sequence related to the third traffic light all transition from step "t" to step "t+6".

In step "t", the subsequence related to the first traffic light is "red, red", and the subsequence related to the second traffic light is "red, red", which are a match. On the other hand, the subsequence related to the first traffic light is "green, green", which is not a match with the subsequence related to the first traffic light and the subsequence related to the second traffic light. However, the most common subsequence is "red, red". Therefore, the output determination unit 170 sets "red" as the output value in correspondence with the current status "red light" related to the determination result of the most common subsequence.

At step "t+1", the subsequence for the first traffic light is "green, red", the subsequence for the second traffic light is "green, red", and the subsequence for the third traffic light is "green, green", so it cannot be said that the consecutive identical judgment results match. Therefore, the output determination unit 170 maintains "red" as the output value.

In step "t+3", the subsequence related to the first traffic light is "green, green", and the subsequence related to the third traffic light is "green, green", which are a match. On the other hand, the subsequence related to the second traffic light is "red, green", which is not a match with the subsequence related to the first traffic light and the subsequence related to the third traffic light. However, the most common subsequence is "green, green". Therefore, the output determination unit 170 sets "green" as the output value in correspondence with the current status "green" related to the determination result of the most common subsequence.

In step "t+4", the subsequence related to the first traffic light is "green, green", and the subsequence related to the third traffic light is "green, green", which are a match. On the other hand, the subsequence related to the second traffic light is "green, red", which is not a match with the subsequence related to the first traffic light and the subsequence related to the third traffic light. However, the most common subsequence is "green, green". Therefore, the output determination unit 170 sets the output value to "green" in correspondence with the current status "green" related to the determination result of the most common subsequence.

In steps "t+5" and "t+6", the subsequence related to the first traffic light, the subsequence related to the first traffic light, and the subsequence related to the first traffic light are all "green, green", so they match. Therefore, the output determination unit 170 sets the output value to "green" in correspondence with the current status "green" related to the determination result in the most frequent subsequence.

If the output value is set using the majority decision result based on the result sequence related to the first traffic light, the majority decision result based on the result sequence related to the second traffic light, and the majority decision result based on the result sequence related to the third traffic light, the output determination unit 170 can set the output value to "green" for the first time in step "t+6".

The output unit 180 outputs the output value determined by the output determination unit 170 as the "current indication state based on the traffic light." Here, the "current indication state based on the traffic light" can be said to be the result obtained by integrating the current indication states of one or more traffic lights.

For example, the "current status based on the traffic light" is output from the output unit 180 to the vehicle control device 400 and used to control the vehicle. Alternatively, the "current status based on the traffic light" may be output from the output unit 180 to a reporting device (not shown) and notified to the occupant via the reporting device.

[Processing procedure of traffic light recognition device]
Next, a process for traffic light recognition by the traffic light recognition device according to the present embodiment will be described with reference to the flowcharts of Figures 2 and 3. Figure 2 is a flowchart showing the process by the traffic light recognition device according to the present embodiment. Figure 3 is a flowchart showing an output determination process.

The traffic light recognition process shown in Figures 2 and 3 may be executed each time an image is captured by the imaging unit 71, or may be executed at each cycle in which image processing is performed on the detection area after an image is captured by the imaging unit 71.

In FIG. 2, in step S101, the controller 100 acquires an image of one or more traffic lights in the direction in which the vehicle is traveling.

In step S103, the controller 100 acquires the vehicle's position via the on-board sensor 73, etc.

In step S105, the controller 100 acquires the positions of one or more traffic lights based on the map information acquired from the map information acquisition unit 75 or the like.

In step S107, the detection area setting unit 140 extracts the traffic light corresponding to the road on which the vehicle is traveling as the target traffic light. Note that if all traffic lights reflected in the acquired image are to be treated as target traffic lights, the processing of steps S103, S105, and S107 may be omitted.

In step S109, the detection area setting unit 140 sets a detection area for each traffic light on the image.

In step S111, the determination unit 150 performs image processing on the detection area set for each traffic light to detect the traffic lights within the detection area, and determines the current status for each traffic light.

In step S113, the determination result storage unit 160 stores the result sequence for each traffic light.

In step S115, the output determination unit 170 performs the output determination process shown in FIG. 3.

In FIG. 3, in step S121, the output determination unit 170 determines whether or not there is a linked traffic light.

If there is an interlocking traffic light (YES in step S121), in step S123, the output determination unit 170 determines whether the subsequences match.

If the subsequences match (YES in step S123), then in step S125, the output determination unit 170 determines an output value based on the subsequence. Then, in step S127, the output determination unit 170 sets the retention count variable to 1. On the other hand, if the subsequences do not match (NO in step S123), the process proceeds to step S133, which will be described later.

If there is no linked traffic light (NO in step S121), in step S131, the output determination unit 170 determines whether the retention count variable is 1 or greater.

If the retention count variable is 1 or more (YES in step S131), in step S133, the output determination unit 170 determines whether the retention count variable is less than or equal to a predetermined number (first predetermined number, second predetermined number).

If the retention count variable is equal to or less than the predetermined number (YES in step S133), in step S135, the output determination unit 170 maintains the output value based on the subsequence (sets the previously output output value as the output value), and in step S137, the output determination unit 170 increments the retention count variable by 1. On the other hand, if the retention count variable exceeds the predetermined number (NO in step S133), the process proceeds to step S141 described below.

If the retention count variable is 0 (NO in step S131), in step S141, the output determination unit 170 determines the output value by majority vote based on the result sequence. Then, in step 143, the output determination unit 170 sets the retention count variable to 0.

After the processing of steps S127, S137, and S143, in step S151, the output determination unit 170 outputs the output value to the output unit 180. After that, the output unit 180 outputs the output value determined by the output determination unit 170 as the "current indication state based on the traffic light."

[Effects of the embodiment]
As described above in detail, the traffic light recognition method and traffic light recognition device according to this embodiment target a plurality of images captured in the vehicle's travel direction, determine the current status of each traffic light for each image, and obtain a result sequence consisting of a plurality of determination results in chronological order for each traffic light. If there are interlocking traffic lights among the plurality of traffic lights, it is determined whether or not subsequences consisting of a predetermined number or more consecutive identical determination results, including the most recent determination result, among the determination results constituting the result sequence related to the interlocking traffic lights match between at least two or more traffic lights. If it is determined that the subsequences match, the current status related to the determination results in the matching subsequences is output as the current status based on the traffic light.

This makes it possible to reduce the delay between when the current signal state of a traffic light changes and when it is actually determined that the current signal state has changed. In particular, it is possible to speed up the change in the judgment result compared to the majority vote result based on the judgment results that make up the result string.

The traffic light recognition method and traffic light recognition device according to this embodiment may also be configured to, in the case where there is an interlocking traffic light and it is determined that the subsequences do not match, continue to output the current status based on the traffic light a number of times not exceeding a first predetermined number of times. This prevents the determination result of the current status of the traffic light from changing frequently in a very short step, making it easier to use the determination result of the current status for vehicle control.

Furthermore, in the traffic light recognition method and traffic light recognition device according to this embodiment, the first predetermined number of times may be less than half the number of judgment results that make up the result string. This makes it possible to prevent the judgment results of the current state of the traffic light from changing too frequently, while also reducing deviations from the majority result based on the judgment results that make up the result string. As a result, it is possible to prevent erroneous judgment results from being output more continuously than necessary.

The traffic light recognition method and traffic light recognition device according to this embodiment may also be capable of outputting the current status related to the judgment result of the most common subsequence among the subsequences related to the interlocking traffic lights as the current status based on the traffic lights when the number of interlocking traffic lights is three or more. This makes it possible to suppress erroneous recognition of the current status of the traffic lights by utilizing the judgment results of the current status of multiple traffic lights. Furthermore, even if the most recent consecutive judgment results of all traffic lights do not match, it is possible to hasten the switching of the judgment result.

Furthermore, the traffic light recognition method and traffic light recognition device according to this embodiment may output, when there is one traffic light, the current indication state associated with the most frequent judgment result among the judgment results constituting the result string related to the traffic lights as the current indication state based on the traffic light. In this way, when there is one traffic light, a judgment can be made using the judgment results constituting the result string.

The traffic light recognition method and traffic light recognition device according to this embodiment may also determine whether an interlocking traffic light has changed from a state in which there is an interlocking traffic light to a state in which there is no interlocking traffic light, and if it is determined that an interlocking traffic light has changed from a state in which there is an interlocking traffic light to a state in which there is no interlocking traffic light, continuously output the current status based on the traffic light a number of times not exceeding a second predetermined number of times. This prevents the determination result of the current status of the traffic light from changing frequently in a very short step, and as a result, makes it easier to use the determination result of the current status for vehicle control.

Furthermore, in the traffic light recognition method and traffic light recognition device according to this embodiment, the second predetermined number of times may be less than half the number of judgment results that make up the result string. This makes it possible to prevent the judgment results of the current state of the traffic light from changing too frequently, while also reducing deviations from the majority result based on the judgment results that make up the result string. As a result, it is possible to prevent erroneous judgment results from being output more continuously than necessary.

Furthermore, in the traffic light recognition method and traffic light recognition device according to this embodiment, the interlocking traffic lights may be multiple traffic lights that correspond to the same stop line on the road on which the vehicle is traveling. This makes it possible to determine the current status of only the traffic lights that the vehicle should obey when traveling. As a result, erroneous recognition of the current status of traffic lights can be suppressed.

Furthermore, the traffic light recognition method and traffic light recognition device according to this embodiment may extract, from among the judgment results constituting the result string, a judgment result that indicates that the current display state is unknown as the first judgment result. From among the judgment results constituting the result string, excluding the first judgment result, the judgment result immediately before the first judgment result and the judgment result immediately after the first judgment result may be extracted as the second judgment result and the third judgment result, respectively. If the second judgment result and the third judgment result are identical, the first judgment result in the result string may be replaced with the second judgment result.

This makes it possible to reduce the time delay between when the actual signal state of the traffic light changes and when the output value changes, even in situations where the current signal state cannot be determined due to flickering of the traffic light, thereby reducing the free-running distance of the vehicle.

For example, as shown in FIG. 9, when determining the current state of a traffic light based on the determination results that constitute a result sequence, delays in determination can be suppressed. In FIG. 9, according to the result sequence before interpolation, the output determination unit 170 can set the output value to "yellow" for the first time at step "t+7". On the other hand, according to the result sequence after interpolation, the output determination unit 170 can set the output value to "yellow" for the first time at step "t+6". In this way, delays in determination can be suppressed.

The traffic light recognition method and traffic light recognition device according to this embodiment may also replace the first judgment result in the result string with the judgment result immediately following the first judgment result if there is no judgment result immediately preceding the first judgment result among the judgment results constituting the result string, excluding the first judgment result. This makes it possible to complement the judgment results even if there is a judgment result at the beginning of the result string indicating that the current status is unknown.

Furthermore, the traffic light recognition method and traffic light recognition device according to this embodiment may replace the first judgment result in the result string with the judgment result immediately preceding the first judgment result if there is no judgment result immediately following the first judgment result among the judgment results constituting the result string, excluding the first judgment result. This makes it possible to complement the judgment results even if there is a judgment result at the beginning of the result string that indicates that the current indication is unknown.

Each of the functions described in the above embodiments may be implemented by one or more processing circuits. Processing circuits include programmed processors, electrical circuits, and even devices such as application specific integrated circuits (ASICs), and circuit components arranged to perform the described functions.

The contents of the present invention have been described above in accordance with the embodiments, but the present invention is not limited to these descriptions, and it will be obvious to those skilled in the art that various modifications and improvements are possible. The descriptions and drawings that form part of this disclosure should not be understood as limiting the present invention. Various alternative embodiments, examples, and operating techniques will be apparent to those skilled in the art from this disclosure.

The present invention naturally includes various embodiments not described here. Therefore, the technical scope of the present invention is determined only by the invention-specific matters related to the scope of the claims that are appropriate from the above explanation.

71 Imaging unit 73 Vehicle-mounted sensor 75 Map information acquisition unit 100 Controller 140 Detection area setting unit 150 Determination unit 160 Determination result storage unit 170 Output determination unit 180 Output unit 400 Vehicle control device

Claims

A traffic light recognition method for controlling a controller connected to an imaging unit mounted on a vehicle, comprising:
The controller:
Using the imaging unit, one or more traffic lights in a traveling direction of the vehicle are imaged a plurality of times to obtain a plurality of images;
A result sequence is obtained for each of the traffic lights, the result sequence being made up of a plurality of judgment results in chronological order, the judgment results being obtained by judging the current state of each of the traffic lights for each of the images;
When there is an interlocking signal among the plurality of signals,
determining whether or not a subsequence consisting of a predetermined number or more of consecutive identical judgment results, including a most recent judgment result, among the judgment results constituting the result sequence related to the interlocking traffic signals, matches between at least two or more of the traffic signals;
A traffic light recognition method characterized in that, when it is determined that the subsequences match, a current status related to the determination result for the matching subsequence is output as a current status based on the traffic light.
The controller, when the interlocking traffic light is present,
2. The traffic light recognition method according to claim 1, characterized in that, when it is determined that the subsequences do not match, the current indication state based on the traffic light is continuously output a number of times not exceeding a first predetermined number of times.
3. The traffic light recognition method according to claim 2, wherein the first predetermined number of times is equal to or less than half the number of judgment results that constitute the result sequence.
When the number of the interlocking traffic lights is three or more, the controller
The traffic light recognition method according to any one of claims 1 to 3, characterized in that a current indication state related to a judgment result in the most frequent subsequence among the subsequences related to the linked traffic lights is output as a current indication state based on the traffic light.
When the number of the traffic lights is 1, the controller
A traffic light recognition method as described in any one of claims 1 to 4, characterized in that the current indication status associated with the most frequent judgment result among the judgment results constituting the result sequence related to the traffic light is output as the current indication status based on the traffic light.
The controller:
determining whether the interlocking traffic signal has changed from a present state to a non-present state;
A traffic light recognition method as claimed in any one of claims 1 to 4, characterized in that when it is determined that the linked traffic light has changed from a present state to a absent state, the current indication state based on the traffic light is continuously output a number of times not exceeding a second predetermined number of times.
7. The traffic light recognition method according to claim 6, wherein the second predetermined number of times is equal to or less than half the number of judgment results that constitute the result sequence.
The traffic light recognition method according to any one of claims 1 to 7, characterized in that the interlocking traffic lights are multiple traffic lights that correspond to the same stop line on the road on which the vehicle is traveling.
The controller:
extracting a determination result indicating that the current state is unknown as a first determination result from among the determination results constituting the result sequence;
extracting, from among the judgment results constituting the result sequence, a judgment result immediately before the first judgment result and a judgment result immediately after the first judgment result as a second judgment result and a third judgment result, respectively;
A traffic light recognition method according to any one of claims 1 to 8, characterized in that when the second judgment result and the third judgment result are identical, the first judgment result in the result sequence is replaced with the second judgment result.
The controller:
extracting a determination result indicating that the current state is unknown as a first determination result from among the determination results constituting the result sequence;
A traffic light recognition method according to any one of claims 1 to 8, characterized in that if there is no judgment result immediately preceding the first judgment result among the judgment results constituting the result sequence, excluding the first judgment result, the first judgment result in the result sequence is replaced with the judgment result immediately following the first judgment result.
The controller:
extracting a determination result indicating that the current state is unknown as a first determination result from among the determination results constituting the result sequence;
A traffic light recognition method according to any one of claims 1 to 8, characterized in that if there is no judgment result immediately following the first judgment result among the judgment results constituting the result sequence, excluding the first judgment result, the first judgment result in the result sequence is replaced with the judgment result immediately preceding the first judgment result.
A traffic light recognition device including an imaging unit and a controller mounted on a vehicle,
The controller:
Using the imaging unit, one or more traffic lights in a traveling direction of the vehicle are imaged a plurality of times to obtain a plurality of images;
A result sequence is obtained for each of the traffic lights, the result sequence being made up of a plurality of judgment results in chronological order, the judgment results being obtained by judging the current state of each of the traffic lights for each of the images;
When there is an interlocking signal among the plurality of signals,
determining whether or not a subsequence consisting of a predetermined number or more of consecutive identical judgment results, including a most recent judgment result, among the judgment results constituting the result sequence related to the interlocking traffic signals, matches between at least two or more of the traffic signals;
A traffic light recognition device characterized in that, when it is determined that the subsequences match, the current status related to the determination result for the matching subsequences is output as the current status based on the traffic light.