Nothing Special   »   [go: up one dir, main page]

WO2020001223A1 - 交通信号灯检测及智能驾驶方法和装置、车辆、电子设备 - Google Patents

交通信号灯检测及智能驾驶方法和装置、车辆、电子设备 Download PDF

Info

Publication number
WO2020001223A1
WO2020001223A1 PCT/CN2019/089062 CN2019089062W WO2020001223A1 WO 2020001223 A1 WO2020001223 A1 WO 2020001223A1 CN 2019089062 W CN2019089062 W CN 2019089062W WO 2020001223 A1 WO2020001223 A1 WO 2020001223A1
Authority
WO
WIPO (PCT)
Prior art keywords
traffic
traffic signal
state
light
attributes
Prior art date
Application number
PCT/CN2019/089062
Other languages
English (en)
French (fr)
Inventor
王贺璋
马宇宸
胡天晓
曾星宇
闫俊杰
Original Assignee
北京市商汤科技开发有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京市商汤科技开发有限公司 filed Critical 北京市商汤科技开发有限公司
Priority to SG11202007333PA priority Critical patent/SG11202007333PA/en
Priority to JP2020550090A priority patent/JP7111827B2/ja
Priority to KR1020207029615A priority patent/KR102447352B1/ko
Publication of WO2020001223A1 publication Critical patent/WO2020001223A1/zh
Priority to US16/944,234 priority patent/US20200353932A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/09623—Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
    • B60W30/18—Propelling the vehicle
    • B60W30/18009—Propelling the vehicle related to particular drive situations
    • B60W30/18154—Approaching an intersection
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
    • B60W30/18—Propelling the vehicle
    • B60W30/18009—Propelling the vehicle related to particular drive situations
    • B60W30/18159—Traversing an intersection
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0221—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory involving a learning process
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G06V20/584—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of vehicle lights or traffic lights
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/04—Detecting movement of traffic to be counted or controlled using optical or ultrasonic detectors
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2552/00—Input parameters relating to infrastructure
    • B60W2552/53—Road markings, e.g. lane marker or crosswalk

Definitions

  • the present disclosure relates to computer vision technology, and more particularly to a method and device for detecting traffic lights and intelligent driving, vehicles, and electronic equipment.
  • Traffic light detection and status determination are important issues in the field of intelligent driving. Traffic lights are important traffic signals and play an irreplaceable role in modern transportation systems. Traffic light detection and status determination can instruct the vehicle to stop and advance during automatic driving to ensure the safe driving of the vehicle.
  • the embodiments of the present disclosure provide a technology for detecting traffic lights and intelligent driving.
  • the detection network includes a region-based full convolution network and a multi-task recognition network, including:
  • a smart driving method including:
  • a traffic light detection device including:
  • a video stream acquiring unit configured to acquire a video stream including a traffic signal light
  • An area determining unit configured to determine a candidate area of a traffic signal light in at least one frame of the video stream
  • An attribute recognition unit is configured to determine at least two attributes of a traffic light in the image based on the candidate area.
  • an intelligent driving device including:
  • a video stream acquisition unit configured to acquire a video stream including a traffic signal based on an image acquisition device provided on a vehicle
  • An area determining unit configured to determine a candidate area of a traffic signal light in at least one frame of the video stream
  • An attribute recognition unit configured to determine at least two attributes of a traffic light in the image based on the candidate area
  • a state determining unit configured to determine a state of the traffic signal light based on at least two attributes of the traffic signal light in the image
  • An intelligent control unit is configured to intelligently control the vehicle according to a state of the traffic signal light.
  • a vehicle including the traffic light detection device according to any one of the above or the intelligent driving device according to any one of the above.
  • an electronic device including a processor, and the processor includes the traffic light detection device according to any one of the above or the intelligent driving device according to any one of the above.
  • an electronic device including: a memory for storing executable instructions;
  • a processor configured to communicate with the memory to execute the executable instructions to complete the operation of the traffic light detection method according to any one of the above, or complete the operation of the intelligent driving method according to any one of the above.
  • a computer-readable storage medium for storing computer-readable instructions, and when the instructions are executed, the traffic signal detection method according to any one of the foregoing or any one of the above is performed.
  • a computer program product including computer-readable code.
  • the computer-readable code runs on a device, a processor in the device executes to implement any of the foregoing.
  • a traffic signal detection and intelligent driving method and device, vehicle, and electronic device are provided to obtain a video stream including the traffic signal; determine a candidate area of the traffic signal in at least one frame of the video stream; based on The candidate area determines at least two attributes of the traffic signal in the image.
  • FIG. 1 is a schematic flowchart of a traffic signal detection method provided by the present disclosure.
  • FIG. 2 is a schematic structural diagram of a traffic light detection device provided by the present disclosure.
  • FIG. 3 is a schematic flowchart of a smart driving method provided by the present disclosure.
  • FIG. 4 is a schematic structural diagram of an intelligent driving device provided by the present disclosure.
  • FIG. 5 is a schematic structural diagram of an electronic device suitable for implementing a terminal device or a server according to an embodiment of the present disclosure.
  • Embodiments of the invention can be applied to a computer system / server, which can operate with many other general or special purpose computing system environments or configurations.
  • Examples of well-known computing systems, environments, and / or configurations suitable for use with computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, based on Microprocessor systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above, and so on.
  • a computer system / server may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system.
  • program modules may include routines, programs, target programs, components, logic, data structures, and so on, which perform specific tasks or implement specific abstract data types.
  • the computer system / server can be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on a local or remote computing system storage medium including a storage device.
  • FIG. 1 is a schematic flowchart of a traffic signal detection method provided by the present disclosure.
  • the method can be executed by any electronic device, such as a terminal device, a server, a mobile device, a vehicle-mounted device, and so on.
  • the method in this embodiment includes:
  • Step 110 Obtain a video stream including a traffic signal.
  • the identification of traffic lights is usually based on the on-board video recorded during the vehicle's travel, and the on-board video is parsed to obtain a video stream including at least one frame of image.
  • the video stream can be installed on the vehicle.
  • the camera device captures the video of the vehicle's forward direction or surrounding environment. If there are traffic lights in the vehicle's forward direction or surrounding environment, it will be captured by the camera device.
  • the captured video stream is the video stream including the traffic signal light.
  • the images in the video stream may include traffic signals in each frame of images, or at least one frame of images may include traffic signals.
  • the step 110 may be executed by the processor calling a corresponding instruction stored in the memory, or may be executed by the video stream obtaining unit 21 executed by the processor.
  • Step 120 Determine a candidate area of a traffic light in at least one frame of the video stream.
  • a candidate region is determined from an image including a traffic signal in the video stream, and the candidate region refers to a region that may include a traffic signal in the image.
  • the detection of the area of the traffic signal can be based on neural networks or other types of detection models.
  • a region-based full convolutional network is used to determine candidate regions of traffic lights in at least one frame of image of the video stream.
  • Region-based, fully convolutional networks are used to detect signal images to obtain candidate regions that may include traffic lights.
  • R-FCN can be regarded as a fast convolutional neural network (Faster Regions (with CNN, Faster RCNN), the detection speed is faster than Faster RCNN.
  • the step 120 may be executed by the processor calling a corresponding instruction stored in the memory, or may be executed by the area determining unit 22 executed by the processor.
  • Step 130 Determine at least two attributes of traffic lights in the image based on the candidate area.
  • the attributes of traffic lights are used to describe traffic lights, which can be defined according to actual needs.
  • the attributes of traffic lights can be used to describe the location or location of traffic lights, such as red, green, and yellow.
  • Etc. attributes, which are used to describe the shape of traffic lights (such as circles, straight arrows, polyline arrows, etc.), and other attributes used to describe other aspects of traffic lights.
  • At least two attributes of the traffic signal light include any two or more of the following: location area, color, and shape.
  • the color of the traffic signal includes three colors of red, yellow, and green
  • the shape includes an arrow shape, a circle, or other shapes.
  • the signals may not be accurately identified. Therefore, in this embodiment, by identifying at least two of a location area, a color, and a shape, for example, when determining a location area and a color of a traffic signal, it is possible to determine a current traffic signal position in the image (corresponding to which direction of the vehicle ),
  • the state of the traffic light display can be determined by color (red, green or yellow corresponding to different states), and the assisted driving or automatic driving can be realized by identifying the different states of the traffic light; when determining the location area and shape of the traffic light, You can determine where the current traffic light is in the image (corresponding to which direction of the vehicle), and determine the status of the traffic light by its shape (for example, arrows pointing in different directions indicate different states, or human figures in different shapes indicate different states );
  • the color of traffic light includes three colors of red, yellow, and green
  • the step 130 may be executed by the processor calling a corresponding instruction stored in the memory, or may be executed by the attribute recognition unit 23 executed by the processor.
  • a method for detecting a traffic signal based on the above embodiments of the present disclosure is to obtain a video stream including a traffic signal; determine a candidate region of the traffic signal in at least one frame of the video stream; and determine at least two of the traffic signal in the image based on the candidate region.
  • This kind of attribute can realize the recognition of various kinds of information of the traffic signal by obtaining at least two attributes of the traffic signal, reduce the recognition time, and improve the accuracy of the traffic signal recognition.
  • operation 130 may include:
  • a multi-task recognition network is used to determine at least two attributes of traffic lights in an image based on candidate regions.
  • At least two attributes of traffic lights are identified through a network. Compared with the case where at least two attributes are identified based on at least two networks, respectively, the size of the network is reduced and the efficiency of attribute recognition of traffic lights is improved. .
  • a multi-task recognition network is used to identify candidate areas that may include traffic lights.
  • the recognition process may include feature extraction and attribute recognition.
  • the multi-task recognition network may include feature extraction branches, and The feature extraction branch is connected to at least two task branches, and different task branches are used to determine different kinds of attributes of the traffic light.
  • Each attribute recognition task requires feature extraction for candidate regions.
  • the feature extraction branches are connected to at least two task branches, so that the feature extraction operations of at least two task branches are combined in the same feature extraction branch.
  • Feature extraction is required for at least two task branches, which reduces the structure of the multi-task recognition network and accelerates the speed of attribute recognition.
  • the process of obtaining at least two attributes may include:
  • the candidate features are processed based on at least two task branches respectively to obtain at least two attributes of traffic lights in the image.
  • the feature extraction branch may include at least one convolution layer, using the candidate region as an input image, and performing feature extraction on the candidate region through the feature extraction branch to obtain candidate features (feature map or feature vector) of the candidate region.
  • candidate features can be obtained by at least two task branches, the position and color of the traffic signal, or the position and shape of the traffic signal, or the color and shape of the traffic signal.
  • the multi-task branch is used to obtain The color, position and shape of the signal light; while checking the position of the signal light, the current state of the signal light can be identified by the color of the signal light, which can be well applied in the field of automatic driving. The accuracy of the signal light recognition can be improved by identifying the shape of the signal light .
  • At least two task branches include, but are not limited to, a detection branch, an identification branch, and a classification branch;
  • the candidate features are processed based on at least two task branches respectively to obtain at least two attributes of traffic lights in the image, including:
  • the position of the candidate feature is detected by the detection branch to determine the location area of the traffic signal;
  • Color classification of candidate features through classification branches, determining the color of the area where the traffic signal is located, and determining the color of the traffic signal;
  • the shape of the candidate feature is identified through the recognition branch, the shape of the area where the traffic signal is located is determined, and the shape of the traffic signal is determined.
  • This embodiment can simultaneously identify any two or three attributes of the location area, color, and shape of the traffic signal through different branches, saving time for multi-task recognition, reducing the size of the detection network, and enabling multi-task recognition.
  • the network is faster in training and application process, and if the location area of the traffic signal is obtained first, the color and shape of the traffic signal can be obtained faster; because the color of the traffic signal is usually only three (red, green and yellow), therefore, for the Color recognition can be implemented using trained classification branches (other network layers except convolution layers in ordinary multi-task recognition networks).
  • the method may further include:
  • the difference between consecutive frames of the video stream may be small, and the location of traffic lights is identified based only on the candidate area of traffic lights in at least one frame of image. It is possible to identify the location area in consecutive frames as the same location area. As a result, the identified location area is inaccurate.
  • key points are identified in the image, the location area of the traffic signal in the image is determined based on the key point, and the traffic signal obtained by the multi-task recognition network is adjusted based on the location area of the key point. Location, improving the accuracy of location area recognition.
  • the key point identification and / or tracking may be implemented based on any one of the existing technologies that can realize key point identification and / or tracking.
  • the tracking of the key points of the traffic lights in the video stream is performed by using a static key point tracking technology to obtain an area where the key points of the traffic lights in the video stream may exist.
  • the location area of the traffic signal lights obtained through the detection branch can easily cause some frames to be missed due to the small gap between consecutive images and the selection of the threshold. Based on the static keypoint tracking technology, the detection effect of the vehicle network on the vehicle video is improved.
  • the characteristic points of an image can be simply understood as the more prominent points in the image, such as corner points, bright points in darker areas, and so on.
  • feature points in the video image are detected and described (Oriented, FAST, Rotated, Brief, ORB) feature points:
  • the definition of the ORB feature points is based on the gray value of the image around the feature points. During detection, a circle around the candidate feature points is considered. If there are enough pixels in the area around the candidate point and the difference between the gray value of the candidate feature point reaches a preset value, the candidate point is considered as a key feature point. In this embodiment, the key points of the traffic signal are identified. Therefore, the key points are the key points of the traffic signal.
  • the key points of the traffic signal can be used to implement static tracking of the traffic signal in the video stream. Not only does it occupy one pixel, that is, the key point of the traffic signal obtained in this embodiment includes at least one pixel, it can be understood that the key point of the traffic signal corresponds to a location area.
  • tracking the key points of the traffic lights in the video stream includes:
  • the two consecutive frames referred to in this embodiment can be two consecutive acquisition frames in the video stream, or two consecutive detection frames in the video stream (because the video stream can be detected frame by frame or can be sampled and detected, so The meanings of both detection frames and acquisition frames are not exactly the same); by associating the key points of traffic lights in multiple consecutive frames of video in the video stream, the key points of traffic lights can be tracked in the video stream, Based on the tracking result, the position and area of at least one frame of image in the video stream can be adjusted.
  • the tracking of the key points of the traffic signal in the video stream can be achieved based on the Hamming distance, the Euclidean distance, the joint Bayesian distance, or the cosine distance between the key points of the traffic signal. Limitation is based on the distance between key points of a traffic light.
  • the Hamming distance is used in data transmission error control coding.
  • the Hamming distance is a concept that represents the number of different bits corresponding to two (same length) words, and performs an exclusive OR operation on the two character strings, and counts them. The result is a number of 1, then this number is the Hamming distance, and the Hamming distance between two images is the number of different data bits between the two images. Based on the Hamming distance between the key points of at least one traffic signal in two frames of signal images, we can know the distance that the signal lights move between the two signal images, and the key points of traffic signals can be tracked.
  • tracking the key points of the traffic signal in the video stream based on the distance between the key points of the traffic signal includes:
  • the key points of the traffic signal are tracked in the video stream.
  • Traffic lights usually do not appear individually, and because traffic lights cannot be represented by a key point in the image, at least one key point of the traffic light is included in the image. For different traffic lights (for example, the Traffic lights, left turn traffic lights) need to be tracked separately. This embodiment overcomes the problem of chaotic tracking of different traffic lights by tracking in consecutive frames based on the key points of the same traffic lights.
  • determining a location area of a key point of the same traffic signal in two consecutive frames of images may be determined based on a smaller value (for example, a minimum value) of a Hamming distance between the key points of at least one traffic signal.
  • the idea of the Brute Force algorithm is to match the first character of the target string S with the first character of the pattern string T. If they are equal, continue to compare the first character of S. Two characters and the second character of T; if they are not equal, then compare the second character of S and the first character of T, and compare them in turn until the final match is obtained.
  • the BruteForce algorithm is a brute force Force algorithm.
  • adjusting the position area of the signal light based on the tracking result includes:
  • the position area of the signal light After adjusting the position area of the signal light based on the tracking result, the position area of the signal light is more stable and more suitable for application in a video scene.
  • the position area corresponding to the key point of the traffic signal light in at least one frame of the video stream can be determined based on the tracking result.
  • the overlapping part between the position area in the tracking result and the position area of the signal light is in the position area of the signal light If the ratio exceeds the set ratio, it can be determined that the tracking result coincides with the location area of the signal light; otherwise, it is determined that the tracking result does not coincide with the location area of the signal light.
  • adjusting the position area of the signal light based on the comparison result includes:
  • the location area corresponding to the key point of the traffic signal is used to replace the location area of the signal light.
  • the comparison result of whether the location area corresponding to the key point of the traffic signal and the location area of the signal light in the signal image are compared can include the following three situations:
  • the location area corresponding to the key point of the traffic signal and the location area of the signal light match (that is, coincidence), that is, the key point location area of the matching traffic signal in the two frames before and after the movement is the same as the detected location area of the signal light, no correction is required; If the location area of the key point of the traffic signal and the location area of the detected signal light are roughly matched, then based on the offset of the location area of the key point of the traffic signal in the previous and subsequent frames, the width and height of the detected signal light remain unchanged.
  • the method may further include:
  • the yellow light in a traffic signal is only a transition state between red and green lights, so the duration of its existence is shorter than that of red and green lights.
  • the detection framework based on R-FCN only inputs a limited number of images at a time, and the number of yellow lights in the image is very small compared to red and green lights. Therefore, it is impossible to effectively train the detection network and improve the sensitivity of the model to yellow lights. Therefore, the present disclosure obtains simultaneous recognition of the position, color, and / or shape of a signal light by training a region-based full convolutional network and a multi-task recognition network.
  • the detection network may further include:
  • the classification network is trained based on the new training image set; the classification network is used to classify the training images based on the color of the traffic lights.
  • the classification network is obtained by removing a candidate regional network (RPN) and a proposal layer from a detection network in the prior art.
  • the classification network may include a multi-task identification network. Feature extraction branch and classification branch; training the classification network based on a new training image set based on a preset scale alone can improve the accuracy of the classification network's color classification of traffic lights.
  • the training image set of the training network is acquired through acquisition, and the R-FCN region-based full convolution network is trained with the acquired training image set; the number of traffic lights and yellow lights in the acquired training image set is adjusted, optionally, preset The number of traffic lights of different colors in the ratio is the same or the number difference is less than the allowable threshold;
  • the color of traffic lights includes red, yellow and green.
  • the ratio of the three colors of red, yellow, and green is preset to the same ratio (such as: red: yellow: green is 1: 1: 1), or the number of red, yellow, and green colors is controlled to be less than the allowable threshold, so that the three The color ratio is close to 1: 1: 1.
  • a new training image set can be constructed by extracting the training signals with corresponding colors from the training image set; or the yellow light images in the training image set can be repeatedly called so that the number of yellow light images is the same as that of the red and green light images. The number is in accordance with the preset ratio), and the classification network is trained with the adjusted new training image set, which overcomes the shortcoming that the number of yellow light images is much smaller than the traffic light image data, and improves the classification network's recognition accuracy of yellow lights.
  • the method before adjusting the parameters of the region-based full convolutional network and the multi-task recognition network based on the training image set, the method further includes:
  • some or all parameters in the multi-task recognition network may be initialized based on the parameters of the trained classification network, for example, the feature extraction branch and classification branch in the multi-task recognition network are initialized with the parameters of the trained classification network;
  • the parameters may include, for example, the size of the convolution kernel, the weight of the convolution connection, and the like.
  • the region-based full convolutional network and multi-task recognition network are trained with the initial training image set.
  • the parameter pairs in the trained classification network are used for detection. Some parameters in the network are initialized.
  • the feature extraction branch and classification branch obtained at this time have a good effect on the color classification of traffic lights and improve the accuracy of yellow light classification.
  • the disclosed traffic signal detection method can be applied in the fields of intelligent driving, high-precision maps, and the like;
  • Car video can be used as input to output the position and status of traffic lights to assist the vehicle's safe driving.
  • It can also be used to build high-precision maps to detect traffic light locations.
  • the method further includes:
  • This embodiment automatically recognizes at least two attributes of the traffic signal and obtains the state of the traffic signal in the video stream, eliminating the need for the driver to distractively observe the traffic signal during the driving process, improving the safety of the vehicle and reducing the risk of human error. Traffic danger caused by mistake.
  • the intelligent driving control includes: sending prompt information or warning information, and / or controlling a driving state of the vehicle according to a state of a traffic signal light.
  • Identifying at least two attributes of traffic lights can provide a basis for intelligent driving.
  • Intelligent driving includes autonomous driving and assisted driving.
  • autonomous driving the driving state of the vehicle is controlled according to the state of the traffic lights (such as: parking, deceleration, steering, etc.)
  • prompt information or warning information to inform the driver of the current status of traffic lights; in the case of assisted driving, usually only the prompt information or warning information is issued, the authority to control the vehicle still belongs to the driver, and the driver according to the prompt The information or warning information controls the vehicle accordingly.
  • the method further includes: storing attributes, states and corresponding images of traffic lights.
  • This embodiment acquires more information (attributes, states, and corresponding images) of the traffic signals by storing the attributes, states, and corresponding images of the traffic signals, and provides more operation basis for intelligent driving. It is also possible to establish a high-precision map based on the time and location corresponding to the stored traffic signal, and determine the location of the traffic lights in the high-precision map based on the image corresponding to the stored traffic signal.
  • the state of the traffic signal light includes, but is not limited to, a traffic permission state, a traffic prohibition state, or a waiting state;
  • the state of the traffic signal is a traffic-permitting state.
  • the state of the traffic signal is a no-traffic state.
  • the state of the traffic signal is a waiting state.
  • the colors of traffic lights include red, green, and yellow, and different colors correspond to different traffic conditions.
  • Red means that vehicles and / or pedestrians are prohibited
  • green means that vehicles and / or pedestrians are allowed
  • yellow means that vehicles and / Or the pedestrian pass needs to pause and wait
  • the auxiliary color can also include shapes such as traffic signals, for example: a plus sign shape (an optional first preset shape) indicates that traffic is allowed, and a fork shape (an optional first The second preset shape) indicates that traffic is prohibited, the minus shape (an optional third preset shape) indicates a waiting state, and the like.
  • performing intelligent driving control on the vehicle according to the state of the traffic signal light includes:
  • control the vehicle In response to the state of the traffic light being a traffic permitted state, control the vehicle to perform one or more operations such as starting, maintaining the driving state, decelerating, turning, turning on the turn signal, turning on the brake light, and other controls needed to control the traffic of the vehicle. ;
  • the color of the traffic signal is green and the shape is a left-pointing arrow
  • the state of operation can achieve safer intelligent driving, improve driving safety, and reduce potential safety hazards caused by human error.
  • the foregoing program may be stored in a computer-readable storage medium.
  • the program is executed, the program is executed.
  • the method includes the steps of the foregoing method embodiment.
  • the foregoing storage medium includes: a ROM, a RAM, a magnetic disk, or an optical disk, and other media that can store program codes.
  • FIG. 2 is a schematic structural diagram of an embodiment of a traffic light detection device according to the present disclosure.
  • the traffic signal detection device of this embodiment may be used to implement the foregoing embodiments of the traffic signal detection methods of the present disclosure.
  • the apparatus of this embodiment includes:
  • the video stream acquiring unit 21 is configured to acquire a video stream including a traffic signal.
  • the identification of traffic lights is usually based on the on-board video recorded during the vehicle's travel, and the on-board video is parsed to obtain a video stream including at least one frame of image.
  • the video stream can be installed on the vehicle.
  • the camera device captures the video of the vehicle's forward direction or surrounding environment. If there are traffic lights in the vehicle's forward direction or surrounding environment, it will be captured by the camera device.
  • the captured video stream is the video stream including the traffic signal light.
  • the images in the video stream may include traffic signals in each frame of images, or at least one frame of images may include traffic signals.
  • the area determining unit 22 is configured to determine a candidate area of a traffic light in at least one frame of an image of a video stream.
  • a candidate region is determined from an image including a traffic signal in the video stream, and the candidate region refers to a region that may include a traffic signal in the image.
  • the detection of the area of the traffic signal can be based on neural networks or other types of detection models.
  • a region-based full convolutional network is used to determine candidate regions of traffic lights in at least one frame of image of the video stream.
  • the region-based full convolutional network (R-FCN) is used to detect the signal image to obtain candidate regions that may include traffic lights.
  • R-FCN can be regarded as an improved version of Faster RCNN, and the detection speed is faster than Faster RCNN.
  • the attribute recognition unit 23 is configured to determine at least two attributes of the traffic signal light in the image based on the candidate region.
  • the attributes of traffic lights are used to describe traffic lights, which can be defined according to actual needs.
  • the attributes of traffic lights can be used to describe the location or location of traffic lights, such as red, green, and yellow.
  • Etc. attributes, which are used to describe the shape of traffic lights (such as circles, straight arrows, polyline arrows, etc.), and other attributes used to describe other aspects of traffic lights.
  • various types of information of a traffic signal are recognized by obtaining at least two attributes of the traffic signal, which reduces the recognition time and improves the accuracy of traffic signal recognition.
  • At least two attributes of the traffic signal light include any two or more of the following: location area, color, and shape.
  • the determination of at least two attributes of a traffic light may be based on a neural network or other type of recognition model.
  • the attribute recognition unit 23 is configured to use a multi-task recognition network to determine at least two attributes of a traffic light in an image based on a candidate region.
  • At least two attributes of traffic lights are identified through a network. Compared with the case where at least two attributes are identified based on at least two networks, respectively, the size of the network is reduced and the efficiency of attribute recognition of traffic lights is improved. .
  • the multi-task recognition network includes a feature extraction branch and at least two task branches respectively connected to the feature extraction branch, and different task branches are used to determine different types of attributes of traffic lights;
  • the attribute recognition unit 23 includes:
  • a feature extraction module configured to perform feature extraction on the candidate region based on the feature extraction branch to obtain candidate features
  • a branch attribute module is configured to process candidate features based on at least two task branches, respectively, to obtain at least two attributes of traffic lights in an image.
  • At least two task branches include, but are not limited to, a detection branch, an identification branch, and a classification branch;
  • the branch attribute module is used to detect the position of candidate features through the detection branch to determine the location area of the traffic signal; to classify the candidate features by color classification to determine the color of the location area of the traffic signal and determine the color of the traffic signal;
  • the branch performs shape recognition on candidate features, determines the shape of the area where the traffic signal is located, and determines the shape of the traffic signal.
  • the method further includes:
  • a key point determining unit configured to identify key points of at least one frame of an image in a video stream, and determine key points of a traffic signal light in the image
  • Key point tracking unit which is used to track the key points of the traffic lights in the video stream to obtain the tracking results
  • a position adjusting unit is configured to adjust a position area of a traffic signal based on a tracking result.
  • the differences between consecutive frames of the video stream may be small.
  • the traffic signal position recognition is based on the candidate signal traffic region in each frame of the image. It is possible to identify the location regions in consecutive frames as the same location region. As a result, the identified location area is inaccurate.
  • keypoints are identified in the image, and the location area of the traffic signal in the image is determined based on the keypoint. Based on the location area of the keypoint, the Location, improving the accuracy of location area recognition.
  • the key point identification and / or tracking may be implemented based on any one of the existing technologies that can realize key point identification and / or tracking.
  • the tracking of the key points of the traffic lights in the video stream is performed by using a static key point tracking technology to obtain an area where the key points of the traffic lights in the video stream may exist.
  • the key point tracking unit is configured to track the key points of the traffic signal in the video stream based on the distance between the key points of the traffic signal in two consecutive frames of images;
  • the two consecutive frames referred to in this embodiment may be two consecutive acquisition frames in the video stream, or two consecutive detection frames in the video stream.
  • the meanings of both detection frames and acquisition frames are not exactly the same); by associating the key points of traffic lights in multiple consecutive frames of video in the video stream, the key points of traffic lights can be tracked in the video stream, Based on the tracking results, the position and area of each frame of the video stream can be adjusted.
  • the tracking of the key points of the traffic signal in the video stream can be achieved based on the Hamming distance, the Euclidean distance, the joint Bayesian distance, or the cosine distance between the key points of the traffic signal. Limitation is based on the distance between key points of a traffic light.
  • the key point tracking unit tracks the key points of the traffic signal in the video stream based on the distance between the key points of the traffic signal, it is used to determine two consecutive frames of images based on the distance between the key points of the traffic signal.
  • the location area of the key points of the same traffic signal in the video; according to the location area of the key points of the same traffic signal in two consecutive frames, the key points of the traffic signal are tracked in the video stream.
  • the position adjustment unit is configured to compare whether the tracking result coincides with the position area of the signal light to obtain a comparison result; and adjust the position area of the signal light based on the comparison result.
  • the position area of the signal light After adjusting the position area of the signal light based on the tracking result, the position area of the signal light is more stable and more suitable for application in a video scene.
  • the position area corresponding to the key point of the traffic signal light in at least one frame of the video stream can be determined based on the tracking result.
  • the overlapping part between the position area in the tracking result and the position area of the signal light is in the position area of the signal light If the ratio exceeds the set ratio, it can be determined that the tracking result coincides with the location area of the signal light; otherwise, it is determined that the tracking result does not coincide with the location area of the signal light.
  • the position adjustment unit adjusts the position area of the signal light based on the comparison result, the position area corresponding to the key point of the traffic signal does not coincide with the position area of the signal light, and the position area corresponding to the key point of the traffic signal is not coincident Replace the location area of the semaphore.
  • it may further include:
  • a pre-training unit configured to train a region-based full convolutional network based on the acquired training image set, where the training image set includes multiple training images with labeled attributes;
  • a training unit for adjusting parameters in a region-based full convolutional network and a multi-task recognition network based on a training image set.
  • the yellow light in a traffic light is only a transition state between red and green lights, so it exists for a shorter period of time than red and green lights.
  • the detection framework based on R-FCN only inputs a limited number of images at a time, and the number of yellow lights in the image is very small compared to red and green lights. Therefore, it is impossible to effectively train the detection network and improve the sensitivity of the model to yellow lights. Therefore, the present disclosure obtains simultaneous recognition of the position, color, and / or shape of a signal light by training a region-based full convolutional network and a multi-task recognition network.
  • the pre-training unit and the training unit may further include:
  • a classification training unit is used to obtain a new training image set whose color proportion of traffic lights is in accordance with a preset ratio based on the training image set; to train a classification network based on the new training image set; the classification network is used to classify the training images based on the color of the traffic signal.
  • the number of traffic lights of different colors in the preset ratio is the same or the number difference is less than the allowable threshold
  • the color of traffic lights includes red, yellow and green.
  • the ratio of the three colors of red, yellow, and green is preset to the same ratio (such as: red: yellow: green is 1: 1: 1), or the number of red, yellow, and green colors is controlled to be less than the allowable threshold, so that the three The color ratio is close to 1: 1: 1.
  • a new training image set can be constructed by extracting the training signals with corresponding colors from the training image set; or the yellow light images in the training image set can be repeatedly called so that the number of yellow light images is the same as that of the red and green light images. The number is in accordance with the preset ratio), and the classification network is trained with the adjusted new training image set, which overcomes the shortcoming that the number of yellow light images is much smaller than the traffic light image data, and improves the classification network's recognition accuracy of yellow lights.
  • the method may further include:
  • An initialization unit is configured to initialize at least some parameters in the multi-task recognition network based on the parameters of the trained classification network.
  • the apparatus in this embodiment may further include:
  • a state determining unit configured to determine a state of a traffic signal based on at least two attributes of the traffic signal in the image
  • the intelligent control unit is used for intelligent driving control of the vehicle according to the state of the traffic signal light.
  • This embodiment automatically recognizes at least two attributes of the traffic signal and obtains the state of the traffic signal in the video stream, eliminating the need for the driver to distractively observe the traffic signal during the driving process, improving the safety of the vehicle and reducing the risk of human error. Traffic danger caused by mistake.
  • the intelligent driving control includes: sending prompt information or warning information, and / or controlling a driving state of the vehicle according to a state of a traffic signal light.
  • it further includes:
  • the storage unit is configured to store attributes, states and corresponding images of traffic lights.
  • the state of the traffic signal light includes, but is not limited to, a traffic permission state, a traffic prohibition state, or a waiting state;
  • a state determining unit configured to determine that the state of the traffic signal is a traffic permitted state in response to the color of the traffic signal being green and / or the shape being a first preset shape
  • the state of the traffic signal is a waiting state.
  • the intelligent control unit is configured to control the vehicle to perform one or more operations of starting, maintaining a driving state, decelerating, turning, turning on a turn signal, and turning on a brake light in response to a state of a traffic signal light being an allowed traffic state;
  • one or more operations of controlling the vehicle to stop, decelerate, and turn on the brake light are controlled.
  • FIG. 3 is a flowchart of an embodiment of a smart driving method according to the present disclosure. As shown in FIG. 3, the method in this embodiment includes:
  • Step 310 Obtain a video stream including a traffic signal based on an image acquisition device provided on the vehicle.
  • the on-board video is parsed to obtain a video stream including at least one frame of the image, for example, the forward or surrounding of the vehicle can be captured by a camera device installed on the vehicle
  • the video of the environment if there are traffic lights in the forward direction of the vehicle or in the surrounding environment, will be captured by the camera device, and the captured video stream is a video stream including traffic lights.
  • the images in the video stream may include traffic signals in each frame of images, or at least one frame of images may include traffic signals.
  • this step 310 may be executed by the processor calling a corresponding instruction stored in the memory, or may be executed by the video stream obtaining unit 21 executed by the processor.
  • Step 320 Determine a candidate area of a traffic signal in at least one frame of the video stream.
  • step 320 may be executed by the processor by calling a corresponding instruction stored in the memory, or may be executed by the area determining unit 22 executed by the processor.
  • Step 330 Determine at least two attributes of a traffic light in the image based on the candidate area.
  • the attributes of traffic lights are used to describe traffic lights, which can be defined according to actual needs.
  • the attributes of traffic lights can be used to describe the location or location of traffic lights, such as red, green, and yellow.
  • Etc. attributes, used to describe the shape of traffic lights (such as circles, straight arrows, polyline arrows, etc.) attributes, and other attributes used to describe other aspects of traffic lights.
  • At least two attributes of the traffic signal light include any two or more of the following: location area, color, and shape.
  • the color of the traffic signal includes three colors of red, yellow, and green
  • the shape includes an arrow shape, a circle, or other shapes.
  • the signals may not be accurately identified. Therefore, in this embodiment, by identifying at least two of a location area, a color, and a shape, for example, when determining a location area and a color of a traffic signal, it is possible to determine a current traffic signal position in the image (corresponding to which direction of the vehicle ),
  • the state of the traffic light display can be determined by color (red, green or yellow corresponding to different states), and the assisted driving or automatic driving can be realized by identifying the different states of the traffic light; when determining the location area and shape of the traffic light, You can determine where the current traffic light is in the image (corresponding to which direction of the vehicle), and determine the status of the traffic light by its shape (for example, arrows pointing in different directions indicate different states, or human figures in different shapes indicate different states );
  • the color of traffic light includes three colors of red, yellow, and green
  • step 330 may be executed by the processor by calling a corresponding instruction stored in the memory, or may be executed by the attribute recognition unit 23 executed by the processor.
  • Step 340 Determine the state of the traffic signal based on at least two attributes of the traffic signal in the image.
  • Existing image processing methods usually can only deal with one task (for example, one of location recognition or color classification), and the traffic light includes information such as location area, color, and shape.
  • the status of the traffic light needs to be determined It is not only necessary to determine the location area of traffic signals, but also at least the color or shape. Therefore, if the general image processing method is applied, at least two neural networks are required to process the video stream, and the processing results need to be synthesized in order to Determine the current state of the traffic signal; this embodiment simultaneously obtains at least two attributes of the traffic signal, determines the state of the traffic signal with at least two attributes, and quickly and accurately identifies the state of the traffic signal.
  • this step 340 may be executed by the processor calling a corresponding instruction stored in the memory, or may be executed by the state determination unit 44 executed by the processor.
  • Step 350 Perform intelligent driving control on the vehicle according to the state of the traffic signal light.
  • step 350 may be executed by the processor by calling corresponding instructions stored in the memory, or may be executed by the intelligent control unit 45 executed by the processor.
  • an image acquisition device on a vehicle can obtain a video stream in real time, and realize the real-time identification of the attributes of traffic lights to determine the status of the traffic lights. Based on the status of the traffic lights, intelligent driving is achieved without the need for the driver to distract from observation during the driving process. Traffic lights reduce the hidden dangers of traffic safety, and to a certain extent reduce the danger of traffic caused by human error. Intelligent driving can include assisted driving and autonomous driving. Generally, assisted driving uses warning lights for warning and automatic driving uses signals for driving control.
  • the intelligent driving control includes: sending prompt information or warning information, and / or controlling a driving state of the vehicle according to a state of a traffic signal light.
  • Identifying at least two attributes of traffic lights can provide a basis for intelligent driving.
  • Intelligent driving includes autonomous driving and assisted driving.
  • autonomous driving the driving state of the vehicle is controlled according to the state of the traffic lights (such as: parking, deceleration, steering, etc.)
  • prompt information or warning information to inform the driver of the current status of traffic lights; in the case of assisted driving, usually only the prompt information or warning information is issued, the authority to control the vehicle still belongs to the driver, and the driver according to the prompt The information or warning information controls the vehicle accordingly.
  • the intelligent driving method provided in the embodiment of the present application further includes:
  • This embodiment acquires more information (attributes, states, and corresponding images) of the traffic signals by storing the attributes, states, and corresponding images of the traffic signals, and provides more operation basis for intelligent driving. It is also possible to establish a high-precision map based on the time and location corresponding to the stored traffic signal, and determine the location of the traffic lights in the high-precision map based on the image corresponding to the stored traffic signal.
  • the states of the traffic signal light include, but are not limited to, a traffic permitted state, a traffic prohibited state, and a waiting state;
  • Step 340 may include:
  • determining that the state of the traffic signal is a state of allowing traffic In response to that the color of the traffic signal is green and / or the shape is a first preset shape, determining that the state of the traffic signal is a state of allowing traffic;
  • the state of the traffic signal is a waiting state.
  • the colors of traffic lights include red, green, and yellow, and different colors correspond to different traffic states.
  • Red means that vehicles and / or pedestrians are not allowed to pass
  • green means that vehicles and / or pedestrians are allowed to pass
  • yellow means that vehicles and / Or pedestrians need to pause and wait
  • auxiliary colors can also include shapes such as traffic signals, such as: plus shape (an optional first preset shape) indicates that traffic is allowed, fork shape (an optional The second preset shape) indicates that traffic is prohibited, the minus shape (an optional third preset shape) indicates a waiting state, and the like.
  • step 350 may include:
  • control the vehicle In response to the state of the traffic light being a traffic permitted state, control the vehicle to perform one or more operations of starting, keeping driving, decelerating, turning, turning on the turn signal, turning on the brake light, and controlling other controls required during the passage of the vehicle ;
  • the foregoing program may be stored in a computer-readable storage medium.
  • the program is executed, the program is executed.
  • the method includes the steps of the foregoing method embodiment.
  • the foregoing storage medium includes: a ROM, a RAM, a magnetic disk, or an optical disk, and other media that can store program codes.
  • FIG. 4 is a schematic structural diagram of an embodiment of an intelligent driving device according to the present disclosure.
  • the intelligent driving device of this embodiment may be used to implement the foregoing intelligent driving method embodiments of the present disclosure.
  • the apparatus of this embodiment includes:
  • the video stream acquiring unit 21 is configured to acquire a video stream including a traffic signal based on an image acquisition device provided on a vehicle.
  • the on-board video is parsed to obtain a video stream including at least one frame of the image, for example, the forward or surrounding of the vehicle can be captured by a camera device installed on the vehicle
  • the video of the environment if there are traffic lights in the forward direction of the vehicle or in the surrounding environment, will be captured by the camera device, and the captured video stream is a video stream including traffic lights.
  • the images in the video stream may include traffic signals in each frame of images, or at least one frame of images may include traffic signals.
  • the area determining unit 22 is configured to determine a candidate area of a traffic light in at least one frame of an image of a video stream.
  • the attribute recognition unit 23 is configured to determine at least two attributes of the traffic signal light in the image based on the candidate region.
  • the attributes of traffic lights are used to describe traffic lights, which can be defined according to actual needs.
  • the attributes of traffic lights can be used to describe the location or location of traffic lights, such as red, green, and yellow.
  • Etc. attributes, used to describe the shape of traffic lights (such as circles, straight arrows, polyline arrows, etc.) attributes, and other attributes used to describe other aspects of traffic lights.
  • the state determining unit 44 is configured to determine a state of the traffic signal light based on at least two attributes of the traffic signal light in the image.
  • Existing image processing methods usually can only deal with one task (for example, one of location recognition or color classification), and the traffic light includes information such as location area, color, and shape.
  • the status of the traffic light needs to be determined It is not only necessary to determine the location area of traffic signals, but also at least the color or shape. Therefore, if the general image processing method is applied, at least two neural networks are required to process the video stream, and the processing results need to be synthesized in order to Determine the current state of the traffic signal; this embodiment simultaneously obtains at least two attributes of the traffic signal, determines the state of the traffic signal with at least two attributes, and quickly and accurately identifies the state of the traffic signal.
  • the intelligent control unit 45 is configured to perform intelligent driving control on the vehicle according to the state of the traffic signal light.
  • an image acquisition device on a vehicle can obtain a video stream in real time, and realize the real-time identification of the attributes of traffic lights to determine the status of the traffic lights. Based on the status of the traffic lights, intelligent driving is achieved without the need for the driver to distract and observe Traffic lights reduce the hidden dangers of traffic safety, and to a certain extent reduce the danger of traffic caused by human error.
  • Intelligent driving can include assisted driving and autonomous driving. Generally, assisted driving uses warning lights for warning and alerting, and automatic driving uses signaling lights for driving control.
  • the intelligent driving control includes: sending prompt information or warning information, and / or controlling a driving state of the vehicle according to a state of a traffic signal light.
  • it further includes:
  • the storage unit is configured to store attributes, states, and corresponding images of traffic lights.
  • At least two attributes of the traffic signal light include any two or more of the following: location area, color, and shape.
  • the states of the traffic signal light include, but are not limited to, a traffic permitted state, a traffic prohibited state, and a waiting state;
  • the state determining unit 44 is configured to determine that the state of the traffic signal is a traffic permitted state in response to the color of the traffic signal being green and / or the shape being a first preset shape;
  • the state of the traffic signal is a waiting state.
  • the intelligent control unit 45 is configured to control the vehicle to perform one or more operations of starting, maintaining a driving state, decelerating, turning, turning on a turn signal, and turning on a brake light in response to a state of a traffic signal being an allowed traffic state. ;
  • one or more operations of controlling the vehicle to stop, decelerate, and turn on the brake light are controlled.
  • a vehicle including the traffic light detection device according to any one of the above embodiments or the intelligent driving device according to any one of the above embodiments.
  • an electronic device including a processor, where the processor includes the traffic light detection device according to any one of the above or the intelligent driving device according to any one of the above embodiments.
  • an electronic device including: a memory for storing executable instructions;
  • a processor configured to communicate with the memory to execute the executable instructions to complete the operation of the traffic signal detection method according to any one of the above embodiments, or to complete the operation of the intelligent driving method according to any one of the above embodiments.
  • An embodiment of the present disclosure further provides an electronic device, which may be, for example, a mobile terminal, a personal computer (PC), a tablet computer, a server, and the like.
  • an electronic device which may be, for example, a mobile terminal, a personal computer (PC), a tablet computer, a server, and the like.
  • FIG. 5 illustrates a schematic structural diagram of an electronic device 500 suitable for implementing a terminal device or a server of an embodiment of the present disclosure.
  • the electronic device 500 includes one or more processors and a communication unit.
  • the one or more processors are, for example, one or more central processing units (CPUs) 501, and / or one or more image processors (GPUs) 513, etc.
  • CPUs central processing units
  • GPUs image processors
  • the processors may be stored in a read-only memory (ROM) 502 or executable instructions loaded from the storage section 508 into the random access memory (RAM) 503 to perform various appropriate actions and processes.
  • the communication unit 512 may include, but is not limited to, a network card, and the network card may include, but is not limited to, an IB (Infiniband) network card.
  • the processor may communicate with the read-only memory 502 and / or the random access memory 503 to execute executable instructions, connect to the communication unit 512 through the bus 504, and communicate with other target devices via the communication unit 512, thereby completing the embodiments of the present disclosure.
  • An operation corresponding to any of the methods is, for example, acquiring a video stream including a traffic signal; determining a candidate region of a traffic signal in at least one frame of an image of the video stream; and determining at least two attributes of the traffic signal in the image based on the candidate region.
  • RAM 503 can also store various programs and data required for the operation of the device.
  • the CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504.
  • ROM502 is an optional module.
  • the RAM 503 stores executable instructions, or writes executable instructions to the ROM 502 at runtime, and the executable instructions cause the central processing unit 501 to perform operations corresponding to the foregoing communication method.
  • An input / output (I / O) interface 505 is also connected to the bus 504.
  • the communication unit 512 may be provided in an integrated manner, or may be provided with a plurality of sub-modules (for example, a plurality of IB network cards) and connected on a bus link.
  • the following components are connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the speaker; a storage portion 508 including a hard disk and the like ; And a communication section 509 including a network interface card such as a LAN card, a modem, and the like.
  • the communication section 509 performs communication processing via a network such as the Internet.
  • the driver 510 is also connected to the I / O interface 505 as necessary.
  • a removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.
  • FIG. 5 is only an optional implementation manner. In the specific practice process, the number and types of the components in FIG. 5 can be selected, deleted, added or replaced according to actual needs. Different functional component settings can also be implemented in separate settings or integrated settings. For example, GPU513 and CPU501 can be set separately or GPU513 can be integrated on CPU501. Communication unit 512 can be set separately or integrated on CPU501 or GPU513. ,and many more. These alternative embodiments all fall within the protection scope of the present disclosure.
  • embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing a method shown in a flowchart, and the program code may include a corresponding Executing instructions corresponding to the method steps provided in the embodiments of the present disclosure, for example, acquiring a video stream including traffic lights; determining candidate areas of traffic lights in at least one frame of video of the video stream; and determining at least two traffic lights in the image based on the candidate areas Kinds of attributes.
  • the computer program may be downloaded and installed from a network through the communication section 509, and / or installed from a removable medium 511.
  • a central processing unit (CPU) 501 When the computer program is executed by a central processing unit (CPU) 501, operations of the above-mentioned functions defined in the method of the present disclosure are performed.
  • a computer-readable storage medium for storing computer-readable instructions, and when the instructions are executed, the traffic signal detection method according to any one of the foregoing or any one of the foregoing is performed.
  • a computer program product including computer-readable code.
  • the computer-readable code runs on a device
  • a processor in the device executes the program to implement The instructions of the traffic signal detection method or the intelligent driving method according to any one of the above.
  • the method and apparatus of the present invention may be implemented by software, hardware, firmware or any combination of software, hardware, firmware.
  • the above-mentioned order of the steps of the method is for illustration only, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated.
  • the present invention can also be implemented as programs recorded in a recording medium, which programs include machine-readable instructions for implementing the method according to the present invention.
  • the present invention also covers a recording medium storing a program for executing the method according to the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Automation & Control Theory (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Traffic Control Systems (AREA)
  • Image Analysis (AREA)
  • Lighting Device Outwards From Vehicle And Optical Signal (AREA)

Abstract

本公开实施例公开了一种交通信号灯检测及智能驾驶方法和装置、车辆、电子设备，其中，交通信号灯检测方法，包括：获取包括有交通信号灯的视频流；确定所述视频流的至少一帧图像中交通信号灯的候选区域；基于所述候选区域确定所述图像中交通信号灯的至少两种属性。

Description

交通信号灯检测及智能驾驶方法和装置、车辆、电子设备
本申请要求在2018年6月29日提交中国专利局、申请号为CN201810697683.9、发明名称为“交通信号灯检测及智能驾驶方法和装置、车辆、电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本公开涉及计算机视觉技术，尤其是一种交通信号灯检测及智能驾驶方法和装置、车辆、电子设备。
背景技术
红绿灯检测及其状态判定是智能驾驶领域的重要问题。红绿灯是重要的交通信号，在现代交通系统中有着不可替代的作用。红绿灯检测及其状态判定可以在自动驾驶中指示车辆的停止、前进，保证车辆的安全行驶。
发明内容
本公开实施例提供一种交通信号灯检测及智能驾驶技术。
本公开实施例的一方面，提供了一种交通信号灯检测方法，所述检测网络包括：基于区域的全卷积网络和多任务识别网络，包括：
获取包括有交通信号灯的视频流；
确定所述视频流的至少一帧图像中交通信号灯的候选区域；
基于所述候选区域确定所述图像中交通信号灯的至少两种属性。
根据本公开实施例的另一个方面，提供了一种智能驾驶方法，包括：
基于设置在车辆上的图像采集装置获取包括有交通信号灯的视频流；
确定所述视频流的至少一帧图像中交通信号灯的候选区域；
基于所述候选区域确定所述图像中交通信号灯的至少两种属性；
基于所述图像中交通信号灯的至少两种属性确定所述交通信号灯的状态；
根据所述交通信号灯的状态对所述车辆进行智能控制。
根据本公开实施例的又一个方面，提供了一种交通信号灯检测装置，包括：
视频流获取单元，用于获取包括有交通信号灯的视频流；
区域确定单元，用于确定所述视频流的至少一帧图像中交通信号灯的候选区域；
属性识别单元，用于基于所述候选区域确定所述图像中交通信号灯的至少两种属性。
本公开实施例的还一方面，提供了一种智能驾驶装置，包括：
视频流获取单元，用于基于设置在车辆上的图像采集装置获取包括有交通信号灯的视频流；
区域确定单元，用于确定所述视频流的至少一帧图像中交通信号灯的候选区域；
属性识别单元，用于基于所述候选区域确定所述图像中交通信号灯的至少两种属性；
状态确定单元，用于基于所述图像中交通信号灯的至少两种属性确定所述交通信号灯的状态；
智能控制单元，用于根据所述交通信号灯的状态对所述车辆进行智能控制。
本公开实施例的再一方面，提供了一种车辆，包括如上任意一项所述的交通信号灯检测装置或如上任意一项所述的智能驾驶装置。
本公开实施例的再一方面，提供了一种电子设备，包括处理器，所述处理器包括如上任意一项所述的交通信号灯检测装置或如上任意一项所述的智能驾驶装置。
本公开实施例的另一方面，提供了一种电子设备，包括：存储器，用于存储可执行指令；
以及处理器，用于与所述存储器通信以执行所述可执行指令从而完成如上任意一项所述交通信号灯检测方法的操作，或完成如上任意一项所述的智能驾驶方法的操作。
本公开实施例的又一方面，提供了一种计算机可读存储介质，用于存储计算机可读取的指令，所述指令被执行时执行如上任意一项所述交通信号灯检测方法或如上任意一项所述的智能驾驶方法的操作。
本公开实施例的另一方面，提供了一种计算机程序产品，包括计算机可读代码，当所述计算机可读代码在设备上运行时，所述设备中的处理器执行用于实现如上任意一项所述交通信号灯检测方法或如上任意一项所述的智能驾驶方法的指令。
基于本公开上述实施例提供的一种交通信号灯检测及智能驾驶方法和装置、车辆、电子设备，获取包括有交通信号灯的视频流；确定视频流的至少一帧图像中交通信号灯的候选区域；基于候选区域确定图像中交通信号灯的至少两种属性，通过获得交通信号灯的至少两种属性实现对信号灯的多种信息的识别，减少了识别时间，并提高了交通信号灯识别的准确率。
下面通过附图和实施例，对本公开的技术方案做进一步的详细描述。
附图说明
构成说明书的一部分的附图描述了本发明的实施例，并且连同描述一起用于解释本发明的原理。
参照附图，根据下面的详细描述，可以更加清楚地理解本发明，其中：
图1为本公开提供的交通信号灯检测方法的一个流程示意图。
图2为本公开提供的交通信号灯检测装置的一个结构示意图。
图3为本公开提供的智能驾驶方法的一个流程示意图。
图4为本公开提供的智能驾驶装置的一个结构示意图。
图5为适于用来实现本公开实施例的终端设备或服务器的电子设备的结构示意图。
具体实施方式
现在将参照附图来详细描述本发明的各种示例性实施例。应注意到：除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本发明的范围。
同时，应当明白，为了便于描述，附图中所示出的各个部分的尺寸并不是按照实际的比例关系绘制的。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本发明及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为说明书的一部分。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
本发明实施例可以应用于计算机系统/服务器，其可与众多其它通用或专用计算系统环境或配置一起操作。适于与计算机系统/服务器一起使用的众所周知的计算系统、环境和/或配置的例子包括但不限于：个人计算机系统、服务器计算机系统、瘦客户机、厚客户机、手持或膝上设备、基于微处理器的系统、机顶盒、可编程消费电子产品、网络个人电脑、小型计算机系统﹑大型计算机系统和包括上述任何 系统的分布式云计算技术环境，等等。
计算机系统/服务器可以在由计算机系统执行的计算机系统可执行指令(诸如程序模块)的一般语境下描述。通常，程序模块可以包括例程、程序、目标程序、组件、逻辑、数据结构等等，它们执行特定的任务或者实现特定的抽象数据类型。计算机系统/服务器可以在分布式云计算环境中实施，分布式云计算环境中，任务是由通过通信网络链接的远程处理设备执行的。在分布式云计算环境中，程序模块可以位于包括存储设备的本地或远程计算系统存储介质上。
图1为本公开提供的交通信号灯检测方法的一个流程示意图。该方法可以由任意电子设备执行，例如终端设备、服务器、移动设备、车载设备等等。如图1所示，该实施例方法包括：
步骤110，获取包括有交通信号灯的视频流。
可选地，对于交通信号灯的识别，通常是以车辆行进过程中记录的车载视频为基础进行的，对车载视频进行解析，获得包括至少一帧图像的视频流，例如可通过安装在车辆上的摄像装置拍摄车辆前向或周围环境的视频，如果车辆前向或周围环境中存在交通信号灯，则会被摄像装置拍摄到，所拍摄的视频流即为包括有交通信号灯的视频流。该视频流中的图像可以每帧图像都包括交通信号灯，或至少有一帧图像中包括交通信号灯。
在一个可选示例中，该步骤110可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的视频流获取单元21执行。
步骤120，确定视频流的至少一帧图像中交通信号灯的候选区域。
可选地，从视频流中包括有交通信号灯的图像中确定候选区域，该候选区域指在图像中可能包括交通信号灯的区域。
交通信号灯的区域的检测可基于神经网络或其他类型的检测模型进行。
在一个或多个可选的实施例中，利用基于区域的全卷积网络，确定视频流的至少一帧图像中交通信号灯的候选区域。通过基于区域的全卷积网络(region-based,fully convolutional networks，R-FCN)对信号图像进行检测，得到可能包括交通信号灯的候选区域，R-FCN可以看做是快速卷积神经网络(Faster Regions with CNN，Faster RCNN)的改进版，检测速度相对Faster RCNN有加快。
在一个可选示例中，该步骤120可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的区域确定单元22执行。
步骤130，基于候选区域确定图像中交通信号灯的至少两种属性。
交通信号灯的属性用于描述交通信号灯，可根据实际需要进行定义，例如，可以包括用于描述交通信号灯绝对位置或相对位置的位置区域属性，用于描述交通信号灯的颜色(如红色、绿色、黄色等)属性，用于描述交通信号灯的形状的(如圆形、直线箭头、折线箭头等)属性，以及用于描述交通信号灯的其他方面的其他属性等。
可选地，交通信号灯的至少两种属性包括以下任意两种或两种以上：位置区域、颜色和形状。
可选地，交通信号灯的颜色包括红、黄、绿三种颜色，形状包括箭头形、圆形或其他形状等，对于不同形状的交通信号灯，如果仅识别其位置，可能无法准确的将信号识别出来，因此，本实施例通过识别位置区域、颜色和形状中的至少两种，例如：当确定交通信号灯的位置区域和颜色，即可确定当前交通信号灯在图像中哪个位置(对应车辆的哪个方向)，通过颜色即可确定交通信号灯显示的状态(红色、绿色或黄色分别对应不同状态)，通过识别到交通信号灯的不同状态可实现辅助驾驶或自动驾驶；当确定交通信号灯的位置区域和形状，即可确定当前交通信号灯在图像中哪个位置(对应车辆的哪个方向)，通过形状即可确定交通信号灯显示的状态(例如：朝向不同方向的箭头表示不同状态，或不同形状的人体图形表示不同状态)；当确定交通信号灯的颜色和形状，可基于颜色和形状相结合确定当前交通信号 灯的状态(例如：指向左侧的绿色箭头表示左转通行，指向前方的红色箭头表示前方禁行)；而当确定交通信号灯的位置区域、颜色和形状时，在获得交通信号灯在图像中哪个位置的基础上，还可以基于颜色和形状相结合确定当前交通信号灯的状态，本实施例通过这三种属性中的两种或两种以上组合，可更突出交通信号灯的属性特征，有利于提高检测、识别等处理效果。
在一个可选示例中，该步骤130可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的属性识别单元23执行。
基于本公开上述实施例提供的一种交通信号灯检测方法，获取包括有交通信号灯的视频流；确定视频流的至少一帧图像中交通信号灯的候选区域；基于候选区域确定图像中交通信号灯的至少两种属性，通过获得交通信号灯的至少两种属性实现对信号灯的多种信息的识别，减少了识别时间，并提高了交通信号灯识别的准确率。
交通信号灯的至少两种属性的确定可基于神经网络或其他类型的识别模型进行。在一个或多个可选的实施例中，操作130可以包括：
利用多任务识别网络，基于候选区域确定图像中交通信号灯的至少两种属性。
本实施例通过一个网络实现对交通信号灯的至少两种属性进行识别，相对于分别基于至少两个网络识别至少两种属性的情况，减小了网络的大小，提高了交通信号灯的属性识别的效率。
通过多任务识别网络对可能包括交通信号灯的候选区域进行识别，识别的过程可以包括特征提取和属性识别两个部分，为了实现这两部分功能，多任务识别网络可以包括特征提取分支、以及分别与特征提取分支连接的至少两个任务分支，不同的任务分支用于确定交通信号灯的不同种类属性。
每个属性识别任务都需要对候选区域进行特征提取，本实施例将特征提取分支分别于至少两个任务分支相连接，使至少两个任务分支的特征提取操作合并在同一特征提取分支进行，不需要对至少两个任务分支分别进行特征提取，缩减了多任务识别网络的结构，加快了属性识别的速度。
可选地，获得至少两种属性的过程可以包括：
基于特征提取分支对候选区域进行特征提取，得到候选特征；
分别基于至少两个任务分支对候选特征进行处理，获得图像中交通信号灯的至少两种属性。
可选地，特征提取分支可以包括至少一层卷积层，将候选区域作为输入图像，通过特征提取分支对候选区域进行特征提取，得到候选区域的候选特征(特征图或者特征向量)，基于该候选特征，通过至少两个任务分支可得到交通信号灯的位置和颜色，或交通信号灯的位置和形状，或交通信号灯的颜色和形状，在一个效果较好的实施例中，通过多任务分支同时得到信号灯的颜色、位置和形状；实现在检查信号灯位置的同时，通过信号灯颜色识别到当前信号灯的状态，在自动驾驶领域可以得到很好的应用，通过对信号灯形状的识别可提高信号灯识别的准确率。
可选地，至少两个任务分支包括但不限于：检测分支、识别分支和分类分支；
分别基于至少两个任务分支对候选特征进行处理，获得图像中交通信号灯的至少两种属性，包括：
经检测分支对候选特征进行位置检测，确定交通信号灯的位置区域；
经分类分支对候选特征进行颜色分类，确定交通信号灯所在位置区域的颜色，确定交通信号灯的颜色；
经识别分支对候选特征进行形状识别，确定交通信号灯所在位置区域的形状，确定交通信号灯的形状。
本实施例可以通过不同分支同时实现对交通信号灯的位置区域、颜色、形状中的任意两种或三种属性进行识别，节省了多任务识别的时间，缩小了检测网络的大小，使多任务识别网络在训练和应用过程都更快，而如果先获得交通信号灯的位置区域，可以更快的获得交通信号灯的颜色和形状；由于信号灯 颜色通常只有三种(红色、绿色和黄色)，因此，对于颜色的识别可采用训练后的分类分支(可以采用普通多任务识别网络中除了卷积层的其他网络层)实现。
实际场景中的红绿灯检测及其状态判定十分困难，首先，受光照、天气等环境因素的干扰，红绿灯的颜色判定十分困难，并且，复杂的现实场景中存在较多相似的干扰，如车灯、路灯等，对红绿灯的检测造成影响。基于本公开上述实施例同时检测交通信号灯的位置区域、颜色、形状中的两种以上，在节省检测时间的基础上，提高了检测的准确度。
在一个或多个可选的实施例中，步骤120之前，还可以包括：
对视频流中的至少一帧图像进行关键点识别，确定图像中的交通信号灯的关键点；
对视频流中的交通信号灯的关键点进行跟踪，得到跟踪结果；
基于跟踪结果对交通信号灯的位置区域进行调整。
在视频流的连续帧之间存在的差异可能很小，仅基于至少一帧图像中交通信号灯的候选区域进行交通信号灯的位置识别，有可能将连续帧中的位置区域识别成相同的位置区域，从而导致识别到的位置区域不准确，本实施例通过图像中进行关键点识别，基于关键点确定交通信号灯在图像中的位置区域，基于该关键点的位置区域调整多任务识别网络获得的交通信号灯的位置，提高了位置区域识别的准确率。
关键点识别和/或跟踪可基于现有技术中可实现关键点识别和/或跟踪的任意一个技术实现。可选地，通过基于静态关键点跟踪技术实现对视频流中的交通信号灯的关键点的跟踪，以获得在视频流中交通信号灯的关键点可能存在的区域。
经检测分支得到的交通信号灯的位置区域，由于连续图像间的细微差距和阈值的选取很容易造成某些帧的漏检测，通过基于静态关键点跟踪技术，提升检测网络对车载视频的检测效果。
图像的特征点可以简单的理解为图像中比较显著的点，如角点、较暗区域中的亮点等。首先对于视频图像中的特征检测和描述(Oriented FAST and Rotated BRIEF，ORB)特征点进行识别：ORB特征点的定义基于特征点周围的图像灰度值，在检测时，考虑候选特征点周围一圈的像素值，如果候选点周围领域内有足够多的像素点与该候选特征点的灰度值差别达到预设值，则认为该候选点为一个关键特征点。本实施例是对交通信号灯的关键点的识别，因此，关键点为交通信号灯的关键点，以该交通信号灯的关键点可实现在视频流中对交通信号灯的静态追踪，由于交通信号灯在图像通常不仅占用一个像素点，即本实施例获得的交通信号灯的关键点包括至少一个像素点，可理解为交通信号灯的关键点对应一个位置区域。
可选地，对视频流中的交通信号灯的关键点进行跟踪，包括：
基于连续两帧图像中交通信号灯的关键点之间的距离；
基于交通信号灯的关键点之间的距离对视频流中的交通信号灯的关键点进行跟踪。
本实施例中所指连续两帧可以为视频流中时序连续的两个采集帧，也可以视频流中时序连续的两个检测帧(由于视频流中可以逐帧检测，也可以采样检测，因此检测帧和采集帧二者的含义并非完全相同)；通过对视频流中多个连续两帧图像的交通信号灯的关键点建立关联，即可实现在视频流中对交通信号灯的关键点进行跟踪，基于跟踪结果即可对视频流中的至少一帧图像进行位置区域的调整。可选地，可基于交通信号灯的关键点之间的汉明距离、欧几里得距离、联合贝叶斯距离或余弦距离等实现在视频流中对交通信号灯的关键点的跟踪，本实施不限制具体基于交通信号灯的关键点之间的哪种距离。
其中，汉明距离是使用在数据传输差错控制编码里面的，汉明距离是一个概念，它表示两个(相同长度)字对应位不同的数量，对两个字符串进行异或运算，并统计结果为1的个数，那么这个数就是汉明距离，两个图像之间的汉明距离为两个图像之间不相同的数据位数量。基于两帧信号图像中至少一个交通信号灯的关键点之间的汉明距离可知两信号图像之间信号灯移动的距离，即可实现对交通信号灯的 关键点的跟踪。
可选地，基于交通信号灯的关键点之间的距离对视频流中的交通信号灯的关键点进行跟踪，包括：
基于交通信号灯的关键点之间的距离，确定连续两帧图像中同一交通信号灯的关键点的位置区域；
根据同一交通信号灯的关键点在连续两帧图像中的位置区域，在视频流中对交通信号灯的关键点进行跟踪。
交通信号灯通常不是单个出现，并且由于交通信号灯在图像中不能由一个关键点进行表示，因此，图像中包括至少一个交通信号灯的关键点，而对于不同交通信号灯(如：同一图像中可同时显示前向交通灯、左转交通灯)需要分别进行跟踪，本实施例通过基于同一交通信号灯的关键点在连续帧中进行跟踪，克服了不同交通信号灯跟踪混乱的问题。
可选地，确定连续两帧图像中同一交通信号灯的关键点的位置区域可基于至少一个交通信号灯的关键点之间的汉明距离的较小值(例如，最小值)进行确定。
例如，可以通过对前后两帧中图像坐标系汉明距离较小的特征点(交通信号灯的关键点)描述子用暴力(Brute Force)算法进行匹配，即对于每对交通信号灯的关键点计算其特征子的汉明距离，基于汉明距离较小(例如，最小)的交通信号灯的关键点实现前后帧中的ORB特征点匹配，实现静态特征点跟踪。同时，由于交通信号灯的关键点的图片坐标系位于信号灯的候选区域内，判定该交通信号灯的关键点是信号灯检测中的静态关键点。暴力(Brute Force)算法，是普通的模式匹配算法，Brute Force算法的思想就是将目标串S的第一个字符与模式串T的第一个字符进行匹配，若相等，则继续比较S的第二个字符和T的第二个字符；若不相等，则比较S的第二个字符和T的第一个字符，依次比较下去，直到得出最后的匹配结果，Brute Force算法是一种蛮力算法。
在一个或多个可选的实施例中，基于跟踪结果对信号灯的位置区域进行调整，包括：
对比跟踪结果与信号灯的位置区域是否重合，得到对比结果；
基于对比结果对信号灯的位置区域进行调整。
基于跟踪结果对信号灯的位置区域进行调整后，使得信号灯的位置区域更加稳定，更适合应用于视频场景中。
本实施例中基于跟踪结果可确定视频流中至少一帧图像中交通信号灯的关键点对应的位置区域，当跟踪结果中的位置区域与信号灯的位置区域之间的重合部分在信号灯的位置区域中占比超出设定比例，即可确定跟踪结果与信号灯的位置区域重合，否则，确定跟踪结果与信号灯的位置区域不重合。
可选地，基于对比结果对信号灯的位置区域进行调整，包括：
响应于交通信号灯的关键点对应的位置区域和信号灯的位置区域不重合，以交通信号灯的关键点对应的位置区域替换信号灯的位置区域。
对比信号图像中的交通信号灯的关键点对应的位置区域和信号灯的位置区域是否重合的对比结果。可以包括以下三种情况：
如果交通信号灯的关键点对应的位置区域和信号灯的位置区域匹配(重合)，即前后两帧中相匹配的交通信号灯的关键点位置区域移动与检测得到的信号灯的位置区域相同，则无需更正；如果交通信号灯的关键点的位置区域和检测得到的信号灯的位置区域大致匹配，则根据前后帧中交通信号灯的关键点的位置区域的偏移，保持检测信号灯的位置宽、高不变的前提下，按照交通信号灯的关键点位置区域移动计算当前帧检测框的位置区域；如果当前帧中没有检测到交通信号灯的位置区域，但上一帧检测得到了交通信号灯的位置区域，则根据交通信号灯的关键点确定当前帧换信号灯的位置区域并没有超出摄像头范围，若没有超出范围，则以交通信号灯的关键点计算的结果确定当前帧交通信号灯的位置区域，以减少漏检。
在一个或多个可选的实施例中，操作120之前，还可以包括：
基于采集的训练图像集训练基于区域的全卷积网络，训练图像集包括多个具有标注属性的训练图像；
基于训练图像集调整基于区域的全卷积网络和多任务识别网络中的参数。
现实场景中，交通信号灯中的黄灯仅是红灯和绿灯中间的一个过渡状态，因此存在的时长与红灯和绿灯相比较短。现有技术中基于R-FCN的检测框架由于一次仅输入有限的图像，图像中的黄灯数目较红灯和绿灯相比极少，无法有效的训练检测网络，提升模型对黄灯的敏感度，因此，本公开通过对基于区域的全卷积网络和多任务识别网络训练获得同时对信号灯的位置、颜色和/或形状进行识别。
为了提高检测网络对黄灯的敏感度，可选地，在基于训练图像集调整基于区域的全卷积网络和多任务识别网络中的参数之前，还可以包括：
基于训练图像集获取交通信号灯的颜色比例符合预设比例的新训练图像集；
基于新训练图像集训练分类网络；分类网络用于基于交通信号灯的颜色对训练图像进行分类。
可选地，该分类网络是现有技术中的检测网络去掉候选区域网络(Region Proposal Network，RPN)和提议(propasal)层得到的，可选地，该分类网络可以对应包括多任务识别网络中的特征提取分支和分类分支；通过单独基于预设比例的新训练图像集训练分类网络，可提高分类网络对交通信号灯的颜色分类的准确率。
通过采集获得训练网络的训练图像集，以采集的训练图像集训练R-FCN基于区域的全卷积网络；对采集得到的训练图像集中红绿灯和黄灯的数量进行调整，可选地，预设比例中不同颜色的交通信号灯的数量相同或者数量差异小于容许阈值；
交通信号灯的颜色包括红色、黄色和绿色。
由于实际黄灯出现的概率远远低于红灯和绿灯，因此，采集到的训练图像中黄灯所占比例远小于红灯和绿灯，本实施例为了提高分类网络的准确性，可选择将红、黄、绿三种颜色的比例预设为相同比例(如：红色:黄色:绿色为1:1:1)，或控制红、黄、绿三种颜色的数量差异小于容许阈值，使三种颜色的比例接近于1:1:1。可通过从训练图像集中抽取交通信号灯为相应颜色的训练图像，构成新训练图像集；或者，将训练图像集中的黄灯图像重复调用，以使黄灯图像的数量与红灯图像和绿灯图像的数量符合预设比例)，以该调整后的新训练图像集训练分类网络，克服黄灯图像数量远小于红绿灯图像数据的缺点，提高了分类网络对黄灯的识别准确率。
可选地，基于训练图像集调整基于区域的全卷积网络和多任务识别网络的参数之前，还包括：
基于训练后的分类网络的参数初始化多任务识别网络中的至少部分参数。
可选地，可基于训练后的分类网络的参数初始化多任务识别网络中的部分或全部参数，例如：以训练后的分类网络的参数对多任务识别网络中特征提取分支和分类分支进行初始化；其中，参数例如可以包括卷积核的大小、卷积连接的权重等等。
在获得对黄灯提高识别准确率的分类网络之后，再以初始的训练图像集训练基于区域的全卷积网络和多任务识别网络，在训练之前，以训练后的分类网络中的参数对检测网络中部分参数进行初始化，此时得到的特征提取分支和分类分支对交通信号灯的颜色分类具有较好的效果，并且提高了对黄灯分类的准确率。
本公开交通信号灯检测方法可以应用在智能驾驶、高精度地图等领域；
可以将车载视频作为输入，输出红绿灯的位置和其状态，辅助车辆的安全行驶。
还可以用于建立高精度地图，检测其中的红绿灯位置。
在一个或多个可选的实施例中，还包括：
基于图像中交通信号灯的至少两种属性确定交通信号灯的状态；
根据交通信号灯的状态对车辆进行智能驾驶控制。
本实施例自动识别到交通信号灯的至少两种属性，并获得了视频流中交通信号灯的状态，无需驾驶员在驾驶过程中分心观察交通信号灯，提高了车辆行驶的安全性，减少了由于人为失误导致的交通危险。
可选地，智能驾驶控制包括：发出提示信息或告警信息，和/或，根据交通信号灯的状态控制车辆的行驶状态。
识别交通信号灯的至少两种属性可为智能驾驶提供基础，智能驾驶包括自动驾驶和辅助驾驶，自动驾驶的情况下，根据交通信号灯的状态控制车辆的行驶状态(如：停车、减速、转向等)，同时还可以发出提示信息或告警信息以告知驾驶员当前交通信号灯的状态；而在辅助驾驶的情况下，通常只发出提示信息或告警信息，控制车辆的权限仍然属于驾驶员，驾驶员根据提示信息或告警信息对车辆进行相应的控制。
可选地，还包括：存储交通信号灯的属性、状态及其对应的图像。
本实施例通过存储交通信号灯的属性、状态及其对应的图像，获取更多交通信号灯的信息(属性、状态及其对应的图像)，为智能驾驶提供更多操作依据。还可以根据存储的交通信号灯对应的时间和位置建立高精度地图，基于存储的交通信号灯对应的图像确定高精度地图中红绿灯的位置。
可选地，交通信号灯的状态包括但不限于：允许通行状态、禁止通行状态或等待状态；
基于图像中交通信号灯的至少两种属性确定交通信号灯的状态，包括以下至少之一：
响应于交通信号灯的颜色为绿色和/或形状为第一预设形状，确定交通信号灯的状态为允许通行状态。
响应于交通信号灯的颜色为红色和/或形状为第二预设形状，确定交通信号灯的状态为禁止通行状态。
响应于交通信号灯的颜色为黄色和/或形状为第三预设形状，确定交通信号灯的状态为等待状态。
根据现行交通法规可知，交通信号灯的颜色包括红色、绿色和黄色，不同颜色对应不同的通行状态，红色表示禁止车辆和/或行人通行，绿色表示允许车辆和/或行人通行、黄色表示车辆和/或行人通需要暂停等待；而辅助颜色的还可以包括交通信号等的形状，例如：加号形状(一种可选的第一预设形状)表示允许通行，叉形状(一种可选的第二预设形状)表示禁止通行，减号形状(一种可选的第三预设形状)表示等待状态等。针对不同交通信号灯的状态提供不同的应对策略，实现自动、半自动的智能驾驶，提高了驾驶的安全性。
可选地，根据交通信号灯的状态对车辆进行智能驾驶控制，包括：
响应于交通信号灯的状态为允许通行状态，控制车辆执行启动、保持行驶状态、减速、转向、开启转向灯、开启刹车灯等控制车辆通行过程中所需的其他控制中的一种或多种操作；
响应于交通信号灯的状态为禁止通行状态或等待状态，控制车辆停车、减速、开启刹车灯、控制车辆禁止通行或处于等待状态过程中所需的其他控制中的一种或多种操作。
例如：当交通信号灯的颜色为绿色且形状为向左指向的箭头时，可控制车辆自动转向(向左)和/或自动开启转向灯(左转向灯)；当交通信号灯的颜色为绿色且形状为向前指向的箭头时，可控制车辆减速行驶通过路口；当然，具体控制车辆如何行驶是根据当前车辆设定的目的地与当前交通信号灯的状态综合的结果；通过自动控制车辆执行对应交通信号灯的状态的操作，可实现安全性更高的智能驾驶，提高了驾驶的安全性，减少了由于人为操作失误导致的安全隐患。
本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于一计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
图2为本公开交通信号灯检测装置一个实施例的结构示意图。该实施例的交通信号灯检测装置可用于实现本公开上述各交通信号灯检测方法实施例。如图2所示，该实施例的装置包括：
视频流获取单元21，用于获取包括有交通信号灯的视频流。
可选地，对于交通信号灯的识别，通常是以车辆行进过程中记录的车载视频为基础进行的，对车载视频进行解析，获得包括至少一帧图像的视频流，例如可通过安装在车辆上的摄像装置拍摄车辆前向或周围环境的视频，如果车辆前向或周围环境中存在交通信号灯，则会被摄像装置拍摄到，所拍摄的视频流即为包括有交通信号灯的视频流。该视频流中的图像可以每帧图像都包括交通信号灯，或至少有一帧图像中包括交通信号灯。
区域确定单元22，用于确定视频流的至少一帧图像中交通信号灯的候选区域。
可选地，从视频流中包括有交通信号灯的图像中确定候选区域，该候选区域指在图像中可能包括交通信号灯的区域。
交通信号灯的区域的检测可基于神经网络或其他类型的检测模型进行。在一个或多个可选的实施例中，利用基于区域的全卷积网络，确定视频流的至少一帧图像中交通信号灯的候选区域。通过基于区域的全卷积网络(R-FCN)对信号图像进行检测，得到可能包括交通信号灯的候选区域，R-FCN可以看做是Faster RCNN的改进版，检测速度相对Faster RCNN有加快。
属性识别单元23，用于基于候选区域确定图像中交通信号灯的至少两种属性。
交通信号灯的属性用于描述交通信号灯，可根据实际需要进行定义，例如，可以包括用于描述交通信号灯绝对位置或相对位置的位置区域属性，用于描述交通信号灯的颜色(如红色、绿色、黄色等)属性，用于描述交通信号灯的形状的(如圆形、直线箭头、折线箭头等)属性，以及用于描述交通信号灯的其他方面的其他属性等。
基于本公开上述实施例提供的一种交通信号灯检测装置，通过获得交通信号灯的至少两种属性实现对信号灯的多种信息的识别，减少了识别时间，并提高了交通信号灯识别的准确率。
可选地，交通信号灯的至少两种属性包括以下任意两种或两种以上：位置区域、颜色和形状。
交通信号灯的至少两种属性的确定可基于神经网络或其他类型的识别模型进行。在一个或多个可选的实施例中，属性识别单元23，用于利用多任务识别网络，基于候选区域确定图像中交通信号灯的至少两种属性。
本实施例通过一个网络实现对交通信号灯的至少两种属性进行识别，相对于分别基于至少两个网络识别至少两种属性的情况，减小了网络的大小，提高了交通信号灯的属性识别的效率。
可选地，多任务识别网络包括特征提取分支、以及分别与所述特征提取分支连接的至少两个任务分支，不同的任务分支用于确定交通信号灯的不同种类属性；
属性识别单元23，包括：
特征提取模块，用于基于特征提取分支对所述候选区域进行特征提取，得到候选特征；
分支属性模块，用于分别基于至少两个任务分支对候选特征进行处理，获得图像中交通信号灯的至少两种属性。
可选地，至少两个任务分支包括但不限于：检测分支、识别分支和分类分支；
分支属性模块，用于经检测分支对候选特征进行位置检测，确定交通信号灯的位置区域；经分类分支对候选特征进行颜色分类，确定交通信号灯所在位置区域的颜色，确定交通信号灯的颜色；经识别分支对候选特征进行形状识别，确定交通信号灯所在位置区域的形状，确定交通信号灯的形状。
在一个或多个可选的实施例中，还包括：
关键点确定单元，用于对视频流中的至少一帧图像进行关键点识别，确定图像中的交通信号灯的关 键点；
关键点跟踪单元，用于对视频流中的交通信号灯的关键点进行跟踪，得到跟踪结果；
位置调整单元，用于基于跟踪结果对交通信号灯的位置区域进行调整。
在视频流的连续帧之间存在的差异可能很小，仅基于每帧图像中交通信号灯的候选区域进行交通信号灯的位置识别，有可能将连续帧中的位置区域识别成相同的位置区域，从而导致识别到的位置区域不准确，本实施例通过图像中进行关键点识别，基于关键点确定交通信号灯在图像中的位置区域，基于该关键点的位置区域调整多任务识别网络获得的交通信号灯的位置，提高了位置区域识别的准确率。
关键点识别和/或跟踪可基于现有技术中可实现关键点识别和/或跟踪的任意一个技术实现。可选地，通过基于静态关键点跟踪技术实现对视频流中的交通信号灯的关键点的跟踪，以获得在视频流中交通信号灯的关键点可能存在的区域。
可选地，关键点跟踪单元，用于基于连续两帧图像中交通信号灯的关键点之间的距离；基于交通信号灯的关键点之间的距离对视频流中的交通信号灯的关键点进行跟踪。
本实施例中所指连续两帧可以为视频流中时序连续的两个采集帧，也可以视频流中时序连续的两个检测帧(由于视频流中可以逐帧检测，也可以采样检测，因此检测帧和采集帧二者的含义并非完全相同)；通过对视频流中多个连续两帧图像的交通信号灯的关键点建立关联，即可实现在视频流中对交通信号灯的关键点进行跟踪，基于跟踪结果即可对视频流中的每帧图像进行位置区域的调整。可选地，可基于交通信号灯的关键点之间的汉明距离、欧几里得距离、联合贝叶斯距离或余弦距离等实现在视频流中对交通信号灯的关键点的跟踪，本实施不限制具体基于交通信号灯的关键点之间的哪种距离。
可选地，关键点跟踪单元基于交通信号灯的关键点之间的距离对视频流中的交通信号灯的关键点进行跟踪时，用于基于交通信号灯的关键点之间的距离，确定连续两帧图像中同一交通信号灯的关键点的位置区域；根据同一交通信号灯的关键点在连续两帧图像中的位置区域，在视频流中对交通信号灯的关键点进行跟踪。
在一个或多个可选的实施例中，位置调整单元，用于对比跟踪结果与信号灯的位置区域是否重合，得到比对结果；基于对比结果对信号灯的位置区域进行调整。
基于跟踪结果对信号灯的位置区域进行调整后，使得信号灯的位置区域更加稳定，更适合应用于视频场景中。
本实施例中基于跟踪结果可确定视频流中至少一帧图像中交通信号灯的关键点对应的位置区域，当跟踪结果中的位置区域与信号灯的位置区域之间的重合部分在信号灯的位置区域中占比超出设定比例，即可确定跟踪结果与信号灯的位置区域重合，否则，确定跟踪结果与信号灯的位置区域不重合。
可选地，位置调整单元基于对比结果对信号灯的位置区域进行调整时，用于响应于交通信号灯的关键点对应的位置区域和信号灯的位置区域不重合，以交通信号灯的关键点对应的位置区域替换信号灯的位置区域。
在一个或多个可选的实施例中，还可以包括：
预训练单元，用于基于采集的训练图像集训练基于区域的全卷积网络，训练图像集包括多个具有标注属性的训练图像；
训练单元，用于基于训练图像集调整基于区域的全卷积网络和多任务识别网络中的参数。
现实场景中，交通信号灯中的黄灯仅是红灯和绿灯中间的一个过渡状态，因此存在的时长较红灯和绿灯较短。现有技术中基于R-FCN的检测框架由于一次仅输入有限的图像，图像中的黄灯数目较红灯和绿灯相比极少，无法有效的训练检测网络，提升模型对黄灯的敏感度，因此，本公开通过对基于区域的全卷积网络和多任务识别网络训练获得同时对信号灯的位置、颜色和/或形状进行识别。
为了提高检测网络对黄灯的敏感度，可选地，预训练单元和训练单元之间还可以包括：
分类训练单元，用于基于训练图像集获取交通信号灯的颜色比例符合预设比例的新训练图像集；基于新训练图像集训练分类网络；分类网络用于基于交通信号灯的颜色对训练图像进行分类。
可选地，预设比例中不同颜色的交通信号灯的数量相同或者数量差异小于容许阈值；
交通信号灯的颜色包括红色、黄色和绿色。
由于实际黄灯出现的概率远远低于红灯和绿灯，因此，采集到的训练图像中黄灯所占比例远小于红灯和绿灯，本实施例为了提高分类网络的准确性，可选择将红、黄、绿三种颜色的比例预设为相同比例(如：红色:黄色:绿色为1:1:1)，或控制红、黄、绿三种颜色的数量差异小于容许阈值，使三种颜色的比例接近于1:1:1。可通过从训练图像集中抽取交通信号灯为相应颜色的训练图像，构成新训练图像集；或者，将训练图像集中的黄灯图像重复调用，以使黄灯图像的数量与红灯图像和绿灯图像的数量符合预设比例)，以该调整后的新训练图像集训练分类网络，克服黄灯图像数量远小于红绿灯图像数据的缺点，提高了分类网络对黄灯的识别准确率。
可选地，在分类训练单元之后还可以包括：
初始化单元，用于基于训练后的分类网络的参数初始化多任务识别网络中的至少部分参数。
在一个或多个可选的实施例中，本实施例装置还可以包括：
状态确定单元，用于基于图像中交通信号灯的至少两种属性确定交通信号灯的状态；
智能控制单元，用于根据交通信号灯的状态对车辆进行智能驾驶控制。
本实施例自动识别到交通信号灯的至少两种属性，并获得了视频流中交通信号灯的状态，无需驾驶员在驾驶过程中分心观察交通信号灯，提高了车辆行驶的安全性，减少了由于人为失误导致的交通危险。
可选地，智能驾驶控制包括：发出提示信息或告警信息，和/或，根据交通信号灯的状态控制车辆的行驶状态。
可选地，还包括：
存储单元，用于存储交通信号灯的属性、状态及其对应的所述图像。
可选地，交通信号灯的状态包括但不限于：允许通行状态、禁止通行状态或等待状态；
状态确定单元，用于响应于交通信号灯的颜色为绿色和/或形状为第一预设形状，确定交通信号灯的状态为允许通行状态；
响应于交通信号灯的颜色为红色和/或形状为第二预设形状，确定交通信号灯的状态为禁止通行状态；
响应于交通信号灯的颜色为黄色和/或形状为第三预设形状，确定交通信号灯的状态为等待状态。
可选地，智能控制单元，用于响应于交通信号灯的状态为允许通行状态，控制车辆执行启动、保持行驶状态、减速、转向、开启转向灯、开启刹车灯中的一种或多种操作；
响应于交通信号灯的状态为禁止通行状态或等待状态，控制车辆停车、减速、开启刹车灯中的一种或多种操作。
本公开实施例提供的交通信号灯检测装置任一实施例的工作过程以及设置方式均可以参照本公开上述相应方法实施例的具体描述，限于篇幅，在此不再赘述。
图3为本公开智能驾驶方法一个实施例的流程图。如图3所示，该实施例方法包括：
步骤310，基于设置在车辆上的图像采集装置获取包括有交通信号灯的视频流。
可选地，以车辆行进过程中记录的车载视频为基础进行的，对车载视频进行解析，获得包括至少一帧图像的视频流，例如可通过安装在车辆上的摄像装置拍摄车辆前向或周围环境的视频，如果车辆前向或周围环境中存在交通信号灯，则会被摄像装置拍摄到，所拍摄的视频流即为包括有交通信号灯的视频 流。该视频流中的图像可以每帧图像都包括交通信号灯，或至少有一帧图像中包括交通信号灯。
在一个可选示例中，该步骤310可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的视频流获取单元21执行。
步骤320，确定视频流的至少一帧图像中交通信号灯的候选区域。
在一个可选示例中，该步骤320可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的区域确定单元22执行。
步骤330，基于候选区域确定图像中交通信号灯的至少两种属性。
交通信号灯的属性用于描述交通信号灯，可根据实际需要进行定义，例如，可以包括用于描述交通信号灯绝对位置或相对位置的位置区域属性，用于描述交通信号灯的颜色(如红色、绿色、黄色等)属性，用于描述交通信号灯的形状的(如圆形、直线箭头、折线箭头等)属性，以及用于描述交通信号灯的其他方面的其他属性。
可选地，交通信号灯的至少两种属性包括以下任意两种或两种以上：位置区域、颜色和形状。
可选地，交通信号灯的颜色包括红、黄、绿三种颜色，形状包括箭头形、圆形或其他形状等，对于不同形状的交通信号灯，如果仅识别其位置，可能无法准确的将信号识别出来，因此，本实施例通过识别位置区域、颜色和形状中的至少两种，例如：当确定交通信号灯的位置区域和颜色，即可确定当前交通信号灯在图像中哪个位置(对应车辆的哪个方向)，通过颜色即可确定交通信号灯显示的状态(红色、绿色或黄色分别对应不同状态)，通过识别到交通信号灯的不同状态可实现辅助驾驶或自动驾驶；当确定交通信号灯的位置区域和形状，即可确定当前交通信号灯在图像中哪个位置(对应车辆的哪个方向)，通过形状即可确定交通信号灯显示的状态(例如：朝向不同方向的箭头表示不同状态，或不同形状的人体图形表示不同状态)；当确定交通信号灯的颜色和形状，可基于颜色和形状相结合确定当前交通信号灯的状态(例如：指向左侧的绿色箭头表示左转通行，指向前方的红色箭头表示前方禁行)；而当确定交通信号灯的位置区域、颜色和形状时，在获得交通信号灯在图像中哪个位置的基础上，还可以基于颜色和形状相结合确定当前交通信号灯的状态。
在一个可选示例中，该步骤330可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的属性识别单元23执行。
步骤340，基于图像中交通信号灯的至少两种属性确定交通信号灯的状态。
现有的图像处理方法通常只能针对一种任务进行处理(例如：位置识别或颜色分类中的一种)，而交通信号灯包括位置区域、颜色和形状等信息，当需要确定交通信号灯的状态时，不仅需要确定交通信号等的位置区域，至少还需要确定颜色或形状，因此，如应用通常的图像处理方法，至少需要两个神经网络对视频流进行处理，还需要对处理结果进行综合，才能确定当前交通信号灯的状态；本实施例同时获得交通信号灯的至少两种属性，以至少两种属性确定交通信号灯的状态，快速准确的识别交通信号灯的状态。
在一个可选示例中，该步骤340可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的状态确定单元44执行。
步骤350，根据交通信号灯的状态对车辆进行智能驾驶控制。
在一个可选示例中，该步骤350可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的智能控制单元45执行。
本实施例通过车辆上的图像采集装置可实时获得视频流，实现实时识别交通信号灯的属性，以确定交通信号灯的状态，基于交通信号灯的状态实现智能驾驶，无需驾驶员在驾驶过程中分心观察交通信号灯，减少了交通安全隐患，一定程度上降低了由于人为失误导致的交通危险。智能驾驶可以包括辅助驾 驶和自动驾驶，通常，辅助驾驶是利用信号灯进行预警提示，自动驾驶利用信号灯进行驾驶控制。
可选地，智能驾驶控制包括：发出提示信息或告警信息，和/或，根据交通信号灯的状态控制车辆的行驶状态。
识别交通信号灯的至少两种属性可为智能驾驶提供基础，智能驾驶包括自动驾驶和辅助驾驶，自动驾驶的情况下，根据交通信号灯的状态控制车辆的行驶状态(如：停车、减速、转向等)，同时还可以发出提示信息或告警信息以告知驾驶员当前交通信号灯的状态；而在辅助驾驶的情况下，通常只发出提示信息或告警信息，控制车辆的权限仍然属于驾驶员，驾驶员根据提示信息或告警信息对车辆进行相应的控制。
可选地，本申请实施例提供的智能驾驶方法还包括：
存储交通信号灯的属性、状态及其对应的图像。
本实施例通过存储交通信号灯的属性、状态及其对应的图像，获取更多交通信号灯的信息(属性、状态及其对应的图像)，为智能驾驶提供更多操作依据。还可以根据存储的交通信号灯对应的时间和位置建立高精度地图，基于存储的交通信号灯对应的图像确定高精度地图中红绿灯的位置。
可选地，交通信号灯的状态包括但不限于：允许通行状态、禁止通行状态和等待状态；
步骤340可以包括：
响应于交通信号灯的颜色为绿色和/或形状为第一预设形状，确定交通信号灯的状态为允许通行状态；
响应于交通信号灯的颜色为红色和/或形状为第二预设形状，确定交通信号灯的状态为禁止通行状态；
响应于交通信号灯的颜色为黄色和/或形状为第三预设形状，确定交通信号灯的状态为等待状态。
根据现行交通法规可知，交通信号灯的颜色包括红色、绿色和黄色，不同颜色对应不同的通行状态，红色表示禁止车辆和/或行人通行，绿色表示允许车辆和/或行人通通行、黄色表示车辆和/或行人通需要暂停等待；而辅助颜色的还可以包括交通信号等的形状，例如：加号形状(一种可选的第一预设形状)表示允许通行，叉形状(一种可选的第二预设形状)表示禁止通行，减号形状(一种可选的第三预设形状)表示等待状态等。针对不同交通信号灯的状态提供不同的应对策略，实现自动、半自动的智能驾驶，提高了驾驶的安全性。
可选地，步骤350可以包括：
响应于交通信号灯的状态为允许通行状态，控制车辆执行启动、保持行驶状态、减速、转向、开启转向灯、开启刹车灯、控制车辆通行过程中所需的其他控制中的一种或多种操作；
响应于交通信号灯的状态为禁止通行状态或等待状态，控制车辆停车、减速、开启刹车灯、控制车辆禁止通行或处于等待状态过程中所需的其他控制中的一种或多种操作。
例如：当交通信号灯的颜色为绿色且形状为向左指向的箭头时，可控制车辆自动转向(向左)和/或自动开启转向灯(左转向灯)；当交通信号灯的颜色为绿色且形状为向前指向的箭头时，可控制车辆减速行驶通过路口；当然，具体控制车辆如何行驶是根据当前车辆设定的目的地与当前交通信号灯的状态综合的结果；通过自动控制车辆执行对应交通信号灯的状态的操作，可实现安全性更高的智能驾驶，减少了由于人为操作失误导致的安全隐患。
本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于一计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
图4为本公开智能驾驶装置一个实施例的结构示意图。该实施例的智能驾驶装置可用于实现本公开 上述各智能驾驶方法实施例。如图4所示，该实施例的装置包括：
视频流获取单元21，用于基于设置在车辆上的图像采集装置获取包括有交通信号灯的视频流。
可选地，以车辆行进过程中记录的车载视频为基础进行的，对车载视频进行解析，获得包括至少一帧图像的视频流，例如可通过安装在车辆上的摄像装置拍摄车辆前向或周围环境的视频，如果车辆前向或周围环境中存在交通信号灯，则会被摄像装置拍摄到，所拍摄的视频流即为包括有交通信号灯的视频流。该视频流中的图像可以每帧图像都包括交通信号灯，或至少有一帧图像中包括交通信号灯。
区域确定单元22，用于确定视频流的至少一帧图像中交通信号灯的候选区域。
属性识别单元23，用于基于候选区域确定图像中交通信号灯的至少两种属性。
交通信号灯的属性用于描述交通信号灯，可根据实际需要进行定义，例如，可以包括用于描述交通信号灯绝对位置或相对位置的位置区域属性，用于描述交通信号灯的颜色(如红色、绿色、黄色等)属性，用于描述交通信号灯的形状的(如圆形、直线箭头、折线箭头等)属性，以及用于描述交通信号灯的其他方面的其他属性。
状态确定单元44，用于基于图像中交通信号灯的至少两种属性确定交通信号灯的状态。
现有的图像处理方法通常只能针对一种任务进行处理(例如：位置识别或颜色分类中的一种)，而交通信号灯包括位置区域、颜色和形状等信息，当需要确定交通信号灯的状态时，不仅需要确定交通信号等的位置区域，至少还需要确定颜色或形状，因此，如应用通常的图像处理方法，至少需要两个神经网络对视频流进行处理，还需要对处理结果进行综合，才能确定当前交通信号灯的状态；本实施例同时获得交通信号灯的至少两种属性，以至少两种属性确定交通信号灯的状态，快速准确的识别交通信号灯的状态。
智能控制单元45，用于根据交通信号灯的状态对车辆进行智能驾驶控制。
本实施例通过车辆上的图像采集装置可实时获得视频流，实现实时识别交通信号灯的属性，以确定交通信号灯的状态，基于交通信号灯的状态实现智能驾驶，无需驾驶员在驾驶过程中分心观察交通信号灯，减少了交通安全隐患，一定程度上降低了由于人为失误导致的交通危险。智能驾驶可以包括辅助驾驶和自动驾驶，通常，辅助驾驶是利用信号灯进行预警提示，自动驾驶利用信号灯进行驾驶控制。
可选地，智能驾驶控制包括：发出提示信息或告警信息，和/或，根据交通信号灯的状态控制车辆的行驶状态。
可选地，还包括：
存储单元，用于存储交通信号灯的属性、状态及其对应的图像。
可选地，交通信号灯的至少两种属性包括以下任意两种或两种以上：位置区域、颜色和形状。
可选地，交通信号灯的状态包括但不限于：允许通行状态、禁止通行状态和等待状态；
状态确定单元44，用于响应于交通信号灯的颜色为绿色和/或形状为第一预设形状，确定交通信号灯的状态为允许通行状态；
响应于交通信号灯的颜色为红色和/或形状为第二预设形状，确定交通信号灯的状态为禁止通行状态；
响应于交通信号灯的颜色为黄色和/或形状为第三预设形状，确定交通信号灯的状态为等待状态。
可选地，智能控制单元45，用于响应于交通信号灯的状态为允许通行状态，控制车辆执行启动、保持行驶状态、减速、转向、开启转向灯、开启刹车灯中的一种或多种操作；
响应于交通信号灯的状态为禁止通行状态或等待状态，控制车辆停车、减速、开启刹车灯中的一种或多种操作。
本公开实施例提供的智能驾驶装置任一实施例的工作过程以及设置方式均可以参照本公开上述相 应方法实施例的具体描述，限于篇幅，在此不再赘述。
根据本公开实施例的另一个方面，提供了一种车辆，包括如上任意一实施例所述的交通信号灯检测装置或如上任意一实施例所述的智能驾驶装置。
根据本公开实施例的另一个方面，提供的一种电子设备，包括处理器，所述处理器包括如上任意一项所述的交通信号灯检测装置或如上任意一实施例所述的智能驾驶装置。
根据本公开实施例的又一个方面，提供的一种电子设备，包括：存储器，用于存储可执行指令；
以及处理器，用于与所述存储器通信以执行所述可执行指令从而完成如上任意一实施例所述交通信号灯检测方法的操作，或完成如上任意一实施例所述的智能驾驶方法的操作。
本公开实施例还提供了一种电子设备，例如可以是移动终端、个人计算机(PC)、平板电脑、服务器等。下面参考图5，其示出了适于用来实现本公开实施例的终端设备或服务器的电子设备500的结构示意图：如图5所示，电子设备500包括一个或多个处理器、通信部等，所述一个或多个处理器例如：一个或多个中央处理单元(CPU)501，和/或一个或多个图像处理器(GPU)513等，处理器可以根据存储在只读存储器(ROM)502中的可执行指令或者从存储部分508加载到随机访问存储器(RAM)503中的可执行指令而执行各种适当的动作和处理。通信部512可包括但不限于网卡，所述网卡可包括但不限于IB(Infiniband)网卡。
处理器可与只读存储器502和/或随机访问存储器503中通信以执行可执行指令，通过总线504与通信部512相连、并经通信部512与其他目标设备通信，从而完成本公开实施例提供的任一项方法对应的操作，例如，获取包括有交通信号灯的视频流；确定视频流的至少一帧图像中交通信号灯的候选区域；基于候选区域确定图像中交通信号灯的至少两种属性。
此外，在RAM 503中，还可存储有装置操作所需的各种程序和数据。CPU501、ROM502以及RAM503通过总线504彼此相连。在有RAM503的情况下，ROM502为可选模块。RAM503存储可执行指令，或在运行时向ROM502中写入可执行指令，可执行指令使中央处理单元501执行上述通信方法对应的操作。输入/输出(I/O)接口505也连接至总线504。通信部512可以集成设置，也可以设置为具有多个子模块(例如多个IB网卡)，并在总线链接上。
以下部件连接至I/O接口505：包括键盘、鼠标等的输入部分506；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分507；包括硬盘等的存储部分508；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分509。通信部分509经由诸如因特网的网络执行通信处理。驱动器510也根据需要连接至I/O接口505。可拆卸介质511，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器510上，以便于从其上读出的计算机程序根据需要被安装入存储部分508。
需要说明的，如图5所示的架构仅为一种可选实现方式，在具体实践过程中，可根据实际需要对上述图5的部件数量和类型进行选择、删减、增加或替换；在不同功能部件设置上，也可采用分离设置或集成设置等实现方式，例如GPU513和CPU501可分离设置或者可将GPU513集成在CPU501上，通信部512可分离设置，也可集成设置在CPU501或GPU513上，等等。这些可替换的实施方式均落入本公开公开的保护范围。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括有形地包含在机器可读介质上的计算机程序，计算机程序包含用于执行流程图所示的方法的程序代码，程序代码可包括对应执行本公开实施例提供的方法步骤对应的指令，例如，获取包括有交通信号灯的视频流；确定视频流的至少一帧图像中交通信号灯的候选区域；基于候选区域确定图像中交通信号灯的至少两种属性。在这样的实施例中，该计算机程序可以 通过通信部分509从网络上被下载和安装，和/或从可拆卸介质511被安装。在该计算机程序被中央处理单元(CPU)501执行时，执行本公开的方法中限定的上述功能的操作。
根据本公开实施例的还一个方面，提供的一种计算机可读存储介质，用于存储计算机可读取的指令，该指令被执行时执行如上任意一项所述交通信号灯检测方法或如上任意一项所述的智能驾驶方法的操作。
根据本公开实施例的再一个方面，提供的一种计算机程序产品，包括计算机可读代码，当计算机可读代码在设备上运行时，该设备中的处理器执行用于实现如上任意一项所述交通信号灯检测方法或如上任意一项所述的智能驾驶方法的指令。
本说明书中各个实施例均采用递进的方式描述，每个实施例重点说明的都是与其它实施例的不同之处，各个实施例之间相同或相似的部分相互参见即可。对于系统实施例而言，由于其与方法实施例基本对应，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
可能以许多方式来实现本发明的方法和装置。例如，可通过软件、硬件、固件或者软件、硬件、固件的任何组合来实现本发明的方法和装置。用于所述方法的步骤的上述顺序仅是为了进行说明，本发明的方法的步骤不限于以上具体描述的顺序，除非以其它方式特别说明。此外，在一些实施例中，还可将本发明实施为记录在记录介质中的程序，这些程序包括用于实现根据本发明的方法的机器可读指令。因而，本发明还覆盖存储用于执行根据本发明的方法的程序的记录介质。
本发明的描述是为了示例和描述起见而给出的，而并不是无遗漏的或者将本发明限于所公开的形式。很多修改和变化对于本领域的普通技术人员而言是显然的。选择和描述实施例是为了更好说明本发明的原理和实际应用，并且使本领域的普通技术人员能够理解本发明从而设计适于特定用途的带有各种修改的各种实施例。

Claims (57)

  1. 一种交通信号灯检测方法，其特征在于，包括：
    获取包括有交通信号灯的视频流；
    确定所述视频流的至少一帧图像中交通信号灯的候选区域；
    基于所述候选区域确定所述图像中交通信号灯的至少两种属性。
  2. 根据权利要求1所述的方法，其特征在于，所述交通信号灯的至少两种属性包括以下任意两种或两种以上：位置区域、颜色和形状。
  3. 根据权利要求1或2所述的方法，其特征在于，所述确定所述视频流的至少一帧图像中交通信号灯的候选区域，包括：利用基于区域的全卷积网络，确定所述视频流的至少一帧图像中交通信号灯的候选区域。
  4. 根据权利要求1-3任一所述的方法，其特征在于，所述基于所述候选区域确定所述图像中交通信号灯的至少两种属性，包括：利用多任务识别网络，基于所述候选区域确定所述图像中交通信号灯的至少两种属性。
  5. 根据权利要求4所述的方法，其特征在于，所述多任务识别网络包括特征提取分支、以及分别与所述特征提取分支连接的至少两个任务分支，不同的任务分支用于确定所述交通信号灯的不同种类属性；
    所述利用多任务识别网络，基于所述候选区域确定所述图像中交通信号灯的至少两种属性，包括：
    基于所述特征提取分支对所述候选区域进行特征提取，得到候选特征；
    分别基于所述至少两个任务分支对所述候选特征进行处理，获得所述图像中交通信号灯的至少两种属性。
  6. 根据权利要求5所述的方法，其特征在于，所述至少两个任务分支包括：检测分支、识别分支和分类分支；
    所述分别基于所述至少两个任务分支对所述候选特征进行处理，获得所述图像中交通信号灯的至少两种属性，包括：
    经所述检测分支对所述候选特征进行位置检测，确定交通信号灯的位置区域；
    经所述分类分支对所述候选特征进行颜色分类，确定交通信号灯所在位置区域的颜色，确定所述交通信号灯的颜色；
    经所述识别分支对所述候选特征进行形状识别，确定所述交通信号灯所在位置区域的形状，确定所述交通信号灯的形状。
  7. 根据权利要求1-6任一所述的方法，其特征在于，所述确定所述视频流的至少一帧图像中交通信号灯的候选区域之前，还包括：
    对所述视频流中的至少一帧图像进行关键点识别，确定所述图像中的交通信号灯的关键点；
    对所述视频流中的交通信号灯的关键点进行跟踪，得到跟踪结果；
    基于所述跟踪结果对所述交通信号灯的位置区域进行调整。
  8. 根据权利要求7所述的方法，其特征在于，所述对所述视频流中的交通信号灯的关键点进行跟踪，包括：
    基于连续两帧所述图像中所述交通信号灯的关键点之间的距离；
    基于所述交通信号灯的关键点之间的距离对所述视频流中的交通信号灯的关键点进行跟踪。
  9. 根据权利要求8所述的方法，其特征在于，所述基于所述交通信号灯的关键点之间的距离对所 述视频流中的交通信号灯的关键点进行跟踪，包括：
    基于所述交通信号灯的关键点之间的距离，确定连续两帧图像中同一交通信号灯的关键点的位置区域；
    根据所述同一交通信号灯的关键点在连续两帧所述图像中的位置区域，在所述视频流中对交通信号灯的关键点进行跟踪。
  10. 根据权利要求7-9任一所述的方法，其特征在于，所述基于所述跟踪结果对所述信号灯的位置区域进行调整，包括：
    对比所述跟踪结果与所述信号灯的位置区域是否重合，得到对比结果；
    基于所述对比结果对所述信号灯的位置区域进行调整。
  11. 根据权利要求10所述的方法，其特征在于，所述基于所述对比结果对所述信号灯的位置区域进行调整，包括：
    响应于所述交通信号灯的关键点对应的位置区域和所述信号灯的位置区域不重合，以所述交通信号灯的关键点对应的位置区域替换所述信号灯的位置区域。
  12. 根据权利要求4-11任一所述的方法，其特征在于，所述确定所述视频流的至少一帧图像中交通信号灯的候选区域之前，还包括：
    基于采集的训练图像集训练所述基于区域的全卷积网络，所述训练图像集包括多个具有标注属性的训练图像；
    基于所述训练图像集调整所述基于区域的全卷积网络和所述多任务识别网络中的参数。
  13. 根据权利要求12所述的方法，其特征在于，所述基于所述训练图像集调整所述基于区域的全卷积网络和所述多任务识别网络中的参数之前，还包括：
    基于所述训练图像集获取交通信号灯的颜色比例符合预设比例的新训练图像集；
    基于所述新训练图像集训练分类网络；所述分类网络用于基于所述交通信号灯的颜色对所述训练图像进行分类。
  14. 根据权利要求13所述的方法，其特征在于，所述预设比例中不同颜色的交通信号灯的数量相同或者数量差异小于容许阈值；
    所述交通信号灯的颜色包括红色、黄色和绿色。
  15. 根据权利要求14所述的方法，其特征在于，所述基于所述训练图像集调整所述基于区域的全卷积网络和所述多任务识别网络的参数之前，还包括：
    基于所述训练后的分类网络的参数初始化所述多任务识别网络中的至少部分参数。
  16. 根据权利要求1-15任一所述的方法，其特征在于，还包括：
    基于所述图像中交通信号灯的至少两种属性确定所述交通信号灯的状态；
    根据所述交通信号灯的状态对所述车辆进行智能驾驶控制。
  17. 根据权利要求16所述的方法，其特征在于，所述智能驾驶控制包括：发出提示信息或告警信息，和/或，根据所述交通信号灯的状态控制所述车辆的行驶状态。
  18. 根据权利要求16或17所述的方法，其特征在于，还包括：
    存储所述交通信号灯的属性、状态及其对应的所述图像。
  19. 根据权利要求16-18任一所述的方法，其特征在于，所述交通信号灯的状态包括：允许通行状态、禁止通行状态或等待状态；
    所述基于所述图像中交通信号灯的至少两种属性确定所述交通信号灯的状态，包括以下至少之一：
    响应于所述交通信号灯的颜色为绿色和/或形状为第一预设形状，确定所述交通信号灯的状态为允 许通行状态；
    响应于所述交通信号灯的颜色为红色和/或形状为第二预设形状，确定所述交通信号灯的状态为禁止通行状态；
    响应于所述交通信号灯的颜色为黄色和/或形状为第三预设形状，确定所述交通信号灯的状态为等待状态。
  20. 根据权利要求19所述的方法，其特征在于，所述根据所述交通信号灯的状态对所述车辆进行智能驾驶控制，包括：
    响应于所述交通信号灯的状态为允许通行状态，控制所述车辆执行启动、保持行驶状态、减速、转向、开启转向灯、开启刹车灯中的一种或多种操作；
    响应于所述交通信号灯的状态为禁止通行状态或等待状态，控制所述车辆停车、减速、开启刹车灯中的一种或多种操作。
  21. 一种智能驾驶方法，其特征在于，包括：
    基于设置在车辆上的图像采集装置获取包括有交通信号灯的视频流；
    确定所述视频流的至少一帧图像中交通信号灯的候选区域；
    基于所述候选区域确定所述图像中交通信号灯的至少两种属性；
    基于所述图像中交通信号灯的至少两种属性确定所述交通信号灯的状态；
    根据所述交通信号灯的状态对所述车辆进行智能驾驶控制。
  22. 根据权利要求21所述的方法，其特征在于，所述智能驾驶控制包括：发出提示信息或告警信息，和/或，根据所述交通信号灯的状态控制所述车辆的行驶状态。
  23. 根据权利要求21或22所述的方法，其特征在于，还包括：
    存储所述交通信号灯的属性、状态及其对应的所述图像。
  24. 根据权利要求21-23任一所述的方法，其特征在于，所述交通信号灯的至少两种属性包括以下任意两种或两种以上：位置区域、颜色和形状。
  25. 根据权利要求24所述的方法，其特征在于，所述交通信号灯的状态包括：允许通行状态、禁止通行状态和等待状态；
    所述基于所述图像中交通信号灯的至少两种属性确定所述交通信号灯的状态，包括：
    响应于所述交通信号灯的颜色为绿色和/或形状为第一预设形状，确定所述交通信号灯的状态为允许通行状态；
    响应于所述交通信号灯的颜色为红色和/或形状为第二预设形状，确定所述交通信号灯的状态为禁止通行状态；
    响应于所述交通信号灯的颜色为黄色和/或形状为第三预设形状，确定所述交通信号灯的状态为等待状态。
  26. 根据权利要求25所述的方法，其特征在于，所述根据所述交通信号灯的状态对所述车辆进行智能驾驶控制，包括：
    响应于所述交通信号灯的状态为允许通行状态，控制所述车辆执行启动、保持行驶状态、减速、转向、开启转向灯、开启刹车灯中的一种或多种操作；
    响应于所述交通信号灯的状态为禁止通行状态或等待状态，控制所述车辆停车、减速、开启刹车灯中的一种或多种操作。
  27. 一种交通信号灯检测装置，其特征在于，包括：
    视频流获取单元，用于获取包括有交通信号灯的视频流；
    区域确定单元，用于确定所述视频流的至少一帧图像中交通信号灯的候选区域；
    属性识别单元，用于基于所述候选区域确定所述图像中交通信号灯的至少两种属性。
  28. 根据权利要求27所述的装置，其特征在于，所述交通信号灯的至少两种属性包括以下任意两种或两种以上：位置区域、颜色和形状。
  29. 根据权利要求27或28所述的装置，其特征在于，所述区域确定单元，用于利用基于区域的全卷积网络，确定所述视频流的至少一帧图像中交通信号灯的候选区域。
  30. 根据权利要求27-29任一所述的装置，其特征在于，所述属性识别单元，用于利用多任务识别网络，基于所述候选区域确定所述图像中交通信号灯的至少两种属性。
  31. 根据权利要求30所述的装置，其特征在于，
    所述多任务识别网络包括特征提取分支、以及分别与所述特征提取分支连接的至少两个任务分支，不同的任务分支用于确定所述交通信号灯的不同种类属性；
    所述属性识别单元，包括：
    特征提取模块，用于基于所述特征提取分支对所述候选区域进行特征提取，得到候选特征；
    分支属性模块，用于分别基于所述至少两个任务分支对所述候选特征进行处理，获得所述图像中交通信号灯的至少两种属性。
  32. 根据权利要求31所述的装置，其特征在于，所述至少两个任务分支包括：检测分支、识别分支和分类分支；
    所述分支属性模块，用于经所述检测分支对所述候选特征进行位置检测，确定交通信号灯的位置区域；经所述分类分支对所述候选特征进行颜色分类，确定交通信号灯所在位置区域的颜色，确定所述交通信号灯的颜色；经所述识别分支对所述候选特征进行形状识别，确定所述交通信号灯所在位置区域的形状，确定所述交通信号灯的形状。
  33. 根据权利要求27-32任一所述的装置，其特征在于，还包括：
    关键点确定单元，用于对所述视频流中的至少一帧图像进行关键点识别，确定所述图像中的交通信号灯的关键点；
    关键点跟踪单元，用于对所述视频流中的交通信号灯的关键点进行跟踪，得到跟踪结果；
    位置调整单元，用于基于所述跟踪结果对所述交通信号灯的位置区域进行调整。
  34. 根据权利要求33所述的装置，其特征在于，所述关键点跟踪单元，用于基于连续两帧所述图像中所述交通信号灯的关键点之间的距离；基于所述交通信号灯的关键点之间的距离对所述视频流中的交通信号灯的关键点进行跟踪。
  35. 根据权利要求34所述的装置，其特征在于，所述关键点跟踪单元基于所述交通信号灯的关键点之间的距离对所述视频流中的交通信号灯的关键点进行跟踪时，用于基于所述交通信号灯的关键点之间的距离，确定连续两帧图像中同一交通信号灯的关键点的位置区域；根据所述同一交通信号灯的关键点在连续两帧所述图像中的位置区域，在所述视频流中对交通信号灯的关键点进行跟踪。
  36. 根据权利要求33-35任一所述的装置，其特征在于，所述位置调整单元，用于对比所述跟踪结果与所述信号灯的位置区域是否重合，得到对比结果；基于所述对比结果对所述信号灯的位置区域进行调整。
  37. 根据权利要求36所述的装置，其特征在于，所述位置调整单元基于所述对比结果对所述信号灯的位置区域进行调整时，用于响应于所述交通信号灯的关键点对应的位置区域和所述信号灯的位置区域不重合，以所述交通信号灯的关键点对应的位置区域替换所述信号灯的位置区域。
  38. 根据权利要求30-37任一所述的装置，其特征在于，还包括：
    预训练单元，用于基于采集的训练图像集训练所述基于区域的全卷积网络，所述训练图像集包括多个具有标注属性的训练图像；
    训练单元，用于基于所述训练图像集调整所述基于区域的全卷积网络和所述多任务识别网络中的参数。
  39. 根据权利要求38所述的装置，其特征在于，还包括：
    分类训练单元，用于基于所述训练图像集获取交通信号灯的颜色比例符合预设比例的新训练图像集；基于所述新训练图像集训练分类网络；所述分类网络用于基于所述交通信号灯的颜色对所述训练图像进行分类。
  40. 根据权利要求39所述的装置，其特征在于，所述预设比例中不同颜色的交通信号灯的数量相同或者数量差异小于容许阈值；
    所述交通信号灯的颜色包括红色、黄色和绿色。
  41. 根据权利要求40所述的装置，其特征在于，还包括：
    初始化单元，用于基于所述训练后的分类网络的参数初始化所述多任务识别网络中的至少部分参数。
  42. 根据权利要求27-41任一所述的装置，其特征在于，还包括：
    状态确定单元，用于基于所述图像中交通信号灯的至少两种属性确定所述交通信号灯的状态；
    智能控制单元，用于根据所述交通信号灯的状态对所述车辆进行智能驾驶控制。
  43. 根据权利要求42所述的装置，其特征在于，所述智能驾驶控制包括：发出提示信息或告警信息，和/或，根据所述交通信号灯的状态控制所述车辆的行驶状态。
  44. 根据权利要求42或43所述的装置，其特征在于，还包括：
    存储单元，用于存储所述交通信号灯的属性、状态及其对应的所述图像。
  45. 根据权利要求42-44任一所述的装置，其特征在于，所述交通信号灯的状态包括：允许通行状态、禁止通行状态或等待状态；
    所述状态确定单元，用于响应于所述交通信号灯的颜色为绿色和/或形状为第一预设形状，确定所述交通信号灯的状态为允许通行状态；
    响应于所述交通信号灯的颜色为红色和/或形状为第二预设形状，确定所述交通信号灯的状态为禁止通行状态；
    响应于所述交通信号灯的颜色为黄色和/或形状为第三预设形状，确定所述交通信号灯的状态为等待状态。
  46. 根据权利要求45所述的装置，其特征在于，所述智能控制单元，用于响应于所述交通信号灯的状态为允许通行状态，控制所述车辆执行启动、保持行驶状态、减速、转向、开启转向灯、开启刹车灯中的一种或多种操作；
    响应于所述交通信号灯的状态为禁止通行状态或等待状态，控制所述车辆停车、减速、开启刹车灯中的一种或多种操作。
  47. 一种智能驾驶装置，其特征在于，包括：
    视频流获取单元，用于基于设置在车辆上的图像采集装置获取包括有交通信号灯的视频流；
    区域确定单元，用于确定所述视频流的至少一帧图像中交通信号灯的候选区域；
    属性识别单元，用于基于所述候选区域确定所述图像中交通信号灯的至少两种属性；
    状态确定单元，用于基于所述图像中交通信号灯的至少两种属性确定所述交通信号灯的状态；
    智能控制单元，用于根据所述交通信号灯的状态对所述车辆进行智能驾驶控制。
  48. 根据权利要求47所述的装置，其特征在于，所述智能驾驶控制包括：发出提示信息或告警信 息，和/或，根据所述交通信号灯的状态控制所述车辆的行驶状态。
  49. 根据权利要求47或48所述的装置，其特征在于，还包括：
    存储单元，用于存储所述交通信号灯的属性、状态及其对应的所述图像。
  50. 根据权利要求47-49任一所述的装置，其特征在于，所述交通信号灯的至少两种属性包括以下任意两种或两种以上：位置区域、颜色和形状。
  51. 根据权利要求50所述的装置，其特征在于，所述交通信号灯的状态包括：允许通行状态、禁止通行状态和等待状态；
    所述状态确定单元，用于响应于所述交通信号灯的颜色为绿色和/或形状为第一预设形状，确定所述交通信号灯的状态为允许通行状态；
    响应于所述交通信号灯的颜色为红色和/或形状为第二预设形状，确定所述交通信号灯的状态为禁止通行状态；
    响应于所述交通信号灯的颜色为黄色和/或形状为第三预设形状，确定所述交通信号灯的状态为等待状态。
  52. 根据权利要求51所述的装置，其特征在于，所述智能控制单元，用于响应于所述交通信号灯的状态为允许通行状态，控制所述车辆执行启动、保持行驶状态、减速、转向、开启转向灯、开启刹车灯中的一种或多种操作；
    响应于所述交通信号灯的状态为禁止通行状态或等待状态，控制所述车辆停车、减速、开启刹车灯中的一种或多种操作。
  53. 一种车辆，其特征在于，包括权利要求27至46任意一项所述的交通信号灯检测装置或权利要求47至52任意一项所述的智能驾驶装置。
  54. 一种电子设备，其特征在于，包括处理器，所述处理器包括权利要求27至46任意一项所述的交通信号灯检测装置或权利要求47至52任意一项所述的智能驾驶装置。
  55. 一种电子设备，其特征在于，包括：存储器，用于存储可执行指令；
    以及处理器，用于与所述存储器通信以执行所述可执行指令从而完成权利要求1至20任意一项所述交通信号灯检测方法的操作，或完成权利要求21至26任意一项所述的智能驾驶方法的操作。
  56. 一种计算机可读存储介质，用于存储计算机可读取的指令，其特征在于，所述指令被执行时执行权利要求1至20任意一项所述交通信号灯检测方法或权利要求21至26任意一项所述的智能驾驶方法的操作。
  57. 一种计算机程序产品，包括计算机可读代码，其特征在于，当所述计算机可读代码在设备上运行时，所述设备中的处理器执行用于实现权利要求1至20任意一项所述交通信号灯检测方法或权利要求21至26任意一项所述的智能驾驶方法的指令。
PCT/CN2019/089062 2018-06-29 2019-05-29 交通信号灯检测及智能驾驶方法和装置、车辆、电子设备 WO2020001223A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
SG11202007333PA SG11202007333PA (en) 2018-06-29 2019-05-29 Traffic light detection method and apparatus, intelligent driving method and apparatus, vehicle, and electronic device
JP2020550090A JP7111827B2 (ja) 2018-06-29 2019-05-29 交通信号灯検出方法、インテリジェントドライブ方法及び装置、車両並びに電子機器
KR1020207029615A KR102447352B1 (ko) 2018-06-29 2019-05-29 교통 신호등 검출 및 지능형 주행을 위한 방법 및 디바이스, 차량, 및 전자 디바이스
US16/944,234 US20200353932A1 (en) 2018-06-29 2020-07-31 Traffic light detection method and apparatus, intelligent driving method and apparatus, vehicle, and electronic device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810697683.9A CN110660254B (zh) 2018-06-29 2018-06-29 交通信号灯检测及智能驾驶方法和装置、车辆、电子设备
CN201810697683.9 2018-06-29

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/944,234 Continuation US20200353932A1 (en) 2018-06-29 2020-07-31 Traffic light detection method and apparatus, intelligent driving method and apparatus, vehicle, and electronic device

Publications (1)

Publication Number Publication Date
WO2020001223A1 true WO2020001223A1 (zh) 2020-01-02

Family

ID=68985543

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/089062 WO2020001223A1 (zh) 2018-06-29 2019-05-29 交通信号灯检测及智能驾驶方法和装置、车辆、电子设备

Country Status (6)

Country Link
US (1) US20200353932A1 (zh)
JP (1) JP7111827B2 (zh)
KR (1) KR102447352B1 (zh)
CN (1) CN110660254B (zh)
SG (1) SG11202007333PA (zh)
WO (1) WO2020001223A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113674544A (zh) * 2021-07-21 2021-11-19 阿波罗智联(北京)科技有限公司 信号机的报警方法、装置、电子设备和可读存储介质
WO2022020709A1 (en) * 2020-07-23 2022-01-27 Waymo Llc Detecting traffic signaling states with neural networks
CN116681935A (zh) * 2023-05-31 2023-09-01 国家深海基地管理中心 一种深海热液喷口自主识别定位方法及系统

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11866046B2 (en) * 2017-12-10 2024-01-09 Anatoly S. Weiser Smart traffic control devices and beacons, methods of their operation, and use by vehicles of information provided by the devices and beacons
US10836393B2 (en) * 2017-12-10 2020-11-17 Anatoly S. Weiser Smart traffic control devices and beacons, methods of their operation, and use by vehicles of information provided by the devices and beacons
CN110688500B (zh) * 2018-06-20 2021-09-14 华为技术有限公司 一种数据库构建方法、一种定位方法及其相关设备
DE112020002604T5 (de) * 2019-05-29 2022-03-03 Mobileye Vision Technologies Ltd. Systeme und verfahren für die fahrzeugnavigation
KR20210052525A (ko) * 2019-06-27 2021-05-10 센스타임 그룹 리미티드 표시등의 지시 정보 인식 방법 및 장치, 전자 기기 및 기억 매체
CN111292531B (zh) * 2020-02-06 2022-07-29 阿波罗智能技术(北京)有限公司 交通信号灯的跟踪方法、装置、设备及存储介质
CN111400531B (zh) * 2020-03-13 2024-04-05 广州文远知行科技有限公司 目标标注方法、装置、设备与计算机可读存储介质
JP7115502B2 (ja) 2020-03-23 2022-08-09 トヨタ自動車株式会社 物体状態識別装置、物体状態識別方法及び物体状態識別用コンピュータプログラムならびに制御装置
JP7359735B2 (ja) * 2020-04-06 2023-10-11 トヨタ自動車株式会社 物体状態識別装置、物体状態識別方法及び物体状態識別用コンピュータプログラムならびに制御装置
JP7388971B2 (ja) 2020-04-06 2023-11-29 トヨタ自動車株式会社 車両制御装置、車両制御方法及び車両制御用コンピュータプログラム
CN112289021A (zh) * 2020-09-24 2021-01-29 深圳一清创新科技有限公司 一种交通信号灯的检测方法、装置及自动驾驶汽车
KR20220083945A (ko) * 2020-12-11 2022-06-21 현대자동차주식회사 신호등 정보 제공 장치, 그를 포함한 차량 시스템 및 그 방법
CN112507951B (zh) * 2020-12-21 2023-12-12 阿波罗智联(北京)科技有限公司 指示灯识别方法、装置、设备、路侧设备和云控平台
US11776281B2 (en) * 2020-12-22 2023-10-03 Toyota Research Institute, Inc. Systems and methods for traffic light detection and classification
CN112699773B (zh) * 2020-12-28 2023-09-01 阿波罗智联(北京)科技有限公司 交通灯识别方法、装置及电子设备
CN112349101B (zh) * 2021-01-08 2021-04-27 深圳裹动智驾科技有限公司 高精地图生成方法、以及识别交通灯的方法及系统
CN113011251B (zh) * 2021-02-03 2024-06-04 深圳大学 一种基于交通灯几何属性的行人交通灯识别方法
US12014549B2 (en) 2021-03-04 2024-06-18 Toyota Research Institute, Inc. Systems and methods for vehicle light signal classification
CN113033464B (zh) * 2021-04-10 2023-11-21 阿波罗智联(北京)科技有限公司 信号灯检测方法、装置、设备以及存储介质
CN113077630B (zh) * 2021-04-30 2022-06-28 安徽江淮汽车集团股份有限公司 基于深度学习的红绿灯检测方法、装置、设备及存储介质
CN113450588B (zh) * 2021-06-28 2023-04-07 通视(天津)信息技术有限公司 交通信号灯时信息的处理方法、装置以及电子设备
CN113469109B (zh) * 2021-07-16 2022-09-13 阿波罗智联(北京)科技有限公司 交通灯识别结果处理方法、装置、路侧设备及云控平台
CN113989774B (zh) * 2021-10-27 2024-11-08 广州小鹏自动驾驶科技有限公司 一种交通灯检测方法、装置、车辆和可读存储介质
CN114067291A (zh) * 2021-11-24 2022-02-18 智道网联科技(北京)有限公司 信号灯检测方法和装置
KR102472649B1 (ko) * 2021-12-28 2022-11-30 포티투닷 주식회사 객체를 트래킹하기 위한 방법 및 장치
CN114782924B (zh) * 2022-05-10 2024-07-12 智道网联科技(北京)有限公司 用于自动驾驶的交通灯检测方法、装置及电子设备
CN114973205B (zh) * 2022-06-28 2024-08-13 深圳一清创新科技有限公司 一种红绿灯跟踪方法、装置、及无人驾驶汽车
CN115984823B (zh) * 2023-02-27 2023-06-09 安徽蔚来智驾科技有限公司 交通信号灯感知方法、车辆控制方法、设备、介质及车辆
WO2024197629A1 (zh) * 2023-03-29 2024-10-03 华为技术有限公司 检测方法、装置以及智能驾驶设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106650641A (zh) * 2016-12-05 2017-05-10 北京文安智能技术股份有限公司 一种交通信号灯定位识别方法、装置及系统
CN106909937A (zh) * 2017-02-09 2017-06-30 北京汽车集团有限公司 交通信号灯识别方法、车辆控制方法、装置及车辆
CN107527511A (zh) * 2016-06-22 2017-12-29 杭州海康威视数字技术股份有限公司 一种智能车辆行车提醒方法及装置
CN107978165A (zh) * 2017-12-12 2018-05-01 南京理工大学 基于计算机视觉的交叉口标志标线与信号灯智能感知方法

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4357137B2 (ja) * 2001-05-11 2009-11-04 富士通マイクロエレクトロニクス株式会社 移動体追跡方法及びシステム
US6999004B2 (en) * 2002-06-17 2006-02-14 Siemens Corporate Research, Inc. System and method for vehicle detection and tracking
CN103489324B (zh) * 2013-09-22 2015-09-09 北京联合大学 一种基于无人驾驶的实时动态红绿灯检测识别方法
KR20150047214A (ko) * 2013-10-24 2015-05-04 현대모비스 주식회사 카메라 센서를 이용한 ｓｃｃ 시스템 및 방법
WO2015077514A1 (en) * 2013-11-20 2015-05-28 Certusview Technologies, Llc Systems, methods, and apparatus for tracking an object
CN103729863B (zh) * 2013-12-06 2016-05-25 南京金智视讯技术有限公司 基于自主学习的交通信号灯全自动定位识别的方法
CN106462727B (zh) * 2014-01-30 2020-02-11 移动眼视力科技有限公司 车辆、车道终止检测系统和方法
WO2015175826A1 (en) * 2014-05-14 2015-11-19 Kapach Yair Systems and methods for detecting traffic signs
US9779314B1 (en) 2014-08-21 2017-10-03 Waymo Llc Vision-based detection and classification of traffic lights
CN105893971A (zh) * 2016-04-01 2016-08-24 上海理工大学 一种基于Gabor和稀疏表示的交通信号灯识别方法
CN106023623A (zh) * 2016-07-28 2016-10-12 南京理工大学 基于机器视觉的车载交通信号与标志的识别及预警方法
CN106570494A (zh) * 2016-11-21 2017-04-19 北京智芯原动科技有限公司 基于卷积神经网络的交通信号灯识别方法及装置
CN106897742B (zh) * 2017-02-21 2020-10-27 北京市商汤科技开发有限公司 用于检测视频中物体的方法、装置和电子设备
CN106837649B (zh) * 2017-03-03 2018-06-22 吉林大学 基于信号灯倒计时识别的自学习智能起停系统
WO2018175441A1 (en) * 2017-03-20 2018-09-27 Mobileye Vision Technologies Ltd. Navigation by augmented path prediction
US10699142B2 (en) * 2017-04-20 2020-06-30 GM Global Technology Operations LLC Systems and methods for traffic signal light detection
CN108108761B (zh) * 2017-12-21 2020-05-01 西北工业大学 一种基于深度特征学习的快速交通信号灯检测方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107527511A (zh) * 2016-06-22 2017-12-29 杭州海康威视数字技术股份有限公司 一种智能车辆行车提醒方法及装置
CN106650641A (zh) * 2016-12-05 2017-05-10 北京文安智能技术股份有限公司 一种交通信号灯定位识别方法、装置及系统
CN106909937A (zh) * 2017-02-09 2017-06-30 北京汽车集团有限公司 交通信号灯识别方法、车辆控制方法、装置及车辆
CN107978165A (zh) * 2017-12-12 2018-05-01 南京理工大学 基于计算机视觉的交叉口标志标线与信号灯智能感知方法

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022020709A1 (en) * 2020-07-23 2022-01-27 Waymo Llc Detecting traffic signaling states with neural networks
US11328519B2 (en) 2020-07-23 2022-05-10 Waymo Llc Detecting traffic signaling states with neural networks
US11783596B2 (en) 2020-07-23 2023-10-10 Waymo Llc Detecting traffic signaling states with neural networks
CN113674544A (zh) * 2021-07-21 2021-11-19 阿波罗智联(北京)科技有限公司 信号机的报警方法、装置、电子设备和可读存储介质
US11955004B2 (en) 2021-07-21 2024-04-09 Apollo Intelligent Connectivity (Beijing) Technology Co., Ltd. Signal machine alarm method and apparatus, electronic device and readable storage medium
CN116681935A (zh) * 2023-05-31 2023-09-01 国家深海基地管理中心 一种深海热液喷口自主识别定位方法及系统
CN116681935B (zh) * 2023-05-31 2024-01-23 国家深海基地管理中心 一种深海热液喷口自主识别定位方法及系统

Also Published As

Publication number Publication date
US20200353932A1 (en) 2020-11-12
JP7111827B2 (ja) 2022-08-02
KR102447352B1 (ko) 2022-09-23
SG11202007333PA (en) 2020-08-28
JP2021519968A (ja) 2021-08-12
CN110660254B (zh) 2022-04-08
CN110660254A (zh) 2020-01-07
KR20200128145A (ko) 2020-11-11

Similar Documents

Publication Publication Date Title
WO2020001223A1 (zh) 交通信号灯检测及智能驾驶方法和装置、车辆、电子设备
JP7106664B2 (ja) 知的運転制御方法および装置、電子機器、プログラムならびに媒体
US11840239B2 (en) Multiple exposure event determination
WO2020048265A1 (zh) 多级目标分类及交通标志检测方法和装置、设备、介质
Wu et al. Applying a functional neurofuzzy network to real-time lane detection and front-vehicle distance measurement
CN108335489A (zh) 高速公路车辆行为语义分析及异常行为监控系统及方法
CN105205785B (zh) 一种可定位的大型车辆运行管理系统及其运行方法
US9965677B2 (en) Method and system for OCR-free vehicle identification number localization
Li et al. Lane detection based on connection of various feature extraction methods
KR20210080459A (ko) 차선 검출방법, 장치, 전자장치 및 가독 저장 매체
Widyantoro et al. Traffic lights detection and recognition based on color segmentation and circle hough transform
Hatolkar et al. A survey on road traffic sign recognition system using convolution neural network
Liu et al. Real-time traffic light recognition based on smartphone platforms
KR20230048434A (ko) 자기 차량의 자율 및 지원 운전에서 환경 특징을 연속적이고 적응적으로 검출하는 컴퓨터 구현 방법
Luo et al. Detection and recognition of obscured traffic signs during vehicle movement
Bhupathi et al. Sharp curve detection of autonomous vehicles using dbscan and augmented sliding window techniques
CA3196453A1 (en) Lane line detection method and apparatus
Kalms et al. Robust lane recognition for autonomous driving
Oza et al. Object Detection using IoT and Machine Learning to Avoid Accident and Improve Road Safety
Li et al. A multiple-lane vehicle tracking method for forward collision warning system applications
Erwin et al. Detection of highway lane using color filtering and line determination
Yang An enhanced three-frame-differencing approach for vehicle detection under challenging environmental conditions
Du et al. Traffic light recognition based on prior knowledge and optimized threshold segmentation
Khalfin et al. Developing, Analyzing, and Evaluating Vehicular Lane Keeping Algorithms Under Dynamic Lighting and Weather Conditions Using Electric Vehicles
Kenk et al. Driving Perception in Challenging Road Scenarios: An Empirical Study

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19827213

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020550090

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 20207029615

Country of ref document: KR

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19827213

Country of ref document: EP

Kind code of ref document: A1