WO2020042984A1

WO2020042984A1 - Vehicle behavior detection method and apparatus

Info

Publication number: WO2020042984A1
Application number: PCT/CN2019/101807
Authority: WO
Inventors: 虞抒沁
Original assignee: 杭州海康威视数字技术股份有限公司
Priority date: 2018-08-28
Filing date: 2019-08-21
Publication date: 2020-03-05
Also published as: CN110866427A

Abstract

Disclosed are a vehicle behavior detection method and apparatus. The vehicle behavior detection method comprises: a vehicle-mounted intelligent device collects a video frame, recognizes a vehicle behavior according to the video frame so as to obtain a recognition result that the vehicle behavior is abnormal, and uploads the recognition result to a server, so that the server performs vehicle behavior alarm according to the recognition result. The present application can effectively recognize various abnormal vehicle behaviors and improves urban traffic efficiency.

Description

Method and device for detecting vehicle behavior

This application claims priority from a Chinese patent application filed with the Chinese Patent Office on August 28, 2018, with application number 201810986173.3, and the invention name is "A Vehicle Behavior Detection Method and Device", the entire contents of which are incorporated herein by reference. in.

Technical field

The present application relates to the field of machine vision, and in particular, to a method and device for detecting vehicle behavior.

Background technique

With the continuous progress of society, the scope of application of video analysis is getting wider and wider. The related digital monitoring system is far from meeting the needs of many applications, which is mainly reflected in the lack of intelligence and the widespread application. Taking the field of traffic accident detection as an example, on the one hand, at present, monitoring is only provided at various checkpoints and intersections for simple violations of evidence. Not only is there a large blind spot in monitoring, but an accident occurs anywhere, and traffic police still need to arrive at the scene for accident and liability identification. In the time period when the accident vehicle occupies the lane and waits for the traffic police to reach the scene, traffic will be seriously affected; on the other hand, behavior recognition based on deep learning technology has begun to be applied to some human-based motion recognition in recent years, but for vehicle accidents or Illegal identification was not noticed.

At present, there are patent applications for similar technologies, such as US20180121731A1, which discloses a method for synthesizing images using a 3D convolutional network and using the LSTM network to realize body image analysis and judgment. However, this method uses external camera acquisition The graphics or text included in the vehicle, etc., identify the graphics or text, it is not disclosed that the vehicle terminal's video detection device can be used to identify traffic violations or accidents; for example, CN108133172A, discloses a method for collecting target vehicle information using machine learning The video images and motion trajectories are analyzed to determine traffic violations and accidents. However, this method does not disclose that the video detection device of the vehicle terminal can be used to perform image acquisition and transmission.

Therefore, in the field of vehicle behavior detection based on video analysis, how to timely and effectively identify various illegal vehicle behaviors such as illegal driving and traffic accidents to improve urban traffic efficiency is still an urgent problem to be solved and improved.

Summary of the Invention

The purpose of the embodiments of the present application is to provide a method and a device for detecting vehicle behavior, so as to realize timely and effective identification of abnormal vehicle behavior, thereby improving urban traffic efficiency. Specific technical solutions are as follows:

In a first aspect, an embodiment of the present application provides a vehicle behavior detection method, which is applied to a vehicle-mounted smart device. The method includes:

Capture video frames;

Identifying vehicle behavior based on the video frame to obtain a recognition result of abnormal vehicle behavior;

The recognition result is uploaded to a server, so that the server performs a vehicle behavior alarm according to the recognition result.

In a second aspect, an embodiment of the present application provides a vehicle behavior detection method, which is applied to a server, and the method includes:

Receiving a recognition result sent by a vehicle-mounted smart device, where the recognition result is a recognition result of abnormal vehicle behavior obtained by the vehicle-mounted smart device based on the collected video frames;

According to the recognition result, a vehicle behavior alarm is performed.

In a third aspect, an embodiment of the present application provides a vehicle behavior detection method, which is applied to a vehicle-mounted smart device, and the method includes:

Capture video frames;

Uploading the video frame to a server, so that the server recognizes vehicle behavior based on the video frame, obtains a recognition result of abnormal vehicle behavior, and performs a vehicle behavior alarm according to the recognition result.

In a fourth aspect, an embodiment of the present application provides a vehicle behavior detection method, which is applied to a server, and the method includes:

Receiving video frames sent by vehicle smart devices;

According to the recognition result, a vehicle behavior alarm is performed.

In a fifth aspect, an embodiment of the present application provides a vehicle behavior detection device applied to a vehicle-mounted smart device, where the device includes:

An acquisition module for acquiring video frames;

A data processing module, configured to identify vehicle behavior based on the video frame, and obtain a recognition result of abnormal vehicle behavior;

A data transmission module, configured to upload the recognition result to a server.

According to a sixth aspect, an embodiment of the present application provides a vehicle behavior detection device, which is applied to a server, and the device includes:

A receiving module, configured to receive a recognition result sent by a vehicle-mounted intelligent device, where the recognition result is a recognition result of abnormal vehicle behavior obtained by the vehicle-mounted intelligent device based on the collected video frames;

A comprehensive alarm module is configured to perform a vehicle behavior alarm according to the recognition result.

In a seventh aspect, an embodiment of the present application provides a vehicle behavior detection device, which is applied to a vehicle-mounted smart device, and the device includes:

An acquisition module for acquiring video frames;

A data transmission module is configured to upload the video frame to a server.

In an eighth aspect, an embodiment of the present application provides a vehicle behavior detection device, which is applied to a server, and the device includes:

A receiving module, configured to receive a video frame sent by a vehicle intelligent device;

In a ninth aspect, an embodiment of the present application provides a vehicle behavior detection system. The system includes: a vehicle-mounted intelligent device and a server;

The in-vehicle intelligent device is used to collect video frames; identify vehicle behaviors based on the video frames to obtain recognition results of abnormal vehicle behaviors; and upload the recognition results to the server;

The server is configured to receive the recognition result sent by the in-vehicle smart device; and perform a vehicle behavior alarm according to the recognition result.

In a tenth aspect, an embodiment of the present application provides a vehicle behavior detection system. The system includes: a vehicle-mounted intelligent device and a server;

The in-vehicle smart device is used to collect video frames; upload the video frames to the server;

The server is configured to receive the video frame sent by the in-vehicle smart device; identify vehicle behavior based on the video frame to obtain a recognition result of abnormal vehicle behavior; and perform a vehicle behavior alarm based on the recognition result.

According to an eleventh aspect, an embodiment of the present application provides a vehicle-mounted smart device, including a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and therefore The processor is caused by the machine executable instructions to execute the method provided by the first aspect of the embodiments of the present application.

In a twelfth aspect, an embodiment of the present application provides a machine-readable storage medium that stores machine-executable instructions. When called and executed by a processor, the machine-executable instructions cause the processor to execute the implementation of the present application. The method provided by the first aspect of the example.

In a thirteenth aspect, an embodiment of the present application provides an application program, which is characterized in that it is used to execute at runtime: the method provided in the first aspect of the embodiment of the present application.

In a fourteenth aspect, an embodiment of the present application provides a server, including a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions that can be executed by the processor. The processor is caused by the machine executable instructions to execute the method provided by the second aspect of the embodiments of the present application.

In a fifteenth aspect, an embodiment of the present application provides a machine-readable storage medium that stores machine-executable instructions. When called and executed by a processor, the machine-executable instructions cause the processor to execute the implementation of the present application. The method provided by the second aspect of the example.

In a sixteenth aspect, an embodiment of the present application provides an application program for executing at runtime: the method provided in the second aspect of the embodiment of the present application.

In a seventeenth aspect, an embodiment of the present application provides a vehicle-mounted smart device, including a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and The processor is caused by the machine executable instructions to execute the method provided by the third aspect of the embodiments of the present application.

In an eighteenth aspect, an embodiment of the present application provides a machine-readable storage medium that stores machine-executable instructions. When called and executed by a processor, the machine-executable instructions cause the processor to execute the implementation of the present application. The method provided by the third aspect of the example.

In a nineteenth aspect, an embodiment of the present application provides an application program for executing at runtime: the method provided in the third aspect of the embodiment of the present application.

In a twentieth aspect, an embodiment of the present application provides a server, including a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processing The machine is caused by the machine executable instructions to execute the method provided by the fourth aspect of the embodiments of the present application.

In a twenty-first aspect, an embodiment of the present application provides a machine-readable storage medium that stores machine-executable instructions. When called and executed by a processor, the machine-executable instructions cause the processor to execute the application. The method provided in the fourth aspect of the embodiment.

In the twenty-second aspect, the embodiment of the present application provides an application program for executing at runtime: the method provided in the fourth aspect of the embodiment of the present application.

This application proposes the following solution: video capture based on vehicle-mounted smart devices such as driving recorders or smartphones, and local processing to identify abnormal vehicle behaviors, upload the recognition results to the server, and perform vehicle alarms through big data analysis to enhance road driving Safety and acceleration of small and micro accidents to help improve urban traffic efficiency.

This application also proposes the following solution: video capture is performed based on a vehicle-mounted smart device such as a driving recorder, or a smartphone, and the collected video frames are uploaded to the server, and abnormal vehicle behavior is identified through big data analysis, and an alarm message is issued to enhance road driving Safety and acceleration of small and micro accidents to help improve urban traffic efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly explain the embodiments of the present application and the technical solutions of the prior art, the following briefly introduces the drawings used in the embodiments and the prior art. Obviously, the drawings in the following description are only the present invention. Some embodiments of the application, for those of ordinary skill in the art, can obtain other drawings according to the drawings without paying creative labor.

1 is a schematic flowchart of a vehicle behavior detection method according to an embodiment of the present application;

2 is a schematic structural diagram of a vehicle behavior detection device applied to a vehicle-mounted smart device according to an embodiment of the present application;

3 is a schematic structural diagram of a vehicle behavior detection device applied to a server according to an embodiment of the present application;

4 is a schematic flowchart of a vehicle behavior detection method according to another embodiment of the present application;

5 is a schematic structural diagram of a vehicle behavior detection device applied to a vehicle-mounted smart device according to another embodiment of the present application;

6 is a schematic structural diagram of a vehicle behavior detection device applied to a server according to another embodiment of the present application;

7 is a schematic diagram of a CNN encoding process in the related art;

8 is a schematic diagram of a 3D CNN encoding process according to an embodiment of the present application;

9 is a schematic diagram of an alarm process of a vehicle behavior detection system according to an embodiment of the present application;

10 is a schematic diagram of an alarm process of a vehicle behavior detection system according to another embodiment of the present application;

11 is a schematic structural diagram of a vehicle-mounted smart device according to an embodiment of the present application;

12 is a schematic structural diagram of a server according to an embodiment of the present application;

13 is a schematic structural diagram of a vehicle-mounted smart device according to another embodiment of the present application;

FIG. 14 is a schematic structural diagram of a server according to another embodiment of the present application.

detailed description

In order to make the purpose, technical solution, and advantages of the present application clearer and clearer, the following describes the present application in detail with reference to the accompanying drawings and examples. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

According to an embodiment of the present application, a vehicle behavior detection method is provided, as shown in FIG. 1, including:

Step 101: a data collection step.

The vehicle-mounted intelligent device may be a device with a video capture function such as a driving recorder, a smart phone, etc. The vehicle-mounted intelligent device is often installed in the front end of the vehicle, and the vehicle-mounted intelligent device performs video collection.

Optionally, step 101 may include: the vehicle-mounted smart device collects multiple video frames at a fixed time interval.

Exemplarily, the in-vehicle intelligent device includes a driving recorder, a mobile communication device, or another camera device having a data transmission function. The vehicle-mounted smart device collects video through its camera module to obtain a video file, where the video file includes multiple video frames. When the camera module performs device collection, the camera module can perform multiple video frames collected at fixed time intervals.

Optionally, step 101 may include: the vehicle-mounted smart device sets a detection range around the vehicle to which it belongs, and photographs one or more vehicles within the detection range to obtain multiple video frames.

Step 102: a data processing step.

This step is mainly a process of identifying vehicle behavior based on video frames, and this step can be performed on a vehicle-mounted smart device. That is, the vehicle-mounted smart device can directly recognize the vehicle behavior based on the collected video frames, and obtain the recognition result of the abnormal vehicle behavior.

Optionally, step 102 may include: the in-vehicle intelligent device inputs video frames into a neural network, and uses the neural network to perform feature extraction on the video frames to generate multi-dimensional behavior feature vectors; each of the multi-dimensional behavior feature vectors The dimensions are logically classified to obtain the confidence level that the behavior feature vectors of each dimension are abnormal vehicle behaviors of different event types; if the confidence level of abnormal vehicle behaviors of any event type is greater than a preset threshold, it is determined that the event type exists in the video frame Abnormal vehicle behavior.

Exemplarily, the above recognition result of abnormal vehicle behavior means that it is determined whether a vehicle in the video frame has abnormal vehicle behavior, and whether the event type of the abnormal vehicle behavior is a type of dangerous driving, traffic accident, etc., and the identification result is obtained. The determination rule is based on whether the confidence level output by the neural network model is greater than a preset threshold. That is, if the confidence level of the dangerous driving is greater than a preset threshold after logical classification, it is determined that the current video frame includes abnormal vehicle behavior whose event type is dangerous driving. If the confidence level of the traffic accident is greater than a preset threshold after logical classification, then the current video frame includes abnormal vehicle behavior with the event type being a traffic accident. The training process of the neural network model is not repeated here.

The types of incidents of abnormal vehicle behavior include dangerous driving, traffic accidents, and so on. Dangerous driving includes chasing, drunk driving, and fatigue driving. Among them, the judgment of chasing driving includes passing through the video frames of the position and distance of the car in front and rear, and the distance changes with time. The situation of changing video frames, such as the shaking of the torso of the driver ’s body, and the judgment of fatigue driving includes the changes of the vehicle ’s driving trajectory and the changes of the driver ’s body, such as the opening and closing of the driver ’s eyes and the position of the driver ’s hands Traffic accidents include certain types / forms of traffic accidents. Among them, traffic accidents include rear-end accidents, overtaking accidents, left-turn accidents, and transition accidents, and morphologically include collisions, scratches, rolling, and rollovers. , Crash, fire, etc.

Exemplarily, the neural network includes, but is not limited to, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM) neural network, and the like.

Exemplarily, the feature mapping structure uses the sigmoid function with a small influence function kernel as the activation function of the convolutional network. The sigmoid function is binary classification, and each class can output independent confidence, so when the difference between classes is small It can give several class outputs with high confidence at the same time, and it will not suppress similar class features when training the model, so as to obtain better model performance.

Exemplarily, step 102 belongs to the smart device to execute locally. The above-mentioned smart device executes locally, which means that during the execution of step 102, the smart device that is local to the vehicle does not need to upload the collected data to the server, and performs the above-mentioned data processing process only on the on-board smart device that is local to a single vehicle.

Step 103: a data transmission step.

Optionally, step 103 may include: the in-vehicle smart device uploads a video frame with an abnormal vehicle behavior and an event type of the abnormal vehicle behavior to the server.

The in-vehicle intelligent device can upload data related to the driving behavior of all vehicles on the road and the recognition results of abnormal vehicle behavior to the server in real time.

For example, the structured analysis data and key video data are uploaded to the server. The structured analysis data includes identification results of abnormal vehicle behavior and / or intermediate process data embodied in a database form. The key video data includes a plurality of frames for determining that there is abnormal vehicle behavior.

Exemplarily, the data transmission mode includes a wireless network communication mode.

Step 104: Comprehensive alarm step.

Optionally, this step includes the server performing vehicle behavior analysis and alarm.

Exemplarily, by analyzing the recognition results of abnormal vehicle behaviors of all vehicles on a road in a certain area uploaded in step 103, alarms and post-processing are performed. Wherein, the alarm includes that the server sends alarm information to the vehicle through the vehicle-mounted smart device via the wireless network according to the received recognition result, and sends the alarm information to the surrounding vehicles of the vehicle through the vehicle-mounted smart device via the wireless network; The type of traffic accident is dealt with according to a certain type / form. Minor accidents will be directly penalized and follow-up clearance measures will be made to restore road traffic to normal as soon as possible to help improve urban traffic efficiency.

Optionally, step 104 may include: sending an alarm message to a vehicle having abnormal vehicle behavior, and a plurality of vehicles having a distance from the vehicle within a preset range.

Exemplarily, a plurality of vehicles whose distances from the vehicle are within a preset range are surrounding vehicles of the vehicle, including all vehicles with a network receiving function enabled within a set range.

The implementation of the above-mentioned alarm information includes audio, video and somatosensory alarm messages to the driver, as well as audio, video and somatosensory alarm messages to the co-pilot or passenger.

According to the scheme proposed in this application, at least the following beneficial effects can be achieved:

1. Able to identify various types of incidents such as illegal driving and traffic accidents;

2.Using in-vehicle intelligent equipment and big data analysis to solve the problem of monitoring blind spots, which can monitor the safe driving of vehicles throughout the road and handle accidents at the monitoring blind spots;

3. Make full use of the local processing capabilities of the vehicle-mounted smart device, reduce the data processing pressure on the server side, and reduce the hardware requirements for the server. Through local processing and the server judges the alarm, the vehicle alarm has a higher real-time nature, and the road safety and And / or the technical effect of speeding up the handling of the accident.

Corresponding to the above steps, the present application proposes a vehicle behavior detection device, as shown in FIG. 2, which is applied to a vehicle-mounted intelligent device. The device includes:

An acquisition module 201 is configured to acquire a video frame.

The data processing module 202 is configured to identify a vehicle behavior based on the video frame, and obtain a recognition result of abnormal vehicle behavior.

A data transmission module 203 is configured to upload the recognition result to a server.

Optionally, the acquisition module 201 may be specifically configured to collect multiple video frames at a fixed time interval.

Optionally, the data processing module 202 may be specifically configured to: input the video frame into a neural network, use the neural network to perform feature extraction on the video frame, and generate a multi-dimensional behavior feature vector; Each dimension in the feature vector is logically classified to obtain the confidence level that the behavior feature vector of each dimension is an abnormal vehicle behavior of a different event type; if the confidence level of the abnormal vehicle behavior of any event type is greater than a preset threshold, determine all The abnormal vehicle behavior of this event type is described in the video frame.

Optionally, the data transmission module 203 may be specifically configured to: upload a video frame with abnormal vehicle behavior and an event type of the abnormal vehicle behavior to a server.

Optionally, the acquisition module 201 may be specifically configured to set a detection range around the vehicle to which the vehicle-mounted smart device belongs, and shoot one or more vehicles within the detection range to obtain multiple video frames.

This application proposes a vehicle behavior detection device, as shown in FIG. 3, which is applied to a server. The device includes:

The receiving module 301 is configured to receive a recognition result sent by a vehicle-mounted smart device, where the recognition result is a recognition result of abnormal vehicle behavior obtained by the vehicle-mounted smart device based on the collected video frames;

A comprehensive alarm module 302 is configured to perform a vehicle behavior alarm according to the recognition result.

Optionally, the comprehensive alarm module 302 may be specifically configured to: issue an alarm message to a vehicle having abnormal vehicle behavior, and a plurality of vehicles having a distance from the vehicle within a preset range.

According to another embodiment of the present application, a vehicle behavior detection method is provided, as shown in FIG. 4, including:

Step 111: a data collection step.

Optionally, step 111 may include: the vehicle-mounted smart device collects multiple videos at a fixed time interval.

Optionally, step 111 may include: the vehicle-mounted smart device sets a detection range around the vehicle to which it belongs, and photographs one or more vehicles within the detection range to obtain multiple video frames.

Step 112: a data transmission step.

Optionally, step 112 may include: the vehicle-mounted smart device uploads the video frame data collected in step 111 to the server in real time.

Exemplarily, the manner of data transmission may include a wireless network communication mode.

Step 113: a data processing step.

Optionally, step 113 may include: the server inputs a video frame into a neural network, and uses the neural network to perform feature extraction on the video frame to generate a multi-dimensional behavior feature vector; and adopts each dimension in the multi-dimensional behavior feature vector. Logically classify and obtain the confidence level that the feature vector of each dimension is abnormal vehicle behavior of different event types; if the confidence level of abnormal vehicle behavior of any event type is greater than a preset threshold, it is determined that there is an abnormality of the event type in the video frame Vehicle behavior.

Exemplarily, the above recognition result of abnormal vehicle behavior means that it is determined that a vehicle in a video frame belongs to a dangerous driving or a traffic accident, and the determination rule for obtaining the recognition result is based on whether the confidence level output by the neural network model is greater than Judging by setting the threshold. That is, if the confidence that the dangerous driving is obtained after the logical classification is greater than a preset threshold, it is determined that the abnormal vehicle behavior of the dangerous driving is included in the current video frame. If the confidence level obtained after the logical classification is that the traffic accident is greater than a preset threshold, then the current video frame includes the abnormal vehicle behavior of the traffic accident. The training process of the neural network model is not repeated here.

Types of incidents of abnormal vehicle behavior include dangerous driving and / or traffic accidents. Dangerous driving includes chasing, drunk driving, and fatigue driving. Among them, the judgment of chasing driving includes passing through the video frames of the position and distance of the car in front and rear, and the distance changes with time. The judgment of drunk driving includes changing the driving trajectory of the vehicle, and the driver ’s body. The situation of changing video frames, such as the shaking of the torso of the driver ’s body, and the judgment of fatigue driving includes the changes of the vehicle ’s driving trajectory and the changes of the driver ’s body, such as the opening and closing of the driver ’s eyes; traffic accidents Including a certain type / form of traffic accidents, among which the types of traffic accidents include rear-end accidents, overtaking accidents, left-turn accidents, and transition accidents, and morphologically include collisions, scratches, rolling, rollovers, crashes, Fire, etc.

Exemplarily, the feature map structure uses a sigmoid function with a small influence function kernel as the activation function of the convolution network, so that the feature map has displacement invariance.

Identifying various illegal driving and traffic accidents on the server side can reduce the hardware requirements for on-board smart devices, make full use of the convenience and efficiency of wireless transmission, and perform complete data processing and analysis on the server side. Through real-time data transmission, the server The terminal data processing makes the vehicle alarm have higher accuracy, and achieves the technical effect of strengthening road safety and / or speeding up the processing of accidents.

Step 114: Comprehensive alarm step.

Exemplarily, an alarm and post-processing are performed by identifying a result of abnormal vehicle behavior obtained in step 113. Wherein, the alarm includes: sending an alarm message to the vehicle through the vehicle-mounted smart device via the wireless network according to the recognition result, and sending an alarm message to the vehicle's surrounding vehicles through the vehicle-mounted intelligent device via the wireless network; the post-processing includes targeting a certain type / Traffic accidents in various forms deal with the types of accidents. Minor accidents directly provide penalties and follow-up clearance measures to restore road traffic to normal as soon as possible to help improve urban traffic efficiency.

Optionally, step 114 may include: sending an alarm message to a vehicle having abnormal vehicle behavior, and a plurality of vehicles having a distance from the vehicle within a preset range.

The above-mentioned implementation manner of issuing the alarm information includes audio, video and somatosensory alarm information to the driver, and also includes audio, video and somatosensory alarm information to the co-pilot or passenger.

3. It can reduce the hardware requirements for in-vehicle smart devices, make full use of the convenience and high efficiency of wireless transmission, and perform complete data processing and analysis on the server side. Through real-time data transmission and server-side data processing, vehicle alarms have a higher Accuracy, achieving the technical effect of enhancing road safety and / or speeding up the handling of accidents.

Corresponding to the above steps, the present application proposes a vehicle behavior detection device, as shown in FIG. 5, which is applied to a vehicle-mounted smart device, and the device includes:

An acquisition module 501, configured to acquire a video frame;

A data transmission module 502 is configured to upload the video frame to a server.

Optionally, the acquisition module 501 may be specifically configured to set a detection range around a vehicle to which the vehicle-mounted smart device belongs, and shoot one or more vehicles within the detection range to obtain multiple video frames.

Optionally, the acquisition module 501 may be specifically configured to collect multiple video frames at a fixed time interval.

This application proposes a vehicle behavior detection device, as shown in FIG. 6, which is applied to a server. The device includes:

A receiving module 601, configured to receive a video frame sent by a vehicle-mounted smart device;

A data processing module 602, configured to identify a vehicle behavior based on the video frame, and obtain a recognition result of abnormal vehicle behavior;

A comprehensive alarm module 603 is configured to perform a vehicle behavior alarm according to the recognition result.

Optionally, the data processing module 602 may be specifically configured to: input the video frame into a neural network, and use the neural network to perform feature extraction on the video frame to generate a multi-dimensional behavior feature vector; Each dimension in the feature vector is logically classified to obtain the confidence level that the behavior feature vector of each dimension is an abnormal vehicle behavior of a different event type; if the confidence level of the abnormal vehicle behavior of any event type is greater than a preset threshold, determine all The abnormal vehicle behavior of this event type is described in the video frame.

Optionally, the comprehensive alarm module 603 may be specifically configured to: issue alarm information to vehicles having abnormal vehicle behavior, and to multiple vehicles within a preset distance from the vehicle, where the abnormal vehicle behavior includes dangerous driving And / or traffic accidents.

An optional embodiment of the present application provides a vehicle behavior detection method, including: Step 101: a data collection step.

The vehicle intelligent device is used for video collection. For the video frames collected by the camera module of the vehicle intelligent device, multiple collected video frames are sent to the subsequent steps for calculation every 0.1 seconds. For example, in the subsequent steps, calculation results need to be given every 0.1 seconds, and the full frame rate of the video is 50 frames per second, and then the result of the interframe sampling is to send 5 frames of pictures every 0.1 seconds into the subsequent steps.

Among them, the vehicle-mounted smart device includes a driving recorder, a mobile communication device, or another camera device with a data transmission function.

Step 102: a data processing step, which specifically includes the following steps:

Step 102-1: The collected continuous video frame pictures, such as 5 frames of pictures, are input into a neural network, and the continuous frame pictures, such as 5 frames of pictures, are subjected to feature extraction through the neural network to generate behavior feature vectors. Among them, a 3D convolutional network is used for feature extraction of a video sequence, and a LSTM network is used for historical feature fusion of features to output a final feature vector.

In the related art, the operation of the video is to individually encode the CNN feature of the video frame, and then use LSTM to process the features of each frame. As shown in Figure 7, the video frames Image (t-2), Image (t-1) , Image (t) respectively encode CNN features, and then use LSTM to process the encoded features of each frame, where h (x) is the hidden state of LSTM, S (t) is the output result vector, and S (t -3), S (t-2), S (t-1) are process vectors, and S (t) is output as the final result of the calculation.

However, the above method has the following problems: a single video frame picture cannot accurately reflect the state of continuous video fragments, and the LSTM network output of the first few pictures cannot participate in training. Therefore, in the steps of this application, the process of FIG. 5 is simplified. As shown in FIG. 8, 5 frames of a video are taken as one input, 3D convolution is used to perform feature coding, and then LSTM is used to encode the feature frame features. Perform processing, where h (x) is the hidden state of LSTM, S (t) is the output result vector, S (t-5) is the process vector, and S (t) is output as the final result of this calculation.

Step 102-2: Logically classify the feature vector S (t) and output a category larger than a preset threshold.

The sigmoid function is used for the final class output of the feature vector S (t).

Specifically, the sigmoid function is used to classify the feature vector S (t) in step 102, and the sigmoid function is defined by the following formula:

Among them, x is a multi-dimensional feature vector S (t) generated by the neural network feature extraction of each event, and the confidence f (x) of each output category corresponds to one dimension on the feature vector. Compare f (x) with a preset threshold (a criterion for distinguishing between dangerous driving or a certain type / form of traffic accidents), and if it is larger than the preset threshold, determine that it is a dangerous driving or a type / form of traffic accidents, Get the recognition result of abnormal vehicle behavior.

Step 102 is performed locally by the smart device. The above-mentioned smart device executes locally, which means that during the execution of step 102, the smart device that is local to the vehicle does not need to upload the collected data to the server, and performs the above-mentioned data processing process only on the on-board smart device that is local to a single vehicle.

In this way, the in-vehicle intelligent device can identify various types of incidents such as driving violations, traffic accidents; if the local processing capabilities of the in-vehicle intelligent device are fully utilized and this function is implemented on the in-vehicle intelligent device, the data processing pressure on the server side can be reduced, and the server's Hardware requirements, through local processing and server judgment alarms, make vehicle alarms have higher real-time performance, and achieve the technical effect of strengthening road safety and / or speeding up the handling of accidents.

Step 103: a data transmission step.

This step includes uploading the recognition result obtained in step 102 and the driving behavior data (structured analysis data and key video data) of all vehicles on the road to the server in real time through the wireless communication network. The structured analysis data includes the event categories represented by the output f (x) of the network model, as well as related video frame data and intermediate result data.

Step 104: Comprehensive alarm step.

This step includes performing a vehicle behavior analysis and alarm on the server.

Based on the above recognition results, when the vehicle is judged to be dangerous driving, based on the comprehensive positioning of GPS and license plate recognition (the license plate recognition is performed by sampling the middle frame image of the video) information, a certain range around the dangerous driving vehicle (exemplarily , Within a circle of a radius of 100 meters), an alarm is issued, and the relative position of the dangerous driving vehicle and its own vehicle is output. Due to the limited accuracy of GPS, the relative position of a dangerous driving vehicle and its own vehicle cannot be accurately given. Therefore, a given range of GPS is adopted. The vehicles in the range form the whole by the relative position of the license plate in the image that each vehicle can see. Location information.

Based on the above recognition results, when a vehicle is judged to be an accident vehicle, the vehicle is alerted to vehicles within 100 meters of the accident vehicle based on the comprehensive positioning of GPS and license plate recognition (the license plate recognition is performed by sampling the middle frame image of the video). , Output the relative position of the accident vehicle and the own vehicle. Due to the limited accuracy of GPS information, the relative position of the accident vehicle and its own vehicle cannot be accurately given, so the GPS given range is used. The vehicles in the range form the whole by the relative position of the license plate in the image that each vehicle can see. Location information.

An optional embodiment of the present application provides a vehicle behavior detection method, including: Step 111: a data collection step.

Among them, the smart device includes a driving recorder, a mobile communication device, or another camera device with a data transmission function.

Step 112: a data transmission step.

Specifically, this step includes uploading the video frames collected in step 111 to the server in real time.

Exemplarily, the manner of data transmission includes a wireless network communication mode.

Step 113: a data processing step, which specifically includes performing the following steps on the server:

Step 113-1: The uploaded continuous video frame pictures such as 5 frames are input to a neural network, and the continuous frame pictures such as 5 frames are subjected to feature extraction through the neural network to generate behavior feature vectors. Among them, a 3D convolutional network is used for feature extraction of a video sequence, and a LSTM network is used for historical feature fusion of features to output a final feature vector.

Step 113-2: Logically classify the feature vector S (t) and output a category larger than a preset threshold.

Specifically, the sigmoid function is used to classify the feature vector S (t) in step 112, and the sigmoid function is defined by the following formula:

Step 11: Comprehensive alarm procedure.

According to an embodiment of the present application, a vehicle behavior detection system is further provided, including: a vehicle-mounted intelligent device and a server;

Exemplarily, the server may be a cloud analysis server.

FIG. 9 shows a schematic diagram of the comprehensive alarm procedure of the vehicle behavior detection system. After the vehicle-mounted smart device of the current vehicle recognizes abnormal vehicle behavior, it sends the recognition result to the server, and the server sends the identification to the vehicle that is determined to have abnormal vehicle behavior. Intelligent devices, on-board intelligent devices of surrounding vehicles judged to be abnormal vehicle behavior, 120 emergency platforms, traffic management agency platforms, etc. issue alarm messages.

According to another embodiment of the present application, a vehicle behavior detection system is provided, including: a vehicle intelligent device and a server;

Exemplarily, the server may be a cloud analysis server.

FIG. 10 shows a schematic diagram of the comprehensive alarm procedure of the vehicle behavior detection system. The on-board intelligent device of the current vehicle sends the collected video frames to the server. After the server recognizes the abnormal vehicle behavior, it judges the abnormal vehicle behavior according to the recognition result. The vehicle-mounted intelligent device, the vehicle-mounted intelligent device of the surrounding vehicles judged to be abnormal vehicle behavior, the 120 emergency platform, the traffic management agency platform, and the like issue alarm information.

An embodiment of the present application further provides a vehicle-mounted smart device. As shown in FIG. 11, the device includes a processor 1101 and a machine-readable storage medium 1102.

A machine-readable storage medium 1102, for storing machine-executable instructions capable of being executed by the processor 1101;

The processor 1101 is configured to be caused to execute the following method steps by the machine-executable instructions stored on the machine-readable storage medium 1102:

Capture video frames;

Optionally, when the processor 1101 executes the captured video frame, it may specifically perform:

Capture multiple video frames at regular time intervals.

Optionally, when the processor 1101 executes the recognition of the vehicle behavior based on the video frame and obtains the recognition result of the abnormal vehicle behavior, it may specifically perform:

Input the video frame into a neural network, and use the neural network to perform feature extraction on the video frame to generate a multi-dimensional behavior feature vector;

Logically classify each dimension in the multi-dimensional behavioral feature vector to obtain the confidence that the behavioral feature vector in each dimension is an abnormal vehicle behavior of a different event type;

If the confidence level of the abnormal vehicle behavior of any event type is greater than a preset threshold, it is determined that the abnormal vehicle behavior of the event type exists in the video frame.

Optionally, when the processor 1101 executes the uploading the recognition result to the server, the processor 1101 may specifically perform:

Upload the video frame with abnormal vehicle behavior and the event type of the abnormal vehicle behavior to the server.

The detection range is set with the vehicle to which the vehicle-mounted smart device belongs as a center, and multiple video frames are obtained by shooting one or more vehicles within the detection range.

The machine-readable storage medium 1102 and the processor 1101 may perform data transmission through a wired connection or a wireless connection, and the vehicle-mounted intelligent device may communicate with other devices through a wired communication interface or a wireless communication interface.

An embodiment of the present application further provides a machine-readable storage medium that stores machine-executable instructions. When called and executed by a processor, the machine-executable instructions cause the processor to execute the embodiment shown in FIG. 11. Steps performed by an in-vehicle smart device.

The embodiment of the present application further provides an application program for executing during execution: the steps performed by the vehicle-mounted smart device in the embodiment shown in FIG. 11.

An embodiment of the present application further provides a server, as shown in FIG. 12, including a processor 1201 and a machine-readable storage medium 1202, where:

A machine-readable storage medium 1202, configured to store machine-executable instructions that can be executed by the processor 1201;

The processor 1201 is configured to be caused to execute the following method steps by the machine-executable instructions stored on the machine-readable storage medium 1202:

According to the recognition result, a vehicle behavior alarm is performed.

Optionally, when the processor 1201 executes the vehicle behavior alarm according to the recognition result, it may specifically perform:

The warning information is issued to a vehicle having abnormal vehicle behavior and a plurality of vehicles having a distance from the vehicle within a preset range.

The machine-readable storage medium 1202 and the processor 1201 may perform data transmission through a wired connection or a wireless connection, and the server may communicate with other devices through a wired communication interface or a wireless communication interface.

An embodiment of the present application further provides a machine-readable storage medium that stores machine-executable instructions. When called and executed by a processor, the machine-executable instructions cause the processor to execute the embodiment shown in FIG. 12. Steps performed by the server.

The embodiment of the present application further provides an application program, which is used to execute at runtime: the steps performed by the server in the embodiment shown in FIG. 12.

An embodiment of the present application further provides a vehicle-mounted smart device. As shown in FIG. 13, the smart device includes a processor 1301 and a machine-readable storage medium 1302, where:

A machine-readable storage medium 1302, configured to store machine-executable instructions that can be executed by the processor 1301;

The processor 1301 is configured to be caused to execute the following method steps by the machine-executable instructions stored on the machine-readable storage medium 1302:

Capture video frames;

Optionally, when the processor 1301 executes the captured video frame, it may specifically perform:

Capture multiple video frames at regular time intervals.

Data can be transmitted between the machine-readable storage medium 1302 and the processor 1301 through a wired connection or a wireless connection, and the vehicle-mounted intelligent device can communicate with other devices through a wired communication interface or a wireless communication interface.

An embodiment of the present application further provides a machine-readable storage medium that stores machine-executable instructions. When called and executed by a processor, the machine-executable instructions cause the processor to execute the embodiment shown in FIG. 13. Steps performed by an in-vehicle smart device.

The embodiment of the present application further provides an application program for executing during execution: the steps performed by the vehicle-mounted smart device in the embodiment shown in FIG. 13.

An embodiment of the present application further provides a server, as shown in FIG. 14, including a processor 1401 and a machine-readable storage medium 1402, where:

A machine-readable storage medium 1402, configured to store machine-executable instructions that can be executed by the processor 1401;

The processor 1401 is configured to be caused to execute the following method steps by the machine-executable instructions stored on the machine-readable storage medium 1402:

Receiving video frames sent by vehicle smart devices;

According to the recognition result, a vehicle behavior alarm is performed.

Optionally, when the processor 1401 executes the recognition of the vehicle behavior based on the video frame and obtains the recognition result of the abnormal vehicle behavior, it may specifically perform:

Optionally, when the processor 1401 executes the vehicle behavior alarm according to the recognition result, it may specifically perform:

Issue warning information to vehicles that have abnormal vehicle behavior and multiple vehicles that are within a preset distance from the vehicle, where the abnormal vehicle behavior includes dangerous driving and / or traffic accidents.

The machine-readable storage medium 1402 and the processor 1401 may perform data transmission through a wired connection or a wireless connection, and the server may communicate with other devices through a wired communication interface or a wireless communication interface.

An embodiment of the present application further provides a machine-readable storage medium that stores machine-executable instructions. When called and executed by a processor, the machine-executable instructions cause the processor to execute the embodiment shown in FIG. 14. Steps performed by the server.

The embodiment of the present application further provides an application program for executing at runtime: the steps performed by the server in the embodiment shown in FIG. 14.

The above machine-readable storage medium may include RAM (Random Access Memory, Random Access Memory), and may also include NVM (Non-volatile Memory, non-volatile memory), such as at least one disk memory. Optionally, the machine-readable storage medium may also be at least one storage device located far from the foregoing processor.

The above processor may be a general-purpose processor, including a CPU (Central Processing Unit), a NP (Network Processor), etc .; it may also be a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit (ASIC), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

For the vehicle behavior detection device, the vehicle behavior detection system, the vehicle-mounted intelligent device, the server, the machine-readable storage medium, and the application program embodiment, since the content of the method involved is basically similar to the foregoing method embodiment, the comparison described It is simple, and the relevant part can refer to the description of the method embodiment.

It should be noted that in this article, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations There is any such actual relationship or order among them. Moreover, the terms "including", "comprising", or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also those that are not explicitly listed Or other elements inherent to such a process, method, article, or device. Without more restrictions, the elements defined by the sentence "including a ..." do not exclude the existence of other identical elements in the process, method, article, or equipment including the elements.

Each embodiment in this specification is described in a related manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, the embodiments of the vehicle behavior detection device, the vehicle behavior detection system, the in-vehicle smart device, the server, the machine-readable storage medium, and the application program are basically similar to the method embodiment, so the description is relatively simple and relevant See the description of the method embodiments.

The above are only preferred embodiments of this application, and are not intended to limit this application. Any modification, equivalent replacement, or improvement made within the spirit and principles of this application shall be included in this application Within the scope of protection.

Claims

A vehicle behavior detection method is characterized in that it is applied to a vehicle-mounted smart device, and the method includes:

Capture video frames;

Identifying vehicle behavior based on the video frame to obtain a recognition result of abnormal vehicle behavior;

The recognition result is uploaded to a server, so that the server performs a vehicle behavior alarm according to the recognition result.
The method according to claim 1, wherein the acquiring a video frame comprises:

Capture multiple video frames at regular time intervals.
The method according to claim 1, wherein the identifying vehicle behavior based on the video frame to obtain a recognition result of abnormal vehicle behavior comprises:

Input the video frame into a neural network, and use the neural network to perform feature extraction on the video frame to generate a multi-dimensional behavior feature vector;

Logically classify each dimension in the multi-dimensional behavioral feature vector to obtain the confidence that the behavioral feature vector in each dimension is an abnormal vehicle behavior of a different event type;

If the confidence level of the abnormal vehicle behavior of any event type is greater than a preset threshold, it is determined that the abnormal vehicle behavior of the event type exists in the video frame.
The method according to claim 1, wherein the uploading the recognition result to a server comprises:

Upload the video frame with abnormal vehicle behavior and the event type of the abnormal vehicle behavior to the server.
The method according to claim 4, wherein said acquiring video frames comprises:

The detection range is set with the vehicle to which the vehicle-mounted smart device belongs as a center, and multiple video frames are obtained by shooting one or more vehicles within the detection range.
A vehicle behavior detection method is characterized in that it is applied to a server, and the method includes:

Receiving a recognition result sent by a vehicle-mounted smart device, where the recognition result is a recognition result of abnormal vehicle behavior obtained by the vehicle-mounted smart device based on the collected video frames;

According to the recognition result, a vehicle behavior alarm is performed.
The method according to claim 6, wherein the performing a vehicle behavior alarm according to the recognition result comprises:

The warning information is issued to a vehicle having abnormal vehicle behavior and a plurality of vehicles having a distance from the vehicle within a preset range.
A vehicle behavior detection method is characterized in that it is applied to a vehicle-mounted smart device, and the method includes:

Capture video frames;

Uploading the video frame to a server, so that the server recognizes vehicle behavior based on the video frame, obtains a recognition result of abnormal vehicle behavior, and performs a vehicle behavior alarm according to the recognition result.
The method according to claim 8, wherein the acquiring a video frame comprises:

The detection range is set with the vehicle to which the vehicle-mounted smart device belongs as a center, and multiple video frames are obtained by shooting one or more vehicles within the detection range.
The method according to claim 8, wherein the acquiring a video frame comprises:

Capture multiple video frames at regular time intervals.
A vehicle behavior detection method is characterized in that it is applied to a server, and the method includes:

Receiving video frames sent by vehicle smart devices;

Identifying vehicle behavior based on the video frame to obtain a recognition result of abnormal vehicle behavior;

According to the recognition result, a vehicle behavior alarm is performed.
The method according to claim 11, wherein the identifying vehicle behavior based on the video frame to obtain a recognition result of abnormal vehicle behavior comprises:

Input the video frame into a neural network, and use the neural network to perform feature extraction on the video frame to generate a multi-dimensional behavior feature vector;

Logically classify each dimension in the multi-dimensional behavioral feature vector to obtain the confidence that the behavioral feature vector in each dimension is an abnormal vehicle behavior of a different event type;

If the confidence level of the abnormal vehicle behavior of any event type is greater than a preset threshold, it is determined that the abnormal vehicle behavior of the event type exists in the video frame.
The method according to claim 11, wherein the performing a vehicle behavior alarm according to the recognition result comprises:

The warning information is issued to a vehicle having abnormal vehicle behavior and a plurality of vehicles having a distance from the vehicle within a preset range.
A vehicle behavior detection device, which is characterized in that it is applied to a vehicle-mounted smart device, and the device includes:

An acquisition module for acquiring video frames;

A data processing module, configured to identify vehicle behavior based on the video frame, and obtain a recognition result of abnormal vehicle behavior;

A data transmission module, configured to upload the recognition result to a server.
A vehicle behavior detection device is characterized in that it is applied to a server, and the device includes:

A receiving module, configured to receive a recognition result sent by a vehicle-mounted intelligent device, where the recognition result is a recognition result of abnormal vehicle behavior obtained by the vehicle-mounted intelligent device based on the collected video frames;

A comprehensive alarm module is configured to perform a vehicle behavior alarm according to the recognition result.
A vehicle behavior detection device, which is characterized in that it is applied to a vehicle-mounted smart device, and the device includes:

An acquisition module for acquiring video frames;

A data transmission module is configured to upload the video frame to a server.
A vehicle behavior detection device is characterized in that it is applied to a server, and the device includes:

A receiving module, configured to receive a video frame sent by a vehicle intelligent device;

A data processing module, configured to identify vehicle behavior based on the video frame, and obtain a recognition result of abnormal vehicle behavior;

A comprehensive alarm module is configured to perform a vehicle behavior alarm according to the recognition result.
A vehicle behavior detection system, characterized in that the system includes: a vehicle intelligent device and a server;

The in-vehicle intelligent device is used to collect video frames; identify vehicle behaviors based on the video frames to obtain recognition results of abnormal vehicle behaviors; and upload the recognition results to the server;

The server is configured to receive the recognition result sent by the in-vehicle smart device; and perform a vehicle behavior alarm according to the recognition result.
A vehicle behavior detection system, characterized in that the system includes: a vehicle intelligent device and a server;

The in-vehicle smart device is used to collect video frames; upload the video frames to the server;

The server is configured to receive the video frame sent by the in-vehicle smart device; identify vehicle behavior based on the video frame to obtain a recognition result of abnormal vehicle behavior; and perform a vehicle behavior alarm based on the recognition result.
An in-vehicle smart device, comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions executable by the processor, and the processor is used by the machine The executable instructions cause the method of any one of claims 1-5 to be performed.
A machine-readable storage medium, characterized in that it stores machine-executable instructions, and when called and executed by a processor, the machine-executable instructions cause the processor to execute any one of claims 1-5 Methods.
An application program, which is configured to execute at runtime: the method according to any one of claims 1-5.
A server includes a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions executable by the processor, and the processor is executable by the machine The instructions cause the method of claim 6 or 7 to be performed.
A machine-readable storage medium, characterized in that it stores machine-executable instructions, and when called and executed by a processor, the machine-executable instructions cause the processor to execute the method according to claim 6 or 7.
An application program, which is used to execute the method according to claim 6 or 7 at runtime.
An in-vehicle smart device, comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions executable by the processor, and the processor is used by the machine The executable instructions cause the method according to any one of claims 8-10 to be performed.
A machine-readable storage medium, characterized in that it stores machine-executable instructions, and when called and executed by a processor, the machine-executable instructions cause the processor to execute any one of claims 8-10 Methods.
An application program, which is configured to execute at runtime: the method according to any one of claims 8-10.
A server includes a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions executable by the processor, and the processor is executable by the machine The instructions cause the method of any one of claims 11-13 to be performed.
A machine-readable storage medium, characterized in that it stores machine-executable instructions, and when called and executed by a processor, the machine-executable instructions cause the processor to execute any one of claims 11-13 Methods.
An application program, which is used to execute at runtime: the method according to any one of claims 11-13.