CN111626275B - Abnormal parking detection method based on intelligent video analysis - Google Patents

Abstract

The invention discloses an abnormal parking detection method based on intelligent video analysis, which comprises the steps of carrying out target detection on a current video frame by using a convolutional neural network model, obtaining attribute information of a detected target in the current video frame, analyzing and judging whether an unmoved vehicle exists in the current video frame by combining a historical tracking queue containing the attribute information of different vehicle targets in a preset time period from a time corresponding to a last video frame of the current video frame to a historical time direction, detecting whether other vehicle targets except the unmoved vehicle target move or not by adding pedestrian detection, and comprehensively judging an abnormal parking condition according to a detection result. The method provided by the invention greatly reduces the false alarm rate and the missing alarm rate of abnormal parking.

Description

Abnormal parking detection method based on intelligent video analysis

Technical Field

The invention relates to the technical field of intelligent video analysis, in particular to an abnormal parking detection method based on intelligent video analysis.

Background

In road traffic, abnormal parking is a traffic violation that affects traffic efficiency and often causes traffic accidents. How to accurately and timely detect the abnormal parking behavior is a key function in the traffic monitoring system. The intelligent video analysis technology is widely applied to abnormal parking detection, but due to the limitation of the prior art, the conditions of false alarm and missed alarm often exist, the accuracy of judging abnormal parking is influenced, and the practicability of the abnormal parking detection function is greatly reduced. In the abnormal parking detection based on intelligent video analysis, target detection is a key, and corresponding functions can be realized only if moving targets such as people and vehicles in the monitoring video are correctly detected.

Target detection algorithms can generally be divided into two broad categories: one is target detection based on a traditional image algorithm, and the other is target detection based on a deep learning algorithm. Research and practice over the last decade has demonstrated that deep learning based target detection is far superior to traditional target detection algorithms, both in speed and accuracy.

Disclosure of Invention

The purpose of the invention is as follows: the abnormal parking detection method based on intelligent video analysis is high in accuracy.

The technical scheme is as follows: the invention provides an abnormal parking detection method based on intelligent video analysis, which realizes abnormal parking detection based on a video image captured by a fixed angle monitoring device, and comprises the following steps:

step 1, acquiring a current video frame and detection time corresponding to the video frame;

step 2, performing target detection on the current video frame by using a convolutional neural network model, acquiring each vehicle target in the current video frame, and further acquiring attribute information of the vehicle target in the current video frame; the attribute information of the vehicle target comprises the position of the vehicle in the corresponding video frame, the feature vector of the vehicle, the first time when the vehicle is detected, the last time when the vehicle is detected, and the target ID of the unique identification of the vehicle;

step 3, judging whether a vehicle target which does not move exists in the current video frame or not according to the attribute information of the vehicle target in the current video frame and the tracking queue, and if so, executing step 4; otherwise, correcting the judgment result according to the steps A to C; the tracking queue is a set of attribute information of different vehicle targets in a preset time period from the moment corresponding to the last video frame of the current video frame to the historical moment direction;

step A, aiming at each vehicle target i in the tracking queue, wherein i is the value of the target ID of the vehicle target in the tracking queue: life Time of vehicle target i Life _ Time_IDiUsing equation (8) to calculate:

Life_Time_IDi＝Now_Time–Update_time_IDi (8)

wherein,the Now _ Time is the detection Time corresponding to the current video frame, Update _ Time_IDiThe detection time when the vehicle target i is detected last time;

compare Life _ Time_IDiAnd a preset Track vehicle target Expiration Threshold value Track _ Expiration _ Threshold, if:

Life_Time_IDi>track _ exception _ Threshold, deleting the vehicle target i including the attribute information thereof from the tracking queue, adding the vehicle target which appears in the current video frame and does not appear in the tracking queue including the attribute information thereof into the tracking queue, and acquiring the updated tracking queue;

step B, comparing the updated data length Track _ Q _ Len in the tracking queue with a preset tracking queue length Threshold value Track _ Q _ Threshold, and if the Track _ Q _ Len is less than the Track _ Q _ Threshold, executing the step 1; otherwise, executing step C;

step C, judging whether a pedestrian target exists in the current video frame, if so, judging that a vehicle target with abnormal parking exists in the current video frame, otherwise, executing the step 1;

step 4, respectively executing the following steps aiming at each vehicle which does not move in the current video frame:

step 401, in combination with the tracking queue, judging whether the staying time of the vehicle target which does not move in the current video frame is larger than a preset abnormal parking time threshold, if so, executing step 402, otherwise, executing step 5;

step 402, judging whether other moving vehicles exist in the current video frame, if so, judging that the vehicle which does not move in the step 401 and has the stay time longer than a preset abnormal parking time threshold value is an abnormal parking, and executing the step 5; otherwise, go to step 403;

step 403, judging whether a pedestrian target exists in the current video frame, if so, judging that the vehicle which does not move and has the stay time longer than a preset abnormal parking time threshold value in the step 401 is abnormal parking, and executing the step 5; otherwise, judging that the vehicle which does not move in the step 401 and has the stay time longer than the preset abnormal parking time threshold value is in non-abnormal parking, and executing the step 5;

and 5, updating the tracking queue information.

In step 2, in the process of performing target detection on the video frame by using the convolutional neural network model: and marking the vehicle target by using the surrounding rectangular frame, acquiring the position coordinates of the surrounding rectangular frame in the corresponding video frame, and marking the acquired position coordinates of the surrounding rectangular frame as the position of the corresponding vehicle in the video frame.

In step 3, the method for determining whether there is a vehicle that does not move in the current video frame specifically includes the following steps:

for each vehicle target j, j in the current video frame as the value of the target ID of the vehicle target in the current video frame, the following operations are executed:

step 301A, for each vehicle target i, i in the tracking queue, the value of the target ID of the vehicle target in the tracking queue: referring to equation (1), the intersection ratio IoU of the bounding rectangular frame of the vehicle object i and the bounding rectangular frame of the vehicle object j is calculated_ij：

IoU_ij＝Inter_ij/Union_ij (1)

Wherein, Inter_ijIs the intersection of the bounding rectangular box of vehicle object i and the bounding rectangular box of vehicle object j, Union_ijIs the union of the bounding rectangular frame of vehicle object i and the bounding rectangular frame of vehicle object j; inter_ijAnd Union_ijThe calculation formula of (a) is as follows:

Inter_ij＝Inter_W_ij*Inter_H_ij (2)

Union_ij＝(right_i–left_i)*(top_i–bottom_i)+(right_j–left_j)*(top_j–bottom_j)-Inter_ij (3)

Inter_W_ij＝max(0，min(right_i,right_j)-max(left_i，left_j)) (4)

Inter_H_ij＝max(0，min(top_i,top_j)-max(bottom_i，bottom_j)) (5)

wherein, Inter _ W_ijIs the width of the intersection of the bounding rectangular frame of the vehicle object i and the bounding rectangular frame of the vehicle object j, Inter _ H_ijThe height of the intersection part of the surrounding rectangular frame of the vehicle target i and the surrounding rectangular frame of the vehicle target j; wherein left_i、top_iAny corner m in the surrounding rectangular frame of the vehicle target i_iAbscissa and ordinate, right_i、bottom_iAt corner m in a bounding rectangular frame of a vehicle object i, respectively_iThe abscissa and ordinate of the corner point of the diagonal position of (a); left_j、top_jAny corner m in the surrounding rectangular frame of the vehicle target j respectively_jAbscissa and ordinate, right_j、bottom_jAt corner m in a bounding rectangular frame of a vehicle object j, respectively_jThe abscissa and ordinate of the corner point of the diagonal position of (a);

step 302A, compare IoU_ijIntersecting with a preset rectangular frame and comparing with a Threshold value IoU _ Threshold if IoU_ijIf the value is less than IoU _ Threshold, the vehicle target i and the vehicle target j are judged not to be at the same spatial position on the road; otherwise, the same spatial positions of the vehicle target i and the vehicle target j on the road are judged, and the step 303A is executed;

step 303A, calculating cosine distance Dist (ID) between the characteristic vector of the vehicle i and the characteristic vector of the vehicle target j according to formula (6)_j,ID_i)：

Wherein r is_IDiApplying a convolutional neural network to the vehicle target i to obtain a normalized feature vector;

the vehicle target j is a vector obtained by transposing a normalized feature vector obtained by applying a convolutional neural network;

step 304A, compare Dist (ID)_j，ID_i) And a preset feature vector threshold value Dist _ ThrAn estimate of Dist (ID)_j，ID_i) Judging that the vehicle i and the vehicle j are the same vehicle and the vehicle j does not move if the distance between the vehicle i and the vehicle j is not larger than the Dist _ Threshold; otherwise, the vehicle j is determined to move.

In step 3, the method for determining whether there is a vehicle that does not move in the current video frame specifically includes the following steps:

for each vehicle target j, j in the current video frame as the value of the target ID of the vehicle target in the current video frame, the following operations are executed:

step 301B, for each vehicle target i, i in the tracking queue, the value of the target ID of the vehicle target in the tracking queue: referring to equation (1), the intersection ratio IoU of the bounding rectangular frame of the vehicle object i and the bounding rectangular frame of the vehicle object j is calculated_ij：

IoU_ij＝Inter_ij/Union_ij (1)

Wherein, Inter_ijIs the intersection of the bounding rectangular box of vehicle object i and the bounding rectangular box of vehicle object j, Union_ijIs the union of the bounding rectangular frame of vehicle object i and the bounding rectangular frame of vehicle object j; inter_ijAnd Union_ijThe calculation formula of (a) is as follows:

Inter_ij＝Inter_W_ij*Inter_H_ij (2)

Union_ij＝(right_i–left_i)*(top_i–bottom_i)+(right_j–left_j)*(top_j–bottom_j)-Inter_ij (3)

Inter_W_ij＝max(0，min(right_i,right_j)-max(left_i，left_j)) (4)

Inter_H_ij＝max(0，min(top_i,top_j)-max(bottom_i，bottom_j)) (5)

wherein, Inter _ W_ijIs the width of the intersection of the bounding rectangular frame of the vehicle object i and the bounding rectangular frame of the vehicle object j, Inter _ H_ijIs a vehicleThe height of the intersection of the bounding rectangular frame of vehicle object i and the bounding rectangular frame of vehicle object j; wherein left_i、top_iAny corner m in the surrounding rectangular frame of the vehicle target i_iAbscissa and ordinate, right_i、bottom_iAt corner m in a bounding rectangular frame of a vehicle object i, respectively_iThe abscissa and ordinate of the corner point of the diagonal position of (a); left_j、top_jAny corner m in the surrounding rectangular frame of the vehicle target j respectively_jAbscissa and ordinate, right_j、bottom_jAt corner m in a bounding rectangular frame of a vehicle object j, respectively_jThe abscissa and ordinate of the corner point of the diagonal position of (a);

step 302B, compare IoU_ijIntersecting with a preset rectangular frame and comparing with a Threshold value IoU _ Threshold if IoU_ijIf the value is less than IoU _ Threshold, the vehicle target i and the vehicle target j are judged not to be at the same spatial position on the road; otherwise, the same spatial positions of the vehicle target i and the vehicle target j on the road are judged, and step 303B is executed;

step 303B, calculating cosine distance Dist' (ID) between the feature vector of vehicle i and the feature vector of vehicle target j according to formula (7)_j，ID_i)：

In the formula,

obtaining a normalized feature vector by applying a convolutional neural network to a vehicle target i in an nth historical video frame with the vehicle target i, wherein M is the number of the historical video frames with the vehicle target i;

the vehicle target j is a vector obtained by transposing a normalized feature vector obtained by applying a convolutional neural network;

step 304B, compare Dist' (ID)_j,ID_i) And a predetermined feature vector Threshold value Dist _ Threshold if Dist' (ID)_j,ID_i) Judging that the vehicle i and the vehicle j are the same vehicle and the vehicle j does not move if the distance between the vehicle i and the vehicle j is not larger than the Dist _ Threshold; otherwise, the vehicle j is determined to move.

In step C, after it is determined that there is a vehicle target with abnormal parking in the current video frame, the method further includes reporting an abnormal parking event.

In step 401, the method further comprises: calculating the dwell time parkingtime of the vehicle object s that is not moving in the current video frame using equation (9)_S：

Parking_time_S＝Update_time_IDs-Init_time_IDd (9)

Wherein Update _ time_IDsInit _ time, which is the time when the vehicle object s that has not moved in the current video frame was last detected_IDdThe time when the vehicle target d corresponding to the vehicle target s which does not move in the current video frame is detected for the first time in the tracking queue; s is the value of the target ID of the vehicle target that does not move in the current video frame, and d is the value of the target ID of the vehicle target in the tracking queue that corresponds to the vehicle target s that does not move in the current video frame.

In step 402 and in step 403, after it is determined that the vehicle which does not move in step 401 and has the stay time longer than the preset abnormal parking time threshold is abnormally parked, the method further includes reporting an abnormal parking event.

In step 5, the method for updating the tracking queue information comprises the following steps:

step 501, replacing the vehicle position, the vehicle characteristic vector and the time of the vehicle which is detected last time of the vehicle in the tracking queue by using the vehicle position, the vehicle characteristic vector and the detection time of the vehicle target which does not move in the current video frame;

step 502, aiming at the vehicle target k except the vehicle target k participating in the operation of the step 701 in the tracking queue, and the survival Time Life _ Time of the vehicle target k_IDkUsing equation (10) to calculate:

Life_Time_IDk＝Now_Time–Update_time_IDk (10)

wherein, the Now _ Time is the detection Time corresponding to the current video frame, Update _ Time_IDkThe detection time when the vehicle target k is detected last time;

compare Life _ Time_IDkAnd a preset tracking vehicle target Expiration Threshold value Track _ Expiration _ Threshold: if:

Life_Time_IDk>track _ Expiration _ Threshold, then remove the vehicle target k, including its attribute information, from the tracking queue; otherwise, the vehicle target k is reserved, and the attribute information of the vehicle target k is not changed; and adding the attribute information of the vehicle target which appears in the current video frame and does not appear in the tracking queue into the tracking queue.

The convolutional neural network model is a YoloV3 convolutional neural network model.

Has the advantages that: compared with the prior art, the method provided by the invention has the following advantages:

(1) the condition that the definition of a video image is not enough or the shielding is serious and the vehicle cannot be correctly tracked is solved, and the missing report rate of abnormal parking is effectively reduced by adding a pedestrian detection method;

(2) and by adding pedestrian detection, detecting whether other vehicle targets except the vehicle target which does not move or not, and comprehensively judging the abnormal parking condition according to the detection result, the false alarm rate and the missing alarm rate of the abnormal parking are greatly reduced.

Drawings

Fig. 1 is a flowchart of an abnormal parking detection method provided according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of object detection provided in accordance with an embodiment of the present invention;

FIG. 3 is a schematic diagram of abnormal parking detection at time t1 according to an embodiment of the present invention;

fig. 4 is a schematic diagram of abnormal parking detection at time t2 according to the embodiment of the present invention.

Detailed Description

The invention is further described below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby.

Referring to fig. 1, the present invention performs abnormal parking detection based on a video image captured by a fixed-angle monitoring device; the method provided by the invention comprises the following steps:

step S1, inputting a first video frame, recording the detection time t1 corresponding to the video frame, and marking the detection time t1 as the time stamp of the video frame; the process advances to step S2.

S2, carrying out target detection on the video frame in the step 1 by using a pre-trained deep convolutional neural network model, and entering the step S3; the deep convolutional neural network model used in the present embodiment is a YoloV3 convolutional neural network model; the method specifically comprises the following steps:

step S201, loading a YoloV3 convolutional neural network model to a GPU display card and setting corresponding weight parameters;

step S202, carrying out target detection on the input video frame by using a YoloV3 convolutional neural network model, and marking the detected target by using a surrounding rectangular frame; when the target detection is carried out, the detected targets comprise vehicle targets and pedestrian targets, and other targets comprising traffic lights, traffic signs and the like are ignored and are not processed;

the rectangular box of each marker has six attributes, which are: the target ID, the target category, the target initial time, the target updating time, the feature vector of the target, the target ID of the unique target identification and the position coordinates of the bounding rectangular frame of the target in the corresponding video frame are used for identifying the detected target;

wherein the object class comprises a vehicle or a pedestrian; the target initial time is the time of the first video frame where the target appears, namely the time when the target is detected for the first time; the target update time is the time of the latest video frame appearing latest by the target, namely the time when the target is detected last time; the characteristic vector of the target is an image characteristic describing the target, the characteristic vectors of different targets are different, and the position coordinates of the target enclosing rectangular frame in the corresponding video frame are used for identifying the position of the corresponding vehicle in the video frame.

Referring to the schematic diagram of object detection shown in fig. 2: each detected target is surrounded by a rectangular frame, and a digital ID is arranged above each surrounding rectangular frame and used for distinguishing different targets; in practical applications, the surrounding rectangular frames of the vehicle object and the pedestrian object can be distinguished by different colors, in this embodiment, the surrounding rectangular frame of the vehicle object is green, and the surrounding rectangular frame of the pedestrian object is red.

Step S3, placing the vehicle targets detected according to the step 1 and the step 2 into a tracking queue Q, and entering the step 1;

each vehicle target in the tracking queue Q is called a tracking vehicle target; each vehicle target includes six attributes: target ID, target category, target initial time, target update time, feature vector of the target and position coordinates of a target surrounding rectangular frame; the tracking queue Q is attribute information of different vehicle targets in a preset time period from the moment corresponding to the last video frame of the current video frame to the historical moment.

In this embodiment, the video frame at the next time instant of the time instant t1 is the current video frame, the detection time mark corresponding to the video frame is t2, and the abnormal parking detection at the time instant t2 is performed according to the video frame at the time instant t2 and the tracking queue Q;

step 1, acquiring a video frame at the time t2, and entering step 2;

step 2, referring to step S202, performing target detection on the input video frame by using a YoloV3 convolutional neural network model, marking the detected target by using a bounding rectangle frame, acquiring attribute information of the detected target, referring the detected vehicle target as a detected vehicle target, and entering step 3;

step 3, comparing each detected vehicle target in the video frame at the time t2 with each tracked vehicle target in the tracking queue Q; judging whether a tracked vehicle target does not move within a certain time period in the video frame at the time t2 according to the comparison result; if yes, executing step 4; otherwise, the judgment result is corrected.

In one embodiment, the method of determining whether the tracked vehicle is moving is as follows:

performing the following operation on each detected vehicle target j in the video frame at the time t2, wherein j is the value of the target ID of the vehicle target in the video frame;

step 301A, for each vehicle i, i in the tracking queue Q, the value of the target ID of the vehicle target in the tracking queue:

representing the bounding rectangle of the vehicle object i as box_i：(left_i，top_i,right_i,bottom_i) The bounding rectangle of the vehicle object j is denoted box_j：(left_j，top_j，right_j，bottom_j) Wherein left_i、top_iAny corner m in the surrounding rectangular frame of the vehicle target i_iAbscissa and ordinate, right_i、bottom_iAt corner m in a bounding rectangular frame of a vehicle object i, respectively_iThe abscissa and ordinate of the corner point of the diagonal position of (a); left_j、top_jAny corner m in the surrounding rectangular frame of the vehicle target j respectively_jAbscissa and ordinate, right_j、bottom_jAt corner m in a bounding rectangular frame of a vehicle object j, respectively_jThe abscissa and ordinate of the corner point of the diagonal position of (a);

in this embodiment, left_i、top_iThe abscissa and the ordinate, right, of the corner point at the upper left corner in the bounding rectangle for the vehicle object i_i、bottom_iRespectively representing the abscissa and the ordinate of a corner point of the vehicle target i surrounding the lower right corner in the rectangular frame; left_j、top_jThe abscissa and the ordinate, right, of the corner point at the upper left corner in the bounding rectangle for the vehicle object j_j、bottom_jRespectively the abscissa and the ordinate of the corner point of the vehicle object j surrounding the lower right corner in the rectangular frame.

Referring to formula (1), a bounding rectangular box of the vehicle object i is calculated_iAnd a bounding rectangular box of the vehicle object j_jCross-over ratio IoU_ij：

IoU_ij＝Inter_ij/Union_ij (1)

Wherein, Inter_ijBounding rectangle box for vehicle object i_iAnd a bounding rectangular box of the vehicle object j_jIntersect, Union_ijBounding rectangle box for vehicle object i_iAnd a bounding rectangular box of the vehicle object j_jA union of (1); inter_ijAnd Union_ijThe calculation formula of (a) is as follows:

Inter_ij＝Inter_W_ij*Inter_H_ij (2)

Union_ij＝(right_i–left_i)*(top_i–bottom_i)+(right_j–left_j)*(top_j–bottom_j)-Inter_ij (3)

wherein, Inter _ W_ijBounding rectangle box for vehicle object i_iAnd a bounding rectangular box of the vehicle object j_jWidth of the intersection, InterH_ijBounding rectangle box for vehicle object i_iAnd a bounding rectangular box of the vehicle object j_jThe height of the intersection; inter _ W_ijAnd Inter _ H_ijThe calculation formula of (a) is as follows:

Inter_W_ij＝max(0，min(right_i,right_j)-max(left_i，left_j)) (4)

Inter_H_ij＝max(0，min(top_i,top_j)-max(bottom_i，bottom_j)) (5)

the Intersection ratio (IoU) is the ratio of the Intersection and the Union of the surrounding rectangular frames of the vehicle target i and the vehicle target j, and the value is between [0 and 1 ]:

when IoU is 0, the bounding rectangular boxes representing the two vehicle targets do not intersect, indicating that vehicle target i and vehicle target j are not the same vehicle, or understood as: even with the same vehicle, the vehicle has moved because they are already located at different road locations on the video frame;

when IoU is equal to 1, the bounding rectangular boxes representing the vehicle object i and the vehicle object j coincide, indicating that the vehicle object i and the vehicle object j are located at the same spatial position on the road at the time t2 and the time t1, but the vehicle object i and the vehicle object j may be the same vehicle or different vehicles; if the bounding rectangular frames of the vehicle object i and the vehicle object j overlap, but the vehicle object i and the vehicle object j are different vehicles, the vehicle object i is separated from the original position at time t2, and the vehicle object j is moved to the original position at time t 2.

Step 302A, due to technical limitations, the target detection algorithm cannot ensure that the generated bounding rectangle can completely and accurately and very stably enclose the vehicle target or the pedestrian target, and even for the same vehicle target which does not move in the time interval of two video frames, the positions of the bounding rectangles generated for the vehicle target or the pedestrian target through the target detection algorithm may have errors, so that IoU _ Threshold is preset in the invention for comparing the calculated intersection and parallel ratio with IoU _ Threshold, so as to improve the accuracy of judgment; 0 is not less than IoU and not more than 1.

Comparison IoU_ijIntersection with a preset rectangular box and comparison with a Threshold IoU _ Threshold:

if IoU_ijLess than IoU _ Threshold, it is determined that vehicle target i and vehicle target j are not at the same spatial location on the road, i.e., vehicle target j has moved in road space relative to vehicle target i;

if IoU is equal to or greater than IoU _ Threshold, then it is determined that vehicle object j has not moved on the spatial road relative to vehicle object i, and after determining that vehicle object j has not moved on the spatial road relative to vehicle object i, step 303A is executed to further determine whether vehicle object j and vehicle object i are the same vehicle.

Step 303A, calculating cosine distance Dist (ID) between the characteristic vector of the vehicle i and the characteristic vector of the vehicle target j according to formula (6)_j,ID_i)：

Wherein r is_IDiA normalized 128-dimensional feature vector obtained by applying a convolutional neural network to the vehicle target i;

transposing a normalized 128-dimensional feature vector obtained by applying a convolutional neural network to a vehicle target j to obtain a vector; dist (ID)_j，ID_i) Ranges between 0 and 2, a value of Dist (ID2, ID1) closer to 0 indicates a greater similarity between two feature vectors, whereas a value of Dist (ID2, ID1) greater indicates a lower similarity between two feature vectors.

Step 304A, compare Dist (ID)_j，ID_i) And a preset feature vector Threshold value Dist _ Threshold, Dist _ Threshold is greater than or equal to 0 and less than or equal to 2:

if Dist (ID)_j，ID_i) Judging that the vehicle i and the vehicle j are the same vehicle and the vehicle j does not move if the distance between the vehicle i and the vehicle j is not larger than the Dist _ Threshold;

if Dist (ID)_j，ID_i) If Dist _ Threshold, then vehicle target i and vehicle target j are not considered to be the same vehicle.

In another embodiment, the method of determining whether the tracked vehicle is moving is as follows:

for each vehicle target j, j in the current video frame as the value of the target ID of the vehicle target in the current video frame, the following operations are executed:

step 301B, for each vehicle target i, i in the tracking queue, the value of the target ID of the vehicle target in the tracking queue: referring to equation (1), the intersection ratio IoU of the bounding rectangular frame of the vehicle object i and the bounding rectangular frame of the vehicle object j is calculated_ij：

IoU_ij＝Inter_ij/Union_ij (1)

Wherein, Inter_ijIs the intersection of the bounding rectangular box of vehicle object i and the bounding rectangular box of vehicle object j, Union_ijA bounding rectangle for vehicle object i and a bounding rectangle for vehicle object jUnion of frames; inter_ijAnd Union_ijThe calculation formula of (a) is as follows:

Inter_ij＝Inter_W_ij*Inter_H_ij (2)

Union_ij＝(right_i–left_i)*(top_i–bottom_i)+(right_j–left_j)*(top_j–bottom_j)-Inter_ij (3)

Inter_W_ij＝max(0，min(right_i,right_j)-max(left_i，left_j)) (4)

Inter_H_ij＝max(0，min(top_i,top_j)-max(bottom_i，bottom_j)) (5)

wherein, Inter _ W_ijIs the width of the intersection of the bounding rectangular frame of the vehicle object i and the bounding rectangular frame of the vehicle object j, Inter _ H_ijThe height of the intersection part of the surrounding rectangular frame of the vehicle target i and the surrounding rectangular frame of the vehicle target j; wherein left_i、top_iRespectively, the abscissa and ordinate of the corner point of the surrounding upper left corner of the rectangular frame of the vehicle object i, right_i、bottom_iRespectively representing the abscissa and the ordinate of a corner point at the lower right corner in a surrounding rectangular frame of the vehicle target i; left_j、top_jRespectively the abscissa and ordinate of the corner point of the surrounding upper left corner of the rectangular frame of the vehicle object j, right_j、bottom_jRespectively representing the abscissa and the ordinate of the corner point at the lower right corner in the surrounding rectangular frame of the vehicle target j;

step 302B, compare IoU_ijIntersecting with a preset rectangular frame and comparing with a Threshold value IoU _ Threshold if IoU_ijIf the value is less than IoU _ Threshold, the vehicle target i and the vehicle target j are judged not to be at the same spatial position on the road; otherwise, the same spatial positions of the vehicle target i and the vehicle target j on the road are judged, and step 303B is executed;

step 303B, in order to avoid the occurrence of errors in single target detection, generating shadows in the calculation of cosine distances between the characteristic vectors of the vehicle target i and the vehicle target jAnd thirdly, for the same vehicle target i, the system keeps the feature vector of the past M frames (M is more than or equal to 1) of the same vehicle target i, and calculates the cosine distance Dist' (ID) between the feature vector of the vehicle i and the feature vector of the vehicle target j according to the formula (7)_j，ID_i)：

In the formula,

obtaining a normalized feature vector by applying a convolutional neural network to a vehicle target i in an nth historical video frame with the vehicle target i, wherein M is the number of the historical video frames with the vehicle target i;

the vehicle target j is a vector obtained by transposing a normalized feature vector obtained by applying a convolutional neural network;

step 304B, compare Dist' (ID)_j，ID_i) And a predetermined feature vector Threshold Dist _ Threshold, 0 ≦ Dist _ Threshold ≦ 2, if Dist' (ID)_j，ID_i) Judging that the vehicle i and the vehicle j are the same vehicle and the vehicle j does not move if the distance between the vehicle i and the vehicle j is not larger than the Dist _ Threshold; otherwise, the vehicle target i and the vehicle target j are not the same vehicle.

According to the method, whether at least one vehicle target does not move in the current video frame is judged, if no vehicle which does not move does not exist in the current video frame, the judgment result is corrected, and the method specifically comprises the following steps:

step A, aiming at each vehicle target i in the tracking queue, wherein i is the value of the target ID of the vehicle target in the tracking queue: life Time of vehicle target i Life _ Time_IDiUsing equation (8) to calculate:

Life_Time_IDi＝Now_Time–Update_time_IDi (8)

wherein, the Now _ Time is the detection Time corresponding to the current video frame, Update _ Time_IDiThe detection time at which the vehicle object i was detected last time.

Compare Life _ Time_IDiAnd a preset tracked vehicle target Expiration Threshold value, wherein the preset tracked vehicle target Expiration Threshold value is Track _ Expiration _ Threshold;

if Life _ Time_IDi>Track _ exception _ Threshold, deleting the vehicle target i from the tracking queue Q; otherwise, keeping tracking the vehicle target i, and not changing any attribute information of the vehicle target i;

adding a new detected vehicle target into a tracking queue, namely: adding the vehicle targets which appear in the current video frame and do not appear in the tracking queue into the tracking queue, including the attribute information of the vehicle targets;

step B, comparing the updated data length Track _ Q _ Len in the tracking queue with a preset tracking queue length Threshold value Track _ Q _ Threshold:

if the Track _ Q _ Len is less than Track _ Q _ Threshold, the road condition is considered to be not congested, and the step 1 is executed;

if the Track _ Q _ Len is larger than or equal to the Track _ Q _ Threshold, the road condition is considered to be congested, and the step C is executed;

wherein the tracking queue length Threshold value Track _ Q _ Threshold is set empirically.

C, judging whether a pedestrian target exists in the current video frame, if so, judging that a vehicle target with abnormal parking exists in the current video frame, and reporting abnormal parking time; otherwise, executing step 1.

Step 4, respectively executing steps 401 to 403 for each vehicle which does not move in the current video frame:

step 401, in combination with the tracking queue Q, judging whether the staying time of the vehicle target which does not move in the video frame at the time t2 is greater than a preset abnormal parking time threshold, if so, executing step 402, otherwise, executing step 5;

according to the method, intersection comparison judgment analysis is carried out on a vehicle target j in a video frame at the time t2 and a surrounding rectangular frame of the vehicle targets in a tracking queue, after the fact that the vehicle target j in the video frame at the time t2 does not change in spatial position on the road relative to the vehicle target i in the tracking queue according to the analysis result of the intersection comparison, calculation analysis of cosine distances between feature vectors of the two vehicle targets is carried out according to the feature vectors of the two vehicle targets, when the two vehicle targets are judged to be the same vehicle according to the analysis result of the cosine distances between the feature vectors of the two vehicle targets, the fact that the vehicle target j in the video frame at the time t2 does not move is judged, the vehicle target j and the vehicle target i are considered to be matched, and the vehicle target i is the vehicle corresponding to the vehicle target j in the tracking queue.

Parking _ time for the dwell time of a vehicle object s that is not moving in the video frame at time t2_sThe calculation formula for the calculation is shown in formula (9):

Parking_time_S＝Update_time_IDs-Init_time_IDd (9)

wherein Update _ time_IDsThe latest time when the vehicle target s which does not move in the current video frame is detected is taken as the time when the vehicle in the current video frame is detected, and the latest time when the vehicle in the current video frame is detected is the detection time corresponding to the current video frame; init _ time_IDdThe time when the vehicle target d corresponding to the vehicle target s which does not move in the current video frame is detected for the first time in the tracking queue; s is the value of the target ID of the vehicle target that does not move in the current video frame, and d is the value of the target ID of the vehicle target in the tracking queue that corresponds to the vehicle target s that does not move in the current video frame.

Step 402, except for the vehicle target which does not move, detecting whether other moving vehicle targets exist in the current video frame, if so, judging that the vehicle which does not move and has the stay time longer than the preset abnormal parking time threshold value in the step 401 is abnormal parking, reporting an abnormal parking event, and executing the step 5; otherwise, step 403 is performed.

The method for detecting whether other vehicle targets move in the current video frame is to monitor the tracking queue and check whether the following two conditions can be met simultaneously:

condition 1. there is a tracked vehicle target in the tracking queue because of the Time of survival Life _ Time_IDi>Track _ Expiration _ Threshold is deleted;

condition 2. a new detected vehicle target is added to the tracking queue.

If the 2 conditions are simultaneously met, indicating that other vehicle targets move in the video; otherwise it indicates that no other vehicle object is moving in the video.

Step 403, judging whether a pedestrian target exists in the current video frame, if so, judging that the vehicle which does not move and has the stay time longer than a preset abnormal parking time threshold value in the step 401 is abnormal parking, reporting an abnormal parking event, and executing the step 5; otherwise, judging that the current road is congested, namely the vehicles which do not move in the step 4 and have the stay time longer than the preset abnormal parking time threshold value are non-abnormally parked, and executing the step 5;

in fig. 3 and 4, a pedestrian target is detected, fig. 3 and 4 are schematic diagrams of abnormal parking monitoring at two different moments, respectively, and it is found by comparison that a vehicle target with a target ID of C1 or C2 in fig. 3 and 4 does not move in a corresponding time period.

Step 5, updating the tracking queue information, and the specific method comprises the following steps:

step 501, respectively updating the target updating time of the vehicle target, the feature vector of the vehicle target and the position coordinates of the target surrounding rectangular frame in the corresponding video frame in the tracking queue to the target updating time of the corresponding vehicle target, the feature vector of the vehicle target and the position coordinates of the target surrounding rectangular frame in the corresponding video frame at the moment t 2; namely: replacing the vehicle position of the vehicle target, the characteristic vector of the vehicle target and the time when the vehicle is detected last time, which correspond to the vehicle target, in the tracking queue by using the vehicle position of the vehicle target, the characteristic vector of the vehicle target and the detection time which do not move in the current video frame;

step 502, aiming at the vehicle target k except the vehicle target k participating in the operation of the step 501 in the tracking queue, and the survival Time Life _ Time of the vehicle target k_IDkUsing equation (10) to calculate:

Life_Time_IDk＝Now_Time–Update_time_IDk (10)

wherein, the Now _ Time is the detection Time corresponding to the current video frame, Update _ Time_IDkThe detection time when the vehicle target k is detected last time;

compare Life _ Time_IDkAnd a preset tracking vehicle target Expiration Threshold value Track _ Expiration _ Threshold:

if Life _ Time_IDk>Track _ Exration _ Threshold, then remove the vehicle target k from the tracking queue; otherwise, the vehicle target k is reserved, and any attribute information of the vehicle target k is not changed; adding the newly detected vehicle target into a tracking queue, namely: and adding the attribute information of the vehicle target which appears in the current video frame and does not appear in the tracking queue into the tracking queue.

The invention provides an abnormal parking detection method based on intelligent video analysis, which comprises the following steps: for the condition that the definition of a video image is not enough or the shielding is serious and the vehicle cannot be correctly tracked, the missing report rate of abnormal parking is effectively reduced by adding a pedestrian detection method; the method has the advantages that the false alarm rate and the false alarm rate of abnormal parking are greatly reduced by adding the methods of pedestrian detection and checking whether other vehicle targets are still moving except the vehicle target which does not move.

The above description is only a preferred embodiment of the present invention, and it will be apparent to those skilled in the art that various modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be considered as the protection scope of the present invention.

Claims (9)

1. An abnormal parking detection method based on intelligent video analysis is characterized in that the method comprises the following steps of:

step 1, acquiring a current video frame and detection time corresponding to the video frame;

step 2, performing target detection on the current video frame by using a convolutional neural network model, acquiring each vehicle target in the current video frame, and further acquiring attribute information of the vehicle target in the current video frame; the attribute information of the vehicle target comprises the position of the vehicle in the corresponding video frame, the feature vector of the vehicle, the first time when the vehicle is detected, the last time when the vehicle is detected, and the target ID of the unique identification of the vehicle;

step 3, judging whether a vehicle target which does not move exists in the current video frame or not according to the attribute information of the vehicle target in the current video frame and the tracking queue, and if so, executing step 4; otherwise, correcting the judgment result according to the steps A to C; the tracking queue is a set of attribute information of different vehicle targets in a preset time period from the moment corresponding to the last video frame of the current video frame to the historical moment direction;

step A, aiming at each vehicle target i in the tracking queue, wherein i is the value of the target ID of the vehicle target in the tracking queue: life Time of vehicle target i Life _ Time_IDiUsing equation (8) to calculate:

Life_Time_IDi＝Now_Time–Update_time_IDi (8)

wherein, the Now _ Time is the detection Time corresponding to the current video frame, Update _ Time_IDiThe detection time when the vehicle target i is detected last time;

compare Life _ Time_IDiAnd a preset Track vehicle target Expiration Threshold value Track _ Expiration _ Threshold, if:

Life_Time_IDi>track _ exception _ Threshold, deleting the vehicle target i including the attribute information thereof from the tracking queue, adding the vehicle target which appears in the current video frame and does not appear in the tracking queue including the attribute information thereof into the tracking queue, and acquiring the updated tracking queue;

step B, comparing the updated data length Track _ Q _ Len in the tracking queue with a preset tracking queue length Threshold value Track _ Q _ Threshold, if:

track _ Q _ Len < Track _ Q _ Threshold, performing step 1; otherwise, executing step C;

step C, judging whether a pedestrian target exists in the current video frame, if so, judging that a vehicle target with abnormal parking exists in the current video frame, otherwise, executing the step 1;

step 4, respectively executing the following steps aiming at each vehicle which does not move in the current video frame:

step 401, in combination with the tracking queue, judging whether the staying time of the vehicle target which does not move in the current video frame is larger than a preset abnormal parking time threshold, if so, executing step 402, otherwise, executing step 5;

step 402, judging whether other moving vehicles exist in the current video frame, if so, judging that the vehicle which does not move in the step 401 and has the stay time longer than a preset abnormal parking time threshold value is an abnormal parking, and executing the step 5; otherwise, go to step 403;

step 403, judging whether a pedestrian target exists in the current video frame, if so, judging that the vehicle which does not move and has the stay time longer than a preset abnormal parking time threshold value in the step 401 is abnormal parking, and executing the step 5; otherwise, judging that the vehicle which does not move in the step 401 and has the stay time longer than the preset abnormal parking time threshold value is in non-abnormal parking, and executing the step 5;

and 5, updating the tracking queue information.

2. The abnormal parking detection method based on intelligent video analysis of claim 1, wherein in step 2, the convolutional neural network model is used for performing target detection on the video frame: and marking the vehicle target by using the surrounding rectangular frame, acquiring the position coordinates of the surrounding rectangular frame in the corresponding video frame, and marking the acquired position coordinates of the surrounding rectangular frame as the position of the corresponding vehicle in the video frame.

3. The abnormal parking detection method based on intelligent video analysis as claimed in claim 2, wherein in step 3, the method for determining whether there is a vehicle that does not move in the current video frame specifically comprises the following steps:

for each vehicle target j, j in the current video frame as the value of the target ID of the vehicle target in the current video frame, the following operations are executed:

step 301A, for each vehicle target i, i in the tracking queue, the value of the target ID of the vehicle target in the tracking queue: referring to equation (1), the intersection ratio IoU of the bounding rectangular frame of the vehicle object i and the bounding rectangular frame of the vehicle object j is calculated_ij：

IoU_ij＝Inter_ij/Union_ij (1)

Wherein, Inter_ijIs the intersection of the bounding rectangular box of vehicle object i and the bounding rectangular box of vehicle object j, Union_ijIs the union of the bounding rectangular frame of vehicle object i and the bounding rectangular frame of vehicle object j; inter_ijAnd Union_ijThe calculation formula of (a) is as follows:

Inter_ij＝Inter_W_ij*Inter_H_ij (2)

Union_ij＝(right_i–left_i)*(top_i–bottom_i)+(right_j–left_j)*(top_j–bottom_j)-Inter_ij(3)

Inter_W_ij＝max(0，min(right_i,right_j)-max(left_i，left_j)) (4)

Inter_H_ij＝max(0，min(top_i,top_j)-max(bottom_i，bottom_j)) (5)

wherein, Inter _ W_ijIs the width of the intersection of the bounding rectangular frame of the vehicle object i and the bounding rectangular frame of the vehicle object j, Inter _ H_ijThe height of the intersection part of the surrounding rectangular frame of the vehicle target i and the surrounding rectangular frame of the vehicle target j; wherein left_i、top_iAny corner m in the surrounding rectangular frame of the vehicle target i_iAbscissa and ordinate, right_i、bottom_iAt corner m in a bounding rectangular frame of a vehicle object i, respectively_iThe abscissa of the diagonally positioned corner point ofA vertical coordinate; left_j、top_jAny corner m in the surrounding rectangular frame of the vehicle target j respectively_jAbscissa and ordinate, right_j、bottom_jAt corner m in a bounding rectangular frame of a vehicle object j, respectively_jThe abscissa and ordinate of the corner point of the diagonal position of (a);

step 302A, compare IoU_ijIntersecting with a preset rectangular frame and comparing with a Threshold value IoU _ Threshold if IoU_ijIf the value is less than IoU _ Threshold, the vehicle target i and the vehicle target j are judged not to be at the same spatial position on the road; otherwise, the same spatial positions of the vehicle target i and the vehicle target j on the road are judged, and the step 303A is executed;

step 303A, calculating cosine distance Dist (ID) between the characteristic vector of the vehicle i and the characteristic vector of the vehicle target j according to formula (6)_j，ID_i)：

Wherein r is_IDiApplying a convolutional neural network to the vehicle target i to obtain a normalized feature vector;

the vehicle target j is a vector obtained by transposing a normalized feature vector obtained by applying a convolutional neural network;

step 304A, compare Dist (ID)_j，ID_i) And a preset feature vector Threshold value Dist _ Threshold, if:

Dist(ID_j，ID_i) Judging that the vehicle i and the vehicle j are the same vehicle and the vehicle j does not move if the distance between the vehicle i and the vehicle j is not larger than the Dist _ Threshold; otherwise, the vehicle j is determined to move.

4. The abnormal parking detection method based on intelligent video analysis as claimed in claim 2, wherein in step 3, the method for determining whether there is a vehicle that does not move in the current video frame specifically comprises the following steps:

for each vehicle target j, j in the current video frame as the value of the target ID of the vehicle target in the current video frame, the following operations are executed:

step 301B, for each vehicle target i, i in the tracking queue, the value of the target ID of the vehicle target in the tracking queue: referring to equation (1), the intersection ratio IoU of the bounding rectangular frame of the vehicle object i and the bounding rectangular frame of the vehicle object j is calculated_ij：

IoU_ij＝Inter_ij/Union_ij (1)

Wherein, Inter_ijIs the intersection of the bounding rectangular box of vehicle object i and the bounding rectangular box of vehicle object j, Union_ijIs the union of the bounding rectangular frame of vehicle object i and the bounding rectangular frame of vehicle object j; inter_ijAnd Union_ijThe calculation formula of (a) is as follows:

Inter_ij＝Inter_W_ij*Inter_H_ij (2)

Union_ij＝(right_i–left_i)*(top_i–bottom_i)+(right_j–left_j)*(top_j–bottom_j)-Inter_ij(3)

Inter_W_ij＝max(0，min(right_i,right_j)-max(left_i，left_j)) (4)

Inter_H_ij＝max(0，min(top_i,top_j)-max(bottom_i，bottom_j)) (5)

wherein, Inter _ W_ijIs the width of the intersection of the bounding rectangular frame of the vehicle object i and the bounding rectangular frame of the vehicle object j, Inter _ H_ijThe height of the intersection part of the surrounding rectangular frame of the vehicle target i and the surrounding rectangular frame of the vehicle target j; wherein left_i、top_iAny corner m in the surrounding rectangular frame of the vehicle target i_iAbscissa and ordinate, right_i、bottom_iMiddle of bounding rectangle frame for vehicle target iAt the corner point m_iThe abscissa and ordinate of the corner point of the diagonal position of (a); left_j、top_jAny corner m in the surrounding rectangular frame of the vehicle target j respectively_jAbscissa and ordinate, right_j、bottom_jAt corner m in a bounding rectangular frame of a vehicle object j, respectively_jThe abscissa and ordinate of the corner point of the diagonal position of (a);

step 302B, compare IoU_ijIntersecting with a preset rectangular frame and comparing with a Threshold value IoU _ Threshold if IoU_ijIf the value is less than IoU _ Threshold, the vehicle target i and the vehicle target j are judged not to be at the same spatial position on the road; otherwise, the same spatial positions of the vehicle target i and the vehicle target j on the road are judged, and step 303B is executed;

step 303B, calculating cosine distance Dist' (ID) between the feature vector of vehicle i and the feature vector of vehicle target j according to formula (7)_j，ID_i)：

In the formula,

obtaining a normalized feature vector by applying a convolutional neural network to a vehicle target i in an nth historical video frame with the vehicle target i, wherein M is the number of the historical video frames with the vehicle target i;

the vehicle target j is a vector obtained by transposing a normalized feature vector obtained by applying a convolutional neural network;

step 304B, compare Dist' (ID)_j，ID_i) And a predetermined feature vector Threshold value Dist _ Threshold if Dist' (ID)_j，ID_i) Judging that the vehicle i and the vehicle j are the same vehicle and the vehicle j does not move if the distance between the vehicle i and the vehicle j is not larger than the Dist _ Threshold; otherwise, the vehicle j is determined to move.

5. The intelligent video analysis-based abnormal parking detection method according to claim 1, wherein in step C, after determining that there is a vehicle target with abnormal parking in the current video frame, the method further comprises reporting an abnormal parking event.

6. The abnormal parking detection method based on intelligent video analysis of claim 1, wherein in step 401, the method further comprises: calculating the dwell time parkingtime of the vehicle object s that is not moving in the current video frame using equation (9)_S：

Parking_time_S＝Update_time_IDs-Init_time_IDd (9)

Wherein Update _ time_IDsInit _ time, which is the time when the vehicle object s that has not moved in the current video frame was last detected_IDdThe time when the vehicle target d corresponding to the vehicle target s which does not move in the current video frame is detected for the first time in the tracking queue; s is the value of the target ID of the vehicle target that does not move in the current video frame, and d is the value of the target ID of the vehicle target in the tracking queue that corresponds to the vehicle target s that does not move in the current video frame.

7. The method for detecting abnormal parking based on intelligent video analysis of claim 1, wherein in step 402 and in step 403, after the vehicle which is not moving in step 401 and has the staying time longer than the preset abnormal parking time threshold value is determined to be abnormally parked, the method further comprises reporting an abnormal parking event.

8. The abnormal parking detection method based on intelligent video analysis of claim 1, wherein in step 5, the method for updating the tracking queue information comprises the following steps:

step 501, replacing the vehicle position, the vehicle characteristic vector and the time of the vehicle which is detected last time of the vehicle in the tracking queue by using the vehicle position, the vehicle characteristic vector and the detection time of the vehicle target which does not move in the current video frame;

step 502, aiming at the vehicle target k except the vehicle target k participating in the operation of the step 701 in the tracking queue, and the survival Time Life _ Time of the vehicle target k_IDkUsing equation (10) to calculate:

Life_Time_IDk＝Now_Time–Update_time_IDk (10)

wherein, the Now _ Time is the detection Time corresponding to the current video frame, Update _ Time_IDkThe detection time when the vehicle target k is detected last time;

compare Life _ Time_IDkAnd a preset tracked vehicle target expiration threshold

Track _ Expiration _ Threshold: if:

Life_Time_IDk>track _ Expiration _ Threshold, then remove the vehicle target k, including its attribute information, from the tracking queue; otherwise, the vehicle target k is reserved, and the attribute information of the vehicle target k is not changed; and adding the attribute information of the vehicle target which appears in the current video frame and does not appear in the tracking queue into the tracking queue.

9. The abnormal parking detection method based on intelligent video analysis of any one of claims 1 to 8, wherein the convolutional neural network model is a yoloV3 convolutional neural network model.

Images