CN112784834A - Automatic license plate identification method in natural scene - Google Patents

Abstract

The invention relates to an automatic license plate recognition method in a natural scene, which comprises three steps of vehicle detection, license plate detection and character recognition. Firstly, carrying out vehicle detection on a given input image by using a YOLOv3 algorithm, then searching a license plate of a detected vehicle by using a deformed plane target detection network WPOD-NET, and correcting the distorted license plate by using affine transformation parameters to form a rectangle of a front view; and finally, inputting the license plate into an LPRNET network, and identifying license plate characters with variable lengths. The method considers the inclination and poor quality of the given image in a natural scene, has less quantity of cascaded models, can identify the license plate with variable length, and has better practicability.

Description

Automatic license plate identification method in natural scene

Technical Field

The invention relates to the technical field of digital image processing, in particular to a cascading algorithm for vehicle detection, license plate detection and character recognition in automatic license plate recognition in a natural scene.

Background

The automatic license plate recognition is to perform operations such as vehicle detection, license plate correction, character segmentation, character recognition and the like on vehicles appearing in the collected image, and convert image information into character information to recognize the identity of the vehicles. In recent years, license plate automatic identification systems have been widely used in traffic-related scenes, such as the fields of stolen vehicle detection, toll control, traffic management, digital security monitoring and large-city parking lot parking management.

At present, the optimization of the automatic license plate recognition system becomes a research hotspot. Recent advances in parallel processing and deep learning have improved many computer vision tasks, and deep Convolutional Neural Networks (CNNs) have become an advanced machine learning technique for vehicle detection, license plate detection, Optical Character Recognition (OCR), and have optimized automatic license plate recognition systems. In addition, some commercial automatic license plate recognition systems (Sighthound, OpenALPR, Amazon knowledge) are also exploring deep learning methods, which are usually deployed in huge data centers, and can process thousands to millions of images every day by network service work, and are continuously improved.

Most of current license plate automatic identification systems collect the front view of a vehicle in a monitoring mode to identify the license plate, but under the natural scene, the vehicle image is easy to incline and the image quality is poor due to the influences of factors such as distance, inclination, blur, illumination change, weather conditions and the like, and the existing method has the following defects:

(1) in a natural scene, the vehicle images collected in a more relaxed manner may cause the viewing angle to be inclined, the license plate may be highly distorted and still readable, and the license plate detection accuracy is low.

(2) Under a natural scene, the image quality of the vehicle is poor under the influence of factors such as illumination and weather, and the accuracy rate of license plate character recognition is low at the moment.

(3) The automatic license plate recognition system comprises a plurality of cascade models, and the recognition speed is low.

Disclosure of Invention

Aiming at the defects, the invention provides an automatic license plate recognition method in a natural scene, which considers the inclination and low quality of the collected vehicle image in the natural scene on the basis of the existing automatic license plate recognition system, optimizes the license plate detection and character recognition and realizes the license plate recognition in the natural scene.

The technical scheme of the invention is as follows:

a method for automatically identifying a license plate in a natural scene is characterized by comprising the following steps:

(1) vehicle detection: firstly, carrying out vehicle detection on a given input image by using a YOLOv3 algorithm;

(2) and (3) detecting the license plate: then, for the detected vehicle, a deformed plane target detection network is used for searching for the license plate, affine transformation parameters are used for correcting the distorted license plate into a rectangle of a front view, and a corrected license plate image is obtained;

(3) character recognition: and finally, inputting the license plate image into an LPRNET network, and performing character recognition on the corrected license plate image to recognize license plate characters with variable lengths.

In the step (1), a classical target detection algorithm YOLOv3 is selected to detect the vehicles in the given image, the related categories of the vehicles are reserved, other categories are deleted, and a YOLOv3 model is not modified.

In the step (2), the network WPOD-NET is detected by using the deformed plane target, and the specific process is as follows:

firstly, the output of a vehicle detection module is zoomed and then sent to a WPOD, and 8-channel feature mapping including 2 target/non-target probabilities and 6 affine transformation parameters is obtained in the forward process;

then, an imaginary rectangular frame of a fixed size (m, n) is set around the center of the cell, and if the probability that the rectangular frame contains an object is higher than a given detection threshold, an affine matrix that transforms the imaginary square (cell) into a license plate region is constructed using partial regression parameters, and the license plate is corrected to an object aligned horizontally and vertically by perspective transformation.

In the step (4), the license plate characters are identified by using the LPRNET network, and the specific process is as follows:

acquiring an original RGB picture as input through a backbone network of an LPRNET network, and calculating the spatial distribution of a large number of features; the output layer of the backbone network is calculated through the full connection layer, then the output layer is tiled to the required size, and finally the output layer is spliced with the output of the backbone network.

On the basis of the existing automatic license plate recognition system, the inclination and low quality of the collected vehicle image in a natural scene are considered, the license plate detection (the license plate detection and the license plate correction model are classified into one model) and the character recognition (the character segmentation and the character recognition model are classified into one model) are optimized, the license plate recognition in the natural scene is realized, and the expected effect is achieved. Compared with the prior art, the method considers the inclination and poor quality of the given image in the natural scene, has less quantity of cascaded models, can identify the license plate with variable length, and has better practicability.

Detailed Description

The method comprises three steps of vehicle detection, license plate detection and character recognition. Firstly, carrying out vehicle detection on a given input image by using a YOLOv3 algorithm; then, for the detected vehicle, searching a license plate by using a deformed plane target detection network (WPOD-NET), and correcting the distorted license plate by using affine transformation parameters to form a rectangle with a front view; and finally, inputting the license plate into an LPRNET network, and identifying license plate characters with variable lengths. The specific implementation mode is as follows:

(1) vehicle detection:

the classical target detection algorithm YOLOv3 is selected to detect vehicles in a given image, the vehicle-related categories are retained, other categories are deleted, and the YOLOv3 model is not modified.

(2) And (3) detecting the license plate:

license plate detection is carried out by using a deformed planar object detection network (WPOD-NET), which is developed by using ideas of YOLO, SSD and Spatial Transform Networks (STN). YOLO and SSD perform fast multi-target detection and recognition based on a single input, but they do not consider spatial transformations, only generate rectangular bounding boxes for each detection. In contrast, STN can be used to detect non-rectangular regions, but it cannot process multiple transforms simultaneously, performing only a single spatial transform on the entire input.

Initially, the output of the vehicle detection module is scaled and sent to the WPOD. The forward process results in an 8-channel feature map, which contains target/non-target probabilities (2) and affine transformation parameters (6). To extract a warped license plate, an imaginary rectangular frame of fixed size (m, n) around the center of the cell is first considered. If the probability that the rectangular box contains the target is higher than a given detection threshold, partial regression parameters are used to construct an affine matrix that transforms the imaginary squares (cells) into the license plate region. Thus, the license plate can be easily corrected to be horizontally and vertically aligned objects (by perspective transformation).

The network architecture of WPOD-NET has a total of 21 convolutional layers, 14 of which are contained in the residual block. The size of all convolution filters is fixed at 3 x 3. The ReLU activation function is used throughout the network, except for the detection block. There are 4 largest pools, size 2 x 2, step 2, which can reduce the input dimension by a factor of 16. Finally, the detection block has two parallel convolutional layers: (i) a probability value for inferring activation by the softmax function; (ii) the other is used to regress affine parameters without an activation function (or equivalently, using an identity function f (x) x as the activation function).

(3) Character recognition:

and identifying the license plate characters by using an LPRNET network. The backbone network of the LPRNET network takes the original RGB picture as input and computes the spatial distribution of a large number of features. The wide convolution (convolution kernel of 1 x 13) takes advantage of the context of the native character and thus replaces the LSTM-based RNN network. The output of the backbone sub-network can be considered as a sequence representing the likelihood of the corresponding character, whose length is equal to the width of the input image. Since the output of the decoder is not consistent with the length of the target character sequence, a CTC loss function is employed for end-to-end training without segmentation, a method widely used to deal with misalignment of input and output sequences.

In order to further improve the expression of the model, the intermediate feature map obtained by the decoder is pre-enhanced, and the embedding is carried out by using a global context relationship. The output layer of the backbone network is calculated through a full connection layer, then the output layer is tiled to the required size, and finally the output layer is spliced with the output of the backbone network. To adjust the depth of the features mapped to each character class, a 1 × 1 convolution is employed.

Claims (4)

1. A method for automatically identifying a license plate in a natural scene is characterized by comprising the following steps:

(1) vehicle detection: firstly, carrying out vehicle detection on a given input image by using a YOLOv3 algorithm;

(2) and (3) detecting the license plate: then, for the detected vehicle, a deformed plane target detection network is used for searching for the license plate, affine transformation parameters are used for correcting the distorted license plate into a rectangle of a front view, and a corrected license plate image is obtained;

(3) character recognition: and finally, inputting the license plate image into an LPRNET network, and performing character recognition on the corrected license plate image to recognize license plate characters with variable lengths.

2. The automatic license plate recognition method in the natural scene as recited in claim 1, wherein: in the step (1), a classical target detection algorithm YOLOv3 is selected to detect the vehicles in the given image, the related categories of the vehicles are reserved, other categories are deleted, and a YOLOv3 model is not modified.

3. The automatic license plate recognition method in the natural scene as recited in claim 1, wherein: in the step (2), the deformation plane target is used for detecting the network WPOD-NET, and the specific process is as follows:

firstly, the output of a vehicle detection module is zoomed and then sent to a WPOD, and 8-channel feature mapping including 2 target/non-target probabilities and 6 affine transformation parameters is obtained in the forward process;

then, an imaginary rectangular frame of a fixed size (m, n) is set around the center of the cell, and if the probability that the rectangular frame contains an object is higher than a given detection threshold, an affine matrix that transforms the imaginary square (cell) into a license plate region is constructed using partial regression parameters, and the license plate is corrected to an object aligned horizontally and vertically by perspective transformation.

4. The automatic license plate recognition method in the natural scene as recited in claim 1, wherein: in the step (4), the LPRNET network is used for identifying the license plate characters, and the specific process is as follows:

acquiring an original RGB picture as input through a backbone network of an LPRNET network, and calculating the spatial distribution of a large number of features; the output layer of the backbone network is calculated through the full connection layer, then the output layer is tiled to the required size, and finally the output layer is spliced with the output of the backbone network.