CN113052170B - Small target license plate recognition method under unconstrained scene - Google Patents
Small target license plate recognition method under unconstrained scene Download PDFInfo
- Publication number
- CN113052170B CN113052170B CN202110300861.1A CN202110300861A CN113052170B CN 113052170 B CN113052170 B CN 113052170B CN 202110300861 A CN202110300861 A CN 202110300861A CN 113052170 B CN113052170 B CN 113052170B
- Authority
- CN
- China
- Prior art keywords
- license plate
- image
- plate region
- scene
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000001514 detection method Methods 0.000 claims abstract description 59
- 238000012549 training Methods 0.000 claims abstract description 34
- 238000013528 artificial neural network Methods 0.000 claims abstract description 15
- 238000005286 illumination Methods 0.000 claims abstract description 13
- 230000006870 function Effects 0.000 claims description 23
- 238000013145 classification model Methods 0.000 claims description 18
- 230000009466 transformation Effects 0.000 claims description 11
- 238000012360 testing method Methods 0.000 claims description 10
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000013518 transcription Methods 0.000 claims description 5
- 230000035897 transcription Effects 0.000 claims description 5
- 230000002457 bidirectional effect Effects 0.000 claims description 3
- 238000003062 neural network model Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 10
- 238000010191 image analysis Methods 0.000 abstract description 9
- 238000012937 correction Methods 0.000 abstract description 4
- 238000013461 design Methods 0.000 description 9
- 239000013598 vector Substances 0.000 description 7
- 238000011176 pooling Methods 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000009826 distribution Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 102100032202 Cornulin Human genes 0.000 description 1
- 101000920981 Homo sapiens Cornulin Proteins 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000011840 criminal investigation Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/24—Aligning, centring, orientation detection or correction of the image
- G06V10/242—Aligning, centring, orientation detection or correction of the image by image rotation, e.g. by 90 degrees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/148—Segmentation of character regions
- G06V30/153—Segmentation of character regions using recognition of characters or words
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/625—License plates
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a small target license plate recognition method under an unconstrained scene, which combines a plurality of image analysis technologies, respectively introduces a neural network for application aiming at license plate region image detection, license plate classification and license plate content recognition, obtains a model under each application through sample training, combines a detection shape correction technology to realize the small target license plate recognition under the unconstrained scene, adopts the high-efficiency image analysis technology in the whole scheme, can accurately recognize the license plate with the license plate length and width of 40 multiplied by 15 pixel level, has great tolerance on factors such as shooting angle, illumination and the like, has wider application scene compared with the license plate recognition algorithm in the prior market, and has high practical value for road traffic and safe cities.
Description
Technical Field
The invention relates to a small target license plate recognition method under an unconstrained scene, and belongs to the technical field of intelligent video and image analysis.
Background
Automatic license plate recognition has been widely used in road traffic, parking lots, and communities, however, these applications are basically performed in limited scenes, and mainly are represented by: the cameras for identifying license plates, such as road traffic, parking lot entrances and exits, have strict requirements on engineering implementation and installation, and the taken license plate pictures are relatively front and clear, so that the accuracy of identification can be ensured.
In a real situation, people often need to accurately identify a small target license plate under an unconstrained scene, wherein the unconstrained scene refers to a situation that angles, definition, illumination conditions and the like of a camera are not explicitly required, a license plate in an image, which is possibly photographed under a large angle, is possibly affected by illumination and even is distorted; the term "small target license plate" refers to the situation that the license plate occupies a relatively small area in the whole video or image, so that the license plate cannot be detected by the conventional algorithm.
The recognition of the small target license plate in the unconstrained scene is a challenging work, but has very important practical value for road traffic, public security criminal investigation and other applications. .
Disclosure of Invention
The technical problem to be solved by the invention is to provide a small target license plate recognition method under an unconstrained scene, which can effectively improve the image analysis efficiency and obtain higher license plate recognition precision by combining various image analysis technologies.
The invention adopts the following technical scheme for solving the technical problems: the invention designs a small target license plate recognition method under an unconstrained scene, which comprises the following steps of i1 to i2, obtaining a license plate region image detection model, and ii1 to ii3, obtaining a license plate content recognition model; and then based on the license plate region image detection model and the license plate content recognition model, executing the following steps A to C to realize the recognition of the target license plate content in the target scene;
step i1., collecting scene sample images containing license plates of specified types, obtaining license plate region images respectively contained in the scene sample images, obtaining four corner coordinates of the license plate region images, and then entering step i2;
step i2., taking each scene sample image as input, taking a license plate region image in each scene sample image and four corner coordinates of the license plate region image as output, and training aiming at a first appointed neural network to obtain a license plate region image detection model;
step i1, sequentially connecting a convolution layer formed by a deep convolution neural network model, a circulation layer formed by a bidirectional circulation neural network and a transcription layer formed by a connection time sequence classification model, constructing a license plate content recognition initial model, and then entering step ii2;
step i2, collecting sample images of each license plate region, obtaining license plate numbers in the sample images of each license plate region, and then entering step ii3;
step ii3, taking each license plate region sample image as input, taking license plate numbers in each license plate region sample image as output, and training for a license plate content recognition initial model to obtain a license plate content recognition model;
step A, obtaining a target scene image, applying a license plate region image detection model, detecting and obtaining a target license plate region image in the target scene image and four corner coordinates of the target license plate region image, and then entering the step B;
b, correcting the target license plate region image to a rectangular shape according to the coordinates of four corner points of the target license plate region image, updating the target license plate region image, and then entering the step C;
and C, identifying the target license plate region image by using a license plate content identification model to obtain a license plate number corresponding to the target license plate region image.
As a preferred technical scheme of the invention: the step i1 comprises the following steps i1-1 to i1-2;
step i1-1, collecting scene sample images containing license plates of specified types, obtaining license plate region images respectively contained in the scene sample images, obtaining four corner coordinates of the license plate region images, and then entering step i1-2;
step i1-2, respectively aiming at each scene sample image, adopting a specified image deformation method to obtain each transformed scene sample image corresponding to the scene sample image, establishing the corresponding relation between each transformed scene sample image and four corner coordinates of a license plate region image and a license plate region image in the scene sample image, and taking each transformed scene sample image as each scene sample image; step i2 is then entered.
As a preferred technical scheme of the invention: the specified image deformation methods comprise illumination distortion methods for performing brightness change, contrast change, saturation change and noise change, geometric distortion methods for performing random scaling, clipping and rotation, and joint methods for performing illumination distortion and geometric distortion.
As a preferred technical scheme of the invention: in the step i2, aiming at a minimum rectangular frame containing a license plate area image, training is performed according to a confidence loss function containing the license plate area image in the minimum rectangular frame, a regression loss function of the minimum rectangular frame and detection loss functions of four corner coordinates of the license plate area image, and a preset overflow condition of each loss function, wherein each scene sample image is taken as an input, each license plate area image in each scene sample image and four corner coordinates of the license plate area image are taken as an output, and a depth convolution network YoloV5 serving as a first appointed neural network is used for obtaining a license plate area image detection model;
regression loss function of minimum rectangular frame of license plate region image:
in the formula, CIoU Loss Representing a detection loss of a minimum rectangular frame including a license plate region image, ioU representing an intersection ratio of the minimum rectangular frame to a true minimum rectangular frame, v representing a distance of aspect ratio of the minimum rectangular frame to the true minimum rectangular frame, α representing a weight coefficient, ρ representing a euclidean distance between center points of the minimum rectangular frame and the true minimum rectangular frame, b and b gt Respectively representing the center points of the detection minimum rectangular frame and the real minimum rectangular frame, c represents the detection minimum rectangular frame and the real minimum momentThe length of the diagonal line of the circumscribed rectangle of the shape frame;
wherein w represents the width of the minimum rectangular frame, h represents the height of the minimum rectangular frame, and w gt Represents the width of a real minimum rectangular frame, h gt Representing the height of a real minimum rectangular frame;
detection loss function of four corner coordinates of license plate region image:
wherein x represents the distance between the diagonal intersection point of the quadrangle formed by the four corner coordinates obtained by detection and the diagonal intersection point of the quadrangle formed by the real four corner coordinates,the detection loss function corresponding to x is shown.
As a preferred technical scheme of the invention: the method also comprises the following steps i1-i2, and after the step i1 is executed, the step i1-i2 is entered;
step i1-i2., according to the license plate region images corresponding to the scene sample images and four corner coordinates of the license plate region images, presetting a license plate region image with a large size range according to rule 1, presetting a license plate region image with a small size range according to rule 2, presetting a license plate region image with an inclination angle exceeding a preset angle threshold value, and presetting a license plate region image with an inclination angle not exceeding a preset angle threshold value according to rule 4, wherein the license plate region images are divided into the following four categories according to the scene sample images:
category 1, license plate region images meeting rule 3 and rule 2;
category 2, license plate region images meeting rule 3 and rule 1;
category 3, license plate region images meeting rule 4 and rule 2;
category 4, license plate region images meeting rule 4 and rule 1, and then entering step i2;
the step i2 comprises the following steps i2-1 to i2-4;
step i2-1, randomly selecting each scene sample image from four categories respectively with equal selection probability to serve as each training scene sample image, and entering the step i2-2;
step i2-2, taking each scene sample image as input, taking a license plate region image in each scene sample image and four corner coordinates of the license plate region image as output, performing iterative training on a first appointed neural network to obtain a detection model, and then entering step i2-3;
step i2-3, testing the detection model based on scene sample images in four categories, obtaining test precision of the detection model corresponding to the four categories respectively, judging whether each test precision meets a preset precision threshold, if so, obtaining the detection model as a license plate area image detection model, otherwise, entering the step i2-4;
step i2-4, respectively increasing the selection probability corresponding to each category according to the preset step length aiming at each category which does not meet the preset precision threshold, then respectively randomly selecting each scene sample image from four categories according to the selection probability corresponding to each category, taking the selected scene sample image as each training scene sample image, and returning to the step i2-2.
As a preferred technical scheme of the invention: the method further comprises the following steps of obtaining a license plate classification model;
collecting various license plate region classification sample images of various types, taking the license plate region classification sample images as input, taking the types corresponding to the license plate region classification sample images as output, and training a residual error network ResNet18 to obtain a license plate classification model;
the license plate content recognition model is obtained in the process of:
step ii2, collecting various license plate region sample images designated in the step ii1, obtaining license plate numbers in the license plate region sample images, and then entering the step ii3;
step i3, respectively aiming at various types, taking sample images of various license plate areas in the types as input, taking license plate numbers in the sample images of various license plate areas as output, training for an initial license plate content recognition model to obtain license plate content recognition models corresponding to the types, and further obtaining license plate content recognition models corresponding to the various types respectively;
the license plate content recognition process in the scene further comprises the following step BC, and after the step B is executed, the step BC is entered;
step BC, applying a license plate classification model, classifying types of the license plate region images to obtain target types corresponding to the license plate region images, and then entering the step C;
and C, applying a license plate content recognition model corresponding to the target type, and recognizing the target license plate region image to obtain a license plate number corresponding to the target license plate region image.
As a preferable technical scheme of the invention, in the obtaining process of the license plate classification model:
collecting and appointing various license plate region classification sample images, and respectively applying random perspective transformation, contrast variation, brightness variation and saturation variation to the license plate region classification sample images to obtain various transformed license plate region classification sample images corresponding to the license plate region classification sample images, wherein the transformed license plate region classification sample images are used as license plate region classification sample images; and then taking each license plate region classification sample image as input, taking the type corresponding to each license plate region classification sample image as output, and training the residual error network ResNet18 to obtain a license plate classification model.
As a preferred technical scheme of the invention: in the step B, according to four corner coordinates of the target license plate area image, new coordinates corresponding to the four corner points under the corresponding rectangular shape are obtained by applying opencv calculation, then an affine transformation matrix is obtained by applying opencv calculation based on the relation between the four corner point coordinates and the new coordinates, finally the target license plate area image is corrected to the rectangular shape according to the affine transformation matrix, the target license plate area image is updated, and then the step C is entered.
Compared with the prior art, the small target license plate recognition method under the unconstrained scene has the following technical effects:
the invention designs a small target license plate recognition method under an unconstrained scene, which combines a plurality of image analysis technologies, respectively introduces a neural network for application aiming at license plate region image detection, license plate classification and license plate content recognition, obtains a model under each application through sample training, combines a detection shape correction technology to realize the small target license plate recognition under the unconstrained scene, adopts the high-efficiency image analysis technology in the whole scheme, can accurately recognize the license plate with the license plate length and width of 40 multiplied by 15 pixel level, has great tolerance on factors such as shooting angle, illumination and the like, has wider application scene compared with the license plate recognition algorithm in the prior market, and has high practical value for road traffic and safe cities.
Drawings
FIG. 1 is a flow chart of a small target license plate recognition method in an unconstrained scene designed by the invention;
FIG. 2 is a schematic flow chart of steps i2-1 to i2-4 in the design of the present invention;
FIG. 3 is a schematic diagram of an application of license plate extraction and correction in the design of the present invention;
FIG. 4 is a ResNet18 network architecture for license plate classification in accordance with the present invention;
FIG. 5 is a license plate content recognition network structure in accordance with the present invention;
fig. 6 shows license plate recognition results in the design of the present invention.
Detailed Description
The following describes the embodiments of the present invention in further detail with reference to the drawings.
The invention designs a small target license plate recognition method under an unconstrained scene, which is implemented by specifically executing the following steps i1 to i2 in practical application to obtain a license plate region image detection model.
Step i1., collecting each scene sample image containing the license plate of each type, obtaining license plate region images correspondingly contained in each scene sample image, obtaining four corner coordinates of the license plate region images, and then entering steps i1-i2.
In practical applications, the step i1 is specifically performed as follows from step i1-1 to step i1-2.
Step i1-1, collecting scene sample images containing license plates of specified types, obtaining license plate region images respectively contained in the scene sample images, obtaining four corner coordinates of the license plate region images, and then entering step i1-2.
Step i1-2, respectively aiming at each scene sample image, adopting a specified image deformation method to obtain each transformed scene sample image corresponding to the scene sample image, establishing the corresponding relation between each transformed scene sample image and four corner coordinates of a license plate region image and a license plate region image in the scene sample image, and taking each transformed scene sample image as each scene sample image; and then proceeds to steps i1-i2.
The specified image distortion methods include an illumination distortion method that performs brightness variation, contrast variation, saturation variation, noise variation, a geometric distortion method that performs random scaling, cropping, rotation, and a joint method that performs illumination distortion and geometric distortion. The geometric distortion can influence the bounding box of the sample label, so that the bounding box of the sample label is updated while the geometric distortion is carried out on the image; for the execution of the illumination distortion and geometric distortion combined method, the illumination distortion and geometric distortion (especially the picture is reduced and rotated) are firstly carried out on four randomly extracted pictures, and then the new pictures are synthesized by cutting and splicing at random positions; by the method, a large amount of small target license plate data in an unconstrained scene is obtained.
Step i1-i2., according to the license plate region images corresponding to the scene sample images and four corner coordinates of the license plate region images, presetting a license plate region image with a large size range according to rule 1, presetting a license plate region image with a small size range according to rule 2, presetting a license plate region image with an inclination angle exceeding a preset angle threshold value, and presetting a license plate region image with an inclination angle not exceeding a preset angle threshold value according to rule 4, wherein the license plate region images are divided into the following four categories according to the scene sample images:
category 1, license plate region images meeting rule 3 and rule 2;
category 2, license plate region images meeting rule 3 and rule 1;
category 3, license plate region images meeting rule 4 and rule 2;
and 4. Meeting license plate region images of the rule 4 and the rule 1, and then entering the step i2.
For the rules herein, such as defining license plates less than 41 pixels in length and less than 16 pixels in width as small-size-range license plate area images, others as large-size-range license plate area images; and defining license plate region images with inclination angles exceeding 30 degrees and license plate region images with inclination angles not exceeding 30 degrees.
In step i2., each scene sample image is taken as input, the license plate region image and four corner coordinates of the license plate region image in each scene sample image are taken as output, and training is performed on the first designated neural network to obtain a license plate region image detection model.
In practical applications, as shown in FIG. 2, the specific design step i2 includes executing the following steps i2-1 to i2-4.
Step i2-1, randomly selecting each scene sample image from four categories with equal selection probability to serve as each training scene sample image, and entering step i2-2.
And i2-2, carrying out iterative training on a first appointed neural network by taking each scene sample image as input and taking a license plate region image and four corner coordinates of the license plate region image in each scene sample image as output to obtain a detection model, and then entering the step i2-3.
Step i2-3, testing the detection model based on scene sample images in four categories, obtaining test precision of the detection model corresponding to the four categories respectively, judging whether each test precision meets a preset precision threshold, if so, obtaining the detection model as a license plate area image detection model, otherwise, entering step i2-4.
Step i2-4, respectively increasing the selection probability corresponding to each category according to the preset step length aiming at each category which does not meet the preset precision threshold, then respectively randomly selecting each scene sample image from four categories according to the selection probability corresponding to each category, taking the selected scene sample image as each training scene sample image, and returning to the step i2-2.
In practical application, for the minimum rectangular frame including the license plate region image, according to the confidence loss function including the license plate region image in the minimum rectangular frame, the regression loss function including the minimum rectangular frame and the detection loss function including four corner coordinates of the license plate region image as follows, and by taking each scene sample image as input and taking four corner coordinates of the license plate region image and the license plate region image in each scene sample image as output in combination with preset overflow conditions of each loss function, training is performed for the depth convolution network YoloV5 serving as the first specified neural network to obtain a license plate region image detection model, wherein for (1) bounding box regression (bounding box regression) loss, (2) confidence loss and (3) classification loss included in the existing depth convolution network YoloV5, further application design is performed, and then the regression loss function including the minimum rectangular frame of the license plate region image is applied as follows:
in the formula, CIoU Loss Representing a detection loss of a minimum rectangular frame including a license plate region image, ioU representing an intersection ratio of the minimum rectangular frame to a true minimum rectangular frame, v representing a distance of aspect ratio of the minimum rectangular frame to the true minimum rectangular frame, α representing a weight coefficient, ρ representing a euclidean distance between center points of the minimum rectangular frame and the true minimum rectangular frame, b and b gt Respectively representing the center points of the minimum detection rectangular frame and the real minimum detection rectangular frame, c represents the minimum detection rectangular frame and the real minimum detection rectangular frameThe length of the diagonal of the circumscribed rectangle of the true minimum rectangle frame.
Wherein w represents the width of the minimum rectangular frame, h represents the height of the minimum rectangular frame, and w gt Represents the width of a real minimum rectangular frame, h gt Representing the height of the true minimum rectangular box.
The detection loss function of the four corner coordinates of the license plate region image is applied as follows:
wherein x represents the distance between the diagonal intersection point of the quadrangle formed by the four corner coordinates obtained by detection and the diagonal intersection point of the quadrangle formed by the real four corner coordinates,the detection loss function corresponding to x is shown.
In the design, obtaining a license plate classification model, specifically designing and collecting various license plate region classification sample images of various types, and respectively aiming at the various license plate region classification sample images, applying random perspective transformation, contrast variation, brightness variation and saturation variation to obtain various transformed license plate region classification sample images corresponding to the license plate region classification sample images, wherein the transformed license plate region classification sample images are used as license plate region classification sample images; and then taking each license plate region classification sample image as input, taking the type corresponding to each license plate region classification sample image as output, and training the residual error network ResNet18 to obtain a license plate classification model.
For the residual network ResNet18, the residual network ResNet18 is designed to be built on an open source deep learning framework PyTorch to realize license plate classification, and the ResNet18 network structure is divided into three parts as shown in FIG. 4:
1) An input section: is a large convolution kernel of size 7x7 and step size 2, and a maximum pooling of size 3x3 and step size 2, by which a 224x224 input image is changed to a feature map of size 56x56, greatly reducing the size required for storage.
2) An intermediate convolution section: the method consists of 4 blocks of block1, block2, block3 and block4, and each block is subjected to convolution by 3*3 to realize information extraction by stacking 2 times.
3) An output section: all feature maps are pulled 1*1 by global adaptive smoothing pooling, i.e., 1x512x7x7 input data is pulled 1x512x1x1, then connected to the full connection layer output.
A training data set and a test set need to be prepared before training, and license plates in the data set are unified to 224x224 resolution sizes through a size operation. Besides the original data set, artificially generated license plate data is additionally added, and is subjected to random perspective transformation and contrast, brightness and saturation transformation for data enhancement. After the dataset preparation is completed, training continues with ResNet18 above, using the cross entropy loss function CrossEntropyLoss.
In addition, the following steps ii1 to ii3 are designed and executed to obtain the license plate content recognition model.
Step ii1. As shown in fig. 5, a convolution layer formed by a deep convolution neural network model, a circulation layer formed by a bidirectional circulation neural network, and a transcription layer formed by a connection time sequence classification model are sequentially connected to construct a license plate content recognition initial model, and then step ii2 is performed.
The license plate content recognition initial model is specifically described as follows:
1) Convolution layer: and extracting features from the input image by using the depth CNN to obtain a feature map.
Before the input image enters the convolution layer, the input image needs to be subjected to scaling pretreatment, the license plate image is scaled to be 32 XW multiplied by 1 in resolution, namely, 32 is high, W is wide and can be any number, 1 is the number of channels, and the license plate image represents a gray image.
The convolution operation consists of a convolution layer and a maximum pooling layer in a standard CNN model, and comprises a series of operations such as convolution, maximum pooling, batch normalization and the like.
Feature images obtained by CNN cannot be directly sent to RNN for training, some adjustment is needed, and feature vector sequences needed by RNN are extracted according to the feature images. Vectors in the extracted feature sequence are sequentially generated from left to right on the feature map for input to the loop layer, each feature vector representing a feature over a certain width on the image, the default width being 1, i.e. a single pixel. Since the CRNN has scaled the input image to the same height, features need only be extracted by a certain width.
2) And (3) a circulating layer: the feature sequence is predicted using a bi-directional RNN (BLSTM), each feature vector in the sequence is learned, and a predictive tag (true value) distribution is output.
A feature vector corresponds to a small rectangular area in the loop layer of fig. 5, and the RNN aims to predict which character the rectangular area is, that is, according to the input feature vector, to predict, so as to obtain a probability distribution of all characters, which is a vector with a length of the number of character categories, and is used as an input of the CTC layer.
3) Transcription layer: and integrating the results of the LSTM network predicted feature sequences, and converting the results into final output results.
The transcription layer uses CTC Loss function, and the goal is to maximize the sum of probabilities of paths contained in labels that may be mapped (de-duplicated, de-nulled) during training (CTC assumes that the outputs of each time slice are independent of each other, the posterior probability of a path is the accumulation of the probabilities of each time slice), and search for paths with the greatest probability according to a given input search at output, and thus map to paths with the greatest likelihood of correct results.
Step i2. Collect each license plate regional sample image, and obtain the license plate number in each license plate regional sample image, then go to step ii3.
And step ii3, taking each license plate region sample image as input, taking the license plate number in each license plate region sample image as output, and training the initial license plate content recognition model to obtain a license plate content recognition model.
For obtaining license plate content recognition models through the steps ii1 to ii3, if the license plate classification models are applied, the following steps ii2 to ii3 are further designed and executed based on the execution of the step ii1.
And step ii2, collecting various license plate region sample images designated in the step ii1, obtaining license plate numbers in the license plate region sample images, and then entering the step ii3.
And step ii3, respectively aiming at various types, taking sample images of various license plate areas in the types as input, taking license plate numbers in the sample images of various license plate areas as output, training a license plate content recognition initial model to obtain license plate content recognition models corresponding to the types, and further obtaining license plate content recognition models corresponding to the various types respectively.
Then based on the obtained license plate region image detection model, license plate classification model and license plate content recognition model, further as shown in fig. 1, the following steps a to C are executed to realize the recognition of the target license plate content in the target scene.
And step A, acquiring a target scene image, applying a license plate region image detection model, detecting and acquiring a target license plate region image in the target scene image and four corner coordinates of the target license plate region image, and then entering the step B.
And B, as shown in fig. 3, according to four corner coordinates of the target license plate region image, applying opencv calculation to obtain new coordinates corresponding to the four corner points under the corresponding rectangular shape, then based on the relation between the four corner coordinates and the new coordinates, applying opencv calculation to obtain an affine transformation matrix, and finally correcting the target license plate region image to the rectangular shape according to the affine transformation matrix, updating the target license plate region image, and then entering step BC.
And step BC, applying a license plate classification model, carrying out type classification on the target license plate region image to obtain a target type corresponding to the target license plate region image, and then entering the step C.
And C, identifying the license plate region image by applying a license plate content identification model corresponding to the target type, and obtaining the license plate number corresponding to the target license plate region image, as shown in fig. 6.
The small target license plate recognition method under the unconstrained scene is designed by the technical scheme, various image analysis technologies are combined, a neural network is respectively introduced for application aiming at license plate region image detection, license plate classification and license plate content recognition, models under the applications are obtained through sample training, the detection shape correction technology is combined, the small target license plate recognition under the unconstrained scene is realized, the whole scheme adopts the efficient image analysis technology, the license plate with the license plate length and width of 40 multiplied by 15 pixel level can be accurately recognized, the tolerance to factors such as shooting angle and illumination is high, compared with the license plate recognition algorithm in the existing market, the method has wider application scene and high practical value for road traffic and safe cities.
The embodiments of the present invention have been described in detail with reference to the drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present invention.
Claims (7)
1. A small target license plate recognition method under an unconstrained scene is characterized by comprising the following steps of: the method comprises the steps of performing the following steps i1 to i2 to obtain a license plate region image detection model, and performing the following steps ii1 to ii3 to obtain a license plate content recognition model; and then based on the license plate region image detection model and the license plate content recognition model, executing the following steps A to C to realize the recognition of the target license plate content in the target scene;
step i1., collecting scene sample images containing license plates of various types, obtaining license plate region images correspondingly contained in the scene sample images, obtaining four corner coordinates of the license plate region images, and then entering step i1-i2;
step i1-i2., according to the license plate region images corresponding to the scene sample images and four corner coordinates of the license plate region images, presetting a license plate region image with a large size range according to rule 1, presetting a license plate region image with a small size range according to rule 2, presetting a license plate region image with an inclination angle exceeding a preset angle threshold value, and presetting a license plate region image with an inclination angle not exceeding a preset angle threshold value according to rule 4, wherein the license plate region images are divided into the following four categories according to the scene sample images:
category 1, license plate region images meeting rule 3 and rule 2;
category 2, license plate region images meeting rule 3 and rule 1;
category 3, license plate region images meeting rule 4 and rule 2;
category 4, license plate region images meeting rule 4 and rule 1, and then entering step i2;
step i2., taking each scene sample image as input, taking a license plate region image in each scene sample image and four corner coordinates of the license plate region image as output, and training aiming at a first appointed neural network to obtain a license plate region image detection model; step i2 includes the following steps i2-1 to i2-4;
step i2-1, randomly selecting each scene sample image from four categories respectively with equal selection probability to serve as each training scene sample image, and entering the step i2-2;
step i2-2, taking each scene sample image as input, taking a license plate region image in each scene sample image and four corner coordinates of the license plate region image as output, performing iterative training on a first appointed neural network to obtain a detection model, and then entering step i2-3;
step i2-3, testing the detection model based on scene sample images in four categories, obtaining test precision of the detection model corresponding to the four categories respectively, judging whether each test precision meets a preset precision threshold, if so, obtaining the detection model as a license plate area image detection model, otherwise, entering the step i2-4;
step i2-4, respectively increasing the selection probability corresponding to each category according to the preset step length aiming at each category which does not meet the preset precision threshold, then respectively randomly selecting each scene sample image from four categories according to the selection probability corresponding to each category, taking the selected scene sample image as each training scene sample image, and returning to the step i2-2;
step i1, sequentially connecting a convolution layer formed by a deep convolution neural network model, a circulation layer formed by a bidirectional circulation neural network and a transcription layer formed by a connection time sequence classification model, constructing a license plate content recognition initial model, and then entering step ii2; step i2, collecting sample images of each license plate region, obtaining license plate numbers in the sample images of each license plate region, and then entering step ii3;
step ii3, taking each license plate region sample image as input, taking license plate numbers in each license plate region sample image as output, and training for a license plate content recognition initial model to obtain a license plate content recognition model;
step A, obtaining a target scene image, applying a license plate region image detection model, detecting and obtaining a target license plate region image in the target scene image and four corner coordinates of the target license plate region image, and then entering the step B;
b, correcting the target license plate region image to a rectangular shape according to the coordinates of four corner points of the target license plate region image, updating the target license plate region image, and then entering the step C;
and C, identifying the target license plate region image by using a license plate content identification model to obtain a license plate number corresponding to the target license plate region image.
2. The method for identifying the small target license plate in the unconstrained scene according to claim 1, wherein the method comprises the following steps: the step i1 comprises the following steps i1-1 to i1-2;
step i1-1, collecting scene sample images containing license plates of specified types, obtaining license plate region images respectively contained in the scene sample images, obtaining four corner coordinates of the license plate region images, and then entering step i1-2; step i1-2, respectively aiming at each scene sample image, adopting a specified image deformation method to obtain each transformed scene sample image corresponding to the scene sample image, establishing the corresponding relation between each transformed scene sample image and four corner coordinates of a license plate region image and a license plate region image in the scene sample image, and taking each transformed scene sample image as each scene sample image; step i2 is then entered.
3. The method for identifying the small target license plate in the unconstrained scene according to claim 2, wherein the method comprises the following steps of: the specified image deformation methods include an illumination distortion method that performs brightness variation, contrast variation, saturation variation, and noise variation, a geometric distortion method that performs random scaling, cropping, rotation, and a joint method that performs illumination distortion and geometric distortion.
4. The method for identifying the small target license plate in the unconstrained scene according to claim 1, wherein the method comprises the following steps: in the step i2, aiming at a minimum rectangular frame containing a license plate area image, training is performed according to a confidence loss function containing the license plate area image in the minimum rectangular frame, a regression loss function of the minimum rectangular frame and detection loss functions of four corner coordinates of the license plate area image, and a preset overflow condition of each loss function, wherein each scene sample image is taken as an input, each license plate area image in each scene sample image and four corner coordinates of the license plate area image are taken as an output, and a depth convolution network YoloV5 serving as a first appointed neural network is used for obtaining a license plate area image detection model;
regression loss function of minimum rectangular frame of license plate region image:
in the formula, CIoU Loss The detection loss of the minimum rectangular frame including the license plate region image is represented by IoU, the intersection ratio of the minimum rectangular frame to the real minimum rectangular frame is represented by v, the distance of the aspect ratio of the minimum rectangular frame to the real minimum rectangular frame is represented by α, the weight coefficient is represented by ρ, the Euclidean distance between the center points of the minimum rectangular frame and the real minimum rectangular frame is represented by b and bb gt Respectively representing the center points of the detection minimum rectangular frame and the real minimum rectangular frame, and c represents the length of the diagonal line of the circumscribed rectangle of the detection minimum rectangular frame and the real minimum rectangular frame;
wherein w represents the width of the minimum rectangular frame, h represents the height of the minimum rectangular frame, and w gt Represents the width of a real minimum rectangular frame, h gt Representing the height of a real minimum rectangular frame;
detection loss function of four corner coordinates of license plate region image:
wherein x represents the distance between the diagonal intersection point of the quadrangle formed by the four corner coordinates obtained by detection and the diagonal intersection point of the quadrangle formed by the real four corner coordinates,the detection loss function corresponding to x is shown.
5. The method for identifying the small target license plate in the unconstrained scene according to claim 1, wherein the method comprises the following steps: the method further comprises the following steps of obtaining a license plate classification model;
collecting various license plate region classification sample images of various types, taking the license plate region classification sample images as input, taking the types corresponding to the license plate region classification sample images as output, and training a residual error network ResNet18 to obtain a license plate classification model;
the license plate content recognition model is obtained in the process of:
step ii2, collecting various license plate region sample images designated in the step ii1, obtaining license plate numbers in the license plate region sample images, and then entering the step ii3;
step i3, respectively aiming at various types, taking sample images of various license plate areas in the types as input, taking license plate numbers in the sample images of various license plate areas as output, training for an initial license plate content recognition model to obtain license plate content recognition models corresponding to the types, and further obtaining license plate content recognition models corresponding to the various types respectively;
the license plate content recognition process in the scene further comprises the following step BC, and after the step B is executed, the step BC is entered; step BC, applying a license plate classification model, classifying types of the license plate region images to obtain target types corresponding to the license plate region images, and then entering the step C;
and C, applying a license plate content recognition model corresponding to the target type, and recognizing the target license plate region image to obtain a license plate number corresponding to the target license plate region image.
6. The method for identifying a small target license plate in an unconstrained scene according to claim 5, wherein in the obtaining process of the license plate classification model:
collecting and designating various license plate region classification sample images, and respectively applying random perspective transformation, contrast variation, brightness variation and saturation variation to the license plate region classification sample images to obtain various transformed license plate region classification sample images corresponding to the license plate region classification sample images, wherein the transformed license plate region classification sample images are used as license plate region classification sample images; and then taking each license plate region classification sample image as input, taking the type corresponding to each license plate region classification sample image as output, and training the residual error network ResNet18 to obtain a license plate classification model.
7. The method for identifying the small target license plate in the unconstrained scene according to claim 1, wherein the method comprises the following steps: in the step B, according to four corner coordinates of the target license plate area image, new coordinates corresponding to the four corner points under the corresponding rectangular shape are obtained by applying opencv calculation, then an affine transformation matrix is obtained by applying opencv calculation based on the relation between the four corner point coordinates and the new coordinates, finally the target license plate area image is corrected to the rectangular shape according to the affine transformation matrix, the target license plate area image is updated, and then the step C is entered.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110300861.1A CN113052170B (en) | 2021-03-22 | 2021-03-22 | Small target license plate recognition method under unconstrained scene |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110300861.1A CN113052170B (en) | 2021-03-22 | 2021-03-22 | Small target license plate recognition method under unconstrained scene |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113052170A CN113052170A (en) | 2021-06-29 |
| CN113052170B true CN113052170B (en) | 2023-12-26 |
Family
ID=76513984
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110300861.1A Active CN113052170B (en) | 2021-03-22 | 2021-03-22 | Small target license plate recognition method under unconstrained scene |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113052170B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113989794B (en) * | 2021-11-12 | 2022-08-02 | 珠海安联锐视科技股份有限公司 | License plate detection and recognition method |
| CN113822278B (en) * | 2021-11-22 | 2022-02-11 | 松立控股集团股份有限公司 | License plate recognition method for unlimited scene |
| CN115100506B (en) * | 2022-06-27 | 2024-05-24 | 平安银行股份有限公司 | Method, server and system for correcting images of screen in double-recording scene |
| CN116246209B (en) * | 2023-03-09 | 2024-02-13 | 彩虹鱼科技(广东)有限公司 | Wide-angle lens biological target detection method based on offset convolution kernel |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018112900A1 (en) * | 2016-12-23 | 2018-06-28 | 深圳先进技术研究院 | License plate recognition method and apparatus, and user equipment |
| CN109086722A (en) * | 2018-08-06 | 2018-12-25 | 汉王科技股份有限公司 | Mix licence plate recognition method, device, electronic equipment |
| CN110427937A (en) * | 2019-07-18 | 2019-11-08 | 浙江大学 | A kind of correction of inclination license plate and random length licence plate recognition method based on deep learning |
| CN111767944A (en) * | 2020-05-27 | 2020-10-13 | 重庆大学 | Deep learning-based single-stage detector design method suitable for multi-scale target detection |
| CN112232240A (en) * | 2020-10-21 | 2021-01-15 | 南京师范大学 | Road sprinkled object detection and identification method based on optimized intersection-to-parallel ratio function |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107704857B (en) * | 2017-09-25 | 2020-07-24 | 北京邮电大学 | End-to-end lightweight license plate recognition method and device |
-
2021
- 2021-03-22 CN CN202110300861.1A patent/CN113052170B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018112900A1 (en) * | 2016-12-23 | 2018-06-28 | 深圳先进技术研究院 | License plate recognition method and apparatus, and user equipment |
| CN109086722A (en) * | 2018-08-06 | 2018-12-25 | 汉王科技股份有限公司 | Mix licence plate recognition method, device, electronic equipment |
| CN110427937A (en) * | 2019-07-18 | 2019-11-08 | 浙江大学 | A kind of correction of inclination license plate and random length licence plate recognition method based on deep learning |
| CN111767944A (en) * | 2020-05-27 | 2020-10-13 | 重庆大学 | Deep learning-based single-stage detector design method suitable for multi-scale target detection |
| CN112232240A (en) * | 2020-10-21 | 2021-01-15 | 南京师范大学 | Road sprinkled object detection and identification method based on optimized intersection-to-parallel ratio function |
Non-Patent Citations (1)
| Title |
|---|
| 基于区域划分和霍夫变换的车牌倾斜校正;王越等;重庆理工大学学报( 自然科学);第31卷(第3期);第97-104页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113052170A (en) | 2021-06-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113052170B (en) | Small target license plate recognition method under unconstrained scene | |
| CN110414507B (en) | License plate recognition method and device, computer equipment and storage medium | |
| CN112686812B (en) | Bank card inclination correction detection method and device, readable storage medium and terminal | |
| US10198657B2 (en) | All-weather thermal-image pedestrian detection method | |
| CN113255659B (en) | License plate correction detection and identification method based on MSAFF-yolk 3 | |
| CN111709416A (en) | License plate positioning method, device and system and storage medium | |
| CN106971185B (en) | License plate positioning method and device based on full convolution network | |
| CN107273832B (en) | License plate recognition method and system based on integral channel characteristics and convolutional neural network | |
| CN110766020A (en) | System and method for detecting and identifying multi-language natural scene text | |
| WO2023082784A1 (en) | Person re-identification method and apparatus based on local feature attention | |
| CN112287941B (en) | License plate recognition method based on automatic character region perception | |
| CN110781882A (en) | License plate positioning and identifying method based on YOLO model | |
| CN110378837A (en) | Object detection method, device and storage medium based on fish-eye camera | |
| CN114429457A (en) | Intelligent fan blade defect detection method based on bimodal fusion | |
| CN103353941B (en) | Natural marker registration method based on viewpoint classification | |
| CN113989794B (en) | License plate detection and recognition method | |
| CN111695373A (en) | Zebra crossing positioning method, system, medium and device | |
| CN116740652B (en) | Method and system for monitoring rust area expansion based on neural network model | |
| CN112347805A (en) | Multi-target two-dimensional code detection and identification method, system, device and storage medium | |
| CN112784834A (en) | Automatic license plate identification method in natural scene | |
| CN116503622A (en) | Data acquisition and reading method based on computer vision image | |
| CN113033482A (en) | Traffic sign detection method based on regional attention | |
| CN113361467A (en) | License plate recognition method based on field adaptation | |
| CN112686247A (en) | Identification card number detection method and device, readable storage medium and terminal | |
| CN114550134A (en) | Deep learning-based traffic sign detection and identification method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |