CN108596277B

CN108596277B - Vehicle identity recognition method and device and storage medium

Info

Publication number: CN108596277B
Application number: CN201810444371.7A
Authority: CN
Inventors: 彭湃; 张有才; 余宗桥; 郭晓威
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2018-05-10
Filing date: 2018-05-10
Publication date: 2020-07-07
Anticipated expiration: 2038-05-10
Also published as: CN108596277A

Abstract

The embodiment of the invention discloses a vehicle identity recognition method, a device and a storage medium; when the vehicle image to be identified needs to be identified, at least one reference vehicle image can be obtained, and then the similarity between the vehicle image to be identified and the reference vehicle image is calculated to obtain the global similarity; extracting image blocks of the area where the preset marker is located from the vehicle image to be recognized and the reference vehicle image respectively to obtain a local image to be recognized and a reference local image, calculating the similarity between the local image to be recognized and the reference local image according to a preset twin neural network model to obtain a local similarity, and then obtaining identity information corresponding to the reference vehicle image, of which the global similarity and the local similarity meet a preset first condition, as identity information of the vehicle to be recognized; the scheme can greatly improve the effectiveness and accuracy of identification.

Description

Vehicle identity recognition method and device and storage medium

Technical Field

The invention relates to the technical field of communication, in particular to a vehicle identity identification method, a vehicle identity identification device and a storage medium.

Background

In recent years, with the development of economic technology, the number of vehicles has also increased greatly, and meanwhile, various illegal cases related to the vehicles have also increased year by year, and the vehicles are accurately identified based on the real-name registration characteristics of the vehicles, so that the system has positive significance for reconnaissance of the cases and guarantee of social security.

In the prior art, the identity of a vehicle is generally determined by recognizing license plate information, for example, a vehicle image may be specifically acquired, license plate information is extracted from the vehicle image and recognized, and then the identity of the vehicle is determined based on a recognition result, and the like. However, in the research and practice processes of the prior art, the inventor of the present invention finds that in some scenes, the license plate information may also have the situations of counterfeiting, missing, blurring, or being difficult to identify, for example, most of the existing suspected vehicles often adopt a fake plate method to hide the real identity of the vehicle, and the like, so that the effectiveness and accuracy of the identification of the existing scheme are not ideal.

Disclosure of Invention

The embodiment of the invention provides a vehicle identity recognition method, a vehicle identity recognition device and a storage medium, which can improve the effectiveness and accuracy of recognition.

The embodiment of the invention provides a vehicle identity recognition method, which comprises the following steps:

acquiring a vehicle image to be identified and at least one reference vehicle image;

calculating the similarity of the vehicle image to be identified and the reference vehicle image to obtain the global similarity;

extracting image blocks of the area where the preset marker is located from the vehicle image to be identified and the reference vehicle image respectively to obtain a local image to be identified and a reference local image;

calculating the similarity of the local image to be recognized and the reference local image according to a preset twin neural network model to obtain the local similarity;

and acquiring identity information corresponding to the reference vehicle image with the global similarity and the local similarity meeting a preset first condition as identity information of the vehicle to be identified.

Correspondingly, the embodiment of the invention also provides a vehicle identity recognition device, which comprises:

the device comprises an acquisition unit, a recognition unit and a processing unit, wherein the acquisition unit is used for acquiring a vehicle image to be recognized and at least one reference vehicle image;

the global calculation unit is used for calculating the similarity between the vehicle image to be identified and the reference vehicle image to obtain the global similarity;

the extraction unit is used for extracting image blocks of areas where preset markers are located from the vehicle image to be identified and the reference vehicle image respectively to obtain a local image to be identified and a reference local image;

the local calculation unit is used for calculating the similarity between the local image to be recognized and the reference local image according to a preset twin neural network model to obtain the local similarity;

and the identification unit is used for acquiring the identity information corresponding to the reference vehicle image with the global similarity and the local similarity meeting the preset first condition, and the identity information is used as the identity information of the vehicle to be identified.

Optionally, in some embodiments, the identification unit may include an operation subunit and a determination subunit, as follows:

the operation subunit may be configured to perform weighted operation on the global similarity and the corresponding local similarity to obtain a comprehensive similarity;

the determining subunit may be configured to acquire identity information corresponding to a reference vehicle image whose comprehensive similarity satisfies a preset second condition, as the identity information of the vehicle to be identified.

Optionally, in some embodiments, the vehicle identification device may further include a setting unit, as follows:

the setting unit may be configured to acquire real identity information of a reference vehicle corresponding to each reference vehicle image, where the identity information includes license plate information and owner information, establish a mapping relationship between each reference vehicle image and the corresponding identity information, and store the mapping relationship;

at this time, the determining subunit may be specifically configured to use a reference vehicle image with a comprehensive similarity satisfying a preset second condition as a target vehicle image, and obtain, according to the mapping relationship, identity information corresponding to the target vehicle image, as identity information of the vehicle to be identified.

Optionally, in some embodiments, the extraction unit may be specifically configured to obtain preset marker information; determining first position information of a marker in the vehicle image to be identified according to the preset marker information, and intercepting an image block of an area where the preset marker is located from the vehicle image to be identified according to the first position information to obtain a local image to be identified; and determining second position information of the marker in the reference vehicle image according to the preset marker information, and intercepting an image block of an area where the preset marker is located from the reference vehicle image according to the second position information to obtain a reference local image.

Optionally, in some embodiments, the obtaining unit may be specifically configured to obtain a vehicle image to be identified, and obtain a candidate set, where the candidate set includes a plurality of reference vehicle images; matching the reference vehicle images in the candidate set with the vehicle images to be identified; filtering the reference vehicle images with the matching degree smaller than a set value to obtain a filtered candidate set; at least one reference vehicle image is obtained from the filtered candidate set.

Optionally, in some embodiments, the vehicle identification apparatus may further include a collecting unit, a combining unit, and a training unit, as follows:

the acquisition unit can be used for acquiring a plurality of vehicle sample images, and the vehicle sample images have real identity information;

the combination unit may be configured to combine two by two the plurality of vehicle sample images to establish a sample pair;

the merging unit may be configured to merge each sample pair into one multi-channel image, and add the multi-channel image to the training sample set;

the training unit can be used for training a preset initial twin model according to a training sample set to obtain a twin neural network model.

Optionally, in some embodiments, the sample pairs include a positive sample pair and a negative sample pair, and the combining unit may be specifically configured to select a vehicle sample image belonging to the same vehicle from the multiple vehicle sample images, and combine two by two the vehicle sample images belonging to the same vehicle to establish the positive sample pair; and selecting vehicle sample images which do not belong to the same vehicle from the plurality of vehicle sample images, and combining the vehicle sample images which do not belong to the same vehicle in pairs to establish a negative sample pair.

Optionally, in some embodiments, the merging unit may be specifically configured to determine color channels of the vehicle sample images in each sample pair, add the color channels to obtain a multi-channel image corresponding to each sample pair, and add the obtained multi-channel image to the training sample set.

Optionally, in some embodiments, the training unit may include a training subunit and a convergence subunit, as follows:

the training subunit may be configured to train an upper half branch network and a lower half branch network of a preset initial twin model according to the training sample set, respectively, to obtain a similarity prediction value of a sample pair corresponding to each multi-channel image in the training sample set;

the convergence subunit may be configured to obtain a true similarity value of each sample pair, and converge the true similarity value and the predicted similarity value to obtain a twin neural network model.

Optionally, in some embodiments, the training subunit may be specifically configured to select one multi-channel image from the training sample set as a current training sample; respectively introducing the current training sample into an upper half branch network and a lower half branch network of a preset initial twin model for training to obtain an output vector of the upper half branch network and an output vector of the lower half branch network; performing one-dimensional full-connection operation on the upper half branch network output vector and the lower half branch network output vector to obtain a similarity prediction value of a sample pair corresponding to the current training sample; and returning to the step of selecting one multi-channel image from the training sample set as the current training sample until all the multi-channel images in the training sample set are trained.

In addition, the embodiment of the present invention further provides a storage medium, where the storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to perform the steps in any one of the vehicle identification methods provided by the embodiments of the present invention.

According to the embodiment of the invention, when the vehicle image to be recognized needs to be recognized, at least one reference vehicle image can be obtained, and then on one hand, the similarity between the vehicle image to be recognized and the reference vehicle image is calculated to obtain the global similarity; on the other hand, extracting image blocks of the area where the preset marker is located from the vehicle image to be recognized and the reference vehicle image respectively to obtain a local image to be recognized and a reference local image, calculating the similarity between the local image to be recognized and the reference local image according to a preset twin neural network model to obtain a local similarity, and then obtaining identity information corresponding to the reference vehicle image, of which the global similarity and the local similarity meet a preset first condition, as the identity information of the vehicle to be recognized; according to the scheme, the vehicle which is most similar to the vehicle to be recognized and has the real identity information can be automatically matched by calculating the global similarity and the local similarity, and the identity of the vehicle to be recognized is recognized, so that compared with the existing scheme that the identity recognition can only be carried out through the license plate information, the situation that the vehicle cannot be recognized or the recognition is wrong due to the fact that the license plate information is fake, lost or fuzzy and the like can be avoided, the dependency on the license plate information of the vehicle to be recognized can be reduced, and the effectiveness and the accuracy of the recognition are greatly improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1a is a schematic view of a scene of a vehicle identification method according to an embodiment of the present invention;

FIG. 1b is a flow chart of a vehicle identification method according to an embodiment of the present invention;

FIG. 1c is a schematic structural diagram of a twin neural network model provided by an embodiment of the present invention;

fig. 2a is a training flow architecture diagram of CNN provided in an embodiment of the present invention;

FIG. 2b is a schematic diagram illustrating the establishment of a training sample set in the vehicle identification method according to the embodiment of the present invention;

FIG. 2c is a diagram of a training process of a twin neural network model according to an embodiment of the present invention;

FIG. 2d is a schematic flow chart of a vehicle identification method according to an embodiment of the present invention;

fig. 2e is a schematic diagram of extracting local features of a vehicle in the vehicle identity recognition method according to the embodiment of the present invention;

FIG. 2f is a diagram illustrating the architecture of a twin neural network model according to an embodiment of the present invention;

FIG. 3a is a schematic structural diagram of a vehicle identification device according to an embodiment of the present invention;

FIG. 3b is a schematic structural diagram of a vehicle identification device according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram of a network device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The embodiment of the invention provides a vehicle identity recognition method, a vehicle identity recognition device and a storage medium.

The vehicle identification device may be specifically integrated in a network device, such as a terminal or a server.

For example, referring to fig. 1a, when a user needs to identify the identity of a vehicle to be identified, an image of the vehicle to be identified (i.e., a vehicle image to be identified) may be provided to a network device, and at the same time, the network device may obtain at least one reference vehicle image, and then calculate the similarity between the vehicle image to be identified and the reference vehicle image to obtain a global similarity, for example, a convolutional neural network model may be used to calculate the similarity between the vehicle image to be identified and the reference vehicle image to obtain the global similarity, and the like; and extracting image blocks of the area where the preset marker is located from the vehicle image to be recognized and the reference vehicle image respectively to obtain a local image to be recognized and a reference local image, calculating the similarity between the local image to be recognized and the reference local image according to a preset twin neural network model to obtain a local similarity, and then obtaining identity information corresponding to the reference vehicle image, wherein the global similarity and the local similarity meet a preset first condition, and the identity information is used as identity information of the vehicle to be recognized, so that the purpose of recognizing the identity of the vehicle is achieved.

The following are detailed below. The order of the following examples is not intended to limit the preferred order of the examples.

The first embodiment,

In the present embodiment, the vehicle identification apparatus will be described from the perspective of a vehicle identification apparatus, which may be specifically integrated in a network device, such as a terminal or a server.

The embodiment of the invention provides a vehicle identity recognition method, which comprises the following steps: the method comprises the steps of obtaining a vehicle image to be recognized and at least one reference vehicle image, calculating the similarity between the vehicle image to be recognized and the reference vehicle image to obtain global similarity, extracting image blocks of areas where preset markers are located from the vehicle image to be recognized and the reference vehicle image respectively to obtain a local image to be recognized and a reference local image, calculating the similarity between the local image to be recognized and the reference local image according to a preset twin neural network model to obtain local similarity, and obtaining identity information corresponding to the reference vehicle image, wherein the global similarity and the local similarity meet preset first conditions, and the identity information serves as identity information of a vehicle to be recognized.

As shown in fig. 1b, the specific process of the vehicle identification method may be as follows:

101. and acquiring a vehicle image to be identified and at least one reference vehicle image.

In the embodiment of the present invention, the reference vehicle mainly refers to a vehicle whose owner's real identity has been confirmed, such as a vehicle whose license plate information shows a normal status, and the vehicle to be identified mainly refers to a vehicle whose identity needs to be identified, such as a vehicle whose owner's identity has not been confirmed, for example, a vehicle whose license plate information does not show a normal status, etc.

The reference vehicle image refers to an image containing a reference vehicle, and the vehicle image to be recognized refers to an image containing a vehicle to be recognized. The reference vehicle image may be an image including the entire reference vehicle or an image including a preset local area of the reference vehicle, and similarly, the vehicle image to be recognized may be an image including the entire vehicle to be recognized or an image including the preset local area of the vehicle to be recognized; the local area may be an area where a specific object on the vehicle is located, where the specific object needs to have distinctive individual features, such as an annual inspection mark attached to a window glass, a pendant and a decoration in the vehicle, and in the embodiment of the present invention, the specific object is referred to as a "preset mark", and mainly refers to the annual inspection mark. The annual inspection mark is a proof of pass obtained when the vehicle passes through the relevant department within a predetermined period, and the next annual inspection time is indicated on the annual inspection mark. Generally, the first annual inspection time of a vehicle depends on license plate receiving time, and then the vehicle needs to be inspected regularly, and the inspection period is different for different vehicle types, such as 1 inspection per year within 5 years of operating passenger cars, more than 5 inspection per year, and 1 inspection per 6 months. Cargo vehicles and large, medium-sized non-commercial passenger vehicles are inspected 1 time per year within 10 years, over 10 years, 1 time every 6 months, and so on, and the annual inspection time on the annual inspection marks of different vehicles is usually different.

The method for acquiring the vehicle image to be recognized and the at least one reference vehicle image may be various, for example, a vehicle identification request triggered by a user may be specifically received, where the vehicle identification request carries the vehicle image to be recognized, and then the at least one reference vehicle image is acquired according to the vehicle identification request.

The reference vehicle image may be obtained by shooting the reference vehicle, intercepting the reference vehicle from a surveillance video, or extracting the reference vehicle image from another gallery.

Optionally, in order to reduce subsequent calculation amount and improve processing efficiency, when obtaining the reference vehicle images, the reference vehicle images may be subjected to preliminary screening to filter out images that are obviously inconsistent with the vehicle to be identified, that is, the step "obtaining at least one reference vehicle image" may include:

the method comprises the steps of obtaining a candidate set, wherein the candidate set comprises a plurality of reference vehicle images, matching the reference vehicle images in the candidate set with the vehicle images to be identified, filtering the reference vehicle images with the matching degree smaller than a set value to obtain a filtered candidate set, and obtaining at least one reference vehicle image from the filtered candidate set.

The matching mode may be set according to the requirements of practical applications, for example, the information of the hanging decoration, the interior decoration, the front side of the vehicle, and/or the back side of the vehicle may be compared, and the obtained similarity may be used as the matching degree. The information such as the hanging decoration and the interior decoration in the vehicle can be obtained through a detection means, the front side and the back side of the vehicle can be obtained through detecting key points of the vehicle, and the specific detection modes can be various and are not described herein any more.

102. And calculating the similarity of the vehicle image to be identified and the reference vehicle image to obtain the global similarity.

The method for calculating the similarity between the vehicle image to be recognized and the reference vehicle image may be various, for example, a common Convolutional Neural Network (CNN) may be used for calculation, that is, the step "calculating the similarity between the vehicle image to be recognized and the reference vehicle image to obtain the global similarity" may include:

and calculating the similarity between the vehicle image to be identified and the reference vehicle image by adopting a preset CNN (CNN) to obtain the global similarity.

Alternatively, another twin neural network model may be used to calculate the global similarity. Namely, the step of "calculating the similarity between the image of the vehicle to be recognized and the image of the reference vehicle to obtain the global similarity" may include:

and calculating the similarity between the vehicle image to be identified and the reference vehicle image by adopting a preset twin neural network model to obtain the global similarity.

The training method of the other twin neural network model is similar to the twin neural network model for calculating the local similarity provided by the embodiment of the invention, for example, a large number of vehicle whole images can be collected as vehicle sample images, then the multiple vehicle sample images are combined pairwise to establish a sample pair, for example, the vehicle sample images belonging to the same vehicle are used as a positive sample pair, the vehicle sample images belonging to different vehicles are used as a negative sample pair, then the preset initial twin model is trained by using the positive sample pair and the negative sample pair to obtain the twin neural network model for calculating the global similarity, then the vehicle image to be identified and the reference vehicle image can be used as an image pair (namely, the image combination is similar to the sample pair), and the image pair is input into the twin neural network model for calculating the global similarity, to calculate the global similarity between the to-be-recognized vehicle image and the reference vehicle image, which will be described in detail later and will not be described herein.

103. And extracting image blocks of the areas where the preset markers are located from the vehicle image to be identified and the reference vehicle image respectively to obtain a local image to be identified and a reference local image.

For example, specifically, preset marker information may be acquired, first position information of a marker in the vehicle image to be recognized is determined according to the preset marker information, and an image block of an area where the preset marker is located is intercepted from the vehicle image to be recognized according to the first position information, so as to obtain a local image to be recognized; and determining second position information of the marker in the reference vehicle image according to the preset marker information, and intercepting an image block of an area where the preset marker is located from the reference vehicle image according to the second position information to obtain a reference local image.

The preset mark can be determined according to the requirements of practical applications, and generally needs to have distinctive individual features, such as an annual inspection mark attached to a window glass, a pendant in a vehicle, decoration, and the like.

The first position information and the second position information may specifically be coordinate information.

104. And calculating the similarity of the local image to be recognized and the reference local image according to a preset twin neural network model to obtain the local similarity.

For example, the local image to be identified and the reference local image may be combined in pairs to obtain a plurality of image pairs, the local image to be identified and the reference local image in each image pair are combined into a multi-channel image, for example, an image with 6 channels, then the multi-channel image is input into the upper half branch network and the lower half branch network of the twin neural network model to obtain an output vector of the upper half branch network and an output vector of the lower half branch network, and then a manhattan distance between the output vector of the upper half branch network and the output vector of the lower half branch network is calculated, and a one-dimensional full-connection operation is performed according to the calculated manhattan distance to obtain a similarity of the image pair corresponding to the multi-channel image, where the similarity is a similarity of the local image to be identified and the reference local image, that is, a local similarity.

Optionally, when the multi-channel image is input into the lower half-branch network, preprocessing may be performed, for example, performing operations such as cropping, down-sampling and/or rotation to obtain a data-enhanced smaller-scale multi-channel image, such as a smaller-scale image, and then, the smaller-scale multi-channel image, such as the smaller-scale image, is introduced into the lower half-branch network of the twin neural network model for calculation. That is, the upper half of the branch network can process the multi-channel image of the original scale, and the lower half of the branch network can process the multi-channel image of the smaller scale.

The twin neural network model may be pre-established by an operation and maintenance person, or may be automatically established by the vehicle identity recognition device, that is, before the step "calculating the similarity between the local image to be recognized and the reference local image according to the pre-established twin neural network model to obtain the local similarity", the vehicle identity recognition device may further include:

(1) a plurality of vehicle sample images are collected, and the vehicle sample images have real identity information, such as normal license plate information, real owner information and the like.

For example, a plurality of vehicle sample images may be collected by taking images of a large number of vehicles, taking a plurality of images of the same vehicle, and the like; alternatively, the plurality of vehicle sample images may be obtained by searching the internet or from a vehicle picture database, and so on.

The plurality of vehicle sample images include images of a plurality of different vehicles and also include different images of the same vehicle (for example, images photographed at different places, different times or different angles), and the images may be an entire image of the vehicle or an image of a local area of the vehicle. The local area may be an area where a predetermined marker is located on the vehicle, and the marker needs to have a distinctive individual feature, such as an annual inspection mark attached to a window glass, a pendant and a decoration in the vehicle, and in the embodiment of the present invention, the local area mainly refers to the annual inspection mark.

(2) And combining the plurality of vehicle sample images pairwise to establish a sample pair.

The sample pair refers to a set formed by combining two vehicle sample images, and the sample pair may be a positive sample pair or a negative sample pair, the positive sample pair refers to vehicle sample images belonging to the same vehicle, such as two images obtained by shooting an annual inspection mark of the same vehicle, and the negative sample pair refers to vehicle sample images belonging to different vehicles, such as two images obtained by shooting an annual inspection mark of different vehicles, and so on.

If the sample pair includes a positive sample pair and a negative sample pair, the step of combining two by two the plurality of vehicle sample images to establish the sample pair may include:

selecting vehicle sample images belonging to the same vehicle from the plurality of vehicle sample images, and combining the vehicle sample images belonging to the same vehicle in pairs to establish a positive sample pair; and selecting the vehicle sample images which do not belong to the same vehicle from the plurality of vehicle sample images, and combining the vehicle sample images which do not belong to the same vehicle in pairs to establish a negative sample pair.

(3) Combining each sample pair into a multi-channel image, and adding the multi-channel image to a training sample set; for example, the following may be specifically mentioned:

and determining color channels of the vehicle sample images in each sample pair, adding the color channels to obtain a multi-channel image corresponding to each sample pair, and adding the obtained multi-channel image to a training sample set.

For example, if each sample pair includes vehicle sample images a and B, where the color channels of the vehicle sample images a and B are 3 channels, namely, Red channel (R, Red), Green channel (G, Green), and Blue channel (B, Blue), the vehicle sample images a and B may be merged into one 6-channel image, namely, into one image including two Red channels, two Green channels, and two Blue channels, and then the 6-channel image is added to the training sample set.

Because each sample pair is combined into a multi-channel image, the calculation amount and the required calculation resources are greatly reduced when the model is trained subsequently, and the efficiency of training the model can be improved.

(4) And training a preset initial twin model according to the training sample set to obtain a twin neural network model.

For example, as shown in fig. 1c, the preset initial twin model may include an upper half-branch network and a lower half-branch network, where the upper half-branch network and the lower half-branch network have the same structure but do not share a weight.

Taking the structure as a convolutional neural network as an example, as shown in fig. 1c, the structure may include four convolutional Layers (Convolution) and one Fully Connected layer (FC), as follows:

and (3) rolling layers: the method is mainly used for feature extraction of an input image (such as a training sample or an image to be identified), wherein the size of a convolution kernel can be determined according to practical application, for example, the sizes of convolution kernels from a first layer of convolution layer to a fourth layer of convolution layer can be (7, 7), (5, 5), (3, 3), (3, 3); optionally, in order to reduce the complexity of the calculation and improve the calculation efficiency, the sizes of the convolution kernels of the four convolution layers may also be set to be (3, 3); optionally, in order to improve the expression capability of the model, a non-Linear factor may be added by adding an activation function, in the embodiment of the present invention, the activation functions are all "relu (Linear rectification function)", and padding (which refers to a space between an attribute definition element border and element content) is all "same", and a "same" padding manner may be simply understood as padding an edge with 0, where the number of left (upper) padding 0 is the same as or less than the number of right (lower) padding 0; optionally, in order to further reduce the amount of computation, downsampling (downsampling) may be performed on all layers or any 1 to 2 layers of the second to fourth convolutional layers, where the downsampling operation is substantially the same as the convolution operation, except that the downsampling convolution kernel is only a maximum value (max) or an average value (average) of corresponding positions, and for convenience of description, in the embodiment of the present invention, the downsampling operation is performed on the second convolutional layer and the third convolutional layer, and specifically, the downsampling operation is specifically max _ po _ ing.

It should be noted that, for convenience of description, in the embodiment of the present invention, both the layer where the activation function is located and the down-sampling layer (also referred to as a pooling layer) are included in the convolution layer, and it should be understood that the structure may also be considered to include the convolution layer, the layer where the activation function is located, the down-sampling layer (i.e., a pooling layer), and a full-connection layer, and of course, may also include an input layer for inputting data and an output layer for outputting data, which are not described herein again.

Full connection layer: the learned features may be mapped to a sample label space, which mainly functions as a "classifier" in the whole convolutional neural network, and each node of the fully-connected layer is connected to all nodes output by the previous layer (e.g., the down-sampling layer in the convolutional layer), where one node of the fully-connected layer is referred to as one neuron in the fully-connected layer, and the number of neurons in the fully-connected layer may be determined according to the requirements of the practical application, for example, in the upper half branch network and the lower half branch network of the twin neural network model, the number of neurons in the fully-connected layer may be set to 512 each, or may be set to 128 each, and so on. Similar to the convolutional layer, optionally, in the fully-connected layer, a non-linear factor may be added by adding an activation function, for example, an activation function sigmoid (sigmoid function) may be added.

Because the upper half branch network and the lower half branch network of the initial twin model can both output a plurality of vectors, and the number of the vectors is consistent with the number of the neurons, for example, if the number of the neurons of the full connection layer of the upper half branch network and the lower half branch network is set to 512, the upper half branch network and the lower half branch network can respectively output 512 vectors; for another example, if the numbers of neurons in the full connection layers of the upper half branch network and the lower half branch network are both set to be 128, the upper half branch network and the lower half branch network may output 128 vectors, and so on, and thus, as shown in fig. 1c, a full connection layer with one dimension may be further set, so as to perform a full connection operation with one dimension on the output vectors of the upper half branch network and the output vectors of the lower half branch network (i.e., map the output vectors into one-dimensional data through full connection), and obtain the similarity corresponding to the input image, such as the similarity between a sample pair corresponding to a certain training sample, and so on.

Based on the structure of the preset initial twin model, the step of training the preset initial twin model according to the training sample set to obtain the twin neural network model may specifically be as follows:

and S1, training the upper half branch network and the lower half branch network of a preset initial twin model respectively according to the training sample set to obtain a similarity prediction value of a sample pair corresponding to each multi-channel image in the training sample set.

For example, a multi-channel image may be selected from the training sample set as a current training sample (i.e., the current training sample is a multi-channel image corresponding to a sample pair, i.e., the multi-channel image corresponds to two vehicle sample images), the current training sample is respectively introduced into the upper half branch network and the lower half branch network of the preset initial twin model for training to obtain an output vector of the upper half branch network and an output vector of the lower half branch network, and performing one-dimensional full-connection operation on the upper half branch network output vector and the lower half branch network output vector to obtain a similarity prediction value of a sample pair corresponding to the current training sample, and returning to execute the step of selecting one multi-channel image from the training sample set as the current training sample until all the multi-channel images in the training sample set are trained.

For example, the current training sample may be specifically led into an upper half branch network of a preset twin neural network model for training, so as to obtain an upper half branch network output vector, and the current training sample is subjected to preset processing, and the processed current training sample is led into a lower half branch network of the preset twin neural network model for training, so as to obtain a lower half branch network output vector, where the preset processing may be determined according to requirements of practical application, for example, the current training sample may be subjected to operations such as clipping, down-sampling, and/or rotation, so as to obtain a data-enhanced current training sample with a smaller scale, that is, the upper half branch network may process a training sample with an original scale, and the lower half branch network may process a training sample with a smaller scale. And then, calculating the Manhattan distance between the output vector of the upper half branch network and the output vector of the lower half branch network, and performing one-dimensional full-connection operation according to the calculated Manhattan distance to obtain a similarity prediction value of a sample pair corresponding to the current training sample.

Optionally, the one-dimensional full-connection operation result may be processed by using an activation function, that is, after the output vector of the upper half branch network and the output vector of the lower half branch network are obtained, a manhattan distance (L) between the output vector of the upper half branch network and the output vector of the lower half branch network may be calculated₁Distance), performing one-dimensional full-connection operation according to the calculated Manhattan distance, and calculating the result of the one-dimensional full-connection operation by adopting a preset activation function to obtain a similarity prediction value of a sample pair corresponding to the current training sample.

The preset activation function may be determined according to requirements of actual applications, and may specifically be a sigmoid, for example.

And S2, acquiring the true similarity value of each sample pair, and converging the true similarity value and the predicted similarity value to obtain the twin neural network model.

For example, the true similarity value and the predicted similarity value may be converged by using a preset loss function to obtain a twin neural network model.

The loss function can be flexibly set according to the actual application requirement, for example, the loss function J can be selected as the cross entropy as follows:

wherein, C is the category number, C is 2, k ∈ (1,2), different values of k represent whether belong to the same vehicle,

for the output similarity prediction value, y_kThe true similarity value is shown. And continuously training by reducing the error between the predicted value of the network similarity and the true value of the similarity so as to adjust the weight to a proper value, thereby obtaining the twin neural network model.

105. And acquiring identity information corresponding to the reference vehicle image with the global similarity and the local similarity meeting a preset first condition as identity information of the vehicle to be identified.

For example, after the global similarity and the local similarity are fused according to a certain proportion, a suitable reference vehicle image is screened as a target vehicle image based on the fusion result, that is, for example, the step "obtaining the identity information corresponding to the reference vehicle image of which the global similarity and the local similarity satisfy the preset first condition" as the identity information of the vehicle to be recognized "may specifically be as follows:

(1) and carrying out weighting operation on the global similarity and the corresponding local similarity to obtain the comprehensive similarity. For example, it can be formulated as follows:

sim＝(1-μ)sim_global+μsim_local；

wherein sim is the comprehensive similarity sim_globalFor global similarity, sim_localFor local similarity, μ is a weight, μ is in a range of (0, 1), and a specific value of μmay be determined according to requirements of practical applications, which is not described herein again.

(2) And acquiring identity information corresponding to the reference vehicle image with the comprehensive similarity meeting the preset second condition as the identity information of the vehicle to be recognized, namely, taking the reference vehicle image with the comprehensive similarity meeting the preset second condition as a target vehicle image, and taking the identity information corresponding to the target vehicle image as the identity information of the vehicle to be recognized.

The preset second condition may be "higher than a preset threshold", or "the first N highest in comprehensive similarity", values of the preset threshold and N may be determined according to requirements of practical applications, N is a positive integer, for example, if N is 10, the obtained multiple comprehensive similarities may be sorted, and then, the first 10 reference vehicle images with higher comprehensive similarities are selected as target vehicle images, and so on, which are not described herein again.

The identity information corresponding to the reference vehicle image may be obtained based on a preset mapping relationship (mapping relationship between the reference vehicle image and the actual identity information of the reference vehicle), that is, optionally, before obtaining the identity information corresponding to the reference vehicle image whose comprehensive similarity satisfies a preset second condition as the identity information of the vehicle to be recognized, the vehicle identity recognition method may further include:

acquiring real identity information of a reference vehicle corresponding to each reference vehicle image, wherein the identity information can comprise license plate information, owner information and the like; and establishing a mapping relation between each reference vehicle image and the corresponding identity information thereof, and storing the mapping relation.

At this time, the step "obtaining identity information corresponding to the reference vehicle image whose comprehensive similarity satisfies the preset second condition as the identity information of the vehicle to be recognized" may specifically be: and taking the reference vehicle image with the comprehensive similarity meeting the preset second condition as a target vehicle image, and acquiring the identity information corresponding to the target vehicle image according to the mapping relation to be used as the identity information of the vehicle to be identified.

As can be seen from the above, in this embodiment, when the vehicle image to be recognized needs to be recognized, at least one reference vehicle image may be obtained, and then, on the one hand, the similarity between the vehicle image to be recognized and the reference vehicle image is calculated to obtain the global similarity; on the other hand, extracting image blocks of the area where the preset marker is located from the vehicle image to be recognized and the reference vehicle image respectively to obtain a local image to be recognized and a reference local image, calculating the similarity between the local image to be recognized and the reference local image according to a preset twin neural network model to obtain a local similarity, and then obtaining identity information corresponding to the reference vehicle image, of which the global similarity and the local similarity meet a preset first condition, as the identity information of the vehicle to be recognized; according to the scheme, the vehicle which is most similar to the vehicle to be recognized and has the real identity information can be automatically matched by calculating the global similarity and the local similarity, and the identity of the vehicle to be recognized is recognized, so that compared with the existing scheme that the identity recognition can only be carried out through the license plate information, the situation that the vehicle cannot be recognized or the recognition is wrong due to the fact that the license plate information is fake, lost or fuzzy and the like can be avoided, the dependency on the license plate information of the vehicle to be recognized can be reduced, and the effectiveness and the accuracy of the recognition are greatly improved.

Example II,

The method according to the preceding embodiment is illustrated in further detail below by way of example.

In the present embodiment, the vehicle recognition apparatus is specifically integrated in a network device, and the global similarity between the vehicle image to be recognized and the reference vehicle image is calculated by the common CNN.

And (I) training a model.

For example, first, the network device may collect a large number of vehicle sample images through multiple paths, where the vehicle sample images may include images of multiple different vehicles and also include different images of the same vehicle, such as images obtained by shooting the same vehicle at different places, different times, or different angles, where the images may be an entire image of the vehicle or a local area of the vehicle, and if the collected images are the entire image of the vehicle, the images of the local area of the vehicle may be obtained by cropping, for example, a local image of an area where an annual inspection mark is located (usually in the upper right corner of a front windshield) may be extracted from the images, and so on.

Secondly, the general CNN can be trained by using the overall images of the vehicles (for performing global similarity calculation), and the twin neural network model can be trained by using the images of the local regions of the vehicles (for performing local similarity calculation), which can be specifically as follows:

(1) training of common CNN;

after the network device screens a plurality of vehicle sample images containing the vehicle as a whole, for example, after removing the weight or some vehicle sample images showing blurriness, the network device adds the remaining vehicle sample images to the training sample set of the CNN, and then the network device can train a preset initial model according to the training sample set to obtain the CNN.

The predetermined initial model (initial CNN) may include four convolutional layers and one full link layer. In order to reduce the complexity of the calculation and improve the calculation efficiency, in this embodiment, the sizes of the convolution kernels of the four convolution layers may be all set to (3, 3), the activation functions all adopt "relu", and the padding ways all set to "same".

Optionally, in order to further reduce the amount of computation, downsampling operations, such as maxporoling, may also be performed in the second convolutional layer and the third convolutional layer.

After the maxporoling operation is performed, the output after the maxporoling operation can be mapped through the fully connected layers, the number of neurons of the fully connected layers can be set to 512 (or to 128, etc.), and sigmoid can be adopted as the activation function.

When model training is needed, the network equipment can select two vehicle sample images from the training sample set as a current training sample; then, as shown in fig. 2a, the current training sample is imported into the initial mode to obtain a similarity prediction value corresponding to the current training sample, the similarity true value of the current training sample is obtained, the similarity true value and the similarity prediction value are converged by using a preset loss function to adjust each parameter in the twin neural network model to a proper value, then, the step of selecting two vehicle sample images from the training sample set as the current training sample can be performed in a returning manner to calculate and converge the similarity prediction values between other vehicle sample images in the training sample set until all the vehicle sample images in the training sample set are calculated and converged, and the required CNN can be obtained.

The loss function may be determined according to the requirements of the actual application, and is not limited herein.

(2) Training a twin neural network model;

the network device may combine two by two a plurality of vehicle sample images including the vehicle local area to establish a sample pair, for example, the vehicle sample images (including the vehicle local area) belonging to the same vehicle may be used as a positive sample pair, the vehicle sample images (including the vehicle local area) belonging to different vehicles may be used as a negative sample pair, then, a color channel of the vehicle sample image in each sample pair is determined, the color channels are added to obtain a multi-channel image corresponding to each sample pair, and the obtained multi-channel image is added to a training sample set of the twin neural network model.

For example, referring to fig. 2B, if the vehicle sample image a1, the vehicle sample image a2, the vehicle sample image A3, and the like are different images of the vehicle a, the vehicle sample images B1, … …, and the vehicle sample image Bn are different images of the vehicle B, the vehicle sample image C is an image of the vehicle C, and the vehicle sample images are all 3-channel (color channel is RGB) images, the network device may combine and merge the vehicle sample images as follows:

combining the vehicle sample image A1 and the vehicle sample image A2 to serve as a positive sample pair, merging the positive sample pair into a multichannel image 1 of 6 channels (two red channels, two green channels and two blue channels), and adding the obtained multichannel image 1 to a training sample set of the twin neural network model;

combining the vehicle sample image A1 and the vehicle sample image A3 to serve as a positive sample pair, merging the positive sample pair into a multi-channel image 2 of 6 channels (two red channels, two green channels and two blue channels), and adding the obtained multi-channel image 2 to a training sample set of the twin neural network model;

combining the vehicle sample image A1 and the vehicle sample image B1 to serve as a negative sample pair, combining the negative sample pair into a multi-channel image 3 of 6 channels (two red channels, two green channels and two blue channels), and adding the obtained multi-channel image 3 into a training sample set of the twin neural network model;

……

combining the vehicle sample image A2 and the vehicle sample image Bn to serve as a negative sample pair, combining the negative sample pair into a multi-channel image n-1 of 6 channels (two red channels, two green channels and two blue channels), and adding the obtained multi-channel image n-1 into a training sample set of the twin neural network model;

the vehicle sample image a2 and the vehicle sample image C are combined as a negative sample pair, and merged into a multichannel image n of 6 channels (two red channels, two green channels, and two blue channels), and the obtained multichannel image n is added to the training sample set of the twin neural network model.

After the training sample set of the twin neural network model is obtained, the network equipment can train the preset initial twin model according to the training sample set to obtain the twin neural network model.

The preset initial twin model (i.e. the initial twin neural network model) may include an upper half branch network and a lower half branch network, and the upper half branch network and the lower half branch network may adopt CNNs with the same structure but do not share weights, that is, the twin neural network model includes two CNNs, where each CNN may include four convolutional layers and one fully connected layer. In order to reduce the complexity of calculation and improve the calculation efficiency, in this embodiment, the sizes of the convolution kernels of the four convolutional layers may be all set to (3, 3), the activation functions all adopt "relu", and the padding modes all set to "same"; optionally, in order to further reduce the amount of computation, downsampling operations, such as maxporoling, may also be performed in the second convolutional layer and the third convolutional layer. After the maxporoling operation is performed, the output after the maxporoling operation can be mapped through a full connection layer, wherein in the embodiment, the number of neurons of the full connection layer can be set to 512 (or to 128, etc.) regardless of the upper half branch network or the lower half branch network, and sigmoid can be adopted as the activation function.

In addition, as shown in fig. 2c, the preset initial twin model may include a one-dimensional full link layer, in addition to the upper half branch network and the lower half branch network, for mapping the output vectors of the upper half branch network and the lower half branch network into one-dimensional data; wherein, the number of neurons in the one-dimensional fully-connected layer is 1, and sigmoid can be adopted as the activation function.

When model training is needed, the network device may select a multi-channel image (one multi-channel image corresponds to one sample pair, i.e., corresponds to two vehicle sample images) from the training sample set of the twin neural network model as a current training sample; then, as shown in fig. 2c, on one hand, the current training sample may be guided into the upper half branch network of the preset initial twin model according to the original scale size to obtain an upper half branch network output vector, and on the other hand, the current training sample may be subjected to operations such as clipping, down-sampling and/or rotation to obtain a smaller scale training sample with data enhancement, and then the smaller scale training sample is guided into the lower half branch network of the preset initial twin model to be trained to obtain a lower half branch network output vector; then, the manhattan distance (L1 distance) between the output vector of the upper half branch network and the output vector of the lower half branch network can be calculated, one-dimensional full connection operation (namely, full connection of a neuron) is carried out according to the calculated manhattan distance, the result of the one-dimensional full connection operation is calculated by adopting an activation function sigmoid to obtain the similarity prediction value of the sample pair corresponding to the current training sample, the similarity real value of the sample pair is obtained, the similarity real value and the similarity prediction value are converged by adopting a preset loss function to adjust each parameter in the twin initial model to a proper value, then, the step of selecting a multi-channel image from the training sample set as the current training sample set can be returned to carry out to calculate and converge the similarity prediction values of other multi-channel images in the training sample set, and obtaining a trained model, namely the required twin neural network model, until all the multi-channel images in the training sample set are calculated and converged.

Wherein the loss function J can be selected as the cross entropy as follows:

wherein, C is the category number, C is 2, k ∈ (1,2), and different values of k represent whether belong to the same vehicle, and are the predicted value of the similarity, which is the true value of the similarity.

It should be noted that, during actual training, the initial twin model may not be pre-trained, and initialization weights are normally distributed, because the number of layers is shallow, the convergence rate is fast, for example, after about 40 epochs, the convergence is performed, that is, the twin neural network model provided by the embodiment of the present invention not only occupies less computing resources (light weight), but also has a fast recognition rate and a high efficiency.

In addition, it should be noted that, in order to ensure the accuracy of the recognition of the common CNN and twin neural network models, in addition to training the CNN and twin neural network models off-line, new vehicle sample images may be collected at regular time to update the training samples in each training sample set, and the CNN and twin neural network models are updated based on the updated training sample set, that is, the CNN and twin neural network models may be continuously learned.

And (II) establishing a reference vehicle identity information base.

Collecting a plurality of reference vehicle images, establishing a candidate set, and acquiring real identity information of a reference vehicle corresponding to each reference vehicle image, such as license plate information, owner information and the like, by the network equipment; then, a mapping relation between each reference vehicle image and the corresponding identity information is established, and the mapping relation is stored in a reference vehicle identity information base.

For example, the network device may specifically acquire the reference vehicle image by shooting a reference vehicle, or extracting the reference vehicle image from a registration database such as a vehicle management center, where the reference vehicle mainly refers to a vehicle whose owner has confirmed the real identity, such as a vehicle whose license plate information is displayed normally.

And (III) identifying the identity of the vehicle.

As shown in fig. 2d, based on the trained model, the specific process of the vehicle identification method may be as follows:

201. the network equipment acquires a vehicle identification request, wherein the vehicle identification request carries a vehicle image to be identified.

For example, the user may specifically obtain the image of the vehicle to be recognized by shooting the vehicle to be recognized or extracting the image from another gallery, and provide the image to the network device.

The vehicle to be identified mainly refers to a vehicle needing identity identification, for example, a vehicle whose owner identity is not confirmed in the monitoring video, such as a vehicle without license plate information or with abnormal license plate information display.

202. After receiving the vehicle identification request, the network device may obtain a candidate set, where the candidate set may include multiple reference vehicle images, and then execute step 203.

The reference vehicle image refers to an image containing a reference vehicle, and the reference vehicle mainly refers to a vehicle with a confirmed real identity of a vehicle owner, such as a vehicle with a normal license plate information display.

203. The network device matches the reference vehicle images in the candidate set with the vehicle images to be identified, filters the reference vehicle images with the matching degree smaller than a set value in the candidate set to obtain a filtered candidate set, and then executes step 204.

The set value is set according to the requirements of practical application, and the matching mode can also be set according to the requirements of practical application, for example, the information such as a hanging decoration, an interior decoration, a front side of a vehicle, a back side of the vehicle and the like in the vehicle can be compared, and the obtained similarity is used as the matching degree, namely, obviously dissimilar reference vehicle images can be filtered. The information such as the hanging decoration and the interior decoration in the vehicle can be obtained through a detection means, the front side and the back side of the vehicle can be obtained through detecting key points of the vehicle, and the specific detection modes can be various and are not described herein any more.

204. The network device determines a reference vehicle image currently in need of processing from the filtered candidate set.

205. And the network equipment calculates the similarity between the vehicle image to be identified and the reference vehicle image which needs to be processed currently by using the trained CNN to obtain the global similarity.

206. The network device extracts the image block of the area where the preset marker is located from the vehicle image to be identified to obtain a local image to be identified, extracts the image block of the area where the preset marker is located from the reference vehicle image to obtain a reference local image, and then executes step 207.

For example, the network device may specifically obtain preset marker information, determine first position information of a marker in the vehicle image to be identified according to the preset marker information, and intercept an image block of an area where the preset marker is located from the vehicle image to be identified according to the first position information to obtain a local image to be identified; and determining second position information of the marker in the reference vehicle image according to the preset marker information, and intercepting an image block of an area where the preset marker is located from the reference vehicle image according to the second position information to obtain a reference local image, and the like.

The preset mark can be determined according to the requirements of practical applications, and the preset mark generally needs to have distinctive individual features, such as an annual inspection mark attached to window glass, a pendant in a vehicle, decoration, and the like.

For example, referring to fig. 2e, the annual survey mark includes a 'test' word on or under which the next year of the vehicle is displayed (e.g. 2010), the number of the test word is surrounded by 1-12 arabic numerals, one of which is punched, and the month of the next vehicle is represented by the arabic numeral that is punched (e.g. 4 which is punched in fig. 2 e), and is generally located at the upper right corner of the front windshield of the vehicle, and since the size of 80 × 80pixel values (pixels) is sufficient to cover a complete annual survey mark, the size of the extracted image block can be generally set to not more than 80 × 80pixels, and of course, the size of the extracted area can be flexibly adjusted according to the actual application scene, and is not limited herein.

Optionally, since the greater the global similarity, the more similar the target vehicle and the vehicle to be identified in appearance, in order to reduce the data processing amount of local feature matching (i.e. annual inspection mark matching, i.e. step 207), only the top M reference vehicle images with the highest global similarity may be selected for annual inspection mark image extraction, so as to ensure that the reference vehicles in the reference vehicle images used for annual inspection mark image extraction are substantially the same as the vehicle to be identified in appearance, such as all belong to the same vehicle type, the same color, the same brand, and the like. Wherein, M is a positive integer, and the specific value can be determined according to the requirement of practical application.

The

steps

205 and 206 may not be executed sequentially.

207. And the network equipment calculates the similarity between the local image to be recognized and the reference local image according to the twin neural network model (namely the twin neural network model trained by the model training part) to obtain the local similarity.

For example, as shown in fig. 2f, the method for calculating the similarity between the local image to be identified and the reference local image may specifically be as follows:

first, the network device may combine the local image to be identified and the reference local image as an "image pair", and combine the local image to be identified and the reference local image in the "image pair" into a multi-channel image, such as the image K of 6 channels.

Secondly, on one hand, the network device may input the multi-channel image, such as the image K with the original size, into the upper half branch network of the twin neural network model for calculation to obtain an upper half branch network vector, and on the other hand, the multi-channel image is clipped, down-sampled and/or rotated to obtain a data-enhanced multi-channel image with a smaller size, such as the image K with a smaller size, and then the multi-channel image with a smaller size, such as the image K with a smaller size, is introduced into the lower half branch network of the twin neural network model for calculation to obtain an output vector of the lower half branch network.

Thereafter, the network device may calculate a manhattan distance (L) between the upper half branch network output vector and the lower half branch network output vector₁Distance), performing one-dimensional full-connection operation (namely, full-connection of one neuron) according to the calculated Manhattan distance, and calculating the result of the one-dimensional full-connection operation by adopting an activation function sigmoid to obtain a similarity prediction value of the image pair, wherein the similarity prediction value of the image pair is the local similarity of the image of the vehicle to be recognized and the image of the reference vehicle.

By analogy, the local similarity of the vehicle image to be identified and other reference vehicle images can be obtained according to the mode.

208. The network device performs weighted operation on the global similarity obtained in step 205 and the local similarity obtained in step 207 to obtain a comprehensive similarity, and then performs step 209. For example, it can be formulated as follows:

sim＝(1-μ)sim_global+μsim_local；

wherein sim is the comprehensive similarity sim_globalIs a whole worldSimilarity, sim_localFor local similarity, μ is a weight, μ is in a range of (0, 1), and a specific value of μmay be determined according to requirements of practical applications, which is not described herein again.

209. The network device takes the reference vehicle image whose integrated similarity satisfies a preset second condition as the target vehicle image, and then performs step 210.

The preset second condition may be that "the preset second condition is higher than a preset threshold," or "the first N highest in comprehensive similarity," values of the preset threshold and N may be determined according to requirements of practical applications, N is a positive integer, for example, if N is 2, the obtained multiple comprehensive similarities may be ranked, then, the first 2 reference vehicle images with higher comprehensive similarity are selected as target vehicle images, and so on.

210. The network device obtains identity information corresponding to the target vehicle image from the reference vehicle identity information base, for example, obtains license plate information and owner information corresponding to the target vehicle image, and the like, and uses the obtained license plate information and owner information as identity information of the vehicle to be identified.

For example, if the target vehicle image is the reference vehicle image a, the corresponding license plate information is "yue B0000", and the owner is "zhang san"; then, at this time, it may be determined that the identity information of the vehicle to be identified may be "yue B0000", and the owner of the vehicle is "zhang san".

For another example, if the target vehicle image is the reference vehicle image B and the reference vehicle image C. The license plate information corresponding to the reference vehicle image B is YueB 0001, and the vehicle owner is Li four; the license plate information corresponding to the reference vehicle image C is Yue B0002, and the vehicle owner is Wangwu; then, at this time, it may be determined that the identity information of the vehicle to be identified may be "yue B0001" and the vehicle owner may be "li si", and may also be "yue B0002" and the vehicle owner may be "wangwu", and then the final vehicle identity information may be determined by other means, such as manual screening.

Optionally, in addition to providing the identification information of the identified vehicle to be identified, a corresponding numerical value of the comprehensive similarity may also be provided, or each numerical value of the global similarity, the local similarity and the comprehensive similarity may also be provided, so that the user can determine the reliability of the identification result accordingly, and further manually perform further screening based on the identification result, which is not described herein again.

As can be seen from the above, in this embodiment, when the vehicle image to be recognized needs to be recognized, at least one reference vehicle image may be obtained, and then, on the one hand, the similarity between the vehicle image to be recognized and the reference vehicle image is calculated to obtain the global similarity; on the other hand, extracting image blocks of the area where the preset marker is located from the vehicle image to be recognized and the reference vehicle image respectively to obtain a local image to be recognized and a reference local image, calculating the similarity between the local image to be recognized and the reference local image according to a preset twin neural network model to obtain a local similarity, then performing weighted operation on the global similarity and the local similarity to obtain a comprehensive similarity, and taking the identity information corresponding to the reference vehicle image with the comprehensive similarity meeting a preset second condition as the identity information of the vehicle to be recognized; according to the scheme, the vehicles which are most similar to the vehicle to be recognized and have the real identity information can be quickly and accurately matched by calculating the global similarity and the local similarity, so that the identities of the vehicle to be recognized, such as the number of the license plate, the owner, and the like, can be recognized, compared with the existing scheme that the identity can only be recognized through the license plate information, the situation that the vehicle cannot be recognized or the recognition is wrong due to the fact that the license plate information is fake, missing or fuzzy and the like can be avoided, the dependency on the license plate information of the vehicle to be recognized can be reduced, and the effectiveness and the accuracy of the recognition are greatly improved.

Example III,

In order to better implement the method, the embodiment of the invention further provides a vehicle identification device, and the vehicle identification device can be specifically integrated in network equipment, such as a terminal or a server and the like.

For example, as shown in fig. 3a, the vehicle identification apparatus includes an obtaining unit 301, a global calculating unit 302, an extracting unit 303, a local calculating unit 304, and a recognizing unit 305, as follows:

(1) an acquisition unit 301;

an acquiring unit 301, configured to acquire a vehicle image to be identified and at least one reference vehicle image.

For example, the obtaining unit 301 may be specifically configured to receive a vehicle identification request triggered by a user, where the vehicle identification request carries a vehicle image to be identified, and then obtain at least one reference vehicle image according to the vehicle identification request.

Optionally, in order to reduce subsequent calculation amount and improve processing efficiency, when reference vehicle images are obtained, the reference vehicle images may be subjected to preliminary screening to filter out images obviously inconsistent with the vehicle to be identified, that is:

the obtaining unit 301 may be specifically configured to obtain a vehicle image to be identified, obtain a candidate set, where the candidate set includes a plurality of reference vehicle images, match the reference vehicle images in the candidate set with the vehicle image to be identified, filter the reference vehicle images whose matching degree is smaller than a set value to obtain a filtered candidate set, and obtain at least one reference vehicle image from the filtered candidate set.

(2) A global computation unit 302;

and the global calculation unit 302 is configured to calculate a similarity between the to-be-recognized vehicle image and the reference vehicle image, so as to obtain a global similarity.

The similarity between the vehicle image to be recognized and the reference vehicle image may be calculated in various ways, for example, the common CNN may be used for calculation, that is:

the global calculating unit 302 may be specifically configured to calculate a similarity between the vehicle image to be identified and the reference vehicle image by using a preset CNN, so as to obtain a global similarity.

Alternatively, another twin neural network model may be used to calculate the global similarity. Namely:

the global calculating unit 302 may be specifically configured to calculate a similarity between the to-be-recognized vehicle image and the reference vehicle image by using a preset twin neural network model, so as to obtain a global similarity.

The twin neural network model may specifically refer to the foregoing embodiments, and will not be described herein.

(3) An extraction unit 303;

the extracting unit 303 is configured to extract image blocks of an area where a preset marker is located from the vehicle image to be identified and the reference vehicle image, respectively, to obtain a local image to be identified and a reference local image;

for example, the extracting unit 303 may be specifically configured to obtain preset marker information, determine first position information of a marker in the vehicle image to be recognized according to the preset marker information, and intercept an image block of an area where the preset marker is located from the vehicle image to be recognized according to the first position information to obtain a local image to be recognized; and determining second position information of the marker in the reference vehicle image according to the preset marker information, and intercepting an image block of an area where the preset marker is located from the reference vehicle image according to the second position information to obtain a reference local image.

The preset marker can be determined according to the requirements of practical application, the preset marker generally needs to have vivid individual characteristics, such as an annual inspection mark adhered to window glass, a pendant and a decoration in a vehicle, and the first position information and the second position information can be specifically coordinate information.

(4) A local calculation unit 304;

and the local calculating unit 304 is configured to calculate a similarity between the local image to be recognized and the reference local image according to a preset twin neural network model, so as to obtain a local similarity.

For example, the local calculating unit 304 may be specifically configured to combine the local image to be identified and the reference local image two by two to obtain a plurality of image pairs, merge the local image to be identified and the reference local image in each image pair into a multi-channel image, input the multi-channel image into the upper half branch network and the lower half branch network of the twin neural network model respectively to obtain an output vector of the upper half branch network and an output vector of the lower half branch network, calculate a manhattan distance between the output vector of the upper half branch network and the output vector of the lower half branch network, and perform a one-dimensional full-connection operation according to the calculated manhattan distance to obtain a local similarity.

(5) An identification unit 305;

the identification unit 305 is configured to obtain identity information corresponding to a reference vehicle image, where the global similarity and the local similarity satisfy a preset first condition, as identity information of a vehicle to be identified.

For example, the identification unit 305 may include an operation subunit and a determination subunit, as follows:

the operation subunit is configured to perform weighted operation on the global similarity and the corresponding local similarity to obtain a comprehensive similarity. For example, it can be formulated as follows:

sim＝(1-μ)sim_global+μsim_local；

The determining subunit is configured to acquire identity information corresponding to a reference vehicle image, of which the comprehensive similarity satisfies a preset second condition, as identity information of the vehicle to be identified.

The identity information corresponding to the reference vehicle image may be obtained based on a preset mapping relationship (mapping relationship between the reference vehicle image and the identity information of the reference vehicle), that is, as shown in fig. 3b, the vehicle identity recognition apparatus may further include a setting unit 306, as follows:

the setting unit 306 may be configured to obtain real identity information of a reference vehicle corresponding to each reference vehicle image, where the identity information includes license plate information and owner information, establish a mapping relationship between each reference vehicle image and its corresponding identity information, and store the mapping relationship.

At this time, the determining subunit may be specifically configured to use the reference vehicle image with the comprehensive similarity satisfying the preset second condition as the target vehicle image, and obtain the identity information corresponding to the target vehicle image according to the mapping relationship, as the identity information of the vehicle to be identified.

The twin neural network model may be pre-established by operation and maintenance personnel, or may be established by the vehicle identity recognition apparatus, that is, as shown in fig. 3b, the vehicle identity recognition apparatus may further include an acquisition unit 307, a combination unit 308, a merging unit 309, and a training unit 310, as follows:

the acquiring unit 307 may be configured to acquire a plurality of vehicle sample images, where the vehicle sample images have real identity information.

The combining unit 308 can be configured to combine the plurality of vehicle sample images two by two to establish a sample pair.

If the sample pair includes a positive sample pair and a negative sample pair, the combining unit 308 may be specifically configured to select a vehicle sample image belonging to the same vehicle from the multiple vehicle sample images, and combine the vehicle sample images belonging to the same vehicle two by two to establish the positive sample pair; and selecting the vehicle sample images which do not belong to the same vehicle from the plurality of vehicle sample images, and combining the vehicle sample images which do not belong to the same vehicle in pairs to establish a negative sample pair.

The merging unit 309 may be configured to merge each sample pair into a multi-channel image, and add the multi-channel image to the training sample set.

For example, the merging unit 309 may be specifically configured to determine color channels of the vehicle sample images in each sample pair, add the color channels to obtain a multi-channel image corresponding to each sample pair, and add the obtained multi-channel image to the training sample set.

The training unit 310 may be configured to train a preset initial twin model according to a training sample set, so as to obtain a twin neural network model.

For example, the training unit 310 may include a training subunit and a convergence subunit, as follows:

the training subunit may be configured to train an upper half branch network and a lower half branch network of a preset initial twin model according to the training sample set, respectively, to obtain a similarity prediction value of a sample pair corresponding to each multi-channel image in the training sample set.

For example, the training subunit may be specifically configured to select a multi-channel image from the training sample set as a current training sample; respectively introducing the current training sample into an upper half branch network and a lower half branch network of a preset initial twin model for training to obtain an output vector of the upper half branch network and an output vector of the lower half branch network; performing one-dimensional full-connection operation on the upper half branch network output vector and the lower half branch network output vector to obtain a similarity prediction value of a sample pair corresponding to the current training sample; and returning to the step of selecting one multi-channel image from the training sample set as the current training sample until all the multi-channel images in the training sample set are trained.

For example, the convergence subunit may be specifically configured to converge the true similarity value and the predicted similarity value by using a preset loss function, so as to obtain a twin neural network model.

The loss function may be determined according to the requirement of the actual application, and is not described herein.

In a specific implementation, the above units may be implemented as independent entities, or may be combined arbitrarily to be implemented as the same or several entities, and the specific implementation of the above units may refer to the foregoing method embodiments, which are not described herein again.

As can be seen from the above, when the vehicle identification apparatus of this embodiment needs to identify a vehicle image to be identified, the obtaining unit 301 may obtain at least one reference vehicle image, and then, on one hand, the global calculating unit 302 calculates the similarity between the vehicle image to be identified and the reference vehicle image to obtain the global similarity; on the other hand, the extracting unit 303 extracts image blocks of the area where the preset marker is located from the vehicle image to be recognized and the reference vehicle image respectively to obtain a local image to be recognized and a reference local image, the local calculating unit 304 calculates the similarity between the local image to be recognized and the reference local image according to the preset twin neural network model to obtain a local similarity, and then the identifying unit 305 obtains identity information corresponding to the reference vehicle image, of which the global similarity and the local similarity meet a preset first condition, as identity information of the vehicle to be recognized; according to the scheme, the vehicle which is most similar to the vehicle to be recognized and has the real identity information can be automatically matched by calculating the global similarity and the local similarity, and the identity of the vehicle to be recognized is recognized, so that compared with the existing scheme that the identity recognition can only be carried out through the license plate information, the situation that the vehicle cannot be recognized or the recognition is wrong due to the fact that the license plate information is fake, lost or fuzzy and the like can be avoided, the dependency on the license plate information of the vehicle to be recognized can be reduced, and the effectiveness and the accuracy of the recognition are greatly improved.

Example four,

The embodiment of the invention also provides network equipment, which can be equipment such as a server or a terminal and the like, and is integrated with any vehicle identity recognition device provided by the embodiment of the invention. Fig. 4 is a schematic diagram illustrating a network device according to an embodiment of the present invention, specifically:

the network device may include components such as a processor 401 of one or more processing cores, memory 402 of one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that the network device architecture shown in fig. 4 does not constitute a limitation of network devices and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components. Wherein:

the processor 401 is a control center of the network device, connects various parts of the entire network device by using various interfaces and lines, and performs various functions of the network device and processes data by running or executing software programs and/or modules stored in the memory 402 and calling data stored in the memory 402, thereby performing overall monitoring of the network device. Optionally, processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 401.

The memory 402 may be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by operating the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to use of the network device, and the like. Further, the memory 402 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 access to the memory 402.

The network device further includes a power supply 403 for supplying power to each component, and preferably, the power supply 403 is logically connected to the processor 401 through a power management system, so that functions of managing charging, discharging, and power consumption are implemented through the power management system. The power supply 403 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.

The network device may also include an input unit 404, where the input unit 404 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

Although not shown, the network device may further include a display unit and the like, which are not described in detail herein. Specifically, in this embodiment, the processor 401 in the network device loads the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 runs the application program stored in the memory 402, thereby implementing various functions as follows:

the method comprises the steps of obtaining a vehicle image to be recognized and at least one reference vehicle image, calculating the similarity between the vehicle image to be recognized and the reference vehicle image to obtain global similarity, extracting image blocks of areas where preset markers are located from the vehicle image to be recognized and the reference vehicle image respectively to obtain a local image to be recognized and a reference local image, calculating the similarity between the local image to be recognized and the reference local image according to a preset twin neural network model to obtain local similarity, and obtaining identity information corresponding to the reference vehicle image, wherein the global similarity and the local similarity meet preset first conditions, and the identity information serves as identity information of a vehicle to be recognized.

The twin neural network model may be established in advance by an operation and maintenance person, or may be established by the vehicle identification device, that is, the processor 401 may further run an application program stored in the memory 402, so as to implement the following functions:

collecting a plurality of vehicle sample images, wherein the vehicle sample images have real identity information, such as normal license plate information, real owner information and the like; combining the plurality of vehicle sample images in pairs to establish a sample pair; combining each sample pair into a multi-channel image, and adding the multi-channel image to a training sample set; and training a preset initial twin model according to the training sample set to obtain a twin neural network model.

The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.

As can be seen from the above, when the network device of this embodiment needs to identify the vehicle image to be identified, at least one reference vehicle image may be obtained, and then, on the one hand, the similarity between the vehicle image to be identified and the reference vehicle image is calculated to obtain the global similarity; on the other hand, extracting image blocks of the area where the preset marker is located from the vehicle image to be recognized and the reference vehicle image respectively to obtain a local image to be recognized and a reference local image, calculating the similarity between the local image to be recognized and the reference local image according to a preset twin neural network model to obtain a local similarity, and then obtaining identity information corresponding to the reference vehicle image, of which the global similarity and the local similarity meet a preset first condition, as the identity information of the vehicle to be recognized; according to the scheme, the vehicle which is most similar to the vehicle to be recognized and has the real identity information can be automatically matched by calculating the global similarity and the local similarity, and the identity of the vehicle to be recognized is recognized, so that compared with the existing scheme that the identity recognition can only be carried out through the license plate information, the situation that the vehicle cannot be recognized or the recognition is wrong due to the fact that the license plate information is fake, lost or fuzzy and the like can be avoided, the dependency on the license plate information of the vehicle to be recognized can be reduced, and the effectiveness and the accuracy of the recognition are greatly improved.

Example V,

It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.

To this end, embodiments of the present invention provide a storage medium having stored therein a plurality of instructions, which can be loaded by a processor to perform the steps of any of the vehicle identification methods provided by the embodiments of the present invention. For example, the instructions may perform the steps of:

The twin neural network model may be established in advance by operation and maintenance personnel, or may be established by the vehicle identity recognition device, and the specific establishment method may refer to the foregoing embodiment.

Wherein the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.

Since the instructions stored in the storage medium can execute the steps in any vehicle identification method provided in the embodiment of the present invention, the beneficial effects that can be achieved by any vehicle identification method provided in the embodiment of the present invention can be achieved, which are detailed in the foregoing embodiments and will not be described herein again.

The vehicle identification method, the vehicle identification device and the storage medium provided by the embodiment of the invention are described in detail, a specific example is applied in the description to explain the principle and the implementation of the invention, and the description of the embodiment is only used for helping to understand the method and the core idea of the invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A vehicle identification method, comprising:

receiving a vehicle identification request triggered by a user, wherein the vehicle identification request carries a vehicle image to be identified;

acquiring at least one reference vehicle image according to the vehicle identity identification request;

acquiring a reference vehicle image of which the global similarity and the local similarity meet a preset first condition;

acquiring identity information corresponding to the reference vehicle image according to the mapping relation, and taking the identity information as the identity information of the vehicle to be identified; the identity information comprises license plate information and vehicle owner information; the mapping relation is the mapping relation between each reference vehicle image and the corresponding identity information;

wherein the twin neural network model is generated by:

collecting a plurality of vehicle sample images, wherein the vehicle sample images have real identity information;

combining the plurality of vehicle sample images pairwise to establish a sample pair;

combining each sample pair into a multi-channel image, and adding the multi-channel image to a training sample set;

selecting a multi-channel image from the training sample set as a current training sample;

respectively introducing the current training sample into an upper half branch network and a lower half branch network of a preset initial twin model for training to obtain an output vector of the upper half branch network and an output vector of the lower half branch network; wherein the upper half branch network and the lower half branch network have the same structure but do not share the weight; the upper half branch network comprises a plurality of convolutional layers and a one-dimensional full-connection layer;

performing one-dimensional full-connection operation on the upper half branch network output vector and the lower half branch network output vector to obtain a similarity prediction value of a sample pair corresponding to the current training sample; the upper half branch network processes a training sample with an original scale, and the lower half branch network processes a training sample with a smaller scale, so that the Manhattan distance between an output vector of the upper half branch network and an output vector of the lower half branch network is calculated, and one-dimensional full-connection operation is performed according to the Manhattan distance;

returning to the step of selecting one multi-channel image from the training sample set as the current training sample until all the multi-channel images in the training sample set are trained;

and acquiring the true similarity value of each sample pair, and converging the true similarity value and the predicted similarity value to obtain the twin neural network model.

2. The method according to claim 1, wherein the obtaining of the identity information corresponding to the reference vehicle image with the global similarity and the local similarity satisfying the preset condition as the identity information of the vehicle to be recognized comprises:

carrying out weighting operation on the global similarity and the corresponding local similarity to obtain comprehensive similarity;

and acquiring identity information corresponding to the reference vehicle image with the comprehensive similarity meeting the preset second condition as the identity information of the vehicle to be identified.

3. The method according to claim 2, wherein before the obtaining of the identity information corresponding to the reference vehicle image with the comprehensive similarity satisfying the preset second condition as the identity information of the vehicle to be recognized, the method further comprises:

acquiring real identity information of a reference vehicle corresponding to each reference vehicle image, wherein the identity information comprises license plate information and vehicle owner information;

establishing a mapping relation between each reference vehicle image and the corresponding identity information thereof, and storing the mapping relation;

the acquiring of the identity information corresponding to the reference vehicle image with the comprehensive similarity meeting the preset second condition is specifically as the identity information of the vehicle to be recognized: and taking the reference vehicle image with the comprehensive similarity meeting the preset second condition as a target vehicle image, and acquiring the identity information corresponding to the target vehicle image according to the mapping relation to be used as the identity information of the vehicle to be identified.

4. The method according to claim 1, wherein the extracting image blocks of the areas where the preset markers are located from the to-be-identified vehicle image and the reference vehicle image respectively to obtain the to-be-identified local image and the reference local image comprises:

acquiring preset marker information;

determining first position information of a marker in the vehicle image to be identified according to the preset marker information, and intercepting an image block of an area where the preset marker is located from the vehicle image to be identified according to the first position information to obtain a local image to be identified;

and determining second position information of the marker in the reference vehicle image according to the preset marker information, and intercepting an image block of an area where the preset marker is located from the reference vehicle image according to the second position information to obtain a reference local image.

5. The method of any one of claims 1 to 4, wherein acquiring at least one reference vehicle image comprises:

acquiring a candidate set, wherein the candidate set comprises a plurality of reference vehicle images;

matching the reference vehicle images in the candidate set with the vehicle images to be identified;

filtering the reference vehicle images with the matching degree smaller than a set value to obtain a filtered candidate set;

at least one reference vehicle image is obtained from the filtered candidate set.

6. The method of claim 1, wherein the sample pairs comprise positive sample pairs and negative sample pairs, and wherein combining the plurality of vehicle sample images two-by-two to create sample pairs comprises:

selecting vehicle sample images belonging to the same vehicle from the plurality of vehicle sample images, and combining the vehicle sample images belonging to the same vehicle in pairs to establish a positive sample pair;

and selecting the vehicle sample images which do not belong to the same vehicle from the plurality of vehicle sample images, and combining the vehicle sample images which do not belong to the same vehicle in pairs to establish a negative sample pair.

7. The method of claim 1, wherein the merging each sample pair into a multi-channel image is added to a training sample set, comprising:

determining a color channel of the vehicle sample image in each sample pair;

adding the color channels to obtain a multi-channel image corresponding to each sample pair;

and adding the obtained multichannel images to a training sample set.

8. A vehicle identification device, comprising:

the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for receiving a vehicle identity identification request triggered by a user, and the vehicle identity identification request carries a vehicle image to be identified; acquiring at least one reference vehicle image according to the vehicle identity identification request;

the identification unit is used for acquiring a reference vehicle image of which the global similarity and the local similarity meet a preset first condition; acquiring identity information corresponding to the reference vehicle image according to the mapping relation, and taking the identity information as the identity information of the vehicle to be identified; the identity information comprises license plate information and vehicle owner information; the mapping relation is the mapping relation between each reference vehicle image and the corresponding identity information;

the device further comprises:

the system comprises an acquisition unit, a display unit and a display unit, wherein the acquisition unit is used for acquiring a plurality of vehicle sample images, and the vehicle sample images have real identity information;

the combination unit is used for combining the plurality of vehicle sample images in pairs to establish a sample pair;

the merging unit is used for merging each sample pair into a multi-channel image and then adding the multi-channel image to the training sample set;

the training unit is used for selecting a multi-channel image from the training sample set as a current training sample;

9. The apparatus of claim 8, wherein the identification unit comprises an operation subunit and a determination subunit;

the operation subunit is configured to perform weighted operation on the global similarity and the corresponding local similarity to obtain a comprehensive similarity;

the determining subunit is configured to acquire identity information corresponding to a reference vehicle image of which the comprehensive similarity satisfies a preset second condition, and use the identity information as the identity information of the vehicle to be identified.

10. The apparatus of claim 9, further comprising a setting unit;

the setting unit is used for acquiring real identity information of a reference vehicle corresponding to each reference vehicle image, wherein the identity information comprises license plate information and owner information, establishing a mapping relation between each reference vehicle image and the corresponding identity information thereof, and storing the mapping relation;

the determining subunit is specifically configured to use the reference vehicle image with the comprehensive similarity meeting a preset second condition as a target vehicle image, and obtain, according to the mapping relationship, identity information corresponding to the target vehicle image, as identity information of the vehicle to be identified.

11. A storage medium storing a plurality of instructions adapted to be loaded by a processor to perform the steps of the vehicle identification method according to any one of claims 1 to 7.