CN116052123A

CN116052123A - Parking space detection method, device, vehicle and equipment based on camera picture

Info

Publication number: CN116052123A
Application number: CN202310042762.7A
Authority: CN
Inventors: 江建山; 何山波; 方志杰; 王志伟; 罗宇亮
Original assignee: GAC Aion New Energy Automobile Co Ltd
Current assignee: GAC Aion New Energy Automobile Co Ltd
Priority date: 2023-01-28
Filing date: 2023-01-28
Publication date: 2023-05-02

Abstract

The embodiment of the application provides a parking space detection method, device, vehicle and equipment based on camera pictures, wherein the method comprises the following steps: acquiring shooting pictures of a plurality of cameras on a vehicle; generating a bird's-eye view according to the pictures shot by the plurality of cameras, and carrying out joint calibration on the bird's-eye view to obtain first vehicle position information in the bird's-eye view; inputting the shooting picture of each camera into a bird's-eye view feature extraction model to obtain bird's-eye view features; inputting the aerial view features into a parking space detection model to obtain second vehicle position information; adjusting the aerial view feature extraction model and the parking space detection model according to the first vehicle position information and the second vehicle position information to obtain a trained aerial view feature extraction model and a trained parking space detection model; and carrying out parking space detection according to the trained aerial view feature extraction model and the trained parking space detection model. By implementing the embodiment, the parking space detection stage and the aerial view stage of the single image can be skipped, aerial view features can be directly generated, and parking space detection can be rapidly realized.

Description

Parking space detection method, device, vehicle and equipment based on camera picture

Technical Field

The application relates to the technical field of image processing, in particular to a parking space detection method, device, vehicle, electronic equipment and computer readable storage medium based on camera pictures.

Background

The automatic parking consists of three parts, namely parking space detection, path planning and path tracking, and even in auxiliary parking, the parking space detection is the most critical link, and how to better make the parking space detection in parking is the following most commonly used technical schemes. Scheme one: inputting two-dimensional images acquired by the multi-angle cameras one by one into a two-dimensional image parking space detection model to perform parking space detection to obtain parking space detection information under a single camera, obtaining three-dimensional parking space detection information by reverse perspective transformation, and then fusing the parking space detection information of the multi-angle cameras to obtain the parking space detection information under a panoramic aerial view; scheme II: the multi-angle cameras acquire two-dimensional images one by one through inverse perspective transformation to obtain three-dimensional images, the three-dimensional images are input into a three-dimensional image parking space detection model to obtain parking space detection information under three dimensions, and then the multi-angle cameras are used for fusing the parking space detection information to obtain parking space detection information under a panoramic aerial view. Scheme III: and (3) the two-dimensional images acquired by the multi-angle cameras enter fusion to obtain an aerial view image, and then the aerial view image is input into a aerial view image parking space detection model to obtain parking space detection information under the panoramic aerial view. According to the scheme I and the scheme II, parking space detection information under a single camera needs to be output firstly, then fusion is carried out, the parking space detection information under the panoramic view can be obtained, and the parking space detection failure can be caused because the angle of view of the single camera is limited. The third scheme needs to fuse the images at multiple angles into the image information under the aerial view, and is a multi-end parking space detection method with long detection time.

Disclosure of Invention

An object of the embodiment of the application is to provide a parking space detection method, device, vehicle and equipment based on a camera picture, which can directly generate a bird's eye view characteristic, eliminate the condition of missed detection and false detection due to a single camera view angle and rapidly realize parking space detection.

In a first aspect, an embodiment of the present application provides a parking space detection method based on a camera frame, including:

acquiring shooting pictures of a plurality of cameras on a vehicle;

generating a bird's-eye view according to the pictures shot by the plurality of cameras, and carrying out joint calibration on the bird's-eye view to obtain first vehicle position information in the bird's-eye view;

inputting the shooting picture of each camera into a bird's-eye view feature extraction model to obtain bird's-eye view features;

inputting the aerial view features into a parking space detection model to obtain second vehicle position information;

adjusting the aerial view feature extraction model and the parking space detection model according to the first vehicle position information and the second vehicle position information to obtain a trained aerial view feature extraction model and a trained parking space detection model;

and carrying out parking space detection according to the trained aerial view feature extraction model and the trained parking space detection model.

In the implementation process, unlike the prior art, the embodiment of the application only constructs the aerial view feature extraction model, extracts the aerial view feature directly based on the aerial view feature extraction model, does not need to synthesize the aerial view in advance when detecting the parking space, does not need to input the shooting picture of a single camera into the parking space detection model for detection, solves the problem that the parking space is only half or less in single-camera shooting, namely, the condition that the parking space is missed in the crossing position of two cameras is solved, also solves the problem of detection delay caused by synthesizing the aerial view in the detection stage, and improves the speed of parking space detection.

Further, the bird's eye view feature extraction model includes: a first feature extraction model;

the step of inputting the shooting picture of each camera into a bird's-eye view feature extraction model to obtain bird's-eye view features comprises the following steps:

inputting the shooting picture of each camera into a first feature extraction model to obtain a first feature map of each camera under a plurality of scales;

and acquiring the aerial view features according to the first feature images of each camera under a plurality of scales.

In the implementation process, the first feature map under multiple scales is generated based on the shooting picture of each camera, so that the shooting picture of the camera is fully utilized, and as many useful features as possible are extracted from the shooting picture of each camera.

Further, the bird's eye view feature extraction model includes: a first feature fusion model;

the step of obtaining the aerial view feature according to the first feature map of each camera under a plurality of scales comprises the following steps:

inputting the first feature images of each camera under a plurality of scales into a first feature fusion model to obtain a second feature image corresponding to each camera;

and fusing the second characteristic map with the internal and external parameters of the camera to obtain the aerial view characteristic.

Further, the parking space detection model includes: a second feature extraction model;

the step of inputting the aerial view features into a parking space detection model to obtain second vehicle position information comprises the following steps:

inputting the aerial view features into a second feature extraction model to obtain first aerial view features with multiple scales;

fusing the first aerial view features of the multiple scales to obtain a second aerial view feature;

and acquiring the second vehicle position information according to the second aerial view characteristic.

In the implementation process, the bird's-eye view features are extracted and fused in multiple scales, so that the proportion of useful features in the second bird's-eye view features is improved, and the accuracy of the acquired second vehicle position information is further improved.

Further, the step of acquiring the second vehicle position information according to the second aerial view feature includes:

generating a task dimension according to the second vehicle position information;

reducing the dimension of the second aerial view feature according to the task dimension to obtain a reduced dimension second aerial view feature;

and acquiring the second vehicle position information according to the second aerial view characteristic after the dimension reduction.

In the implementation process, task dimensions are generated according to the information of the second vehicle position information, the dimensions of the second aerial view features are reduced, the features are converted into dimensions corresponding to the identification tasks, and the full utilization of the second aerial view features is achieved.

Further, the parking space information includes: a plurality of corner coordinates of the parking space;

the second aerial view features after the dimension reduction comprise second aerial view sub-features of a plurality of channels;

the plurality of channels of the second bird's eye view feature includes: the parking space comprises a first channel corresponding to the coordinates of a plurality of angular points of the parking space and a second channel for clustering;

the step of obtaining the second vehicle position information according to the second aerial view feature after the dimension reduction includes:

analyzing the second aerial view sub-feature of the first channel by using a hetmap algorithm to obtain a plurality of initial angular point coordinates;

Acquiring a plurality of parking space characteristic values corresponding to the initial angular point coordinates on the second channel according to the initial angular point coordinates;

clustering the parking space characteristic values to obtain a clustering result;

and generating a plurality of corner coordinates of the parking space according to the clustering result.

Further, the first channel includes: the parking space angle sensor comprises a plurality of first sub-channels, wherein each first sub-channel is used for acquiring one angle point coordinate of the parking space;

the step of generating the coordinates of the plurality of corner points of the parking space according to the clustering result comprises the following steps:

judging whether channels to which a plurality of initial angular point coordinates corresponding to the parking space characteristic values in the clustering result belong comprise all the first sub-channels or not;

and if so, screening the initial angular point coordinates corresponding to the parking space characteristic values of the clustering result to obtain a plurality of angular point coordinates of the parking space.

Further, the step of screening the initial angular point coordinates corresponding to the parking space feature values of the clustering result to obtain a plurality of angular point coordinates of the parking space includes:

screening a plurality of initial corner coordinates belonging to the same first sub-channel in the clustering result to obtain a screened initial corner coordinate corresponding to the same first sub-channel;

And generating a plurality of corner coordinates of the parking space according to the screened initial corner coordinates corresponding to each first sub-channel in the clustering result.

Further, the step of screening the plurality of initial corner coordinates belonging to the same first sub-channel in the clustering result to obtain a screened initial corner coordinate corresponding to the same first sub-channel includes:

obtaining a plurality of least square values according to the parking space characteristic values corresponding to a plurality of initial angular point coordinates belonging to the same first sub-channel in the clustering result and the parking space characteristic values corresponding to other first sub-channels in the clustering result;

and taking the initial angular point coordinate corresponding to the least square value in the least square values as the initial angular point coordinate corresponding to the same channel after screening.

Further, the second bit information includes: the parking space berthing performance; the plurality of channels of the second bird's eye view feature includes: a third channel corresponding to the berthing property of the parking space;

determining the poisability corresponding to each feature point of the third channel;

Determining the poisability of the coordinates of a plurality of angular points of the parking space according to the poisability corresponding to each characteristic point;

and determining the berthing property of the parking space according to the berthing property of the plurality of corner coordinates.

Further, the second bit information includes: offset of a plurality of corner points of the parking space;

the plurality of channels of the second bird's eye view feature includes: a fourth channel corresponding to the offset of the plurality of corner points of the parking space;

determining offset characteristic values corresponding to a plurality of angular point coordinates of the parking space in the second aerial view sub-characteristic of the fourth channel;

and taking the offset characteristic values corresponding to the coordinates of the plurality of corner points of the parking space as the offset of the plurality of corner points of the parking space.

Further, the step of adjusting the aerial view feature extraction model and the parking space detection model according to the first vehicle position information and the second vehicle position information to obtain a trained aerial view feature extraction model and a trained parking space detection model includes:

acquiring scaling of the aerial view and the second aerial view characteristic after the dimension reduction;

Scaling the first vehicle position information according to the scaling ratio to obtain scaled first vehicle position information;

acquiring a loss value according to the scaled first vehicle position information and the scaled second vehicle position information;

and adjusting the aerial view characteristic extraction model and the parking space detection model according to the loss value to obtain a trained aerial view characteristic extraction model and a trained parking space detection model.

Further, the step of determining the berthing property of the parking space according to the berthing properties of the plurality of corner coordinates includes:

acquiring the number of corner points with poisability in a plurality of corner points of the parking space;

acquiring the number of corner points which do not have the poisability in the plurality of corner points of the parking space;

judging whether the number of the corner points with the poisability is more than that of the corner points without the poisability;

if yes, judging that the parking space has the berthability;

if not, judging that the parking space does not have the berthability.

Further, the step of inputting the photographed image of each camera into the first feature extraction model to obtain a first feature map of each camera under a plurality of scales includes:

acquiring the installation position of each camera;

Inputting the shooting picture of each camera into a first feature extraction model to obtain a first initial feature map corresponding to a preset scale output by the first feature extraction model;

and determining a second initial feature map of a second preset scale corresponding to the installation position of each camera from the first initial feature maps corresponding to all the first preset scales, and taking the second initial feature map of the second preset scale as the first feature map of each camera under a plurality of scales.

In the implementation process, because the mounting positions of the cameras are different, the weights of the features extracted from the photographed images in the parking space recognition process are different, and the second initial feature images of the second preset scale corresponding to the mounting positions of each camera are determined from the first initial feature images of all the first preset scales corresponding to the mounting positions of each camera based on the mounting positions of the cameras, and the second initial feature images of the second preset scale are used as the first feature images of each camera under multiple scales.

Further, the step of determining a first initial feature map of a preset scale corresponding to the installation position of each camera from the first initial feature maps of all preset scales to obtain a first feature map of each camera under a plurality of scales includes:

If the installation position of the camera is matched with the advancing direction of the vehicle, determining a first initial feature map corresponding to all first preset scales as a first feature map of the camera;

and if the installation position of the camera is not matched with the advancing direction of the vehicle, determining a first initial characteristic diagram corresponding to a preset second preset scale as the first characteristic diagram of the camera.

In the implementation process, the direction of the vehicle determines the direction of the parking space, so that by judging whether the installation position of the camera is matched with the direction of the vehicle, whether the first initial feature map corresponding to the first preset scale for generating the shooting picture of the camera is screened is further determined, and the proportion of useful information in the first feature map under a plurality of scales can be further improved.

In the implementation process, the first feature images under multiple scales are fused, the information is further concentrated, the second feature images are fused with the internal and external parameters of the camera, and the aerial view features are obtained, so that the aerial view features comprise the performance parameters of the camera, and the recognition effect is further improved.

Further, the step of determining a first initial feature map of a second preset scale corresponding to the installation position of each camera from the first initial feature maps corresponding to all the first preset scales to obtain a first feature map of each camera under a plurality of scales includes:

In a second aspect, an embodiment of the present application provides a parking space detection device based on a camera frame, including;

A shooting picture acquisition module for acquiring shooting pictures of a plurality of cameras on a vehicle;

the calibrating module is used for generating a bird's-eye view according to the pictures shot by the plurality of cameras, and carrying out joint calibration on the bird's-eye view to obtain first vehicle position information in the bird's-eye view;

the feature extraction module is used for inputting the shooting picture of each camera into the aerial view feature extraction model to obtain aerial view features;

the parking space information acquisition module is used for acquiring second vehicle position information according to the aerial view characteristics;

the adjusting module is used for adjusting the aerial view feature extraction model according to the first vehicle position information and the second vehicle position information to obtain a trained aerial view feature extraction model;

and the parking space detection module is used for carrying out parking space detection according to the trained aerial view feature extraction model.

In a third aspect, an electronic device provided in an embodiment of the present application includes: a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method according to any one of the first aspects when the computer program is executed.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium having instructions stored thereon, which when executed on a computer, cause the computer to perform the method according to any of the first aspects.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flow chart of a parking space detection method based on a camera picture according to an embodiment of the present application;

fig. 2 is a schematic diagram of an installation position and a view angle of a camera view angle according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of a parking space detection device based on a camera frame according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance.

Example 1

Referring to fig. 1, an embodiment of the present application provides a parking space detection method based on a camera image, including:

s101: acquiring shooting pictures of a plurality of cameras on a vehicle;

s102: generating a bird's-eye view according to the pictures shot by the plurality of cameras, and carrying out joint calibration on the bird's-eye view to obtain first vehicle position information in the bird's-eye view;

s103: inputting the shooting picture of each camera into a bird's-eye view feature extraction model to obtain bird's-eye view features;

s104: inputting the aerial view features into a parking space detection model to obtain second vehicle position information;

s105: adjusting the aerial view feature extraction model and the parking space detection model according to the first vehicle position information and the second vehicle position information to obtain a trained aerial view feature extraction model and a trained parking space detection model;

s106: and carrying out parking space detection according to the trained aerial view feature extraction model and the trained parking space detection model.

In the implementation process, unlike the prior art, the embodiment of the application only constructs the aerial view feature extraction model, extracts the aerial view feature directly based on the aerial view feature extraction model, does not need to synthesize the aerial view in advance when the parking space is detected, does not need to input the shooting picture of a single camera into the parking space detection model for detection, solves the problem of inaccurate identification caused by limited visual angle of the single camera, solves the problem of detection delay caused by synthesizing the aerial view, and improves the speed of parking space detection.

In one possible embodiment, the bird's eye view feature extraction model includes: a first feature extraction model; s103 includes: inputting the shooting picture of each camera into a first feature extraction model to obtain a first feature map of each camera under a plurality of scales; and acquiring aerial view features according to the first feature map of each camera under a plurality of scales.

Preferably, the first feature extraction model is resnet18.

In one possible embodiment, the bird's eye view feature extraction model includes: a first feature fusion model; the method for acquiring the aerial view features according to the first feature map of each camera under a plurality of scales comprises the following steps: inputting a first feature map of each camera under a plurality of scales into a first feature fusion model to obtain a second feature map corresponding to each camera;

Preferably, the first feature fusion model is a FPN network.

In one possible embodiment, the parking space detection model includes: a second feature extraction model; inputting the aerial view features into a parking space detection model to obtain second vehicle position information, wherein the method comprises the following steps: inputting the aerial view features into a second feature extraction model to obtain first aerial view features with multiple scales; fusing the first aerial view features of the multiple scales to obtain second aerial view features; and acquiring second vehicle position information according to the second aerial view characteristic.

Preferably, the second feature extraction model is resnet18.

In a possible implementation manner, the step of acquiring the second vehicle location information according to the second aerial view feature includes: generating a task dimension according to the second vehicle position information; reducing the dimension of the second aerial view feature according to the task dimension to obtain a reduced dimension second aerial view feature; and acquiring the second vehicle position information according to the second aerial view characteristic after the dimension reduction.

The second vehicle position information of the application is 17 dimensions and is provided with 17 channels, and tasks corresponding to the 17 channels are respectively 1-4 parking space corner points, and an abscissa and an ordinate of 1-4 parking space corner point offset; the berthing performance of 1-4 parking stall angular points and the clustering of the parking stall angular points.

In one possible embodiment, the parking space information includes: a plurality of corner coordinates of the parking space; the second aerial view features after the dimension reduction comprise second aerial view sub-features of a plurality of channels; the plurality of channels of the second bird's eye view feature includes: the parking space comprises a first channel corresponding to the coordinates of a plurality of angular points of the parking space and a second channel for clustering; the step of obtaining the second vehicle position information according to the second aerial view feature after the dimension reduction includes: analyzing the second aerial view sub-feature of the first channel by using a hetmap algorithm to obtain a plurality of initial angular point coordinates; acquiring a plurality of parking space characteristic values corresponding to the initial angular point coordinates on the second channel according to the initial angular point coordinates; clustering the parking space characteristic values to obtain a clustering result; and generating a plurality of corner coordinates of the parking space according to the clustering result.

For example, local maximum value calculation is performed on the second aerial view feature corresponding to the first channel of the 1-4 parking space corner points, feature points larger than a preset value are obtained, position index coordinates (such as ((x 1, y 1), (x 2, y 2) and …) corresponding to the points larger than the preset value are obtained, then the parking space feature values are obtained on the second aerial view feature of the second channel according to the index coordinates, the plurality of parking space feature values are clustered by using a CFDP ((Clustering byfast search and find of density peaksd) clustering algorithm to obtain a clustering result, and a plurality of corner coordinates of the parking space are generated according to the clustering result.

In one possible embodiment, the first channel comprises: the parking space angle sensor comprises a plurality of first sub-channels, wherein each first sub-channel is used for acquiring one angle point coordinate of the parking space; the step of generating the coordinates of the plurality of corner points of the parking space according to the clustering result comprises the following steps: judging whether channels to which a plurality of initial angular point coordinates corresponding to the parking space characteristic values in the clustering result belong comprise all the first sub-channels or not; and if so, screening the initial angular point coordinates corresponding to the parking space characteristic values of the clustering result to obtain a plurality of angular point coordinates of the parking space.

Each clustering result comprises initial corner coordinates of one parking space, and finally, only 4 corner points on 4 parking space corner channels are formed, and each channel is only one. If there are multiple spaces, there are multiple clusters.

Illustratively, the first channel has four first sub-channels; the clustering result comprises the following steps: the method comprises the steps of determining a first parking place characteristic value, a second parking place characteristic value and a third parking place characteristic value, wherein the first parking place characteristic value is determined by index coordinates in a first sub-channel, the second parking place characteristic value is determined by index coordinates in a second first sub-channel, the third parking place characteristic value is determined by index coordinates in a third first sub-channel, and a clustering result lacks a parking place characteristic value determined by index coordinates corresponding to a fourth first sub-channel, so that a channel to which a plurality of initial corner coordinates corresponding to the parking place characteristic value in the clustering result belong is determined to not comprise the four first sub-channels, and the clustering result is directly excluded and is not used as a basis for generating four corner coordinates of a parking place.

In a possible implementation manner, the step of screening the initial angular point coordinates corresponding to the parking space feature values of the clustering result to obtain a plurality of angular point coordinates of the parking space includes:

For example, for the clustering result of the parking space feature values with more than 1 channel number, each channel with a plurality of parking space feature values is processed, and other parking space feature values with more than 1 parking space feature value in the channel are removed, so that 4 values of the clustering result correspond to the channels corresponding to 1-4 corner points, and the number of the parking spaces and the coordinate position information of four corner points of each parking space are obtained.

In a possible implementation manner, the step of screening the plurality of initial corner coordinates belonging to the same first sub-channel in the clustering result to obtain a screened initial corner coordinate corresponding to the same first sub-channel includes:

In one possible implementation, the second vehicle location information includes: the parking space berthing performance; the plurality of channels of the second bird's eye view feature includes: a third channel corresponding to the berthing property of the parking space;

the step of obtaining the second vehicle position information according to the second aerial view feature after the dimension reduction includes: determining the poisability corresponding to each feature point of the third channel; determining the poisability of the coordinates of a plurality of angular points of the parking space according to the poisability corresponding to each characteristic point; and determining the berthing property of the parking space according to the berthing property of the plurality of corner coordinates.

Illustratively, on the second aerial view sub-feature of the berthing corresponding channel, decoding is performed on the corresponding coordinates according to the calculated coordinates of the corner points of the vehicle to calculate the berthing of the 1-4 corner points.

In one possible implementation, the second vehicle location information includes: offset of a plurality of corner points of the parking space; the plurality of channels of the second bird's eye view feature includes: a fourth channel corresponding to the offset of the plurality of corner points of the parking space; the step of obtaining the second vehicle position information according to the second aerial view feature after the dimension reduction includes: determining offset characteristic values corresponding to a plurality of angular point coordinates of the parking space in the second aerial view sub-characteristic of the fourth channel; and taking the offset characteristic values corresponding to the coordinates of the plurality of corner points of the parking space as the offset of the plurality of corner points of the parking space.

Illustratively, on the second aerial view sub-feature of the channel corresponding to the offset, decoding is performed on the corresponding coordinates according to the obtained parking space corner coordinates to obtain the offset of 1-4 corner points. The offset includes: the fourth channel is provided with eight fourth sub-channels, and the fourth sub-channels are respectively used for acquiring the abscissa offset value and the ordinate offset value of the four parking space corner points.

And obtaining the parking space angular point coordinates based on the offset of the parking space angular point coordinates and the vehicle angular point coordinates.

The step of adjusting the aerial view feature extraction model and the parking space detection model according to the first vehicle position information and the second vehicle position information to obtain a trained aerial view feature extraction model and a trained parking space detection model comprises the following steps: acquiring a bird's-eye view and the scaling of the bird's-eye view characteristics after the dimension reduction; scaling the first vehicle position information according to the scaling ratio to obtain scaled first vehicle position information; acquiring a loss value according to the scaled first vehicle position information and the scaled second vehicle position information; and adjusting the aerial view characteristic extraction model and the parking space detection model according to the loss value to obtain a trained aerial view characteristic extraction model and a trained parking space detection model.

Illustratively, the first location information also includes: parking space angular point coordinates, offset of the parking space angular point coordinates and berthing attributes.

On the second aerial view sub-feature of the channel corresponding to the 1-4 parking space angular points, the joint labeling is scaled according to the scaling of the sizes of the original image and the second aerial view sub-feature of the corresponding channel, for example, the scaling is 4, the coordinates (403,406) are labeled, the coordinates (100, 101) of the joint labeling point coordinates on the aerial view feature of the corresponding channel are obtained, the characteristic points are assigned to be 1 according to the parking space numbers corresponding to the joint labeling point coordinates, and the rest are all 0. And carrying out Gaussian FocalLoss calculation on the re-marked second aerial view sub-feature and the unremarked second aerial view sub-feature, and multiplying by a mask, wherein the mask is a matrix with value 1 on the position coordinates with angular points and the rest of the matrix being 0, namely only the coordinate positions with the angular points of the parking space are calculated, the rest of the matrix is not calculated, and finally the number of the whole parking space is divided to obtain a loss value corresponding to Gaussian FocalLoss. The four corner points are the same. For the offset of 1-4 angular points, the offset of each angular point is respectively placed on the corresponding coordinate positions of the angular points on two channels (the offset of each angular point is divided into the offset of the abscissa and the offset of the ordinate), meanwhile, mask is used, then L1Loss is used for calculating a Loss value, the number of the whole parking spaces is divided, the Loss value corresponding to the offset value is obtained, the poisable attribute is similar to the offset of 1-4 angular points, only the value of 10 or 100 is assigned on the corresponding coordinate positions, 100 represents poisable, 10 is not poisable, and GaussianFocalloss calculation is carried out. And calculating the corresponding parking space corner clustering channels by using a binary_cross sentropy method.

In one possible implementation manner, the step of inputting the shot image of each camera into the first feature extraction model to obtain a first feature map of each camera at multiple scales includes: acquiring the installation position of each camera; inputting the shooting picture of each camera into a first feature extraction model to obtain a first initial feature map corresponding to a preset scale output by the first feature extraction model; determining a second initial feature map of a second preset scale corresponding to the installation position of each camera from the first initial feature maps corresponding to all the first preset scales, and taking the second initial feature map of the second preset scale as the first feature map of each camera under a plurality of scales.

In the embodiment of the application, the resnet18 is adopted as a first feature extraction model, the first feature extraction model generates four scale feature graphs, namely, an original feature graph 1/4, an original feature graph 1/8, an original feature graph 1/16 and an original feature graph 1/32, and one or more scale first initial feature graphs are selected from the 4 scale feature graphs of each camera to serve as the first feature graphs based on the installation position of each camera.

In a possible implementation manner, determining a first initial feature map of a preset scale corresponding to an installation position of each camera from first initial feature maps of all preset scales, and obtaining a first feature map of each camera under multiple scales, including:

and if the installation position of the camera is not matched with the advancing direction of the vehicle, determining a first initial characteristic diagram corresponding to a preset second preset scale as a first characteristic diagram of the camera.

Referring to fig. 2, a schematic diagram of the installation positions of the cameras and the viewing angles of different cameras of the vehicle is provided in the embodiment of the application.

When the current foreground direction of the vehicle is left turning, for the front right camera and the rear right camera, only acquiring a first initial feature map with the scale of the original feature map 1/16 as a first feature map; for other cameras, the first initial feature map for all scales is taken as the feature map. The right turn direction is the same and will not be described again here.

When feature fusion is carried out, for cameras which do not comprise feature images of all scales in the first initial feature image, generating a unit matrix of the missing scales and inputting the unit matrix into the FPN network. For example, when the current foreground direction of the vehicle is left turn, for the front right camera and the rear right camera, only the first initial feature map with the scale of the original feature map 1/16 is obtained as the first feature map; in the process of fusing the feature maps with different scales, generating an original feature map 1/4, an original feature map 1/8 and an identity matrix of the original feature map 1/32, inputting the identity matrix into the FPN network, and generating a second feature map of the front right camera and the rear right camera.

Example 2

Referring to fig. 3, an embodiment of the present application provides a parking space detection device based on a camera image, including;

a photographed-image acquisition module 1 for acquiring photographed images of a plurality of cameras on a vehicle;

the calibrating module 2 is used for generating a bird's-eye view according to the pictures shot by the plurality of cameras, and carrying out joint calibration on the bird's-eye view to obtain first vehicle position information in the bird's-eye view;

the feature extraction module 3 is used for inputting the shooting picture of each camera into a bird's-eye view feature extraction model to obtain bird's-eye view features;

the parking space information acquisition module 4 is used for acquiring second vehicle position information according to the aerial view characteristics;

the adjusting module 5 is used for adjusting the aerial view feature extraction model according to the first vehicle position information and the second vehicle position information to obtain a trained aerial view feature extraction model;

And the parking space detection module 6 is used for carrying out parking space detection according to the trained aerial view feature extraction model.

In one possible embodiment, the bird's eye view feature extraction model includes: a first feature extraction model; the feature extraction module 3 is further configured to input a shot picture of each camera into the first feature extraction model, so as to obtain a first feature map of each camera under multiple scales;

and acquiring aerial view features according to the first feature map of each camera under a plurality of scales.

In a possible embodiment, the feature extraction module 3 is further configured to obtain the installation position of each camera; inputting the shooting picture of each camera into a first feature extraction model to obtain a first initial feature map corresponding to a preset scale output by the first feature extraction model; determining a second initial feature map of a second preset scale corresponding to the installation position of each camera from the first initial feature maps corresponding to all the first preset scales, and taking the second initial feature map of the second preset scale as the first feature map of each camera under a plurality of scales.

In one possible embodiment, the parking space information includes: a plurality of corner coordinates of the parking space;

The second aerial view features after the dimension reduction comprise second aerial view sub-features of a plurality of channels; the plurality of channels of the second bird's eye view feature includes: the parking space comprises a first channel corresponding to the coordinates of a plurality of angular points of the parking space and a second channel for clustering; the parking space information acquisition module 4 is further used for analyzing the second aerial view sub-features of the first channel by using a hetmap algorithm to obtain a plurality of initial angular point coordinates; acquiring a plurality of parking space characteristic values corresponding to the initial angular point coordinates on the second channel according to the initial angular point coordinates; clustering the parking space characteristic values to obtain a clustering result; and generating a plurality of corner coordinates of the parking space according to the clustering result.

In one possible embodiment, the first channel comprises: the parking space angle sensor comprises a plurality of first sub-channels, wherein each first sub-channel is used for acquiring one angle point coordinate of the parking space; the parking space information acquisition module 4 is further configured to determine whether channels to which a plurality of initial angular point coordinates corresponding to the parking space feature values in the clustering result belong include all the first sub-channels; and if so, screening the initial angular point coordinates corresponding to the parking space characteristic values of the clustering result to obtain a plurality of angular point coordinates of the parking space.

In a possible implementation manner, the parking space information obtaining module 4 is further configured to obtain a plurality of least square values according to the parking space feature values corresponding to the initial angular point coordinates belonging to the same first sub-channel in the clustering result and the parking space feature values corresponding to the other first sub-channels in the clustering result; and taking the initial angular point coordinate corresponding to the least square value in the least square values as the initial angular point coordinate corresponding to the same channel after screening.

In one possible implementation, the second vehicle location information includes: the parking space berthing performance; the plurality of channels of the second bird's eye view feature includes: a third channel corresponding to the berthing property of the parking space; the parking space information acquisition module 4 is further used for determining the berthing property corresponding to each feature point of the third channel; determining the poisability of the coordinates of a plurality of angular points of the parking space according to the poisability corresponding to each characteristic point; and determining the berthing property of the parking space according to the berthing property of the plurality of corner coordinates.

In one possible implementation, the second vehicle location information includes: offset of a plurality of corner points of the parking space; the plurality of channels of the second bird's eye view feature includes: a fourth channel corresponding to the offset of the plurality of corner points of the parking space; the parking space information acquisition module 4 is further configured to determine offset feature values corresponding to a plurality of corner coordinates of the parking space in the second aerial view sub-feature of the fourth channel; and taking the offset characteristic values corresponding to the coordinates of the plurality of corner points of the parking space as the offset of the plurality of corner points of the parking space.

In one possible implementation, the second vehicle location information includes: offset of a plurality of corner points of the parking space; the second vehicle location information includes: offset of a plurality of corner points of the parking space; the parking space information acquisition module 4 is further configured to determine offset feature values corresponding to a plurality of corner coordinates of the parking space in the second aerial view sub-feature of the fourth channel; and taking the offset characteristic values corresponding to the coordinates of the plurality of corner points of the parking space as the offset of the plurality of corner points of the parking space.

In a possible implementation manner, the parking space information obtaining module 4 is further configured to obtain a scaling ratio of the aerial view and the second aerial view feature after the dimension reduction; scaling the first vehicle position information according to the scaling ratio to obtain scaled first vehicle position information; acquiring a loss value according to the scaled first vehicle position information and the scaled second vehicle position information; and adjusting the aerial view characteristic extraction model and the parking space detection model according to the loss value to obtain a trained aerial view characteristic extraction model and a trained parking space detection model.

In a possible implementation manner, the parking space information obtaining module 4 is further configured to obtain the number of corner points with poisability in the plurality of corner points of the parking space; acquiring the number of corner points which do not have the poisability in the plurality of corner points of the parking space; judging whether the number of the corner points with the poisability is more than that of the corner points without the poisability; if yes, judging that the parking space has the berthability; if not, judging that the parking space does not have the berthability.

In a possible implementation manner, the parking space information acquisition module 4 is further used for acquiring the installation position of each camera; inputting the shooting picture of each camera into a first feature extraction model to obtain a first initial feature map corresponding to a preset scale output by the first feature extraction model; and determining a second initial feature map of a second preset scale corresponding to the installation position of each camera from the first initial feature maps corresponding to all the first preset scales, and taking the second initial feature map of the second preset scale as the first feature map of each camera under a plurality of scales.

In a possible implementation manner, the feature extraction module 3 is further configured to determine, as the first feature map of the camera, a first initial feature map corresponding to all first preset dimensions if the installation position of the camera matches the advancing direction of the vehicle; and if the installation position of the camera is not matched with the advancing direction of the vehicle, determining a first initial characteristic diagram corresponding to a preset second preset scale as a first characteristic diagram of the camera.

In one possible embodiment, the bird's eye view feature extraction model includes: a first feature fusion model; the feature extraction module 3 is further configured to input a first feature map of each camera under multiple scales to the first feature fusion model, so as to obtain a second feature map corresponding to each camera; and fusing the second characteristic map with the internal and external parameters of the camera to obtain the aerial view characteristic.

In one possible embodiment, the parking space detection model includes: a second feature extraction model; the parking space information acquisition module 4 is further used for inputting the aerial view features into a second feature extraction model to obtain first aerial view features with multiple scales; fusing the first aerial view features of the multiple scales to obtain second aerial view features; and acquiring second vehicle position information according to the second aerial view characteristic.

The application further provides an electronic device, please refer to fig. 4, and fig. 4 is a block diagram of an electronic device according to an embodiment of the application. The electronic device may include a processor 41, a communication interface 42, a memory 43, and at least one communication bus 44. Wherein the communication bus 44 is used to enable direct connection communication of these components. The communication interface 42 of the electronic device in the embodiment of the present application is used for performing signaling or data communication with other node devices. The processor 41 may be an integrated circuit chip with signal processing capabilities.

The processor 41 may be a general-purpose processor including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but may also be a Digital Signal Processor (DSP), application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. The general purpose processor may be a microprocessor or the processor 41 may be any conventional processor or the like.

The Memory 43 may be, but is not limited to, random access Memory (Random Access Memory, RAM), read Only Memory (ROM), programmable Read Only Memory (Programmable Read-OnlyMemory, PROM), erasable Read Only Memory (Erasable ProgrammableRead-Only Memory, EPROM), electrically erasable Read Only Memory (ElectricErasable Programmable Read-Only Memory, EEPROM), etc. The memory 43 has stored therein computer readable instructions which, when executed by the processor 41, can cause the electronic device to perform the steps involved in the above-described method embodiments.

Optionally, the electronic device may further include a storage controller, an input-output unit.

The memory 43, the memory controller, the processor 41, the peripheral interface, and the input/output unit are electrically connected directly or indirectly to each other to realize data transmission or interaction. For example, the components may be electrically coupled to each other via one or more communication buses 44. The processor 41 is arranged to execute executable modules stored in the memory 43, such as software functional modules or computer programs comprised by the electronic device.

The input-output unit is used for providing the user with the creation task and creating the starting selectable period or the preset execution time for the task so as to realize the interaction between the user and the server. The input/output unit may be, but is not limited to, a mouse, a keyboard, and the like.

It will be appreciated that the configuration shown in fig. 4 is merely illustrative, and that the electronic device may also include more or fewer components than shown in fig. 4, or have a different configuration than shown in fig. 4. The components shown in fig. 4 may be implemented in hardware, software, or a combination thereof.

The embodiment of the application further provides a computer readable storage medium, on which instructions are stored, and when the instructions run on a computer, the computer program is executed by a processor to implement the method of the method embodiment, so that repetition is avoided, and no further description is given here.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners as well. The apparatus embodiments described above are merely illustrative, for example, flow diagrams and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In addition, the functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.

The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored on a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only memory (ROM), a random access memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The above is only an example of the present application, and is not intended to limit the scope of the present application, and various modifications and variations will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application. It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.

The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes or substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.

Claims

1. The parking space detection method based on the camera picture is characterized by comprising the following steps of:

acquiring shooting pictures of a plurality of cameras on a vehicle;

generating a bird's eye view according to the shooting pictures of the cameras;

performing joint calibration on the aerial view to obtain first vehicle position information in the aerial view;

2. The camera-screen-based parking space detection method according to claim 1, wherein the bird's eye view feature extraction model comprises: a first feature extraction model;

3. The camera-screen-based parking space detection method according to claim 2, wherein the bird's eye view feature extraction model includes: a first feature fusion model;

4. The camera frame-based parking space detection method according to claim 1, wherein the parking space detection model comprises: a second feature extraction model;

5. The method for detecting a parking space based on a camera frame according to claim 4, wherein the step of acquiring the second vehicle position information according to the second bird's eye view feature comprises:

6. The camera frame-based parking space detection method according to claim 5, wherein the parking space information includes: a plurality of corner coordinates of the parking space;

7. The camera frame based parking spot detection method according to claim 6, wherein the first channel comprises: the parking space angle sensor comprises a plurality of first sub-channels, wherein each first sub-channel is used for acquiring one angle point coordinate of the parking space;

8. The method for detecting a parking space based on a camera picture according to claim 7, wherein the step of screening the initial corner coordinates corresponding to the feature values of the parking space of the clustering result to obtain a plurality of corner coordinates of the parking space comprises:

9. The method for detecting a parking space based on a camera picture according to claim 8, wherein the step of screening the plurality of initial corner coordinates belonging to the same first sub-channel in the clustering result to obtain a screened initial corner coordinate corresponding to the same first sub-channel comprises the steps of:

and taking the initial angular point coordinate corresponding to the least square value in the plurality of least square values as the initial angular point coordinate corresponding to the same first sub-channel after screening.

10. The parking spot detection method according to claim 9, wherein the second vehicle spot information includes: the parking space berthing performance; the plurality of channels of the second bird's eye view feature includes: a third channel corresponding to the berthing property of the parking space;

11. The parking spot detection method according to claim 9, wherein the second vehicle spot information includes: offset of a plurality of corner points of the parking space;

12. The method for detecting a parking space based on a camera frame according to claim 5, wherein the step of adjusting the aerial view feature extraction model and the parking space detection model according to the first and second vehicle position information to obtain a trained aerial view feature extraction model and a trained parking space detection model comprises:

13. The camera frame-based parking space detection method according to claim 10, wherein the step of determining the berthability of the parking space according to the berthability of the plurality of corner coordinates comprises:

if yes, judging that the parking space has the berthability;

if not, judging that the parking space does not have the berthability.

14. The parking space detection method based on the camera images according to claim 2, wherein the step of inputting the photographed image of each camera into the first feature extraction model to obtain the first feature map of each camera at a plurality of scales comprises:

Acquiring the installation position of each camera;

15. The parking space detection method based on the camera frame according to claim 14, wherein the step of determining the first initial feature map of the second preset scale corresponding to the installation position of each camera from the first initial feature maps corresponding to all the first preset scales to obtain the first feature map of each camera under multiple scales includes:

16. Parking stall detection device based on camera picture, characterized by, include:

the parking space information acquisition module is used for inputting the aerial view features into a parking space detection model to obtain second vehicle position information;

the adjustment module is used for adjusting the aerial view feature extraction model and the parking space detection model according to the first vehicle position information and the second vehicle position information to obtain a trained aerial view feature extraction model and a trained parking space detection model;

and the parking space detection module is used for carrying out parking space detection according to the trained aerial view feature extraction model and the trained parking space detection model.

17. A vehicle, characterized by comprising: the parking spot detection device of claim 16.

18. An electronic device, comprising: memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method according to any one of claims 1-15 when the computer program is executed.

19. A computer readable storage medium having instructions stored thereon which, when run on a computer, cause the computer to perform the method of any of claims 1-15.