CN110889829A - A monocular ranging method based on fisheye lens - Google Patents

Abstract

The invention relates to a target object ranging method based on a monocular fisheye camera, comprising the following steps: 1) using the fisheye lens to take pictures to obtain a distorted image; 2) sending the distorted image into a trained neural network to obtain the target The bounding frame coordinates of the object, and frame the target object in the image according to the obtained frame frame coordinates; 3) Perform image processing on the framed image area to obtain the contour map of the target object; 4) Use feature point detection on the contour map algorithm to obtain the target feature points; 5) According to the distorted image correction formula, the corrected feature point coordinates are obtained; 6) According to the known distances between the feature points of objects in reality corresponding to the feature points in the image, use the coordinate conversion formula in A mathematical equation is established between the image and the world coordinate system, and the distance between the fisheye camera and the target object is obtained by solving. Compared with the traditional method, the method has the advantages of low cost, large recognition range, fast detection speed and high accuracy.

Description

A monocular ranging method based on fisheye lens

technical field

The invention relates to a monocular ranging method based on a fisheye lens, belonging to the fields of optics and computer vision.

Background technique

Vision sensors have attracted a lot of attention in recent years, and vision sensor-based ranging methods have also become a research hotspot because they can collect a wide range of environmental information, and are inexpensive and easy to use. According to the number of visual sensors used, visual ranging methods can be mainly divided into three types: monocular ranging, binocular ranging and multi-eye ranging.

At present, LiDAR is usually required to obtain high-precision depth information from a target. However, due to its high price, it is still mainly in the stage of technical research and testing, and it is still far from large-scale market applications. In addition, with the rapid development of artificial intelligence in recent years, vision has gradually become the focus of research, but some shortcomings have also been found, such as the use of binocular ranging technology is limited by the baseline, which leads to the size of the equipment and the load capacity of the transportation platform. The coordination between them is insufficient; the depth estimation based on RGB-D has a short range, which is difficult to apply in practice, and is greatly affected by environmental spatial changes, so the outdoor performance is not ideal; the ranging method based on ordinary pinhole cameras, Because the visible range of the camera is small, it cannot obtain as much visual information as possible, and the recognition efficiency is low.

Fisheye camera not only has the advantages of low price and small size, but also has a large viewing angle that can reach or even exceed 180 degrees. Only one can reach the shooting range of 2-3 ordinary cameras, and the photos taken are rich in information. Therefore, many shortcomings of the above sensors can be effectively overcome. Therefore, obtaining depth information using a single-eye fisheye camera has gradually become one of the research focuses in the field of computer vision.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a monocular ranging method based on a fisheye lens.

In order to solve the above technical problems, the technical solution of the present invention is to provide a monocular ranging method based on a fisheye lens, and the method includes the following steps:

1) Use a fisheye lens to take a photo to obtain a distorted image;

2) Send the distorted image into the trained neural network to obtain the bounding box coordinates of the target object, and frame the target object in the image according to the obtained frame coordinates;

3) Image processing is performed on the framed image area to obtain the contour map of the target object;

4) Use the feature point detection algorithm on the contour map to obtain the target feature points;

5) According to the distorted image correction formula, the corrected feature point coordinates are obtained;

6) According to the known distances between the feature points of objects in reality corresponding to the feature points in the image, use the coordinate conversion formula to establish a mathematical equation between the image and the world coordinate system, and solve to obtain the distance between the fisheye camera and the target object.

In the step 1), the angle of view of the fisheye lens is 180 degrees, and the center of the target object and the optical center of the fisheye lens are on the same horizontal plane.

In the step 2), the algorithm used for the distorted image is a target detection algorithm based on deep learning, and the algorithm includes the following parts:

a) Using the sub-channel convolution method in MobileNet V2 to extract the high-level features of the distorted images in the training data set;

b) Use FPN to perform feature fusion on the features screened by different convolutional layers, and send the fused features into the classification sub-network and the positioning sub-network to obtain the classification and positioning errors;

c) Use the loss function to learn and train the deep neural network, and obtain the optimized model after training. The loss function is:

是当预设框为正预设框时值为1，否则为0，l_i和p_i分别为位置偏移和标签；L_reg表示位置损失函数，L_cls表示分类损失函数。where x is the input image, θ is the model parameter, m is the number of preset frames, α and β are the weights that balance the localization and classification losses,

is 1 when the preset frame is a positive preset frame, otherwise 0, l _i and p _i are the position offset and label, respectively; L _reg represents the position loss function, and L _cls represents the classification loss function.

In the step 3), the contour recognition of the image includes the following steps:

a) Read the image to be detected, and perform grayscale processing on the image, that is,

b) Set a threshold to binarize the grayscale image;

c) use the Canny operator to detect the contour of the grayscale image, obtain the contour points and store them in the array, and use contour tracking to the contour to obtain a continuous contour point set P(i);

d) Use OpenCV to draw the outline in the black and white image of the original size.

In the step 4), the feature point detection method based on the contour mainly solves the feature point by an indirect method. First, decompose the target object contour point set P(i) into the X and Y coordinate axes to obtain two one-dimensional discrete curves X(i) and Y(i), and then solve the curves X(i) and Y( i) The concave rate of each point:

Among them, L is the interpolation step size; after obtaining the concave rate, multi-scale is used to improve the detection accuracy and the robustness of the algorithm to noise at a small scale, and the feature points of the target object can be obtained by using an adaptive threshold.

In the step 5), the correction of the distorted image needs to adopt the distortion coefficient matrix K=[K ₁ , K ₂ , . . . , K ₅ ] obtained by Zhang Zhengyou’s calibration method;

Zhang Zhengyou's calibration method needs to use a printed checkerboard (the black and white spacing is known), and paste it on a flat plate, and then take several pictures (10-20 pictures) of the checkerboard, and then use the Harris feature to detect in the picture Finally, through the analytical solution estimation method, the internal parameters and distortion coefficients of the fisheye lens can be calculated.

In the step 6), the conversion expression between the pixel coordinate system and the world coordinate system is:

为世界坐标系的坐标。Among them, (μ, v) is the coordinates of the pixel coordinate system, dx, dy are the size of the image unit pixel in the horizontal and vertical directions, (c _x , _cy ) is the image center point, f is the focal length of the fisheye lens, R is the rotation matrix, t is the transfer vector,

are the coordinates of the world coordinate system.

Compared with the traditional technology, the present invention has the following advantages:

1. Low cost: the present invention only needs to use a monocular camera and an embedded device at least to complete the whole process of ranging;

Second, the recognition range is large: the present invention adopts a large and wide-angle fisheye lens. Compared with the traditional algorithm, the detection range is wider and the efficiency is higher under the same number of lenses;

3. Fast detection speed and high accuracy: the present invention uses a lightweight deep neural network, which is not only faster but also more accurate compared with the traditional target detection algorithm; the present invention uses an indirect method to solve the feature points of the target object, which ensures accurate In the case of high rate, feature points can be identified more quickly.

Description of drawings

Fig. 1 is the flow chart of the monocular ranging method based on fisheye lens of the present invention;

Fig. 2 is the target detection network structure diagram of the present invention;

Fig. 3 is the imaging principle diagram of the fisheye lens of the present invention;

FIG. 4 is a schematic diagram of distance measurement for establishing mathematical equations between coordinate systems in the present invention.

Detailed ways

In order to illustrate the advantages of the present invention and the implementation process of the solution more clearly, the present invention will be further described below with reference to specific embodiments and accompanying drawings. It should be understood that the embodiments explained below are not used to limit the implementation conditions of the present invention, but are only used to explain the present invention.

The present invention provides a target detection and ranging method based on a monocular fisheye lens, which uses a single fisheye lens as a detection sensor to collect images. The specific implementation process is shown in Figure 1, and the method includes the following steps:

Step 1. Take a photo with a fisheye lens to get a distorted image:

The angle of view of the fisheye lens is 180 degrees, and the center of the target object is on the same horizontal plane as the optical center of the fisheye lens. Depending on the application, you need to use a fisheye lens to take pictures in three places:

1) Take 10-20 black and white chessboard photos with a spacing of 20mm using a fisheye lens, so as to use the Zhang Zhengyou calibration method for calibration, and obtain the fisheye lens distortion coefficient K=[k ₁ , k ₂ , k ₃ , k ₄ , k ₅ ] and internal parameters [dx, dy, c _x , c _y ];

2) Make the optical center of the fisheye lens and the center of the target object at the same height, and then take about 1000 photos from different angles and different distances and different scenes to train the image detection deep network model. The structure of the deep network model is shown in Figure 2:

The deep convolution part adopts the sub-channel convolution method in MobileNet V2 to extract the high-level features of the distorted images in the training dataset, that is, feature maps. Compared with traditional convolution, this method separates the regions and channels of the image, reduces the amount of parameters, and calculates faster;

Use FPN to perform feature fusion on the feature maps screened by different convolutional layers, and send the fused features into the classification sub-network and the positioning sub-network to obtain the classification and positioning errors;

The loss function is used for deep neural network learning and training, and the loss function is:

是当预设框为正预设框时值为1，否则为0，l_i和p_i分别为位置偏移和标签；L_reg表示位置损失函数，L_cls表示分类损失函数。where x is the input image, θ is the model parameter, m is the number of preset frames, α and β are the weights that balance the localization and classification losses,

is 1 when the preset frame is a positive preset frame, otherwise 0, li and p _i are the position offset and label _, respectively; L _reg represents the position loss function, and L _cls represents the classification loss function.

3) Take a picture of the target directly, and use this method to measure the distance of the target.

Step 2: Send the distorted image to the trained neural network to obtain the bounding frame coordinates of the target object, and according to the obtained frame frame coordinates:

The frame coordinates are expressed in the form of (x, y, w, h), where (x, y) are the coordinates of the upper left corner of the bounding box, and (w, h) are the width and height of the bounding box. Frame the target object.

Step 3: Perform image processing on the framed image area to obtain the contour map of the target object:

1) Read the image to be detected, and perform grayscale processing on the image, that is,

2) Set a threshold to binarize the grayscale image;

3) According to the coordinates of the frame obtained in step 2, set the pixel value of the area outside the frame to 0, then use the Canny operator to detect the contour of the grayscale image, obtain the contour points and store them in the array, and use contour tracking for the contour to obtain continuous contour points. set P(i);

4) Use OpenCV to draw the outline in the black and white image of the original image size.

Step 4. Use the feature point detection algorithm on the contour map to obtain the target feature points:

The feature point detection method based on contour mainly solves the feature point by indirect method. First, decompose the target object contour point set P(i) into the X and Y coordinate axes to obtain two one-dimensional discrete curves X(i) and Y(i), and then solve the curves X(i) and Y( i) The concave rate of each point:

Among them, L is the interpolation step size; after obtaining the concave rate, multi-scale is used to improve the detection accuracy and the robustness of the algorithm to noise at a small scale, and the feature points of the target object can be obtained by using an adaptive threshold.

Step 5. Obtain the corrected feature point coordinates according to the distorted image correction formula:

The imaging principle of the fisheye lens is shown in Figure 3. In reality, the target P is imaged at the point p' of the image plane through the optical center of the fisheye lens, and the length of O'p' is set as r', combined with the distortion coefficient K, as follows:

r′(θ)=k ₁ θ+k ₂ θ ³ +…+k ₅ θ ⁹

大小，进而求得针孔模型下正常点p坐标：The angle of incidence θ can be obtained, so Op length r = ftan θ. The coordinates (x', y') of p' are also known, so we can get

size, and then obtain the p-coordinate of the normal point under the pinhole model:

Step 6. According to the known distances between the feature points of objects in reality corresponding to the feature points in the image, use the coordinate conversion formula to establish a mathematical equation between the image and the world coordinate system, and solve to obtain the distance between the fisheye camera and the target object:

The conversion expression between the known pixel coordinate system and the world coordinate system is:

Assuming that the world coordinate system is located at the position shown in Figure 4, then R=I, T=[0 0 d] ^T , the above formula can be transformed into:

d can be obtained by synthesizing the coordinates of the known feature points and the length between them.

则摄像机光心到目标质心的距离

Select the midpoint of the two feature points as the target centroid, then the distance from the centroid to the origin in the world coordinate system

Then the distance from the camera optical center to the target mass center

Claims (8)

1. a monocular ranging method based on fisheye lens, is characterized in that, comprises the following steps: Step 1. Use a fisheye lens to take a photo to obtain a distorted image; Step 2, sending the distorted image into the trained neural network to obtain the outer frame coordinates of the target object, and frame the target object in the image according to the obtained frame frame coordinates; Step 3: Perform image processing on the framed image area to obtain a contour map of the target object; Step 4: Use a feature point detection algorithm on the contour map to obtain target feature points; Step 5. Obtain the corrected feature point coordinates according to the distorted image correction formula; Step 6: According to the known distance between the feature points of the real object corresponding to the feature points in the image, use the coordinate conversion formula to establish a mathematical equation between the image and the world coordinate system, and solve to obtain the distance between the fisheye camera and the target object. 2. a kind of monocular ranging method based on fisheye lens as claimed in claim 1, is characterized in that, in described step 1, fisheye lens angle of view is 180 degrees, the center of target object and fisheye lens optical center at the same level. 3. a kind of monocular ranging method based on fisheye lens as claimed in claim 1 is characterized in that, in described step 2, the algorithm used to distorted image is the target detection algorithm based on deep learning, this algorithm Includes the following parts: Using the sub-channel convolution method in MobileNet V2 to extract the high-level features of the distorted images in the training dataset; Use FPN to perform feature fusion on the features screened by different convolutional layers, and send the fused features into the classification sub-network and the positioning sub-network to obtain the classification and positioning errors; The loss optimization function is used for deep neural network learning and training, and the optimized model after training is obtained. The loss function is:

where x is the input image, θ is the model parameter, m is the number of preset frames, α and β are the weights that balance the localization and classification losses,

is 1 when the preset frame is a positive preset frame, otherwise 0, l _i and p _i are the position offset and label, respectively; L _reg represents the position loss function, and L _cls represents the classification loss function. 4. a kind of monocular ranging method based on fisheye lens as claimed in claim 1, is characterized in that, in described step 3, carries out outline recognition to the image and comprises the following steps: Read the image to be detected, and perform grayscale processing on the image, that is,

Set the threshold to binarize the grayscale image; The Canny operator is used to detect the contour of the grayscale image, and the contour points are obtained and stored in the array, and contour tracking is used for the contour to obtain a continuous contour point set P(i); Use OpenCV to draw the contours in the original size black and white image. 5. a kind of monocular ranging method based on fisheye lens as claimed in claim 1 is characterized in that, in described step 4, the feature point detection method based on contour is to solve feature point by indirect method, its Proceed as follows: First, decompose the target object contour point set P(i) into the X and Y coordinate axes to obtain two one-dimensional discrete curves X(i) and Y(i), and then solve the curves X(i) and Y( i) Curvature of each point:

The upper part is the interpolation step size; after the curvature is obtained, the detection accuracy and the robustness of the algorithm to noise are improved by using multi-scale in a small scale, and the feature points of the target object are obtained by using an adaptive threshold. 6. a kind of monocular ranging method based on fisheye lens as claimed in claim 1 is characterized in that, in described step 5, the correction of distorted image adopts the distortion coefficient matrix K=[ K ₁ , K ₂ , ..., K ₅ ]; Zhang Zhengyou calibration method: use a printed checkerboard (the black and white spacing is known), and paste it on a flat plate, then take 10-20 pictures of the checkerboard, and then use the Harris feature to detect feature points in the picture, and finally pass The analytical solution estimation method calculates the internal parameters and distortion coefficients of the fisheye lens. 7. a kind of monocular ranging method based on fisheye lens as claimed in claim 1 is characterized in that, in described step 6, the conversion expression between pixel coordinate system and world coordinate system is:

Among them, (μ, v) is the coordinates of the pixel coordinate system, dx, dy are the size of the image unit pixel in the horizontal and vertical directions, (c _x , _cy ) is the image center point, f is the focal length of the fisheye lens, R is the rotation matrix, t is the transfer vector,

are the coordinates of the world coordinate system. 8 . The monocular ranging method based on a fisheye lens according to claim 2 , wherein the target object is a cube. 9 .

Images