CN110222749B - A method for matching visible light image and infrared image - Google Patents

Abstract

The invention discloses a visible light image and infrared image matching method, which comprises the following steps: (1) respectively extracting maximum stable extremum regions from the visible light image and the infrared image; (2) respectively carrying out ellipse fitting on the maximum stable extremum regions of the visible light image and the infrared image, and normalizing the maximum stable extremum regions into a standard circular region; (3) establishing an FBP model, and describing texture information in the standard circle by using binary coding; (4) matching the codes of the regions by using Hamming distance. According to the visible light image and infrared image matching method, an FBP algorithm firstly extracts a most stable extremum region as a region to be matched, then the FBP algorithm is used for carrying out feature description on each region, finally a threshold value is set, and a Hamming distance is used for judging whether each region is matched or not. The experimental result proves that the method has good performance in positioning precision, algorithm speed, stability, flexibility and real-time property.

Description

A method for matching visible light image and infrared image

technical field

The invention relates to the technical field of image processing, in particular to a visible light and infrared image matching method based on MSER and improved LBP algorithm.

Background technique

In multi-mode image matching, infrared images have the advantages of outstanding detection ability at night, better anti-occlusion ability, and object temperature perception; visible light imaging can retain rich scene information. Compared with single-mode images, multi-mode images can describe the imaging characteristics of scenes or objects from different aspects, and the obtained information is more reliable and complementary. Multimode images are acquired by different types of sensors, such as a combination of visible light and infrared images, and a combination of visible light and synthetic aperture radar. The night-time detection ability, fog penetration ability, and anti-blocking ability of infrared sensors are typical complements with the preservation of rich detailed information of the scene in visible light imaging. Therefore, visible light and infrared imaging have become more common in multi-mode image matching. a combination of . Image matching is a method of seeking similar image targets through the analysis of the corresponding relationship, similarity and consistency of image content, features, structure, relationship, texture and grayscale, etc. For single-mode image matching, the current mainstream SIFT, Algorithms such as SURF and Harris can obtain highly robust results. For multi-mode image matching, due to the different working bands of imaging, common feature detection algorithms cannot correctly extract the same feature regions or feature points in multi-mode images.

A lot of research has been done at home and abroad on the subject of visible light and infrared image matching. The method of using visible light combined with infrared image matching was earlier based on the image contour extraction algorithm, using the LOG operator to extract the edge, using the chain code to describe the edge, or further extracting the corner points on the edge and using the invariant moment. In seconds, feature description-based matching is performed on lines or points at the end. In view of the poor effect of visible light combined with infrared image contour feature extraction, Coiras et al. further proposed a segmentation-based matching method, using the extracted lines to construct virtual triangles, and then matching based on triangles. This method has strong robustness However, it is limited to images with many non-parallel line features.

In addition, the current visible light and infrared image matching still has the shortcomings of large amount of calculation and low efficiency.

SUMMARY OF THE INVENTION

In order to solve the problem that the existing image registration cannot guarantee the accuracy and high speed of the visible light and infrared image registration at the same time, the present invention proposes a visible light image and infrared image matching method, which can solve the above problems.

In order to solve the above-mentioned technical problems, the present invention adopts the following technical solutions to be realized:

A method for matching a visible light image and an infrared image, comprising the following steps:

(1), extract the maximum stable extreme value area from the visible light image and the infrared image respectively;

(2) Perform ellipse fitting on the maximum stable extreme value regions of the visible light image and the infrared image respectively, and normalize them into a standard circular region;

(3), establish the FBP model, use binary coding to describe the texture information in the standard circle, including:

(31), draw a circle with the center of the standard circular area as the center of the circle and r as the radius, and delineate a circular area as the second circular area, where r=R/2, and R is the radius of the standard circular area;

(32), select several boundary points on the circumference of the second circular area, draw a circle with each boundary point as the center and r as the radius, and then demarcate several corresponding circular areas as the edge circular area , wherein each boundary point is evenly distributed on the circumference of the second circular area;

(33), use LBP algorithm to encode each area and standard circular area respectively, and each area includes the second circular area and all edge circular areas;

(4), use the Hamming distance to match the codes of each area, including:

(41), calculate the weight of each area;

(42), calculate the Hamming distance of each area;

(43), use the Hamming distance of each area and the weight of each area to calculate the total Hamming distance;

(44) Comparing the total Hamming distance with the set threshold, if the total Hamming distance is not greater than the set threshold, the visible light image and the infrared image meet the matching conditions, otherwise, they do not match.

Further, in step (1), the MSER algorithm is used to extract the maximum stable extreme value region, including:

(11), divide the image into different connected regions, mark these regions, and complete the extraction of MSER blocks;

(12) Gradually increase the threshold pixels from 0 to 255, divide the image into different connected areas, mark these areas, and complete the MSER block division;

其中，q_i表示第i时刻MSER块的面积变化率，Q_i表示第i次递增时MSER块的面积，Q_i+Δ表示第i+Δ次递增时MSER块的面积，Q_i-Δ表示第i-Δ次递增时MSER块的面积；(13), calculate the area change rate of the MSER block, and complete the extraction of the MSER block. The calculation method is:

Among them, qi represents the area change rate of the MSER block at the _ith time, Qi represents the area of the MSER block at the _i -th increment, Qi _+Δ represents the area of the MSER block at the i+Δ-th increment, and Qi _-Δ represents the area of the MSER block. The area of the MSER block at the i-Δth increment;

(14), judging whether the _MSER block is the maximum stable extreme value region, when qi is the smallest, the current region is the maximum stable extreme value region.

Further, in step (2), the least squares method is used to perform ellipse fitting on the maximum stable extreme value region.

Further, between step (2) and step (3), the step of performing Gaussian blurring on the entire image is also included.

Further, in step (33), before each region is encoded by LBP algorithm, it also includes steps:

Draw a circle with the center of the standard circular area as the center and 3 pixels as the radius, delimit a circular area as the third circular area, and calculate the average value of all pixel values in the third circular area as the standard circle The center pixel value of the area.

Further, in step (33), encoding each region using the LBP algorithm includes:

(331), select several coding points respectively on the boundary of each region;

(332), comparing the pixel value of the coding point with the central pixel value of the region where the coding point is located, if the pixel value of the coding point is not less than the central pixel value, the coding point is coded as 1, otherwise, the coding is 0.

Further, the weight calculation method of each area in step (41) is:

其中，w_j为第j个区域的权重，j为整数且0＜j≤n，n为第二圆形区域以及所有边缘圆形区域的总数量，(x_j,y_j)为第j个区域的中心坐标，(x_c.y_c)为标准圆形区域的中心坐标，r_j为第j个区域的半径。

Among them, w _j is the weight of the j-th area, j is an integer and 0<j≤n, n is the total number of the second circular area and all edge circular areas, (x _j ,y _j ) is the j-th circular area The center coordinates of the area, (x _c .y _c ) is the center coordinate of the standard circular area, and r _j is the radius of the jth area.

Further, the Hamming distance calculation method for each region is:

其中d_j为第j个区域的汉明距离，x[i]为第j个区域的编码，y[i]为标准圆形区域的编码。

where d _j is the Hamming distance of the jth area, x[i] is the code of the jth area, and y[i] is the code of the standard circular area.

Further, the calculation method of the total Hamming distance d is:

Compared with the prior art, the advantages and positive effects of the present invention are: the visible light image and infrared image matching method of the present invention extracts the maximum stable extreme value region, and establishes an FBP model to process and describe the texture information of the feature region. The FBP algorithm first extracts the most stable extreme value area as the area to be matched, then uses the FBP algorithm to describe each area, and finally sets the threshold, and uses the Hamming distance to determine whether each area matches. The experimental results prove that the invention has good performance in positioning accuracy, algorithm speed, stability, flexibility and real-time performance.

Other features and advantages of the present invention will become more apparent upon reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a flowchart of an embodiment of a method for matching a visible light image and an infrared image proposed by the present invention;

Fig. 2a is a visible light original image of scene 1 in an experimental example of a method for matching a visible light image and an infrared image proposed by the present invention;

Fig. 2b is the original infrared image of scene 1 in an experimental example of the visible light image and infrared image matching method proposed by the present invention;

Fig. 3a is the MSER extraction result in Fig. 2a;

Fig. 3b is the MSER extraction result in Fig. 2b;

Fig. 4a is the ellipse fitting result of the MSER region in Fig. 3a;

Fig. 4b is the ellipse fitting result of the MSER region in Fig. 3b;

Fig. 5a is a partial diagram of ellipse fitting of one of the MSER regions in Fig. 3a;

Fig. 5b is the standard circular area normalized by the MSER area in Fig. 5a;

Fig. 6a is a partial texture map in Fig. 2a;

Fig. 6b is a partial texture map in Fig. 2b;

Fig. 7 Image registration results in Fig. 2a and Fig. 2b;

FIG. 8 is a schematic diagram of the area delineated by the FBP model.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Embodiment 1, this embodiment proposes a method for matching a visible light image and an infrared image, including the following steps:

S1. Extract the maximum stable extreme value region from the visible light image and the infrared image respectively;

S2. Perform ellipse fitting on the maximum stable extreme value regions of the visible light image and the infrared image respectively, and normalize them into a standard circular region;

S3, establish an FBP model, and use binary coding to describe the texture information in the standard circle, including:

S31. As shown in Figure 8, draw a circle with the center O of the standard circular area as the center and r as the radius, and delimit a circular area as the second circular area, where r=R/2, and R is the standard circle the radius of the shaped area;

S32. Select a number of boundary points on the circumference of the second circular area, such as point O1 in Figure 8, draw a circle with each boundary point as the center and r as the radius, and then delimit several corresponding circular areas again , is an edge circular area, in which each boundary point is evenly distributed on the circumference of the second circular area; Figure 8 shows the area divided by the FBP model.

S33, encode each area and the standard circular area respectively, and each area includes the second circular area and all edge circular areas;

S4. Use the Hamming distance to match the codes of each area, including:

S41. Calculate the weight of each area;

S42. Calculate the Hamming distance of each area;

S43, using the Hamming distance of each area and the weight of each area to calculate the total Hamming distance;

S44. Compare the total Hamming distance with the set threshold. If the total Hamming distance is not greater than the set threshold, the visible light image and the infrared image meet the matching conditions, otherwise, they do not match. The visible light image and infrared image matching method of this embodiment extracts the maximum stable extreme value region, and establishes an FBP model to process and describe the texture information of the feature region. The FBP algorithm first extracts the most stable extreme value area as the area to be matched, then uses the FBP algorithm to describe each area, and finally sets the threshold, and uses the Hamming distance to determine whether each area matches. The experimental results prove that the invention has good performance in positioning accuracy, algorithm speed, stability, flexibility and real-time performance.

In the step of describing texture information, the traditional LBP algorithm is simple and fast, but LBP only compares the gray value of the center pixel and the gray value of the pixel on the surrounding circular boundary when describing the feature area. LBP can only describe the edge information of the feature area, and the obtained texture features are robust. However, if the detected feature area is large and the texture is complex, the LBP algorithm only describes the edge information of the feature area when describing the feature area. A lot of image texture information will be lost, resulting in a large error in the final registration result. In step S3 of this scheme, a new algorithm FBP is proposed, which further expands the LBP area by using sector segmentation and ring segmentation, so that FBP can effectively describe the feature area when the detected feature area is large and the texture is complex.

In step S1, the MSER algorithm is used to extract the maximum stable extreme value region, including:

S11. Divide the image into different connected regions, mark these regions, and complete the extraction of the MSER block;

S12. Gradually increase the threshold pixels from 0 to 255, divide the image into different connected areas, mark these areas, and complete the MSER block division;

其中，q_i表示第i时刻MSER块的面积变化率，Q_i表示第i次递增时MSER块的面积，Q_i+Δ表示第i+Δ次递增时MSER块的面积，Q_i-Δ表示第i-Δ次递增时MSER块的面积；S13. Calculate the area change rate of the MSER block, and complete the extraction of the MSER block. The calculation method is as follows

Among them, qi represents the area change rate of the MSER block at the _ith time, Qi represents the area of the MSER block at the _i -th increment, Qi _+Δ represents the area of the MSER block at the i+Δ-th increment, and Qi _-Δ represents the area of the MSER block. The area of the MSER block at the i-Δth increment;

S14. Determine whether the _MSER block is the maximum stable extreme value region. When qi is the smallest, the current region is the maximum stable extreme value region.

In order to extract the maximum value area and the minimum value area in the image at the same time, the MSER algorithm includes both forward and reverse extraction processes. In the forward extraction process, the most stable extreme value region in the forward direction is determined according to its area change rate to extract the maximum value region in the original image, which is recorded as MSER+. In the reverse extraction process, the gray value of the original image is reversed first, and the most stable extreme value region in the reversed image is extracted, which is recorded as MSER-. Generally speaking, forward and reverse extraction can stably extract the relevant features of the corresponding objects in the image. The good performance of MSER is due to the similarity of its extraction process to the attention mechanism of the human visual system, focusing on the "conspicuous part" area. boundaries and their evolution.

The FBP method makes the final extracted feature code contain more texture information by dividing the circle, and the final feature code is more robust, and the FBP model supports adaptive expansion, and the FBP model can be specified according to actual needs. The rules are extended to take into account both time performance and feature encoding accuracy.

The shape of the most stable extreme value regions extracted by the MSER algorithm is arbitrary. In order to facilitate processing, these most stable extreme value regions need to be fitted, such as ellipse fitting, polygon fitting, and circular fitting. Generally speaking, since the eigenvalues and eigenvectors of the feature region covariance matrix uniquely determine an ellipse, ellipse fitting will be performed on the extracted feature regions. However, considering that the concept of the LBP algorithm will be used to describe the feature area, and the circle is inherently invariant to rotation, here we choose to first perform ellipse fitting on the feature area, and then normalize the fitted ellipse into a circle to facilitate the description of the feature area. In step S2 of this embodiment, the least squares method is used to perform ellipse fitting on the maximum stable extreme value region. Then, the fitted ellipse is normalized into a circle to form a standard circular area to facilitate the description of the feature area.

In order to reduce the influence of noise, a step of Gaussian blurring is also included between step S2 and step S3 to make the pixel gray value of the feature area smoother, and the value of the original pixel has the largest Gaussian distribution. All have the largest weight, and adjacent pixels get smaller and smaller as they get farther and farther from the original pixel. This image blur preserves edge effects better than other equalization blur filters.

In step S33, before encoding each region using the LBP algorithm, the following steps are further included:

Since the central pixel value of the standard circular area needs to be used as a benchmark in the subsequent texture description steps, the accuracy of this value has a great influence on the accuracy in the subsequent steps. In order to prevent abnormal pixel values in the center of the standard circular area, This leads to low calculation accuracy. In this scheme, the center of the standard circular area is used as the center of the circle and 3 pixels are used as the radius to draw a circle. A circular area is designated as the third circular area, and all pixel values of the third circular area are calculated. The average value of , as the center pixel value of the standard circular area. By delineating a small area around the center of the standard circular area, and taking the average value of the pixel values in the small area as the central pixel value of the standard circular area, even if the real central pixel value is abnormal, it will not affect the subsequent The precision of the calculation.

In step S33, encoding each region using the LBP algorithm includes:

S331, select several code points respectively on the boundary of each area;

S332. Compare the pixel value of the coding point with the central pixel value of the region where the coding point is located. If the pixel value of the coding point is not less than the central pixel value, the coding point is coded as 1; otherwise, the coding is 0. That is, the encoding of the jth region is:

Among them, _gi is the pixel value of the ith code point, and g _c is the center pixel value of the region where the code point is located.

j=0,1,2,...,n, in this way, n encoded values can be finally obtained, and the n encoded values have grayscale invariance, but at this time, they do not have rotation invariance. First, rotate the circle continuously. The LBP encoding in the neighborhood selects the LBP feature with the smallest LBP feature value as the LBP feature of the center pixel from these LBP feature values. There are n coding values in this FBP method, and the rotation-invariant LBP feature is applied to the circle in the 3×3 neighborhood at the center of the circle to calculate a rotation direction, and the remaining n-1 codes are all based on this rotation direction. Perform rotation encoding, and finally obtain n encoded values with grayscale invariance and rotation invariance.

In the registration process, the concept of equivalence mode is used for reference, the coding dimension is reduced, and the Hamming distance is finally used as the metric.

The weight calculation method of each area in step S41 is:

其中，w_j为第j个区域的权重，j为整数且0＜j≤n，n为第二圆形区域以及所有边缘圆形区域的总数量，(x_j,y_j)为第j个区域的中心坐标，(x_c.y_c)为标准圆形区域的中心坐标，r_j为第j个区域的半径。|(x_j,y_j)-(x_c.y_c)|表示第j个区域的中心到标准圆中心的距离。

Among them, w _j is the weight of the j-th area, j is an integer and 0<j≤n, n is the total number of the second circular area and all edge circular areas, (x _j ,y _j ) is the j-th circular area The center coordinates of the area, (x _c .y _c ) is the center coordinate of the standard circular area, and r _j is the radius of the jth area. |(x _j ,y _j )-(x _c .y _c )| represents the distance from the center of the jth region to the center of the standard circle.

The Hamming distance calculation method for each area is as follows:

其中d_j为第j个区域的汉明距离，x[i]为第j个区域的编码，y[i]为标准圆形区域的编码，

表示异或运算。

where d _j is the Hamming distance of the jth area, x[i] is the code of the jth area, y[i] is the code of the standard circular area,

Represents an exclusive OR operation.

The calculation method of the total Hamming distance d is:

Experimental example:

In order to verify the effectiveness of the algorithm of the present invention, the FBP algorithm is used to register the visible light image and the infrared image in the two scenarios respectively. The experimental environment is: processor Intel(R) Xeon(R) CPU E5-2603v3@1.60GHz1.50GHz, memory 16.0GB, window7 system, coding environment is VS2015, opencv2.4.13 computing platform. In the experiment, two sets of roadside surveillance video frame sequence images were selected respectively. Figure 2a and Figure 2b are the first frame images in scene 1.

In the experiment, the images of scene 1 and scene 2 were first converted into grayscale images, and the MSER+ and MSER- operations were performed on the grayscale images to extract the most stable extreme value regions in the images. The stable extreme value area is similar to the observation of the human eye, highlighting the conspicuous area. The results are shown in Figure 3a and Figure 3b, and the white area is the most stable extreme value area extracted.

The most stable extremum regions of the extracted mser are irregular in shape, which is not conducive to subsequent image processing. It is necessary to perform ellipse fitting on each region. The final fitted ellipse region will be used as the candidate image registration feature region. The combined results are shown in Figure 4a and Figure 4b.

The FBP algorithm proposed by this method is aimed at the circular area, and it is not difficult to find that the MSER area is fitted by an ellipse, and the processing of the elliptical area is obviously not as convenient as the processing of the circular area. In order to establish the FBP algorithm model, the ellipse is regularized into a circular area by the transformation matrix, and the normalized results are shown in Fig. 5a and Fig. 5b.

The normalized circular area is a texture image. The imaging mechanism of visible light image is different from that of infrared image, so the information contained in the image is different. However, both visible light and infrared images retain the complete image outline information. The relationship between the pixels can generate feature descriptors, and the texture information of the feature area of scene 1 is shown in Figure 6a and Figure 6b.

Taking the coordinate point of the center of the ellipse area as the final matching point position, the infrared image and the visible light image are processed by the FBP algorithm respectively, and the FBP code value is obtained for registration. For scene 1, the FBP model is used for code registration, and the result is shown in Figure 7. Show. It can be seen from Figure 7 that the FBP algorithm can correctly find the matching area and realize the registration of visible light images and infrared images.

Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by those of ordinary skill in the art within the scope of the present invention should also belong to protection scope of the present invention.

Claims (8)

1. A visible light image and infrared image matching method is characterized by comprising the following steps:

(1) respectively extracting maximum stable extremum regions from the visible light image and the infrared image;

(2) respectively carrying out ellipse fitting on the maximum stable extremum regions of the visible light image and the infrared image, and normalizing the maximum stable extremum regions into a standard circular region;

(3) establishing an FBP model, describing texture information in the standard circle by using binary coding, and comprising the following steps:

(31) drawing a circle by taking the center of the standard circular area as a circle center and R as a radius, and defining a circular area as a second circular area, wherein R is R/2, and R is the radius of the standard circular area;

(32) selecting a plurality of boundary points on the circumference of the second circular area, drawing a circle by taking each boundary point as a circle center and r as a radius, and then dividing a plurality of corresponding circular areas into edge circular areas, wherein each boundary point is uniformly distributed on the circumference of the second circular area;

(33) respectively coding each area and the standard circular area based on an LBP algorithm, drawing a circle by taking the center of the standard circular area as a circle center and 3 pixels as radiuses, delimiting a circular area as a third circular area, and calculating the average value of all pixel values of the third circular area as the central pixel value of the standard circular area;

(34) respectively coding each area and the standard circular area, wherein each area comprises a second circular area and all edge circular areas;

(4) and matching the codes of the regions by using the Hamming distance, comprising:

(41) calculating the weight of each region;

(42) calculating the Hamming distance of each region;

(43) calculating the total Hamming distance by using the Hamming distance of each region and the weight of each region;

(44) and comparing the total Hamming distance with a set threshold, if the total Hamming distance is not greater than the set threshold, the visible light image and the infrared image accord with the matching condition, otherwise, the visible light image and the infrared image do not match.

2. The visible light image and infrared image matching method according to claim 1, wherein in the step (1), extracting the maximally stable extremal region by using the MSER algorithm includes:

(11) dividing the image into different connected regions, marking the regions and finishing the extraction of the MSER block;

(12) gradually increasing the threshold value pixel from 0 to 255, dividing the image into different connected regions, marking the regions, and completing MSER block division;

(13) calculating the area change rate of the MSER block to complete the extraction of the MSER block, wherein the calculation method comprises the following steps

Wherein q is_iDenotes the rate of change of area, Q, of the MSER block at time i_iDenotes the area of the MSER block at the i-th increment, Q_i+ΔDenotes the area of the MSER block at i + Δ increments, Q_i-ΔRepresents the area of the MSER block at the i-delta increments;

(14) judging whether the MSER block is the mostLarge stable extremal region, when q_iWhen the minimum value is smaller, the current region is the maximum stable extremum region.

3. The method for matching a visible light image with an infrared image according to claim 1, wherein in the step (2), the least square method is adopted to perform ellipse fitting on the maximum stable extremum region.

4. The visible light image and infrared image matching method according to claim 1, further comprising a step of performing gaussian blurring processing on the whole image between the step (2) and the step (3).

5. The visible-light image and infrared image matching method as claimed in claim 1, wherein the encoding of each region by using the LBP algorithm in step (33) comprises:

(331) respectively selecting a plurality of coding points on the boundary of each area;

(332) comparing the pixel value of the coding point with the central pixel value of the area where the coding point is located, if the pixel value of the coding point is not less than the central pixel value, the coding point is coded as 1, otherwise, the coding point is coded as 0.

6. The visible light image and infrared image matching method according to claim 5, wherein the weight calculation method for each region in step (41) is:

wherein, w_jIs the weight of the jth region, j is an integer and is more than 0 and less than or equal to n, n is the total number of the second circular region and all the edge circular regions, (x)_j,y_j) Is the center coordinate of the jth region, (x)_c.y_c) Is the center coordinate of the standard circular region, r_jIs the radius of the jth region.

7. The visible light image and infrared image matching method according to claim 6, wherein the hamming distance calculation method for each region is:

wherein d is_jIs the Hamming distance of the jth region, x [ i ]]For coding of the jth region, y [ i ]]Coding of standard circular area.

8. The method for matching a visible light image with an infrared image according to claim 7, wherein the total hamming distance d is calculated by:

Images