CN107578419A - A Stereo Image Segmentation Method Based on Consistent Contour Extraction - Google Patents

Abstract

The embodiment of the invention discloses a kind of stereo-picture dividing method based on uniformity contours extract, including input stereo-picture left view, left view segmentation figure picture and stereo-picture right view；According to stereo-picture or so view, Stereo matching item is calculated；Stereo-picture right view segmentation probability graph is estimated according to stereo-picture left view, left view segmentation figure picture and stereo-picture right image, and extracts and correct the lines of outline of estimating of segmentation probability graph, and further calculates the profile item based on image patch；According to Stereo matching item and the profile item based on image patch, profile probability graph is synthesized；The optimal profile line of profile probability graph is calculated using geodesic curve distance；Optimal profile line is scanned, obtains the right view segmentation figure picture of stereo-picture.Implement the present invention, the stereo-picture segmentation effect of high quality can be obtained, there is faster operational efficiency again.

Description

A Stereo Image Segmentation Method Based on Consistent Contour Extraction

technical field

The invention relates to the technical field of computer graphics and image processing, in particular to a stereoscopic image segmentation method based on consistent contour extraction.

Background technique

In recent years, stereo media has attracted more and more attention, making the editing of stereo images more and more important. However, stereoscopic image segmentation is the basis of stereoscopic image editing, so it is particularly urgent to perform accurate stereoscopic image segmentation.

So far, the segmentation method of a single image has been very mature, such as the paper "grabcut" published by Carsten Rother, Vladimir Kolmogorov, and Andrew Blake (2004) on ACM Trans. Graph.23(3):309-314: Interactive fore- ground extraction using iterated graph cuts, the GrabCut method proposed by it, the user only needs to draw a rectangular frame near the object or use colored lines to indicate the required foreground and background areas, and can obtain high-quality image segmentation results, and has been used on OpenCV Real-time performance is achieved with GPU acceleration.

However, stereoscopic image segmentation is not limited to single image segmentation, but is based on a single image segmentation operation for the left and right views of the stereoscopic image, but this most direct stereoscopic image segmentation method not only requires a lot of user interaction, but also very It is difficult to guarantee the stereo segmentation results with left and right consistency.

In order to overcome the problems of the above methods, many scholars have begun to study the method of image segmentation for the left and right views of the stereo image using the stereo disparity map. At present, the stereoscopic image segmentation methods developed by many scholars can be roughly divided into three categories: stereoscopic image segmentation methods based on regions, stereoscopic image segmentation methods based on colored lines, and stereoscopic image segmentation methods based on contour extraction.

The region-based stereo image segmentation method uses the stereo correlation between each pixel in the left and right view images, and derives the corresponding right view segmentation result of the stereo image according to the segmentation result of the left view of the stereo image, but the disadvantage is that not only the segmentation effect is serious It depends on the quality of the stereo disparity map, and calculating the stereo disparity map will also consume a lot of time. For example, Wan-Yen Lo, Jeroen van Baar, Claude Knaus, Matthias Zwicker, and Markus Gross (2010) published the paper Stereoscopic 3dcopy&paste on ACM Trans.Graph.29(6):147:1-147:10, through stereo disparity The figure establishes the consistency relationship between the left and right views of the stereo image, and obtains the consistent right view segmentation result of the stereo image according to the segmentation results of the left view.

The stereoscopic image segmentation method based on colored lines is different from the region-based stereoscopic image segmentation method. This method only transfers the user's colored lines through stereo consistency, and then performs stereoscopic image segmentation on the left and right views of the stereoscopic image separately, but still needs to calculate the stereoscopic image. Correlation, as well as performing image segmentation operations on the left and right views of the stereo image respectively, consume a lot of time. For example, the paper Fastinteractive stereo image segmentation published by Wei Ma, Luwei Yang, Yu Zhang, and Lijuan Duan (2016) on Multimedia Tools and Applications, 75(18): 10935-10948 proposed to use the sparse stereo consistency of stereo images to Replace the original dense stereo disparity map to improve the efficiency of stereo image segmentation; another example, W H.Emrah Tasli and A.Aydin Alatan (2013) published a paper on Image Commun,28(10):1374-1389 User assisted disparity remapping for Stereo images, proposed to use sparse stereo consistency to automatically migrate the colored lines in the left view of the stereo image to the right view of the stereo image, and then use the energy optimization model based on graph cuts to perform image segmentation operations on the left and right views of the stereo image to improve The effect of stereoscopic image segmentation.

Stereo image segmentation methods based on contour extraction are still relatively few. For example, Ran Ju, Tongwei Ren, and GangshanWu (2015) published the paper Stereosnakes: Contour based consistent object extraction for stereo images on The IEEE International Conference on Computer Vision (ICCV), which proposes to segment the outline of the object according to the left view of the stereo image, combined with the stereo Correlation and image color histogram are used to derive the object contour of the right view of the stereo image. Although this method improves the efficiency of image segmentation, it is difficult to obtain the image segmentation results in the occluded area because the image color histogram is used to predict the image occlusion area. Satisfactory quality, and the method assumes that the number of pixels in the contour map of the right view of the stereo image is the same as the number of contour pixels in the left view. Due to the existence of occluded areas, this assumption is not valid, so the method is not satisfactory in extracting the contours of occluded areas.

To sum up, the stereoscopic image segmentation methods studied by many scholars cannot obtain high-quality stereoscopic image segmentation effects, nor can they have fast operating efficiency. Therefore, it is necessary to provide a stereoscopic image segmentation method that can not only obtain high-quality stereoscopic image segmentation effect, but also has faster operating efficiency.

Contents of the invention

The purpose of the embodiment of the present invention is to provide a stereoscopic image segmentation method based on consistent contour extraction, which can not only obtain high-quality stereoscopic image segmentation effect, but also have faster operating efficiency.

In order to solve the above-mentioned technical problem, the embodiment of the present invention provides a kind of stereoscopic image segmentation method based on consistent contour extraction, comprising steps:

S1. Input the left view of the stereo image, the segmented image of the left view and the right view of the stereo image;

S2. Calculate a stereo matching item according to the left and right views of the stereo image;

S3. Estimating the right view segmentation probability map of the stereo image according to the left view of the stereo image, the segmented image of the left view, and the right view of the stereo image, and extracting and correcting the estimate according to the estimated right view segmentation probability map of the stereo image Contour lines, and further calculate contour items based on image patches according to the revised estimated contour lines;

S4. Synthesize a contour probability map according to the calculated stereo matching item and the contour item based on the image patch;

S5. Calculate the optimal contour line of the synthesized contour probability map by using the geodesic distance;

S6. Scan the calculated optimal contour line to obtain the right view segmented image of the stereo image.

Wherein, in the step S2, the stereo matching item is obtained by the formula to fulfill;

Among them, p _i , p _j are the pixel position index, p _i is the pixel on the contour line of the left view segmented image, is the m×m local area centered on p _i , m=5, d(p _i ) is the parallax value of pixel p _i , C _AD is the absolute error sum algorithm, and c represents the three channels of the color image.

Wherein, the specific steps of "estimated stereoscopic image right view segmentation probability map" in the step S3 include:

Construct the eigenvectors of the stereo image left view and the right view stereo image respectively, and the eigenvectors are constructed as follows: In the formula, is a m×m local area centered on p _i , is an m×m image patch centered on p _i , m=5;

Using a preset search strategy to search for the image patch most similar to each pixel in the stereo image right view in the left view of the stereo image;

By the formula I _Rmask (p _i )=I _Lmask (p _j ), estimate the right view segmentation probability map of the stereo image; where, I _Lmask is the left view segmentation image, p _i , p _j are the pixel position index, p _j is the right In-view pixel p _i is the center position of the most similar image patch found in the left view of the stereo image.

Wherein, the step of searching for the image patch most similar to each pixel in the right view of the stereo image in the left view of the stereo image using a preset search strategy specifically includes:

The occluded pixel in the right view of the stereo image adopts a global range search strategy to search for the most similar patch; wherein, the global range search strategy is to determine the search range as the entire image of the left view of the stereo image;

The non-occluded pixels in the right view of the stereo image use a scan line search strategy to search for the most similar patch; wherein, the scan line search strategy is to determine the search range as the left view of the stereo image along the same horizontal line at [0,d _max ] parallax Within the range; d _max is the maximum parallax value of the estimated stereo image;

Among them, the distinction between the occluded pixel and the non-occluded pixel in the right view of the stereo image is realized through the following steps, which specifically include: using the scan line search strategy to search in the left view of the stereo image and the most similar pixel P _r in the right view of the stereo image The pixel of the patch, the searched pixel is set to P _l , and then the scan line search strategy is also used to search in the right view of the stereo image

Wherein, in the step S3, the specific steps of "according to the estimated right view segmentation probability map of the stereoscopic image, extracting and correcting the estimated contour lines" include:

Use progressive scanning of the right-view segmentation probability map of the stereoscopic image, and after scanning to a certain pixel in the right-view segmentation probability map of the stereoscopic image is non-zero and contains at least one non-zero pixel in the 3×3 area centered on this pixel, The pixel point is reserved as a contour point, and the retained contour point is further drawn into an estimated contour line;

Count and estimate the number of closed curves in the contour lines, and arrange them according to the area enclosed by the closed curves from small to large;

Set the area threshold A _h = βA _k , remove the closed curves whose area is smaller than the threshold A _h in the estimated contour lines, and obtain the revised estimated contour lines; where β is the weight parameter, set to 0.8, and A _k is The area of the kth closed curve that has been sorted, k is the number of closed curves in the left view segmented image.

Wherein, in the step S3 "according to the estimated contour line after the correction, calculate the contour item based on the image patch" is through the formula To achieve; wherein, d _c (q _i ) is the shortest distance from the pixel point q _i to the estimated contour line, D is the weight parameter, set to D=10, T _d is the threshold, set to 20; C _r is the corrected estimated contour line, d(q _i , q _j ) is the Euler distance between two pixels.

Wherein, the "contour probability map" in the step S4 is calculated by the formula C _C (p _i ,d(p _i ))=C _S (p _i ,d(p _i ))×C _O (p _i ,d(p i ) _i )) to realize; among them, C _S is a stereo matching item, and C _O is a contour item based on an image patch.

Wherein, the "optimum contour line" in the step S5 is obtained by solving the formula The optimal solution is realized; where, C _m is the contour line of the left view segmentation image, N(p _i , p _i-1 ) is the contour continuity item, when the pixel p _i-1 is the 8 neighbors of the pixel p _i , The value of the contour continuity item is 0, otherwise the value of the contour continuity item is ∞.

Wherein, the step S6 specifically includes:

Scan from left to right line by line. If the pixel value is 0, it means that the point is the background. Until the point with a pixel value of 1 is scanned, it means that the point is a left contour point. Continue to scan until the point with a pixel value of 0 is scanned. , indicating the end of the left contour line, the point is the foreground, and then continue to scan until a point with a pixel value of 1 is scanned again and the pixel value corresponding to the next pixel point is 0, indicating that the point is a right contour point,

Fill the pixels between the left contour point and the right contour point as 1, repeat this step until the entire optimal contour line is filled completely, and obtain the right view segmentation image of the stereo image.

Implementing the embodiment of the present invention has the following beneficial effects:

The present invention utilizes the epipolar constraints of the stereo image, and in the energy optimization framework, in addition to the stereo matching item, the contour item based on the image patch is added at the same time, and the contour probability map is synthesized according to these two items, and then the geodesic distance is used to calculate optimal contour. The proposed stereoscopic image segmentation method based on consistent contour extraction can perform consistent image segmentation on stereoscopic images, especially in image occlusion areas, and can also achieve good stereoscopic segmentation results, so as to achieve both high-quality stereoscopic images Segmentation effect, and the purpose of faster operation efficiency.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, obtaining other drawings based on these drawings still belongs to the scope of the present invention without any creative effort.

Fig. 1 is a flow chart of a stereoscopic image segmentation method based on consistent contour extraction provided by an embodiment of the present invention;

Fig. 2 is a frame diagram of the process flow of the stereoscopic image segmentation technology of the present invention;

Fig. 3 is the strategy diagram of two kinds of search most similar image patches that the present invention proposes, global range search strategy and scan line search strategy;

Fig. 4 is the optimization result figure that the present invention adopts different cost matching items;

Fig. 5 is a comparison diagram between the present invention and the current mainstream stereoscopic image segmentation method;

Fig. 6 is the result figure that the present invention carries out image segmentation to multiple groups of stereoscopic images, wherein the first to third columns are the stereoscopic image left view, right view and left view segmentation images successively, the fourth column and the fifth column are obtained by the method of the present invention Right view optimal contour map and optimal right view segmented image, the sixth column is the standard right view segmented image.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

As shown in Figure 1, in the embodiment of the present invention, a kind of stereoscopic image segmentation method that proposes based on consistent outline extraction, comprises steps:

Step S1, input stereoscopic image left view, left view segmentation image and stereoscopic image right view;

The specific process is as follows: input stereo image left view I _L , left view segmented image I _Lmask , and stereo image right view I _R .

Step S2, calculating stereo matching items according to the left and right views of the stereo image;

The specific process is, through the formula Calculate the stereo matching term C _S ;

Among them, p _i , p _j are the pixel position index, p _i is the pixel on the contour line of the left view segmented image, is the m×m local area centered on p _i , m=5, d(p _i ) is the parallax value of pixel p _i , C _AD is the absolute error sum algorithm, and c represents the three channels of the color image.

Step S3: Estimating the right view segmentation probability map of the stereo image according to the left view of the stereo image, the segmented image of the left view and the right view of the stereo image, and extracting and correcting the prediction probability map according to the estimated right view segmentation probability map of the stereo image. Estimating contour lines, and further calculating contour items based on image patches according to the revised estimated contour lines;

The specific process is, (a) predicting the right view segmentation probability map of the stereo image;

(i) Construct the feature vectors of the left view of the stereo image and the stereo image of the right view respectively, and the construction of the feature vector is as follows: In the formula, is a local area of m×m centered on p _i , is an m×m image patch centered on p _i , m=5;

(ii) using a preset search strategy (two search strategies: a global range search strategy and a scan line search strategy) to search for the image patch most similar to each pixel in the stereo image right view in the left view of the stereo image;

The occluded pixels in the right view use the global range search strategy to search for the most similar patch, and the non-occluded pixels in the right view use the scan line search strategy to search for the most similar patch; that is, the global range search strategy is to determine the search range as the left view of the stereo image The whole image; the scan line search strategy is to determine the search range as the left view of the stereo image along the same horizontal line within the parallax range of [0, d _max ]; d _max is the maximum parallax value of the estimated stereo image;

The distinction between occluded pixels and non-occluded pixels in the right view of the stereo image is achieved through the following steps, which specifically include: using the scan line search strategy to search for the most similar patch to a certain pixel P _r in the left view of the stereo image Pixel, the searched pixel is set as P _l , and then the same scan line search strategy is used to search for the pixel of the most similar patch to the pixel P _l in the left view of the stereo image in the right view of the stereo image, and the searched pixel is set as P′ _r ; If the pixel P _r and the pixel P' _r are the same point, then the pixel P _r is a non-occluded pixel; otherwise, if the pixel P _r and the pixel P' _r are different points, then the pixel P _r is an occluded pixel.

(iii) By the formula I _Rmask (p _i )=I _Lmask (p _j ), estimate the right view segmentation probability map of the stereo image; where, I _Lmask is the left view segmentation image, p _i , p _j are the pixel position index, p _j is the center position of the most similar image patch found by pixel p _i in the right view in the left view of the stereo image.

(b) According to the estimated stereo image right view segmentation probability map, extract the estimated contour lines and correct the estimated contour lines;

(i) The right-view segmentation probability map of the stereoscopic image is scanned progressively, and a certain pixel in the right-view segmentation probability map of the stereoscopic image to be scanned is non-zero and contains at least one non-zero pixel in the 3×3 area centered on the pixel After the pixel, the pixel point is reserved as a contour point, and the reserved contour point is further drawn into an estimated contour line;

(ii) Statistically estimate the number of closed curves in the contour lines, and arrange them according to the area enclosed by the closed curves from small to large;

(iii) Set the area threshold A _h = βA _k , remove the closed curves whose area is smaller than the threshold A _h in the estimated contour lines, and obtain the revised estimated contour lines; where β is a weight parameter, set to 0.8, A _k is the area of the kth closed curve that has been sorted, and k is the number of closed curves in the left view segmented image.

(c) Calculate the contour item based on the image patch according to the estimated contour line;

According to the estimated contour line, calculate the contour item C _O based on the image patch:

In the formula, d _c (q _i ) is the shortest distance from the pixel point q _i to the estimated contour line, D is the weight parameter, set to D=10, T _d is the threshold, set to 20; if the pixel point q _i The shortest distance to the estimated contour line exceeds the set threshold T _d , which means that the pixel point cannot become the final contour point, and the value of d _c (q _i ) is infinite;

The calculation method of the shortest distance d _c (q _i ) from the pixel point q _i to the estimated contour line is as follows:

In the formula, C _r is the corrected estimated contour in the right view of the stereo image, and d(q _i , q _j ) is the Euler distance between two pixels.

Step S4, according to the stereo matching item calculated and the contour item based on the image patch, synthesize a contour probability map;

The specific process is to synthesize the contour probability map C _C according to the stereo matching item and the contour item based on the image patch:

C _C (p _i ,d(p _i ))＝C _S (p _i ,d(p _i ))×C _O (p _i ,d(p _i ))

In the formula, C _S is the stereo matching item, which can accurately calculate the consistent contour of the unoccluded area in the left and right views, but the pixel-by-pixel matching does not take into account the occlusion problem; C _O is the contour item based on the image patch, the item The probability that the pixel in the right view becomes a contour point is estimated, especially the probability that the occluded area becomes a contour point can be detected very well.

Step S5, using geodesic distance to calculate the optimal contour of the synthetic contour probability map;

The specific process is to use the geodesic distance to calculate the optimal contour map to solve the following optimization problem:

That is, by solving the formula The optimal solution is realized; among them, C _m is the contour line of the left view segmented image, N(p _i , p _i-1 ) is the contour continuity item, because the closed contour has continuity, so when the pixel p _i-1 is a pixel When there are 8 neighbors of p _i , the value of the contour continuity item is 0, otherwise the value of the contour continuity item is ∞. It can be seen that the solution of the optimization problem is transformed into finding the disparity value d(p _i ), so that the energy The value E(p _i , d(p _i )) takes the minimum value; it should be noted that in the method of the present invention, the points on the contour line of the left view segmented image and the points on the contour line of the right view segmented image that need to be extracted are allowed to exist One-to-many correspondence. The optimal contour line of the contour probability map is calculated using geodesic distance, which includes three steps:

(i) Determine the starting and ending points of the contour; the centroid of each closed contour curve after correction is recorded as M=(M _L1 , M _L2 ,...,M _Lm ), first search for the minimum value of the composite contour probability map along the abscissa of the centroid , this point is marked as the starting point p _m of the contour line. In order to determine the starting position of the contour line, a breadth-first search strategy is used to search. The search strategy is divided into four steps: (1) The initialization queue only contains one pixel p _t =p _m ; (2) search for the smallest and second smallest pixel (not including pixel p _t ) in the 8 neighborhoods of pixel p _t , and mark them as p _s and _pe respectively; (3) if three pixels p _t , p _s The vertical coordinates of p and p _e are not equal to each other, that is, p _t [y]≠p _s [y]≠p _e [y], where [y] is the vertical coordinate of the extracted pixel; then the starting point p _start of the contour line is set to p _s , the end point p _end is set to p _e , and the program ends; otherwise, delete the first pixel of the queue, insert the pixels p _s and p _e into the queue, and enter the fourth step of the program; (4) dequeue the first pixel of the queue and assign it to the pixel p _t , repeat step 2 until the queue is empty;

(ii) calculate the geodesic distance from the starting point p _start of the contour line to the end point p _end of the contour line;

Before calculating the geodesic distance from the starting point p _start of the contour line to the end point p _end of the contour line, the contour probability map must be an unclosed area, and the following strategy can be adopted: if the pixel is adjacent to p _t and the ordinate of the pixel and p _t is the same, the value of the point in the contour probability map is set to infinity;

The distance between two adjacent pixels in the synthetic contour probability map is calculated as follows:

d _H (p _i ,p _j )＝|C _C (p _i ,d(p _i ))-C _C (p _j ,d(p _j ))|

Along the curve C={p ₁ ,p ₂ ,L,p _m } (two points before and after are located in the neighborhood of 8), the geodesic distance from the starting point p _start of the contour line to the end point p _end of the contour line is calculated as follows:

Geodesic distance is defined as the minimum distance of all curves connecting the start point to the end point

Because the curve obtained by the geodesic distance is the minimum energy value of all the curves connecting the starting point and the end point, and the two points before and after the curve are continuous, so the curve obtained by calculating the geodesic distance is the right-view profile; the profile The value of the point is marked as 1, and the value of other points is marked as 0;

Step S6, scanning the calculated optimal contour line to obtain the right view segmented image of the stereo image.

The specific process is to scan from left to right line by line. If the pixel value is 0, it means that the point is the background, until the point with a pixel value of 1 is scanned, it means that the point is the left contour point, and continue scanning until the pixel value is scanned. A point of 0 indicates the end of the left contour line, which is the foreground point, and then continues to scan until a point with a pixel value of 1 is scanned again and the pixel value corresponding to the next pixel point is 0, indicating that the point is a right contour point ,

Fill the pixels between the left contour point and the right contour point as 1, repeat this step until the entire optimal contour line is filled completely, and obtain the right view segmentation image of the stereo image.

As shown in Figure 2, the working principle of the stereoscopic image segmentation method based on consistent contour extraction in the embodiment of the present invention is further described. Figure 2 is a frame diagram of the stereoscopic image segmentation technical process. The method can be summarized as the following six steps: First, the user inputs the left and right views of the stereo image and the segmentation map of the left view of the stereo image; then calculates the stereo matching item according to the input left and right views of the stereo image; and calculates the contour item based on the image patch according to the input left and right views of the stereo image and the left view segmentation image ; Then according to the stereo matching item and the contour item based on the image patch, the contour probability map is synthesized; then the optimal contour line is calculated by using the geodesic distance; finally, the right view segmentation map of the stereo image is obtained according to the contour line.

Fig. 3 is the application scenario figure of two kinds of search most similar image patch strategies that the present invention proposes, including global range search strategy and scan line search strategy.

In order to verify the influence of the stereo matching item and the image patch-based contour item in the energy optimization framework of the present invention on extracting the right view segmentation map of the stereo image, Fig. 4 shows the optimization results of the present invention using different cost matching items. As can be seen from Figure 4, the optimized energy framework using only stereo matching items can match the contour lines of the left and right views of the stereo image well for the unoccluded area, but for the occluded area, because the pixels in the right view are in the left view Matching points cannot be found in the occluded area, so the stereo matching item has a large deviation in the occluded area. Compared with the contour item based on the image patch proposed by the method of the present invention, it can handle the contour line extraction of the occluded area well, and the combined stereo matching item And the contour item based on the image patch further improves the accuracy of the optimal contour line in the right view, and obtains high-quality stereo image right view segmentation results.

In order to quantitatively calculate the effect of stereoscopic image segmentation, we calculated the misjudgment rate between the optimal right-view segmentation map obtained by this method and the standard right-view segmentation map. The calculation formula of the false positive rate PUM (Percentage of unmatching pixels) is as follows:

In the formula, p is the pixel position index, N is the number of pixels in the entire stereoscopic right view, M _S (p) is the optimal right view segmentation map obtained by this method, M ^G (p) is the standard right view segmentation map, # For the counting operation, count the number of pixels satisfying the condition |MS (p) _{-M G} (p)|≠0. The lower the misjudgment rate, the closer the obtained optimal right view segmentation map is to the standard right view segmentation map.

Fig. 5 shows the effect comparison between the present invention and the current mainstream stereoscopic image segmentation method. Contour tracking methods, such as Contour tracking (CT) aim to track the movement of objects, and this method does not need to be too precise in tracking the contours of objects; video segmentation methods, such as Snapcut (SN), are objects that are characterized by the movement of objects in the video Segmentation; Cooperative segmentation methods, such as Co-segmentation (CO), perform collaborative segmentation through two or more pictures of different perspectives. These three methods all perform image segmentation based on the characteristics of object motion in multiple photos or videos, and Stereoscopic images have stricter restrictions, such as epipolar constraints. It can be seen from Figure 5 that the misjudgment rates of these three methods are relatively high. Compared with stereo image segmentation methods, such as Stereo cut (SC) and Stereo snake (SS), epipolar constraints are added to stereo image segmentation. , Stereo cut (SC) is a stereoscopic segmentation method based on regions, and the image segmentation effect of this method depends on the probability density function, while Stereo snake (SS) and the method of the present invention are stereoscopic segmentation methods based on contour extraction, compared to Stereo snake (SS) uses a simple stereoscopic image color histogram to derive the contour of the right-view segmented object of the stereoscopic image. The contour item based on the image patch proposed by the method of the present invention can better handle the contour extraction of the image occlusion area.

Among them, the Contour tracking (CT) method was published by Chih-Yuan Chung and Homer H Chen in 2010 in the paper Video object extraction via mrf-based contour tracking in IEEE Transactions on Circuits and Systems for Video Technology Volume 20, Issue 1, Page 149-155 propose.

The Snapcut(SN) method was published by Xue Bai, Jue Wang, David Simons, and Guillermo Sapiro in 2009 in the paper Video snapcut: Robustvideo object cutout using localized classifiers on ACM Trans.Graph.28 Volume 3 Issue 70:1-70:11 propose.

The Co-segmentation (CO) method was published by D.Batra, A.Kowdle, D.Parikh, J.Luo, and T.Chen in 2010 in In Computer Vision and Pattern Recognition (CVPR), 2010IEEEConference on, pages 3169-3176 The paper icoseg: Interactive co-segmentation with intelligent scribble guidance was proposed.

The Stereo cut (SC) method was proposed by B.L.Price and S.Cohen in the paper Stereocut: Consistent interactive object selection in stereo image pairs published in In2011International Conference on Computer Vision, pages 1148-1155 in 2011.

The Stereo snake (SS) method was proposed by Ran Ju, Tongwei Ren, and Gangshan Wu in the paper Stereosnakes: Contour based consistent object extraction for stereo images published in In The IEEE International Conference on Computer Vision (ICCV), 1724-1732 in 2015 .

In order to quantitatively calculate the misjudgment rate of the inventive method and the mainstream stereoscopic image segmentation method, we tested 31 groups of stereoscopic image pairs (the stereoscopic image left and right views and the standard stereoscopic image left and right view segmentation map come from the Middlebury data set (see http: //vision.middlebury.edu/stereo/) and Adobe open datasets (see https://sourceforge.net/adobe/adobedatasets/home/Home/), these datasets have very strong texture features and include different depth objects), the quantitative stereo image segmentation accuracy is shown in Table 1.

Table 1

It can be seen from Table 1 that the method of the present invention, the Stereo cut (SC) and the Stereo snake (SS) methods adopt stricter epipolar constraints, so the misjudgment rate is lower than the other three methods, which need special explanation Yes, the misjudgment rate of the method of the present invention is the lowest. Therefore, the method of the present invention obtains the highest quality stereoscopic image segmentation effect.

Fig. 6 is the result that the present invention carries out image segmentation to multiple sets of stereoscopic images, wherein the first to third columns are the left view, right view and left view segmentation images of the stereoscopic images, the fourth column and the fifth column are the right images obtained by the method of the present invention View optimal contour map and optimal right view segmented image, the sixth column is the standard right view segmented image. As can be seen from Fig. 6, the method of the present invention achieves high-quality stereoscopic segmentation results for the stereoscopic image segmentation of occluded regions, color overlapping, and camera motion.

Implementing the embodiment of the present invention has the following beneficial effects:

The present invention utilizes the epipolar constraints of the stereo image, and in the energy optimization framework, in addition to the stereo matching item, the contour item based on the image patch is added at the same time, and the contour probability map is synthesized according to these two items, and then the geodesic distance is used to calculate optimal contour. The proposed stereoscopic image segmentation method based on consistent contour extraction can perform consistent image segmentation on stereoscopic images, especially in image occlusion areas, and can also achieve good stereoscopic segmentation results, so as to achieve both high-quality stereoscopic images Segmentation effect, but also the purpose of faster operating efficiency.

Those of ordinary skill in the art can understand that all or part of the steps in the method of the above-mentioned embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage Media such as ROM/RAM, magnetic disk, optical disk, etc.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (9)

1. A stereoscopic image segmentation method based on consistent contour extraction, characterized in that, comprising steps: S1. Input the left view of the stereo image, the segmented image of the left view and the right view of the stereo image; S2. Calculate a stereo matching item according to the left and right views of the stereo image; S3. Estimating the right view segmentation probability map of the stereo image according to the left view of the stereo image, the segmented image of the left view, and the right view of the stereo image, and extracting and correcting the estimate according to the estimated right view segmentation probability map of the stereo image Contour lines, and further calculate contour items based on image patches according to the revised estimated contour lines; S4. Synthesize a contour probability map according to the calculated stereo matching item and the contour item based on the image patch; S5. Calculate the optimal contour line of the synthesized contour probability map by using the geodesic distance; S6. Scan the calculated optimal contour line to obtain a right-view segmented image of the stereo image. 2. stereoscopic image segmentation method as claimed in claim 1, is characterized in that, in described step S2, stereo matching item is by formula to fulfill; Among them, p _i , p _j are the pixel position index, p _i is the pixel on the contour line of the left view segmented image, is the m×m local area centered on p _i , m=5, d(p _i ) is the parallax value of pixel p _i , C _AD is the absolute error sum algorithm, and c represents the three channels of the color image. 3. stereoscopic image segmentation method as claimed in claim 2, is characterized in that, in described step S3, the specific step of " estimation stereoscopic image right view segmentation probability figure " comprises: Construct the eigenvectors of the stereo image left view and the right view stereo image respectively, and the eigenvectors are constructed as follows: In the formula, is a m×m local area centered on p _i , is an m×m image patch centered on p _i , m=5; Search for the image patch most similar to each pixel in the right view of the stereo image in the left view of the stereo image using a preset search strategy; By the formula I _Rmask (p _i )=I _Lmask (p _j ), estimate the right view segmentation probability map of the stereo image; where, I _Lmask is the left view segmentation image, p _i , p _j are the pixel position indexes, and p _j is the right In-view pixel p _i is the center position of the most similar image patch found in the left view of the stereo image. 4. stereoscopic image segmentation method as claimed in claim 3, is characterized in that, the step of searching for the image patch most similar to each pixel point in the stereoscopic image right view in the stereoscopic image left view using preset search strategy Specifically include: The occluded pixel in the right view of the stereo image adopts a global range search strategy to search for the most similar patch; wherein, the global range search strategy is to determine the search range as the entire image of the left view of the stereo image; The non-occluded pixels in the right view of the stereo image use a scan line search strategy to search for the most similar patch; wherein, the scan line search strategy is to determine the search range as the left view of the stereo image along the same horizontal line at [0,d _max ] parallax Within the range; d _max is the maximum parallax value of the estimated stereo image; Among them, the distinction between the occluded pixel and the non-occluded pixel in the right view of the stereo image is realized through the following steps, which specifically include: using the scan line search strategy to search in the left view of the stereo image and the most similar pixel P _r in the right view of the stereo image The pixel of the patch, the searched pixel is set as P _l , and then the same scan line search strategy is used to search the pixel in the right view of the stereo image and the pixel P _l in the left view of the stereo image is the most similar patch pixel, and the searched pixel is set as P′ _r ; if the pixel P _r and the pixel P′ _r are the same point, then the pixel P _r is a non-occluded pixel; otherwise, if the pixel P _r and the pixel P′ _r are different points, then the pixel P _r is an occluded pixel. 5. The stereoscopic image segmentation method according to claim 3, characterized in that, in the step S3, the specific steps of "according to the estimated right-view segmentation probability map of the stereoscopic image, extracting and correcting the estimated contour lines" include : Use progressive scanning of the right-view segmentation probability map of the stereoscopic image, and after scanning to a certain pixel in the right-view segmentation probability map of the stereoscopic image is non-zero and contains at least one non-zero pixel in the 3×3 area centered on this pixel, The pixel point is reserved as a contour point, and the retained contour point is further drawn into an estimated contour line; Count and estimate the number of closed curves in the contour lines, and arrange them according to the area enclosed by the closed curves from small to large; Set the area threshold A _h = βA _k , remove the closed curves whose area is smaller than the threshold A _h in the estimated contour lines, and obtain the revised estimated contour lines; where β is the weight parameter, set to 0.8, and A _k is The area of the kth closed curve that has been sorted, k is the number of closed curves in the left view segmented image. 6. The stereoscopic image segmentation method according to claim 5, characterized in that, "according to the corrected estimated contour line, calculate the contour item based on the image patch" in the step S3 by the formula To achieve; wherein, d _c (q _i ) is the shortest distance from the pixel point q _i to the estimated contour line, D is the weight parameter, set to D=10, T _d is the threshold, set to 20; C _r is the corrected estimated contour line, d(q _i , q _j ) is the Euler distance between two pixels. 7. The stereoscopic image segmentation method according to claim 6, wherein the "contour probability map" in the step S4 is obtained by the formula C _C (p _i , d(p _i ))=C _S (p _i , d(p _i ))×C _O (p _i ,d(p _i )); among them, C _S is the stereo matching item, and C _O is the contour item based on the image patch. 8. stereoscopic image segmentation method as claimed in claim 6, is characterized in that, " optimum contour line " in described step S5 is by solving formula The optimal solution is realized; where, C _m is the contour line of the left view segmentation image, N(p _i , p _i-1 ) is the contour continuity item, when the pixel p _i-1 is the 8 neighbors of the pixel p _i , The value of the contour continuity item is 0, otherwise the value of the contour continuity item is ∞. 9. stereoscopic image segmentation method as claimed in claim 6, is characterized in that, described step S6 specifically comprises: Scan from left to right line by line. If the pixel value is 0, it means that the point is the background. Until the point with a pixel value of 1 is scanned, it means that the point is a left contour point. Continue to scan until the point with a pixel value of 0 is scanned. , indicating the end of the left contour line, the point is the foreground, and then continue to scan until a point with a pixel value of 1 is scanned again and the pixel value corresponding to the next pixel point is 0, indicating that the point is a right contour point, Fill the pixels between the left contour point and the right contour point with 1, repeat this step until the entire optimal contour line is filled completely, and obtain the right view segmentation image of the stereo image.