KR20200095251A - Apparatus and method for estimating optical flow and disparity via cycle consistency - Google Patents

Abstract

The present invention provides an apparatus and a method for estimating optical flow and disparity which can reduce learning time. The apparatus for estimating optical flow and disparity comprises: a stereo image acquisition unit to acquire stereo images of a plurality of frames; and an estimation unit which includes a plurality of convolutional neural networks (hereafter, CNNs) having the same structure and having the same learning weight by learning a pattern recognition method in advance, and simultaneously estimates and outputs the disparity between a left image and a right image distinguished in accordance with a time difference and optical flow for images of continuous frames in a stereo image set of two continuous frames transferred from the stereo image acquisition unit. The plurality of CNNs of the estimation unit cycles the remaining images in a previously designated forward direction and reverse direction from each pixel of an image among four images of the stereo image set of the two continuous frames inputted during learning to backpropagate total loss including cycle consistency loss acquired in accordance with a cycle transition result representing the sum of changes of the positions of searched response points.

Description

A device and method for estimating optical flow and disparity based on cycle consistency {APPARATUS AND METHOD FOR ESTIMATING OPTICAL FLOW AND DISPARITY VIA CYCLE CONSISTENCY}

The present invention relates to an apparatus and method for estimating optical flow and disparity, and to an apparatus and method for simultaneously estimating optical flow and disparity based on cycle consistency.

Analysis of dense correspondence between multiple images is a fundamental task for a variety of applications in image processing and computer vision, especially Advanced Driver Assistance System (ADAS) and It is mainly used in a variety of vehicle applications including autonomous driving systems.

Correspondence point analysis of an image is mainly used in image stitching, alignment, recognition, stereo matching, and optical flow.

On the other hand, as research on deep learning techniques using artificial neural networks develops, the proportion of using deep learning techniques for disparity estimation and optical flow estimation for stereo matching is increasing. Although many studies have been conducted on disparity estimation and optical flow estimation using deep learning techniques, disparity estimation for matching stereo images and optical flow estimation for inferring the motion of an object in a number of consecutive frames are independently conducted. Research has been conducted.

However, recently, it has been found that when disparity and optical flow are simultaneously learned and estimated using a deep learning technique, performance is improved compared to individual estimation of disparity and optical flow. However, in the past, when training to estimate disparity and optical flow individually or when training to estimate disparity and optical flow at the same time, training data containing ground truth as a label is basically used. It has been studied based on the supervised learning method used.

Currently, training data for supervising disparity and optical flow individually is insufficient, but training data for supervised learning disparity and optical flow at the same time is particularly insufficient. In addition, the lack of training data has a problem that results in a significant decrease in accuracy when estimating optical flow and disparity.

However, generating training data including labeled verification data requires a very long time, effort, and cost. Therefore, a method of generating virtual learning data by combining a virtual image and verification data has been studied, but in most cases, when learning using the virtual learning data, the required performance cannot be displayed due to the lack of realism and variability of the virtual image. to be.

Accordingly, there is a need for a method in which the optical flow and disparity estimation apparatus learn to simultaneously infer disparity and optical flow using an unsupervised learning method from a conventional stereo image without requiring training data labeled with verification data.

Korean Registered Patent No. 10-1849605 (Registered on 2018.04.11)

An object of the present invention is to provide an apparatus and method for estimating optical flow and disparity, which exhibits high simultaneous estimation performance of optical flow and disparity even when learned in an unsupervised learning method that does not require learning data.

Another object of the present invention is to provide an apparatus and method for estimating optical flow and disparity that can reduce learning time by simultaneously performing learning for estimating optical flow and disparity.

Another object of the present invention is an apparatus and method for estimating optical flow and disparity that are unsupervised learning based on cycle coherence of corresponding points between the left and right images of consecutive frames in a stereo image of a number of consecutive frames without verification data To provide.

An optical flow and disparity estimation apparatus according to an embodiment of the present invention for achieving the above object comprises: a stereo image acquisition unit for obtaining a stereo image of a plurality of frames; And a plurality of convolutional neural networks (hereinafter referred to as CNNs) having the same structure and having the same learning weight by pre-learning the pattern recognition method, and are sequentially transmitted from the stereo image set of two consecutive frames transmitted from the stereo image acquisition unit. An estimating unit for simultaneously estimating and outputting an optical flow for the images of the frame and disperity between the left and right images classified according to parallax; Including, the plurality of CNNs of the estimation unit are searched by cycling the remaining images in each of the predetermined forward and reverse directions from each pixel of one image among four images of a stereo image set of two consecutive frames input at the time of learning. The total loss including the cycle coherence loss obtained according to the cycle transition result representing the sum of the changes in the positions of the corresponding points may be backpropagated and learned with the updated learning weight.

The cycle coherence loss may be obtained by reflecting a cycle reliability map indicating whether a corresponding point exists between two images in a cycle path in a forward direction and a reverse direction among the four images to the cycle transition result.

When the cycle coherence loss for each pixel in the forward direction and the reverse direction exceeds a predetermined threshold, the cycle coherence loss of the corresponding pixel may be output as a threshold value.

The total loss is a pixel value and a gradient of a corresponding point for each pixel between two images of the four images according to frame order, between two images of the same frame, and between two images having different parallaxes among the four images during training. The restoration loss obtained according to the value may be additionally included.

The restoration loss may be obtained by further reflecting a reliability map indicating whether a corresponding point exists between two images according to a frame order, between two images of the same frame, and between two images having different parallaxes from a frame.

The total loss may further include an optical flow smoothing loss for limiting a change in optical flow obtained according to a frame order in the four images and a disparity smoothing loss for limiting a change in disparity obtained according to a parallax.

The optical flow and disparity estimation apparatus further includes a learning unit that is combined while learning a plurality of CNNs of the estimator to obtain the learning weight, and the learning unit has the same structure as the plurality of CNNs of the estimator, and the 4 An offset acquisition unit composed of a plurality of Siamese CNNs for obtaining a positional change of a corresponding point as an offset for each of two images of different combinations among the images; A loss measuring unit for calculating the cycle coherence loss, the restoration loss, the optical flow smoothing loss, and the disparity smoothing loss by using a plurality of offsets obtained from each of the plurality of sham CNNs of the offset obtaining unit; And applying a predetermined loss weight to each of the cycle coherence loss, the restoration loss, the optical flow smoothing loss, and the disparity smoothing loss to obtain the total loss and backpropagating to the plurality of Siamese CNNs, and the plurality of A lossy backpropagation unit that updates the learning weights for the Siamese CNNs and, when learning on the plurality of Siamese CNNs is completed, transfers the learning weights to the plurality of CNNs of the estimation unit; It may include.

An optical flow and disparity estimation method according to another embodiment of the present invention for achieving the above object includes: obtaining a stereo image of multiple frames; And a plurality of convolutional neural networks (hereinafter referred to as CNNs) having the same structure and having the same learning weight by pre-learning the pattern recognition method, from the stereo image set of two consecutive frames among the stereo images of the plurality of frames. Simultaneously estimating and outputting a disparity between the left image and the right image divided according to the optical flow of the images of and the parallax; Including, the plurality of CNNs are searched corresponding points by cycling the remaining images in each of the predetermined forward and reverse directions from each pixel of one of the four images of the stereo image set of two consecutive frames input at the time of training The total loss including the cycle coherence loss obtained according to the cycle transition result representing the sum of the changes in the position of is backpropagated and can be learned with the updated learning weight.

Therefore, the optical flow and disparity estimation apparatus and method according to an embodiment of the present invention use unsupervised learning of optical flow and disparity at the same time by using cycle consistency in which the cycle for corresponding points in the stereo image of consecutive frames should be the same. Since it can be trained in a method, it is learned in an unsupervised learning method without requiring training data including verification data, so that it has very high optical flow and disparity estimation performance, and can greatly reduce the learning time. In addition, it is possible to increase the accuracy of learning based on cycle consistency based on the reliability map, and further improve the optical flow and disparity estimation performance by further backpropagating and learning the recovery loss and the smoothing loss.

1 shows a schematic structure of an optical flow and disparity estimation apparatus according to an embodiment of the present invention.
FIG. 2 shows a detailed configuration of the learning unit of FIG. 1.
FIG. 3 is a diagram for describing an operation of an offset acquisition unit of FIG. 2.
4 shows a detailed configuration of the loss measurement unit of FIG. 2.
5 is a diagram for explaining the concept of cycle coherence loss.
6 is a diagram for explaining the concept of a reliability map.
7 and 8 are diagrams for explaining the concept of restoration loss.
9 shows an optical flow and disparity estimation method according to an embodiment of the present invention.
10 and 11 show results of comparing the performance of the optical flow and disparity estimation apparatus and method according to the present embodiment.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings and the contents described in the accompanying drawings, which illustrate preferred embodiments of the present invention.

Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention can be implemented in various different forms, and is not limited to the described embodiments. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part “includes” a certain component, this means that other components may be further included, rather than excluding other components, unless specifically stated to the contrary. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware or software or hardware. And software.

1 shows a schematic structure of an optical flow and disparity estimation apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the optical flow and disparity estimation apparatus of the present embodiment includes a stereo image acquisition unit 100, an estimation unit 200, and a learning unit 300.

The stereo image acquisition unit 100 acquires a stereo image to which optical flow and disparity are to be estimated. Here, the stereo image is an image that can be acquired from a stereo camera, and is an image composed of a plurality of consecutive frames having two different viewpoints. The stereo image may be obtained as a vertical image, a left image, etc. according to the structure of a stereo camera, but it is assumed here that a left image and a right image are acquired as an example.

The stereo image acquisition unit 100 transmits a stereo image of two consecutive frames among the acquired stereo images of a plurality of frames to the estimation unit 200.

The stereo image acquisition unit 100 transfers the t-th frame and the t+1-th frame to the estimating unit 200, for example. That is, the left and right images of the t-th frame, that is, a stereo image set (l ₁ , r ₁ ) and a stereo image set (l ₂ , r ₂ ) of the t+1 th frame are transferred.

The estimating unit 200 includes an optical flow estimating unit 210 and a disparity estimating unit 220 each implemented with a pre-learned artificial neural network, for example a convolution neural network (CNN), and a stereo image The optical flow and disparity are estimated and output from the two stereo image sets transmitted from the acquisition unit 100.

The optical flow estimation unit 210 may include two CNNs for estimating optical flow by searching for a movement of a corresponding point in each of the left image and the right image of two consecutive frames, that is, the movement of the object, and the disparity estimating unit Reference numeral 220 may include at least one CNN for estimating a disparity that is a position difference of a corresponding point between the left image and the right image of each frame.

Here, a plurality of CNNs included in the optical flow estimating unit 210 and the disparity estimating unit 220 may be implemented as a siamese CNN that has the same structure and is learned at the same time and applied with the same weight.

Although optical flow searches for the motion of an object in consecutive frames, disparity has a difference in searching for the difference between the left image and the right image of the same frame, but basically, the positional shift between corresponding pixels in two images, that is, It is similar in that it estimates the corresponding point variation. Therefore, it can be implemented as a Siamese CNN having the same structure and learning at the same time to which the same weight is applied.

The optical flow estimating unit 210 includes two CNNs, and the t-th of _two sets of stereo images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) transmitted from the stereo image acquisition unit 100 The correspondence point shift (F _l1,l2 ) between the left images (l ₁ , l ₂ ) of the frame and the t+1th frame and the correspondence point shift (F _r1,r2 ) between the right images (r ₁ , r ₂ ) It is estimated by optical flow.

In addition, the disparity estimator 220 is the left image (l) of the t-th frame of the two stereo image sets ((l ₁ , r ₁ ), (l ₂ , r ₂ )) transmitted from the stereo image acquisition unit 100. _The variation of the corresponding point (F _l1,r1 ) from ₁ ) to the right image (r ₁ ) is estimated as disparity. At this time, the disparity estimating unit 220 may also estimate a shift of the corresponding point (F _l2,r2 ) from the left image (l ₂ ) to the right image (r ₂ ) of the t+1th frame including a plurality of CNNs. . Also, depending on the case, the corresponding point shift from the right image (r ₁ , r ₂ ) to the left image (l ₁ , l ₂ ) of each of the t-th frame and t+1-th frame ((F _r1,l1 ), (F _{r2) ,l2} )) can also be estimated.

As described above, in the optical flow and disparity estimation apparatus according to the present embodiment, the estimating unit 200 is composed of a plurality of pre-learned Siamese CNNs, and calculates optical flow and disparity from the transmitted stereo images of two frames. It can be estimated and printed at the same time.

However, in order for the plurality of Siamese CNNs of the estimating unit 200 to accurately estimate optical flow and disparity, they must be learned in advance.

Accordingly, the apparatus for estimating optical flow and disparity according to the embodiment may further include a learning unit 300 for learning the estimating unit 200 without learning data labeled with separate verification data.

Like the estimation unit 200, the learning unit 300 receives a two-frame stereo image set ((l ₁ , r ₁ ), (l ₂ , r ₂ )) from the stereo image acquisition unit 100. And, for the transmitted stereo image set of two frames ((l ₁ , r ₁ ), (l ₂ , r ₂ )), 8 offsets were calculated using 8 Siamese CNNs having the same configuration as the estimator 200. The weights of the 8 siamese CNNs are updated by backpropagating them to 8 siamese CNNs by measuring losses from the obtained 8 offsets. That is, eight Siamese CNNs are trained.

Here, since the 8 Siamese CNNs have the same configuration as the estimating unit 200, the weights of the updated 8 Siamese CNNs are applied as weights of a number of Siamese CNNs included in the estimating unit 200, and as a result, the estimating unit ( 200) of Siamese CNNs can be trained.

In addition, the learning unit 300 may measure various types of losses from eight offsets and backpropagation in order to improve the learning performance, that is, to increase the accuracy of estimating optical flow and disparity of the estimating unit 200. Here, as an example, cycle coherence loss, recovery loss, and smoothing loss are measured and backpropagated.

The learning unit 300 is a configuration for only learning the estimation unit 200, and may be omitted after a plurality of Siamese CNNs of the estimation unit are learned. That is, when the optical flow and disparity estimation apparatus are actually used, the learning unit 300 may be excluded. In addition, the learning unit 300 may be configured as a separate learning device that acquires weights for a plurality of Siamese CNNs.

FIG. 2 shows a detailed configuration of the learning unit of FIG. 1, FIG. 3 is a diagram for explaining the operation of the offset acquisition unit of FIG. 2, and FIG. 4 shows a detailed configuration of the loss measurement unit of FIG. 2.

Referring to FIG. 2, the learning unit 300 includes an offset acquisition unit 310, a loss measurement unit 320, and a loss backpropagation unit 330.

As shown in FIG. 3, the offset acquisition unit 310 is, respectively, from the two stereo image sets ((l ₁ , r ₁ ), (l ₂ , r ₂ )) transmitted from the stereo image acquisition unit 100. It includes 8 Siamese CNNs having an encoder that extracts a feature map by receiving corresponding two images and a decoder that obtains an offset from the extracted feature map. And, of the eight Siamese CNNs, four Siamese CNNs constitute the optical flow offset acquisition unit 311, and the remaining four Siamese CNNs constitute the disparity offset acquisition unit 312.

Table 1 shows an example of a Siamese CNN encoder and decoder structure.

As shown in Table 1, the encoder may include, for example, 10 convolution layers, and the decoder may include 12 layers in which a convolution layer and an upconvolution layer are combined.

The optical flow offset acquisition unit 311 includes four Siamese CNNs of the same structure, and each CNN is a set of two stereo images transmitted from the stereo image acquisition unit 100 ((l ₁ , r ₁ ), (l _2, r ₂₎₎ left picture (l _1, l ₂₎ and right images (r _1, r ₂₎ 4 of optical flow offset ((F _{l1, l2)} of the forward and reverse directions in accordance with the passage of time for each , (F _r2,r1 ), (F _l2,l1 ), (F _r1,r2 )) are obtained.

Two of the four Siamese CNNs of the optical flow offset acquisition unit 311 are two sets of stereo images transmitted from the stereo image acquisition unit 100 in the same manner as the optical flow estimator 210 of the estimating unit 200 (( In l ₁ , r ₁ ), (l ₂ , r ₂ )), the corresponding point shift between the left images (l ₁ , l ₂ ) from the t-th frame to the t+1-th frame in the forward direction over time (F The corresponding point shifts (F _r1,r2 ) between _l1,l2 ) and right images (r ₁ , r ₂ ) are acquired as optical flow offsets.

And the remaining two siamese CNNs of the optical flow offset acquisition unit 311 are the corresponding point shifts (F _{l2, l1} ) between the left images (l ₁ , l ₂ ) from the t+1 th frame to the t th frame in the reverse direction of time. ) And the right images (r ₂ , r ₁ ) to obtain a corresponding point shift (F _r1,r2 ) as an optical flow offset.

On the other hand, the disparity offset acquisition unit 312 also includes four Siamese CNNs, and each CNN includes two stereo image sets ((l ₁ , r ₁ ), (l ₂ ) transmitted from the stereo image acquisition unit 100 , r ₂ )), the corresponding point shift from the left image (l ₁ , l ₂ ) to the right image (r ₁ , r ₂ ) of each of the t-th and t+1th frames ((F _l1,r1 ), (F _{l2 ,r2} )) and the corresponding point shift ((F _r1,l1 ), (F _r2,l2 )) from the right image (r ₁ , r ₂ ) to the left image (l ₁ , l ₂ ) as a disparity offset .

That is, the offset acquisition unit 310 uses 8 Siamese CNNs to provide 4 sets of 2 stereo images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) transmitted from the stereo image acquisition unit 100. By searching for the corresponding point variation in different combinations for the image, 8 offsets (4 optical flow offsets ((F _l1,l2 ), (F _r2,r1 ), (F _l2,l1 ), (F _r1,r2 )) ) And four disparity offsets ((F _l1,r1 ), (F _r2,l2 ), (F _r1,l1 ), (F _l2,r2 ))).

Here, if the location of the pixel (p) in the image a is (p _x , p _y ) and the location of the corresponding point (q) in the image b is (q _x , q _y ), the shift of the corresponding point from image a to image b, That is, the offset (F _a,b ) is calculated as in Equation 1.

In this embodiment, when the offset acquisition unit 310 acquires 8 offsets including 8 Siamese CNNs, the learning unit 300 accurately performs unsupervised learning based on cycle coherence described below without additional training data. It is to make it possible.

Referring to FIG. 4, the loss measurement unit 320 may include a cycle coherence loss measurement unit 321, a restoration loss measurement unit 322, a smoothing loss measurement unit 323, and a reliability map generation unit 324. .

As described above, in the present embodiment, the loss measurement unit 320 measures three losses of cycle coherence loss, restoration loss, and smoothing loss in order to learn eight Siamese CNNs of the offset acquisition unit 310.

First, the cycle coherence loss measurement unit 321 is determined from the four images of the stereo image set ((l ₁ , r ₁ ), (l ₂ , r ₂ )) of two frames transmitted from the stereo image acquisition unit 100. The loss is measured using cycle coherence that the sum of the variations of the corresponding points cycled in the specified order must have an offset of zero. In addition, the restoration loss measurement unit 322 measures the loss by using the pixel consistency that the pixel values of each corresponding point should be similar. Finally, the smoothing loss measurement unit 323 measures the loss by using a smoothing characteristic that the pixel values of the pixels surrounding the corresponding point must be smoothly changed in the rest area excluding the boundary between motion and disparity.

On the other hand, the reliability map generation unit 324 includes the cycle coherence loss measurement unit 321 and the restoration loss measurement unit 322 for areas where there is no corresponding point such as an area disappeared due to movement of an object and an occlusion area. By measuring the loss, a reliability map is created to prevent errors in the loss measurement.

Hereinafter, the operation of the loss measurement unit 320 of FIG. 4 will be described in detail with reference to FIGS. 5 to 9.

5 is a diagram for explaining the concept of cycle coherence loss.

Cycle coherence means that the corresponding points cycled in the four images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) of a stereo image set of two consecutive frames must be matched. If it is assumed that the 8 Siamese CNNs of the offset acquisition unit 310 are normally learned, as shown in (a) of FIG. 5, t+1 from a specific pixel of the left image 1 ₁ of the t-th frame The corresponding point of the left image (l ₁ ) of the t-th frame cycled through the left image (l ₂ ) of the th frame, the right image (r ₂ ) of the t+1 th frame, and the right image (r ₁ ) of the t th frame is It should be the same as the pixel of the left image (l ₁ ) of the first t-th frame. That is, the sum of the variations of each corresponding point acquired by the offset acquisition unit 310 should be zero.

And this cycle coherence is not only the forward direction (l ₁ -> l ₂ -> r ₂ -> r ₁ ) shown in Fig. 5(a), but also the reverse direction (l ₁ -> r ₁ -) shown in (b). > r ₂ -> l ₂ ) should also be satisfied.

If the sum of the corresponding point variations in at least one of the forward and reverse cycles is not 0, it can be determined that at least one CNN of the 8 CNNs has acquired the variation for the wrong corresponding point, resulting in a cycle coherence loss.

)를 수학식 2과 같이 정의한다.Accordingly, the cycle coherence loss measurement unit 321 calculates the sum of the shifts of the corresponding points in the forward and reverse directions. And in order to calculate the sum of the corresponding point variations, in this embodiment, first, the transfer operator (

) Is defined as in Equation 2.

)는 a, b 및 c의 3개의 영상에 대해 a 영상으로부터 b 영상을 거쳐 c 영상으로의 전이되는 오프셋(F_a,b)의 합을 나타낸다.Equation 2 is the transfer operator (

) Represents the sum of the offsets (F _a,b ) transitioned from the a image to the c image through the b image for three images a, b, and c.

)를 수학식 2의 전이 연산자(

)를 이용하면, 수학식 3과 같이 표현될 수 있다.And similarly to Figure 5, the cycle transition results that are sequentially transitioned for four images of a, b, c, and d (

) To the transfer operator in Equation 2 (

) Can be expressed as in Equation 3.

는 a, b, c, d의 4개의 영상에 대해 a 영상으로부터 b 영상 방향으로의 사이클 전이 결과를 나타낸다.Referring to Equation 3,

Represents the result of the cycle transition from the a image to the b image direction for four images of a, b, c, and d.

로 표현되고, (b)의 역방향 사이클 전이 결과는

로 표현될 수 있다.Accordingly, if Equation 3 is reflected in (a) and (b) of Fig. 5, the result of the forward cycle transition of (a) is

And the reverse cycle transition result in (b) is

Can be expressed as

)와 역방향 사이클 전이 결과(

)가 모두 0이 되어야 하며, 전이 결과가 0이 아니면, 사이클 일관성 손실이 발생된 것으로 볼 수 있다. 즉 사이클 일관성 손실(L _c )은 순방향 사이클 전이 결과(

)와 역방향 사이클 전이 결과(

)의 합(

+

)으로 계산될 수 있다.To satisfy cycle consistency, the forward cycle transition result (

) And reverse cycle transition result (

) Must be all 0, and if the transition result is not 0, it can be considered that cycle coherence loss has occurred. That is, the cycle coherence loss ( L _c ) is the result of the forward cycle transition (

) And reverse cycle transition result (

) Sum (

+

) Can be calculated.

)와 역방향 사이클 전이 결과(

)의 합을 단순 계산하는 경우, 계산된 사이클 일관성 손실(L _c )에 오류가 발생하게 된다.However, as described above, a pixel without a corresponding point may be included between the four images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) of the two stereo image sets. Therefore, the result of the forward cycle transition for all pixels (p) of the image (

) And reverse cycle transition result (

If the sum of) is simply calculated, an error occurs in the calculated cycle coherence loss ( L _c ).

Accordingly, the loss measurement unit 320 additionally includes a reliability map generation unit 324 to prevent an error of the cycle coherence loss L _c . The reliability map generator 324 is a two-way repetitive correspondence point shift obtained between two images according to each combination in four images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) of two stereo image sets. The reliability of each pixel is determined based on the difference, that is, the difference in the offset.

6 is a diagram for explaining the concept of a reliability map. In FIG. 6, (a) and (b) are diagrams of a reliability map generated between two images of image a (in this case l ₁ ) and image b (in this case, l ₂ ). Represent the concept.

)과, (b)에 도시된 b 영상에서 a 영상으로 전이 후 다시 b 영상으로 전이된 오프셋의 합(

) 사이의 차를 기반으로 a 영상과 b 영상 사이의 신뢰도 맵을 계산한다.The reliability map generator 324 is the sum of the offsets transferred from image a to image b and then transferred to image a again (

), and the sum of the offsets transferred from the b image to the a image and then to the b image (

) Based on the difference between the a and b images.

)과 (b)에 도시된 오프셋의 합(

)은 모두 0으로 나타나야 하며, 적어도 0에 근접한 값을 나타내야 한다.Referring to FIG. 6, when the CNNs of the offset acquisition unit 310 are normally learned, the sum of the offsets shown in (a) (

) And the sum of the offsets shown in (b) (

) Must appear as 0, and must represent at least a value close to 0.

)과 오프셋의 합(

) 사이의 차 또한 0에 근사된 값을 가져야 한다. 이에 a 영상과 b 영상의 두 영상 사이의 신뢰도 맵(M_a,b)은 수학식 4에 따라 계산될 수 있다.So the sum of the offsets (

) Plus the offset (

The difference between) must also have a value approximating 0. Accordingly, a reliability map (M _a,b ) between two images of image a and image _b may be calculated according to Equation 4.

In Equation 4, H(x) is a step function that outputs 1 if x is greater than or equal to 0, and outputs 0 otherwise, and ∥∥ ₂ is an L2 norm function. And α ₁ and α ₂ having predetermined values are margin values.

는 오프셋의 합(

)이 오프셋의 합(

)과 마진(α₁)의 합보다 작은 경우에 1로 출력되어 신뢰할 수 있음을 의미한다. 그러나 첫번째 항목만으로 신뢰도를 계산하는 경우, 오프셋의 합(

)과 오프셋의 합(

)이 마진(α₁)보다 작은 차이를 갖되 둘 다 매우 큰 값을 갖는 경우, 신뢰도를 잘못 판단하게 되는 문제가 발생할 수 있다. 이에 두번째 항목인

가 오프셋의 합(

)이 마진(α₂)보다 작은 경우에 1로 출력되어 신뢰할 수 있음을 의미한다. 즉 수학식 4는 오프셋의 합(

)과 오프셋의 합(

)이 마진(α₁) 이내의 차를 갖고, 오프셋의 합(

)이 마진(α₂)보다 작은 값을 갖는 경우에 신뢰할 수 있음을 나타낸다.The first item in Equation 4

Is the sum of the offsets (

) Is the sum of the offsets (

If it is smaller than the sum of) and margin (α ₁ ), it is output as 1, which means that it is reliable. However, when calculating the reliability with only the first item, the sum of the offsets (

) Plus the offset (

) Has a difference smaller than the margin (α ₁ ), but both have very large values, a problem of incorrectly determining the reliability may occur. The second item

Is the sum of the offsets (

If) is smaller than the margin (α ₂ ), it is output as 1, which means that it is reliable. That is, Equation 4 is the sum of the offsets (

) Plus the offset (

) Has the difference within the margin (α ₁ ), and the sum of the offsets (

) Is less than the margin (α ₂ ).

And the cycle coherence loss measurement unit 321 calculates the forward and reverse cycle transition results for four images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) of a stereo image set of two frames, Correspondingly, the reliability map generator 324 may also generate forward and reverse reliability maps between the four images, respectively.

)은 수학식 5와 같이 계산될 수 있다.When the reliability map generator 324 generates a reliability map for the cycle transition from the a image to the b image direction for four images in the order of a, b, c, and d, the cycle reliability map (

) Can be calculated as in Equation 5.

)에 기반하여, 사이클 손실(L _c )을 수학식 6에 따라 획득할 수 있다.In addition, the cycle coherence loss measurement unit 321 is a cycle reliability map obtained from the reliability map generation unit 324 (

), the cycle loss L _c can be obtained according to Equation 6.

과

는 도5 의 (a) 및 (b)에 나타난 순방향 사이클 손실 및 역방향 사이클 손실을 나타내고, 각각 수학식 7과 같이 계산될 수 있다.here

and

Denotes the forward cycle loss and the reverse cycle loss shown in (a) and (b) of FIG. 5, and can be calculated as in Equation 7 respectively.

로서 ∥∥₁은 L1 norm 함수이며, T는 기지정된 문턱값을 나타낸다. 즉 ψ(x)는 x의 L1 norm 값과 기지정된 문턱값(T) 중 작은 값을 출력하는 함수로서, 사이클 손실의 이상 출력값을 제거하기 위해 적용된다.In Equation 7, ψ() is a truncation function,

As ∥∥ ₁ is an L1 norm function, and T represents a predetermined threshold. That is, ψ(x) is a function for outputting the smaller value of the L1 norm value of x and the predetermined threshold value T, and is applied to remove the abnormal output value of the cycle loss.

Meanwhile, even if the cycle loss measurement unit 321 measures the cycle loss in the forward and reverse directions, and the cycle loss is very low, in some cases, the cycle loss may be measured low through an incorrect cycle path. That is, even if an inaccurate correspondence point is passed during the transition between the four images, the cycle loss may be measured as 0 or a low value close to 0 as it is finally transferred to the initial pixel.

In order to compensate for this problem, the restoration loss measurement unit 322 measures a pixel value and a gradient consistency error between corresponding points as restoration loss. Given the offset (F _a,b ) between the a-image and the b-image, the restoration loss ( L ^r _a,b ) between the two images by applying the reliability map (M _a,b ) is obtained according to Equation 8 I can.

는 그래디언트 연산자이고, γ는 픽셀값과 그래디언트의 균형을 조절하기 위한 밸런스값이다. 수학식 8에서 그래디언트 연산자는 조명의 변화에 강건한 복원 손실을 획득하기 위해 적용되며, 기존에도 옵티컬플로우 추정 시에 널리 이용되고 있는 연산자이다.here

Is the gradient operator, and γ is the balance value for adjusting the balance between the pixel value and the gradient. In Equation 8, the gradient operator is applied to obtain a restoration loss robust to a change in lighting, and is an operator widely used in estimating optical flow.

In addition, the application of the reliability map (M _a,b ) in Equation 8 is to prevent the restoration loss for a region without a corresponding point from being reflected as in the cycle coherence loss calculation.

7 and 8 are diagrams for explaining the concept of restoration loss. In FIG. 7 (a) is an optical flow diagram in two stereo image sets ((l ₁ , r ₁ ), (l ₂ , r ₂ )). Forward and reverse recovery loss, (b) represents the forward and reverse recovery loss of disparity, and (a) and (b) of FIG. 8 represent the forward and reverse combination recovery loss of optical flow and disparity.

As shown in Figs. 7 and 8, similarly to the cycle coherence loss, the recovery loss should be taken into account for both the optical flow and disparity, and the forward and reverse recovery loss of each combination of optical flow and disparity along the transition path.

In FIG. 8, the recovery loss for the optical flow and disparity combination is not immediately acquired from the left image (l ₁ ) of the t-th frame and the right image (r ₂ ) of the t+1th frame, but the left of the t+1th frame. Acquiring through the image (l ₂ ) acquires an offset between the left image (l ₁ ) of the t-th frame and the right image (r ₂ ) of the t+1-th frame among 8 sham CNNs of the offset acquisition unit 310 This is because there is no CNN. If the offset acquisition unit 310 includes more than 8 Siamese CNNs, the recovery loss for the optical flow and disparity combination is different from that of FIG. 8, the left image (l ₁ ) and t+ of the t-th frame. It may be obtained directly from the offset between the right image r ₂ of the first frame.

However, an increase in the number of Siamese CNNs of the offset acquisition unit 310 complicates the structure of the offset acquisition unit 310, thereby increasing the cost for configuring the learning unit 300. Therefore, in the present embodiment, in consideration of efficiency, it is assumed that the offset acquisition unit 310 includes eight Siamese CNNs.

Accordingly, the restoration loss measurement unit 322 performs optical flow and optical flow for 4 images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) of the stereo image set of two frames shown in FIGS. 7 and 8. The total reconstruction loss is obtained according to Equation 9 by reflecting all of the forward and reverse recovery losses of the disparity, optical flow, and disparity combination.

On the other hand, the smoothing loss measurement unit 323 according to the characteristics that the optical flow and disparity rapidly change in response to the boundary of the object, that is, a discontinuous part in the image, while the internal area of the object, that is, a continuous part, changes smoothly. Measure the smoothing loss.

The smoothing loss measurement unit 323 may obtain a smoothing loss for two images of image a and image b according to Equation 10.

Here, β has a predetermined value as a smoothness bandwidth adjuster.

However, since the offset of the optical flow representing the difference between successive frames is often smaller than the offset of the disparity representing the difference between different points in time, the lossy backpropagation unit 330 then performs a smoothing loss for the optical flow ( L _o It is not desirable to apply the same loss weights to) and the smoothing loss ( L _d ) for disparity.

Accordingly, the smoothing loss measurement unit 323 divides it so that different loss weights (w _o , w _d ) can be applied to the smoothing loss ( L _o ) for the optical flow and the smoothing loss ( L _d ) for the disparity. Can be measured.

The loss back propagation unit 330 is determined by Equation 11 for each of the cycle coherence loss ( L _c ), the recovery loss ( L _r ), and the smoothing loss ( L _o , L _d ) measured by the loss measurement unit 320. The loss weight (w _c , w _r , w _o , w _d ) is applied and summed to calculate the total loss ( L ).

In addition, the loss backpropagation unit 330 backpropagates the calculated total loss L to the eight Siamese CNNs of the offset acquisition unit 310 to learn by updating the weights of the eight Siamese CNNs.

That is, the learning unit 300 may perform unsupervised learning on the eight Siamese CNNs of the offset acquisition unit 310 by using a general stereo image that is not labeled with the verification data as training data. Here, the learning unit 300 may repeatedly learn until the total loss becomes less than or equal to a predetermined reference loss value, or repeatedly learn up to a predetermined number of reference learning times.

And by transferring the weight of the CNN corresponding to the CNN included in the estimator 200 among the weights of the learned 8 Siamese CNNs to the CNN of the estimating unit 200, the CNN of the estimating unit 200 is the same It can have an effect.

In the above, the offset obtaining unit 310 and the estimating unit 200 of the learning unit 300 are configured separately, but in some cases, the offset obtaining unit 310 of the learning unit 300 is used as the estimating unit 200. May be. That is, a plurality of CNNs of the estimating unit 200 may be directly learned based on the losses backpropagated by the lossy backpropagation unit 330.

9 shows an optical flow and disparity estimation method according to an embodiment of the present invention.

Referring to FIGS. 1 to 8, the optical flow and disparity estimation method of FIG. 9 will be described. First, the stereo image acquisition unit 100 generates a training stereo image for learning eight Siamese CNNs of the learning unit 300. The obtained stereo image set ((l ₁ , r ₁ ), (l ₂ , r ₂ )) of two consecutive frames from the acquired training stereo image is transmitted to the learning unit 300 (S10).

Here, the learning stereo image refers to a stereo image that is simply used at the time of learning, rather than the existing training data where the verification data is separately labeled.

When the learning stereo image is obtained, the learning unit 300 performs learning to acquire learning weights for a plurality of CNNs of the estimating unit 200 using the learning stereo image (S20).

The offset acquisition unit 310 of the learning unit 300 uses 8 siamese CNNs having the same structure as the plurality of CNNs of the estimating unit 200 to a stereo image set of two frames ((l ₁ , r ₁ ), (l ₂ , r ₂ )) with 4 optical flow offsets ((F _l1,l2 ), (F _r2,r1 ), (F _l2,l1 ), (F _r1,r2 )) and 4 disparity offsets ( A total of eight offsets of (F _l1,r1 ), (F _r2,l2 ), (F _r1,l1 ), (F _l2,r2 ))) are obtained (S21).

The learning unit 300 learns 8 Siamese CNNs by measuring losses from the obtained 8 offsets and backpropagating them to 8 Siamese CNNs.

In order to learn 8 Siamese CNNs, first, the reliability map generation unit 324 of the loss measurement unit 320 uses 8 offsets to form 4 images ((l ₁ , r ₁ ) of a stereo image set of 2 frames. , (l ₂ , r ₂ )), a reliability map between the two images is generated with a predetermined combination different from each other, and a cycle reliability map is generated based on the generated reliability map (S22).

,

)를 획득하고, 획득된 사이클 전이 결과(

,

)와 생성된 사이클 신뢰도 맵을 기반으로 2개 프레임의 스테레오 영상 세트의 4개의 영상((l₁, r₁), (l₂, r₂))에 대해 기지정된 순방향 및 역방향으로의 사이클 일관성 손실(L _c )을 수학식 6 및 7에 따라 계산한다(S23).In addition, the cycle coherence loss measurement unit 321 is in the forward and reverse directions for 4 images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) of a stereo image set of 2 frames from 8 offsets. The cycle transition result of (

,

), and the obtained cycle transition result (

,

) And the cycle coherence loss in the forward and reverse directions for the four images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) of a two-frame stereo image set based on the generated cycle reliability map. ( L _c ) is calculated according to Equations 6 and 7 (S23).

On the other hand, the restoration loss measurement unit 322 is the difference between the pixel value, the gradient difference and the reliability between the corresponding points among the four images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) of the stereo image set of two frames. The restoration loss L _r is calculated using the map according to Equations 8 and 9 (S24). Here, as shown in Equation 9, the restoration loss measurement unit 322 sums the forward and reverse restoration losses of the optical flow, the forward and reverse restoration losses of the disparity, and the combination restoration loss of the optical flow and the disparity in the forward and reverse directions. The recovery loss ( L _r ) can be calculated.

And the smoothing loss measurement unit 323 is 2 images according to the optical flow in the 4 images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) of the stereo image set of two frames ((l ₁ , l ₂ ), (r ₁ , r ₂ )) smoothing loss ( L _o ) and smoothing loss between 2 images ((l ₁ , r ₁ ), (l ₂ , r ₂ )) according to disparity ( L _d ) Is calculated according to Equation 10 (S25).

The loss back propagation unit 330 is determined by Equation 11 for each of the cycle coherence loss ( L _c ), the recovery loss ( L _r ), and the smoothing loss ( L _o , L _d ) measured by the loss measurement unit 320. The loss weight (w _c , w _r , w _o , w _d ) is applied and summed to calculate the total loss ( L ) (S26).

Then, the calculated total loss ( L ) is backpropagated to the eight Siamese CNNs of the offset acquisition unit 310, and training is performed by updating the learning weights of the eight Siamese CNNs (S27).

The learning unit 300 determines whether the number of learning of the eight Siamese CNNs of the offset acquisition unit 310 is equal to or greater than a predetermined reference number of learning (S28). If it is less than the reference number of learning, eight offsets are obtained again (S21). However, if it is more than the reference number of training, the repetition is terminated, and training weights corresponding to a plurality of CNNs of the estimating unit 200 among the training weights of the 8 Siamese CNNs are transferred to the corresponding CNNs (S29). That is, by transferring the learned weights to the plurality of CNNs of the estimating unit 200, the plurality of CNNs are converted to the learned state.

Thereafter, the stereo image acquisition unit 100 acquires a stereo image for which optical flow and disparity are to be obtained, and a stereo image set ((l ₁ , r ₁ ), (l ₂ ) by two consecutive frames from the obtained stereo image. , r ₂ )) is transferred to the estimation unit 200 (S30).

Two CNNs constituting the optical flow estimator 210 among the plurality of CNNs of the estimating unit 200 that are configured identically to the eight Siamese CNNs of the learning unit 300 and are trained by receiving a learning weight are two stereo images. Correspondence point shift (F _{l1, l2} ) and right images (r1,) between the left images (l1, l2) of the t-th frame and the t+1th frame of the set ((l1, r1), (l2, r2)) The corresponding point variation (F _r1, r2) between _r2 ) is estimated as an optical flow, and at least one CNN constituting the disparity estimator 220 is from the left image (l1) of the t-th frame to the right image (r1). The corresponding point variation (F _l1,r1 ) of is estimated as disparity (S40). In addition, the shift of the correspondence point (F _l2,r2 ) from the left image (l2) to the right image (r2) of the t+1th frame can be estimated together as disparity, and in some cases, the tth frame and the t+1th frame The corresponding point transition ((F _r1,l1 ), (F _r2,l2 )) from the right image (r1, r2) of each frame to the left image (l1, l2) can also be estimated as disparity.

Hereinafter, the performance of the optical flow and disparity estimation apparatus and method according to the present embodiment will be reviewed.

Table 2 shows the results of comparing the optical flow estimation performance according to the present embodiment with the conventional optical flow estimation method for FlyingThings3D, Sintel clean/final and KITTI 2012/2015 data sets.

Among the data sets used here, FlyingThings3D provides a labeled stereo image with verification data for optical flow, disparity, and disparity change, and provides a data set including 21,818 labels and 4,248 test stereo images. Sintel includes a clean version of the data set and a final version of the data set each containing 1064 stereo images. And KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) 2012 and KITTI 2015 provide data sets containing 197 and 200 labels and 195 and 200 test stereo images, respectively.

The stereo image used for training has a size of 512 X 256, and iterative learning was performed 1700k, 850k and 200k times for each data set, and the loss weights (w _c , w _r , w _o , w _d ) were each w _c = 1, w _r = 1, w _o = 13.25 and w _d = 3. And α ₁ = 1, α ₂ = 20, β = 1, γ = 1, T = 10.

However, for the KITTI data set, w _r = 0.3 and γ = 1 were applied to consider the change in lighting.

In Table 2, the suffix -FC indicates the case where each optical flow estimation method was learned using the FlyingChairs data set, -FT indicates the case where it was trained using the FlyingThings3D data set, and -K indicates the case learned using the KITTI data set. It indicates the case. And -S denotes a case learned using the Sintel data set. And +ft- represents the fine-tuned result using two data sets.

All results in Table 2 represent the average endpoint error (EPE) in pixels when compared to the ground truth, and when the EPE exceeds 3% or 5%, the estimated optical flow is considered to have an error. The percentage of error pixels (P1-all) is expressed as %.

Referring to Table 2, this embodiment, which is unsupervised learning with the FlyingChairs data set, has superior performance for the KITTI 2012/2015 data set than the existing DSTFlow-FC or Occlusion-aware-FC that has been unsupervised learning with the FlyingChairs data set. It can be seen that it represents. Especially, for Sintel clean/final data set, it shows the best performance among all unsupervised learning methods.

And this embodiment, unsupervised learning with the KITTI 2012/2015 data set, shows the best performance for the KITTI 2012/2015 data set, and this embodiment unsupervised learning with the Sintel clean/final data set is the KITTI 2012/2015 data. It shows similar performance to FlowNet using supervised learning method for three.

Meanwhile, Table 3 shows the results of comparing the disparity estimation performance according to the present embodiment with the conventional disparity estimation method for the FlyingThings3D and KITTI 2012/2015 data sets.

Table 3 shows the mean absolute error (MAE) and log root mean square error (RMSE) comparing the disparity estimation result with the verification data, and D1-all is an error that is considered an error when MAE is 3% or 5% or more. Represents the percentage of pixels.

Referring to Table 3, it can be seen that this embodiment, unsupervised learning with the FlyingThings3D data set, provides similar performance to DispNet supervised learning with the same FlyingThings3D data set for the KITTI 2012/2015 data set. In addition, it can be seen that this embodiment, which is unsupervised learning with the KITTI 2012/2015 data set, shows better performance than all supervised learning and unsupervised learning in terms of average MAE and log RMSE for the KITTI 2012/2015 data set.

Table 4 shows the results of quantitative comparison of the disparity estimation performance of the present embodiment.

Referring to Table 4, it can be seen that the disparity estimation performance of the present embodiment is superior to all unsupervised learning methods, and shows a performance close to that of the supervised learning method.

10 and 11 show results of comparing the performance of the optical flow and disparity estimation apparatus and method according to the present embodiment.

FIG. 10 shows a result of qualitative comparison of the optical flow and disparity estimation performance of the present embodiment. In FIG. 10, (a) shows the original stereo images, and (b) shows the ground-truth optical flow. And (c) denotes an optical flow estimated in the case of learning by the unsupervised learning method of the present embodiment. In addition, (d) denotes the verification data disparity, and (e) denotes the estimated disparity in the case of learning by the unsupervised learning method of the present embodiment.

The first 6 rows from the top of Fig. 10 show the case where FT is first studied and fine-tuned learning is performed with +ft-S, and then 6 rows are unsupervised learning with the KITTI 2012/2015 data set, and the KITTI 2012/2015 data set. The results of estimating optical flow and disparity are shown. The bottom 3 row shows the results of estimating optical flow and disparity for the FlyingThings3D data set when unsupervised learning is performed with the FlyingThings3D data set.

As shown in FIG. 10, it can be seen that the present embodiment can estimate optical flow and disparity with excellent performance comparable to ground-truth.

Table 5 shows a performance comparison for the case where the learning unit 300 uses each or both of the reliability map and the cycle consistency, and learns each or both of the optical flow and disparity.

Referring to Table 5, it can be seen that the best performance in FlyingThings3D and Sintel clean/final is obtained when both the reliability map and cycle consistency are used, and optical flow and disparity are simultaneously learned.

11 is a diagram showing qualitatively the optical flow and disparity estimation performance according to the learning method. The upper part shows the optical flow estimation result, and the lower part shows the disparity estimation result. And (a) shows the case of learning without using both the cycle coherence loss and the reliability map, (b) the case where the cycle coherence loss is not used, (c) the case where the reliability map is not used, and (d) Is a case where optical flow and disparity are individually learned, (e) shows a case where optical flow and disparity are simultaneously learned using a cycle coherence loss and reliability map according to the present embodiment, and (f) is Show verification data.

As shown in FIG. 11, it can be seen that the EPE and MAE compared to the verification data are the best when optical flow and disparity are simultaneously learned using the cycle coherence loss and reliability map according to the present embodiment.

As a result, the optical flow and disparity estimation apparatus and method according to the present embodiment do not need to separately learn the optical flow and disparity, and thus, not only can improve the learning speed, but also exhibit better performance than the individual learning method. , Optical flow and disparity can be estimated at the same time.

The method according to the present invention may be implemented as a computer program stored in a medium for execution on a computer. Computer readable media herein can be any available media that can be accessed by a computer, and can also include any computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and ROM (readable) Dedicated memory), RAM (random access memory), CD (compact disk)-ROM, DVD (digital video disk)-ROM, magnetic tape, floppy disk, optical data storage, and the like.

Although the present invention has been described with reference to the embodiments shown in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

100: stereo image acquisition unit 200: estimation unit
300: learning unit 210: optical flow estimation unit
220: disparity estimation unit 310: offset acquisition unit
320: loss measurement unit 330: loss back propagation unit
311: optical flow offset acquisition unit 312: disparity offset acquisition unit
321: cycle consistency loss measurement unit 322: restoration loss measurement unit
323: smoothing loss measurement unit 324: reliability map generation unit

Claims (10)

A stereo image acquisition unit that acquires a stereo image of multiple frames; And
Consecutive frames from a stereo image set of two consecutive frames transmitted from the stereo image acquisition unit, including a plurality of convolutional neural networks (hereinafter referred to as CNN) having the same structure and having the same learning weight by pre-learning the pattern recognition method An estimating unit for simultaneously estimating and outputting disparity between the left image and the right image divided according to the optical flow and parallax of the images of Including,
The plurality of CNNs of the estimation unit are
Cycle representing the sum of changes in the position of the searched corresponding points by cycling the remaining images from each pixel of one of the four images of the stereo image set of two consecutive frames input at the time of learning in each of the predetermined forward and reverse directions A device for estimating optical flow and disparity that is learned with the updated learning weight by backpropagating a total loss including a cycle coherence loss obtained according to a transition result. The method of claim 1, wherein the cycle coherence loss is
The optical flow and disparity estimation apparatus obtained by reflecting a cycle reliability map indicating whether a corresponding point exists between two images in a cycle path in a forward direction and a reverse direction among the four images to the cycle transition result. The method of claim 2, wherein the cycle coherence loss is
An optical flow and disparity estimation apparatus for outputting a cycle coherence loss of a corresponding pixel as a threshold when the cycle coherence loss for each pixel in the forward direction and the reverse direction exceeds a predetermined threshold value. The method of claim 1, wherein the total loss is
At the time of learning, between two images according to the frame order of the four images, between two images of the same frame, and between two images with different parallaxes from the frame, acquired according to the pixel value and the gradient value of the corresponding point for each pixel Optical flow and disparity estimating device additionally including the restoration loss. The method of claim 4, wherein the restoration loss is
An optical flow and disparity estimation apparatus obtained by further reflecting a reliability map indicating whether a corresponding point exists between two images according to a frame order, between two images of the same frame, and between two images having different parallaxes from a frame. The method of claim 4, wherein the total loss is
Optical flow and disparity estimating apparatus further including an optical flow smoothing loss for limiting a change in optical flow acquired according to a frame order in the four images and a disparity smoothing loss for limiting a change in disparity obtained according to a parallax . The method of claim 6, wherein the optical flow and disparity estimation apparatus
Further comprising a learning unit that is combined while training a plurality of CNNs of the estimation unit to obtain the learning weight,
The learning unit
An offset acquisition unit comprising a plurality of Siamese CNNs having the same structure as the plurality of CNNs of the estimation unit and obtaining a positional change of a corresponding point for each of two images of different combinations among the four images;
A loss measuring unit for calculating the cycle coherence loss, the restoration loss, the optical flow smoothing loss, and the disparity smoothing loss by using a plurality of offsets obtained from each of the plurality of sham CNNs of the offset obtaining unit; And
By applying a known loss weight to each of the cycle coherence loss, the restoration loss, the optical flow smoothing loss, and the disparity smoothing loss, the total loss is obtained and backpropagated to the plurality of Siamese CNNs, and the plurality of Siamese A lossy backpropagation unit that updates the training weights for the CNNs and, when the training on the plurality of Siamese CNNs is completed, transfers the training weights to the plurality of CNNs of the estimation unit; Optical flow and disparity estimation apparatus comprising a. Obtaining a multi-frame stereo image; And
Using a plurality of convolutional neural networks (hereinafter referred to as CNNs) having the same structure and having the same learning weight by pre-learning the pattern recognition method, consecutive frames from the stereo image set of two consecutive frames among the stereo images of the plurality of frames are Simultaneously estimating and outputting a disparity between the left and right images classified according to the optical flow of the images and the parallax; Including,
The plurality of CNNs
Cycle representing the sum of changes in the position of the searched corresponding points by cycling the remaining images from each pixel of one of the four images of the stereo image set of two consecutive frames input at the time of learning in each of the predetermined forward and reverse directions A method for estimating optical flow and disparity learned with the updated learning weight by backpropagating a total loss including a cycle coherence loss obtained according to a transition result. The method of claim 8, wherein the optical flow and disparity estimation method
Further comprising a learning step of learning the plurality of CNNs,
The learning step is
Using a plurality of Siamese CNNs having the same structure as the plurality of CNNs, acquiring a plurality of offsets representing a position change of a corresponding point for each of two images of different combinations among the four images input during training ;
Using the plurality of offsets, a cycle reliability map indicating whether a corresponding point exists between the cycle transition result and each of two images in a cycle path in a forward direction and a reverse direction among the four images is obtained, and the cycle reliability map is obtained. Reflecting the cycle transition result to obtain the cycle coherence loss;
Reconstruction loss obtained according to the pixel value and the gradient value of the corresponding point for each pixel between two images according to the frame order of the four images, between two images of the same frame, and between two images having different parallaxes from the frame Calculating;
Calculating an optical flow smoothing loss for limiting a change in optical flow obtained according to a frame order in the four images and a disparity smoothing loss for limiting a change in disparity obtained according to a parallax;
Obtaining the total loss by applying a predetermined loss weight to each of the cycle coherence loss, the restoration loss, the optical flow smoothing loss, and the disparity smoothing loss;
Backpropagating the total loss to the plurality of Siamese CNNs to update learning weights for the plurality of Siamese CNNs; And
Transmitting the learning weights to the plurality of CNNs when learning for the plurality of Siamese CNNs is completed; Optical flow and disparity estimation method comprising a. The method of claim 9, wherein the learning step
Generating a reliability map indicating whether a corresponding point exists between two images of different combinations among the four images; And
Generating a cycle reliability map indicating whether a corresponding point exists between two images in a cycle path in a forward direction and a reverse direction among the four images using the reliability map; Further comprising,
The step of obtaining the cycle coherence loss
Reflecting the cycle reliability map to the cycle transition result to obtain the cycle coherence loss,
The step of calculating the restoration loss
An optical flow and disparity estimation method for obtaining the restoration loss by reflecting a reliability map obtained between two images according to a frame order, between two images of the same frame, and between two images having different parallaxes.

Images