KR101261577B1 - Apparatus and method for encoding and decoding multi view video - Google Patents

Abstract

The present invention relates to a method and apparatus for encoding and decoding a multiview video. A multi-view video encoding apparatus according to the present invention comprises: a disparity prediction unit configured to receive a current frame and a reference frame and predict a disparity vector between the frames in a predetermined block unit; A parallax deviation calculator for calculating a deviation or deviation of the predicted parallax vectors; And an encoder which encodes a multi-view video using the calculated deviation or deviation of the disparity vector. According to the present invention, the compression rate of a multi-view video can be improved.

Multiview Video, Compression Rate, Parallax Vectors, Frames, Deviation, Coding

Description

Apparatus and method for encoding and decoding multi view video}

1 illustrates a direct mode of a B picture.

2 illustrates a method of predicting a motion vector in a spatial domain.

3 is a block diagram illustrating a configuration of a multiview video encoding apparatus according to an embodiment of the present invention.

4 shows a frame sequence photographed by a multiview camera.

5 is a view for explaining the case where the deviation of the parallax vector of the present invention is constant.

FIG. 6 is a graph showing the relationship between parallax vectors when the variation of the parallax vector is constant as shown in FIG. 5; FIG.

7 is a block diagram illustrating a configuration of a multiview video encoding apparatus according to an embodiment of the present invention.

8 is a diagram illustrating a multiview video encoding method according to an embodiment of the present invention.

9 is a block diagram showing the configuration of a multi-view video decoding apparatus according to an embodiment of the present invention.

The present invention relates to a method and apparatus for encoding and decoding a multiview video, and to a multiview video encoding method and apparatus for improving the compression rate of multiview video encoding, and a multiview video decoding method and apparatus.

Recently, the new H.264 video coding standard has attracted attention due to the higher coding efficiency than the previous standard. This new standard not only considers a typical bi-directional B prediction slice, but also allows for variable block sizes ranging from 16x16 to 4x4, quadtree structures, multiple reference frames, intra prediction, for motion compensation in loop deblocking filters, It relies on several new features, such as considering context-adaptive entropy coding. Unlike standards such as MPEG-2 and MPEG-4 Part 2, B slices use multiple predictions from images coming from the same direction (forward or reverse), while they can be used as reference pictures of other slices. However, the features described above with respect to this standard require the prediction mode to have a fairly high rate of bits for motion information.

To alleviate this problem, skip (SKIP) and direct (DIRECT) modes were introduced for the P slice and the B slice, respectively. These modes enable the prediction of the motion of any block of the picture to be currently encoded using previously encoded motion vector information. Thus, no additional motion data for the macroblock or block is encoded. The motion for these modes is obtained using the spatial (SKIP) or temporal (DIRECT) relationship of the motion of the adjacent MB or picture.

1 is a diagram for explaining a direct mode of a B picture.

In the direct mode, in order to predict the motion of an arbitrary block of a B picture to be currently encoded, a forward motion vector and a backward motion vector are obtained using a motion vector of a co-located block of a P picture, which is the next picture in time. To save.

In order to calculate the forward motion vector MV _LO and the reverse motion vector MV _L1 of the direct mode block 102 for which the motion in the B picture 110 is to be predicted, the direct in the reference picture 1 picture 120 which is the next picture in time. A co-located block 104, which is a block at the same location as the mode block 102, finds a motion vector MV for the reference list 0 picture 130 referenced by the motion vector. Then, the forward motion vector MV _LO and the reverse motion vector MV _L1 of the direct mode block 102 of the B picture 110 are calculated by Equation 1 below.

Here, MV represents a motion vector of the corresponding block 104 of the reference list 1 picture 120. TD _D indicates the distance between the reference list 0 picture 130 and the reference list 1 picture 120, and TD _B indicates the distance between the B picture 110 and the reference list picture 0 130.

2 is a diagram illustrating a method of predicting a motion vector in a spatial domain.

According to the H.264 standard used for encoding video data, a frame is divided into blocks having a predetermined size, and a motion search is performed to search for the most similar block by referring to an adjacent frame that has already been encoded. That is, among the motion vectors of the three macroblocks of the left (4), the top (2), and the top right (3) of the current macroblock (c), the intermediate value is determined as the predicted value of the motion vector. This motion vector prediction can be represented by Equation 2.

As described above, a method of encoding a video using spatial relevance as well as temporal relevance has been proposed, and a method of improving compression rate and processing speed is required for a multi-view video having more information than general video.

An object of the present invention is to provide a multi-view video encoding method and apparatus for improving the compression ratio of a multi-view video by using the relation between the parallax vectors for the multi-view video captured by the multi-view camera.

Another object of the present invention is to provide a multi-view video decoding method and apparatus for decoding a multi-view video encoded by using the correlation between disparity vectors with respect to a multi-view video.

The present invention is to solve the above technical problem, the multi-view video encoding apparatus according to an aspect of the present invention, receiving a current frame and a reference frame to predict the parallax vector between the frames in a predetermined block unit Prediction unit; A parallax deviation calculator for calculating a deviation or deviation of the predicted parallax vectors; And an encoder which encodes a multi-view video using the calculated deviation or deviation of the disparity vector.

Preferably, the deviation between the predicted parallax vectors is constant.

The encoding mode selection unit may be configured to select whether to encode a multiview video using a deviation of parallax or to encode a multiview video using a deviation of a deviation according to a predetermined criterion, and to set flag information according to the selected encoding mode. The encoder further encodes a multi-view video with the set flag information based on the set flag information.

Preferably, the parallax deviation calculator comprises: a first buffer that stores the predicted parallax vector; A first difference calculating unit calculating a deviation between the parallax vectors stored in the first buffer; A second buffer for storing the deviation; And a second order calculating unit calculating a deviation of the deviation.

According to another aspect of the present invention, there is provided a multi-view video encoding method comprising: receiving a current frame and a reference frame and predicting a parallax vector between the frames in a predetermined block unit; Calculating a deviation or deviation of the deviation between the predicted parallax vectors; And encoding the multiview video using the deviation of the parallax vector or the deviation of the deviation.

In order to solve the above technical problem, a multi-view video decoding apparatus according to another aspect of the present invention, encoding mode check unit for identifying flags indicating a multi-view video encoding mode included in the encoded multi-view video; A parallax vector determiner that determines a parallax vector for a multiview video according to the identified flag; And a decoder which decodes a multiview video using the determined disparity vector, wherein a flag indicating a multiview video encoding mode is encoded using the deviation of the disparity vector or the deviation of the multiview video. Indicates.

Preferably, the parallax vector determination unit, when the flag indicates that the flag is encoded using the deviation of the parallax vector, when the deviation of the parallax vector of the block at the current i position is dDV _i , the parallax vector DV _{i −} of the previous block. The parallax vector DV _i of the block at the current i position is determined by adding ₁ and the deviation dDV _i of the parallax vector.

Preferably, the parallax vector determining unit, when indicating that the flag is encoded using the deviation of the deviation of the parallax vector, when the deviation of the deviation of the parallax vector of the block at the current i position is ddDV _i , the parallax of the previous block deviation vector dDV _{i - 1} and the time difference plus the deviation ddDV _i of the deviation of the vector and the current determines the i blocks deviation dDV _i of the differential vector of the location, the previous block difference vector DV _{i - 1} and dDV deviation of the determined time difference vector _{Add i} to determine the parallax vector DV _i of the block at the current i position.

In order to solve the above technical problem, a multi-view video decoding method according to another aspect of the present invention includes: identifying flags indicating a multi-view video encoding mode included in an encoded multi-view video; Determining a parallax vector for a multiview video according to the identified flag; And decoding the multiview video using the determined disparity vector, wherein the flag indicating the multiview video encoding mode is encoded using the deviation of the disparity vector or the deviation of the multiview video. Indicates.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

3 is a block diagram illustrating a configuration of a multiview video encoding apparatus according to an embodiment of the present invention.

The apparatus for encoding a multiview video according to an embodiment of the present invention includes a multiview image buffer 310, a predictor 320, a parallax / motion compensation unit 330, a difference image encoder 340, and an entropy encoder 350. ).

In FIG. 3, the proposed multi-view video encoding apparatus receives a multi-view video source that is typically obtained from multiple camera systems or other possible methods. The input multiview video is stored in the multiview image buffer 310. The multiview image buffer 310 provides the stored multiview video source data to the predictor 320 and the difference image encoder 340.

The predictor 320 includes a parallax predictor 322 and a motion predictor 324 to perform motion prediction and parallax prediction on a stored multi-view video source.

The parallax / motion compensation is performed by the parallax / motion compensation unit 330 using the motion vector and the parallax vector predicted by the parallax predictor 322 and the motion predictor 324. The parallax / motion compensation unit 330 provides the differential image encoder 340 with a reconstructed image using the predicted motion and parallax vector.

The difference image encoder 340 may provide a difference image between the original image provided from the multiview image buffer 310 and the reconstructed image compensated by the parallax / motion compensation unit 330 to provide a better image quality and a three-dimensional effect. The encoding is performed and provided to the entropy encoder 350.

The entropy encoder 350 receives the information about the disparity vector and the motion vector generated by the predictor 320 and the residual image from the difference image encoder 340, and generates a bit stream for multi-view video source data.

The parallax information predicted by the parallax prediction unit 322 of FIG. 3 is used for parallax compensation, such as motion information for motion prediction and motion compensation used in conventional codecs such as H.264 and MPEG 2/4. . As attempts have been made to reduce motion information and increase coding efficiency in H.264, an attempt should be made to reduce disparity information and increase coding efficiency in MVC.

According to an embodiment of the present invention, the disparity predicting unit 322 calculates the disparity between the predicted disparity vectors or the deviation of the disparity vectors in order to predict the disparity in a predetermined block unit and improve the coding efficiency. The multi-view video encoding of the present invention is performed using the calculated deviation or deviation of the deviation.

4 is a diagram illustrating a frame sequence photographed by a multiview camera.

The horizontal axis represents the number of cameras for capturing a multiview image with the viewpoint axis. The vertical axis represents the number of time sequences on the time axis. If the multiview cameras are arranged in parallel, there will be a high association between the multiview frames in the same time sequence. Even if the multi-view cameras are not arranged in parallel, the images captured from the multi-view cameras can be corrected in the same state as those taken by the multi-view cameras arranged in parallel. Therefore, the multi-view video may be encoded by using the correlation between the multi-view frames.

FIG. 5 is a diagram illustrating a case in which the variation of the parallax vector is constant, and FIG. 6 is a graph illustrating the relationship between the parallax vectors when the variation of the parallax vector is constant as shown in FIG. 5.

Vc and Vr in FIG. 5 are one line of MBs in any frames in the same time sequence. Vr is an already encoded reference view frame, and Vc is a frame predicted using Vr for current encoding. 6 shows a change in the size of DV according to the disparity vector DV of each MB. In Figure 6 the vertical axis represents the magnitude of the DV.

In FIG. 5, when the current MB is at position i, the previous MB is at position i-1 located to the left of the current MB. As shown in Figs. 5 and 6, assuming that the DVs of a group of consecutive MBs in the MB line change linearly, the difference between the DV _i of the MB at a given position and the DV _i-1 of the previous MB of the MB DDV _i is constant.

For example, in FIG. 6, dDV ₂ = DV ₂ -DV ₁ , dDV ₃ = DV ₃ -DV ₂ , dDV ₄ = DV ₄ -DV ₃ , and dDV _n = DV _n −DV _{n −1} The calculated dDV becomes a constant value. Therefore, if the parallax vector is encoded using the value of the first DV and the value of dDV, the decoding apparatus can predict the DV of each MB using this value.

The relationship between dDVs, which are deviations of the parallaxes of adjacent MBs, may be represented by ddDV, which is the deviations of the parallaxes. When the deviation of the parallax deviation of the MB of the current i position is referred to as ddDV _i , there is a relationship of ddDV _i = dDV _i − dDV _{i − 1} , and as shown in FIGS. 5 and 6, The deviation ddDV becomes zero. Therefore, when using the multi-view video coding ddDV _i again, in the decoding apparatus, using a ddDV _i and dDV _{i -1} from the previous MB of the current MB can be calculated dDV _i. That is, the deviation dDVi of the parallax vector of the macroblock at the current position i may be calculated as in Equation 3 below.

dDV _i = dDV _{i -1} + ddDV _i

When dDV _i is calculated, the parallax vector DV _i of the macroblock at the current position i may also be calculated by Equation 4.

DV _i = DV _{i -1} + dDV _i

As described above, the multi-view video is encoded using the deviation dDV of the parallax or the deviation ddDV of the parallax deviation, thereby improving the coding efficiency of the multiview video. In addition, when ddDV is encoded, compression efficiency may be further improved as compared with encoding dDV. In addition, by using Equations 3 and 4, it is possible to determine DV required when decoding a multi-view video coded using dDV or ddDV.

7 is a block diagram illustrating a configuration of a multiview video encoding apparatus according to an embodiment of the present invention.

A multi-view video encoding apparatus according to an embodiment of the present invention includes a parallax predictor 710, a parallax deviation calculator 720, an encoding mode selector 730, and an encoder 740. The parallax deviation calculation unit 720 may include a first buffer 721 for temporarily storing a parallax vector, a first order calculating unit 722 for calculating a deviation of the parallax vector, and a second buffer for temporarily storing the deviation of the parallax vector ( 723 and a second order execution unit 724 for calculating a deviation of the deviation.

The current frame and the reference frame, which are consecutive frames, are input to the disparity predictor 710. The disparity predictor 710 predicts the disparity vector DV in a predetermined block unit, for example, macroblock unit, between two frames. In this case, the deviation between the disparity vectors in units of blocks may be constant.

The parallax deviation calculator 720 calculates the deviation or deviation of the deviation between the predicted parallax vectors. In more detail, the predicted parallax is first input to the first buffer 721 of the parallax deviation calculator 720 and temporarily stored in the first buffer 721. Then, the parallax vector in units of blocks is input to the first-order operation performing unit 722, and when the parallax vector of the block at the current i position is DV _i , the difference between the parallax vector DV _i and the previous parallax vector DV _{i -1} is determined. The deviation dDV _i of the parallax vector is calculated.

The calculated deviation dDV _i of the parallax vector is input to the second buffer 723 and temporarily stored. The deviation dDV _i of the parallax vector temporarily stored in the second buffer 723 is inputted to the second order operation performing unit 724 again, and the deviation of the deviation of the parallax vector which is the difference between the deviation dDV _{i −1} between the dDV _i and the previous parallax vector. ddDV _i is calculated.

The encoder 740 encodes a multiview video by using the deviation of the parallax vector or the deviation of the deviation calculated by the parallax deviation calculator 720.

An encoding mode selection unit 730 whether to encode the multi-view video using the back dDV _i whether using ddDV _i encoding the multiview video is again selected according to predetermined criteria. The code screen mode selection unit 730 sets the flag Flag_ddDV for distinguishing a case using a case of using a dDV _i encoding the multiview video again and ddDV _i for encoding the multiview video is passed back to the encoding unit 740 do.

When the encoding mode selector 730 selects an encoding mode for encoding a multiview video using dDV _i , the encoding mode selector 730 may set the flag Flag_ddDV to 0 and transmit the same to the encoder 740. In addition, when the encoding mode selector 730 selects an encoding mode for encoding a multiview video using ddDV _i , the encoding mode selector 730 may set the flag Flag_ddDV to 1 and transmit it to the encoder 740.

The encoder 740 may perform multi-view video encoding according to the flag value transmitted from the encoding mode selector 730. When Flag_ddDV is set to 0, the encoder 740 encodes a multi-view video using dDV _i together with flag information. When Flag_ddDV is set to 1, the encoding unit 740 encodes a multi-view video using ddDV _i together with flag information.

8 is a diagram illustrating a multi-view video encoding method according to an embodiment of the present invention.

The current frame and the reference frame are received to predict a parallax vector between the frames in a predetermined block unit (S810). The deviation between the predicted parallax vectors or the deviation of the parallax vectors is calculated (S820). A multiview video is encoded using the deviation of the parallax vector or the deviation of the parallax vector (S830). Selecting between encoding according to a predetermined criterion between steps S 820 and S 830 based on a predetermined criterion, and performing encoding according to a predetermined criterion. In this case, the multiview video may be encoded according to the selected encoding mode.

9 is a block diagram illustrating a configuration of a multiview video decoding apparatus according to an embodiment of the present invention.

The multi-view video decoding apparatus according to an embodiment of the present invention includes an encoding mode checker 910, a disparity vector determiner 920, and a decoder 930.

The encoding mode checking unit 910 checks flags indicating a multiview video encoding mode included in the received multiview video.

The parallax vector determiner 920 determines a parallax vector for a multiview video according to the identified flag.

Hereinafter, a process of determining the parallax vector by the parallax vector determiner 920 will be described. The flag is encoded using the deviation of the parallax vector, and it is assumed that the deviation of the parallax vector of the predetermined block at the current i position, for example, the macroblock, is dDV _i . At this time, the time difference from the previous macroblock vector DV _{i -} in addition to determining the time difference vector DV of the macroblock of the current _i i _i where the deviation dDV of the _first and the parallax vector.

When the flag indicates that the flag is encoded using the deviation of the parallax vector, the parallax vector is determined as follows. First, when the deviation of the parallax vector of the macroblock at the current i position is ddDV _i , the deviation dDV _{i − 1} of the parallax vector of the previous macroblock and the deviation ddDV _i of the deviation of the parallax vector are added to the macro at the current i position. Determine the deviation dDV _i of the parallax vector of the block. Then, a determination is made prior to the macro-block differential vector by adding the deviation dDV _i of the differential vector is determined and the time difference DV _i-1 of the macroblocks of the current i location vector DV _i.

As described above, the decoder 930 decodes a multiview video using the identified flag, that is, the disparity vector determined according to the encoding mode.

The method according to the invention can be embodied as computer readable code on a computer readable recording medium. The code and code segments implementing the above program can be easily deduced by a computer programmer in the field. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, and the like, and may also include those implemented in the form of carrier waves (eg, transmission over the Internet). do. The computer-readable recording medium may also be distributed over a networked computer system and stored and executed in computer readable code in a distributed manner.

The above description is only one embodiment of the present invention, and those skilled in the art to which the present invention pertains may implement it in a modified form without departing from the essential characteristics of the present invention. Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be construed to include various embodiments within the scope of the claims.

As described above, according to the present invention, since the deviation of the parallax vector for each block or the deviation of the deviation is encoded using the relation between the parallax vectors, the compression rate of the multiview video can be improved.

According to the present invention, a method and apparatus for decoding a multiview video for decoding a multiview video encoded by using a correlation between disparity vectors with respect to a multiview video can be provided.

Claims (16)

A disparity predicting unit configured to receive a current frame and a reference frame and predict a disparity vector between the frames in a predetermined block unit; A parallax deviation calculator for calculating a deviation between the predicted parallax vectors or a deviation of the deviation; An encoding mode selector configured to select whether to encode a multiview video using the deviation between the parallax vectors or to encode the multiview video using the deviation of the deviation, and to set flag information according to the selected encoding mode ; And And an encoder which encodes a multiview video by using the calculated deviation of the disparity vector or the deviation of the deviation. The method of claim 1, And a deviation between the predicted disparity vectors is constant. delete The method of claim 1, wherein the parallax deviation calculator, A first buffer for storing the predicted parallax vector; A first difference calculating unit calculating a deviation between the parallax vectors stored in the first buffer; A second buffer for storing the deviation; And And a second order calculation performing unit calculating the deviation of the deviation. Receiving a current frame and a reference frame and predicting a parallax vector between the frames in a predetermined block unit; Calculating a deviation or deviation of the deviation between the predicted parallax vectors; Selecting whether to encode a multiview video using a deviation between the parallax vectors or to encode a multiview video using the deviation of the deviation according to a predetermined criterion; Setting flag information according to the selected encoding mode; And And encoding a multiview video using the deviation of the parallax vector or the deviation of the deviation. The method of claim 5, And a deviation between the predicted disparity vectors is constant. delete The method of claim 5, wherein the calculating of the deviation between the predicted parallax vectors or the deviation of the deviation comprises: Storing the predicted parallax vector; Calculating a deviation between the parallax vectors; Storing the deviation; And Multi-view video encoding method comprising the step of calculating the deviation of the deviation. An encoding mode checking unit which checks flags indicating multi-view video encoding modes included in the encoded multi-view video; A parallax vector determiner that determines a parallax vector for the multi-view video according to the checked flag; And And a decoder which decodes a multiview video using the determined disparity vector. And a flag indicating the multi-view video encoding mode indicates whether the multi-view video is encoded using a deviation between parallax vectors or is encoded using the deviation. The method of claim 9, wherein the parallax vector determining unit, When the flag indicates that the flag is encoded using the deviation of the parallax vector, when the deviation of the parallax vector of the block at the current i position is dDV _i , the deviation of the parallax vector DV _{i - 1} of the previous block and the parallax vector A multi-view video decoding apparatus, wherein dDV _i is added to determine a disparity vector DV _i of a block at a current i position. The method of claim 9, wherein the parallax vector determining unit, When the flag indicates that the deviation is encoded using the deviation of the parallax vector, the deviation of the parallax vector of the block at the current i position is ddDV _i , and the deviation of the parallax vector of the previous block dDV _{i - 1} And the deviation ddDV _i of the deviation of the parallax vector is added to determine the deviation dDV _i of the parallax vector of the block at the current i location, Differential vector DV of the previous block _i-1 and adding the deviation dDV _i of the determined parallax vector multiview video decoding apparatus, characterized in that for determining the time difference vector DV _i of a block of the current i location. Identifying flags indicating a multiview video encoding mode included in the encoded multiview video; Determining a parallax vector for the multi-view video according to the identified flag; And Decoding a multi-view video using the determined disparity vector; And a flag indicating the multiview video encoding mode indicates whether the multiview video is encoded by using a deviation between parallax vectors or by using the deviation of the deviation. The method of claim 12, wherein determining the parallax vector, When the flag indicates that the flag is encoded by using the deviation of the parallax vector, assuming that the deviation of the parallax vector of the block at the current i position is dDV _i , And determining the disparity vector DV _i of the block at the current i position by adding the disparity vector DV _{i - 1} of the previous block and the deviation dDV _i of the disparity vector. The method of claim 12, wherein determining the parallax vector, When the flag indicates that the flag is encoded using the deviation of the deviation of the parallax vector, when the deviation of the deviation of the parallax vector of the block at the current i position is ddDV _i , Deviation of the parallax vector of the previous block dDV _{i - 1} and adding the deviation ddDV _i of the deviation of the parallax vector and determining the deviation dDV _i of the differential vector of the block i in the current position; And And determining the parallax vector DV _i of the block at the current i position by adding the parallax vector DV _{i - 1} of the previous block and the determined deviation vector dDV _i . A computer-readable recording medium having recorded thereon a program for implementing the method according to any one of claims 5, 6 and 8. A computer-readable recording medium having recorded thereon a program for implementing the method according to any one of claims 12 to 14.

Images