KR102528365B1 - Video Encoding and Decoding using Resolution Enhancement Scheme - Google Patents

Abstract

The present invention provides a technique for encoding or decoding an image to which a resolution enhancement technique is selectively applied to improve image compression efficiency, and a reference picture buffer management technique and related information signaling technique necessary for selectively applying the resolution enhancement technique. Suggest.

Description

Video Encoding and Decoding using Resolution Enhancement Scheme {Video Encoding and Decoding using Resolution Enhancement Scheme}

The present invention relates to image encoding or decoding, and more particularly, to encoding and decoding of images selectively applying a resolution enhancement technique.

The contents described in this part merely provide background information on the present embodiment and do not constitute prior art.

As the resolution of an image increases, various efforts are being made to efficiently encode an image. The resolution enhancement technology has been mainly dealt with in the field of computer vision and image processing in the past, but various attempts have been made to introduce the resolution enhancement technology and the computer vision technology to the recent image compression technology. The paper "Adaptive downsampling / upsampling for better video compression at low bit rate" contributed to IEEE ISCAS in 2008, based on the distribution characteristics of frequency-converted DCT coefficients, four modes (normal, horizontal down-sampling, vertical down-sampling, 1/2 down-sampling) was defined to attempt down/up sampling in the frequency domain. The paper "Adaptive downsampling for high-definition video coding" contributed to IEEE TCSVT in 2014 designed a down/up sampling filter based on power spectral density to minimize the downsampling error of video frames. The paper "Convolutional neural network-based block up-sampling for intra frame coding" contributed to IEEE TCSVT in 2017 is a convolutional neural network (CNN)-based resolution enhancement technology that has recently shown excellent performance within the screen of HEVC. By applying it to the prediction process, the BD rate was reduced by 5.5% in the AI (all intra) environment.

The existing interpolation technique used to improve resolution uses a method such as taking a weighted average using pixel values of a low-resolution image in which the high-frequency component of the original high-resolution image has already been lost. Restoration is limited. Resolution enhancement techniques that have recently attracted attention include a patch matching method based on examples of such high-frequency components, a method based on motion registration, a method based on deep-learning, and the like. These resolution enhancement techniques can generate an image with a quality close to that of an original high-resolution image from a low-resolution image.

Deep learning-based resolution enhancement techniques are generally based on CNNs. This CNN-based super-resolution technology can be seen as performing a process of sequentially applying a 3D filter to non-linearly map a low-resolution image input to a high-resolution image. The corresponding 3D filter uses filter coefficients already learned through training.

An object of the present invention is to increase image compression efficiency by selectively applying a resolution enhancement technique to encode or decode an image.

According to one aspect of the present invention, in a method for decoding an image using a resolution enhancement technique, resolution mode information is decoded from a bitstream to determine whether a coding block, which is a block to be decoded, has been encoded using the resolution enhancement technique. identifying whether or not; decoding syntax elements for reconstructing the coding block from the bitstream and restoring the coding block using the decoded syntax elements; When the coding block is not coded using a resolution enhancement technique, the reconstructed coding block is stored in a first reference picture buffer, the reconstructed coding block is converted to a low resolution to generate a low resolution block, and the low resolution block is used as a second reference. storing in a picture buffer; and when the coding block is encoded by the resolution enhancement technique, storing the reconstructed coding block in the second reference picture buffer, converting the reconstructed coding block to a high resolution by the resolution enhancement technique, and generating a high resolution block; It provides an image decoding method comprising the step of storing the high-resolution block in the first reference picture buffer.

According to another aspect of the present invention, an apparatus for decoding an image using a resolution enhancement technique includes: a first reference picture buffer for storing a picture reconstructed in high resolution; a second reference picture buffer for storing a picture reconstructed in a low resolution; a decoder that decodes resolution mode information from a bitstream, identifies whether a coding block, which is a block to be decoded, has been coded using a resolution enhancement technique, and decodes syntax elements for reconstructing the coding block; and a reconstructor for reconstructing the coding block using syntax elements for reconstructing the coding block, wherein the reconstructor converts the reconstructed coding block into the first coding block when the coding block is not encoded using a resolution enhancement technique. Store in a reference picture buffer, generate a low-resolution block by converting the reconstructed coding block to a low resolution, store the low-resolution block in a second reference picture buffer, and when the coding block is coded with a resolution enhancement technique, the reconstructed coding block A coding block is stored in the second reference picture buffer, the reconstructed coding block is converted to a high resolution by the resolution enhancement technique to generate a high resolution block, and the high resolution block is stored in the first reference picture buffer. It provides a video decoding device that does.

1 is an exemplary block diagram of an image encoding apparatus according to an embodiment of the present invention;
2 is an exemplary flowchart illustrating a method for managing a reference picture buffer by an image encoding apparatus according to an embodiment of the present invention;
3 is an exemplary block diagram of a video decoding apparatus according to an embodiment of the present invention;
4 is an exemplary flowchart illustrating a method of managing a reference picture buffer by a video decoding apparatus according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. It should be noted that in adding identification codes to components of each drawing, the same components have the same symbols as much as possible even if they are displayed on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted.

The present disclosure relates to encoding or decoding of an image to selectively apply a resolution enhancement technique. The image encoding apparatus receives and encodes a high resolution (HR) image, and receives and encodes a low resolution (LR) image obtained by down-sampling the high resolution image. Then, an image having high encoding efficiency is selected from among the encoded high-resolution and low-resolution images and transmitted to the image decoding device. The video decoding apparatus decodes the received data and restores the video. If the received data corresponds to a high-resolution video, the restored video can be displayed as it is. It is displayed after conversion to high resolution.

In order to selectively apply such a resolution enhancement technique, a different reference picture management is required so that the video encoding apparatus and the video decoding apparatus can encode or decode an image by referring to the same reference picture. In addition, proper signaling of syntax indicating whether the encoded image is encoded using a resolution enhancement technique, that is, whether the encoded image is high resolution or low resolution obtained by downsampling the high resolution, is required.

1 is an exemplary block diagram of an image encoding apparatus according to an embodiment of the present invention.

An image encoding apparatus according to an embodiment of the present invention includes a first encoding module 1100, a second encoding module 1200, an evaluator 1300, a bitstream generator 1400, a first reference picture buffer ( 1500) and a second reference picture buffer 1600. Each component of the first encoding module 1100, the second encoding module 1200, the evaluator 1300, and the bitstream generator 1400 may be implemented as a hardware chip or implemented as software. The above microprocessor may be implemented to execute software functions corresponding to each component.

The first encoding module 1100 receives a high-resolution original image and performs encoding, and the second encoding module 1200 receives a low-resolution original image and performs encoding.

An image is composed of a plurality of pictures, each picture may be divided into one or more slices, and each slice may be divided into one or more coding tree units (CTUs). CTU is recursively partitioned using a tree structure. A leaf node in the tree structure becomes a coding unit (CU), which is a basic unit of encoding. As a tree structure, a QuadTree (QT) in which a parent node (or parent node) is divided into four subnodes (or child nodes) of the same size, or a QT structure and a parent node are divided into two subnodes A QTBT (QuadTree plus BinaryTree) structure using a binary tree (BinaryTree, BT) structure may be used. That is, QTBT can be used to divide a CTU into multiple CUs.

In the QTBT (QuadTree plus BinaryTree) structure, the CTU may first be divided into the QT structure. Quadtree splitting can be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of leaf nodes allowed by QT. If the leaf node of the quad tree is not larger than the maximum block size (MaxBTSize) of the root node allowed in BT, it can be further partitioned into a BT structure. In BT, a plurality of partition types may exist. For example, in some examples, there may be two types of horizontally splitting a block of a corresponding node into two equal-sized blocks (ie, symmetric horizontal splitting) and a vertical splitting type (ie, symmetric vertical splitting). In addition, a type in which a block of a corresponding node is divided into two blocks having an asymmetric shape may additionally exist. The asymmetric form may include a form in which the block of the corresponding node is divided into two rectangular blocks having a size ratio of 1:3, or a form in which the block of the corresponding node is divided in a diagonal direction.

A CU can have various sizes depending on the QTBT division from the CTU. Hereinafter, a block corresponding to a CU to be encoded or decoded (ie, a leaf node of QTBT) is referred to as a 'coding block'. A coding block becomes a basic unit for encoding and decoding.

The block partitioning structure of the high-resolution image input to the first encoding module 1100 and the low-resolution image input to the second encoding module 1120 may or may not be the same depending on the coding unit that distinguishes the high resolution and the low resolution. For example, if a high-resolution image and a low-resolution image are classified on a picture-by-picture basis, the block partitioning structure within a picture may be different. Conversely, if a high-resolution image and a low-resolution image are distinguished in units of coding blocks, the partitioning structure of the coding blocks must be the same.

Meanwhile, although the first encoding module 1100 and the second encoding module 1120 are shown as physically separated modules in FIG. 1, the present invention is not limited thereto. For example, the first encoding module 1100 and the second encoding module 1120 may be physically configured as one module, and a high-resolution image and a low-resolution image may be sequentially encoded.

The first encoding module 1100 receives a high-resolution image in picture units and encodes the input picture F. And, after reconstructing the encoded picture, a reconstructed picture (F') is generated. To this end, the first encoding module 1100 includes an intra prediction unit 1102, an inter prediction unit 1104, a subtractor 1106, a transform unit 1108, a quantization unit 1110, an encoding unit 1112, and an inverse quantization unit. 1114, an inverse transform unit 1116, an adder 1118, and a filter unit 1120.

Coding blocks within a picture may be each predictively coded. Prediction of a coding block is generally done using an intra-prediction technique (using data from a picture containing the coding block) or an inter-prediction technique (using data from a picture coded prior to the picture containing the coding block). can be performed To this end, the first encoding module 1100 includes an intra prediction unit 1102 and an inter prediction unit 1104.

The intra prediction unit 1102 predicts pixels in the current coding block using pixels (reference pixels) located around the current coding block in the current picture including the coding block to be currently encoded. A plurality of intra prediction modes exist according to prediction directions, and neighboring pixels to be used and operation expressions are defined differently according to each prediction mode. The intra prediction unit 1102 selects one intra prediction mode from among a plurality of intra prediction modes, and predicts a current coding block by using neighboring pixels (reference pixels) determined according to the selected intra prediction mode and an arithmetic expression. Information on the selected intra prediction mode is encoded by the encoder 1112.

The inter prediction unit 1104 generates a prediction block for a current coding block through a motion compensation process. Prior to the current picture, a block most similar to the current coding block is searched within the encoded and decoded reference picture, and a prediction block for the current coding block is generated using the searched block. Then, a motion vector corresponding to displacement between the current coding block in the current picture and the prediction block in the reference picture is generated. In general, motion estimation is performed on the luma component, and a motion vector calculated based on the luma component is used for both the luma component and the chroma component. Motion information including information on reference pictures used to predict the current coding block and information on motion vectors is encoded by the encoder 1112.

The subtractor 1106 subtracts the prediction block generated by the intra prediction unit 1102 or the inter prediction unit 1104 from the current coding block to generate a residual block.

The transform unit 1108 transforms the residual signal in the residual block having pixel values in the spatial domain into transform coefficients in the frequency domain. The transform unit 1108 may transform the residual signals in the residual block by using the size of the current coding block as a transform unit, or divide the residual block into a plurality of smaller subblocks and use the residual signal as a transform unit of the subblock size. can also convert them. There may be various methods of dividing a residual block into smaller subblocks. For example, it may be divided into subblocks having the same predefined size, or a QT (quadtree) method partitioning using a residual block as a root node may be used.

The quantization unit 1110 quantizes the transform coefficients output from the transform unit 1108 and outputs the quantized transform coefficients to the encoder 1112.

The encoder 1112 encodes the quantized transform coefficients using a coding scheme such as CABAC to generate a bitstream. In addition, the encoder 150 encodes information such as CTU size, MinQTSize, MaxBTSize, MaxBTDepth, MinBTSize, QT splitting flag, BT splitting flag, and splitting type related to block splitting so that the video decoding apparatus can perform the same operation as the video encoding apparatus. Blocks can be split.

The encoder 1112 encodes information about a prediction type indicating whether the current coding block is coded by intra prediction or inter prediction, and encodes intra prediction information or inter prediction information according to the prediction type. . When the current coding block is intra predicted, a syntax element for an intra prediction mode is encoded as intra prediction information. When the current coding block is inter-predicted, the encoder 1112 encodes syntax elements for motion information.

The inverse quantization unit 1114 inversely quantizes the quantized transform coefficients output from the quantization unit 1110 to generate transform coefficients. The inverse transform unit 1116 transforms transform coefficients output from the inverse quantization unit 1114 from a frequency domain to a spatial domain to restore a residual block.

The adder 1118 restores the current coding block by adding the reconstructed residual block and the prediction block generated by the intra predictor 1102 or the inter predictor 1104. The pixels in the reconstructed current coding block are used as reference pixels when intra-predicting the next block.

The filter unit 1120 performs deblocking filtering on boundaries between reconstructed blocks in order to remove blocking artifacts caused by block-by-block encoding/decoding.

Meanwhile, the video encoding apparatus according to the embodiment of the present invention further includes a down sampler (D/S, 1122). The downsampler 1122 generates a reconstructed picture DS(F') converted to a low resolution by downsampling the reconstructed picture F' after deblocking filtering.

The second encoding module 1200 receives a low-resolution image obtained by downsampling the high-resolution image in units of pictures and encodes the input picture f. After reconstructing the encoded picture, the reconstructed picture (f') is transformed using a resolution enhancement technique to generate a reconstructed picture (SR(f')) converted to high resolution. To this end, the second encoding module 1200 includes an intra prediction unit 1202, an inter prediction unit 1204, a subtractor 1206, a transform unit 1208, a quantization unit 1210, an encoding unit 1212, and an inverse quantization unit. A unit 1214, an inverse transform unit 1216, an adder 1218, and a filter unit 1220 are included. These components include an intra prediction unit 1102, an inter prediction unit 1104, a subtractor 1106, a transform unit 1108, a quantization unit 1110, an encoding unit 1112, Since the inverse quantization unit 1114, the inverse transform unit 1116, the adder 1118, and the filter unit 1120 perform the same function, detailed descriptions of each component of the second encoding module 1200 are avoided. omit for

The second encoding module 1200 further includes a super-resolution conversion unit (S/R, 1222), and the super-resolution conversion unit 1222 improves the resolution of the reconstructed picture f' output from the filter unit 1220. By transforming using the technique, a reconstructed picture SR(f') converted to high resolution is generated. As a resolution enhancement technique, an example-based patch matching method, a motion registration-based method, or a deep-learning-based method may be used.

The evaluator 1300 compares the encoding cost by the first encoding module 1100 with the encoding cost by the second encoding module 1200, and compares the data encoded by the first encoding module 1100 with the second encoding cost. 2 Resolution mode information for selecting one of data encoded by the encoding module 1200 is generated. As the encoding cost, a rate-distortion cost or the like may be used. Rate is an indicator representing the amount of data required for encoding, and distortion is an indicator representing loss of original data, that is, a difference between an original image and a reconstructed image after encoding.

The evaluator 1300 receives the amount of data required when the first encoding module encodes a picture from the encoding unit 1112, and the high-resolution original picture F and the reconstructed picture output from the filter unit 1120 Calculate the difference between (F'). Through this, a rate-distortion cost in the case of encoding by the first encoding module 1100 is calculated. In addition, the evaluator 1300 receives the amount of data required when the second encoding module encodes a picture from the encoding unit 1212, and uses the high-resolution original picture F and the super-resolution conversion unit 1222 to process the data. Calculate the difference between the reconstructed pictures (SR(f')) converted to high resolution output. Through this, a rate-distortion cost when the resolution enhancement technique is used in the second encoding module 1100 is calculated. Then, resolution mode information is generated by comparing the two rate-distortion costs. The resolution mode information is a control signal for managing the reference picture buffer and for allowing the bitstream generator 1400 to select one of the encoded data generated by the first encoding module and the encoded data generated by the second encoding module. works as

Based on the resolution mode information generated by the evaluator 1300, the bitstream generation unit 1400 includes encoded data output from the encoding unit 1112 of the first encoding module and the encoding unit 1212 of the second encoding module. ), selects any one of the encoded data output from ), generates the selected encoded data as a bitstream, and transmits it to the video decoding device. The bitstream includes resolution mode information generated by the evaluator 1300 .

Meanwhile, it has been described above that the evaluator 1300 calculates the rate-distortion cost in units of pictures and generates resolution mode information, but the present invention is not limited thereto. For example, the estimator 1300 may calculate the rate-distortion cost on a per coding block, CTU, or slice basis. In this case, resolution mode information is generated in units of coding blocks, CTUs, or slices. This means that the resolution mode information of the present invention can be coded with a syntax of any one level of CU, CTU, slice header, and picture parameter set (PPS) corresponding to a coding block and signaled to a video decoding device. .

The first reference picture buffer 1500 and the second reference picture buffer 1600 store a high-resolution reference picture and a low-resolution reference picture, respectively. Hereinafter, a method for the video encoding apparatus to manage reference picture buffers according to resolution mode information will be described.

2 is an exemplary flowchart illustrating a method of managing a reference picture buffer by an image encoding apparatus according to an embodiment of the present invention.

As mentioned above, the unit for which the evaluator 1300 of the video encoding apparatus compares the encoding cost between the first and second encoding modules may be a CU, CTU, slice, or picture. Hereinafter, the expression 'image area' is used to collectively refer to these units. That is, the image area may be any one of CU, CTU, slice, or picture.

The video encoding apparatus converts the encoding cost (cost(HR)) when the high-resolution image area F is encoded by the first encoding module and the low-resolution image area f obtained by downsampling the high-resolution image area F into the second encoding module. The encoding cost (cost(LR)) at the time of encoding by the encoding module is compared (S202). The video encoding device generates resolution mode information corresponding to the comparison result.

If cost(HR) is less than or equal to cost(LR) (eg, resolution mode information = 0), the video encoding device converts the data (high resolution image area F) encoded by the encoder 1112 of the first encoding module to Encoded data) is controlled to be output to the bitstream generator 1400 (S204). At this time, the bitstream generated by the bitstream generator 1400 includes resolution mode information indicating that the high resolution image area F is encoded. Then, the reconstructed high-resolution image area F' is stored in the first reference picture buffer 1500 (S206), and an image area DS(F') obtained by downsampling the reconstructed high-resolution image area F' is provided. 2 It is stored in the reference picture buffer 1600 (S208). For example, referring to FIG. 1 , when cost(HR) is less than or equal to cost(LR) (eg, resolution mode information = 0), switches 1 to 3 (sw1 to sw3) are all switched to position A. Accordingly, the encoded data output by the encoding unit 1112 of the first encoding module 1100 is output to the bitstream generator 1400 . In addition, the reconstructed high-resolution image area F' is stored in the first reference picture buffer 1500, and the image area DS(F') obtained by downsampling the reconstructed high-resolution image area F' is a second reference picture buffer 1500. It is stored in the picture buffer 1600.

On the other hand, if cost(HR) is greater than cost(LR) (eg, resolution mode information = 1), the video encoding apparatus converts the data (high resolution image area F) encoded by the encoder 1212 of the second encoding module to The down-sampled low-resolution image region (f) is controlled to be output to the bitstream generator 1400 (S210). At this time, the bitstream generated by the bitstream generator 1400 includes resolution mode information indicating that the low resolution image area f is encoded, that is, indicating that a resolution enhancement technique is used. Then, the reconstructed low-resolution image area f' is stored in the second reference picture buffer 1600 (S212), and the reconstructed low-resolution image area f' is converted to a high-resolution image area (SR) using a resolution enhancement technique. (f')) is stored in the first reference picture buffer 1500 (S214). For example, referring to FIG. 1, if cost(HR) is greater than cost(LR) (eg, resolution mode information = 1), Switches 1 to 3 (sw1 to sw3) are all switched to position B. Accordingly, the encoded data output by the encoder 1212 of the second encoding module 1200 is output to the bitstream generator 1400 . In addition, the reconstructed low-resolution image area f' is stored in the second reference picture buffer 1600, and the reconstructed low-resolution image area f' is converted to a high-resolution image area SR(f') through a resolution enhancement technique. )) is stored in the first reference picture buffer 1500.

In the present disclosure, encoding is performed by selectively applying a resolution enhancement technique in units of pictures, slices, CTUs, or CUs. That is, a high resolution image area may be directly encoded or a low resolution image area may be encoded after converting the high resolution image area into a low resolution image area. Therefore, the low-resolution image area may be referred to for predictive encoding of the high-resolution image area, and the high-resolution image area may be referred to for predictive encoding of the low-resolution image area. Therefore, if the reference picture buffer is managed as described above depending on whether the high-resolution image area is coded or the low-resolution image area is coded (whether the resolution enhancement technique is used), cross-reference between the high-resolution image area and the low-resolution image area becomes possible, so that more It is possible to encode an image efficiently.

3 is an exemplary block diagram of a video decoding apparatus according to an embodiment of the present invention.

An image decoding apparatus according to an embodiment of the present invention includes a decoder 3100, a reconstructor 3200, a first reference picture buffer 3300, and a second reference picture buffer 3400. Each component of the decoder 3100 and the restorer 3200 may be implemented as a hardware chip or as software, and may be implemented so that one or more microprocessors execute software functions corresponding to each component. .

The decoder 3100 decodes resolution mode information in units of image areas from the bitstream received from the video encoding apparatus. As described above, an image area may be any one of CU, CTU, slice, and picture.

In addition, the decoder 3100 decodes the bitstream received from the video encoding apparatus, extracts information related to block division, determines a current coding block to be decoded, and predicts information and a residual signal necessary for restoring the current coding block. extract information about

The decoder 3100 determines the size of the CTU by extracting information about the CTU size from a Sequence Parameter Set (SPS) or a Picture Parameter Set (PPS), and divides the picture into CTUs of the determined size. In addition, the CTU is divided using the tree structure by determining the CTU as the top layer of the tree structure, that is, the root node, and extracting division information for the CTU. For example, when splitting a CTU using a QTBT structure, first, a first flag (QT_split_flag) related to splitting of QT is extracted and each node is split into four nodes of a lower layer. Then, for a node corresponding to a leaf node of QT, a second flag (BT_split_flag) related to BT splitting and split type information are extracted to split the corresponding leaf node into a BT structure.

Meanwhile, when the decoder 3100 determines a current block to be decoded through partitioning of a tree structure, it extracts information about a prediction type indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra prediction, the decoder 3100 extracts a syntax element for an intra prediction mode of the current block. When the prediction type information indicates inter prediction, the decoder 3100 extracts syntax elements for motion information.

Also, the decoder 3100 extracts information on quantized transform coefficients of the current block as information on the residual signal.

The reconstructor 3200 reconstructs the encoded coding blocks. To this end, the reconstructor 3200 includes an inverse quantization unit 3202, an inverse transform unit 3204, an intra prediction unit 3206, an inter prediction unit 3208, an adder 3210, and a filter unit 3212. .

The inverse quantization unit 3202 inversely quantizes the quantized transform coefficients, and the inverse transform unit 3204 inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals to generate a residual block for the current coding block. .

The intra prediction unit 3206 is activated when the prediction type of the current coding block is intra prediction. The intra prediction unit 3206 determines the intra prediction mode of the current coding block among a plurality of intra prediction modes from the syntax element for the intra prediction mode extracted from the decoder 3100, and determines the intra prediction mode of the current coding block according to the intra prediction mode. A current block is predicted using reference pixels.

The inter-prediction unit 3208 determines a reference picture and a motion vector for inter-prediction of the current coding block using syntax elements for motion information extracted from the decoder 3100. Then, a current coding block is predicted using the determined reference picture and motion vector.

The adder 3210 restores the current coding block by adding the residual block output from the inverse transform unit and the prediction block output from the inter prediction unit or intra prediction unit. Pixels in the reconstructed current coding block are used as reference pixels when intra-predicting a block to be decoded later.

The filter unit 3212 performs deblocking filtering on boundaries between reconstructed blocks in order to remove blocking artifacts generated by block-by-block decoding.

Meanwhile, the reconstructor 3200 further includes a down sampler (D/S. 3214) and a super-resolution converter 3216. The down sampler 3214 and the super-resolution converter 3216 are selectively activated according to the resolution mode information extracted from the bitstream.

The downsampler 3214 is activated when the resolution mode information indicates high-resolution encoding, and converts the reconstructed picture into a low-resolution picture by downsampling. The reconstructor 3200 stores the reconstructed picture before downsampling in the first reference picture buffer 3300 and stores the low resolution picture after downsampling in the second reference picture buffer 3400 .

The super-resolution converter 3216 is activated when the resolution mode information indicates low-resolution coding, that is, when the resolution enhancement technique is used, and converts the reconstructed picture using the resolution enhancement technique to generate a high-resolution picture. . The reconstructor 3200 stores the reconstructed picture before applying the resolution enhancement technique in the second reference picture buffer 3400, and stores the high-resolution picture after applying the resolution enhancement technique in the first reference picture buffer 3300.

The high-resolution reference picture stored in the first reference picture buffer 3300 and the low-resolution reference picture stored in the second reference picture buffer 3400 are used to predict blocks to be decoded later. For example, when a coding block to be decoded is coded in high resolution, the reconstructor 3200 obtains a reference region used for prediction of the corresponding coding block from the first reference picture buffer 3300. On the other hand, when a coding block to be decoded later is coded at a low resolution (ie, when a resolution enhancement technique is used), the reconstructor 3200 uses the second reference picture buffer 3400 as a reference used for prediction of the coding block. get an area

For example, when the resolution mode information indicates high-resolution encoding, the reconstructor 3200 switches switch 4 (sw4) to position A to predict the current coding block using reference pictures stored in the first reference picture buffer 3300. make it possible In addition, the reconstructor 3200 switches all switches 1 to 3 (sw1 to sw3) to positions A to convert a reconstructed current picture including reconstructed coding blocks output from the filter unit into a low-resolution picture through the downsampler 3214. , and the reconstructed current picture before downsampling is stored in the first reference picture buffer 3300, and the low resolution picture after downsampling is stored in the second reference picture buffer 3400.

On the other hand, when the resolution mode information indicates low-resolution coding (when the resolution resolution technique is applied), the reconstructor 3200 switches switch 4 (sw4) to the B position to retrieve the reference picture stored in the second reference picture buffer 3400. It is possible to predict the current coding block using . In addition, the reconstructor 3200 switches all of the switches 1 to 3 (sw1 to sw3) to the B positions to convert the reconstructed current picture including the reconstructed coding blocks output from the filter unit through the super-resolution converter 3216. After converting to a high-resolution picture, the reconstructed current picture before conversion is stored in the second reference picture buffer 3400, and the high-resolution picture after conversion is stored in the first reference picture buffer 3300.

4 is an exemplary flowchart illustrating a method of managing a reference picture buffer by a video decoding apparatus according to an embodiment of the present invention.

The video decoding apparatus decodes resolution mode information from the bitstream (S402), decodes syntax elements necessary for restoring the current coding block, and then restores the current coding block using the decoded syntax elements (S404).

The video decoding apparatus determines whether the resolution mode information indicates high-resolution coding or low-resolution coding, that is, whether a resolution enhancement technique is applied (S406).

When the resolution mode information indicates high-resolution encoding, the video decoding apparatus stores the reconstructed coding block in the first reference picture buffer 3300 (S408). Then, the reconstructed coding block is converted to a low resolution through downsampling (S410), and the block converted to a low resolution is stored in a second reference picture buffer (S412).

When the resolution mode information indicates low-resolution coding (when a resolution enhancement technique is applied), the video decoding apparatus stores the reconstructed coding block in the second reference picture buffer 3400 (S414). Then, the reconstructed coding block is converted to high resolution using a resolution enhancement technique (S416), and the block converted to high resolution is stored in the first reference picture buffer 3300 (S418).

In the above, in order to selectively apply the resolution enhancement technique, a technique for designing two buffers for storing a high-resolution reference picture and a low-resolution reference picture, respectively, and managing the reference picture buffer has been described. However, it is also possible to manage the reference picture buffer in other ways. For example, it is also possible to use one reference picture buffer for storing high-resolution reference pictures. In this case, high-resolution reference pictures may be configured in one reference picture buffer as follows. When high-resolution coding is applied, the reconstructed block is stored in the reference picture buffer as it is, and when low-resolution coding is applied, the reconstructed block is converted to high resolution using resolution enhancement technology, and the block converted to high resolution is stored in the reference picture buffer. . In addition, when a block to be encoded or decoded is encoded in high resolution, the reference picture stored in the reference picture buffer is output as it is, so that the corresponding block can be predicted using the high resolution reference picture. If a block to be encoded or decoded is coded in a low resolution, a reference picture stored in a reference picture buffer is downsampled and output so that the corresponding block can be predicted using the low resolution reference picture.

Hereinafter, a method of signaling resolution mode information so that the video encoding apparatus and the video decoding apparatus can manage the reference picture buffer in the above-described manner will be described.

A. Signaling of resolution mode information

Resolution mode information may always be signaled, but a flag indicating whether resolution mode information is signaled at a higher level may be encoded first. When the corresponding flag is 0, resolution mode information is not signaled. In this case, all blocks belonging to the higher level perform high-resolution encoding like normal encoding/decoding. That is, the resolution enhancement technique is not applied. If the corresponding flag is 1, resolution mode information is signaled at a lower level of the higher level. For example, Table 1 is an example of encoding a corresponding flag as a sequence parameter set (SPS) syntax ( sps _super_resolution_coding_enabled_flag ).

Example A-1

Resolution mode information may be signaled in units of pictures, slices, CTUs, or CUs. That is, when signaled at the picture level, this is the syntax of a picture parameter set (PPS), when signaled at the slice level, this is the syntax of a slice header, when signaled at the CTU level, this is CTU syntax, and when signaled at the CU level, it is syntax of a slice header. It is encoded as the syntax of CU. Tables 2 to 5 below show syntax structures in each case.

In Tables 2 to 5, the resolution mode information is marked as low_resolution_coding_mode . If the value of low_resolution_coding_mode is 0, it means high-resolution coding, and if the value of low_resolution_coding_mode is 1, it means low-resolution coding (that is, applying a resolution enhancement technique).

That resolution mode information is signaled at a specific level means that all blocks belonging to the specific level are coded in the same resolution mode. For example, when the resolution mode information is included in the slice header and indicates high resolution encoding, all blocks in the corresponding slice are encoded in high resolution. When the resolution mode information is included in the CU level syntax, the resolution mode is individually determined for each coding block.

Meanwhile, a level at which resolution mode information is signaled may be adaptively determined. In this case, level information indicating a level at which resolution mode information is signaled is additionally encoded. The video decoding apparatus decodes the resolution mode information at a level indicated by the level information. For example, level information may be signaled in SPS or PPS, as shown in Table 6 or 7 below.

In Tables 6 and 7, level information is indicated as super_resolution_unit_type . super_resolution_unit_type can have the values shown in Table 8.

For example, if super_resolution_unit_type is 0, resolution mode information is signaled as PPS syntax as shown in Table 9, and if super_resolution_unit_type is 1, it is signaled as slice header syntax as shown in Table 10.

Example A-2

When the CU size (coding block size) is small, there is a high possibility that a large gain in compression efficiency cannot be obtained even if a resolution enhancement technique is used. Therefore, in this embodiment, the resolution enhancement technique is not applied to a CU having a size smaller than the critical size. That is, in this embodiment, the resolution mode information is selectively encoded according to the size of the CU (size of the coding block) at the CU level. That is, if the size of the CU is smaller than the critical size, the resolution mode information is not encoded, and if the CU size is larger than the critical size, the resolution mode information is encoded. Table 11 shows a syntax structure according to this embodiment.

Meanwhile, the threshold size may be a fixed value shared by the video encoding apparatus and the video decoding apparatus, or may be signaled at a higher level of the CU. That is, the video encoding apparatus determines a threshold size applied to a higher level and transmits information on the determined threshold size to the video decoding apparatus as syntax in the upper level. After determining the threshold size by decoding information indicating the threshold size, the video decoding apparatus decodes resolution mode information only for coding blocks having a size greater than or equal to the threshold among coding blocks belonging to a higher level. Table 12 is an example in which information on the threshold size is encoded as PPS syntax, but the present invention is not limited thereto, and information on the threshold size may be included as syntax of a higher level of a CU, for example, a CTU or a slice header.

Example A-3

In general, an image region coded using intra prediction, for example, an intra random access point (IRAP) picture, an I (intra) slice, or an intra coding block, which is a random access picture, occupies a large amount of data in the entire bitstream. Accordingly, the present embodiment selectively applies the resolution enhancement technique only to an image region coded using intra prediction, and performs only high-resolution encoding without applying the resolution enhancement technique to an image region not using intra prediction.

For example, when a resolution enhancement technique is selectively applied to an intra random access point (IRAP) picture, when nal_unit_type of the NAL unit header indicates an intra random access point (IRAP) picture, the resolution mode information is PPS or slice header Can be included as a syntax of. As described in the embodiment A-1, which level of syntax to include the resolution mode information may be indicated through level information. Alternatively, it may be fixed information shared by the video encoding apparatus and the video decoding apparatus.

When the resolution enhancement technique is selectively applied to I (intra) slices, when the slice_type syntax included in the slice header indicates I slices, it may be included as a slice header or CTU or CU level syntax. As described in the embodiment A-1, which level of syntax to include the resolution mode information may be indicated through level information. Alternatively, it may be fixed information shared by the video encoding apparatus and the video decoding apparatus. Table 13 is an example of a case where resolution mode information is signaled as CU level syntax when slice_type indicates I slice.

When the resolution enhancement technique is selectively applied to intra coding blocks, pred _mode_flag (information on a prediction type indicating whether a coding block is inter-predicted or intra-predicted), which is one of the CU-level syntaxes, indicates intra prediction When doing so, the resolution mode information is encoded. Table 14 is an example of encoding resolution mode information as CU level syntax when a coding block is an intra block using intra prediction.

Example A-4

Since there is a high correlation between adjacent images, a picture group composed of adjacent images is highly likely to have the same characteristics. Therefore, in this embodiment, the low_resolution_coding_mode value determined in the first picture in the picture group is referred to by the remaining pictures in the group.

As an example, based on an I (intra) picture consisting of only I slices, pictures up to immediately before the next I picture may be generated as one group. Here, the I picture may be an intra random access point (IRAP) picture. In this case, when nal _unit_type of the NAL unit header indicates an intra random access point (IRAP) picture, resolution mode information may be included as syntax of the PPS.

The super-resolution converter 1222 of the image encoding device of FIG. 1 and the super-resolution converter 3216 of the image decoding device of FIG. 3 may use any one resolution enhancement technique, but the high-frequency component is converted to an example based It is also possible to use a plurality of resolution enhancement techniques, such as a patch registration method, a motion registration-based method, or a deep-learning-based method. In this case, it is required to signal selection information indicating which resolution enhancement technique is applied among a plurality of resolution enhancement techniques so that the video encoding apparatus and the video decoding apparatus can use the same resolution enhancement technique.

B. Signaling of information indicating resolution enhancement techniques

The video encoding apparatus encodes a syntax indicating which one of a plurality of resolution enhancement techniques is used, and the video decoding apparatus converts an image region encoded in a low resolution into a high resolution one by using a resolution enhancement technique indicated by the corresponding syntax. Hereinafter, various methods of encoding syntax for indicating a resolution enhancement technique will be described.

Example B-1

Indicating whether the syntax for indicating the resolution enhancement technique is coded before encoding the syntax A flag may be additionally coded. If the corresponding flag is 0, syntax for indicating a resolution enhancement technique is not encoded. In this case, the video encoding apparatus and the video decoding apparatus use a default resolution enhancement technique. On the other hand, if the corresponding flag is 1, syntax for indicating a resolution enhancement technique is encoded.

Table 15 is an example of encoding these syntaxes in SPS.

in table 15 If sps _super_resolution_list_enabled_flag is 0, sps _super_resolution_list_id , a syntax for indicating resolution enhancement techniques, is not encoded. If sps_super_resolution_list_enabled_flag is 1, sps _super_resolution_list_id is encoded and the resolution enhancement technique indicated by sps_super_resolution_list_id is applied to all blocks that refer to the same SPS.

Meanwhile, these syntaxes may be encoded at the PPS level as shown in Table 16. In Table 16, if pps_super_resolution_list_enabled_flag is 1, pps_super_resolution_list_id , which is a syntax for indicating a resolution enhancement technique, is encoded, and the resolution enhancement technique indicated by pps_super_resolution_list_id is applied to all blocks in the corresponding picture.

In Tables 15 and 16 above, it has been described that two syntaxes (a syntax for indicating resolution enhancement and a flag indicating whether the syntax is encoded) are signaled at the same level, but in this embodiment, the two syntaxes are at different levels. may be signaled. That is, it is also possible to encode the flag at a higher level and encode a syntax for indicating a resolution enhancement technique at a lower level. For example, a flag may be coded at the SPS level, and a syntax for indicating a resolution enhancement technique may be coded at the PPS level.

Example B-2

In the case of a resolution enhancement technique based on deep-learning, it may be required to additionally signal filter coefficient information from an image encoding device to an image decoding device. This embodiment relates to additional signaling of filter coefficients when a resolution enhancement technique based on deep-learning is applied.

As an example, a flag indicating whether the deep learning-based resolution enhancement technique is applied may be encoded, and if the flag indicates that the deep learning-based resolution enhancement technique is applied, filter coefficients may be encoded.

Table 17 is an example of encoding related syntaxes at the SPS level.

Referring to Table 17, it indicates whether the deep learning-based resolution enhancement technique is applied. Encode sps_super_resolution_list_data_present_flag , and if sps_super_resolution_list_data_present_flag is 1 ( indicating that deep learning-based resolution enhancement technique is applied), filter coefficients are encoded through super_resolution_list_data(). If sps_super_resolution_list_data_present_flag is 0 (indicating that it is not a deep learning-based resolution enhancement technique), one of the resolution enhancement techniques other than the deep learning-based resolution enhancement technique encodes sps_super_resolution_list_id to indicate.

As another example, the value of sps _super_resolution_list_id is obtained without separately encoding a flag indicating whether the deep learning-based resolution enhancement technique is applied. If a deep learning-based resolution enhancement technique is indicated, filter coefficients may be encoded. For example, if there are a total of four resolution enhancement techniques and sps_super_resolution_list_id = 3 indicates a deep learning-based resolution enhancement technique, filter coefficients can be encoded as shown in Table 18.

Referring to Table 18, when sps _super_resolution_list_id is greater than 2 (ie, 3), filter coefficients for applying the deep learning-based resolution enhancement technique are encoded through super_resolution_list_data().

Meanwhile, in Tables 17 and 18, it has been described that related syntaxes are coded at the SPS level, but as shown in Tables 19 and 20 below, they may be coded at the PPS level.

As another example, as described in Tables 6 to 8 of Embodiment A-1, when the level at which the resolution mode information is signaled is adaptively determined, the filter coefficient may be differently encoded according to the level at which the resolution mode information is signaled. there is. For example, if super_resolution_unit_type of Table 8 is 0, information about filter coefficients applied in picture units is coded, and if super_resolution_unit_type is 1, information about filter coefficients applied in slice units is coded. Table 21 is an example of encoding related syntaxes at the SPS level, but encoding at the PPS level is also possible.

The above description is merely an example of the technical idea of the present embodiment, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights of this embodiment.

Claims (8)

A method for decoding an image using a resolution enhancement technique,
Decoding resolution mode information from a bitstream and identifying whether a coding block, which is a decoding target block, has been coded using a resolution enhancement technique;
decoding syntax elements for reconstructing the coding block from the bitstream and restoring the coding block using the decoded syntax elements;
When the coding block is not encoded using the resolution enhancement technique, the reconstructed coding block is stored in a first buffer, the reconstructed coding block is converted to a low resolution to generate a low resolution block, and the low resolution block is stored in a second buffer. doing; and
When the coding block is encoded by the resolution enhancement technique, the reconstructed coding block is stored in the second buffer, and the reconstructed coding block is converted to a high resolution by the resolution enhancement technique to generate a high resolution block. Including the step of storing in the first buffer,
Decoding the resolution mode information,
decoding level information indicating a level including the resolution mode information among a plurality of levels from the bitstream; and
Decoding the resolution information from a level indicated by the level information,
The plurality of levels include at least two or more of a picture level, a slice level, a coding tree unit (CTU) level, and a coding unit level. According to claim 1,
The step of restoring the coding block,
obtaining a reference region used to predict the coding block from the first buffer when the coding block is not coded using the resolution enhancement technique; and
Acquiring a reference region used to predict the coding block from the second buffer when the coding block is coded with the resolution enhancement technique
Image decoding method comprising a. delete delete A method for decoding an image using a resolution enhancement technique,
Decoding resolution mode information from a bitstream to identify whether a coding block, which is a decoding target block, has been coded using a resolution enhancement technique;
decoding syntax elements for reconstructing the coding block from the bitstream and restoring the coding block using the decoded syntax elements;
When the coding block is not encoded using the resolution enhancement technique, the reconstructed coding block is stored in a first buffer, the reconstructed coding block is converted to a low resolution to generate a low resolution block, and the low resolution block is stored in a second buffer. doing; and
When the coding block is encoded by the resolution enhancement technique, the reconstructed coding block is stored in the second buffer, and the reconstructed coding block is converted to a high resolution by the resolution enhancement technique to generate a high resolution block. Including the step of storing in the first buffer,
The resolution mode information,
A syntax element of the coding unit,
The video decoding method characterized in that the bitstream is decoded when the size of the coding block is greater than the threshold size. According to claim 5,
The video decoding method further comprising decoding information indicating the threshold size from the bitstream and determining the threshold size based on the information indicating the threshold size. A method for decoding an image using a resolution enhancement technique,
Decoding resolution mode information from a bitstream to identify whether a coding block, which is a decoding target block, has been coded using a resolution enhancement technique;
decoding selection information for indicating one of a plurality of resolution enhancement techniques from the bitstream;
decoding syntax elements for reconstructing the coding block from the bitstream and restoring the coding block using the decoded syntax elements;
When the coding block is not encoded using the resolution enhancement technique, the reconstructed coding block is stored in a first buffer, the reconstructed coding block is converted to a low resolution to generate a low resolution block, and the low resolution block is stored in a second buffer. doing; and
When the coding block is encoded using the resolution enhancement technique, the reconstructed coding block is stored in the second buffer, and the reconstructed coding block is converted into a high resolution block by the resolution enhancement technique indicated by the selection information. Generating and storing the high-resolution block in the first buffer.
Image decoding method comprising a. delete

Images