RU2794483C1 - Reference picture oversampling signalling with indication of constant window size in video bitstream - Google Patents

Abstract

FIELD: video encoding and decoding.

SUBSTANCE: invention is related specifically to signalling the size of an image or portions of an image that may vary from image to image. Method for decoding an encoded video bitstream is proposed, which includes: obtaining the first flag indicating whether a constant image size is used in an encoded video sequence including the current image; based on the first flag indicating that a constant image size is used, decoding the current image without performing reference image resampling; based on the first flag indicating that a constant image size is not used, obtaining a second flag indicating whether the match window size is signalled; based on the second flag indicating that the match window size is signalled, obtaining the match window size, determining a resampling factor between the current picture and the reference picture based on the match window size, and performing resampling of the reference picture on the current picture using the resampling factor.

EFFECT: increase of efficiency of encoding and decoding a video bitstream.

20 cl, 11 dwg, 1 tbl

Description

Article I. CROSS-REFERENCE TO RELATED APPLICATION

[1] This application claims priority of U.S. Provisional Application No. 62/903,601, filed September 20, 2019, and U.S. Patent Application No. 17/009,979, filed September 2, 2020, which are incorporated herein in their entirety.

Article II The technical field to which the invention belongs

[2] The disclosed subject matter relates to video coding and decoding, and more specifically to signaling the size of an image or portions of an image that may vary from image to image or from image part to image part.

Article III. State of the Art

[3] It is known to encode and decode video using motion-compensated inter-picture prediction. Uncompressed digital video may consist of a series of images, each image having a spatial size of, for example, 1920 x 1080 luma samples and associated chrominance samples. A series of images can have a fixed or variable image rate (informally also known as frame rate), such as 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920x1080 brightness sample resolution at 60Hz frame rate) requires bandwidth close to 1.5 Gbps. An hour of such video requires more than 600 GB of disk space.

[4] One of the goals of video encoding and decoding may be to reduce the redundancy of the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression, as well as a combination of both, can be used. Lossless compression refers to methods in which an exact copy of the original signal can be restored from a compressed source signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal suitable for the intended application. In the case of video, lossy compression is widely used. The amount of distortion allowed depends on the application; for example, users of certain consumer streaming applications can tolerate higher distortion than users of broadcast television applications. The achievable compression ratio may reflect the following: higher distortion resolution/tolerance may result in a higher compression ratio.

[5] The video encoder and video decoder may use techniques from several broad categories including, for example, motion compensation, transformation, quantization, and entropy coding, some of which will be presented below.

[6] Historically, video encoders and video decoders tended to work with a given picture size, which in most cases was defined and held constant for a coded video sequence (CVS), group of pictures (GOP), or similar multi-picture time frame. For example, in MPEG-2, the system design is known to change the horizontal resolution (and hence the image size) depending on factors such as scene activity, but only for I-pictures, hence typically for GOP. The resampling of reference pictures to use different resolutions in CVS is known, for example, from ITU-T Rec. H.263, Annex P. However, here the image size is not changed, only the reference pictures are resampled, with the result that only parts of the image canvas can potentially be used (in in the case of downsampling) or capture only parts of the scene (in the case of upsampling). In addition, H.263 Annex Q allows resampling of an individual macroblock by a factor of two (in each dimension), up or down. Again, the image size remains the same. The macroblock size is fixed in H.263 and therefore does not need to be transmitted.

[7] Image resizing in predicted images has become more common in modern video coding. For example, VP9 allows you to resample the reference image and change the resolution for the entire image. Similarly, some of the proposals made regarding VVC (including, for example, Hendry, et. al., "On adaptive resolution change (ARC) for VVC", JVT document JVET-M0135-v1, January 9-19, 2019, incorporated herein in its entirety) allow resampling of entire reference pictures with a different, higher or lower resolution. This document proposes to encode different resolution candidates in a sequence parameter set and refer to them using the syntax elements for each image in the image parameter set.

Article IV Disclosure of the essence of the invention

[8] In one embodiment, a method is provided for decoding an encoded video bitstream using at least one processor, the method including: obtaining a first flag indicating whether a constant image size is used in an encoded video sequence including a current image; based on the first flag indicating that a constant image size is used, decoding the current image without performing reference image resampling; based on the first flag indicating that a constant image size is not used, obtaining a second flag indicating whether the match window size is signaled; based on a second flag indicating that the match window size is signaled, obtaining the match window size, determining a resampling factor between the current picture and the reference picture based on the match window size, and performing resampling of the reference picture on the current picture using the resampling factor.

[9] In one embodiment, an apparatus is provided for decoding an encoded video bitstream, the apparatus including at least one memory element configured to store program code; and at least one processor configured to read the program code and operate in accordance with the instructions of the program code, the program code including: a first receive code configured to cause the at least one processor to receive a first flag indicating whether a constant image size in the encoded video sequence including the current image; a decoding code configured to cause at least one processor, based on the first flag indicating that a constant image size is used, to decode the current image without performing reference image resampling; a second obtaining code, configured to cause at least one processor, based on the first flag indicating that a constant image size is not used, to obtain a second flag indicating whether a matching window size is signaled; and execution code configured to cause at least one processor, based on a second flag indicating that a match window size is signaled, to obtain a match window size, determine a resampling ratio between the current picture and a reference picture based on the match window size, and perform resampling reference image on the current image using the resampling factor.

[10] In one embodiment, a non-transitory computer-readable medium is provided that stores instructions including: one or more instructions that, when executed by one or more processors of a device for decoding an encoded video bitstream, cause one or more processors to: receive a first flag indicating whether a constant image size is used in the encoded video sequence including the current image; based on the first flag indicating that a constant image size is used, decode the current image without performing reference image resampling; based on the first flag indicating that a constant image size is not used, obtain a second flag indicating whether the match window size is signaled; based on the second flag indicating that the match window size is signaled, obtain the match window size, determine a resampling ratio between the current picture and the reference picture based on the match window size, and perform resampling of the reference picture on the current picture using the resampling factor.

Article V Brief Description of Drawings

[11] Additional features, essence and various advantages of the disclosed subject matter will be more apparent from the following detailed description and the accompanying drawings, in which:

[12] FIG. 1 is a schematic illustration of a simplified block diagram of a communication system according to an embodiment.

[13] FIG. 2 is a schematic illustration of a simplified block diagram of a communication system in accordance with an embodiment.

[14] FIG. 3 is a schematic illustration of a simplified block diagram of a decoder according to an embodiment.

[15] FIG. 4 is a schematic illustration of a simplified block diagram of an encoder according to an embodiment.

[16] FIG. 5 is a schematic illustration of ARC/RPR parameter signaling options according to an embodiment.

[17] FIG. 6A-6B show schematic illustrations of examples of syntax tables in accordance with an embodiment.

[18] FIG. 7 is a schematic illustration of the transmission of an image size and a matching window in an SPS, in accordance with embodiments.

[19] FIG. 8 is a schematic illustration of the transmission of image size and matching window in PPS according to embodiments.

[20] FIG. 9 shows a flowchart of an exemplary process for decoding an encoded video bitstream in accordance with an embodiment.

[21] FIG. 10 is a schematic illustration of a computer system in accordance with an embodiment.

Article I. Implementation of the invention

[22] Figure 1 illustrates a simplified block diagram of a communication system (100) according to an embodiment of the present disclosure. System (100) may include at least two terminals (110 and 120) interconnected via network (150). For unidirectional data transmission, the first terminal (110) may encode video data at the local location for transmission to another terminal (120) via the network (150). The second terminal (120) may receive the encoded video data of another terminal from the network (150), decode the encoded data, and display the recovered video data. One-way communication may be common in media serving applications and the like.

[23] FIG. 1 shows a second pair of terminals (130, 140) providing support for bi-directional encoded video transmission, which may occur, for example, during a video conference. For bi-directional data transmission, each terminal (130, 140) may encode video data captured at a local location for transmission to another terminal over a network (150). Each terminal (130, 140) may also receive the encoded video data transmitted by the other terminal, may decode the encoded data, and may display the recovered video data on a local display device.

[24] In Fig. 1, terminals (110-140) may be depicted as servers, personal computers, and smartphones, but the principles of the present disclosure cannot be limited to this. Embodiments of the present disclosure find application in laptop computers, tablet computers, media players, and/or specialized video conferencing equipment. Network (150) is any number of networks that transmit encoded video data between terminals (110-140), including, for example, wired and/or wireless networks. The communication network (150) may communicate on circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks and/or the Internet. For the purposes of this discussion, the architecture and topology of the network (150) may not be essential to the operation of this disclosure, unless explained herein below.

[25] Figure 2 illustrates, as an example of application for the disclosed subject matter, the placement of a video encoder and a video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, videoconferencing, digital television, storage of compressed video on digital media including CD, DVD, memory card, etc., and so on. .

[26] The streaming system may include a capture subsystem (213), which may include a video source (201), such as a digital camera, producing, for example, a stream (202) of uncompressed video samples. This stream (202) of samples, depicted with a thick line to emphasize the large amount of data compared to encoded video bitstreams, can be processed by an encoder (203) connected to a camera (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (204), drawn with a thin line to emphasize the smaller amount of data compared to the sample stream, may be stored on the streaming server (205) for future use. One or more streaming clients (206, 208) can access the streaming server (205) to extract copies (207, 209) of the encoded video bitstream (204). The client (206) may include a video decoder (210) that decodes an incoming copy of the encoded video bitstream (207) and produces an outgoing stream (211) of video samples that can be displayed on a display (212) or other visualization device (not shown). In some streaming systems, video bitstreams (204, 207, 209) may be encoded according to certain video coding/compression standards. Examples of these standards include ITU-T Rec. H.265. A video coding standard is under development, informally known as Universal Video Coding or VVC. The disclosed subject matter may be used in the context of VVC.

[27] FIG. 3 may be a functional block diagram of a video decoder (210) according to an embodiment of the present disclosure.

[28] The receiver (310) may receive one or more encoded video sequences to be decoded by the decoder (210); in the same or another embodiment, one coded video sequence at a time, where the decoding of each coded video sequence is independent of other coded video sequences. An encoded video sequence may be received from a channel (312), which may be a hardware/software connection to a storage device that stores the encoded video data. The receiver (310) may receive encoded video data with other data, such as encoded audio data and/or ancillary data streams, which may be sent to their respective using entities (not shown). The receiver (310) may separate the encoded video sequence from other data. To combat network jitter, a buffer memory (315) can be connected between the receiver (310) and the entropy decoder/parser (320) (hereinafter "parser"). When the receiver (310) receives data from a storage/transfer device with sufficient bandwidth and controllability, or from an isosynchronous network, buffer memory (315) may not be needed or may be small. For use in best quality packet networks such as the Internet, a buffer (315) may be required, which may be relatively large and may preferably be of an adaptive size.

[29] The video decoder (210) may include a parser (320) for recovering symbols (321) from the entropy encoded video sequence. The categories of these symbols include information used to control the operation of the decoder (210) and potentially information to control a visualization device such as a display (212) which is not an integral part of the decoder but can be connected to it as shown. in Fig.3. The control information for the rendering device(s) may be in the form of incremental enhancement information (SEI messages) or video usability information (VUI) parameter set fragments (not shown). The parser (320) may parse/entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be in accordance with a video coding technology or standard, and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so on. The parser (320) may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOPs), pictures, tiles, slices, bricks, macroblocks, coding tree units (CTUs), coding units (CUs), blocks, transform units (TUs), prediction units (PUs), and so on. . A tile may indicate a rectangular area of a CU/CTU in a particular column and row of tiles in an image. A brick may point to a rectangular area of CU/CTU rows within a particular tile. A slice may point to one or more image bricks that are contained in a NAL unit. A sub-image may point to a rectangular area of one or more slices in an image. The entropy decoder/parser may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[30] The parser (320) may perform an entropy decoding/parsing operation of the video sequence received from the buffer (315) to create symbols (321).

[31] Symbol recovery (321) may include many different modules depending on the type of encoded video image or parts thereof (such as: external and internal image, external and internal block) and other factors. Which modules are involved and how can be controlled by the subgroup control information parsed from the encoded video sequence by the parser (320). The flow of such subgroup control information between the parser (320) and the plurality of modules is not shown below for clarity.

[32] In addition to the functional blocks already mentioned, the decoder 210 can be conceptually subdivided into a number of functional modules, as described below. In a practical implementation operating under commercial constraints, many of these modules interact closely with each other and can be at least partially integrated into each other. However, for purposes of describing the disclosed subject matter, the conceptual division into functional modules below is appropriate.

[33] The first module is the scaling/inverse transform module (351). The scaling/inverse transform module (351) receives the quantized transform coefficient, as well as control information, including which transform to use, block size, quantization factor, quantization scaling matrices, and so on, in the form of symbol(s) (321) from the parser ( 320). It can output blocks containing sample values that can be entered into the aggregator (355).

[34] In some cases, the output samples of the scaling/inverse transform module (351) may refer to an intra-coded block; that is, to a block that does not use prediction information from previously restored images, but can use prediction information from previously restored portions of the current image. Such prediction information may be provided by the intra prediction module (352). In some cases, the intra-prediction module (352) generates a block of the same size and shape as the reconstructed block using the already reconstructed surrounding information extracted from the current (partially reconstructed) image (358). The aggregator (355), in some cases, adds for each sample the prediction information generated by the intra-prediction module (352) to the output sample information provided by the scaling/inverse transform module (351).

[35] In other cases, the output samples of the scaling/inverse transform module (351) may refer to a block with intercoding and potentially with motion compensation. In such a case, the motion compensation prediction module (353) may access the reference picture memory (357) to retrieve samples used for prediction. After compensating for the movement of the selected samples according to the symbols (321) associated with the block, these samples can be added by the aggregator (355) to the output of the scaler/demap module (in this case called residual samples or residual signal) to generate output information. readings. Reference picture memory addresses from where the motion compensation prediction module selects prediction samples may be driven by motion vectors available to the motion compensation prediction module in the form of symbols (321), which may have X, Y and reference picture components, for example. Motion compensation may also include interpolation of sample values retrieved from reference picture memory when accurate sub-sample motion vectors are used, motion vector prediction mechanisms, and so on.

[36] The output samples of the aggregator (355) may be subjected to various loop filtering methods in the loop filter module (356). The video compression technologies may include in-loop filtering technologies that are controlled by parameters included in the encoded video bitstream and provided to the loop filtering module (356) as symbols (321) from the parser (320), but may also respond to metainformation received during decoding the previous (in the order of decoding) parts of the encoded image or encoded video sequence, as well as responding to previously restored and filtered by loop filtering sample values.

[37] The output of the loop filter module (356) may be a stream of samples that can be output to the renderer (212) and also stored in reference picture memory for use in future inter-picture prediction.

[38] Certain encoded pictures after full recovery can be used as reference pictures for future prediction. Once the encoded picture has been completely reconstructed and the encoded picture has been identified as a reference picture (eg, by parser (320)), the current reference picture (358) may become part of the reference picture buffer (357), and fresh current picture memory may be reallocated before the beginning of the reconstruction of the next encoded picture.

[39] Video decoder 210 may perform decoding operations according to a predetermined video compression technology, which may be documented in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax given by the video compression technology or standard used, in the sense that it conforms to the syntax of the video compression technology or standard as specified in the video compression technology document or standard, and in particular their profile document. It may also be necessary for compliance that the complexity of the encoded video sequence be within limits defined by the level of technology or video compression standard. In some cases, levels limit the maximum image size, the maximum frame rate, the maximum recovery sampling rate (measured in megasamples per second, for example), the maximum reference image size, and so on. The limits set by the layers may, in some cases, be further limited by hypothetical reference decoder (HRD) specifications and metadata for managing the HRD buffer transmitted in the encoded video sequence.

[40] In an embodiment, the receiver (310) may receive additional (redundant) data with encoded video. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (210) to correctly decode the data and/or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, and so on.

[41] FIG. 4 may be a functional block diagram of a video encoder (203) according to an embodiment of the present disclosure.

[42] The encoder (203) may receive video samples from a video source (201) (which is not part of the encoder) that may capture video image(s) for encoding by the encoder (203).

[43] The video source (201) may provide the source video sequence for encoding by the encoder (203) in the form of a digital stream of video samples, which may have any suitable bit depth (eg: 8 bit, 10 bit, 12 bit, ...), any color space (eg BT.601 Y CrCB, RGB, …) and any suitable sample structure (eg Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (201) may be a storage device storing pre-rendered video. In a videoconferencing system, the video source (203) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual images that, when viewed sequentially, convey motion. The images themselves can be organized as a spatial array of pixels, where each pixel can contain one or more samples depending on the sample structure used, color space, etc. One skilled in the art can easily understand the relationship between pixels and samples. The description below is based on readings.

[44] According to an embodiment, the encoder (203) may encode and compress images of the source video sequence into the encoded video sequence (443) in real time or with any other time constraints as required by the application. Providing an appropriate coding rate is one of the functions of the controller (450). The controller controls other functional modules, as described below, and is functionally associated with these modules. The connection is not shown for clarity. The parameters set by the controller may include parameters related to speed control (picture skip, quantizer, speed-distortion optimization methods lambda value, ...), image size, group of pictures (GOP) layout, maximum motion vector search range, and so on. . A person skilled in the art can easily determine other functions of the controller (450) as they may relate to a video encoder (203) optimized for a particular system design.

[45] Some video encoders operate in what one of ordinary skill in the art would readily recognize as a "coding loop". As a simplified description, an encoding loop may consist of an encoder coding part (430) (hereinafter "source encoder") (responsible for creating symbols based on the input picture to be encoded and the reference picture(s)), and a (local) decoder (433) built into the encoder (203) that decompresses the symbols to create sample data that the (remote) decoder can also create (because any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies considered in the disclosed subject matter) . This reconstructed sample stream is entered into the reference picture memory (434). Because decoding a stream of symbols results in bit-accurate results, regardless of the location of the decoder (local or remote), the contents of the reference picture buffer are also bit-accurate between the local encoder and the remote encoder. In other words, the prediction portion of the encoder "sees" as reference picture samples exactly the same sample values that the decoder would "see" using prediction during decoding. This fundamental principle of reference picture synchronism (and the resulting drift if synchronism cannot be maintained due to, for example, channel errors) is well known to those skilled in the art.

[46] The operation of the "local" decoder (433) may be the same as that of the "remote" decoder (210), which has already been described in detail above in the description of FIG. 3. Briefly referring also to FIG. 4, however, since the symbols are available and encoding/decoding of the symbols into the encoded video sequence by the entropy encoder (445) and parser (320) can be lossless, the entropy decoding part of the decoder (210) including the channel ( 312), receiver (310), buffer (315), and parser (320) cannot be fully implemented in the local decoder (433).

[47] At this point, it can be observed that any decoding technology other than parsing/entropy decoding that is present in a decoder must also necessarily be present in a substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter focuses on the operation of the decoder. The description of encoding technologies can be abbreviated as they are the inverse of fully described decoding technologies. Only in some areas a more detailed description is required, which is given below.

[48] As part of its operation, the source encoder (430) may perform motion-compensated predictive encoding that predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence, which have been designated "reference frames". Thus, the coding engine (432) encodes the differences between the pixel blocks of the input frame and the pixel blocks of the reference frame(s) that can be selected as the prediction reference(s) for the input frame.

[49] The local video decoder (433) may decode the encoded video data of the frames, which may be designated as reference frames, based on the symbols generated by the source encoder (430). The operations of the encoding engine (432) may preferably be lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence can typically be a copy of the original video sequence with some errors. The local video decoder (433) replicates the decoding processes that may be performed by the video decoder on the reference frames and may cause the recovered reference frames to be stored in the reference picture cache (434). Thus, the encoder (203) can locally store copies of the reconstructed reference frames that share content as the reconstructed reference frames to be received by the far-end video decoder (in the absence of transmission errors).

[50] The predictor (435) may perform predictive searches for the encoding engine (432). That is, for a new picture to be encoded, the predictor (435) may search the reference picture memory (434) for sample data (as reference pixel block candidates) or certain metadata such as reference picture motion vectors, block shapes, and so on. further, which can serve as a suitable prediction reference for new images. The predictor (435) may operate on a block-by-pixel basis of samples to find appropriate prediction references. In some cases, as determined by the search results obtained by the predictor (435), the input image may have prediction references taken from a plurality of reference pictures stored in the reference picture memory (434).

[51] The controller (450) may control the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[52] The output signals of all the above functional modules can be entropy encoded in an entropy encoder (445). The entropy encoder translates symbols generated by various functional modules into an encoded video sequence by lossless symbol compression according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, and so on.

[53] The transmitter (440) may buffer the encoded video sequence(s) generated by the entropy encoder (445) to prepare it for transmission over a communication channel (460), which may be a hardware/software connection to a storage device that will store the encoded video data. . The transmitter (440) may merge the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and/or ancillary data streams (sources not shown).

[54] The controller (450) may control the operation of the encoder (203). During encoding, the controller (450) may assign to each encoded picture a certain type of encoded picture, which may affect the encoding methods that can be applied to the corresponding picture. For example, images can often be assigned to one of the following frame types:

[55] An intra-picture (I-picture) may be one that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow various types of internal images, including, for example, images with independent decoder update. The person skilled in the art will be aware of these variants of I-images and their respective uses and features.

[56] A predictive picture (P-picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most one motion vector and a reference index to predict sample values of each block.

[57] A bi-directional predictive picture (B-picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most two motion vectors and reference indices to predict sample values of each block. Similarly, multi-prediction images may use more than two reference images and associated metadata to reconstruct a single block.

[58] Source images can typically be spatially divided into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded on a block-by-block basis. The blocks may be predictively encoded with reference to other (already encoded) blocks, as determined by the coding assignment applied to the corresponding block images. For example, I-picture blocks may be predictively encoded or they may be predictively encoded with reference to already encoded blocks of the same picture (spatial prediction or intra prediction). P-picture pixel blocks may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one previously encoded reference picture. Blocks of B-pictures may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one or two previously encoded reference pictures.

[59] The video encoder (203) may perform encoding operations in accordance with a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In operation, the video encoder (203) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore follow the syntax given by the video coding technology or standard being used.

[60] In an embodiment, transmitter (440) may transmit additional data with encoded video. Video encoder (430) may include such data as part of an encoded video sequence. The additional data may comprise temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant images and slices, incremental enhancement information (SEI) messages, video usability information (VUI) parameter set fragments, and so on.

[61] Recently, some attention has been drawn to the aggregation of a compressed area, or the extraction of multiple semantically independent parts of an image into a single video image. In particular, in the context of e.g. 360 encoding or certain surveillance applications, multiple semantically independent source images (e.g., the surface of six cubes of a cube-projected 360 scene, or individual camera inputs in the case of a multi-camera surveillance setup) may require separate adaptive resolution settings to cope with different activity of each scene at a given time. In other words, encoders at a given point in time may choose to use different resampling factors for different semantically independent images that make up the entire 360-degree or observation scene. When combined into a single picture, there is a need to perform resampling of the reference picture and to have adaptive resolution coding signaling available for portions of the encoded picture.

[62] Below are a few terms that are referenced in the remainder of this description.

[63] A sub-picture may refer, in some cases, to a rectangular arrangement of samples, blocks, macroblocks, coding units, or the like, which are semantically grouped and which can be independently encoded at a changed resolution. One or more subimages may form an image. One or more encoded sub-pictures may form an encoded picture. One or more sub-images may be assembled into an image, and one or more sub-images may be extracted from the image. In certain environments, one or more encoded sub-pictures may be collected in the compressed area without being re-coded to the sample level in the encoded picture, and in the same or other cases, one or more encoded sub-pictures may be extracted from the encoded picture in the compressed area.

[64] Hereinafter, reference picture resampling (RPR) or adaptive resolution change (ARC) may refer to mechanisms that allow changing the resolution of an image or sub-picture in a coded video sequence, for example, by reference picture resampling. Hereinafter, RPR/ARC parameters refer to control information necessary to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, output and/or reference picture resolutions, various control flags, and so on.

[65] In embodiments, encoding and decoding may be performed on a single semantically independent encoded video image. Before describing the consequences of encoding/decoding multiple sub-pictures with independent RPR/ARC parameters and their implied additional complexity, RPR/ARC parameter transmission options must be described.

[66] FIG. 5 shows several embodiments of RPR/ARC parameter signaling. As noted for each of the embodiments, they may have certain advantages and certain disadvantages in terms of coding efficiency, complexity, and architecture. A video coding standard or technology may choose one or more of these or prior art embodiments for transmitting the RPR/ARC parameters. The embodiments may not be mutually exclusive and may possibly be interchangeable depending on the needs of the application, the standards used, or the choice of encoder.

[67] RPR/ARC parameter classes may include:

[68] upsampling/downsampling factors, separate or combined in the X and Y dimensions

[69] upsampling/downsampling factors with the addition of a temporal dimension, indicating an increase/decrease in size at a constant rate for a given number of images

[70] - either of the above two options may include encoding one or more supposedly short syntax elements that may point to a table containing the coefficient(s).

[71] is the resolution in the X or Y dimension in units of samples, blocks, macroblocks, coding units (CU), or any other suitable granularity of the input image, output image, reference image, encoded image, in combination or individually. If there is more than one resolution (eg, one for the input image, one for the reference image), then in some cases one set of values may be derived from another set of values. This can be communicated, for example, by using flags. See below for a more detailed example.

[72] - "warp" coordinates like those used in H.263 Annex P, again with the appropriate granularity as described above. Annex P of H.263 defines one efficient way to encode such warp coordinates, but other, potentially more efficient ways may also be developed. For example, the reverse Huffman encoding with variable length warp coordinates according to Annex P can be replaced by a binary encoding of an appropriate length, where the length of the binary codeword can be, for example, obtained from the maximum image size, perhaps multiplied by a certain factor and shifted by a certain value, so that take into account "deformation" beyond the boundaries of the maximum image size.

[73] - Upsampling and/or downsampling filter parameters. In embodiments, there may be only one filter for upsampling and/or downsampling. However, in embodiments, it may be desirable to provide more flexibility in filter design, and this may require passing filter parameters. Such parameters may be selected by an index into a list of possible filter designs, the filter may be fully specified (e.g., by a list of filter coefficients using suitable entropy coding techniques), the filter may be implicitly selected by appropriate upsampling/downsampling relationships that , in turn, are transmitted according to any of the mechanisms mentioned above, and so on.

[74] The following description assumes encoding a finite set of upsampling/downsampling coefficients (the same coefficient to be used in the X and Y dimensions) indicated by a codeword. This codeword may be encoded in a variable length, for example, using the Exponential Golomb code common to certain syntax elements in video coding specifications such as H.264 and H.265. One suitable mapping of values to upsampling/downsampling factors could, for example, follow Table 1:

TABLE 1

A codeword Exponential Golomb Code Source/Target resolution 0 1 1/1 1 010 1/1.5 (50% increase) 2 011 1.5/1 (50% reduction) 3 00100 1/2 (100% increase) 4 00101 2/1 (100% reduction)

[75] Many similar mappings can be developed according to the needs of the application and the capabilities of the upscaling and downscaling mechanisms available in the video compression technology or standard. The table can be expanded to more values. Values may also be represented by entropy encoding mechanisms other than Exponential Golomb codes, such as using binary encoding. This may have certain advantages when the oversampling factors were of interest outside of the video processing engines themselves (primarily the encoder and decoder), for example via MANE. It should be noted that for situations where a resolution change is not required, Exponential Golomb can be chosen, which is short; there is only one bit in the table above. This may have an advantage in coding efficiency over using binary codes for the most common case.

[76] The number of entries in the table, as well as their semantics, can be completely or partially customizable. For example, the basic table structure may be conveyed in a "high" parameter set, such as a sequence or a decoder parameter set. In embodiments, one or more such tables may be defined in a video coding technology or standard and may be selected, for example, by a decoder or sequence parameter set.

[77] The following is a description of how the upsampling/downsampling ratio (ARC information) encoded as described above can be included in a video coding technology or standard syntax. Similar considerations may apply to one or more codewords driving the upsampling/downsampling filters. See the discussion below for when a filter or other data structure requires relatively large amounts of data.

[78] As shown in FIG. 5, the H.263 application P includes ARC information (502) in the form of four warp coordinates in the picture header (501), specifically, in the H.263 PLUSPTYPE header extension (503). This may be a reasonable design choice when a) there is an image header available and b) frequent changes to the ARC information are expected. However, the overhead when using H.263 style transmission can be quite high, and the scaling factors may not be related to the borders of the image, since the image header may be temporary.

[79] In the same or another embodiment, the transmission of ARC parameters may correspond to the detailed example as shown in FIGS. 6A-6B. Fig. 6A-6B depict syntax diagrams in a representation type using a notation that roughly corresponds to C-style programming, such as that used in video coding standards since at least 1993. Lines in bold indicate syntax elements present in the bitstream, lines without bold often indicate control flow or setting variables.

[80] As shown in FIG. 6A, a tile group header (601), as an exemplary header syntax structure applicable to a (possibly rectangular) portion of a picture, may conventionally contain a variable length syntax element encoded with the exponential Golomb code dec_pic_size_idx (602 ) (highlighted in bold). The presence of this syntax element in a tile group header can be restricted using adaptive resolution (603) - here the flag value is not bold, meaning that the flag is present in the bitstream at the point where it occurs in the syntax diagram. The fact whether adaptive resolution is used for this image or parts of it can be conveyed in any high-level syntactic structure inside or outside the bitstream. In the example shown, this is passed in a sequence parameter set, as shown below.

[81] FIG. 6B also shows a selection of a set (610) of sequence parameters. The first syntax element shown is adaptive_pic_resolution_change_flag (611). When true, this flag may indicate the use of adaptive resolution, which in turn may require certain control information. In the example, such control information is conditionally present based on the value of the flag based on the if() statement in the parameter set (612) and the tile group header (601).

[82] When adaptive resolution is used, the output resolution is encoded in units of samples (613) in this example. Position number 613 refers to both output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together can determine the resolution of the output image. Elsewhere in a video coding technology or standard, certain restrictions may be defined for any value. For example, the level definition may limit the number of total output samples that can be the product of the values of these two syntax elements. Also, certain video coding technologies or standards, or external technologies or standards such as, for example, system standards, may restrict the numbering range (for example, one or both dimensions must be divisible by a power of 2) or width-to-height ratio (for example, width and height must be in a ratio such as 4:3 or 16:9). Such restrictions may be introduced to facilitate hardware implementations or for other reasons, and are well known in the art.

[83] In some applications, it may be desirable for the encoder to instruct the decoder to use a specific reference picture size instead of implicitly assuming that this size is the size of the output picture. In this example, the syntax element reference_pic_size_present_flag (614) conveys the conditional presence of the dimensions (615) of the reference picture (again, the position number refers to both width and height).

[84] Finally, a table of possible decoding image width and height is shown. Such a table may be expressed, for example, by specifying a table (num_dec_pic_size_in_luma_samples_minus1) (616). "Minus1" may refer to the interpretation of the meaning of this syntax element. For example, if the encoded value is zero, there is one entry in the table. If the value is five, there are six table entries. Then, for each "row" in the table, the width and height of the decoded picture are included in the syntax (617).

[85] Represented table entries (617) can be indexed using the dec_pic_size_idx (602) syntax element in the tile group header, thereby allowing different decoded sizes—essentially scaling factors—for each tile group.

[86] Some video coding technologies or standards, such as VP9, support spatial scalability by implementing certain forms of reference picture resampling (transmitted quite differently than in the disclosed subject matter) in combination with temporal scalability to provide spatial scalability. In particular, some reference pictures may be upsampled using ARC-style techniques to a higher resolution to form the base of the spatial enhancement layer. These upsampled images can be enhanced using normal high resolution prediction engines to add detail.

[87] The embodiments discussed herein may be used in such an environment. In some cases, in the same or another embodiment, a value in the NAL unit header, such as a temporal identifier field, may be used to indicate not only the temporal but also the spatial layer. This may have certain advantages for certain system configurations; for example, an existing Selected Forwarding Units (SFU) architecture created and optimized for selected temporal slices forwarded based on the NAL Unit Header Temporary Identifier value can be used without modification for scalable environments. In order to make this possible, there may be a requirement for a mapping between the encoded picture size and the temporal layer indicated by the temporal identifier field in the NAL unit header.

[88] Recently, some attention has been drawn to the aggregation of a compressed area, or the extraction of multiple semantically independent parts of an image into a single video image. In particular, in the context of e.g. 360 encoding or certain surveillance applications, multiple semantically independent source images (e.g., the six cube surface of a cube-projected 360 scene, or individual camera inputs in the case of a multi-camera surveillance setup) may require separate adaptive resolution settings to cope with different activity of each scene at a given time. In other words, encoders at a given point in time may choose to use different resampling factors for different semantically independent pictures that make up the entire 360-degree or observation scene. When combined into a single picture, there is a need to perform resampling of the reference picture and to have adaptive resolution coding signaling available for portions of the encoded picture.

[89] In embodiments, not all samples of the reconstructed image are intended for output. The encoder may indicate the rectangular portion of the image to be output using the match box. The matching window may be described or indicated, for example, by offsets to the left and right from the edges of the image, as determined by the size of the image. Certain use cases can be identified where lookup windows can matter, including stretched unfolding, spatial assembly of views in Multiview systems, or 360 systems where lookup windows can point to one of several cubemap surfaces for output.

[90] Since not all applications require the use of a match window, and since match window parameters may require a certain number of bits in the bitstream, and therefore, when not used, coding efficiency may be detrimental, and the availability of such parameters may be limited by a flag.

[91] In embodiments, the fit window size may be signaled in a Picture Parameter Set (PPS). The match window parameters, which may indicate the size of the match window, can be used to calculate the resampling factor when the match window size of the reference picture is different from the window size of the current picture. The decoder may need to recognize the size of the matching window of each picture to determine if a resampling process is required. When the oversampling factor is not 1, the size of the output image is not constant in the coded video sequence (CVS), and special processing and post-processing of the output images, such as zoom in/out for display, may be used.

[92] In embodiments, a flag that indicates whether the decoded/output image is the same size and the resampling factor is 1 in the coded video sequence (CVS)/bitstream may be signaled in a high-level parameter set such as a decoding parameter set ( DPS), Video Parameter Set (VPS), or Sequence Parameter Set (SPS). The flag can be used to negotiate a session for video streaming or decoder configuration and display settings.

[93] Referring to FIG. 7, a constant_pic_size_flag (704) of 1 may indicate that the image sizes in CVS are the same. However, constant_pic_size_flag equal to 0 may indicate that image sizes for images in CVS may or may not be the same. If the value of constant_pic_size_flag is 1, the sps_conformance_window_flag (705) may be represented in the SPS (701). Here, the sps_conformance_window_flag equal to 1 may indicate that the conformance crop window offset parameters follow in an appropriate position, eg, next in the SPS. However, a sps_conformance_window_flag of 0 may indicate that there are no offset parameters for a conformance crop window offset parameter.

[94] In embodiments, sps_conf_win_left_offset (706), sps_conf_win_right_offset (707), sps_conf_win_top_offset (708), and sps_conf_win_bottom_offset (709) may indicate image samples in CVS that are output from the image decoding process as a rectangular area specified in image coordinates for output .

[95] In embodiments, when the sps_conf_win_left_offset, sps_conf_win_right_offset, sps_conf_win_top_offset, and sps_conf_win_bottom_offset syntax elements are not present, the values of sps_conf_win_left_offset, sps_conf_win_right_offset, sps_conf_win_top_offset, and sps_conf_win_bottom_offset may be output as 0.

[96] In embodiments, as shown in FIG. 8, pic_width_in_luma_samples (802) may determine the width of each decoded picture based on PPS (801) in units of luma samples. In embodiments, pic_width_in_luma_samples may not be equal to 0, may be an integer multiple of Max( 8, MinCbSizeY), and may be less than or equal to pic_width_max_in_luma_samples. If absent, the value of pic_width_in_luma_samples may be inferred to be equal to pic_width_max_in_luma_samples. Here, pic_height_in_luma_samples (803) may indicate the height of each decoded picture based on PPS in units of luma samples. In this case, pic_height_in_luma_samples in some cases may not be equal to 0, but may be an integer multiple of Max (8, MinCbSizeY), and may be less than or equal to pic_height_max_in_luma_samples. If absent, the value of pic_height_in_luma_samples may be inferred to be equal to pic_height_max_in_luma_samples.

[97] In embodiments, as still shown in FIG. 8, conformance_window_flag (804) equal to 1 may indicate that the conformance cropping window offset parameters follow in a suitable location, such as following in PPS (801). However, a conformance_window_flag of 0 may indicate that there are no conformance cropping window offset parameters. At the same time, conf_win_left_offset (805), conf_win_right_offset (806), conf_win_top_offset (807) and conf_win_bottom_offset (808) may indicate image samples based on PPS, which are output from the decoding process, in the form of a rectangular area specified in image coordinates for output.

[98] In the same embodiment, when the syntax elements conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are not present, the values of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset can be inferred to be equal to the values of sps_conf_win_left_offset, sps_conf_win_left_offset, sps_conf_win_bottom_offset respectively.

[99] In embodiments, the matching crop window may contain brightness samples with horizontal image coordinates from SubWidthC * conf_win_left_offset to pic_width_in_luma_samples - (SubWidthC * conf_win_right_offset + 1) and vertical image coordinates from SubHeightC * conf_win_top_offset to pic_height_in_luma_samples - ( Subwin_C *_conf_off) , inclusive.

[100] SubWidthC * (conf_win_left_offset + conf_win_right_offset) may be less than pic_width_in_luma_samples, and SubHeightC * (conf_win_top_offset + conf_win_bottom_offset) may be less than pic_height_in_luma_samples.

[101] The variables PicOutputWidthL and PicOutputHeightL can be obtained as shown in Equation 1 and Equation 2 below:

PicOutputWidthL = pic_width_in_luma_samples - SubWidthC * (conf_win_right_offset + conf_win_left_offset) (Equation 1)

PicOutputHeightL = pic_height_in_luma_samples - SubHeightC * (conf_win_bottom_offset + conf_win_top_offset) (Equation 2)

[102] In embodiments, the fractional reference picture resampling interpolation process may be processed as follows.

[103] The input to this process may be a luma location (xSb, ySb) indicating the top left count of the current coding sub-block relative to the top left luminance count of the current image, an sbWidth variable indicating the width of the current coding sub-block, an sbHeight variable indicating the height of the current coding sub-block , the motion vector offset mvOffset, the refined motion vector refMvLX, the selected reference image sample array refPicLX, the half sample interpolation filter index hpelIfIdx, the bidirectional optical flow flag bdofFlag, and the cIdx variable indicating the color component index of the current block.

[104] The output of this process may be: an array of predSamplesLX (sbWidth + brdExtSize) x (sbHeight + brdExtSize) prediction sample values.

[105] The prediction block boundary extension size brdExtSize can be obtained as shown in Equation 3 below:

brdExtSize = ( bdofFlag | | ( inter_affine_flag[ xSb ][ ySb ] && sps_affine_prof_enabled_flag ) ) ? 2:0 (Equation 3)

[106] The variable fRefWidth may be set equal to PicOutputWidthL of the reference image in luminance samples. The variable fRefHeight can be set to PicOutputHeightL of the reference image in luminance samples. The motion vector mvLX can be set to (refMvLX - mvOffset).

[107] If cIdx is 0, the following may apply:

– The scaling factors and their fixed-point representations can be determined according to Equation 4 and Equation 5 below:

hori_scale_fp = ( ( fRefWidth << 14 ) + ( PicOutputWidthL >> 1 ) ) / PicOutputWidthL (Equation 4)

vert_scale_fp = ( ( fRefHeight << 14 ) + ( PicOutputHeightL >> 1 ) ) / PicOutputHeightL (Equation 5)

– Let ( xIntL, yIntL ) be the brightness location, given in units of an integer sample, and ( xFracL, yFracL ) be the offset, given in units of 1/16 of a sample. These variables may be used in this clause to specify the locations of fractional samples within the refPicLX arrays of reference samples.

– The top left coordinate of the bounding box to fill with the reference sample (xSbInt _L ,ySbInt _L ) can be set to ( xSb + ( mvLX[ 0 ] >> 4 ), ySb + ( mvLX[ 1 ] >> 4 ) ).

– For each luminance sample location (xL ₌ 0..sbWidth - 1 + brdExtSize, yL ₌ 0..sbHeight - 1 + brdExtSize) within the array of predicted luma samples predSamplesLX, the corresponding value of the predicted luma sample predSamplesLX [ _xL ] [ _yL ] is obtained as follows :

– Let (refxSb _L , refySb _L ) and (refx _L ,refy _L ) be the luma locations pointed to by the motion vector (refMvLX, refMvLX) given in units of 1/16 of a sample. The variables refxSb _L , refx _L , refySb _L , and refy _L can be obtained as shown in Equations 6-9 below:

refxSb _L = ( ( xSb<< 4 ) + refMvLX[ 0 ] ) * hori_scale_fp (Equation 6)

refx _L = ( ( Sign( refxSb ) * ( ( Abs( refxSb ) + 128 ) >> 8 )

+ x _L * ( ( hori_scale_fp + 8 ) >> 4 ) ) + 32 ) >> 6 (Equation 7)

refySb _L = ( ( ySb << 4 ) + refMvLX[ 1 ] ) * vert_scale_fp (Equation 8)

refyL = ( ( ( Sign( refySb ) * ( ( Abs( refySb ) + 128 ) >> 8 ) + yL *

( ( vert_scale_fp + 8 ) >> 4 ) ) + 32 ) >> 6 (Equation 9)

– The variables xInt _L , yInt _L ,xFrac _L and yFrac _L can be derived as shown in Equations 10-13 below:

xInt _L = refx _L >> 4 (Equation 10)

yInt _L = refy _L >> 4 (Equation 11)

xFrac _L = refx _L & 15 (Equation 12)

yFrac _L = refy _L & 15 (Equation 13)

– If bdofFlag is TRUE or (sps_affine_prof_enabled_flag is TRUE and inter_affine_flag [xSb] [ySb] is TRUE) and one or more of the following conditions are true, the predicted luminance sample value predSamplesLX [x _L ] [y _L ] can be obtained by referring to the process of extracting an integer luminance sample as specified in the corresponding clause of the video coding specification, with (xInt _L + ( xFrac _L >> 3) - 1),yInt _L + (yFrac _L >> 3) - 1) and refPicLX as input .

– x _L is equal to 0.

– x _L is equal to sbWidth + 1.

– y _L is equal to 0.

– y _L is equal to sbHeight + 1.

– Otherwise, the predicted luma sample value predSamplesLX [xL] [yL] can be obtained by referring to the 8-position luma sample interpolation filtering process as specified in the corresponding clause of the video coding specification with (xIntL - (brdExtSize > 0 ? 1 : 0 ), yIntL − ( brdExtSize > 0 ? 1: 0)), (xFracL, yFracL), (xSbInt _L , ySbInt _L ), refPicLX, hpelIfIdx, sbWidth, sbHeight and (xSb, ySb) as input.

[108] Otherwise (cIdx is not equal to 0), the following may apply:

1. Let (xIntC, yIntC) be the chromaticity location, given in units of an integer sample, and (xFracC, yFracC) be the offset, given in units of 1/32 of a sample. These variables may be used in this clause to specify the common locations of fractional samples within the refPicLX arrays of reference samples.

2. The upper left coordinate of the bounding box to fill with the reference sample (xSbIntC, ySbIntC) is set to ((xSb / SubWidthC) + (mvLX [0] >> 5), (ySb / SubHeightC) + (mvLX [1]) >> 5 )).

3. For each chroma sample location (xC = 0..sbWidth - 1, yC = 0 .. sbHeight - 1) within the arrays of predicted chroma samples predSamplesLX, the corresponding value of the predicted chroma sample predSamplesLX [xC] [yC] can be obtained as follows:

– Let (refxSb _C , refySb _C ) and (refx _C ,refy _C ) be the chromaticity locations pointed to by the motion vector (mvLX [0], mvLX [1]) given in units of 1/32 of a sample. The variables refxSb _C , refySb _C , refx _C and refy _C can be obtained as shown in Equations 14-17 below:

refxSb _C =( (xSb / SubWidthC << 5) + mvLX[0]) * hori_scale_fp (Equation 14)

refx _C = ((Sign (refxSb _C )* ((Abs (refxSb _C ) + 256) >> 9)

+ xC * ((hori_scale_fp + 8) >> 4)) + 16) >> 5 (Equation 15)

refySb _C =( (ySb / SubHeightC<< 5) + mvLX [1]) * vert_scale_fp (Equation 16)

refy _C = ((Sign (refySb _C )* ((Abs (refySb _C ) + 256) >> 9)

+ yC * ((vert_scale_fp + 8) >> 4)) + 16) >> 5 (Equation 17)

– Variables xInt _C , yInt _C , xFrac _C and yFrac _C can be obtained as shown in Equations 18-21 below:

xInt _C = refx _C >> 5 (Equation 18)

yInt _C = refy _C >> 5 (Equation 19)

xFrac _C = refy _C & 31 (Equation 20)

yFrac _C = refy _C & 31 (Equation 21)

[109] The value of the predicted sample predSamplesLX [xC] [yC] can be obtained by referring to the above process with (xIntC, yIntC), (xFracC, yFracC), (xSbIntC, ySbIntC), sbWidth, sbHeight and refPicLX as input.

[110] FIG. 9 shows an exemplary flowchart of a process 900 for decoding an encoded video bitstream. In some implementations, one or more of the process blocks of FIG. 9 may be executed by decoder 210. In some implementations, one or more of the process blocks of FIG. 9 may be performed by another device or group of devices separate from or including decoder 210, such as encoder 203.

[111] As shown in FIG. 9, process 900 may include obtaining a first flag indicating whether a constant image size is used in an encoded video sequence including the current image (block 910).

[112] As further shown in FIG. 9, process 900 may include determining, based on a first flag, whether a constant image size is being used (block 920).

[113] As further shown in FIG. 9, process 900 may include, based on a first flag indicating that a constant picture size is being used (YES at step 920), decoding the current picture without performing reference picture resampling (step 930).

[114] As further shown in FIG. 9, based on the first flag indicating that a constant image size is not used (NO at block 920), process 900 may proceed to block 940, block 950, block 960, and block 970.

[115] As further shown in FIG. 9, process 900 may include obtaining a second flag indicating whether the match window size is being signaled (block 940).

[116] As further shown in FIG. 9, process 900 may include, based on a second flag indicating that a match window size is being signaled, obtaining a match window size (block 950), determining an oversampling factor between the current picture and a reference picture based on the match window size (block 960), and performing resampling of the reference picture on the current picture using the specified resampling factor (block 970).

[117] In one embodiment, the fit window size may be signaled as at least one offset distance from the edge of the current image.

[118] In one embodiment, the first flag may be signaled in a sequence parameter set (SPS) and the second flag may be signaled in one of the SPS and the picture parameter set (PPS).

[119] In one embodiment, the second flag may be signaled in the SPS and may indicate whether the SPS match window parameters are signaled in the SPS.

[120] In one embodiment, based on a second flag indicating that the SPS matching window parameters are signaled to the SPS, the size of the matching window can be obtained based on the SPS matching window parameters.

[121] In one embodiment, based on a first flag indicating that the image size is not constant, process 900 may include obtaining a third flag indicating whether PPS matching window parameters are signaled in the PPS.

[122] In one embodiment, based on a second flag indicating that the SPS matching window parameters are signaled in the SPS and a third flag indicating that the PPS matching window parameters are not signaled in the PPS, the size of the matching window can be obtained based on the parameters SPS matching windows.

[123] In one embodiment, based on a second flag indicating that the SPS matching window parameters are not signaled in the SPS and a third flag indicating that the PPS matching window parameters are signaled in the PPS, the size of the matching window can be obtained based on the parameters PPS matching windows.

[124] Although FIG. 9 shows exemplary blocks of process 900, in some implementations, process 900 may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those depicted in FIG. Additionally or alternatively, two or more blocks of process 900 may be executed in parallel.

[125] In addition, the proposed methods can be implemented by processing circuits (eg, one or more processors or one or more integrated circuits). In one example, one or more processors execute a program that is stored on a non-transitory computer-readable medium to implement one or more of the proposed methods.

[126] The methods described above may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 10 illustrates a computer system 1000 suitable for implementing certain embodiments of the disclosed subject matter.

[127] Computer software may be coded using any suitable machine code or computer language, which may be the object of assembly, compilation, linking, or similar mechanisms to produce code containing instructions that can be executed directly or through interpretation, execution of microcode, and and etc., computer central processing units (CPUs), graphics processing units (GPUs), etc.

[128] Instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[129] The components shown in FIG. 10 for computer system 1000 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of computer software implementing embodiments of the present disclosure. The component configuration is also not to be interpreted as having any dependency or requirement related to any one or combination of components illustrated in the exemplary embodiment of computer system 1000.

[130] Computer system 1000 may include certain human-machine interface input devices. Such a human-machine interface input device may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement of a control glove), audio input (e.g., voice, handclaps), visual input (for example: gestures), olfactory input (not shown). HMI devices can also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, ambient sound), images (e.g. scanned images, photographic images taken from a camera for still images), video (eg 2D video, 3D video including stereoscopic video).

[131] Human interface input devices may include one or more of (only one of each): keyboard 1001, mouse 1002, trackpad 1003, touch screen 1010 and associated graphics adapter 1050, control glove, joystick 1005, microphone 1006 , scanner 1007, camera 1008.

[132] Computer system 1000 may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback from a touch screen (1010), operating glove (1004), or joystick (1005), but there may also be tactile feedback devices that do not serve as input devices), audio output devices (for example: speakers (1009), headphones (not shown)), image output devices (for example, screens (1010), including CRT screens, LCD screens, plasma screens, OLED- screens, each with or without touchscreen input capability, each with or without tactile feedback capability - some of which may be capable of outputting two-dimensional visual output or more than three-dimensional output by means such as stereographic output; virtual reality goggles (not shown), holographic displays and smoke tanks (not shown) and printers (not shown).

[133] The computer system (1000) may also include human accessible storage devices and associated media such as optical media including CD/DVD ROM/RW (1020) with CD/DVD or similar media (1021), flash drive (1022), removable hard drive or solid state drive (1023), previously developed magnetic media such as tape and floppy disk (not shown), specialized ROM/ASIC/PLD based devices such as dongles (not shown) and so on.

[134] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other time signals.

[135] Computer system 1000 may also include interface(s) to one or more communication networks (1155). Networks can be, for example, wireless, wired, optical. In addition, networks can be local, wide area, metropolitan, automotive, industrial, real-time, latency tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless local area networks (LAN), cellular networks, including global systems for mobile communications (GSM), third generation (3G), fourth generation (4G), fifth generation (5G), Long Term Evolution (LTE) networks, etc., wired television or wireless global digital networks, including cable TV, satellite TV and terrestrial broadcast TV, automotive and industrial, including CAN bus, and so on. Some networks typically require external network interface adapters (1154) that are connected to certain general purpose data ports or peripheral buses (1149) (for example, the Universal Serial Bus (USB) ports of the 1000 computer system); others are typically integrated into the core of computer system 1000 by being attached to a system bus as described below (eg, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). For example, network 1055 may be connected to peripheral bus 1049 using network interface 1054. Using any of these networks, computer system 1000 may communicate with other entities. Such communication can be unidirectional, receive-only (for example, broadcast television), unidirectional, send-only (for example, CAN bus to certain CAN-bus devices), or bidirectional, for example, to other computer systems using a local or wide area digital network. . Each of these networks and network interfaces (1154) may use specific protocols and protocol stacks as described above.

[136] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be attached to the core 1040 of the computer system 1000.

[137] The core (1040) may include one or more central processing units (CPUs) (1041), graphics processing units (GPUs) (1042), custom field programmable gate array processors (FPGAs) (1043), hardware accelerators (1044) for specific tasks, etc. These devices, along with Read Only Memory (ROM) 1045, Random Access Memory (RAM) 1046, high capacity internal storage such as user-inaccessible internal hard drives, Solid State Drives (SSDs) ) and so on 1047 may be connected via the system bus 1048. In some computer systems, the system bus 1048 may be available as one or more physical connectors to allow expansion with additional CPUs, GPUs, etc. Peripherals may be connected either directly to the core system bus 1048 or via a peripheral bus 1049. Peripheral bus architecture includes peripheral component interconnection (PCI), USB, and the like.

[138] CPU 1041, GPU 1042, FPGA 1043, and accelerators 1044 may execute certain instructions, which in combination may constitute the aforementioned computer code. This computer code may be stored in ROM 1045 or RAM 1046. Transient data may also be stored in RAM 1046, while persistent data may be stored in, for example, internal mass storage 1047. Fast storage and retrieval to any of the storage devices may be available through use of cache memory, which may be closely associated with one or more CPU 1041, GPU 1042, mass storage 1047, ROM 1045, RAM 1046, etc.

[139] The computer-readable medium may contain computer code for performing various computer-implemented operations. The media and computer code may be specially designed and constructed for the purposes of this disclosure, or they may be well known and available to those skilled in the art of computer software.

[140] By way of example, and not limitation, a computer system having architecture 1000, and in particular core 1040, may provide functionality as a result of having processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) that executes software embodied on one or more tangible, machine-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as certain core storage devices 1040 that are of a non-transitory nature, such as internal mass storage 1047 or ROM 1045. implementation of the present disclosure may be stored in such devices and executed by the core 1040. The computer-readable medium may include one or more memories or chips in accordance with specific needs. The software may cause the core 1040, and in particular the processors therein (including the CPU, GPU, FPGA, etc.) to execute certain processes or certain parts of the specific processes described herein, including the definition of data structures stored in RAM 1046 and modifying such data structures in accordance with processes defined by the software. In addition, or alternatively, a computer system may provide functionality as a result of logic embedded in hardware circuitry or otherwise embodied in circuitry (e.g., accelerator 1044) that can operate in place of or in conjunction with software to perform certain processes or particular parts of particular processes described in this document. The link to the software may include logic and vice versa where applicable. A reference to a computer-readable medium may include circuitry (eg, an integrated circuit (IC)) storing software to execute, circuitry embodying logic to execute, or both, where applicable. The present disclosure includes any suitable combination of hardware and software.

[141] Although this disclosure describes several exemplary embodiments, there are variations, permutations, and various replacement equivalents that fall within the scope of the disclosure. Thus, it will be appreciated that those skilled in the art will be able to develop numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the disclosure and thus fall within its spirit and scope.

Claims (40)

1. A method for decoding an encoded video bitstream using at least one processor, including: obtaining a first flag indicating whether a constant image size is used in the encoded video sequence including the current image; based on the first flag indicating that a constant image size is used, decoding the current image without performing reference image resampling; based on the first flag indicating that a constant image size is not used, obtaining a second flag indicating whether the match window size is signaled; based on a second flag indicating that the match window size is being signaled: getting the match window size, determining a resampling factor between the current picture and the reference picture based on the match window size, and performing resampling of the reference image on the current image using the resampling factor. 2. The method of claim 1, wherein the size of the match window is signaled as at least one offset distance from the edge of the current image. 3. The method of claim 1, wherein the first flag is signaled in a sequence parameter set (SPS), while the second flag is signaled in one of the SPS and the picture parameter set (PPS). 4. The method of claim 3, wherein the second flag is signaled to the SPS and indicates whether the SPS match window parameters are signaled to the SPS. 5. The method of claim 4, wherein based on the second flag indicating that the SPS match window parameters are signaled to the SPS, the match window size is obtained based on the SPS match window parameters. 6. The method of claim 4, further comprising, based on the first flag indicating that the image size is not constant, obtaining a third flag indicating whether PPS matching window parameters are signaled in the PPS. 7. The method of claim 6, wherein, based on the second flag indicating that the SPS matching window parameters are signaled in the SPS and the third flag indicating that the PPS matching window parameters are not signaled in the PPS, the matching window size is obtained based on the parameters. SPS matching windows. 8. The method of claim 6, wherein based on the second flag indicating that the SPS matching window parameters are not signaled in the SPS and the third flag indicating that the PPS matching window parameters are signaled in the PPS, the size of the matching window is obtained based on the window parameters. PPS compliance. 9. A device for decoding an encoded video bitstream, the device comprising: at least one memory element configured to store program code; And at least one processor configured to read the program code and operate in accordance with the instructions of the program code, the program code including: a first acquisition code configured to cause at least one processor to obtain a first flag indicating whether a constant image size is used in a coded video sequence including the current image; a decoding code configured to cause at least one processor, based on the first flag indicating that a constant image size is used, to decode the current image without performing resampling of the reference image; a second obtaining code, configured to cause at least one processor, based on the first flag indicating that a constant image size is not used, to obtain a second flag indicating whether a matching window size is signaled; And execution code configured to direct at least one processor, based on a second flag indicating that a match window size is signaled, to obtain a match window size, determine a resampling ratio between the current picture and a reference picture based on the match window size, and perform resampling of the reference images on the current image using the resampling factor. 10. The apparatus of claim 9, wherein the size of the match window is signaled as at least one offset distance from the boundary of the current image. 11. The apparatus of claim 9, wherein the first flag is signaled in a sequence parameter set (SPS), while the second flag is signaled in one of the SPS and the picture parameter set (PPS). 12. The apparatus of claim 11, wherein the second flag is signaled to the SPS and indicates whether the SPS match window parameters are signaled to the SPS. 13. The apparatus of claim 12, wherein based on the second flag indicating that the SPS match window parameters are signaled to the SPS, the match window size is obtained based on the SPS match window parameters. 14. The apparatus of claim 12, wherein the program code further includes a third acquisition code configured to cause at least one processor, based on the first flag indicating that the image size is not constant, to obtain a third flag indicating whether the PPS matching window parameters are signaled in the PPS. 15. The apparatus of claim 14, wherein, based on the second flag indicating that the SPS matching window parameters are signaled in the SPS and the third flag indicating that the PPS matching window parameters are not signaled in the PPS, the matching window size is obtained based on the parameters SPS matching windows. 16. The apparatus of claim 14, wherein based on the second flag indicating that the SPS matching window parameters are not signaled in the SPS and the third flag indicating that the PPS matching window parameters are signaled in the PPS, the size of the matching window is obtained based on the window parameters. PPS compliance. 17. A non-transitory computer-readable medium that stores instructions containing: one or more instructions that, when executed by one or more processors of a device for decoding an encoded video bitstream, cause one or more processors to: obtain a first flag indicating whether a constant image size is used in the encoded video sequence including the current image; based on the first flag indicating that a constant image size is used, decode the current image without performing reference image resampling; based on the first flag indicating that a constant image size is not used, obtain a second flag indicating whether the match window size is signaled; based on a second flag indicating that the match window size is being signaled: get match window size, determine a resampling ratio between the current picture and the reference picture based on the size of the match window, and resampling the reference image on the current image using the resampling factor. 18. The carrier of claim 17, wherein the first flag is signaled in a sequence parameter set (SPS), while the second flag is signaled in one of the SPS and the picture parameter set (PPS). 19. The carrier of claim 18, wherein the second flag is signaled to the SPS and indicates whether the SPS match window parameters are signaled to the SPS. 20. The medium of claim 19, wherein the one or more instructions further cause at least one processor, based on a first flag indicating that the image size is not constant, to obtain a third flag indicating whether PPS matching window parameters are signaled in the PPS.

Images