CN115956363A

CN115956363A - Content adaptive online training method and device for post filtering

Info

Publication number: CN115956363A
Application number: CN202280004506.2A
Authority: CN
Inventors: 丁鼎; 蒋薇; 王炜; 刘杉
Original assignee: Tencent America LLC
Current assignee: Tencent America LLC
Priority date: 2021-05-27
Filing date: 2022-05-24
Publication date: 2023-04-11
Also published as: JP2023532397A; EP4128764A1; WO2022251828A1; US20220385896A1; EP4128764A4; US11979565B2; KR20230010259A

Abstract

The embodiment of the application discloses a video decoding method, a video decoding device and a non-transitory computer readable storage medium. The apparatus includes a processing circuit. The processing circuit is used for receiving an image or video, wherein the image or video comprises at least one block; decoding first post-filtering parameters in the image or video corresponding to the at least one block to be reconstructed, wherein the first post-filtering parameters are applied to one or more of the at least one block, the first post-filtering parameters have been updated by a post-filtering module in a post-filtering neural network NN, training the post-filtering NN based on a training dataset; determining the post-filtering NN corresponding to the at least one block in the video decoder based on the first post-filtering parameters; and decoding the at least one block based on the determined post-filtering NN corresponding to the at least one block.

Description

Content adaptive online training method and device for post filtering

Incorporation by reference

This application claims priority from U.S. application Ser. No. 17/749,641, entitled "method and apparatus for content adaptive Online training for post Filtering" filed on 20/5/2022 and U.S. provisional application Ser. No. 63/194,057, entitled "method for content adaptive Online training for post Filtering", filed on 27/5/2021, the entire contents of which are incorporated herein by reference.

Technical Field

The embodiment of the application relates to a video coding and decoding technology.

Background

The background description provided herein is intended to present the background of the application in its entirety. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description, is not admitted to be prior art by inclusion in this application, nor is it explicitly shown or suggested that it would qualify as prior art against the present application.

Video encoding and decoding may be performed by an inter-picture prediction technique with motion compensation. Uncompressed digital video may comprise a series of pictures, each picture having spatial dimensions of, for example, 1920x1080 luma samples and related chroma samples. The series of pictures has a fixed or variable picture rate (also informally referred to as frame rate), e.g. 60 pictures per second or 60Hz. Uncompressed video has very large bit rate requirements. For example, a video (1920 × 1080 luminance sample resolution, 60Hz frame rate) of 1080p60 4. One hour of such video requires more than 600GB of storage space.

One purpose of video encoding and decoding is to reduce redundant information of an input video signal by compression. Video compression can help reduce the bandwidth or storage requirements described above, by two or more orders of magnitude in some cases. Lossless and lossy compression, as well as combinations of both, may be employed. Lossless compression refers to a technique for reconstructing an exact copy of an original signal from a compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough that the reconstructed signal is useful for the intended application. Lossy compression is widely used for video. The amount of distortion tolerated depends on the application. For example, some users consuming streaming applications may tolerate higher distortion than users consuming television applications. The achievable compression ratio reflects: higher allowable/tolerable distortion may result in higher compression ratios.

Video encoders and decoders may utilize several broad classes of techniques, including, for example: motion compensation, transformation, quantization and entropy coding.

Video codec techniques may include known intra-coding techniques. In intra coding, sample values are represented without reference to samples or other data of a previously reconstructed reference picture. In some video codecs, a picture is spatially subdivided into blocks of samples. When all sample blocks are encoded in intra mode, the picture may be an intra picture. Intra pictures and their derivatives (e.g., stand-alone decoder refresh pictures) can be used to reset the decoder state and thus can be used as the first picture in the encoded video bitstream and video session, or as still images. The samples of the intra block may be used for transform, and the transform coefficients may be quantized prior to entropy encoding. Intra prediction may be a technique that minimizes sample values in the pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficient, the fewer bits are needed to represent the block after entropy encoding at a given quantization step size.

As is known from techniques such as MPEG-2 generation coding, conventional intra-frame coding does not use intra-frame prediction. However, some newer video compression techniques include: techniques are attempted to derive data chunks from, for example, surrounding sample data and/or metadata obtained during spatially adjacent encoding/decoding and prior to the decoding order. This technique was later referred to as an "intra-prediction" technique. It is noted that, at least in some cases, intra prediction uses only the reference data of the current picture being reconstructed, and not the reference data of the reference picture.

There may be many different forms of intra prediction. When more than one such technique may be used in a given video coding technique, the techniques used may be coded in intra-prediction mode. In some cases, a mode may have sub-modes and/or parameters, and these modes may be encoded separately or included in a mode codeword. Which codeword is used for a given mode/sub-mode/parameter combination affects the coding efficiency gain through intra prediction, and so does the entropy coding technique used to convert the codeword into a bitstream.

H.264 introduces an intra prediction mode that improves in h.265 and further improves in newer Coding techniques, such as Joint development mode (JEM), universal Video Coding (VVC), and reference Set (BMS). The prediction block may be formed using values of neighboring samples belonging to the existing samples. Depending on the direction, the sample values of neighboring samples are copied to the prediction block. The reference to the direction used may be encoded in the bitstream or may be predicted itself.

Referring to fig. 1A, the bottom right depicts a subset of nine prediction directions known from the 33 possible prediction directions of h.265 (corresponding to 33 angular modes out of the 35 intra prediction modes). The point (101) where the arrows converge represents the sample being predicted. The arrows indicate the direction in which the sample is being predicted. For example, arrow (102) represents the prediction of a sample (101) from at least one sample with an angle of 45 degrees to the horizontal from the upper right. Similarly, arrow (103) represents the prediction of a sample (101) from at least one sample with 22.5 degree angle to the horizontal at the bottom left.

Still referring to fig. 1A, a square block (104) comprising 4 x 4 samples is shown at the top left (indicated by the thick dashed line). The square block (104) consists of 16 samples, each labeled with "S", and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second (from the top) sample in the Y dimension and the first sample in the X dimension (starting from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the X and Y dimensions. Since the block is 4 × 4 samples, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. The reference sample is labeled with "R" and its Y position (e.g., row index) and X position (column index) relative to the block (104). In h.264 and h.265, the prediction sample and the block are adjacent at the time of reconstruction, and therefore, a negative value does not need to be used.

Intra picture prediction can be performed by copying the reference sample values from the neighbouring samples occupied by the signaled prediction direction. For example, it is assumed that the coded video bitstream comprises signaling indicating, for the block, a prediction direction coinciding with the arrow (102), i.e. the samples are predicted from at least one predicted sample having an upper right angle of 45 degrees to the horizontal direction. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, the sample S44 is predicted from the reference sample R08.

In some cases, the values of multiple reference samples may be combined, for example by interpolation, to compute the reference sample, especially when the direction cannot be divided exactly by 45 degrees.

With the development of video coding technology, the number of directions is gradually increasing. In h.264 (2003), nine different directions can be represented. There are 33 increases in H.265 (2013) and JEM/VVC/BMS, and up to 65 orientations can be supported at the time of this application. Experiments have been conducted to identify the most likely directions, and some techniques in entropy coding can be used to identify these most likely directions with a small number of bits, with some unlikely directions being lost to reception. Further, the directions themselves may sometimes be predicted from neighboring directions used by neighboring, decoded blocks.

Fig. 1B shows a schematic diagram (110) depicting 65 intra prediction directions according to JEM to show the number of prediction directions increasing over time.

The mapping of intra prediction direction bits in the encoded video bitstream representing the direction may differ according to different video encoding techniques; and may range, for example, from simple direct mapping of prediction directions to intra-prediction modes, to codewords, to complex adaptation schemes involving the most probable mode, and similar techniques. In all cases, however, there is statistically less likelihood that some directions will appear in the video content than others. Since the goal of video compression is to reduce redundancy, in a well-working video codec, these unlikely directions are represented using more bits than the more likely directions.

Motion compensation may be a lossy compression technique and may involve the following: a specimen data block from a previously reconstructed picture or a portion of a reconstructed picture (reference picture) is spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV) for prediction of the newly reconstructed picture or portion of the picture. In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or three dimensions, where the third dimension represents the reference picture in use (the latter may be indirectly the temporal dimension).

In some video compression techniques, an MV applied to a certain sample data region may be predicted from other MVs, e.g., from those MVs that are related to another sample data region spatially adjacent to the region being reconstructed and that precede the MV in decoding order. This can greatly reduce the amount of data required to encode the MVs, thereby eliminating redundant information and increasing the amount of compression. MV prediction can be done efficiently, for example, when coding an input video signal derived from a camera (called natural video), there is a statistical possibility that regions with areas larger than the applicable region of a single MV will move in similar directions, and thus, in some cases, prediction can be done with similar motion vectors derived from MVs of neighboring regions. This results in the MVs found for a given region being similar or identical to the MVs predicted from the surrounding MVs and, after entropy encoding, can in turn be represented by a smaller number of bits than the number of bits used when directly encoding the MVs. In some cases, MV prediction may be an example of lossless compression of a signal (i.e., MV) derived from an original signal (i.e., a sample stream). In other cases, MV prediction itself may be lossy, for example due to rounding errors that occur when calculating the predicted values from several surrounding MVs.

Various MV prediction mechanisms are described in H.265/HEVC (ITU-T H.265 recommendation, "High Efficiency Video Coding", 2016 (12 months) to Hi-Fi). Among the various MV prediction mechanisms provided by h.265, described herein is a technique referred to hereinafter as "spatial merging.

Referring to fig. 2, a current block (201) includes samples that have been found by an encoder during a motion search process, which samples can be predicted from a previous block of the same size that has resulted in a spatial offset. In addition, the MVs may be derived from metadata associated with one or at least two reference pictures, rather than directly encoding the MVs. For example, the MVs associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (202-206, respectively) are derived (in decoding order) from the metadata of the most recent reference picture using the MVs. In h.265, MV prediction can use the prediction value of the same reference picture that the neighboring block is also using.

Disclosure of Invention

The embodiment of the application provides a method and a device for video coding and decoding. In some examples, an apparatus for video decoding includes a processing circuit. The processing circuit is used for receiving an image or video, wherein the image or video comprises at least one block; decoding first post-filtering parameters in the image or video corresponding to the at least one block to be reconstructed, wherein the first post-filtering parameters are applied to one or more of the at least one block, the first post-filtering parameters have been updated by a post-filtering module in a post-filtering neural network NN, training the post-filtering NN based on a training dataset; determining the post-filtering NN corresponding to the at least one block in the video decoder based on the first post-filtering parameters; and decoding the at least one block based on the determined post-filtering NN corresponding to the at least one block.

In an embodiment, the processing circuit decodes a second post-filtering parameter corresponding to the at least one block in the image or video; determining the post-filtering NN further based on a second post-filtering parameter; wherein the second post-filtering parameters are applied to a second block of the at least one block, the second block being different from the one or more blocks of the at least one block, the second post-filtering parameters having been updated by the post-filtering module in the post-filtering NN.

In an embodiment, the first post-filtering parameter corresponds to a second image to be reconstructed, and the processing circuit is configured to: decoding the second image based on the determined post-filtering NN.

In an embodiment, the first post-filtering parameter is different from the second post-filtering parameter, the first post-filtering parameter being adaptive to the content of a first block of the at least one block, the second post-filtering parameter being adaptive to the content of the second block.

In an embodiment, the first post-filtering parameter is updated based on bias terms or weight coefficients in the post-filtering NN.

In an embodiment, the post-filtering NN is configured with initial parameters; the processing circuitry is to: updating at least one of the initial parameters using the first post-filter parameter.

In an embodiment, the coding information corresponding to the at least one block indicates a difference between the first post-filtering parameter and one of the initial parameters, the processing circuit is configured to: and determining the first post-filtering parameter according to the sum of the difference value and one of the initial parameters.

In an embodiment, the first post-filtering parameters are updated in (i) a single layer of the post-filtering NN, (ii) multiple layers of the post-filtering NN, or (iii) all layers of the post-filtering NN.

In an embodiment, the number of layers in the post-filtering NN depends on a step size, or a number of step sizes corresponding to different blocks of the at least one block.

Embodiments of the present application further provide a non-transitory computer-readable storage medium having stored thereon instructions, which when executed by at least one processor, implement the above-described video decoding method.

Drawings

Other features, properties, and various advantages of the disclosed subject matter will be further apparent from the following detailed description and the accompanying drawings, in which:

FIG. 1A is a diagram of an exemplary subset of intra prediction modes;

FIG. 1B is a diagram of exemplary intra prediction directions;

FIG. 2 shows a schematic diagram of a current block (201) and its surrounding samples according to an embodiment;

FIG. 3 is a schematic diagram of a simplified block diagram of a communication system (300) according to an embodiment;

fig. 4 is a schematic diagram of a simplified block diagram of a communication system (400) according to another embodiment;

FIG. 5 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment;

FIG. 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment;

FIG. 7 shows a block diagram of an encoder according to another embodiment;

FIG. 8 shows a block diagram of a decoder according to another embodiment;

FIG. 9A illustrates an example of block-wise image encoding according to an embodiment of the present application;

fig. 9B illustrates an exemplary NIC framework according to an embodiment of the present application;

fig. 10 shows an exemplary CNN of a primary encoder network according to an embodiment of the application;

fig. 11 shows an exemplary CNN of a primary decoder network according to an embodiment of the application;

fig. 12 shows an exemplary CNN of a super-encoder according to an embodiment of the application;

FIG. 13 shows an exemplary CNN of a super decoder according to an embodiment of the present application;

FIG. 14 illustrates an exemplary CNN of a context model network according to an embodiment of the present application;

fig. 15 shows an exemplary CNN of an entropy parameter network according to an embodiment of the present application;

FIG. 16A illustrates an exemplary video encoder according to an embodiment of the present application;

FIG. 16B shows an exemplary video decoder according to an embodiment of the present application;

FIG. 17 shows an exemplary video encoder according to an embodiment of the present application;

FIG. 18 shows an exemplary video decoder according to an embodiment of the present application;

FIG. 19A illustrates an example of block-wise image filtering according to an embodiment of the present application;

fig. 19B illustrates an exemplary post-filtering module and NIC framework according to embodiments of the application;

FIG. 20A illustrates an exemplary video decoder according to an embodiment of the present application;

FIG. 20B shows an exemplary video decoder according to an embodiment of the present application;

21A-21C illustrate an exemplary deblocking process according to embodiments of the present application;

fig. 22 shows an example of a boundary region comprising samples of more than two blocks according to an embodiment of the present application;

FIG. 23 illustrates an exemplary deblocking process based on multiple deblocking models according to embodiments of the present application;

FIG. 24 illustrates an exemplary enhancement process according to an embodiment of the present application;

FIG. 25 illustrates an exemplary enhancement process according to an embodiment of the present application;

FIG. 26 shows an exemplary image level enhancement process according to an embodiment of the application;

FIG. 27 shows a flow chart of an encoding method according to an embodiment of the present application;

FIG. 28 shows a flow chart of a decoding method according to an embodiment of the application;

FIG. 29 is a schematic diagram of a computer system, according to an embodiment.

Detailed Description

Fig. 3 shows a simplified block diagram of a communication system (300) according to an embodiment of the present application. The communication system (300) comprises a plurality of terminal devices which can communicate with each other via, for example, a network (350). For example, a communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via a network (350). In the example of fig. 3, the first pair of terminal devices (310) and (320) performs a unidirectional transfer of data. For example, the terminal device (310) may encode video data (e.g., a video picture stream captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data may be transmitted in the form of at least one encoded video bitstream. The terminal device (320) may receive encoded video data from the network (350), decode the encoded video data to recover the video picture, and display the video picture according to the recovered video data. One-way data transmission may be common in media service applications and the like.

In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bi-directional transmission of encoded video data, such as may occur during a video conference. For bi-directional transmission of data, in an example, each of the end devices (330) and (340) may encode video data (e.g., a video picture stream captured by the end device) for transmission to the other of the end devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other of the terminal devices (330) and (340), and may decode the encoded video data to recover the video picture, and may display the video picture on an accessible display device according to the recovered video data.

In the example of fig. 3, the terminal devices (310), (320), (330), and (340) may be illustrated as a server, a personal computer, and a smartphone, but the principles of the present application may not be limited thereto. Embodiments of the present application may be applied to laptop computers, tablet computers, media players and/or dedicated video conferencing equipment. Network (350) represents any number of networks that transport encoded video data between terminal devices (310), (320), (330), and (340), including, for example, wired/wired and/or wireless communication networks. The communication network (350) may exchange data in circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and/or the internet. For purposes of this discussion, the architecture and topology of the network (350) may be immaterial to the operation of the present application, unless explained below.

By way of example, fig. 4 illustrates the placement of a video encoder and a video decoder in a streaming environment. The subject matter disclosed herein is equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, and the like.

The streaming system may include an acquisition subsystem (413), which may include a video source (401), such as a digital camera, that creates an uncompressed video picture stream (402). In an embodiment, the video picture stream (402) includes samples taken by a digital camera. The video picture stream (402) is depicted as a thick line to emphasize a high data amount video picture stream compared to the encoded video data (404) (or the encoded video bitstream), the video picture stream (402) being processable by an electronic device (420), the electronic device (420) comprising a video encoder (403) coupled to a video source (401). The video encoder (403) may comprise hardware, software, or a combination of hardware and software to implement or embody aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video codestream (404)) is depicted as a thin line to emphasize the lower data amount of the encoded video data (404) (or encoded video codestream (404)) as compared to the video picture stream (402), which may be stored on a streaming server (405) for future use. At least one streaming client subsystem, such as client subsystem (406) and client subsystem (408) in fig. 3, may access the streaming server (405) to retrieve a copy (407) and a copy (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in the electronic device (330). The video decoder (410) decodes incoming copies (407) of the encoded video data and generates an output video picture stream (411) that may be presented on a display (412), such as a display screen, or another presentation device (not depicted). In some streaming systems, encoded video data (404), video data (407), and video data (409), such as video streams, may be encoded according to certain video encoding/compression standards. Examples of such standards include ITU-T H.265. In an embodiment, the Video Coding standard under development is informally referred to as next generation Video Coding (VVC), and the present application may be used in the context of the VVC standard.

It should be noted that electronic device (420) and electronic device (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

Fig. 5 is a block diagram of a video decoder (510) according to an embodiment of the present disclosure. The video decoder (510) may be disposed in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receive circuit). The video decoder (510) may be used in place of the video decoder (410) in the fig. 4 embodiment.

The receiver (531) may receive at least one encoded video sequence to be decoded by the video decoder (510); in the same or another embodiment, the encoded video sequences are received one at a time, wherein each encoded video sequence is decoded independently of the other encoded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware/software link to a storage device that stores encoded video data. The receiver (531) may receive encoded video data as well as other data, e.g. encoded audio data and/or auxiliary data streams, which may be forwarded to their respective usage entities (not indicated). The receiver (531) may separate the encoded video sequence from other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder/parser (520) (hereinafter "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (515) may be disposed external (not labeled) to the video decoder (510). While in other cases a buffer memory (not labeled) is provided external to the video decoder (510), e.g., to prevent network jitter, and another buffer memory (515) may be configured internal to the video decoder (510), e.g., to handle playout timing. The buffer memory (515) may not be required to be configured or may be made smaller when the receiver (531) receives data from a store/forward device with sufficient bandwidth and controllability or from an isochronous network. Of course, for use over traffic packet networks such as the internet, a buffer memory (515) may also be required, which may be relatively large and may be of adaptive size, and may be implemented at least partially in an operating system or similar element (not labeled) external to the video decoder (510).

The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (510), as well as potential information to control a display device, such as a display screen (512), that is not an integral part of the electronic device (530), but may be coupled to the electronic device (530), as shown in fig. 5. The control Information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse/entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be performed in accordance with video coding techniques or standards and may follow various principles, including variable length coding, huffman coding, arithmetic coding with or without contextual sensitivity, and so forth. A parser (520) may extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the group. A subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and so on. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

The parser (520) may perform entropy decoding/parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

The reconstruction of the symbol (521) may involve a number of different units depending on the type of the encoded video picture or portion of the encoded video picture (e.g., inter and intra pictures, inter and intra blocks), among other factors. Which units are involved and the way they are involved can be controlled by subgroup control information parsed from the coded video sequence by a parser (520). For the sake of brevity, such a subgroup control information flow between parser (520) and the following units is not described.

In addition to the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into several functional units as described below. In a practical embodiment, operating under business constraints, many of these units interact closely with each other and may be integrated with each other. However, for the purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate.

The first unit is a scaler/inverse transform unit (551). The scaler/inverse transform unit (551) receives the quantized transform coefficients as symbols (521) from the parser (520) along with control information including which transform scheme to use, block size, quantization factor, quantization scaling matrix, etc. The sealer/inverse transform unit (551) may output a block comprising sample values, which may be input into an aggregator (555).

In some cases, the output samples of sealer/inverse transform unit (551) may belong to an intra-coded block; namely: predictive information from previously reconstructed pictures is not used, but blocks of predictive information from previously reconstructed portions of the current picture may be used. Such predictive information may be provided by an intra picture prediction unit (552). In some cases, the intra picture prediction unit (552) generates surrounding blocks of the same size and shape as the block being reconstructed using the reconstructed information extracted from the current picture buffer (558). For example, the current picture buffer (558) buffers a partially reconstructed current picture and/or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information generated by the intra prediction unit (552) to the output sample information provided by the scaler/inverse transform unit (551) on a per sample basis.

In other cases, the output samples of sealer/inverse transform unit (551) may belong to inter-coded and potential motion compensated blocks. In this case, the motion compensated prediction unit (553) may access a reference picture memory (557) to fetch samples for prediction. After motion compensation of the extracted samples according to the sign (521), these samples may be added by an aggregator (555) to the output of the scaler/inverse transform unit (551), in this case referred to as residual samples or residual signals, thereby generating output sample information. The fetching of prediction samples by the motion compensated prediction unit (553) from addresses within the reference picture memory (557) may be controlled by motion vectors, and the motion vectors are used by the motion compensated prediction unit (553) in the form of the symbols (521), the symbols (521) comprising, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values fetched from a reference picture memory (557), motion vector prediction mechanisms, etc., when using sub-sample exact motion vectors.

The output samples of the aggregator (555) may be employed by various loop filtering techniques in the loop filter unit (556). The video compression techniques may include in-loop filter techniques that are controlled by parameters included in an encoded video sequence (also referred to as an encoded video bitstream), and which are available to the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of previous (in decoding order) portions of the encoded picture or encoded video sequence, as well as to sample values previously reconstructed and loop filtered.

The output of the loop filter unit (556) may be a sample stream that may be output to a display device (512) and stored in a reference picture memory (557) for subsequent inter picture prediction.

Once fully reconstructed, some of the coded pictures may be used as reference pictures for future prediction. For example, once the encoded picture corresponding to the current picture is fully reconstructed and the encoded picture is identified (by, e.g., parser (520)) as a reference picture, current picture buffer (558) may become part of reference picture memory (557) and a new current picture buffer may be reallocated before starting reconstruction of a subsequent encoded picture.

The video decoder (510) may perform decoding operations according to predetermined video compression techniques, such as in the ITU-T h.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence conforms to the syntax of the video compression technique or standard and the configuration files recorded in the video compression technique or standard. In particular, the configuration file may select certain tools from all tools available in the video compression technology or standard as the only tools available under the configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the limits defined by the level of video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in units of, e.g., mega samples per second), the maximum reference picture size, and so on. In some cases, the limits set by the hierarchy may be further defined by a Hypothetical Reference Decoder (HRD) specification and metadata signaled HRD buffer management in the encoded video sequence.

In an embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of an encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and/or more accurately reconstruct the original video data. The additional data may be in the form of, for example, a temporal, spatial, or signal-to-noise ratio (SNR) enhancement layer, a redundant slice, a redundant picture, a forward error correction code, and so forth.

Fig. 6 is a block diagram of a video encoder (603) according to an embodiment of the disclosure. The video encoder (603) is disposed in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) may be used in place of the video encoder (403) in the embodiment of fig. 4.

Video encoder (603) may receive video samples from a video source (601) (not part of electronics (620) in the fig. 6 embodiment) that may capture video images to be encoded by video encoder (603). In another embodiment, the video source (601) is part of an electronic device (620).

The video source (601) may provide a source video sequence in the form of a stream of digital video samples to be encoded by the video encoder (603), which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit \8230;), any color space (e.g., bt.601y CrCB, RGB \8230;) and any suitable sampling structure (e.g., Y CrCB 4. In the media service system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. Video data may be provided as a plurality of individual pictures that are given motion when viewed in sequence. The picture itself may be constructed as an array of spatial pixels, where each pixel may comprise at least one sample, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples can be readily understood by those skilled in the art. The following description focuses on the sample.

According to an embodiment, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real-time or under any other temporal constraint required by an application. It is a function of the controller (650) to implement the appropriate encoding speed. In some embodiments, the controller (650) controls and is functionally coupled to other functional units as described below. For simplicity, the couplings are not labeled in the figures. The parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be used to have other suitable functions relating to the video encoder (603) optimized for a certain system design.

In some embodiments, the video encoder (603) operates in an encoding loop. As a brief description, in an embodiment, an encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, e.g., a stream of symbols, based on input pictures and reference pictures to be encoded) and a (local) decoder (633) embedded in a video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a similar manner as a (remote) decoder creates sample data (since in the video compression techniques considered herein any compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit accurate results independent of decoder location (local or remote), the content in the reference picture store (634) also corresponds bit accurately between the local encoder and the remote encoder. In other words, the reference picture samples that the prediction portion of the encoder "sees" are identical to the sample values that the decoder would "see" when using prediction during decoding. This reference picture synchronization philosophy (and the drift that occurs if synchronization cannot be maintained due to, for example, channel errors) is also used in some related techniques.

The operation of the "local" decoder (633) may be the same as a "remote" decoder, such as the video decoder (510) that has been described in detail above in connection with fig. 5. However, referring briefly to fig. 5 additionally, when symbols are available and the entropy encoder (645) and parser (520) are able to losslessly encode/decode the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), may not be fully implemented in the local decoder (633).

At this point it can be observed that any decoder technique other than the parsing/entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the present application focuses on decoder operation. The description of the encoder techniques may be simplified because the encoder techniques are reciprocal to the fully described decoder techniques. A more detailed description is only needed in certain areas and is provided below.

During operation, in some embodiments, the source encoder (630) may perform motion compensated predictive coding. The motion compensated predictive coding predictively codes an input picture with reference to at least one previously coded picture from the video sequence that is designated as a "reference picture". In this way, the encoding engine (632) encodes the difference between a pixel block of an input picture and a pixel block of a reference picture, which may be selected as a prediction reference for the input picture.

The local video decoder (633) may decode encoded video data, which may be designated as reference pictures, based on the symbols created by the source encoder (630). The operation of the encoding engine (632) may be a lossy process. When the encoded video data can be decoded at a video decoder (not shown in fig. 6), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates a decoding process that may be performed on reference pictures by the video decoder, and may cause reconstructed reference pictures to be stored in the reference picture cache (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has common content (no transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

Predictor (635) may perform a prediction search for coding engine (632). That is, for a new picture to be encoded, predictor (635) may search reference picture memory (634) for sample data (as candidate reference pixel blocks) or some metadata, such as reference picture motion vectors, block shapes, etc., that may be referenced as appropriate predictions for the new picture. The predictor (635) may perform operations on a block-by-block basis to find a suitable prediction reference. In some cases, it may be determined from search results obtained by predictor (635) that the input picture may have prediction references taken from multiple reference pictures stored in reference picture memory (634).

The controller (650) may manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding the video data.

The outputs of all of the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) losslessly compresses the symbols generated by the various functional units according to techniques such as huffman coding, variable length coding, arithmetic coding, etc., to convert the symbols into an encoded video sequence.

The transmitter (640) may buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which may be a hardware/software link to a storage device that will store the encoded video data. The transmitter (640) may combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and/or an auxiliary data stream (sources not shown).

The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain encoded picture type to each encoded picture, but this may affect the encoding techniques applicable to the respective picture. For example, pictures may be generally assigned to any of the following picture types:

intra pictures (I pictures), which may be pictures that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs tolerate different types of intra pictures, including, for example, independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of variants of picture I and their corresponding applications and features.

Predictive pictures (P pictures), which may be pictures that may be encoded and decoded using intra prediction or inter prediction that uses at most one motion vector and reference index to predict sample values of each block.

Bi-predictive pictures (B-pictures), which may be pictures that can be encoded and decoded using intra-prediction or inter-prediction that uses at most two motion vectors and reference indices to predict sample values of each block. Similarly, multiple predictive pictures may use more than two reference pictures and associated metadata for reconstructing a single block.

A source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4 x 4, 8 x 8, 4 x 8, or 16 x 16 samples) and encoded block-wise. These blocks may be predictively coded with reference to other (coded) blocks, which are determined according to the coding allocation applied to their respective pictures. For example, a block of an I picture may be non-predictive encoded, or the block may be predictive encoded (spatial prediction or intra prediction) with reference to already encoded blocks of the same picture. The pixel block of the P picture can be prediction-coded by spatial prediction or by temporal prediction with reference to one previously coded reference picture. A block of a B picture may be prediction coded by spatial prediction or by temporal prediction with reference to one or two previously coded reference pictures.

The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as the ITU-T h.265 recommendation. In operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the encoded video data may conform to syntax specified by the video coding technique or standard used.

In an embodiment, the transmitter (640) may transmit the additional data while transmitting the encoded video. The source encoder (630) may take such data as part of an encoded video sequence. The additional data may include temporal/spatial/SNR enhancement layers, redundant pictures and slices, among other forms of redundant data, SEI messages, VUI parameter set segments, and the like.

The captured video may be provided as a plurality of source pictures (video pictures) in a time sequence. Intra-picture prediction, often abbreviated as intra-prediction, exploits spatial correlation in a given picture, while inter-picture prediction exploits (temporal or other) correlation between pictures. In an embodiment, the particular picture being encoded/decoded, referred to as the current picture, is partitioned into blocks. When a block in a current picture is similar to a reference block in a reference picture that has been previously encoded in video and is still buffered, the block in the current picture may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

In some embodiments, bi-directional prediction techniques may be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, e.g., a first reference picture and a second reference picture that are both prior to the current picture in video in decoding order (but may be past and future, respectively, in display order). A block in a current picture may be encoded by a first motion vector pointing to a first reference block in a first reference picture and a second motion vector pointing to a second reference block in a second reference picture. In particular, the block may be predicted by a combination of a first reference block and a second reference block.

Furthermore, merge mode techniques may be used in inter picture prediction to improve coding efficiency.

According to some embodiments disclosed herein, prediction such as inter-picture prediction and intra-picture prediction is performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into Coding Tree Units (CTUs) for compression, the CTUs in the pictures having the same size, e.g., 64 × 64 pixels, 32 × 32 pixels, or 16 × 16 pixels. In general, a CTU includes three Coding Tree Blocks (CTBs), which are one luminance CTB and two chrominance CTBs. Further, each CTU may be further split into at least one Coding Unit (CU) in a quadtree. For example, a 64 × 64-pixel CTU may be split into one 64 × 64-pixel CU, or 4 32 × 32-pixel CUs, or 16 × 16-pixel CUs. In an embodiment, each CU is analyzed to determine a prediction type for the CU, e.g., an inter prediction type or an intra prediction type. Furthermore, depending on temporal and/or spatial predictability, a CU is split into at least one Prediction Unit (PU). In general, each PU includes a luma Prediction Block (PB) and two chroma blocks PB. In an embodiment, a prediction operation in encoding (encoding/decoding) is performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values (e.g., luma values), such as 8 × 8 pixels, 16 × 16 pixels, 8 × 16 pixels, 16 × 8 pixels, and so on.

Fig. 7 is a diagram of a video encoder (703) according to another embodiment of the present disclosure. A video encoder (703) is used to receive a processing block of sample values within a current video picture in a sequence of video pictures, such as a prediction block, and encode the processing block into an encoded picture that is part of an encoded video sequence. In this embodiment, a video encoder (703) is used in place of the video encoder (403) in the embodiment of fig. 4.

In an HEVC embodiment, a video encoder (703) receives a matrix of sample values for a processing block, e.g., a prediction block of 8 × 8 samples, etc. A video encoder (703) determines whether to encode the processing block using intra mode, inter mode, or bi-directional prediction mode using, for example, rate-distortion (RD) optimization. When encoding a processing block in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and when the processing block is encoded in inter mode or bi-prediction mode, the video encoder (703) may encode the processing block into the encoded picture using inter-prediction or bi-prediction techniques, respectively. In some video coding techniques, the merge mode may be an inter-picture prediction sub-mode, in which a motion vector is derived from at least one motion vector predictor without resorting to coded motion vector components outside of the predictor. In some other video coding techniques, there may be motion vector components that are applicable to the subject block. In an embodiment, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining a processing block mode.

In the embodiment of fig. 7, the video encoder (703) includes an inter encoder (730), an intra encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in fig. 7.

The inter encoder (730) is configured to receive samples of a current block (e.g., a processed block), compare the block to at least one reference block in a reference picture (e.g., blocks in previous and subsequent pictures), generate inter prediction information (e.g., redundant information descriptions according to inter coding techniques, motion vectors, merge mode information), and calculate an inter prediction result (e.g., a predicted block) using any suitable technique based on the inter prediction information. In some embodiments, the reference picture is a decoded reference picture that is decoded based on encoded video information.

The intra encoder (722) is used to receive samples of a current block (e.g., a processing block), in some cases compare the block to a block already encoded in the same picture, generate quantized coefficients after transformation, and in some cases also generate intra prediction information (e.g., intra prediction direction information according to at least one intra coding technique). In an embodiment, the intra encoder (722) also calculates an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same picture.

The general purpose controller (721) is used to determine general purpose control data and control other components of the video encoder (703) based on the general purpose control data. In an embodiment, a general purpose controller (721) determines a mode of a block and provides a control signal to a switch (726) based on the mode. For example, when the mode is intra mode, the general purpose controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select and add intra prediction information in the code stream; and when the mode is an inter mode, the general purpose controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select and add inter prediction information in the code stream.

A residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or the inter encoder (730). A residual encoder (724) is operative to encode the residual data to generate transform coefficients based on the residual data. In an embodiment, a residual encoder (724) is used to convert residual data from the time domain to the frequency domain and generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). A residual decoder (728) is used to perform the inverse transform and generate decoded residual data. The decoded residual data may be suitably used by an intra encoder (722) and an inter encoder (730). For example, inter encoder (730) may generate a decoded block based on decoded residual data and inter prediction information, and intra encoder (722) may generate a decoded block based on decoded residual data and intra prediction information. The decoded blocks are processed appropriately to generate a decoded picture, and in some embodiments, the decoded picture may be buffered in a memory circuit (not shown) and used as a reference picture.

The entropy coder (725) is for formatting the codestream to produce coded blocks. The entropy encoder (725) generates various information according to a suitable standard such as the HEVC standard. In an embodiment, the entropy encoder (725) is used to obtain general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other suitable information in the code stream. It should be noted that, according to the disclosed subject matter, there is no residual information when a block is encoded in the merge sub-mode of the inter mode or bi-prediction mode.

Fig. 8 is a diagram of a video decoder (810) according to another embodiment of the present disclosure. A video decoder (810) is configured to receive encoded images as part of an encoded video sequence and decode the encoded images to generate reconstructed pictures. In an embodiment, a video decoder (810) is used in place of the video decoder (410) in the fig. 3 embodiment.

In the fig. 8 embodiment, video decoder (810) includes an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together as shown in fig. 8.

An entropy decoder (871) is operable to reconstruct from an encoded picture certain symbols representing syntax elements constituting the encoded picture. Such symbols may include, for example, a mode used to encode the block (e.g., intra mode, inter mode, bi-prediction mode, a merge sub-mode of the latter two, or another sub-mode), prediction information (e.g., intra prediction information or inter prediction information) that may identify certain samples or metadata for prediction by an intra decoder (872) or an inter decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, and so forth. In an embodiment, when the prediction mode is inter or bi-directional prediction mode, inter prediction information is provided to an inter decoder (880); and providing the intra prediction information to an intra decoder (872) when the prediction type is an intra prediction type. The residual information may be inverse quantized and provided to a residual decoder (873).

An inter-frame decoder (880) is configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

An intra-frame decoder (872) is used to receive intra-frame prediction information and generate a prediction based on the intra-frame prediction information.

A residual decoder (873) is used to perform inverse quantization to extract dequantized transform coefficients and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also need some control information (to obtain the quantizer parameters QP) and that information may be provided by the entropy decoder (871) (data path not labeled as this is only low-level control information).

The reconstruction module (874) is configured to combine the residuals output by the residual decoder (873) and the prediction results (which may be output by the inter prediction module or the intra prediction module) in the spatial domain to form a reconstructed block, which may be part of a reconstructed picture, which in turn may be part of a reconstructed video. It should be noted that other suitable operations, such as deblocking operations, may be performed to improve visual quality.

It should be noted that video encoder (403), video encoder (603), and video encoder (703), and video decoder (410), video decoder (510), and video decoder (810) may be implemented using any suitable techniques. In an embodiment, the video encoder (403), the video encoder (603), and the video encoder (703) and the video decoder (410), the video decoder (510), and the video decoder (810) may be implemented using at least one integrated circuit. In another embodiment, the video encoder (403), the video encoder (603), and the video encoder (703), and the video decoder (410), the video decoder (510), and the video decoder (810) may be implemented using at least one processor executing software instructions.

Video encoding techniques related to neural network image compression techniques and/or neural network video compression techniques, such as Artificial Intelligence (AI) based neural Network Image Compression (NIC), are described. Aspects of the present application include content adaptive online training in a NIC, such as a content adaptive online training NIC method with post-filtering, for a neural network based end-to-end (E2E) optimized image coding framework. The Neural Networks (NN) may include Artificial Neural Networks (ANN), such as Deep Neural Networks (DNN) or Convolutional Neural Networks (CNN).

In an embodiment, the related hybrid video codec is difficult to optimize as a whole. For example, improvements of a single module (e.g., encoder) in a hybrid video codec may not result in codec gain in overall performance. In the NN-based video coding framework, different modules may be jointly optimized from input to output by performing a learning process or training process (e.g., a machine learning process) to improve the final goal (e.g., rate-distortion performance, such as the rate-distortion loss L described in this application), thus yielding an E2E optimized NIC.

An exemplary NIC framework or system may be described as follows. The NIC framework may use the input block x as an input to a neural network encoder (e.g., an encoder based on a neural network such as DNN) to compute a compressed representation (e.g., a compact representation)

The compressed representation may be compact, for example for storage and transmission purposes. Neural network decoders (e.g., decoders based on neural networks such as DNN) can use compression representations &>

Reconstructing an output block (also called reconstruction block) as input

In various embodiments, an input block x and a reconstruction block +>

In the spatial domain, compression means->

In a different domain than the spatial domain. In some examples, representing ÷ compression>

Quantization and entropy coding are performed.

In some examples, the NIC framework may use a Variational Auto Encoder (VAE) structure. In the VAE structure, the neural network encoder may directly use the entire input block x as an input to the neural network encoder. The entire input block x may be computed by a set of neural network layers working as black boxesCompressed representation

Compressed representation->

Is the output of the neural network encoder. The neural network decoder may represent the entire compression->

As an input. Compressed representation->

Reconstruction blocks can be calculated by another set of neural network layers working as another black box>

The rate-distortion (R-D) loss L @, may be optimized>

Implementing a reconstructed block @, using a trade-off over parameter lambda>

Is lost->

And compact representation->

Is the trade-off between R and R.

A neural network (e.g., an ANN) can learn to perform tasks from examples without task-specific programming. The ANN may be configured with connected nodes or artificial neurons. A connection between nodes may transmit a signal from a first node to a second node (e.g., a receiving node), and the signal may be modified by a weight, which may be indicated by a weight coefficient for the connection. The receiving node may process at least one signal received from at least one node (i.e., at least one input signal of the receiving node) and then generate an output signal by applying a function to the input signal. The function may be a linear function. In an example, the output signal is a weighted sum of the at least one input signal. In an example, the output signal is further modified by an offset, which may be indicated by an offset term, so that the output signal is a sum of the offset and a weighted sum of the at least one input signal. The function may comprise a non-linear operation, e.g. a weighted sum or sum of a weighted sum of the bias and the at least one input signal. The output signal may be transmitted to at least one node (at least one downstream node) connected to the receiving node. The ANN may be represented or configured by parameters (e.g., weights for connections and/or offsets). The weights and/or biases may be obtained by training an ANN with examples, wherein the weights and/or biases may be iteratively adjusted. The trained ANN, configured with the determined weights and/or the determined biases, may be used to perform a task.

The nodes in the ANN may be organized in any suitable architecture. In various embodiments, nodes in the ANN are organized into layers, including an input layer for receiving at least one input signal of the ANN, and an output layer for outputting at least one output signal from the ANN. In an embodiment, the ANN further comprises at least one layer, such as at least one hidden layer between the input layer and the output layer. Different layers may perform different types of transformations on the respective inputs of the different layers. Signals may propagate from the input layer to the output layer.

An ANN having multiple layers between an input layer and an output layer may be referred to as a DNN. In an embodiment, the DNN is a feed-forward network where data flows from the input layer to the output layer without a looping back (loopback). In an example, the DNN is a fully connected network where each node in one layer is connected to all nodes in the next layer. In an embodiment, the DNN is a Recurrent Neural Network (RNN), where data may flow in any direction. In an embodiment, the DNN is CNN.

The CNN may include an input layer, an output layer, and at least one hidden layer between the input layer and the output layer. The at least one hidden layer may comprise at least one convolutional layer (e.g., used in an encoder) that performs a convolution, such as a two-dimensional (2D) convolution. In an embodiment, the 2D convolution performed in the convolutional layer is between the convolution kernel (also referred to as a filter or channel, such as a 5 × 5 matrix) and the input signal to the convolutional layer (e.g., a 2D matrix such as a 2D block, a 256 × 256 matrix). In various examples, the size of the convolution kernel (e.g., 5 × 5) is smaller than the size of the input signal (e.g., 256 × 256). Thus, the portion (e.g., the 5 × 5 area) of the input signal (e.g., the 256 × 256 matrix) covered by the convolution kernel is smaller than the area (e.g., the 256 × 256 area) of the input signal, and thus may be referred to as a receptive field in a corresponding node in the next layer.

During convolution, the dot products of the convolution kernels and the corresponding respective receptive fields in the input signal are computed. Thus, each element of the convolution kernel is a weight that is applied to the corresponding sample in the perceptual domain, and thus the convolution kernel includes a weight. For example, a convolution kernel represented by a 5 × 5 matrix has 25 weights. In some examples, a bias is applied to the output signal of the convolutional layer, and the output signal is based on a sum of the dot product and the bias.

The convolution kernel can be moved along the input signal (e.g., 2D matrix), the size of the movement being referred to as stride, such that the convolution operation generates a feature map (feature map) or activation map (e.g., another 2D matrix) as input to the next layer in the CNN. For example, the input signal is a 2D block, with 256 × 256 samples, and a stride of 2 samples (e.g., a stride of 2). For stride 2, the convolution kernel is shifted by 2 samples along the X-direction (e.g., horizontal direction) and/or the Y-direction (e.g., vertical direction).

Multiple convolution kernels may be applied to an input signal in the same convolution layer to generate multiple feature maps, respectively, where each feature map may represent a particular feature of the input signal. Typically, there are convolutional layers of N channels (i.e., N convolutional kernels), each convolutional kernel has M × M samples, and the step S may be specified as Conv: m cN sS. For example, a convolutional layer with 192 channels, each convolutional core has 5 × 5 samples, and steps of 2 are specified as Conv: 5X 5c192 s2. The at least one hidden layer may comprise at least one deconvolution layer (e.g., for use in a decoder) that performs deconvolution, such as 2D deconvolution. Deconvolution is the inverse process of convolution. Deconvolution layers with 192 channels, each deconvolution kernel having 5 × 5 samples, and step 2 is specified as DeConv: 5X 5c192 s2.

In various embodiments, CNN has the following benefits. The number of learnable parameters (i.e., parameters to be trained) in the CNN may be significantly less than the number of learnable parameters in the DNN, such as feed-forward DNN. In CNNs, a relatively large number of nodes may share the same filter (e.g., the same weights) and the same bias (if a bias is used), and thus memory footprint may be reduced because a single bias and a single weight vector may be used across all of the receptive fields that share the same filter. For example, for an input signal having 100 × 100 samples, a convolutional layer having a convolution kernel of 5 × 5 samples has 25 learnable parameters (e.g., weights). If an offset is used, 26 learnable parameters (e.g., 25 weights and one offset) are used for one channel. If the convolutional layer has N channels, the total learnable parameter is 26N. On the other hand, for a fully connected layer in DNN, a weight of 100 × 100 (i.e., 10000) is used for each node in the next layer. If the next layer has L nodes, the total learnable parameter is 10000 xL.

The CNN may further include at least one other layer, such as at least one pooling layer, at least one fully-connected layer that may connect each node in one layer to each node in another layer, and/or at least one normalization layer, among others. The layers in the CNN may be arranged in any suitable order and in any suitable architecture (e.g., feed-forward architecture, recursive architecture). In an example, the convolutional layer is followed by at least one other layer, such as at least one pooling layer, at least one fully-connected layer, and/or at least one normalization layer, and so on.

The pooling layer may combine outputs from multiple nodes at one layer into a single node in the next layer, thereby reducing the size of the data. The following describes the pooling operation of the pooling layer with the feature map as input. The description may be suitably adapted to other input signals. The feature map may be divided into sub-regions (e.g., rectangular sub-regions), and the features in the respective sub-regions may be individually sub-sampled (or pooled) into a single value, for example, by taking the average in the average pooling or the maximum in the maximum pooling.

The pooling layer may perform pooling, such as local pooling, global pooling, maximum pooling, and/or average pooling, among others. Pooling is a form of non-linear down-sampling. Local pooling combines a small number of nodes (e.g., local clusters of nodes, such as 2 x 2 nodes) in a feature graph. Global pooling may combine all nodes of, for example, a feature map.

The pooling layer may reduce the size of the representation, thus reducing the number of parameters, memory footprint, and computational load in the CNN. In an example, pooled layers are inserted between consecutive convolutional layers in the CNN. In an example, the pooling layer is followed by an activation function, such as a rectifying linear unit (ReLU) layer. In an example, pooling layers are omitted between successive convolutional layers in the CNN.

The normalization layer may be ReLU, leaky (leak) ReLU, generalized split normalization (GDN), or Inverse GDN (IGDN), etc. By setting negative values to zero, the ReLU may apply a non-saturating activation function to remove negative values from an input signal (such as a signature). The leakage ReLU may have a small slope (e.g., 0.01) for negative values, rather than a flat slope (e.g., 0). Accordingly, if the value x is greater than 0, the output from the leakage ReLU is x. Otherwise, the output from the leakage ReLU is the value x multiplied by a small slope (e.g., 0.01). In an example, the slope is determined prior to training, so the slope is not learned during training.

In NN-based image compression methods, such as DNN-based or CNN-based image compression methods, the entire image is not encoded directly, but rather a block-based or block-by-block encoding scheme is effectively used for compressing images in DNN-based video coding standards, such as FVC. The entire image may be partitioned into blocks of the same (or different) size, and these blocks may be compressed separately. In embodiments, the image may be segmented into blocks of equal or unequal size. The segmented blocks may be compressed instead of the image. Fig. 9A illustrates an example of block-wise image encoding according to an embodiment of the present application. The image (980) may be partitioned into blocks, such as blocks (981) - (996). The blocks (981) - (996) may be compressed, for example, according to a scan order. In the example shown in fig. 9A, blocks (981) - (989) have been compressed, and blocks (990) - (996) will be compressed.

An image may be considered a block. In an embodiment, the image is compressed without being divided into blocks. The entire image may be the input to the E2E NIC framework.

In the following description, a signal is used to refer to an image or a block for the sake of brevity. Thus, the input signal may refer to an input image or input block, the encoded signal may refer to an encoded image or encoded block, and the reconstructed signal may refer to a reconstructed image or reconstructed block.

Fig. 9B illustrates an exemplary NIC framework (900) (e.g., a NIC system) according to an embodiment of the present application. The NIC framework (900) may be based on a neural network, such as DNN and/or CNN. The NIC framework (900) may be used to compress (e.g., encode) signals (e.g., blocks and/or images) and decompress (e.g., decode or reconstruct) compressed signals (e.g., blocks and/or images) (e.g., encoded blocks or images). The NIC framework (900) may include two sub-neural networks, i.e., a first sub-NN (951) and a second sub-NN (952), implemented using neural networks.

The first sub-NN (951) may be similar to an auto-encoder and may be trained to generate a compressed signal (e.g., a compressed block or a compressed image) of an input signal (e.g., an input block or an input image) x

And for the compression signal->

Decompression is performed to obtain a reconstructed signal (e.g., a reconstructed block or a reconstructed image) <>

. The first sub-NN (951) may include a plurality of components (or modules) such as a main encoder neural network (or main encoder network) (911), a quantizer (912), an entropy encoder (913), an entropy decoder (914), and a main decoder neural network (or main encoder network) (915). Referring to fig. 9B, the primary encoder network (911) may generate a potential value or potential representation y from an input signal x (e.g., a signal to be compressed or encoded). In an example, the primary encoder network (911) is implemented using CNN. The relationship between the potential representation y and the input signal x can be described using equation 2.

y＝f ₁ (x；θ ₁ ) Equation 2 wherein the parameter θ ₁ Representing parameters such as weights and offsets used in the convolution kernel in the primary encoder network (911) (if offsets are used in the primary encoder network (911)).

The potential representation y may be quantized using a quantizer (912) to generate quantized potential values

. Quantified potential value->

May be compressed, for example, by an entropy encoder (913) using lossless compression to generate a compressed representation of the input signal x ≦ ≦ for the input signal x>

Is (e.g., an encoded signal) based on the compression signal (e.g., the encoded signal)>

(931). The entropy encoder (913) may use entropy encoding techniques such as huffman coding or arithmetic coding, or the like. In an example, the entropy encoder (913) uses arithmetic coding and is an arithmetic encoder. In an example, the encoded signal (931) is transmitted in an encoded code stream.

The encoded signal (931) may be decompressed (e.g., entropy decoded) by an entropy decoder (914) to generate an output. Entropy decoder(914) An entropy decoding technique corresponding to the entropy encoding technique used in the entropy encoder (913) may be used, such as huffman coding or arithmetic coding. In an example, the entropy decoder (914) uses arithmetic decoding and is an arithmetic decoder. In an example, lossless compression is used in the entropy encoder (913), lossless decompression is used in the entropy decoder (914), and noise such as due to transmission of the encoded signal (931) is negligible, the output from the entropy decoder (914) is a quantized latent value

。

The master decoder network (915) may apply quantization to the potential values

Decoding to generate a reconstruction signal->

. In an example, the primary decoder network (915) is implemented using CNN. Reconstructed signal->

(i.e., the output of the master decoder network (915)) and the quantized potential value->

The relationship between (i.e., the inputs of the primary decoder network (915)) may be described using equation 3.

Wherein the parameter theta ₂ Representing parameters such as weights and offsets used in the convolution kernel in the primary decoder network (915) (if offsets are used in the primary decoder network (915)). Thus, the first sub-NN (951) may compress (e.g., encode) the input signal x to obtain an encoded signal (931), and decompress (e.g., decode) the encoded signal (931) to obtain a reconstructed signal

. The reconstructed signal @, due to the quantization loss introduced by the quantizer (912)>

May be different from the input signal x.

The second sub-NN (952) may be directed to quantized potential values

Learning entropy models (e.g., prior probability models) for entropy coding. Thus, the entropy model may be a conditional entropy model, such as a Gaussian Mixture Model (GMM), a Gaussian Scale Model (GSM) depending on the input block x. The second sub-NN (952) may include a context model NN (916), an entropy parameter NN (917), a super encoder (921), a quantizer (922), an entropy encoder (923), an entropy decoder (924), and a super decoder (925). The entropy model used in the context model NN (916) may be a potential value (e.g., a quantized potential value ≦ ≦>

) The autoregressive model of (1). In an example, the super encoder (921), the quantizer (922), the entropy encoder (923), the entropy decoder (924), and the super decoder (925) form a super neural network (e.g., a super-a-first NN). The super neural network may represent information, i.e., information useful for correcting context-based predictions. Data from the context model NN (916) and the super neural network may be combined by the entropy parameters NN (917). The entropy parameters NN (917) may generate parameters, such as mean and scale parameters, for an entropy model, such as a conditional gaussian entropy model (e.g., GMM).

Referring to fig. 9B, at the encoder side, the quantized potential values from quantizer (912)

Is fed into the context model NN (916). At the decoder side, the quantized potential value ≦ from the entropy decoder (914)>

Is fed into the context model NN (916). The context model NN (916) may be implemented using a neural network such as a CNN. Context model NN (916) can be based on context @>

Generating an output o _cm,i The context is the quantized potential value { (916) } available to the context model NN (916)>

. Context->

May comprise previously quantized potential values at the encoder side or previously entropy decoded quantized potential values at the decoder side. Output o of context model NN (916) _cm,i And an input (e.g., based on a predetermined criterion)>

) The relationship between them can be described using equation 4.

Wherein the parameter theta ₃ Representing parameters such as weights and biases used in the convolution kernels in the context model NN (916) (if biases are used in the context model NN (916)).

Output o from context model NN (916) _cm,i And an output o from the super decoder (925) _hc Is fed to an entropy parameter NN (917) to generate an output o _ep . The entropy parameter NN (917) may be implemented using a neural network such as CNN. Output o of entropy parameter NN (917) _ep And input (e.g., o) _cm,i And o _hc ) The relationship between them can be described using equation 5.

o _ep ＝f ₄ (o _cm,i ,o _hc ；θ ₄ ) Equation 5 whichMiddle, parameter θ ₄ Representing parameters such as weights and biases used in the convolution kernel in the entropy parameter NN (917) (if biases are used in the entropy parameter NN (917)). Output o of entropy parameter NN (917) _ep Can be used to determine (e.g., adjust) an entropy model, so the conditional entropy model can be, for example, via the output o from the super-decoder (925) _hc But depends on the input signal x. In the example, output o _ep Including parameters such as mean and scale parameters for adjusting the entropy model (e.g., GMM). Referring to fig. 9B, the entropy encoder (913) and the entropy decoder (914) may employ an entropy model (e.g., a conditional entropy model) in entropy encoding and entropy decoding, respectively.

The second sub NN (952) may be described as follows. The potential value y may be fed to a super encoder (921) to generate a super potential value z. In an example, the super-encoder (921) is implemented using a neural network, such as CNN. The relationship between the super potential value z and the potential value y may be described using equation 6.

z＝f ₅ (y；θ ₅ ) Equation 6 wherein the parameter θ ₅ Representation parameters such as weights and offsets used in the convolution kernel in the super-encoder (921) (if offsets are used in the super-encoder (921)).

The hyper-latent value z is quantized by a quantizer (922) to generate a quantized latent value

. Quantified potential value->

The compression may be performed by an entropy encoder (923), e.g., using lossless compression, to generate side information, such as encoded bits (932) from a super neural network. The entropy encoder (923) may use entropy encoding techniques such as huffman coding or arithmetic coding, or the like. In an example, the entropy encoder 923 uses arithmetic coding and is an arithmetic encoder. In an example, side information such as encoded bits (932) may be transmitted in an encoded code stream, e.g., along with an encoded signal (931).

Side information such as coded bits (932)To be decompressed (e.g., entropy decoded) by an entropy decoder (924) to generate an output. The entropy decoder (924) may use entropy coding techniques such as huffman coding or arithmetic coding, or the like. In an example, the entropy decoder (924) uses arithmetic decoding and is an arithmetic decoder. In an example, lossless compression is used in the entropy encoder (923), lossless decompression is used in the entropy decoder (924), and noise, such as due to transmission of side information, is negligible, the output from the entropy decoder (924) may be quantized latent values

. The super decoder (925) may determine the quantized potential value @>

Decoding to generate an output o _hc . Output o _hc And quantized potential value>

The relationship therebetween can be described using equation 7.

Wherein the parameter theta ₆ Representing parameters such as weights and offsets used in the convolution kernel in the super-decoder (925) (if offsets are used in the super-decoder (925)).

As described above, compressed or encoded bits (932) may be added as side information to the encoded code stream, which enables the entropy decoder (914) to use a conditional entropy model. Thus, the entropy model may be signal-dependent (e.g., block-dependent or image-dependent) and spatially adaptive, and thus may be more accurate than the fixed entropy model.

The NIC framework (900) may be suitably modified, for example, to omit at least one component shown in fig. 9B, to modify at least one component shown in fig. 9B, and/or to include at least one component not shown in fig. 9B. In an example, a NIC framework using a fixed entropy model includes a first sub-NN (951) and does not include a second sub-NN (952). In an example, the NIC framework includes components in the NIC framework (900) other than the entropy encoder (923) and the entropy decoder (924).

In an embodiment, at least one component in the NIC framework (900) shown in fig. 9B is implemented using at least one neural network, such as at least one CNN. Each NN-based component (e.g., primary encoder network (911), primary decoder network (915), context model NN (916), entropy parameters NN (917), super-encoder (921), or super-decoder (925)) in a NIC framework (e.g., NIC framework (900)) may include any suitable architecture (e.g., with any suitable combination of layers), including any suitable type of parameters (e.g., weights, offsets, and/or combinations of weights and offsets, etc.), and including any suitable number of parameters.

In an embodiment, the primary encoder network (911), the primary decoder network (915), the context model NN (916), the entropy parameters NN (917), the super-encoder (921), and the super-decoder (925) are implemented using respective CNNs.

Fig. 10 shows an exemplary CNN of a primary encoder network (911) according to an embodiment of the application. For example, the master encoder network (911) includes four sets of layers, where each set of layers includes a convolutional layer 5 x 5c192 s2, followed by a GDN layer. At least one of the layers shown in fig. 10 may be modified and/or omitted. At least one additional layer may be added to the primary encoder network (911).

Fig. 11 shows an exemplary CNN of a primary decoder network (915) according to an embodiment of the application. For example, the primary decoder network (915) includes three sets of layers, where each set of layers includes an inverse convolutional layer 5 x 5c192 s2, followed by an IGDN layer. In addition, three sets of layers are followed by a deconvolution layer 5 × 5c3 s2, followed by IGDN layers. At least one of the layers shown in fig. 11 may be modified and/or omitted. At least one additional layer may be added to the primary decoder network (915).

Fig. 12 shows an exemplary CNN of a super (hyper) encoder (921) according to an embodiment of the present application. For example, the super encoder (921) includes a convolutional layer 3 × 3c192 s1 (followed by a leaky ReLU), a convolutional layer 5 × 5c192 s2 (followed by a leaky ReLU), and a convolutional layer 5 × 5c192 s2. At least one of the layers shown in fig. 12 may be modified and/or omitted. At least one additional layer may be added to the super-encoder (921).

Fig. 13 shows an exemplary CNN of a super decoder (925) according to an embodiment of the present application. For example, the super decoder (925) includes an deconvolution layer 5 × 5c192 s2 (followed by a leaky ReLU), a deconvolution layer 5 × 5c288 s2 (followed by a leaky ReLU), and a deconvolution layer 3 × 3c384 s1. At least one of the layers shown in fig. 13 may be modified and/or omitted. At least one additional layer may be added to the super-decoder (925).

Fig. 14 shows an exemplary CNN of a context model NN (916) according to an embodiment of the present application. For example, the context model NN (916) includes a mask convolution 5 × 5c384 s1 for context prediction, so the context in equation 4

Including a finite context (e.g., a 5 x 5 convolution kernel). The convolutional layer in fig. 14 may be modified. At least one additional layer may be added to the context model NN (916). />

Fig. 15 shows an exemplary CNN of an entropy parameter NN (917) according to an embodiment of the present application. For example, entropy parameters NN (917) include convolutional layer 1 × 1c640 s1 (followed by leaky ReLU), convolutional layer 1 × 1c512s1 (followed by leaky ReLU), and convolutional layer 1 × 1c384 s1. At least one of the layers shown in fig. 15 may be modified and/or omitted. At least one additional layer may be added to the entropy parameter NN (917).

The NIC framework (900) may be implemented using CNNs, as described with reference to fig. 10-15. The NIC framework (900) may be suitably modified such that at least one component (e.g., (911), (915), (916), (917), (921), and/or (925)) in the NIC framework (900) is implemented using any suitable type of neural network (e.g., a CNN or non-CNN based neural network). At least one other component of the NIC framework (900) may be implemented using at least one neural network.

A NIC framework (900) including a neural network (e.g., CNN) may be trained to learn parameters used in the neural network. For example, when using CNN, the parameter θ can be learned separately during training ₁ -θ ₆ Such as weights and biases used in the convolution kernels in the primary encoder network (911) (if biases are used in the primary encoder network (911)), weights and biases used in the convolution kernels in the primary decoder network (915) (if biases are used in the primary decoder network (915)), weights and biases used in the convolution kernels in the super-encoder (921) (if biases are used in the super-encoder (921)), weights and biases used in the convolution kernels in the super-decoder (925) (if biases are used in the super-decoder (925)), weights and biases used in at least one convolution kernel in the context model NN (916) (if biases are used in the context model NN (916)), and weights and biases used in the convolution kernels in the context parameter NN (917) (if biases are used in the entropy parameter NN (917)).

In an example, referring to fig. 10, the primary encoder network (911) includes four convolutional layers, where each convolutional layer has a 5 × 5 convolutional kernel and 192 channels. Thus, the number of weights used in the convolution kernel in the primary encoder network (911) is 19200 (i.e., 4 × 5 × 5 × 192). The parameters used in the primary encoder network (911) include 19200 weights and optionally offsets. When a bias and/or at least one additional NN is used in the primary encoder network (911), at least one additional parameter may be included.

Referring to fig. 9b, the nic framework (900) includes at least one component or module built on at least one neural network. The at least one component may include at least one of a primary encoder network (911), a primary decoder network (915), a super encoder (921), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). At least one component may be trained separately. In an example, a training process is used to learn the parameters of each component separately. At least one component may be co-trained as a group. In an example, a training process is used to jointly learn parameters of a subset of at least one component. In an example, a training process is used to learn parameters for all of the at least one component, hence the term E2E optimization.

During training of at least one component in the NIC framework (900), a weight (or weight coefficient) of the at least one component may be initialized. In an example, the weights are initialized based on at least one pre-trained corresponding neural network model (e.g., DNN model, CNN model). In an example, the weights are initialized by setting them to random numbers.

For example, after the weights are initialized, a set of training blocks may be employed to train at least one component. The set of training blocks may include any suitable blocks having at least one of any suitable size. In some examples, the set of training blocks includes blocks from original images, natural images, and/or computer-generated images in the spatial domain. In some examples, the set of training blocks includes a block, from a residual block or residual image having residual data in the spatial domain. The residual data may be calculated by a residual calculator (e.g., residual calculator (723)). In some examples, the original image and/or the residual image including the residual data may be used directly to train a neural network in a NIC framework, such as the NIC framework (900). Thus, the original image, the residual image, the blocks from the original image, and/or the blocks from the residual image may be used to train a neural network in the NIC framework.

For the sake of brevity, the training process is described below with the training block as an example. The description may be suitably adapted to the training image. The training blocks t in the training block set may be generated by the encoding process in fig. 9B to be a compressed representation (e.g., encoded information (e.g., of the incoming codestream)). The encoded information may be used to calculate and reconstruct the reconstructed block by the decoding process described in FIG. 9B

。

For the NIC framework (900), two competing goals, such as reconstruction quality and bit consumption, are balanced. Quality loss function (e.g. distortion or distortion loss)

Can be used to indicate reconstruction quality, such as reconstruction (e.g., reconstruction block @)>

) And the original block(e.g., training blocks). The rate (or rate loss) R may be used to indicate the bit consumption of the compressed representation. In an example, the rate loss R further comprises side information, e.g. for determining a context model.

For neural network image compression, a quantized differentiable approximation may be used for E2E optimization. In various examples, noise injection is used to model quantization during training of neural network-based image compression, so quantization is modeled by noise injection rather than being performed by a quantizer (e.g., quantizer (912)) I. Thus, training with noise injection can approximate the quantization error in variance (variational). A Bit Per Pixel (BPP) estimator may be used to model the entropy encoder, so entropy encoding is modeled by the BPP estimator rather than performed by the entropy encoder (e.g., (913)) and the entropy decoder (e.g., (914)). Thus, the rate loss R in the loss function L shown in equation 1 during the training process can be estimated, for example, based on a noise injection and BPP estimator. In general, a higher rate R may allow for a lower distortion D, while a lower rate R may result in a higher distortion D. Thus, the compromise hyperparameter λ in equation 1 can be used to optimize the joint R-D penalty L, where L is the sum of λ D and R. The training process may be used to adjust parameters of at least one component (e.g., (911), (915)) in the NIC framework (900) such that the joint R-D loss L is minimized or optimized. In some examples, a compromise hyperparameter λ may be used to optimize joint rate-distortion (R-D) losses, such as:

where E is used to measure the distortion between the decoded block and the original block residual before encoding, which is the regularization loss of residual encoding/decoding DNN and encoding/decoding DNN. β is a hyper-parameter that balances the importance of regularization loss.

Various models can be used to determine the distortion loss D and the rate loss R, and thus, the joint R-D loss L in equation 1. In an example, distortion loss

Expressed as peak signal-to-noise ratio (PSNR), which is a metric based on mean square error, multiscale structural similarity (MS-SSIM) quality index, a weighted combination of PSNR or MS-SSIM, or the like.

In an example, the goal of the training process is to train an encoding neural network (e.g., encoding DNN), such as a video encoder to be used at the encoder side, and to train a decoding neural network (e.g., decoding DNN), such as a video decoder to be used at the decoder side. In an example, referring to fig. 9B, the coding neural network may include a primary encoder network (911), a super encoder (921), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The decoding neural network may include a master decoder network (915), a super-decoder (925), a context model NN (916), and an entropy parameter NN (917). The video encoder and/or the video decoder may comprise at least one other component, based on the at least one NN, and/or not based on the at least one NN.

A NIC framework, such as NIC framework (900), may be trained in an E2E manner. In an example, during training, the encoding neural network and the decoding neural network are jointly updated in an E2E manner based on the back-propagation gradient.

After the parameters of the neural network in the NIC framework (900) are trained, at least one component in the NIC framework (900) may be used to encode and/or decode the block. In an embodiment, at the encoder side, the video encoder is configured to encode the input signal x into an encoded signal (931) to be transmitted in a codestream. The video encoder may include a number of components in the NIC framework (900). In an embodiment, at the decoder end, a corresponding video decoder is configured to decode an encoded signal (931) in a code stream into a reconstructed signal

. The video decoder may include a number of components in the NIC framework (900). />

In an example, the video encoder includes all components in the NIC framework (900), such as when content adaptive online training is employed.

Fig. 16A shows an exemplary video encoder (1600A) according to an embodiment of the application. The video encoder (1600A) comprises, for example, the main encoder network (911), the quantizer (912), the entropy encoder (913), and the second sub-NN (952) described with reference to fig. 9B. Fig. 16B shows an exemplary video decoder (1600B) according to an embodiment of the application. The video decoder (1600B) may correspond to the video encoder (1600A). The video decoder (1600B) may include a master decoder network (915), an entropy decoder (914), a context model NN (916), an entropy parameter NN (917), an entropy decoder (924), and a super-decoder (925). Referring to fig. 16A-16B, at the encoder side, the video encoder (1600A) may generate an encoded signal (931) and encoded bits (932) to be transmitted in a codestream. At the decoder end, a video decoder (1600B) may receive and decode the encoded signal (931) and the encoded bits (932).

Fig. 17-18 illustrate an exemplary video encoder (1700) and corresponding video decoder (1800), respectively, according to embodiments of the present application. Referring to fig. 17, the encoder (1700) includes a main encoder network (911), a quantizer (912), and an entropy encoder (913). An example of a primary encoder network (911), quantizer (912) and entropy coder (913) is described with reference to fig. 9B. Referring to fig. 18, the video decoder (1800) includes a main decoder network (915) and an entropy decoder (914). An example of the master decoder network (915) and the entropy decoder (914) is described with reference to fig. 9B. Referring to fig. 17 and 18, the video encoder (1700) may generate an encoded signal (931) to be included in the bitstream. The video decoder (1800) may receive and decode the encoded signal (931).

Post-filtering or post-processing may be applied to the reconstructed signal, such as the reconstructed block or the reconstructed image, to determine a post-filtered signal (e.g., a post-filtered image or a post-filtered block). Post-filtering may be applied to reduce distortion loss of the post-filtered signal, e.g., the distortion loss of the post-filtered signal may be less than the distortion loss of the corresponding reconstructed signal.

Fig. 19A illustrates an example of block-by-block image filtering according to an embodiment of the present application. Instead of post-filtering the entire image directly (1980), a block-based filtering mechanism may be applied to the post-filtering. The entire image (1980) is partitioned into blocks of the same size (e.g., wxH samples) or of different sizes (1981) - (1996), and the blocks (1981) - (1996) are post-filtered separately. In an embodiment, a post-filtering module is applied to blocks (1981) - (1996) instead of images (1980).

In an embodiment, an image (e.g., a reconstructed image) is post-filtered without being segmented into blocks, so the entire image is the input to the post-filtering module.

Fig. 19B illustrates an exemplary post-filtering module (1910) and a NIC framework (e.g., NIC framework (900)) according to embodiments of the present application. The NIC framework (900) is described with reference to fig. 9B. Reconstructed signal from main decoder network (915)

Can be fed into a post-filtering module (1910) and post-filtered signal->

May be generated by a post-filter module (1910). In an example, a reconstruction block +>

Distortion loss in->

Is reduced to a post-filter signal>

Is lost->

In which the post-filter signal>

Is lost->

Less than the reconstruction block->

Is lost->

。

The post-filtering module (1910) may be implemented using any suitable filtering method that can reduce distortion loss and improve visual quality. According to embodiments of the application, the post-filtering module (1910) is NN-based and is implemented using NN (e.g., post-filtering NN), such as post-filtering DNN or post-filtering CNN.

The post-filtering NN may include any suitable architecture (e.g., with any suitable combination of layers), include any suitable type of parameters (e.g., weights, biases, and/or a combination of weights and biases, etc.), and include any suitable number of parameters, as described herein.

In an example, a post-filtering NN is implemented using a CNN that includes at least one convolutional layer. The post-filtering NN may comprise at least one additional layer described in the present application, such as at least one pooling layer, at least one fully connected layer, and/or at least one normalization layer, etc. The layers in the post-filter NN may be arranged in any suitable order and in any suitable architecture (e.g., a feed-forward architecture or a recursive architecture). In an example, the convolutional layer is followed by at least one other layer, such as at least one pooling layer, at least one fully-connected layer, and/or at least one normalizing layer, among others. Each of the convolution layers in the post-filtering NN may include any suitable number of channels and any suitable convolution kernels and steps.

The post-filtered NN may be trained based on training signals (e.g., training images and/or training blocks) that are similar or identical to those described above with reference to fig. 9B. In some examples, characteristics of an input signal (e.g., an input image or input block) to be compressed (e.g., encoded) and/or transmitted are significantly different from characteristics of a training signal. Therefore, post-filtering a reconstructed signal corresponding to the input signal using post-filtering NN trained with the training signal may result in relatively poor R-D loss L (e.g., relatively large distortion). Accordingly, aspects of the present application describe a content adaptive online training method for post filtering. In a content adaptive online training method for post filtering, at least one parameter of a post filter NN may be determined based on at least one input signal to be compressed (e.g., encoded) and/or transmitted. The at least one input signal may comprise raw data or residual data.

In an embodiment, the at least one input signal is at least one block in the input image. At least one block may be used to determine at least one parameter in the post-filtering NN by optimizing the rate-distortion performance, and the content adaptive online training method for post-filtering may be referred to as a block-wise content adaptive online training method for post-filtering.

In an embodiment, the at least one input signal is at least one input image. At least one input image may be used to determine at least one parameter in the post-filtering NN by optimizing the rate-distortion performance, and the content-adaptive online training method for post-filtering may be referred to as an image-based content-adaptive online training method for post-filtering.

Postfilter information indicative of the determined at least one parameter may be encoded into the video bitstream together with the encoded at least one signal (e.g., the encoded at least one block or the encoded at least one image). At the decoder side, a post-filtering NN may be determined based on the determined at least one parameter, and the reconstructed at least one signal may be post-filtered, achieving better compression performance by using the determined at least one parameter. The reconstructed at least one signal may be reconstructed by the video decoder based on the encoded at least one signal, respectively.

According to aspects of the present application, content adaptive online training may be applied in post-filtering (such as NN-based post-filtering). NN-based post-filtering (e.g., DNN-based post-filtering or CNN-based post-filtering) may be implemented in an NN-based image encoding framework (e.g., NIC framework (900)) or other image compression methods.

The post-filtering module (1910) may be implemented using NN (e.g., DNN or CNN) and may be applied to the reconstructed signal

. The NN in the post-filtering module may be referred to as a post-filtering NN, such as post-filtering DNN or post-filtering CNN. The output of the post-filter module (1910) is a post-filtered signal ≥ h>

. According to embodiments of the present application, content adaptive online training of the post-filter module (1910) may be implemented to reduce the R-D penalty @, as shown in equation 9>

。

Wherein and a post-filtered signal

Corresponding R-D loss->

May include post-filtering the signal->

Is lost->

Encoded signal->

Is lost R->

And signaling rate loss or bit consumption R (p) of post-filtering information in the encoded video bitstream. The post-filtering information may indicate at least one parameter of the post-filtering NN.

For simplicity, the content adaptive online training of the post-filtering module (1910) may be referred to as post-filtering training. The block-wise content adaptive online training method for postfiltering may be referred to as block-wise postfiltering training. The image-based content adaptive online training method for post-filtering may be referred to as image-based post-filtering training.

In an embodiment, the goal of post-filter training is to reduce (e.g., minimize) the R-D loss L _p So that R-D loses L _p Less than the reconstructed signal

R-D loss L->

As described in equation 1. Comparing equation 1 and equation 8, the rate lost @>

Remain unchanged. Increased R-D loss L due to rate loss R (p) _p Therefore, the post-filtering training can significantly reduce the distortion loss

So that>

According to an embodiment of the application, at least one parameter in the post-filtering NN may be determined, for example, in post-filtering training based on at least one input signal. Referring to fig. 19B, an input signal x is fed into the NIC framework (900), and an encoded signal is generated in the encoding process

. Generating a reconstruction signal->

. In an embodiment, at least one parameter in the post-filtering NN may be based on the reconstruction signal->

To be determined. In an example, at least one parameter in the post-filtering NN is based on the reconstruction signal->

Is iteratively updated to reduce distortion loss >>

. The post-filtering NN therefore depends on the reconstruction signal ≥>

The reconstructed signal depends on the input signal x to be encoded.

The post-filtering NN may be configured with initial parameters (e.g., initial weights and/or initial biases), for example, prior to post-filtering training. In an embodiment, the post-filtered NN is pre-trained based on a training dataset comprising training blocks and/or training images, and the initial parameters comprise pre-training parameters (e.g. pre-training weights and/or pre-training biases). In an embodiment, the post-filtering NN is not pre-trained. The initial parameter may be a random parameter.

In an embodiment, at least one of the initial parameters in the post-filtering NN is based on a reconstructed signal determined from the input signal x in post-filtering training

Are iteratively updated. At least one initial parameter may be updated (e.g., replaced) by at least one parameter (e.g., at least one replacement parameter) determined in post-filtering training. In an example, initial parameters are updatedThe entire set of numbers. In an example, a subset of the initial parameters is updated by at least one replacement parameter, and a remaining subset of the initial parameters is kept unchanged by post-filtering training.

Post-filtering training may be used as a pre-processing step (e.g., a pre-encoding step) to improve the compression performance of any image compression method.

The determined at least one parameter may be associated with the encoded signal

Encoded together into a video bitstream. Encoded signal

Is lost by the rate loss R (` is `)>

) Indicating a rate loss of the encoded at least one parameter, denoted by a rate loss R (p). Post-filtering training is agnostic to the image compression codec and may be implemented with any suitable type of image compression codec. The image compression codec may be NN-based, such as the NIC framework (900) shown in fig. 9B and 19B. The image compression codec may be implemented without the NN, such as in some implementations of fig. 5-8.

The post-filtering module (1910) may be applied to the reconstructed signal by using the determined at least one parameter (e.g., at least one replacement parameter) in the post-filtering NN

(e.g., reconstructed image or reconstructed block) and achieve better distortion performance, such as smaller R-D loss L _p . As described with reference to equation 8, R (p) represents the bit consumption of at least one parameter (e.g., at least one replacement parameter) encoded into the video bitstream.

The above description may be suitably modified when a plurality of input signals are used to determine at least one parameter, wherein the determined at least one parameter is shared by the plurality of input signals.

Referring to fig. 19B, at the encoder side, the reconstructed signal in the post-filtering training may be based

To determine at least one parameter of a post-filtering NN in a post-filtering module (1910). Can generate an encoded signal->

. In an embodiment, post-filtering information indicative of at least one parameter of the post-filtering NN may be encoded. Post-filtering information and encoded signal->

And (4) correspondingly. Encoded signal->

And post-filtering information may be included in the encoded video stream.

Fig. 20A shows an exemplary video decoder (2000A) according to an embodiment of the application. The video decoder (2000A) may include the components of the video decoder (1600B) described with reference to fig. 16B and the post-filtering module (1910) described with reference to fig. 19B.

Fig. 20B shows an exemplary video decoder (2000B) according to an embodiment of the application. The video decoder (2000B) may comprise the components of the video decoder (1800) described with reference to fig. 18 and the post-filtering module (1910) described with reference to fig. 19B.

On the decoder side, referring to fig. 19B, 20A, and 20B, the post-filtering information may be decoded by a video decoder (e.g., video decoder (2000A) or (2000B)). At least one parameter of a post-filtering NN in the post-filtering module (1910) may be obtained based on the post-filtering information. In an embodiment, the post-filtering NN is configured with initial parameters, and the at least one parameter is at least one replacement parameter. The at least one initial parameter may be updated with the at least one replacement parameter, respectively.

In an example, the post-filtering information is indicative of a difference between the at least one parameter and a respective at least one of the initial parameters. The at least one parameter may be determined from the difference and a sum of the respective at least one initial parameter of the at least one initial parameter.

A post-filtering NN in a post-filtering module (1910) in a video decoder (e.g., video decoder (2000A) or (2000B)) may be determined based on post-filtering information. In an embodiment, at least one initial parameter in the post-filtering NN is updated based on the at least one parameter, e.g. replaced by the at least one initial parameter.

Referring to FIG. 19B, FIG. 20A and FIG. 20B, the encoded signal

Can be decoded by the video decoder (2000A) or (2000B). For example, the master decoder network (915) reconstructs an encoded signal &>

To generate a reconstruction signal->

. The reconstructed signal may be reconstructed on the basis of a post-filtering NN>

Post-filtering, NN, is performed, which is determined (e.g., updated) based on at least one parameter. In an example, a reconstructed signal &basedon an updated post-filter NN>

Post-filtering is performed.

In some examples, at least one parameter indicated by the post-filtering information is decompressed and then used to update the post-filtering NN.

According to embodiments of the present application, the post-filtering training may be referred to as a fine tuning process, wherein at least one of the initial parameters (e.g., at least one pre-training parameter) in the post-filtering NN may be updated (e.g., fine tuned) based on at least one input signal to be encoded and/or transmitted. The at least one input signal may be different from the training signal used to obtain the initial parameters. Thus, the post-filtering NN may be adapted to target the content of at least one input signal.

In an example, the at least one input signal comprises at least one input signal to be encoded and/or transmitted. In an example, the at least one input signal includes a single input signal (e.g., a single input image or a single input block), and the post-filter training is performed on the single input signal. A post-filtering NN is determined (e.g., trained or updated) based on a single input signal. The post-filtering NN determined at the decoder end may be used for post-filtering reconstructed signals (e.g. reconstructed images or reconstructed blocks) corresponding to a single input signal, and optionally other reconstructed signals corresponding to other input signals. The post-filter information may be encoded into the video bitstream along with the encoded single input signal (e.g., encoded pictures or encoded blocks).

In an embodiment, the at least one input signal comprises a plurality of input signals, and the post-filter training is performed on the plurality of input signals. A post-filtering NN is determined (e.g., trained or updated) based on a plurality of input signals. The post-filtering NN determined at the decoder end may be used for post-filtering of reconstructed signals corresponding to a plurality of input signals and optionally other reconstructed signals corresponding to other input signals. The post-filter information may be encoded into the video bitstream along with the encoded plurality of input signals.

Rate loss R _p May be increased as post-filtering information is identified in the video bitstream. When the at least one input signal comprises a single input signal, postfiltering information is signaled for each encoded signal, and a rate loss R is signaled _p Is used to indicate the rate loss R due to signaling of post-filtering information for each signal _p Is increased. When the at least one input signal comprises a plurality of input signals, signaling and receiving the plurality of input signalsIndividual input signals share post-filtering information and a loss of utilization R _p Indicating the rate loss R due to signaling of post-filtering information for each signal _p Is increased. Since the post-filtering information is shared by multiple input signals, the rate loss R _p May be less than the rate loss R _p First increase of (b). Thus, in some examples, it may be advantageous to determine (e.g., train or update) the post-filtering NN using multiple input signals.

The post-filtering NN may be specified by different types of parameters, such as weights, offsets, etc. The post-filtering NN may be configured with suitable initial parameters such as weights, offsets or a combination of weights and offsets. When at least one CNN is used, the weights may include elements in a convolution kernel. As described above, a subset or the entire set of initial parameters may be updated by post-filter training.

In an embodiment, the at least one initial parameter that is updated is in a single layer (e.g., a single convolutional layer) of the post-filtering NN. In an embodiment, the at least one initial parameter that is updated is in multiple or all layers (e.g., multiple or all convolutional layers) of the post-filtering NN.

One or more types of parameters may be used to specify the post-filtering NN. In an embodiment, the at least one initial parameter to be updated is at least one bias term, and only the at least one bias term is replaced by the determined at least one parameter. In an embodiment, the at least one initial parameter to be updated is a weight, and only the weight is replaced by the determined at least one parameter. In an embodiment, the at least one initial parameter to be updated comprises a weight and a bias term and is replaced by the determined at least one parameter. In an example, the entire set of initial parameters is replaced by the determined at least one parameter.

The post-filter training may include a plurality of epochs (e.g., iterations) in which at least one initial parameter is updated during the iteration. The post-filtering training may be stopped when the training loss has flattened or is about to flatten. In an example, when training is lost (e.g., R-D loses L _p ) Below a first thresholdAt value, the post-filtering training is stopped. In an example, the post-filtering training is stopped when the difference between two consecutive training losses is below a second threshold.

The two superparameters (e.g., step size and maximum number of steps) may be related to a loss function (e.g., R-D loss L _p ) Together for post-filtering training. The maximum number of iterations may be used as a threshold for the maximum number of iterations to terminate post-filter training. In an example, the post-filter training stops when the number of iterations reaches a maximum number of iterations.

The step size may indicate a learning rate of an online training process (e.g., post-filter training). The step size may be used in a gradient descent algorithm or back propagation calculation performed in post-filtering training. The step size may be determined using any suitable method.

In block-wise post-filtering training, the step size of each block in the image may be different. In an embodiment, the images may be assigned different step sizes in order to achieve better compression results (e.g., better R-D loss Lp).

In image-based post-filtering training, the step size may be different for each image. In an embodiment, different input images may be assigned different step sizes in order to achieve better compression results (e.g., better R-D loss Lp).

In an embodiment, different step sizes are used for signals (e.g., blocks or images) having different types of content to achieve the best results. The signal may include an input signal (e.g., input signal x), an encoded signal (e.g., encoded signal)

) A reconstruction signal (e.g., a reconstruction signal->

) And the like. The input signals are described with reference to figures 9B and 19B (e.g., input signal x), an encoded signal (e.g., encoded signal @>

) And a corresponding reconstruction signal (e.g., reconstruction signal +)>

) The relationship (2) of (c). Different types may refer to different variances. In an example, the step size is determined based on a variance of a signal (e.g., a block or an image) used to update the post-filtering NN. For example, the step size for signals with high variance is larger than the step size for signals with low variance, where high variance is larger than low variance. />

In an embodiment, the step size is selected based on characteristics of the signal (e.g., block or image), such as the RGB variance of the signal. In an embodiment, the step size is selected based on the RD performance (e.g., R-D penalty L) of the signal (e.g., block or image). Multiple sets of parameters (e.g., multiple alternative sets of parameters) may be generated based on different step sizes, and the set with better compression performance (e.g., smaller R-D loss) may be selected.

In an embodiment, the first step size may be used to run a certain number of iterations (e.g., 100). The second step size (e.g., the first step size plus or minus a size increment) may then be used to run a number of iterations. Results from the first step size and the second step size may be compared to determine the step size to use. More than two steps may be tested to determine the optimal step size.

The step size may be varied during post-filter training. The step size may have an initial value at the beginning of the post-filtering training, and the initial value may be reduced (e.g., halved) at a later stage of the post-filtering training, e.g., after a certain number of iterations to achieve finer adjustment (tuning). The step size or learning rate may be changed by the scheduler during the iterative post-filtering training. The scheduler may include a parameter adjustment method for adjusting the step size. The scheduler may determine the value of the step size such that the step size may be increased, decreased, or kept constant in multiple intervals. In an example, the learning rate is changed in each step by the scheduler. A single scheduler or multiple different schedulers may be used for different blocks or different images. Thus, multiple sets of parameters may be generated based on multiple schedulersAnd may be selected to have better compression performance (e.g., smaller R-D loss L _p ) One of a plurality of parameter sets.

In an embodiment, multiple learning rate schedules are assigned for different signals (e.g., different blocks or different images) in order to achieve better compression results. In an example, all blocks in an image share the same learning rate schedule. In an example, a group of images share the same learning rate schedule. In an embodiment, the selection of the learning rate schedule is based on characteristics of the signal (e.g., block or image), such as the RGB variance of the signal. In an embodiment, the selection of the learned rate schedule is based on the RD performance of the signal.

In embodiments, different types of parameters (e.g., biases or weights) in the post-filtering NN may be determined (e.g., updated) using different signals (e.g., different blocks or different images). For example, a first signal (e.g., a first block) is used to update at least one bias in the post-filtering NN, and a second signal (e.g., a second block) is used to update at least one weight in the post-filtering NN.

In an embodiment, multiple signals (e.g., multiple blocks in an image or multiple images) are used to determine (e.g., update) the same at least one parameter. In an example, all blocks in the image are used to determine (e.g., update) the same at least one parameter.

In an embodiment, the at least one initial parameter to be updated is selected based on a characteristic of the signal (e.g., a block or an image), such as the RGB variance of the signal. In an embodiment, the at least one initial parameter to update is selected based on the RD performance of the signal.

At the end of the post-filtering training, at least one updated parameter may be calculated for the respective determined at least one parameter (e.g., the respective at least one replacement parameter). In an embodiment, the at least one updated parameter is calculated as a difference between the determined at least one parameter (e.g., the respective at least one replacement parameter) and the corresponding at least one initial parameter (e.g., the at least one pre-training parameter). In an embodiment, the at least one updated parameter is the determined at least one parameter, respectively.

In an embodiment, how to obtain the at least one updated parameter from the respective determined at least one parameter depends on the signal used in the post-filtering training (e.g. input signal x, reconstructed signal)

). Different methods may be used for different signals. In an example, the updated parameters of the post-filtering NN applied to the first block are calculated as the difference between the replacement parameters obtained based on the first block and the corresponding pre-training parameters. In an example, the updated parameters of the post-filtering NN applied to the second block are replacement parameters obtained based on the second block.

In an embodiment, different blocks have different relationships between at least one updated parameter and at least one replacement parameter. For example, for a first block, at least one updated parameter is calculated as a difference between at least one replacement parameter and a corresponding at least one pre-training parameter. For the second block, the at least one updated parameter is at least one replacement parameter, respectively.

In an embodiment, how to obtain the at least one updated parameter from the respective determined at least one parameter does not depend on the signal used in the post-filtering training (e.g. input signal x, reconstructed signal)

). In an example, all blocks share the same way to update at least one parameter in the post-filtering NN. In an embodiment, a plurality of blocks (e.g., all blocks) in the image have the same relationship between the at least one updated parameter and the at least one replacement parameter.

In an embodiment, the relationship between the at least one updated parameter and the at least one replacement parameter is selected based on a characteristic of the signal (e.g. a block or an image), such as the RGB variance of the signal. In an embodiment, the relation between the at least one updated parameter and the at least one replacement parameter is selected based on the RD performance of the signal.

In an embodiment, the at least one updated parameter may be generated from the determined at least one parameter (e.g., the at least one replacement parameter), for example using some linear or non-linear transformation, and the at least one updated parameter is the at least one representative parameter generated based on the determined at least one parameter. The determined at least one parameter is transformed into at least one updated parameter for better compression.

In an example, the at least one updated parameter may be compressed, for example, using LZMA2, which is a variant of Lempel-Ziv-Markov chain algorithm (LZMA) or bzip2 algorithm, or the like. In an example, the compression is omitted for the at least one updated parameter. In some embodiments, the at least one updated parameter may be encoded into the video stream as post-filtering information, where the post-filtering information may indicate the determined at least one parameter (e.g., the at least one replacement parameter).

In an embodiment, the compression method for the at least one updated parameter is different for different signals (e.g. different blocks or different images). For example, for a first block, LZMA2 is used to compress at least one updated parameter, and for a second block, bzip2 is used to compress at least one updated parameter. In an embodiment, the same compression method is used to compress at least one updated parameter for a plurality of blocks (e.g., all blocks) in an image. In an embodiment, the compression method is selected based on characteristics of the signal (e.g., block or image), such as the RGB variance of the signal. In an embodiment, the compression method is selected based on the RD performance of the signal.

In an embodiment, the structure (e.g., architecture) of the post-filtering NN for each signal is the same. The structure of post-filtering NN may include multiple layers, how to organize and connect different nodes and/or layers, feed-forward architecture, recursive architecture, DNN and/or CNN, etc. In an example, the structure refers to a plurality of convolutional layers, and different blocks have the same number of convolutional layers.

In an embodiment, different structures of the post-filtering NN correspond to different signals. In an example, the post-filtering NN for post-filtering different blocks has a different number of convolutional layers.

In the examples, L is lost based on R-D _p A comparison with the R-D penalty L determines whether to post-filter the signal (e.g., block or image) using a post-filtering module. In an embodiment, different R-D losses L based on different post-filter modules _ps To select the best R-D performance (e.g., minimum R-D loss L) _p ) The post-filter module.

According to an embodiment of the present application, each signal (e.g., each block or each image) corresponds to a post-filter NN determined based on the signal in post-filter training. The post-filtering NN may be updated independently of another post-filtering NN. For example, the post-filtering NN corresponding to the first signal is updated independently of the post-filtering NN corresponding to the second signal.

According to an embodiment of the present application, a post-filtering NN corresponding to a signal (e.g., a second signal) may be updated based on a post-filtering NN corresponding to another signal (e.g., a first signal). In an example, first post-filtering information in an encoded code stream corresponding to a first signal is decoded. The first post-filtering information may indicate a first parameter of a post-filtering NN in the video decoder. A post-filtering NN in the video decoder corresponding to the first signal may be determined based on the first parameter indicated by the first post-filtering information. In an example, the first signal includes a first encoded block to be reconstructed. The first signal may be decoded based on the determined post-filtering NN corresponding to the first signal. For example, the first signal is reconstructed and the reconstructed first signal is post-filtered based on the determined post-filtering NN.

At least one parameter in the post-filtering NN corresponding to the first signal may be used to update the post-filtering NN corresponding to the second signal. For example, the distribution of pixels in blocks of the same image may be similar, and thus at least one parameter to be updated of the post-filters NN corresponding to different blocks may be reduced. According to an embodiment of the present application, the second post-filtering information in the encoded code stream corresponding to the second signal may be decoded. The second post-filtering information may indicate a second parameter. The second signal is different from the first signal. The post-filtering NN corresponding to the first signal may be updated based on the second parameter. The updated post-filtering NN corresponds to the second signal and is configured with first parameters and second parameters. The second signal may be decoded (e.g., post-filtered) based on the updated post-filtering NN corresponding to the second signal.

In an embodiment, different post-filtering NNs may be applied to signals (e.g., blocks or images) having different sizes. In general, the number of parameters in the post-filtering NN may increase with the size (e.g., width, height, or area) of the signal (e.g., block or image).

In an embodiment, different post-filtering NNs are applied to the same signal (e.g., image or block) corresponding to different compression quality targets.

The NIC framework and post-filtering NN may comprise any type of neural network and use any neural network based image compression method, such as a context-super-a (superpior) encoder-decoder framework, a scale-super-a encoder-decoder framework, a gaussian mixture likelihood framework and variants of the gaussian mixture likelihood framework, an RNN based recursive compression method and variants of the RNN based recursive compression method, and the like.

Compared with the related E2E image compression method, the post-filtering training method and apparatus in the present application may have the following benefits. And the coding and decoding efficiency is improved by utilizing an adaptive online training mechanism. Various types of pre-training frameworks and quality metrics may be accommodated using a flexible and generic framework. For example, by using online training, certain pre-training parameters in the post-filtering NN may be replaced by at least one block or at least one image to be encoded and transmitted.

Video coding techniques may include filtering operations performed on reconstructed samples such that artifacts caused by lossy compression (such as by quantization) may be reduced. A deblocking filtering process is used in one such filtering operation, where block boundaries (e.g., boundary regions) between two adjacent blocks may be filtered such that a smoother transition of sample values from one block to another may be achieved.

In some related examples (e.g., HEVC), a deblocking filtering process may be applied to samples adjacent to block boundaries. The deblocking filtering process may be performed on each CU in the same order as the decoding process. For example, the deblocking filtering process may be performed by first performing horizontal filtering on a vertical boundary of an image and then performing vertical filtering on a horizontal boundary of the image. For the luminance component and the chrominance component, filtering may be applied to the 8 × 8 block boundary determined to be filtered. In an example, 4 x 4 block boundaries are not processed in order to reduce complexity.

The Boundary Strength (BS) may be used to indicate the degree or strength of the deblocking filtering process. In an embodiment, a value of 2 for BS indicates strong filtering, 1 indicates weak filtering, and 0 indicates no deblocking filtering.

Blocks of an image may be reconstructed from an encoded video bitstream using any suitable method, such as embodiments herein. For example, a block may be reconstructed using, for example, a video decoder (e.g., (1600B), (1800), (2000A), or (2000B)) that includes a neural network (e.g., CNN). non-NN based video decoders may be used to reconstruct blocks. According to some embodiments of the present application, post-filtering or post-processing may be performed on one of the regions of the first two neighboring reconstructed blocks of the reconstructed block using at least one post-processing NN. The first two neighboring reconstructed blocks may have a first shared boundary and include boundary regions with samples on both sides of the first shared boundary. The plurality of regions of the first two adjacent reconstructed blocks may include a boundary region and a non-boundary region outside the boundary region. One of the plurality of regions may be replaced with a post-processed region of the plurality of regions of the first two neighboring reconstructed blocks. The post-processing performed may be deblocking the bounding region, enhancing at least one non-bounding region, and/or a combination of deblocking and enhancing, among others.

One or more deblocking methods may be used to reduce artifacts among blocks (e.g., reconstructed blocks in an image). To reduce artifacts among blocks, such as artifacts in boundary regions, at least one NN-based deblocking model may be used. The NN-based deblocking model may be a DNN-based deblocking model, a CNN-based deblocking model, or the like. The NN-based deblocking model may be implemented using NN (such as DNN or CNN, etc.).

In an embodiment, one of the plurality of regions is a border region. The at least one post-processing NN includes at least one deblocking NN, and deblocking may be performed on the boundary region using the at least one deblocking NN. The bounding region may be replaced with a deblocked bounding region. Examples of deblocking are shown in fig. 21A to 21C, fig. 22, fig. 23, and fig. 26.

Fig. 21A-21C illustrate an exemplary deblocking process (2100) according to embodiments of the present application. Referring to fig. 21A, the image (2101) may be divided into a plurality of blocks (2111) - (2114). For simplicity, four equal-sized blocks (2111) - (2114) are illustrated in fig. 21A. In general, an image may be partitioned into any suitable number of blocks, and the sizes of the blocks may be different or the same, and the description may be modified as appropriate. In some examples, regions that include artifacts, such as artifacts caused by partitioning an image into blocks, may be processed by deblocking.

In an example, the blocks (2111) - (2114) are reconstructed blocks from a primary decoder network (915). The first two adjacent reconstructed blocks of reconstructed blocks (2111) - (2114) may include blocks (2111) and (2113) separated by a first shared boundary (2141). Blocks (2111) and (2113) may include boundary region a with samples on both sides of the first shared boundary (2141). Referring to fig. 21A through 21B, the boundary area a may include sub-boundary areas A1 and A2 located in the blocks (2111) and (2113), respectively.

Two adjacent reconstructed blocks of reconstructed blocks (2111) - (2114) may include blocks (2112) and (2114) separated by a second shared boundary (2142). Blocks (2112) and (2114) may include boundary regions B with samples on both sides of the second shared boundary (2142). Referring to fig. 21A through 21B, the boundary region B may include sub-boundary regions B1 and B2 located in blocks (2112) and (2114), respectively.

Two adjacent reconstructed blocks of reconstructed blocks (2111) - (2114) may include blocks (2111) and (2112) separated by a shared boundary (2143). Blocks (2111) and (2112) may include a boundary region C with samples on both sides of the shared boundary (2143). Referring to fig. 21A through 21B, the boundary region C may include sub-boundary regions C1 and C2 located in the blocks (2111) and (2112), respectively.

Two adjacent reconstructed blocks of reconstructed blocks (2111) - (2114) may include blocks (2113) and (2114) separated by a shared boundary (2144). Blocks (2113) and (2114) may include a boundary region D with samples on both sides of the shared boundary (2144). Referring to fig. 21A through 21B, the boundary region D may include sub-boundary regions D1 and D2 located in the blocks (2113) and (2114), respectively.

The sub-bounding regions A1-D1 and A2-D2 (as well as the bounding regions A-D) may have any suitable size (e.g., width and/or height). In the embodiment shown in FIG. 21A, the sub-boundary regions A1, A2, B1, and B2 have the same size m n, where n is the width of the blocks (2111) - (2114) and m is the height of the sub-boundary regions A1, A2, B1, and B2. Both m and n are positive integers. In an example, m is four pixels or four samples. Therefore, the boundary areas a and B have the same size of 2m × n. The sub-boundary regions C1, C2, D1, and D2 have the same size n × m, where n is the height of the blocks (2111) - (2114) and m is the width of the sub-boundary regions C1, C2, D1, and D2. Therefore, the boundary areas C and D have the same size n × 2m. As described above, the sub-bounding regions and the bounding regions may have different sizes, such as different widths and/or different heights, etc. For example, the sub-boundary regions A1 and A2 may have different heights. In an example, the sub-boundary regions C1 and C2 may have different widths. The boundary regions a and B may have different widths. The boundary regions C and D may have different heights.

Referring to fig. 21A through 21B, the boundary area a includes m rows of samples (e.g., m rows of samples) in the block (2111) from the first shared boundary (2141) and m rows of samples (e.g., m rows of samples) in the block (2113) from the first shared boundary (2141). The boundary region C includes m rows of samples (e.g., m columns of samples) in the block (2111) from the shared boundary (2143) and m rows of samples (e.g., m columns of samples) in the block (2112) from the shared boundary (2143).

Deblocking may be performed on at least one of the boundary regions a-D using at least one deblocking NN, such as deblocking NN based on at least one DNN, at least one CNN, or at least one any suitable NN. In an example, the at least one deblocking NN includes a deblocking NN (2130). In an example, a deblocking NN (2130) is implemented using a CNN that includes at least one convolutional layer. The deblocking NN (2130) may include at least one additional layer described herein, such as at least one pooling layer, at least one fully connected layer, and/or at least one normalization layer, among others. The layers in deblocking NN (2130) may be arranged in any suitable order and in any suitable architecture (e.g., feed forward architecture, recursive architecture). In an example, the convolutional layer is followed by at least one other layer, such as at least one pooling layer, at least one fully-connected layer, and/or at least one normalizing layer, among others.

Deblocking can be performed on the boundary areas A-D using deblocking NN (2130). At least one of the boundary regions a-D comprises an artifact. Artifacts may be caused by individual neighboring blocks. At least one of the boundary regions a-D may be sent to the deblocking NN (2130) to reduce artifacts. Thus, the input of the deblocking NN (2130) comprises at least one of the boundary regions A-D, and the output from the deblocking NN (2130) comprises at least one of the boundary regions A-D that was deblocked.

Referring to fig. 21B, the boundary areas a-D include artifacts caused by respective neighboring blocks. The boundary regions a-D may be sent to the deblocking NN (2130) to reduce artifacts. The output from the deblocking NN (2130) includes the deblocked boundary region a '-D'. In an example, artifacts in the deblocked border region a '-D' are reduced compared to artifacts in the border region a-D.

Referring to FIGS. 21B and 21C, the boundary regions A-D in the image (2101) are updated, for example, by being replaced by deblocked boundary regions A '-D'. Thus, an image (2150) is generated and includes deblocked boundary regions A '-D' and non-boundary regions (2121) - (2124).

At least one sample may be in a plurality of border regions. When multiple bounding regions are replaced by corresponding deblocked bounding regions, any suitable method may be used to determine the value of one of the at least one shared sample.

Referring to fig. 21A, a sample S is in boundary regions a and C. After the boundary regions a 'and C' are obtained, the following method may be used to obtain the value of the sample S. In an example, the boundary region a is replaced by a deblocked boundary region a 'and subsequently the boundary region C is replaced by a deblocked boundary region C'. Therefore, the value of the sample S is determined by the value of the sample S in the deblock boundary region C'.

In an example, the boundary region C is replaced by a deblocked boundary region C 'and subsequently the boundary region a is replaced by a deblocked boundary region a'. Therefore, the value of the sample S is determined by the value of the sample S in the deblock boundary region a'.

In an example, the value of the sample S is determined by an average (e.g., a weighted average) of the value of the sample S in the deblock boundary region a 'and the value of the sample S in the deblock boundary region C'.

The border region may include more than two blocks of samples. Fig. 22 shows an example of a boundary region including samples of more than two blocks according to an embodiment of the present application. The single border area AB may include border areas a and B. The boundary region AB may include samples on both sides of a shared boundary (2141) between two adjacent reconstructed blocks (2111) and (2113), and include samples on both sides of a shared boundary (2142) between two adjacent reconstructed blocks (2112) and (2114). The single border area CD may include border areas C and D. The boundary region CD may include samples on both sides of a shared boundary (2143) between two adjacent reconstructed blocks (2111) and (2112), and include samples on both sides of a shared boundary (2144) between two adjacent reconstructed blocks (2113) and (2114).

Deblocking NN, such as deblocking NN (2130), may perform deblocking on at least one of the boundary areas AB and CD to generate at least one deblocked boundary area in the boundary area. Referring to fig. 22, the boundary areas AB and CD are sent to the deblocking NN (2130), and deblocked boundary areas AB 'and CD' are generated. The deblocked boundary regions AB 'and CD' may replace the boundary regions AB and CD in the image (2101), thus generating an image (2250). The image (2250) may include the deblocked border region AB '-CD' and the non-border regions (2121) - (2124).

According to embodiments of the present application, a multi-model deblocking method may be used. Different deblocking models can be applied to different types or classes of bounding regions to remove artifacts. A classification module may be applied to classify the bounding regions into different categories. Any classification module may be applied. In an example, the classification module is NN-based. In an example, the classification module is not based on NN. The bounding regions may be sent to different deblocking models according to respective categories.

In an embodiment, the at least one deblocking NN includes a plurality of deblocking NNs implemented based on different deblocking models, respectively. It may be determined which of a plurality of deblocking NN is to be applied to the bounding region. Deblocking may be performed on the boundary region using the determined deblocking NN. In an example, which of the multiple deblocking NNs to apply is determined by a classification module based on the NNs (e.g., also referred to as classification NNs), such as DNNs or CNNs.

FIG. 23 shows an exemplary deblocking process (2300) based on multiple deblocking models, according to an embodiment of the application. The classification module (2310) may classify the bounding regions A-D into at least one category. For example, bounding regions C-D are classified into a first category, bounding region B is classified into a second category, and bounding region A is classified into a third category. Different deblocking models can be applied to different classes of bounding regions. In fig. 23, deblocking NN (2330) may be used to perform deblocking, such as multi-model deblocking based on multiple deblocking models (e.g., deblocking models 1-L). L is a positive integer. When L is 1, the deblocking NN (2330) includes a single deblocking model. When L is greater than 1, deblocking NN (2330) includes a plurality of deblocking models.

In an example, the deblocking model 1 is applied to at least one boundary region (e.g., C and D) in the first category and generates at least one deblocked boundary region (e.g., C "and D"). The deblocking model 2 is applied to at least one boundary region (e.g., B) in the second category and at least one deblocked boundary region (e.g., B ") is generated. The deblocking model 3 is applied to at least one boundary region (e.g., a ") in the third category and a deblocked boundary region (e.g., a") is generated. The deblocked bounding region A "-D" may replace the corresponding bounding region A-D in the image (2101), thus generating the image (2350). The image (2350) may include the deblocked border region A "-D" and the non-border regions (2121) - (2124).

Any suitable metric may be applied to classify or categorize the boundary region. In an example, the bounding regions are classified according to their content. For example, boundary regions with high frequency content (e.g., content with relatively large variance) and boundary regions with low frequency content (e.g., content with relatively small variance) are classified into different categories corresponding to different deblocking models. The strength of the artifacts in the boundary region can be used to classify the boundary region. The multi-model deblocking method may be applied to any suitable boundary region, such as a boundary region between two or more blocks (e.g., A, B, C, D, AB, and/or CD). The frequency of the boundary region may be determined based on a maximum difference of the samples within the boundary region. In an example, a first difference value of samples near a first edge in a first side of a shared boundary is determined. In an example, a second difference of samples near a second edge in a second side of the shared boundary is determined. In an example, a first difference and a second difference are determined.

Deblocking NN (e.g., deblocking NN (2130) in fig. 21B or deblocking NN (2330) in fig. 23) may be applied to remove artifacts among blocks. In an example, more deblocking may be performed on samples (or pixels) near the shared boundary than on samples (or pixels) further from the shared boundary. Referring back to fig. 21A, sample S is closer to the shared boundary than sample F (2141), so sample S can be deblocked more than sample F.

The deblocking model in the deblocking NN (e.g., deblocking NN (2130) in fig. 21B or deblocking NN (2330) in fig. 23) may include at least one convolutional layer. For example, a CNN-based attention mechanism (e.g., a non-local attention, pinch and fire network (SENet)) and/or a residual neural network (ResNet) (e.g., including a set of CNNs or convolutional neural networks (convnets) and activation functions) may be used, among others. For example, the DNN used for image super-resolution may be used, for example, by changing the output size to be the same as the input size. In image super-resolution, the resolution of an image may be increased from a low resolution to a high resolution.

The above describes how deblocking is performed on at least one boundary region using NN or other learning-based methods. In some examples, the video encoder and/or the video decoder may select between an NN-based deblocking method or a non-NN-based deblocking method. Selection may be made at various levels, such as at a slice level, a picture level, a group of pictures, and/or a sequence level, among others. A flag may be used to signal the selection. The selection may be inferred from the content of the bounding region.

In addition to the methods and embodiments described herein, the video encoder and/or video decoder may apply various levels of boundary strength, for example, when the NN-derived adjustment on pixels or samples is at a default level of Boundary Strength (BS). By analyzing boundary conditions and block coding characteristics, BSs of different levels may be assigned to modify (e.g., zoom in or out) the default adjustment.

According to embodiments of the present application, the at least one post-processing NN may comprise at least one enhanced NN. At least one of the non-boundary regions of adjacent reconstructed blocks may be enhanced with at least one enhancement NN. At least one of the non-border regions may be replaced with an enhanced at least one of the non-border regions.

The reconstructed image (e.g., image (2101) in fig. 21A) may be sent to an enhancement module to generate an enhanced image (e.g., a final reconstructed image). In some embodiments where deblocking is performed, the reconstructed image may be sent to an enhancement module after artifact reduction by using deblocking NN. To enhance the quality of the image, an NN-based post-enhancement model (e.g., at least one DNN-based or at least one CNN-based post-enhancement model) may be used in a post-enhancement module such as the post-enhancement NN (2430) in fig. 24.

Fig. 24 shows an exemplary enhancement process (2400) according to an embodiment of the application. In some examples, non-boundary regions (2121) - (2124) (e.g., remaining regions other than the boundary regions a-D) in the image (2101) are not sent to a deblocking module (e.g., deblocking NN (2130)). In an example, the non-boundary region (e.g., non-boundary region (2121)) is from a reconstruction block (e.g., (2111)) in the image (e.g., (2101)), and the size of the boundary region may be (n-m) × (n-m). As described with reference to fig. 21A, n is a side length (e.g., width and/or height) of a reconstruction block (e.g., (2111)), and m is a side length of a sub-boundary region (e.g., A1) for deblocking. The at least one non-boundary region (2121) - (2124) may be sent to an enhancement module to further improve the quality of at least one of the non-boundary regions (2121) - (2124). The enhanced at least one of the non-boundary regions may replace at least one of the non-boundary regions (2121) - (2124) in the image. Referring to fig. 24, the non-boundary regions (2121) - (2124) are fed into a post-enhancement NN (2430) to generate enhanced non-boundary regions (2121 ') - (2124'). The enhanced non-boundary regions (2121 ') - (2124') may replace the enhanced non-boundary regions (2121) - (2124) to generate an enhanced image (2450).

In an example, the non-bounding region overlaps the bounding region such that a portion of the non-bounding region is in the bounding region. In an example, the non-boundary region is the entire encoded block. Referring to fig. 24, a block (2111) may be a non-boundary region, and thus the non-boundary region (2111) is adjacent to other adjacent blocks such as (2112) - (2113).

In some embodiments, the at least one augmented NN is based on a plurality of augmented models (e.g., post-augmented models), respectively. Which of the plurality of enhancement models to apply to the non-boundary region may be determined, for example, by a classification module. The non-boundary region may be enhanced with the determined enhancement model. In an example, which of the plurality of augmented models to apply is determined by a classification module based on NN (e.g., also referred to as classification NN), such as DNN or CNN. The classification module (e.g., classification module (2510)) used in the post-enhancement process (e.g., (2500)) may be the same as or different from the classification module (e.g., classification module (2310)) used in the deblocking process (e.g., (2300)). The classification module (e.g., classification module (2510)) used in the post-enhancement process may include an NN (e.g., DNN or CNN). In an example, the classification module (e.g., classification module (2510)) used in the post-enhancement process does not include the NN.

Fig. 25 illustrates an exemplary enhancement process (2500), such as a multi-model post enhancement module according to an embodiment of the present application.

The classification module (2510) may classify the non-boundary regions (2121) - (2124) into at least one class. For example, the non-boundary regions (2122) - (2123) are classified into a first category, and the non-boundary regions (2121) and (2124) are classified into a second category. Different enhancement models (e.g., post-enhancement models) may be applied to different classes of non-bounding regions. In fig. 25, an augmented NN (2530) may be used to perform augmentation, such as multi-model augmentation based on multiple augmentation models (e.g., augmentation models 1-J). J is a positive integer. When J is 1, the augmented NN (2530) includes a single augmented model. When J is greater than 1, the augmented NN (2530) includes a plurality of augmented models.

In an example, enhancement model 1 is applied to at least one non-bounding region (e.g., (2122) - (2123)) in the first category and at least one enhanced non-bounding region (e.g., (2122 ") - (2123")) is generated. The augmented model 2 is applied to at least one non-bounding region (e.g., (2121) and (2124)) in the second class and at least one augmented non-bounding region (e.g., (2121 ") and (2124")) is generated. An enhanced non-boundary region (2121 ") - (2124") may replace the corresponding non-boundary region (2121) - (2124), wherein the enhanced image (2550) includes the enhanced non-boundary region (2121 ") - (2124") and the boundary region a-D.

Any suitable metric may be applied to classify or categorize the non-boundary regions. In an example, the non-bounding regions are classified according to their content. For example, non-boundary regions with high frequency content (e.g., content with relatively large variance) and non-boundary regions with low frequency content (e.g., content with relatively small variance) are classified into different categories corresponding to different enhancement models.

The image may be enhanced at the block level as described with reference to fig. 21 to 25. An enhancement model (e.g., a post-enhancement model) may enhance the entire image. FIG. 26 shows an exemplary image-level enhancement process (2600) to enhance an entire image according to an embodiment of the application. The image (2101) includes non-boundary regions (2121) - (2124) and boundary regions A-D, as depicted in FIG. 21A. In an example, as described above, the image (2101) is a reconstructed image including the reconstruction blocks (2111) - (2114). Artifacts in the border areas may be reduced and non-border areas may be enhanced with improved visual quality.

Referring to FIG. 26, an image (2101) including bounding regions A-D and non-bounding regions (2121) - (2124) may be fed into an enhancement module (2630). The enhancement module (2630) may generate enhanced bounding regions E-H corresponding to the bounding regions A-D, respectively, for example, by deblocking the bounding regions A-D. The enhancement module (2630) may generate enhanced non-border regions (2621) - (2624) corresponding to the non-border regions (2121) - (2124), respectively. The enhanced border regions E-H may replace border regions A-D, respectively, and the enhanced non-border regions (2621) - (2624) may replace non-border regions (2121) - (2124), respectively, thus generating an enhanced image (2650) based on the reconstructed image (2101).

In an example, the image-based enhancement module (2630) includes an enhancement NN that may perform both deblocking and enhancement. In an example, the image-based enhancement module (2630) includes an enhancement NN that may perform enhancement and a deblocking NN that may perform deblocking.

The enhancing modules (e.g., (2430), (2530), and (2630)) described with reference to fig. 24 to 26 may enhance the quality of the image. The boost modules (e.g., (2430), (2530), and (2630)) may include at least one convolutional layer. CNN-based attention mechanisms (e.g., non-local attention, sentet) and/or ResNet (e.g., including a set of CNNs or convolutional neural networks (convnets) and activation functions) may be used, among others. For example, the DNN used for image super-resolution may be used, for example, by changing the output size to be the same as the input size.

The boundary regions and non-boundary regions in the image may be processed by the enhancement NN and the deblocking NN in any suitable order, such as sequentially or simultaneously. In an example, the boundary region is deblocked by the deblocking NN, and subsequently, the non-boundary region is processed by the enhancement NN. In an example, non-boundary regions are processed by the enhancement NN, and then boundary regions are deblocked by the deblocking NN.

According to embodiments of the application, an enhanced NN (e.g., (2430), (2530), or (2630)), a deblocked NN (e.g., (2130) or (2330)), and/or a classified NN (e.g., (2310) or (2510)) may include any neural network architecture, may include any number of layers, may include at least one sub-neural network (as described herein), and may be trained using any suitable training images or training blocks. The training image may comprise an original image or an image containing residual data. The training block may be from an original image or an image that includes residual data.

Content adaptive online training may be applied to update at least one pre-training parameter in one of an enhanced NN (e.g., (2430), (2530), or (2630)), a deblocked NN (e.g., (2130) or (2330)), and/or a classified NN (e.g., (2310) or (2510)), as described herein.

The enhanced NN (e.g., (2430), (2530), or (2630)), the deblock NN (e.g., (2130) or (2330)), and/or the class NN (e.g., (2310) or (2510)) may be trained separately, e.g., a single deblock NN is trained to determine pre-training parameters in the deblock NN. The enhanced NN (e.g., (2430), (2530), or (2630)), the deblocked NN (e.g., (2130) or (2330)), and/or the classified NN (e.g., (2310) or (2510)) may be trained (pre-trained or on-line) as a component in the NIC framework. For example, at least one of the NIC framework (900) and the enhanced NN (e.g., (2430), (2530), or (2630)), the deblocking NN (e.g., (2130) or (2330)), and/or the classification NN (e.g., (2310) or (2510)) may be jointly trained.

In an embodiment, the post-filtering NN in post-filtering module (1910) may include at least one of an enhancement NN (e.g., (2430), (2530), or (2630)), a deblocking NN (e.g., (2130) or (2330)), and a classification NN (e.g., (2310) or (2510)). An enhanced NN (e.g., (2430), (2530), or (2630)) and a deblocked NN (e.g., (2130) or (2330)) may be applied to the reconstructed image or reconstructed block.

The enhancement process (e.g., (2400), (2500), or (2600)) may be referred to as a post-enhancement process, e.g., when the enhancement process is performed after the block is reconstructed. For the same reason, the boost module (e.g., (2430), (2530), or (2630)) may be referred to as a post-boost module or post-boost NN.

In some examples, the reconstructed image (2101) includes residual data.

Fig. 27 shows a flowchart outlining an encoding process (2700) according to an embodiment of the present application. Process (2700) may be used to encode an input signal (e.g., an input block or an input image). In various embodiments, process (2700) is performed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330), and (340), processing circuitry that performs the functions of a video encoder (e.g., (403), (603), (703), (1600A), or (1700)), and so on. In an example, the processing circuit performs a combination of functions, such as (i) one of the video encoder (403), the video encoder (603), and the video encoder (703), and (ii) one of the video encoder (1600A) or the video encoder (1700). In some embodiments, process (2700) is implemented in software instructions such that when the processing circuit executes the software instructions, the processing circuit performs process (2700). The process starts (S2701), and proceeds to (S2710).

At (S2710), an encoded signal is generated based on the input signal (e.g., x) using any suitable method (e.g.,

). In an example, as described with reference to fig. 9B and 19B, a coded signal +is generated based on an input signal x>

. The input signal may be an input block or an input image. />

At (S2720), based on a reconstructed signal in post-filtering training (e.g.,

) At least one parameter of a post-filtering NN in a post-filtering module (e.g., (1910)) is determined. Referring to FIG. 19B, a NIC framework (900) may be used that based on an encoded signal ≦ based on>

Generating a reconstruction signal>

. Post-filter training may be implemented in a post-filter module (1910) to determine at least one parameter of a post-filter NN. The post-filtering NN may be configured with initial parameters, and at least one of the initial parameters may be iteratively updated in post-filtering training. The at least one initial parameter may be replaced by the at least one parameter.

At (S2730), postfilter information indicating at least one parameter of the postfilter NN is encoded. Post-filtering information and encoded signal

And (4) correspondingly. In some examples, an encoded signal ≧ may be transmitted in an encoded video bitstream>

And post-filter information. The process (2700) proceeds to (S2799), and ends.

Process (2700) can be adapted to various scenarios as appropriate, and the steps in process (2700) can be adjusted accordingly. At least one of the steps in process (2700) may be modified, omitted, repeated, and/or combined. Process (2700) may be implemented using any suitable order. At least one additional step may be added.

Fig. 28 shows a flowchart outlining the decoding process (2800) according to an embodiment of the application. The process (2800) may be used for reconstruction and post-filtering of an encoded signal (e.g., an encoded block or an encoded image). In various embodiments, process (2800) is performed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330), and (340), processing circuitry that performs the functions of video decoder (1600B), processing circuitry that performs the functions of a video decoder (e.g., (2000A) or (2000B)). In an example, the processing circuit performs a combination of functions, such as (i) one of the video decoder (410), the video decoder (510), and the video decoder (810), and (ii) one of the video decoder (2000A) or the video decoder (2000B). In some embodiments, process (2800) is implemented in software instructions, such that when the software instructions are executed by the processing circuitry, the processing circuitry performs process (2800). The process starts at (S2801), and proceeds to (S2810).

At (S2810), first post-filtering information corresponding to a first encoded signal in the encoded video bitstream is decoded. The first encoded signal may be an encoded block or an encoded picture. The first post-filtering information may indicate a first parameter (e.g., a first post-filtering parameter) in the video decoder. Post-filtering NN may be implemented by a post-filtering module (e.g., (1910)) in a video decoder (e.g., (2000A) or (2000B)).

In an example, the first encoded signal includes a first encoded image to be reconstructed, and the first post-filtering information corresponds to the first encoded image.

In an example, the first post-filtering information further corresponds to a second encoded image to be reconstructed.

The first parameter (e.g., the first post-filtering parameter) may be a bias term or a weight coefficient in the post-filtering NN. In an example, the first post-filtering parameter is updated based on a bias term or weight coefficient in the post-filtering NN.

The first post-filtering information may indicate at least one parameter of the post-filtering NN in various ways. In an example, the post-filtering NN is configured with initial parameters. The first post-filtering information indicates a difference between the first parameter and one of the initial parameters, from which the first parameter may be determined. In another example, the first post-filtering information is directly indicative of the at least one parameter.

In an example, the number of layers in the post-filtering NN depends on the size (e.g., width, height, or area) of the first encoded signal.

In an example, the number of layers in the post-filtering NN depends on the step size, or the number of step sizes corresponding to different ones of the at least one block.

In an example, an image or video is received, the image or video including at least one block. A first post-filtering parameter corresponding to at least one block to be reconstructed in the image or video is decoded. The first post-filtering parameter may be applied to one or more of the at least one block. The first post-filter parameters are updated by a post-filter module in the post-filter NN, and the post-filter NN is trained based on a training data set.

At (S2820), a post-filtering NN in the video decoder corresponding to the first encoded signal is determined based on, for example, a first parameter (e.g., a first post-filtering parameter) indicated by the first post-filtering information. For example, at least one of the initial parameters configured for the post-filtering NN is updated (e.g., replaced) with a first parameter (e.g., a first post-filtering parameter).

At (S2830), the first encoded signal is decoded based on a post-filtering NN corresponding to the first encoded signal. The first encoded signal may be reconstructed to generate a reconstructed signal. The reconstructed signal may be post-filtered based on the determined post-filtering NN to generate a post-filtered signal.

In an example, the first encoded signal includes an encoded block, which may be reconstructed. The reconstruction block may be post-filtered based on the determined post-filtering NN to generate a post-filtered block.

In an example, the first encoded signal includes a first encoded image, and the first encoded image is decoded based on a post-filtering NN corresponding to the first encoded image.

In an example, the second encoded image is decoded based on a post-filtering NN corresponding to the first encoded image.

The process (2800) proceeds to (S2899), and ends.

The process (2800) may be adapted to various scenarios as appropriate, and the steps in the process (2800) may be adjusted accordingly. At least one of the steps in the process (2800) may be modified, omitted, repeated, and/or combined. The process (2800) may be implemented using any suitable order. At least one additional step may be added.

In an example, a second post-filtering parameter corresponding to at least one block in an image or video is decoded. A post-filtering NN may be determined based on the second post-filtering parameters. The second post-filtering parameter may be applied to a second block of the at least one block. The second block may be different from one or more of the at least one block. The second post-filtering parameters have been updated by the post-filtering module in the post-filtering NN.

The first post-filtering parameter may be different from the second post-filtering parameter, the first post-filtering parameter being adaptive to the content of a first block of the at least one block, the second post-filtering parameter being adaptive to the content of a second block. The first block may be different from the second block.

The first post-filtering parameter may be determined based on the content of a first block of the at least one block and the second post-filtering parameter may be determined based on the content of a second block.

In an example, the first post-filtering parameters correspond to a second image to be reconstructed and the second image is decoded based on the determined post-filtering NN.

In an example, second post-filtering information corresponding to a second encoded signal in the encoded video stream is decoded. The second post-filtering information indicates a second parameter. The second encoded signal is different from the first encoded signal. The post-filtering NN corresponding to the first encoded signal may be updated based on the second parameters. The updated post-filtering NN corresponds to the second encoded signal and may be configured with first parameters and second parameters. The first and second parameters may comprise (i) two different weight coefficients, (ii) two different biases, or (iii) weight coefficients and biases of the updated post-filtering NN. The second encoded signal may be decoded based on an updated post-filtering NN corresponding to the second encoded signal.

In an example, the first post-filtering parameters are updated in (i) a single layer of the post-filtering NN, (ii) multiple layers of the post-filtering NN, or (iii) all layers of the post-filtering NN.

The embodiments in this application may be used alone or in any order in combination. Further, each of the method (or embodiment), the encoder and the decoder may be implemented by a processing circuit (e.g., at least one processor or at least one integrated circuit). In one example, at least one processor executes a program stored in a non-transitory computer readable storage medium.

The present application does not impose any limitations on the methods used for encoders (such as neural network-based encoders), decoders (such as neural network-based decoders). The at least one neural network used in the encoder and/or decoder etc. may be any suitable type of at least one neural network, such as DNN and CNN etc.

Therefore, the content adaptive online training method of the present application may adapt to different types of NIC frameworks, such as different types of encoded DNN, decoded DNN, encoded CNN, and/or decoded CNN, and so on.

The techniques described above may be implemented as computer software via computer readable instructions and physically stored in at least one computer readable storage medium. For example, fig. 29 illustrates a computer system (2900) suitable for implementing certain embodiments of the disclosed subject matter.

The computer software may be encoded in any suitable machine code or computer language, and may be encoded by any suitable mechanism for assembling, compiling, linking, etc., code comprising instructions that are directly executable by at least one computer Central Processing Unit (CPU), graphics Processing Unit (GPU), etc., or executable by code translation, microcode, etc.

The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, internet of things devices, and so forth.

The components illustrated in FIG. 29 for computer system (2900) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the application. Neither should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiments of computer system (2900).

The computer system (2900) may include some human interface input devices. Such human interface input devices may respond to input from at least one human user by tactile input (e.g., keyboard input, swipe, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gesture), olfactory input (not shown). The human-machine interface device may also be used to capture certain media that need not be directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still-image cameras), video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

The human interface input device may include at least one of the following (only one of which is depicted): keyboard (2901), mouse (2902), touch pad (2903), touch screen (2910), data glove (not shown), joystick (2905), microphone (2906), scanner (2907), camera (2908).

The computer system (2900) may also include certain human interface output devices. Such a human interface output device may stimulate the perception of at least one human user by, for example, tactile output, sound, light, and smell/taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (2910), a data glove (not shown), or joystick (2905), but there may also be tactile feedback devices that do not act as input devices), audio output devices (e.g., speakers (2909), headphones (not shown)), visual output devices (e.g., screens (2910) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light emitting diode screens, each with or without touch screen input functionality, each with or without tactile feedback functionality — some of which may output two-dimensional visual output or more than three-dimensional output by means such as stereoscopic picture output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

The computer system (2900) may also include human-accessible storage devices and their associated media, such as optical media including compact disk read-only/rewritable (CD/DVD ROM/RW) (2920) with CD/DVD or similar media (2921), thumb drive (2922), removable hard drive or solid state drive (2923), conventional magnetic media such as magnetic tape and floppy disk (not shown), ROM/ASIC/PLD based application specific devices such as security dongle (not shown), and so forth.

Those skilled in the art will also appreciate that the term "computer-readable storage medium" used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

The computer system (2900) may also include an interface (2954) to at least one communication network (2955). For example, the network may be wireless, wired, optical. The network may also be a local area network, a wide area network, a metropolitan area network, a vehicular network, an industrial network, a real-time network, a delay tolerant network, and so forth. The network also includes ethernet, wireless local area networks, local area networks such as cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable, satellite, and terrestrial broadcast television), automotive and industrial networks (including CANBus), and so forth. Some networks typically require external network interface adapters for connecting to some general purpose data ports or peripheral buses (2949) (e.g., a USB port of computer system (2900)); other systems are typically integrated into the core of computer system (2900) (e.g., ethernet interface integrated into a PC computer system or cellular network interface integrated into a smart phone computer system) by connecting to a system bus as described below. Using any of these networks, computer system (2900) may communicate with other entities. The communication may be unidirectional, for reception only (e.g., wireless television), unidirectional for transmission only (e.g., CAN bus to certain CAN bus devices), or bidirectional, for example, to other computer systems over a local or wide area digital network. Each of the networks and network interfaces described above may use certain protocols and protocol stacks.

The human interface device, human accessible storage device, and network interface described above may be connected to the core (2940) of the computer system (2900).

The core (2940) may include at least one Central Processing Unit (CPU) (2941), a Graphics Processing Unit (GPU) (2942), a special purpose programmable processing unit in the form of a Field Programmable Gate Array (FPGA) (2943), a hardware accelerator (2944) for certain tasks, a graphics adapter (2950), and so on. These devices, as well as Read Only Memory (ROM) (2945), random access memory (2946), internal mass storage (e.g., internal non-user accessible hard drives, solid state drives, etc.) (2947), etc., may be connected via a system bus (2948). In some computer systems, the system bus (2948) may be accessed in the form of at least one physical plug, so as to be extensible through additional central processing units, graphics processing units, and the like. The peripheral devices may be attached directly to the system bus (2948) of the core or connected through a peripheral bus (2949). In an example, the screen (2910) may be connected to a graphics adapter (2950). The architecture of the peripheral bus includes peripheral controller interface PCI, universal serial bus USB, etc.

The CPU (2941), GPU (2942), FPGA (2943), and accelerator (2944) may execute certain instructions, which in combination may constitute the computer code. The computer code may be stored in ROM (2945) or RAM (2946). The transitional data may also be stored in RAM (2946), while the persistent data may be stored in, for example, internal mass storage (2947). Fast storage and retrieval of any memory device can be achieved by using cache memory, which can be closely associated with at least one CPU (2941), GPU (2942), mass storage (2947), ROM (2945), RAM (2946), and the like.

The computer-readable storage medium may have computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present application, or they may be of the kind well known and available to those having skill in the computer software arts.

By way of example, and not limitation, a computer system having architecture (2900), and in particular cores (2940), may provide functionality as a processor (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in at least one tangible computer-readable storage medium. Such computer-readable storage media may be media associated with the user-accessible mass storage described above, as well as certain memory having a non-volatile core (2940), such as core internal mass storage (2947) or ROM (2945). Software implementing various embodiments of the present application may be stored in such a device and executed by the core (2940). The computer-readable storage medium may include one or more storage devices or chips, according to particular needs. The software may cause the core (2940), and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (2946) and modifying such data structures in accordance with the software-defined processes. Additionally or alternatively, the computer system may provide functionality that is logically hardwired or otherwise embodied in circuitry (e.g., accelerator (2944)) that may operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. Where appropriate, reference to software may include logic and vice versa. Where appropriate, reference to a computer-readable storage medium may include storage of circuitry (e.g., an Integrated Circuit (IC)) that executes software, circuitry that contains execution logic, or both. The present application includes any suitable combination of hardware and software.

Appendix A: acronyms

JEM: federated development model

VVC: universal video coding

BMS: reference set

MV: motion vector

HEVC: efficient video coding

SEI: supplemental enhancement information

VUI: video usability information

GOP: picture group

TU: conversion unit

PU (polyurethane): prediction unit

And (3) CTU: coding tree unit

CTB: coding tree block

PB: prediction block

HRD: hypothetical reference decoder

SNR: signal to noise ratio

A CPU: central processing unit

GPU: graphics processing unit

CRT: cathode ray tube having a shadow mask with a plurality of apertures

LCD: liquid crystal display device

An OLED: organic light emitting diode

CD: optical disk

DVD: digital video CD

ROM: read-only memory

RAM: random access memory

ASIC: application specific integrated circuit

PLD: programmable logic device LAN: local area network

GSM: global mobile communication system

LTE: long term evolution

CANBus: controller area network bus

USB: universal serial bus

PCI: peripheral device interconnect

FPGA: field programmable gate array

SSD: solid state drive

IC: integrated circuit with a plurality of transistors

CU: coding unit

NIC: neural network image compression

R-D: rate distortion

E2E: end-to-end

And (3) ANN: artificial neural network

DNN: deep neural network

CNN: convolutional neural network

While the application has described several exemplary embodiments, various modifications, arrangements, and equivalents of the embodiments are within the scope of the application. It will thus be appreciated that those skilled in the art will be able to devise various systems and methods which, although not explicitly shown or described herein, embody the principles of the application and are thus within its spirit and scope.

Claims

1. A method of video decoding in a video decoder, the method comprising:

receiving an image or video, the image or video comprising at least one block;

decoding first post-filtering parameters in the image or video corresponding to the at least one block to be reconstructed, wherein the first post-filtering parameters are applied to one or more of the at least one block, the first post-filtering parameters have been updated by a post-filtering module in a post-filtering neural network NN, the post-filtering NN is trained based on a training data set;

determining the post-filtering NN corresponding to the at least one block in the video decoder based on the first post-filtering parameters; and a (C) and (D) and,

decoding the at least one block based on the determined post-filtering NN corresponding to the at least one block.

2. The method of claim 1, further comprising:

decoding a second post-filtering parameter corresponding to the at least one block in the image or video;

determining the post-filtering NN further based on a second post-filtering parameter;

wherein the second post-filtering parameter is applied to a second block of the at least one block,

the second block is different from the one or more blocks of the at least one block,

the second post-filtering parameters have been updated by the post-filtering module in the post-filtering NN.

3. The method of claim 1, wherein the first post-filtering parameter corresponds to a second image to be reconstructed, the method further comprising:

decoding the second image based on the determined post-filtering NN.

4. The method of claim 2, wherein the first post-filtering parameter is different from the second post-filtering parameter, wherein the first post-filtering parameter is adaptive to a content of a first block of the at least one block, and wherein the second post-filtering parameter is adaptive to a content of the second block.

5. The method according to claim 1, wherein the first post-filtering parameters are updated based on bias terms or weight coefficients in the post-filtering NN.

6. The method according to claim 1, wherein the post-filtering NN is configured with initial parameters;

the determining the post-filtering NN comprises: updating at least one of the initial parameters using the first post-filter parameter.

7. The method of claim 6, wherein the coding information corresponding to the at least one block indicates a difference between the first post-filtering parameter and one of the initial parameters, the method further comprising:

and determining the first post-filtering parameter according to the sum of the difference value and one of the initial parameters.

8. Method according to claim 1, characterized in that the first post-filtering parameters are updated in (i) a single layer of the post-filtering NN, (ii) multiple layers of the post-filtering NN, or (iii) all layers of the post-filtering NN.

9. The method of claim 1, wherein the number of layers in the post-filtering NN depends on a step size, or a number of step sizes corresponding to different ones of the at least one block.

10. A video decoding apparatus, comprising:

a processing circuit to:

receiving an image or video, the image or video comprising at least one block;

decoding first post-filtering parameters in the image or video corresponding to the at least one block to be reconstructed, wherein the first post-filtering parameters are applied to one or more of the at least one block, the first post-filtering parameters have been updated by a post-filtering module in a post-filtering neural network NN, training the post-filtering NN based on a training dataset;

determining the post-filtering NN corresponding to the at least one block in the video decoder based on the first post-filtering parameters; and a process for the preparation of a coating,

11. The apparatus of claim 10, wherein the processing circuit is configured to:

12. The apparatus of claim 10, wherein the first post-filtering parameter corresponds to a second image to be reconstructed, and wherein the processing circuitry is configured to:

decoding the second image based on the determined post-filtering NN.

13. The apparatus of claim 11, wherein the first post-filtering parameter is different from the second post-filtering parameter, wherein the first post-filtering parameter is adaptive to a content of a first block of the at least one block, and wherein the second post-filtering parameter is adaptive to a content of the second block.

14. The apparatus of claim 10, wherein the first postfiltering parameter is updated based on a bias term or a weight coefficient in the postfiltering NN.

15. The apparatus according to claim 10, wherein the post-filtering NN is configured with initial parameters;

the processing circuitry is to: updating at least one of the initial parameters using the first post-filter parameter.

16. The apparatus of claim 15, wherein the coding information corresponding to the at least one block indicates a difference between the first post-filtering parameter and one of the initial parameters, and wherein the processing circuit is configured to:

17. The apparatus according to claim 10, characterized in that the first post-filtering parameters are updated in (i) a single layer of the post-filtering NN, (ii) multiple layers of the post-filtering NN, or (iii) all layers of the post-filtering NN.

18. The apparatus of claim 10, wherein the number of layers in the post-filtering NN depends on a step size, or a number of step sizes corresponding to different ones of the at least one block.

19. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, implement:

receiving an image or video, the image or video comprising at least one block;

20. The non-transitory computer-readable storage medium of claim 19, wherein the instructions, when executed by at least one processor, implement:

decoding second post-filtering parameters corresponding to the at least one block in the image or video;