CN113039804B - Bit stream merging - Google Patents

Abstract

A video encoder (2) for providing an encoded video representation (12), wherein the video encoder (2) is configured to provide a video stream (12) comprising encoding parameter information describing a plurality of encoding parameters (20, 22, 26, 30), encoding video content information and one or more merge identifiers indicating whether the encoded video representation (12) can be merged with another encoded video representation and/or how the encoded video representation (12) can be merged with another encoded video representation.

Description

Bitstream merging

Technical Field

The present application relates to video encoding/decoding.

Background technique

Video synthesis is used in many applications to present a synthesis of multiple video sources to the user. Common examples are picture-in-picture (PiP) synthesis and mixing overlays with video content, e.g. for advertising or user interfaces. Producing such synthesis in the pixel domain requires parallel decoding of the input video bitstreams, which is computationally complex and even not feasible on devices with a single hardware decoder or other limited resources. For example, in current IPTV system designs, capable set-top boxes perform the synthesis and are a major service cost factor due to their complexity, distribution and limited lifespan. Reducing these cost factors has led to continuous efforts to virtualize the set-top box functionality, such as moving user interface generation to cloud resources. With this approach, only the video decoder (so-called zero client) is the only hardware that remains at the customer's premises. The latest state of the art in such system designs is synthesis based on transcoding, i.e., synthesis in its simplest form: decoding, pixel-domain synthesis and re-encoding before or during transmission. In order to reduce the workload of the entire decoding and encoding cycle, it was first proposed to perform PiP synthesis in the transform coefficient domain instead of the pixel domain. Since then, many techniques have been proposed to merge or shorten the individual synthesis steps and apply them to current video codecs. However, transcoding-based approaches for general synthesis are still computationally complex, which hurts the scalability of the system. Depending on the transcoding method, such synthesis may also affect the rate-distortion (RD) performance.

In addition, there are a wide range of applications built on tiles, where a tile is a spatial subset of a video plane that is encoded independently of neighboring tiles. Tile-based streaming systems for 360° video work by splitting the 360° video into tiles, which are encoded into sub-bitstreams at various resolutions and merged into a single bitstream on the client side depending on the current user viewing orientation. Another application involving the merging of sub-bitstreams is, for example, the combination of traditional video content with banner ads. In addition, the recombination of sub-bitstreams can also be a key part of video conferencing systems, where multiple users send their respective video streams to a receiver that eventually merges the streams. Even further, before merging the resulting tile bitstreams back into a single bitstream, tile-based cloud coding systems rely on fail-safe checks, in which the video is divided into tiles and the tile encoding is distributed to separate and independent instances. In all of these applications, the streams are merged to allow decoding on a single video decoder with known consistency points. Merging in this context refers to lightweight bitstream rewriting that does not require full decoding and encoding operations on the bitstream or entropy coded data for pixel value reconstruction. However, techniques for ensuring successful bitstream merging (i.e., consistency of the resulting merged bitstream) are derived from many different application scenarios.

For example, conventional codecs support a technique called motion constrained tile sets (MCTS), in which the encoder constrains inter prediction between pictures to be limited to the boundaries of tiles or pictures, i.e., sample values or syntax element values that do not belong to the same tile or are outside the picture boundaries are not used. This technique is derived from region of interest (ROI) decoding, in which the decoder can decode only specific sub-parts of the bitstream and the coded picture without encountering unresolvable dependencies and avoiding drift. Another recent technology in this context is the picture structure (SOP) SEI message, which gives an indication of the applied bitstream structure (i.e., picture order and inter prediction reference structure). This information is provided by summarizing the picture order count (POC) value, the valid sequence parameter set (SPS) identifier, and the reference picture set (RPS) index into the valid SPS for each picture between two random access points (RAPs). Based on this information, the bitstream structure can be identified, which can help the transcoder, middlebox or media-aware network entity (MANE) or media player to operate or change the bitstream, such as adjusting the bitrate, dropping frames or fast forwarding.

Although both of the above exemplary signaling techniques are necessary in understanding whether lightweight merging of sub-bitstreams can be performed without significant syntax changes or even full transcoding, they are far from sufficient. In more detail, lightweight merging in this context is characterized by interleaving the NAL units of the source bitstreams with only a small rewrite operation, i.e., writing the jointly used parameter set with the new image size and tile structure so that each bitstream to be merged will be located in a separate tile area. The next level of merging complexity consists of a small rewrite of the slice header elements, ideally without changing the variable length codes in the slice header. There are other levels of merging complexity, for example, re-running entropy coding on the slice data to change specific syntax elements that are entropy coded but can be changed without pixel value reconstruction, which can be considered beneficial and quite lightweight compared to full transcoding with decoding and encoding of the video (which is not considered lightweight).

In a merged bitstream, all slices must refer to the same parameter sets. Lightweight merging may not be possible when the parameter sets of the original bitstreams use even significantly different settings, because many parameter set syntax elements have further impact on slice header and slice payload syntax, and their impact varies in degree. The deeper the involvement of a syntax element in the decoding process, the more complex the merging/rewriting becomes. Some noteworthy general categories of syntax dependencies (of parameter sets and other structures) can be distinguished as follows.

A. Syntax Existence Indication

B. Value Calculation Dependency

C. Strip Payload Encoding Tool Control

a. Syntax used in the early stages of the decoding process (e.g. coefficient sign hiding, block partitioning restrictions)

b. Syntax used later in the decoding process (e.g., motion compensation, loop filter) or general decoding process control (reference pictures, bitstream order)

D. Source format parameters

For category A, the parameter set carries a number of presence flags (e.g., dependent_slice_segments_enabled_flag or output_flag_present_flag) for various tools. In the case where there are differences in these flags, the flags can be set to enable in the joint parameter set, and the default values can be explicitly written to the merged slice header of the slices in the bitstream that do not include the syntax elements before the merge, i.e., in this case, the merge requires changes to the parameter set and the slice header syntax.

For category B, the signaled values of the parameter set syntax and parameters of other parameter sets or slice headers may be used in the calculation, for example, in HEVC, the slice quantization parameter (QP) of a slice is used to control the coarseness of the quantization of the transform coefficients of the residual signal of the slice. The signaling of the slice QP (SliceQpY) in the bitstream depends on the QP signaling at the picture parameter set (PPS) level, as shown below:

SliceQpY=26+init_qp_minus26 (from PPS)+slice_qp_delta (from slice header)

Since each slice in the (merged) coded video picture needs to reference the same active PPS, the difference in init_qp_minus26 in the PPS of the individual streams to be merged requires adjustment of the slice header to reflect the new common value of init_qp_minus26. That is, in this case, the merge requires changes to the parameter sets and slice header syntax, as made for category 1.

For category C, other parameter set syntax elements control coding tools that affect the bitstream structure of the slice payload. Subcategories C.a and C.b can be distinguished according to the degree to which the syntax elements participate in the decoding process and the complexity associated with making changes to these syntax elements (related to the degree to which the syntax elements participate in the decoding process), i.e., which processes between entropy coding and pixel level reconstruction the syntax elements participate in.

For example, one element in category C.a is sign_data_hiding_enabled_flag, which controls the derivation of sign data for coded transform coefficients. Sign data hiding can be easily disabled, and the corresponding inferred sign data is explicitly written into the slice payload. However, these changes to the slice payload in this category do not need to be fully decoded into the pixel domain before encoding the pixel-by-pixel merged video again. Another example is an inferred block partitioning decision, or any other syntax that can easily write inferred values to the bitstream. That is, in this case, the merge requires changes to the parameter set, the slice header syntax, and the slice payload that requires entropy decoding/encoding.

However, subcategory C.b requires syntax elements related to processes far down the decoding chain, so many complex decoding processes must be performed anyway in a way that avoiding remaining decoding steps is undesirable in terms of implementation and computation. For example, differences in syntax elements associated with motion compensation constraints "Temporal Motion Constraints Tile Sets SEI (Structure of Picture Information) Messages" or various syntax related to the decoding process (e.g., "SEI messages") make pixel-level transcoding inevitable. Compared to subcategory C.a, there are many encoder decisions that affect the slice payload in ways that cannot be changed without full pixel-level transcoding.

For category D, there are parameter set syntax elements (e.g., chroma subsampling indicated by chroma_format_idc) where different values make pixel level transcoding inevitable for merging sub-bitstreams. That is, in this case, merging requires a full decoding, pixel level merging, and a full encoding process.

The above list is by no means exhaustive, but it is clear that various parameters affect the benefits of merging sub-bitstreams into a common bitstream in different ways, and that tracking and analyzing these parameters is cumbersome.

Summary of the invention

An object of the present invention is to provide a video codec to efficiently merge video bit streams.

This object is achieved by the subject matter of the claims of the present application.

The basic idea of the present invention is to improve the merging of multiple video streams by including one or more merging identifiers. This method can reduce the load on computing resources and can also speed up the merging process.

According to an embodiment of the present application, a video encoder is configured to provide a video stream including encoding parameter information, encoded video content information (i.e., encoded using encoding parameters defined by the parameter information), and one or more merge identifiers, wherein the encoding parameter information describes multiple encoding parameters, and the one or more merge identifiers indicate whether the encoded video representation can be merged with another encoded video representation and/or how the encoded video representation can be merged with another encoded video representation (e.g., which complexity to use). Which complexity to use can be determined based on the parameter value defined by the parameter information. The merge identifier can be a concatenation of multiple encoding parameters, which must be equal in two different encoded video representations so that the two different encoded video representations can be merged using a predetermined complexity. In addition, the merge identifier can be a hash value of the concatenation of multiple encoding parameters, which must be equal in two different encoded video representations so that the two different encoded video representations can be merged using a predetermined complexity.

According to an embodiment of the present application, a merge identifier may indicate a merge identifier type, wherein the merge identifier type indicates an association with the complexity of a merge process, or generally, may indicate a "suitable" merge method of merging by rewriting a parameter set, or by rewriting a parameter set and a slice header, or by rewriting a parameter set, a slice header, or a slice payload, wherein the merge identifier is associated with the merge identifier type, wherein the merge identifier associated with the merge identifier type includes those encoding parameters that must be equal in two different encoded video representations, so that the two different encoded video representations can be merged using the complexity of the merge process indicated by the merge identifier type. The value of the merge identifier type may indicate a merge process, wherein the video encoder is configured to switch between at least two of the following values of the merge identifier type; a first value of the merge identifier type, which indicates a merge process rewritten by a parameter set; a second value of the merge identifier type, which indicates a merge process rewritten by a parameter set and a slice header; and a third value of the merge identifier type, which indicates a merge process rewritten by a parameter set, a slice header, and a slice payload.

According to an embodiment of the present application, multiple merge identifiers are associated with different complexities of the merge process, for example, each identifier indicates a parameter set/hash and the type of the merge process. The encoder can be configured to check whether the encoding parameters evaluated to provide the merge identifier are the same in all units of the video sequence, and to provide the merge identifier based on the check.

According to an embodiment of the present application, multiple encoding parameters may include merge-related parameters, which must be the same in different video streams (i.e., encoded video representations) to allow the merging to be less complex than the merging performed by full-pixel decoding, and wherein the video encoder is configured to provide one or more merge identifiers based on the merge-related parameters, i.e., full-pixel decoding is performed in the absence of common parameters between two encoded video representations, i.e., it is not possible to reduce the complexity of the merging process. The merge-related parameters include one or more or all of the following parameters: parameters describing motion constraints at tile boundaries (e.g., motion constraint tile set supplemental enhancement information (MCTS SEI)); information about the picture group (GOP) structure (i.e., mapping of coding order to display order); random access point indication; for example, temporal layering with picture structure SOP; SEI, parameters describing the chroma coding format and parameters describing the luma coding format, for example, a set including at least the chroma format and bit depth luma/chroma; parameters describing advanced motion vector prediction; parameters describing sample adaptive offset; parameters describing temporal motion vector prediction; and parameters describing loop filters and other coding parameters, i.e., rewriting parameter sets (including reference picture sets, basic quantization parameters, etc.), slice headers, slice payloads to merge two coded video representations.

According to an embodiment of the present application, when a merge parameter associated with a second complexity of a merge process is determined based on a second encoding parameter set, a merge identifier associated with a first complexity of the merge process can be determined based on a first encoding parameter set, the second complexity being higher than the first complexity, the second encoding parameter set being a proper subset of the first encoding parameter set. A merge identifier associated with a third complexity of the merge process can be determined based on a third encoding parameter set, the third complexity being higher than the second complexity, the third encoding parameter set being a proper subset of the second encoding parameter set. The video encoder is configured to determine a merge identifier associated with the first complexity of the merge process based on a coding parameter set, the coding parameter set being, for example, a first set that must be equal in two different video streams (e.g., encoded video representations) to allow merging of video streams in the following manner: only modifying a parameter set that is applicable to multiple slices while keeping the slice header and slice payload unchanged, i.e., a merge process rewritten (only) by a parameter set. The video encoder may be configured to determine a merge identifier associated with a second complexity of a merge process based on a set of encoding parameters, e.g., a second set that must be equal in two different video streams (e.g., encoded video representations) to allow a merge of the video streams in a manner that modifies a set of parameters applicable to multiple slices and also modifies a slice header while keeping the slice payload unchanged, i.e., a merge process through rewriting of parameter sets and slice headers. The video encoder may be configured to determine a merge identifier associated with a third complexity of a merge process based on a set of encoding parameters, e.g., a third set that must be equal in two different video streams (e.g., encoded video representations) to allow a merge of the video streams in a manner that modifies a set of parameters applicable to multiple slices and also modifies a slice header and a slice payload, but does not perform full pixel decoding and pixel re-encoding, i.e., a merge process through rewriting of parameter sets, slice headers and slice payloads.

According to an embodiment of the present application, a video merger for providing a merged video representation based on multiple encoded video representations (e.g., video streams), wherein the video merger is configured to receive multiple video streams including encoding parameter information, encoded video content information (i.e., encoded using encoding parameters defined by the parameter information) and one or more merge identifiers, wherein the encoding parameter information describes multiple encoding parameters, and the one or more merge identifiers indicate whether the encoded video representation can be merged with another encoded video representation and/or how the encoded video representation can be merged with another encoded video representation. The video merger is configured to determine the usage of the merging method (e.g., merging type; merging process) based on the merge identifier (i.e., based on a comparison of the merge identifiers of different video streams). The video merger is configured to select a merging method from multiple merging methods based on the merge identifier. The video merger can be configured to select between at least two of the following merger methods; a first merger method, wherein the first merger method is a merger of video streams in the following manner: only a parameter set that can be applied to multiple slices is modified while keeping a slice header and a slice payload unchanged; a second merger method, wherein the second merger method is a merger of video streams in the following manner: a parameter set that can be applied to multiple slices is modified and a slice header is also modified while keeping a slice payload unchanged; and a third merger method, wherein the third merger method is a merger of video streams in the following manner: a parameter set that can be applied to multiple slices is modified and a slice header and a slice payload are also modified, but full pixel decoding and pixel re-encoding are not performed, which is a merger method with different complexity according to one or more merger identifiers.

According to an embodiment of the present application, a video merger is configured to compare the merge identifiers of two or more video streams associated with the same given merge method or associated with the same merge identifier type, and decide whether to use the given merge method to perform the merge based on the result of the comparison. The video merger can be configured to: if the comparison indicates that the merge identifiers of two or more video streams associated with the given merge method are equal, then the given merge method is used to selectively perform the merge. The video merger can be configured to: if the comparison of the merge identifiers indicates that the merge identifiers of two or more video streams associated with the given merge method are different, then a merge method with a higher complexity than the given merge method associated with the compared merge identifier is used, that is, without further comparing the encoding parameters themselves. The video merger may be configured to selectively compare encoding parameters that must be equal in the two or more video streams to allow the video streams to be merged using the given merger method if a comparison of the merger identifiers indicates that the merger identifiers of two or more video streams associated with a given merger method are equal, and wherein the video merger is configured to selectively use the given merger method to perform the merger if a comparison of the encoding parameters (i.e., encoding parameters that must be equal in the two or more video streams to allow the video streams to be merged using the given merger method) indicates that the encoding parameters are equal, and wherein the video merger is configured to perform the merger using a merger method with a higher complexity than the given merger method if a comparison of the encoding parameters indicates that the encoding parameters include differences.

According to an embodiment of the present application, a video merger may be configured to: compare merger identifiers associated with merger methods having different complexities, i.e., compare hashes, and wherein the video merger is configured to: identify a merger method of the lowest complexity, the associated merger identifiers for which are equal in two or more video streams to be merged; and wherein the video merger is configured to: compare sets of encoding parameters (i.e., individual encoding parameters, not hashed versions thereof), which must be equal in two or more video streams to be merged, to allow the merger to be performed using the identified merger method. and, wherein different, typically overlapping sets of coding parameters are associated with merging methods of different complexity, and wherein the video merger is configured to merge the two or more video streams using the identified merging method if the comparison indicates that the coding parameters of the coding parameter set associated with the identified merging method are equal in the video streams to be merged, and wherein the video merger is configured to merge the two or more video streams using a merging method of higher complexity than the identified merging method if the comparison indicates that the coding parameters of the coding parameter set associated with the identified merging method include differences in the video streams to be merged. The video merger is configured to determine which coding parameters should be modified in the merging process (i.e., merging the video streams) based on one or more differences between merge identifiers associated with the same merging method or "merge identifier type" of different video streams to be merged.

According to an embodiment of the present application, a video merger is configured to obtain joint coding parameters or a joint coding parameter set associated with slices of all video streams to be merged based on the coding parameters of the video streams to be merged, for example, a sequence parameter set SPS and a picture parameter set PPS, and, for example, in the case where the values of all coding parameters of one coded video representation and another coded video representation are the same, include the joint coding parameters in the merged video stream, update the coding parameters by copying the common parameters in the case of any difference between the coding parameters of the coded video representations, update the coding parameters based on the main coded video representation (i.e., one coded video representation is the main coded video representation and the other coded video representation is the sub (sub) coded video representation), for example, some coding parameters (e.g., the total picture size) can be adapted according to the combination of video streams. The video merger is configured to adapt the coding parameters associated with each video slice (e.g., defined in the slice header), or, for example, when a merging method with a complexity higher than the minimum complexity is used, so as to obtain a modified slice to be included in the merged video stream. The adapted encoding parameters comprise parameters representing a picture size of the merged coded video representation, wherein the picture size is calculated based on picture sizes of the coded video representations to be merged (ie in each dimension and in the context of their spatial arrangement).

According to an embodiment of the present application, a video encoder for providing a coded video representation (i.e., a video stream), wherein the video encoder can be configured to provide coarse-grained capability requirement information, such as level information; level 3 or level 4 or level 5, which describes the compatibility of the video stream with a video decoder, such as the decodability of the video stream by a video decoder having a capability level among multiple predetermined capability levels; and wherein the video encoder is configured to provide fine-grained capability requirement information, such as merge level limit information, which describes how large a fraction of an allowable capability requirement (i.e., decoder capability) associated with one of the predetermined capability levels is required to decode the coded video representation, and/or describes how much of the allowable capability requirement is contributed by the coded video representation (i.e., a sub-bitstream) to the merged video stream into which the video stream is merged. The video encoder is configured to provide fine-grained capability requirement information such that the fine-grained capability requirement information includes a ratio value or a percentage value that refers to one of the predetermined capability levels. The video encoder is configured to provide fine-grained capability requirement information such that the fine-grained capability requirement information includes reference information and score information, and wherein the reference information describes which of the predetermined capability levels the score information refers to, so that the fine-grained capability requirement information as a whole describes the score of one of the predetermined capability levels.

According to an embodiment of the present application, a video merger is used to provide a merged video representation based on multiple encoded video representations (i.e., video streams), wherein the video merger can be configured to receive multiple video streams, the multiple video streams including encoding parameter information, encoded video content information (e.g., encoded using encoding parameters defined by the parameter information), coarse-grained capability requirement information and fine-grained capability requirement information (e.g., merge level restriction information), the encoding parameter information describes multiple encoding parameters, the coarse-grained capability requirement information is, for example, level information - level 3 or level 4 or level 5, which describes the compatibility of the video stream with a video decoder, i.e., the decodability of the video stream by the video decoder, the video decoder having one of multiple predetermined capability levels, wherein the video merger is configured to merge two or more video streams according to the coarse-grained capability requirement information and the fine-grained capability requirement information. The video merger can be configured to: i.e., without violating the permissible capability requirement (i.e., such capability requirement of the merged video stream is consistent with one of the predetermined capability levels), determine which video streams can be a merged video stream or can be included in the merged video stream according to the fine-grained capability requirement information. The video merger may be configured to determine whether a valid merged video stream (e.g., without violating the allowed capability requirements, i.e., such that the capability requirements of the merged video stream are consistent with one of the predetermined capability levels) can be obtained by merging two or more video streams according to the fine-grained capability requirement information. The video merger is configured to synthesize the fine-grained capability requirement information of the plurality of video streams to be merged, e.g., to determine which video streams can be included, or to determine whether a valid merged video stream can be obtained.

According to an embodiment of the present application, a video decoder for decoding a provided video representation, wherein the video decoder is configured to receive a video representation comprising multiple sub-video streams (i.e., multiple sub-bitstreams of the entire bitstream), the multiple sub-video streams comprising encoding parameter information, encoded video content information, coarse-grained capability requirement information, and fine-grained capability requirement information, wherein the encoding parameter information describes multiple encoding parameters, and the coarse-grained capability requirement information describes the compatibility of the video stream with a video decoder having a capability level in multiple predetermined capability levels, i.e., the respective information is carried, for example, in an SEI message, wherein the video decoder is configured to determine whether the combined capability requirements described by the fine-grained capability requirement information of the multiple sub-video streams to be merged match the predetermined restrictions to be complied with (i.e., the level-specific restrictions of the decoder). The video decoder may be further configured to parse the received coarse-grained capability requirement information and fine-grained capability requirement information to obtain an indication of the capability level and a score of the allowed capability requirements.

BRIEF DESCRIPTION OF THE DRAWINGS

The preferred embodiments of the present application are described below with reference to the accompanying drawings, in which:

FIG1 shows a block diagram of an apparatus for providing an encoded video representation as an example of a video encoder, in which the bitstream merging concept according to an embodiment of the present application can be implemented;

FIG2 shows a schematic diagram illustrating an example of a bitstream structure according to an embodiment of the present application;

FIG3 shows a block diagram of an apparatus for providing an encoded video representation as another example of a video encoder, in which the bitstream merging concept according to an embodiment of the present application can be implemented;

FIG4 is a schematic diagram showing an example of encoding parameters according to an embodiment of the present application;

FIG5a, FIG5b show a detailed sequence parameter set (SPS) example indicated in FIG4;

FIG. 6 a and FIG. 6 b show a detailed picture parameter set (PPS) example indicated in FIG. 4 ;

7a to 7d show detailed stripe header examples indicated in FIG. 4;

FIG8 shows a detailed structure of a picture in the Supplemental Enhancement Information (SEI) message indicated in FIG4 ;

FIG. 9 shows a detailed motion constraint tile set in the SEI message indicated in FIG. 4 ;

FIG10 shows a block diagram of an apparatus for providing a merged video representation as an example of a video merger, in which the bitstream merging concept according to an embodiment of the present application can be implemented;

FIG11 shows a schematic diagram illustrating a process of determining a merge complexity according to an embodiment of the present application;

FIG12 is a schematic diagram showing a bitstream structure of multiple video representations to be merged and a bitstream structure of a merged video representation according to the bitstream merging concept of the present application;

FIG13 shows a block diagram of an apparatus for providing a merged video representation as another example of a video merger, in which the bitstream merging concept according to an embodiment of the present application can be implemented;

FIG. 14 shows a block diagram of an apparatus for providing encoded video representation as an example of a video encoder, which provides merged video representation capability requirement information that can be implemented according to an embodiment of the present application; and

FIG. 15 shows a block diagram of an apparatus for providing a merged video representation as an example of a video merger, which provides merged video representation capability requirement information that can be implemented according to an embodiment of the present application.

Detailed ways

The following description sets forth specific details, such as specific embodiments, processes, techniques, etc., for purposes of explanation rather than limitation. Those skilled in the art will appreciate that, in addition to these specific details, other embodiments may be used. For example, although the following description is facilitated using non-limiting example applications, the technology may be applied to any type of video codec. In some instances, detailed descriptions of well-known methods, nodes, interfaces, circuits, and devices are omitted so as not to obscure the description with unnecessary detail.

In the following description, the same or equivalent elements or elements having the same or equivalent functions are denoted by the same or equivalent reference numerals.

The invention herein aims to provide future video codecs such as VVC (Versatile Video Coding) with a means of putting an indication in each sub-bitstream that allows identification of sub-bitstreams that can be merged together into a legal bitstream, or identification of sub-bitstreams that cannot be merged together into a legal bitstream at a given level of complexity. The indication, referred to hereinafter as the "merge identifier", also provides information about a suitable merging method through an indication referred to as the "merge identifier type". Assuming that two bitstreams carry a "merge identifier" and its same value, the sub-bitstreams can be merged into a new joint bitstream at a given level of merging complexity (related to the merging method).

FIG1 shows a video encoder 2 for providing an encoded video representation (i.e., a video stream 12) based on a provided (input) video stream, comprising an encoder core 4, the encoder core 4 comprising a coding parameter determination unit 14 and a merge identifier provision 6. The provided video stream and the video stream 12 respectively have a bitstream structure, for example, as shown in FIG2 as a simplified configuration. The bitstream structure consists of a plurality of network abstraction layer (NAL) units, each NAL unit comprising various parameters and/or data, such as a sequence parameter set (SPS) 20, a picture parameter set (PPS) 22, an instantaneous decoder refresh (IDR) 24, a supplemental enhancement information (SEI) 26 and a plurality of slices 28. The SEI 26 comprises various messages, i.e., the structure of a picture, a motion constraint tile set, etc. The slice 28 comprises a header 30 and a payload 32. The coding parameter determination unit 14 determines the coding parameters based on the SPS 20, the PPS 22, the SEI 26 and the slice header 30. According to the present application, IDR 24 is not a necessary factor for determining encoding parameters, but IDR 24 can be optionally included for determining encoding parameters. A merge identifier is provided 6 to provide a merge identifier, which indicates whether a coded video stream can be merged with another coded video stream and/or how (at which complexity) the coded video stream can be merged with another coded video stream. The merge identifier determines a merge identifier, the merge identifier indicates a merge identifier type, the merge identifier type representation is associated with the complexity of the merge process, or generally indicates a "suitable" merge method, for example, merging by rewriting parameter sets, or merging by rewriting parameter sets and slice headers, or merging by rewriting parameter sets, slice headers, and slice payloads, wherein the merge identifier is associated with the merge identifier type, wherein the merge identifier associated with the merge identifier type includes those encoding parameters that must be equal in two different coded video representations, so that the two different coded video representations can be merged using the complexity of the merge process represented by the merge identifier type.

As a result, the encoded video stream includes encoding parameter information describing a plurality of encoding parameters, encoded video content information, and one or more merge identifiers. In this embodiment, the merge identifier is determined based on the encoding parameters. However, the merge identifier value can also be set according to the will of the encoder operator with little guarantee of conflict avoidance (which may be sufficient for a closed system). Third-party entities such as DVB (Digital Video Broadcasting), ATSC (Advanced Television Systems Committee) can define the value of the merge identifier to be used within their system.

Considering another embodiment according to the present invention using FIGS. 3 to 9 , the merge identifier types are described below.

FIG. 3 shows an encoder 2 (2a), which includes an encoder core 4 and a merge identifier 6a, and indicates a data stream in the encoder 2a. The merge identifier 6a includes a first hash component 16a and a second hash component 16b, which provide a hash value as a merge identifier to indicate a merge type. That is, the merge value can be formed by concatenating the coded values of the syntax element (coding parameter) set defined by the bitstream, hereinafter referred to as a hash set. The merge identifier value can also be formed by concatenating the above-mentioned coded values into a well-known hash function (e.g., MD5, SHA-3) or any other suitable function. As shown in FIG. 3, the input video stream includes input video information 10, and the input video stream is processed at the encoder core 4. The encoder core 4 encodes the input video content and stores the encoded video content information in the payload 32. The encoding parameter determination component 14 receives parameter information including SPS, PPS, slice header and SEI message. The parameter information is stored in each corresponding unit and is received at the merge identifier provider 6a, i.e., at the first hash component 16a and the second hash component 16b, respectively. The first hash component 16a generates a hash value indicating, for example, the merge identifier type 2 based on the encoding parameter, and the second hash component 16b generates a hash value indicating, for example, the merge identifier type 1 based on the encoding parameter.

The content of the hash set (ie which syntax element (ie merge parameter) values are concatenated to form the merge identifier value) determines the quality of the mergeability indication with respect to the above syntax category.

For example, the merge identifier type indicates the appropriate merge method for the merge identifier, i.e., the different levels of mergeability corresponding to the syntax elements merged into the hash set:

Type 0 - Merge identifier for merging via parameter set override

Type 1 - Merge identifier for merging with parameter set and stripe header rewrite

Type 2 - Merge identifier for merging by parameter set, slice header, and slice payload rewrite

For example, given two input sub-bitstreams to be merged in the context of the present application, a device may compare the values of the merge identifier and the merge identifier type and conclude on the prospects of using the method associated with the merge identifier type on both sub-bitstreams.

The following table gives the mapping between syntax element categories and the associated merging methods.

Grammatical categories Merge Method A 0 B 1 C.a 2 C.b, D Fully transcoded

As mentioned above, the syntax categories are by no means exhaustive, but it is clear that various parameters affect the benefits of merging sub-bitstreams into a common bitstream in different ways, and it is cumbersome to keep track of and analyze these parameters. In addition, the syntax categories and merging methods (types) do not correspond exactly, so for example, some parameters are required for category B, but the same parameters are not required for merging method 1.

Generate a merged identifier value according to two or more of the above identifier type values, and write these merged identifier values to the bitstream to allow the device to easily identify the applicable merge method. Merge method (type) 0 indicates a first value of the merged identifier type, merge method (type) 1 indicates a second value of the merged identifier type, and merge method (type) 3 indicates a third value of the merged identifier type.

The following example syntax should be incorporated into the hash set.

● Temporal motion constraints tile set SEI message, which indicates motion constraints at tile and picture boundaries (merge methods 0, 1, 2, syntax category C.b)

● Structure of the Picture Information SEI message, which defines the GOP structure (i.e., coding order to display order mapping, random access point indication, temporal layering, reference structure) (Merge methods 0, 1, 2, syntax category C.b)

●Parameter set syntax element value

○ Reference image set (merging method 0, syntax category B)

○ Chroma format (combination method 0, 1, 2, syntax category D)

○Base QP, Chroma QP offset (Merge method 0, syntax category B)

○ Bit depth Luma/Chroma (merging methods 0, 1, 2, syntax class D)

○hrd parameter

■ Initial arrival delay (merging method 0, grammatical class B)

■ Initial removal delay (merge method 0, syntax category B)

○Coding tools

■Coding block structure (maximum/minimum block size, inferred segmentation) (merging method 0, 1, syntax category C.a)

■Transformation size (minimum/maximum) (merge method 0, 1, syntax category C.a)

■PCM block usage (merging method 0, 1, syntax category C.a)

■Advanced motion vector prediction (merge methods 0, 1, 2, syntax class C.b)

■ Sample Adaptive Offset (Merge Method 0, Syntax Category C.b)

■Temporal motion vector prediction (merge method 0, 1, 2, syntax category C.b)

■Intra-frame smoothing (merging method 0, 1, syntax category C.a)

■Related strips (merging method 0, syntax category A)

■Symbol hiding (merging methods 0, 1, syntax category C.a)

■Weighted prediction (merging method 0, grammatical category A)

■Transquant bypass (merge method 0, 1, syntax category C.a)

■ Entropy coding synchronization (merge method 0, 1, syntax category C.a) (Skup4)

■ Loop filter (merging methods 0, 1, 2, syntax category C.b)

●Strip header value

○Parameter set ID (merge method 0, syntax category C.a)

○ Reference image set (merging method 0, 1, grammatical category B)

●Use implicit CTU address signal to transmit cp. (Referenced by European Patent Application No.: EP 18153516) (Merge Method 0, Syntax Element A)

That is, for the first value of the merge identifier type, i.e. type 0, the syntax elements (parameters) motion constraints at tile and picture boundaries, GOP structure, reference picture sets, chroma format, basic quantization parameter and chroma quantization parameter, bit depth luminance/chroma, assumed reference decoder parameters including parameters on initial arrival delay and parameters on initial removal delay, coding block structure, transform minimum and/or maximum size, pulse code modulation block usage, advanced motion vector prediction, sample adaptive offset, temporal motion vector prediction, description of intra-frame smoothing, dependent slices, sign hiding, weighted prediction, transform quantization bypass, entropy coding synchronization, loop filter, slice header value including parameter set ID, slice header value including reference picture set, and use of implicit coded transform unit address signaling should be incorporated into the hash set.

For the second value of the merge identifier type, i.e. type 1, syntax elements (parameters): motion constraints at tile and picture boundaries, GOP structure, chroma format, bit depth luma/chroma, coding block structure, transform minimum and/or maximum size, pulse code modulation block usage, advanced motion vector prediction, sample adaptive offset, temporal motion vector prediction, intra-frame smoothing, sign hiding, transform quantization bypass, entropy coding synchronization, loop filter, and slice header values including reference picture sets.

For the third value of the merge identifier type, type 2, syntax elements (parameters): motion constraints at tile and picture boundaries, GOP structure, chroma format, bit depth luma/chroma, advanced motion vector prediction, sample adaptive offset, temporal motion vector prediction, loop filter.

FIG4 shows a schematic diagram illustrating an example of coding parameters according to an embodiment of the present application. In FIG4 , reference numeral 40 represents type 0, and a syntax element belonging to type 0 is indicated by a dotted line. Reference numeral 42 represents type 1, and a syntax element belonging to type 1 is indicated by a normal line. Reference numeral 44 represents type 2, and a syntax element belonging to type 1 is indicated by a dashed line.

5a and 5b are examples of a sequence parameter set (SPS) 20, and syntax elements required for type 0 are indicated by reference numeral 40. In the same manner, syntax elements required for type 1 are indicated by reference numeral 42, and syntax elements required for type 2 are indicated by reference numeral 44.

6 a and 6 b are examples of a picture parameter set (PPS) 22 , where syntax elements required for type 0 are indicated by reference numeral 40 , and syntax elements required for type 1 are indicated by reference numeral 42 .

7a to 7d are examples of a slice header 30, and only one syntax element of the slice header is necessary for type 0, as indicated by reference numeral 40 in FIG. 7c.

FIG. 8 is an example of a picture structure (SOP) 26 a , and all syntax elements belonging to the SOP are required for type 2 .

FIG. 9 is an example of a motion constraint tile set (MCTS) 26 b , and all syntax elements belonging to the MCTS are type 2 required.

As described above, the merge identifier 6a of Figure 3 uses hash functions at hash components 16a and 16b to generate a merge identifier value. In the case where two or more merge identifier values are generated using a hash function, the hashes are connected in the following manner: the hash function input to the second merge identifier containing additional elements in the hash set relative to the first merge identifier uses the first identifier value (hash result) instead of the individual syntax element values to concatenate the inputs to the hash function.

In addition, the presence of the merge identifier also provides the following guarantee: the syntax elements incorporated into the hash set have the same value in all access units (AUs) of a coded video sequence (CVS) and/or bitstream. In addition, this guarantee has the form of a constraint flag in the profile/level syntax of the parameter set.

Considering another embodiment according to the present invention using FIGS. 10 to 13 , the merging process is described below.

FIG. 10 shows a video merger for providing a merged video stream based on multiple encoded video representations. Video merger 50 includes a receiver 52, a merge method identifier 54, and a merge processor 56, and receiver 52 receives input video stream 12 (12a and 12b shown in FIG. 12). Merged video stream 60 is sent to a decoder. In the case where video merger 50 is included in a decoder, the merged video stream is sent to a user device or any other device to display the merged video stream.

The merging process is driven by the above-mentioned merging identifier and the merging identifier type. The merging process may only need to generate the interleaving of parameter sets plus NAL units, which is the lightest form of the merging associated with the merging identifier type value 0, i.e. the first complexity. Figure 12 shows an example of the first complexity merging method. As shown in Figure 12, parameter sets, i.e., the SPS1 of video stream 12a and the SPS2 of video stream 12b are merged (generating the merged SPS based on SPS1 and SPS2), and the PPS1 of video stream 12a and the PPS of video stream 12b are merged (generating the merged PPS based on PPS1 and PPS2). IDR is optional data, so description is omitted. In addition, the strips 1,1 and 1,2 of video stream 12a and the strips 2,1 and 2,2 of video stream 12b are interleaved, as shown in the merged video stream 60. In Figures 10 and 12, as examples, two video streams 12a and 12b are input. However, more video streams may also be input and merged in the same manner.

The merge process may also include rewriting the slice header in the bitstream during NAL unit interleaving associated with a merge identifier type value of 1 (i.e., second complexity), when needed. Additionally and finally, there are cases where syntax elements in the slice payload require adjustment, which is associated with a merge identifier type value of 2 (i.e., third complexity), and requires entropy decoding and encoding during NAL unit interleaving. The merge identifier and merge identifier type drive the decision to select one of the merge processes to be performed and its details.

The input to the merging process is a list of input sub-bitstreams, which also represents the spatial arrangement. The output of the process is a merged bitstream. Generally, in all bitstream merging processes, a parameter set for a new output bitstream needs to be generated, which can be based on the parameter set of the input sub-bitstream (e.g., the first input sub-bitstream). Necessary updates to the parameter set include picture size. For example, the picture size of the output bitstream is calculated as the sum of the picture sizes of the input bitstreams in each dimension and in the context of their spatial arrangement.

The bitstream merging process requires that all input sub-bitstreams carry the same value of the merge identifier and at least one instance of the merge identifier type. In one embodiment, the merging process is performed associated with the minimum value of the merge identifier type for which all sub-bitstreams carry the same value of the merge identifier.

For example, the difference in the merge identifier value having a certain merge identifier type value is used in the merge process to determine the details of the merge process according to the difference value of the merge identifier type that matches the merge identifier value. For example, as shown in FIG11, when the first merge identifier 70 having a merge identifier type value equal to 0 does not match between the two input sub-bitstreams 70a and 70b, but the second merge identifier 80 having a merge identifier type value equal to 1 matches between the two sub-bitstreams 80a and 80b, the difference in the bit position of the first merge identifier 70 indicates the (slice header related) syntax elements that need to be adjusted in all slices.

FIG13 shows a video merger 50 (50a), which includes a receiver (not shown) and a merger processor 56, and the merger method identifier includes a merger identifier comparator 54a and a coding parameter comparator 54b. In the case where the input video stream 12 includes a hash value as a merger identifier value, the value of each input video stream is compared at the merger identifier comparator 54a. For example, when both input video streams have the same merger identifier value, the respective coding parameters of each input video stream are compared at the coding parameter comparator 54b. The merger method is determined based on the coding parameter comparison result, and the merger processor 56 merges the input video streams 12 by using the determined merger method. In the case where the merger identifier value (hash value) also indicates the merger method, individual coding parameter comparisons are not required.

Three merging methods are explained above - the first complexity merging method, the second complexity merging method and the third complexity merging method. The fourth merging method is to merge video streams using full pixel decoding and pixel re-encoding. If all three merging methods are not applicable, the fourth merging method is applied.

The following describes a process for identifying the merge result level according to another embodiment of the present invention using Figures 14 and 15. The identification of the merge result level means putting information into the sub-bitstream and how much the sub-bitstream contributes to the level restriction of the merged bitstream including the sub-bitstream.

FIG. 14 shows an encoder 2 ( 2 b ) comprising an encoder core including encoding parameter determination 14 , merge identifier provision (not shown) and granularity capability provider 8 .

In general, when sub-bitstreams are to be merged into a joint bitstream, an indication of how much each sub-bitstream contributes to the level-specific restrictions of the codec system that the potential merged bitstream must comply with is critical to ensuring that a legal joint bitstream is created. Although traditionally, the codec level granularity is quite coarse (e.g., distinguishing between primary resolutions such as 720p, 1080p, or 4K), the merged level restriction indication requires finer granularity. The granularity of this traditional level indication is not enough to represent the contribution of each sub-bitstream to the merged bitstream. Given that the number of tiles to be merged is unknown in advance, a reasonable compromise needs to be found between flexibility and bitrate overhead, but typically, it far exceeds the traditional level restriction granularity. An exemplary use case is 360-degree video streaming, where the service provider needs the freedom to choose between different tile structures (e.g., 12, 24, or 96 tiles per 360-degree video), where each tile stream will contribute 1/12, 1/24, or 1/96 of the entire level restriction (e.g., 8K assuming equal rate distribution). Even further, assuming that the rate distribution between tiles is not uniform (e.g., to achieve constant quality across the video plane), an arbitrarily fine granularity may be required.

For example, such signaling would be a signaled ratio and/or percentage of an otherwise signaled level. For example, in a conference scenario with four participants, each participant would send a legitimate level 3 bitstream that also includes an indication, i.e., level information included in the coarse granularity capability information, that the sent bitstream complies with 1/3 and/or 33% of the level 5 limits, i.e., this information may be included in the fine granularity capability information, and a receiver of multiple such streams (i.e., the video merger 50 (50b) shown in FIG. 15) would therefore, for example, know that the three bitstreams can be merged into a single joint bitstream that complies with level 5.

In Figure 15, the video merger is shown as including a receiver 52 and a merging processor 56, and multiple video streams are input into the video merger. The video merger can also be included in a video decoder, or the video decoder is not shown in the figure, but the receiver can work as a decoder. The decoder (not shown) receives a bitstream, which includes multiple sub-bitstreams, multiple coding parameters, coarse granularity capability information and fine granularity capability information. The coarse granularity capability information and the fine granularity capability information are carried in the SEI message, and the decoder parses the received information and interprets the level and score as the level restrictions available to the decoder. Then, the decoder checks whether the bitstream it is encountering actually follows the restrictions represented. The result of the inspection is provided to the video merger or the merging processor. Therefore, as previously mentioned, it is possible that the video merger can know that multiple sub-bitstreams can be merged into a single joint bitstream.

The granularity capability information may have indications of ratios and/or percentages as a vector of values, with each dimension relating to another aspect of the codec level limitations, e.g., maximum allowed number of luma samples per second, maximum picture size, bitrate, buffer fullness, number of tiles, etc. Additionally, the ratios and/or percentages refer to a general codec level for the video bitstream.

[0013] Hereinafter, additional embodiments and aspects of the invention will be described, which may be used alone or in combination with any of the features, functions, and details described herein.

A first aspect relates to a video encoder for providing an encoded video representation, wherein the video encoder is configured to provide a video stream comprising encoding parameter information, encoded video content information, and one or more merge identifiers, wherein the encoding parameter information describes a plurality of encoding parameters, and the one or more merge identifiers indicate whether the encoded video representation can be merged with another encoded video representation and/or how the encoded video representation can be merged with another encoded video representation.

According to a second aspect referring to the first aspect, in the video encoder, which complexity to use is determined based on a parameter value defined by parameter information.

According to the third aspect referring to the first or second aspect, in the video encoder, the merge identifier is a concatenation of a plurality of encoding parameters.

According to the fourth aspect referring to any one of the first to third aspects, in the video encoder, the merge identifier is a hash value of a concatenation of a plurality of encoding parameters.

According to the fifth aspect referring to any one of the first to fourth aspects, in the video encoder, the merge identifier indicates a merge identifier type representing complexity of a merge process.

According to the sixth aspect with reference to the fifth aspect, in the video encoder, the value of the merge identifier type indicates a merge process, wherein the video encoder is configured to switch between at least two of the following values of the merge identifier type: a first value of the merge identifier type, indicating a merge process rewritten by a parameter set; a second value of the merge identifier type, indicating a merge process rewritten by a parameter set and a slice header; and a third value of the merge identifier type, indicating a merge process rewritten by a parameter set, a slice header, and a slice payload.

According to the seventh aspect referring to any one of the first to sixth aspects, in the video encoder, a plurality of merge identifiers are associated with different complexities of the merge process.

According to the eighth aspect with reference to any one of the first to seventh aspects, in the video encoder, the encoder is configured to check whether the encoding parameters evaluated to provide the merge identifier are the same in all units of the video sequence, and to provide the merge identifier based on the check.

According to the ninth aspect with reference to any one of the first to eighth aspects, in the video encoder, the multiple encoding parameters include merge-related parameters, which must be the same in different video streams to allow the merge to have a lower complexity than the merge through full-pixel decoding, and the video encoder is configured to provide the one or more merge identifiers based on the merge-related parameters.

According to the tenth aspect with reference to the ninth aspect, in the video encoder, the merge-related parameters include one or more or all of the following parameters: parameters describing motion constraints at tile boundaries, information about a group of pictures (GOP) structure, parameters describing a chroma coding format and parameters describing a luminance coding format, parameters describing advanced motion vector prediction, parameters describing sample adaptive offset, parameters describing temporal motion vector prediction, and parameters describing a loop filter.

According to the eleventh aspect referring to any one of the first to seventh aspects, in the video encoder, when a merge parameter associated with a second complexity of a merge process is determined based on a second encoding parameter set, a merge identifier associated with a first complexity of the merge process is determined based on a first encoding parameter set, the second complexity is higher than the first complexity, and the second encoding parameter set is a proper subset of the first encoding parameter set.

According to the twelfth aspect with reference to the eleventh aspect, in the video encoder, a merge identifier associated with a third complexity of the merging process is determined based on a third encoding parameter set, wherein the third complexity is higher than the second complexity, and the third encoding parameter set is a proper subset of the second parameter set.

According to the thirteenth aspect referring to the eleventh or twelfth aspect, the video encoder is configured to determine a merge identifier associated with a first complexity of the merging process based on a set of encoding parameters that must be equal in two different video streams to allow merging of the video streams in the following way: only the set of parameters that can be applied to multiple slices is modified while keeping the slice header and the slice payload unchanged.

According to the fourteenth aspect with reference to the eleventh or twelfth aspect, the video encoder is configured to determine a merge identifier associated with the first complexity based on one or more or all of the following parameters:

Parameters indicating motion constraints at tile and picture boundaries,

Parameters that define the GOP structure,

Parameters describing the reference picture set,

Parameters describing the chroma format,

Parameters describing the base quantization parameters and the chrominance quantization parameters,

Parameters describing the bit depth luma/chroma,

parameters describing hypothesized reference decoder parameters, the hypothesized reference decoder parameters comprising parameters relating to an initial arrival delay and parameters relating to an initial removal delay,

Parameters describing the structure of the coding block,

Parameters describing the minimum and/or maximum size of the transform,

Describes the parameters used by the Pulse Code Modulation block,

Describes the parameters of advanced motion vector prediction,

Parameters describing sample adaptive offset,

Parameters describing the temporal motion vector prediction,

Parameters describing intra-frame smoothing,

Parameters describing the relevant strip,

Describes the parameters hidden by the symbol,

Parameters describing the weighted prediction,

Parameters describing the transform quantization bypass,

Parameters describing entropy coding synchronization,

Describe the parameters of the loop filter,

Parameters describing the stripe header value including the parameter set ID,

Parameters describing the slice header value including the reference picture set, and

Describes the parameters transmitted using implicitly encoded transform unit address signals.

According to the fifteenth aspect referring to any one of the eleventh to fourteenth aspects, the video encoder is configured to determine a merge identifier associated with a second complexity of the merging process based on a set of encoding parameters that must be equal in two different video streams to allow merging of the video streams in the following manner: modifying a set of parameters that can be applied to multiple slices and also modifying a slice header while keeping the slice payload unchanged.

According to the 16th aspect referring to any one of the 11th to 15th aspects, the video encoder is configured to determine the merge identifier associated with the second complexity based on one or more or all of the following parameters:

Parameters indicating motion constraints at tile and picture boundaries,

Parameters that define the GOP structure,

Parameters describing the chroma format,

Parameters describing the bit depth luma/chroma,

Parameters describing the structure of the coding block,

Parameters describing the minimum and/or maximum size of the transform,

Describes the parameters used by the Pulse Code Modulation block,

Describes the parameters of advanced motion vector prediction,

Parameters describing sample adaptive offset,

Parameters describing the temporal motion vector prediction,

Parameters describing intra-frame smoothing,

Describes the parameters hidden by the symbol,

Parameters describing the transform quantization bypass,

Parameters describing entropy coding synchronization,

Describe the parameters of the loop filter, and

Parameters describing the slice header values including the reference picture set.

According to the seventeenth aspect referring to any one of the eleventh to sixteenth aspects, the video encoder is configured to determine a merge identifier associated with a third complexity of a merging process based on a set of coding parameters that must be equal in two different video streams to allow merging of the video streams in the following manner: modifying a set of parameters that can be applied to multiple slices and also modifying a slice header and a slice payload, but not performing full pixel decoding and pixel re-encoding.

According to the 18th aspect referring to any one of the 11th to 17th aspects, the video encoder is configured to determine a merge identifier associated with the third complexity based on one or more or all of the following parameters:

Parameters indicating motion constraints at tile and picture boundaries,

Parameters that define the GOP structure,

Parameters describing the chroma format,

Parameters describing the bit depth luma/chroma,

Describes the parameters of advanced motion vector prediction,

Parameters describing sample adaptive offset,

Parameters describing the temporal motion vector prediction, and

Describes the parameters of the loop filter.

According to aspect 19 with reference to any one of aspects 1 to 18, the video encoder is configured to: apply a hash function to a concatenation of a second merge identifier and one or more encoding parameters that are not considered when determining the second merge identifier so as to obtain a first merge identifier associated with a first complexity of the merging process, wherein the second merge identifier is associated with a second complexity of the merging process, and the first complexity is lower than the second complexity.

According to the 20th aspect with reference to any one of the first to nineteenth aspects, the video encoder is configured to: apply a hash function to the concatenation of a third merge identifier and one or more encoding parameters that are not considered when determining the third merge identifier, so as to obtain a second merge identifier associated with a second complexity of the merging process, wherein the third merge identifier is associated with a third complexity of the merging process, and the second complexity is lower than the third complexity.

A twenty-first aspect relates to a video merger for providing a merged video representation based on multiple encoded video representations, wherein the video merger is configured to receive multiple video streams including encoding parameter information, encoded video content information and one or more merge identifiers, the encoding parameter information describing multiple encoding parameters, and the one or more merge identifiers indicating whether the encoded video representation can be merged with another encoded video representation and/or how the encoded video representation can be merged with another encoded video representation; wherein the video merger is configured to decide the usage of the merging method based on the merge identifier.

According to the twenty-second aspect with reference to the twenty-first aspect, the video merger is configured to select a merging method from a plurality of merging methods according to the merging identifier.

According to the twenty-third aspect of reference twenty-second aspect, the video merger is configured to select between at least two of the following merger methods based on one or more merger identifiers: a first merger method, wherein the first merger method is a merger of video streams in the following manner: only a parameter set that can be applied to multiple slices is modified while keeping a slice header and a slice payload unchanged; a second merger method, wherein the second merger method is a merger of video streams in the following manner: a parameter set that can be applied to multiple slices is modified and a slice header is also modified while keeping a slice payload unchanged; and a third merger method, wherein the third merger method is a merger of video streams in the following manner: a parameter set that can be applied to multiple slices is modified and a slice header and a slice payload are also modified, but full pixel decoding and pixel re-encoding are not performed.

According to the twenty-fourth aspect based on the twenty-third aspect, the video merger is configured to: selectively use a fourth merger method based on the one or more merger identifiers, and the fourth merger method is a merger of video streams using full pixel decoding and pixel re-encoding.

According to aspect 25 with reference to any one of aspects 22 to 24, the video merger is configured to compare merger identifiers of two or more video streams associated with the same given merger method to decide whether to use the given merger method to perform the merger based on the result of the comparison.

According to the twenty-sixth aspect of reference twenty-fifth aspect, the video merger is configured to: if the comparison indicates that the merger identifiers of the two or more video streams associated with the given merger method are equal, then use the given merger method to selectively perform the merger.

According to aspect twenty-seven of reference aspect twenty-fifth, the video merger is configured to: if the comparison indicates that the merger identifiers of the two or more video streams associated with the given merger method are different, use a merger method with a higher complexity than the given merger method.

According to aspect twenty-eight based on aspect twenty-seven, the video merger is configured to: selectively compare encoding parameters of the two or more video streams that must be equal to allow the merging of the video streams using the given merging method when the comparison indicates that the merging identifiers of the two or more video streams associated with the given merging method are equal, and wherein the video merger is configured to: selectively perform the merging using the given merging method if the comparison of the encoding parameters indicates that the encoding parameters are equal, and wherein the video merger is configured to: perform the merging using a merging method with a higher complexity than the given merging method if the comparison of the encoding parameters indicates that the encoding parameters include differences.

According to aspect 29 with reference to any one of aspects 21 to 28, the video merger is configured to compare merge identifiers associated with merge methods having different complexities, and wherein the video merger is configured to merge two or more video streams using a merge method having a lowest complexity, the associated merge identifiers for the merge method having the lowest complexity being equal in the two or more video streams to be merged.

According to the 30th aspect with reference to any one of the 22nd to 29th aspects, the video merger is configured to compare merge identifiers associated with merge methods having different complexities, and wherein the video merger is configured to identify the merge method of lowest complexity, the associated merge identifiers for the merge method of lowest complexity being equal in two or more video streams to be merged; and wherein the video merger is configured to compare a set of encoding parameters that must be equal in two or more video streams to be merged to allow merging using the identified merge method, and wherein the video merger is configured to: if the comparison indicates that the encoding parameters of the encoding parameter set associated with the identified merge method are equal in the video streams to be merged, then merge the two or more video streams using the identified merge method.

According to aspect 31 with reference to any one of aspects 22 to 30, the video merger is configured to determine which encoding parameters should be modified during the merger process based on one or more differences between merger identifiers of different video streams to be merged.

According to aspect 32 with reference to aspect 31, the video merger is configured to determine which encoding parameters should be modified in a merging method having a given complexity based on one or more differences between merging identifiers of different video streams to be merged, wherein the merging identifier is associated with a merging method having a complexity lower than the given complexity.

According to aspect 33 with reference to any one of aspects 21 to 32, the video merger is configured to obtain joint coding parameters associated with slices of all video streams to be merged based on coding parameters of the video streams to be merged, and include the joint coding parameters in the merged video stream.

According to the thirty-fourth aspect with reference to the thirty-third aspect, the video merger is configured to adapt coding parameters respectively associated with the individual video slices to obtain modified slices to be included in the merged video stream.

According to the thirty-fifth aspect with reference to the thirty-fourth aspect, in the video merger, the adapted encoding parameters include parameters representing a picture size of the merged encoded video representation, wherein the picture size is calculated based on the picture size of the encoded video representation to be merged.

A thirty-sixth aspect relates to a method for providing an encoded video representation, comprising: providing a video stream including encoding parameter information, encoded video content information and one or more merge identifiers, wherein the encoding parameter information describes multiple encoding parameters, and the one or more merge identifiers indicate whether the encoded video representation can be merged with another encoded video representation and/or how the encoded video representation can be merged with another encoded video representation.

A thirty-seventh aspect relates to a method for providing a merged video representation based on multiple encoded video representations, comprising: receiving multiple video streams including encoding parameter information, encoded video content information and one or more merge identifiers, wherein the encoding parameter information describes multiple encoding parameters, and the one or more merge identifiers indicate whether the encoded video representation can be merged with another encoded video representation and/or how the encoded video representation can be merged with another encoded video representation; and selecting a merging method from multiple merging methods based on the merging identifier.

A thirty-eighth aspect relates to a merging method for merging two or more video streams, comprising: providing common coding parameter information based on coding parameter information of different video streams while keeping the encoded video content information unchanged; selecting a merging process based on the unchanged encoded video content information; and using the selected merging process to merge the two or more video streams.

A thirty-ninth aspect relates to a computer program having a program code for executing any one of the methods according to aspects thirty-six to thirty-eight when the program code is run on a computer.

The fortieth aspect relates to a data stream generated by any one of the methods according to aspects thirty-sixth to thirty-eighth.

Although some aspects have been described in the context of an apparatus or system, it will be clear that these aspects also represent the description of the corresponding method, wherein a block or device corresponds to a method step or a feature of a method step. Similarly, the aspects described in the context of a method step also represent the description of the features of the corresponding block or item or corresponding apparatus and/or system. Some or all of the method steps may be performed by (or using) a hardware device (such as a microprocessor, a programmable computer or an electronic circuit). In some embodiments, one or more of the most important method steps may be performed by such a device.

The data stream of the present invention may be stored on a digital storage medium, or may be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium (eg, the Internet).

Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or in software. The implementation may be performed using a digital storage medium (e.g., a floppy disk, DVD, Blu-Ray, CD, ROM, PROM and EPROM, EEPROM or flash memory) on which electronically readable control signals are stored, which cooperate (or are capable of cooperating) with a programmable computer system to perform the corresponding method. Thus, the digital storage medium may be computer readable.

Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.The program code may, for example, be stored on a machine readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment of the inventive method is therefore a data carrier (or a digital storage medium or a computer-readable medium) on which is recorded the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium is typically tangible and/or non-transitory.

Therefore, another embodiment of the inventive method is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection (eg, via the Internet).

A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

Another embodiment according to the invention comprises an apparatus or system configured to transmit a computer program to a receiver (e.g. electronically or optically), the computer program being used to perform one of the methods described herein. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, comprise a file server for transmitting the computer program to the receiver.

In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. Typically, the method is preferably performed by any hardware device.

The devices described herein may be implemented using hardware devices, or using computers, or using a combination of hardware devices and computers.

The apparatus described herein or any component of an apparatus described herein may be implemented at least partially in hardware and/or software.

The methods described herein may be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

Any component of a method described herein or an apparatus described herein may be performed at least in part by hardware and/or by software.

The above embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and variations of the arrangements and details described herein will be apparent to other persons skilled in the art. Therefore, it is intended that the scope of the present invention be limited only by the scope of the appended patent claims and not by the specific details given by way of the description and explanation of the embodiments herein.

Claims (24)

1. A video encoder for providing an encoded video representation, comprising:

A processor;

A memory storing a computer program that, when executed by the processor, causes the video encoder to:

Providing coarse grain capability requirement information describing: compatibility of video streams with video decoders having a capability level of a plurality of predetermined capability levels, and

Providing fine granularity capability requirement information describing: which score of the allowable capability requirements associated with one of the predetermined capability levels is needed to decode the encoded video representation to enable a video combiner to combine two or more video streams according to the coarse granularity capability requirement information and the fine granularity capability requirement information.

2. The video encoder of claim 1, wherein the computer program, when executed by the processor, further causes the video encoder to: providing the fine granularity capability requirement information such that the fine granularity capability requirement information includes a ratio value or a percentage value.

3. The video encoder of claim 1, wherein the computer program, when executed by the processor, further causes the video encoder to: the fine granularity capability requirement information is provided such that the fine granularity capability requirement information includes a ratio value or a percentage value that references a predetermined capability level described by the coarse granularity capability requirement information.

4. The video encoder of claim 1, wherein the computer program, when executed by the processor, further causes the video encoder to: providing the fine granularity capability requirement information such that the fine granularity capability requirement information includes reference information and score information, and

Wherein the reference information describes which of the predetermined capability levels the score information references.

5. The video encoder of claim 1, wherein the computer program, when executed by the processor, further causes the video encoder to: the fine granularity capability requirement information is provided such that the fine granularity capability requirement information describes capability requirements of the encoded video representation in terms of capability requirements with respect to a plurality of standards.

6. The video encoder of claim 1, wherein the computer program, when executed by the processor, further causes the video encoder to: the fine granularity capability requirement information is provided such that the fine granularity capability requirement information includes a plurality of score values associated with different criteria.

7. The video encoder of claim 1, wherein the computer program, when executed by the processor, further causes the video encoder to: providing the fine granularity capability requirement information such that the fine granularity capability requirement information includes one or more score values describing one or more of the following criteria:

-a fraction of the maximum number of allowed luminance samples per second;

-a fraction of the maximum image size;

-a fraction of the maximum bit rate;

-a fraction of buffer fullness; and

-A fraction of the maximum number of tiles.

8. The video encoder of claim 1, wherein the computer program, when executed by the processor, further causes the video encoder to: providing such resolution to the fine granularity capability requirement information.

9. A video combiner for providing a combined video representation based on a plurality of encoded video representations, the video combiner comprising:

A processor;

a memory storing a computer program that, when executed by the processor, causes the video combiner to:

receiving a plurality of video streams including encoding parameter information describing a plurality of encoding parameters, encoding video content information, coarse granularity capability requirement information describing compatibility of the video streams with a video decoder having a capability level of a plurality of predetermined capability levels, and fine granularity capability requirement information describing which score of an allowable capability requirement associated with one of the predetermined capability levels is required to decode an encoded video representation, and

Two or more video streams are merged according to the coarse granularity capability requirement information and the fine granularity capability requirement information.

10. The video combiner of claim 9, wherein the computer program, when executed by the processor, further causes the video combiner to: which video streams can be included in the combined video stream is determined based on the fine granularity capability requirement information.

11. The video combiner of claim 9, wherein the computer program, when executed by the processor, further causes the video combiner to: it is decided whether or not a valid combined video stream can be obtained by combining two or more video streams according to the fine granularity capability requirement information.

12. The video combiner of claim 9, wherein the computer program, when executed by the processor, further causes the video combiner to: the fine granularity capability requirement information of the multiple video streams to be combined is integrated.

13. The video combiner of claim 9, wherein the fine granularity capability requirement information describes capability requirements of the encoded video representation in terms of capability requirements with respect to a plurality of standards; and

Wherein the computer program, when executed by the processor, further causes the video combiner to: it is determined whether the combined capability requirement of the multiple video streams to be combined as described by the fine granularity capability requirement information is within predetermined limits for all criteria.

14. The video combiner of claim 9, wherein the fine granularity capability requirement information comprises a plurality of score values relating to different criteria.

15. The video combiner of claim 9, wherein the fine granularity capability requirement information describes one or more score values for one or more of the following criteria:

-a fraction of the maximum number of allowed luminance samples per second;

-a fraction of the maximum image size;

-a fraction of the maximum bit rate;

-a fraction of buffer fullness; and

-A fraction of the maximum number of tiles.

16. A video decoder, wherein the video decoder comprises the video combiner of claim 9.

17. A method of providing an encoded video representation, comprising:

Providing coarse grain capability requirement information describing: compatibility of video streams with video decoders having a capability level of a plurality of predetermined capability levels, and

Providing fine granularity capability requirement information describing: which fraction of the allowable capability requirement associated with one of the predetermined capability levels is needed to decode the encoded video representation to enable a video combiner to combine two or more video streams according to the coarse granularity capability requirement information and the fine granularity capability requirement information.

18. A method for providing a combined video representation based on a plurality of encoded video representations, comprising:

Receiving a plurality of video streams comprising encoding parameter information describing a plurality of encoding parameters, encoding video content information, coarse granularity capability requirement information describing compatibility of the video streams with a video decoder having a capability level of a plurality of predetermined capability levels, and fine granularity capability requirement information describing which fraction of an allowable capability requirement associated with one of the predetermined capability levels is required to decode an encoded video representation, wherein two or more video streams are combined according to the coarse granularity capability requirement information and the fine granularity capability requirement information.

19. A computer readable storage medium storing a computer program for performing the method of claim 17 or 18 when run on a computer.

20. A video decoder for decoding a provided video representation, wherein the video decoder comprises:

A processor;

A memory storing a computer program that, when executed by the processor, causes the video decoder to: receiving a video representation comprising a plurality of video streams, the plurality of video streams comprising encoding parameter information, encoding video content information, coarse granularity capability requirement information, and fine granularity capability requirement information, the encoding parameter information describing a plurality of encoding parameters, the coarse granularity capability requirement information describing compatibility of the video streams with a video decoder having a capability level of a plurality of predetermined capability levels, the fine granularity capability requirement information describing which fraction of a permissible capability requirement associated with one of the predetermined capability levels is required to decode the encoded video representation, and

A determination is made as to whether the combined capability requirements described by the fine granularity capability requirement information of the plurality of video streams to be combined match a predetermined limit to be adhered to.

21. The video decoder of claim 20, wherein the computer program, when executed by the processor, further causes the video decoder to: the received coarse granularity capability requirement information and fine granularity capability requirement information are parsed to obtain an indication of the capability level and a fraction of the allowable capability requirement.

22. The video decoder of claim 20, wherein the fine granularity capability requirement information describes capability requirements of the encoded video representation in terms of capability requirements with respect to a plurality of standards; and

Wherein the computer program, when executed by the processor, further causes the video decoder to: when the combined capability requirements of the plurality of video streams to be combined are within predetermined limits with respect to all criteria, it is determined that the combined capability requirements of the plurality of video streams to be combined are matched.

23. The video decoder of claim 20, wherein the fine granularity capability requirement information includes a plurality of score values associated with different criteria.

24. The video decoder of claim 20, wherein the fine granularity capability requirement information includes one or more score values describing one or more of the following criteria:

-a fraction of the maximum number of allowed luminance samples per second;

-a fraction of the maximum image size;

-a fraction of the maximum bit rate;

-a fraction of buffer fullness; and

-A fraction of the maximum number of tiles.