RU2406254C2 - Video processing with scalability - Google Patents

Abstract

FIELD: information technologies.

SUBSTANCE: video data of improvement level is included into component of network abstraction level (NAL); one or several syntactic elements are included into NAL to indicate, whether NAL component includes video data of improvement level, and one or several syntactic elements to indicate at least one of the following: whether video data of improvement level include video data with intraframe coding in NAL component; whether NAL component includes parametre of sequence, set of image parametres, layer of reference image or section of data of reference image layer; and number of nonzero coefficients with value exceeding one, in units of intraframe coding in video data of improvement level.

EFFECT: realisation of expansion of versions of International Telecommunication Union standard for efficient scalable video coding.

49 cl, 16 dwg, 18 tbl

Description

Priority claim under US Code 35 U.S.C. §119

This application claims the priority of provisional application US No. 60/787310, filed March 29, 2006, provisional US application No. 60/789320, filed March 29, 2006, and provisional US application No. 60/833445, filed July 25, 2006, the contents of each of which is fully incorporated by reference.

FIELD OF THE INVENTION

The present invention relates to the processing of digital video data and, in particular, to methods for processing scalable video data.

BACKGROUND

Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital broadcast systems, wireless devices, personal digital assistants (PDAs), laptop computers, desktop computers, video game consoles, digital cameras, and digital recording devices , cellular or satellite radiotelephones, etc. Digital video devices can provide significant advantages over standard analog video systems for processing and transmitting video sequences.

For coding digital video sequences, various coding standards for video data are set. For example, the Cinematography Expert Group (MPEG) has developed several standards, including MPEG-1, MPEG-2, and MPEG-4. Other examples include the International Telecommunication Union (ITU) -T H.263 standard and the ITU-T H.264 standard, as well as their analogue ISO / IEC MPEG-4, Part 10, i.e. Advanced Video Coding (AVC). These video encoding standards support enhanced transmission of video sequences by encoding data in a compressed form.

SUMMARY OF THE INVENTION

In general, video processing methods are described here that involve the use of syntax elements and semantics to support fairly simple extensions for multimedia processing with video scalability. Syntax elements and semantics can be applied to multimedia broadcasting and determining the bitstream format and encoding process, which supports fairly simple video scalability.

The syntax element and semantics can be applied to the components of the network abstraction layer (NAL). According to some aspects, these methods can be applied to implement extensions with fairly simple video scalability for devices that are otherwise compliant with the ITU-T H.264 standard. In some aspects, NAL components can generally conform to the H.264 standard. In particular, NAL components carrying base layer video data may conform to the H.264 standard, while NAL components carrying enhancement layer video data may include one or more added or modified syntax elements.

According to one aspect, the invention provides a method for transmitting scalable digital video data, the method comprising including enhancement level video data in a network abstraction layer (NAL) block and including one or more syntax elements in a NAL component to indicate whether the NAL component includes video level data improvements.

According to another aspect, the invention provides an apparatus for transmitting scalable digital video data, the apparatus comprising a Network Abstraction Layer (NAL) component module that includes encoded video enhancement layer components in a NAL component and includes one or more syntax elements in a NAL component to indicate whether includes NAL component video data enhancement level.

According to yet another aspect, the invention provides a processor for transmitting scalable digital video data, the processor being configured to include enhancement layer video data in a network abstraction layer (NAL) component and include one or more syntax elements in a NAL component to indicate whether the NAL component includes improvement level video data.

According to a further aspect, the invention provides a method for processing scalable digital video data, the method comprising receiving video data of an enhancement level in a network abstraction layer (NAL) component, receiving one or more syntax elements in a NAL component to indicate whether the NAL component includes video level data enhancements and decoding of digital video in the NAL component based on this indication.

According to yet another aspect, the invention provides an apparatus for processing scalable digital video data, the apparatus comprising a network abstraction layer (NAL) component module that receives enhancement level video data in a NAL component and receives one or more syntax elements in a NAL component to indicate whether includes a NAL component video data of the enhancement level, and a decoder that decodes the digital video data in the NAL component based on this indication.

According to a further aspect, the invention provides a processor for processing scalable digital video data, the processor being configured to receive enhancement level video data in a Network Abstraction (NAL) component, receive one or more syntax elements in a NAL component to indicate whether the NAL component includes layer video data enhancements, and decoding of digital video in the NAL component based on this indication.

The methods described in this invention can be implemented in a digital video coding and / or video decoding device with hardware, software, firmware, or any combination thereof. When implemented by software, the software may run on a computer. The software may be initially stored as instructions, program code, or the like. Accordingly, the invention also contemplates the use of a computer program product for encoding a digital video containing computer-readable media, the computer-readable media containing codes causing the computer to perform the methods and functions of the present invention.

Further details of various aspects of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will become apparent from the description and drawings, as well as from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a digital multimedia broadcasting system supporting video scalability.

FIG. 2 is a diagram illustrating video frames at a basic level and an enhancement level of a scalable video bitstream.

FIG. 3 is a block diagram illustrating exemplary components of a broadcast server and subscriber unit in the digital multimedia broadcast system of FIG. one.

FIG. 4 is a block diagram illustrating exemplary components of a video decoder for a subscriber unit.

FIG. 5 is a flowchart illustrating a process for decoding video data of a base layer and an enhancement layer in a scalable video bitstream.

FIG. 6 is a block diagram illustrating the combination of base-level and enhancement coefficients in a video decoder for single-level decoding.

FIG. 7 is a block diagram illustrating a combination of base layer coefficients and enhancement layer in a video decoder.

FIG. 8 is a flowchart illustrating a process for encoding a scalable video bitstream to combine many different exemplary syntax elements to support fairly simple video scalability.

FIG. 9 is a flowchart illustrating a process for decoding a scalable video bitstream to process many different exemplary syntax elements to support fairly simple video scalability.

Figures 10 and 11 are diagrams illustrating a partition of macroblocks (MB) and quarter blocks for spatial brightness prediction modes.

FIG. 12 is a flowchart illustrating a macroblock (MB) decoding process of a base layer and an enhancement layer to create a single MB layer.

FIG. 13 is a diagram illustrating a filtering process for releasing luminance and chrominance.

FIG. 14 is a diagram illustrating rules for describing samples across a horizontal or vertical boundary for 4x4 blocks.

FIG. 15 is a block diagram illustrating an apparatus for transmitting scalable digital video data.

FIG. 16 is a block diagram illustrating an apparatus for decoding scalable digital video data.

DETAILED DESCRIPTION OF THE INVENTION

Scalable video coding can be used to provide scalable signal-to-noise ratio (SNR) in video compression applications. Temporal and spatial scalability is also possible. With SNR scalability, for example, encoded video data includes a base layer and an enhancement layer. The basic level carries the minimum amount of data required for video decoding, and provides a basic level of quality. The enhancement layer carries additional data that improves the quality of the decoded video data.

In general, a baseline can be thought of as a bitstream containing encoded video data that represents the first level of spatio-temporal-SNR scalability defined by this specification. The enhancement layer bitstream can only be decoded together with the base layer, that is, it contains links to decoded base layer video data that is used to create the final decoded video data.

By using hierarchical modulation at the physical layer, the base layer and enhancement layer can be transmitted on the same carrier or subcarriers, but with different transmission characteristics, which leads to different packet error rates (PER). The baseline has a lower PER for more reliable reception over the entire coverage area. The decoder can decode only the base level or the base level plus the enhancement level if the enhancement level is reliably received and / or depending on other criteria.

In general, this invention describes video processing methods that involve the use of syntax elements and semantics to support fairly simple extensions for processing multimedia with video scalability. These methods are particularly suitable for multimedia broadcasting and determine the format of the bitstream and the encoding process, which support a fairly simple video scalability. In a number of aspects, these methods can be applied to implement extensions for fairly simple video scalability for devices that otherwise comply with the H.264 standard. For example, extensions may represent potential modifications to future versions or extensions of the H.264 standard or other standards.

The H.264 standard was developed by the ITU-T Video Coding Expert Group and the ISO / IEC Cinematography Expert Group (MPEG) as a partnership product known as the Joint Video Development Group (JVT). The H.264 standard is described in ITU-T Recommendation H.264, Advanced video coding for generic audiovisual services, by the ITU-T Study Group and dated March 2005, where this standard is called the H.264 standard or the H.264 specification. either a standard or an H.264 / AVC specification.

The methods described in this invention involve the use of syntactic elements and semantics of the enhancement level, designed to implement effective processing of basic level video data and the enhancement level by a video decoder. Various syntactic elements and semantics that can be used together or separately on a selective basis will be described here. Rather simple video scalability provides two levels of spatio-temporal-SNR scalability by dividing the bitstream into two types of syntax objects, designated as the base level and the enhancement level.

Encoded video data and scalable extensions are carried in the Network Abstraction Layer (NAL) components. Each NAL component is a network transmission component, which can take the form of a packet containing an integer number of bytes. NAL components carry either baseline data or enhancement layer data. According to some aspects of the invention, some of the NAL components may actually comply with the H.264 / AVC standard. However, various principles of the present invention can be applied to other types of NAL components. In general, the first byte of the NAL component includes a header indicating the data type in the NAL component. The rest of the NAL component carries payload data corresponding to the type indicated in the header. The nal_unit_type header has a five-bit value that indicates one of thirty-two different types of NAL components, with nine values reserved for future use. Four of the nine reserved NAL component types are reserved for scalability. The specialized nal_unit_type header can be used to indicate that the NAL component is a specialized NAL component, which may include enhancement level video data for use in scalable applications.

The syntax and semantics of the base layer bitstream in the NAL component can generally conform to the applicable standard, for example, the H.264 standard, possibly with some limitations. As examples of restrictions, image parameter sets may have MbaffFRameFlag flag equal to 0, sequence parameter sets may have frame_mbs_only_flag flag equal to 1, and image flag B may be 0. The syntax and semantics of the enhancement level bitstream for NAL components are defined in this invention For effective support, simple enough extensions for video scalability are enough. For example, the semantics of network abstraction layer (NAL) components that carry enhancement layer data can be modified with respect to the H.264 standard to introduce new types of NAL components that specify the data structure type of the raw bit serial payload (RBSP) in the NAL component level of improvement.

Enhancement level NAL components can carry syntax elements with many different enhancement level indications to assist the video decoder in processing the NAL component. Various indications may include an indication of whether the NAL component includes enhancement layer video data with intra-frame coding at the enhancement layer, an indication of whether the decoder should use the addition of enhancement layer video data to the base layer data in the pixel domain or in the transform domain, and / or an indication of whether the video data of the enhancement layer includes any residual data related to the video data of the base layer.

Enhancement level NAL components may also carry syntax elements indicating whether the NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of the reference image layer. Other syntax elements may identify blocks in the video of the enhancement layer containing non-zero values of the transform coefficients, indicate the number of non-zero coefficients with a magnitude greater than one in the blocks with intraframe coding in the video data of the enhancement level, and indicate coded block patterns for blocks with intraframe coding in the video data of the enhancement level . The above information may be useful in maintaining efficient and orderly decoding.

The methods described in this invention can be used in conjunction with any of a variety of different predictive video coding standards, such as MPEG-1, MPEG-2 or MPEG-4, ITU H.263 or H.264, or MPEG ISO / IEC -4, part 10, that is, advanced video coding (AVC), which is essentially identical to the H.264 standard. For illustrative purposes, the following describes the use of these methods to support fairly simple extensions for video scaling associated with the H.264 standard. Accordingly, this invention, in particular, involves the adaptation, expansion or modification of the H.264 standard, as described here, to provide sufficiently simple video scalability, but can also be applied to other standards.

According to some aspects, the present invention contemplates its applicability to video coding in the enhanced H.264 standard for providing real-time video services in terrestrial mobile multimedia broadcast (TM3) systems using the FLO radio interface specification (forward only communication), “Forward Link Only Air Interface Specification for Terrestrial Mobile Multimedia Multicast ”, published as TIA-1099 (“ FLO specification ”). The FLO specification includes examples defining the syntax and semantics of bitstreams, as well as decoding processes suitable for providing services over the FLO radio interface.

As mentioned above, scalable video coding has two levels: a basic level and an improvement level. According to some aspects, a plurality of improvement levels can be provided that provide progressively increasing quality levels, i.e., scalability of the signal-to-noise ratio. However, a single level of improvement is described herein for illustrative purposes. Based on the use of hierarchical modulation at the physical layer, the base layer and one or more enhancement layers can be transmitted on the same carrier or subcarriers, but with different transmission characteristics, which leads to different packet error rates (PER). The base layer has a lower PER frequency. Then, the decoder can decode only the base level or the base level plus the enhancement level, depending on their availability and / or other criteria.

If decoding is performed on a client device, such as a mobile phone or other small portable device, restrictions may occur due to computational complexity and memory requirements. Accordingly, scalable coding can be designed such that decoding of the base layer plus the enhancement layer does not significantly increase the computational complexity and memory requirements compared to single-level coding. Effective decoding of basic and enhancement data can be supported by appropriate syntax elements and associated semantics.

As an example of a possible hardware implementation, a subscriber device may comprise a hardware core with three modules: a motion estimation module for processing motion compensation, a transform module for processing dequantization and inverse transform operations, and a release module for processing the release of decoded video data. Each module can be configured to simultaneously process one macroblock (MB). However, access to the subsections of each module can be difficult.

For example, the inverse luminance conversion between MBs may be based on 4 × 4 blocks, and in the conversion module it may be necessary to perform 16 transformations in series for all 4 × 4 blocks. In addition, to speed up the decoding process, you can use the pipelined mode of operation of the three modules. Therefore, interrupts to enable processes for scalable decoding can slow down the progress of decoding.

In a scalable coding scheme according to one aspect of the present invention, in a decoder, data from a base layer and an enhancement layer can be combined at the same level, for example, in a general purpose microprocessor. In this case, the incoming data coming from the microprocessor looks like a single level of data and can be processed by the hardware core as one level. Thus, according to some aspects of the invention, scalable decoding is transparent to the hardware core. It may not be necessary to reorder the hardware kernel modules. Decoding the data of the basic level and the level of improvement as data of one level according to some aspects may slightly increase the complexity of decoding and slightly increase (or not increase at all) the memory requirements.

If the improvement level is lost due to the high PER frequency or for some other reason, only the basic level data remains. Therefore, it will be possible to perform standard single-level decoding for data of the basic level, either small changes to standard, non-scalable decoding may be required, or changes will not be required at all. However, if there is both a basic level and a data enhancement level, the decoder can decode both levels and create video with the quality of the enhancement level, increasing the signal-to-noise ratio of the resulting video data for presentation on the display device.

In this invention, a decoding procedure is described for the case where both the base layer and the enhancement layer are received and are available. However, it should be apparent to those skilled in the art that the decoding procedure described herein is also applicable to single-level decoding of only the base layer. Also, for scalable decoding and standard, single-level (base) decoding, the same hardware core can be shared. In addition, minor scheduling (or not at all) may be required to control scheduling in the hardware core to process both decoding of the base layer only and decoding of the base layer plus the enhancement layer.

Some of the tasks related to scalable decoding can be performed on a general purpose microprocessor. Their implementation may include two-level entropy decoding, combining the coefficients of two levels and supplying control information to a digital signal processor (DSP). The control information supplied to the DSP may include quantization parameter (QP) values and several non-zero coefficients in each 4x4 block. QP values can be sent to the DSP for dequantization and can also be used in conjunction with information about nonzero coefficients in the hardware core for release. The DSP can access blocks in the hardware core to complete other operations. However, the methods described herein need not be limited to any particular hardware implementation or architecture.

In this description, bidirectional predictive B-frames generated by the forward / backward prediction method can be encoded in a standard way under the assumption that B-frames can be carried at both levels. In this description, the main attention is paid to the processing of I- and P-frames and / or layers, which can appear at a basic level, level of improvement, or at both levels. In the general case, a single-level decoding process is described which combines operations for bit streams of the base layer and the enhancement layer in order to minimize decoding complexity and power consumption.

As an example, to combine the base level and the improvement level, the coefficients of the base level can be reduced to the SNR scale of the improvement level. For example, baseline coefficients can simply be multiplied by a scale factor. If the difference in the quantization parameter (QP) between the base level and the enhancement level is a multiple of, for example, 6, then the base level coefficients can be scaled to the enhancement level using a simple bit shift operation. The result will be a scaled version of the baseline data that can be combined with the enhancement layer data to enable single-level decoding of both the base layer and the enhancement layer on a combined basis, as if they were at the common bitstream level.

Using the decoding of one level, rather than two different levels on an independent basis, it is possible to simplify the components of the decoder necessary for processing, to alleviate planning restrictions and reduce power consumption. To provide simplified scalability, the NAL components of the enhancement level bitstream include various syntax elements and semantics designed to facilitate decoding so that the video decoder can respond to the presence of both base layer data and extension layer data in various NAL components. Examples of syntax elements, semantics, and processing functions are described below with reference to the drawings.

In FIG. 1 is a block diagram illustrating a digital multimedia broadcasting system 10 supporting video scalability. In the example of FIG. 1, system 10 includes a broadcast server 12, a transmitting tower 14, and a plurality of subscriber units 16A, 16B. Broadcast server 12 receives digital multimedia content from one or more sources and encodes this multimedia content, for example, in accordance with any of the above video encoding standards, for example, H.264. The multimedia content encoded by the broadcast server 12 can be arranged as separate bitstreams to support various channels selected by the user associated with the subscriber device 16. The broadcast server 12 can receive digital multimedia content live or in the form of archived multimedia content from various providers content.

Broadcast server 12 may include or be coupled to a modulator / transmitter that includes appropriate components of radio frequency (RF) modulation, filtering, and amplification to drive one or more antennas associated with transmit tower 14 to deliver encoded multimedia content received from the broadcast server 12, through a wireless channel. According to some aspects, the broadcast server 12 may be generally configured to provide real-time video services in terrestrial mobile multimedia broadcast (TM3) systems according to the FLO specification. The modulator / transmitter can transmit multimedia data in accordance with any of a variety of different wireless communication methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal frequency division multiplexing (OFDM) or any combination of these methods.

Each subscriber device 16 may be in any device capable of decoding and presenting digital multimedia data: a direct digital broadcasting system, a wireless communication device such as a cellular or satellite radiotelephone, personal digital assistant (PDA), a laptop computer, a desktop computer, video game console or the like Subscriber devices 16 may support wired and / or wireless reception of multimedia data. In addition, some subscriber units 16 may be equipped with means for encoding and transmitting multimedia data, as well as supporting voice and data applications, including video telephony, video streaming, and the like.

To support scalable video, the broadcast server 12 encodes the source video data to create separate base-level bitstreams and enhancement levels for multiple video channels. Transmission on these channels is usually performed simultaneously, so that the subscriber unit 16A, 16B can at any time choose another channel for viewing.

Therefore, the user device 16A, 16B under the control of the user can select one channel for watching sports programs, and then select another channel for watching news or some other scheduled program, just like when watching TV. In general, each channel includes a base layer and an enhancement layer that are transmitted with different PER frequencies.

In the example of FIG. 1, two subscriber units 16A, 16B are shown. However, the system 10 may include any number of subscriber units 16A, 16B in a given coverage area. It is noteworthy that many subscriber devices 16A, 16B can access the same channels to simultaneously view the same content. In FIG. 1 shows the location of subscriber units 16A and 16B with respect to the transmitting tower 14, in which one subscriber unit 16A is closer to the transmitting tower and the other subscriber unit 16B is further from the transmitting tower. Since the base layer is encoded at a low PER frequency, it must be reliably received and decoded by any subscriber unit 16 in the coverage area used. As shown in FIG. 1, the base level is received by both subscriber units 16A, 16B. However, the subscriber 16B is further from the transmitting tower 14 and does not accept the level of improvement with sufficient reliability.

Closer subscriber device 16A is capable of providing higher quality video, since both basic level and enhancement level data are available to it, while subscriber device 16B is capable of presenting video with only the minimum quality level provided by the basic level data. Therefore, the video received by the subscriber units 16 is scalable in the sense that a pre-decoded enhancement level can be added to the base level to increase the signal-to-noise ratio of the decoded video data. However, scalability is only possible when there is improvement level data. As described below, with the enhancement layer data, the syntax elements and semantics associated with the enhancement layer NAL components help the video decoder in the subscriber unit 16 realize scalability of the video data. In this description, and in particular, in the drawings, the term "improvement" for short may be shortened to the abbreviation "exp" (or "ENH").

In FIG. 2 is a diagram illustrating video frames in the base layer 17 and the scalable bitstream video enhancement layer 18. Base layer 17 is a bitstream containing encoded video data that represents a first level of spatio-temporal-SNR scalability. Improvement layer 18 is a bitstream containing encoded video data that represents a second level of spatio-temporal-SNR scalability. In the general case, the bitstream of the enhancement layer can only be decoded together with the base layer, and not independently. Improvement level 18 contains links to decoded video data in the base layer 17. These links can be used either in the transform domain or in the pixel domain to create final decoded video data.

Base level 17 and enhancement level 18 may comprise I-frames (with intra-frame coding), P-frames (with predictive coding) and B-frames (generated by the forward / backward prediction method). P-frames in enhancement level 18 rely on references to P-frames in base level 17. By decoding frames in enhancement level 18 and base level 17, a video decoder is able to improve the quality of decoded video. For example, base layer 17 may include video encoded at a minimum frequency of 15 frames per second, while enhancement level 18 may include video encoded at a higher frequency of 30 frames per second. To support encoding with different quality levels, the base level 17 and enhancement level 18 can be encoded with a large quantization parameter (QP) and a lower QP, respectively.

In FIG. 3 is a block diagram illustrating exemplary components of a broadcast server 12 and a subscriber device 16 in the digital multimedia broadcasting system 10 of FIG. 1. As shown in FIG. 3, the broadcast server 12 includes one or more video sources 20 or an interface for various video sources. Broadcast server 12 also includes video decoder 22, NAL component module 23, and modulator / transmitter 24. Subscriber device 16 includes receiver / demodulator 26, NAL component module 27, video decoder 28, and video display device 30. The receiver / demodulator 26 receives video data from the modulator / transmitter 24 through the communication channel 15. Video decoder 22 includes a base layer encoder module 32 and an enhancement level encoder module 34. Video decoder 28 includes a level combiner (base / enhancement) module 38 and a base / enhancement entropy decoder 40.

The base layer encoder 32 and the enhancement layer encoder 34 receive common video data. A base layer encoder 32 encodes video data with a first quality level. Enhancement level encoder 34 encodes refinements that, when added to the base level, enhance the video to a second, higher quality level. The NAL component module 23 processes the encoded bitstream from video encoder 22 and creates NAL components containing encoded video data from a base layer and an enhancement layer. The NAL component module 23 may be a separate unit, as shown in FIG. 3, or be integrated or otherwise integrated with the video encoder 22. Some NAL components carry basic layer data, while other NAL components carry enhancement layer data. According to the present invention, at least some of the NAL components include syntax elements and semantics that assist video decoder 28 in decoding the base layer and enhancement layer data without significantly increasing complexity. For example, in a NAL component that includes video data of the enhancement layer, a NAL component that includes video data of the base layer, or both, one or more syntax elements indicating the presence of video data of the enhancement layer in the NAL component may be provided.

Modulator / transmitter 24 includes suitable components for modulation / demodulation, amplification, filtering, and frequency conversion to support modulation and wireless transmission of NAL components created by NAL component module 23. The receiver / demodulator 26 includes suitable modulation / demodulation, amplification, filtering and frequency conversion components to support the wireless reception of NAL components transmitted by the broadcast server. According to some aspects, the broadcast server 12 and the subscriber device 16 may be equipped with two-way communication, so that the broadcast server 12, the subscriber device 16, or both of them include transmitting and receiving components, and both are capable of encoding and decoding video data. According to other aspects, the broadcast server 12 may be a subscriber device 16 equipped with encoding, decoding, transmission and reception of video data using encoding of the base level and the level of improvement. Thus, scalable video processing for video data transmitted between two or more subscriber devices is also contemplated herein.

The NAL component module 27 extracts the syntax elements from the received NAL components and supplies the corresponding information to the video decoder 28 for use in decoding the video data of the base layer and the enhancement layer. The NAL component module 27 may be a separate unit, as shown in FIG. 3, or be integrated or otherwise integrated with video decoder 28. Entropy decoder 40 of the base level / level of improvement applies entropy decoding to the received video data. If there is improvement level data, then the level combiner (base / enhancement) module 38 combines the coefficients from the base level and the enhancement level using the guidelines provided by the NAL component module 27 to support single-level decoding of the combined information. Video decoder 28 decodes the combined video data to create output video data driving the display device 30. The syntax elements found in each NAL component and the semantics of the syntax elements control the video decoder 28 when combining and decoding the received video data of the base level and the enhancement level.

The various components in the broadcast server 12 and the subscriber unit 16 can be implemented using any suitable combination of hardware, software, and firmware. For example, video encoder 22 and NAL component module 23, as well as NAL component module 27 and video decoder 28 can be implemented using one or more general purpose microprocessors, digital signal processors (DSPs), hardware cores, application specific integrated circuits (ASICs), gateways user programmable matrices (FPGAs) or any combination thereof. In addition, various components can be implemented in a video encoder-decoder (CODEC). In some cases, a number of aspects of the methods disclosed herein may be performed by a DSP processor that activates various hardware components in the hardware core to speed up the encoding process.

With regard to aspects whose functionality is implemented by software, for example, functionality implemented by a processor or DSP, the invention also contemplates the possibility of using computer-readable media containing codes as part of a computer program product. When executed on a machine, codes cause the machine to perform one or more aspects of the methods described herein. Machine-readable media can contain random-access memory (RAM), for example, random-access dynamic synchronous memory (SDRAM), read-only memory (ROM), non-volatile random-access memory (NVRAM), electrically erasable programmable read-only memory ( EEPROM), flash memory, etc.

In FIG. 4 is a block diagram illustrating exemplary components of video decoder 28 for subscriber unit 16. In the example of FIG. 4, as in FIG. 3, video decoder 28 includes a base layer / enhancement entropy decoder module 40 and a layer combiner (base / extension) module 38. In FIG. 4 also shows a base level error recovery module 44 plus an enhancement layer and a dequantization module 46, as well as a dequantization and prediction module 48. In FIG. 4 also shows a post-processing module 50 that receives the output of the video decoder 28 and the display device 30.

The entropy base level / enhancement level decoder 40 applies entropy decoding to the video data received by the video decoder 28. The level combiner (base / extension) module 38 combines the base level and enhancement level video data for a given frame or macroblock when there is enhancement level data, that is, when the level data The improvements have been successfully adopted. As will be described below, the level combiner (base / extension) module 38 may first, based on the syntax elements present in the NAL component, determine whether the NAL component contains enhancement level data. If it contains, then combiner module 38 combines the base layer data for the corresponding frame with the enhancement layer data, for example, by scaling the base layer data. Thus, combiner module 38 creates a single-level bitstream that can be decoded by video decoder 28 without processing multiple layers. Other syntax elements and their associated semantics in the NAL component can specify how to combine and decode data at the base level and enhancement level.

Error recovery module 44 corrects errors in the decoded output of combiner module 38. The de-quantization module 46 and the inverse transform module 48 apply the functions of de-quantization and inverse transform, respectively, to the output of the error recovery module 44, creating decoded video output for the post-processing module 55. The post-processing module 50 may perform any of a variety of different video enhancement functions, such as deblocking, suppressing reverb, smoothing, sharpening, or the like. If there is improvement level data for a frame or macroblock, video decoder 28 has the ability to create higher quality video for submission to post-processing module 50 and display device 30. If there is no improvement level data, then the decoded video is created with the minimum quality level provided by the basic level.

In FIG. 5 is a flowchart illustrating a process for decoding video data of a base layer and an enhancement layer in a scalable video bitstream. In the general case, if the improvement level is lost due to the high frequency of packet errors or due to lack of reception, only the basic level data remains available. Therefore, standard single-level decoding will be performed. However, if there is data of both levels: basic and improvement, video decoder 28 will decode both levels and create video with the quality of the improvement level. As shown in FIG. 5, after decoding of a group of pictures (GOP) has begun (54), the NAL component module 27 determines whether the incoming NAL components include enhancement layer data or only base layer data (58). If the NAL components include only basic layer data, then video decoder 28 applies standard single-level decoding to the basic layer data (60) and continues to the end of the GOP group (62).

If the NAL components include not only base layer data (58), that is, some of the NAL components include enhancement layer data, then video decoder 28 performs base layer I-decoding (64) and enhancement layer I-decoding (ENH) ( 66). In particular, video decoder 28 decodes all I-frames in a base layer and an enhancement layer. Video decoder 28 moves the contents of the memory (68) to control the decoding of I-frames for both the base level and the enhancement level. In essence, the base layer and the enhancement layer provide two I-frames for a single I-frame, that is, the Ie frame of the enhancement layer and the Ib frame of the base layer. For this reason, moving the contents of memory can be used.

To decode the I-frame in the presence of data from both levels, two-pass decoding can be implemented, which is generally performed as follows. First, the basic layer frame Ib is restored as a normal I-frame. The I-frame of the enhancement layer is then restored as a P-frame. The reference frame for the reconstructed P-frame of the enhancement layer is the reconstructed I-frame of the base layer. In the resulting P-frame, all motion vectors are equal to zero. Accordingly, the decoder 28 decodes the reconstructed frame as a P-frame with zero motion vectors, providing transparent scalability.

Compared to single-level decoding, the decoding of an enhancement level Ie frame is generally equivalent to the decoding time of a standard I frame and a P frame. If the frequency of I-frames does not exceed one frame per second, then significant complication is not required. If this frequency is more than one I-frame per second, for example, due to a scene change or for some other reason, then a coding algorithm is configured to encode the above-mentioned I-frames only at a basic level. If simultaneous existence of both Ib and Ie frames in the decoder is possible, then Ie can be stored in the frame buffer with a difference from Ib. In this case, when frame Ie is restored as the P-frame, the memory indices can be moved and the memory occupied by Ib can be freed. Then, the decoder 28 performs the processing of moving the memory indices depending on whether there is an enhancement level bitstream. If the amount of memory for this is too small, then it is possible to overwrite Ie on Ib, since all motion vectors are zero.

After decoding the I-frames (64, 66) and moving the contents of the memory (68), the combiner module 38 combines the data of the P-frames of the base level and the enhancement level into a single level (70). Then, the de-quantization module 46 and the inverse transform module 48 decode a single P-frame level (72). In addition, the de-quantization module 46 and the inverse transform module 48 decode the B-frames (74).

After decoding the P-frame data (72) and the B-frame data (74), the process ends (62) if the entire GOP group (76) has been processed. If the GOP group has not yet been decoded, then the process continues by performing another iteration of combining the data of the base-level P-frame and the enhancement level (70), decoding the resulting single-level P-frame data (72), and decoding the B-frames (74) . This process continues until the end of the GOP group (76) is reached, which is the moment the process ends.

In FIG. 6 is a block diagram illustrating the combination of base level coefficients and enhancement level in video decoder 28. As shown in FIG. 6, the coefficients of the P-frame of the base layer are subjected to de-quantization 80 and inverse transform 82, for example, using inverse transform module 46 and inverse transform and prediction module 48, respectively (Fig. 4), and then are summed by adder 84 with residual data from buffer 86, representing the reference frame, to create a decoded output of the P-frame of the basic level. However, if there is improvement level data, the base level coefficients are scaled (88) to match the quality of the improvement level coefficients.

Then, adder 90 sums the scalable base level coefficients and enhancement level coefficients for a given frame to create combined base level / enhancement level data. This combined data is de-quantized 92 and inversely converted 94, and then summed by the adder 96 with the residual data from the buffer 98. The output is the combined decoded data of the base level and level of improvement, giving an increased level of quality relative to the base level, but may be required only single-level processing.

In general, in the buffers 86 and 98 of the base level and the enhancement level, the reconstructed reference video data specified by the configuration files in order to compensate for movement can be stored. When receiving bitstreams of both the base level and the enhancement level, simply scaling the base level discrete cosine transform (DCT) coefficients and summing them with the enhancement level DCT coefficients can support single-level coding, in which only a single dequantization and reverse DCT operation is performed for two data levels .

In some aspects, scaling of the base layer data may be accompanied by a simple bit shift operation. For example, if the quantization parameter (QP) of the base level is six levels higher than the parameter QP of the enhancement level, that is, if QP _b -QP _e = 6, then the combined data of the base level and enhancement level can be expressed as follows:

where C _enh 'represents the combined coefficient after scaling the C _base coefficient of the base level and adding it to the original coefficient C _{enh of} the enhancement level, and Q _e ^-1 represents the dequantization operation applied to the enhancement level.

In FIG. 7 is a flowchart illustrating a combination of base layer coefficients and enhancement layer in a video decoder. As shown in FIG. 7, the NAL component module 27 determines when the subscriber unit 16 receives the video data of both levels: the base level and the enhancement level (100), for example, by accessing the syntax elements of the NAL component indicating the extension type of the NAL component. If base level and enhancement level video data is received, then the NAL component module 27 also checks one or more additional syntax elements in the given NAL component to determine whether each base macroblock (MB) has non-zero coefficients (102). If so (YES branch from step 102), then combiner 28 converts the improvement level coefficients into the sum of the existing improvement level coefficients for the corresponding combined MB plus zoomed-in base level coefficients for the combined MB (104).

In this case, the coefficients for the de-quantization module 46 and the inverse transform module 48 are the sum of the scaled base-level coefficients and the enhancement-level coefficients, which is expressed as COEFF = SCALEDBASE_COEFF + ENH_COEFF (104). Thus, combiner 38 combines the enhancement level and the base level data into a single level for the dequantization module 46 and the video decoder 28 inverse transform module 48. If the base level MB combined with the enhancement level does not have nonzero coefficients (branch NO from step 102), then then the coefficients of the improvement level are not cumulative with the coefficients of the base level. Instead, the coefficients for the de-quantization module 46 and the inverse transform module 48 are coefficients of the enhancement level, which can be represented as COEFF = ENH_COEFF (108). The dequantization module 46 and the inverse transform module 48 decode the MB (106) using either the enhancement level coefficients (108) or the combined base level and enhancement level coefficients (104).

In FIG. 8 is a flowchart illustrating a process for encoding a scalable video data bitstream to include a plurality of exemplary syntax elements to support fairly simple video scalability. These various syntax elements can be inserted into NAL components carrying video data of the enhancement layer to identify the type of data carried in the NAL component and exchange information to assist in decoding video data of the enhancement layer. In general, syntax elements with appropriate semantics can be created by the NAL component module 23 and inserted into the NAL components before being transmitted from the broadcast server 12 to the subscriber 16. As one example, the NAL component module 23 can set the parameter type of the NAL component, for example, nal_unit_type in the component NAL equal to the selected value (for example, 30) to indicate that the NAL component is a specialized NAL component, which may include video data of the enhancement level. Other syntax elements and associated values, as described herein, may be generated by the NAL component module 23 to facilitate processing and decoding of the enhancement layer video data carried in the various NAL components. One or more syntax elements may be included in the first NAL component, including base layer video data, the second NAL component, including enhancement level video, or both, to indicate the presence of enhancement level video data in the second NAL component.

Syntax elements and semantics are described in more detail below. In FIG. Figure 8 shows the process in relation to the transmission of video data as a basic level, and the level of improvement. In most cases, both types of video data will be transmitted: the base level and the level of improvement. However, some subscriber units 16 will only receive NAL components that carry basic level video data due to distance to the transmitting tower 14, interference, or other factors. However, from the point of view of the broadcast server 12, the base layer video data and the enhancement layer video data are sent regardless of the ability of some subscriber units 16 to receive both levels.

As shown in FIG. 8, encoded base layer video data and encoded enhancement layer video data from the base layer encoder 32 and enhancement layer encoder 34, respectively, are received by the NAL component module 23 and inserted into the respective NAL components as a payload. In particular, the NAL component module 23 inserts the base layer encoded video data into the first NAL (110) component and inserts the enhancement layer encoded video data into the second NAL component (112). Supporting video decoder 28, the NAL component module 23 inserts into the first NAL component a value indicating that the NAL component type for the first NAL component is an RBSP payload containing base layer video data (114). In addition, the NAL component module 23 inserts into the second NAL component a value indicating that the type of the extended NAL component for the second NAL component is RBSP containing video data of the enhancement level (116). These values can be associated with specific syntax elements. In this way, the NAL component module 27 in the subscriber unit 16 can distinguish between NAL components containing the base layer video data and the enhancement layer video data and detect when the video decoder 28 should initiate the processing of scalable video data. The base layer bitstream can strictly correspond to the exact H.264 format, while the enhancement layer bitstream can include an extended syntax element of the bitstream, for example, “extended_nal_unit_type” in the header of the NAL component. From the point of view of video decoder 28, the syntax element in the header of the NAL component, for example, “extension flag” indicates the bit stream of the enhancement layer and starts the corresponding processing by the video decoder.

If the enhancement layer data includes (I) intra-frame encoded data (118), then the NAL component module 23 inserts a syntax element value into the second NAL component to indicate the presence of internal data (120) in the enhancement layer data. In this way, the NAL component module 27 may send information to the video decoder 28 indicating that internal (I) processing of the video data of the enhancement layer in the second NAL component is necessary, assuming that the second NAL component is securely received by the subscriber unit 16. In any case, does the layer improving the data with intra-frame encoding or not (118), the NAL component module 23 also inserts the value of the syntax element into the second NAL component to indicate whether to add the base layer video data to video data of the enhancement level in the pixel region or the transform region (122), depending on which region is specified by the enhancement level encoder 34.

If residual data (124) is present in the enhancement layer, then the NAL component module 23 inserts a value into the second NAL component to indicate the presence of residual information in the enhancement layer (126). In any case, whether residual data is present or not, the NAL component module 23 also inserts a value into the second NAL component to indicate the size of the set of parameters carried in the second NAL component (128). As further shown in FIG. 8, the NAL component module 23 also inserts a value into the second NAL component, that is, the NAL component carrying enhancement level video data to identify blocks with intraframe coding, for example, macroblocks (MB) having non-zero coefficients greater than one (130).

In addition, the NAL component module 23 inserts a value into the second NAL component to indicate coded block patterns (CBP) for blocks with intraframe coding in the enhancement layer video data carried by the second NAL component (132). The identification of blocks with intraframe coding with nonzero coefficients greater than one and the identification of CBP patterns for block patterns with interframe coding helps the video decoder 28 in the subscriber unit 16 when performing scalable video decoding. In particular, the NAL component module 27 detects various syntax elements and issues instructions to the entropy decoder 40 and combiner 38 for efficiently processing the base level and enhancement video data in order to decode them.

For example, the presence of enhancement level data in a NAL component may be indicated by the nal_unit_type syntax element, which indicates the specialized NAL component for which a specific decoding process is specified. A nal_unit_type value in an undefined range of the H.264 standard, for example, a value of 30, can be used to indicate that the NAL component is a specialized NAL component. The syntax element “extension_flag” in the header of the NAL component indicates that the specialized NAL component includes the enhanced RBSP of the NAL component. Therefore, nal_unit_type and extension_flag together can indicate whether the NAL component includes enhancement level data. The syntax element “extended_nal_unit_type” indicates the particular type of enhancement layer data contained in the NAL component.

An indication of whether video decoder 28 should be used in the pixel or transform region can be shown by the syntax element “decoding_mode_flag” in the header of the enhancement layer “enh_slice_header”. An indication of whether there is intra-frame encoded data at the enhancement level may be provided by the syntax element refine_ intra_mb_flag. Indication of blocks with intraframe coding having nonzero coefficients and an internal CBP pattern can be shown with syntax elements, such as “enh_intra16 × 16_macroblock_cbp ()” for MB 16 × 16 with intraframe coding at the macroblock level of the enhancement level (enh_macroblock_layer) and “coded_block_pattern for the mode with intraframe coding of 4x4 blocks in enh_macroblock_layer. The intraframe-encoded CBP template can be specified with the syntax element “enh_coded_block_pattern” in enh_macroblock_layer. The specific names of the syntax elements, although they were given for illustrative purposes, are subject to change. Accordingly, these names should not be construed as limiting the functions and indications associated with the syntactic elements mentioned.

In FIG. 9 is a flowchart illustrating a process for decoding a scalable video bitstream to process a plurality of different exemplary syntax elements to support fairly simple video scalability. The decoding process shown in FIG. 9 is generally the reverse of the coding process shown in FIG. 8, in the sense that it highlights the processing of various syntax elements in the resulting enhancement level NAL component. As shown in FIG. 9, after receiving the NAL component by the receiver / demodulator 26 (134), the NAL component module 27 determines whether the NAL component includes a syntax element value indicating that the NAL component contains video data of the enhancement level (136). If not, then decoder 28 uses only basic level video processing (138). However, if the type of the NAL component indicates the data of the enhancement layer, then the NAL component module 27 analyzes the given NAL component to find other syntax elements related to the video data of the enhancement layer. Additional syntax elements help the decoder 28 to provide efficient and orderly decoding of video data as a basic level, and the level of improvement.

For example, the NAL component module 27 determines whether the video data of the enhancement level in the NAL component includes intra-frame encoded data (142), for example, by detecting a suitable syntax element value. In addition, the NAL component module 27 parses the NAL component to detect syntax elements indicating in which area (pixel or transform) the base level and enhancement level should be summed (144), whether the presence of residual data in the enhancement level is indicated (146 ) and whether the parameter set and the volume of this parameter set are indicated (148). The NAL component module 27 also detects syntax elements identifying blocks with intraframe coding with nonzero coefficients greater than unity (150) in the enhancement level, and syntax elements indicating CBP patterns for blocks with interframe coding in the video data of enhancement level (152). Based on the definitions provided by the specified syntax elements, the NAL component module 27 provides the video decoder 28 with appropriate instructions for their use in decoding video data of the base level and enhancement level (154).

In the examples of FIGS. 8 and 9, enhancement level NAL components can carry syntax elements with many different enhancement level indications to assist video decoder 28 in processing the NAL component. For example, these various indications may include an indication of whether the NAL component contains video data of the enhancement layer with intra-frame encoding, an indication of whether the decoder should use the addition of video data of the enhancement level to the base layer data in the pixel or transform region, and / or an indication of whether the enhancement layer video data includes any residual data related to the basic layer video data. In addition to the above examples, enhancement layer NAL components can also carry syntax elements indicating whether the NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of a reference image layer.

Other syntax elements may identify blocks in video data of the enhancement layer containing non-zero coefficient values, indicate the number of non-zero coefficients in blocks with intraframe coding in video data of the enhancement level with a value greater than one, and indicate coded block patterns for blocks with interframe coding in video data of the enhancement level. Again, the examples proposed in figures 8 and 9 should not be construed as limiting. The NAL components of the enhancement layer can provide many additional syntax elements and semantics, some of which are discussed below.

The following describes in more detail examples of the syntax of the improvement level with a discussion of the applied semantics. According to some aspects, as discussed above, NAL components can be used in encoding and / or decoding multimedia data, including base layer video data and enhancement layer video data. In these cases, the general syntax and structure of the enhancement layer NAL components may be the same as in the H.264 standard. However, it will be apparent to those skilled in the art that other blocks may be used. Alternatively, you can enter new values for the NAL component type (nal_unit_type), which specify the data structure type of the raw byte serial payload (RBSP) contained in the enhancement level NAL component.

In general, the enhancement level syntax described in this invention can be characterized by semantics with low overhead and low complexity, such as single-level decoding. The syntax of macroblocks of the enhancement level can be characterized by high compression efficiency, and this syntax can define syntax elements for block patterns (CBP) of 16 × 16 enhancement levels with intraframe coding, CBP patterns for macroblocks with interframe encoding of the enhancement level and a new entropy decoding using contextual coding tables -adaptive coding with variable length codes (CAVLC) for macroblocks MB level enhancement with intraframe coding.

At low overhead, the syntax of the layers and macroblocks (MB) defines the relationship between the layer of the enhancement layer and the combined layer of the base layer. Prediction modes for macroblocks and motion vectors in the base layer syntax can be expressed in the base layer syntax. MB modes for enhancement level can be obtained from MB modes for a combined base level. The Enhanced Encoder Block Level Pattern (CBP) MB can be decoded in two different ways, depending on the CBP pattern for the MB combined base layer.

With low complexity, single-level decoding can be performed by simply combining operations for base-level bitstreams and an enhancement level to simplify the decoder and reduce power consumption. In this case, the coefficients of the base level can be converted to the scale of the level of improvement, for example, by multiplying by the scale factor, which can be performed by bit shift based on the difference of the quantization parameters (QP) between the base level and the level of improvement.

Also, for low complexity, a refine_intra_mb_flag syntax element indicating the presence of MB with intra-frame coding in layer P of the enhancement layer may be provided. By default, refine_ intra_mb_flag == 0 can be set to enable single-level decoding. In this case, there is no refinement for macroblock MB with intra-frame coding at the enhancement level. This will not adversely affect visual quality, even if MB macroblocks with intraframe coding are encoded with basic layer quality. In particular, MBs with intraframe coding usually correspond to newly appeared visual information, and the human eye is not sensitive to it at the beginning. However, the refine_intra_mb_flag = 1 value may still be provided for the extension.

With high compression efficiency, a CBP pattern for 16 × 16 MB with intra-frame enhancement level coding can be provided so that a portion of the intra-frame coded coefficients for the 16 × 16 enhancement level macroblocks is determined based on base level brightness prediction modes for 16 × 16 macroblock with intra-frame encoding. The 16 × 16 MB CBP with intra-frame coding of the enhancement layer is decoded in two different ways depending on the CBP of the macroblock of the combined base layer. In case 1, when not all the AC coefficients of the base level are zero, CBP for macroblocks of 16 × 16 enhancement levels with intraframe coding are decoded according to the H.264 standard. A syntax element (e.g., BaseLayerAcCoefficientsAllZero) may be provided as a flag that indicates whether all the AC coefficients of the corresponding macroblock in the base layer are zero. In case 2, when all the baseline AC coefficients are zero, a new approach can be provided for transmitting CBP macroblocks 16 × 16 with intra-frame coding. Namely, the enhancement level MB is divided into 4 subsections of MB depending on the prediction modes for 16 × 16 brightness macroblocks with intraframe coding of the base level.

CBP can be provided for the MV level of enhancement with intra-frame coding to specify which of the six 8x8 luma and chroma blocks contains non-zero coefficients. CBPs for MB enhancement layer are decoded in two ways depending on CBP for MB combined base layer. In case 1, when the CBP for the MB of the combined base level (base_coded_block_pattern or base_cbp) is zero, the CBP for the MB of the enhancement level (enh_coded_block_pattern or enh_cbp) is decoded according to the H.264 standard. In case 2, when base_coded_block_pattern is not equal to zero, a new approach can be provided for the expression enh_coded_block_pattern. For a base level 8x8 block with non-zero coefficients, one bit is used to indicate whether the 8x8 block of the combined enhancement level has non-zero coefficients. The status of other 8x8 blocks is represented by variable-length coding (VLC).

As a further refinement, new entropy decoding (CAVLC tables) may be provided for macroblocks of the MB level of enhancement with intraframe coding to represent the number of non-zero coefficients in the MB of the enhancement level with intraframe coding. The syntax element enh_coeff_token 0 (16 may represent the number of non-zero coefficients from 0 to 16, provided that there are no coefficients greater than 1. The syntactic element enh_coeff_token 17 represents that there is at least one non-zero coefficient greater than 1. In this case, (enh_coeff_token 17), the standard approach is used to decode the total number of nonzero coefficients and coefficients with a unit at the end.enh_coeff_token (0 (16) is decoded using one of eight VLC tables, based on the context.

In this description, various abbreviations shall be interpreted as defined in clause 4 of the H.264 standard. Rules can be interpreted as specified in clause 5 of the H.264 standard, and formats of source, encoded, decoded and output data, scanning processes and neighboring relationships can be interpreted as specified in clause 6 of the H.264 standard.

In addition, based on the purposes of this description, the following definitions can be applied. The term “baseline” generally refers to a bitstream containing encoded video data that represents the first level of spatio-temporal-SNR scalability defined by this description. The basic layer bitstream can be decoded by any suitable decoder with an extended H.264 profile. The syntax element BaseLayerAcCoefficientsAllZero is a variable, which, being not equal to 0, indicates that all AC coefficients of the combined macroblock in the base level are zero.

The syntax element BaseLayerIntra16x16PredMode has values 0, 1, 2, or 3, which correspond to Intra_16 × 16_Vertical, Intra_16 × 16_Horizontal, Intra_16 × 16_DC and Intra_16 × 16_Planar, respectively. This variable is equal to the Intra16x16PredMode variable, as specified in clause 8.3.3 of the H.264 standard. The syntax element BaseLayerMBType is a variable that indicates the type of combined macroblock at the base level. This variable may be equal to the syntax element mb_type specified in clause 7.3.5 of the H.264 standard.

The term “base layer” (or base_layer_slice) refers to a layer that is encoded according to clause 7.3.3 of the H.264 standard and which has a corresponding enhancement layer layer specified in this description with the same image serial number as defined in clause 8.2 .1 H.264 standard. The BaseLayerSliceType (or base_layer_slice_type) element is a variable that indicates the type of the co-layer in the base layer. This variable is equal to the syntax element slice_type, as specified in clause 7.3.3 of the H.264 standard.

The term “enhancement level” generally refers to a bitstream containing encoded video data that represents a second level of spatio-temporal-SNR scalability. The enhancement layer bitstream can only be decoded together with the base layer, that is, it contains links to decoded base layer video data that are used to create the final decoded video data.

A quarter macroblock refers to one quarter of the macroblock samples that are the result of partitioning the macroblock. This definition is similar to the definition of a sub-macroblock in the H.264 standard, except that quarter macroblocks can take a non-square (e.g., rectangular) shape. The term “quarter macroblock section” refers to a block of luminance samples and two corresponding blocks of chroma samples resulting from a split of a quarter macroblock for inter-frame prediction or intra-frame refinement. This definition can be identical to the definition of a sub-macroblock section in the H.264 standard, except that the term “intraframe” is introduced in this description.

The term “macroblock section” refers to a block of luma samples and two corresponding blocks of chroma samples resulting from a macroblock partition for inter-frame prediction or intra-frame refinement. This definition is identical to the definition in the H.264 standard except that the term "intraframe" is introduced in this description. In addition, the shapes of the macroblock partitions defined in this description may differ from those defined in the H.264 standard.

Enhancement level syntax

RBSP Syntax

Table 1 below provides examples of RBSP types for fairly simple video scalability.

Table 1
Raw Byte Serial Load and RBSP Tail Bits RBSP Description Sequence parameter set RBSP The sequence parameter set is sent only at the basic level. RBSP image parameter set A set of image parameters is sent only at a basic level. Layer Data Partition RBSP Syntax Enhancement layer layer data section RBSP syntax conforms to H.264

As indicated above, the syntax of the RBSP enhancement layer may be the same as in the specified standard except that the sequence parameter set and the image parameter set can be sent at a basic level. For example, the RBSP syntax of the sequence parameter set, the RBSP syntax of the image parameter set, and the RBSP of the layer data section encoded in the enhancement layer may have the syntax specified in clause 7 of ITU-T H.264.

In the various tables in this specification, all syntax elements may have the appropriate syntax and semantics specified in the ITU-T H.264 standard, within the specified syntax elements described in the H.264 standard, unless otherwise specified. In general, syntax elements and semantics not described in the H.264 standard are described in this invention.

In the various tables in this description, a column marked “C” lists the categories of syntax elements that can be represented in the NAL component that can correspond to the categories in the H.264 standard. In addition, syntax elements with the syntax category “All” may be present, as defined by the syntax and semantics of the RBSP data structure.

The presence or absence of any syntactic elements of a specific listed category is determined based on the syntax and semantics of the corresponding RBSP data structure. The descriptor column defines the descriptor, for example, f (n), u (n), b (n), ue (v), se (v), me (v), ce (v), which can generally correspond to descriptors, defined in the H.264 standard, unless otherwise specified in this description.

Advanced NAL Component Syntax

The syntax for NAL components in relation to extensions for video scalability according to one aspect of this invention can be generally defined, as shown in Table 2 below.

table 2
NAL Component Syntax for Extensions nal_unit (NumBytesInNALunit) { FROM Descriptor forbidden zero bit All f (1) nal_ref_idc All u (2) nal_unit_type / * equal to 30 * / All u (5) reserved_zero_lbit All u (1) extension_flag All u (1) if (! extension flag) { enh_profile_idc All u (3) reserved_zero_3bits All u (3) } otherwise { extended_nal_unit_type All u (6) NumBytesInRBSP = 0 for (i = 1; i <NumBytesInNALunit; i ++) { if (i + 2 <NumBytesInNALunit && next_bits (24) == 0x000003) { rbsp_byte [NumBytesInRBSP ++] All b (8) rbsp_byte [NumBytesInRBSP ++] All b (8) i + = 2 emulation_prevention_three_byte / * equal to 0x03 * / All f (8) } otherwise rbsp_byte [NumBytesInRBSP ++] All b (8) } } }

In table 2 above, nal_unit_type is set to 30 to indicate a specific extension for processing the enhancement level. When nal_unit_type is set to the selected value, for example, 30, the NAL component indicates that it carries enhancement level data that starts processing the enhancement level by decoder 28. The nal_unit_type value provides a unique dedicated nal_unit_type to support the processing of additional syntax modifications of the enhancement level bitstream in addition to the standard bitstream H.264. As an example, nal_unit_type can be set to 30 to indicate that the NAL component includes enhancement level data and trigger processing of additional syntax elements that may be present in the NAL component, such as, for example, extension_flag and extended_nal_unit_type. For example, the syntax element extended_nal_unit_type is set to some value specifying the type of extension. In particular, extended_nal_unit_type may indicate the type of NAL component of the enhancement level. The extended_nal_unit_type element may indicate the type of RBSP data structure for enhancement level data in the NAL component. For B layers, the syntax of the layer headers may conform to the H.264 standard. Further in this description, the applied semantics will be described in detail.

Layer header syntax

For I-layers and P-layers at the enhancement level, the syntax of the layer header can be defined, as shown in Table 3A below. Other parameters for the enhancement layer layer, including reference frame information, can be obtained from the superimposed layer of the base layer.

Table 3A
Layer header syntax enh_slice_header () { FROM Descriptor first_mb_in_slice 2 ue (v) enh_slice_type 2 ue (v) pic_parameter_set_id 2 ue (v) frame_num 2 u (v) If (pic_order_cnt_type == 0) { pic_order_cnt_lsb 2 u (v) if (pic_order_present_flag &&! field_pic flag) delta_pic_order_cnt_bottom 2 ue (v) } If (pic_order_cnt_type == 1 &&! Delta_pic_order_always_zero_flag) { delta_pic_order_cnt [0] 2 se (v) if (pic_order_present_flag &&! field_pic_flag) delta_pic_order_cnt [1] 2 se (v) } if (redundant_pic_cnt_present_flag) redundant_pic_cnt 2 ue (v) decoding_mode 2 ue (v) if (base_layer_slice_type! = I) Refine_intra_MB 2 f (l) Slice_qp_delta 2 se (v) }

The base_layer_slice element may refer to a layer encoded, for example, according to clause 7.3.3 of the H.264 standard, which has a corresponding layer of the enhancement level encoded according to table 2 with the same serial number as defined, for example, in clause 8.2.1 of standard H .264. The base_layer_slice_type element refers to the type of the base layer layer, for example, as specified in clause 7.3 of the H.264 standard. Other parameters for the enhancement layer, including reference frame information, are obtained from the combined base layer.

In the layer header syntax, the refine_intra_MB parameter indicates whether the enhancement level video data in the NAL component includes intra-frame encoded video data. If refine_intra_MB is 0, then intraframe coding exists only at the basic level. Accordingly, intra-frame decoding of the enhancement layer may be omitted. If refine_intra_MB is 1, then video data with intraframe coding is present both at the base level and at the enhancement level. In this case, data with intra-frame encoding of the enhancement layer can be processed to improve the quality of data with intra-frame encoding of the base layer.

Layer Data Syntax

An exemplary syntax for the layer data may be provided as specified in Table 3B below.

Table 3B
Layer Data Syntax enh_slice_data () { FROM Descriptor CurrMbAddr = first_mb_in_slice moreDataFlag = 1 do { if (moreDataFlag) { if (BaseLayerMbType! = SKIP && (refine_intra_mb_flag ||
(BaseLayerSliceType! = I && BaseLayerMbType! = I))) enh_macroblock_layer () } CurrMbAddr = NextMbAddress (CurrMbAddr) moreDataFlag = more_rbsp_data () } while (moreDataFlag) }

Macroblock Level Syntax

Exemplary syntax for macroblocks of the enhancement layer can be provided, as shown in Table 4 below.

Table 4
Syntax MB Improvement Level enh_macroblock_layer () { FROM Descriptor tor if (MbPartPredMode (BaseLayerMbType, 0) == Intra_16 × 16) { enh_intral6 × 16 macroblock_cbp () if (mb_intral6 × 16_luma_flag || mb_intral6 × 16 chroma_flag) { mb_qp_delta 2 se (v) enh_residual () 3 | 4 } } otherwise, if (MbPartPredMode (BaseLayerMbType, 0) == Intra_4 × 4) { coded_block_pattern 2 me (v) if (CodedBlockPatternLuma> 0 || CodedBlockPatternChroma> 0) { mb_qp_delta enh_residual () } } otherwise { enh_coded_block_pattern 2 me (v) EnhCodedBlockPatternLuma = enh_coded_block_pattern% 16 EnhCodedBlockPatternChroma = enh_coded_block_pattern / 16 if (EnhCodedBlockParternLuma> 0 || EnhCodedBlockPatternChroma> 0)
{ mb_qp_delta 2 se (v) residual ()
/ * Standard compliant syntax as specified in clause 7.3.5.3 [1] * / } } }

Other parameters for the macroblock extension level are obtained from the macroblock level of the base level for the corresponding macroblock in the corresponding base_layer_slice parameter.

In Table 4 above, the syntax element enh_coded_block_pattern generally indicates whether the enhancement layer video data in the enhancement layer MB includes residual data related to the base layer data. Other parameters for the macroblock extension level are obtained from the macroblock level of the base level for the corresponding macroblock in the corresponding base_layer_slice parameter.

Coded Block Pattern (CBP) Syntax for Intraframe Encoding Macroblocks

For 4x4 macroblocks with intraframe coding, the CBP syntax may be the same as in the H.264 standard, for example, as in clause 7 of the H.264 standard. For intra-frame 16x16 macroblocks, a new syntax can be provided for encoding CBP information, as shown in Table 5 below.

Table 5
Syntax CBP macroblock 16 × 16 with intraframe coding enh_intral6 × 16_macroblock_cbp () { FROM Descriptor mb_intral6 × 16_luma_flag 2 u (1) if (mb_intral6 × 16_luma_flag) { if (BaseLayerAcCoefficientsAllZero) for (mbPartIdx = 0; mbPartIdx <4; mbPartIdx ++) { mb_intral6 × 16_luma_part_flag [mbPartIdx] 2 u (1) if (mb_intral6 × 16_luma_part flag [mbPartIdx]) for (qtrMbPartIdx = 0; qtrMbPartIdx <4; qtrMbPartIdx ++) qtr_mb_intral_6 × 16_luma_part_flag
[mbPartldx] [qtrMbPartldxl 2 u (1) mb_intral6 × 16_chroma_flag 2 u (1) if (mb_intral6 × 16_chroma_flag) { Mb_intral6 × 16_chroma_ac_flag 2 u (1) }

Residual Data Syntax

The syntax for the residual MBs with intraframe coding in the enhancement layer, that is, the syntax of the residual enhancement layer data, may be as shown in Table 6A below. For residual MBs with intraframe coding, the syntax may conform to the H.264 standard.

Table 6A
Residual MB Syntax with Intraframe Encoding enh_residual () { FROM Descriptor if (MbPartPredMode (BaseLayerMbType, 0) == Intra_16 × 16) enh_residual_block_cavlc (Intra16 × 16DCLevel, 16) 3 for (mbPartldx = 0; mbPartldx <4; mbPartIdx ++) for (qtrMbPartldx = 0; qtrMbPartldx <4; qtrMbPartIdx ++) if (MbPartPredMode (BaseLayerMbType, 0) == Intra_16 × 16 &&
BaseLayerAcCoefficientsAllZero) { if (mb_intral6xl6_luma_part_flag [mbPartIdx] &&
qtr_mb_intra16 × 16_luma_part_flag [mbPartIdx] [qtrMbPartIdx]) enh_residual_block_cavlc (Intra16x16ACLevel [mbPartldx * 4 + qtrMbPartId
x], 15) 3 otherwise for (i = 0; i <15; i ++) Intra16 × 16ACLevel [mbPartldx * 4 + qtrMbPartldx] [i] = 0 otherwise if (EnhCodedBlockPatternLuma & (1 "mbPartldx)) { if (MbPartPredMode (BaseLayerMbType, 0) == Intra_16 × 16) enh residual block cavlc (
Intra16 × 16ACLevel [mbPartldx * 4 + qtrMbPartldx], 15) 3 otherwise enh residual block cavlc (
LumaLevel [mbPartldx * 4 + qtrMbPartldx], 16) 3 | 4 } otherwise { if (MbPartPredMode (BaseLayerMbType, 0) == Intra_16 × 16) for (i = 0; i <15; i ++) Intra16 × 16ACLevel [mbPartldx * 4 + qtrMbPartldx] [i] = 0 otherwise for (i = 0; i <16; i ++) LumaLevel [mbPartldx * 4 + qtrMbPartldx] [i] = 0 } for (iCbCr = 0; iCbCr <2; iCbCr ++) if (EnhCodedBlockPatternChroma & 3) / * chroma DC residual present
* / residual_block (ChromaDCLevel [iCbCr], 4) 3 | 4 otherwise for (i = 0; i <4; i ++) ChromaDCLevel [iCbCr] [i] = 0 for (iCbCr = 0; iCbCr <2; iCbCr ++) for (qtrMbPartldx = 0; qtrMbPartldx <4; qtrMbPartIdx ++) if (EnhCodedBlockPatternChroma & 2)
/ * chroma AC residual present * / residual_block (ChromaACLevel [iCbCr] [qtrMbPartldx], 15) 3 | 4 otherwise for (i = 0; i <15; i ++) ChromaACLevel [iCbCr] [qtrMbPartldx] [i] = 0 }

Other parameters for the remainder of the enhancement layer are obtained from the remainder of the baseline for the combined macroblock in the corresponding layer of the baseline.

CAVLC Syntax Residual Blocks

The syntax for context adaptive variable-length coding (CAVLC) coding of residual blocks may be as defined in Table 6B below.

Table 6B
CAVLC Syntax Residual Blocks enh_residual_block_cavlc (coeffLevel, maxNumCoeff) { FROM Descriptor for (i = 0; i <maxNumCoeff; i ++) coeffLevel [i] = 0 if ((MbPartPredMode (BaseLayerMbType, 0) == Intra_16 × 16 &&
mb_intra16 × 16_luma_flag) || (MbPartPredMode (BaseLayerMbType, 0) ==
Intra_4 × 4 && CodedBlockPatternLuma) { enh_coeff_token 3 | 4 ce (v) if (enh_coeff_token == 17) { / * Standard compliant syntax as specified in clause 7.3.5.3.1 of H.264 * / } otherwise { if (TotalCoeff (enh_coeff_token)> 0) { for (i = 0; i <TotalCoeff (enh_coeff_token); i ++) enh_coeff_sign_flag [i] 3 | 4 u (l) level [i] = 1-2 * enh_coeff_sign_flag if (TotalCoeff (enh_coeff_token) <maxNumCoeff) { total_zeros 3 | 4 ce (v) zerosLeft = total_zeros } otherwise zerosLeft = 0 for (i = 0; i <Totalcoeff (enh_coeff token) -1; i ++) { if (zerosLeft> 0) { ran_before 3 | 4 ce (v) ran [i] = ran_before } otherwise ran [i] = 0 zerosLeft = zerosLeft-ran [i] } ran [TotalCoeff (enh_coeff_token) -1] = zerosLeft coeffNum = -1 for (i = TotalCoeff (enh_coeff_token) -1; i> = 0; i-) { coeffNum + = ran [i] + 1 coeffLevel [coeffNum] = level [i] } } } otherwise { / * Suitable standard syntax specified in clause 7.3.5.3.1 of the H.264 standard * / } }

Other parameters for the CAVLC residual blocks of the enhancement layer can be obtained from the CAVLC residual blocks of the base layer for the combined macroblock in the corresponding layer of the base layer.

Improvement Level Semantics

The following describes the semantics of the improvement level. The semantics of the enhancement layer NAL components may essentially be the same as the syntax of the NAL components defined by the H.264 standard for the syntax elements defined in the H.264 standard. New syntax elements not described in the H.264 standard have the semantics used described in this invention. The semantics of the RBSP enhancement layer and RBSP tail bits may be the same as in the H.264 standard.

Semantics of Advanced NAL Components

We turn to the table 2 above, where the Forbidden_zero_bit parameter is specified as in clause 7 of the specification of the H.264 standard. A value of nal_ref_idc, not equal to 0, determines that the content of the extended NAL component contains a set of sequence parameters, a set of image parameters, a reference image layer, or a data section of the reference image layer. A nal_ref_idc value of 0 for an extended NAL component containing a layer or layer data section indicates that the layer or layer data section is part of a non-reference image. The nal_ref_idc value will not be 0 for the sequence parameter set or NAL components of the image parameter set.

When nal_ref_idc is 0 for one layer or an extended NAL component of a layer data section of a particular image, it will be 0 for all layers and extended NAL components of a layer data section of a given image. The nal_ref_idc value will not be 0 for extended NAL components with instant decoding update (IDR), that is, NAL components with extended_nal_unit_type equal to 5, as shown in table 7 below. In addition, nal_ref_idc will be 0 for all extended NAL components having extended_nal_unit_type equal to 6, 9, 10, 11, or 12, as shown in Table 7 below.

The nal_unit_type value is 30 in the “not defined” range of the H.264 standard to indicate the specialized NAL component for which the decoding process is specified in this description. The nal_unit_type value is not 30, as defined in clause 7 of the H.264 standard.

The extension_flag value is a one-bit flag. When the extension_flag value is 0, it determines that the next 6 bits are reserved. When extension_flag is equal to 1, this determines that this NAL component contains the RBSP of the extended NAL component.

The reserved value or reserved_zero_1bit is a one-bit flag used for future extensions to the application matching nal_unit_type of 30. The value enh_profile_idc indicates the profile to which the bitstream corresponds. The value reserved_zero_3bits is a 3-bit field reserved for future use. The value of extended_nal_unit_type is set below in table 7:

Table 7
Enhanced NAL Component Type Codes extended_nal_unit_type Extended NAL Component Content and RBSP Syntax Structure FROM 0 Not set one Coded image layer, slice_layer_without_partitioning_rbsp () without IDR 2, 3, 4 2 Encoded data section A of the slice_data_partition_a_layer_rbsp () layer 2 3 Encoded data section B of the slice_data_partition_b_layer_rbsp () layer 3 four C coded data section of slice_data_partition_c_layer_rbsp () layer four 5 IDR encoded image layer slice_layer_without_partitioning_rbsp () 2, 3 6 Additional Enhancement Information (SEI) sei_rbsp () 5 7 Seq_parameter_set_rbsp () sequence parameter set 0 8 Pic_parameter_set_rbsp () image parameter set one 9 Access block delimiter access_unit_delimiter_rbsp () 6 10 ... 23 Reserved 24 ... 63 Not set

Extended NAL components that use extended_nal_unit_type equal to 0 or in the range 24 to 63 inclusive do not affect the decoding process described in this invention. Types of the advanced NAL component with values of 0 and from 24 to 63 can be used as defined by this application. The decoding process for these values (0 and 24 through 63) nal_unit_type is not specified. In this example, decoders can ignore, that is, remove from the bitstream and discard the contents of all extended NAL components that use the reserved extended_nal_unit_type values. This potential requirement allows future definitions of compatible extensions. The rbsp_byte and emulation_prevention_three_byte values are specified in clause 7 of the H.264 standard specification.

The semantics of RBSP

The semantics of the RBSP enhancement level are defined as in clause 7 of the H.264 standard specification.

Layer header semantics

For layer header semantics, the syntax element first_mb_in_slice sets the address of the first macroblock in the layer. When random layer order is not allowed, the value of first_mb_in_slice will be no less than the value of first_mb_in_slice for any other layer of the current image that precedes the current layer in decoding order. The address of the first macroblock of the layer can be obtained as follows. The value of first_mb_in_slice is the address of the first macroblock in the layer, and first_mb_in_slice is in the range from 0 to PicSizeInMbs - 1 inclusive, where PicSizeInMbs is the number of megabytes in the image.

The enh_slice_type element sets the encoding type of the layer according to table 8 below.

Table 8
Name association with enh_slice_type values enh_slice_type Name enh_slice_type 0 P (P-layer) one B (B-layer) 2 I (I-layer) 3 SP (SP layer) or not used four SI (SI layer) or not used 5 P (P-layer) 6 B (B-layer) 7 I (I-layer) 8 SP (SP layer) or not used 9 SI (SI layer) or not used

Enh_slice_type values in the range of 5 to 9, in addition to the encoding type of the current layer, determine that all other layers of the current encoded image have an enh_slice_type value equal to the current enh_slice_type value or equal to the current slice_type-5 value. In alternative aspects, enh_slice_type values of 3, 4, 8, and 9 may not be used. When extended_nal_unit_type is 5, according to the instant decoding update (IDR) image, slice_type may be 2, 4, 7, or 9.

The syntax element pic_parameter_set_id is set to pic_parameter_set_id for the corresponding base_layer_slice. The frame_num element in the NAL component of the enhancement layer will be the same as the aligned layer of the base layer.

Similarly, the pic_order_cnt_lsb element in the NAL component of the enhancement layer will be the same as pic_order_cnt_lsb for the combined base layer (base_layer_slice). The semantics for delta_pic_order_cnt_bottom, delta_pic_order_cnt [o], delta_pic_order_cnt [1] and the semantics of redundant_pic_cnt are defined as in clause 7.3.3 of the H.264 standard. The decoding_mode_flag element defines the decoding process for the enhancement layer, as shown in Table 9 below.

Table 9
Decoding_mode_flag specification decoding_mode_flag Process 0 Adding in the Pixel Area one Adding in the area of coefficients

In Table 9 above, the addition in the pixel region indicated by the decoding_mode_flag value of 0 in the NAL component means that the enhancement layer should be added to the base layer in the pixel region to support single-level decoding. Adding in the coefficient area indicated by decoding_mode_flag equal to 1 in the NAL component means that the enhancement layer can be added to the base layer in the coefficient field to support single-level decoding. Thus, decoding_mode_flag provides a syntax element that indicates whether the decoder should use the addition of enhancement layer video data to the base layer data in the pixel region or in the transform domain.

Adding in the pixel region adds the enhancement layer to the base layer in the pixel region as follows:

where Y is the brightness, Cb is the color of blue, and Cr is the color of red, and where Clip1Y is a mathematical function that has the following form:

Clip1 _Y (x) = Clip3 (0, (1 << BitDepth _Y ) -1, x),

and Clip1C represents the following mathematical function:

Clip1 _C (x) = Clip3 (0, (1 << BitDepth _C ) -1, x),

and where Clip3 is described in this document. The mathematical functions of Clip1y, Clip1c, and Clip3 are defined in the H.264 standard.

Adding in the area of coefficients leads to adding a layer of the level of improvement to the layer of the base level in the field of coefficients as follows:

LumaLevel [i] [j] = k LumaLevel [i] [j] base + LumaLevel [i] [j] enh

ChromaLevel [i] [j] = k ChromaLevel [i] [j] base + ChromaLevel [i] [j] enh

Where k is the scale factor used to bring the base-level coefficients to the QP scale of the enhancement level.

The refine_intra_MB syntax element in the enhancement layer NAL component determines whether to refine the macroblock MBs with intraframe coding at the enhancement layer in layers other than I-layers. If refine_intra_MB is 0, then the intra-frame encoded MBs are not refined at the enhancement level, and these MBs at the extension level will be skipped. If refine_intra_MB is 1, then MBs with intra-frame coding at the enhancement level are refined.

The slice_qp_delta element sets the initial value of the brightness quantization parameter QP _{Y to} be used for all macroblocks in the layer until it is changed to mb_qp_delta at the macroblock level. The initial quantization parameter QP _Y for the layer is calculated as:

The value of slice_qp_delta can be limited, so that QPY will be in the range from 0 to 51 inclusive. The pic_init_qp_minus26 value indicates the initial QP value.

Layer data semantics

The semantics of the data of the enhancement level layer can be specified as in clause 7.4.4 of the H.264 standard.

Macroblock Level Semantics

For macroblock level semantics, the enh_coded_block_pattern element determines which of the six 8x8 blocks (luminance and color) can contain non-zero levels of transform coefficient. The semantics of the mb_qp_delta element can be specified as in clause 7.4.5 of the H.264 standard. The semantics for the coded_block_pattern syntax element can be specified as in clause 7.4.5 of the H.264 standard.

Semantics of the coded block template (CBP) of a 16x16 macroblock with intraframe coding

For I-layers and P-layers, when refine_ intra_mb_flag is 1, the semantics of 16 × 16 CBP with intraframe coding are determined by the following description. Macroblocks that have a prediction mode for a macroblock of a combined base level with intra-frame coding (Intra_16 × 16) can be divided into 4 quarter macroblocks depending on the values of their AC coefficients and the prediction mode of the Intra_16 × 16 macroblock of a combined base level (BaseLayerIntra16 × 16PredMode). If all base level AC coefficients are equal to zero, and at least one enhancement level AC coefficient is not equal to zero, then the macroblock of the enhancement level is divided into 4 sections of the macroblock depending on BaseLayerIntra16 × 16PredMode.

This macroblock partitioning creates partitions called quarter macroblocks. Each quarter macroblock can be further divided into 4x4 sections of a quarter macroblock. Figures 10 and 11 are diagrams illustrating a partition of macroblocks and quarter macroblocks. In FIG. 10 shows sections of a macroblock of an enhancement layer based on prediction modes for 16 × 16 base layer blocks with intraframe coding and their indices corresponding to spatial positions. In FIG. 11 shows sections of a quarter macroblock of an enhancement level based on the sections of the macroblock shown in FIG. 10, and their indices corresponding to spatial positions.

In FIG. 10 shows an Intra_16 × 16_Vertical mode with four MB partitions, each of 4 * 16 brightness samples and corresponding chroma samples, Intra_16 × 16_Horizontal mode with 4 macroblock sections, each of 16 * 4 brightness samples and corresponding chroma samples, and Intra_16 × 16_DC or Intra_16 mode × 16_Planar with 4 sections of the macroblock, each of 8 * 8 brightness samples and corresponding color samples.

In FIG. 11 shows 4 quarter vertical sections of a macroblock, each of 4 * 4 brightness samples and corresponding chroma samples, 4 quarter horizontal sections of a macroblock, each of 4 * 4 brightness samples and corresponding chroma samples, and 4 quarter DC or Planar macroblock sections, each of 4 * 4 brightness samples and corresponding chroma samples.

Each section of the macroblock is designated as mbPartIdx. Each quarter macroblock section is designated as qtrMbPartIdx. Both mbPartIdx and qtrMbPartIdx can have values of 0, 1, 2, or 3. Sections of the macroblock and quarter macroblock are scanned to refine intraframe coding, as shown in Figures 10 and 11. The rectangles correspond to the sections. The number in each rectangle specifies the index scan of the macroblock section or the scan section of the quarter macroblock.

The mb_intra16 × 16_luma_flag element, equal to 1, determines that at least one coefficient in Intra16 × 16ACLevel is not equal to zero. The intra16 × 16_luma_flag parameter, equal to 0, determines that all coefficients in Intra16 × 16ACLevel are equal to zero.

The mb_intra16 × 16_luma_part_flag [mbPartIdx] element, equal to 1, determines that there is at least one nonzero coefficient in the Intra16 × 16ACLevel in the macroblock section of mbPartIdx. The mb_intra16x16_luma_part_flag [mbPartIdx] element, equal to 0, determines that all coefficients in the Intra16 × 16ACLevel in the macroblock section of mbPartIdx are equal to zero.

The qtr_mb_intra16 × 16_luma_part_flag [mbPartIdx] [qtrMbPartIdx] element equal to 1 determines that there is at least one nonzero coefficient in the Intra16 × 16ACLevel in the quarter macroblock section qtrMbPartIdx.

The qtr_mb_intra16 × 16_luma_part_flag [mbPartIdx] [qtrMbPartIdx] element, equal to 0, determines that all coefficients in Intra16 × 16ACLevel in the quarter macroblock section qtrMbPartIdx are equal to zero. The mb_intra16 × 16_chroma_flag element, equal to 1, determines that at least one color coefficient is not equal to zero.

The mb_intra16 × 16_chroma_flag element, equal to 0, determines that all chroma coefficients are equal to zero. The mb_intra16x16_chroma_AC_flag element, equal to 1, determines that at least one color coefficient in mb_ChromaACLevel is not equal to zero. The mb_intra16x16_chroma_AC_flag element, equal to 0, determines that all coefficients in mb_ChromaACLevel are equal to zero.

Residual Data Semantics

The semantics of the residual data, with the exception of the CAVLC semantics of the residual blocks described in this invention, may be the same as defined in clause 7.4.5.3 of the H.264 standard.

Semantics of CAVLC Residual Blocks

CAVLC semantics can be provided as follows. In particular, enh_coeff_token sets the total number of nonzero transform coefficient levels when scanning the transform coefficient level. The TotalCoeff (enh_coeff_token) function returns the number of non-zero levels of the conversion coefficient obtained from enh_coeff_token, as follows:

1. When enh_coeff_token is 17, TotalCoeff (enh_coeff_token) is specified as in clause 7.4.5.3.1 of the H.264 standard.

2. When enh_coeff_token is not equal to 17, TotalCoeff (enh_coeff_token) is equal to enh_coeff_token.

The value enh_coeff_sign_flag sets the sign of the nonzero level of the transform coefficient. The semantics of total_zeros is defined as in clause 7.4.5.3.1 of the H.264 standard. The semantics of run_be for e are defined as in clause 7.4.5.3.1 of the H.264 standard.

Decoding Processes for Extensions

I-layer decoding

The decoding processes for scalability extensions are described in more detail below. To decode the I-frame in the presence of data from both the base level and the enhancement level, two-pass decoding can be implemented in decoder 28. The two-pass decoding process can generally be performed as described above, after which iterative processing is performed as follows. First, the basic layer frame Ib is restored as a normal I-frame. Then, the I-frame of the combined enhancement level is restored as a P-frame. Then, the reference frame for this P-frame will be the reconstructed base-level I-frame. Again, all motion vectors in the reconstructed P-frame of the enhancement level will be zero.

If there is an enhancement level, each macroblock of the enhancement level is decoded as residual data using mode information from the combined macroblock at the base level. The base layer I-layer (Ib) can be decoded as in clause 8 of the H.264 standard. After the decoding of both the macroblock of the enhancement level and the combined macroblock of the basic level is performed, you can use the addition in the pixel region, as defined in clause 2.1.2.3 of the H.264 standard, to create the final restored block.

P-layer decoding

In the process of decoding P-layers, the base layer and the enhancement layer share the same mode and motion information that is transmitted in the base layer. Information for macroblocks with intraframe coding exists at both levels. In other words, the bits belonging to the macroblocks with intraframe coding exist only at the basic level, but do not exist at the improvement level, while the coefficients of the macroblock MBs with intraframe coding are distributed at both levels.

Improvement level macroblocks that have skipped macroblocks at the combined base level are also skipped.

If refine_intra_mb_flag is equal to 1, then the information belonging to macroblocks with intraframe coding exists at both levels, and decoding_mode_flag should be equal to 0. Otherwise, when refine_intra_mb_flag is equal to 0, then the information belonging to macroblocks with intraframe coding exists only at the basic level, and macroblocks of the enhancement layer having macroblocks with intra-frame coding of the combined base layer are skipped.

According to one aspect of the P-layer coding scheme, two-level data on the coefficients of MB macroblocks with intra-frame coding can be combined in a general-purpose microprocessor immediately after entropy decoding and before de-quantization, since the de-quantization module is located in the hardware core and is involved in pipelining with other modules. Therefore, the total number of macroblocks MB to be processed by the DSP processor and the hardware core can be exactly the same as in the case of single-level decoding, and the hardware core will only perform single-level decoding. In this case, you may not need to change the schedule of the hardware kernel.

In FIG. 12 is a flowchart illustrating a decoding process of a P layer. As shown in FIG. 12. video decoder 28 performs entropy decoding of macroblocks of the base level. If the current base layer MB is intra-frame encoded MB or skipped (162), then video decoder 28 proceeds to the next base layer MB (164). However, if the MB is not an intra-frame encoded MB or is not skipped, then video decoder 28 performs entropy decoding for the MB combined enhancement level (166), and then combines the two data layers (168), that is, the base level MB after entropy decoding and MB after entropy decoding for a combined level of improvement, to create a single level of data for operations of de-quantization and inverse transformation. The tasks shown in FIG. 12 can be performed on a general purpose microprocessor before transmitting a single combined data layer to the hardware core for de-quantization and inverse transformation. Based on the procedure shown in FIG. 12, the buffer management (dpb) of the decoded image is the same or nearly the same as single-level decoding, and additional memory may not be required.

Decoding of macroblocks with intraframe coding for enhancement level

When decoding macroblocks with intraframe coding for an enhancement level during entropy decoding of transform coefficients, CAVLC coding may require contextual information that is processed differently when decoding the base layer and decoding the enhancement level. This contextual information includes the number of non-zero levels of the transform coefficient (specified by the TotalCoeff parameter (coeff_token)) in the block of transform coefficient levels located to the left of the current block (blkA) and the block of transform coefficient levels located above the current block (blkB).

In entropy decoding of macroblocks with intraframe coding for the enhancement level with combined macroblocks of the base level with non-zero coefficients, the context for decoding coeff_token is the number of non-zero coefficients in the blocks of the combined base level. In the entropy decoding of macroblocks with intraframe coding for the enhancement level with the combined macroblock of the base level, where all coefficients are zero, the context for decoding coeff_token is the context of the enhancement level, and nA and nB are the number of nonzero transform coefficient levels specified by the TotalCoeff (coeff_token) parameter in the block blkA of the enhancement level, located to the left of the current block, and the blkB block of the base level, located above the current block, respectively. After entropy decoding, information is stored by decoder 28 for decoding other macroblocks and unlocking. The TotalCoeff (coeff_token) of each transform block is saved only when decoding the base layer without decoding the enhancement layer. This information is used as a context for entropy decoding of other macroblocks and for controlling release. In video decoding of the enhancement level, the TotalCoeff parameter (enh_coeff_token) is used as a context and for controlling release.

According to one aspect of the invention, the hardware core in decoder 28 is configured to process entropy decoding. According to this aspect, the DSP may be configured to inform the hardware core when decoding a P frame with zero motion vectors. The standard P-frame is decoded in the hardware core, and scalable decoding is transparent. Again, compared to single-level decoding, the decoding of the I-frame of the enhancement level is generally equivalent to the decoding time of the standard I-frame and P-frame.

If the frequency of I-frames does not exceed one frame per second, then the complication is negligible. If this frequency exceeds one I-frame per second (due to a scene change or for some other reason), then the encoding algorithm can provide encoding of such I-frames only at a basic level.

The process of getting enh_coeff_token

The following describes the process for obtaining enh_coeff_token. The syntax element enh_coeff_token can be decoded using one of the eight VLC codes defined in Tables 10 and 11 below. The enh_coeff_sign_flag element specifies the sign of the non-zero level of the transform coefficient. The VLCs in Tables 10 and 11 are based on statistical information from 27 decoded MPEG2 sequences. Each VLC sets a TotalCoeff value (enh_coeff_token) for a given codeword enh_coeff_token. The choice of VLC depends on the variable numcoeff_vlc, which is obtained as follows. If the combined block of the basic level has non-zero coefficients, then the following procedure is applied:

If (base_nC <2) Numcoeff_vlc = 0; Otherwise, if (base_nC <4)

Numcoeff_vlc = 1; Otherwise, if (base_nC <8) Numcoeff_vlc = 2; Otherwise Numcoeff_vlc = 3;

Otherwise, nC is found using a suitable method from the H.264 standard, and Numcoeff_vlc is obtained as follows:

If (nC <2) Numcoeff_vlc = 4; Otherwise, if (nC <4) Numcoeff_vlc = 5; Otherwise, if (nC <8) Numcoeff_vlc = 6; Otherwise Numcoeff_vlc = 7;

Table 10
Code tables for decoding enh_coeff_token, numcoeff_vlc = 0-3 enh_coeff_token numcoeff_vlc = 0 numcoeff_vlc = 1 numcoeff_vlc = 2 numcoeff_vlc = 3 0 10 101 11110 1001 1 one eleven 01 101 1111 2 00 00 00 110 i 010 111 01 01 four 0110 one hundred 110 00 5 0111 0 1100 one hundred 101 6 0111 101 11010 1110 1110 7 0111 1001 1101 101 1111 10 10010 8 0111 1000 1 1101 1001 1111 1111 1000 11 9 0111 1000 01 1101 1000 1 1111 1110 1 1000 101 10 0111 1000 001 1101 1000 01 1111 1110 01 1000 1000 eleven 0111 1000 0001 1 1101 1000 001 1111 1110 001 1000 1001 00 12 0111 1000 0001 0 1101 1000 0001 1111 1110 0001 1000 1001 01 13 0111 1000 0000 0 1101 1000 0000
eleven 1111 1110 0000 00 1000 1001 100 fourteen 0111 1000 0000 10 1101 1000 0000 00 1111 1110 0000 01 1000 1001 101 fifteen 0111 1000 0000
110 1101 1000 0000 01 1111 1110 0000 10 1000 1001 110 16 0111 1000 0000
111 1101 1000 0000 10 1111 1110 0000
eleven 1000 1001 111 17 0111 11 1101 11 1111 110 1000 0

Table 11
Code tables for decoding enh_coeff_token, numcoeff_vlc = 4-7 enh_coeff_token numcoeff_vlc = 4 numcoeff_vlc = 5 numcoeff_vlc = 6 numcoeff_vlc = 7 0 one eleven 10 1010 one 01 10 01 1011 2 001 01 00 one hundred 3 0001 001 110 1100 four 0000 1 0001 1110 0000 5 0000 00 0000 1 1111 0 0001 6 0000 0101 0000 01 1111 10 0010 7 0000 0100 1 0000 000 1111110 0011 8 0000 0100 01 0000 0011 1 1111 1110 1 0100 9 0000 0100 001 0000 0011 01 1111 1110 01 0101 10 0000 0100 0000 0000 0011 000 1111 1110 0011 0110 eleven 0000 0100 0001
eleven 0000 0011 001 00 1111 1110 0000 0 0111 12 0000 0100 0001 00 0000 0011 0010 1 1111 1110 0000 1 11010 13 0000 0100 0001 010 0000 0011 0011 00 1111 1110 0001 0 1101 1 fourteen 0000 0100 0001
011 0000 0011 0011 01 1111 1110 0001 1 11100 fifteen 0000 0100 0001 100 0000 0011 0011 10 1111 1110 0010 0 11101 16 0000 0100 0001 101 0000 0011 0011
eleven 1111 1110 0010 1 11110 17 0000 011 0000 0010 1111 1111 1111 1

Inter-frame coding macroblock decoding for enhancement level

The following describes the decoding of inter-frame coded macroblocks for the enhancement layer. For inter-frame coded macroblocks (excluding missing macroblocks), decoder 28 decodes residual information from both the base layer and the enhancement layer. Thus, the decoder 28 can be configured to provide two entropy decoding processes that may be required for each macroblock.

If both the base level and the improvement level have non-zero coefficients for the macroblock, then context information from neighboring macroblocks is used at both levels to decode coeff_token. Each level uses its contextual information.

After entropy decoding, information is stored as context information for entropy decoding of other blocks and release. For base level decoding, the decoded TotalCoeff parameter (coeff_token) is stored. For decoding the enhancement level, the base layer decoded TotalCoeff parameter (coeff_token) and the enhancement level decoded TotalCoeff parameter (enh_coeff_token) are separately stored separately. The TotalCoeff parameter (coeff_token) is used as the context for decoding the coeff_token of the macroblock of the base layer, including macroblocks with intraframe coding that exist only in the base layer. The sum TotalCoeff (coeff_token) + TotalCoeff (enh_coeff_token) is used as a context for decoding macroblocks with inter-frame coding in the enhancement level.

Decoding of macroblocks with intraframe coding for enhancement level

For inter-frame coding macroblocks (MBs), with the exception of missing MBs, if any, residual information can be encoded at both the base level and the enhancement level. Thus, for each MB, two entropy decoding processes are used, for example, as shown in FIG. 5. If we assume that both levels have non-zero coefficients for MB, then for decoding coeff_token, context information for neighboring MBs is provided at both levels. Each level has its own contextual information.

After entropy decoding, some information is stored for entropy decoding of other MBs and release. If base level video decoding is performed, then the decoded base level TotalCoeff (coeff_token) is stored. If video decoding of the enhancement level is performed, then the decoded base level TotalCoeff (coeff_token) and the enhancement level decoded TotalCoeff (enh_coeff_token) are separately stored.

The TotalCoeff parameter (coeff_token) is used as a context for decoding the coeff_token MB of the base layer, including intra-frame encoded MBs that exist only in the base layer. For decoding MB with interframe coding in the enhancement layer, the sum of the base level TotalCoeff (coeff_token) and enhancement level TotalCoeff (enh_coeff_token) are used as context. In addition, this amount can also be used as a parameter to release video data of the enhancement level.

Since dequantization involves intensive calculations, coefficients from two levels can be combined before dequantization in a general-purpose microprocessor, so that the hardware core will dequantize once for each MB with one QP. Both levels can be combined in a microprocessor, for example, as described in the next section.

Decoded Coded Block Pattern (CBP)

The enhancement level macroblock CBP, enh_coded_block_pattern, indicates coded block patterns in enhancement level video data. In some cases, enh_coded_block_pattern may be shortened to enh_cbp, for example, as in tables 12-15 below. When decoding CBP with high CBP compression efficiency, the macroblock enhancement level, enh_coded_block_pattern, can be encoded in two different ways depending on the base_coded_block_pattern cbp MB pattern of the combined base level.

For case 1, when base_coded_block_pattern = 0, enh_coded_block_pattern can be encoded according to the H.264 standard, for example, in the same way as the base level. For case 2, when base_coded_block_pattern is not 0, the following approach can be used to obtain enh_coded_block_pattern. This approach can include three steps:

Step 1. At this step, for each 8x8 brightness block, where the corresponding base level coded_block_pattern bit is 1, one bit is selected. Each bit is an enh_coded_block_pattern bit for a combined 8x8 block of enhancement level. This selected bit can be considered as a refinement bit. It should be noted that the 8x8 block is used as an example for explanation. Obviously, you can use other blocks with different sizes.

Step 2. Bearing in mind the number of nonzero 8x8 brightness blocks and the cbp color block at the basic level, there are 9 combinations shown in Table 12 below. Each combination is a context for decoding the rest of the information about enh_coded_block_pattern. In table 12, cbpbC stands for cbp of the base level chroma, and (cbpb, Y (b8) represents the number of non-zero 8x8 base-level brightness blocks. The cbpeC and cbpeY columns show the new cbp format for unencrypted information about enh_coded_block_pattern, excluding contexts 4 and 9. In cbpeY ““ x ”stands for one bit for an 8x8 block of brightness, while in cbpeC“ xx ”stands for 0, 1, or 2.

Code tables for decoding enh_coded_block_pattern based on various contexts are defined below in tables 13 and 14.

Step 3. For contexts 4 and 9, enh_chroma_coded_block_pattern (which can be shortened to enh_chroma_cbp) is decoded separately using the code dictionary in table 15, shown below.

Code dictionaries for different contexts are shown in tables 13 and 14. These code dictionaries are based on statistical information on 27 decoded MPEG2 sequences.

Table 13
Huffman codewords for context 1-3 for enh_coded_block_pattern (enh_cbp) context 1 context 2 context 3 symbol the code enh_cbp the code enh_cbp the code enh_cbp 0 10 0 eleven 0 0 one one 001 one 00 3 10 0 2 110 four one hundred one 111 3 3 1110 2 011 2 1101 2 four 0001 3 1011 four 1100 0 four 5 0100 5 0101 7 1100 1 5 6 0000 6 1010 0 5 7 1100 7 1010 1 6 8 0101 8 0 100 0 8 9 1101 10 10 0100 10 eleven 10 1111 00 12 0100 111 10 eleven 1101 11 fifteen 0100 110 9 12 1111 01 9 13 1111 110 eleven fourteen 1111 111 13 fifteen 1111 101 fourteen 16 1101 011 16 17 1101 001 23 eighteen 1101 0101 17 19 1111 1000 eighteen twenty 1101 0000 19 21 1111 1001 twenty 22 1101 0100 21 23 1101 0001 22

Table 14
Huffman codewords for context 5-7 for enh_coded_block_pattern (enh_cbp) context 5 context 6 context 7 context 8 symbol the code enh_cbp the code enh_cbp the code enh_cbp the code enh_cbp 0 one 0 01 0 10 0 0 0 one 0000 four 101 one 00 one one one 2 0010 8 001 2 01 2 3 0111 0 one one hundred four eleven 3 four 0101 0 10 000 5 5 0001 0 eleven 110 7 6 0101 1 12 1110 3 7 0011 1 13 1111 6 8 0001 1 fourteen 9 0110 1 fifteen 10 0111 1 2 eleven 01100 3 12 0100 1 5 13 0011 0 7 fourteen 0100 00 6 fifteen 0100 01 9

Step 3. For contexts 4-9, enh_cbp colors can be decoded separately using the codebook shown in Table 15 below.

Table 15
Codeword for enh_chroma_block_pattern (enh_chroma_cbp) enh_chroma_cbp the code 0 0 one 10 2 eleven

The process of obtaining quantization parameters

The following describes the process of obtaining quantization parameters (QP _S ). The syntax element mb_qp_delta for each macroblock transmits the QP of the macroblock. Base Level Nominal QP (QPb) also represents the QP used to quantize at the base level specified using mb_qp_delta in the macroblocks in base_layer_slice. The nominal enhancement level QP (QPe) is also the QP that is used to quantize the enhancement level specified using mb_qp_delta in enh_macroblock_layer. When receiving a QP to save a bit instead of sending mb_qp_delta for each macroblock of the enhancement layer, the QP difference between the base layer and the enhancement layer can be kept constant. In this way, the difference mb_qp_delta for QP between the two layers is sent only on a frame basis.

Based on QP _b and QP _{e, the} difference QP, designated as delta_layer_qp, is defined as:

delta_layer_qp = QP _b -QP _e

The QP _eY for QP quantization used for the enhancement level is obtained based on two factors: (a) the existence of nonzero coefficient levels at the base level and (b) delta_layer_qp. To support a single dequantization operation for improvement level coefficients, the delta_layer_qp parameter can be limited, so delta_layer_qp% 6 = 0. With these two given values, QP is obtained as follows:

1. If the combined base-level MB does not have non-zero coefficients, then the nominal QPe will be used, since only the improvement level coefficients will need to be decoded.

Q _Pe.Y = QP _e

2. If delta_layer_qp% 6 = 0, then QPe is still used for the improvement level, regardless of whether or not there are non-zero coefficients. This is based on the fact that the size of the quantization step doubles with each QP increment of 6.

The following operation describes the process of de-quantization (denoted as Q-1) to combine the coefficients of the base level and the level of improvement, defined as Cb and Ce, respectively.

F _e = Q ^-1 ((Cb (QP _b ) << (delta_layer_qp / 6)) + C _e (QP _e ))

where F _e denotes the coefficients of the level of improvement after dequantization, and Q ^-1 indicates the function of dequantization.

If a nearby base level block has a nonzero coefficient and delta_layer_qp% 6 ≠ 0, then when decanting the coefficients of the basic level and the improvement level, QP _b and QP _{e are used,} respectively. The coefficients of the improvement level are obtained as follows:

Fe = Q ^-1 (C _b (QP _b )) + Q ^-1 (C _e (QP _e ))

Obtaining the QP color parameters (QP _{base, C} and QP _{enh, C} ) is based on the QP brightness parameters (QP _{b, Y} and QP _{e, Y} ). First calculate qP _I as follows:

qP _I = Clip3 (0, 51, QP _xY + chroma_qp_index_offset)

where x stands for “b” for the base level or “e” for the enhancement level, chroma_qp_index_offset is defined in the image parameter set, and Clip3 is the following mathematical function

The value of QP _{x, C} can be determined from table 16 below.

Table 16
Description of QP _xC as a function of qP _I qP _I <30 thirty 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 fifty 51 QP _xC qP _I 29th thirty 31 32 32 33 34 34 35 35 36 36 37 37 37 38 38 38 39 39 39 39

For video data of the enhancement level, the QP parameters for MB obtained during dequantization are used for release.

Release

When releasing for all edges of a 4x4 block of a frame, except for the edges at the frame boundary and any edges for which the release filter is disabled by the disable_deblocking_filter_idc parameter, a release filter can be used. The specified filtering process is performed on the basis of macroblocks (MB) after completing the process of constructing a frame with all macroblocks in the frame, processed in ascending order of macroblock addresses.

In FIG. 13 is a diagram illustrating the operation of a luminance and chroma deblocking filter. The release filter is triggered separately for the luminance and chrominance components. For each macroblock, the vertical edges are first filtered from left to right, and then the horizontal edges are filtered from top to bottom. For a 16x16 macroblock, a luminance release filter operates at four edges with 16 samples, and a deblocking filter for each color component operates at two edges with 8 samples, in the horizontal direction and in the vertical direction, for example, as shown in FIG. 13. The brightness boundaries in the macroblock to be filtered are shown in FIG. 13 solid lines. The color boundaries in the macroblock to be filtered are shown in FIG. 13 dashed lines.

In FIG. 13, reference numerals 170, 172 show vertical edges for filtering luminance and chroma, respectively. Reference numerals 174, 176 indicate respectively horizontal edges for filtering luminance and chrominance. Sample values above and to the left of the current macroblock that have already been modified as a result of the release filter in previous macroblocks are used as input to the release filter in the current macroblock and can be further modified during filtering of the current macroblock. Sampled values modified during filtering of vertical edges are used as input for filtering horizontal edges for the same macroblock.

In the H.264 standard, the MB modes, the number of non-zero levels of the transform coefficient and the motion information are used to decide on the border filtering intensity. The macroblock QP parameters are used to obtain a threshold value that indicates whether input samples are filtered. When releasing a basic level, these pieces of information are used directly. Information is generated for the video enhancement layer. In this example, the filtering process is applied to a set of eight samples across the horizontal or vertical edge of a 4x4 block, denoted as pi and qi, where i is 0, 1, 2, or 3, as shown in FIG. 14, with edge 178 being between p0 and q0. In FIG. 14 pi and qi are set for i = from 0 to 3.

To decode the I-frame of the enhancement layer, a decoded base layer I-frame and the addition of an inter-layer predicted remainder may be required. For the reconstructed base layer I-frame, a deblocking filter is used before predicting the enhancement level I-frame. Applying the standard method for releasing an I-frame to release an I-frame of an enhancement layer may not be desirable. As an alternative, the following criteria can be used to obtain the desired border filtering intensity (bS). The variable bS can be obtained as follows. BS is set to 2 if any of the following conditions is true:

but. A 4x4 luminance block containing the p0 sample contains nonzero transform coefficient levels and is located in a macroblock encoded using the prediction mode for a 4x4 macroblock with intraframe coding; or

b. A 4x4 luminance block containing a q0 sample contains non-zero transform coefficient levels and is located in a macroblock encoded using the prediction mode for a 4x4 macroblock with intraframe coding.

If none of the above conditions is met, then the value of bS is set equal to 1.

For P-frames, the residual information of macroblocks with intraframe coding, in addition to missing macroblocks, can be encoded both at the basic level and at the enhancement level. Due to a single decoding, the coefficients from these two levels are combined. Since the number of nonzero levels of the transform coefficient is used to decide on the intensity of the border filtering during release, it is important to determine how to calculate the number of nonzero levels of transform coefficients for each 4x4 block at the improvement level used in the release. Unreasonable increase or decrease in this amount can either lead to excessive smoothing of the image, or cause the phenomenon of "blockiness" of the image. The bS variable is obtained as follows:

1. If the edge of the block is also the edge of the macroblock, and both samples p ₀ and q ₀ are in the frame macroblocks and any of the samples p ₀ or q ₀ is in the macroblock encoded using the prediction mode for the macroblock with intraframe coding, then the value of bS equal to 4.

2. Otherwise, if any of the samples p ₀ or q ₀ is in the macroblock encoded using the prediction mode for the macroblock with intraframe coding, then the value for bS is 3.

3. Otherwise, if at the basic level, a 4x4 luminance block containing a p ₀ sample or a 4x4 luminance block containing a q ₀ sample contains nonzero conversion coefficient levels, or at an improvement level, a 4x4 luminance block containing a p ₀ sample or block 4x4 brightness, containing the q ₀ sample, contains non-zero levels of the conversion coefficient, then the value for bS is CP2.

4. Otherwise, a value of 1 is displayed for bS, or the standard approach is used alternatively.

Channel Switch Frames

The channel switching frame may be encapsulated in one or more NAL components with information for further improvement (SEI), and it may be called a channel switching frame (CSF) with SEI. In one example, the CSF frame with SEI has a payloadTypefield of 22. The RBSP syntax for the SEI message is specified as in clause 7.3.2.3 of the H.264 standard. The syntax of the RBI SEI and CSF messages with the SEI may be provided as shown in Tables 17 and 18 below.

Table 17
Syntax SEI RBSP sei_rbsp () { FROM Descriptor do sei message () 5 while (more_rbsp_data ()) rbsp_trailing_bits () 5 }

Table 18
Message Syntax SEI CSF sei_message () { FROM Descriptor 22 / * payloadType * / 5 f (8) payloadType = 22 payloadSize = 0 while (next_bits (8) = = OxFF) { ff_byte / * equal to OxFF * / 5 f (8) payloadSize + = 255 } last_payload_size_byte 5 u (8) payloadSize + = last_payload_size_byte channel_switch_frame_slice_data 5 }

The syntax of the data of the layer of the channel switching frame may be identical to the syntax of the I-layer or P-layer of the basic level, which is specified in clause 7 of the H.264 standard. A Channel Switching Frame (CSF) may be encapsulated in an independent transmission protocol packet, allowing visibility at random access points in an encoded bitstream. There are no level restrictions for transmitting a channel switching frame. This frame may be contained either at the base level or at the enhancement level.

When decoding a channel switching frame, if a channel change request is triggered, the channel switching frame in the requested channel will be decoded. If the channel switching frame is contained in the SEI CSF message, then the decoding process used for the base layer I layer will be used to decode the CSI SEI. The P-layer matching the CSI SEI will not be decoded, and the B-image with the output order before the channel switching frame is discarded. The decoding process of future images (in terms of output order) does not change.

In FIG. 15 is a block diagram illustrating an apparatus 180 for transmitting scalable digital video data with a variety of different exemplary syntax elements to support fairly simple video scalability. The device 180 includes a module 182 for incorporating base layer video data into a first NAL component, a module 184 for incorporating enhancement layer video data into a second NAL component, and a module 186 for including one or more syntax elements in at least one of the first and second components NAL to indicate the presence of an enhancement level in the second NAL component of video data. In one example, device 180 may form part of a broadcast server 12, as shown in Figures 1 and 3, and may be implemented in hardware, software, or firmware or using any suitable combination thereof. For example, module 182 may include one or more aspects of the base layer encoder 32 and the NAL component module 23 of FIG. 3, which encode the base layer video data and input it into the NAL component. In addition, module 184, for example, may include one or more aspects of the enhancement layer encoder 34 and the NAL component module 23 that encode the enhancement layer video data and insert them into the NAL component. Module 186 may include one or more aspects of a NAL component module 23, which includes one or more syntax elements in at least one of the first and second NAL components to indicate the presence of enhancement level video data in the second NAL component. In one example, one or more syntax elements are provided in a second NAL component in which video data of enhancement level is provided.

In FIG. 16 is a flowchart illustrating a digital video decoding apparatus 188 that decodes a scalable video data bitstream for processing a plurality of various exemplary syntax elements supporting fairly simple video scalability. Digital video decoding device 188 may reside in a subscriber device, for example, subscriber device 16 of FIG. 1 or FIG. 3, video decoder 14 of FIG. 1 and may be implemented in hardware, software, or firmware, or using any suitable combination thereof. Apparatus 188 includes a module 190 for receiving base layer video data in a first NAL component, module 192 for receiving extension layer video data in a second NAL component, module 194 for receiving one or more syntax elements in at least one of the first and second components NAL, to indicate the presence of an improvement level in the second NAL component, and a module 196 for decoding digital video in the second NAL based on an indication provided by one or more syntax elements in the second NAL unit. In one aspect, one or more syntax elements are provided in a second NAL component, where enhancement level video data is provided. Module 190, for example, may include a receiver / demodulator 26 of subscriber unit 16 of FIG. 3. In this example, module 192 may also include a receiver / demodulator 26. In some exemplary configurations, module 194 may include a NAL component module, such as the NAL component module 27 of FIG. 3, which processes syntax elements in NAL components. Module 196 may include a video decoder, such as video decoder 28 of FIG. 3.

The methods described herein can be implemented in hardware, software, hardware-software, or any combination thereof. When implemented by software, these methods can be implemented at least partially by one or more stored or transmitted commands, or code, on a computer-readable medium. A computer-readable medium may include a computer storage medium, a transmission medium, or both, and may also include any medium supporting transfer of a computer program from one place to another. The storage medium may be any available medium that can be accessed by a computer.

For example, but not as a limitation, such a computer-readable medium may comprise random-access memory (RAM), such as synchronous random-access memory (SDRAM), read-only memory (ROM), non-volatile random-access memory (NVRAM), ROM, Read-Only Electrically Erasable Programmable Memory (EEPROM), EEPROM, Flash Memory, CD-ROM, or Other Optical Disk Storage, Magnetic Disk Storage or Other Magnetic Storage Devices -keeping, or any other medium which can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

Also, any connection will correctly be assigned to computer-readable media. For example, if the software is transferred from a Web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL line or wireless technologies such as infrared, radio and microwave, are included in the definition of media. As used herein, the term “disc” includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk or a laser disc operating in the blue region of the spectrum where discs typically reproduce data from using magnetic properties, while discs reproduce data using optical phenomena, such as using lasers. Combinations of the above should also be included within the scope of the term “computer-readable medium”.

The code associated with a computer-readable medium of a computer program product can be executed by a computer, for example, one or more processors, for example, one or more digital signal processors (DSP), general-purpose microprocessors, application-specific specialized integrated circuits (ASICs), logic arrays programmed by the user, or other equivalent integrated or discrete logic circuits. According to some aspects, the functionality described herein may be provided in specialized software modules or hardware modules configured for encoding and decoding, or included in an integrated video encoder-decoder (CODEC).

Various aspects of the invention have been described herein. These and other aspects are within the scope of the following claims.

Claims (49)

1. A method for transmitting scalable digital video data, the method comprising the steps of:
include enhancement layer video data in a network abstraction layer (NAL) component;
include one or more syntax elements in the NAL component to indicate whether the NAL component includes enhancement level video data and one or more syntax elements to indicate at least one of:
whether the video data enhancement level in the NAL component includes video data with intraframe coding;
whether the NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of a reference image layer; and
the number of non-zero coefficients with a value greater than one in the blocks of intraframe coding in the video data of the enhancement level. 2. The method according to claim 1, further comprising the step of including one or more syntax elements in the NAL component to indicate the data structure type of the raw byte serial payload (RBSP) for the enhancement level data in the NAL component. 3. The method according to claim 1, in which the NAL component is the first NAL component, the method further comprising the steps of including base-level video data in the second NAL component and including one or more syntax elements in at least one of the first and the second NAL component, to indicate whether the decoder should use the addition of a pixel region or a transform region of the enhancement layer video data to the base layer video data. 4. The method according to claim 1, in which the NAL component is the first NAL component, the method further comprising the steps of including base-level video data in the second NAL component and including one or more syntax elements in at least one of the first and the second NAL component, to indicate whether the video data of the enhancement layer includes any residual data related to the video data of the base layer. 5. The method according to claim 1, further comprising the step of including one or more syntax elements in a NAL component for identifying blocks in the enhancement layer video data containing syntax elements with non-zero transform coefficients. 6. The method according to claim 1, further comprising the step of including one or more syntax elements in a NAL component to indicate coded block patterns for inter-frame encoded blocks in the enhancement layer video data. 7. The method according to claim 1, wherein the NAL component is a first NAL component, the method further comprising adding base layer video data to a second NAL component, and wherein the enhancement layer video data is encoded to increase the signal-to-noise ratio of the base layer video data . 8. The method according to claim 1, wherein the step of including one or more syntax elements in the NAL component to indicate whether the NAL component includes enhancement level video data comprises the step of setting the type parameter of the NAL component in the NAL to the selected value to indicate that the NAL component includes video data of the enhancement level. 9. A device for transmitting scalable digital video data, the device comprising:
a network abstraction layer (NAL) component module that includes enhancement layer video data in a NAL component and includes one or more syntax elements in a NAL component to indicate whether the NAL component includes enhancement layer video data;
moreover, the NAL component module includes one or more syntax elements in the NAL component to indicate at least one of:
whether the video data enhancement level in the NAL component includes video data with intraframe coding;
whether the NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of a reference image layer; and
the number of non-zero coefficients with a value greater than one in blocks with intraframe coding in video data of the enhancement level. 10. The device according to claim 9, in which the NAL component module includes one or more syntax elements in the NAL component to indicate the data structure type of the raw byte sequential payload (RBSP) data level enhancement in the NAL component. 11. The device according to claim 9, in which the NAL component is the first NAL component, wherein the NAL component module includes base layer video data in the second NAL component, and wherein the NAL component module includes one or more syntax elements in at least one of the first and second NAL components to indicate whether the decoder should use the addition of a pixel region or a transform region of the enhancement layer video data to the base layer video data. 12. The device according to claim 9, in which the NAL component is the first NAL component, wherein the NAL component module includes base layer video data in the second NAL component, and wherein the NAL component module includes one or more syntax elements in at least one of the first and second NAL components, to indicate whether the video data of the enhancement layer includes any residual data related to the video data of the base layer. 13. The device according to claim 9, in which the NAL component module includes one or more syntax elements in the NAL component to identify blocks in the video level enhancement containing syntax elements with non-zero transform coefficients. 14. The device according to claim 9, in which the NAL component module includes one or more syntax elements in the NAL component for indicating coded block patterns for inter-frame encoded blocks in the enhancement layer video data. 15. The device according to claim 9, in which the NAL component is the first NAL component, wherein the NAL component module includes base layer video data in the second NAL component, and in which the encoder encodes the enhancement layer video data to increase the signal-to-noise ratio of the base layer video data. 16. The device according to claim 9, in which the NAL component module sets the parameter type of the NAL component in the NAL component to the selected value to indicate that the NAL component includes video data of the enhancement level. 17. A processor for transmitting scalable digital video data, wherein the processor is configured to include enhancement layer video data in a network abstraction layer (NAL) component, include one or more syntax elements in a NAL component to indicate whether the NAL component includes enhancement layer video data, and enable one or more syntax elements in a NAL component to indicate at least one of:
whether the video data enhancement level in the NAL component includes video data with intraframe coding;
whether the NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of a reference image layer; and
the number of non-zero coefficients with a value greater than one in blocks with intraframe coding in video data of the enhancement level. 18. A device for transmitting scalable digital video data, the device comprising:
means for incorporating video data of the enhancement layer into a component of a network abstraction layer (NAL); and
means for including one or more syntax elements in the NAL component to indicate whether the NAL component includes video data of the enhancement level, and one or more syntax elements to indicate at least one of:
whether the video data enhancement level in the NAL component includes video data with intraframe coding;
whether the NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of a reference image layer; and
the number of non-zero coefficients with a value greater than one in blocks with intraframe coding in video data of the enhancement level. 19. The apparatus of claim 18, further comprising means for including one or more syntax elements in the NAL component for indicating a data structure type of an unprocessed byte sequential payload (RBSP) of enhancement level data in the NAL component. 20. The device according to p, in which the NAL component is the first NAL component, and the device further comprises means for incorporating basic level video data into the second NAL component and means for including one or more syntax elements in at least one of the first and the second component of the NAL, to indicate whether the decoder should use the addition of the pixel region or the area of the conversion of the video data of the enhancement level to the video data of the base level. 21. The device according to p, in which the NAL component is the first NAL component, and the device further comprises means for incorporating basic level video data into the second NAL component and means for including one or more syntax elements in at least one of the first and the second NAL component to indicate whether the video data of the enhancement layer includes any residual data related to the video data of the base layer. 22. The apparatus of claim 18, further comprising means for including one or more syntax elements in a NAL component for identifying blocks in enhancement level video data containing syntax elements with non-zero transform coefficients. 23. The apparatus of claim 18, further comprising means for including one or more syntax elements in a NAL component for indicating coded block patterns for inter-frame encoded blocks in the enhancement layer video data. 24. The apparatus of claim 18, wherein the NAL component is a first NAL component, the device further comprising means for incorporating base layer video data into a second NAL component, and wherein the enhancement layer video data increases the signal-to-noise ratio of the base layer video data. 25. The device according to p, in which the means for including one or more syntax elements in the NAL component to indicate whether the NAL component includes video data of the enhancement level, comprises means for setting the type parameter of the NAL component in the NAL component to the selected value to indicate that the NAL component includes video data level enhancement. 26. A computer-readable medium on which a computer program product is stored, the computer program product comprising computer-executable codes for causing a computer to implement a method for transmitting scalable digital video data, comprising:
code to include video data of the enhancement layer in a component of the network abstraction layer (NAL); and
code to include one or more syntax elements in the NAL component to indicate whether the NAL component includes video data of the enhancement level, and one or more syntax elements to indicate at least one of:
whether the video data enhancement level in the NAL component includes video data with intraframe coding;
whether the NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of a reference image layer; and
the number of non-zero coefficients with a value greater than one in blocks with intraframe coding in video data of the enhancement level. 27. A method for processing scalable digital video data, the method comprising the steps of:
receiving enhancement layer video data in a network abstraction layer (NAL) component;
receiving one or more syntax elements in the NAL component to indicate whether the NAL component includes enhancement layer video data;
decode the digital video data in the NAL component based on the indication; and
detect one or more syntax elements in the NAL component to determine at least one of:
whether the enhancement level video data in the NAL component includes intra-frame encoded video data, and
the number of non-zero coefficients with a value greater than one in blocks with intraframe coding in video data of the enhancement level. 28. The method of claim 27, further comprising detecting one or more syntax elements in the NAL component to determine the data structure type of the raw byte serial payload (RBSP) data of the enhancement layer in the NAL component. 29. The method of claim 27, wherein the NAL component is a first NAL component, the method further comprising the steps of:
receiving base layer video data in a second NAL component;
detecting one or more syntax elements in at least one of the first and second NAL components to determine if the enhancement layer video data includes any residual data related to the base layer video data; and
skip decoding of the enhancement layer video data if it is determined that the enhancement layer video data does not include any residual data related to the base layer video data. 30. The method according to item 27, in which the NAL component is the first NAL component, and the method further comprises the steps of:
receiving base layer video data in a second NAL component;
detecting one or more syntax elements in at least one of the first and second NAL components to determine whether the first NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of the reference image layer;
detecting one or more syntax elements in at least one of the first and second NAL components to identify blocks in enhancement level video data containing syntax elements with non-zero transform coefficients; and
detecting one or more syntax elements in at least one of the first and second NAL components to determine whether to add a pixel region or a video transform region of the enhancement layer to the base layer video for decoding digital video data. 31. The method of claim 27, further comprising detecting one or more syntax elements in the NAL component to determine coded block patterns for inter-frame encoded blocks in the enhancement layer video data. 32. The method according to item 27, in which the NAL component is the first NAL component, and the method further comprises the step of including the base layer video data in the second NAL component, and wherein the enhancement layer video data is encoded to increase the signal-to-noise ratio of the base layer video data . 33. The method of claim 27, wherein the step of taking one or more syntax elements in the NAL component to indicate whether the NAL component includes enhancement level video data, comprises the step of taking the type parameter of the NAL component in the NAL component, which is set to the selected value to indicate that the NAL component includes video data of the enhancement level. 34. A device for processing scalable digital video data, the device comprising:
a network abstraction layer (NAL) component module that receives the enhancement layer video data into the NAL component and receives one or more syntax elements into the NAL component to indicate whether the NAL component includes the enhancement layer video data; and
a decoder that decodes digital video data in the NAL component based on the indication;
moreover, the NAL component module detects one or more syntax elements in the NAL component to determine at least one of:
whether the enhancement level video data in the NAL component includes intra-frame encoded video data, and
the number of non-zero coefficients with a value greater than one in blocks with intraframe coding in video data of the enhancement level. 35. The device according to clause 34, in which the NAL component module detects one or more syntax elements in the NAL component to determine the type of data structure of the raw byte serial payload (RBSP) data level enhancement in the NAL component. 36. The device according to clause 34, in which the NAL component is the first NAL component, and the NAL component module receives base level video data in the second NAL component, and the NAL component module detects one or more syntax elements in at least one of the first and second NAL components to determine whether the enhancement layer video data includes any residual data related to the base layer video data, and the decoder skips decoding the enhancement layer video data if it is determined that in deodannye enhancement layer does not comprise any residual data pertaining to video data baseline. 37. The device according to clause 34, in which the NAL component is the first NAL component, and the NAL component module:
receives base layer video data in a second NAL component;
detects one or more syntax elements in at least one of the first and second NAL components to determine whether the first NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of the reference image layer;
detects one or more syntax elements in at least one of the first and second NAL components for identifying blocks in the video data of the enhancement level containing syntax elements with non-zero transform coefficients; and
detects one or more syntax elements in at least one of the first and second NAL components to determine whether to use the addition of a pixel region or a transform region of the enhancement layer video data to the base layer video to decode digital video data. 38. The device according to clause 34, in which the NAL processing module detects one or more syntax elements in the NAL component to determine the encoded block patterns for blocks with inter-frame coding in the video data level enhancement. 39. The apparatus of claim 34, wherein the NAL component is a first NAL component, wherein the NAL component module includes base layer video data in a second NAL component, and wherein the enhancement layer video data is encoded to enhance the signal-to-noise ratio of the base layer video data. 40. The device according to clause 34, in which the NAL component module accepts the parameter type of the NAL component in the NAL component, which is set equal to the selected value to indicate that the NAL component includes video data level enhancement. 41. A processor for processing scalable digital video data, the processor being configured to:
receiving video data of the enhancement layer in a component of a network abstraction layer (NAL);
receiving one or more syntax elements in the NAL component to indicate whether the NAL component includes video data of the enhancement level and one or more syntax elements to indicate at least one of:
whether the video data enhancement level in the NAL component includes video data with intraframe coding;
whether the NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of a reference image layer; and
the number of non-zero coefficients with a value greater than one in blocks with intraframe coding in the video data of the enhancement level; and
decoding digital video data in a NAL component based on the indication. 42. A device for processing scalable digital video data, the device comprising:
means for receiving video data of the enhancement layer in a network abstraction layer (NAL) component;
means for receiving one or more syntax elements in the NAL component to indicate whether the NAL component includes video data of the enhancement level;
means for decoding digital video data in a NAL component based on the indication; and
means for detecting one or more syntax elements in the NAL component to determine at least one of:
whether the enhancement level video data in the NAL component includes intra-frame encoded video data, and
the number of non-zero coefficients with a value greater than one in blocks with intraframe coding in video data of the enhancement level. 43. The device according to § 42, further comprising a means for detecting one or more syntax elements in the NAL component to determine the type of data structure of the raw byte sequential payload (RBSP) data of the enhancement level in the NAL component. 44. The device according to § 42, in which the NAL component is the first NAL component, and the device further comprises:
means for receiving base level video data in a second NAL component;
means for detecting one or more syntax elements in at least one of the first and second NAL components to determine whether the enhancement layer video data includes any residual data related to the base layer video data; and
means for skipping decoding of the video data of the enhancement layer, if it is determined that the video data of the enhancement layer does not include any residual data related to the base layer video data. 45. The device according to § 42, in which the NAL component is the first NAL component, and the device further comprises:
means for receiving base level video data in a second NAL component;
means for detecting one or more syntax elements in at least one of the first and second NAL components to determine whether the first NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of the reference image layer;
means for detecting one or more syntax elements in at least one of the first and second NAL components for identifying blocks in enhancement level video data containing syntax elements with non-zero transform coefficients; and
means for detecting one or more syntax elements in at least one of the first and second NAL components to determine whether to add the enhancement layer video data to the base layer video data in the pixel region or in the transform domain to decode the digital video data. 46. The device according to § 42, further comprising means for detecting one or more syntax elements in the NAL component for determining encoded block patterns for inter-frame encoded blocks in the video data of the enhancement level. 47. The device according to § 42, in which the NAL component is the first NAL component, the device further comprising means for incorporating base layer video data into the second NAL component, and wherein the enhancement layer video data is encoded to enhance the signal-to-noise ratio of the base layer video data. 48. The device according to § 42, in which the means for receiving one or more syntax elements in the NAL component for indicating whether the corresponding NAL component includes video data of the enhancement level, comprises means for receiving a parameter type of the NAL component in the NAL component, which is set equal to a selected value to indicate that the NAL component includes enhancement level video data. 49. A computer-readable medium on which a computer program product is stored, the computer program product comprising computer-executable codes for causing a computer to implement a method for processing scalable digital video data, comprising:
code for receiving video data of the enhancement level in a component of the network abstraction layer (NAL); and
code for receiving one or more syntax elements in a NAL component
to indicate whether the NAL component includes video data of the enhancement level, and one or more syntax elements to indicate at least one of:
whether the video data enhancement level in the NAL component includes video data with intraframe coding;
whether the NAL component includes a sequence parameter, a set of image parameters, a reference image layer, or a data section of a reference image layer; and
the number of non-zero coefficients with a value greater than one in blocks with intraframe coding in the video data of the enhancement level; and
code for decoding digital video data in a NAL component based on the indication.

Images