JP6768096B2 - Image coding method and image decoding device - Google Patents

Description

The present invention relates to an image coding method and an image coding device.

H. In the conventional image coding method represented by the ITU-T standard called 26x and the ISO / IEC standard called MPEG-x, the encoder and the decoder are operated in order to operate the encoder and the decoder in synchronization. As a framework for ensuring compatibility with the above, the provision of conformity (standard conformity) is defined (see P.214 to P.226 of Non-Patent Document 2).

In addition, a concept called a virtual reference decoder (HRD) that virtually models the buffer management of the decoder has been introduced. By using the HRD, it is possible to prevent a decoder failure such as an underflow in which the image data is not contained in the buffer at the decoding timing and an overflow in which the image data exceeds the prepared buffer size.

Specifically, the virtual stream scheduler (Hypostical Stream Scheduler: HSS) manages the input of the bitstream to the HRD coded picture buffer (CPB). There are two input methods, a fixed bit rate and a variable bit rate.

In the case of a constant bit rate, a bit stream is always input to the CPB of the HRD at a constant bit rate. If you want to encode an image at a constant bit rate, you need to check for both underflow and overflow.

In the case of a variable bit rate, the bit rate for inputting the bit stream to the CPB of the HRD is variable. As a result, when there is no free space in the CPB capacity of the HRD, the bitstream input can be suspended, so that an overflow does not occur. Therefore, when encoding at a variable bit rate, only underflow needs to be checked.

Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11, 12th Meeting: Geneva, CH, 14-23. 2013, “High Efficiency Video Coding (HEVC) ext specialization draft 10 (for FDIS & Last Call)”, http: // phenix. int-every. fr / jct / dock_end_user / documents / 12_Geneva / wg11 / JCTVC-L1003-v34. zip Sakae Okubo et al., "H.265 / HEVC Textbook", Impress Japan, 2013

In such an image coding method, it is desired that the amount of processing for generating a bit stream satisfying conformity can be reduced.

Therefore, an object of the present invention is to provide an image coding method or an image coding apparatus capable of reducing the amount of processing for generating a bit stream satisfying conformance.

The image coding method according to one aspect of the present invention is an image coding method for generating a bit stream having time scalability by encoding a plurality of images, and the bit stream is encoded at a fixed bit rate. When the selection step of selecting whether to encode at a variable bit rate or when the bit stream is selected to be encoded at a constant bit rate, (1) a part of the bit stream and the plurality of Among the images, the first bit rate type of the bit stream including the encoded data of the plurality of images belonging to the uppermost time layer is set to a fixed bit rate, and (2) is a part of the bit stream and described above. A setting step of setting the second bit rate type of the sub-bit stream containing the encoded data of the images belonging to the time layer other than the uppermost time layer among the plurality of images to the variable bit rate, and each of the plurality of images. , A coding step that prohibits reference to an image whose time layer is higher than that of the image to be processed and encodes the data.

It should be noted that these general or specific embodiments may be realized in a recording medium such as a system, method, integrated circuit, computer program or computer readable CD-ROM, system, method, integrated circuit, computer program. And any combination of recording media may be realized.

The present invention can provide an image coding method or an image coding apparatus capable of reducing the amount of processing for generating a bit stream satisfying conformance.

FIG. 1 is a diagram showing an example of a bit stream having time scalability according to the first embodiment. FIG. 2 is a block diagram showing the structure of the image coding apparatus according to the first embodiment. FIG. 3 is a flowchart of the coding process according to the first embodiment. FIG. 4 is a diagram showing an example of a bitstream data structure according to the first embodiment. FIG. 5 is a diagram for explaining a bit rate type determination process according to the first embodiment. FIG. 6 is a flowchart of a calculation process for calculating the bit rate control information and the time scalability information according to the first embodiment. FIG. 7 is a diagram showing a CPB of HRD when encoded at a constant bit rate so as to satisfy the conformity according to the first embodiment. FIG. 8 is a diagram showing a CPB of HRD when the time resolution is reduced to 1/4 while maintaining the fixed bit rate according to the first embodiment. FIG. 9 is a diagram showing CPB of HRD when the time resolution is halved with the variable bit rate according to the first embodiment. FIG. 10 is an overall configuration diagram of a content supply system that realizes a content distribution service. FIG. 11 is an overall configuration diagram of the digital broadcasting system. FIG. 12 is a block diagram showing a configuration example of a television. FIG. 13 is a block diagram showing a configuration example of an information reproduction / recording unit that reads / writes information to / from a recording medium such as an optical disc. FIG. 14 is a diagram showing a structural example of a recording medium which is an optical disc. FIG. 15A is a diagram showing an example of a mobile phone. FIG. 15B is a block diagram showing a configuration example of a mobile phone. FIG. 16 is a diagram showing the structure of the multiplexed data. FIG. 17 is a diagram schematically showing how each stream is multiplexed in the multiplexed data. FIG. 18 is a diagram showing in more detail how the video stream is stored in the PES packet sequence. FIG. 19 is a diagram showing the structure of TS packets and source packets in the multiplexed data. FIG. 20 is a diagram showing a data structure of PMT. FIG. 21 is a diagram showing an internal configuration of multiplexed data information. FIG. 22 is a diagram showing an internal configuration of stream attribute information. FIG. 23 is a diagram showing steps for identifying video data. FIG. 24 is a block diagram showing a configuration example of an integrated circuit that realizes the moving image coding method and the moving image decoding method of each embodiment. FIG. 25 is a diagram showing a configuration for switching the drive frequency. FIG. 26 is a diagram showing steps for identifying video data and switching the drive frequency. FIG. 27 is a diagram showing an example of a look-up table in which a standard of video data and a drive frequency are associated with each other. FIG. 28A is a diagram showing an example of a configuration in which the module of the signal processing unit is shared. FIG. 28B is a diagram showing another example of the configuration in which the module of the signal processing unit is shared.

(Knowledge that became the basis of the present invention)
Image coding standard H. Image coding methods such as 265 / HEVC (see Non-Patent Document 1) have a function called time scalability. A time identifier called TemporalId is inserted into a bitstream having time scalability (see P.211 to P.212 of Non-Patent Document 2). This makes it possible for the image decoding device to output the decoded image at a plurality of time resolutions. For example, when the transmission rate is insufficient for a bitstream in which a moving image of 60 fps (frames per second) is encoded, the image encoding device (transmitter) drops the frame rate from the bitstream to 30 fps or 15 fps. By cutting out the sub-bit stream and transmitting the obtained sub-bit stream, it is possible to prevent frame dropping and deterioration of image quality in the image decoding device (receiver).

FIG. 1 is a diagram showing an example of a 60 fps bit stream to which TemporalId is added. The vertical axis represents TemporalId, and the horizontal axis represents the output order (POC: Picture Orderer Count) of pictures. Further, the picture at the source of the arrow can be used to generate a predicted image of the picture at the tip of the arrow. For example, for a picture with POC = 2, pictures with POC = 0 and POC = 4 can be used as reference images.

The bitstream in this example is encoded in three layers with different TemporalIds. This layer is called a sublayer or a time layer. When the time direction prediction of the current picture to be encoded or decoded is performed, there is a limitation that a picture having a value of TemporalId larger than the TemporalID of the current picture cannot be referred to. For example, when the picture with POC = 2 is the current picture, since TemporalId = 1 is added to the current picture, the pictures with POC = 1 and POC = 3 with TemporalId = 3 cannot be referred to.

This limitation allows the image encoding device to easily remove large TemporalId pictures from the bitstream when the time resolution is reduced. For example, the image encoding device removes the sublayer of TemporalId = 2 from the bitstream when generating the subbitstream of 30 fps, and removes the sublayer of TemporalId ≥ 1 from the bitstream when generating the subbitstream of 15 fps. In this way, since the frame rate can be converted without re-encoding, the load on the image encoding device (transmitting device) can be reduced.

However, when encoding a bit stream that satisfies the conformity at all time resolutions by this method, there is a problem that the amount of processing increases and the load on the image coding apparatus increases.

Specifically, when generating a normal (non-time-scalable) bitstream, the image encoding device may encode the bitstream so that the bitstream satisfies the conformity. On the other hand, in a bit stream having time scalability, a sub bit stream is generated by the above-mentioned clipping, and the sub bit stream is transmitted to the image decoding device. Therefore, in such a bitstream, not only the original bitstream containing the coding information of all the sublayers, but also each subbitstream generated from the bitstream needs to satisfy the conformity. Thus, it is difficult to encode a single bitstream so that the original bitstream and all of the multiple subbitstreams satisfy conformance, and to generate such a bitstream. In addition, the processing in the image coding apparatus becomes complicated and the processing amount greatly increases.

Therefore, the image coding method according to one aspect of the present invention is an image coding method that generates a bit stream having time scalability by encoding a plurality of images, and is the first from a fixed bit rate and a variable bit rate. When a selection step for selecting a 1-bit rate type, a determination step for determining a time layer for each of the plurality of images, and a fixed bit rate as the first bit rate type are selected, (1) belong to all time layers. The second bit rate type of the bit stream containing the encoded data of the plurality of images is set to a constant bit rate, and (2) a time layer of the uppermost layer of the plurality of images, which is a part of the bit stream. The setting step of setting the third bit rate type of the sub-bit stream containing the encoded data of the images belonging to the time layers other than the variable bit rate to the variable bit rate, and each of the plurality of images has a higher time layer than the image to be processed. A coding step for encoding by prohibiting reference to, time scalability information indicating the time layer of the plurality of images, bit rate control information indicating the second bit rate type and the third bit rate type, and the like. A generation step of generating the bitstream including the encoded plurality of images is included.

This sets the bitrate type of the subbitstream to variable bitrate, even if the bitrate type of the bitstream containing the images of all time layers is set to constant bitrate. As a result, when the bitstream is generated, it is not necessary to consider the overflow of the sub-bitstream, and it is only necessary to control the bitstream including the images of all time layers so that the overflow does not occur. Thereby, the image coding method can reduce the processing amount of the image coding apparatus when encoding a bit stream having a plurality of time resolutions and satisfying the conformity at each time resolution.

For example, in the coding step, a virtual reference decoder that virtually models the buffer management of the image decoding device is used so that the image decoding device can process the bit stream without failure at the second bit rate type. The plurality of images may be encoded.

For example, the time scalability information may include a time identifier indicating the time layer to which each of the plurality of images belongs, or a number of patterns having a time resolution that identifies one or more time layers to be decoded.

Further, the image coding device according to one aspect of the present invention is an image coding device that generates a bit stream having time scalability by encoding a plurality of images, and is the first from a fixed bit rate and a variable bit rate. When a selection unit that selects a 1-bit rate type, a determination unit that determines the time layer of each of the plurality of images, and a fixed bit rate are selected as the first bit rate type, (1) belongs to all time layers. The second bit rate type of the bit stream containing the coded data of the plurality of images is set to a fixed bit rate, and (2) a time layer of the uppermost layer of the plurality of images, which is a part of the bit stream. The setting unit that sets the third bit rate type of the subbit stream containing the encoded data of the images belonging to the time layers other than the variable bit rate to the variable bit rate, and each of the plurality of images has an image whose time layer is higher than that of the image to be processed. A coding unit for encoding by prohibiting reference to the above, time scalability information indicating the time layer of the plurality of images, bit rate control information indicating the second bit rate type and the third bit rate type, and the like. It includes a generation unit that generates the bit stream including the plurality of encoded images.

It should be noted that these comprehensive or specific embodiments may be realized in a recording medium such as a system, method, integrated circuit, computer program or computer readable CD-ROM, system, method, integrated circuit, computer program. And any combination of recording media may be realized.

Hereinafter, embodiments will be described in detail with reference to the drawings. In addition, all the embodiments described below show comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, the order of steps, etc. shown in the following embodiments are examples, and are not intended to limit the present invention. Further, among the components in the following embodiments, the components not described in the independent claims indicating the highest level concept are described as arbitrary components.

(Embodiment 1)
The image coding apparatus according to the present embodiment sets the bit rate type of the sub bit stream to a variable bit rate regardless of the bit rate type of the original bit stream including the coding information of all the sub layers. As a result, the image encoding device can control the bit rate without considering the overflow of the sub-bit stream. Therefore, the processing amount of the image coding apparatus can be reduced.

First, the configuration of the image coding device 200 according to the present embodiment will be described. FIG. 2 is a block diagram showing the structure of the image coding apparatus 200 according to the present embodiment.

The image coding device 200 creates a bit stream 261 which is a coded bit stream by encoding an input image 251 which is an input moving image or an image bit stream for each block. As shown in FIG. 2, the image coding device 200 includes a bit rate control unit 201, a time scalability information addition unit 202, a subtractor 203, a conversion unit 204, a quantization unit 205, and an inverse quantization unit. It includes 206, an inverse conversion unit 207, an adder 208, a block memory 209, a frame memory 210, an intra prediction unit 211, an inter prediction unit 212, and an entropy coding unit 213.

The bit rate control unit 201 controls the quantization unit 205 using the HRD in order to generate a bit stream 261 that satisfies the conformity. Further, the bit rate control information 263 is output to the entropy encoding unit 213.

The bit rate control information 263 is information indicating each bit rate type of the bit stream having selectable time resolutions (60 fps, 30 fps, and 15 fps in the example shown in FIG. 1). Here, the bit rate type is a fixed bit rate or a variable bit rate. For example, the bit rate control information 263 is the flag information cbr_flag [TemporalId].

cbr_flag [TemporalId] contains as many flags as there are selectable time resolutions. Here, the TemporalId in cbr_flag [TemporalId] shows the largest value among the TemporalIds of the reproduced picture. For example, in the example shown in FIG. 1, when all the pictures at 60 fps are reproduced, the largest TemporalId among the TemporalIds of the reproduced pictures is "2", so that it is stored in cbr_frame [2]. The bit rate type is used. Further, when reproducing at 15 fps, the largest TemporalId among the TemporalIds of the reproduced picture is "0", so the bit rate type stored in cbr_flag [0] is used.

For example, cbr_flag [TemporalId] = 1 indicates a constant bit rate, and cbr_flag [TemporalId] = 0 indicates a variable bit rate.

In the following, the bitstream having the maximum time resolution (bitstream 261) is referred to as the uppermost layer bitstream, and the bitstream having a lower time resolution generated from the bitstream 261 is referred to as the lower layer bitstream or subbitstream. Call. Further, the bit rate type of the uppermost bit stream is called the uppermost bit rate type, and the bit rate type of the lower layer bit stream is called the lower bit rate type.

The time scalability information giving unit 202 outputs the time scalability information 264 to the entropy coding unit 213. The time scalability information 264 specifically indicates the TemporalId of each image.

The input image 251 is input to the subtractor 203 in units of a plurality of pictures, one picture, one slice, and the like. The subtractor 203 calculates the residual signal 252, which is the difference between the input image 251 and the predicted image 260, and outputs the residual signal 252 to the conversion unit 204.

The conversion unit 204 converts the residual signal 252 into a frequency coefficient 253, and outputs the obtained frequency coefficient 253 to the quantization unit 205. The quantization unit 205 quantizes the input frequency coefficient 253 and outputs the obtained quantization coefficient 254 to the inverse quantization unit 206 and the entropy coding unit 213.

The processing of the conversion unit 204 and the quantization unit 205 may be executed in order in each processing unit in units of conversion units (TU: Transform Unit), or one or more matrices having a coefficient corresponding to the TU size. In some cases, it is executed in a batch using multiplication.

The inverse quantization unit 206 inversely quantizes the quantization coefficient 254 output from the quantization unit 205, and outputs the obtained frequency coefficient 255 to the inverse conversion unit 207. The inverse conversion unit 207 converts the frequency coefficient 255 into a residual signal 256 by performing inverse frequency conversion with respect to the frequency coefficient 255, and outputs the obtained residual signal 256 to the adder 208.

The adder adds the residual signal 256 output from the inverse conversion unit 207 to the prediction image 260 output from the intra prediction unit 211 or the inter prediction unit 212, and further predicts the obtained reconstructed image 257. Therefore, it is output to the block memory 209 or the frame memory 210.

The processes of the inverse quantization unit 206 and the inverse conversion unit 207 may be executed in order in TU units, or may be collectively performed by multiplying one or more matrices having a coefficient corresponding to the TU size. In some cases. Although the terms inverse quantization and inverse transformation are used here to clarify the explanation, inverse quantization and inverse transformation differ from quantization and transformation only in the value of the coefficient, and matrix multiplication is used. Since it is the process used, it may be called quantization and transformation.

The intra prediction unit 211 searches the inside of the reconstructed image 257 stored in the block memory 209 for each prediction unit (PU: Precision Unit), copies a part of the image obtained by the search, or multiplies the weight. Is applied to create a predicted image 258 predicted to be similar to the input image 251.

The inter-prediction unit 212 searches the reconstructed image 257 stored in the frame memory 210 for each PU, and detects one or more images that are most similar to or are likely to be similar to the input image 251. Image 259 is generated. Further, one of the predicted image 258 and the predicted image 259 is selected as the predicted image 260.

The entropy encoding unit 213 includes bit rate control information 263 from the bit rate control unit 201, time scalability information 264 from the time scalability information giving unit 202, quantization coefficient 254 from the quantization unit 205, and an intra prediction unit. The bit stream 261 is output by encoding the prediction information from 211 and the prediction information from the inter-prediction unit 212.

Next, the image coding process according to the present embodiment will be described. FIG. 3 is a flowchart of the image coding process according to the present embodiment.

In step S301, the image encoding device 200 generates bit rate control information 263 and time scalability information 264. The image coding apparatus 200 entropy-encodes the generated bit rate control information 263 and time scalability information 264, and inserts the encoded bit rate control information 263 and time scalability information 264 into the header of the bit stream 261.

In step S302, the image encoding device 200 creates a predicted image 260 by performing intra prediction or inter prediction.

In step S303, the image coding device 200 calculates the residual signal 252, which is the difference between the predicted image 260 and the input image 251.

In step S304, the image encoding device 200 calculates the frequency coefficient 253 by frequency-converting the residual signal 252.

In step S305, the image coding apparatus 200 calculates the quantization width by bit rate control, and calculates the quantization coefficient 254 by quantizing the frequency coefficient 253 using the obtained quantization width. Specifically, the image coding apparatus 200 calculates the quantization width by performing bit rate control using HRD with the bit rate type of the uppermost layer.

In step S306, the image coding apparatus 200 entropy-encodes the prediction information and the quantization coefficient 254, and inserts the encoded prediction information and the quantization coefficient 254 into the bit stream 261.

The process of step S301 will be described in detail below with reference to FIGS. 4 to 9.

FIG. 4 is a diagram showing an example of the data structure of the bit stream 261 generated by the image encoding device 200. The bitstream 261 includes header portions such as VPS (Video Parameter Set), APS (Adaptation Parameter Set), SPS (Sequence Parameter Set), and PPS (Picture Parameter Set), and picture data which is encoded image data. Includes. The picture data includes a slice header (SH) and slice data. The slice data includes encoded image data contained in the slice. The slice data includes a block header (BH) and block data. The block data includes encoded image data contained in the block.

The bit rate control information 263 and the time scalability information 264 are encoded by the entropy encoding unit 213 and inserted into any of VPS, APS, SPS, PPS, and SH.

FIG. 5 is a diagram for explaining a calculation process of the bit rate control information 263 by the bit rate control unit 201. The bit rate control unit 201 determines the bit rate type of each layer according to the bit rate type specified by the external parameter 262.

Specifically, when a fixed bit rate is specified by the external parameter 262, the bit rate control unit 201 sets the bit rate type of the uppermost layer (TemporalId = 2 in this example) to the fixed bit rate, and sets the maximum bit rate. The bit rate type other than the upper layer (TemporalId = 0 and 1 in this example) is set to the variable bit rate. When the variable bit rate is specified by the external parameter 262, the bit rate control unit 201 sets the bit rate type of all layers (TemporalId = 0 to 2 in this example) to the variable bit rate.

That is, the bit rate control unit 201 sets the bit rate type of the uppermost layer to be the same as the bit rate type specified by the external parameter 262. Further, the bit rate control unit 201 always sets all the bit rate types other than the uppermost layer to variable bit rates without depending on the bit rate type specified by the external parameter 262.

FIG. 6 is a flowchart of the calculation process of the bit rate control information 263.

In step S401, the bit rate control unit 201 acquires the bit rate type at the time resolution that encodes all the pictures, which is indicated by the external parameter 262.

In step S402, the time scalability information giving unit 202 acquires the time scalability information indicated by the external parameter 262. Here, the time scalability information indicates the number of sublayers, that is, the number of TempralIds, or the TemporalId to be set for each picture. The TempralId information may be the TempralId information set for each picture in the GOP unit, or the TempralID information for each slice type. Further, the time scalability information giving unit 202 generates time scalability information 264 based on the time scalability information indicated by the external parameter 262, and outputs the generated time scalability information 264 to the bit rate control unit 201. Here, the time scalability information 264 indicates, for example, the number of sublayers, that is, the number of TemporalIds.

In step S403, the bit rate control unit 201 determines whether the bit rate type acquired in step S401 is a constant bit rate.

When the bit rate type is a constant bit rate (Yes in S403), in step S404, the bit rate control unit 201 sets the bit rate type of the uppermost layer to a fixed bit rate. Specifically, the bit rate control unit 201 sets “1” (constant bit rate) in cbr_flag [2] of TemporalId = 2.

On the other hand, when the bit rate type is not a fixed bit rate, that is, a variable bit rate (No in S403), in step S405, the bit rate control unit 201 sets the uppermost layer bit rate type to a variable bit rate. Specifically, the bit rate control unit 201 sets “0” (variable bit rate) in cbr_flag [2] of TemporalId = 2.

After step S404 or S405, in step S406, the bit rate control unit 201 sets the bit rate types of all layers except the top layer to variable bit rates. Specifically, the bit rate control unit 201 sets “0” (variable bit rate) in cbr_flag [TemporalId] of TemporalId ≦ 1.

In step S407, the bit rate control unit 201 outputs cbr_flag [TemporalId], which is bit rate control information 263 indicating the bit rate type of each layer, to the entropy coding unit 213.

In steps S401 and S402, the bit rate type and the time scalability information are input from the outside as the external parameter 262, but as at least one of the bit rate type and the time scalability information, it is fixed in advance in the image encoding device 200. Values may be used.

Hereinafter, it will be described that the bitstream 261 that satisfies the conformity at all time resolutions can be created by the present embodiment.

FIG. 7 is a diagram showing an example of the CPB occupancy of HRD with respect to a bit stream when a moving image of 60 fps is encoded while satisfying conformity at a fixed bit rate. The vertical axis of FIG. 7 shows the CPB occupancy, and the horizontal axis shows the time. In addition, the CPB capacity of HRD is indicated by a horizontal line. As shown in FIG. 7, in the coding of 60 fps, the coding is performed without causing an overflow.

For the sake of explanation, the example shown in FIG. 1 is used for the hierarchical structure of the bitstream. Here, by excluding the picture with TemporalId ≧ 1 from all the pictures, a bit stream of 15 fps can be created from the bit stream of 60 fps. The CPB occupancy of HRD at that time is shown in FIG. Further, FIG. 8 is a diagram for comparison, and shows a case where cbr_flag [0] = 1, that is, the bit rate type at 15 fps is a constant bit rate. In this case, as shown in FIG. 8, an overflow occurs.

On the other hand, in the present embodiment, as described above, cbr_flag [TemporalId] = 0 (TemporalId ≦ 1) is set for the lower layer bitstream other than 60 fps. That is, bit rate types other than 60 fps are set to variable bit rates. At a variable bit rate, if there is no free space in the CPB capacity, the bitstream input can be paused. As a result, overflow can be avoided. FIG. 9 is a diagram showing the CPB occupancy of HRD when the bit rate type of 15 fps is a variable bit rate. As shown in FIG. 9, the occurrence of overflow can be prevented by setting the bit rate type to a variable bit rate.

Further, the same target bit rate may be set for bitstreams having different time resolutions (60 fps, 30 fps and 15 fps described above). In this case, the amount of input to the CPB per hour is constant in a plurality of bitstreams having different time resolutions. Further, when the time resolution is changed from 60 fps to 15 fps, the amount of buffer extracted from the CPB is reduced. Therefore, if it is encoded so that underflow does not occur at the highest time resolution, underflow does not occur at all time resolutions.

Here, the information indicating the withdrawal time from the CPB is encoded by, for example, the Supplimental Enhance Information (SEI) set in the sequence.

According to the present embodiment, by controlling the bit rate so that underflow and overflow do not occur only at the highest time resolution that encodes all the pictures, the conformity is achieved at all the time resolutions set by TemporalId. The satisfying bitstream can be encoded.

As described above, according to the configuration of the present embodiment, the image can be appropriately encoded so as to satisfy the conformity regardless of the transmission rate while suppressing the increase in the processing amount.

As described above, the image coding apparatus 200 according to the present embodiment generates a bit stream 261 having time scalability by encoding a plurality of images.

The image coding apparatus 200 selects a first bit rate type from a fixed bit rate and a variable bit rate (S401), and determines a time layer for each of the plurality of images (S402). For example, the image encoding device 200 selects the bit rate type indicated by the external parameter 262 or a predetermined bit rate type. Further, the image coding device 200 determines the time layer of each of the plurality of images based on the time scalability information indicated by the external parameter 262.

When a fixed bit rate is selected as the first bit rate type (Yes in S403), the image encoding device 200 (1) is the first bit stream 261 containing encoded data of a plurality of images belonging to all time layers. The 2-bit rate type is set to a fixed bit rate (S404), and (2) the encoded data of an image that is a part of the bit stream 261 and belongs to a time layer other than the uppermost time layer among the above-mentioned plurality of images. Set the third bit rate type of the including subbit stream to variable bit rate (S406).

Next, the image coding apparatus 200 encodes each of the plurality of images by prohibiting reference to an image having a time layer higher than that of the image to be processed (S302 to S305).

For example, the image encoding device 200 uses a virtual reference decoder (HRD) that virtually models the buffer management of the image decoding device, and the image decoding device can process the bit stream 261 without failure in the second bit rate type. Multiple images are encoded as such. Specifically, the image coding device 200 controls the quantization width by bit rate control using HRD so that underflow and overflow do not occur in the image decoding device.

Further, the image coding apparatus 200 determines the quantization width by performing bit rate control in the second bit rate type in coding a plurality of images belonging to all time layers, and is a sub-bit having a low time resolution. Bit rate control in the third bit rate type is not performed for each of the streams.

Next, the image coding apparatus 200 includes time scalability information 264 indicating a time layer of a plurality of images, bit rate control information 263 indicating a second bit rate type and a third bit rate type, and a plurality of encoded images. Generates a bitstream 261 containing (S301 and S306). Here, the time scalability information 264 includes a time identifier (TemporalId) indicating a time layer to which each of the plurality of images belongs, or a number of patterns having a time resolution that identifies one or more time layers to be decoded.

As described above, the image coding apparatus 200 according to the present embodiment sets the bit rate type of the sub bit stream to a variable bit rate regardless of the bit rate type of the uppermost bit stream including the coding information of all the sub layers. To do. As a result, the image encoding device 200 can control the bit rate without considering the overflow of the sub-bit stream. Therefore, the processing amount of the image coding apparatus 200 can be reduced.

Further, as described above, the target bit rates of the uppermost layer bit stream and the sub bit stream may be set to be the same. This eliminates the need to consider subbitstream underflow.

As a result, the image encoding device 200 does not need to consider the overflow and underflow of the sub-bit stream, so that the bit rate control may be performed only for the uppermost bit stream. Therefore, the image encoding device 200 can generate a bit stream having time scalability that satisfies the conformity by the same processing as a normal bit stream (which does not have time scalability).

Although the image coding method and the image coding device according to the embodiment have been described above, the present invention is not limited to this embodiment.

For example, the present invention may be realized as an image decoding method or an image decoding device that decodes a bit stream generated by the image coding method or the image coding device according to the above embodiment.

Further, each processing unit included in the image coding apparatus according to the above embodiment is typically realized as an LSI which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip so as to include a part or all of them.

Further, the integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connection and settings of circuit cells inside the LSI may be used.

In each of the above embodiments, each component may be configured with dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

In other words, the image coding device includes a processing circuit (processing circuit) and a storage device (store) electrically connected to the processing circuit (accessible from the processing circuit). The processing circuit includes at least one of dedicated hardware and a program execution unit. Further, when the processing circuit includes a program execution unit, the storage device stores the software program executed by the program execution unit. The processing circuit uses the storage device to execute the image coding method according to the above embodiment.

Further, the present invention may be the software program or a non-temporary computer-readable recording medium on which the program is recorded. Needless to say, the above program can be distributed via a transmission medium such as the Internet.

In addition, the numbers used above are all exemplified for the purpose of specifically explaining the present invention, and the present invention is not limited to the illustrated numbers.

In addition, the order in which the steps included in the above image coding method are executed is for exemplifying the present invention in detail, and may be an order other than the above. Further, a part of the above steps may be executed at the same time (parallel) with other steps.

Although the image coding method and the image coding device according to one or more aspects of the present invention have been described above based on the embodiments, the present invention is not limited to the embodiments. As long as it does not deviate from the gist of the present invention, one or more of the present embodiments may be modified by those skilled in the art, or may be constructed by combining components in different embodiments. It may be included within the scope of the embodiment.

(Embodiment 2)
Each of the above-described embodiments is performed by recording a program for realizing the configuration of the moving image coding method (image coding method) or the moving image decoding method (image decoding method) shown in each of the above embodiments on a storage medium. It becomes possible to easily carry out the processing shown in the above embodiment in an independent computer system. The storage medium may be a magnetic disk, an optical disk, a magneto-optical disk, an IC card, a semiconductor memory, or the like, as long as it can record a program.

Further, an application example of the moving image coding method (image coding method) and the moving image decoding method (image decoding method) shown in each of the above embodiments and a system using the same will be described here. The system is characterized by having an image coding / decoding device including an image coding device using an image coding method and an image decoding device using an image decoding method. Other configurations in the system can be changed appropriately as the case may be.

FIG. 10 is a diagram showing an overall configuration of a content supply system ex100 that realizes a content distribution service. The communication service providing area is divided into desired sizes, and fixed radio stations ex106, ex107, ex108, ex109, and ex110 are installed in each cell.

This content supply system ex100 has a computer ex111, a PDA (Personal Digital Assistant) ex112, a camera ex113, a mobile phone ex114, and a game machine ex115 via the Internet service provider ex102 and the telephone network ex104, and the base station ex106 to the Internet ex101. Each device such as is connected.

However, the content supply system ex100 is not limited to the configuration as shown in FIG. 10, and any element may be combined and connected. Further, each device may be directly connected to the telephone network ex104 from the base station ex106, which is a fixed radio station, without going through ex110. Further, the devices may be directly connected to each other via short-range radio or the like.

The camera ex113 is a device capable of shooting a moving image such as a digital video camera, and the camera ex116 is a device capable of shooting a still image and a moving image such as a digital camera. The mobile phone ex114 is a GSM (registered trademark) (Global System for Mobile Communications) system, a CDMA (Code Division Multiple Access) system, a W-CDMA (Wideband-Code Division Multiple Access) system, or an LTE (Long Term Evolution) system. The method, HSPA (High Speed Packet Access) mobile phone, PHS (Personal Handyphone System), or the like may be used.

In the content supply system ex100, the camera ex113 and the like are connected to the streaming server ex103 through the base station ex109 and the telephone network ex104, so that live distribution and the like become possible. In the live distribution, the content (for example, a live music image) shot by the user using the camera ex113 is encoded as described in each of the above embodiments (that is, according to one aspect of the present invention). (Functions as the image encoding device), and transmits to the streaming server ex103. On the other hand, the streaming server ex103 streams the content data transmitted to the requested client. Examples of the client include a computer ex111, a PDAex112, a camera ex113, a mobile phone ex114, a game machine ex115, and the like, which can decode the encoded data. Each device that has received the distributed data decodes and reproduces the received data (that is, functions as an image decoding device according to one aspect of the present invention).

The captured data may be encoded by the camera ex113, the streaming server ex103 that performs the data transmission processing, or may be shared with each other. Similarly, the decryption process of the delivered data may be performed by the client, the streaming server ex103, or may be shared with each other. Further, not only the camera ex113 but also the still image and / or the moving image data taken by the camera ex116 may be transmitted to the streaming server ex103 via the computer ex111. The coding process in this case may be performed by any of the camera ex116, the computer ex111, and the streaming server ex103, or may be shared with each other.

Further, these coding / decoding processes are generally performed by the computer ex111 or the LSI ex500 of each device. The LSI ex500 may be one chip or may be composed of a plurality of chips. In addition, software for video coding / decoding is embedded in some recording medium (CD-ROM, flexible disk, hard disk, etc.) that can be read by a computer ex111 or the like, and the coding / decoding processing is performed using the software. You may. Further, when the mobile phone ex114 is equipped with a camera, the moving image data acquired by the camera may be transmitted. The moving image data at this time is the data encoded by the LSI ex500 of the mobile phone ex114.

Further, the streaming server ex103 may be a plurality of servers or a plurality of computers, and may disperse data for processing, recording, and distribution.

As described above, in the content supply system ex100, the client can receive and reproduce the encoded data. In this way, in the content supply system ex100, the client can receive, decode, and reproduce the information transmitted by the user in real time, and even a user who does not have special rights or equipment can realize personal broadcasting.

Not limited to the example of the content supply system ex100, as shown in FIG. 11, the digital broadcasting system ex200 also includes at least a moving image coding device (image coding device) or moving image decoding according to each of the above embodiments. Any of the devices (image decoding devices) can be incorporated. Specifically, at the broadcasting station ex201, the multiplexed data in which music data or the like is multiplexed with the video data is transmitted via radio waves to the communication or the satellite ex202. This video data is data encoded by the moving image coding method described in each of the above embodiments (that is, data encoded by the image coding apparatus according to one aspect of the present invention). In response to this, the broadcasting satellite ex202 transmits radio waves for broadcasting, and the radio waves are received by the home antenna ex204 capable of receiving satellite broadcasting. A device such as a television (receiver) ex300 or a set-top box (STB) ex217 decodes and reproduces the received multiplexed data (that is, functions as an image decoding device according to one aspect of the present invention).

Further, the reader / recorder ex218 also reads and decodes the multiplexed data recorded on the recording medium ex215 such as a DVD or BD, or encodes the video signal on the recording medium ex215 and, in some cases, multiplexes and writes the music signal. It is possible to implement the moving image decoding device or the moving image coding device shown in each of the above embodiments. In this case, the reproduced video signal is displayed on the monitor ex219, and the video signal can be reproduced in another device or system by the recording medium ex215 in which the multiplexed data is recorded. Further, a moving image decoding device may be mounted in a set-top box ex217 connected to a cable ex203 for cable television or an antenna ex204 for satellite / terrestrial broadcasting, and this may be displayed on a television monitor ex219. At this time, the moving image decoding device may be incorporated in the television instead of the set-top box.

FIG. 12 is a diagram showing a television (receiver) ex300 using the moving image decoding method and the moving image coding method described in each of the above embodiments. The television ex300 has a tuner ex301 that acquires or outputs multiplexed data in which audio data is multiplexed on video data via an antenna ex204 or a cable ex203 that receives the above broadcast, and democratizes the received multiplexed data. Alternatively, the modulation / demodulation unit ex302 that modulates the multiplexed data to be transmitted to the outside, and the video data and audio data that separate the demodulated multiplexed data into video data and audio data, or are encoded by the signal processing unit ex306. The multiplexing / separation unit ex303 for multiplexing the data is provided.

Further, the television ex300 includes an audio signal processing unit ex304 and a video signal processing unit ex305 (an image coding device or an image according to one aspect of the present invention) that decodes each of the audio data and the video data or encodes the respective information. It has a signal processing unit ex306 having a function as a decoding device), a speaker ex307 that outputs a decoded audio signal, and an output unit ex309 having a display unit ex308 such as a display that displays the decoded video signal. Further, the television ex300 has an interface unit ex317 having an operation input unit ex312 and the like for receiving input of user operation. Further, the television ex300 has a control unit ex310 that controls each unit in an integrated manner, and a power supply circuit unit ex311 that supplies electric power to each unit. In addition to the operation input unit ex312, the interface unit ex317 has a bridge ex313 connected to an external device such as a reader / recorder ex218, a slot unit ex314 for mounting a recording medium ex216 such as an SD card, and an external recording such as a hard disk. It may have a driver ex315 for connecting to media, a modem ex316 for connecting to a telephone network, and the like. The recording medium ex216 is capable of electrically recording information by a stored non-volatile / volatile semiconductor memory element. Each part of the television ex300 is connected to each other via a synchronization bus.

First, a configuration will be described in which the television ex300 decodes and reproduces the multiplexed data acquired from the outside by the antenna ex204 or the like. The television ex300 receives a user operation from the remote controller ex220 or the like, and separates the multiplexed data demodulated by the modulation / demodulation unit ex302 by the multiplexing / separation unit ex303 based on the control of the control unit ex310 having a CPU or the like. Further, the television ex300 decodes the separated audio data by the audio signal processing unit ex304, and decodes the separated video data by the video signal processing unit ex305 using the decoding method described in each of the above embodiments. The decoded audio signal and video signal are output to the outside from the output unit ex309, respectively. At the time of output, it is preferable to temporarily store these signals in buffers ex318, ex319, etc. so that the audio signal and the video signal are reproduced in synchronization. Further, the television ex300 may read the multiplexed data from the recording media ex215 and ex216 such as a magnetic / optical disk and an SD card, not from broadcasting or the like. Next, a configuration will be described in which the television ex300 encodes an audio signal or a video signal and transmits it to the outside or writes it to a recording medium or the like. The television ex300 receives a user operation from a remote controller ex220 or the like, encodes an audio signal in the audio signal processing unit ex304 based on the control of the control unit ex310, and outputs a video signal in the video signal processing unit ex305 according to each of the above embodiments. It is encoded using the coding method described in. The encoded audio signal and video signal are multiplexed by the multiplexing / separating unit ex303 and output to the outside. At the time of multiplexing, it is preferable to temporarily store these signals in buffers ex320, ex321, etc. so that the audio signal and the video signal are synchronized. As shown in the figure, a plurality of buffers ex318, ex319, ex320, and ex321 may be provided, or one or more buffers may be shared. Further, in addition to the figures shown, data may be stored in the buffer as a cushioning material for avoiding system overflow or underflow even between the modulation / demodulation unit ex302 and the multiplexing / separation unit ex303, for example.

In addition to acquiring audio data and video data from broadcasting and recording media, the TV ex300 has a configuration that accepts AV input from a microphone or camera, and performs encoding processing on the data acquired from them. May be good. Although the television ex300 has been described here as a configuration capable of the above-mentioned coding processing, multiplexing, and external output, these processing cannot be performed, and only the above-mentioned reception, decoding processing, and external output are possible. It may be a configuration.

When the reader / recorder ex218 reads or writes the multiplexed data from the recording medium, the decoding process or the coding process may be performed by either the television ex300 or the reader / recorder ex218, or the television ex300. The reader / recorder ex218 may share the work with each other.

As an example, FIG. 13 shows the configuration of the information reproduction / recording unit ex400 when reading or writing data from an optical disc. The information reproduction / recording unit ex400 includes elements ex401, ex402, ex403, ex404, ex405, ex406, and ex407 described below. The optical head ex401 irradiates the recording surface of the recording medium ex215, which is an optical disk, with a laser spot to write information, detects the reflected light from the recording surface of the recording medium ex215, and reads the information. The modulation recording unit ex402 electrically drives the semiconductor laser built in the optical head ex401 and modulates the laser beam according to the recorded data. The reproduction / demodulation unit ex403 amplifies the reproduction signal by electrically detecting the reflected light from the recording surface by the photodetector built in the optical head ex401, separates and demodulates the signal component recorded on the recording medium ex215, and is necessary. Information is played back. The buffer ex404 temporarily holds the information for recording on the recording medium ex215 and the information reproduced from the recording medium ex215. The disc motor ex405 rotates the recording medium ex215. The servo control unit ex406 moves the optical head ex401 to a predetermined information track while controlling the rotational drive of the disc motor ex405, and performs laser spot tracking processing. The system control unit ex407 controls the entire information reproduction / recording unit ex400. In the above-mentioned read / write processing, the system control unit ex407 uses various information held in the buffer ex404, generates / adds new information as necessary, and modifies the modulation recording unit ex402 and the reproduction / demodulation unit. This is realized by recording and reproducing information through the optical head ex401 while coordinating the ex403 and the servo control unit ex406. The system control unit ex407 is composed of, for example, a microprocessor, and executes those processes by executing a read / write program.

In the above, the optical head ex401 has been described as irradiating the laser spot, but it may be configured to perform higher density recording using near-field light.

FIG. 14 shows a schematic diagram of the recording medium ex215, which is an optical disc. A guide groove (groove) is formed in a spiral shape on the recording surface of the recording medium ex215, and address information indicating an absolute position on the disk is recorded in advance on the information track ex230 by changing the shape of the groove. This address information includes information for specifying the position of the recording block ex231, which is a unit for recording data, and the recording block is specified by playing back the information track ex230 and reading the address information in a device that performs recording or reproduction. Can be done. Further, the recording medium ex215 includes a data recording area ex233, an inner peripheral area ex232, and an outer peripheral area ex234. The area used for recording user data is the data recording area ex233, and the inner circumference area ex232 and the outer circumference area ex234 arranged on the inner circumference or the outer circumference from the data recording area ex233 are used for specific purposes other than recording user data. Used. The information reproduction / recording unit ex400 reads / writes encoded audio data, video data, or multiplexed data obtained by multiplexing those data with respect to the data recording area ex233 of such recording media ex215.

In the above description, an optical disc such as a single-layer DVD or BD has been described as an example, but the present invention is not limited to these, and an optical disc having a multi-layer structure and capable of recording other than the surface may be used. In addition, an optical disc having a structure for multidimensional recording / playback, such as recording information in the same place on a disk using light of various different wavelength colors or recording different layers of information from various angles. It may be.

Further, in the digital broadcasting system ex200, it is also possible for a car ex210 having an antenna ex205 to receive data from a satellite ex202 or the like and to reproduce a moving image on a display device such as a car navigation ex211 having the car ex210. As the configuration of the car navigation ex211, for example, among the configurations shown in FIG. 12, a configuration including a GPS receiving unit can be considered, and the same can be considered for a computer ex111, a mobile phone ex114, and the like.

FIG. 15A is a diagram showing a mobile phone ex114 using the moving image decoding method and the moving image coding method described in the above embodiment. The mobile phone ex114 includes an antenna ex350 for transmitting and receiving radio waves to and from the base station ex110, a camera unit ex365 capable of taking images and still images, an image captured by the camera unit ex365, an image received by the antenna ex350, and the like. The camera includes a display unit ex358 such as a liquid crystal display that displays the decoded data. The mobile phone ex114 further includes a main body having an operation key unit ex366, an audio output unit ex357 such as a speaker for outputting audio, an audio input unit ex356 such as a microphone for inputting audio, and a captured image. In the memory unit ex367 that stores encoded data such as still images, recorded audio, received video, still images, mail, etc. or decoded data, or in the interface unit with the recording media that also stores data. A certain slot portion ex364 is provided.

Further, a configuration example of the mobile phone ex114 will be described with reference to FIG. 15B. The mobile phone ex114 has a power supply circuit unit ex361, an operation input control unit ex362, and a video signal processing unit ex355 with respect to a main control unit ex360 that collectively controls each part of the main body unit including the display unit ex358 and the operation key unit ex366. , Camera interface unit ex363, LCD (Liquid Crystal Display) control unit ex359, modulation / demodulation unit ex352, multiplexing / separation unit ex353, audio signal processing unit ex354, slot unit ex364, memory unit ex367 are connected to each other via bus ex370. ing.

When the call end and the power key are turned on by the user's operation, the power circuit unit ex361 activates the mobile phone ex114 in an operable state by supplying electric power to each unit from the battery pack.

The mobile phone ex114 converts the voice signal picked up by the voice input unit ex356 in the voice call mode into a digital voice signal by the voice signal processing unit ex354 based on the control of the main control unit ex360 having a CPU, ROM, RAM and the like. , This is subjected to spectrum diffusion processing by the modulation / demodulation unit ex352, digital-analog conversion processing and frequency conversion processing by the transmission / reception unit ex351, and then transmitted via the antenna ex350. Further, the mobile phone ex114 amplifies the received data received via the antenna ex350 in the voice call mode, performs frequency conversion processing and analog-digital conversion processing, and performs spectrum reverse diffusion processing by the modulation / demodulation unit ex352, and the voice signal processing unit. After converting to an analog audio signal with ex354, this is output from the audio output unit ex357.

Further, when the e-mail is transmitted in the data communication mode, the text data of the e-mail input by the operation of the operation key unit ex366 of the main body unit or the like is sent to the main control unit ex360 via the operation input control unit ex362. The main control unit ex360 performs spread spectrum processing on the modulation / demodulation unit ex352, digital-to-analog conversion processing and frequency conversion processing on the transmission / reception unit ex351, and then transmits the text data to the base station ex110 via the antenna ex350. .. When receiving an e-mail, the received data is processed in the reverse manner and output to the display unit ex358.

When transmitting video, still images, or video and audio in the data communication mode, the video signal processing unit ex355 compresses the video signal supplied from the camera unit ex365 by the moving image coding method shown in each of the above embodiments. It is encoded (that is, functions as an image coding device according to one aspect of the present invention), and the encoded video data is transmitted to the multiplexing / separating unit ex353. Further, the audio signal processing unit ex354 encodes the audio signal picked up by the audio input unit ex356 while the camera unit ex365 is capturing the video, still image, etc., and sends the encoded audio data to the multiplexing / separation unit ex353. To do.

The multiplexing / separating unit ex353 multiplexes the encoded video data supplied from the video signal processing unit ex355 and the encoded audio data supplied from the audio signal processing unit ex354 by a predetermined method, and is obtained as a result. The multiplexed data is subjected to spectrum diffusion processing by the modulation / demodulation unit (modulation / demodulation circuit unit) ex352, digital-analog conversion processing and frequency conversion processing by the transmission / reception unit ex351, and then transmitted via the antenna ex350.

Decrypts the multiplexed data received via the antenna ex350 when receiving video file data linked to a homepage, etc. in data communication mode, or when receiving e-mail with video and / or audio attached. In order to do so, the multiplexing / separating unit ex353 separates the multiplexed data into a bit stream of video data and a bit stream of audio data, and processes the video data encoded via the synchronization bus ex370 as a video signal. In addition to supplying the coded audio data to the unit ex355, the encoded audio data is supplied to the audio signal processing unit ex354. The video signal processing unit ex355 decodes the video signal by decoding by the video decoding method corresponding to the video coding method shown in each of the above embodiments (that is, the image according to one aspect of the present invention). (Functions as a decoding device), the video and still images included in the moving image file linked to the homepage are displayed from the display unit ex358 via the LCD control unit ex359. Further, the audio signal processing unit ex354 decodes the audio signal, and the audio output unit ex357 outputs the audio.

Further, the terminals such as the mobile phone ex114 are referred to as transmission / reception terminals having both an encoder and a decoder, as well as transmission terminals having only an encoder and receiving terminals having only a decoder, as in the case of the television ex300. There are three possible implementation formats. Further, the digital broadcasting system ex200 has been described as receiving and transmitting multiplexed data in which music data and the like are multiplexed with video data, but data in which character data and the like related to video are multiplexed in addition to audio data. It may be the video data itself instead of the multiplexed data.

As described above, it is possible to use the moving image coding method or moving image decoding method shown in each of the above-described embodiments for any of the above-mentioned devices / systems, and by doing so, in each of the above-described embodiments. The described effect can be obtained.

Further, the present invention is not limited to the above-described embodiment, and various modifications or modifications can be made without departing from the scope of the present invention.

(Embodiment 3)
The moving image coding method or device shown in each of the above embodiments is appropriately switched between the moving image coding method or device conforming to different standards such as MPEG-2, MPEG4-AVC, and VC-1, as necessary. By doing so, it is also possible to generate video data.

Here, when a plurality of video data conforming to different standards are generated, it is necessary to select a decoding method corresponding to each standard when decoding. However, since it is not possible to identify which standard the video data to be decoded conforms to, there arises a problem that an appropriate decoding method cannot be selected.

In order to solve this problem, the multiplexed data in which audio data or the like is multiplexed with the video data is configured to include identification information indicating which standard the video data conforms to. The specific configuration of the multiplexed data including the video data generated by the moving image coding method or the apparatus shown in each of the above embodiments will be described below. The multiplexed data is a digital stream in the MPEG-2 transport stream format.

FIG. 16 is a diagram showing the structure of the multiplexed data. As shown in FIG. 16, the multiplexed data is obtained by multiplexing one or more of a video stream, an audio stream, a presentation graphics stream (PG), and an interactive graphics stream. The video stream shows the main and sub-video of the movie, the audio stream (IG) shows the main audio part of the movie and the sub-audio that mixes with the main audio, and the presentation graphics stream shows the subtitles of the movie. Here, the main image indicates a normal image displayed on the screen, and the sub image is an image displayed on a small screen in the main image. In addition, the interactive graphics stream shows an interactive screen created by arranging GUI components on the screen. The video stream is encoded by the moving image coding method or device shown in each of the above embodiments, or a moving image coding method or device conforming to the conventional standards such as MPEG-2, MPEG4-AVC, and VC-1. ing. The audio stream is encoded by a method such as Dolby AC-3, Dolby Digital Plus, MLP, DTS, DTS-HD, or linear PCM.

Each stream contained in the multiplexed data is identified by PID. For example, 0x1011 for video streams used for movie images, 0x1100 to 0x111F for audio streams, 0x1200 to 0x121F for presentation graphics, and 0x1400 to 0x141F for interactive graphics streams. 0x1B00 to 0x1B1F are assigned to the video stream used for the sub video, and 0x1A00 to 0x1A1F are assigned to the audio stream used for the sub audio to be mixed with the main audio.

FIG. 17 is a diagram schematically showing how the multiplexed data is multiplexed. First, the video stream ex235 composed of a plurality of video frames and the audio stream ex238 composed of a plurality of audio frames are converted into PES packet sequences ex236 and ex239, respectively, and converted into TS packets ex237 and ex240, respectively. Similarly, the data of the presentation graphics stream ex241 and the interactive graphics ex244 are converted into PES packet sequences ex242 and ex245, respectively, and further converted into TS packets ex243 and ex246. The multiplexing data ex247 is configured by multiplexing these TS packets into one stream.

FIG. 18 shows in more detail how the video stream is stored in the PES packet sequence. The first stage in FIG. 18 shows a video frame sequence of a video stream. The second row shows the PES packet sequence. As shown by arrows yy1, yy2, yy3, yy4 in FIG. 18, a plurality of Video Presentation Units I-pictures, B-pictures, and P-pictures in the video stream are divided into pictures and stored in the payload of the PES packet. .. Each PES packet has a PES header, and the PES header stores a PTS (Presentation Time-Stamp) which is a picture display time and a DTS (Decoding Time-Stamp) which is a picture decoding time.

FIG. 19 shows the format of the TS packet that is finally written to the multiplexed data. The TS packet is a 188-byte fixed-length packet composed of a 4-byte TS header having information such as a PID that identifies a stream and a 184-byte TS payload that stores data, and the PES packet is divided and stored in the TS payload. To. In the case of a BD-ROM, a 4-byte TP_Extra_Header is added to the TS packet to form a 192-byte source packet, which is written to the multiplexed data. Information such as ATS (Arrival_Time_Stamp) is described in TP_Extra_Header. ATS indicates the transfer start time of the TS packet to the PID filter of the decoder. Source packets are lined up in the multiplexed data as shown in the lower part of FIG. 19, and the number incremented from the beginning of the multiplexed data is called SPN (source packet number).

In addition to each stream such as video, audio, and subtitles, the TS packets included in the multiplexed data include PAT (Program Association Table), PMT (Program Map Table), PCR (Program Lock Reference), and the like. The PAT indicates what the PID of the PMT used in the multiplexed data is, and the PID of the PAT itself is registered as 0. The PMT has the PID of each stream such as video, audio, and subtitles included in the multiplexed data and the attribute information of the stream corresponding to each PID, and also has various descriptors related to the multiplexed data. The descriptor includes copy control information that instructs whether to allow or disallow copying of multiplexed data. PCR corresponds to ATS in which the PCR packet is transferred to the decoder in order to synchronize ATC (Arrival Time Clock) which is the time axis of ATS and STC (System Time Clock) which is the time axis of PTS / DTS. Has information on STC time.

FIG. 20 is a diagram illustrating the data structure of the PMT in detail. At the beginning of the PMT, a PMT header describing the length of the data included in the PMT is arranged. Behind it, a plurality of descriptors related to the multiplexed data are arranged. The copy control information and the like are described as descriptors. After the descriptor, a plurality of stream information about each stream included in the multiplexed data is arranged. The stream information is composed of a stream descriptor in which the stream type, the PID of the stream, and the attribute information of the stream (frame rate, aspect ratio, etc.) are described in order to identify the compression codec of the stream. There are as many stream descriptors as there are streams in the multiplexed data.

When recording on a recording medium or the like, the multiplexed data is recorded together with the multiplexed data information file.

As shown in FIG. 21, the multiplexed data information file is management information of multiplexed data, has a one-to-one correspondence with the multiplexed data, and is composed of multiplexed data information, stream attribute information, and an entry map.

As shown in FIG. 21, the multiplexed data information is composed of a system rate, a reproduction start time, and a reproduction end time. The system rate indicates the maximum transfer rate of the multiplexed data to the PID filter of the system target decoder described later. The ATS interval included in the multiplexed data is set to be less than or equal to the system rate. The playback start time is set to the PTS of the first video frame of the multiplexed data, and the playback end time is set to the PTS of the video frame at the end of the multiplexed data plus the playback interval of one frame.

As shown in FIG. 22, as the stream attribute information, the attribute information for each stream included in the multiplexed data is registered for each PID. The attribute information has different information for each of the video stream, audio stream, presentation graphics stream, and interactive graphics stream. The video stream attribute information includes what kind of compression codec the video stream was compressed with, what the resolution of the individual picture data that makes up the video stream is, what the aspect ratio is, and the frame rate. It has information such as how much it is. The audio stream attribute information includes what compression codec the audio stream was compressed with, how many channels the audio stream contains, what language it supports, what the sampling frequency is, and so on. Has the information of. This information is used for initializing the decoder before the player plays it.

In the present embodiment, the stream type included in the PMT is used among the above-mentioned multiplexed data. When the multiplexed data is recorded on the recording medium, the video stream attribute information included in the multiplexed data information is used. Specifically, in the moving image coding method or apparatus shown in each of the above embodiments, the moving image coding shown in each of the above embodiments is applied to the stream type or video stream attribute information included in the PMT. Provide steps or means to set unique information indicating that the video data is generated by the method or device. With this configuration, it becomes possible to distinguish between the video data generated by the moving image coding method or the apparatus shown in each of the above embodiments and the video data conforming to other standards.

Further, FIG. 23 shows the steps of the moving image decoding method in the present embodiment. In step exS100, the stream type included in the PMT or the video stream attribute information included in the multiplexed data information is acquired from the multiplexed data. Next, in step exS101, it is determined whether or not the stream type or the video stream attribute information indicates that it is the multiplexed data generated by the moving image coding method or the apparatus shown in each of the above embodiments. To do. Then, when it is determined that the stream type or the video stream attribute information is generated by the moving image coding method or apparatus shown in each of the above embodiments, in step exS102, each of the above implementations is performed. Decoding is performed by the moving image decoding method shown in the form. Further, when it is shown that the stream type or the video stream attribute information conforms to the conventional standards such as MPEG-2, MPEG4-AVC, and VC-1, in step exS103, the conventional method is used. Decoding is performed by a moving image decoding method that conforms to the standard.

In this way, by setting a new eigenvalue in the stream type or the video stream attribute information, it is possible to determine whether or not the video stream can be decoded by the moving image decoding method or device shown in each of the above embodiments when decoding. You can judge. Therefore, even when multiplexed data conforming to different standards is input, an appropriate decoding method or device can be selected, so that the data can be decoded without causing an error. Further, the moving image coding method or device shown in the present embodiment, or the moving image decoding method or device can be used for any of the above-mentioned devices and systems.

(Embodiment 4)
The moving image coding method and apparatus, moving image decoding method and apparatus shown in each of the above embodiments are typically realized by an LSI which is an integrated circuit. As an example, FIG. 24 shows the configuration of the LSI ex500 integrated into one chip. The LSI ex500 includes elements ex501, ex502, ex503, ex504, ex505, ex506, ex507, ex508, and ex509 described below, and each element is connected via a bus ex510. The power supply circuit unit ex505 is activated by supplying electric power to each unit when the power supply is on.

For example, when performing coding processing, the LSI ex500 is controlled by the microphone ex117 and the camera ex113 by the AVI / Oex509 based on the control of the control unit ex501 having the CPU ex502, the memory controller ex503, the stream controller ex504, the drive frequency control unit ex512, and the like. The AV signal is input from the above. The input AV signal is temporarily stored in an external memory ex511 such as SDRAM. Based on the control of the control unit ex501, the accumulated data is appropriately divided into a plurality of times according to the processing amount and the processing speed and sent to the signal processing unit ex507, and the signal processing unit ex507 encodes the audio signal and / or the video. The signal is encoded. Here, the video signal coding process is the coding process described in each of the above embodiments. The signal processing unit ex507 further performs processing such as multiplexing the encoded audio data and the encoded video data in some cases, and outputs the stream I / Oex506 to the outside. The output multiplexed data is transmitted to the base station ex107 or written to the recording medium ex215. It is advisable to temporarily store the data in the buffer ex508 so that the data will be synchronized when multiplexing.

In the above description, the memory ex511 has been described as an external configuration of the LSI ex500, but it may be a configuration included inside the LSI ex500. The buffer ex508 is not limited to one, and may include a plurality of buffers. Further, the LSI ex500 may be integrated into one chip or a plurality of chips.

Further, in the above, it is assumed that the control unit ex501 has a CPU ex502, a memory controller ex503, a stream controller ex504, a drive frequency control unit ex512, and the like, but the configuration of the control unit ex501 is not limited to this configuration. For example, the signal processing unit ex507 may be configured to further include a CPU. By providing a CPU inside the signal processing unit ex507, the processing speed can be further improved. Further, as another example, the CPU ex502 may be configured to include a signal processing unit ex507 or, for example, an audio signal processing unit that is a part of the signal processing unit ex507. In such a case, the control unit ex501 is configured to include a signal processing unit ex507 or a CPU ex502 having a part thereof.

Although it is referred to as an LSI here, it may be referred to as an IC, a system LSI, a super LSI, or an ultra LSI depending on the degree of integration.

Further, the method of making an integrated circuit is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connection and settings of circuit cells inside the LSI may be used. Such a programmable logic device typically loads a program constituting software or firmware from a memory or the like to load the moving image coding method or moving image shown in each of the above embodiments. The image decoding method can be executed.

Furthermore, if an integrated circuit technology that replaces an LSI appears due to advances in semiconductor technology or another technology derived from it, it is naturally possible to integrate functional blocks using that technology. There is a possibility of adaptation of biotechnology.

(Embodiment 5)
When decoding the video data generated by the moving image coding method or device shown in each of the above embodiments, the video data conforming to the conventional standards such as MPEG-2, MPEG4-AVC, and VC-1 is decoded. It is conceivable that the processing amount will increase as compared with the case. Therefore, in LSIex500, it is necessary to set a drive frequency higher than the drive frequency of CPUex502 when decoding video data conforming to the conventional standard. However, when the drive frequency is increased, there arises a problem that the power consumption increases.

In order to solve this problem, a moving image decoding device such as a television ex300 or LSI ex500 is configured to identify which standard the video data conforms to and switch the drive frequency according to the standard. FIG. 25 shows the configuration ex800 in the present embodiment. The drive frequency switching unit ex803 sets the drive frequency high when the video data is generated by the moving image coding method or apparatus shown in each of the above embodiments. Then, the decoding processing unit ex801 that executes the moving image decoding method shown in each of the above embodiments is instructed to decode the video data. On the other hand, when the video data is video data conforming to the conventional standard, compared with the case where the video data is generated by the moving image coding method or the apparatus shown in each of the above embodiments. Set the drive frequency low. Then, the decoding processing unit ex802, which conforms to the conventional standard, is instructed to decode the video data.

More specifically, the drive frequency switching unit ex803 includes the CPU ex502 of FIG. 24 and the drive frequency control unit ex512. Further, the decoding processing unit ex801 that executes the moving image decoding method shown in each of the above embodiments and the decoding processing unit ex802 that conforms to the conventional standard correspond to the signal processing unit ex507 of FIG. 24. The CPUex502 identifies which standard the video data conforms to. Then, the drive frequency control unit ex512 sets the drive frequency based on the signal from the CPU ex502. Further, the signal processing unit ex507 decodes the video data based on the signal from the CPU ex502. Here, for the identification of the video data, for example, it is conceivable to use the identification information described in the third embodiment. The identification information is not limited to the information described in the third embodiment, and may be any information that can identify which standard the video data conforms to. For example, it is possible to identify which standard the video data conforms to based on an external signal that identifies whether the video data is used for a television or a disc. In some cases, identification may be based on such an external signal. Further, it is conceivable that the drive frequency in the CPUex 502 is selected based on, for example, a look-up table in which the video data standard as shown in FIG. 27 and the drive frequency are associated with each other. The look-up table is stored in the buffer ex508 or the internal memory of the LSI, and the CPU ex502 can select the drive frequency by referring to the lookup table.

FIG. 26 shows the steps to implement the method of this embodiment. First, in step exS200, the signal processing unit ex507 acquires identification information from the multiplexed data. Next, in step exS201, the CPU ex502 identifies whether or not the video data is generated by the coding method or apparatus shown in each of the above embodiments based on the identification information. When the video data is generated by the coding method or apparatus shown in each of the above embodiments, in step exS202, the CPU ex502 sends a signal for setting the drive frequency high to the drive frequency control unit ex512. Then, the drive frequency control unit ex512 sets a high drive frequency. On the other hand, when it is shown that the video data conforms to the conventional standards such as MPEG-2, MPEG4-AVC, and VC-1, the CPU ex502 drives the signal for setting the drive frequency low in step exS203. It is sent to the frequency control unit ex512. Then, in the drive frequency control unit ex512, the drive frequency is set to be lower than that in the case where the video data is generated by the coding method or the apparatus shown in each of the above embodiments.

Further, the power saving effect can be further enhanced by changing the voltage applied to the LSI ex500 or the device including the LSI ex500 in conjunction with the switching of the drive frequency. For example, when the drive frequency is set low, it is conceivable to set the voltage applied to the LSI ex500 or the device including the LSI ex500 lower than when the drive frequency is set high.

Further, as for the setting method of the drive frequency, when the processing amount at the time of decoding is large, the drive frequency may be set high, and when the processing amount at the time of decoding is small, the drive frequency may be set low. Not limited to the method. For example, when the amount of processing for decoding video data conforming to the MPEG4-AVC standard is larger than the amount of processing for decoding video data generated by the moving image coding method or apparatus shown in each of the above embodiments. It is conceivable to reverse the setting of the drive frequency as described above.

Further, the method of setting the drive frequency is not limited to the configuration in which the drive frequency is lowered. For example, when the identification information indicates that it is the video data generated by the moving image coding method or the apparatus shown in each of the above embodiments, the voltage applied to the LSI ex500 or the apparatus including the LSI ex500 is set high. However, when it is shown that the video data conforms to the conventional standards such as MPEG-2, MPEG4-AVC, and VC-1, it is conceivable to set the voltage applied to the LSI ex500 or the device including the LSI ex500 low. Be done. Further, as another example, when the identification information indicates that it is the video data generated by the moving image coding method or the apparatus shown in each of the above embodiments, the driving of the CPU ex502 is stopped. If it is shown that the video data conforms to the conventional standards such as MPEG-2, MPEG4-AVC, and VC-1, there is a margin in processing, so the drive of CPUex502 should be suspended. Is also possible. Even when the identification information indicates that it is the video data generated by the moving image coding method or the apparatus shown in each of the above embodiments, if there is a margin in the processing, the CPU ex502 is temporarily driven. It is also possible to stop it. In this case, it is conceivable to set the stop time shorter than in the case where the video data conforms to the conventional standards such as MPEG-2, MPEG4-AVC, and VC-1.

In this way, power saving can be achieved by switching the drive frequency according to the standard to which the video data conforms. Further, when the LSI x500 or the device including the LSI ex500 is driven by using the battery, the life of the battery can be extended along with the power saving.

(Embodiment 6)
A plurality of video data conforming to different standards may be input to the above-mentioned devices / systems such as televisions and mobile phones. As described above, in order to enable decoding even when a plurality of video data conforming to different standards are input, the signal processing unit ex507 of the LSI ex500 needs to support the plurality of standards. However, if the signal processing unit ex507 corresponding to each standard is used individually, there arises a problem that the circuit scale of the LSI ex500 becomes large and the cost increases.

In order to solve this problem, a decoding processing unit for executing the moving image decoding method shown in each of the above embodiments, and decoding conforming to conventional standards such as MPEG-2, MPEG4-AVC, and VC-1. The configuration is such that a part is shared with the processing unit. An example of this configuration is shown in ex900 of FIG. 28A. For example, the moving image decoding method shown in each of the above embodiments and the moving image decoding method conforming to the MPEG4-AVC standard are processed in processing such as entropy coding, dequantization, deblocking filter, and motion compensation. Some of the contents are common. For common processing contents, the decoding processing unit ex902 corresponding to the MPEG4-AVC standard is shared, and for other processing contents not corresponding to the MPEG4-AVC standard and specific to one aspect of the present invention, a dedicated decoding processing unit is used. A configuration using ex901 is conceivable. In particular, since one aspect of the present invention is characterized by hierarchical coding, for example, a dedicated decoding processing unit ex901 is used for hierarchical coding, and other entropy decoding, dequantization, and deblocking are performed. -It is conceivable to share the decoding processing unit for either the filter, the motion compensation, or all the processing. Regarding the sharing of the decoding processing unit, regarding the common processing content, the decoding processing unit for executing the moving image decoding method shown in each of the above embodiments is shared, and the processing content peculiar to the MPEG4-AVC standard is shared. May be configured to use a dedicated decoding processing unit.

Further, another example of partially sharing the processing is shown in ex1000 of FIG. 28B. In this example, a dedicated decoding processing unit ex1001 corresponding to the processing content peculiar to one aspect of the present invention, a dedicated decoding processing unit ex1002 corresponding to the processing content peculiar to other conventional standards, and one aspect of the present invention. It is configured to use the common decoding processing unit ex1003 corresponding to the processing contents common to the moving image decoding method according to the above and the moving image decoding method of other conventional standards. Here, the dedicated decoding processing units ex1001 and ex1002 are not necessarily specialized in one aspect of the present invention or processing contents peculiar to other conventional standards, but can execute other general-purpose processing. May be good. It is also possible to implement the configuration of this embodiment in LSI ex500.

As described above, the circuit scale of the LSI can be reduced by sharing the decoding processing unit for the processing contents common to the moving image decoding method according to one aspect of the present invention and the moving image decoding method of the conventional standard. Moreover, it is possible to reduce the cost.

The present invention can be used in image coding devices or image coding methods.

200 Image coding device 201 Bit rate control unit 202 Time scalability information assignment unit 203 Subtractor 204 Converter 205 Quantizer 206 Inverse quantization unit 207 Inverse conversion unit 208 Adder 209 Block memory 210 Frame memory 211 Intra prediction unit 212 Inter Prediction unit 213 Entropy coding unit 251 Input image 252,256 Residual signal 253,255 Frequency coefficient 254 Quantization coefficient 257 Reconstructed image 258,259,260 Prediction image 261 Bitstream 262 External parameter 263 Bit rate control information 264 Hours scalability information

Claims (3)

An image coding method that generates a bitstream with time scalability by encoding a plurality of images.
A selection step of selecting whether to encode the bitstream at a fixed bit rate or a variable bit rate.
If you choose to encode the bitstream at a constant bit rate,
(1) The first bit rate type of the bit stream, which is a part of the bit stream and includes the encoded data of the plurality of images belonging to the uppermost time layer among the plurality of images, is set to a fixed bit rate. (2) Variable bit rate type of the second bit rate of the sub-bit stream that is a part of the bit stream and contains the encoded data of the images belonging to the time layer other than the uppermost time layer among the plurality of images. Setting steps to set the rate and
Each of the plurality of images includes a coding step of encoding by prohibiting reference to an image having a time layer higher than that of the image to be processed.
Image coding method. An image coding method that encodes multiple images to generate a bitstream with time scalability.
The plurality of images are assigned to any of the plurality of layers and encoded to generate the bitstream.
The bit stream contains control information indicating whether the bit stream is encoded at a variable bit rate or a constant bit rate for each of the plurality of layers.
Of the plurality of layers, the control information corresponding to the uppermost time layer indicates a fixed bit rate.
Of the plurality of layers, the control information corresponding to the time layer other than the top layer indicates a variable bit rate.
The image referred to when each of the plurality of images is encoded is an image belonging to a time layer below the time layer to which the image to be processed belongs.
Image coding method. An image decoding device that decodes a bitstream with time scalability.
A circuit with a buffer and
With memory,
The circuit
Decoding the first control information indicating which of the plurality of layers the one or more images are assigned to,
The bit stream is decoded for a second control information indicating whether it is encoded at a variable bit rate or a constant bit rate for each of a plurality of layers.
Based on the first control information and the second control information, the bitstream is taken out of the buffer at a constant bit rate or out of the buffer at a variable bit rate.
Decoding the bitstream to generate multiple images
The image referred to when generating each of the plurality of images is an image belonging to the time layer below the time layer to which the image to be processed belongs.
The bitstream is (1) a first bitstream containing encoded data of an image belonging to the uppermost time layer among a plurality of layers, or (2) a part of the first bitstream. Yes, it is a second bitstream containing encoded data of an image belonging to a time layer other than the top layer among a plurality of layers.
When the bitstream is the first bitstream, the second control information indicates a constant bit rate, and when the bitstream is the second bitstream, the second control information is variable bits. An image decoding device that indicates the rate.

Images