JP6986671B2 - Receiver and receiving method - Google Patents

Description

The present invention relates to a receiving device and a receiving method.

With the sophistication of broadcasting and communication services, the introduction of ultra-high-definition video content such as 8K (7680 x 4320 pixels: also referred to as 8K4K below) and 4K (3840 x 2160 pixels: also referred to as 4K2K below) is being considered. Has been done. The receiving device needs to decode and display the coded data of the received ultra-high-definition moving image in real time, but the processing load at the time of decoding is particularly large for the moving image having a resolution such as 8K. It is difficult to decode a moving image with one decoder in real time. Therefore, a method of reducing the processing load per decoder and achieving real-time processing by parallelizing the decoding processes using a plurality of decoders is being studied.

Further, the coded data is multiplexed and transmitted based on a multiplexing method such as MPEG-2 TS (Transport Stream) or MMT (MPEG Media Transport). For example, Non-Patent Document 1 discloses a technique for transmitting encoded media data for each packet according to MMT.

Information technology-High efficiency coding and media delivery in heterogeneous environment-Part1: MPEG media transfer (MMT), ISO / IEC-1 DIS

By the way, there are various multiplexing methods used in the TV broadcasting system, and for example, the multiplexing method includes a TS method and an IP multiplexing method. Therefore, in a receiving device corresponding to a plurality of methods, when each method is individually mounted, there is a problem that the circuit scale is large and the cost increases.

The present invention provides a receiving device that can be easily shared with an existing implementation and can be realized at low cost.

In order to achieve the above object, the receiving device according to one aspect of the present invention includes the first multiplexing data of the first multiplexing method and the second multiplexing of the second multiplexing method different from the first multiplexing method. A second of receiving a broadcast signal in which at least the multiplexed data composed of at least the first multiplexed data of the data is modulated, demodating the received broadcast signal, and outputting the multiplexed data obtained by demodating the received broadcast signal. 1 processing unit and a conversion unit that converts the multiplexing method of the first multiplexing data among the output multiplexing data to the second multiplexing method and outputs the converted data obtained by the conversion. , Equipped with.

Further, the receiving device according to one aspect of the present invention is a second packet in which the first packet constituting the first multiplexing data of the first multiplexing method is stored, and is different from the first multiplexing method. The first conversion data composed of the second packet used in the second multiplexing method and the second packet composed of the second packet obtained by converting the first packet into the second multiplexing method. A demultiplexing process in which a receiving unit that receives the converted data in which the converted data is multiplexed and the converted data received by the receiving unit are demultiplexed into the first converted data and the second converted data. The first packet is extracted from the second packet constituting the demultiplexing unit and the first conversion data obtained by the demultiplexing process, and the extracted first packet constitutes the first packet. The first decoding unit that performs the first decoding process in the first multiplexing method for one data and the second packet that constitutes the second conversion data obtained by the demultiplexing process are configured. The second decoding unit that performs the second decoding processing in the second multiplexing method for the second data, the first decoding data obtained by the first decoding processing, and the second decoding processing are obtained. It also includes an output unit that outputs the second decoded data.

It should be noted that these general or specific embodiments may be realized in recording media such as methods, systems, integrated circuits, computer programs or computer-readable CD-ROMs, methods, systems, integrated circuits, computer programs. And may be realized by any combination of recording media.

The receiving device of the present invention can be easily shared with an existing implementation and can be realized at low cost.

FIG. 1 is a diagram showing an example of dividing a picture into slice segments. FIG. 2 is a diagram showing an example of a PES packet string in which picture data is stored. FIG. 3 is a diagram showing a division example of the picture according to the first embodiment. FIG. 4 is a diagram showing a division example of a picture according to a comparative example of the first embodiment. FIG. 5 is a diagram showing an example of data of the access unit according to the first embodiment. FIG. 6 is a block diagram of the transmission device according to the first embodiment. FIG. 7 is a block diagram of the receiving device according to the first embodiment. FIG. 8 is a diagram showing an example of an MMT packet according to the first embodiment. FIG. 9 is a diagram showing another example of the MMT packet according to the first embodiment. FIG. 10 is a diagram showing an example of data input to each decoding unit according to the first embodiment. FIG. 11 is a diagram showing an example of MMT packet and header information according to the first embodiment. FIG. 12 is a diagram showing another example of data input to each decoding unit according to the first embodiment. FIG. 13 is a diagram showing a division example of the picture according to the first embodiment. FIG. 14 is a flowchart of the transmission method according to the first embodiment. FIG. 15 is a block diagram of the receiving device according to the first embodiment. FIG. 16 is a flowchart of the receiving method according to the first embodiment. FIG. 17 is a diagram showing an example of MMT packet and header information according to the first embodiment. FIG. 18 is a diagram showing an example of MMT packet and header information according to the first embodiment. FIG. 19 is a diagram showing the configuration of the MPU. FIG. 20 is a diagram showing the structure of MF metadata. FIG. 21 is a diagram for explaining a data transmission order. FIG. 22 is a diagram showing an example of a method of performing decoding without using header information. FIG. 23 is a block diagram of the transmission device according to the second embodiment. FIG. 24 is a flowchart of the transmission method according to the second embodiment. FIG. 25 is a block diagram of the receiving device according to the second embodiment. FIG. 26 is a flowchart of an operation for specifying the MPU head position and the NAL unit position. FIG. 27 is a flowchart of an operation of acquiring initialization information based on a transmission order type and decoding media data based on the initialization information. FIG. 28 is a flowchart of the operation of the receiving device when the low delay presentation mode is provided. FIG. 29 is a diagram showing an example of the transmission order of MMT packets when auxiliary data is transmitted. FIG. 30 is a diagram for explaining an example in which the transmitting device generates auxiliary data based on the configuration of the moof. FIG. 31 is a diagram for explaining the reception of auxiliary data. FIG. 32 is a flowchart of a reception operation using auxiliary data. FIG. 33 is a diagram showing the configuration of an MPU composed of a plurality of movie fragments. FIG. 34 is a diagram for explaining the transmission order of MMT packets when the MPU having the configuration of FIG. 33 is transmitted. FIG. 35 is a first diagram for explaining an operation example of a receiving device when one MPU is composed of a plurality of movie fragments. FIG. 36 is a second diagram for explaining an operation example of the receiving device when one MPU is composed of a plurality of movie fragments. FIG. 37 is a flowchart of the operation of the receiving method described with reference to FIGS. 35 and 36. FIG. 38 is a diagram showing a case where non-VCL NAL units are individually used as data units and aggregated. FIG. 39 is a diagram showing a case where non-VCL NAL units are collectively used as a data unit. FIG. 40 is a flowchart of the operation of the receiving device when packet loss occurs. FIG. 41 is a flowchart of the receiving operation when the MPU is divided into a plurality of movie fragments. FIG. 42 is a diagram showing an example of a picture prediction structure in each TemporalId when achieving time scalability. FIG. 43 is a diagram showing the relationship between the decoding time (DTS) and the display time (PTS) in each picture of FIG. 42. FIG. 44 is a diagram showing an example of a picture prediction structure that requires picture delay processing and reorder processing. FIG. 45 is a diagram showing an example in which an MPU composed of an MP4 format is divided into a plurality of movie fragments and stored in an MMTP payload and an MMTP packet. FIG. 46 is a diagram for explaining a calculation method and a problem of PTS and DTS. FIG. 47 is a flowchart of the reception operation when the DTS is calculated using the information for calculating the DTS. FIG. 48 is a diagram for explaining a method of storing a data unit in the payload in MMT. FIG. 49 is an operation flow of the transmission device according to the third embodiment. FIG. 50 is an operation flow of the receiving device according to the third embodiment. FIG. 51 is a diagram showing an example of a specific configuration of the transmission device according to the third embodiment. FIG. 52 is a diagram showing an example of a specific configuration of the receiving device according to the third embodiment. FIG. 53 is a diagram showing a method of storing a non-timed medium in an MPU and a method of transmitting an MMTP packet. FIG. 54 is a diagram showing an example in which a plurality of divided data obtained by dividing a file are packetized and transmitted. FIG. 55 is a diagram showing another example of packetizing and transmitting each of a plurality of divided data obtained by dividing the file. FIG. 56 is a diagram showing the syntax of the loop for each file in the asset management table. FIG. 57 is an operation flow for specifying the divided data number in the receiving device. FIG. 58 is an operation flow for specifying the number of divided data in the receiving device. FIG. 59 is an operation flow for determining whether or not to operate the fragment counter in the transmitting device. FIG. 60 is a diagram for explaining a method of specifying the number of divided data and the divided data number (when the fragment counter is utilized). FIG. 61 is an operation flow of the transmission device when the fragment counter is utilized. FIG. 62 is an operation flow of the receiving device when the fragment counter is utilized. FIG. 63 is a diagram showing a service configuration when the same program is transmitted by a plurality of IP data flows. FIG. 64 is a diagram showing an example of a specific configuration of the transmission device. FIG. 65 is a diagram showing an example of a specific configuration of the receiving device. FIG. 66 is an operation flow by the transmission device. FIG. 67 is an operation flow by the receiving device. FIG. 68 shows a receive buffer model based on the receive buffer model defined in ARIB STD-B60, particularly when only a broadcast transmission line is used. FIG. 69 is a diagram showing an example in which a plurality of data units are aggregated and stored in one payload. FIG. 70 is an example in which a plurality of data units are aggregated and stored in one payload, and is a diagram showing an example in which a video signal in NAL size format is used as one data unit. FIG. 71 is a diagram showing the configuration of the payload of the MMTP packet whose data unit length is not shown. FIG. 72 is an example shown in the extend area assigned to each packet. FIG. 73 shows the operation flow of the receiving device. FIG. 74 is a diagram showing an example of a specific configuration of the transmission device. FIG. 75 is a diagram showing an example of a specific configuration of the receiving device. FIG. 76 is an operation flow by the transmission device. FIG. 77 is an operation flow by the receiving device. FIG. 78 is a diagram showing an MMT / TLV protocol stack defined in ARIB STD-B60. FIG. 79 is a diagram showing a configuration of a TLV packet. FIG. 80 is a diagram showing an example of a block diagram of the receiving device. FIG. 81 is a diagram for explaining a time stamp descriptor. FIG. 82 is a diagram for explaining leap second adjustment. FIG. 83 is a diagram showing the relationship between the NTP time, the MPU time stamp, and the MPU presentation timing. FIG. 84 is a diagram for explaining a correction method for correcting the time stamp on the transmitting side. FIG. 85 is a diagram for explaining a correction method for correcting a time stamp in a receiving device. FIG. 86 is an operation flow by the transmitting side (transmitting device) when the transmitting side (transmitting device) corrects the MPU time stamp. FIG. 87 is an operation flow by the receiving device when the transmitting side (transmitting device) corrects the MPU time stamp. FIG. 88 is an operation flow by the transmitting side (transmitting device) when the MPU time stamp is corrected in the receiving device. FIG. 89 is an operation flow by the receiving device when the MPU time stamp is corrected in the receiving device. FIG. 90 is a diagram showing an example of a specific configuration of the transmission device. FIG. 91 is a diagram showing an example of a specific configuration of the receiving device. FIG. 92 is an operation flow by the transmission device. FIG. 93 is an operation flow by the receiving device. FIG. 94 is a diagram showing an extended example of the MPU extended time stamp descriptor. FIG. 95 is a diagram for explaining a case where a discontinuity occurs in the MPU sequence number due to the adjustment of the MPU sequence number. FIG. 96 is a diagram for explaining a case where the packet sequence numbers become discontinuous at the timing of switching from the normal equipment to the redundant equipment. FIG. 97 is an operation flow by the receiving device when a discontinuity of the MPU sequence number and the packet sequence number occurs. FIG. 98 is a diagram for explaining a correction method for correcting a time stamp in a receiving device when a leap second is inserted. FIG. 99 is a diagram for explaining a correction method for correcting a time stamp in a receiving device when a leap second is deleted. FIG. 100 is an operation flow by the receiving device. FIG. 101 is a diagram showing an example of a specific configuration of a transmission / reception system. FIG. 102 is a diagram showing a specific configuration of the receiving device. FIG. 103 is an operation flow by the receiving device. FIG. 104 is a diagram for explaining a correction method for correcting a time stamp on the transmitting side (transmitting device) when a leap second is inserted. FIG. 105 is a diagram for explaining a correction method for correcting a time stamp on the transmitting side (transmitting device) when a leap second is deleted. FIG. 106 is a diagram showing an operation flow of the transmission device described with reference to FIG. 104. FIG. 107 is a diagram showing an operation flow of the receiving device described with reference to FIG. 104. FIG. 108 is a diagram showing an operation flow of the transmission device described with reference to FIG. 105. FIG. 109 is a diagram showing an operation flow of the receiving device described with reference to FIG. 105. FIG. 110 is a diagram showing an example of a specific configuration of the transmission device. FIG. 111 is a diagram showing an example of a specific configuration of the receiving device. FIG. 112 is an operation flow (transmission method) by the transmission device. FIG. 113 is an operation flow (reception method) by the receiving device. FIG. 114 is a diagram showing details of the protocol stack diagram of the MMT / TLV method defined in ARIB STD-B60. FIG. 115 is a diagram showing a block diagram of the receiving device. FIG. 116 is a diagram showing a general broadcast protocol multiplexed with MPEG-2 TS Systems. FIG. 117 is a diagram showing a block diagram of a receiving device that receives broadcast signals multiplexed by the TS method. FIG. 118A is a diagram showing a configuration of a conventional receiving device when processing a single method. FIG. 118B is a diagram showing a configuration of a conventional receiving device that independently processes a plurality of methods. FIG. 118C is a diagram showing an example of the configuration of the receiving device when the method conversion means is used. FIG. 118D is a diagram showing another example of the receiver configuration when the method conversion means is used. FIG. 118E is a diagram showing another example of the configuration of the receiving device when the method conversion means is used. FIG. 118F is a diagram showing another example of the configuration of the receiving device when the method conversion means is used. FIG. 119 is a diagram showing a modified example of the configuration of the receiving device when the method conversion means is used. FIG. 120 is a diagram showing a detailed example of the method conversion means. FIG. 121 is a diagram showing a reception processing flow for receiving using the receiving device shown in FIG. 118F. FIG. 122 is a diagram showing a processing flow of the method conversion means in FIG. 119. FIG. 123 is a diagram showing a processing flow of the method processing means B in FIG. 119. FIG. 124 is a diagram showing an example of a specific configuration of the receiving device. FIG. 125 is a diagram showing another example of a specific configuration of the receiving device. FIG. 126 is an operation flow (reception method) by the receiving device. FIG. 127 is an operation flow (reception method) by the receiving device.

The receiving device according to one aspect of the present invention is at least the first multiplexing of the first multiplexing data of the first multiplexing method and the second multiplexing data of the second multiplexing method different from the first multiplexing method. A first processing unit that receives a broadcast signal in which the multiplexed data composed of data is modulated, demolishes the received broadcast signal, and outputs the multiplexed data obtained by demodating the received broadcast signal, and is output. Among the multiplexed data, the first multiplexed data multiplexing method is converted into the second multiplexing method, and a conversion unit for outputting the converted data obtained by the conversion is provided.

Such a receiving device can be easily shared with an existing implementation and can be realized at low cost.

Further, the conversion unit extracts the first data of a part of the first multiplexing data, and uses the first packet constituting the first data as the second packet used in the second multiplexing method. A first conversion unit that performs the first conversion stored in the above and outputs the first conversion data composed of the second packet obtained by the first conversion, and a part of the rest of the first multiplexed data. The first packet constituting the second data is extracted, the extracted first packet is converted into the second packet of the second multiplexing method, and the second conversion is performed, and the second conversion is performed. A second conversion unit that outputs the second conversion data composed of the second packet, and a multiplexing unit that performs a multiplexing process for multiplexing the output first conversion data and the second conversion data. The data obtained by the multiplexing process may be output as the conversion data.

Further, the first data and the second data may be data of different media.

Further, the first conversion unit assigns a first identifier indicating that the second packet is obtained by the first conversion to the second packet of the first conversion data, and the second conversion unit , A second identifier indicating that the second packet is obtained by the second conversion may be added to the second packet of the second conversion data.

Further, a retransmitting unit that retransmits the converted data output by the conversion unit to another receiving device may be provided.

Further, a storage unit that stores the conversion data output by the conversion unit in the storage device may be provided.

Further, a retransmitting unit that retransmits the converted data stored in the storage device to another receiving device may be provided.

Further, a second processing unit may be provided which performs a decoding process for decoding the converted data output by the conversion unit and outputs the decoded data obtained by the decoding process.

Further, a second processing unit is provided which performs a decoding process for decoding the conversion data output by the conversion unit and outputs the decoded data obtained by the decoding process, and the second processing unit provides the conversion. A demultiplexing unit that performs a demultiplexing process that demultiplexes the converted data output by the unit into the first converted data and the second converted data, and the first demultiplexing unit obtained by the demultiplexing process. 1 A first decoding unit that performs a first decoding process in the first multiplexing method on the first data composed of the first packet extracted from the second packet constituting the conversion data. Second decoding in which the second decoding process in the second multiplexing method is performed on the second data composed of the second packet constituting the second conversion data obtained by the demultiplexing process. The first decoded data obtained by the first decoding process and the second decoded data obtained by the second decoding process may be output as the decoded data.

Further, the receiving device further uses the first control information of the first decoded data and the second control information of the second decoded data to obtain one of the first control information and the second control information. An adjustment unit may be provided for making adjustments to match the other.

Further, the first control information is the first reference clock information, the second control information is the second reference clock information, and the adjusting unit has the first reference clock information and the second reference clock information. The first decoded data and the second decoded data may be synchronized by adjusting one to match the other.

Further, the first multiplexing method may be an MMT / TLV method, and the second multiplexing method may be a TS method.

The receiving device according to one aspect of the present invention is a second packet in which a first packet constituting the first multiplexing data of the first multiplexing method is stored, and is a second packet different from the first multiplexing method. The first conversion data composed of the second packet used in the multiplexing method and the second conversion data composed of the second packet obtained by converting the first packet into the second multiplexing method. Performs demultiplexing processing in which the receiving unit that receives the converted data multiplexed with the data and the conversion data received by the receiving unit are demultiplexed into the first conversion data and the second conversion data. The first packet is extracted from the demultiplexing unit and the second packet constituting the first conversion data obtained by the demultiplexing process, and the extracted first packet constitutes the first data. The first decoding unit that performs the first decoding process in the first multiplexing method and the second packet that constitutes the second conversion data obtained by the demultiplexing process are configured. A second decoding unit that performs a second decoding process on the two data in the second multiplexing method, a first decoded data obtained by the first decoded process, and a second decoded unit obtained by the second decoded process. 2 An output unit for outputting decoded data is provided.

It should be noted that these comprehensive or specific embodiments may be realized in recording media such as methods, systems, integrated circuits, computer programs or computer-readable CD-ROMs, methods, systems, integrated circuits, and the like. It may be realized by any combination of computer programs or recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that all of the embodiments described below are comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, the order of steps, etc. shown in the following embodiments are examples, and are not intended to limit the present invention. Further, among the components in the following embodiments, the components not described in the independent claim indicating the highest level concept are described as arbitrary components.

(Knowledge that became the basis of the present invention)
In recent years, the resolution of displays such as TVs, smartphones, and tablet terminals has been increasing. Especially for broadcasting in Japan, 8K4K (resolution is 8K x 4K) service is scheduled for 2020. Since it is difficult to decode an ultra-high resolution moving image such as 8K4K in real time with a single decoder, a method of performing decoding processing in parallel using a plurality of decoders is being studied.

Since the coded data is multiplexed and transmitted based on a multiplexing method such as MPEG-2 TS or MMT, the receiving device needs to separate the encoded data of the moving image from the multiplexed data prior to decoding. .. Hereinafter, the process of separating the coded data from the multiplexed data is referred to as demultiplexing.

When parallelizing the decoding process, it is necessary to distribute the coded data to be decoded to each of the decoders. When distributing the coded data, it is necessary to analyze the coded data itself, and since the bit rate is very high especially for contents such as 8K, the processing load related to the analysis is large. Therefore, there is a problem that the demultiplexing part becomes a bottleneck and reproduction in real time cannot be performed.

By the way, H. is standardized by MPEG and ITU. 264 and H. In a moving image coding method such as 265, the transmission device can divide a picture into a plurality of regions called slices or slice segments, and encode each of the divided regions so that they can be independently decoded. Therefore, for example, H. In the case of 265, the receiving device that receives the broadcast can realize parallelization of the decoding process by separating the data for each slice segment from the received data and outputting the data of each slice segment to a separate decoder. ..

FIG. 1 is a diagram showing an example of dividing one picture into four slice segments in HEVC. For example, the receiving device comprises four decoders, each of which decodes any one of the four slice segments.

In conventional broadcasting, the transmitting device stores one picture (access unit in the MPEG system standard) in one PES packet, and multiplexes the PES packet into a TS packet string. Therefore, the receiving device separates the payload of the PES packet, analyzes the data of the access unit stored in the payload, separates each slice segment, and decodes the data of each separated slice segment. Needed to be output to.

However, the present inventor has found that there is a problem that it is difficult to perform this processing in real time because the amount of processing when analyzing the data of the access unit and separating the slice segments is large.

FIG. 2 is a diagram showing an example in which the data of a picture divided into slice segments is stored in the payload of a PES packet.

As shown in FIG. 2, for example, the data of a plurality of slice segments (slice segments 1 to 4) are stored in the payload of one PES packet. Further, the PES packet is multiplexed into the TS packet string.

(Embodiment 1)
In the following, as a moving image coding method, H.I. The case where 265 is used will be described as an example, but H. This embodiment can also be applied when another coding method such as 264 is used.

FIG. 3 is a diagram showing an example in which the access unit (picture) in the present embodiment is divided into division units. The access unit is H. A feature called tiles introduced by 265 divides the tiles horizontally and vertically into two equal parts, for a total of four tiles. Also, slice segments and tiles are associated one-to-one.

The reason for dividing into two equal parts in the horizontal and vertical directions will be described. First, at the time of decoding, a line memory for storing one horizontal line of data is generally required, but in an ultra-high resolution such as 8K4K, the size in the horizontal direction becomes large, so that the size of the line memory increases. In mounting the receiving device, it is desirable that the size of the line memory can be reduced. Vertical division is required to reduce the size of the line memory. Vertical division requires a data structure called tiles. For these reasons, tiles are used.

On the other hand, since images generally have a high horizontal correlation, the coding efficiency is improved if a wide range can be referred to in the horizontal direction. Therefore, from the viewpoint of coding efficiency, it is desirable that the access unit is divided in the horizontal direction.

By dividing the access unit into two equal parts in the horizontal and vertical directions, these two characteristics can be compatible with each other, and both the mounting surface and the coding efficiency can be considered. When a single decoder can decode a 4K2K moving image in real time, the 8K4K image is divided into four equal parts, and each slice segment is divided into 4K2K to obtain a receiver. Can decode 8K4K images in real time.

Next, the reason why the tiles obtained by dividing the access unit in the horizontal and vertical directions and the slice segments have a one-to-one correspondence will be described. H. In 265, the access unit is composed of a plurality of units called NAL (Network Adjustment Layer) units.

The payload of the NAL unit is an access unit delimiter that indicates the start position of the access unit, SPS (Sequence Parameter Set) that is initialization information at the time of decoding that is commonly used for each sequence, and the initial at the time of decoding that is commonly used in the picture. Either PPS (Picture Parameter Set), which is the information to be converted, SEI (Supplemental Enchantment Information), which is not necessary for the decoding process itself but is necessary for processing and displaying the decoding result, and coded data of the slice segment. To store. The header of the NAL unit contains type information for identifying the data stored in the payload.

Here, the transmitting device transmits the coded data to MPEG-2 TS, MMT (MPEG Media Transport), and MPEG DASH (Dynamic Adaptive).
When multiplexing with a multiplexing format such as Streaming over HTTP) or RTP (Real-time Transport Protocol), the basic unit can be set in the NAL unit. In order to store one slice segment in one NAL unit, it is desirable to divide the access unit into slice segment units when dividing the access unit into areas. For this reason, the transmitter associates tiles with slice segments in a one-to-one relationship.

As shown in FIG. 4, the transmission device can collectively set tiles 1 to 4 into one slice segment. However, in this case, all the tiles are stored in one NAL unit, and it is difficult for the receiving device to separate the tiles in the multiplexing layer.

The slice segment includes an independent slice segment that can be independently decoded and a reference slice segment that refers to the independent slice segment. Here, a case where the independent slice segment is used will be described.

FIG. 5 is a diagram showing an example of data of an access unit divided so that the boundaries between the tile and the slice segment match as shown in FIG. The data of the access unit includes the NAL unit in which the access unit delimiter placed at the beginning is stored, the NAL units of SPS, PPS, and SEI placed after that, and tiles 1 to 4 placed after that. Includes data from the slice segment in which the data is stored. The data of the access unit may not include a part or all of the NAL units of SPS, PPS and SEI.

Next, the configuration of the transmission device 100 according to the present embodiment will be described. FIG. 6 is a block diagram showing a configuration example of the transmission device 100 according to the present embodiment. The transmission device 100 includes a coding unit 101, a multiplexing unit 102, a modulation unit 103, and a transmission unit 104.

The coding unit 101 converts the input image into, for example, H.I. Encoded data is generated by encoding according to 265. Further, the coding unit 101 divides the access unit into four slice segments (tiles) and encodes each slice segment, for example, as shown in FIG.

The multiplexing unit 102 multiplexes the coded data generated by the coding unit 101. The modulation unit 103 modulates the data obtained by the multiplexing. The transmission unit 104 transmits the modulated data as a broadcast signal.

Next, the configuration of the receiving device 200 according to the present embodiment will be described. FIG. 7 is a block diagram showing a configuration example of the receiving device 200 according to the present embodiment. The receiving device 200 includes a tuner 201, a demodulation unit 202, a demultiplexing unit 203, a plurality of decoding units 204A to 204D, and a display unit 205.

The tuner 201 receives the broadcast signal. The demodulation unit 202 demodulates the received broadcast signal. The demodulated data is input to the demultiplexing unit 203.

The demultiplexing unit 203 separates the demodulated data into division units, and outputs the data for each division unit to the decoding units 204A to 204D. Here, the division unit is a division area obtained by dividing the access unit, and is, for example, H.A. The slice segment at 265. Further, here, the 8K4K image is divided into four 4K2K images. Therefore, there are four decoding units 204A to 204D.

The plurality of decoding units 204A to 204D operate in synchronization with each other based on a predetermined reference clock. Each decoding unit decodes the coded data of the division unit according to the DTS (Decoding Time Stamp) of the access unit, and outputs the decoding result to the display unit 205.

The display unit 205 generates an 8K4K output image by integrating a plurality of decoding results output from the plurality of decoding units 204A to 204D. The display unit 205 displays the generated output image according to the PTS (Presentation Time Stamp) of the access unit acquired separately. When integrating the decoding results, the display unit 205 performs filter processing such as a deblock filter so that the boundaries are not visually conspicuous in the boundary areas of the division units adjacent to each other, such as the boundaries of tiles. May be good.

In the above description, the transmitting device 100 and the receiving device 200 that transmit or receive the broadcast have been described as an example, but the content may be transmitted and received via the communication network. When the receiving device 200 receives the content via the communication network, the receiving device 200 separates the multiplexed data from the IP packet received by the network such as Ethernet.

In broadcasting, the transmission line delay from the transmission of the broadcast signal to the arrival at the receiving device 200 is constant. On the other hand, in a communication network such as the Internet, the transmission path delay until the data transmitted from the server reaches the receiving device 200 is not constant due to the influence of congestion. Therefore, the receiving device 200 often does not perform strict synchronous reproduction based on the reference clock as in PCR in the MPEG-2 TS of broadcasting. Therefore, the receiving device 200 may display the output image of 8K4K according to the PTS on the display unit without strictly synchronizing the decoding units.

Further, due to congestion of the communication network or the like, the decoding process of all the division units may not be completed at the time indicated by the PTS of the access unit. In this case, the receiving device 200 skips the display of the access unit, or delays the display until the decoding of at least four division units is completed and the generation of the 8K4K image is completed.

The content may be transmitted and received in combination with broadcasting and communication. In addition, this method can also be applied when reproducing multiplexed data stored in a recording medium such as a hard disk or a memory.

Next, when MMT is used as the multiplexing method, a method for multiplexing access units divided into slice segments will be described.

FIG. 8 is a diagram showing an example of packetizing the data of the access unit of HEVC into MMT. SPS, PPS, SEI, etc. do not necessarily have to be included in the access unit, but the case where they exist is illustrated here.

NAL units arranged before the first slice segment in the access unit such as the access unit delimiter, SPS, PPS, and SEI are collectively stored in the MMT packet # 1. Subsequent slice segments are stored in separate MMT packets for each slice segment.

As shown in FIG. 9, the NAL unit arranged before the first slice segment in the access unit may be stored in the same MMT packet as the first slice segment.

Also, if NAL units such as End-of-Sequence or End-of-Bitstream indicating the end of the sequence or stream are added after the final slice segment, they will be in the same MMT packet as the final slice segment. Stored. However, since NAL units such as End-of-Sequence or End-of-Bitstream are inserted at the end point of the decoding process or the connection point of two streams, the receiving device 200 inserts these NAL units. , It may be desirable to be easily available in the multiplexing layer. In this case, these NAL units may be stored in an MMT packet separate from the slice segment. Thereby, the receiving device 200 can easily separate these NAL units in the multiplexing layer.

As the multiplexing method, TS, DASH, RTP, or the like may be used. Also in these methods, the transmission device 100 stores different slice segments in different packets. This can ensure that the receiving device 200 can separate slice segments in the multiplexing layer.

For example, when TS is used, the coded data is PES packetized in slice segment units. When RTP is used, the coded data is RTP packetized in slice segment units. Even in these cases, the NAL unit and the slice segment arranged before the slice segment may be packetized separately as in MMT packet # 1 shown in FIG.

When TS is used, the transmission device 100 indicates a unit of data stored in a PES packet, such as by using a data alignment descriptor. Further, since DASH is a method of downloading MP4 format data units called segments by HTTP or the like, the transmission device 100 does not packetize the coded data at the time of transmission. Therefore, the transmitting device 100 creates a subsample for each slice segment so that the receiving device 200 can detect the slice segment in the multiplexing layer in the MP4, and stores the information indicating the storage position of the subsample in the header of the MP4. You may.

Hereinafter, MMT packetization of slice segments will be described in detail.

As shown in FIG. 8, by packetizing the coded data, the data commonly referred to at the time of decoding all the slice segments in the access unit such as SPS and PPS is stored in the MMT packet # 1. In this case, the receiving device 200 concatenates the payload data of the MMT packet # 1 with the data of each slice segment, and outputs the obtained data to the decoding unit. In this way, the receiving device 200 can easily generate input data to the decoding unit by concatenating the payloads of a plurality of MMT packets.

FIG. 10 is a diagram showing an example in which input data to the decoding units 204A to 204D is generated from the MMT packet shown in FIG. The demultiplexing unit 203 concatenates the payload data of the MMT packet # 1 and the MMT packet # 2, so that the decoding unit 204A generates the data necessary for decoding the slice segment 1. The demultiplexing unit 203 similarly generates input data from the decoding unit 204B to the decoding unit 204D. That is, the demultiplexing unit 203 generates the input data of the decoding unit 204B by concatenating the payload data of the MMT packet # 1 and the MMT packet # 3. The demultiplexing unit 203 generates the input data of the decoding unit 204C by concatenating the payload data of the MMT packet # 1 and the MMT packet # 4. The demultiplexing unit 203 generates the input data of the decoding unit 204D by concatenating the payload data of the MMT packet # 1 and the MMT packet # 5.

The demultiplexing unit 203 removes NAL units such as the access unit delimiter and SEI, which are not necessary for the decoding process, from the payload data of the MMT packet # 1, and only the SPS and PPS NAL units required for the decoding process. May be separated and added to the data of the slice segment.

When the coded data is packetized as shown in FIG. 9, the demultiplexing unit 203 outputs the MMT packet # 1 including the head data of the access unit in the multiplexing layer to the first decoding unit 204A. .. Further, the demultiplexing unit 203 analyzes the MMT packet including the head data of the access unit in the multiplexing layer, separates the SPS and PPS NAL units, and slices the separated SPS and PPS NAL units from the second onward. By adding to each of the segment data, input data for each of the second and subsequent decoding units is generated.

Further, the receiving device 200 uses the information contained in the header of the MMT packet to store the type of data stored in the MMT payload, and when the slice segment is stored in the payload, the slice segment in the access unit. It is desirable to be able to identify the index number. Here, the data types are the data before the slice segment (the NAL units arranged before the first slice segment in the access unit are collectively referred to as this), and the data of the slice segment. Either. When storing a unit obtained by fragmenting an MPU such as a slice segment in an MMT packet, a mode for storing an MFU (Media Fragment Unit) is used. When this mode is used, the transmission device 100 uses, for example, a Data Unit, which is a basic unit of data in the MFU, as a sample (a data unit in the MMT, which corresponds to an access unit) or a subsample (a sample). Can be set to a divided unit).

At this time, the header of the MMT packet includes a field called a Fragmentation indicator and a field called a Fragmentation counter.

The Fragmentation indicator indicates whether the data stored in the payload of the MMT packet is a fragment of the Data unit, and if it is a fragment, the fragment is the first or last fragment in the Data unit, or , Indicates whether the fragment is neither the beginning nor the end. In other words, the Fragmentation indicator included in the header of a certain packet is (1) the Data unit, which is the basic data unit, contains only the packet, and (2) the Data unit is divided into a plurality of packets and stored. The packet is the first packet of the Data unit, (3) the Data unit is divided and stored in a plurality of packets, and the packet is a packet other than the beginning and the end of the Data unit, and (4). This is identification information indicating whether the Data unit is divided into a plurality of packets and stored, and the packet is the last packet of the Data unit.

The Fragment counter is an index number indicating which fragment the data stored in the MMT packet corresponds to in the Data unit.

Therefore, the transmitting device 100 sets the sample in the MMT to the Data unit, and sets the pre-slice segment data and each slice segment to each fragment unit of the Data unit, so that the receiving device 200 sets the header of the MMT packet. The information contained in can be used to identify the type of data stored in the payload. That is, the demultiplexing unit 203 can generate input data to the decoding units 204A to 204D by referring to the header of the MMT packet.

FIG. 11 is a diagram showing an example in which the sample is set to Data unit and the pre-slice segment data and the slice segment are packetized as a fragment of Data unit.

The pre-slice segment data and the slice segment are divided into five fragments from fragment # 1 to fragment # 5. Each fragment is stored in a separate MMT packet. At this time, the values of the Fragmentation indicator and the Fragmentation counter included in the header of the MMT packet are as shown in the figure.

For example, Fragment indicator is a binary 2-bit value. The Fragment indicator of MMT packet # 1 at the beginning of Data unit, the Fragment indicator of MMT packet # 5 at the end, and the Fragment indicator from MMT packet # 2 to MMT packet # 4 which are packets in between are different from each other. Set to a value. Specifically, the Fragment indicator of the MMT packet # 1 at the beginning of the Data unit is set to 01, the Fragment indicator of the final MMT packet # 5 is set to 11, and from the MMT packet # 2 which is a packet in between. The Fragment indicator up to MMT packet # 4 is set to 10. If the Data unit contains only one MMT packet, the Fragment indicator is set to 00.

Further, the Fragment counter is 4 which is a value obtained by subtracting 1 from 5 which is the total number of fragments in MMT packet # 1, decreases by 1 in order in subsequent packets, and is 0 in the final MMT packet # 5. be.

Therefore, the receiving device 200 can identify the MMT packet storing the pre-slice segment data by using either the Fragment indicator or the Fragment counter. Further, the receiving device 200 can identify the MMT packet containing the Nth slice segment by referring to the Fragment counter.

The header of the MMT packet separately includes the sequence number in the MPU of the Movie Fragment to which the Data unit belongs, the sequence number of the MPU itself, and the sequence number in the Movie Fragment of the sample to which the Data unit belongs. By referring to these, the demultiplexing unit 203 can uniquely determine the sample to which the Data unit belongs.

Further, since the demultiplexing unit 203 can determine the index number of the fragment in the Data unit from the Fragment counter or the like, the slice segment stored in the fragment can be uniquely specified even when packet loss occurs. For example, even if the fragment # 4 shown in FIG. 11 cannot be acquired due to packet loss, the demultiplexing unit 203 knows that the fragment received next to the fragment # 3 is the fragment # 5, so that the fragment # 5 is used. The slice segment 4 stored in can be correctly output to the decoding unit 204D instead of the decoding unit 204C.

When a transmission line that guarantees no packet loss is used, the demultiplexing unit 203 refers to the header of the MMT packet and refers to the type of data stored in the MMT packet or the slice segment. The arriving packet may be processed periodically without determining the index number of. For example, if the access unit is transmitted by pre-slice data and a total of five MMT packets of four slice segments, the receiving device 200 has determined the pre-slice data of the access unit to start decoding. By processing the received MMT packets in order, the pre-slice data and the data of the four slice segments can be acquired in order.

Hereinafter, a modified example of packetization will be described.

The slice segment does not necessarily have to be divided in both the horizontal direction and the vertical direction in the plane of the access unit, and the access unit may be divided only in the horizontal direction as shown in FIG. However, it may be divided only in the vertical direction.

Also, if the access unit is divided only in the horizontal direction, tiles do not need to be used.

Further, the number of in-plane divisions in the access unit is arbitrary and is not limited to four. However, the area size of slice segments and tiles is H. It must be at least the lower limit of a coding standard such as 265.

The transmission device 100 may store identification information indicating an in-plane division method in the access unit in an MMT message, a TS descriptor, or the like. For example, information indicating the number of divisions in the horizontal direction and the vertical direction in the plane may be stored. Alternatively, the identification information peculiar to the division method, such as being divided into two equal parts in the horizontal direction and the vertical direction as shown in FIG. 3, or being divided into four equal parts in the horizontal direction as shown in FIG. May be assigned. For example, when the access unit is divided as shown in FIG. 3, the identification information indicates mode 1, and when the access unit is divided as shown in FIG. 1, the identification information indicates mode 1. ..

Further, the multiplexing layer may include information indicating restrictions on the coding conditions related to the in-plane division method. For example, information indicating that one slice segment is composed of one tile may be used. Alternatively, the reference block for motion compensation when decoding a slice segment or tile is limited to the slice segment or tile at the same position on the screen, or the block within a predetermined range in the adjacent slice segment. Information indicating that, etc. may be used.

Further, the transmission device 100 may switch whether or not to divide the access unit into a plurality of slice segments according to the resolution of the moving image. For example, the transmission device 100 does not perform in-plane division when the moving image to be processed has a resolution of 4K2K, and may divide the access unit into four when the moving image to be processed is 8K4K. good. By prescribing the division method for the 8K4K moving image in advance, the receiving device 200 determines the presence / absence of in-plane division and the dividing method by acquiring the resolution of the received moving image, and decodes the image. The operation can be switched.

Further, the receiving device 200 can detect the presence or absence of in-plane division by referring to the header of the MMT packet. For example, when the access unit is not divided and the Data unit of MMT is set in the sample, the fragment of Data unit is not performed. Therefore, the receiving device 200 can determine that the access unit is not divided when the value of the Fragment counter included in the header of the MMT packet is always zero. Alternatively, the receiving device 200 may detect whether or not the value of the Fragmentation indicator is always 01. The receiving device 200 can determine that the access unit is not divided even when the value of the Fragmentation indicator is always 01.

Further, the receiving device 200 can handle a case where the number of divisions in the plane of the access unit and the number of decoding units do not match. For example, if the receiving device 200 includes two decoding units 204A and 204B capable of decoding 8K2K encoded data in real time, the demultiplexing unit 203 may refer to the decoding unit 204A with 8K4K encoded data. Outputs two of the four slice segments that make up.

FIG. 12 is a diagram showing an operation example when the MMT packetized data as shown in FIG. 8 is input to the two decoding units 204A and 204B. Here, it is desirable that the receiving device 200 can integrate and output the decoding results of the decoding units 204A and 204B as they are. Therefore, the demultiplexing unit 203 selects the slice segment to be output to each of the decoding units 204A and 204B so that the decoding results of the decoding units 204A and 204B are spatially continuous.

Further, the demultiplexing unit 203 may select the decoding unit to be used according to the resolution or frame rate of the coded data of the moving image. For example, when the receiving device 200 includes four 4K2K decoding units, if the resolution of the input image is 8K4K, the receiving device 200 performs the decoding process using all four decoding units. Further, if the resolution of the input image is 4K2K, the receiving device 200 performs the decoding process using only one decoding unit. Alternatively, if the demultiplexing unit 203 can decode 8K4K in real time by a single decoding unit even if the in-plane is divided into four, the demultiplexing unit 203 integrates all the division units into one decoding unit. Output to.

Further, the receiving device 200 may determine the decoding unit to be used in consideration of the frame rate. For example, when the receiving device 200 includes two decoding units whose upper limit of the frame rate that can be decoded in real time is 60 fps when the resolution is 8K4K, there is a case where 120 fps coded data is input at 8K4K. be. At this time, assuming that the in-plane is composed of four division units, the slice segment 1 and the slice segment 2 are input to the decoding unit 204A, and the slice segment 3 and the slice segment 4 are combined, as in the example of FIG. It is input to the decoding unit 204B. Since each of the decoding units 204A and 204B can decode up to 120 fps in real time if the resolution is 8K2K (the resolution is half of 8K4K), the decoding process is performed by these two decoding units 204A and 204B.

Further, even if the resolution and the frame rate are the same, the profile or level in the coding method, or H.I. 264 or H. If the coding method itself such as 265 is different, the processing amount will be different. Therefore, the receiving device 200 may select a decoding unit to be used based on this information. If the receiving device 200 cannot decode all the coded data received by broadcasting or communication, or if all the slice segments or tiles constituting the area selected by the user cannot be decoded, the decoding unit Decryptable slice segments or tiles may be automatically determined within the processing range of. Alternatively, the receiving device 200 may provide a user interface for the user to select an area to be decoded. At this time, the receiving device 200 may display a warning message indicating that all the areas cannot be decoded, or may display information indicating the number of decodeable areas, slice segments, or tiles.

Further, the above method can be applied to a case where an MMT packet storing a slice segment of the same coded data is transmitted and received using a plurality of transmission paths such as broadcasting and communication.

Further, the transmission device 100 may perform coding so that the regions of the slice segments overlap each other in order to make the boundaries of the division units inconspicuous. In the example shown in FIG. 13, the 8K4K picture is divided into four slice segments 1 to 4. Each of the slice segments 1 to 3 is, for example, 8K × 1.1K, and the slice segment 4 is 8K × 1K. Also, adjacent slice segments overlap each other. By doing so, at the boundary in the case of being divided into four, which is indicated by the dotted line, motion compensation at the time of coding can be efficiently executed, so that the image quality of the boundary portion is improved. In this way, deterioration of image quality at the boundary portion is reduced.

In this case, the display unit 205 cuts out an 8K × 1K area from the 8K × 1.1K area and integrates the obtained area. In addition, the transmission device 100 separately transmits information indicating whether or not the slice segments are overlapped and encoded and the range of the overlap is included in the multiplexing layer or the encoded data. May be good.

The same method can be applied when tiles are used.

Hereinafter, the operation flow of the transmission device 100 will be described. FIG. 14 is a flowchart showing an operation example of the transmission device 100.

First, the coding unit 101 divides the picture (access unit) into a plurality of slice segments (tiles) which are a plurality of regions (S101). Next, the coding unit 101 encodes each of the plurality of slice segments so that they can be independently decoded, thereby generating coded data corresponding to each of the plurality of slice segments (S102). The coding unit 101 may encode a plurality of slice segments with a single coding unit, or may perform parallel processing by the plurality of coding units.

Next, the multiplexing unit 102 multiplexes the plurality of coded data by storing the plurality of coded data generated by the coding unit 101 in a plurality of MMT packets (S103). Specifically, as shown in FIGS. 8 and 9, the multiplexing unit 102 stores a plurality of coded data in a plurality of coded data so that the coded data corresponding to different slice segments is not stored in one MMT packet. Store in MMT packet. Further, as shown in FIG. 8, the multiplexing unit 102 stores control information commonly used for all decoding units in the picture, and a plurality of MMT packets # 2 to # in which a plurality of coded data are stored. It is stored in MMT packet # 1, which is different from 5. Here, the control information includes at least one of the access unit delimiter, SPS, PPS and SEI.

The multiplexing unit 102 may store the control information in the same MMT packet as any one of the plurality of MMT packets in which the plurality of coded data are stored. For example, as shown in FIG. 9, the multiplexing unit 102 stores the control information in the first MMT packet (MMT packet # 1 in FIG. 9) among the plurality of MMT packets in which the plurality of coded data are stored. You may.

Finally, the transmission device 100 transmits a plurality of MMT packets. Specifically, the modulation unit 103 modulates the data obtained by the multiplexing, and the transmission unit 104 transmits the modulated data (S104).

FIG. 15 is a block diagram showing a configuration example of the receiving device 200, and is a diagram showing in detail the configuration of the demultiplexing unit 203 and the subsequent stage shown in FIG. 7. As shown in FIG. 15, the receiving device 200 further includes a decoding instruction unit 206. Further, the demultiplexing unit 203 includes a type discrimination unit 211, a control information acquisition unit 212, a slice information acquisition unit 213, and a decoding data generation unit 214.

Hereinafter, the operation flow of the receiving device 200 will be described. FIG. 16 is a flowchart showing an operation example of the receiving device 200. Here, the operation for one access unit is shown. When the decryption process of a plurality of access units is executed, the process of this flowchart is repeated.

First, the receiving device 200 receives, for example, a plurality of packets (MMT packets) generated by the transmitting device 100 (S201).

Next, the type determination unit 211 acquires the type of coded data stored in the received packet by analyzing the header of the received packet (S202).

Next, the type determination unit 211 determines whether the data stored in the received packet is the pre-slice segment data or the slice segment data based on the type of the acquired coded data (S203). ..

When the data stored in the received packet is the pre-slice segment data (Yes in S203), the control information acquisition unit 212 acquires the pre-slice segment data of the access unit to be processed from the payload of the received packet, and the slice concerned. The pre-segment data is stored in the memory (S204).

On the other hand, when the data stored in the received packet is slice segment data (No in S203), the receiving device 200 uses the header information of the received packet to display a plurality of data stored in the received packet. It is determined which region of the regions is the coded data. Specifically, the slice information acquisition unit 213 acquires the index number Idx of the slice segment stored in the received packet by analyzing the header of the received packet (S205). Specifically, the index number Idx is an index number in the Movie Fragment of the access unit (sample in MMT).

The process of step S205 may be collectively performed in step S202.

Next, the decoding data generation unit 214 determines a decoding unit for decoding the slice segment (S206). Specifically, the index number Idx and the plurality of decoding units are associated with each other in advance, and the decoding data generation unit 214 decodes the slice segment of the decoding unit corresponding to the index number Idx acquired in step S205. Determine the decoding unit to be used.

As described in the example of FIG. 12, the decoding data generation unit 214 has a resolution of the access unit (picture), a method of dividing the access unit into a plurality of slice segments (tiles), and a plurality of reception devices 200. The decoding unit that decodes the slice segment may be determined based on at least one of the processing capabilities of the decoding unit. For example, the decoding data generation unit 214 determines the method of dividing the access unit based on the identification information in the descriptor such as the message of MMT or the section of TS.

Next, the decoding data generation unit 214 includes control information commonly used for all decoding units in the picture contained in any of the plurality of packets, and each of the plurality of coded data of the plurality of slice segments. By combining with and, a plurality of input data (combined data) to be input to a plurality of decoding units is generated. Specifically, the decoding data generation unit 214 acquires slice segment data from the payload of the received packet. The decoding data generation unit 214 generates input data to the decoding unit determined in step S206 by combining the pre-slice segment data stored in the memory in step S204 and the acquired slice segment data. (S207).

After step S204 or S207, if the data of the received packet is not the final data of the access unit (No in S208), the processing after step S201 is performed again. That is, the above process is repeated until the input data to the plurality of decoding units 204A to 204D corresponding to all the slice segments included in the access unit are generated.

The timing at which the packets are received is not limited to the timing shown in FIG. 16, and a plurality of packets may be received in advance or sequentially and stored in a memory or the like.

On the other hand, when the data of the received packet is the final data of the access unit (Yes in S208), the decoding command unit 206 outputs a plurality of input data generated in step S207 to the corresponding decoding units 204A to 204D. (S209).

Next, the plurality of decoding units 204A to 204D generate a plurality of decoded images by decoding a plurality of input data in parallel according to the DTS of the access unit (S210).

Finally, the display unit 205 generates a display image by combining the plurality of decoded images generated by the plurality of decoding units 204A to 204D, and displays the displayed image according to the PTS of the access unit (S211).

The receiving device 200 acquires the DTS and PTS of the access unit by analyzing the payload data of the MMT packet that stores the header information of the MPU or the header information of the Movie Fragment. Further, when TS is used as the multiplexing method, the receiving device 200 acquires the DTS and PTS of the access unit from the header of the PES packet. When RTP is used as the multiplexing method, the receiving device 200 acquires the DTS and PTS of the access unit from the header of the RTP packet.

Further, the display unit 205 may perform a filter process such as a deblock filter at the boundary of adjacent division units when integrating the decoding results of the plurality of decoding units. Since the filtering process is not required when displaying the decoding result of a single decoding unit, the display unit 205 performs the processing depending on whether or not the filtering process is performed on the boundary of the decoding result of the plurality of decoding units. It may be switched. Whether or not the filtering process is necessary may be predetermined depending on the presence or absence of division and the like. Alternatively, information indicating whether filtering is necessary may be separately stored in the multiplexing layer. In addition, information necessary for filter processing such as filter coefficients may be stored in the SPS, PPS, SEI, or slice segment. The decoding units 204A to 204D or the demultiplexing unit 203 acquire these information by analyzing the SEI, and output the acquired information to the display unit 205. The display unit 205 performs filtering processing using these information. When these information are stored in the slice segment, it is desirable that the decoding units 204A to 204D acquire the information.

In the above description, an example is shown in which the types of data stored in the fragment are two types, the pre-slice segment data and the slice segment, but the data types may be three or more. In this case, the case classification according to the type is performed in step S203.

Further, the transmission device 100 may fragment the slice segment and store it in the MMT packet when the data size of the slice segment is large. That is, the transmission device 100 may fragment the pre-slice segment data and the slice segment. In this case, if the access unit and the Data unit are set to be equal as in the packetization example shown in FIG. 11, the following problems occur.

For example, when the slice segment 1 is divided into three fragments, the slice segment 1 is divided into three packets having a Fragment counter value of 1 to 3 and transmitted. Further, in the slice segment 2 and later, the Fragment counter value becomes 4 or more, and the value of the Fragment counter cannot be associated with the data stored in the payload. Therefore, the receiving device 200 cannot specify the packet that stores the head data of the slice segment from the information in the header of the MMT packet.

In such a case, the receiving device 200 may analyze the data of the payload of the MMT packet to specify the start position of the slice segment. Here, H. 264 or H. In 265, as a format for storing the NAL unit in the multiplexing layer, a format called a byte stream format in which a start code consisting of a specific bit string is added immediately before the NAL unit header, and a field indicating the size of the NAL unit are added. There are two types, a format called the NAL size format.

The byte stream format is used in MPEG-2 systems and RTPs and the like. The NAL size format is used in MP4, as well as DASH and MMT using MP4.

When the byte stream format is used, the receiving device 200 analyzes whether the head data of the packet matches the start code. If the start data of the packet matches the start code, the receiving device 200 obtains the type of the NAL unit from the subsequent NAL unit header, and whether the data included in the packet is the slice segment data. You can detect if.

On the other hand, in the case of the NAL size format, the receiving device 200 cannot detect the start position of the NAL unit based on the bit string. Therefore, in order to acquire the start position of the NAL unit, the receiving device 200 needs to shift the pointer by reading data by the size of the NAL unit in order from the head NAL unit of the access unit.

However, if the size of the subsample unit is indicated in the header of the MPU or Movie Fragment in the MMT and the subsample corresponds to the pre-slice data or the slice segment, the receiving device 200 is based on the size information of the subsample. The starting position of each NAL unit can be specified. Therefore, the transmitting device 100 may include information indicating whether or not the subsample unit information exists in the MPU or Movie Fragment in the information acquired by the receiving device 200 at the start of data reception, such as the MPT in the MMT. ..

The MPU data is an extension based on the MP4 format. In MP4, H. 264 or H. There is a mode in which the parameter set such as 265 SPS and PPS can be stored as sample data, and a mode in which the parameter set cannot be stored. In addition, information for identifying this mode is shown as the entry name of SampleEntry. When a storable mode is used and the parameter set is included in the sample, the receiver 200 acquires the parameter set by the method described above.

On the other hand, when the non-storable mode is used, the parameter set is stored as a Decoder Specific Information in SampleEntry, or is stored using a stream for the parameter set. Here, since the stream for the parameter set is not generally used, it is desirable that the transmitting device 100 stores the parameter set in the Decoder Special Information. In this case, the receiving device 200 analyzes the SampleEntry transmitted as the metadata of the MPU or the metadata of the Movie Fragment in the MMT packet, and acquires the parameter set referred to by the access unit.

When the parameter set is stored as sample data, the receiving device 200 can acquire the parameter set necessary for decoding by referring only to the sample data without referring to SampleEntry. At this time, the transmission device 100 does not have to store the parameter set in the SampleEntry. By doing so, the transmission device 100 can use the same SampleEntry in different MPUs, so that the processing load of the transmission device 100 at the time of generating the MPU can be reduced. Further, there is an advantage that the receiving device 200 does not need to refer to the parameter set in the SampleEntry.

Alternatively, the transmission device 100 may store one default parameter set in the SampleEntry and store the parameter set referred to by the access unit in the sample data. In the conventional MP4, since the parameter set is generally stored in the SampleEntry, if the parameter set does not exist in the SampleEntry, there may be a receiving device that stops the reproduction. This problem can be solved by using the above method.

Alternatively, the transmitter 100 may store the parameter set in the sample data only if a parameter set different from the default parameter set is used.

Since it is possible to store the parameter set in the SampleEntry in both modes, the transmitting device 100 may always store the parameter set in the VisualSampleEntry, and the receiving device 200 may always acquire the parameter set from the VisualSampleEntry.

In the MMT standard, the header information of MP4 such as Moov and Moof is transmitted as MPU metadata or movie fragment metadata, but the transmission device 100 does not necessarily transmit MPU metadata and movie fragment metadata. You don't have to. Further, the receiving device 200 determines whether SPS and PPS are stored in the sample data based on the service of the ARIB (Association of Radio Industries and Businesses) standard, the type of the asset, the presence / absence of transmission of the MPU meta, and the like. It is also possible to judge.

FIG. 17 is a diagram showing an example in which the pre-slice segment data and each slice segment are set to different Data units.

In the example shown in FIG. 17, the pre-slice segment data and the data sizes from slice segment 1 to slice segment 4 are Length # 1 to Length # 5, respectively. The Field values of Fragmentation indicator, Fragment counter, and Offset included in the header of the MMT packet are as shown in the figure.

Here, Offset indicates the bit length (offset) from the beginning of the coded data of the sample (access unit or picture) to which the payload data belongs to the first byte of the payload data (encoded data) included in the MMT packet. Offset information. Although the Fragment counter value is described as starting from a value obtained by subtracting 1 from the total number of fragments, it may start from another value.

FIG. 18 is a diagram showing an example when the Data unit is fragmented. In the example shown in FIG. 18, the slice segment 1 is divided into three fragments and stored in MMT packet # 2 to MMT packet # 4, respectively. Also in this case, assuming that the data size of each fragment is Length # 2_1 to Length # 2_3, the values of each field are as shown in the figure.

As described above, when the data unit such as the slice segment is set to Data unit, the head of the access unit and the head of the slice segment can be determined as follows based on the field value of the MMT packet header.

The head of the payload in the packet whose Offset value is 0 is the head of the access unit.

The beginning of the payload of the packet whose Offset value is different from 0 and whose Fragmentation indicator value is 00 or 01 is the beginning of the slice segment.

Further, when the data unit fragmentation does not occur and the packet loss does not occur, the receiving device 200 is stored in the MMT packet based on the number of slice segments acquired after detecting the head of the access unit. You can identify the index number of the slice segment.

Further, even when the data unit of the data before the slice segment is fragmented, the receiving device 200 can similarly detect the access unit and the head of the slice segment.

Further, even when packet loss occurs or when SPS, PDU, and SEI included in the pre-slice segment data are set to different Data units, the receiving device 200 slices based on the analysis result of the MMT header. By identifying the MMT packet that stores the start data of the segment and then analyzing the header of the slice segment, the start position of the slice segment or tile in the picture (access unit) can be identified. The amount of processing related to the analysis of the slice header is small, and the processing load does not matter.

As described above, each of the plurality of coded data of the plurality of slice segments is associated one-to-one with the basic data unit (Data unit) which is a unit of data stored in one or more packets. Further, each of the plurality of coded data is stored in one or more MMT packets.

The header information of each MMT packet includes Fragmentation indicator (identification information) and Offset (offset information).

The receiving device 200 determines that the receiving device 200 is the head of the encoded data of each slice segment at the head of the payload data included in the packet having the header information including the Fragmentation indicator having a value of 00 or 01. .. Specifically, the head of the payload data included in the packet having the header information including the Offset whose value is not 0 and the Fragmentation indicator whose value is 00 or 01 is the head of the coded data of each slice segment. Is determined.

Further, in the example of FIG. 17, the head of the Data unit is either the head of the access unit or the head of the slice segment, and the value of the Fragmentation indicator is 00 or 01. Further, the receiving device 200 refers to the type of the NAL unit and determines whether the head of the Data Unit is the access unit delimiter or the slice segment. It is also possible to detect the beginning of the slice segment or the beginning of the slice segment.

In this way, the transmitting device 100 packetizes the NAL unit so that the beginning of the NAL unit always starts from the beginning of the payload of the MMT packet, so that the pre-slice segment data is divided into a plurality of Data units. The receiving device 200 can detect the head of the access unit or the slice segment by analyzing the Packet indicator and the NAL unit header. The type of NAL unit exists in the first byte of the NAL unit header. Therefore, the receiving device 200 can acquire the type of the NAL unit by additionally analyzing one byte of data when analyzing the header portion of the MMT packet.

In the case of audio, the receiving device 200 may detect the head of the access unit, and may determine based on whether the value of the Fragmentation indicator is 00 or 01.

Further, as described above, when the coded data encoded so as to be able to be divided and decoded is stored in the PES packet of the MPEG-2 TS, the transmitting device 100 can use the data alignment descriptor. be. Hereinafter, an example of a method of storing the coded data in the PES packet will be described in detail.

For example, in HEVC, the transmitting device 100 can use the data alignment descriptor to indicate whether the data stored in the PES packet is an access unit, a slice segment, or a tile. The types of alignment in HEVC are defined as follows:

Alignment type = 8 indicates a slice segment of HEVC. Alignment type = 9 indicates a slice segment or access unit of HEVC. Alignment type = 12 indicates a slice segment or tile of HEVC.

Therefore, the transmitting device 100 can indicate that the data of the PES packet is either the slice segment or the pre-slice segment data by using, for example, type 9. Since the type indicating a slice instead of the slice segment is also specified separately, the transmission device 100 may use the type indicating a slice instead of the slice segment.

Further, the DTS and PTS included in the header of the PES packet are set only in the PES packet including the head data of the access unit. Therefore, in the receiving device 200, if the type is 9 and the DTS or PTS field is present in the PES packet, the entire access unit or the first division unit in the access unit is stored in the PES packet. It can be judged.

Further, even if the transmitting device 100 uses a field such as transport_priority indicating the priority of the TS packet that stores the PES packet including the head data of the access unit, the receiving device 200 can distinguish the data included in the packet. good. Further, the receiving device 200 may determine the data included in the packet by analyzing whether or not the payload of the PES packet is the access unit delimiter. Also, the data_alignnment_indicator in the PES packet header indicates whether data is stored in the PES packet according to these types. If this flag (data_alignnment_indicator) is set to 1, the data stored in the PES packet is guaranteed to follow the type shown in the data alignment descriptor.

Further, the transmission device 100 may use the data alignment descriptor only when the PES packet is formed in a unit that can be divided and decoded such as a slice segment. As a result, the receiving device 200 can determine that the coded data is PES packetized in units that can be divided and decoded when the data alignment descriptor exists, and if the data alignment descriptor does not exist, the code is coded. It can be determined that the converted data is converted into a PES packet for each access unit. If data_alignnment_indicator is set to 1 and the data alignment descriptor does not exist, it is specified in the MPEG-2 TS standard that the unit of PES packetization is an access unit.

If the data alignment descriptor is included in the PMT, the receiving device 200 determines that the PES packet is formed in units that can be divided and decoded, and inputs to each decoding unit based on the packetized unit. Data can be generated. Further, when the receiving device 200 does not include the data identification descriptor in the PMT and it is determined that parallel decoding of the coded data is necessary based on the program information or other descriptor information, the receiving device 200 does not include the data alignment descriptor. , The input data to each decoding unit is generated by analyzing the slice header of the slice segment. When the coded data can be decoded by a single decoding unit, the receiving device 200 decodes the data of the entire access unit by the decoding unit. When the information indicating whether or not the coded data is composed of a unit that can be divided and decoded such as a slice segment or a tile is separately indicated by a PMT descriptor or the like, the receiving device 200 uses the descriptor of the descriptor. It may be determined whether or not the coded data can be decoded in parallel based on the analysis result.

Further, since the DTS and PTS included in the header of the PES packet are set only in the PES packet including the head data of the access unit, when the access unit is divided into PES packets, the second and subsequent PESs are set. The packet does not contain information indicating the DTS and PTS of the access unit. Therefore, when the decoding processes are performed in parallel, the decoding units 204A to 204D and the display unit 205 use DTS and PTS stored in the header of the PES packet including the head data of the access unit.

(Embodiment 2)
In the second embodiment, a method of storing NAL size format data in an MP4 format-based MPU in MMT will be described. In the following, a storage method in the MPU used for MMT will be described as an example, but such a storage method can also be applied to DASH, which is the same MP4 format base.

[How to store in MPU]
In the MP4 format, a plurality of access units are collectively stored in one MP4 file. In the MPU used for MMT, data for each medium is stored in one MP4 file, and the data can include any number of access units. Since the MPU is a unit that can be decoded by itself, for example, an access unit in the GOP unit is stored in the MPU.

FIG. 19 is a diagram showing the configuration of the MPU. The head of the MPU is ftyp, mmpu, and moov, which are collectively defined as MPU metadata. Initialization information common to files and MMT hint tracks are stored in moov.

Further, in the moof, initialization information and size for each sample or subsample, information that can specify the presentation time (PTS) and the decoding time (DTS) (sample_duration, sample_size, sample_composition_time_offset), data_offset indicating the position of the data, and the like are included. Stored.

Further, each of the plurality of access units is stored in mdat (mdat box) as a sample. The data of moof and mdat excluding the sample is defined as movie fragment metadata (hereinafter referred to as MF metadata), and the sample data of mdat is defined as media data.

FIG. 20 is a diagram showing the structure of MF metadata. As shown in FIG. 20, the MF metadata consists more specifically of the moof box (moof), type, lens, and data, and the mdat box (mdat) type and lens.

When storing the access unit in MP4 data, H. 264 and H. There is a mode in which the parameter set such as 265 SPS and PPS can be stored as sample data, and a mode in which the parameter set cannot be stored.

Here, in the above-mentioned non-storable mode, the parameter set is stored in the Decoder Special Information of SampleEntry in moov. Further, in the above-mentioned storable mode, the parameter set is included in the sample.

The MPU metadata, MF metadata, and media data are each stored in the MMT payload, and the fragment type (FT) is stored in the header of the MMT payload as an identifier that can identify these data. FT = 0 indicates that it is MPU metadata, FT = 1 indicates that it is MF metadata, and FT = 2 indicates that it is media data.

In FIG. 19, an example in which the MPU metadata unit and the MF metadata unit are stored in the MMT payload as a data unit is shown, but units such as ftyp, mmpu, moov, and moof are data as a data unit. It may be stored in the MMT payload on a unit-by-unit basis. Similarly, FIG. 19 illustrates an example in which a sample unit is stored in the MMT payload as a data unit. However, data units may be configured in sample units or NAL unit units, and such data units may be stored in the MMT payload in data unit units. Such data units may be further stored in the MMT payload in fragmented units.

[Conventional transmission methods and issues]
Conventionally, when encapsulating a plurality of access units in the MP4 format, moov and moof are created when all the samples stored in MP4 are prepared.

When transmitting the MP4 format in real time by broadcasting or the like, for example, if the sample stored in one MP4 file is in GOP unit, moov and moof are created after the time sample in GOP unit is accumulated. There is a delay associated with encapsulation. Due to such encapsulation on the transmitting side, the End-to-End delay is always increased by the GOP unit time. This makes it difficult to provide services in real time, and in particular, when live content is transmitted, it leads to deterioration of services for viewers.

FIG. 21 is a diagram for explaining a data transmission order. When MMT is applied to broadcasting, as shown in FIG. 21 (a), it is transmitted by being placed on MMT packets in the order of MPU configuration (MMT packets # 1, # 2, # 3, # 4, # 5, # 6). (Transmission in the order of) causes a delay due to encapsulation in the transmission of the MMT packet.

In order to prevent the delay due to this encapsulation, as shown in FIG. 21 (b), MPU header information such as MPU metadata and MF metadata is not transmitted (packets # 1 and # 2 are not transmitted, and packet # is not transmitted. A method (sending 3- # 6 in this order) has been proposed. Further, as shown in FIG. 20 (c), the media data is transmitted first without waiting for the creation of the MPU header information, and the MPU header information is transmitted after the media data is transmitted (# 3- # 6, #). (1), # 2 is transmitted in this order) method is conceivable.

The receiving device decodes the MPU header information without using the MPU header information when the MPU header information is not transmitted, and the receiving device decodes the MPU header information when the MPU header information is postponed to the media data. Wait for the information to be obtained before decrypting.

However, the conventional MP4 compliant receiving device is not guaranteed to decode without using the MPU header information. Further, when the receiving device performs decoding by special processing without using the MPU header, if the conventional transmission method is used, the decoding process becomes complicated and there is a high possibility that real-time decoding becomes difficult. Further, when the receiving device waits for the acquisition of the MPU header information and then performs decoding, it is necessary to buffer the media data until the receiving device acquires the header information. No decryption was guaranteed.

Therefore, as shown in FIG. 20D, the transmission device according to the second embodiment transmits the MPU metadata before the media data by storing only the information common to the MPU metadata. Then, the transmission device according to the second embodiment transmits the MF metadata in which the generation is delayed after the media data. This provides a transmission method or a reception method that can guarantee the decoding of media data.

Hereinafter, the receiving method when each of the transmission methods (a)-(d) of FIG. 21 is used will be described.

In each transmission method shown in FIG. 21, first, MPU metadata, MFU metadata, and media data are configured in this order.

After configuring the MPU data, when the transmitting device transmits data in the order of MPU metadata, MF metadata, and media data as shown in FIG. 21 (a), the receiving device uses the following (A-1). ) And (A-2) can be used for decoding.

(A-1) The receiving device acquires the MPU header information (MPU metadata and MF metadata), and then decodes the media data using the MPU header information.

(A-2) The receiving device decodes the media data without using the MPU header information.

All of these methods cause a delay due to encapsulation on the transmitting side, but have the advantage that the receiving device does not need to buffer the media data in order to acquire the MPU header. When buffering is not performed, it is not necessary to install a memory for buffering, and no buffering delay occurs. Further, since the method (A-1) performs decoding using the MPU header information, it can also be applied to a conventional receiving device.

When the transmitting device transmits only media data as shown in FIG. 21 (b), the receiving device can perform decoding by the method of (B-1) below.

(B-1) The receiving device decodes the media data without using the MPU header information.

Further, although not shown, when the MPU metadata is transmitted before the transmission of the media data in FIG. 21 (b), the decoding can be performed by the method (B-2) below.

(B-2) The receiving device decodes the media data using the MPU metadata.

Both of the above methods (B-1) and (B-2) have advantages in that there is no delay due to encapsulation on the transmitting side and there is no need to buffer media data for acquiring the MPU header. Is. However, since neither of the methods (B-1) and (B-2) perform decoding using the MPU header information, special processing may be required for decoding.

When the transmitting device transmits data in the order of media data, MPU metadata, and MF metadata as shown in (c) of FIG. 21, the receiving device has the following (C-1) and (C-2). Decryption can be performed by either method.

(C-1) The receiving device decodes the media data after acquiring the MPU header information (MPU metadata and MF metadata).

(C-2) The receiving device decodes the media data without using the MPU header information.

When the above method (C-1) is used, it is necessary to buffer the media data in order to acquire the MPU header information. On the other hand, when the above method (C-2) is used, it is not necessary to perform buffering for acquiring MPU header information.

Further, in any of the above methods (C-1) and (C-2), no delay due to encapsulation occurs on the transmitting side. Further, since the method (C-2) does not use MPU header information, special processing may be required.

When the transmitting device transmits data in the order of MPU metadata, media data, and MF metadata as shown in (d) of FIG. 21, the receiving device has the following (D-1) and (D-2). ) Can be decrypted.

(D-1) The receiving device acquires the MPU metadata, further acquires the MF metadata, and then decodes the media data.

(D-2) After acquiring the MPU metadata, the receiving device decodes the media data without using the MF metadata.

When the above method (D-1) is used, it is necessary to buffer the media data for MF metadata acquisition, but in the case of the above (D-2) method, it is necessary to acquire MF metadata. There is no need to buffer.

Since the above method (D-2) does not perform decoding using MF metadata, special processing may be required.

As described above, if the MPU metadata and the MF metadata can be used for decoding, there is an advantage that the conventional MP4 receiving device can also be used for decoding.

In FIG. 21, the MPU data is composed of MPU metadata, MFU metadata, and media data in this order, and in moof, position information (offset) for each sample or subsample is determined based on this configuration. ing. The MF metadata also includes data (box size and type) other than the media data in the mdat box.

Therefore, when the receiving device identifies the media data based on the MF metadata, the receiving device reconstructs the data in the order in which the MPU data was configured, regardless of the order in which the data was transmitted. , MPU metadata moov or MF metadata moof is used for decoding.

In FIG. 21, the MPU data is configured in the order of MPU metadata, MFU metadata, and media data, but even if the MPU data is configured in an order different from that of FIG. 21 and the position information (offset) is determined. good.

For example, the MPU data may be configured in the order of MPU metadata, media data, and MF metadata, and negative position information (offset) may be shown in the MF metadata. In this case as well, regardless of the order in which the data is transmitted, the receiving device reconstructs the data in the order in which the MPU data is configured on the transmitting side, and then performs decoding using moov or moof.

The transmitting device may signal information indicating the order in which the MPU data is configured, and the receiving device may reconstruct the data based on the signaled information.

As described above, the receiving device uses the packetized MPU metadata, the packetized media data (sample data), and the packetized MF metadata as shown in FIG. 21 (d). Receive in order. Here, the MPU metadata is an example of the first metadata, and the MF metadata is an example of the second metadata.

Next, the receiving device reconstructs the MPU data (MP4 format file) including the received MPU metadata, the received MF metadata, and the received sample data. Then, the sample data included in the reconstructed MPU data is decoded using the MPU metadata and the MF metadata. The MF metadata is metadata including data that can be generated only after the sample data is generated on the transmitting side (for example, the lens stored in the mbox).

More specifically, the operation of the receiving device is performed by each component constituting the receiving device. For example, the receiving device includes a receiving unit that receives the data, a reconstructing unit that reconstructs the MPU data, and a decoding unit that decodes the MPU data. Each of the receiving unit, the generating unit, and the decoding unit is realized by a microcomputer, a processor, a dedicated circuit, or the like.

[How to perform decryption without using header information]
Next, a method of performing decoding without using header information will be described. Here, a method of decoding without using the header information in the receiving device will be described regardless of whether the header information is sent or not on the transmitting side. That is, this method can be applied to any of the transmission methods described with reference to FIG. 21. However, some decoding methods are decryption methods that can be applied only in the case of a specific transmission method.

FIG. 22 is a diagram showing an example of a method of performing decoding without using header information. In FIG. 22, only the MMT payload and the MMT packet containing only the media data are shown, and the MMT payload and the MMT packet containing the MPU metadata and the MF metadata are not shown. Further, in the following description of FIG. 22, it is assumed that the media data belonging to the same MPU is continuously transmitted. Further, a case where a sample is stored in the payload as media data will be described as an example, but in the following description of FIG. 22, a NAL unit may be stored as a matter of course, or a fragmented NAL unit is stored. May be.

In order to decode the media data, the receiving device must first acquire the initialization information required for decoding. If the media is video, the receiving device must acquire initialization information for each sample, specify the start position of the MPU, which is a random access unit, and acquire the start position of the sample and the NAL unit. .. Further, the receiving device needs to specify the decoding time (DTS) and the presentation time (PTS) of the sample, respectively.

Therefore, the receiving device can perform decoding without using the header information by using, for example, the following method. When the NAL unit unit or the unit obtained by fragmenting the NAL unit is stored in the payload, "sample" may be read as "NAL unit in the sample" in the following description.

<Random access (= specify the first sample of MPU)>
There are the following methods 1 and 2 for the receiving device to specify the head sample of the MPU when the header information is not transmitted. When the header information is transmitted, the method 3 can be used.

[Method 1] The receiving device acquires a sample included in the MMT packet having ‘RAP_flag = 1’ in the MMT packet header.

[Method 2] The receiving device acquires a sample in which'sample number = 0'in the MMT payload header.

[Method 3] When the receiving device transmits at least one of the MPU metadata and the MF metadata to at least one of the front and the back of the media data, the receiving device is a fragment type in the MMT payload header. (FT) acquires the sample contained in the MMT payload switched to the media data.

In Method 1 and Method 2, when a plurality of samples belonging to different MPUs are mixed in one payload, it is impossible to determine which NAL unit is a random access point (RAP_flag = 1 or sample number = 0). Therefore, there is a restriction that different MPU samples are not mixed in one payload, or when different MPU samples are mixed in one payload, if the last (or first) sample is a random access point, RAP_flag. It is necessary to have restrictions such as setting 1 to 1.

Further, in order for the receiving device to acquire the start position of the NAL unit, it is necessary to shift the data read pointer by the size of the NAL unit in order from the head NAL unit of the sample.

When the data is fragmented, the receiving device can identify the data unit by referring to the fragment_indicator or the fragment_number.

<DTS of sample determined>
There are the following method 1 and method 2 as a method for determining the DTS of the sample.

[Method 1] The receiving device determines the DTS of the first sample based on the prediction structure. However, since this method requires analysis of coded data and may be difficult to decode in real time, the following method 2 is desirable.

[Method 2] The receiving device separately transmits the DTS of the head sample, and acquires the transmitted DTS of the head sample. As a method of transmitting the DTS of the first sample, for example, there are a method of transmitting the DTS of the MPU head sample using MMT-SI, a method of transmitting the DTS of each sample using the MMT packet header extension area, and the like. The DTS may be an absolute value or a relative value with respect to PTS. Further, it may be signaled on the transmitting side whether or not the DTS of the first sample is included.

In both method 1 and method 2, the DTS of the subsequent samples is calculated assuming that the frame rate is fixed.

As a method of storing the DTS of each sample in the packet header, in addition to using the extended area, there is a method of storing the DTS of the sample included in the MMT packet in the 32-bit NTP time stamp field in the MMT packet header. If the DTS cannot be represented by the number of bits (32 bits) of one packet header, the DTS may be represented by using a plurality of packet headers. Further, the DTS may be expressed by combining the NTP time stamp field of the packet header and the extended area. If the DTS information is not included, it is a known value (for example, ALL0).

<Determination of sample PTS>
The receiving device acquires the PTS of the first sample from the MPU time stamp descriptor for each asset included in the MPU. The receiving device calculates the subsequent sample PTS from parameters indicating the display order of the samples such as POC, assuming that the frame rate is fixed. As described above, in order to calculate DTS and PTS without using the header information, transmission at a fixed frame rate is indispensable.

Further, when the MF metadata is transmitted, the receiving device receives the relative time information of DTS and PTS from the first sample shown in the MF metadata and the time stamp of the MPU first sample shown in the MPU time stamp descriptor. The absolute values of DTS and PTS can be calculated from the absolute values.

When calculating the DTS and PTS by analyzing the coded data, the receiving device may calculate using the SEI information included in the access unit.

<Initialization information (parameter set)>
[For video]
For video, the parameter set is stored in the sample data. Further, when the MPU metadata and the MF metadata are not transmitted, it is guaranteed that the parameter set required for decoding can be obtained by referring only to the sample data.

Further, as shown in FIGS. 21 (a) and 21 (d), when the MPU metadata is transmitted before the media data, it may be specified that the parameter set is not stored in the SampleEntry. In this case, the receiving device refers only to the parameter set in the sample without referring to the parameter set of SingleEntry.

Further, when the MPU metadata is transmitted before the media data, the parameter set common to the MPU and the default parameter set are stored in the SampleEntry, and the receiving device stores the parameter set of the SampleEntry and the parameter set in the sample. You may refer to it. By storing the parameter set in the SampleEntry, it becomes possible to perform decoding even with a conventional receiving device that cannot be reproduced unless the parameter set exists in the SampleEntry.

[For audio]
In the case of audio, the LATM header is required for decoding, and in MP4, it is essential that the LATM header is included in the sample entry. However, if the header information is not transmitted, it is difficult for the receiving device to acquire the LATM header, so that the LATM header is separately included in the control information such as SI. The LATM header may be included in the message, table, or descriptor. The LATM header may be included in the sample.

The receiving device acquires the LATM header from SI or the like before starting the decoding, and starts decoding the audio. Alternatively, as shown in FIGS. 21 (a) and 21 (d), if the MPU metadata is transmitted before the media data, the receiving device receives the LATM header before the media data. It is possible. Therefore, when the MPU metadata is transmitted before the media data, it is possible to perform decoding even by using a conventional receiving device.

<Others>
The transmission order and the type of transmission order may be notified as MMT packet header, payload header, or control information such as MPT and other tables, messages, and descriptors. The transmission order type here is, for example, the transmission order of the four types (a) to (d) in FIG. 21, and an identifier for identifying each type can be obtained before the start of decoding. It should be stored in the place.

Further, as the transmission order type, different types may be used for audio and video, or a common type may be used for audio and video. Specifically, for example, audio is transmitted in the order of MPU metadata, MF metadata, and media data as shown in FIG. 21 (a), and video is shown in FIG. 21 (d). As such, MPU metadata, media data, and MF metadata may be transmitted in this order.

By the method as described above, the receiving device can perform decoding without using the header information. Further, when the MPU metadata is transmitted before the media data (FIG. 21 (a) and FIG. 21 (d)), the conventional receiving device can also perform decoding.

In particular, since the MF metadata is transmitted after the media data ((d) in FIG. 21), it is possible to perform decoding even with a conventional receiving device without causing a delay due to encapsulation.

[Configuration and operation of transmitter]
Next, the configuration and operation of the transmitter will be described. FIG. 23 is a block diagram of the transmission device according to the second embodiment, and FIG. 24 is a flowchart of the transmission method according to the second embodiment.

As shown in FIG. 23, the transmission device 15 includes a coding unit 16, a multiplexing unit 17, and a transmission unit 18.

The coding unit 16 displays the video or audio to be coded, for example, H.I. Encoded data is generated by encoding according to 265 (S10).

The multiplexing unit 17 multiplexes (packets) the coded data generated by the coding unit 16 (S11). Specifically, the multiplexing unit 17 packets each of the sample data, the MPU metadata, and the MF metadata constituting the MP4 format file. The sample data is data in which a video signal or an audio signal is encoded, MPU metadata is an example of first metadata, and MF metadata is an example of second metadata. The first metadata and the second metadata are both metadata used for decoding the sample data, but the difference between them is that the second metadata can be generated only after the sample data is generated. Is to include.

Here, the data that can be generated only after the sample data is generated is, for example, data other than the sample data stored in mdat in the MP4 format (data in the header of mdat, that is, type and lens shown in FIG. 20. ). Here, the second metadata may include length, which is at least a part of this data.

The transmission unit 18 transmits a packetized MP4 format file (S12). The transmission unit 18 transmits, for example, an MP4 format file by the method shown in FIG. 21 (d). That is, the packetized MPU metadata, the packetized sample data, and the packetized MF metadata are transmitted in this order.

Each of the coding unit 16, the multiplexing unit 17, and the transmitting unit 18 is realized by a microcomputer, a processor, a dedicated circuit, or the like.

[Receiver configuration]
Next, the configuration and operation of the receiving device will be described. FIG. 25 is a block diagram of the receiving device according to the second embodiment.

As shown in FIG. 25, the receiving device 20 includes a packet filtering unit 21, a transmission order type determination unit 22, a random access unit 23, a control information acquisition unit 24, a data acquisition unit 25, and PTS and DTS calculation. A unit 26, an initialization information acquisition unit 27, a decoding command unit 28, a decoding unit 29, and a presentation unit 30 are provided.

[Operation of receiving device 1]
First, when the medium is a video, the operation for the receiving device 20 to specify the MPU head position and the NAL unit position will be described. FIG. 26 is a flowchart of such an operation of the receiving device 20. Here, it is assumed that the transmission order type of the MPU data is stored in the SI information by the transmission device 15 (multiplexing unit 17).

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering to acquire the transmission order type of MPU data (S21).

Next, the transmission order type determination unit 22 determines (determines) whether or not the MPU header information (at least one of the MPU metadata or the MF metadata) is included in the data after packet filtering (S22). When the MPU header information (is included (Yes in S22), the random access unit 23 identifies the MPU head sample by detecting that the fragment type of the MMT payload header is switched to the media data (S23). ).

On the other hand, when the MPU header information is not included (No in S22), the random access unit 23 specifies the MPU head sample based on the RAP_flag of the MMT packet header or the sample number of the MMT payload header (S24).

Further, the transmission order type determination unit 22 determines whether or not the packet-filtered data includes MF metadata (S25). When it is determined that the MF metadata is included (Yes in S25), the data acquisition unit 25 reads the NAL unit based on the sample, the offset of the subsample, and the size information contained in the MF metadata. As a result, the NAL unit is acquired (S26). On the other hand, when it is determined that the MF metadata is not included (No in S25), the data acquisition unit 25 reads the data of the size of the NAL unit in order from the head NAL unit of the sample to read the NAL unit. Acquire (S27).

Even if it is determined in step S22 that the MPU header information is included, the receiving device 20 may specify the MPU head sample by using the process of step S24 instead of the process of step S23. Further, when it is determined that the MPU header information is included, the process of step S23 and the process of step S24 may be used together.

Further, the receiving device 20 may acquire the NAL unit by using the process of step S27 without using the process of step S26 even when it is determined in step S25 that the MF metadata is included. Further, when it is determined that the MF metadata is included, the process of step S23 and the process of step S24 may be used in combination.

Further, it is assumed that the case where it is determined in step S25 that the MF metadata is included and the MF data is transmitted after the media data. In this case, the receiving device 20 may buffer the media data, wait until the MF metadata is acquired, and then perform the process of step S26, or the receiving device 20 may not wait for the acquisition of the MF metadata. It may be determined whether or not to perform the process of step S27.

For example, the receiving device 20 may determine whether to wait for acquisition of MF metadata based on whether or not it has a buffer having a buffer size capable of buffering media data. Further, the receiving device 20 may determine whether or not to wait for the acquisition of MF metadata based on whether or not the End-to-End delay becomes small. Further, the receiving device 20 may mainly perform the decoding process using the process of step S26, and may use the process of step S27 in the case of the processing mode when packet loss or the like occurs.

If the transmission order type is predetermined, steps S22 and S26 may be omitted. In this case, the receiving device 20 considers the buffer size and the End-to-End delay, and MPU. The method of specifying the first sample and the method of specifying the NAL unit may be determined.

If the transmission order type is known in advance, the transmission order type determination unit 22 is unnecessary in the receiving device 20.

Further, although not described in FIG. 26, the decoding instruction unit 28 acquires data based on the PTS and DTS calculated by the PTS and DTS calculation unit 26, and the initialization information acquired by the initialization information acquisition unit 27. The data acquired in the unit is output to the decoding unit 29. The decoding unit 29 decodes the data, and the presentation unit 30 presents the decoded data.

[Operation of receiving device 2]
Next, the operation of the receiving device 20 to acquire the initialization information based on the transmission order type and to decode the media data based on the initialization information will be described. FIG. 27 is a flowchart of such an operation.

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering and acquires the transmission order type (S301).

Next, the transmission order type determination unit 22 determines whether or not the MPU metadata has been transmitted (S302). When it is determined that the MPU metadata has been transmitted (Yes in S302), the transmission order type determination unit 22 determines whether or not the MPU metadata has been transmitted before the media data as a result of the analysis in step S301. (S303). When the MPU metadata is transmitted before the media data (Yes in S303), the initialization information acquisition unit 27 is based on the common initialization information included in the MPU metadata and the initialization information of the sample data. To decode the media data (S304).

On the other hand, when it is determined that the MPU metadata is transmitted after the media data (No in S303), the data acquisition unit 25 buffers the media data until the MPU metadata is acquired (S305). , The process of step S304 is performed after the MPU metadata is acquired.

If it is determined in step S302 that the MPU metadata has not been transmitted (No in S302), the initialization information acquisition unit 27 decodes the media data based only on the initialization information of the sample data. (S306).

If the transmission side is guaranteed to decode the media data only based on the initialization information of the sample data, the process of step S306 is used without performing the process based on the determination of step S302 and step S303.

Further, the receiving device 20 may determine whether or not to buffer the media data before step S305. In this case, the receiving device 20 proceeds to the process of step S305 when it is determined to buffer the media data, and proceeds to the process of step S306 when it is determined not to buffer the media data. The determination as to whether or not to buffer the media data may be performed based on the buffer size and the occupied amount of the receiving device 20, and for example, the one having the smaller End-to-End delay is selected. The determination may be made in consideration of the −to—End delay.

[Operation of receiver 3]
Here, the details of the transmission method and the reception method in the case where the MF metadata is transmitted after the media data ((c) in FIG. 21 and (d) in FIG. 21) will be described. Hereinafter, the case of FIG. 21D will be described as an example. In the transmission, only the method (d) of FIG. 21 is used, and the transmission order type signaling is not performed.

As described above, when data is transmitted in the order of MPU metadata, media data, and MF metadata, as shown in (d) of FIG.
(D-1) The receiving device 20 decodes the media data after acquiring the MPU metadata and further acquiring the MF metadata.
(D-2) After acquiring the MPU metadata, the receiving device 20 decodes the media data without using the MF metadata.
There are two possible decryption methods.

Here, D-1 requires buffering of media data for acquiring MF metadata, but since decoding can be performed using MPU header information, it can be decoded by a conventional MP4 compliant receiving device. It becomes. Further, D-2 does not require buffering of media data for acquiring MF metadata, but cannot be decoded using MF metadata, so that special processing is required for decoding.

Further, the method (d) of FIG. 21 has an advantage that the MF metadata is transmitted after the media data, so that the delay due to encapsulation does not occur and the End-to-End delay can be reduced.

The receiving device 20 can select the above two decoding methods according to the capability of the receiving device 20 and the quality of service provided by the receiving device 20.

The transmitting device 15 must ensure that the decoding operation in the receiving device 20 can reduce the occurrence of buffer overflow and underflow. As an element for defining a decoder model in the case of decoding using the method of D-1, for example, the following parameters can be used.

-Buffer size for reconstructing MPU (MPU buffer)

For example, buffer size = maximum rate × maximum MPU time × α, and the maximum rate is the profile of the coded data, the upper limit rate of the level + the overhead of the MPU header. Further, the maximum MPU time is the maximum time length of GOP when 1 MPU = 1 GOP (video).

Here, the audio may be a GOP unit common to the video, or may be another unit. α is a margin for preventing overflow, and may be multiplied or added to the maximum rate × maximum MPU time. When multiplied, α ≧ 1, and when added, α ≧ 0.

-Upper limit of decoding delay time from data input to MPU buffer to decoding (TSTD_delay in MPEG-TS STD)

For example, at the time of transmission, the DTS is set so that the acquisition completion time of the MPU data in the receiver <= DTS in consideration of the upper limit of the maximum MPU time and the decoding delay time.

Further, the transmission device 15 may add DTS and PTS according to the decoder model in the case of decoding using the method of D-1. As a result, the transmitting device 15 guarantees the decoding of the receiving device that performs decoding using the method of D-1, and at the same time, transmits auxiliary information necessary for decoding using the method of D-2. You may.

For example, the transmitting device 15 can guarantee the operation of the receiving device for decoding using the method of D-2 by signaling the prebuffering time in the decoder buffer when decoding using the method of D-2.

The prebuffering time may be included in SI control information such as messages, tables, and descriptors, or may be included in the headers of MMT packets and MMT payloads. Further, the SEI in the coded data may be overwritten. The DTS and PTS for decoding using the method of D-1 are stored in the MPU time stamp descriptor, SampleEntry, and the DTS and PTS for decoding using the method of D-2, or the prebuffering time. It may be described in SEI.

When the receiving device 20 supports only the MP4 compliant decoding operation using the MPU header, the receiving device 20 selects the decoding method D-1 and supports both D-1 and D-2. If so, you may select either one.

The transmission device 15 may add DTS and PTS so as to guarantee the decoding operation of one (D-1 in this description), and may further transmit auxiliary information for assisting the decoding operation of one.

Further, when the method of D-2 is used, the End-to-End delay may be larger due to the delay caused by the prebuffering of the MF metadata as compared with the case where the method of D-1 is used. Is high. Therefore, when the receiving device 20 wants to reduce the End-to-End delay, the receiving device 20 may select and decode the method of D-2. For example, the receiving device 20 may always use the method of D-2 when it is desired to always reduce the End-to-End delay. Further, the receiving device 20 may use the method of D-2 only when the receiving device 20 is operating in the low delay presentation mode, such as live content, channel selection, and zapping operation, which is desired to be presented with low delay.

FIG. 28 is a flowchart of such a receiving method.

First, the receiving device 20 receives the MMT packet and acquires the MPU data (S401). Then, the receiving device 20 (transmission order type determining unit 22) determines whether or not to present the program in the low delay presentation mode (S402).

When the program is not presented in the low delay presentation mode (No in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) acquires random access and initialization information using the header information (S405). ). Further, the receiving device 20 (PTS, DTS calculation unit 26, decoding instruction unit 28, decoding unit 29, presentation unit 30) performs decoding and presentation processing based on the PTS and DTS assigned on the transmitting side (S406).

On the other hand, when the program is presented in the low delay presentation mode (Yes in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) uses a decoding method that does not use header information to perform random access. Acquire initialization information (S403). Further, the receiving device 20 performs decoding and presentation processing based on auxiliary information for decoding without using the PTS, DTS and header information given on the transmitting side (S404). In addition, in step S403 and step S404, processing may be performed using MPU metadata.

[Transmission / reception method using auxiliary data]
The transmission / reception operation in the case where the MF metadata is transmitted after the media data (in the case of (c) of FIG. 21 and the case of (d) of FIG. 21) has been described above. Next, a method will be described in which the transmission device 15 can start decoding earlier and reduce the End-to-End delay by transmitting auxiliary data having a part of the function of the MF metadata. Here, an example in which auxiliary data is further transmitted based on the transmission method shown in FIG. 21 (d) will be described, but methods using auxiliary data are shown in FIGS. 21 (a) to 21 (c). It is also applicable to the transmission method.

FIG. 29 (a) is a diagram showing an MMT packet transmitted by the method shown in FIG. 21 (d). That is, the data is transmitted in the order of MPU metadata, media data, and MF metadata.

Here, sample # 1, sample # 2, sample # 3, and sample # 4 are samples included in the media data. Here, an example in which the media data is stored in the MMT packet in sample units will be described, but the media data may be stored in the MMT packet in units of NAL units, or stored in units of divided NAL units. May be done. In some cases, a plurality of NAL units are aggregated and stored in an MMT packet.

As described in D-1 above, in the case of the method shown in (d) of FIG. 21, that is, when the data is transmitted in the order of MPU metadata, media data, and MF metadata, the MPU metadata is used. After the acquisition, there is a method of further acquiring the MF metadata and then decoding the media data. Such a method of D-1 requires buffering of media data for acquiring MF metadata, but since decoding is performed using MPU header information, D-1 is also used in a conventional MP4 compliant receiving device. Method -1 has the advantage of being applicable. On the other hand, the receiving device 20 has a drawback that it has to wait for the start of decoding until the MF metadata is acquired.

On the other hand, as shown in FIG. 29 (b), in the method using the auxiliary data, the auxiliary data is transmitted before the MF metadata.

The MF metadata contains information indicating the DTS, PTS, offset and size of all the samples contained in the movie fragment. On the other hand, the auxiliary data includes information indicating the DTS, PTS, offset, and size of some of the samples included in the movie fragment.

For example, the MF metadata contains information for all samples (sample # 1-sample # 4), while the auxiliary data contains information for some samples (samples # 1- # 2). ..

In the case shown in FIG. 29 (b), since the sample # 1 and the sample # 2 can be decoded by using the auxiliary data, the End-to-End is used with respect to the transmission method of the D-1. The delay is small. It should be noted that the auxiliary data may include any combination of sample information, and the auxiliary data may be repeatedly transmitted.

For example, in FIG. 29 (c), when the auxiliary information is transmitted at the timing of A, the transmission device 15 includes the information of the sample # 1 in the auxiliary information, and when the auxiliary information is transmitted at the timing of B, the auxiliary information is transmitted. Include the information of sample # 1 and sample # 2 in the auxiliary information. When the transmission device 15 transmits the auxiliary information at the timing of C, the auxiliary information includes the information of the sample # 1, the sample # 2, and the sample # 3.

The MF metadata includes information on sample # 1, sample # 2, sample # 3, and sample # 4 (information on all samples in the movie fragment).

Auxiliary data does not necessarily have to be sent immediately after generation.

In the header of the MMT packet or MMT payload, a type indicating that auxiliary data is stored is specified.

For example, when the auxiliary data is stored in the MMT payload using the MPU mode, the fragment_type field value (eg, FT = 3) specifies a data type indicating that it is auxiliary data. The auxiliary data may be data based on the structure of moof, or may have other structures.

When the auxiliary data is stored in the MMT payload as a control signal (descriptor, table, message), a descriptor tag indicating that the auxiliary data is used, a table ID, a message ID, and the like are specified.

Further, PTS or DTS may be stored in the header of the MMT packet or MMT payload.

[Example of generating auxiliary data]
Hereinafter, an example in which the transmission device generates auxiliary data based on the configuration of the moof will be described. FIG. 30 is a diagram for explaining an example in which the transmitting device generates auxiliary data based on the configuration of the moof.

In normal MP4, a movie is created for a movie fragment, as shown in FIG. The moof contains information indicating the DTS, PTS, offset, and size of the sample contained in the movie fragment.

Here, the transmission device 15 constructs an MP4 (MP4 file) using only a part of the sample data among the sample data constituting the MPU, and generates auxiliary data.

For example, as shown in FIG. 30A, the transmitting device 15 generates MP4 using only sample # 1 among the samples # 1 to # 4 constituting the MPU, and the header of moof + mdat is used. Use as auxiliary data.

Next, as shown in FIG. 30 (b), the transmission device 15 generates MP4 by using sample # 1 and sample # 2 among the samples # 1 to # 4 constituting the MPU. The header of moof + mdat is used as the next auxiliary data.

Next, as shown in FIG. 30 (c), the transmission device 15 uses the sample # 1, the sample # 2, and the sample # 3 among the samples # 1 to # 4 constituting the MPU to transmit the MP4. It is generated, and the header of moof + mdat is used as the next auxiliary data.

Next, as shown in FIG. 30 (d), the transmission device 15 generates all MP4s among the samples # 1 to # 4 constituting the MPU, of which the header of moof + mdat is the movie fragment metadata. It becomes.

Here, the transmission device 15 generates auxiliary data for each sample, but auxiliary data may be generated for each N sample. The value of N is an arbitrary number, and for example, when the auxiliary data is transmitted M times when one MPU is transmitted, N = all samples / M may be set.

The information indicating the offset of the sample in moof may be the offset value after the sample entry area of the subsequent sample number is secured as the NULL area.

Auxiliary data may be generated so as to be configured to fragment the MF metadata.

[Example of reception operation using auxiliary data]
The reception of the auxiliary data generated as described with reference to FIG. 30 will be described. FIG. 31 is a diagram for explaining the reception of auxiliary data. In FIG. 31A, the number of samples constituting the MPU is 30, and auxiliary data is generated and transmitted every 10 samples.

In FIG. 30A, the auxiliary data # 1 is a sample # 1- # 10, the auxiliary data # 2 is a sample # 1- # 20, and the MF metadata is a sample # 1- # 30. Information is included respectively.

Although samples # 1- # 10, samples # 11- # 20, and samples # 21- # 30 are stored in one MMT payload, they may be stored in sample units or NAL units, or fragments. Or aggregated units may be stored.

The receiving device 20 receives packets of MPU meta, sample, MF meta, and auxiliary data, respectively.

The receiving device 20 concatenates the sample data in the order of reception (behind), and after receiving the latest auxiliary data, updates the auxiliary data so far. Further, the receiving device 20 can configure a complete MPU by finally replacing the auxiliary data with the MF metadata.

When the receiving device 20 receives the auxiliary data # 1, the receiving device 20 concatenates the data as shown in the upper part of FIG. 31 (b) to form the MP4. As a result, the receiving device 20 can parse the samples # 1 to # 10 using the MPU metadata and the information of the auxiliary data # 1, and the information of the PTS, DTS, offset, and size included in the auxiliary data. Decryption can be performed based on.

Further, when the receiving device 20 receives the auxiliary data # 2, the receiving device 20 concatenates the data as shown in the middle stage of FIG. 31 (b) to form the MP4. As a result, the receiving device 20 can parse the samples # 1 to # 20 using the information of the MPU metadata and the auxiliary data # 2, and the information of the PTS, DTS, offset, and size included in the auxiliary data can be obtained. Decryption can be performed based on this.

Further, when the receiving device 20 receives the MF metadata, the receiving device 20 concatenates the data as shown in the lower part of FIG. 31 (b) to form the MP4. This allows the receiver 20 to parse samples # 1- # 30 using MPU metadata and MF metadata, based on the PTS, DTS, offset, and size information contained in the MF metadata. Can be decrypted.

In the absence of auxiliary data, the receiving device 20 can acquire the sample information only after receiving the MF metadata, so that it is necessary to start decoding after receiving the MF metadata. However, when the transmitting device 15 generates and transmits the auxiliary data, the receiving device 20 can acquire the sample information using the auxiliary data without waiting for the reception of the MF metadata, so that the decoding start time is shortened. be able to. Further, the transmitting device 15 can generate auxiliary data based on the moof described with reference to FIG. 30, so that the receiving device 20 can parse the conventional MP4 parser as it is.

In addition, the newly generated auxiliary data and MF metadata include sample information that overlaps with the auxiliary data transmitted in the past. Therefore, even if the past auxiliary data could not be acquired due to packet loss, etc., MP4 can be reconstructed by using the newly acquired auxiliary data and MF metadata, and the sample information (PTS, DTS, size, etc.) can be obtained. And offset) can be obtained.

The auxiliary data does not necessarily have to include the information of the past sample data. For example, the auxiliary data # 1 may correspond to the sample data # 1 to # 10, and the auxiliary data # 2 may correspond to the sample data # 11- # 20. For example, as shown in FIG. 31 (c), the transmitting device 15 may sequentially transmit the complete MF metadata as the data unit and the fragmented unit of the data unit as the auxiliary data.

Further, the transmission device 15 may repeatedly transmit auxiliary data or may repeatedly transmit MF metadata as a countermeasure against packet loss.

The MMT packet and MMT payload in which the auxiliary data is stored include the MPU sequence number and the asset ID as well as the MPU metadata, the MF metadata, and the sample data.

The reception operation using the auxiliary data as described above will be described with reference to the flowchart of FIG. FIG. 32 is a flowchart of a reception operation using auxiliary data.

First, the receiving device 20 receives the MMT packet and analyzes the packet header and the payload header (S501). Next, the receiving device 20 analyzes whether the fragment type is auxiliary data or MF metadata (S502), and if the fragment type is auxiliary data, overwrites and updates the past auxiliary data (S503). .. At this time, if there is no past auxiliary data of the same MPU, the receiving device 20 uses the received auxiliary data as it is as new auxiliary data. Then, the receiving device 20 acquires a sample and decodes it based on the MPU metadata, the auxiliary data, and the sample data (S507).

On the other hand, when the fragment type is MF metadata, the receiving device 20 overwrites the past auxiliary data with the MF metadata in step S505 (S505). Then, the receiving device 20 acquires a sample in the form of a complete MPU based on the MPU metadata, the MF metadata, and the sample data, and performs decoding (S506).

Although not shown in FIG. 32, in step S502, the receiving device 20 stores the data in a buffer when the fragment type is MPU metadata, and when the fragment type is sample data, the data is added to the back of each sample. Store the concatenated data in the buffer.

If the auxiliary data cannot be acquired due to packet loss, the receiving device 20 can overwrite the sample with the latest auxiliary data or decode the sample by using the past auxiliary data.

The transmission cycle and the number of transmissions of the auxiliary data may be predetermined values. Information on the transmission cycle and the number of times (count, countdown) may be transmitted together with the data. For example, the data unit header may store a time stamp such as a transmission cycle, a transmission count, and initial_cpb_removal_delay.

The CPB buffer model can be followed by transmitting auxiliary data containing information on the first sample of the MPU at least once prior to the initial_cpb_removal_delay. At this time, the value based on the picture timing SEI is stored in the MPU time stamp descriptor.

The transmission method in the reception operation in which such auxiliary data is used is not limited to the MMT method, and can be applied to the case of streaming transmission of a packet composed of an ISOBMFF file format such as MPEG-DASH.

[Sending method when one MPU consists of multiple movie fragments]
In the description after FIG. 19 above, one MPU is composed of one movie fragment, but here, a case where one MPU is composed of a plurality of movie fragments will be described. FIG. 33 is a diagram showing the configuration of an MPU composed of a plurality of movie fragments.

In FIG. 33, the samples (# 1- # 6) stored in one MPU are stored separately in two movie fragments. The first movie fragment is generated based on samples # 1- # 3, and a corresponding moof box is generated. The second movie fragment is generated based on samples # 4- # 6 and a corresponding moof box is generated.

The headers of the moof box and mdat box in the first movie fragment are stored in the MMT payload and the MMT packet as movie fragment metadata # 1. On the other hand, the headers of the moof box and the mdat box in the second movie fragment are stored in the MMT payload and the MMT packet as the movie fragment metadata # 2. In FIG. 33, the MMT payload in which the movie fragment metadata is stored is hatched.

The number of samples constituting the MPU and the number of samples constituting the movie fragment are arbitrary. For example, two movie fragments may be configured by using the number of samples constituting the MPU as the number of samples in the GOP unit and the number of samples in half of the GOP unit as the movie fragment.

Here, an example in which one MPU contains two movie fragments (moof box and mdat box) is shown, but one MPU may contain three or more movie fragments instead of two. Further, the sample stored in the movie fragment may be divided into an arbitrary number of samples instead of the number of equally divided samples.

In FIG. 33, the MPU metadata unit and the MF metadata unit are stored in the MMT payload as data units, respectively. However, the transmitting device 15 may store units such as ftyp, mmpu, moov, and moof as data units in the MMT payload in units of data units, or store the data units in the MMT payload in units fragmented. May be good. Further, the transmission device 15 may store the data unit in the MMT payload in the aggregated unit.

Further, in FIG. 33, the sample is stored in the MMT payload in sample units. However, the transmitting device 15 may configure a data unit in a NAL unit unit or a unit in which a plurality of NAL units are combined, not in a sample unit, and store the data unit in the MMT payload in the data unit unit. Further, the transmitting device 15 may store the data unit in the MMT payload in the fragmented unit, or may store the data unit in the MMT payload in the aggregated unit.

In FIG. 33, MPUs are configured in the order of moof # 1, mdat # 1, moof # 2, and mdat # 2, and offset # 1 is given an offset assuming that the corresponding mdat # 1 is attached to the back. There is. However, the offset may be given by assuming that mdat # 1 is attached before moof # 1. However, in this case, the movie fragment metadata cannot be generated in the form of moof + mdat, and the headers of moof and mdat are transmitted separately.

Next, the transmission order of the MMT packets when the MPU having the configuration described with reference to FIG. 33 is transmitted will be described. FIG. 34 is a diagram for explaining the transmission order of MMT packets.

(A) of FIG. 34 shows a transmission order when MMT packets are transmitted in the configuration order of the MPU shown in FIG. 33. Specifically, FIG. 34A shows MPU meta, MF meta # 1, media data # 1 (sample # 1- # 3), MF meta # 2, and media data # 2 (sample # 4- # 6). ) Is shown in this order.

(B) of FIG. 34 transmits MPU meta, media data # 1 (sample # 1- # 3), MF meta # 1, media data # 2 (sample # 4- # 6), and MF meta # 2 in this order. An example is shown.

In FIG. 34 (c), media data # 1 (sample # 1- # 3), MPU meta, MF meta # 1, media data # 2 (sample # 4- # 6), and MF meta # 2 are transmitted in this order. An example is shown.

MF Meta # 1 is generated using Samples # 1- # 3, and MF Meta # 2 is generated using Samples # 4- # 6. Therefore, when the transmission method of FIG. 34 (a) is used, the transmission of the sample data is delayed due to encapsulation.

On the other hand, when the transmission method of FIG. 34 (b) and FIG. 34 (c) is used, the sample can be transmitted without waiting for the generation of the MF meta, so that the delay due to encapsulation is increased. It does not occur and the End-to-End delay can be reduced.

Further, also in the transmission order of FIG. 34 (a), one MPU is divided into a plurality of movie fragments, and the number of samples stored in the MF meta is smaller than that in the case of FIG. The amount of delay due to encapsulation can be made smaller than in the case.

In addition to the method shown here, for example, the transmission device 15 may connect the MF meta # 1 and the MF meta # 2 and transmit them together at the end of the MPU. In this case, the MF metas of different movie fragments may be aggregated and stored in one MMT payload. Further, the MF metas of different MPUs may be aggregated together and stored in the MMT payload.

[Reception method when one MPU consists of multiple movie fragments]
Here, an operation example of the receiving device 20 that receives and decodes the MMT packets transmitted in the transmission order described in FIG. 34 (b) will be described. 35 and 36 are diagrams for explaining such an operation example.

The receiving device 20 receives the MMT packet including the MPU meta, the sample, and the MF meta transmitted in the transmission order as shown in FIG. 35, respectively. The sample data are concatenated in the order of reception.

The receiving device 20 concatenates the data as shown in (1) of FIG. 36 with T1 which is the time when the MF meta # 1 is received, and constitutes the MP4. As a result, the receiving device 20 can acquire the samples # 1 to # 3 based on the MPU metadata and the information of the MF meta # 1, and the information of the PTS, DTS, offset, and size included in the MF meta. Decoding can be performed based on.

Further, the receiving device 20 concatenates the data to T2, which is the time when the MF meta # 2 is received, as shown in (2) of FIG. 36, and constitutes the MP4. Thereby, the receiving device 20 can acquire the sample # 4- # 6 based on the MPU metadata and the information of the MF meta # 2, and is based on the information of the PTS, DTS, offset, and size of the MF meta. Can be decrypted. Further, the receiving device 20 concatenates the data as shown in (3) of FIG. 36 and configures MP4 to obtain samples # 1 to # 6 based on the information of MF meta # 1 and MF meta # 2. You may get it.

By dividing one MPU into a plurality of movie fragments, the time until the first MF meta is acquired in the MPU is shortened, so that the decoding start time can be shortened. In addition, the buffer size for accumulating the sample before decoding can be reduced.

The transmission device 15 transmits (or receives) the first sample of the movie fragment to the transmission (or reception) of the MF meta corresponding to the movie fragment, which is shorter than the initial_cpb_removal_delay specified by the encoder. The division unit of the movie fragment may be set so as to be. By setting in this way, the receive buffer can follow the cpb buffer, and low-delay decoding can be realized. In this case, absolute time based on initial_cpb_removal_delay can be used for PTS and DTS.

Further, the transmitting device 15 may divide the movie fragment at equal intervals, or may divide the subsequent movie fragment at intervals shorter than the previous movie fragment. As a result, the receiving device 20 can always receive the MF meta including the information of the sample before decoding the sample, and continuous decoding becomes possible.

The following two methods can be used to calculate the absolute time of PTS and DTS.

(1) The absolute time of PTS and DTS is determined based on the reception time (T1 or T2) of MF meta # 1 or MF meta # 2, and the relative time of PTS and DTS included in MF meta.

(2) The absolute time of PTS and DTS is determined based on the absolute time signaled from the transmitting side such as the MPU time stamp descriptor and the relative time of PTS and DTS included in the MF meta.

Further, the absolute time signaled by the (2-A) transmission device 15 may be an absolute time calculated based on internal_cpb_removal_delay specified by the encoder.

Further, the absolute time signaled by the (2-B) transmission device 15 may be an absolute time calculated based on the predicted value of the reception time of the MF meta.

The MF meta # 1 and the MF meta # 2 may be repeatedly transmitted. By repeatedly transmitting the MF meta # 1 and the MF meta # 2, the receiving device 20 can acquire the MF meta again even if the MF meta cannot be acquired due to packet loss or the like.

An identifier indicating the order of movie fragments can be stored in the payload header of the MFU containing the samples constituting the movie fragment. On the other hand, the identifier indicating the order of the MF metas constituting the movie fragment is not included in the MMT payload. Therefore, the receiving device 20 identifies the order of the MF meta by packet_sequence_number. Alternatively, the transmitting device 15 stores an identifier indicating which movie fragment the MF meta belongs to in the control information (message, table, descriptor), MMT header, MMT payload header, or data unit header for signaling. You may.

The transmitting device 15 transmits the MPU meta, the MF meta, and the sample in a predetermined transmission order, and the receiving device 20 performs reception processing based on a predetermined predetermined transmission order. May be. Further, the transmitting device 15 may signal the transmission order, and the receiving device 20 may select (determine) the reception process based on the signaling information.

The reception method as described above will be described with reference to FIG. 37. FIG. 37 is a flowchart of the operation of the receiving method described with reference to FIGS. 35 and 36.

First, the receiving device 20 determines (identifies) whether the data contained in the payload is MPU metadata, MF metadata, or sample data (MFU) according to the fragment type shown in the MMT payload (S601). , S602). When the data is sample data, the receiving device 20 buffers the sample and waits for the reception of the MF metadata corresponding to the sample and the start of decoding (S603).

On the other hand, in step S602, when the data is MF metadata, the receiving device 20 acquires sample information (PTS, DTS, position information, and size) from the MF metadata, and uses the acquired sample information. The sample is obtained based on the above, and the sample is decoded and presented based on the PTS and DTS (S604).

Although not shown, when the data is MPU metadata, the MPU metadata includes initialization information necessary for decoding. Therefore, the receiving device 20 accumulates this and uses it for decoding the sample data in step S604.

When the receiving device 20 stores the received MPU data (MPU metadata, MF metadata, and sample data) in the storage device, the receiving device 20 rearranges the received MPU data into the MPU configuration described with reference to FIG. 19 or 33. After that, it accumulates.

On the transmitting side, a packet sequence number is assigned to the MMT packet for the packet having the same packet ID. At this time, the packet sequence number may be assigned after the MMT packet including the MPU metadata, the MF metadata, and the sample data is sorted in the transmission order, or the packet sequence number is assigned in the order before the sorting. May be good.

When the packet sequence numbers are assigned in the order before sorting, the receiving device 20 can sort the data in the MPU configuration order based on the packet sequence numbers, which facilitates storage.

[How to detect the beginning of the access unit and the beginning of the slice segment]
A method of detecting the beginning of an access unit or the beginning of a slice segment based on the information of the MMT packet header and the MMT payload header will be described.

Here, when the non-VCL NAL unit (access unit delimiter, VPS, SPS, PPS, SEI, etc.) is collectively stored in the MMT payload as a data unit, and the non-VCL NAL unit is used as a data unit and a data unit. Will be aggregated and stored in one MMT payload.

FIG. 38 is a diagram showing a case where non-VCL NAL units are individually used as data units and aggregated.

In the case of FIG. 38, the head of the access unit is the head data of the MMT payload including the data unit having the fragment_type value of MFU, the aggregation_flag value of 1, and the offset value of 0. At this time, the Fragmentation_indicator value is 0.

Further, in the case of FIG. 38, the head of the slice segment is the head data of the MMT payload having a fragment_type value of MFU, an aggregation_flag value of 0, and a fragmentation_indicator value of 00 or 01.

FIG. 39 is a diagram showing a case where non-VCL NAL units are collectively used as a data unit. The field value of the packet header is as shown in FIG. 17 (or FIG. 18).

In the case of FIG. 39, at the head of the access unit, the head data of the payload in the packet whose Offset value is 0 is the head of the access unit.

Further, in the case of FIG. 39, the head of the slice segment is a value different from 0 in the Offset value, and the head data of the payload of the packet having the fragmentation indicator value of 00 or 01 is the head of the slice segment.

[Reception processing when packet loss occurs]
Normally, when transmitting MP4 format data in an environment where packet loss occurs, the receiving device 20 restores the packet by ALFEC (Application Layer FEC), packet retransmission control, or the like.

However, if packet loss occurs when AL-FEC is not used in streaming such as broadcasting, the packet cannot be restored.

The receiving device 20 needs to restart decoding of video and audio after data is lost due to packet loss. For that purpose, the receiving device 20 needs to detect the head of the access unit or the NAL unit and start decoding from the head of the access unit or the NAL unit.

However, since the start code is not attached to the head of the MP4 format NAL unit, the receiving device 20 cannot detect the head of the access unit or the NAL unit even if the stream is analyzed.

FIG. 40 is a flowchart of the operation of the receiving device 20 when packet loss occurs.

The receiving device 20 detects packet loss by a Packet access number, a packet counter, a fragment counter, or the like in the header of the MMT packet or the MMT payload (S701), and determines which packet has disappeared from the context (S702). ..

When it is determined that no packet loss has occurred (No in S702), the receiving device 20 configures an MP4 file and decodes the access unit or the NAL unit (S703).

When it is determined that packet loss has occurred (Yes in S702), the receiving device 20 generates a NAL unit corresponding to the NAL unit having packet loss from dummy data, and constitutes an MP4 file (S704). When the receiving device 20 puts dummy data in the NAL unit, it indicates that the type of the NAL unit is dummy data.

Further, the receiving device 20 detects the head of the next access unit or NAL unit based on the methods described in FIGS. 17, 18, 38, and 39, and inputs the head data to the decoder. Decoding can be resumed (S705).

When a packet loss occurs, the receiving device 20 may restart decoding from the beginning of the access unit and the NAL unit based on the information detected based on the packet header, or the NAL unit of dummy data. Decoding may be restarted from the beginning of the access unit and the NAL unit based on the header information of the reconstructed MP4 file including.

When the receiving device 20 stores an MP4 file (MPU), the packet data (NAL unit or the like) lost due to packet loss may be separately acquired from broadcasting or communication and stored (replaced).

At this time, when the receiving device 20 acquires the lost packet from the communication, the receiving device 20 obtains the information of the lost packet (packet ID, MPU sequence number, packet sequence number, IP data flow number, IP address, etc.) from the server. To notify and acquire the packet. The receiving device 20 may acquire not only the lost packet but also the packet group before and after the lost packet at the same time.

[How to configure a movie fragment]
Here, a method of configuring a movie fragment will be described in detail.

As described with reference to FIG. 33, the number of samples constituting the movie fragment and the number of movie fragments constituting one MPU are arbitrary. For example, the number of samples constituting the movie fragment and the number of movie fragments constituting one MPU may be a fixed fixed number or may be dynamically determined.

Here, by configuring the movie fragment so as to satisfy the following conditions on the transmitting side (transmitting device 15), it is possible to guarantee low-delay decoding in the receiving device 20.

The conditions are as follows.

The transmitting device 15 receives sample data so that the receiving device 20 can always receive the MF meta including the information of the sample before the decoding time (DTS (i)) of any sample (Sample (i)). MF meta is generated and transmitted using the divided unit as a movie fragment.

Specifically, the transmission device 15 constructs a movie fragment using a sample encoded before DTS (i) (including the i-th sample).

As a method for dynamically determining the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU so as to guarantee low-delay decoding, for example, the following method is used.

(1) At the start of decoding, the decoding time DTS (0) of the sample Sample (0) at the head of the GOP is a time based on internal_cpb_removal_delay. The transmitting device constitutes a first movie fragment using the coded sample at a time before DTS (0). Further, the transmission device 15 generates MF metadata corresponding to the first movie fragment and transmits it at a time before DTS (0).

(2) The transmission device 15 also configures the movie fragment so as to satisfy the above conditions in the subsequent samples.

For example, assuming that the first sample of the movie fragment is the kth sample, the MF meta of the movie fragment containing the kth sample is transmitted by the decoding time DTS (k) of the kth sample. When the coding completion time of the l-th sample is before DTS (k) and the coding completion time of the (l + 1) th sample is after DTS (k), the transmission device 15 is k-th. A movie fragment is constructed using the lth sample from the sample of.

The transmission device 15 may form a movie fragment using the k-th sample to the sample less than the l-th sample.

(3) After the coding of the last sample of the MPU is completed, the transmission device 15 constructs a movie fragment using the remaining samples, generates MF metadata corresponding to the movie fragment, and transmits the movie fragment.

Note that the transmission device 15 may configure a movie fragment using some of the coded samples instead of using all the coded samples.

In the above, an example is shown in which the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU are dynamically determined based on the above conditions so as to guarantee low-delay decoding. rice field. However, the method for determining the number of samples and the number of movie fragments is not limited to such a method. For example, the number of movie fragments constituting one MPU may be fixed to a predetermined value, and the number of samples may be determined so as to satisfy the above conditions. Further, the number of movie fragments constituting one MPU and the time for dividing the movie fragment (or the code amount of the movie fragment) are fixed to predetermined values, and the number of samples may be determined so as to satisfy the above conditions.

Also, if the MPU is split into multiple movie fragments, information indicating whether the MPU is split into multiple movie fragments, the attributes of the split movie fragment, or the attributes of the MF meta for the split movie fragment. May be sent.

Here, the attribute of the movie fragment is information indicating whether the movie fragment is the first movie fragment of the MPU, the last movie fragment of the MPU, or any other movie fragment.

Further, the attribute of the MF meta corresponds to whether the MF meta is the MF meta corresponding to the first movie fragment of the MPU, the MF meta corresponding to the last movie fragment of the MPU, or other movie fragments. This is information indicating whether or not the MF meta is to be used.

The transmission device 15 may store and transmit the number of samples constituting the movie fragment and the number of movie fragments constituting one MPU as control information.

[Operation of receiver]
The operation of the receiving device 20 based on the movie fragment configured as described above will be described.

The receiving device 20 determines the absolute time of each of the PTS and the DTS based on the absolute time signaled from the transmitting side such as the MPU time stamp descriptor and the relative time of the PTS and the DTS included in the MF meta.

The receiving device 20 processes as follows based on the information of whether or not the MPU is divided into a plurality of movie fragments, and if the MPU is divided, based on the attributes of the divided movie fragments. ..

(1) When the movie fragment is the first movie fragment of the MPU, the receiving device 20 determines the absolute time of the PTS of the first sample included in the MPU time stamp descriptor and the relative time of the PTS and DTS included in the MF meta. It is used to generate absolute times for PTS and DTS.

(2) When the movie fragment is not the first movie fragment of the MPU, the receiving device 20 uses the relative time of the PTS and the DTS included in the MF meta without using the information of the MPU time stamp descriptor, and the PTS and the DTS. Generates the absolute time of.

(3) When the movie fragment is the last movie fragment of the MPU, the receiving device 20 resets the PTS and DTS calculation processing (relative time addition processing) after calculating the absolute times of the PTS and DTS of all the samples. do. The reset process may be performed in the movie fragment at the beginning of the MPU.

The receiving device 20 may determine whether or not the movie fragment is divided as described below. Further, the receiving device 20 may acquire the attribute information of the movie fragment as described below.

For example, the receiving device 20 may determine whether or not the unit is divided based on the identifier movie_fragment_sequence_number field value indicating the order of the movie fragments shown in the MMTP (MMT Protocol) payload header.

Specifically, the receiving device 20 has a case where the number of movie fragments contained in one MPU is 1, the movie_fragment_sequence_number field value is 1, and the field value is 2 or more. , The MPU may be determined to be divided into a plurality of movie fragments.

Further, the receiving device 20 has the MPU when the number of movie fragments contained in one MPU is 1, the movie_fragment_sequence_number field value is 0, and the field value is a value other than 0. May be determined to be divided into a plurality of movie fragments.

Similarly, the attribute information of the movie fragment may be determined based on movie_fragment_sequence_number.

Even if movie_fragment_sequence_number is not used, it may be determined whether or not the movie fragment is divided and the attribute information of the movie fragment by counting the transmission of the movie fragment and the MF meta included in one MPU.

With the configuration of the transmitting device 15 and the receiving device 20 as described above, the receiving device 20 can receive the movie fragment metadata at shorter intervals than the MPU, and can start decoding with a low delay. Further, it is possible to perform decoding with low delay by using the decoding process based on the MP4 parsing method.

As described above, the reception operation when the MPU is divided into a plurality of movie fragments will be described with reference to a flowchart. FIG. 41 is a flowchart of the receiving operation when the MPU is divided into a plurality of movie fragments. It should be noted that this flowchart illustrates the operation of step S604 in FIG. 37 in more detail.

First, the receiving device 20 acquires MF metadata based on the data type shown in the MMTP payload header when the data type is MF meta (S801).

Next, the receiving device 20 determines whether or not the MPU is divided into a plurality of movie fragments (S802), and if the MPU is divided into a plurality of movie fragments (Yes in S802), the received MF. It is determined whether or not the metadata is the metadata at the beginning of the MPU (S803). When the received MF metadata is the MF metadata at the beginning of the MPU (Yes in S803), the receiving device 20 sets the absolute time of the PTS shown in the MPU time stamp descriptor, and the PTS and the PTS shown in the MF metadata. The absolute time of PTS and DTS is calculated from the relative time of DTS (S804), and it is determined whether or not it is the last metadata of MPU (S805).

On the other hand, when the received MF metadata is not the MF metadata at the beginning of the MPU (No in S803), the receiving device 20 does not use the information of the MPU time stamp descriptor and shows the relative PTS and DTS in the MF metadata. The absolute time of PTS and DTS is calculated using the time (S808), and the process proceeds to step S805.

If it is determined in step S805 that it is the last MF metadata of the MPU (Yes in S805), the receiving device 20 resets the calculation process of PTS and DTS after calculating the absolute time of PTS and DTS of all the samples. do. If it is determined in step S805 that it is not the last MF metadata of the MPU (No in S805), the receiving device 20 ends the process.

If it is determined in step S802 that the MPU is not divided into a plurality of movie fragments (No in S802), the receiving device 20 acquires sample data based on the MF metadata transmitted after the MPU. Then, PTS and DTS are determined (S807).

Then, although not shown, the receiving device 20 finally performs a decoding process and a presentation process based on the determined PTS and DTS.

[Problems that occur when splitting movie fragments and their solutions]
So far, we have described a method of reducing the End-to-End delay by dividing a movie fragment. From here, the problems newly generated when the movie fragment is divided and the solutions thereof will be described.

First, as a background, the picture structure in the coded data will be described. FIG. 42 is a diagram showing an example of a picture prediction structure in each TemporalId when achieving time scalability.

In coding methods such as MPEG-4 AVC and HEVC (High Efficiency Video Coding), time-wise scalability is realized by using B-pictures (bidirectional reference prediction pictures) that can be referred to by other pictures. can.

The TemporalId shown in FIG. 42 (a) is an identifier of the hierarchy of the coded structure, and the TemporalId indicates that the larger the value, the deeper the hierarchy. The square block indicates a picture, Ix in the block is an I picture (in-screen prediction picture), Px is a P picture (forward reference prediction picture), and Bx and bx are B pictures (bidirectional reference prediction picture). show. The x of Ix / Px / Bx indicates the display order, and indicates the order in which the pictures are displayed. The arrows between the pictures indicate a reference relationship, and for example, the picture of B4 indicates that I0 and B8 are used as reference images to generate a predicted image. Here, it is prohibited for one picture to use another picture having a TemporalId larger than its own TemporalId as a reference image. The hierarchy is defined to provide time scalability. For example, when all the pictures are decoded in FIG. 42, a video of 120 fps (frame per second) is obtained, but only the hierarchy whose Temporal Id is 0 to 3 is specified. Is decoded to obtain a video of 60 fps.

FIG. 43 is a diagram showing the relationship between the decoding time (DTS) and the display time (PTS) in each picture of FIG. 42. For example, the picture I0 shown in FIG. 43 is displayed after the decoding of B4 is completed so that a gap does not occur in the decoding and the display.

As shown in FIG. 43, when the B picture is included in the prediction structure, the decoding order and the display order are different. Therefore, the receiving device 20 decodes the picture, delays the picture, and rearranges the picture. (Reorder) Processing is required.

The example of the picture prediction structure in the time direction scalability has been described above. However, even when the time direction scalability is not used, the picture delay processing and the reorder processing may be required depending on the prediction structure. be. FIG. 44 is a diagram showing an example of a picture prediction structure that requires picture delay processing and reorder processing. The numbers in FIG. 44 indicate the decoding order.

As shown in FIG. 44, depending on the prediction structure, the first sample in the decoding order and the first sample in the presentation order may be different. In FIG. 44, the first sample in the presentation order is the decoding order. Is the 4th sample. Note that FIG. 44 shows an example of the prediction structure, and the prediction structure is not limited to such a structure. Also in other prediction structures, the sample that is the first in the decoding order and the sample that is the first in the presentation order may be different.

FIG. 45 is a diagram showing an example in which an MPU composed of an MP4 format is divided into a plurality of movie fragments and stored in an MMTP payload and an MMTP packet, as in FIG. 33. The number of samples constituting the MPU and the number of samples constituting the movie fragment are arbitrary. For example, two movie fragments may be configured by using the number of samples constituting the MPU as the number of samples in the GOP unit and the number of samples in half of the GOP unit as the movie fragment. One sample may be one movie fragment, or the samples constituting the MPU may not be divided.

FIG. 45 shows an example in which one MPU contains two movie fragments (moof box and mdat box), but one MPU does not have to contain two movie fragments. The movie fragment contained in one MPU may be three or more, or may be the number of samples included in the MPU. Further, the sample stored in the movie fragment may be divided into an arbitrary number of samples instead of the number of equally divided samples.

The movie fragment metadata (MF metadata) contains information on the PTS, DTS, offset, and size of the sample contained in the movie fragment, and the receiving device 20 receives the sample when decoding the sample. PTS and DTS are extracted from the MF meta containing the information of, and the decoding timing and the presentation timing are determined.

From here on, for detailed explanation, the absolute value of the decoding time of the i-sample is described as DTS (i), and the absolute value of the presentation time is described as PTS (i).

Of the time stamp information stored in the moof in the MF meta, the information of the i-th sample is specifically the relative value of the decoding time of the i-th sample and the (i + 1) th sample, and the i-th sample. It is a relative value of the decoding time and the presentation time of the sample of, and these are hereinafter referred to as DT (i) and CT (i).

Movie fragment metadata # 1 contains DT (i) and CT (i) of samples # 1- # 3, and movie fragment metadata # 2 includes DT (i) of samples # 4- # 6. ) And CT (i) are included.

Further, the PTS absolute value of the access unit at the head of the MPU is stored in the MPU time stamp descriptor or the like, and the receiving device 20 outputs the PTS and DTS based on the PTS_MPU of the access unit at the head of the MPU and CT and DT. calculate.

FIG. 46 is a diagram for explaining a method and a problem of calculating PTS and DTS when the MPU is composed of the samples of # 1 to # 10.

FIG. 46 (a) shows an example in which the MPU is not divided into movie fragments, FIG. 46 (b) shows an example in which the MPU is divided into two movie fragments in units of 5 samples, and FIG. 46 (c) shows an example. ) Shows an example in which the MPU is divided into 10 movie fragments for each sample.

As described with reference to FIG. 45, when PTS and DTS are calculated using the MPU time stamp descriptor and the time stamp information (CT and DT) in MP4, the sample that is the first in the presentation order in FIG. 44. Is the fourth in the decoding order. Therefore, the PTS stored in the MPU time stamp descriptor becomes the PTS (absolute value) of the fourth sample in the decoding order. Hereinafter, this sample will be referred to as an A sample. Further, the first sample in the decoding order is called a B sample.

Since the absolute time information related to the time stamp is only the information of the MPU time stamp descriptor, the receiving device 20 calculates the PTS (absolute time) and DTS (absolute time) of the other samples until the A sample arrives. Can not. The receiving device 20 cannot also calculate the PTS and DTS of the B sample.

In the example of (a) of FIG. 46, the A sample is included in the same movie fragment as the B sample and is stored in one MF meta. Therefore, the receiving device 20 can determine the DTS of the B sample immediately after receiving the MF meta.

In the example of (b) of FIG. 46, the A sample is included in the same movie fragment as the B sample and is stored in one MF meta. Therefore, the receiving device 20 can determine the DTS of the B sample immediately after receiving the MF meta.

In the example of (c) of FIG. 46, the A sample is included in a movie fragment different from the B sample. Therefore, the receiving device 20 can determine the DTS of the B sample only after receiving the CT of the movie fragment containing the A sample and the MF meta including the DT.

Therefore, in the case of the example of (c) of FIG. 46, the receiving device 20 cannot start decoding immediately after the arrival of the B sample.

As described above, when the movie fragment containing the B sample does not include the A sample, the receiving device 20 decodes the B sample only after receiving the MF meta related to the movie fragment containing the A sample. I can't start.

This problem occurs because the movie fragment is divided before the A sample and the B sample are not stored in the same movie fragment when the first sample in the presentation order and the first sample in the decoding order do not match. do. Also, this issue arises regardless of whether the MF meta is postponed or postponed.

In this way, when the first sample in the presentation order and the first sample in the decoding order do not match, and the A sample and the B sample are not stored in the same movie fragment, immediately after receiving the B sample. DTS cannot be determined. Therefore, the transmission device 15 separately transmits information that can calculate the DTS (absolute value) of the B sample or the DTS (absolute value) of the B sample on the receiving side. Such information may be transmitted using control information, packet headers, and the like.

The receiving device 20 calculates the DTS (absolute value) of the B sample using such information. FIG. 47 is a flowchart of the reception operation when the DTS is calculated using such information.

The receiving device 20 receives the movie fragment at the head of the MPU (S901), and determines whether or not the A sample and the B sample are stored in the same movie fragment (S902). When stored in the same movie fragment (Yes in S902), the receiving device 20 calculates the DTS using only the MF meta information without using the DTS (absolute time) of the B sample, and starts decoding. (S904). In step S904, the receiving device 20 may determine the DTS using the DTS of the B sample.

On the other hand, when the A sample and the B sample are not stored in the same movie fragment in step S902 (No in S902), the receiving device 20 acquires the DTS (absolute time) of the B sample, determines the DTS, and decodes it. Is started (S903).

In the above description, the absolute value of the decoding time and the absolute value of the presentation time of each sample are calculated using the MF meta (time stamp information stored in the MOOF of MP4 format) in the MMT standard. Although an example has been described, it goes without saying that the MF meta may be replaced with arbitrary control information that can be used for the calculation of the absolute value of the decoding time of each sample and the absolute value of the presentation time. As an example of such control information, the relative value CT (i) of the decoding time of the i-th sample and the (i + 1) th sample described above is used as the presentation time of the i-th sample and the (i + 1) th sample. The control information replaced with the relative value, and the relative value of the decoding time of the i-th sample and the (i + 1) th sample CT (i) and the relative value of the presentation time of the i-th sample and the (i + 1) th sample. There is control information including both.

(Embodiment 3)
[Overview]
In the third embodiment, a method of transmitting content and a data structure in the case of transmitting content such as video, audio, subtitles, and data broadcasting by broadcasting will be described. That is, the transmission method and the data structure of the content specialized for the reproduction of the broadcast stream will be described.

In the third embodiment, an example in which the MMT method (hereinafter, also simply referred to as MMT) is used as the multiplexing method will be described, but other multiplexing methods such as MPEG-DASH or RTP may be used. ..

First, the details of the method of storing the data unit (DU: Data Unit) in the payload in the MMT will be described. FIG. 48 is a diagram for explaining a method of storing a data unit in the payload in MMT.

In the MMT, the transmission device stores a part of the data constituting the MPU as a data unit in the MMTP payload and transmits it with a header. The header includes an MMTP payload header and an MMTP packet header. The unit of the data unit may be a NAL unit unit or a sample unit.

FIG. 48A shows an example in which the transmitting device aggregates a plurality of data units and stores them in one payload. In the example of FIG. 48 (a), a data unit header (DUH: Data Unit Header) and a data unit length (DUL: Data Unit Lens) are added to the heads of each of the plurality of data units, and the data unit header and the data unit header and Multiple data units with the data unit length are collectively stored in the payload.

FIG. 48B shows an example of storing one data unit in one payload. In the example of FIG. 48 (b), a data unit header is added to the head of the data unit and stored in the payload. FIG. 48 (c) shows an example in which one data unit is divided, a data unit header is added to the divided data unit, and the data unit is stored in the payload.

The data unit includes timed-MFU, which is a medium containing information on synchronization such as video, audio, or subtitles, non-timed-MFU, MPU metadata, MF metadata, etc., which is a medium not containing information on synchronization such as files. The data unit header is defined according to the type of data unit. The MPU metadata and the MF metadata do not have a data unit header.

Further, although the transmitting device cannot aggregate different types of data units in principle, it may be specified so that different types of data units can be aggregated. For example, when the size of the MF metadata is small, such as when each sample is divided into movie fragments, the number of packets can be reduced and the transmission capacity can be reduced by aggregating the MF metadata and the media data. You can also do it.

When the data unit is MFU, a part of MPU information such as information for configuring MPU (MP4) is stored as a header.

For example, the header of timed-MFU includes movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter, and the header of non-timed-MFU includes item_D. The meaning of each field is shown in standards such as ISO / IEC2608-1 or ARIB STD-B60. Hereinafter, the meaning of each field specified in such a standard will be described.

movie_fragment_sequence_number indicates the sequence number of the movie fragment to which the MFU belongs, and is also shown in ISO / IEC14496-12.

sample_number indicates the sample number to which the MFU belongs and is also shown in ISO / IEC14496-12.

The offset indicates the offset amount of the MFU in the sample to which the MFU belongs, in bytes.

The priority indicates the relative importance of the MFU in the MPU to which the MFU belongs, and indicates that the MFU having a large priority number is more important than the MFU having a small priority number.

Dependency_counter indicates the number of MFUs for which the decoding process depends on the MFU (that is, the number of MFUs for which the decoding process cannot be performed unless the decoding process is performed). For example, when the MFU is HEVC and the B picture or the P picture refers to the I picture, the B picture or the P picture cannot be decoded unless the I picture is decoded.

Therefore, when the MFU is a sample unit, the degree_counter in the MFU of the I picture indicates the number of pictures that refer to the I picture. When the MFU is a NAL unit unit, the degree_counter in the MFU belonging to the I picture indicates the number of NAL units belonging to the picture referring to the I picture. Further, in the case of a time-directed hierarchically coded video signal, since the MFU of the expansion layer depends on the MFU of the base layer, the degree_counter in the MFU of the base layer indicates the number of MFUs of the expansion layer. This field can only be generated after the number of dependent MFUs has been determined.

item_iD indicates an identifier that uniquely identifies the item.

[MP4 non-support mode]
As described with reference to FIGS. 19 and 21, as a method for transmitting the MPU in the MMT, a method of transmitting MPU metadata or MF metadata before or after the media data, and only the media data are used. There is a way to send. Further, in the receiving device, there are a method of performing decoding using a receiving device and a receiving method compliant with MP4, and a method of decoding without using a header.

As a data transmission method specialized for broadcasting stream reproduction, for example, there is a transmission method that does not support MP4 reconstruction in a receiving device.

As a transmission method that does not support MP4 reconstruction in the receiving device, for example, there is a method that does not transmit metadata (MPU metadata and MF metadata) as shown in FIG. 21 (b). In this case, the field value of the fragment type (information indicating the type of the data unit) included in the MMTP packet is fixed at 2 (= MFU).

When the metadata is not transmitted, as described above, the received data cannot be decoded as MP4 by an MP4 compliant receiving device or the like, but it can be decoded without using the metadata (header). Is.

Therefore, the metadata is not necessarily essential information for broadcasting stream decoding and reproduction. Similarly, the information in the data unit header in the timing-MFU described with reference to FIG. 48 is information for reconstructing the MP4 in the receiving device. Since it is not necessary to reconstruct the MP4 in the broadcast stream reproduction, the information of the data unit header (hereinafter, also referred to as the timing-MFU header) in the timing-MFU is not necessarily the information necessary for the broadcast stream reproduction.

The receiving device can easily reconstruct the MP4 by using the metadata and the information for reconstructing the MP4 in the data unit header (hereinafter, also referred to as MP4 configuration information). However, the receiving device cannot reconfigure the MP4 even if only one of the metadata and the MP4 configuration information in the data unit header is transmitted. The merit of transmitting only one of the metadata and the information for reconstructing MP4 is small, and generating and transmitting unnecessary information causes an increase in processing and a decrease in transmission efficiency.

Therefore, the transmission device controls the data structure and transmission of the MP4 configuration information by using the following method. The transmitting device determines whether to indicate MP4 configuration information in the data unit header based on whether the metadata is transmitted. Specifically, the transmitting device shows the MP4 configuration information in the data unit header when the metadata is transmitted, and does not show the MP4 configuration information in the data unit header when the metadata is not transmitted.

As a method of not showing MP4 configuration information in the data unit header, for example, the following method can be used.

1. 1. The transmitting device sets the MP4 configuration information to reserved and does not operate it. As a result, it is possible to reduce the processing amount (processing amount of the transmitting device) on the transmitting side that generates MP4 configuration information.

2. 2. The transmitting device deletes the MP4 configuration information and compresses the header. As a result, the amount of processing on the transmitting side that generates MP4 configuration information can be reduced, and the transmission capacity can be reduced.

When the MP4 configuration information is deleted and the header is compressed, the transmission device may indicate a flag indicating that the MP4 configuration information has been deleted (compressed). Flags are indicated in headers (MMTP packet headers, MMTP payload headers, data unit headers), control information, and the like.

Further, the information on whether or not the metadata is transmitted may be predetermined, or may be separately signaled to a header or control information and transmitted to the receiving device.

For example, information as to whether the metadata corresponding to the MFU is transmitted may be stored in the MFU header.

On the other hand, the receiving device can determine whether the MP4 configuration information is shown based on whether the metadata is being transmitted.

Here, if the order in which the data is transmitted (eg, MPU metadata, MF metadata, media data, etc.) is determined, the receiving device is based on whether the metadata was received before the media data. May be determined.

When the MP4 configuration information is shown, the receiving device can use the MP4 configuration information to reconfigure the MP4. Alternatively, the receiving device can use the MP4 configuration information for detecting the head of another access unit or NAL unit.

The MP4 configuration information may be the whole or a part of the timing-MFU header.

Similarly, in the non-timed-MFU header, the transmitting device also uses the item in the non-timed-MFU header based on whether or not the metadata is transmitted.
You may decide whether to indicate id.

The transmitter may show MP4 configuration information only in either the timing-MFU or the non-timed-MFU. If the MP4 configuration information is shown to only one of them, the transmitter determines whether to show the MP4 configuration information based on whether the metadata is transmitted and whether it is timing-MFU or non-timed-MFU. do. The receiving device can determine whether the metadata is transmitted and whether the MP4 configuration information is indicated based on the timing / non-timed flag.

In the above description, the transmitting device determines whether or not to indicate the MP4 configuration information based on whether or not the metadata (both MPU metadata and MF metadata) is transmitted. However, the transmitting device may not show the MP4 configuration information when a part of the metadata (either MPU metadata or MF metadata) is not transmitted.

The transmitting device may also determine whether to indicate MP4 configuration information based on information other than metadata.

For example, a mode such as MP4 supported mode / MP4 unsupported mode is defined, and the transmitting device indicates MP4 configuration information in the data unit header in the case of MP4 supported mode, and data in the case of MP4 unsupported mode. The MP4 configuration information may not be shown in the unit header. Further, the transmitting device transmits metadata in the MP4 support mode and indicates MP4 configuration information in the data unit header, and in the MP4 non-support mode, the data unit header does not transmit the metadata. Also, the MP4 configuration information may not be shown.

[Operation flow of transmitter]
Next, the operation flow of the transmitter will be described. FIG. 49 is an operation flow of the transmission device.

The transmitting device first determines whether or not to transmit the metadata (S1001). When the transmitting device determines that the metadata is to be transmitted (Yes in S1002), the transmission device proceeds to step S1003, generates MP4 configuration information, stores it in a header, and transmits it (S1003). In this case, the transmitting device also generates and transmits metadata.

On the other hand, when it is determined that the metadata is not transmitted (No in S1002), the transmission device transmits the MP4 configuration information without generating it and storing it in the header (S1004). In this case, the transmitter does not generate and transmit the metadata.

Whether or not the metadata is transmitted in step S1001 may be predetermined, and whether or not the metadata is generated inside the transmitting device and whether or not the metadata is transmitted inside the transmitting device. It may be determined based on.

[Operation flow of receiver]
Next, the operation flow of the receiving device will be described. FIG. 50 is an operation flow of the receiving device.

The receiving device first determines whether or not the metadata is being transmitted (S1101). Whether or not metadata is being transmitted can be determined by monitoring the fragment type in the MMTP packet payload. Further, it may be predetermined whether or not the transmission is performed.

When it is determined that the metadata is transmitted (Yes in S1102), the receiving device reconfigures the MP4 and executes the decoding process using the MP4 configuration information (S1103). On the other hand, when it is determined that the metadata is not transmitted (No in S1102), the MP4 reconstruction process is not performed and the decryption process is executed without using the MP4 configuration information (S1104).

The receiving device can detect a random access point, detect the head of an access unit, detect the head of a NAL unit, and the like without using MP4 configuration information by using the methods described so far, and can perform decoding processing. , Packet loss can be detected, and recovery from packet loss can be processed.

For example, the head of the access unit is the head data of the MMT payload having an aggregation_flag value of 1. At this time, the Fragmentation_indicator value is 0.

Further, the head of the slice segment is the head data of the MMT payload having an aggregation_flag value of 0 and a fragmentation_indicator value of 00 or 01.

The receiving device can detect the head of the access unit and detect the slice segment based on the above information.

The receiving device analyzes the NAL unit header in the packet including the head of the data unit whose fragmentation_indicator value is 00 or 01, the type of the NAL unit is the AU delimiter, and the type of the NAL unit is the slice segment. It may be detected that.

[Broadcast simple mode]
So far, as a method of transmitting data specialized for broadcasting stream reproduction, a method of not supporting MP4 configuration information in a receiving device has been described, but the method of transmitting data specialized for broadcasting stream reproduction is not limited to this. No.

For example, the following method may be used as a data transmission method specialized for broadcasting stream reproduction.

-The transmitting device does not have to use AL-FEC in the fixed reception environment of broadcasting. When AL-FEC is not used, FEC_type in the MMTP packet header is always fixed at 0.

-The transmission device may always use AL-FEC in the mobile reception environment of broadcasting and the communication UDP transmission mode. When AL-FEC is used, FEC_type in the MMTP packet header is always 0 or 1.

-The transmitter does not have to perform bulk transmission of assets. When bulk transmission of the asset is not performed, the location_information indicating the number of transmission locations of the asset inside the MPT may be fixed to 1.

The transmitter does not have to carry out hybrid transmission of assets, programs, and messages.

Further, for example, when the broadcast simple mode is defined and the broadcast simple mode is used, the MP4 non-support mode is set, or the above-mentioned data transmission method specialized for broadcast stream reproduction is used. May be. Whether or not it is in the broadcast simple mode may be predetermined, or the transmitting device may store a flag indicating that the broadcasting simple mode is used as control information and transmit it to the receiving device.

Further, based on whether or not the transmitting device is in the MP4 non-supporting mode (whether or not metadata is transmitted) described with reference to FIG. 49, when the transmitting device is in the MP4 non-supporting mode, the above-mentioned broadcasting simple mode is used. The data transmission method specialized for broadcasting stream reproduction shown in the above may be used.

In the case of the broadcast simple mode, the receiving device can perform the decoding process without reconfiguring the MP4, assuming that it is the MP4 non-support mode.

Further, in the case of the broadcast simple mode, the receiving device can determine that the function is specialized for broadcasting and can perform reception processing specialized for broadcasting.

As a result, in the broadcast simple mode, by using only the functions specialized for broadcasting, not only unnecessary processing for the transmitting device and the receiving device can be reduced, but also unnecessary information is not compressed and transmitted. Therefore, the transmission overhead can be reduced.

When the MP4 non-support mode is used, hint information supporting a storage method other than the MP4 configuration may be shown.

As the storage method other than the MP4 configuration, for example, there are a method of directly storing MMT packets and IP packets, a method of converting MMT packets into MPEG-2 TS packets, and the like.

In the case of the MP4 non-support mode, a format that does not conform to the MP4 configuration may be used.

For example, in the case of MP4 non-support mode, the data stored in the MFU may be in a format with a byte start code instead of a format in which the size of the NAL unit is added to the beginning of the NAL unit in MP4 format. good.

In MMT, the asset type indicating the asset type is described by 4CC registered in MP4REG (http://www.mp4ra.org), and when HEVC is used as a video signal,'HEV1' or'HVC1'is used. Be done. 'HVC1' is a format in which the parameter set may be included in the sample, and'HEV1'is a format in which the parameter set is not included in the sample and the parameter set is included in the sample entry in the MPU metadata.

In the case of broadcast simple mode or MP4 non-support mode, it may be specified that the parameter set is always included in the sample when the MPU metadata and the MF metadata are not transmitted. Further, regardless of whether the asset type indicates "HEV1" or "HVC1", it may be specified that the asset type always takes the form of "HVC1".

[Supplement 1: Transmitter]
As described above, when the metadata is not transmitted, the MP4 configuration information is set to reserved, and the transmission device that is not operated can be configured as shown in FIG. 51. FIG. 51 is a diagram showing an example of a specific configuration of the transmission device.

The transmission device 300 includes a coding unit 301, an imparting unit 302, and a transmission unit 303. Each of the coding unit 301, the adding unit 302, and the transmitting unit 303 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The coding unit 301 encodes a video signal or an audio signal to generate sample data. The sample data is specifically a data unit.

The adding unit 302 adds header information including MP4 configuration information to the sample data which is the data in which the video signal or the audio signal is encoded. The MP4 configuration information is information for reconstructing the sample data as an MP4 format file on the receiving side, and the content differs depending on whether or not the presentation time of the sample data is set.

As described above, the addition unit 302 has a header (header information) of timed-MFU, which is an example of sample data (sample data including information on synchronization) in which the presentation time is determined, such as move_fragment_sequence_number, sample_number, office, and priority. , And MP4 configuration information such as header_counter.

On the other hand, the addition unit 302 has MP4 configuration information such as item_id in the header (header information) of timing-MFU, which is an example of sample data (sample data that does not include information about synchronization) for which the presentation time is not defined. To include.

Then, when the metadata corresponding to the sample data is not transmitted by the transmission unit 303 (for example, in the case of (b) of FIG. 21), the granting unit 302 determines whether or not the presentation time of the sample data is set. Header information that does not include MP4 configuration information is added to the sample data according to the question.

Specifically, when the presentation time of the sample data is determined, the addition unit 302 assigns the header information that does not include the first MP4 configuration information to the sample data, and the presentation time of the sample data is determined. If not, header information including the second MP4 configuration information is added to the sample data.

For example, as shown in step S1004 of FIG. 49, the granting unit 302 sets the MP4 configuration information to reserved (fixed value) when the metadata corresponding to the sample data is not transmitted by the transmitting unit 303. It does not substantially generate MP4 configuration information and does not substantially store it in the header (header information). The metadata includes MPU metadata and movie fragment metadata.

The transmission unit 303 transmits sample data to which header information is added. More specifically, the transmission unit 303 packetizes the sample data to which the header information is added by the MMT method and transmits the packet data.

As described above, in the transmission method and the reception method specialized for the reproduction of the broadcast stream, it is not necessary to reconfigure the data unit to MP4 in the receiving device. When the receiving device does not need to be reconfigured to MP4, the processing of the transmitting device is reduced by not generating unnecessary information such as MP4 configuration information.

On the other hand, the transmitting device must send necessary information, but it is necessary to maintain consistency with the standard so that it is not necessary to separately send extra additional information and the like.

According to the configuration such as the transmission device 300, by setting the area where the MP4 configuration information is stored to a fixed value, the MP4 configuration information is not transmitted, and only the necessary information is transmitted based on the standard, which is unnecessary. The effect is that you do not have to send additional information. That is, the configuration of the transmission device and the processing amount of the transmission device can be reduced. Further, the transmission efficiency can be improved by preventing unnecessary data from being transmitted.

[Supplement 2: Receiver]
Further, the receiving device corresponding to the transmitting device 300 may be configured as shown in FIG. 52, for example. FIG. 52 is a diagram showing another example of the configuration of the receiving device.

The receiving device 400 includes a receiving unit 401 and a decoding unit 402. The receiving unit 401 and the decoding unit 402 are realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving unit 401 is sample data in which a video signal or an audio signal is encoded, and is provided with header information including MP4 configuration information for reconstructing the sample data as an MP4 format file. Receive data.

The decoding unit 402 decodes the sample data without using the MP4 configuration information when the metadata corresponding to the sample data is not received by the receiving unit and the presentation time of the sample data is set. do.

For example, as shown in step S1104 of FIG. 50, when the receiving unit 401 does not receive the metadata corresponding to the sample data, the decoding unit 402 executes the decoding process without using the MP4 configuration information.

Thereby, the configuration of the receiving device 400 and the processing amount in the receiving device 400 can be reduced.

(Embodiment 4)
[Overview]
In the fourth embodiment, a method of storing an asynchronous (non-timed) medium that does not include information related to synchronization such as a file in the MPU and a method of transmitting an MMTP packet will be described. In the fourth embodiment, MPU in MMT will be described as an example, but DASH, which is the same MP4 base, can also be applied.

First, the details of the storage method of the non-timed media (hereinafter, also referred to as “asynchronous media data”) in the MPU will be described with reference to FIG. 53. FIG. 53 is a diagram showing a method of storing a non-timed medium in an MPU and a method of transmitting an MMTP packet.

The MPU that stores the non-timed media is composed of boxes such as ftyp, mmpu, moov, and meta, and stores information about files to be stored in the MPU. A plurality of idat boxes can be stored in the meta box, and one file is stored as an item in the idat box.

A part of the ftyp, mmpu, moov, and meta boxes constitutes one data unit as MPU metadata, and the item or idat box constitutes a data unit as MFU.

After the data unit is aggregated or fragmented, the data unit header, the MMTP payload header, and the MMTP packet header are added and transmitted as an MMTP packet.

Note that FIG. 53 shows an example in which File # 1 and File # 2 are stored in one MPU. The MPU metadata is not divided, and the MFU is divided and stored in the MMTP packet, but the present invention is not limited to this, and may be aggregated or fragmented according to the size of the data unit. Further, the MPU metadata does not have to be transmitted, in which case only the MFU is transmitted.

The itemID (identifier that uniquely identifies the item) is shown in the header information such as the data unit header, and the packet sequence number (sequence number for each packet) and the MPU sequence number are shown in the MMTP payload header and MMTP packet header. (MPU sequence number, unique number within the asset.) Is included.

The data structures of the MMTP payload header and MMTP packet header other than the data unit header are the same as those of the timed media (hereinafter, also referred to as “synchronous media data”) described so far, and include aggregation_flag, fragmentation_indicator, fragment_counter, and the like. ..

Next, a specific example of the header information when the file (= Item = MFU) is divided into packets will be described with reference to FIGS. 54 and 55.

54 and 55 are diagrams showing an example of packetizing and transmitting each of a plurality of divided data obtained by dividing the file. Specifically, FIGS. 54 and 55 show information (packet sequence number, fragment counter, fragmentation) included in any of the data unit header, the MMTP payload header, and the MMTP packet header, which are header information for each divided MMTP packet. Indicator, MPU sequence number, item ID). Note that FIG. 54 is a diagram showing an example in which File # 1 is divided into M pieces (M <= 256), and FIG. 55 is an example in which File # 2 is divided into N pieces (256 <N). It is a figure which shows.

The divided data number indicates the index of the divided data from the beginning of the file, and this information is not transmitted. That is, the divided data number is not included in the header information. The divided data number is a number assigned to the packet corresponding to each of the plurality of divided data obtained by dividing the file, and is a number assigned by adding 1 in ascending order from the first packet. be.

The packet sequence number is a sequence number of packets having the same packet ID, and in FIGS. 54 and 55, the divided data at the beginning of the file is set as A, and consecutive numbers are assigned up to the divided data at the end of the file. The packet sequence number is a number assigned by adding 1 in ascending order from the divided data at the beginning of the file, and is a number corresponding to the divided data number.

The fragment counter indicates the number of the plurality of divided data after the divided data among the plurality of divided data obtained by dividing one file. Further, the fragment counter indicates a remainder obtained by dividing the number of divided data by 256 when the number of divided data, which is the number of the plurality of divided data obtained by dividing one file, exceeds 256. In the example of FIG. 54, since the number of divided data is 256 or less, the field value of the fragment counter is (M-divided data number). On the other hand, in the example of FIG. 55, since the number of divided data exceeds 256, the value is obtained by dividing (N-divided data number) by 256 ((N-divided data number)% 256).

The fragmentation indicator indicates the state of division of the data stored in the MMTP packet, and whether it is the first divided data, the last divided data, or other divided data in the divided data unit. Alternatively, it is a value indicating whether the data unit is one or more undivided data units. Specifically, the fragmentation indicator is "01" for the first split data, "11" for the last split data, "10" for the remaining split data, and "10" for the unsplit data unit. 00 ".

In the present embodiment, when the number of divided data exceeds 256, it is described as indicating the remainder obtained by dividing the number of divided data by 256, but the number of divided data is not limited to 256 and may be another number (predetermined number). You may.

As shown in FIGS. 54 and 55, when the file is divided and the conventional header information is added to each of the plurality of divided data obtained by dividing the file and transmitted, the MMTP packet received by the receiving device is transmitted. There is no information on which number of divided data in the original file the data stored in is (divided data number), the number of divided data in the file, or the divided data number and the number of divided data can be derived. Therefore, in the conventional transmission method, even if the MMTP packet is received, the divided data number and the number of divided data of the data stored in the received MMTP packet cannot be uniquely detected.

For example, when it is known that the number of divided data is 256 or less and the number of divided data is 256 or less in advance as shown in FIG. 54, the divided data number and the number of divided data can be determined by referring to the fragment counter. It is possible to identify. However, when the number of divided data is 256 or more, the divided data number and the number of divided data cannot be specified.

When the number of divided data in a file is limited to 256 or less, when the data size that can be transmitted in one packet is x [bytes], the maximum size of the file that can be transmitted is limited to x * 256 [bytes]. File. For example, in broadcasting, x = 4k [bytes] is assumed, and in this case, the maximum size of a file that can be transmitted is limited to 4k * 256 = 1M [bytes]. Therefore, when it is desired to transmit a file exceeding 1 [Mbytes], the number of divided data of the file cannot be limited to 256 or less.

Further, for example, since the first divided data and the last divided data of the file can be detected by referring to the fragmentation indicator, the number of MMTP packets is counted until the MMTP packet including the last divided data of the file is received. Since it is possible to calculate the divided data number and the number of divided data by combining with the packet sequence number after receiving the MMTP packet containing the last divided data of the file, the fragmentation indicator and the packet sequence The divided data number and the number of divided data may be signaled by combining with the number. However, when reception is started from an MMTP packet containing divided data in the middle of the file (that is, divided data that is neither the divided data at the beginning of the file nor the divided data at the end of the file), the divided data number or divided data of the divided data is started. The number cannot be specified. The divided data number and the number of divided data of the divided data can be specified only after receiving the MMTP packet including the divided data at the end of the file.

The following method is used to uniquely determine the divided data number and the number of divided data of the divided data of the file at the time when the problem described with reference to FIGS. 54 and 55, that is, the packet containing the divided data of the file is received from the middle. Is used.

First, the divided data number will be described.

For the divided data number, the packet sequence number in the divided data at the beginning of the file (item) is signaled.

As a signaling method, it is stored in the control information that manages the file. Specifically, in FIGS. 54 and 55, the packet sequence number A of the divided data at the beginning of the file is stored in the control information. The receiving device acquires the value of A from the control information and calculates the divided data number from the packet sequence number shown in the packet header.

The divided data number of the divided data is obtained by subtracting the packet sequence number A of the first divided data from the packet sequence number of the divided data.

As control information for managing files, for example, there is an asset management table defined in ARIB STD-B60. In the asset management table, file size, version information, etc. are shown for each file, and they are stored in a data transmission message and transmitted. FIG. 56 is a diagram showing the syntax of the loop for each file in the asset management table.

If the area of the existing asset management table cannot be expanded, signaling may be performed using a 32-bit area of a part of the item_info_byte field indicating item information. A flag indicating whether or not the packet sequence number in the divided data at the beginning of the file (item) is indicated in a part of the item_info_byte area may be indicated in, for example, the reserved_future_use field of the control information.

When a file such as a data carousel is repeatedly transmitted, a plurality of packet sequence numbers may be indicated, or the packet sequence number at the beginning of the file to be transmitted immediately after may be indicated.

Not limited to the packet sequence number of the divided data at the beginning of the file, any information may be used as long as it is information that associates the divided data number of the file with the packet sequence number.

Next, the number of divided data will be described.

The order of loops for each file included in the asset management table may be defined as the file transmission order. As a result, since the packet sequence number at the beginning of two consecutive files in the transmission order can be known, the first packet sequence number of the file transmitted before can be subtracted from the first packet sequence number of the file transmitted later. It is possible to specify the number of divided data of the transmitted file. That is, for example, when File # 1 shown in FIG. 54 and File # 2 shown in FIG. 55 are consecutive files in this order, the last packet sequence number of File # 1 and the first packet sequence of File # 2 Consecutive numbers are assigned to the numbers.

Further, by specifying the method of dividing the file, it may be specified so that the number of divided data of the file can be specified. For example, when the number of divided data is N, the size of each of the 1st to (N-1) th divided data is L, and the size of the Nth divided data is a fraction (item_size-L * (N-1)). By specifying, the number of divided data can be calculated back from item_size shown in the asset management table. In this case, the integer value rounded up from (item_size / L) is the number of divided data. The method of dividing the file is not limited to this.

Further, the number of divided data may be stored directly in the asset management table.

The receiving device receives the control information by using the above method, and calculates the number of divided data based on the control information. Further, the packet sequence number corresponding to the divided data number of the file can be calculated based on the control information. If the timing of receiving the packet of the divided data is earlier than the timing of receiving the control information, the divided data number or the number of divided data may be calculated at the timing of receiving the control information.

When signaling the divided data number or the number of divided data by using the above method, the divided data number and the number of divided data are not specified based on the fragment counter, and the fragment counter becomes unnecessary data. Therefore, in the transmission of asynchronous media, if the divided data number and the information that can specify the number of divided data are signaled by using the above method or the like, the fragment counter may not be operated or the header may be compressed. .. As a result, the processing amount of the transmitting device and the receiving device can be reduced, and the transmission efficiency can be improved. That is, when transmitting asynchronous media, the fragment counter may be reserved. Specifically, the value of the fragment counter may be a fixed value of, for example, "0". Further, when receiving asynchronous media, the fragment counter may be ignored.

When storing synchronous media such as video and audio, the transmission order of MMTP packets in the transmitting device and the arrival order of MMTP packets in the receiving device match, and the packets are not retransmitted. In such a case, if it is not necessary to detect and reconstruct the packet loss, the fragment counter may not be operated. In other words, in this case, the fragment counter may be reserved.

In addition, it is possible to detect a random access point, detect the head of an access unit, detect the head of a NAL unit, etc. without using a fragment counter, and it is possible to perform decoding processing, detect packet loss, and recover from packet loss. Can be processed.

Further, in the transmission of real-time contents such as live broadcasting, lower delay transmission is required, and it is required to sequentially packetize and transmit the coded data. However, in the transmission of real-time content, the conventional fragment counter cannot determine the number of divided data when the first divided data is transmitted. Therefore, in the transmission of the first divided data, all the coding of the data unit is completed and the divided data is transmitted. There will be a delay after the number has been determined. Even in such a case, this delay can be reduced by not operating the fragment counter by using the above method.

FIG. 57 is an operation flow for specifying the divided data number in the receiving device.

The receiving device acquires the control information in which the file information is described (S1201). The receiving device determines whether or not the packet sequence number at the beginning of the file is indicated in the control information (S1202), and if the packet sequence number at the beginning of the file is indicated in the control information (Yes in S1202), the file is filed. The packet sequence number corresponding to the divided data number of the divided data of is calculated (S1203). Then, after acquiring the MMTP packet in which the divided data is stored, the receiving device identifies the divided data number of the file from the packet sequence number stored in the packet header of the acquired MMTP packet (S1204).

On the other hand, when the control information does not indicate the packet sequence number at the beginning of the file (No in S1202), the receiving device acquires the MMTP packet containing the divided data at the end of the file, and then acquires the packet header of the MMTP packet. The fragmented data number is specified by using the fragment indicator stored in the packet sequence number and the packet sequence number (S1205).

FIG. 58 is an operation flow for specifying the number of divided data in the receiving device.

The receiving device acquires the control information in which the file information is described (S1301). The receiving device determines whether or not the control information includes information that can calculate the number of divided data of the file (S1302), and if it determines that the information includes information that can calculate the number of divided data, the receiving device determines. (Yes in S1302), the number of divided data is calculated based on the information included in the control information (S1303). On the other hand, when the receiving device determines that the number of divided data cannot be calculated (No in S1302), the receiving device acquires the MMTP packet containing the divided data at the end of the file and then stores it in the packet header of the acquired MMTP packet. The number of divided data is specified by using the fragment indicator and the packet sequence number (S1304).

FIG. 59 is an operation flow for determining whether or not to operate the fragment counter in the transmitting device.

First, the transmission device determines whether the medium to be transmitted (hereinafter, also referred to as “media data”) is a synchronous medium or an asynchronous medium (S1401).

When the result of the determination in step S1401 is the synchronous media (synchronous media in S1402), does the transmitting device match the transmission / reception MMTP packet order in the environment for transmitting the synchronous media and does not require packet reconstruction at the time of packet loss? It is determined whether or not (S1403). When the transmitting device determines that it is unnecessary (Yes in S1403), the transmitting device does not operate the fragment counter (S1404). On the other hand, when it is determined that the transmission device is not unnecessary (No in S1403), the transmission device operates the fragment counter (S1405).

When the result of the determination in step S1401 is asynchronous media (asynchronous media in S1402), the transmitting device is fragmented based on whether the divided data number or the number of divided data is signaled using the method described above. Decide whether to operate the counter. Specifically, the transmitting device does not operate the fragment counter (S1404) when the divided data number or the number of divided data is signaled (Yes in S1406). On the other hand, when the divided data number or the number of divided data is not signaled (No in S1406), the transmitting device operates a fragment counter (S1405).

When the transmission device does not operate the fragment counter, the value of the fragment counter may be reserved or the header may be compressed.

The transmitting device may determine whether to signal the above-mentioned divided data number or the number of divided data based on whether or not the fragment counter is operated.

When the synchronous media does not operate the fragment counter, the transmitting device may signal the divided data number and the number of divided data by using the method described above in the asynchronous media. Conversely, the operation of synchronous media may be determined based on whether asynchronous media operates fragment counters. In this case, whether or not to operate the fragment can be the same for the synchronous media and the asynchronous media.

Next, a method of specifying the number of divided data and the divided data number (when utilizing the fragment counter) will be described. FIG. 60 is a diagram for explaining a method of specifying the number of divided data and the divided data number (when the fragment counter is utilized).

As described with reference to FIG. 54, when it is known that the number of divided data is 256 or less and the number of divided data is 256 or less in advance, the divided data number or division can be obtained by referring to the fragment counter. It is possible to specify the number of data.

When the number of divided data in a file is limited to 256 or less, and the data size that can be transmitted in one packet is x [bytes], the maximum size of the file that can be transmitted is limited to x * 256 [bytes]. For example, in broadcasting, x = 4k [bytes] is assumed, and in this case, the maximum size of a file that can be transmitted is limited to 4k * 256 = 1M [bytes].

When the file size exceeds the maximum size of the file that can be transmitted, the file is divided in advance so that the size of the divided file is x * 256 [bytes] or less. Each of the plurality of divided files obtained by dividing the file is treated as one file (item), further divided within 256, and the divided data obtained by further division is converted into an MMTP packet. Stored and transmitted.

Information indicating that the item is a divided file, the number of divided files, and the sequence number of the divided files may be stored in the control information and transmitted to the receiving device. Further, these pieces of information may be stored in the asset management table, or may be shown using a part of the existing field item_info_byte.

If the item is one of a plurality of split files obtained by splitting one file, the receiver identifies the other split file and reconstructs the original file. Can be done. Further, in the receiving device, the number of divided data and the divided data number can be uniquely specified by using the number of divided files of the divided file, the index of the divided files, and the fragment counter in the control information. Further, the number of divided data and the divided data number can be uniquely specified without using the packet sequence number or the like.

Here, it is desirable that the item_id of each of the plurality of divided files obtained by dividing one file is the same as each other. When another item_id is assigned, the item_id of the first divided file may be indicated in order to uniquely refer to the file from other control information or the like.

Further, the plurality of divided files may always belong to the same MPU. When a plurality of files are stored in the MPU, different types of files may not be stored, and a file obtained by dividing one file may be stored without fail. The receiving device can detect the update of the file by checking the version information for each MPU without checking the version information for each item.

FIG. 61 is an operation flow of the transmission device when the fragment counter is utilized.

First, the transmitting device confirms the size of the file to be transmitted (S1501). Next, the transmitting device determines whether or not the file size exceeds x * 256 [bytes] (x is the data size that can be transmitted in one packet. For example, the MTU size) (S1502), and the file size is x. If it exceeds * 256 [bytes] (Yes in S1502), the file is divided so that the size of the divided file is less than x * 256 [bytes] (S1503). Then, the divided file is transmitted as an item, and information about the divided file (for example, being a divided file, a sequence number in the divided file, etc.) is stored in the control information and transmitted (S1504). On the other hand, if the file size is less than x * 256 [bytes] (No in S1502), the file is transmitted as an item as usual (S1505).

FIG. 62 is an operation flow of the receiving device when the fragment counter is utilized.

First, the receiving device acquires and analyzes control information related to the transmission of a file such as an asset management table (S1601). Next, the receiving device determines whether the desired item is a split file (S1602). When the receiving device determines that the desired file is a divided file (Yes in S1602), the receiving device acquires information for reconstructing the file such as the divided file and the index of the divided file from the control information (S1603). .. Then, the receiving device acquires the items constituting the divided file and reconstructs the original file (S1604). On the other hand, when the receiving device determines that the desired file is not a divided file (No in S1602), the receiving device acquires the file as usual (S1605).

In short, the transmitting device signals the packet sequence number of the divided data at the beginning of the file. Further, the transmitting device signals information that can specify the number of divided data. Alternatively, the transmitting device defines a division rule that can specify the number of division data. Further, the transmitting device performs reserved or header compression without operating the fragment counter.

When the packet sequence number of the data at the beginning of the file is signaled, the receiving device identifies the divided data number and the number of divided data from the packet sequence number of the divided data at the beginning of the file and the packet sequence number of the MMTP packet. ..

From another point of view, the transmitting device divides the file and divides the data into each divided file and transmits the data. Signal the information (sequence number, number of divisions, etc.) that associates the divided files.

The receiving device specifies the divided data number and the number of divided data by the fragment counter and the sequence number of the divided file.

As a result, the divided data number and the divided data can be uniquely specified. Further, since the divided data number of the divided data can be specified at the time when the divided data in the middle is received, the waiting time can be reduced and the memory can be reduced.

Further, by not operating the fragment counter, the configuration of the transmission / reception device can reduce the processing amount and improve the transmission efficiency.

FIG. 63 is a diagram showing a service configuration when the same program is transmitted by a plurality of IP data flows. Here, the data of a part (video / audio) of the program with the service ID = 2 is transmitted by the IP data flow using the MMT method, and the data having the same service ID but different from the part of the data is the advanced BS. An example of transmission by an IP data flow using a data transmission method (in this example, the file transmission protocol is different but the same protocol may be used) is shown.

The transmitting device multiplexes the IP data in the receiving device so that the data composed of a plurality of IP data flows can be guaranteed to be aligned by the decoding time.

The receiving device can realize the guaranteed receiver operation by processing the data composed of a plurality of IP data flows based on the decoding time.

[Supplement: Transmitter and receiver]
As described above, the transmission device that transmits data without operating the fragment counter can also be configured as shown in FIG. 64. Further, the receiving device that receives data without operating the fragment counter can also be configured as shown in FIG. 65. FIG. 64 is a diagram showing an example of a specific configuration of the transmission device. FIG. 65 is a diagram showing an example of a specific configuration of the receiving device.

The transmission device 500 includes a division unit 501, a component unit 502, and a transmission unit 503. Each of the division unit 501, the component unit 502, and the transmission unit 503 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving device 600 includes a receiving unit 601, a determination unit 602, and a constituent unit 603. Each of the receiving unit 601 and the determining unit 602 and the constituent unit 603 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

A detailed description of each component of the transmitting device 500 and the receiving device 600 will be given in the description of the transmitting method and the receiving method, respectively.

First, the transmission method will be described with reference to FIG. FIG. 66 is an operation flow (transmission method) by the transmission device.

First, the division unit 501 of the transmission device 500 divides the data into a plurality of division data (S1701).

Next, the component unit 502 of the transmission device 500 configures a plurality of packets by adding header information to each of the plurality of divided data and forming a packet (S1702).

Then, the transmission unit 503 of the transmission device 500 transmits the configured plurality of packets (S1703). The transmission unit 503 transmits the divided data information and the value of the invalidated fragment counter. The divided data information is information for specifying the divided data number and the number of divided data. Further, the divided data number is a number indicating which division data the divided data is among the plurality of divided data. The number of divided data is the number of a plurality of divided data.

As a result, the processing amount of the transmission device 500 can be reduced.

Next, the receiving method will be described with reference to FIG. 67. FIG. 67 is an operation flow (reception method) by the receiving device.

First, the receiving unit 601 of the receiving device 600 receives a plurality of packets (S1801).

Next, the determination unit 602 of the receiving device 600 determines whether or not the divided data information has been acquired from the received plurality of packets (S1802).

Then, when the determination unit 602 determines that the divided data information has been acquired (Yes in S1802), the component unit 603 of the receiving device 600 receives the data without using the value of the fragment counter included in the header information. Data is composed of a plurality of packets (S1803).

On the other hand, when the determination unit 602 determines that the divided data information has not been acquired (No in S1802), the configuration unit 603 uses the values of the fragment counter included in the header information to receive a plurality of received data. Data may be configured from packets (S1804).

As a result, the processing amount of the receiving device 600 can be reduced.

(Embodiment 5)
[Overview]
In the fifth embodiment, a transmission packet (TLV packet) transmission method when the NAL unit is stored in the multiplexing layer in the NAL size format will be described.

As described in the first embodiment, H.I. 264 and H. When storing the 265 NAL unit in the multiplexing layer, there are the following two types of formats. One is a format called a byte stream format in which a start code consisting of a specific bit string is added immediately before the NAL unit header. The other is a format called NAL size format that adds a field indicating the size of the NAL unit. The byte stream format is used in an MPEG-2 system, RTP, etc., and the NAL size format is used in MP4, DASH, MMT, etc. using MP4.

In the byte stream format, the start code is composed of 3 bytes, and any byte (byte whose value is 0) can be added.

On the other hand, in the general MP4 NAL size format, the size information is indicated by either 1 byte, 2 bytes, and 4 bytes. This size information is indicated by the lengthSizeMinusOne field in the HEVC sample entry. When the value of the field is "0", it is 1 byte, when it is "1", it is 2 bytes, and when it is "3", it is 4 bytes.

Here, in ARIB STD-B60 "Media transport method by MMT in digital broadcasting" standardized in July 2014, when the NAL unit is stored in the multiplexing layer, the output of the HEVC encoder is a byte stream. In this case, the byte start code is removed, and the size of the NAL unit in byte units indicated by 32 bits (unsigned integer) is added immediately before the NAL unit as length information. The MPU metadata including the HEVC sample entry is not transmitted, and the size information is fixed to 32 bits (4 bytes).

Further, in ARIB STD-B60 "Media transport method by MMT in digital broadcasting", in the receive buffer model considered at the time of transmission in order to guarantee the buffer operation in the receiving device, the buffer before decoding the video signal. Is defined as CPB.

However, there are the following problems. CPB in the MPEG-2 system and HRD in HEVC are defined on the premise that the video signal is in the byte stream format. Therefore, for example, when the rate of the transmission packet is controlled on the premise that the transmission packet is in the byte stream format with the start code of 3 bytes, the transmission packet in the NAL size format to which the size area of 4 bytes is added is received. The receiving device may not be able to meet the receive buffer model in ARIB STD-B60. Further, since the receive buffer model in ARIB STD-B60 does not show a specific buffer size and extraction rate, it is difficult to guarantee the buffer operation in the receiving device.

Therefore, in order to solve the above problems, the receive buffer model for guaranteeing the buffer operation in the receiver is defined as follows.

FIG. 68 shows a receive buffer model based on the receive buffer model defined in ARIB STD-B60, particularly when only a broadcast transmission line is used.

The receive buffer model includes a TLV packet buffer (first buffer), an IP packet buffer (second buffer), an MMTP buffer (third buffer), and a pre-decoding buffer (fourth buffer). Note that the broadcast transmission line does not require a de-jitter buffer or a buffer for FEC, so it is omitted.

The TLV packet buffer receives a TLV packet (transmission packet) from a broadcast transmission path, and has a variable length packet header (IP packet header, a full header during IP packet compression, and an IP packet compression) stored in the received TLV packet. An IP packet composed of a compressed header) and a variable-length payload is converted into an IP packet (first packet) having a fixed-length IP packet header decompressed by the header, and the IP packet obtained by the conversion. Is output at a constant bit rate.

The IP packet buffer converts an IP packet into an MMTP packet (second packet) having a packet header and a variable length payload, and outputs the MMTP packet obtained by the conversion at a constant bit rate. The IP packet buffer may be merged with the MMTP buffer.

The MMTP buffer converts the output MMTP packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bit rate.

The pre-decoding buffer sequentially accumulates the output NAL units, generates an access unit from the accumulated plurality of NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit.

In the receive buffer model shown in FIG. 68, the MMTP buffer and the pre-decoding buffer, which are buffers other than the TLV packet buffer and the IP packet buffer in the previous stage, are characteristic of following the receive buffer model in the MPEG-2 TS.

For example, the MMTP buffer (MMTP B1) in the video is composed of a transport buffer (TB) in the MPEG-2 TS and a buffer corresponding to the multiplexing buffer (MB). Further, the MMTP buffer (MMTP Bn) in voice is composed of a buffer corresponding to the transport buffer (TB) in MPEG-2 TS.

The buffer size of the transport buffer is the same as that of MPEG-2 TS and is a fixed value. For example, it is n times the MTU size (n may be a decimal number or an integer, and is 1 or more).

Further, the MMTP packet size is specified so that the overhead rate of the MMTP packet header is smaller than the overhead rate of the PES packet header. Thereby, as the extraction rate from the transport buffer, the extraction rates RX1, RXn, and RXs of the transport buffer in the MPEG-2 TS can be applied as they are.

The size of the multiplexing buffer and the extraction rate are MPEG-2, respectively.
MB size in TS and RBX1.

In addition to the above receive buffer model, the following restrictions are set in order to solve the problem.

The HRD regulation of HEVC is premised on a byte stream format, and MMT is a NAL size format in which a 4-byte size area is added to the beginning of a NAL unit. Therefore, at the time of coding, rate control is performed so as to satisfy HRD in the NAL size format.

That is, the transmitting device controls the rate of the transmission packet based on the above-mentioned reception buffer model and constraints.

In the receiving device, by performing reception processing using the above signal, it is possible to perform a decoding operation that causes underflow or overflow.

Even if the size area at the beginning of the NAL unit is not 4 bytes, the rate is controlled so as to satisfy the HRD in consideration of the size area at the beginning of the NAL unit.

The extraction rate of the TLV packet buffer (bit rate when the TLV packet buffer outputs an IP packet) is set in consideration of the transmission rate after expanding the IP header.

That is, after inputting a TLV packet having a variable data size, removing the TLV header and decompressing (restoring) the IP header, the transmission rate of the output IP packet is taken into consideration. In other words, the amount of increase / decrease in the header is taken into consideration with respect to the input transmission rate.

Specifically, because the data size is variable length, packets with IP header compression and packets without IP header compression coexist, and the size of the IP header differs depending on the packet type such as IPv4 and IPv6. , The transmission rate of the output IP packet is not unique. Therefore, the average packet length of the variable length data size is determined, and the transmission rate of the IP packet output from the TLV packet is determined.

Here, in order to specify the maximum transmission speed after decompressing the IP header, the transmission rate is determined on the assumption that the IP header is always compressed.

Further, when the packet types of IPv4 and IPv6 are mixed, or when the packet types are specified without distinction, the transmission rate is determined assuming an IPv6 packet having a large header size and a large increase rate after header expansion. ..

For example, when the average packet length of the TLV packet input to the TLV packet buffer is S, all the IP packets stored in the TLV packet are IPv6 packets, and the header is compressed, the removal of the TLV header and the removal of the TLV header. The maximum output transmission rate after decompression of the IP header is
Input rate x {S / (S + IPv6 header compression amount)}
Will be.

More specifically, the average packet length S of the TLV packet is
S = 0.75 x 1500 (1500 assumes maximum MTU size)
Set as a reference,
IPv6 header compression amount = TLV header length-IPv6 header length-UDP header length
= 3-40-8
If, the maximum output transmission rate after removing the TLV header and expanding the IP header is
Input rate x 1.0417 ≒ Input rate x 1.05
Will be.

FIG. 69 is a diagram showing an example in which a plurality of data units are aggregated and stored in one payload.

In the MMT method, when the data unit is aggregated, the data unit length and the data unit header are added before the data unit, as shown in FIG. 69.

However, for example, when a video signal in the NAL size format is stored as one data unit, as shown in FIG. 70, one data unit has two fields indicating the size, which are duplicated as information. .. FIG. 70 is an example in which a plurality of data units are aggregated and stored in one payload, and is a diagram showing an example in which a video signal in NAL size format is used as one data unit. Specifically, both the size of the first size area in the NAL size format (referred to as "size area" in the following description) and the data unit length field located before the data unit header in the MMTP payload header are both sizes. It is a field indicating, and is duplicated as information. For example, when the length of the NAL unit is L bytes, the size area shows L bytes, and the data unit length field shows L bytes + “size area length” (byte). Although the values indicated by the size area and the data unit length field do not completely match, it can be said that they overlap because the other value can be easily calculated from one value.

In this way, when the data including the size information of the data is stored as a data unit and a plurality of the data units are aggregated and stored in one payload, the size information is duplicated, so that the overhead is large. There is a problem that the transmission efficiency is poor.

Therefore, in the transmission device, when the data including the size information of the data is stored as a data unit and a plurality of the data units are aggregated and stored in one payload, as shown in FIGS. 71 and 72. It is conceivable to store it.

As shown in FIG. 71, it is conceivable that the NAL unit including the size area is stored as a data unit, and the data unit length conventionally included in the MMTP payload header is not shown. FIG. 71 is a diagram showing the configuration of the payload of the MMTP packet whose data unit length is not shown.

Further, as shown in FIG. 72, a flag indicating whether or not the data unit length is indicated and information indicating the length of the size area may be newly stored in the header. The place where the information indicating the length of the flag or the size area is stored may be indicated in units of data units such as a data unit header, or may be indicated in units of aggregation of a plurality of data units (packet unit). FIG. 72 is an example shown in the extend area assigned to each packet. The storage location of the newly shown information is not limited to this, and may be an MMTP payload header, an MMTP packet header, or control information.

On the receiving side, if the flag indicating whether or not the data unit length is compressed indicates that the data unit length is compressed, the length information of the size area inside the data unit is acquired and the size area is acquired. By acquiring the size area based on the length information of, the data unit length can be calculated using the acquired length information of the size area and the size area.

By the above method, the amount of data can be reduced on the transmitting side and the transmission efficiency can be improved.

It should be noted that the overhead may be reduced by reducing the size area instead of reducing the data unit length. When the size area is reduced, information indicating whether or not the size area is reduced or information indicating the length of the data unit length field may be stored.

The length information is also included in the MMTP payload header.

When the NAL unit including the size area is stored as a data unit, the payload size area in the MMTP payload header may be reduced regardless of whether it is aggregated or not.

Further, even when the data not including the size area is stored as a data unit, the payload size area in the MMTP payload header may be reduced if it is aggregated and the data unit length is indicated.

When the payload size area is reduced, the flag indicating whether or not the payload size area is reduced, the length information of the reduced size field, or the length information of the size field not reduced may be shown in the same manner as described above.

FIG. 73 shows the operation flow of the receiving device.

As described above, the transmitting device stores the NAL unit including the size area as a data unit, and the data unit length included in the MMTP payload header is not shown in the MMTP packet.

In the following, a flag indicating whether or not the data unit length is indicated and a case where the length information of the size area is indicated in the MMTP packet will be described as an example.

The receiving device determines whether or not the data unit includes a size area and the data unit length is reduced based on the information transmitted from the transmitting side (S1901).

When it is determined that the data unit length has been reduced (Yes in S1902), the length information of the size area inside the data unit is acquired, and then the size area inside the data unit is analyzed to calculate the data unit length. Acquired by (S1903).

On the other hand, when it is determined that the data unit length has not been reduced (No in S1902), the data unit length is calculated from either the data unit length or the size area inside the data unit as usual (S1904).

If the receiving device is known in advance, it is not necessary to transmit the flag indicating whether or not the data unit length is reduced and the length information of the size area. In this case, the receiving device performs the processing shown in FIG. 73 based on the predetermined information.

[Supplement: Transmitter and receiver]
As described above, the transmission device that performs rate control so as to satisfy the definition of the receive buffer model at the time of coding can also be configured as shown in FIG. 74. Further, the receiving device that receives and decodes the transmission packet transmitted from the transmitting device can also be configured as shown in FIG. 75. FIG. 74 is a diagram showing an example of a specific configuration of the transmission device. FIG. 75 is a diagram showing an example of a specific configuration of the receiving device.

The transmission device 700 includes a generation unit 701 and a transmission unit 702. Each of the generation unit 701 and the transmission unit 702 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving device 800 includes a receiving unit 801, a first buffer 802, a second buffer 803, a third buffer 804, a fourth buffer 805, and a decoding unit 806. Each of the receiving unit 801, the first buffer 802, the second buffer 803, the third buffer 804, the fourth buffer 805, and the decoding unit (decoder) 806 is, for example, a microcomputer, a processor, a dedicated circuit, or the like. Realized by.

A detailed description of each component of the transmitting device 700 and the receiving device 800 will be given in the description of the transmitting method and the receiving method, respectively.

First, the transmission method will be described with reference to FIG. 76. FIG. 76 is an operation flow (transmission method) by the transmission device.

First, the generation unit 701 of the transmission device 700 generates a coded stream by performing rate control so as to satisfy a predetermined specification by the reception buffer model in order to guarantee the buffer operation of the reception device (S2001).

Next, the transmission unit 702 of the transmission device 700 packetizes the generated coded stream and transmits the transmission packet obtained by packetizing (S2002).

Since the reception buffer model used in the transmission device 700 includes the first to fourth buffers 802 to 805 of the configuration of the reception device 800, the description thereof will be omitted.

Thereby, the transmitting device 700 can guarantee the buffer operation of the receiving device 800 when data is transmitted by using a method such as MMT.

Next, the receiving method will be described with reference to FIG. 77. FIG. 77 is an operation flow (reception method) by the receiving device.

First, the receiving unit 801 of the receiving device 800 receives a transmission packet composed of a fixed-length packet header and a variable-length payload (S2101).

Next, the first buffer 802 of the receiving device 800 is a fixed-length packet header in which a packet composed of a variable-length packet header and a variable-length payload stored in the received transmission packet is decompressed. It is converted into the first packet to have, and the first packet obtained by the conversion is output at a constant bit rate (S2102).

Next, the second buffer 803 of the receiving device 800 is obtained by converting the first packet obtained by conversion into a second packet composed of a packet header and a variable length payload, and converting the first packet. The second packet is output at a constant bit rate (S2103).

Next, the third buffer 804 of the receiving device 800 converts the output second packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bit rate (S2104).

Next, the fourth buffer 805 of the receiving device 800 sequentially accumulates the output NAL units, generates an access unit from the accumulated plurality of NAL units, and decodes the generated access unit corresponding to the access unit. It is output to the decoder at the timing of (S2105).

Then, the decoding unit 806 of the receiving device 800 decodes the access unit output by the fourth buffer (S2106).

As a result, the receiving device 800 can perform a decoding operation that causes underflow or overflow.

(Embodiment 6)
[Overview]
In the sixth embodiment, a transmission method and a reception method when leap second adjustment is performed on the time information that is the reference of the reference clock in the transmission method of MMT / TLV will be described.

FIG. 78 is a diagram showing an MMT / TLV protocol stack defined in ARIB STD-B60.

In the MMT method, data such as video and audio are input to multiple MPUs (Media) in packets.
It is stored in each first data unit such as Presentation Unit) or MFU (Media Fragment Unit), and an MMTP packet header is added to generate an MMTP packet as a predetermined packet (convert it into an MMTP packet). Further, by adding an MMTP packet header to control information such as a control message in MMTP, an MMTP packet as a predetermined packet is generated. The MMTP packet header is provided with a field for storing a 32-bit short format NTP (Network Time Protocol: specified in IETF RFC 5905), and can be used for QoS control of a communication line or the like.

Further, the reference clock of the transmitting side (transmitting device) is synchronized with the 64-bit long format NTP specified in RFC 5905, and based on the synchronized reference clock, PTS (Presentation Time Stamp) and DTS (Decode Time) are used. A time stamp such as Stamp) is added to the synchronized media. Further, the reference clock information on the transmitting side is transmitted to the receiving side, and the receiving device generates a system clock in the receiving device based on the reference clock information received from the transmitting side.

Specifically, PTS and DTS are stored in the MPU time stamp descriptor which is the control information of MMTP and the MPU extended time stamp descriptor, are stored in the MP table for each asset, and are converted into MMTP packets as control messages. After that, it is transmitted.

The MMTP packetized data is added with a UDP header and an IP header and encapsulated in an IP packet. At this time, in the IP header and UDP header, a set of packets having the same source IP address, destination IP address, source port number, destination port number, and protocol type is defined as an IP data flow. Since the header of the IP packet of the same IP data flow is redundant, the header is compressed in some IP packets.

Further, as the reference clock information, the 64-bit NTP time stamp is stored in the NTP packet and is stored in the IP packet. At this time, in the IP packet for storing the NTP packet, the source IP address, the destination IP address, the source port number, the destination port number, and the protocol type are fixed values, and the header of the IP packet is not compressed.

FIG. 79 is a diagram showing a configuration of a TLV packet.

As shown in FIG. 79, the TLV packet can include transmission control information such as an IP packet, a compressed IP packet, an AMT (Addless Map Table) and a NIT (Network Information Table) as data, and these data can be used as data. It is identified using an 8-bit data type. Further, in the TLV packet, the data length (in bytes) is indicated by using a 16-bit field, and the data value is stored after that. Further, the TLV packet has 1 byte of header information before the data type, and the header information is stored in a total of 4 bytes of header area. Further, the TLV packet is mapped to the transmission slot in the advanced BS transmission method, and the mapping information is stored in the TMCC (Transmission and Multiplexing Control) control information.

FIG. 80 is a diagram showing an example of a block diagram of the receiving device.

In the receiving device, first, the broadcast signal received by the tuner is decoded and error-corrected by the demodulation means in the transmission path coded data, and the TLV packet is extracted. Then, the TLV / IP DEMUX means performs TLV DEMUX processing and IP DEMUX processing. The TLV DEMUX processing is performed according to the data type of the TLV packet. For example, if the TLV packet has a compressed IP packet, the compressed header of the compressed IP packet is restored. In IP DEMUX, processing such as header analysis of IP packets and UDP packets is performed, and MMTP packets and NTP packets are extracted.

The NTP clock generation means reproduces the NTP clock from the extracted NTP packet. In MMTP DEMUX, filtering processing of components such as video and audio and control information is performed based on the packet ID stored in the extracted MMTP packet header. From the control information acquisition means, the time stamp descriptor stored in the MP table is acquired, and the PTS / DTS calculation means calculates the PTS and DTS for each access unit. The time stamp descriptor includes both the MPU time stamp descriptor and the MPU extended time stamp descriptor.

The access unit reproduction means converts the filtered video, audio, etc. from the MMTP packet into the data of the unit to be presented. Specifically, the data of the unit to be presented is a NAL unit or an access unit of a video signal, an audio frame, a presentation unit of subtitles, or the like. The decoding / presenting means decodes and presents the access unit at the time when the PTS / DTS of the access unit matches, based on the reference time information of the NTP clock.

The configuration of the receiving device is not limited to this.

Next, the time stamp descriptor will be described.

FIG. 81 is a diagram for explaining a time stamp descriptor.

PTS and DTS are stored in the MPU time stamp descriptor as the first control information and the MPU extended time stamp descriptor as the second control information, which are MMT control information, and are stored in the MP table for each asset for control. It is transmitted after being converted into an MMTP packet as a message.

FIG. 81 (a) is a diagram showing the structure of the MPU time stamp descriptor defined in ARIB STD-B60. In the MPU time stamp descriptor, for each of the plurality of MPUs, the first (first) AU in the presentation order of the plurality of access units (AUs) as the second data unit stored in the MPU (hereinafter referred to as the AU). , The presentation time information (first time information) indicating the PTS (absolute value indicated by 64-bit NTP) of the "head AU") is stored. That is, the presentation time information of the MPU given to the MPU is stored in the control information of the MMTP packet and transmitted.

FIG. 81 (b) shows the structure of the MPU extended time stamp descriptor. The MPU extended time stamp descriptor stores information for calculating the PTS and DTS of the AU contained in each MPU of the plurality of MPUs. The MPU extended time stamp descriptor contains relative information (second time information) from the PTS of the first AU of the MPU stored in the MPU time stamp descriptor, and the PTS of each of the plurality of AUs contained in the MPU and the PTS of each of the plurality of AUs contained in the MPU. The DTS can be calculated based on both the MPU timestamp descriptor and the MPU extended timestamp descriptor. That is, the PTS and DTS of the AUs other than the first AU included in the MPU are based on the PTS of the first AU stored in the MPU time stamp descriptor and the relative information stored in the MPU extended time stamp descriptor. Can be calculated.

In other words, the second time information is relative time information for calculating the PTS or DTS of each of the plurality of AUs together with the first time information. That is, the second time information is information indicating the PTS or DTS of each of the plurality of AUs together with the first time information.

NTP is reference time information based on Coordinated Universal Time (UTC). UTC adjusts leap seconds (hereinafter referred to as "leap second adjustment") in order to adjust the difference from astronomical time based on the rotation speed of the earth. The leap second adjustment is specifically an adjustment performed at 9:00 am Japan time to insert or delete one second.

FIG. 82 is a diagram for explaining leap second adjustment.

FIG. 82 (a) is a diagram showing an example of leap second insertion in Japan time. As shown in (a) of FIG. 82, in the leap second insertion, after 8:59:59 Japan time, it becomes 8:59:59 at the timing when it originally becomes 9:00:59, and 8:59:59 units. Repeated twice.

FIG. 82 (b) is a diagram showing an example of leap second deletion in Japan time. As shown in FIG. 82 (b), in the leap second deletion, after 8:59:58 Japan time, it becomes 9:00 at the timing when it is originally 8:59:59, and 8:59:59 units. 1 second is deleted.

In addition to the 64-bit time stamp, a 2-bit leap_indicator is stored in the NTP packet. The leap_indicator is a flag for notifying in advance that a leap second adjustment is performed, and indicates a leap second insertion when leap_indicator = 1 and a leap second deletion when leap_indicator = 2. Advance notification includes a method of notifying from the beginning of the month in which leap second adjustment is performed, a method of notifying 24 hours in advance, and a method of starting notification at any time. Further, leap_indicator becomes 0 at the time (9:00:00) when the leap second adjustment is completed. For example, in the case of prior notice from 24 hours before, the time immediately before the leap second adjustment is performed on the day when the leap second adjustment is performed from 9:00 on the day before the leap second adjustment is performed in Japan time (that is, the leap second). In the case of insertion, it is the time of the first 8:59:59, and in the case of leap second deletion, the time of 8:59:58), the leap_indicator may be "1" or "2". Shown.

Next, the problems at the time of leap second adjustment will be described.

FIG. 83 is a diagram showing the relationship between the NTP time, the MPU time stamp, and the MPU presentation timing. The NTP time is a time indicated by NTP. The MPU time stamp is a time stamp indicating the PTS of the first AU in the MPU. The MPU presentation timing is a timing at which the receiving device should present the MPU according to the MPU time stamp. Specifically, (a) to (c) of FIG. 83 show the NTP time, the MPU time stamp, and the case where the leap second is inserted, the leap second is deleted, and the leap second is deleted, respectively. It is a figure which shows the relationship of the MPU presentation time.

Here, the case where the NTP time (reference clock) on the sending side is synchronized with the NTP server and the NTP time (system clock) on the receiving side is synchronized with the NTP time on the sending side will be described as an example. do. In this case, the receiving device reproduces based on the time stamp stored in the NTP packet transmitted from the transmitting side. Further, in this case, since both the NTP time on the sending side and the NTP time on the receiving side are synchronized with the NTP server, the leap second is adjusted by ± 1 second. Further, it is assumed that the NTP time in FIG. 83 is common to the NTP time on the sending side and the NTP time on the receiving side. In addition, it is assumed that there is no transmission delay.

The MPU time stamp in FIG. 83 indicates the time stamp of the first AU in the presentation order among the plurality of AUs included in each of the plurality of MPUs, and is generated (set) based on the NTP time indicated by the arrow. Will be done. Specifically, the MPU is presented by adding a predetermined time (for example, 1.5 seconds in FIG. 83) to the NTP time as the reference time information at the timing of generating the MPU presentation time information. Generate time information. The generated MPU timestamp is stored in the MPU timestamp descriptor.

The receiving device presents the MPU at the MPU presentation time based on the time stamp stored in the MPU time stamp descriptor.

In FIG. 83, the reproduction time of one MPU is described as 1 second, but the reproduction time of one MPU may be other reproduction time, for example, 0.5 second. , 0.1 seconds may be used.

In the example of (a) of FIG. 83, the receiving device can sequentially present MPU # 1- # 5 based on the time stamp stored in the MPU time stamp descriptor.

However, in FIG. 83 (b), the presentation times of MPU # 2 and MPU # 3 overlap due to the leap second insertion. Therefore, if the receiving device presents the MPU based on the time stamp stored in the MPU time stamp descriptor, there are two MPUs to be presented in the same time zone of 9:00:00. Therefore, it is not possible to determine which of the two MPUs should be presented. Further, since the receiving device has the MPU presentation time (8:59:59) indicated by the time stamp of MPU # 1 twice, which MPU presentation time should be presented among the two MPU presentation times? I can't judge.

Further, in FIG. 83 (c), the receiving device does not have the MPU presentation time (8:59:59) indicated by the MPU time stamp of MPU # 3 at the NTP time due to the deletion of the leap second, so that the MPU # 3 cannot be presented.

When the receiving device performs the decoding process and the presentation process of the MPU that is not based on the time stamp, the above-mentioned problems can be solved. However, it is difficult for a receiving device that performs processing based on a time stamp to perform different processing (processing that is not based on a time stamp) only when a leap second occurs.

Next, a method of solving the problem of leap second adjustment by correcting the time stamp on the transmitting side will be described.

FIG. 84 is a diagram for explaining a correction method for correcting the time stamp on the transmitting side. Specifically, FIG. 84 (a) is a diagram showing an example of leap second insertion, and FIG. 84 (b) is a diagram showing an example of leap second deletion.

First, the case of leap second insertion will be described.

As shown in FIG. 84 (a), when the leap second is inserted, the time immediately before the leap second insertion (that is, up to the first 8:59:59 unit in the NTP time) is set as the A region, and after the leap second insertion. (That is, after the second 8:59:59 in the NTP time) is set as the B region. The area A and the area B are temporal areas, and are time zones or periods. The MPU time stamp in FIG. 84 (a) is the same as the time stamp described in FIG. 83, and is a time stamp generated (set) based on the NTP time at the timing of adding the MPU time stamp.

In the case of leap second insertion, the method of correcting the time stamp on the transmitting side will be specifically described.

The transmitting side (transmitting device) performs the following processing.

1. 1. The timing for adding the MPU time stamp (the timing indicated by the source of the arrow in FIG. 84 (a)) is included in the A area, and the MPU time stamp (value of the MPU time stamp before correction) is 9:00:00. When indicating the following, the MPU time stamp is decremented by 1 second, corrected by -1 second, and the corrected MPU time stamp is stored in the MPU time stamp descriptor. That is, if the MPU time stamp is generated based on the NTP time included in the A region and the MPU time stamp indicates after 9:00, the MPU time stamp is corrected by -1 second. In addition, "9:00:00" here is the time corresponding to Japan time, which is the reference time for leap second adjustment (that is, the time calculated by adding 9 hours to UTC time). Is. In addition, correction information, which is information indicating that the correction has been made separately, is transmitted to the receiving device.

2. 2. When the timing of adding the MPU time stamp (the timing indicated by the source of the arrow in FIG. 84 (a)) is the B region, the MPU time stamp is not corrected. That is, if the MPU time stamp is generated based on the NTP time included in the B area, the MPU time stamp is not corrected.

In the receiving device, the MPU is presented based on the MPU time stamp and the correction information indicating whether or not the MPU time stamp has been corrected (that is, whether or not information indicating that the correction has been made is included).

If the receiving device determines that the MPU time stamp has not been corrected (that is, if it determines that it does not contain information indicating that it has been corrected), the time stamp stored in the MPU time stamp descriptor is the receiving device. The MPU is presented at the time that matches the NTP time (including both before and after the correction). That is, in the case of the MPU time stamp stored in the MPU time stamp descriptor of the MPU transmitted before the MPU whose MPU time stamp is corrected, the MPU time stamp is before the leap second insertion (that is, the first 8). : 59: 59 or earlier) The MPU is presented at the timing that matches the NTP time. If the MPU time stamp stored in the MPU time stamp descriptor of the received MPU is a time stamp transmitted after the corrected MPU time stamp, after the leap second is inserted (that is, the second 8:59: The MPU is presented at the timing that matches the NTP time of 59 units or later).

Further, when the MPU time stamp stored in the MPU time stamp descriptor of the received MPU is corrected, the receiving device inserts the time stamp stored in the MPU time stamp descriptor into a leap second in the receiving device ( That is, the MPU is presented based on the NTP time (after the second 8:59:59 unit).

Information indicating that the MPU time stamp value has been corrected is stored and transmitted in a control message, a descriptor, a table, MPU metadata, MF metadata, an MMTP packet header, or the like.

Next, the case of leap second deletion will be described.

As shown in FIG. 84 (b), when the leap second is deleted, the time immediately before the leap second deletion (that is, immediately before 9:00 in the NTP time) is set as the C area, and the time after the leap second deletion (that is, that is). , After 9:00 in NTP time) is set as the D area. The C region and the D region are temporal regions, and are time zones or periods. The MPU time stamp in FIG. 84 (b) is the same as the time stamp described in FIG. 83, and is a time stamp generated (set) based on the NTP time at the timing of adding the MPU time stamp.

In the case of leap second deletion, the method of correcting the time stamp on the transmitting side will be specifically described.

The transmitting side (transmitting device) performs the following processing.

1. 1. The timing for adding the MPU time stamp (the timing indicated by the source of the arrow in FIG. 84 (b)) is included in the C region, and the MPU time stamp (value of the MPU time stamp before correction) is 8:59:59. When indicating the following, the MPU time stamp is added by 1 second, corrected by +1 second, and the corrected MPU time stamp is stored in the MPU time stamp descriptor. That is, if the MPU time stamp is generated based on the NTP time included in the C region and the MPU time stamp indicates after 8:59:59, the MPU time stamp is corrected by +1 second. In addition, "8:59:59" here is the time corresponding to the Japanese time as the reference time for leap second adjustment (that is, the time calculated by adding 9 hours to the UTC time). It is the time obtained by subtracting -1 second from. In addition, correction information, which is information indicating that the correction has been made separately, is transmitted to the receiving device. In this case, the correction information does not necessarily have to be transmitted.

2. 2. When the timing of adding the MPU time stamp (the timing indicated by the source of the arrow in FIG. 84 (b)) is the D region, the MPU time stamp is not corrected. That is, if the MPU time stamp is generated based on the NTP time included in the D area, the MPU time stamp is not corrected.

In the receiving device, the MPU is presented based on the MPU time stamp. If there is correction information indicating whether or not the MPU time stamp has been corrected, the MPU may be presented based on the MPU time stamp and the correction information.

By the above processing, even when the leap second is adjusted to the NTP time, the receiving device can present a normal MPU using the MPU time stamp stored in the MPU time stamp descriptor. ..

It should be noted that the receiving side may be notified by signaling whether the timing of adding the MPU time stamp is the A region, the B region, the C region, or the D region. That is, the MPU time stamp is generated based on the NTP time included in the A region, the NTP time included in the B region, or the NTP time included in the C region. It may be signaled whether it is generated based on the NTP time included in the D area and notified to the receiving side. In other words, identification information indicating whether or not the MPU time stamp (presentation time) is generated based on the reference time information (NTP time) before leap second adjustment may be transmitted. Since this identification information is given based on the leap_indicator included in the NTP packet, it is immediately before the leap second adjustment is performed on the day when the leap second adjustment is performed from 9:00 on the day before the leap second adjustment is performed in Japan time. It is set based on the time up to the time (that is, the time of the first 8:59:59 in the case of leap second insertion, and the time of 8:59:58 in the case of leap second deletion). Information indicating whether or not it is an MPU time stamp. That is, was the identification information generated based on the NTP time from the time before the MPU time stamp to the time immediately before the leap second adjustment by a predetermined period (for example, 24 hours) to the time immediately before the time? It is information indicating whether or not.

Next, a method of solving the problem of leap second adjustment by correcting the time stamp in the receiving device will be described.

FIG. 85 is a diagram for explaining a correction method for correcting a time stamp in a receiving device. Specifically, FIG. 85 (a) is a diagram showing an example of leap second insertion, and FIG. 85 (b) is a diagram showing an example of leap second deletion.

First, the case of leap second insertion will be described.

As shown in FIG. 85 (a), at the time of leap second insertion as in FIG. 84 (a), the time immediately before the leap second insertion (that is, until the first 8:59:59 unit in the NTP time). Is defined as the A region, and the time after the leap second insertion (that is, after the second 8:59:59 in the NTP time) is defined as the B region. The area A and the area B are temporal areas, and are time zones or periods. The MPU time stamp in FIG. 85 (a) is the same as the time stamp described in FIG. 83, and is a time stamp generated (set) based on the NTP time at the timing of adding the MPU time stamp.

In the case of leap second insertion, the method of correcting the time stamp in the receiving device will be specifically described.

The transmitting side (transmitting device) performs the following processing.

-The MPU time stamp is stored in the MPU time stamp descriptor and transmitted to the receiving device without correcting the generated MPU time stamp.

-Information indicating whether the timing of adding the time stamp of the MPU is the A region or the B region is transmitted to the receiving device as identification information. That is, the identification information indicating whether the MPU time stamp is generated based on the NTP time included in the A region or the NTP time included in the B region is transmitted to the receiving device.

The receiving device performs the following processing.

In the receiving device, the MPU time stamp is corrected based on the MPU time stamp and the identification information indicating whether the timing at which the MPU time stamp is applied is the A region or the B region.

Specifically, the following processing is performed.

1. 1. The timing for adding the MPU time stamp (the timing indicated by the source of the arrow in FIG. 85 (a)) is included in the A area, and the MPU time stamp (value of the MPU time stamp before correction) is 9:00:00. When indicating the following, the MPU time stamp is decremented by 1 second and corrected by -1 second. That is, if the MPU time stamp is generated based on the NTP time included in the A region and the MPU time stamp indicates after 9:00, the MPU time stamp is corrected by -1 second. In addition, "9:00:00" here is the time corresponding to Japan time, which is the reference time for leap second adjustment (that is, the time calculated by adding 9 hours to UTC time). Is.

2. 2. When the timing of adding the MPU time stamp (the timing indicated by the source of the arrow in FIG. 85 (a)) is the B region, the MPU time stamp is not corrected. That is, if the MPU time stamp is generated based on the NTP time included in the B area, the MPU time stamp is not corrected.

When the MPU time stamp is not corrected, the MPU is presented at the time when the MPU time stamp stored in the MPU time stamp descriptor matches the NTP time (including before and after correction) of the receiving device.

That is, in the case of the MPU time stamp received before the MPU time stamp to be corrected, the MPU is presented at the time corresponding to the NTP time before the leap second insertion (that is, before the first 8:59:59 unit). Further, in the case of the MPU time stamp received after the MPU time stamp to be corrected, the MPU is presented at the time corresponding to the NTP time after the leap second is inserted (after the second 8:59:59 unit).

When correcting the MPU time stamp, the corrected MPU time stamp presents the MPU based on the NTP time after the leap second is inserted in the receiving device (that is, after the second 8:59:59 unit).

On the transmitting side, the identification information indicating whether the timing at which the MPU time stamp is added is in the A area or the B area includes control messages, descriptors, tables, MPU metadata, MF metadata, and MMTP packet headers. Store in such as and transmit.

Next, the case of leap second deletion will be described.

As shown in FIG. 85 (b), when the leap second is deleted, the time immediately before the leap second deletion (that is, until just before 9:00 in the NTP time) is C as in the case of FIG. 84 (a). The area is defined as the time after the leap second is deleted (that is, after 9:00:00 in the NTP time) as the area D. The C region and the D region are temporal regions, and are time zones or periods. The MPU time stamp in FIG. 85 (b) is the same as the time stamp described in FIG. 83, and is a time stamp generated (set) based on the NTP time at the timing of adding the MPU time stamp.

In the case of leap second deletion, the method of correcting the time stamp in the receiving device will be specifically described.

The transmitting side (transmitting device) performs the following processing.

-The MPU time stamp is stored in the MPU time stamp descriptor and transmitted to the receiving device without correcting the generated MPU time stamp.

-Information indicating whether the timing of adding the time stamp of the MPU is the C region or the D region is transmitted to the receiving device as identification information. That is, the identification information indicating whether the MPU time stamp is generated based on the NTP time included in the C region or the NTP time included in the B region is transmitted to the receiving device.

The receiving device performs the following processing.

In the receiving device, the MPU time stamp is corrected based on the MPU time stamp and the identification information indicating whether the timing at which the MPU time stamp is added is the C region or the D region.

Specifically, the following processing is performed.

1. 1. The timing for adding the MPU time stamp (the timing indicated by the source of the arrow in FIG. 85 (b)) is included in the C region, and the MPU time stamp (value of the MPU time stamp before correction) is 8:59:59. When indicating the following, the MPU time stamp value is added by 1 second and corrected by +1 second. That is, if the MPU time stamp is generated based on the NTP time included in the C region and the MPU time stamp indicates after 8:59:59, the MPU time stamp is corrected by +1 second. In addition, "8:59:59" here is the time corresponding to the Japanese time as the reference time for leap second adjustment (that is, the time calculated by adding 9 hours to the UTC time). It is the time obtained by subtracting -1 second from.

2. 2. When the timing of adding the MPU time stamp (the timing indicated by the source of the arrow in FIG. 85 (b)) is the D region, the MPU time stamp is not corrected. That is, if the MPU time stamp is generated based on the NTP time included in the D area, the MPU time stamp is not corrected.

In the receiving device, the MPU is presented based on the MPU time stamp and the corrected MPU time stamp.

By the above processing, even when the leap second is adjusted to the NTP time, the receiving device can present a normal MPU using the MPU time stamp stored in the MPU time stamp descriptor. ..

Even in this case, the timing of adding the MPU time stamp is in the A region, the B region, or the C region, as in the case of correcting the MPU time stamp on the transmitting side described with reference to FIG. 84. It may be signaled whether it is in the D area or in the D area, and the receiving side may be notified. Since the details of the notification are the same, the description thereof will be omitted.

It should be noted that the information indicating whether or not the MPU time stamp has been corrected and whether the timing at which the MPU time stamp is applied is in the A region, the B region, or the C region, which are described with reference to FIGS. 84 and 85. , Additional information (identification information) such as information indicating whether the NTP packet is in the D region may be enabled when the leap_indicator of the NTP packet indicates the deletion (leap_indicator = 2) or the insertion (leap_indicator = 1) of the leap second. It may be effective from an arbitrary predetermined time (for example, 3 seconds before the leap second adjustment), or may be dynamically enabled.

Further, the timing at which the validity period of the additional information ends and the timing at which the signaling of the additional information ends may be set to the leap_indicator of the NTP packet, or at any predetermined time (for example, 3 seconds before the leap second adjustment). It may be enabled from, or it may be enabled dynamically.

Since the 32-bit time stamp stored in the MMTP packet, the time stamp information stored in the TMCC, and the like are also generated and given based on the NTP time, the same problem occurs. Therefore, in the case of 32-bit time stamps and time stamp information stored in TMCC, the time stamps are corrected by the same method as in FIGS. 84 and 85, and processing based on the time stamps is performed in the receiving device. can do. For example, when storing the additional information for the 32-bit time stamp stored in the MMTP packet, it may be indicated by using the extended area of the MMTP packet header. In this case, it indicates that it is additional information to the extended type of the multi-header type. Further, as for the time when the leap_indicator is set, additional information may be indicated by using some bits of the 32-bit or 64-bit time stamps.

In the example of FIG. 84, the corrected MPU time stamp is stored in the MPU time stamp descriptor, but the MPU time stamp before and after the correction (that is, the uncorrected MPU time stamp and the corrected MPU time stamp). ) May be sent to the receiver. For example, the MPU time stamp descriptor before correction and the MPU time stamp descriptor after correction may be stored in the same MPU time stamp descriptor, or may be stored in two MPU time stamp descriptors, respectively. good. In this case, it is identified whether the MPU time stamp is the uncorrected MPU time stamp or the corrected MPU time stamp based on the arrangement order of the two MPU time stamp descriptors and the description order in the MPU time stamp descriptor. May be good. Further, the MPU time stamp before correction may always be stored in the MPU time stamp descriptor, and the corrected time stamp may be stored in the MPU extended time stamp descriptor when making corrections.

In the present embodiment, the time in Japan time (9:00 am) has been described as an example as the NTP time, but the time is not limited to the time in Japan time. The leap second adjustment is based on UTC time and is corrected worldwide at the same time. The Japanese time is a time advanced by 9 hours with respect to the UTC time, and is represented by a value (+9) with respect to the UTC time. In this way, the time adjusted to a different time may be adopted depending on the time difference according to the place.

As described above, by correcting the time stamp on the transmitting side or the receiving device based on the information indicating the timing at which the time stamp is added, normal reception processing using the time stamp becomes possible.

A receiving device that continues the decoding process sufficiently before the leap second adjustment time may be capable of decoding processing and presentation processing without using a time stamp, but the receiving device selected immediately before the leap second adjustment time is used. , The time stamp cannot be determined, and it may not be possible to present it until after the leap second adjustment is completed. Even in that case, by using the correction method in the present embodiment, reception processing using the time stamp becomes possible, and channel selection becomes possible even immediately before the leap second adjustment time.

FIG. 86 shows the operation flow of the transmitting side (transmitting device) and FIG. 87 shows the operation flow of the receiving device when the MPU time stamp is corrected on the transmitting side (transmitting device) described with reference to FIG. 84.

First, the operation flow of the transmitting side (transmitting device) will be described with reference to FIG. 86.

When the leap second is inserted or deleted, it is determined whether the timing of adding the MPU time stamp is the A region, the B region, the C region, or the D region (S2201). The case where leap seconds are not inserted or deleted is not shown.

Here, the areas A to D are defined as follows.

Area A: Time until just before the leap second insertion (until the first 8:59:59)
Area B: Time after leap second insertion (after the second 8:59:59)
Area C: Time until just before leap second deletion (until 9:00:00)
Area D: Time after leap second deletion (after 9:00:00)

In step S2201, when it is determined that the area is A and the MPU time stamp indicates after 9:00, the MPU time stamp is corrected by -1 second, and the corrected MPU time stamp is used as the MPU time. It is stored in the stamp descriptor (S2202).

Then, the correction information indicating that the MPU time stamp has been corrected is signaled and transmitted to the receiving device (S2203).

If it is determined in step S2201 that the area is C and the MPU time stamp indicates 8:59:59 or later, the MPU time stamp is corrected by +1 second, and the corrected MPU time stamp is used as the MPU. It is stored in the time stamp descriptor (S2205).

Further, when it is determined in step S2201 that the area is the B area or the D area, the MPU time stamp is stored in the MPU time stamp descriptor without being corrected (S2204).

Next, the operation flow of the receiving device will be described with reference to FIG. 87.

It is determined whether or not the MPU time stamp is corrected based on the information signaled from the transmitting side (S2301).

When it is determined that the MPU time stamp has been corrected (Yes in S2301), the MPU is presented based on the MPU time stamp at the NTP time after the leap second adjustment is performed in the receiving device (S2302).

When it is determined that the MPU time stamp is not corrected (No in S2301), the MPU is presented based on the MPU time stamp at the NTP time.

When correcting the MPU time stamp, the MPU corresponding to the MPU time stamp is presented in the section where the leap second is inserted.

On the contrary, when the MPU time stamp is not corrected, the MPU corresponding to the MPU time stamp is not presented in the section where the leap second is inserted, but is presented in the section where the leap second is not inserted.

FIG. 88 shows the operation flow on the transmitting side and FIG. 89 shows the operation flow of the receiving device when the MPU time stamp is corrected in the receiving device described with reference to FIG. 85.

First, the operation flow of the transmitting side (transmitting device) will be described with reference to FIG. 88.

Based on the leap_indicator of the NTP packet, it is determined whether the leap second adjustment (insertion or deletion) is performed (S2401).

When it is determined that the leap second adjustment is performed (Yes in S2401), the timing of assigning the MPU time stamp is determined, the identification information is signaled, and the identification information is transmitted to the receiving device (S2402).

On the other hand, when it is determined that the leap second adjustment is not performed (No in S2041), the process ends with the normal operation.

Next, the operation flow of the receiving device will be described with reference to FIG. 89.

Based on the identification information signaled by the transmitting side (transmitting device), it is determined whether the timing of adding the MPU time stamp is the A region, the B region, the C region, or the D region. (S2501). Here, since the areas A to D are the same as those defined above, the description thereof will be omitted. As in FIG. 87, the case where the leap second is not inserted or deleted is not shown.

In step S2501, when it is determined that the area is A and the MPU time stamp indicates after 9:00, the MPU time stamp is corrected by -1 second (S2502).

In step S2501, when it is determined that the region is C and the MPU time stamp indicates 8:59:59 or later, the MPU time stamp is corrected by +1 second (S2504).

If it is determined in step S2501 that the area is B or D, the MPU time stamp is not corrected (S2503).

The receiving device further presents the MPU based on the corrected MPU time stamp in a process (not shown).

When correcting the MPU time stamp, the MPU corresponding to the MPU time stamp is presented based on the MPU time stamp at the NTP time after the leap second adjustment is performed.

On the contrary, when the MPU time stamp is not corrected, the MPU corresponding to the MPU time stamp is not presented in the section where the leap second is inserted, but is presented in the section where the leap second is not inserted.

In short, the transmitting side (transmitting device) determines the timing of assigning the MPU time stamp corresponding to the MPU for each of the plurality of MPUs. Then, as a result of the determination, when the timing is the time immediately before the leap second insertion and the MPU time stamp indicates 9:00: 00 or later, the MPU time stamp is corrected by -1 second. In addition, the correction information indicating that the MPU time stamp has been corrected is signaled and transmitted to the receiving device. Further, as a result of the determination, if the timing is the time immediately before the leap second deletion and the MPU time stamp indicates after 8:59:59, the MPU time stamp is corrected by +1 second.

Further, when the receiving device indicates that the MPU time stamp is corrected based on whether or not the MPU time stamp indicated by the correction information signaled by the transmitting side (transmitting device) is corrected. , The MPU is presented based on the MPU time stamp at the NTP time after the leap second adjustment has been performed. If the MPU time stamp is not corrected, the MPU is presented based on the MPU time stamp at the NTP time before leap second adjustment.

Further, the transmitting side (transmitting device) determines the timing at which the MPU time stamp corresponding to the MPU is added to each of the plurality of MPUs, and signals the determined result. The receiving device performs the following processing based on the information signaled by the transmitting side indicating the timing of adding the MPU time stamp. Specifically, when the information indicating the timing indicates the time immediately before the leap second insertion and the MPU time stamp indicates after 9:00:00, the MPU time stamp is set to-. Correct for 1 second. If the information indicating the timing is the time immediately before the leap second deletion and the MPU time stamp indicates after 8:59:59, the MPU time stamp is corrected by +1 second.

When the MPU time stamp is corrected, the receiving device presents the MPU based on the MPU time stamp at the NTP time after the leap second adjustment is performed. If the MPU time stamp is not corrected, the MPU is presented based on the MPU time stamp at the NTP time before leap second adjustment.

In this way, by determining the timing of assigning the MPU time stamp in the leap second adjustment, the MPU time stamp can be corrected, the receiving device can determine which MPU should be presented, and the MPU time stamp description. Appropriate reception processing using the child and MPU extended time stamp descriptor becomes possible. That is, even when the leap second is adjusted to the NTP time, the receiving device can present a normal MPU using the MPU time stamp stored in the MPU time stamp descriptor.

[Supplement: Transmitter and receiver]
As described above, the transmission device that stores and transmits the data constituting the coded stream in the MPU can also be configured as shown in FIG. 90. Further, the receiving device that receives the MPU in which the data constituting the coded stream is stored can be configured as shown in FIG. 91. FIG. 90 is a diagram showing an example of a specific configuration of the transmission device. FIG. 91 is a diagram showing an example of a specific configuration of the receiving device.

The transmission device 900 includes a generation unit 901 and a transmission unit 902. Each of the generation unit 901 and the transmission unit 902 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving device 1000 includes a receiving unit 1001 and a reproducing unit 1002. Each of the receiving unit 1001 and the reproducing unit 1002 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

A detailed description of each component of the transmitting device 900 and the receiving device 1000 will be given in the description of the transmitting method and the receiving method, respectively.

First, the transmission method will be described with reference to FIG. 92. FIG. 92 is an operation flow (transmission method) by the transmission device.

First, the generation unit 901 of the transmission device 900 generates presentation time information (MPU time stamp) indicating the presentation time of the MPU as the first data unit based on the NTP time as the reference time information received from the outside. (S2601).

Next, is the transmission unit 902 of the transmission device 900 generated based on the presentation time information generated by the MPU, the generation unit 901, and the NTP time before the leap second adjustment of the presentation time information (MPU time stamp)? The identification information indicating whether or not it is transmitted is transmitted (S2602).

As a result, the receiving device that has received the information transmitted from the transmitting device 900 can reproduce the MPU based on the identification information even when the leap second adjustment is performed, so that the MPU is reproduced at the intended time. can.

Next, the receiving method will be described with reference to FIG. 93. FIG. 93 is an operation flow (reception method) by the receiving device.

First, whether or not the receiving unit 1001 of the receiving device 1000 is generated based on the MPU, the presentation time information indicating the presentation time of the MPU, and the NTP time before the leap second adjustment of the presentation time information (MPU time stamp). (S2701).

Next, the reproduction unit 1002 of the reception device 1000 reproduces the MPU received by the reception unit 1001 based on the presentation time information (MPU time stamp) and the identification information received by the reception unit 1001 (S2702).

As a result, the receiving device 1000 can reproduce the MPU at the intended time even when the leap second adjustment is performed.

In the sixth embodiment, the reproduction means a process including at least one process of decoding and presentation.

(Modified Example of Embodiment 6)
Next, a specific example of a specific signaling method for additional information (identification information) described with reference to FIGS. 84 and 85 will be described.

Here, a method of signaling by a combination of the MPU time stamp descriptor shown in FIG. 81 (a) and the MPU extended time stamp descriptor shown in FIG. 81 (b) will be described.

FIG. 94 is a diagram showing an extended example of the MPU extended time stamp descriptor. The underlined field in FIG. 94 is a field newly added to the MPU extended time stamp descriptor shown in FIG. 81 (b).

The NTP_leap_indicator in FIG. 94 indicates a flag as to whether or not the MPU extended time stamp descriptor as the second control information indicates additional information (identification information) related to the leap second adjustment of NTP. When this flag is set, mpu_presentation_time_type in a loop for each of a plurality of MPUs is valid.

In the mpu_presentation_time_type in FIG. 94, when the MPU time stamp is corrected as described in FIG. 84 on the transmitting side (transmitting device), is the mpu_presentation_time of the same sequence number described in the MPU time stamp stamp descriptor corrected? Show me how. Further, in the mpu_presentation_time_type, when the MPU time stamp is corrected in the receiving device as described with reference to FIG. 85, the timing at which the mpu_presentation_time of the same sequence number described in the MPU time stamp stamp descriptor is given is the A region. It indicates whether it is a B region, a C region, or a D region.

That is, the MPU extended time stamp descriptor stores identification information indicating whether or not the MPU time stamp is generated based on the time information (NTP time) before leap second adjustment. Further, the MPU extended time stamp descriptor stores the identification information corresponding to the MPU in each loop of the plurality of MPUs.

In step S2602, the transmission unit 902 of the transmission device 900 has the MPU, the MPU time stamp descriptor as the first control information in which the presentation time information (MPU time stamp) generated by the generation unit 901 is stored, and the said. The MPU extended time stamp descriptor as the second control information in which the identification information indicating whether or not the presented time information is generated based on the time information (NTP time) before the leap second adjustment is stored is transmitted.

In step S2701, the receiving unit 1001 of the receiving device 1000 receives the MPU, the MPU time stamp descriptor, and the MPU extended time stamp descriptor. Then, the reproduction unit 1002 of the receiving device 1000 reproduces the MPU received by the receiving unit 1001 based on the MPU time stamp descriptor and the MPU extended time stamp descriptor received by the receiving unit 1001. As described above, the receiving device 1000 analyzes both the MPU time stamp descriptor and the MPU extended time stamp descriptor, so that the decoding process using the MPU time stamp can be performed at the time of leap second adjustment. Therefore, the receiving device 1000 can reproduce the MPU at the intended time even when the leap second adjustment is performed. In addition, since the identification information is stored in the MPU extended time stamp descriptor, the time information (NTP time) before the leap second adjustment of the MPU time stamp is maintained while the MPU time stamp descriptor is configured to correspond to the existing standard (MPEG). ) Can be extended to signal whether or not it was generated. In this way, since the conventional configuration can be used, even if the signaling function is expanded, the design change can be minimized, and the manufacturing cost of the transmitting side (transmitting device) and the receiving device can be reduced. ..

The timing of setting the NTP_leap_indicator may be the same as that of the leap_indicator in the NTP packet, or may be any time while the leap_indicator is standing. That is, the NTP_leap_indicator is determined to be a value corresponding to the value of the leap_indicator in the NTP packet corresponding to the NTP time as the reference for which the MPU time stamp is generated.

(Modification 2 of Embodiment 6)
The problem related to the leap second adjustment described in the present embodiment is not limited to the case where the time based on the NTP time is used, but also the problem which occurs similarly even when the time based on the UTC time is used. be.

When the time system is a time system based on UTC and the presented time information (MPU time stamp) is generated and given based on the time system, the same method as the method shown so far should be used. Therefore, even at the time of leap second adjustment, the presentation time information can be corrected and processed on the receiving device side by using the presentation time information and signaling information (additional information, identification information).

For example, in ARIB STD-B60, when transmitting an event message for the purpose of notifying the application running on the receiver of the time (time designation information) for specifying the operation from the broadcasting station, the event message descriptor. Is stored in the MMTP packet, and the time when the event message occurs is indicated in the event message descriptor. In this case, the data constituting the application is stored in the MPU as the first data unit and transmitted. There are multiple ways to specify the time when the event message occurs, and it is indicated by time_mode. In time_mode, when the time when the event message occurs is expressed in UTC, the UTC time is given based on the time information before the leap second adjustment or the time information after the leap second adjustment. It signals identification information indicating whether it is time information. That is, the identification information is information indicating whether or not the presentation time information is generated based on the time information (NTP time) before the leap second adjustment, similar to the identification information described in the present embodiment. It is desirable that such identification information (signaling information) be shown in the event message descriptor. For example, it can be signaled using the reserved_future_use field of the event message descriptor.

That is, in step S2601, the generation unit 901 of the transmission device 900 generates time designation information indicating the operation time of the application based on the NTP time as the reference time information received from the outside. Then, in the transmission unit 902 of the transmission device 900, in step S2602, the MPU, the time designation information indicating the operation time of the application (time when the event message occurs) generated by the generation unit 901, and the time information are combined. An event message descriptor as control information in which identification information indicating whether or not the time information is before leap second adjustment is stored is transmitted.

The transmitting side (transmitting device) determines the time zone or program that requires leap second adjustment, and prohibits the use of time_mode expressed in UTC in the time zone or program that requires leap second adjustment, and is involved in UTC. It may be specified that no time_mode is used.

Alternatively, the transmitting side (transmitting device) calculates the UTC time when the event message is generated, determines whether or not the time when the event message is generated is after the leap second adjustment, and if it is after the leap second adjustment, the leap second is adjusted. The second-adjusted UTC time may be calculated and stored in the event message descriptor. Further, it may be signaled separately whether or not the presentation time information is adjusted for leap seconds.

Similarly, the UTC (Universal Time Coordinated) -NPT (Normal Play Time) reference descriptor shows the relationship between UTC and NPT using the UTC_Reference and NPT_Reference fields. Whether the time information shown in the UTC_Reference field or NPT_Reference field is the time information given based on the reference time information (NTP time) before the leap second adjustment or the time information given based on the reference time information after the leap second adjustment. Signal. That is, the UTC-NPT reference descriptor includes the presentation time information generated by the generation unit 901 of the transmission device 900 and the identification information indicating whether or not the presentation time information is the time information (NTP time) before the leap second adjustment. Is the control information that stores. The identification information can be signaled using the reserved field or the like in the UTC-NPT reference descriptor.

Further, the method of designating the presentation time of the subtitle or the character supermarket described in ARIB STD-B60 or ARIB STD-B62 is indicated by TMD (time control mode) in the additional identification information (Additional_Alib_Subtitle_Info). The additional identification information is stored in the MPT asset loop.

TMD (time control mode) is MPU time stamp and NPT (Normal Play)
In the case of the mode which is the time starting from Time), the presentation time of the subtitle or the character super is shown in the MPU extended time stamp descriptor or the UTC-NPT reference descriptor. With the above signaling information, it is possible for the receiving device to correct and process the time information by using the time information and the signaling information even at the time of leap second adjustment.

When the time control mode = '0000' (UTC_TTML description), the time code in the ARIB-TTML subtitle indicates the presentation time corresponding to UTC. In this case, it signals whether the UTC time indicated by the time code is the time information given based on the time information before the leap second adjustment or the time information given based on the time information after the leap second adjustment. .. It is desirable that the signaling information transmitted at this time is shown in the ARIB-TTML subtitles.

When the time control mode = '0010' (UTC_TTML description), it is the time starting from the UTC time shown in the reference_start_time field in the additional identification information. In this case, signaling is performed in the additional identification information in the same manner as in the case where the time control mode = '0000' (UTC_TTML description). In this case, on the transmitting side (transmitting device), the UTC time indicated in the reference_start_time field is a predetermined period (for example, the time until 9:00: 00) immediately before the leap second adjustment is performed. Information indicating whether or not the time is from the time before (24 hours) (that is, 9:00:00 on the day before the leap second adjustment) to the time immediately before the time is transmitted as identification information.

It is stipulated that the transmitting side (transmitting device) determines the time zone and program that require leap second adjustment, prohibits the use of the time control mode expressed in UTC, and uses the time control mode that is not related to UTC. May be good.

The same applies to the MH-Expire descriptor.

(Embodiment 7)
A general broadcasting station facility has a plurality of transmission systems, and one system is operated as a normal facility for an actual broadcasting service, and the other is operated as a redundant system facility for backup. As a result, it is possible to switch to the redundant equipment when an error occurs in the service provided to the viewer due to the transmission equipment or the like. Here, in the MMT / TLV method, the following phenomenon occurs when switching redundant equipment.

1. 1. Adjustment of MPU sequence number Adjustment of MPU sequence number is performed as a preprocessing for switching redundant equipment. As shown in FIG. 95, when the MPU sequence number is adjusted, the MPU sequence number may be discontinuous before and after the adjustment. FIG. 95 shows an example in which MPU sequence numbers are duplicated.

The access unit and the MPU time stamp for the access unit are the same before and after the adjustment of the MPU sequence number, but the same MPU sequence number is assigned to different MPUs, so that the two MPUs have the same MPU sequence number. The same MPU sequence number is assigned across.

As a result, the MPU sequence number stored in the MMTP packet header and the MPU sequence number stored in the MPU time stamp descriptor and the MPU extended time stamp descriptor are duplicated in different MPUs. Similarly, the MPU sequence number may be skipped or returned.

2. 2. Discontinuity of packet sequence number Because the packet sequence number and packet counter are different between the normal equipment and the redundant equipment, the packet sequence number and the packet counter become discontinuous when switching from the normal equipment to the redundant equipment. May be. As shown in FIG. 96, in normal operation, the packet sequence numbers are continuous, but may be skipped, duplicated, or returned. Note that FIG. 96 is a diagram for explaining a case where the packet sequence numbers become discontinuous at the timing of switching from the normal equipment to the redundant equipment.

When the MPU sequence number or the packet sequence number becomes discontinuous as in 1 or 2 above, there is a problem that the receiving device that operates by believing the continuity of these sequence numbers may malfunction.

The access unit and the MPU time stamp for the access unit are maintained in continuity even when the MPU sequence number or the packet sequence number is discontinuous, and continuous data is transmitted to the receiving device.

The reception process when the discontinuity of the MPU sequence number and the packet sequence number as described with reference to FIGS. 95 and 96 occurs will be described.

When the MPU sequence number or the packet sequence number is discontinuous, the receiving device that performs the process using the MPU sequence number or the packet sequence number as shown below may malfunction.

As a process using the MPU sequence number, for example, there is a process of detecting the MPU boundary by switching the MPU sequence number in the MMTP packet header.

Further, as a process using the MPU sequence number, for example, a time stamp for the MPU sequence number is described in the MPU time stamp descriptor and the MPU extended time stamp descriptor, and the time stamp is used for decoding and / or Present.

As a process using the packet sequence number, for example, the packet sequence number is used for detecting packet loss, and when the packet sequence number is skipped, it is determined as packet loss, and when the packet sequence number is returned, it is determined as packet replacement. There is processing.

In order to prevent malfunction due to processing using these MPU sequence numbers and packet sequence numbers, the receiving device determines the discontinuity of the MPU sequence number and the packet sequence number, and if it is determined to be discontinuous, the MPU sequence Perform processing that does not use numbers or packet sequence numbers.

Here, for example, the following method can be used as a method for determining the discontinuity of the MPU sequence number and the packet sequence number.

First, one of the above-mentioned determination methods may be to signal additional information indicating switching to redundant equipment. As a result, the receiving device can perform reception processing without using the MPU sequence number or the packet sequence number based on the additional information.

As a method of signaling additional information, there is a method of storing and transmitting the additional information in a control message, a descriptor, a table, MPU metadata, MF metadata, an MMTP packet header, an extension header of an MMTP packet, or the like.

Next, a method of detecting the discontinuity using other information in the receiving device will be described.

For example, as information indicating the MPU boundary, there are a method of detecting the switching of the MPU sequence number in the MMTP payload header and a method of using RAP_flag which is a flag indicating whether or not it is a random access point in the MMTP packet header. In addition, it is possible to detect the MPU boundary by switching packets in which Sample_number becomes 0, switching fragment types, and the like. It is also possible to detect the MPU boundary by counting the number of access units.

Here, when there is a possibility that the switching of the redundant system equipment is carried out, the MPU boundary is determined by using another method without determining the MPU boundary using the switching of the MPU sequence number. For example, a packet having RAP_flag of 1 is determined to be the first packet in the MPU. Here, if the MPU sequence number of the packet after the MPU boundary is the same as the MPU sequence number of the previous packet, it is determined that the MPU sequence number is duplicated in the MMTP packet.

Alternatively, the packet in which RAP_flag is 1 is determined to be the first packet in the MPU. Here, when the MPU sequence number of the packet determined to be the first packet is skipped or returned from the MPU sequence number of the preceding packet, it is determined that the MPU sequence number is discontinuous in the MMTP packet.

If the discontinuity of the MPU sequence number in the MMTP payload header always precedes the discontinuity of the MPU sequence number or the packet sequence number in the MPU time stamp descriptor or the MPU extended time stamp descriptor, it is based on the determination result. Therefore, reception processing that does not use a time stamp or processing that does not use a packet sequence number may be performed.

Further, the discontinuity of the MPU sequence number in the MMTP payload header of the video packet is the discontinuity of the MPU sequence number in the MMTP payload header of the voice packet, and the MPU sequence number in the MPU time stamp descriptor and the MPU extended time stamp descriptor of the voice. And if it always occurs before the discontinuity of the packet sequence number, the reception process that does not use the audio time stamp or the process that does not use the packet sequence number is based on the detection result of the discontinuity of the MPU sequence number of the video packet. May be carried out. The determination result may be determined as the timing at which switching to the redundant equipment starts.

The receiving device does not use the MPU sequence number or the packet sequence number only when the additional information indicating the switching to the redundant system equipment is indicated by using the method of signaling the additional information indicating the switching to the redundant system equipment. Processing may be performed.

The receiving device detects discontinuity between the RAP_flag and the MPU sequence number only when the additional information indicating switching to the redundant system equipment is indicated by using the method of signaling the additional information indicating the switching to the redundant system equipment. Processing may be carried out.

Here, the determination using the time stamp descriptor in the receiving device will be described. Normally, a time stamp for the MPU sequence number is described and transmitted multiple times. However, if the time stamp value is updated even though the MPU sequence number is the same, the receiving device receives the MPU sequence number. Is determined to be duplicated.

Also, if the relationship between the time stamp value and the MPU sequence number does not correspond, such as when the MPU sequence number is updated by 2 or more even though the time stamp value is updated for the presentation time of one MPU, it is received. The device determines that a discontinuity in the MPU sequence number has occurred.

When updating the MPU time stamp or the MPU extended time stamp, the version in the MP table may not be updated. Further, version information may be defined separately, and the version of the time stamp information and the version of the information other than the time stamp information may be transmitted separately. As a result, the version frequency of information other than the time stamp information can be reduced, and the reception process can be reduced.

If the MPU sequence number is discontinuous, the relationship between the MPU time stamp and the time stamp information cannot be trusted. Therefore, the descriptor may be signaled that the time stamp information is untrustworthy information. For example, using the same fields as NTP_leap_indicator and mpu_presentation_time_type described with reference to FIG. 94, it may be shown that the timing of adding the MPU time stamp is the timing before and after the redundant system switching, and the time stamp is unreliable. Further, it may be shown in the MPU extended time stamp descriptor on the assumption that it is used in combination with the MPU time stamp descriptor.

By using the above method and determining that the MPU sequence number and the packet sequence number are discontinuous in the receiving device, it is possible to perform processing without using the MPU sequence number and the packet sequence number, which causes a malfunction in synchronous reproduction. It will be possible to prevent it.

The operation flow of the receiving device when the discontinuity of the MPU sequence number and the packet sequence number as described above occurs will be described. FIG. 97 is an operation flow by the receiving device when a discontinuity of the MPU sequence number and the packet sequence number occurs.

The receiving device determines the MPU boundary from the packet header or payload header of the MMTP packet. Specifically, the MPU boundary is determined by determining that the packet in which RAP_flag is 1 is the packet at the head of the MPU. Further, the MPU sequence number is determined, and it is determined whether or not the MPU sequence number of the previous packet is incremented by 1 (S2801).

When it is determined that RAP_flag is 1 and the MPU sequence number is increased by 1 (Yes in S2801), it is assumed that the discontinuity of the MPU sequence number has not occurred, and the reception process using the MPU sequence number is performed. (S2803).

On the other hand, when it is determined that RAP_flag is 0, or even if RAP_flag is 1, it is not incremented by 1 from the MPU sequence number of the previous packet (No in S2801), a discontinuity in the MPU sequence number occurs. It is assumed that the reception process is performed without using the MPU sequence number (S2802).

(Embodiment 8)
In the sixth embodiment, the correction of the MPU time stamp has been mainly described, but a method of correcting the PTS or DTS for each AU contained in the MPU in the receiving device will be described. That is, a correction method for correcting the time stamp indicating the PTS or DTS of each of the plurality of other AUs excluding the head AU among the plurality of AUs included in the MPU will be described.

FIG. 98 is a diagram for explaining a correction method for correcting a time stamp in a receiving device when a leap second is inserted. FIG. 98 is the same case as in FIG. 85 (a).

Here, the NTP time on the transmitting side is synchronized with the NTP server, and the NTP time on the receiving side is reproduced based on the time stamp stored in the NTP packet transmitted from the transmitting side. A case where the time is synchronized with the NTP time on the transmitting side will be described as an example. In this case, both the NTP time on the transmitting side and the NTP time on the receiving side are adjusted by +1 second when the leap second is inserted.

It is assumed that the NTP time in FIG. 98 is common to the NTP time on the transmitting side and the NTP time on the receiving side. In the following, it is assumed that there is no transmission delay. Further, although only the method of correcting the PTS of the AU in the MPU will be described below, the DTS of the AU in the MPU can also be corrected by using the same method. In FIG. 98, the number of AUs included in one MPU is assumed to be 5, but the number of AUs included in the MPU is not limited to this.

Similar to the transmission method described in FIG. 85 (a), the NTP time immediately before the leap second insertion (up to 8:59:59 for the first time) is set to the A region, and the NTP time after the leap second insertion (second time). 8:59:59 or later) is defined as the B region.

The transmitting side (transmitting device) performs the following processing.

-As described with reference to FIG. 83 in the sixth embodiment, the MPU time stamp which is the presentation time information (first time information) of the MPU is generated.

-The MPU time stamp is stored in the MPU time stamp descriptor and transmitted to the receiving device without correcting the generated MPU time stamp. That is, the first time information is time information that is not corrected on the transmitting side of the MPU at the time of leap second adjustment. Further, the identification information indicating whether the timing of adding the MPU time stamp is the A region or the B region is transmitted to the receiving device. Specifically, an MMTP packet containing a plurality of MPUs is transmitted. This MMTP packet includes, as control information, an MPU time stamp descriptor including an MPU time stamp as first time information, an MPU extended time stamp descriptor including relative information as second time information, and identification information. ..

In FIG. 98, the timing at which the MPU time stamp is added is indicated by the arrow.

The receiving device performs the following processing.

As described in the sixth embodiment, in the MPU extended time stamp descriptor, the second time information for calculating the PTS or DTS for each AU is stored in the MPU time stamp descriptor of the head AU of the MPU. It is stored as relative information from PTS. At this time, the relative information stored in the MPU extended time stamp descriptor is not a value considering the insertion of leap seconds.

The receiving device first receives the MMTP packet including the MPU, the first time information, the second time information, and the identification information transmitted by the transmitting side (transmitting device), and the receiving device first receives the MMTP packet including the identification information, and the “for each AU” in FIG. 98. As shown in "PTS", all AUs contained in the MPU based on both the first time information stored in the MPU time stamp descriptor and the second time information stored in the MPU extended time stamp descriptor. Calculate PTS (DTS) for each.

Next, the receiving device corrects the PTS for each AU. In the AU to be corrected, the timing of assigning the MPU time stamp of the MPU to which the AU belongs (the timing indicated by the source of the arrow in FIG. 98) is in the A region, and the PTS of each AU before correction is 9:00: It is an AU indicating after 00. The AUs to be corrected are the AUs underlined in FIG. 98, AU # 5 in MPU # 1, and AU # 1-AU # 5 in MPU # 2. When the receiving device determines that it corresponds to the AU to be corrected for PTS, the receiving device corrects the PTS of the AU for -1 second.

That is, in the correction, for each of the plurality of AUs, the MPU time stamp of the MPU storing the AU is generated based on the reference time information before the leap second adjustment, and the PTS or the calculated PTS of the AU or It is determined whether or not the DTS satisfies the correction condition after a predetermined time (9:00:00 in the case of leap second insertion). Then, in the correction, the PTS or DTS of the AU determined to satisfy the above correction condition is corrected in the determination.

The timing of assigning the MPU time stamp of the MPU to which the AU belongs is in the A region, and the AU in which the PTS is not corrected (AU up to AU # 4 in MPU # 1 in FIG. 98) is before the leap second insertion (1). The AU is presented at the time that matches the NTP time of the 8:59:59: before the unit. Further, the PTS-corrected AU or the AU after that (AU # 5 or later in MPU # 1 in FIG. 98) is set to the NTP time after the leap second is inserted (after the second 8:59:59 unit). Present the AU at the same time.

It should be noted that the identification information indicating the timing of assigning the MPU time stamp needs to be presented to at least the MPU including the AU to be corrected. That is, it is not always necessary to present the identification information to the MPU that does not include the AU to be corrected.

FIG. 99 is a diagram for explaining a correction method for correcting a time stamp in a receiving device when a leap second is deleted. FIG. 99 is the same case as in FIG. 85 (b).

The NTP time on the transmitting side is synchronized with the NTP server, and the NTP time on the receiving side is reproduced based on the time stamp stored in the NTP packet transmitted from the transmitting side. The case of synchronizing with the NTP time will be described as an example. In this case, both the NTP time on the transmitting side and the NTP time on the receiving side are adjusted by -1 second when the leap second is deleted.

It is assumed that the NTP time in FIG. 99 is common to the NTP time on the transmitting side and the NTP time on the receiving side. In the following, it is assumed that there is no transmission delay. Further, although only the method of correcting the PTS of the AU in the MPU will be described below, the DTS of all the AUs in the MPU can be corrected by using the same method. In FIG. 99, the number of AUs included in one MPU is described as 5, but the number of AUs included in the MPU is not limited to this.

Similar to the transmission method described with reference to FIG. 85, the NTP time immediately before the leap second deletion (up to 8:59:58) is in the C area, and the NTP time after the leap second deletion (after 9:00:00) is D. It is an area.

The transmitting side (transmitting device) performs the following processing.

-As described with reference to FIG. 83 in the sixth embodiment, the MPU time stamp which is the presentation time information (first time information) of the MPU is generated.

-The MPU time stamp is stored in the MPU time stamp descriptor and transmitted to the receiving device without correcting the generated MPU time stamp. That is, the first time information is time information that is not corrected on the transmitting side of the MPU at the time of leap second adjustment. Further, the identification information indicating whether the timing of adding the MPU time stamp is the C region or the D region is transmitted to the receiving device. Specifically, an MMTP packet containing a plurality of MPUs is transmitted. The MMTP packet has the same configuration as the MMTP packet described with reference to FIG. 98.

In FIG. 99, the timing at which the MPU time stamp is added is indicated by the arrow.

The receiving device performs the following processing.

As described in the sixth embodiment, in the MPU extended time stamp descriptor, the second time information for calculating the PTS or DTS for each AU is stored in the MPU time stamp descriptor of the head AU of the MPU. It is stored as relative information from PTS. At this time, the relative information stored in the MPU extended time stamp descriptor is not a value considering the insertion of leap seconds.

First, the receiving device receives the MMTP packet including the MPU, the first time information, the second time information, and the identification information transmitted by the transmitting side (transmitting device), and the “for each AU” in FIG. 99. As shown in "PTS", all AUs contained in the MPU based on both the first time information stored in the MPU time stamp descriptor and the second time information stored in the MPU extended time stamp descriptor. Calculate PTS (DTS) for each.

Next, the receiving device corrects the PTS for each AU. In the AU to be corrected, the timing of assigning the MPU time stamp of the MPU to which the AU belongs (the timing indicated by the source of the arrow in FIG. 99) is in the C region, and the PTS of each AU before correction is 8:59: It is an AU indicating 59 or later. The AUs to be corrected are the AUs underlined in FIG. 99, AU # 5 in MPU # 2, and AU # 1-AU # 5 in MPU # 3. When the receiving device determines that the AU corresponds to the AU to be corrected by the PTS, the receiving device corrects the PTS of the AU by +1 second.

That is, in the correction, for each of the plurality of AUs, the MPU time stamp of the MPU storing the AU is generated based on the reference time information before the leap second adjustment, and the PTS or the calculated PTS of the AU or It is determined whether or not the DTS satisfies the correction condition after a predetermined time (8:59:59 in the case of leap second deletion). Then, in the correction, the PTS or DTS of the AU determined to satisfy the above correction condition is corrected in the determination.

In the receiving device, synchronous reproduction of all AUs can be performed based on the NTP time in which the leap second is deleted and the corrected PTS.

It should be noted that the identification information indicating the timing of assigning the MPU time stamp needs to be presented to at least the MPU including the AU to be corrected. That is, it is not always necessary to present the identification information to the MPU that does not include the AU to be corrected.

FIG. 100 is a diagram showing an operation flow of the receiving device described with reference to FIGS. 98 and 99.

Since the operation flow on the transmitting side is the same as the operation flow shown in FIG. 88, the description thereof will be omitted.

The receiving device first calculates the time stamps (PTS and DTS) for each AU using the MPU time stamp descriptor and the MPU extended time stamp descriptor (S2901).

Next, based on the identification information signaled by the transmitting side, whether the timing of assigning the MPU time stamp of the MPU to which the AU belongs is the A region, the B region, or the C region. , D region (S2902). Here, since the areas A to D are the same as those defined above, the description thereof will be omitted. As in FIG. 89, the case where the leap second is not inserted or deleted is not shown.

If it is determined in step S2902 that the MPU time stamp of the MPU to which the AU belongs is assigned based on the NTP time in the A region, is the AU time stamp calculated in step S2901 after 9:00: 0:? It is determined whether or not (S2903).

When it is determined that the AU time stamp is after 9:00 (Yes in step S2903), the AU time stamp value is corrected by -1 second (S2904).

If it is determined in step S2902 that the MPU time stamp of the MPU to which the AU belongs is given based on the NTP time in the C region, whether or not the AU time stamp calculated in step S2901 is after 8:59:59. Is determined (S2905).

When it is determined that the AU time stamp is after 8:59:59 (Yes in step S2905), the AU time stamp value is corrected by +1 second (S2906).

On the other hand, if it is determined in step S2902 that the MPU time stamp of the MPU to which the AU belongs is given based on the NTP time of the B area or the D area, or if the time stamp value of the AU is not 90000 or more in step S2903. If it is determined (No in step S2903), or if it is determined in step S2905 that the AU time stamp value is not after 8:59:59 (No in step S2905), the AU time stamp is not corrected (No). Step S2907), the operation of the receiving device is terminated.

As described above, as represented by the MPU time stamp descriptor and the MPU extended time stamp descriptor, the first time information indicating the absolute value of the MPU time stamp and the second time indicating the relative value to the absolute value are shown. When the information is transmitted, the transmitting side transmits the identification information indicating the region of the NTP time to which the absolute value of the time stamp is given, together with the absolute value of the time stamp.

In the receiving device, after calculating the time stamps (PTS and DTS) of all AUs based on the first time information indicating the absolute value of the time stamp of the MPU and the second time information indicating the relative value, the respective time stamps and the respective time stamps are used. , The time stamp is corrected by determining whether or not to correct the time stamp based on the identification information indicating the area of the NTP time which is the base to which the absolute value of the MPU time stamp which is the base of the time stamp is given. ..

Information indicating whether or not the time stamp has been corrected for each AU in the receiving device, and whether the timing at which the MPU time stamp of the MPU to which each AU belongs is the A region, the B region, or the C region. , Additional information such as information indicating whether it is the D region does not have to be shown in all MPUs, but may be shown in a section including an MPU including at least the AU to be corrected.

For example, when the time difference between the time stamp (PTS or DTS) in any AU and the NTP time that is the basis for assigning the MPU time stamp of the MPU to which the AU belongs is N seconds, the leap second is inserted or the leap second is used. When the seconds are deleted, the period during which the AU to be corrected exists is at least N seconds before the leap second adjustment. Therefore, the area A when the leap second is inserted or the area C when the leap second is deleted needs to be a region indicating a period of at least N seconds before the leap second adjustment.

On the contrary, the time indicating the A region when the leap second is inserted and the C region when the leap second is deleted may be defined as M seconds, and a constraint may be given so that a time stamp is always given so that N <M.

Further, the B region when the leap second is inserted or the D region when the leap second is deleted does not necessarily have to be signaled, and when the leap second is inserted, the information indicating that the region A has been transferred to the B region is transmitted to the leap second. At the time of deletion, it is sufficient to show at least the information indicating that the area has been changed from the C area to the D area.

In short, the receiving device first calculates the time stamps of all access units based on the first time information (absolute time information) and the second time information (relative time information), and all access units. For each of the above, the time stamp of the access unit is corrected based on the identification information indicating whether or not the NTP time, which is the basis for calculating the MPU time stamp of the MPU to which the access unit belongs, is before the leap second adjustment. Specifically, when the identification information indicates the time immediately before the leap second insertion and the time stamp of the access unit indicates after 9:00:00, the time stamp is corrected by -1 second. If the identification information indicates the time immediately before the leap second deletion and the time stamp of the access unit indicates after 8:59:59, the time stamp is corrected by +1 second. Further, in the case of an access unit whose time stamp has been corrected, the receiving device presents the access unit based on the reference time information after the leap second adjustment is performed.

In this way, by determining the timing of assigning the MPU time stamp of the MPU to which the access unit belongs at the time of leap second adjustment, it is possible to correct the time stamp of all access units, and the MPU time stamp descriptor and MPU extension time can be corrected. Appropriate reception processing using the stamp descriptor becomes possible. That is, even when leap second adjustment is performed on the reference time information that is the reference clock of the transmitting side and the receiving device, a plurality of access units stored in the MPU can be reproduced at the intended time. ..

[Supplement: Transmission / reception system and receiver]
As described above, the transmission / reception system including the transmission device for storing and transmitting the data constituting the coded stream in the MPU and the receiving device for receiving the transmitted MPU can also be configured as shown in FIG. 101. Is. Further, the receiving device can also be configured as shown in FIG. 102. FIG. 101 is a diagram showing an example of a specific configuration of a transmission / reception system. FIG. 102 is a diagram showing an example of a specific configuration of the receiving device.

The transmission / reception system 1100 includes a transmission device 1200 and a reception device 1300.

The transmission device 1200 includes a generation unit 1201 and a transmission unit 1202. Each of the generation unit 1201 and the transmission unit 1202 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The generation unit 1201 generates first time information (MPU time stamp) indicating the presentation time of the first data unit (MPU) based on the reference time information (NTP time) received from the outside.

The transmission unit 1202 includes a first data unit (MPU), a first time information (MPU time stamp) generated by the generation unit 1201, and a plurality of second data including the first time information (MPU time stamp). Second time information (relative information) indicating the presentation time (PTS) or decoding time (DTS) of each unit (AU) is transmitted, and identification information is transmitted.

The receiving device 1300 includes a receiving unit 1301, a calculating unit 1302, and a correction unit 1303. Each of the receiving unit 1301, the calculation unit 1302, and the calculation unit 1303 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

A detailed description of each component of the receiving device 1300 will be given in the description of the receiving method.

The receiving method will be described with reference to FIG. 103. FIG. 103 is an operation flow (reception method) by the receiving device.

First, the receiving unit 1301 of the receiving device 1300 receives the first data unit (MPU), the first time information (MPU time stamp), the second time information (relative information), and the identification information (S3001). ).

Next, the calculation unit 1302 of the receiving device 1300 uses the first time information (MPU time stamp) and the second time information (relative information) received by the receiving unit 1301 to be received by the receiving unit 1302. The second data unit (AU) of the second data unit (AU) produces a presentation time (PTS) or a decoding time (DTS), respectively (S3002).

Next, the correction unit 1303 of the receiving device 1300 may present the presentation time (PTS) of each of the plurality of second data units (AU) calculated by the calculating unit 1302 based on the identification information received by the receiving unit 1301. The decoding time (DTS) is corrected (S3003).

As a result, the receiving device 1300 can normally reproduce the plurality of second data units stored in the first data unit even when the leap second adjustment is performed.

In the eighth embodiment, the reproduction means a process including at least one of decoding and presentation.

(Embodiment 9)
In the eighth embodiment, the method of correcting the PTS or DTS for each AU contained in the MPU in the receiving device will be described, but in the ninth embodiment, the method of correcting the PTS or DTS in the transmitting device will be described. .. That is, a correction method for correcting the time stamp indicating the PTS or DTS of each of the plurality of other AUs excluding the head AU among the plurality of AUs included in the MPU will be described.

FIG. 104 is a diagram for explaining a correction method for correcting a time stamp on the transmitting side (transmitting device) when a leap second is inserted. FIG. 104 is the same case as in FIG. 84 (a).

Here, the NTP time on the transmitting side is synchronized with the NTP server, and the NTP time on the receiving side is reproduced based on the time stamp stored in the NTP packet transmitted from the transmitting side. A case where the time is synchronized with the NTP time on the transmitting side will be described as an example. In this case, both the NTP time on the transmitting side and the NTP time on the receiving side are adjusted by +1 second when the leap second is inserted.

It is assumed that the NTP time in FIG. 104 is common to the NTP time on the transmitting side and the NTP time on the receiving side. In the following, it is assumed that there is no transmission delay. Further, although only the method of correcting the PTS of the AU in the MPU will be described below, the DTS of the AU in the MPU can also be corrected by using the same method. In FIG. 104, the number of AUs included in one MPU is assumed to be 5, but the number of AUs included in the MPU is not limited to this.

The transmitting side (transmitting device) performs the following processing.

1. 1. As described with reference to FIG. 83 in the sixth embodiment, the generation process for generating the MPU time stamp which is the presentation time information (first time information) of the MPU is performed.

2. 2. When the timing of assigning the MPU time stamp (timing indicated by the source of the arrow in FIG. 104) is included in the A area and the MPU time stamp (MPU time stamp value before correction) indicates 9:00: 00 or later. , The MPU time stamp value is corrected by -1 second, and the corrected MPU time stamp is stored in the MPU time stamp descriptor. That is, if the MPU time stamp is generated based on the NTP time included in the A region and the MPU time stamp indicates after 9:00, the MPU time stamp is corrected by -1 second. In addition, "9:00:00" here is the time corresponding to Japan time, which is the reference time for leap second adjustment (that is, the time calculated by adding 9 hours to UTC time). Is.

Further, when the timing of adding the MPU time stamp (the timing indicated by the source of the arrow in FIG. 104) is the B region, the MPU time stamp is not corrected. That is, if the MPU time stamp is generated based on the NTP time included in the B area, the MPU time stamp is not corrected.

In the MPU extended time stamp descriptor, the second time information for calculating the PTS or DTS for each AU is stored as relative information from the PTS of the head AU of the MPU stored in the MPU time stamp descriptor. ing. At this time, the relative information stored in the MPU extended time stamp descriptor is not a value considering the insertion of leap seconds.

Therefore, in the generation process of 1 above, the MPU time stamp is corrected when the MPU time stamp as the first time information satisfies the correction condition. Therefore, the first time information is the time information to be corrected at the time of leap second adjustment. The correction condition is that the MPU time stamp is generated based on the NTP time before the leap second adjustment, and is after 9:00:00.

3. 3. Performs a transmission process of transmitting an MMTP packet containing a plurality of MPUs to a receiving device. The MMTP packet includes, as control information, an MPU time stamp descriptor including MPU time stamp as first time information, an MPU extended time stamp descriptor including relative information as second time information, and identification information.

Here, the identification information uses the MPU time stamp and the relative information when the MPU time stamp does not correspond to the correction condition of the MPU time stamp in the generation process of 1 above and the transmitting side does not correct the MPU time stamp. Information for the receiving device to identify the MPU including the AU that needs to be corrected by inserting a leap second for the PTS or the DTS when the receiving device calculates the PTS or DTS for each AU in the MPU. ..

That is, the identification information is an AU presented by the PTS that needs to be corrected or presented by the DTS that needs to be corrected by performing leap second adjustment with respect to the reference time information. , Information indicating that an AU other than the leading AU (hereinafter referred to as "correction required AU") may be included in the MPU. In other words, the identification information can be said to be information indicating whether or not the receiving device is made to determine whether or not the correction required AU is included in the MPU.

The identification information indicates, for example, whether or not the MPU time stamp is information generated based on the NTP time before leap second adjustment in the generation process of 1 above and not corrected in the correction process of 2 above. Information. That is, the identification information includes the first identification information indicating whether or not the MPU time stamp is generated based on the NTP time before the leap second adjustment, and the second identification information indicating that the MPU time stamp has been corrected.

As described above, when the identification information indicates that the MPU time stamp is information generated based on the NTP time before leap second adjustment in the generation process and is not corrected in the correction process, the correction required AU is determined. Indicates that it may be included in the MPU. On the contrary, when the identification information indicates that the MPU time stamp is not generated based on the NTP time before leap second adjustment in the generation process or is corrected in the correction process, the correction required AU is required. Indicates that there is no possibility of being included in the MPU (that is, the receiving device is not made to determine whether or not the AU requiring correction is included in the MPU).

The processing on the transmitting side (transmitting device) will be specifically described with reference to FIG. 104.

In the example shown in FIG. 104, it is MPU # 2 that corresponds to the correction condition in the correction process of 2 above, and as shown by the underline, the MPU time stamp of MPU # 2 is corrected by -1 second on the transmitting side. The MPU time stamps of other MPUs are not corrected. The MPU time stamp calculated as described above is shown in the MPU time stamp descriptor, while the PTS (DTS) for each AU in the MPU is expressed from the MPU time stamp in the MPU extended time stamp descriptor (not shown) as before. Shown by relative information.

Further, MPU # 1 in FIG. 104 is an MPU including an AU requiring correction. Here, FIG. 104 shows the PTS for each AU calculated based on the MPU time stamp descriptor and the MPU extended time stamp descriptor. Although the PTS of AU # 5 in MPU # 1 is calculated as 9: 00: 00.1, it is actually the AU that should be presented at the second 8:59:59 after the leap second insertion, and- A 1 second correction is required. That is, AU # 5 in MPU # 1 is an AU requiring correction.

As described above, in the transmission device, the timing of assigning the MPU time stamp (the timing indicated by the source of the arrow in FIG. 104) is the region A, and the correction condition is not satisfied (that is, the MPU time stamp before correction). Is before 9:00 as a predetermined time, and at least one of all AU PTSs (DTSs) in the MPU calculated using the uncorrected MPU time stamp and relative information. It can be determined that the MPU that meets the conditions (the above values are after 9:00: 00) is an MPU that includes a correction-required AU, and information indicating that the MPU includes a correction-required AU is transmitted from the transmitting device to the receiving device. Signal to. In other words, the MPU should be presented in the MPU at 8:59:59 for the first time and at 8:59:59 for the second time when the MPU time stamp is not corrected. It can also be said that "MPU in which both of them exist".

The receiving device performs the following processing.

1. 1. Performs reception processing to receive MMTP packets transmitted by the transmitting side (transmitting device).

2. 2. A calculation process is performed to calculate the PTS (DTS) of all AUs based on the MPU time stamp descriptor and the MPU extended time stamp descriptor included in the control information of the MMTP packet.

3. 3. Based on the identification information signaled by the transmitting side (transmitting device) indicating whether or not the MPU may include the correction required AU, the identification information indicates that the correction required AU may be included. Performs a determination process for determining whether or not an AU requiring correction is included in the MPU. Specifically, it is determined whether or not the PTS (DTS) of all AUs included in the MPU is after 9:00, and the AU whose PTS (DTS) is after 9:00 is corrected. By determining that the AU is required, it is determined whether or not the MPU includes the AU requiring correction. Then, the PTS (DTS) of the correction required AU is corrected for -1 second.

The processing of the receiving device will be specifically described with reference to FIG. 104.

In FIG. 104, since the transmitting side signals that MPU # 1 is an MPU that may include a correction-required AU, the PTS calculated based on the time stamp descriptor in MPU # 1 is 9:00: AU # 5, which indicates 00 or later, is the target of correction, and correction is performed for -1 second. As a result, after the correction, the PTS of AU # 5 in MPU # 1 becomes 8: 59: 59.1.

If the transmitting side signals that the MPU does not include the correction required AU, the receiving device does not perform the above determination process. Therefore, for the plurality of AUs stored in the MPU, the PTS (DTS) of the plurality of AUs calculated based on the MPU time stamp descriptor and the MPU extended time stamp descriptor is not corrected. By doing the above, the PTS (DTS) of all AUs can be calculated.

The receiving device is corrected by either the transmitting device or the receiving device based on the information indicating whether the MPU time stamp is corrected on the transmitting side or the PTS (DTS) is corrected on the receiving device. When presenting or decoding using the time stamp, the presentation or decoding is always performed based on the NTP time after the leap second insertion is performed (after the second 8:59:59 unit). On the other hand, when presenting or decoding using the time stamp before correction, presentation or decoding is performed based on the NTP time before the leap second insertion is performed (before the first 8:59:59). Decrypt. In the example of FIG. 104, the AUs up to AU # 4 in MPU # 1 are presented based on the NTP time before the leap second insertion is performed (before the first 8:59:59), and MPU # 1 is presented. The AUs after AU # 5 in the above are presented based on the NTP time after the leap second insertion is performed (after the second 8:59:59 unit).

In the present embodiment, it has been explained that the transmitting side stores the relative value from the MPU time stamp without considering the leap second insertion as in the conventional case, but the correction by the leap second insertion is performed. Also, in consideration of the above, the relative value after correcting the time stamp of AU # 5 in MPU # 1 by -1 second may be stored. At this time, it is separately signaled that the AU is corrected and is presented or decoded based on the NTP time after the leap second insertion (after the second 8:59:59 second).

The receiving device calculates the PTS or DTS of all AUs based on the MPU time stamp descriptor and the MPU extended time stamp descriptor. At this time, since the relative value has already been corrected, it is not necessary to correct the AU time stamp in the receiving device, the information indicating whether or not the MPU time stamp has been corrected, and the PTS or DTS of the AU in the MPU. All AUs can be presented or decoded based on information indicating whether or not has been corrected.

FIG. 105 is a diagram for explaining a correction method for correcting a time stamp on the transmitting side (transmitting device) when a leap second is deleted. FIG. 105 is the same case as in FIG. 84 (b).

Here, the NTP time on the transmitting side is synchronized with the NTP server, and the NTP time on the receiving side is reproduced based on the time stamp stored in the NTP packet transmitted from the transmitting side. A case where the time is synchronized with the NTP time on the transmitting side will be described as an example. In this case, both the NTP time on the transmitting side and the NTP time on the receiving side are adjusted by -1 second when the leap second is inserted.

It is assumed that the NTP time in FIG. 105 is common to the NTP time on the transmitting side and the NTP time on the receiving side. In the following, it is assumed that there is no transmission delay. Further, although only the method of correcting the PTS of the AU in the MPU will be described below, the DTS of the AU in the MPU can also be corrected by using the same method. In FIG. 104, the number of AUs included in one MPU is assumed to be 5, but the number of AUs included in the MPU is not limited to this.

The transmitting side (transmitting device) performs the following processing.

1. 1. As described with reference to FIG. 83 in the sixth embodiment, the generation process for generating the MPU time stamp which is the presentation time information (first time information) of the MPU is performed.

2. 2. When the timing of assigning the MPU time stamp (timing indicated by the source of the arrow in FIG. 105) is included in the C area and the MPU time stamp (MPU time stamp value before correction) indicates 8:59:59 or later. , The MPU time stamp value is corrected by +1 second, and the corrected MPU time stamp is stored in the MPU time stamp descriptor. That is, if the MPU time stamp is generated based on the NTP time included in the C region and the MPU time stamp indicates after 8:59:59, the MPU time stamp is corrected by +1 second. In addition, "8:59:59" here is the time corresponding to the Japanese time as the reference time for leap second adjustment (that is, the time calculated by adding 9 hours to the UTC time). It is the time obtained by subtracting -1 second from.

Further, when the timing of adding the MPU time stamp (the timing indicated by the source of the arrow in FIG. 105) is the D region, the MPU time stamp is not corrected. That is, if the MPU time stamp is generated based on the NTP time included in the D area, the MPU time stamp is not corrected.

In the MPU extended time stamp descriptor, the second time information for calculating the PTS or DTS for each AU is stored as relative information from the PTS of the head AU of the MPU stored in the MPU time stamp descriptor. ing. At this time, the relative information stored in the MPU extended time stamp descriptor is not a value considering the deletion of leap seconds.

Therefore, in the generation process of 1 above, the MPU time stamp is corrected when the MPU time stamp as the first time information satisfies the correction condition. Therefore, the first time information is the time information to be corrected at the time of leap second adjustment. The correction condition is that the MPU time stamp is generated based on the NTP time before the leap second adjustment, and is after 8:59:59.

3. 3. Performs a transmission process of transmitting an MMTP packet containing a plurality of MPUs to a receiving device. The MMTP packet includes, as control information, an MPU time stamp descriptor including MPU time stamp as first time information, an MPU extended time stamp descriptor including relative information as second time information, and identification information.

Here, the identification information uses the MPU time stamp and the relative information when the MPU time stamp does not correspond to the correction condition of the MPU time stamp in the generation process of 1 above and the transmitting side does not correct the MPU time stamp. Information for the receiving device to identify the MPU including the AU that needs to be corrected by deleting the leap second for the PTS or the DTS when the receiving device calculates the PTS or DTS for each AU in the MPU. ..

Here, since the identification information is the same as the identification information described in the leap second insertion, the description thereof will be omitted.

The processing on the transmitting side (transmitting device) will be specifically described with reference to FIG. 105.

In the example shown in FIG. 105, it is MPU # 3 that corresponds to the condition for correcting the MPU time stamp in the correction process of 2 above, and as shown by the underline, the MPU time stamp of MPU # 3 is increased by +1 on the transmitting side. Seconds are corrected, and MPU time stamps of other MPUs are not corrected. The MPU time stamp calculated as described above is shown in the MPU time stamp descriptor, while the PTS (DTS) for each AU in the MPU is expressed from the MPU time stamp in the MPU extended time stamp descriptor (not shown) as before. Shown as relative information.

Further, MPU # 2 in FIG. 105 is an MPU including an AU requiring correction.

Here, 105 shows the PTS for each AU calculated based on the MPU time stamp descriptor and the MPU extended time stamp descriptor. Although the PTS of AU # 5 in MPU # 2 is calculated as 8:59:59.1, it is actually the AU that should be presented at 9:00:00 after the leap second is deleted, and the correction is +1 second. is required. That is, AU # 5 in MPU # 2 is an AU requiring correction.

As described above, in the transmission device, the timing of assigning the MPU time stamp (the timing indicated by the source of the arrow in FIG. 105) is in the C region and does not satisfy the correction condition (that is, the MPU time stamp before correction). Is before 8:59:59 as a predetermined time, and at least one of all AU PTSs (DTSs) in the MPU calculated using the uncorrected MPU time stamp and relative information. It can be determined that the MPU that meets the conditions (the above values are after 8:59:59) is an MPU that includes a correction-required AU, and information indicating that the MPU includes a correction-required AU is transmitted from the transmitting device to the receiving device. Signal to. In other words, when the MPU time stamp is not corrected, the MPU should be presented in the MPU at 8:59:58 before the leap second is deleted, and at 9:00:00 after the leap second is deleted. It can also be said that there is an MPU in which both AUs to be presented exist.

The receiving device performs the following processing.

1. 1. Performs reception processing to receive MMTP packets transmitted by the transmitting side (transmitting device).

2. 2. A calculation process is performed to calculate the PTS (DTS) of all AUs based on the MPU time stamp descriptor and the MPU extended time stamp descriptor included in the control information of the MMTP packet.

3. 3. Based on the identification information signaled by the transmitting side (transmitting device) indicating whether or not the MPU may include the correction required AU, the identification information indicates that the correction required AU may be included. Performs a determination process for determining whether or not an AU requiring correction is included in the MPU. Specifically, it is determined whether or not the PTS (DTS) of all AUs included in the MPU is after 8:59:59, and the AU whose PTS (DTS) is after 8:59:59 is corrected. By determining that the AU is required, it is determined whether or not the MPU includes the AU requiring correction. Then, the PTS (DTS) of the correction required AU is corrected for +1 second.

In FIG. 105, since the transmitting side signals that MPU # 2 is an MPU that may include a correction-required AU, the PTS calculated based on the time stamp descriptor in MPU # 2 is 8:59 :. AU # 5, which indicates 59 or later, is the target of correction, and correction is performed for +1 second. As a result, after the correction, the PTS of AU # 5 in MPU # 2 becomes 9:00:00.1.

If the transmitting side signals that the MPU does not include the correction required AU, the receiving device does not perform the above determination process. Therefore, for the plurality of AUs stored in the MPU, the PTS (DTS) of the plurality of AUs calculated based on the MPU time stamp descriptor and the MPU extended time stamp descriptor is not corrected. By doing the above, the PTS (DTS) of all AUs can be calculated.

The receiving device can present or decode the AU using the corrected PTS (DTS).

In the present embodiment, it has been explained that the transmitting side stores the relative value from the MPU time stamp without considering the leap second deletion as in the conventional case, but the correction by the leap second deletion is performed. Also, in consideration of the above, the relative value after correcting the time stamp of AU # 5 in MPU # 1 by +1 second may be stored.

The receiving device calculates the PTS or DTS of all AUs based on the MPU time stamp descriptor and the MPU extended time stamp descriptor. At this time, since the relative value has already been corrected, it is not necessary for the receiving device to correct the time stamp of the AU, and all AUs can be presented or decoded.

In addition, among the correction methods for leap second insertion or leap second deletion described so far, the transmitting side (transmitting device) may be able to select which correction method to use. When the transmitting side (transmitting device) selects a correction method, the transmitting side signals to the receiving device which method is used for the correction. The receiving device can switch the correction, presentation operation, or decoding operation in the receiving device based on the signaled information.

FIG. 106 is a diagram showing an operation flow of the transmission device described with reference to FIG. 104. FIG. 107 is a diagram showing an operation flow of the receiving device described with reference to FIG. 104.

In FIG. 106, the transmitting device determines whether the timing of assigning the time stamp of the MPU is in the A region or the B region (S3101). Here, since the A region and the B region are the same as those defined above, the description thereof will be omitted. As in FIG. 89, the case where the leap second is not inserted is not shown.

If it is determined in step S3101 that the area is B, the process is terminated without correcting the MPU time stamp (S3102).

On the other hand, when it is determined in step S3101 that the region is A, it is determined whether or not the MPU time stamp is after 9:00:00 (S3103).

When it is determined that the MPU time stamp is after 9:00 (Yes in step S3103), the MPU time stamp is corrected by -1 second, and the correction of the MPU time stamp is signaled to the receiving device (S3104). , End the process.

On the other hand, when it is determined that the MPU time stamp is not after 9:00 (No in step S3103), the MPU time stamp descriptor and the MPU extended time stamp descriptor of all AUs included in the MPU are used. Calculate the time stamps (PTS and DTS) (S3105).

Next, it is determined whether or not the AU whose calculated time stamp is after 9:00: 00 is included in the MPU (S3106).

When it is determined that the calculated time stamp includes an AU after 9:00 (Yes in S3106), the receiving device is informed that the time stamp of the AU in the MPU needs to be corrected. Signaling (S3107).

On the other hand, when it is determined that the AU whose calculated time stamp is after 9:00: 00 is not included in the MPU (No in S3106), the process ends.

In FIG. 107, the receiving device calculates the time stamps of all AUs using the MPU time stamp descriptor and the MPU extended time stamp descriptor (S3201).

Next, it is determined whether or not the time stamp of the AU in the MPU needs to be corrected based on the signaled identification information (S3202).

When it is determined that the AU time stamp in the MPU needs to be corrected (Yes in step S3202), the AU time stamp indicating that the time stamp is after 9:00 is corrected by -1 second (S3205).

On the other hand, when it is determined that the correction of the AU time stamp in the MPU is not necessary (No in step S3202), it is determined whether or not the MPU time stamp is corrected based on the signaled identification information (S3203). ).

When it is determined that the MPU time stamp is not corrected (No in step S3203), the AU is presented based on the NTP time before the leap second insertion for the time stamp before the corrected time stamp, and the corrected time is presented. For the time stamp after the stamp, the AU is presented based on the NTP time after the leap second insertion (S3204).

After step S3205, or if it is determined in step S3203 that the time stamp has been corrected (Yes in step S3203), the AU is presented based on the NTP time after the leap second insertion has been performed (S3206).

FIG. 108 is a diagram showing an operation flow of the transmission device described with reference to FIG. 105. FIG. 109 is a diagram showing an operation flow of the receiving device described with reference to FIG. 105.

In FIG. 108, the transmitting device determines whether the timing of adding the time stamp of the MPU is in the C region or the D region (S3301). Here, since the C region and the D region are the same as those defined above, the description thereof will be omitted. As in FIG. 89, the case where the leap second is not deleted is not shown.

If it is determined in step S3301 that it is in the D region, the MPU time stamp is not corrected (S3302), and the process ends.

On the other hand, when it is determined in step S3301 that it is in the C region, it is determined whether or not the MPU time stamp is after 8:59:59 (S3303).

If it is determined that the MPU time stamp is after 8:59:59 (Yes in step S3303), the MPU time stamp is corrected by +1 second (S3304), and the process ends.

On the other hand, when it is determined that the MPU time stamp is not after 8:59:59 (No in step S3303), the MPU time stamp descriptor and the MPU extended time stamp descriptor of all AUs included in the MPU are used. Calculate the time stamps (PTS and DTS) (S3305).

Next, it is determined whether or not an AU whose calculated time stamp is after 8:59:59 is included (S3306).

If the calculated time stamp is after 8:59:59 but it is determined that the AU is included (Yes in S3306), the receiving device indicates that the receiving device needs to correct the time stamp of the AU in the MPU. Signal to (S3307).

On the other hand, when it is determined that the AU whose calculated time stamp is after 8:59:59 is not included in the MPU (No in S3306), the process ends.

In FIG. 109, the receiving device calculates the time stamps of all AUs using the MPU time stamp descriptor and the MPU extended time stamp descriptor (S3401).

Next, it is determined whether or not the time stamp of the AU in the MPU needs to be corrected based on the signaled identification information (S3402).

When it is determined that the AU time stamp in the MPU needs to be corrected (Yes in step S3402), the AU time stamp indicating that the time stamp is after 8:59:59 is corrected by +1 second (S3403).

After step S3403 or when it is determined in step S3402 that the time stamp of the AU in the MPU does not need to be corrected (No in step S3402), the AU is presented based on the NTP time.

According to this, (i) MPU, (ii) MPU time stamp, and (iii) a second data unit presented at the presentation time requiring correction or decoded at the decoding time requiring correction. An identification indicating that the first data unit may include a second data unit other than the first second data unit in the presentation order among the plurality of second data units. Send information and. Therefore, even when the leap second adjustment is performed with respect to the reference time information which is the reference clock of the transmitting side and the receiving device, the receiving device is stored in the first data unit. 2 data units can be played back at the intended time.

[Supplement: Transmitter and receiver]
As described above, the transmission device that stores and transmits the data constituting the coded stream in the MPU can also be configured as shown in FIG. 110. Further, the receiving device that receives the MPU in which the data constituting the coded stream is stored can also be configured as shown in FIG. 111. FIG. 110 is a diagram showing an example of a specific configuration of the transmission device. FIG. 111 is a diagram showing an example of a specific configuration of the receiving device.

The transmission device 1400 includes a generation unit 1401 and a transmission unit 1402. Each of the generation unit 1401 and the transmission unit 1402 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving device 1500 includes a receiving unit 1501 and a determining unit 1502. Each of the receiving unit 1501 and the determining unit 1502 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

A detailed description of each component of the transmitting device 1400 and the receiving device 1500 will be given in the description of the transmitting method and the receiving method, respectively.

First, the transmission method will be described with reference to FIG. 112. FIG. 112 is an operation flow (transmission method) by the transmission device.

First, the generation unit 1401 of the transmission device 1400 generates the first time information indicating the presentation time of the MPU as the first data unit based on the reference time information received from the outside (S3501).

Next, the transmission unit 1402 of the transmission device 1400 transmits the first data unit (MPU), the first time information (MPU time stamp), and the identification information (S3502). Here, the first time information (MPU time stamp) is the information generated by the generation unit 1401. Further, the identification information is a second data unit (access unit) presented at the presentation time (PTS) requiring correction or decoded at the decoding time (DTS) requiring correction, and is a plurality of data units (access units). The first data unit (MPU) includes a second data unit (access unit) other than the first second data unit (access unit) in the presentation order of the second data unit (access unit). Information that indicates that there is a possibility of

As a result, the receiving device that has received the information transmitted from the transmitting device 1400 can reproduce the AU in the MPU based on the identification information even when the leap second adjustment is performed, so that the AU in the MPU can be used. It can be played at the intended time.

Next, the receiving method will be described with reference to FIG. 113. FIG. 113 is an operation flow (reception method) by the receiving device.

First, the receiving unit 1501 of the receiving device 1500 receives the first data unit (MPU), the first time information (MPU time stamp), and the identification information (S3601). The first time information and the identification information are the same as the first time information and the identification information described in the transmission device 1400.

Next, the identification information received by the receiving unit 1501 indicates that the determination unit 1502 of the receiving device 1500 may include another second data unit (access unit). It is determined whether or not the unit (MPU) includes another second data unit (access unit) (S3602).

As a result, the receiving device 1500 can reproduce the AU in the MPU at the intended time even when the leap second adjustment is performed.

(Embodiment 10)
FIG. 114 is a diagram showing details of the protocol stack diagram of the MMT / TLV method defined in ARIB STD-B60, and is a diagram showing FIG. 78 in detail.

In the MMT / TLV method, video, audio, subtitles / character supermarkets, applications, etc. are transmitted in MMTP packets. Subtitles and character supermarkets are TTML-coded. As for the application, the HTML5 application is stored in the MMTP packet.

Further, FIG. 115 is a diagram showing a block diagram of the receiving device, and is a more detailed diagram of FIG. 80.

The receiving device 1600 extracts the TLV packet by decoding the transmission line coded data in the demodulation means 1602 with respect to the broadcast signal received by the tuner 1601 and performing error correction or the like on the data obtained by the decoding. ..

The TLV / IP / UDP DEMUX means 2603 performs TLV DEMUX processing and IP / UDP DEMUX processing. The TLV / IP / UDP DEMUX means 2603 performs processing according to the data type in the DEMUX processing of TLV. For example, in the case of a compressed IP packet, the compressed header is restored. The TLV / IP / UDP DEMUX means 2603 processes TLV-SI (AMT, NIT) when the data type is TLV-SI. The TLV / IP / UDP DEMUX means 2603 performs processing such as discarding when the data type is a null packet. The TLV / IP / UDP DEMUX means 2603 performs processing such as header analysis of IP packets and UDP packets in the IP / UDP DEMUX processing, and extracts MMTP packets and NTP packets.

The descrambled means 1604 descrambles the MMTP packet if it is scrambled. The descramble means 1604 acquires the scramble key information for descrambling from the EMM section, the ECM section, and the like in the control information analysis means 1605. When the descramble means 1604 is scrambled in IP packet units, it descrambles in IP packet units.

The NTP clock generation means 1606 reproduces the NTP clock from the NTP packet.

The MMT DEMUX means 1607 performs filtering processing of components such as video, audio, subtitles, character supermarkets, applications, and control information based on the packet ID stored in the MMTP packet header. The MMT DEMUX means 1607 describes a time stamp from the control information analysis means 1605 when it refers to both the time stamp descriptor (MPU time stamp descriptor and the MPU extended time stamp descriptor) stored in the MP table. (Described as a child) is acquired, and the PTS and DTS for each access unit are calculated by the PTS / DTS calculation means.

The media processing means 1608 converts MMTP packets into data in units for presenting filtered video, audio, subtitles, character supervision, applications, and the like. Specifically, the unit to be presented is a NAL unit or access unit of a video signal, an audio frame, a unit for presenting subtitles, or the like.

The decoding / presenting means 1609 decodes and presents the access unit at the time when the PTS / DTS of the access unit matches, based on the time information of the NTP clock. Further, when the decoding presentation means 1609 is mounted as a different LSI, a different device, or a different device (for example, a television, a display, a recording device, a set-top box, etc.), the decoding presentation means 1609 is an external output means or a communication means. , Interface means 1610 and the like are used to connect the respective processing blocks.

The configuration of the receiving device 1600 is not limited to the configuration shown in FIG. 115. In addition to MMT, there are DASH / ROUTE method, TSover IP method, etc. as the IP multiplexing method, and DVB-T2, DVB-S2, DVB-C2 as the broadcasting transmission method corresponding to the IP multiplexing method. , ATSC3.0, etc. In addition to the TLV method, the Layer2 protocol that serves as an interface between the IP multiplexing method and the broadcast transmission method includes a GSE method, an ALP (ATSC Link-Layer Protocol) method, and the like.

In addition, there are interface standards, communication standards, communication multicasts, etc. that use the IP multiplexing method. In the case of these methods as well, the receiving device has the same configuration as that shown in FIG. 115.

FIG. 116 is a diagram showing a general broadcasting protocol multiplexed by MPEG-2 TS Systems (hereinafter referred to as “TS system”).

In the TS method, video and audio, subtitles / character supermarkets, and applications are stored in PES or section format, and are transmitted as TS packets.

In conventional ARIB terrestrial digital broadcasting and BS digital broadcasting, subtitles and character supermarkets are ARIB subtitle encoded. The application is also data broadcast BML encoded.

Further, FIG. 117 is a diagram showing a block diagram of a receiving device that receives broadcast signals multiplexed by the TS method.

The receiving device 1700 extracts the TS packet by decoding the transmission line coded data in the demodulation means 1702 with respect to the broadcast signal received by the tuner 1701 and performing error correction or the like on the data obtained by the decoding. ..

The descrambled means 1703 descrambles the TS packet if it is scrambled. The descramble means 1703 acquires the scramble key information for descrambling from the EMM section, the ECM section, and the like in the control information analysis means 1704.

The reference clock generation means 1705 extracts the PCR from the TS packet and reproduces the PCR clock.

The TS DEMUX means 1706 performs filtering processing of components such as video, audio, subtitles, character supervision, application, and control information based on the PID (Packet Identifier) stored in the TS packet header.

The media processing means 1707 converts the TS packet into data of a unit for presenting filtered video, audio, subtitles, character super, application, etc., and further extracts a time stamp (PTS, DTS) from the PES packet header. The unit to be presented is a NAL unit or access unit of a video signal, an audio frame, a unit for presenting subtitles, or the like.

The decoding presentation means 1708 decodes and presents the access unit at the time when the PTS / DTS of the access unit matches, based on the time information of the PCR clock. Further, when each processing unit is mounted as a different LSI, a different device, or a different device (for example, a television, a display, a recording device, a set-top box, etc.), an external output means, a communication means, an interface means 1709, etc. are used. Use to connect each processing block.

The configuration of the receiving device 1700 is not limited to the configuration shown in FIG. 117. Transmission methods corresponding to the TS method include satellite broadcasting, terrestrial broadcasting, cable broadcasting transmission methods standardized by DVB, ISDB, ATSC, etc., interface standards, communication standards, communication multicast methods, and the like. In the case of these methods as well, the receiving device has the same configuration as that of FIG. 117.

There are two major broadcasting or communication multicast methods: a method based on the IP multiplexing method described with reference to FIGS. 114 and 115, and a method based on the TS method described with reference to FIGS. 116 and 117. .. Further, the receiving device includes a receiving device corresponding to a single method of receiving processing, a receiving device corresponding to a plurality of methods of receiving processing, and examples thereof are shown below.

118A, 118B, 118C, 118D, 118E, and 118F are configuration examples of the receiving device, and the first method is, for example, the MMT / TLV method described with reference to FIGS. 114 and 115. The first multiplexing method), and the second method is, for example, the TS method (second multiplexing method) described with reference to FIGS. 116 and 117. The first method and the second method may be reversed, or three or more methods may be mixed. Further, the processing means A and the processing means B are processing means in which some functions of the plurality of processing means described with reference to FIGS. 115 and 117 are integrated. Although an example of roughly dividing into the processing means A and the processing means B is shown here, any of the processing means of the plurality of processing means described with reference to FIGS. 115 and 117 may be combined and configured. And among the plurality of processing means described with reference to FIG. 117, the processing of one processing means may be further divided into a combination of processing means, or three or more processing means in which the plurality of processing means are combined may be combined. It may be configured by the processing means of.

FIG. 118A is a diagram showing a configuration of a conventional receiving device when processing a single method. In this case, the receiving device processes the signal transmitted by the first method by the processing means A1811 of the first method, and then processes it by the processing means B1812 of the first method.

FIG. 118B is a diagram showing a configuration of a conventional receiving device that independently processes a plurality of methods. In this case, the receiving device processes the signal transmitted by the first method by the processing means A1821 of the first method, then processes it by the processing means B1822 of the first method, and transmits it by the second method. The signal is processed by the processing means A1823 of the second method, and then processed by the processing means B1824 of the second method.

On the other hand, the configuration when the method conversion means as the conversion unit is used is shown below.

FIG. 118C is a diagram showing an example of the configuration of the receiving device when the method conversion means is used. In this case, the receiving device processes the signal transmitted by the first method by the processing means A1831 of the first method, and then transfers the output of the first method processing means A1831 to the second method by the method conversion means 1832. The conversion is performed, and the subsequent processing is processed by the second method processing means B1833.

FIG. 118D is a diagram showing another example of the receiver configuration when the method conversion means is used. In this case, the receiving device processes the signal transmitted by the first method by the processing means A1841 of the first method, and then processes a part of the output of the processing means A1841 of the first method by the first method. The means B1842 processes, the remaining part is converted into the second method by the method conversion means 1843, and the subsequent processing is processed by the second method processing means B1844.

FIG. 118E is a diagram showing another example of the configuration of the receiving device when the method conversion means is used. The receiving device in this case receives the signal transmitted by the first method and the signal transmitted by the second method, and processes the signal transmitted by the first method by the processing means A1851 of the first method. Then, the output of the first method processing means A1851 is converted into the second method by the method conversion means 1852, and the subsequent processing is processed by the second method processing means B1854. On the other hand, the receiving device processes the signal transmitted by the second method by the processing means A1853 of the second method, and then processes the output of the second method processing means A1853 by the second method processing means B1854. .. That is, the receiving device processes both the signal transmitted by the first method and the signal transmitted by the second method by the processing means B of the second method.

FIG. 118F is a diagram showing another example of the configuration of the receiving device when the method conversion means is used. The receiving device in this case receives the signal transmitted by the first method and the signal transmitted by the second method, and processes the signal transmitted by the first method by the processing means A1861 of the first method. After that, a part of the output of the processing means A1861 of the first method is processed by the processing means B1862 of the first method, and the remaining part is converted into the second method by the method conversion means 1863, and the subsequent processing. Is processed by the second method processing means B1865. On the other hand, the receiving device processes the signal transmitted by the second method by the processing means A1864 of the second method, and then processes the output of the second method processing means A1864 by the second method processing means B1865. .. That is, the receiving device processes a part of the signal transmitted by the first method and the signal transmitted by the second method by the processing means B1865 of the second method.

In addition, as shown in FIG. 118D or FIG. 118F, when the signal transmitted by the first method is processed by both the first method processing means B1842, 1862 and the second method processing means B1844, 1865, Information for synchronization (for example, time information, clock information, etc.) may be shared between the first method processing means B1842, 1862 and the second method processing means B1844, 1865. That is, in this case, adjustment may be made between the first method processing means B1842, 1862 and the second method processing means B1844, 1865 to match one process with the other process.

For example, in FIGS. 118C to 118F, the first method processing means A1831, 1851, 1861 and the second method processing means A1853, 1864, which perform the processing of the previous stage, receive and receive the broadcast signal in which the multiplexed data is modulated. This is the first processing unit that performs a process of demodulating the broadcast signal and outputting the multiplexed data obtained by the demodulation. The multiplexing data referred to here is based on at least the first multiplexing data of the first multiplexing data of the first multiplexing method and the second multiplexing data of the second multiplexing method different from the first multiplexing method. It is composed data.

The method conversion means 1832, 1843, 1852, 1863 secondly uses the first multiplexing data multiplexing method among the multiplexing data output by the first method processing means A1831, 1851, 1861 as the first processing unit. It is a conversion unit that converts to a multiplexing method and outputs the conversion data obtained by the conversion.

Further, for example, in FIGS. 118C to 118F, the second method processing means B1833, 1844, 1854, 1865, which performs the processing after the method conversion means 1832, 1843, 1852, 1863, are the method conversion means 1832, 1843, 1852. It is a second processing unit that performs a decoding process for decoding the converted data output by 1863 and outputs the decoded data obtained by the decoding process.

The receiving device having the method conversion means 1832, 1843, 1852, 1863 as described above has the following effects.

For example, by having the method conversion means 1832, 1843, 1852, 1863, it is not necessary to perform all of the subsequent processing by the first method processing means B, and at least a part of the subsequent processing is performed by the second method processing means. Processing is feasible. For example, when only the second method processing means B is provided as an existing implementation, it is not necessary to newly implement the first method processing means B. Alternatively, for example, when the second method processing means B is a device (hardware or software) having high performance, only the signal requiring high-performance processing is converted by using the method conversion means to perform the second method. By processing by the method processing means B and processing only the signal that does not require high-performance processing by the first method processing means B, the first method processing means B does not require high performance. Therefore, there is an effect of reducing the circuit scale and cost reduction can be realized. Here, the second method processing means B that requires high-precision processing is implemented by using hardware, and the first method processing means B that does not require high-performance processing is the second method processing means B. By implementing using the software in the above, it is possible to implement using hardware having only the function of the second method processing means B. In addition, by being able to flexibly realize various configurations, it is possible to implement effectively for compatibility with existing systems, circuit scale, and cost reduction.

FIG. 119 is a diagram showing a modified example of the configuration of the receiving device when the method conversion means is used.

Here, the first method will be described as the MMT / TLV method, and the second method will be described as the TS method.

As shown in FIG. 119, the receiving device 1900 includes a first method processing means A1910, a method conversion means 1920, and a method processing means B1930.

The processing means A1910 of the first method receives an MMT / TLV method signal, performs processing such as TLV Demux, and transfers a part of the signal of the MMTP packet to the first method converting means 1921 as the first conversion unit. , The remaining part of the signal is output to the second method conversion means 1922 as the second conversion unit.

The method conversion means 1920 includes a first method conversion means 1921, a second method conversion means 1922, and a second method multiplexing means 1923 as a multiplexing unit. The first method conversion means 1921 stores the MMTP packet in the TS packet while maintaining the data structure, and outputs the data to the second method multiplexing means 1923. That is, the first method conversion means extracts the first data of a part of the first multiplexed data, and the first packet (MMTP packet) constituting the first data is used in the second multiplexing method. The first conversion stored in the second packet (TS packet) is performed, and the first conversion data composed of the second packet obtained by the first conversion is output.

Further, the second method conversion means 1922 extracts data from the other MMT packet, converts it into a TS packet, and outputs the data to the second method multiplexing means. That is, the second method conversion means 1922 extracts the first packet (MMTP packet) constituting the second data of a part of the remaining part of the first multiplexed data, and the extracted first packet is second multiplexed. The second conversion to be converted into the second packet (TS packet) of the method is performed, and the second conversion data composed of the second packet obtained by the second conversion is output.

Here, the first data and the second data may be data of different media. For example, the first data may be data indicating video / audio, and the second data may be data indicating subtitles, applications, and the like.

Then, the second method multiplexing means 1923 multiplexes and outputs the TS packet format data output by the first method converting means 1921 and the second method converting means 1922 by using the TS method. That is, the second method multiplexing means 1923 performs the multiplexing process for multiplexing the output first conversion data and the second conversion data.

As described above, the method conversion means 1920 outputs a signal on one interface by multiplexing using either the first method or the second method. That is, the method conversion means 1920 outputs the data obtained by the multiplexing process as conversion data.

The second method processing means B1920 includes a second method demultiplexing means 1931 as a demultiplexing unit, a first method processing means B1932, and a second method processing means B1933. The second method demultiplexing means 1931 receives the signal multiplexed by the TS method and demultiplexes it. That is, the second method demultiplexing means 1931 performs demultiplexing processing in which the conversion data output by the method conversion means 1920 is demultiplexed into the first conversion data and the second conversion data.

Further, the second method demultiplexing processing means 1931 converts the TS packet converted by the first method converting means 1921, which constitutes the first conversion data obtained by demultiplexing, into an MMTP packet. It is converted and output to the first method processing means B. Further, the second method demultiplexing processing means 1931 uses the TS packet converted by the second method converting means 1922, which constitutes the second conversion data obtained by demultiplexing, as a second method. Output to the method processing means B1933.

Then, the first method processing means B1932 processes the MMTP packet. Specifically, the first method processing means B1932 is composed of a first packet (MMTP packet) extracted from a second packet (TS packet) constituting the first conversion data obtained by the demultiplexing process. The first decoding process by the first multiplexing method is performed on the first data.

Further, the second method processing means B1933 processes the TS packet. Specifically, the second method processing means B1933 performs second multiplexing with respect to the second data configured by the second packet (TS packet) constituting the second conversion data obtained by the demultiplexing process. The second decoding process is performed by the conversion method.

The method processing means B1930 uses the first control information of the first decoded data and the second control information of the second decoded data to adjust one of the first control information and the second control information to match the other. By doing so, it may have an adjusting unit for adjusting the first decoded data and the second decoded data. Specifically, the first control information is the first reference clock information, the second control information is the second reference clock information, and the adjusting unit is one of the first reference clock information and the second reference clock information. The first decoded data and the second decoded data are synchronized by making an adjustment to match the other with the other.

As described above, the method processing means B1930 may output the first decoded data obtained by the first decoding process and the second decoded data obtained by the second decoding process as the decoded data.

FIG. 120 is a diagram showing a detailed example of the method conversion means.

As shown in FIG. 120, the first method processing means A2010 includes a tuner 2011, a demodulation means 2012, a TLV / IP / UDP Demux means 2013, a first method descramble means 2014, a control information analysis means 2015, and a first method. It is composed of a reference clock generating means 2016, an MMT Demox means 2017, and the like. The tuner 2011, demodulation means 2012, TLV / IP / UDP Demux means 2013, first method descramble means 2014, control information analysis means 2015, first reference clock generation means 2016, and MMT Demux means 2017 are shown in FIG. 115, respectively. Because it corresponds to the tuner 1601, demodulation means 1602, TLV / IP / UDP Demux means 1603, descramble means 1604, control information analysis means 1605, NTP clock generation means 1606, and MMT Demux means 1607 of the receiving device 1600 described in the above. The explanation is omitted.

The method conversion means 2020 includes media conversion means media conversion means (first media conversion means 2021 and second media conversion means 2022), clock generation means (second reference clock generation means 2023 and difference information generation means 2024). It is composed of a second method multiplexing means 2025, a second method scrambling means 2026, and the like.

The method conversion means 2020 may use different media conversion means for each medium. For example, in the method conversion means 2020, the first medium (for example, video, audio) uses the second method conversion means 1922 of the method conversion means 1920 described with reference to FIG. 119, and the second medium (for example, subtitles). May be configured to use the first method conversion means 1921 of the method conversion means 1920.

Further, the method conversion means 2020 generates and multiplexes the difference information between the first reference clock information and the second reference clock in the reference clock information conversion means and the difference information generation means. That is, the adjustment unit described above may be provided by the method conversion means 2020.

Hereinafter, a specific example of the method conversion means for converting the signal received by the MMT / TLV method into the TS method will be described.

(Video, audio)
A case where the method conversion means converts the video data and the audio data from the first method to the second method will be described.

When converting in the second method conversion means 1922, the video signal and the audio signal stored in the input MMTP packet are converted into each access unit, an elemental stream (ES) is generated, and then a PES header is added. By doing so, it is repacketized. Specifically, the second method conversion means 1922 extracts the data stored in the MMTP packet, divides the data when the MFU is aggregated, and receives all the data when the MFU is fragmented. Join.

When the MFU of the video signal is a NAL unit unit, a plurality of NAL units are further combined to form an access unit unit. Here, when the video signal is transmitted with the asset type'hvc1', a non-VCL NAL unit is acquired from the MFU to form an access unit, and when the video signal is transmitted with the asset type'have1', the video signal is transmitted. , The non-VCLNAL unit is acquired from the separately transmitted MPU metadata to configure the access unit. Further, in the case of the NAL size format, the 4-byte NAL size area is deleted and a byte start code is added to convert the 4-byte NAL size area into a byte stream for output.

When the MFU of the audio signal is in the frame unit, the MFU is set in the access unit unit. When the MFU is in the LATM / LOAS stream format and is in the AudioMaxConfig () format, synchronization information and length information may be added and converted to the AudioSyncStream () format, or when the MFU is in the Raw data stream format. May separately acquire AudioMaxSync () from MPU metadata, control information, or the like and convert it into a LATM / LOAS stream format. It may be converted to other stream formats.

(Timestamp information)
A case where the method conversion means converts the time stamp information from the first method to the second method will be described.

Time stamp information (decoding time or presentation time, or information for calculating decoding time and presentation time) such as video and audio is acquired from the received MMTP packet and converted into PTS and DTS for each access unit. For example, in the control information analysis means, the presentation time of the first access unit is acquired from the MPU time stamp descriptor in the presentation order in the MPU, and the PTS of all the access units in the MPU is also used in combination with the MPU extended time stamp descriptor. And calculate the DTS. The calculated DTS and PTS are stored in the header of the PES packet including the corresponding video or audio access unit, and the PES packet is generated and output. In the first method conversion means 1921, the MMTP packet in which the synchronous MPU / MFU is stored may be stored in the PES packet or the TS packet and output without converting to the PES packet format. As a transmission method at the time of output, it may be stored in a PES packet as a private stream, stored in a private area of a TS packet and transmitted, may be sectioned as control information, or may be stored in an IP packet. It may be stored and transmitted via Ether, or may be any transmission means such as transmission using a communication means such as USB or I2C.

(Reference clock information)
A case where the method conversion means converts the reference clock information from the first method to the second method will be described. For example, the operations of the first reference clock generation means 2016, the second reference clock generation means 2023, and the difference information generation means 2024 shown in FIG. 120 will be described.

In the MMT / TLV method, the time stamp is based on the transmission system clock synchronized with the UTC, and the time stamp value based on the system clock is separately stored in the NTP packet as the reference clock information and transmitted in order to notify the system clock information. To. The receiving device reproduces the receiver system clock based on the reference clock information stored in the NTP packet in the first reference clock generating means 2016 of the first method processing means A2010. The method conversion means 2020 converts the system clock information (first reference clock information) reproduced based on the NTP packet into STC (second reference clock information), and converts the time stamp (PCR) into the adaptation field of the TS packet. It may be stored in the NTP packet and output, or the time stamp value of the NTP packet may be directly converted into a PCR value and stored in the TS packet for output. For example, the method conversion means 2020 converts NTP with a power of 2 accuracy to PCR with an accuracy of 27 MHz. Further, for example, the method conversion means 2020 extracts a specific 33 bits from a 64-bit NTP and uses it as PCR.

(Leap second adjustment of reference clock information)
Next, the leap second adjustment method when the leap second adjustment is performed for the reference clock information stored in the NTP packet will be described. When a leap_indicator indicating leap second adjustment is indicated in the NTP packet, the receiving device corrects the PTS and DTS based on the mpu_presentation_time_leap_indicator stored in the MPU extended time stamp descriptor, and then sets the corrected PTS and DTS in the PES packet header. Store in. Alternatively, the mpu_presentation_time_leap_indicator information may be separately generated and output without correcting the PTS and DTS, or the information indicating that leap second correction is required for the PTS and DTS for each access unit may be generated and output. good. The method processing means B1930 that received the PTS and DTS that is not leap second corrected corrects the PTS and DTS based on the information indicating that the leap second correction is necessary. At the time of leap second adjustment, the system clock of the receiver may also perform leap second adjustment of ± 1 second at the same time, or the time stamp may be corrected without performing leap second adjustment. If a ± 1 second discontinuity occurs in the STC during leap second adjustment, enable the PCR discontinuity flag in the TS packet header. The receiving device may independently generate a 27 MHz system clock (STC) and transmit the time stamp value as PCR. In that case, PTS and DTS of video and audio are added and stored in the PES packet header based on the STC generated independently. Even if a leap_indicator indicating leap second adjustment is indicated in the NTP packet, the PTS and DTS based on the independently generated system clock may not perform leap second adjustment.

When each of the first method processing means B1932 and the second method processing means B1933 is provided as in the method processing means B1930 of FIG. 119, the receiving device independently generates the NTP clock generated based on the NTP packet. It has two clocks of the system clock (STC), and the difference information generation means may generate the difference information indicating the time correspondence relationship between the clocks. In this case, the generated difference information and the time stamp (PCR) based on the STC independently generated are multiplexed and output. Method Processing means B1930 receives PCR and regenerates STC. Since the media processed by the second method processing means B1933 is provided with PTS and DTS based on STC (second reference clock information), decoding is performed at the timing when the PTS and DTS match the second reference clock information. Alternatively, the presentation is carried out. On the other hand, since the media processed by the first method processing means 1910 is provided with PTS and DTS based on NTP (first reference clock information), it corresponds to the second reference clock information based on the difference information. It is converted into a time, and decoding or presentation is performed at a timing that matches the second reference clock information. When the leap second adjustment is performed on the NTP clock, a leap_indicator indicating that the leap second adjustment is performed may be output together with the difference information.

Further, as a transmission method at the time of output, it may be stored in a PES packet as a private stream, stored in a private area of a TS packet and transmitted, may be sectioned as control information, or may be IP. It may be stored in a packet for Ether transmission, or may be any transmission means such as transmission using a communication means such as USB or I2C. In these cases, an identifier that can identify the information indicating the time correspondence between the clocks or the information indicating the leap second correction flag is assigned and output. The reference_start_time in the subtitles and the UTC-NPT reference descriptor in the event message also have a leap second flag, and leap second information and difference information are generated and output using the same method.

When generating a PES packet, if the access unit and the data reception timing of the time stamp information of the access unit are different, one of the data is accumulated and the PES packet is generated when both data are collected. .. For example, when it is guaranteed that the time stamp information of a certain access unit can be received N seconds before the decryption time of the access unit, the access unit is accumulated up to N seconds before the decryption time of the access unit, and the access unit is accumulated. A PES packet is generated and output N seconds before the decryption time of the access unit.

(subtitles)
A case where the method conversion means converts the subtitle / character supermarket stored in the input MMTP packet from the first method to the second method by storing it in the TS packet will be described.

The input data is, for example, a format in which caption data encoded by TTML is stored in a synchronous MPU / MFU and stored in an MMTP packet. The second method conversion means 1922 extracts TTML data from the MMTP packet and converts it into a section format in the TS method. The section format is output in the data carousel method.

The first method conversion means 1921 stores the MMTP packet in which the synchronous MPU / MFU is stored in the PES packet or the TS packet and outputs the packet without converting to the TS section format. As a transmission method at the time of output, it may be stored in a PES packet as a private stream, stored in a private area of a TS packet and transmitted, may be sectioned as control information, or may be stored in an IP packet. It may be stored and transmitted via Ether, or may be any transmission means such as transmission using a communication means such as USB or I2C.

(application)
A case where the method conversion means converts the application data stored in the input MMTP packet from the first method to the second method by storing the application data in the TS packet will be described.

The input data is, for example, a format in which application data encoded by HTML5 is stored in an asynchronous MPU / MFU and stored in an MMTP packet. The second method conversion means 1922 extracts HTML data from the MMTP packet and converts it into a section format in the TS method. The section format is output in the data carousel method.

The first method conversion means 1921 stores the MMTP packet in which the asynchronous MPU / MFU is stored in the PES packet or the TS packet and outputs the packet without converting to the TS section format. As a transmission method at the time of output, it may be stored in a PES packet as a private stream, stored in a private area of a TS packet and transmitted, may be sectioned as control information, or may be stored in an IP packet. It may be stored and transmitted via Ether, or may be any transmission means such as transmission using a communication means such as USB or I2C.

Control information related to application transmission (for example, MH-AIT, application control information descriptor, data transmission message, etc.) is also converted into TS format and output. The control information input in the MMT message format may be converted into the TS section format and output, the control information input in the MMT message format may be stored in the PES packet as a private stream, or the TS packet may be stored. It may be stored in the private area of and transmitted. It should be noted that only the M2 section message may be converted into the TS section format and output, and the other messages may be output without being converted into the TS section format.

(Control information)
The control information that is the entry point of the service includes PAT and PMT in the TS method, PLT, MPT, AMT and the like in the MMT / TLV method.

If there is a component that uses the second method conversion means 1922, the control information is converted so as to include the information of the component. For example, when video and audio are converted using the second method conversion means 1922 and subtitles and applications are converted using the first method conversion means 1921, service information and video and audio are transmitted from the MPT. Converts the information in PMT. At the same time, a PAT indicating PMT location information is generated and output.

If there is a component that uses the first method conversion means 1921, the control information is stored in a PES packet or a TS packet and output.

(Packet header conversion)
In the header part of the MMTP packet, a packet counter indicating the continuity of the MMTP packet, a packet sequence number indicating the continuity of each asset, a fragment counter indicating the continuity of the divided MFU, and a fragment indicator are shown. When a packet loss due to a transmission error or the like is detected by monitoring the field indicating these continuities shown in the input MMTP packet, the continuity index (continuity counter) in the TS packet is set to a discontinuous value. Output. The transport error indicator in the TS packet may be set to 1 to generate invalid data and output, or a null TS packet may be inserted and output in order to match the transmission rate. Alternatively, when packet loss occurs, the data may be restored by copying the previous data, for example. The restored data is stored in PES packets and TS packets and output. In this case, a continuous value is set for the continuity index and output.

The packet ID (16 bits) shown in the MMTP packet is converted into the PID (13 bits) of the TS packet. For example, a rule or table for associating the usable area of the packet ID and the PID is prepared in advance, and conversion is performed according to the table. When the MMTP packet is stored in the PES packet or TS packet as private data for each MMTP packet header by using the first method conversion means 1921, the TS packet is stored regardless of any data stored in the MMTP packet. The PID in the above is output with a PID indicating that the MMTP packet is stored as it is while maintaining the data structure of the MMTP packet. When receiving the output TS packet, if the TS packet is given a PID indicating that the data structure of the MMTP packet is stored, the MMTP packet is extracted from the TS packet and processed by MMTP Demux. To carry out.

In the case of video and audio, the payload unit start indicator is set to 1 for the TS packet including the beginning of the access unit. In the MMTP packet for transmitting video, the head of the NAL unit is identified by the payload_indicator of 00 or 01, and the head of the NAL unit is identified as the MMTP packet including the head of the access unit by the header type of the AU delimiter. By doing so, the payload unit start indicator is set to 1 for the first packet when the MMTP packet is converted into a TS packet.

In the MMTP packet that transmits voice, the head of the access unit is identified by the fragment_indicator of 00 or 01, and the payload unit start indicator is set to 1 for the head packet when the MMTP packet is converted into a TS packet. ..

If the MMTP packet is scrambled, it is descrambled and re-scrambled by the scramble method in the TS method.

When scrambling an MMTP packet, a multi-type extension header is added to the extension header area in the MMTP packet, and if the multi-extension header type is an extension header for scrambling, whether it is scrambled or not, and the scramble key is an even number. Information on whether it is a key or an odd key is stored.

When directly converting the MMTP packet header to the TS packet header, the scramble control bit in the extension header of the MMTP packet is converted into the transport scramble control bit in the TS packet. If the extension header for scrambling is not shown in the MMTP packet, the transport scramble control bit in the TS packet is set to 0 as no scrambling.

When re-scrambled, the first scramble method may be used instead of the second scramble method.

FIG. 121 is a diagram showing a reception processing flow for receiving using the receiving device shown in FIG. 118F.

The receiving device 1860 receives signals transmitted by a plurality of different methods (a signal transmitted by the first method and a signal transmitted by the second method) (S3701).

Next, the receiving device 1860 determines whether the received signal is the first method or the second method (S3702).

When the received signal is the second method (second method in S3702), the receiving device 1860 executes the process by using the second method processing means A1864 (S3703).

On the other hand, when the received signal is the first method (first method in S3702), the receiving device 1860 executes the process by using the first method processing means A1861 (S3705).

Next, the receiving device 1860 determines whether the output of the first method processing means A1861 is processed by the first method processing means B1862 or the second method processing means B1865 (S3706). ..

When the receiving device 1860 determines that processing is performed using the second method processing means B1865 (Yes in step S3706), the method conversion means 1863 converts the first method into the second method (S3707).

Then, the receiving device 1860 converted the signal of the second method processed by the second method processing means A1864 and the signal of the first method by the method conversion means 1863 into the signal of the second method. All of the signals are processed by using the second method processing means B1865 (S3704).

On the other hand, when the receiving device 1860 determines that the processing is performed by using the first method processing means B1862 (No in step S3706), the receiving device 1860 executes the processing by using the first method processing means B1962 (S3708).

As described above, in the receiving device 1860 that receives the signals of a plurality of different methods (first method and second method), by having the method conversion means 1863, the signal of any method can be the second method. The processing can be executed in the processing means B1865.

If there is no signal to execute processing using the first method processing means B, step S3706 and step S3708 may be omitted, and the output of step S3705 may be connected to step S3707 as a reception processing flow.

FIG. 122 is a diagram showing a processing flow of the method conversion means 1920 in FIG. 119.

The method conversion means 1920 acquires a signal output from the first method processing means A1910 (S3801).

Next, it is determined whether or not the acquired signal is a signal to be processed by the second method processing means B1933 (S3802).

When the method conversion means 1920 is a signal processed by the second method processing means B1933 (Yes in step S3802), in the second method conversion means 1922, the packet of the first method is converted into the packet of the second method. Convert to a packet (S3803). Then, the method conversion means 1920 assigns an identifier for identifying a component, control information, or the like (S3804). That is, the second method conversion means 1922 assigns a second identifier indicating that it is the second packet (TS packet) obtained by the second conversion to the second packet of the second conversion data.

On the other hand, when the method conversion means 1920 is not a signal to be processed by the second method processing means B1933 (No in step S3802), the first method conversion means 1921 converts the packet of the first method into the second method. Store in a packet (S3805). Further, the method conversion means 1920 assigns an identifier indicating that the packet of the first method is stored in the packet of the second method (S3806). That is, the first method conversion means 1921 assigns the first identifier indicating that it is the second packet (TS packet) obtained by the first conversion to the second packet of the first conversion data.

The second method multiplexing means 1923 is packetized by the second method from step S3801 to step S3806, and the packet to which the identifier is given is multiplexed and output to the method processing means B1930 (S3807).

FIG. 123 is a diagram showing a processing flow of the method processing means B1930 in FIG. 119.

The method processing means B1930 acquires a signal multiplexed by the second method output from the method conversion means 1920 (S3901).

Next, the method processing means B1930 determines from the identifier of the packet whether or not the packet of the first method is a stored packet (S3902).

When the method processing means B1930 is a packet in which the packet of the first method is stored (Yes in step S3902), the method processing means B1930 extracts the packet of the first method and executes the processing by the first method processing means B1932 (Yes). S3903).

On the other hand, when the method processing means B1930 is not a packet in which the packet of the first method is stored (No in step S3902), the method processing means B1930 extracts the packet of the second method and executes the processing by the second method processing means B1933. (S3904).

In addition, the second method processing means B1833, 1844, 1854, 1865 in the receiving devices 1830 to 1860 of FIGS. 118C to 118F may be configured to include a second method transmitting means as a retransmission unit (not shown). A retransmitting device may be configured by using a replaced configuration. In this case, the signal received by the first method can be retransmitted by the second method.

That is, the configuration may include a retransmission unit that retransmits the conversion data output by the method conversion means 1832, 1843, 1852, 1863 as the conversion unit to another receiving device.

It should be noted that the second method processing means B1833, 1844, 1854, 1865 in the receiving devices 1830 to 1860 of FIGS. 118C to 118F are configured to include or replace the second method storage means as a storage unit (not shown). The storage device may be configured by the configuration. In this case, the signal received by the first method can be stored by the second method.

That is, the configuration may include a storage unit that stores the conversion data output by the method conversion means 1832, 1843, 1852, 1863 as the conversion unit in the storage device. The storage device is realized by, for example, an auxiliary storage device composed of a hard disk drive, a non-volatile memory, and the like.

Further, in the receiving devices 1830 to 1860 of FIGS. 118C to 118F, it is said that the processing is performed by two processing means, that is, the processing on the front stage side and the processing on the rear stage side, but it may be composed of three or more processing means. I also explained. That is, as described above, when there is a retransmission unit, the configuration does not include the second method processing means B1833, 1844, 1854, 1865 and the first method processing means 1842, 1862 on the rear stage side. May be good. That is, the second method processing means B1833, 1844, 1854, 1865 and the first method processing means 1842, 1862 on the latter stage side may be configured to be provided in another receiving device.

Specifically, the signal transmitted from the retransmitting device of the second method and the signal reproduced from the storage device of the second method are the second method processing means B1833, 1844 provided in the other receiving device. It can be reproduced at 1854 and 1865. Further, the storage device also has a retransmission unit, and the conversion data stored in the storage device may be retransmitted to another receiving device. In addition to the method of accumulating the signal after processing by the method conversion means 1832, 1843, 1852, 1863, when accumulating the signal of the first method before processing by the method conversion means and reproducing the accumulated signal. In addition, it may be reproduced by using the method conversion means and the second method processing means.

[Supplement: Receiver]
The receiving device can also be configured as shown in FIG. 124. The receiving device can also be configured for FIG. 125. FIG. 124 is a diagram showing an example of a specific configuration of the receiving device. FIG. 125 is a diagram showing another example of a specific configuration of the receiving device.

The receiving device 2100 includes a first processing unit 2101 and a conversion unit 2102. Each of the first processing unit 2101 and the conversion unit 2102 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like. That is, each processing unit constituting the receiving device 2100 may be realized by software or hardware.

The receiving device 2200 includes a receiving unit 2201, a demultiplexing unit 2202, a first decoding unit 2203, a second decoding unit 2204, and an output unit 2205. Each of the receiving unit 2201, the demultiplexing unit 2202, the first decoding unit 2203, the second decoding unit 2204, and the output unit 2205 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like. That is, each processing unit constituting the receiving device 2200 may be realized by software or hardware.

A detailed description of each component of the receiving devices 2100 and 2200 will be given in the description of the corresponding receiving method, respectively.

First, the receiving method of the receiving device 2100 will be described with reference to FIG. 126. FIG. 126 is an operation flow (reception method) by the receiving device.

First, the first processing unit 2101 of the receiving device 2100 is the first multiplexing data of the first multiplexing method (MMT / TLV method) and the second multiplexing method (TS method) different from the first multiplexing method. (2) A broadcast signal in which the multiplexed data composed of at least the first multiplexed data of the multiplexed data is modulated is received, the received broadcast signal is demoted, and the multiplexed data obtained by demodating is output (. S4001).

Next, the conversion unit 2102 of the receiving device 2100 converts the multiplexing method of the first multiplexing data among the output multiplexing data into the second multiplexing method, and outputs the converted data obtained by the conversion. (S4002).

As a result, if the data converted into the second multiplexing method by the receiving device 2100 is used, the processing unit of the conventional TS method can be used as it is. Therefore, in the subsequent processing, sharing with the existing implementation can be easily realized and can be realized at low cost.

Next, the receiving method of the receiving device 2200 will be described with reference to FIG. 127. FIG. 127 is an operation flow (reception method) by the receiving device.

First, the receiving unit 2201 of the receiving device 2200 is a second packet (TS packet) in which the first packet (MMTP packet) constituting the first multiplexing data of the first multiplexing method (MMT / TLV method) is stored. Therefore, the first conversion data composed of the second packet used in the second multiplexing method (TS method) different from the first multiplexing method and the first packet are converted into the second multiplexing method. Receives the converted data multiplexed with the second converted data composed of the second packet obtained in (S4101).

Next, the demultiplexing unit 2202 of the receiving device 2200 performs a demultiplexing process of demultiplexing the converted data received by the receiving unit 2201 into the first converted data and the second converted data (S4102).

Next, the first decoding unit 2203 of the receiving device 2200 extracts the first packet from the second packet constituting the first conversion data obtained by the demultiplexing process, and the extracted first packet constitutes the first packet. The first decoding process by the first multiplexing method is performed on the first data (S4103).

Further, the second decoding unit 2204 of the receiving device 2200 uses the second multiplexing method for the second data formed by the second packet constituting the second converted data obtained by the demultiplexing process. 2 Decoding process is performed (S4104).

Finally, the output unit 2205 of the receiving device 2200 outputs the first decoded data obtained by the first decoding process and the second decoded data obtained by the second decoding process (S4105).

As a result, the receiving device 2200 can use the data converted to the second multiplexing method, so that, for example, the processing unit of the conventional TS method can be used as it is. Therefore, the receiving device 2200 can be easily shared with the existing mounting and can be realized at low cost.

(Other embodiments)
Although the transmitting device, the receiving device, the transmitting method and the receiving method according to the embodiment have been described above, the present invention is not limited to this embodiment.

Further, each processing unit included in the transmission device and the reception device according to the above embodiment is typically realized as an LSI which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip so as to include a part or all of them.

Further, the integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connection and settings of circuit cells inside the LSI may be used.

In each of the above embodiments, each component may be configured with dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

In other words, the transmitting device and the receiving device include a processing circuit and a storage device electrically connected to the processing circuit (accessible from the control circuit). The processing circuit includes at least one of dedicated hardware and a program execution unit. Further, when the processing circuit includes a program execution unit, the storage device stores the software program executed by the program execution unit. The processing circuit uses the storage device to execute the transmission method or the reception method according to the above embodiment.

Further, the present invention may be the software program or a non-temporary computer-readable recording medium in which the program is recorded. Needless to say, the above program can be distributed via a transmission medium such as the Internet.

In addition, the numbers used above are all exemplified for the purpose of specifically explaining the present invention, and the present invention is not limited to the exemplified numbers.

In addition, the division of functional blocks in the block diagram is an example, and multiple functional blocks can be realized as one functional block, one functional block can be divided into multiple, and some functions can be transferred to other functional blocks. You may. Further, the functions of a plurality of functional blocks having similar functions may be processed by a single hardware or software in parallel or in a time division manner.

Further, the order in which the steps included in the above transmission method or reception method are executed is for exemplifying the present invention in detail, and may be an order other than the above. Further, a part of the above steps may be executed simultaneously with other steps (parallel).

Although the transmitting device, the receiving device, the transmitting method and the receiving method according to one or more aspects of the present invention have been described above based on the embodiments, the present invention is not limited to this embodiment. No. As long as it does not deviate from the gist of the present invention, one or more of the present embodiments may be modified by those skilled in the art, or may be constructed by combining components in different embodiments. It may be included within the scope of the embodiment.

The present invention can be applied to a device or device that performs media transport such as video data and audio data.

15, 100, 300, 500, 700, 900, 1200, 1400 Transmitter 16, 101, 301 Coding unit 17, 102 Multiplexing unit 18, 104 Transmitter unit 20, 200, 400, 600, 800, 1000, 1300, 1500, 1810, 1820, 1830, 1840, 1850, 1860, 2100, 2200 Receiver 21 Packet filtering unit 22 Transmission order type determination unit 23 Random access unit 24, 212 Control information acquisition unit 25 Data acquisition unit 26 Calculation unit 27 Initialization Information acquisition unit 28, 206 Decoding command unit 29, 204A, 204B, 204C, 204D, 402 Decoding unit 30 Presentation unit 201 Tuner 202 Demodulation unit 203 Demultiplexing unit 205 Display unit 211 Type discrimination unit 213 Slice information acquisition unit 214 Decoding data Generation unit 302 Grant unit 303, 503, 702, 902, 1202, 1402 Transmission unit 401, 601, 801, 1001, 1301, 1501, 2201 Reception unit 501 Division unit 502, 603 Configuration unit 602 Judgment unit 701, 901, 1201, 1401 Generation unit 802 First buffer 803 Second buffer 804 Third buffer 805 Fourth buffer 806 Decoding unit 1002 Playback unit 1302 Calculation unit 1303 Correction unit 1502 Judgment unit 1811, 1821, 1831, 1841, 1851, 1861 Method processing means A of 1
1812, 1822, 1842, 1862 First method processing means B
1823, 1853, 1864 Second method Processing means A
1824, 1833, 1844, 1854, 1865 Second method processing means B
1832, 1843, 1852, 1863 Method conversion means 2101 1st processing unit 2102 Conversion unit 2202 Demultiplexing unit 2203 1st decoding unit 2204 2nd decoding unit 2205 Output unit

Claims (4)

The multiplexing data composed of at least the first multiplexing data of the first multiplexing data of the first multiplexing method and the second multiplexing data of the second multiplexing method different from the first multiplexing method is modulated. A first processing unit that receives the broadcast signal, demolishes the received broadcast signal, and outputs the multiplexed data obtained by demodulating the received broadcast signal.
Among the output multiplexing data, the first multiplexing data multiplexing method is converted to the second multiplexing method, and a conversion unit for outputting the converted data obtained by the conversion is provided .
The conversion unit
A first conversion is performed in which the first data, which is a part of the first multiplexing data, is extracted and the first packet constituting the first data is stored in the second packet used in the second multiplexing method. A first conversion unit that outputs the first conversion data composed of the second packet obtained by the first conversion.
A second packet that extracts the first packet constituting the second data of the remaining part of the first multiplexing data and converts the extracted first packet into the second packet of the second multiplexing method. A second conversion unit that performs conversion and outputs second conversion data composed of the second packet obtained by the second conversion.
It has a multiplexing unit that performs a multiplexing process for multiplexing the output first conversion data and the second conversion data.
A receiving device that outputs the data obtained by the multiplexing process as the converted data. It is a second packet in which the first packet constituting the first multiplexing data of the first multiplexing method is stored, and is composed of a second packet used in a second multiplexing method different from the first multiplexing method. Receives the conversion data in which the first conversion data to be generated and the second conversion data composed of the second packet obtained by converting the first packet to the second multiplexing method are multiplexed. Receiver and
A demultiplexing unit that performs demultiplexing processing that demultiplexes the converted data received by the receiving unit into the first conversion data and the second conversion data.
The first packet is extracted from the second packet constituting the first conversion data obtained by the demultiplexing process, and the first data is composed of the extracted first packet. The first decoding unit that performs the first decoding process in the multiplexing method,
A second decoding unit that performs a second decoding process in the second multiplexing method on the second data configured by the second packet constituting the second conversion data obtained by the demultiplexing process. When,
A receiving device including an output unit for outputting the first decoded data obtained by the first decoding process and the second decoded data obtained by the second decoding process. The multiplexing data composed of at least the first multiplexing data of the first multiplexing data of the first multiplexing method and the second multiplexing data of the second multiplexing method different from the first multiplexing method is modulated. Receives the broadcast signal, demodulates the received broadcast signal, and outputs the multiplexed data obtained by demodulating.
Of the output multiplexing data, the multiplexing method of the first multiplexing data is converted to the second multiplexing method, and the converted data obtained by the conversion is output.
In the output of the conversion data,
A first conversion is performed in which the first data, which is a part of the first multiplexing data, is extracted and the first packet constituting the first data is stored in the second packet used in the second multiplexing method. Then, the first conversion data composed of the second packet obtained by the first conversion is output.
A second packet that extracts the first packet constituting the second data of the remaining part of the first multiplexing data and converts the extracted first packet into the second packet of the second multiplexing method. The conversion is performed, and the second conversion data composed of the second packet obtained by the second conversion is output.
A multiplexing process for multiplexing the output first conversion data and the second conversion data is performed.
A receiving method for outputting the data obtained by the multiplexing process as the converted data. It is a second packet in which the first packet constituting the first multiplexing data of the first multiplexing method is stored, and is composed of a second packet used in a second multiplexing method different from the first multiplexing method. Receives the conversion data in which the first conversion data to be generated and the second conversion data composed of the second packet obtained by converting the first packet to the second multiplexing method are multiplexed. ,
A demultiplexing process is performed in which the received conversion data is demultiplexed into the first conversion data and the second conversion data.
The first packet is extracted from the second packet constituting the first conversion data obtained by the demultiplexing process, and the first data is composed of the extracted first packet. Perform the first decoding process in the multiplexing method,
The second decoding process by the second multiplexing method is performed on the second data composed of the second packet constituting the second conversion data obtained by the demultiplexing process.
A receiving method for outputting the first decoded data obtained by the first decoding process and the second decoded data obtained by the second decoding process.

Images