JP4596693B2 - Streaming method and system for executing the same - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a streaming method, and more particularly to a streaming method in which a server transmits multimedia data to a terminal through the Internet, and the terminal reproduces the data while receiving the data.
[0002]
[Prior art]
(Description of multimedia data encoding and compression method and buffer model)
There are various types of multimedia data used for transmission on the Internet, such as moving images, still images, audio, text, and data in which they are multiplexed. In the video, H. Coding compression methods such as H.263, MPEG1, 2, 4 are well known, JPEG is used for still images, MPEG audio is used for audio, and G.264 is used for audio. 729 and so on.
[0003]
In the present invention, since the focus is on streaming reproduction, moving images and audio are targets for transmission. Here, an MPEG video that is a representative of the moving image compression method, particularly an MPEG1 (ISO / IEC 11172) video and an MPEG2 (ISO / IEC 13818) video that have a relatively simple mechanism will be described.
[0004]
MPEG video mainly has the following two features in order to realize highly efficient data compression. First, in compression of moving image data, a compression method using temporal correlation characteristics between frames is adopted in addition to a compression method using spatial frequency characteristics which has been conventionally performed. In MPEG, each frame (also referred to as a picture) constituting a stream is divided into an I frame (an intra-frame encoded picture), a P frame (a picture using intra-frame encoding and a reference relationship from the past), a B frame ( Data compression is performed by classifying into three types (pictures using intra-frame coding and past and future reference relationships). In these three types, the I frame is the largest (that is, the amount of information is the largest), followed by P and B. Although greatly dependent on the compression algorithm, the ratio of the information amount is approximately I: P: B = 4: 2: 1. In general, an MPEG video stream includes 15 frames (= 1 GOP) as a unit, and 1 GOP includes 1 I frame, 4 P frames, and 10 B frames.
[0005]
The second feature of MPEG video is that dynamic code amount allocation according to image complexity can be performed in units of pictures. An MPEG decoder includes a decoder buffer. By decoding data after storing the data in the decoder buffer, a large amount of code can be assigned to a complex image that is difficult to compress. In moving image compression, not limited to MPEG, the standard decoder buffer capacity is mostly defined by the standard. In the case of MPEG1 or MPEG2, the standard decoder buffer has a capacity defined as 224 Kbytes in the standard, and the MPEG encoder must generate picture data so that the decoder buffer occupancy transitions within this capacity.
[0006]
19A to 19C are diagrams for explaining a conventional streaming method. FIG. 19A is a diagram showing a video frame, FIG. 19B is a diagram schematically showing a transition of buffer occupancy, and FIG. 19C is a diagram showing a configuration example of a conventional terminal. . In FIG. 19C, the terminal includes a video buffer, a video decoder, an I / P rearrangement buffer, and a switch. The video buffer corresponds to the decoder buffer described above, and the transferred data is stored in the video buffer and then decoded by the video decoder. The decoded data is rearranged in order of reproduction time through the I, P rearrangement buffer and the switch.
[0007]
In FIG. 19B, the vertical axis represents the buffer occupation amount (data accumulation amount of the video buffer), the horizontal axis represents time, and the thick line in the figure represents the temporal transition of the buffer occupation amount. The slope of the thick line corresponds to the video bit rate and indicates that data is input to the buffer at a constant rate. Further, the buffer occupancy is reduced at regular intervals (33.3667 msec). This is because the data of each frame is decoded at regular intervals. In addition, the intersection of the oblique dotted line and the time axis indicates the time when the data in each video frame starts to be transferred to the video buffer. Accordingly, the transfer start time of frame X shown in FIG. 19A is t1, and the transfer start time of frame Y is t2.
[0008]
19A and 19B, from the time t1 when the start frame X of the video starts to be input to the video buffer to the time when decoding is first executed (the first falling position of the bold line in the figure). Is generally called vbv_delay time. Since the first decoding is performed at the moment when the video buffer is full, the vbv_delay time is usually the time from the start of data input until the video buffer with a capacity of 224 Kbytes is full. This is the initial delay time (waiting time for cueing) from the start until video playback is started through the decoder.
[0009]
When the frame Y in FIG. 19A is a complex image, as shown in FIG. 19B, the amount of code is large, so that it is earlier than the decoding time of frame Y (t3 in the figure). Data transfer to the video buffer must be started from time (t2 in the figure). However, the amount of pictures that occupy the buffer is within the allowable range of 224 Kbytes, no matter how complex the image is.
[0010]
If data is transferred to the video buffer so that the buffer transition shown in FIG. 19B is properly maintained, it is guaranteed by the MPEG standard that no streaming failure occurs due to an underflow or overflow of the video buffer. ing.
[0011]
(Description of network transfer jitter absorption receive buffer)
However, as shown in FIG. 20, when the server 201 and the terminal 202 are connected via the network 203 and the MPEG data in the storage 210 is distributed, the time for generating the packet by the generation module 211 and the network devices 204 and 205 The data transfer rate fluctuates due to the transfer procedure time, the transmission delay time accompanying the congestion of the network 203, and the like. Therefore, in reality, the buffer transition shown in FIG. 19B is not maintained. As a method for mitigating and absorbing such fluctuations (jitter) in the transfer rate, it is conceivable that content having a coding rate sufficiently smaller than the network bandwidth is first flowed. However, this method is not appropriate because it is necessary to provide high-quality video and audio using network resources as effectively as possible. Therefore, in general, the network devices 204 and 205 are provided with transmission / reception buffers 206 and 207 having appropriate capacities, respectively, so that data is normally transferred somewhat forward and there is no shortage when data transfer is delayed. A supplementary method is adopted.
[0012]
Here, providing the reception buffer 207 on the terminal 202 side means that, in the buffer transition of FIG. 19B, the upper limit of the buffer occupancy amount is increased from the 224 Kbytes that is the standard of the decoder buffer 208 to the accumulation amount by the reception buffer 207. It is roughly equivalent to raising the height by that amount. FIGS. 21A and 21B show the buffer occupancy before and after the addition of the reception buffer 297 side by side. Note that FIG. 21A shows the same buffer transition as in FIG. 19B.
[0013]
By adding the reception buffer 207, the allowable range of buffer transition is expanded. As a result, the buffer transition of FIG. 19B, that is, the buffer transition of FIG. 21A becomes as shown in FIG. Even if the transfer rate decreases, underflow can be avoided. On the other hand, the vbv_delay time is increased by a time corresponding to the accumulation amount by the reception buffer 207, and the start of decoding by the decoder 209 and the start of playback by the playback device 212 are delayed. That is, the cue time becomes longer by the time required for data storage in the reception buffer 207.
[0014]
[Problems to be solved by the invention]
As is apparent from the above, when streaming multimedia data such as MPEG in a network environment where reliability and transmission speed are guaranteed such as a small LAN, it is basically determined by the codec standard. As long as the system design properly observes the playback initial delay time (vbv_delay) and the decoder buffer transition, the decoder playback will not underflow or overflow, and streaming playback will not fail.
[0015]
However, in a wide area network environment such as the Internet, since the transfer jitter due to the transmission characteristic variation of the communication path is so large that it cannot be ignored, the conventional terminal 202 has a decoder buffer (vbv buffer) defined by the codec standard. In addition, there are many cases where a buffer for absorbing transfer jitter, such as the reception buffer 207 of FIG. At this time, the following problems exist.
[0016]
In general, the capacity of a buffer for absorbing jitter mounted on a terminal varies depending on the model. For this reason, even if the same data is distributed under the same conditions, streaming playback can be performed without failure on a model with a large buffer capacity. However, on a small model, there is a case in which it fails without absorbing jitter.
[0017]
In order to solve this problem, for example, the amount of memory installed in the terminal may be increased to ensure a sufficient buffer capacity for absorbing jitter. However, the amount of installed memory is one of the main factors that determine the price of a terminal, and there is a demand to suppress it as much as possible. In addition, if the buffer capacity for absorbing jitter is too large, a new problem arises that the cue time until the start of reproduction becomes long and the user is frustrated.
[0018]
Therefore, the object of the present invention is to avoid the failure of streaming playback due to buffer underflow or overflow even if the buffer capacity of the terminal varies depending on the model or the transmission capacity of the network fluctuates. Furthermore, it is to provide a streaming method capable of achieving both the avoidance of failure of streaming reproduction and the reduction of waiting time at the time of cueing.
[0019]
[Means for Solving the Problems and Effects of the Invention]
A first invention is a streaming method in which a server transmits stream data to a terminal through a network, and the terminal reproduces the stream data while receiving the stream data,
A target amount determination step in which the terminal determines a target amount of stream data to be stored in its buffer in relation to its buffer capacity and network transmission capability;
The delay time from when the terminal writes the head data of the stream to its own buffer until the data is read and playback is started, as long as it does not exceed the value obtained by dividing the buffer capacity by the transmission capacity Determine the delay time determination step,
The terminal notifies the server of the determined target time and delay time;
When the server transmits the stream data to the terminal through the network, the server includes a control step of controlling the transmission speed based on the notified target amount and delay time.
[0020]
In the first invention, the terminal determines a target amount according to its own buffer capacity and the transmission capacity of the network, and further within a range not exceeding a value obtained by dividing the buffer capacity by the transmission capacity, Determine the delay time. Since the server controls the transmission speed based on the target amount and delay time determined by the terminal in this way, even if the terminal buffer capacity varies depending on the model or the transmission capacity of the network fluctuates, the buffer amount and transmission capacity As a result, it is possible to avoid failure of streaming playback due to buffer underflow or overflow. In addition, since the delay time is determined independently of the target amount, it is possible to achieve both the avoidance of failure of streaming playback and the reduction of the waiting time at the time of cueing.
[0021]
Here, the reason why the delay time is limited to a value obtained by dividing the buffer capacity by the transmission capability is that there is a risk that streaming playback may fail if the delay time exceeds this value. As long as the value does not exceed this value, the delay time may be determined to any value. However, when determining the value, the balance between the tolerance to fluctuations in transmission capability and the waiting time at the time of cueing is considered.
[0022]
According to a second invention, in the first invention,
In the control step, the server
The transmission speed is controlled so that the amount of stream data stored in the buffer of the terminal changes without exceeding the target amount in the vicinity of the target amount.
[0023]
In the second aspect of the invention, since the accumulated amount transitions in the vicinity of the target amount without exceeding the target amount, the buffer underflow and overflow are unlikely to occur.
[0024]
According to a third invention, in the second invention,
In the control step, the server predicts and calculates the amount of stream data stored in the terminal buffer based on the transmission speed, the delay time, and the speed at which the terminal decodes the stream data. .
[0025]
In the third aspect, since the server predicts and calculates the accumulation amount and performs transmission speed control based on the amount, the accumulation amount can be shifted in the vicinity of the target amount so as not to exceed the target amount.
[0026]
Here, the terminal may notify the server of the current accumulation amount, and the server may perform transmission speed control based on the notification. However, in this case, since it takes time to transmit information from the terminal to the server, the server performs transmission speed control based on the past accumulation amount. Therefore, it is not always possible to make a transition so that the accumulated amount does not exceed the target amount in the vicinity of the target amount.
[0027]
According to a fourth invention, in the first invention,
A detection step in which the terminal detects that the transmission capability of the network has changed across a predetermined threshold;
A target amount changing step in which the terminal changes the target amount according to the detection result in the detecting step; and
The terminal further includes a step of notifying the server of the target amount after the change,
In the control step, when the server receives the notification of the changed target amount, the amount of stream data accumulated in the terminal buffer does not exceed the changed target amount in the vicinity of the changed target amount. The transmission speed is controlled so as to transit.
[0028]
In the fourth aspect, when the transmission capacity changes across the threshold, the target amount is changed by the terminal. The server follows the change in the target amount by controlling the transmission speed so that the change does not exceed the target amount after the change in the vicinity of the target amount after the change.
[0029]
A fifth invention is the fourth invention,
When detecting that the transmission capability of the network has decreased across the first threshold in the detection step, the terminal changes the direction to increase the target amount in the target amount change step,
In the control step, the server controls the transmission speed to increase in accordance with the increase of the target amount.
[0030]
In the fifth aspect, when the transmission capacity changes across the first threshold, the target amount is increased by the terminal. The server follows the increase in the target amount by increasing the transmission speed.
[0031]
According to a sixth invention, in the fifth invention,
The first threshold is a value approximately in the middle between the maximum realizable transmission capability and the transmission capability at which stream data transfer loss begins to occur.
[0032]
In the sixth aspect of the invention, when the transmission capability is decreasing, the transmission rate is increased to increase the accumulation amount before the transfer loss of stream data starts to occur. As a result, it is possible to prevent the streaming reproduction from failing when the transmission capability is reduced.
[0033]
According to a seventh invention, in the fourth invention,
When detecting that the transmission capability of the network has dropped across the second threshold value smaller than the first threshold value in the detection step, the terminal changes the direction to decrease the target amount in the target amount change step,
In the control step, the server controls the transmission speed to decrease in accordance with the target amount being changed in the decreasing direction.
[0034]
In the seventh aspect, when the transmission capacity changes across the second threshold, the target amount is decreased by the terminal. The server follows the decrease in the target amount by reducing the transmission rate.
[0035]
In an eighth aspect based on the seventh aspect,
The second threshold value is a value corresponding to a transmission capability at which a transfer loss of stream data starts to occur.
[0036]
In the eighth aspect of the invention, when the transmission capacity declines and stream data transfer loss begins to occur, the transmission speed is reduced. This is to prevent the lost data from being retransmitted.
[0037]
Here, when reducing the transmission rate, the server must skip frame transmission at a frequency corresponding to the decrease rate. When a frame is skipped, the quality of video and audio obtained by playing back the terminal is degraded. In order to suppress this deterioration in quality, in the ninth invention, a frame that is not in time for the reproduction time is selected as a skipped frame. In the following tenth invention, as a frame to be skipped, a frame with low importance and a frame with high importance but not in time for the reproduction time are selected.
[0038]
In a ninth aspect based on the eighth aspect,
In the target amount changing step, when the terminal changes in a direction to decrease the target amount, in the control step, the server sequentially compares the reproduction time of each frame constituting the stream to be transmitted with the current time, thereby reproducing the reproduction time. Is characterized by skipping transmission of frames older than the current time, thereby controlling the transmission rate in a decreasing direction.
[0039]
In the eleventh aspect of the invention, frames that do not meet the playback time are selectively skipped, so that deterioration in quality due to a decrease in transmission speed can be suppressed as compared with random skipping.
[0040]
In a tenth aspect based on the eighth aspect,
In the target amount changing step, when the terminal changes in a direction to decrease the target amount, in the control step, the server sequentially compares the importance of each frame constituting the stream to be transmitted with the reference value,
For frames whose importance is less than the reference value, all transmissions are skipped,
For frames whose importance is greater than or equal to the reference value, each playback time is sequentially compared with the current time, and transmission is skipped only when the playback time is older than the current time, thereby controlling the transmission speed in a decreasing direction. It is characterized by doing.
[0041]
In the tenth aspect of the invention, a frame with low importance and a frame with high importance but not in time for the reproduction time are selectively skipped, so the transmission rate is higher than when skipping randomly. It is possible to suppress the deterioration of the quality due to the decrease in the quality.
[0042]
Here, as in the tenth aspect of the invention, a method that considers the importance in addition to whether or not it is in time for the reproduction time when selecting a frame to be skipped is typically used for an MPEG video frame. Used. In this case, when the transmission speed is reduced, the P and B frames are skipped as low importance frames, while the I frame is skipped as a high importance frame unless it is not in time for the playback time. Therefore, quality degradation of the reproduced image due to a decrease in transmission speed can be minimized. In the case of an MPEG audio frame, there is no difference in importance between frames, so it is only necessary to consider whether or not it is in time for the playback time.
[0043]
An eleventh invention is a system comprising a server that transmits stream data through a network and a terminal that reproduces the stream data while receiving the data.
The terminal
A target amount determining means for determining a target amount of stream data to be accumulated in its own buffer in relation to its own buffer capacity and network transmission capability;
Arbitrarily determine the delay time from writing the head data of the stream to its own buffer until reading the data and starting playback within a range not exceeding the value obtained by dividing the buffer capacity by the transmission capacity Delay time determining means, and
Means for notifying the server of the determined target time and delay time;
The server includes control means for controlling the transmission speed based on the notified target amount and delay time when transmitting stream data to the terminal through the network.
[0044]
A twelfth aspect of the invention is a terminal that is used together with a server that transmits stream data over a network and that plays back the stream data while receiving the stream data.
The server is equipped with a control means for controlling the transmission speed based on the notified target amount and delay time when transmitting stream data to the terminal through the network,
A target amount determining means for determining a target amount of stream data to be accumulated in its own buffer in relation to its own buffer capacity and network transmission capability;
Arbitrarily determine the delay time from writing the head data of the stream to its own buffer until reading the data and starting playback within a range not exceeding the value obtained by dividing the buffer capacity by the transmission capacity Delay time determining means, and
Means for notifying the server of the determined target time and delay time.
[0045]
A thirteenth aspect of the invention is a server that is used together with a terminal that reproduces while receiving stream data, and that transmits the stream data through a network.
On the device,
A target amount determining means for determining a target amount of stream data to be accumulated in its own buffer in relation to its own buffer capacity and network transmission capability;
Arbitrarily determine the delay time from writing the head data of the stream to its own buffer until reading the data and starting playback within a range not exceeding the value obtained by dividing the buffer capacity by the transmission capacity Delay time determining means, and
A means to notify the server of the determined target time and delay time is provided.
When transmitting stream data to the terminal through the network, the control means for controlling the transmission speed based on the notified target amount and delay time,
The control means controls the transmission speed so that the amount of stream data accumulated in the buffer of the terminal changes without exceeding the target amount in the vicinity of the target amount.
[0046]
A fourteenth invention is a program describing a streaming method as in the first invention.
[0047]
A fifteenth aspect of the invention is a recording medium on which a program like the fourteenth aspect of the invention is recorded.
[0048]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration example of a server client system that executes a streaming method according to an embodiment of the present invention. In FIG. 1, the system includes a server 101 and a terminal 102 that operates as a client thereof. Video and audio data are accumulated on the server 101 side. This data is data encoded and compressed by MPEG. In response to a request from the terminal 102, the server 101 packetizes the accumulated data and generates a stream. Then, the generated stream is transmitted to the terminal 102 through the network 103. The terminal 102 receives and decodes the stream transmitted from the server 101, and displays and outputs the obtained video and audio.
[0049]
FIG. 2 is a block diagram showing the configuration of the server 101 in FIG. In FIG. 2, the server 101 includes a storage device 411, a transmission / reception module 402, a generation module 405, a RAM 404, a CPU 412, and a ROM 413. The storage device 411 stores video and audio data. Data in the storage device 411 is given to the generation module 405. The generation module 405 includes a read buffer 407, a packet generation circuit 406, and a packet generation buffer 408. The generation module 405 packetizes given data to generate a stream.
[0050]
The transmission / reception module 402 includes a network controller 410 and a transmission buffer 409, and transmits the stream generated by the generation module 405 to the terminal 102 via the network 103. Also, information transmitted from the terminal 102 via the network 103 is received.
[0051]
Information from the terminal 102 received by the transmission / reception module 402 is written in the RAM 404. A server control program is stored in the ROM 413, and the CPU 412 executes the program in the ROM 413 while referring to information from the terminal stored in the RAM 404, thereby controlling the transmission / reception module 402 and the generation module 405. I do. Although the program is stored in the ROM 413 here, the program may be stored in a storage medium other than the ROM, such as a hard disk or a CD-ROM.
[0052]
FIG. 3 is a block diagram showing a configuration of the terminal 102 of FIG. In FIG. 3, the terminal 102 includes a transmission / reception module 507, a playback module 510, a display device 511, a ROM 502, and a CPU 503. The transmission / reception module 507 includes a network controller 506 and a reception buffer 505, and receives a stream transmitted from the server 101 via the network 103. Information from the CPU 503 is transmitted to the server 101 via the network 103.
[0053]
A stream received by the transmission / reception module 507 is input to the reproduction module 510. The reproduction module 510 includes a decoder buffer 508 and a decoder 509, and decodes and reproduces an input stream. The data reproduced by the reproduction module 510 is given to the display device 511, and the display device 511 converts the data into video and displays it.
[0054]
A terminal control program is stored in the ROM 502, and the CPU 503 executes the program in the ROM 502, thereby controlling the transmission / reception module 507, the reproduction module 510, and the display device 511.
[0055]
The operation of the system configured as described above will be described below. FIG. 4 is a sequence diagram for explaining the overall operation of the system of FIG. FIG. 4 shows a transmission / reception layer and a control layer on the server 101 side, and a transmission / reception layer and a control layer on the terminal 102 side, and commands and streams exchanged between these layers are arranged in time series. ing.
[0056]
First, the overall operation of this system will be described with reference to FIG. In FIG. 4, the command “SETUP” is first transmitted from the terminal 102 to the server 101. The server 101 performs initial setting according to “SETUP”. When the setting is completed, “OK” is returned from the server 101 to the terminal 102.
[0057]
When “OK” is returned from the server 101, the command “PLAY” is transmitted from the terminal 102 to the server 101. The server 101 prepares for transmission, and when the preparation is completed, “OK” is returned from the server 101 to the terminal 102.
[0058]
When “OK” is returned from the server 101, the terminal 102 shifts to a state of waiting for a stream. Following the response of “OK”, the server 101 starts transmitting a stream.
[0059]
Thereafter, the command “TEARDOWN” is transmitted from the terminal 102 to the server 101, and the server 101 ends the stream transmission in response to “TEARDOWN”. When the transmission is completed, “OK” is returned from the server 101 to the terminal 102.
When “OK” is returned from the server 101, the terminal 102 leaves the stream standby state.
[0060]
The above is an overview of the overall operation of the system, and as long as it has been described above, the operation of the system is the same as the conventional one. The operation of this system differs from the conventional one in the following two points (1) and (2).
(1) Parameters “S_target” and “T_delay” are attached to the command “SETUP” from the terminal 102 to the server 101, and the server 101 controls the transmission speed based on these parameters when transmitting the stream. .
[0061]
In the above (1), “S_target” is a target value of the amount of data stored in the buffer by the terminal 102, and the total capacity of the buffers (reception buffer 505 and decoder buffer 508 in the example of FIG. 3) provided in the terminal 102. ("S_max") and the transmission capability of the network 103. Therefore, “S_target” generally varies depending on the model of the terminal 102.
[0062]
Further, “T_delay” is a time from when the terminal 102 writes the head data to the buffer until the data is read and decoding is started (that is, cue delay time), and “S_target” is a transmission speed (described later). The value obtained by dividing by is determined as an arbitrary value within a range not exceeding the maximum value. That is, although the condition “within a range not exceeding the value obtained by dividing“ S_target ”by the transmission rate” is attached, the terminal 102 can determine “T_delay” independently of “S_target”.
[0063]
The “transmission speed” refers to the amount of information transmitted per unit time. For example, when the number of packets transmitted per unit time is determined, the amount of data packed in each packet is increased or decreased to increase the amount of information transmitted. The speed can be controlled. In addition, when the amount of data to be packed in each packet is determined, the transmission speed can be controlled by expanding / contracting the time interval between the packet and the next packet. Alternatively, the transmission speed can also be controlled by performing both simultaneously (that is, increasing or decreasing the amount of data packed in each packet and expanding or contracting the time interval between the packets and the next packet). In the present embodiment, the speed control is performed by increasing or decreasing the amount of data packed in each packet.
[0064]
(2) The terminal 102 can change “S_target” as needed during the delivery of the stream. In this case, the changed “S_target” is notified from the terminal 102 to the server 101, and thereafter, the server 101 controls the transmission speed based on the changed “S_target”.
[0065]
In (2) above, “S_target” is changed in accordance with the change in the transmission capability of the network 103. Specifically, when the terminal 102 is a mobile phone, it is possible to detect electric field strength (for example, four levels of strength, medium, weak, and out of service area). “S_target” is changed on the assumption that the transmission capability varies. For example, when the electric field strength changes from “strong” to “medium”, the terminal 102 changes the value of “S_target” to a larger value, and when it changes from “medium” to “weak”, the terminal changes the value of “S_target”. Change to a smaller value.
The operation of this system is mainly different from the above two points.
[0066]
Next, a specific example of the overall operation of this system will be described in detail. In FIG. 4, before starting the stream reproduction, the terminal 102 extracts a parameter group specific to the terminal from the ROM 502 according to the terminal control program. This parameter group includes the total capacity (maximum data amount that can be actually stored in the terminal 102) S_max, which is the sum of the reception buffer 505 and the decoder buffer 508. On the other hand, it is assumed that the CPU 503 knows the encoding / compression rate Vr of the stream data to be received and the video or audio frame generation period Tfrm by the procedure for obtaining the stream reproduction auxiliary data in advance. Further, it is assumed that the CPU 503 detects the transmission capability of the network 103, for example, the received radio wave intensity and the communication speed (information such as 64 Kbps connection or 32 Kbps connection in the case of PHS) through the network interface.
[0067]
The CPU 503 uses the S_max, Vr, Tfrm, the transmission capacity of the network 103 (for example, effective transfer rate = networkRate), and the like, a target amount S_target indicating how much data is stored in the buffer in the terminal 102, and The pre-buffering time (that is, the cue delay time) T_delay until the stream playback is started is determined.
[0068]
Here, the essential meaning of S_target is that the streaming playback can be normally continued without interruption if the storage buffer amount of the terminal changes so that it does not exceed the vicinity of S_target in the streaming playback to be started from now. It is a value. As described above, when T_delay is large, the cue delay time becomes long, but it becomes strong against the transfer jitter of the network 103. However, if the delay time is too long, it is inappropriate as a service specification. Therefore, when determining T_delay, it is necessary to consider the balance between the tolerance against transfer jitter and the waiting time at the time of cueing.
[0069]
In addition, instead of T_delay or in combination with T_delay, a filling amount S_delay that determines how many bytes of data are filled in the buffer in terminal 102 to start decoding may be determined. When the terminal 102 determines only S_delay and notifies it to the server 101, it is possible to convert S_delay to T_delay on the server 101 side using an expression of T_delay = S_delay / networkRate. Further, the value of S_delay may be a filling rate rS (%) with respect to the total buffer amount S_max (in this case, the conversion formula is S_delay = S_max * rS / 100).
[0070]
When the terminal 102 prepares S_target and T_delay (and / or S_delay), the terminal 102 issues a SETUP command that prompts the server 101 to prepare for stream delivery, as shown in FIG. The SETUP command includes S_target and T_delay (and / or S_delay) as arguments. When the server 101 receives the SETUP, the server 101 stores the argument in the RAM 404 and performs initial setting for stream delivery. Specifically, the CPU 412 of the server 101 RAM An argument is extracted from 404, and then, for example, an operation of reading a source file of the stream from the storage device 411 and writing it in the buffer 407 and setting of parameters of the packet generation circuit 406 that packetizes the read data are performed. The packet generation circuit 406 is not necessarily dedicated hardware, and is a program (software algorithm) that causes the CPU 412 of the server 101 (for example, realized by a workstation) to execute similar packetization processing. May be.
[0071]
The two values of S_target and T_delay (and / or S_delay) are also passed to the packet generation circuit 406. The packet generation circuit 406 calculates an optimum rate control parameter using these values, and as a result, packets are generated and transmitted at a rate suitable for stream delivery to the terminal 102. When the preparation for transmitting the packet into the network 103 is completed normally, OK is returned from the transmission / reception layer to the control layer as shown in FIG. 4, and then OK is returned to the terminal 102 for the SETUP command. Thus, the stream distribution preparation is completed in this system.
[0072]
Next, the terminal 102 issues a PLAY command that prompts the server 101 to start stream distribution. When receiving the PLAY, the server 101 starts distributing stream data. The terminal 102 receives and accumulates stream data from the server 101. Then, after the pre-buffering time (T_delay) has elapsed from the start of accumulation, decoding and reproduction of stream data is started. At this time, it goes without saying that the stream distribution is performed based on an appropriate rate control parameter set at the time of SETUP.
[0073]
At the end of stream reproduction, a TEARDOWN command is issued from the terminal 102 to the server 101. When the server 101 receives TEARDOWN, it performs a stream distribution end process and completes all procedures. The above is a specific operation example of this system.
[0074]
Hereinafter, the operation of the terminal 102 will be described in detail. The terminal 102 is a mobile phone that can be connected to the Internet, and has a function of detecting electric field strength (received radio wave strength). FIG. 5 is a flowchart showing the operation of the terminal 102 of FIG. In FIG. 5, first, the terminal 102 determines values of two parameters S_target and T_delay (step S101).
[0075]
Here, the processing content of the said step S101 is demonstrated concretely. FIG. 6 is a diagram showing the contents stored in the ROM 502 of FIG. As illustrated in FIG. 6, the ROM 502 stores a terminal control program, a table 601 in which electric field strength and S_target are described in association with each other, and a parameter T_delay value. As the value of the parameter S_target, three values of S_target 1 corresponding to the electric field strength “strong”, S_target 2 corresponding to “medium”, and S_target 3 corresponding to “weak / out of range” are stored. On the other hand, only one parameter T_delay value is stored.
[0076]
The three values S_target 1 to 3 are determined so as to satisfy the following relationship.
S_target3 <S_target1 <S_target2 ≦ S_max On the other hand, the value T_delay is determined not to exceed the value obtained by dividing the value S_max by the effective transmission capability of the network 103.
[0077]
As an example, if S_max is 512 (KB), S_target1 = 256 (KB), S_target2 = 384 (KB), S_target3 = 128 (KB), and the like. Further, assuming that the effective transmission capacity of the network 103 is 384 (Kbps), that is, 48 (KB / sec), T_delay is an arbitrary value (for example, 4) within a range not exceeding 512 ÷ 48≈10.7 (seconds). Second or 3 seconds).
[0078]
In step S101, S_target1 as an initial value and a value T_delay are read from the ROM 502.
[0079]
Here, S_targets 1 to 3 and the value of T_delay are calculated in advance and stored in the ROM 502, and the CPU 503 reads out necessary values from the ROM 502, but instead, the total capacity of the buffer and The effective transmission capability of the network 103 and a program for calculating the values of S_target and T_delay may be stored in the ROM 502. In this case, the CPU 503 reads the capacity, speed, and program from the ROM 502 whenever necessary, and calculates the values of S_target and T_delay. Further, here, there is only one value of T_delay, but a plurality of values may be prepared and selected from them. The above is the processing content of step S101.
[0080]
In FIG. 5 again, the terminal 102 attaches the S_target and T_delay determined in step S101 to the SETUP command and transmits them to the server 101 (step S102). In response, a stream is sent from the server 101. During stream transmission, the server 101 performs transmission speed control based on S_target and T_delay notified from the terminal (the operation on the server side will be described later).
[0081]
Next, the terminal 102 receives a stream sent from the server 101 and starts an operation of writing to the buffer (step S103). Specifically, as shown in FIG. 3, the stream sent through the network 103 is first written into the reception buffer 505 via the network controller 506. When time elapses and the reception buffer 505 becomes full, the stream in the reception buffer 505 is sequentially read from the head data and written to the decoder buffer 508.
[0082]
Next, the terminal 102 determines whether or not the time T_delay has elapsed from the start of buffering (step S104). If the determination result is negative, the terminal 102 waits until it becomes affirmative. If the determination result in step S104 is affirmative, the terminal 102 starts an operation of reading a stream from the buffer and decoding / reproducing it (step S105). Specifically, in FIG. 3, the CPU 503 measures the elapsed time from the start of buffering, and at the moment when the measurement result coincides with T_delay in the ROM 502, the playback module 510 is instructed to stream the stream in the decoder buffer 508. Are sequentially read from the head data and input to the decoder 509 is started.
[0083]
Next, the terminal 102 determines whether or not the transmission capability of the network 103 has changed across the threshold (step S106). Specifically, this determination is performed as follows. For example, a host computer (not shown) that manages the network 103 distributes information regarding the transmission capability of the network 103 to the terminal 102 as needed via the network 103, and the terminal 102 is based on information from the host computer. Determine if there is a change.
[0084]
In this case, specifically, as shown in FIG. 3, information regarding transmission capability is sent to the CPU 503 through the transmission / reception module 507. The ROM 502 stores a threshold value in advance, and the CPU 503 compares the sent information, the previous information held, and the threshold value in the ROM 502 with each other, so that the transmission capability crosses the threshold value. It can be determined whether or not it has changed.
[0085]
Alternatively, when the host computer that manages the network 103 does not have a function of distributing information related to the transmission capability to the terminal 102, the terminal 102 can make the determination for example as follows. That is, when the terminal 102 is a mobile phone, as shown in FIG. 7 (described later), it has a function of detecting the surrounding electric field strength and displaying the detection result as “strong / medium / weak / out of service area”. Yes. If this change in electric field strength is regarded as a change in transmission capability of the network 103, the terminal 102 can easily detect.
[0086]
If the determination result of step S106 is affirmative, the terminal 102 determines a new S_target (step S107) and transmits it to the server 101 (step S108). On the other hand, if the determination result of step S106 is negative, steps S107 and S108 are skipped and step S109 (described later) is executed.
[0087]
Here, the processing contents of steps S106 and S107 will be described in detail. Hereinafter, a case where the terminal 102 is a mobile phone and S_target is changed according to a change in electric field strength will be described. FIG. 7 is a schematic diagram showing a distribution of electric field strength in a certain area and a change in transmission capability accompanying the movement of the terminal. FIG. 7A shows an electric field strength distribution in an area including three relay stations B1 to B3. In FIG. 7A, concentric circles centering on the relay stations B1 to B3 are equi-electric field curves that can connect points having the same electric field strength.
[0088]
For example, the electric field strength is “strong” in the concentric circle 703 closest to the relay station, and “medium” in the region between the concentric circle 703 and the next concentric circle 704. Furthermore, the region between the concentric circle 704 and the concentric circle 705 is “weak”, and the region outside the concentric circle 705 is “out of range”. However, concentric circles centering on each relay station partially intersect each other, and there are only a few regions where the electric field strength is “out of range”.
[0089]
Now, the terminal 102 is about to move from the vicinity of the relay station B1 to the vicinity of the relay B2 along the route indicated by the arrow 702. FIG. 7B shows the electric field strength along the arrow 702 in FIG. 7A (this can be regarded as the transmission capability of the network 103). As shown in FIG. 7B, the electric field strength is “strong” when the terminal 102 is in the vicinity of the relay station, and “medium”, “weak”, “out of service” as the terminal 102 moves away from the relay station B1. It will become weaker gradually. Immediately after the relay station B1 becomes “out of service area”, the terminal 102 enters the “service area” of the relay station B2, and the radio wave intensity gradually increases as “weak”, “medium”, and “strong”. .
[0090]
The terminal 102 that moves as described above determines that the transmission capability of the network 103 has changed across the threshold A at the moment when the electric field strength changes from “strong” to “medium”, and determines a new S_target. At the moment when the state changes from “medium” to “weak”, it is determined that the transmission capacity has changed across the threshold B, and a new S_target is determined. Conversely, the moment when the transmission capacity changes from “weak” to “medium”, it is determined that the transmission capacity has changed across the threshold value B, a new S_target is determined, and the moment when it changes from “medium” to “strong” Then, it is determined that the transmission capability has changed across the threshold A, and a new S_target is determined.
[0091]
In general, the threshold A is a value approximately between the maximum transmission capability that can be realized in the network 103 and the transmission capability at which a stream transfer loss starts to occur. The threshold value B is a value corresponding to a transmission capability at which a stream transfer loss starts to occur.
[0092]
The new S_target is determined as follows by referring to the table 601 (FIG. 6) in the ROM 502. FIG. 8 is a flowchart showing details of step S107 in FIG. In FIG. 8, the terminal 102 first determines whether or not the electric field strength after the change is “strong” (step S201). If the determination result is affirmative, the terminal 102 determines a new S_target as S_target1 (step S202). . If the determination result of step S201 is negative, the process jumps to step S202 and proceeds to step S203.
[0093]
Next, the terminal 102 determines whether or not the electric field strength after the change is “medium” (step S203). If the determination result is affirmative, the terminal 102 determines a new S_target as S_target2 (step S204). If the determination result of step S203 is negative, the process jumps to step S204 and proceeds to step S205.
[0094]
Next, the terminal 102 determines whether or not the changed electric field strength is “weak / out of range” (step S205). If the determination result is affirmative, the terminal 102 determines a new S_target as S_target3 (step S206). Thereafter, the flow returns to the flow of FIG. If the determination result of step S205 is negative, the process jumps to step S206 and returns to the flow of FIG.
[0095]
Therefore, when the terminal 102 moves along the arrow 702 in FIG. 7A, the terminal 102 changes the value of the parameter S_target from S_target1 → S_target2 → S_target3 → S_target2 → S_target1 as the electric field strength changes. To change. As a specific example, it is changed in the order of 256 (KB) → 384 (KB) → 128 (KB) → 384 (KB) → 128 (KB). The above is a specific processing example of step S106 and step S107.
[0096]
In FIG. 5 again, when the terminal 102 transmits a new S_target to the server 101 in step S108, the server 101 changes the value of the parameter S_target to a value newly notified from the terminal 102 accordingly. Continue transmission speed control.
[0097]
Next, the terminal 102 determines whether or not to end the streaming reproduction (step S109). When the terminal 102 ends, the terminal 102 transmits the command TEARDOWN to the server 101 and stops receiving and buffering the stream (step S110). Then, the reproduction process is stopped (step S111). On the other hand, when the streaming reproduction is continued, the terminal 102 returns to step S106 and repeats the same processing as described above. The above is the operation of the terminal 102.
[0098]
Next, the operation of the server 101 will be described in detail. In order to simplify the description, the server 101 uses MPEG1 video (ISO / IEC 11172-2), MPEG2 video (ISO / IEC 13818-2), or MPEG2-AAC audio (ISO / IEC 13818-7). It is assumed that encoding is performed using an encoding compression algorithm that generates a frame with a fixed period Tfrm, and the encoded data is packetized with a fixed period Ts.
This packetization is performed in units of frames.
[0099]
First, an overview of stream transmission speed control performed by the server 101 will be described with reference to FIGS. 9 to 11 are diagrams illustrating how the data amount (buffer occupancy) accumulated in the buffer of the terminal 102 changes according to the stream transmission speed control performed by the server 101. FIG. The server 101 controls the transmission speed of the stream so that the buffer occupancy changes as shown in FIGS. 9 to 11 in the terminal 102 of the transmission destination.
[0100]
FIG. 9 shows how the buffer occupancy approaches S_target. FIG. 10 shows how the buffer occupancy approaches S_target2 when the value of S_target is changed to a larger value (S_target2) while the buffer occupancy is in the vicinity of S_target. ing. FIG. 11 shows a state in which the buffer occupancy approaches S_target3 when the value of S_target is changed to a smaller value (S_target3) while the buffer occupancy is in the vicinity of S_target. ing.
[0101]
9 to 11, “S_max” is the total buffer capacity of the terminal 102, and “Sum” is the buffer occupation amount. “Delta (0, 1, 2,...)” Indicates the amount of data that the server 101 transmits per unit time Ts (that is, the amount of data packed in one packet). Here, the unit time Ts is a cycle in which the server 101 transmits a packet, and is a fixed value. “L (0, 1, 2,...)” Is a data amount per frame.
[0102]
When the server 101 receives a notification of the value of T_delay from the terminal 102, the server 101 controls the transmission speed of the stream based on the value. This speed control is performed by changing the amount of data packed in one packet.
[0103]
In FIG. 9, the packet (i = 0) transmitted first by the server 101 is packed with data of the amount delta0, and the buffer occupation amount Sum is delta0 at time t = 0. When the unit time Ts elapses, the next packet (i = 1) is sent, which is filled with data of the quantity delta1. Therefore, at time t = Ts, Sum is {delta0 + delta1}. Thereafter, every time the unit time Ts elapses, packets (i = 2, 3,...) Are sent one after another, and delta2, delta3,... Are added to Sum.
[0104]
On the other hand, at time t = T_delay before the third packet (i = 2) is sent, a process of reading and decoding data from the buffer is started. Since decoding is performed on a frame basis, L0, L1, L2,... Are subtracted from Sum every fixed period Tfrm after t = T_delay.
[0105]
That is, after the time t = 0, the buffer occupancy Sum is gradually increased by adding delta0, delta1,... Every period Ts. On the other hand, L0, L1, L2,... Are subtracted every cycle Tfrm after time t = T_delay. Accordingly, during the period until the Sum reaches the target value S_target, the amount of data packed in one packet is made larger than the standard (more generally, the transmission speed is increased), and the writing speed to the buffer is increased. From then on, if the amount of data packed in one packet is returned to the standard and the writing speed and the reading speed are balanced, Sum can be shifted in the vicinity of the target value S_target. It becomes possible.
[0106]
When such transmission speed control is performed, as shown in FIGS. 10 and 11, even when the target value S_target is changed to a new target value (S_target2, 3) in the middle, Sum is changed to the new target value (S_target2). , 3) in the vicinity.
[0107]
That is, in FIG. 10, when S_target is changed to a larger value (S_target2) in a state where Sum is transitioning in the vicinity of target value S_target, server 101 changes to a subsequent packet (i = 3, 4). By increasing the amount of data to be packed, the writing speed to the buffer is made faster than the reading speed from the buffer. After Sum reaches the new target value S_target2, the data amount packed in one packet is returned to the standard, and the writing speed and the reading speed may be balanced.
[0108]
In FIG. 11, when S_target is changed to a smaller value (S_target3) while Sum is transitioning in the vicinity of the target value S_target, the server 101 changes to subsequent packets (i = 3, 4). By reducing the amount of data to be packed, the writing speed to the buffer is made lower than the reading speed from the buffer. After Sum reaches the new target value S_target3, the data amount packed in one packet is returned to the standard, and the writing speed and the reading speed may be balanced.
[0109]
Next, transmission rate control by the server 101 as described above will be described in detail. FIG. 12 is a flowchart illustrating an example of a transmission rate control algorithm performed by the server 101. In FIG. 12, first, the terminal 102 detects its own buffer occupation amount (Sum), and the server 101 receives a notification of the buffer occupation amount Sum from the terminal 102 (step S301). Next, the server 101 determines whether or not the buffer occupancy amount Sum notified in step S301 has changed in the vicinity of the target value S_target specified from the terminal 102 (step S302). If the determination result is affirmative, the current transmission rate is maintained.
[0110]
When the determination result of step S302 is negative, the server 101 determines whether or not the buffer occupation amount Sum notified in step S301 is larger than the target value S_target (step S303). If the determination result is negative, the transmission speed is changed to a speed higher than the current speed (step S304), and then the process proceeds to step S306. On the other hand, if the determination result of step S303 is affirmative, the transmission speed is changed to a speed slower than the current speed (step S305), and then the process proceeds to step S306.
[0111]
In step S306, it is determined whether or not to continue the speed control operation. If the determination result is affirmative, the process returns to step S301 and the same operation as described above is repeated. On the other hand, if the determination result is negative, the operation is terminated. The above is an example of transmission speed control performed by the server 101.
[0112]
By the way, in the example of FIG. 12, the terminal 102 detects its own buffer occupation amount and notifies the server 101 of it. However, in that case, the terminal 102 detects the buffer occupancy at the current time. In addition, since it takes time to transmit information from the terminal 102 to the server 101, the server 101 performs transmission rate control based on the past buffer occupancy for the transmission delay time, and the buffer occupancy is set to S_target. It is actually difficult to make a transition in the vicinity of.
[0113]
On the other hand, in another example described below (see FIGS. 13 and 14), the buffer occupancy is changed in the vicinity of S_target by performing transmission rate control based on the buffer occupancy at a certain time in the future. It becomes possible to make it. In this case, the server 101 does not receive the Sum notification from the terminal 102 but predicts and calculates the buffer occupancy Sum on the terminal 102 side at a future time. This prediction calculation is performed as follows.
[0114]
That is, in FIG. 2, the ROM 413 stores a packet transmission cycle Ts (fixed value) and a decoding cycle Tfrm (fixed value) in advance. When generating a packet, the CPU 412 stores the amount of data (delta 0, delta 1, delta 2,...) Packed in the packet in the RAM 404. Further, when the stream is transmitted, the data amount (L0, L1, L2,...) Of each frame is stored in the RAM 404.
[0115]
The RAM 404 also stores T_delay previously notified from the terminal 102, and the CPU 412 stores Ts and Tfrm in the ROM 413, delta (0, 1, 2,...), L (0, 1) in the RAM 404. , 2,...) And T_delay to perform a predetermined calculation, the buffer occupancy Sum at a future time can be calculated. By performing such a calculation process, the server 101 can predict how the buffer occupation amount Sum changes on the terminal 102 side (see FIGS. 9 to 11).
[0116]
Hereinafter, with regard to a specific operation example in which the server 101 predicts and calculates the buffer occupation amount Sum of the terminal 102 and performs transmission rate control, the buffer transition diagram of FIG. 9, the flowcharts of FIGS. 13 and 14, and the packet configuration diagram of FIG. Will be described.
[0117]
In FIG. 9, S_max is the maximum value of the effective accumulation amount of the buffer in the terminal 102 (this is simply called “total capacity of the buffer”), and S_target is accumulated in the buffer in the terminal 102 in the current streaming. T_delay is a set value for the cue delay time. The meaning of each parameter has already been described. In the following description, it is assumed that S_target and T_delay are notified from the terminal 102.
[0118]
In the present embodiment, in order to facilitate understanding, an example in which packets are generated / distributed every fixed time period Ts (packet distribution at a time corresponding to i = n: n is an integer) is shown. In addition, when a packet is delivered at a time corresponding to i = n (t = i * Ts), the accumulation amount Sum in the reception buffer 505 and the decoder buffer 508 of the terminal 102 is an amount of data corresponding to several frames. However, this is because packets are generated and distributed to the terminal 102 in a pattern in which a plurality of frames are inserted into one packet as shown in FIG. . Actually, the packet delivery takes a transfer time, and the buffer occupancy does not increase instantaneously as shown in the figure (inclination = slanted line of networkRate), but it is assumed that it is handled as a simple model. In addition, the buffer occupancy is reduced stepwise after time t = T_delay, indicating that stream playback has started at the terminal 102 at that time. That is, data is processed by the decoder 509 for each frame length L = L [k] (k is an integer) for each frame reproduction period Tfrm.
[0119]
FIG. 13 and FIG. 14 are flowcharts showing another example of the transmission control algorithm performed by the server 101 to realize the buffer transition of FIG. FIG. 13 is an overview of the algorithm, and FIG. 14 is a flowchart showing an example of the function mkPacket in step S404 of FIG. A program describing such an algorithm is stored in the ROM 413 (see FIG. 2), and the CPU 412 performs various calculations and controls according to this program, and as a result, the buffer transition of FIG. 9 is realized. In order to simplify the explanation, it is assumed that the distribution stop in the middle of the stream is not considered this time. Hereinafter, each step will be described sequentially.
[0120]
In FIG. 9, the server 101 receives and stores S_target and T_delay sent from the terminal 102 (step S401). Specifically, in FIG. 2, the values of S_target and T_delay sent from the terminal 102 via the network 103 are written into the RAM 404 through the network controller 410.
[0121]
Here, the terminal 102 determines the values of the parameters S_target and T_delay and notifies the result to the server 101. Alternatively, the server 101 may store these values in advance, or The model information (total buffer capacity, etc.) of the terminal 102 may be stored, and the server 101 may calculate the parameter value based on the model information.
[0122]
Next, each variable is initialized (steps S402 and S403). The meaning of each variable will be described later with reference to FIG. When the initialization is completed, processing after step S404, that is, processing for generating a packet with the function mkPacket and sending it to the network 103 is started. In this example, the generated packet is delivered to the terminal 102 at a fixed period Ts. Therefore, the server 101 adjusts the timing in step S405, and then sends it out in step S406. When this series of processing is completed, the CPU 412 updates the execution counter i of the function mkPacket, returns to step S404, and loops. When all the reading and packetization of the stream data are completed, the CPU 412 exits the function mkPacket and returns to the step S404 with the determination result FALSE. At this time, the CPU 412 regards the distribution as being completed, and completes the algorithm. The above is the outline of this transmission control algorithm.
[0123]
Next, a detailed algorithm of the function mkPacket shown in step S404 will be described. First, each variable will be described. Sum is the total amount of data stored in the reception buffer 505 and the decoder buffer 508 in the terminal 102, L is the data amount of the frame, and delta is packetized after the function mkPacket is called this time. The sum of the data amount (that is, the amount of data packed in one packet), in is a counter indicating the number of frames of the stream source read from the storage device 411, and out is decoded by the decoder 509 in the terminal 102 The counter indicates the number of frames that have been received, dts is the time at which the decoder 509 decodes the frame, and grid is the upper limit of dts advanced when processing one loop of the previous function mkPacket.
[0124]
In FIG. 14, the function mkPacket is roughly divided into two algorithms: a packet generation algorithm A1 and a decoding amount calculation algorithm A2. In the former, in the first step (S501), the CPU 412 clears delta. In the subsequent step S502, the CPU 412 determines whether or not a frame of L = L [in] (already read) can be used for the current packet generation. The criteria for determination are (a) adding L to Sum does not exceed S_target, and (b) the amount of data packetized by the current function call (the amount of data packed in one packet this time) delta Even if L is added, the upper limit value deltaMax of the data amount that can be packed in one packet is not exceeded.
[0125]
Here, deltaMax is the inequality shown in FIG.
(DeltaMax + hdr) / Ts <NetworkRate
Is the maximum value of the amount of data that can be distributed to the terminal within the period Ts, and can be calculated from the effective transfer rate (transmission capability) of the network 103. If it is determined to be true in step S502, the CPU 412 proceeds to step S503 and packetizes the frame of L = L [in]. In subsequent step S504, CPU 412 updates Sum and delta as packetization is performed. In subsequent step S <b> 505, the CPU 412 reads the data of the next frame from the read buffer 407 and the frame length L from the RAM 404.
Then, it is determined whether or not L is greater than zero.
[0126]
If the determination result in step S505 is negative, that is, if L = 0, the CPU 412 regards that all data has been read (end of file detection) and exits the function. Then, RETURN is performed with the determination result FALSE in step S404 of the main flow (FIG. 13). On the other hand, if the determination result is affirmative, that is, if L> 0, the CPU 412 proceeds to the next step (S506), and adds L [in] into the array length (stores it in the RAM 404). As will be described later, this is for use in the decoding amount calculation algorithm A2. Next, the CPU 412 proceeds to step S507, updates the read frame number counter in, and loops to step S502.
[0127]
As the packet generation by the above loop is repeated, the values of Sum and delta increase gradually. If it is determined in step S502 that Sum or delta has become sufficiently large, the CPU 412 exits this loop and enters the decoding amount calculation algorithm A2.
[0128]
In the decoding amount calculation algorithm A2, in the first step (S508), it is determined whether i * Ts is equal to or greater than grid. The purpose of step S508 is to determine whether or not it is time to start decoding in the terminal 102. Specifically, since grid is initially set to T_delay, it is determined that decoding at terminal 102 has not yet started while function call counter number i is small and t = i * Ts is less than grid. The In FIG. 9, the time corresponding to i = 0 and i = 1 corresponds to this.
[0129]
If the determination result of step S508 is negative, the CPU 412 exits the function without performing subtraction of frame data by decoding. On the other hand, when i becomes sufficiently large and the packet generation time t = i * Ts becomes equal to or greater than grid, the CPU 412 regards that the terminal 102 has already started decoding, and performs the subtraction process on the frame data. In FIG. 9, the time when i is 2 or more corresponds to this. In the subsequent loop from step S509 to step S512, the amount of frame data length [out] to be decoded within the time between the current grid time and the next grid time (= grid + Ts) is subtracted from Sum, and The number of decoded frames out is counted up.
[0130]
In step S511 in the above loop, Tfrm is added to dts every time one frame is decoded, and this embodiment uses an encoding method that generates frames with a fixed time interval Tfrm. It comes from that. In step S512, the CPU 412 determines whether there is a frame to be decoded at the current time interval Ts. If the determination result in step S512 is negative, that is, if it is determined that there is no longer a frame to be decoded at the current time interval Ts, the CPU 412 exits from the loop (steps S509 to S512) and proceeds to step S513. In step S513, the CPU 412 updates the variable grid to the next grid time. Then, the function is exited, and RETURN is performed with the determination result TRUE in step S404 of the main flow (FIG. 13).
[0131]
With the above algorithm, as shown in FIG. 9, the buffer occupancy Sum can be changed within the terminal 102 so that it is always in the vicinity of S_target and does not exceed S_target. Therefore, even if there are a plurality of types of terminals 102 and the total capacity Smax of the buffer varies depending on the models, if S_target is set to an appropriate value according to the Smax of each terminal 102, the buffer overflows and underflows. It can be prevented from occurring.
[0132]
In this example, a packet is generated with a pattern in which a plurality of frames are inserted into one packet as shown in FIG. 15A. Instead, one frame is included in one packet as shown in FIG. 15B. It is also possible to generate a packet with a pattern for inserting. In this case, in step S502 in FIG.
delta + (L + hdr) <= deltaMax
And the latter half of step S504
delta + = (L + hdr)
Just do it.
[0133]
Further, in the present embodiment, for the sake of simplicity, an encoding method for generating frames at a fixed time interval Tfrm is used. However, an encoding method to be used—for example, MPEG4 video (ISO / IEC 14496-2) — If the decoding amount calculation algorithm A2 is designed according to the above, it goes without saying that it is not always necessary to generate frames at fixed time intervals. Further, the algorithm does not necessarily need to handle data in units of frames. For example, it may be an algorithm that handles data in units of slices or packs of MPEG1 or MPEG2 system streams.
[0134]
On the other hand, when the value of S_target is changed halfway in step S502 of FIG. 14, the present algorithm instantly generates a packet with the changed new S_target as a target. The state of buffer transition when the value of S_target is changed in the middle is shown in FIGS. In FIG. 10, when S_target is changed to S_target2 at the time of i = 3 (S_target <S_target2 ≦ S_max), a large amount of frame data is packetized for a while after the change (delta3 and delta4 in the figure). Sum reaches the vicinity of the new target value S_target2.
[0135]
As shown in FIG. 11, when S_target is changed to S_target3 at the time of i = 2 (S_target3 <S_target), a small amount (delta4) or 0 (delta3) of frame data is packetized. On the other hand, Sum is consumed by decoding, so that Sum reaches near the new target value S_target3. If such a mechanism is used, the buffer occupancy Sum in the terminal 102 can be dynamically increased or decreased according to the transmission capability of the network 103 (or the radio wave reception state of the terminal 102), as will be described below. Application becomes possible.
[0136]
In FIG. 7A, a case where a user having a mobile phone 701 (corresponding to the terminal 102 in FIG. 1) moves from the area of the relay station B1 to the area of the relay station B2 as indicated by an arrow 702 in the figure. Think. Along with the movement, the task of accepting a call from the mobile phone 701 is handed over from the relay station B1 to the relay station B2. At this time, the received radio wave intensity of the mobile phone 701 changes as shown in the graph of FIG. 7B. In this model, in order to simplify the explanation, the place where the radio field intensity changes from strong to medium (or medium to strong) is changed to the threshold A concerning the transmission capability of the network 103, and the place where medium to weak (or weak to medium) changes. A threshold value C is defined as a threshold value C, where the light intensity changes from weak to out of service (or from out of service to weak).
[0137]
In FIG. 7B, it is assumed that the user holding the mobile phone 701 has moved by the distance d1 and the transmission capability has fallen below the threshold A. At this time, the mobile phone 701 changes S_delay to a larger value (S_target2) and notifies the server 101 of the value, as shown in FIG. This facilitates the generation and transmission of new packets by the server 101 in preparation for a decrease in transmission capacity that is expected to continue thereafter, and thereby, as much data as possible (this is denoted by Δt) as long as possible. This is because it is stored in the internal buffer. Even if the transmission capacity falls below the threshold A, no packet transfer loss occurs while the transmission capacity is equal to or higher than the threshold B. Therefore, the transmission speed can be increased.
[0138]
When the user moves to reach the distance d2, the transmission capability falls below the threshold value B, and packet transfer loss starts to occur. At this time, the mobile phone 701 changes S_target to a small value (S_target3) and notifies the server 101 of the value, as shown in FIG. This is to suppress generation and transmission of new packets by the server 101 as much as possible in preparation for a decrease in transmission capability that is expected to continue further. The reason for suppressing packet generation and transmission is as follows.
[0139]
For example, when the cellular phone 701 employs a PHS PIAFS system as a communication system, when a packet transmission loss occurs, data retransmission processing based on a PIAFS layer protocol that is a link layer is performed. This is because if a new packet is generated and transmitted during the retransmission process, it will interfere with the retransmission process, which is not preferable.
[0140]
When the user moves to reach the distance d3, the transmission capability falls below the threshold C, and packet transfer becomes difficult for a moment. However, when the user further moves to reach the distance d4, the transmission capability exceeds the threshold value B, and the handover of the task for accepting the call is completed, so that the mobile phone 701 is now based on S_target3. Is returned to the value S_target, and the server 101 is notified of the value, thereby increasing the data storage amount (ie, the buffer occupation amount Sum). In addition, since the handover time of PHS or the like is normally completed in about 2 to 3 seconds even at a speed at which a person walks, if the above Δt is secured for about 3 to 4 seconds, handover occurs. In addition, streaming playback on the mobile phone 701 can be continued without delay.
[0141]
By the way, as shown in FIG. 11, when the set value of S_target is changed to a smaller value in the middle of stream distribution, the judgment sentence in step S502 in the algorithm of FIG. There may be cases where it cannot be sent. When such a case frequently occurs, even when the corner packet is delivered to the terminal 102, the time (Presentation Time) at which the frame data in the packet should be reproduced has passed, and the data is wasted. Sometimes. In such a case, skipping the frame data for which the playback time has passed is more efficient as long as unnecessary data does not flow through the network 103.
[0142]
FIG. 16 is a flowchart showing another example of the function mkPacket in step S404 of FIG. The function mkPacket in FIG. 16 includes steps (S601 and S602) for skipping the transmission of data past the reproduction time when the server 101 decreases the transmission speed. That is, in the algorithm of FIG. 16, only step S601 and step S602 are added as compared to FIG. The other steps are exactly the same as in FIG. 14, and are given the same reference numbers. In step S601, the CPU 412 determines that the in-th frame data to be transmitted from now on is not the 0th frame data, and the playback time is longer than the out-th frame data that is considered to have been decoded by the terminal 102. Judging whether it is after.
[0143]
If this determination result is true, the CPU 412 assumes that the data of the in-th frame is in time for the playback time at the terminal, packetizes the data in step S503, and sends it to the terminal 102. In the case of false, it is considered that there is no in-th frame data, and L = 0 is set in step S602. As a result, it is always determined to be true in step S502, and in the packetization in step S503, it is possible to advance the transmission frame without copying unnecessary frame data. If there is such a frame skip, the playback at the decoder 509 skips for the time Tfrm, so information notifying the terminal 102 of this fact is described in the packets of FIGS. 15A and 15B. Shall be. For example, an area for describing such reproduction time information may be provided in the header.
[0144]
The algorithm shown in FIG. 16 is a sufficiently effective technique when there is no difference in priority (importance) between frames as in MPEG audio. However, in MPEG video, as already explained in the introduction of the conventional example, a meaningful image can be reconstructed by itself if it is an I frame. Without a reference frame, a meaningful image cannot be reconstructed. In this case, when performing frame decimation in the algorithm of FIG. 16, while preferentially transmitting I frames in time for playback time, skipping all P and B frames allows the transfer rate of the network 103 to be reduced. Even in a slow situation, the terminal 102 can be provided with higher quality video.
[0145]
FIG. 17 is a flowchart showing yet another example of the function mkPacket in step S404 of FIG. In the function mkPacket of FIG. 17, when the server 101 decreases the transmission speed, a step for skipping transmission of low priority data and high priority data that has passed the reproduction time (S505 ′, S601, S602, S701, and S702) are included. That is, in the algorithm of FIG. 17, steps S601, S602, S701, and S702 are added, and step S505 is replaced with step S505 ′, compared to FIG. Step S505 ′ is obtained by adding a function of detecting priority order pri to the function NexTfrm to step S505. The other steps are exactly the same as in FIGS. 14 and 16, and are given the same reference numerals.
[0146]
Accordingly, in the algorithm of FIG. 17, steps S701 and S702 are added and step S505 is replaced with S505 ′ as compared with FIG.
[0147]
In order to execute the algorithm of FIG. 17, a function for notifying the server 101 of information (reception status information) indicating the reception status detected by the terminal 102 is required. FIG. 18 shows a configuration example of a server client system having such a function. In FIG. 18, the terminal 102 includes a detection unit 801 that detects a reception state. Between the terminal 102 and the server 101, a notification unit 802 that notifies the server 101 of the detected reception state information is provided. The server 101 includes a holding unit 803 and holds the notified reception state information.
[0148]
In FIG. 17 again, when the mkPacket function is called, step S701 is executed prior to step S501. In step S <b> 701, the server 101 (the CPU 412 thereof) refers to the information in the holding unit 803 (the reception state on the terminal 102 side) and determines whether or not the transmission capability of the network 103 is below the threshold B. As a result of this determination, if the threshold value B is below the threshold B, then the slow flag is true, and otherwise it is false.
[0149]
In step S505 ′, the priority of the next frame is detected. In subsequent step S702, it is determined whether the priority of the data of the frame is high and the slow flag is true. If this determination result is affirmative, that is, if the slow flag indicating that the transfer speed of the network 103 is slow is true and the frame has a high priority, the process proceeds to step S601, and whether or not the frame has passed the playback time. Is determined. On the other hand, if the determination result is negative, the process proceeds to step S602 and L = 0 is set (that is, the frame is skipped even if it is in time for the reproduction time). The subsequent processing is exactly the same as the processing in FIGS.
[0150]
As described above, according to the present embodiment, the terminal 102 determines a target amount according to its own buffer capacity and the transmission capability of the network 103, and further obtains a value obtained by dividing the target amount by the transmission capability. The delay time is determined within a range not exceeding. Since the server 101 controls the transmission speed based on the target amount and the delay time determined by the terminal 102 in this way, even if the buffer capacity of the terminal 102 differs depending on the model or the transmission capability of the network 103 varies, the buffer 101 Transmission speed control according to the amount and transmission capability can be performed, and as a result, failure of streaming playback due to buffer underflow or overflow can be avoided. In addition, since the delay time is determined independently of the target amount, it is possible to achieve both the avoidance of failure of streaming playback and the reduction of the waiting time at the time of cueing.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration example of a server client system that executes a streaming method according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a configuration of the server 101 in FIG.
FIG. 3 is a block diagram illustrating a configuration of the terminal 102 of FIG.
4 is a sequence diagram for explaining the overall operation of the system of FIG. 1. FIG.
FIG. 5 is a flowchart showing an operation of the terminal 102 of FIG.
6 is a diagram showing the contents stored in the ROM 502 of FIG. 3;
FIG. 7 is a schematic diagram showing an electric field intensity distribution (A) in a certain area and a change in transmission capability (B) accompanying the movement of the terminal.
FIG. 8 is a flowchart showing details of step S107 in FIG. 5;
9 is a diagram illustrating how the buffer occupancy of the terminal 102 changes (approaching S_target) according to transmission speed control performed by the server 101 in FIG. 1; FIG.
FIG. 10 shows a state in which the terminal 102 performs transmission speed control performed by the server 101 in FIG. 1 when the value of S_target is changed to a larger value (S_target2) while the buffer occupancy is changing in the vicinity of S_target. It is a figure which shows how the buffer occupation amount changes.
11 shows a state in which the terminal 102 is subjected to transmission speed control performed by the server 101 in FIG. 1 when the value of S_target is changed to a smaller value (S_target3) while the buffer occupancy is changing in the vicinity of S_target. It is a figure which shows how the buffer occupation amount changes.
12 is a flowchart showing an example of a transmission rate control algorithm performed by the server 101 of FIG.
FIG. 13 is a flowchart showing another example of a transmission rate control algorithm performed by the server 101 to realize the buffer transition of FIGS. 9 to 11;
14 is a flowchart illustrating an example of a function mkPacket in step S404 in FIG.
15 is a diagram illustrating a configuration example of a packet generated by the server 101 in FIG. 1 (A is a case where a plurality of frames are inserted into one packet, and B is a case where one frame is inserted into one packet).
16 is a flowchart showing another example of the function mkPacket in step S404 in FIG.
FIG. 17 is a flowchart showing still another example of the function mkPacket in step S404 in FIG.
FIG. 18 is a block diagram showing another configuration example of the server client system that executes the streaming method according to the embodiment of the present invention.
FIG. 19 is a diagram for explaining a conventional streaming method (A is a video frame, B is a transition of buffer occupancy, and C is a configuration example of a conventional terminal).
FIG. 20 is a block diagram illustrating a configuration example of a server / client system that executes a conventional streaming method;
FIG. 21 is a diagram for explaining how the transition of buffer occupancy changes by adding a reception buffer (A is before addition and B is after addition);
[Explanation of symbols]
101 server
102 terminals
103 network
402,507 Transmission / reception module
404 RAM
405 generation module
406 Packet generation circuit
407 Read buffer
408 Packet generation buffer
409 Transmission buffer
410,506 Network controller
411 Storage device
412 and 503 CPU
413, 502 ROM
505 Receive buffer
508 Decoder buffer
509 decoder
510 Playback module
511 Display device

Claims (15)

A streaming method in which a server transmits stream data to a terminal through a network, and the terminal reproduces while receiving the stream data,
A target amount determination step in which the terminal determines a target amount of stream data to be stored in its buffer in relation to its buffer capacity and network transmission capability;
The delay time from when the terminal writes the head data of the stream to its own buffer until the data is read and playback is started, as long as it does not exceed the value obtained by dividing the buffer capacity by the transmission capacity Determine the delay time determination step,
The terminal notifies the server of the determined target time and delay time;
A streaming method comprising a control step of controlling a transmission rate based on a notified target amount and delay time when a server transmits stream data to a terminal through a network. In the control step, the server
The transmission rate is controlled so that the amount of stream data accumulated in a buffer of a terminal does not exceed the target amount in the vicinity of the target amount. Streaming method.   In the control step, the server predicts and calculates the amount of stream data stored in the buffer of the terminal based on the transmission speed, the delay time, and the speed at which the terminal decodes the stream data. The streaming method according to claim 2. A detection step in which the terminal detects that the transmission capability of the network has changed across a predetermined threshold;
According to the detection result in the detection step, the terminal further includes a target amount changing step in which the terminal changes the target amount, and a step in which the terminal notifies the server of the changed target amount,
In the control step, when the server receives the notification of the changed target amount, the amount of stream data accumulated in the terminal buffer exceeds the changed target amount in the vicinity of the changed target amount. The streaming method according to claim 1, wherein the transmission speed is controlled so as to make a transition without any change. When detecting that the transmission capability of the network has dropped across the first threshold in the detection step, the terminal changes the direction to increase the target amount in the target amount change step,
5. The streaming method according to claim 4, wherein, in the control step, the server controls the transmission speed to increase in accordance with an increase in the target amount.   6. The streaming according to claim 5, wherein the first threshold is a value approximately in the middle between a maximum transmission capability that can be realized and a transmission capability at which a transfer loss of stream data starts to occur. Method. When the detection step detects that the transmission capability of the network has dropped across the second threshold value that is smaller than the first threshold value, the terminal changes the direction so as to decrease the target amount in the target amount change step. ,
5. The streaming method according to claim 4, wherein, in the control step, the server controls the transmission speed to decrease in accordance with the target amount being changed in a decreasing direction.   The streaming method according to claim 7, wherein the second threshold value is a value corresponding to a transmission capability at which stream data transfer loss starts to occur.   In the target amount changing step, when the terminal changes in a direction to decrease the target amount, in the control step, the server sequentially compares the reproduction time of each frame constituting the stream to be transmitted with the current time, 9. The streaming method according to claim 8, wherein transmission of a frame whose playback time is older than the current time is skipped, and thereby the transmission speed is controlled in a decreasing direction. In the target amount changing step, when the terminal changes in a direction to decrease the target amount, in the control step, the server sequentially compares the importance of each frame constituting the stream to be transmitted with a reference value,
For frames whose importance is less than the reference value, all transmissions are skipped,
For frames whose importance is greater than or equal to the reference value, each playback time is sequentially compared with the current time, and transmission is skipped only when the playback time is older than the current time, thereby controlling the transmission speed in a decreasing direction. The streaming method according to claim 8, wherein: A system comprising a server that transmits stream data over a network and a terminal that reproduces the stream data while receiving the data,
The terminal
A target amount determining means for determining a target amount of stream data to be accumulated in its own buffer in relation to its own buffer capacity and network transmission capability;
Arbitrarily determine the delay time from writing the head data of the stream to its own buffer until reading the data and starting playback within a range not exceeding the value obtained by dividing the buffer capacity by the transmission capacity A delay time determining means, and means for notifying a server of the determined target time and delay time;
The server comprises control means for controlling a transmission speed based on the notified target amount and delay time when transmitting stream data to a terminal through a network. A terminal that is used together with a server that transmits stream data through a network and that plays back the stream data while receiving the stream data,
The server is equipped with a control means for controlling the transmission speed based on the notified target amount and delay time when transmitting stream data to the terminal through the network,
A target amount determining means for determining a target amount of stream data to be accumulated in its own buffer in relation to its own buffer capacity and network transmission capability;
Arbitrarily determine the delay time from writing the head data of the stream to its own buffer until reading the data and starting playback within a range not exceeding the value obtained by dividing the buffer capacity by the transmission capacity A terminal comprising delay time determining means and means for notifying a server of the determined target time and delay time. A server that is used together with a terminal that reproduces while receiving stream data and that transmits the stream data through a network,
On the device,
A target amount determining means for determining a target amount of stream data to be accumulated in its own buffer in relation to its own buffer capacity and network transmission capability;
Arbitrarily determine the delay time from writing the head data of the stream to its own buffer until reading the data and starting playback within a range not exceeding the value obtained by dividing the buffer capacity by the transmission capacity A delay time determining means, and means for notifying the server of the determined target time and delay time;
When transmitting stream data to the terminal through the network, the control means for controlling the transmission speed based on the notified target amount and delay time,
The control means controls the transmission speed so that the amount of stream data stored in the buffer of the terminal changes without exceeding the target amount in the vicinity of the target amount. . A program that describes a streaming method in which a server transmits stream data to a terminal through a network and the terminal plays back the stream data while receiving the stream data,
A target amount determination step in which the terminal determines a target amount of stream data to be stored in its buffer in relation to its buffer capacity and network transmission capability;
The delay time from when the terminal writes the head data of the stream to its own buffer until the data is read and playback is started, as long as it does not exceed the value obtained by dividing the buffer capacity by the transmission capacity Determine the delay time determination step,
The terminal notifies the server of the determined target time and delay time;
A program describing a streaming method including a control step of controlling a transmission rate based on a notified target amount and delay time when a server transmits stream data to a terminal through a network. A recording medium that records a program in which a server transmits stream data to a terminal through a network and a terminal describes a streaming method for reproducing the stream data while receiving the stream data,
A target amount determination step in which the terminal determines a target amount of stream data to be stored in its buffer in relation to its buffer capacity and network transmission capability;
The delay time from when the terminal writes the head data of the stream to its own buffer until the data is read and playback is started, as long as it does not exceed the value obtained by dividing the buffer capacity by the transmission capacity Determine the delay time determination step,
The terminal notifies the server of the determined target time and delay time;
A recording medium on which is recorded a program describing a streaming method including a control step of controlling a transmission speed based on a notified target amount and delay time when a server transmits stream data to a terminal through a network.

Images