JP4071140B2 - Data communication apparatus, method and program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to data communication technology, and in particular, to communication technology for performance data and video data.
[0002]
[Prior art]
In communication of performance data between remote locations, it will be possible to have more enjoyable communication if the video of the other party captured by the camera can be displayed. However, compared with performance data (MIDI data), the amount of video data is large and the communication load becomes large. In order to perform an ensemble, it is necessary to transmit and receive performance data as fast as possible, so it is not very desirable to communicate video data that affects the communication state.
[0003]
In addition, when capturing video data with a camera, the load on the CPU increases, which may affect other processes. At this time, it is conceivable that communication of performance data when performing an ensemble is hindered.
[0004]
As described above, it is desired that the other party's video can be seen in the ensemble between the remote locations, but this actually causes a delay in the performance and is difficult to realize.
[0005]
Moreover, the following patent document 1 is disclosed.
[0006]
[Patent Document 1]
JP 2000-20055 JP
[Problems to be solved by the invention]
An object of the present invention is to provide a data communication technique capable of preventing communication delay of performance data when transmitting performance data and video data.
[0008]
[Means for Solving the Problems]
According to one aspect of the present invention, performance data input means for inputting performance data, video data input means for inputting video data from a camera, performance data transmission means for transmitting the input performance data, and the performance Provided by a data communication device having a video data transmission means that does not permit transmission of the input video data before the lapse of a predetermined time after data input and permits transmission of the input video data after the lapse of a predetermined time Is done.
According to another aspect of the present invention, a performance data input step for inputting performance data, a video data input step for inputting video data from a camera, a performance data transmission step for transmitting the input performance data, A data communication method comprising: a video data transmission step that does not permit transmission of the input video data before the elapse of a predetermined time after performance data input and permits transmission of the input video data after the elapse of a predetermined time. Provided.
According to still another aspect of the present invention, a performance data input step for inputting performance data, a video data input step for inputting video data from a camera, a performance data transmission step for transmitting the input performance data, Before the predetermined time elapses after the performance data is input, transmission of the input video data is not permitted, and after the predetermined time has elapsed, a video data transmission step of permitting transmission of the input video data is executed by a computer. A program is provided.
[0009]
According to the present invention, since transmission of the input video data is not permitted before the elapse of a predetermined time after the performance data is input, it is possible to prevent a delay in transmission of the performance data based on the transmission of the video data. Further, after a predetermined time has elapsed after the performance data is input, transmission of the input video data is permitted, so that the video data can be transmitted.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 shows a first configuration example of a data communication system according to an embodiment of the present invention. The two terminals 110 and 120 are directly connected so that two performers can perform ensembles at remote locations, and communicate performance data, audio data, and video data with each other. For example, the terminal 110 is a server and the terminal 120 is a client. The terminals 110 and 120 each have a keyboard 101, a sound source 102, a microphone 103, a speaker 104, a camera 105, and a display 106.
[0011]
When the performer plays the keyboard (electronic musical instrument) 101, the keyboard 101 generates performance data and supplies it to the terminal 110. The performance data is, for example, MIDI (Musical Instrument Digital Interface) data. When the MIDI data is input from the keyboard 101, the terminal 110 outputs the MIDI data to the sound source 102 and transmits it to the terminal 120 via the Internet (network) 130. The sound source 102 generates a tone signal based on the MIDI data, and causes the speaker 104 to generate a tone. Similarly, at the terminal 120, a musical sound is generated by the sound source 102 and the speaker 104.
[0012]
The microphone 103 generates a sound signal based on the sound such as the performer's singing voice and outputs the sound signal to the terminal 110. When an analog audio signal is input from the microphone 103, the terminal 110 converts the audio signal into digital audio data, generates a sound from the speaker 104, and transmits the sound to the terminal 120 via the Internet 130. Similarly, at the terminal 120, sound is produced from the speaker 104.
[0013]
The camera 105 generates video data such as a performer or a score and outputs it to the terminal 110. When the video data is input from the camera 105, the terminal 110 displays the video data on the display 106 and transmits it to the terminal 120 via the Internet 130. Similarly, the terminal 120 displays the video on the display 106.
[0014]
Similarly, at the terminal 120, the generated MIDI data, audio data, and video data are reproduced and transmitted to the terminal 110. In the terminal 110, the same MIDI data, audio data, and video data are reproduced. Thereby, two performers can perform an ensemble in a remote place in real time.
[0015]
Under the Internet 130 environment, the terminals 110 and 120 communicate audio data and video data via a TCP (Transmission Control Protocol) port 131 and communicate MIDI data via a UDP (User Datagram Protocol) port 132. . MIDI data is required to be faster (real time) than audio data and video data. Therefore, the MIDI data uses the UDP port 132, and the audio data and video data use the TCP port 131. Only the UDP port may be used without dividing the TCP port 131 and the UDP port 132. In that case, it is preferable to prioritize the data and send / receive MIDI data with priority. In the two-person mode, of the two terminals 110 and 120, the one that requests the connection first is the server, and the other is the client.
[0016]
FIG. 2 shows a second configuration example of the data communication system according to the embodiment of the present invention. The four terminals 211 to 214 are connected via a session server 260 so that four performers can perform ensembles at remote locations, and communicate performance data, audio data, and video data with each other. This mode is used when there are three or more people, but the case of four people will be described as an example. The four terminals 211 to 214 are all clients, and the session server 260 is a server. Similarly to FIG. 1, the terminals 211 to 214 each have a keyboard 101, a sound source 102, a microphone 103, a speaker 104, a camera 105, and a display 106.
[0017]
The terminals 211 to 214 and the session server 260 are connected via the Internet (network) 250. Under the Internet 250 environment, the terminals 211 to 214 communicate audio data, video data, and the like via the TCP port 251, and communicate MIDI data and the like via the UDP port 252. The terminals 211 to 214 communicate MIDI data, audio data, and video data with each other via the session server 260. Thereby, three or more (for example, four) performers can perform an ensemble in real time in a remote place.
[0018]
As shown in FIG. 1, when performing an ensemble with two terminals, it is preferable to communicate by directly connecting the two terminals. If the terminals are directly connected, high-speed communication can be performed. However, when three or more terminals are directly connected, the load increases because each terminal communicates with a plurality of terminals. Therefore, as shown in FIG. 2, when performing an ensemble with three or more terminals, it is preferable to connect and communicate with three or more terminals via the session server 260. When a terminal is connected via the session server 260, each terminal only needs to communicate with the session server 260 only, so that the load on each terminal can be reduced.
[0019]
FIG. 3 shows a flow of data processing when the first terminal 310 and the second terminal 320 perform communication. The vertical axis is time. The left column is the process of the first terminal 310, and the right column is the process of the second terminal 320.
[0020]
First, MIDI data will be described. When the key is pressed by the first terminal 310, the first terminal 310 transmits MIDI data to the second terminal 320 to perform sound generation processing. The second terminal 320 receives the MIDI data and performs sound generation processing. When the key is released at the first terminal 310, the first terminal 310 transmits MIDI data to the second terminal 320 and performs a mute process. The second terminal 320 receives the MIDI data and performs a mute process.
[0021]
Similarly, when the key is depressed at the second terminal 320, the second terminal 320 transmits MIDI data to the first terminal 310 to perform sound generation processing. The first terminal 310 receives the MIDI data and performs sound generation processing. When the key is released from the second terminal 320, the second terminal 320 transmits MIDI data to the first terminal 310 and performs a mute process. The first terminal 310 receives the MIDI data and performs a mute process.
[0022]
Next, audio data will be described. Audio data is recorded in the recording buffer. Every predetermined interval (for example, 0.45 seconds) 301, the recording buffer becomes full (full) and is played back. When the recording buffer becomes full in the first terminal 310, the first terminal 310 encodes the audio data and transmits it to the second terminal 320. The second terminal 320 receives the audio data, decodes it, sets it in the reproduction buffer, and reproduces it. Similarly, when the recording buffer becomes full in the second terminal 320, the second terminal 320 encodes the audio data and transmits it to the first terminal 310. The first terminal 310 receives the audio data, decodes it, sets it in the reproduction buffer, and reproduces it.
[0023]
Next, video data will be described. For the video data, a camera capture command is issued at predetermined intervals (for example, 0.5 seconds) 302. When a capture command is issued at the first terminal 310, the first terminal 310 JPEG-compresses the video data, transmits it to the second terminal 320, and displays the video data. The second terminal 320 receives the JPEG data, decompresses the JPEG, and displays it. Similarly, when a capture command is given by the second terminal 320, the second terminal 320 JPEG-compresses the video data, transmits it to the first terminal 310, and displays the video data. The first terminal 310 receives the JPEG data, decompresses the JPEG, and displays it.
[0024]
Compared to MIDI data, the amount of video data is large. Therefore, video data input processing and transmission processing cause a delay in the transmission of MIDI data. MIDI data is required to be real-time compared to video data. That is, the performance delay has a greater adverse effect than the video delay. Hereinafter, a method of preventing communication delay of MIDI data when transmitting MIDI data and video data will be described.
[0025]
FIG. 4 is a diagram illustrating a data communication method according to the present embodiment. The horizontal axis indicates time. The terminal performs preview processing of the camera 105 (FIG. 1), for example, at regular intervals Tprv. The preview process is a process for acquiring video data from the camera 105, inputting it, and storing it in a memory. The interval Tprv is, for example, 0.033 [s].
[0026]
The interval Tdsp corresponds to the interval 302 in FIG. 3 and is 0.5 [s], for example, and is an interval of capture video data transmission. The terminal performs transmission processing of video data stored in the memory by preview at timings t1, t2, t3, etc. at regular intervals Tdsp.
[0027]
Next, the case where the MIDI data M1 and M2 are input from the keyboard 101 (FIG. 1) or the case where they are input (received) from another terminal via the network will be described. The MIDI data M1 is input between timings t1 and t2, and the MIDI data M2 is input between timings t2 and t3. The initial value of the counter value 401 is 0. Until the MIDI data M1 is input, the counter value 401 is zero. When the MIDI data M1 is input, the counter value 401 is set to a predetermined value (for example, 1 [s]). Thereafter, the counter value 401 is decremented at intervals of Tchk (for example, Tchk = 5 [ms]). Next, when the MIDI data M2 is input, the counter value 401 is set to a predetermined value again. After that, the counter value 401 is again decremented by the interval Tchk. Eventually, the counter value 401 becomes zero.
[0028]
Periods T1 and T3 are periods in which the counter value 401 is zero. The period T2 is a period in which the counter value 401 is 1 or more. Since the counter value 401 is 0 in the periods T1 and T3, the preview process at the interval Tchk and the video data transmission process at the interval Tdsp are permitted. In the period T2, since the counter value 401 is 1 or more, the preview process at the interval Tprv and the video data transmission process at the interval Tdsp are not permitted.
[0029]
In the period T2 before the predetermined time elapses after the MIDI data M1 and M2 are input, the video data preview (input) processing and transmission processing are not permitted, so that transmission delay of MIDI data based on the video data can be prevented. In the periods T1 and T3 after the lapse of a predetermined time after the MIDI data is input, the video data preview (input) processing and transmission processing are permitted, so that the video data can be transmitted.
[0030]
In the above description, the case where the interval Tprv for previewing (inputting) video data is shorter than the interval Tdsp for transmitting video data has been described as an example. The interval Tprv for previewing (inputting) video data may be the same as the interval Tdsp for transmitting video data. Thereby, it is possible to reduce the load due to the preview processing of the terminal.
[0031]
The receiving terminal cannot periodically receive video data when MIDI data is being input. At that time, the receiving terminal continues to display the previously received video data.
[0032]
FIG. 5 is a hardware configuration diagram of the computer of the terminal and the session server. A central processing unit (CPU) 502, a ROM 503, a RAM 504, a network interface 505, an input device 506, an output device 507, and an external storage device 508 are connected to the bus 501.
[0033]
The CPU 502 performs data processing and calculation, and controls the above-described constituent units connected via the bus 501. The ROM 503 stores a boot program in advance, and the computer is activated when the CPU 502 executes the boot program. A computer program is stored in the external storage device 508, and the computer program is copied to the RAM 504 and executed by the CPU 502. This computer can execute the processes of FIGS. 6 to 8 described later and the like by executing a computer program.
[0034]
The external storage device 508 is, for example, a hard disk storage device or the like, and the stored content does not disappear even when the power is turned off. The external storage device 508 can record a computer program, MIDI data, audio data, video data, and the like on a recording medium, and can read out the computer program and the like from the recording medium.
[0035]
The network interface 505 can transmit and receive computer programs and the like in addition to MIDI data, audio data, and video data to and from a network (Internet, MIDI, and the like). The input device 506 is, for example, a keyboard and a pointing device (mouse), and can perform various designations or inputs. The output device 507 is a display or the like and can display various information.
[0036]
FIG. 6 is a flowchart showing processing of the terminal when MIDI data is input from the keyboard 101 (FIG. 1) or input (received) from another terminal via the network. In step S601, the preview process is stopped. That is, the terminal stops the process of periodically taking and inputting video data from the camera. The preview process is not performed until the start of the preview process is instructed in step S704 in FIG. Next, in step S602, a constant is set in the counter. In step S603, the input MIDI data is transmitted to another terminal via the network. Other terminals receive the MIDI data and perform sound generation processing and the like. In step S604, the MIDI data is processed. If the MIDI data is a key-on event, sound generation processing is performed, and if the MIDI data is a key-off event, mute processing is performed. Thus, the process ends.
[0037]
FIG. 7 is a flowchart showing timer interrupt processing performed at the interval Tchk (FIG. 4). In step S701, it is checked whether or not the counter value is greater than zero. If greater than 0, the process proceeds to step S702, and if not greater, the process ends. In step S702, the counter value is decremented. In step S703, it is checked whether the counter value is zero. If it is 0, the process proceeds to step S704, and if it is not 0, the process ends. In step S704, preview processing is started. Thereafter, the terminal periodically captures and inputs video data from the camera. Thus, the process ends.
[0038]
FIG. 8 is a flowchart showing timer interrupt processing performed at the interval Tdsp (FIG. 4). In step S801, it is checked whether the counter value is 0 or not. If it is 0, the process proceeds to step S802, and if it is not 0, the process ends. In step S802, the video data captured from the camera by the preview process is acquired as capture data. In step S803, the captured video data is transmitted to another terminal via the network. Other terminals receive and display the video data. In step S804, the captured video data is displayed on the display. Thus, the process ends.
[0039]
As described above, according to the present embodiment, the preview (input) process, the transmission process, and the display process of the video data are not permitted before the predetermined time has elapsed after the MIDI data is input. Delay can be prevented. In addition, after a predetermined time has elapsed after the MIDI data is input, video data preview (input) processing, transmission processing, and display processing are permitted, so that the video data can be transmitted, and the self video can be transmitted to the partner terminal. Can be displayed.
[0040]
This embodiment can be realized by a computer executing a program. Also, means for supplying a program to a computer, for example, a computer-readable recording medium such as a CD-ROM recording such a program, or a transmission medium such as the Internet for transmitting such a program is also applied as an embodiment of the present invention. Can do. A computer program product such as a computer-readable recording medium in which the above program is recorded can also be applied as an embodiment of the present invention. The above program, recording medium, transmission medium, and computer program product are included in the scope of the present invention. As the recording medium, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.
[0041]
The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.
[0042]
【The invention's effect】
As described above, since transmission of the input video data is not permitted before the elapse of a predetermined time after the performance data is input, performance data transmission delay due to transmission of the video data can be prevented. In addition, after a predetermined time has elapsed after the performance data is input, transmission of the input video data is permitted, so that the video data can be transmitted.
[Brief description of the drawings]
FIG. 1 is a diagram showing a first configuration example of a data communication system according to an embodiment of the present invention.
FIG. 2 is a diagram showing a second configuration example of the data communication system according to the embodiment of the present invention.
FIG. 3 is a diagram illustrating a flow of data processing when a first terminal and a second terminal perform communication.
FIG. 4 is a diagram showing a data communication method according to the present embodiment.
FIG. 5 is a hardware configuration diagram of a computer of a terminal and a session server.
FIG. 6 is a flowchart showing processing of a terminal when MIDI data is input from a keyboard.
FIG. 7 is a flowchart showing timer interrupt processing performed at an interval Tchk.
FIG. 8 is a flowchart showing timer interrupt processing performed at an interval Tdsp.
[Explanation of symbols]
101 Keyboard 102 Sound Source 103 Microphone 104 Speaker 105 Camera 106 Display 110, 120 Terminal 130 Internet 131 TCP Port 132 UDP Port 211-214 Terminal 250 Internet 251 TCP Port 252 UDP Port 260 Session Server 501 Bus 502 CPU
503 ROM
504 RAM
505 Network interface 506 Input device 507 Output device 508 External storage device

Claims (11)

Performance data input means for inputting performance data;
Video data input means for inputting video data from the camera;
Performance data transmission means for transmitting the input performance data;
A data communication apparatus comprising: a video data transmitting unit that does not permit transmission of the input video data before a predetermined time elapses after the performance data is input and permits transmission of the input video data after a predetermined time elapses . 2. The data communication apparatus according to claim 1, wherein the video data input means does not input video data from the camera before a predetermined time elapses after the performance data is input, and inputs video data from the camera after a predetermined time elapses. 3. The data communication apparatus according to claim 2, wherein the video data transmitting means transmits the input video data at regular intervals after a predetermined time has elapsed after the performance data is input. 4. The data communication apparatus according to claim 3, wherein the video data input means inputs video data from the camera at a constant interval after a predetermined time has elapsed after the performance data is input. 5. The data communication apparatus according to claim 4, wherein an interval for inputting the video data is shorter than an interval for transmitting the video data. 5. The data communication apparatus according to claim 4, wherein an interval for inputting the video data is the same as an interval for transmitting the video data. The data according to any one of claims 1 to 6, wherein the video data transmission means starts counting of a counter when the performance data is input, and permits transmission of video data when the counter reaches a predetermined value. Communication device. The data communication device according to claim 1, wherein the performance data input unit inputs performance data from an electronic musical instrument. The data communication apparatus according to claim 1, wherein the performance data is MIDI data. A performance data input step for inputting performance data;
A video data input step for inputting video data from the camera;
A performance data transmission step of transmitting the input performance data;
A data communication method comprising: a video data transmission step that does not permit transmission of the input video data before a predetermined time elapses after the performance data is input and permits transmission of the input video data after a predetermined time elapses . A performance data input step for inputting performance data;
A video data input step for inputting video data from the camera;
A performance data transmission step of transmitting the input performance data;
Before the predetermined time elapses after the performance data is input, transmission of the input video data is not permitted, and after the predetermined time has elapsed, a video data transmission step of permitting transmission of the input video data is executed by a computer. Program for.

Images