JP2002158983A - Multicast video conference terminal and video conference system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video conference terminal in which a participator can recognize the speaker in a conference easily. SOLUTION: A video signal 104 is digitized at a video input section 4 and encoded at a video encoding section 5 before being transmitted as encoded video data 110. A vocal information processing section 3 transmits vocal information 113 based on voice level data 102 from a voice input section 1, lip motion information 107 from a lip motion discriminating section 7, and right of speaking information 112 from a right of speaking token processing section 13. The video encoding section 5 alters the encoding method of video data 105 upon receiving the vocal information 113. Encoded video data 120 from other terminal is decoded at a video decoding section 11 and a composited image is delivered from a video output section 10 to a display monitor. A composited image where the image only at a part from the speaker terminal has a high quality is presented on the display monitor.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video conference terminal device and a video conference system, and more particularly to a video conference terminal device and a video conference system capable of easily recognizing speakers in a conference.

[0002]

2. Description of the Related Art As a typical configuration example of a multipoint video conference system using video and audio, a multipoint control unit (hereinafter sometimes simply referred to as "MCU") is used. Some use an intra-group broadcast function called multicast.

When using an MCU, as shown in FIG. 5, a video conference terminal device 32-1 installed at a plurality of points is used.
To 32-n are connected to the MCU 31 and the MCU 31
Receives audio and video from each of the terminal devices 32-1 to 32-n, synthesizes them, and
By transmitting to -n, a video conference can be realized. The MCU 31 and the terminal devices 32-1 to 32-32
n may be connected via a network.

When using multicast, as shown in FIG. 6, audio and video from each terminal device 41-1 to 41-n are transmitted to all other terminal devices 41-1 to 41-n.
−n via the network 42, and the terminal devices 41-1 to 41-n can realize a video conference by reproducing the received audio and video individually or in combination.

[0005] It is indispensable for the teleconference participant to recognize the speaker in the conference in order for the conference to proceed smoothly, but the terminal devices 32-1 to 32-n and 41-1 to 41-n are In any case, since the voice from the other terminal device is synthesized and reproduced, the conference participant cannot easily identify the speaker terminal, that is, the terminal device to which the speaking conference participant belongs. Therefore, in a system using the MCU, the MCU identifies the speaker terminal, transmits information identifying the terminal device that is speaking to each terminal device, and each terminal device determines the terminal device number based on the received identification signal. Is displayed to identify the speaker terminal. Also, by transmitting the video signal from the speaker terminal to another terminal device as a video signal different from the video signal from another terminal device, for example, a video signal with improved video quality, the speaker terminal can be identified. Can also be performed.

[0006]

However, when displaying the number of the speaker terminal, it is necessary to check the video of the corresponding terminal device, and it is difficult for the user to immediately recognize the speaker. If there are a plurality of speakers, the recognition is further delayed. Also, when displaying an image with improved image quality, it is easy to recognize the speaker, but the burden on the MCU increases, and it is necessary to transmit high-quality images from all terminal devices to the MCU in advance. There is a lot of waste. further,
When attempting to recognize a speaker in a system using multicasting, the burden of specifying a speaker terminal and processing a video signal from the speaker terminal is large, which is not practical.

An object of the present invention is to provide a video conference terminal device that enables a video conference participant to easily recognize a speaker in a conference, and a video conference system using such a video conference terminal device. Aim.

[0008]

SUMMARY OF THE INVENTION A multicast video conference terminal apparatus of the present invention is used in a multipoint video conference system for transmitting and receiving audio information and video information between multipoints. An utterance information processing unit that transmits utterance information indicating that the information is valid or that audio information is being transmitted, and performs an encoding process on an input video signal to create encoded video information to be transmitted A video encoding processing unit, and a video information reproduction processing unit that performs a reproduction process of the received encoded video information, wherein the utterance information is also input to the video encoding processing unit, and the video encoding processing unit When the utterance information is input, the content of the encoding process of the input video signal is changed. In this way, when the conference participant is speaking, the changed video information transmitted on the transmitting side is received, so the receiving side can identify the speaker terminal without performing any special processing. Can output a natural image.

The video encoding processing section performs an encoding process with improved video quality when the utterance information is input.

[0010] The encoding process for improving the video quality may include at least one of a process of changing a monochrome image to a color image, a process of changing a non-zoom image to a zoom image of a speaker, and an improvement of a frame rate. Is the choice.

[0011] The multicast video conference terminal device of the present invention further has a floor token processing unit for transmitting and receiving a floor token circulated between a plurality of multicast video conference terminals participating in the video conference. The utterance information processing section outputs the utterance information when the terminal device user inputs a utterance voice while holding the floor right token. In this case,
It is possible to prevent the collision of the speech at the start of the speech of the conference participant.

[0012] The determination of the presence or absence of a speech input by the user of the terminal device is based on an input level of an input speech signal to the terminal device and lip movement information of the user obtained from an input video signal. It is what is done.

[0013] Further, when the voice information processing unit transmits the voice information, the voice right token processing unit stops the circulation of the voice right token, and outputs the voice voice from another multicast video conference terminal device. While receiving the information, when the stop of the circulation of the floor right token continues for a certain period of time, the speech is transmitted between other multicast video conference terminals except the multicast video conference terminal transmitting the voice information. This is to restart the circulation of the right token.

[0014] In addition, the fixed period until the circulation of the floor token is reopened is at least the maximum delay time from the time of voice input to the time of reproduction on another multicast video conference terminal or the maximum delay time. It is the time obtained by adding the time that the contents can be grasped.

[0015] The circulation of the floor token is performed by unicast or multicast.

Further, the multicast video conference terminal device of the present invention is arranged such that, when the terminal device user has made a speech input and the speech information is not output,
It outputs a warning indicating that the remark is not valid or that the voice information has not been transmitted.

Further, the warning is performed by displaying the warning in the field of view of the speaker.

Further, the utterance information indicating that the transmitted voice information is valid is transmitted when the voice input of the terminal device user has been lost for a certain period of time.

Further, the video information reproduction processing unit includes a video decoding unit that performs a decoding process by lowering a frame rate of a video signal from a terminal device having a lower priority.

Further, as the terminal device for lowering the frame rate, at least one of a non-speaker terminal and a terminal which has transmitted utterance information earlier among a plurality of speaker terminals is selected.

The teleconference system of the present invention is a system in which a plurality of the above-mentioned multicast video conference terminal devices are mutually connected.

Further, the interconnection uses a network.

[0023]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below with reference to FIGS. FIG. 1 is a schematic functional block diagram of an embodiment of a multicast video conference terminal device (hereinafter, may be simply referred to as a “terminal device”) of the present invention. This terminal device is used by being connected to a network as in the system shown in FIG. 6, and receives an audio signal 100 from a microphone, a video signal 104 from a camera, and encoded audio data 103.
And the video data 108 to another terminal device via the network. Further, it receives encoded audio data 116 and video data 120 from another terminal device via a network, and sends an audio signal 119 based on the received encoded audio data 116 to a speaker,
Is output to the display monitor based on the video signal 122. Furthermore, while transmitting and receiving the floor right tokens 110 and 111, the terminal device transmits voice information 113 indicating that the terminal device holds the floor and outputs voice information to another terminal device. The same utterance information 114 is received.

An audio signal 100 input from a microphone is digitized by an audio input unit 1, and digitized audio data 101 is sent to an audio encoding unit 2. The audio encoding unit 2 encodes the audio data 101 by a predetermined encoding method, and transmits the encoded audio data 101 as encoded audio data 103. Further, the voice input unit 1 sends voice level data 102 indicating the voice level of the voice signal from the microphone to the utterance information processing unit 3.

The video signal 104 input from the camera is digitized by the video input unit 4, and the digitized video data 105 is encoded by the video encoding unit 5 by a predetermined encoding method. Then, it is transmitted as encoded video data 110. The video data 105 is also sent to the lip extracting unit 6, and the lip is extracted. The extracted lip image data 106 is sent to the lip movement discriminating unit 7 and
When a movement of a predetermined size is detected, the lip movement information 10
7 is sent to the utterance information processing unit 3. The movement of the lips is performed by detecting changes in a plurality of lip image frames.

The utterance information processing unit 3 receives the utterance level data 102 from the voice input unit 1, the lip movement information 107 from the lip movement discriminating unit 7, and the utterance right information 112 from the utterance right token processing unit 13 described later. The utterance information 113 is transmitted based on the utterance information. The speech information 113 indicates to other terminal devices that the encoded speech data 103 transmitted from the speech encoding unit 2 is valid, and that the speaker holds the right to speak and includes speech information during speech. It is shown. This utterance information 113
Is also sent to the floor token processing unit 13 to stop the circulation of the floor token.

The utterance information 113 is stored in the video encoding unit 5.
Also sent to. When the utterance information 113 is input, the video encoding unit 5 changes the encoding method of the video data 105. The reason for changing the video encoding method is to easily identify the speaker at the time of reproduction. Therefore, it is preferable to change to a method that relatively improves the video quality. Examples of improving the video quality include a change from a monochrome image to a color image, a change from a non-zoom image to a speaker's zoom image, and an improvement in frame rate. Further, these may be combined.

The encoded voice data 103 and the utterance information 11 transferred to another terminal device via the network
3, and the encoded video data 108 include the ID of the terminal device.
Is added.

The floor token processing unit 13 receives the token 110 circulating through a plurality of terminal devices participating in the video conference at a predetermined timing, and transmits the token 111. While the token is being acquired, the right to speak 112 is output to the speech information processing unit 3. When the voice is detected during the acquisition of the token and the voice information 113 is transmitted, the voice right token processing unit
Stop sending. Therefore, as will be described later, at least for a certain period, the other terminal cannot obtain the right to speak, and the uttered voice is transmitted to the other terminal device as coded voice data, and voice information with less collision and noise is generated. Will be played.

As a method of circulating the floor token, there is a unicast method as shown in FIG. 2A or a multicast method as shown in FIG. 2B. The unicast method is to receive a floor token from a predetermined terminal device and transmit the floor token to a predetermined terminal device. For example, if the terminal device of FIG. 1 is a terminal device C that circulates by unicast, the terminal device B receives the floor right token 110 from the terminal device B, and
The floor token 111 is transmitted. Unicast transmission has a small amount of communication and a simple configuration. However, when one terminal device does not operate normally, a token is not transmitted at all and the system is completely stopped. On the other hand, the multicast method sends a token to all terminal devices,
The example of FIG. 2B shows a state in which the terminal device B transmits that "the token has been turned from B to C" to all other terminal devices. This method is resistant to network failures, and all terminal devices can grasp the current state of the system, so that more complicated control can be handled.

Encoded audio data 11 from another terminal device
6 is sent to a speaker via an audio decoding unit 8, a mixer unit 9, and an audio output unit 10. The utterance information receiving unit 14
Then, the utterance information 114 transmitted by another terminal device is received. The encoded audio data 116 is decoded by the audio decoding unit 8 and sent to the mixer unit 9 to synthesize audio signals from a plurality of terminal devices. The mixer unit 9 outputs the utterance information 115 from the utterance information receiving unit 14. Selectively synthesizes only the audio data transmitted from the terminal device designated by the user. Therefore, since only the audio signal obtained from the effective encoded audio data of the speaker drives the speaker via the audio output unit 10, noise is effectively reduced and the performance of the video conference terminal is improved.

From another terminal device, the encoded video data 120
Are decoded by the video decoding unit 11, and are sent to the display monitor where the images are synthesized by the video output unit 10.
Since the synthesized image displayed on the display monitor is a high-quality image only from the speaker terminal, the speaker can be easily recognized. Further, since no special processing is performed on the receiving side, the amount of data processing of the terminal device does not increase significantly. In particular, when the transmitting side transmits a zoom image, the amount of video data hardly changes,
There is almost no increase in throughput.

If the patrol is stopped when the talker token is acquired, the talk right token does not circulate and other speakers cannot speak. Resumes patrol.

Referring to FIG. 3, the circulation of the floor token will be described in more detail. The figure shows an example of patrol by unicast. FIG. 3A shows a case where no one is speaking, and the speaking right token circulates in the order of A, B, C, D, E, A,. FIG.
(b) is a case in which C speaks while acquiring the speaking right token. At this time, the circulation of the speaking right token is stopped, and the utterance information is transmitted from C to another terminal device. FIG. 3C shows a state where the circulation of the floor token has been resumed. In this case, while A holds the right to speak and continues to speak, D, E, A,
B, D,... The patrol is resumed by the terminal device that is to acquire the invention right token next to the terminal device that has stopped the floor right token, in this case, the floor right token processing unit 13 of D. D's voice token processing unit 13
Indicates that the floor token from C is not circulated and that C
By transmitting the utterance information from, it is possible to recognize that the floor token has been stopped at C. Then, if this state is continued for a certain period of time, the floor right token is transmitted to E to restart the tour. In this tour, B can recognize that the previous floor right token has been stopped at C, which is the destination of the normal floor right token, and therefore transmits the floor right token to D.

In consideration of preventing the collision of the utterance, it is necessary that the certain period of time before resuming the circulation of the speaking right token is at least longer than the maximum delay time from the time of voice input to the time of reproduction on another multicast video conference terminal. is there. Preferably, it is desirable that the maximum delay time be added to the time required to grasp the content of the utterance of the speaker.

When the circulation of the floor token is resumed in such a manner, voice information from a plurality of terminal devices may collide with each other. However, unlike the collision at the beginning of the speech, the conference does not proceed smoothly. .

The video decoding unit 11 decodes the coded video data 120 transmitted from a plurality of terminal devices. However, the number of terminal devices participating in the conference increases or the number of terminal devices transmitting voice information increases. In this case, the decoding capability of the video signal may be exceeded. In this case, the video decoding unit 11
Performs a decoding process by lowering the frame rate of a video signal from a terminal device having a lower priority. The process of changing the frame rate according to the processing device capability is disclosed in, for example, JP-A-8-163556. As a method of selecting a terminal device for lowering the frame rate, there are a method of selecting a non-speaker terminal and a method of selecting one of a plurality of speaker terminals to which utterance information has been transmitted earlier. in this case,
The utterance information 115 from the utterance information receiving unit 14 is input to the video decoding unit 11.

According to the present invention, the voice information uttered only when the floor right token is acquired can be reproduced by another terminal. Therefore, even if the speaker speaks, it is not recognized by the other terminal device. Things happen. If such a situation occurs frequently, the smooth progress of the conference is hindered. Therefore, it is effective to warn the speaker that other terminal devices have not recognized the situation. This warning is issued when the sound level data 102 is greater than a certain value and the lip movement information 107
This is performed when the utterance information 113 is not transmitted even though is detected. The warning can be issued by various means, but it is preferable that the warning be displayed in the visual field of the speaker, for example, on a display monitor.

Regarding the transmission / reception processing of the audio signal, the terminal device shown in FIG. 1 performs the transmission processing of the encoded audio data irrespective of the acquisition of the right to speak, and selectively performs the synthesis processing on the receiving side. However, the transmission process may be selectively performed. FIG. 4 shows a main part of a schematic functional block diagram in that case. In the terminal device of FIG. 4, the encoded audio data from the audio encoding unit 2 is transmitted via the audio information transmission control unit 15. The audio information transmission control unit 15 transmits the encoded audio data from the audio encoding unit 2 only while the utterance information 113 is being transmitted. With this configuration, since only the encoded audio data from the terminal device that is transmitting the utterance information is input to the audio decoding unit 8, the mixer unit 9 needs to perform the selective synthesis based on the received utterance information 115. Disappears.

[0040]

As is apparent from the above description, according to the present invention, a video conference participant can easily recognize a speaker in a conference, and a smooth multipoint video conference can be held. it can.

[Brief description of the drawings]

FIG. 1 is a schematic functional block diagram of an embodiment of the present invention.

FIG. 2 is a diagram for explaining a token patrol method;

FIG. 3 is a diagram illustrating a method of circulating a floor token

FIG. 4 is a functional block diagram of a main part of an example in which a transmission process of encoded audio data is changed.

FIG. 5 is a configuration example of a multipoint video conference system using an MCU.

FIG. 6 is a configuration example of a multipoint video conference system using a multicast terminal device.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Audio input part 2 ... Audio encoding part 3 ... Speech information processing part 4 ... Video input part 5 ... Video encoding part 6 ... Lip extraction part 7 ... Lip movement discriminating unit 8 ... Audio decoding unit 9 ... Mixer unit 10 ... Audio output unit 11 ... Video decoding unit 12 ... Video output unit 13 ... Speaking right token processing unit 14 ... .Speech information receiving unit 15: Voice information transmission control unit

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Toshio Oka 1006 Kadoma Kadoma, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. 72) Inventor Satoshi Senga 1006 Kazuma Kadoma, Kadoma-shi, Osaka Matsushita Electric Industrial Co., Ltd. F-term (reference) 5C064 AA02 AB01 AB02 AC04 AC09 AC12 AC16 AD02 AD13

Claims (17)

[Claims] 1. A multicast video conference terminal device used in a multipoint video conference system for transmitting and receiving audio information and video information between multipoints, wherein the transmitted audio information is valid or An utterance information processing unit that transmits utterance information indicating that transmission is in progress; a video encoding processing unit that performs an encoding process on an input video signal and creates encoded video information to be transmitted; A video information reproduction processing unit that performs a reproduction process of encoded video information, wherein the utterance information is also input to the video encoding processing unit, and the video encoding processing unit is configured to input the vocal information,
A multicast video conference terminal device for changing the encoding process of an input video signal. 2. The multicast video conference terminal device according to claim 1, wherein the video encoding processing unit is configured to:
A multicast video conference terminal device that performs an encoding process with improved video quality. 3. The multicast video conference terminal device according to claim 2, wherein the encoding process for improving the video quality includes a process of changing a monochrome image to a color image, and a process of changing a non-zoom image to a zoom image of a speaker. To a multicast video conference terminal device, which is at least one of a change process to a frame rate and an improvement in a frame rate. 4. The multicast video conference terminal device according to claim 1, further comprising: transmitting and receiving a floor right token circulated among a plurality of multicast video conference terminal devices participating in the video conference. A voice token processing unit for performing processing, wherein the voice information processing unit outputs the voice information when the voice voice input of the terminal device user is received when the voice token is held. Multicast video conference terminal. 5. The multicast video conference terminal device according to claim 4, wherein the determination of the presence or absence of a speech voice input by the terminal device user is based on an input level of an input voice signal to the terminal device and an input video signal. A multicast video conference terminal device that is performed based on the user's lip movement information obtained from the user. 6. The multicast video conference terminal device according to claim 4, wherein said floor token processing unit circulates said floor token when said voice information processing unit transmits said voice information. And while receiving the utterance information from the other multicast video conference terminal device, when the stop of the circulation of the floor token continued for a certain period of time,
A multicast video conference terminal device for resuming the circulation of the floor token among other multicast video conference terminal devices except the multicast video conference terminal device transmitting the utterance information. 7. The multicast video conference terminal device according to claim 6, wherein at least a certain period of time from the time of voice input to the time of reproduction on another multicast video conference terminal device is reached for a certain period of time until the circulation of the floor token is restarted. Multicast videoconferencing terminal device that is the maximum delay time. 8. The multicast video conference terminal device according to claim 7, wherein, during a predetermined period until resuming the circulation of the floor right token, the maximum delay time is set to such a time that the contents of the speaker can be grasped. Multicast video conference terminal device that is the added time. 9. The multicast video conference terminal device according to claim 4, wherein said floor token is a multicast circulated between said plurality of multicast video conference terminal devices by unicast. Video conference terminal device. 10. The multicast video conference terminal device according to claim 4, wherein the floor token is circulated among the plurality of multicast video conference terminal devices by multicast. Conference terminal device. 11. The multicast video conference terminal device according to any one of claims 1 to 10, wherein said terminal device user has a speech input and said speech information is not output. Is a multicast video conference terminal device that outputs a warning indicating that a statement is not valid or that audio information has not been transmitted. 12. The multicast video conference terminal device according to claim 11, wherein the warning is based on a display within a field of view of a speaker. 13. The multicast video conference terminal device according to claim 1, wherein the utterance information indicating that the transmitted audio information is valid is a utterance of the user of the terminal device. A multicast video conference terminal device, in which transmission is stopped when voice input is lost for a predetermined time. 14. The multicast video conference terminal device according to claim 1, wherein the video information reproduction processing unit lowers a frame rate of a video signal from a terminal device having a lower priority. Multicast video conference terminal device including a video decoding unit that performs decoding processing on the video signal. 15. The multicast video conference terminal device according to claim 14, wherein, as the terminal device for lowering the frame rate, a non-speaker terminal, or a terminal that has transmitted utterance information to a destination among a plurality of speaker terminals. A multicast video conference terminal device that selects at least one of the following. 16. A video conference system in which a plurality of the multicast video conference terminal devices according to claim 1 are connected to each other. 17. The video conference system according to claim 16, wherein the interconnection uses a network.