JP5613102B2 - CONFERENCE DEVICE, CONFERENCE METHOD, AND CONFERENCE PROGRAM - Google Patents

Description

  The present invention relates to a conference apparatus, a conference method, and a conference program for performing a remote conference via a network.

  The demand for remote conferences is increasing for the purpose of reducing environmental problems (ecology) and business trip expenses in corporate activities. Conventional conference systems include, for example, a telephone (voice) conference system and a Web (video) conference system that operates on a browser. In the Web conference system, if a browser is started, a Web camera is installed, and a headset is attached, a conference can be held while watching each other's participants. For example, Patent Document 1 describes a multipoint communication system that can acquire or transfer a right to speak.

JP 2004-248145 A

  Such a conference system has a drawback that the quality of video and audio is low. For example, the video resolution is low, the video display size is limited by the small display size, and there is a delay in transmission between video and audio. For this reason, it becomes difficult to read the expressions and gestures of other conference participants that exist remotely.

  As one of the problems that occur due to this, it is not known who is likely to speak next, and at the same time, a collision of utterances in which a plurality of conference participants start speaking. In a meeting where utterance collisions occur frequently, mental stress accumulates in the meeting participants and the progress of the meeting is delayed.

  In the technique of Patent Document 1, a conscious action for acquiring the right to speak is necessary, and the smooth progress of the conference may be hindered. In face-to-face communication, there are a case where a person expresses his intention to speak and a person speaks, and a case where he speaks by transmitting a desire to speak from a natural gesture (exchange of non-linguistic information). The latter case is not considered.

  The present invention has been made in view of the above circumstances, and the purpose of the present invention is to provide a conference device that allows the participant's speech desires to be easily recognized even when the facial expressions and gestures of the participants cannot be read. To provide a meeting method and a meeting program.

  In order to achieve the above object, the present invention provides a conference device that performs a remote conference via a network, the video acquisition means for acquiring video data of the self-participant who uses the conference device, and the self-participant Via a network, an action detecting means for detecting a predetermined action assumed to have an utterance desire from video data; an utterance desire level calculating means for calculating an utterance desire level based on the action detected by the action detecting means; Receiving means for receiving video data and utterance desire levels of other participants from other conference devices, video generating means for generating conference videos in which video data of the self-participants and other participants are arranged, self-participants and others Video editing means for setting information indicating the utterance desire level of the participant to the conference video, and display means for displaying the conference video edited by the video editing means.

  The present invention is a conference method for performing a remote conference via a network, wherein the conference device acquires a video acquisition step of acquiring video data of a self-participant who uses the conference device, and the video data of the self-participant An action detecting step for detecting a predetermined action assumed to have an utterance desire, an utterance desire level calculating step for calculating an utterance desire level based on the action detected in the action detecting step, and another meeting via the network A receiving step of receiving video data and speech desire levels of other participants from the device, a video generating step of generating a conference video in which video data of the self-participants and other participants are arranged, and the self-participants and other participants' A video editing step for setting information indicating an utterance desire level in the conference video, and a table for displaying the conference video edited in the video editing step. And the step, is carried out.

  The present invention is a conference program for causing a computer to execute the conference method.

  ADVANTAGE OF THE INVENTION According to this invention, even if it is a case where a participant's facial expression and gesture cannot be read, the conference apparatus, the conference method, and the conference program which can recognize each participant's speech desire easily can be provided.

1 is an overall configuration diagram of a conference system according to an embodiment of the present invention. It is a flowchart which shows operation | movement of this embodiment. It is a figure which shows an example of a meeting image | video.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is an overall configuration diagram of a conference system according to an embodiment of the present invention. The conference system of the present embodiment is a system for a plurality of conference participants (hereinafter referred to as “participants”) in a remote place to perform a remote conference using a plurality of conference devices 1 connected to a network 9. is there.

  The conference system shown in the figure includes a plurality of conference devices 1 used by each of a plurality of participants, and a conference server 8 connected to these conference devices 1 via a network 9.

  The conference apparatus 1 is, for example, a PC or the like, and includes a video camera 11 that captures images of the self-participant who uses the conference apparatus 1, a microphone 12 that collects sound spoken by the self-participant, and a conference video. A display device (display) 13 for displaying and a speaker 14 for outputting the voices of other participants transmitted from the conference server 8 are provided.

  In addition, the conference apparatus 1 includes a video acquisition unit 21, a motion detection unit 22, an utterance desire level calculation unit 23, a memory 24, a voice acquisition unit 25, a conflict detection unit 26, a video generation unit 27, and a video. An editing unit 28 and a communication unit 29 are provided.

  The video acquisition unit 21 captures video data captured by the video camera 11. The motion detection unit 22 detects a predetermined motion that is assumed to be uttered from the video data of the self-participant.

  The sound acquisition unit 25 takes in sound data collected from the microphone 12. The conflict detection unit 26 detects the conflict from the voice data of the participant.

  The utterance desire level calculation means 23 calculates the utterance desire level based on the motion detected by the motion detector 22 and the conflict detected by the conflict detector 26 and stores it in the memory 24.

  The communication unit 29 transmits the video data captured by the video acquisition unit 21 to the conference server 8 via the network 9 and sends it to the video generation unit 27. In addition, the communication unit 29 transmits the audio data captured by the audio acquisition unit 25 to the conference server 8 via the network 9. In addition, the communication unit 29 receives the audio data of other participants from the conference server 8 via the network 9, and outputs / reproduces the audio data of the other participants via the network 9. Received from the conference server 8 and sent to the video generation unit 27. The communication unit 29 transmits / receives information stored in the memory 24 to / from the other conference apparatus 1 via the network 9 and the conference server 8.

  The video generation unit 27 generates a conference video in which the video data of the self-participant captured by the video acquisition unit 21 and the video data of other participants received by the communication unit 29 are arranged. The video editing unit 28 sets information indicating the utterance desire levels of the self-participant and other participants in the conference video generated by the video generation unit 27 and displays the information on the display device 13.

  The conference server 8 receives the participant's video data, audio data, and information in the memory 24 input from each conference device 1 used by each participant, and transmits them to the other conference devices 1.

  The conference device 1 and the conference server 8 can use, for example, a general-purpose computer system including a CPU, a memory, an external storage device such as an HDD, an input device, and an output device. In this computer system, each function of each device is realized by the CPU executing a predetermined program loaded on the memory. For example, the functions of the conference device 1 and the conference server 8 are as follows. The CPU of the conference device 1 is a program for the conference device 1, and the CPU of the conference server 8 is a program for the conference server 8, respectively. It is realized by executing.

  Further, the program for the conference apparatus 1 and the program for the conference server 8 can be stored in a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, an MO, a DVD-ROM, or distributed via a network. You can also

  Next, the operation of this embodiment will be described.

  Each participant sits in front of the conference apparatus 1 and conducts a remote conference via the network.

  The video acquisition unit 21 captures the video data of the self-participant captured by the video camera 11 and sends it to the operation detection unit 22.

  The motion detection unit 22 detects a predetermined motion of the participant from the video data. The predetermined motion detected here is, for example, a motion that is assumed to have an utterance desire such as “move the hand to the mouth”, “raise hand”, “whit”, “move the body sideways”, and the like.

  The operation detection unit 22 detects such an operation by performing image processing on the video data. For example, in the “whispering” operation, the face direction is detected, and when the face direction moves up and down, it is determined as “whispering”. The “move the hand to the mouth” operation, for example, identifies a skin-colored area that is not adjacent to the face area as a hand area, and when this hand area overlaps the face area, Is determined.

  In the “raising hand” operation, for example, a skin-colored region that is not adjacent to the face region is identified as a hand region, and the center of gravity of the hand region has a predetermined height in the y-axis direction of the image (video data). If (threshold) is exceeded, it is determined that the hand is raised. The “move the body sideways” operation is performed, for example, when face detection is performed, and the center point of the detected face area exceeds the predetermined threshold per unit time and moves in the x-axis direction of the image. "Move to".

  The voice acquisition unit 25 collects the voice of the self-participant input from the microphone 12 and sends it to the conflict detection unit 26. When the voice data is input and the self-participant is not a speaker, the conflict detection unit 26 determines that all the input voice data is “contrast”. For example, each conference device stores a speaker flag in the memory 24 for determining whether or not the self-participant is a speaker. The initial value of the speaker flag is “0” (non-speaker). When this is “1” (speaker), it is determined that the self-participant is a speaker.

  The conference server 8 manages the speaker flags of all conference devices (all participants) and determines who is the current speaker. For example, when voice data is input from a certain conference device when the speaker flags of all the participants are “0”, the participant of the conference device that has performed the input is designated as a speaker, and the participation is performed. The speaker flag of the speaker is set to “1”. That is, the conference server 8 updates the speaker flag in the conference server 8 and transmits a speaker designation notification to the conference device designated as the speaker, and sets the speaker flag in the memory 24 of the conference device to “0”. To "1". Note that, when the participant no longer inputs voice data, the speaker designation is canceled and the speaker flag is updated from “1” to “0”.

  In addition, the conflict detection unit 26, when the speaker flag of the self-participant in the memory 24 is “0” (non-speaker) and the speaker flag of the participant other than the self-participant is “1”. When the voice data is input, it is determined that the voice data is “consideration”. At this time, the speaker flag of the participant remains “0”. When the voice data is input, the conflict detection unit 26 inquires of the conference server 8 whether or not there is another participant whose speaker flag is “1”, and another participant whose speaker flag is “1”. If there is a person, it is determined that it is “contradictory”.

  When the voice data is input when the speaker flag of the self-participant in the memory 24 is “1” (speaker), the conflict detection unit 26 does not make an inquiry to the conference server 8 and Is not determined. At this time, the speaker flag in the memory 24 remains “1”.

  Then, the motion information detected by the motion detector 22 and the conflict information detected by the conflict detector 26 are sent to the utterance desire level calculator 23.

  FIG. 2 is a flowchart showing processing of the utterance desire level calculation unit 23. The utterance desire level calculation unit 23 repeatedly performs the process shown in FIG. 2 at a predetermined timing.

  The utterance desire level calculation unit 23 is not shown when the motion detection unit 22 detects any one of “move the hand to the mouth”, “whit”, and “move the body sideways” (S11: YES). “1” is added to the value of the counter in the storage unit (S12). Note that the initial value of the counter is “0”.

  Further, the utterance desire level calculation unit 23 adds “5” to the value of the counter when the motion detection unit 22 detects the “raising hand” motion (S13: YES) (S14).

  Further, the utterance desire level calculation unit 23 adds “3” to the value of the counter when the conflict detection unit 26 detects “conflict” (S15: YES) (S16).

  And the utterance desire level calculation part 23 calculates an utterance desire level based on the value of the counter added by S11 to S16. For example, when the utterance desire level is divided into six levels from level 0 to level 5, the value of the counter is converted into any level and stored in the memory 24 (S17).

Then, when the converted level value is equal to or higher than a predetermined value (for example, “5”) (S18: YES), the utterance desire level calculation unit 23 determines display of the predetermined mark and stores the mark display information in the memory 24. (S19). In addition, a figure, a symbol, a character, etc. can be used for a predetermined mark. For example, a raised hand mark reminiscent of a raised hand may be used as the predetermined mark.
As described with reference to FIG. 2, the memory 24 stores the utterance desire level of the self-participant and mark display information when the utterance desire level is “5” or more. The self-participant's utterance desire level and mark display information are transmitted to all other conference apparatuses 1 by the communication unit 29 via the network 9. Further, the utterance desire level and mark display information of other participants transmitted from all other conference apparatuses 1 via the network 9 are received by the communication unit 29 and stored in the memory 24. Thus, the speech desire level and mark display information of each participant are shared by all the conference apparatuses 1.

  The video generation unit 27 and the video editing unit 28 are stored in the memory 24, the video data of the self-participant captured by the video camera 11, the video data of other participants received from the conference server 8 via the network 9, and the memory 24. A conference video is generated and edited using the utterance desire level and mark display information of each participant.

  FIG. 3 is a diagram illustrating an example of a conference video. As shown in the figure, the video generation unit 27 arranges the video data of each participant at a predetermined position in a tile shape. In the example shown in the figure, four participants are participating in the conference.

  Then, the video editing unit 28 edits the conference video generated by the video generation unit 27 with reference to the memory 24 in which each participant's speech desire level and mark display information are stored. Specifically, as shown in the figure, information indicating the utterance desire level (in the example shown, an indicator) is set in the vicinity of the video data of the corresponding participant according to the utterance desire level of each participant. For example, in the case of a participant whose utterance desire level is “2”, the indicator is lit according to the utterance desire level, for example, two indicators are lit from the bottom.

  For a participant in which mark display information is stored, a predetermined mark (in the illustrated example, a hand raising mark) is set in the vicinity of the participant's video data. The display device 13 displays the conference video edited by the video editing unit 28.

  Further, when there is a speaker (when there is a participant whose speaker flag is “1”), the video editing unit 28 surrounds the video data of the speaker with a frame of a predetermined color, for example. It is good also as displaying so that it may stand out.

  In the present embodiment described above, it is possible to easily (naturally) recognize each participant's utterance desire by displaying information indicating the utterance desire level of each participant in a remote conference via a network. . Therefore, the participant can proceed with a smooth conference while drawing out each participant's utterance desires without operating an input device such as a mouse or a keyboard in order to show the utterance intention to other participants. For example, it is possible to prevent utterance collisions and realize a smooth speaker change.

  In this embodiment, one of the actions detected by the action detection unit 22 is “raising hands”. Therefore, when the participant wants to explicitly indicate his / her intention to speak, To raise the level of utterance desire.

  In the present embodiment, a predetermined mark (for example, a raised hand mark) is displayed in the vicinity of the video data of a participant whose utterance desire level is a predetermined value or more, and a participant with a high utterance desire can be easily recognized at a glance. Can be recognized.

  In addition, this invention is not limited to the said embodiment, Many deformation | transformation are possible within the range of the summary. For example, in the present embodiment, the utterance desire level is calculated based on the motion detected by the motion detector 22 and the conflict detected by the conflict detector 26. However, the utterance desire level may be calculated based only on the motion detected by the motion detection unit 22. Further, the utterance desire level may be calculated based only on the conflict detected by the conflict detection unit 26.

1: Conference device 11: Video camera 12: Microphone 13: Display device 14: Speaker 21: Video acquisition unit 22: Motion detection unit 23: Speech request level calculation unit 24: Memory 25: Audio acquisition unit 26: Affinity detection unit 27: Video generation unit 28: Video editing unit 29: Communication unit 8: Conference server 9: Network

Claims (7)

A conference device for performing a remote conference via a network,
Video acquisition means for acquiring video data of self-participants who use the conference device;
Action detecting means for detecting a predetermined action assumed to have utterance desire from the video data of the participant;
Utterance desire level calculating means for calculating an utterance desire level based on the action detected by the action detecting means;
Receiving means for receiving video data and utterance desire levels of other participants from other conference devices via a network;
Video generation means for generating a conference video in which video data of self-participants and other participants are arranged;
Video editing means for setting information indicating the speech desire level of the self-participant and other participants in the conference video;
And a display unit for displaying the conference video edited by the video editing unit. The conference device according to claim 1,
The video editing means sets a predetermined mark in the vicinity of video data of a participant whose utterance desire level is a predetermined value or more. The conference device according to claim 1 or 2,
Voice acquisition means for acquiring voice data of the self-participant who uses the conference device;
A conflict detection means for detecting conflict from the audio data of the participant;
The conference apparatus characterized in that the utterance desire level calculation means calculates an utterance desire level based on the action detected by the action detection means and the interaction detected by the interaction detection means. A conference method for conducting a remote conference via a network,
The conference equipment
A video acquisition step of acquiring video data of the self-participant who uses the conference device;
An operation detecting step for detecting a predetermined operation assumed to have an utterance desire from the video data of the participant;
An utterance desire level calculating step for calculating an utterance desire level based on the motion detected in the motion detection step;
A receiving step of receiving video data and utterance desire levels of other participants from other conference devices via the network;
A video generation step for generating a conference video in which video data of the self-participant and other participants are arranged;
A video editing step for setting information indicating the utterance desire level of the self-participant and other participants in the conference video;
And a display step of displaying the conference video edited in the video editing step. The conference method according to claim 4,
In the video editing step, a predetermined mark is set in the vicinity of video data of a participant whose utterance desire level is equal to or higher than a predetermined value. The conference method according to claim 4 or 5, wherein:
An audio acquisition step of acquiring audio data of the self-participant using the conference device;
A conflict detection step of detecting a conflict from the audio data of the participant; and
The utterance desire level calculating step calculates an utterance desire level based on the motion detected in the motion detection step and the conflict detected in the conflict detection step.   A conference program for causing a computer to execute the conference method according to claim 4.

Images