JP2019215502A - Server, sound data evaluation method, program, and communication system - Google Patents

Abstract

To provide a server that allows one teacher to teach speaking to a plurality of students in parallel.SOLUTION: A server 50 can communicate with a plurality of communication terminals 10, 70A to 70C via a network, and comprises: communication means that transmits first sound data respectively to the plurality of second communication terminals 70A to 70C according to an instruction from the first communication terminal 10, and receives a plurality of pieces of second sound data respectively uttered by second users of the plurality of second communication terminals; and generation means that, on the basis of the first sound data and the second sound data acquired respectively from the plurality of second communication terminals, generates evaluations for the respective plurality of pieces of second sound data. The communication means transmits the evaluations for the respective plurality of pieces of second sound data obtained from the generation means to the first communication terminal of the first user different from the second users.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a server, a sound data evaluation method, a program, and a communication system.

  There are many needs for students who want to learn correct pronunciation and utterance by actually speaking out and receiving feedback from teachers, such as foreign language classes such as English conversation and announcement classes. At the time of learning, the teacher and the students gather in the same place such as a classroom, and the teacher gives guidance to the sounds emitted by the students with examples.

  In recent years, a form in which teachers and students at remote locations learn pronunciation and utterance using the Internet has been put to practical use. For example, in a foreign language classroom, a sound education system in which a one-to-one instruction is provided using a communication system used for a video conference or the like is already known.

  There is also known a sound education system that assumes that a plurality of students receive guidance simultaneously for one teacher (for example, see Patent Document 1). Patent Literature 1 discloses a system in which a student can answer a set problem via a terminal, and the teacher confirms the question and instructs a plurality of students.

  However, the conventional technique has a problem that it is difficult for one teacher to instruct a plurality of students in parallel. If the teacher and the student are one-to-one, the teacher can check the utterance of the student and provide detailed guidance. However, when there are a plurality of students, it takes time for the teacher to give advice to the students one by one. On the other hand, if a plurality of students utter simultaneously, the time can be reduced, but since the evaluation target is sound, it is difficult for the teacher to simultaneously evaluate the utterances of the plurality of students, and it is difficult to provide detailed guidance.

  The present invention has been made in view of the above problems, and has as its object to provide a server that allows one teacher to instruct a plurality of students in parallel.

  In view of the above-mentioned problem, the present invention is a server capable of communicating with a plurality of communication terminals via a network, wherein the first sound data is transmitted to a plurality of first terminals by an instruction of the first communication terminal operated by a first user. Communication means for transmitting to each of the plurality of second communication terminals and receiving a plurality of second sound data respectively emitted by a second user of the plurality of second communication terminals; and Generating means for generating an evaluation for each of the plurality of second sound data based on the second sound data obtained from each of the plurality of second communication terminals, wherein the communication means The evaluation obtained by the generating means is transmitted to the first communication terminal.

  A server can be provided in which one teacher can instruct a plurality of students to speak in parallel.

It is an example of a figure explaining the outline of processing which a communication system evaluates a student's voice. It is an example of the schematic diagram of a communication system. FIG. 3 is a hardware configuration diagram of an example of a terminal. FIG. 2 is a hardware configuration diagram of an example of a management system. FIG. 3 is a software configuration diagram of an example of a terminal. FIG. 2 is an example of a functional block diagram of a terminal constituting a part of the communication system and a management system. FIG. 4 is a diagram illustrating an example of an authentication management table, a terminal management table, and a group management table. It is a figure showing an example of a sound recording management table and a result management table. FIG. 5 is an example of a sequence diagram illustrating a process in which a terminal logs in to a management system. It is an example of the sequence diagram which shows the procedure by which the management system evaluates the voice data of a student (recorded file is output). It is a figure showing an example of a teacher screen T1. It is a figure showing an example of teacher screen T2a and T2b. It is a figure showing an example of a teacher screen T3. It is a figure showing an example of student screen S1. It is a figure showing an example of student screen S2. It is a figure showing an example of student screen S3. It is an example of the sequence diagram which shows the procedure by which the management system evaluates the voice data of a student (teacher utters). It is an example of a figure explaining voice recognition typically. FIG. 3 is another example of a schematic diagram of a communication system. FIG. 3 is another example of a schematic diagram of a communication system. FIG. 5 is an example of a sequence diagram illustrating a process in which a terminal logs in to a management system. FIG. 3 is a diagram illustrating a configuration example of a Web application in a terminal.

  Hereinafter, a communication system and a sound data evaluation method performed by the communication system will be described with reference to the drawings as an example of an embodiment for implementing the present invention.

<Outline of processing>
FIG. 1 is an example of a diagram illustrating an outline of a process in which a communication system 1 of the present embodiment evaluates a student's voice. In FIG. 1, a terminal 10 used by a teacher and terminals 70A to 70C used by students (all correspond to communication terminals described later) are connected to a management system 50 via a communication network so as to be able to communicate.
(1) First, the teacher utters an example voice (or a pre-registered example voice may be used). The model voice is transmitted to the terminals 70A to 70C via the management system 50 and can be listened to by the student.
(2) Next, the student uses the teacher's voice as a model and utters a voice almost at the same time (at least in some time zones). The student's voice is transmitted to the management system 50.
(3) The management system 50 compares the model voice with the student voice to evaluate how close the student voice is to the model.
(4) The management system 50 transmits the evaluation results of all the students to the terminal 10 of the teacher. Since the terminal 10 displays the evaluation results of all the students, the teacher can check the evaluation results of each student and perform detailed guidance to improve the utterance of each student.

  As described above, in the communication system 1 according to the present embodiment, even in a mode in which a remote teacher instructs a plurality of students to utter, the management system 50 compares the teacher's voice with each student's voice. Can teach a lot of students' voices in parallel.

<Terminology>
A meeting originally refers to meeting and discussing or gathering and discussing, but in the present embodiment, it is not necessary to gather at the same place because the meeting is discussed via the Internet. Meetings for sound evaluation and guidance are also forms of meetings.

  The sound data may be any data as long as air vibration is converted into an electric signal. In the present embodiment, a description will be given using voice data uttered by a person as an example. Also, the communication system 1 that teaches English utterance as a predetermined language will be described, but the present invention can also be applied to a predetermined language other than English (Chinese, French, Spanish, Russian, German, Arabic, etc.). The sound data is not limited to sound, but may be sound emitted from a musical instrument in a music classroom or the like.

  The sound refers to a state in which the air vibrates, and the sound data is information or data. In the present embodiment, the two are not strictly distinguished.

  Instructing in parallel means that even if multiple students participate in one videoconference and the time of sound production overlaps, teaching equivalent to one-on-one lessons is possible. . Teaching in parallel may be referred to as "teaching at the same time."

<System configuration example>
FIG. 2 is a schematic diagram of the communication system 1 according to the present embodiment. As shown in FIG. 2, the communication system 1 manages a plurality of communication terminals 10a, 10b, 10c, an example of which is a video conference terminal, a plurality of communication terminals 70x, 70y, an example of which is a smartphone, and manages each communication terminal. The communication management system 50 and the relay device 30 are configured. Hereinafter, the “communication terminal” is referred to as a “terminal”, and the “communication management system” is referred to as a “management system”.

  FIG. 2 illustrates three terminals 10a, 10b, and 10c and two terminals 70x and 70y, but the number is not limited thereto. Further, an arbitrary terminal among the plurality of terminals 70x and 70y is represented as “terminal 70”, and an arbitrary terminal among the plurality of terminals 10a, 10b, and 10c is represented as “terminal 10”. The terminals 10 and 70 may be a general-purpose computer having a communication function, an electronic whiteboard, a car navigation terminal, an electronic signboard (digital signage), or the like.

  The management system 50 is an information processing device having a server function, and is a so-called server (corresponding to a server in the claims). Each of the terminals 10 and 70 is a computer having a corresponding client function. The terminals 10 and 70, the relay device 30, and the management system 50 can communicate with each other via the communication network 2 such as the Internet, a mobile phone network, a LAN (Local Area Network), WiFi (Wireless Fidelity), or Bluetooth (registered trademark). . The communication network 2 also includes a base station 2a at the end of the mobile phone network. Although FIG. 2 shows one base station 2a, the number is not limited to this.

  The relay device 30 relays content data such as audio data (including audio data), video (image) data, and text data between the terminals 10 and 70.

  The terminal 10 is arranged in a living room (home, classroom, or the like) where a teacher is located as an example of a base, and the terminal 70 is arranged in each student's home or the like as an example of a base. Teachers can be located anywhere, and students can be located anywhere. In addition, the teacher may use the terminal 70, or the student may use the terminal 10. In the present embodiment, for convenience of explanation, a description will be given assuming that a teacher uses the terminal 10 and a student uses the terminal 70.

  The management system 50, the relay device 30, and the terminals 10, 70 may be arranged in the same country or region, or may be arranged in different countries or regions. The user of the terminal 10 is, for example, a teacher, and the user of the terminal 70 is, for example, a student.

  The communication system 1 includes a data providing system for transmitting content data in one direction from one terminal to another terminal via a management system 50, and information and emotions between a plurality of terminals via a management system. Communication systems to communicate are included. This communication system is a system for mutually transmitting information, emotions, and the like between a plurality of terminals via a management system, and examples thereof include a video conference system and a video telephone system.

<Example of hardware configuration>
<< Example of hardware configuration of terminal >>
FIG. 3A is a hardware configuration diagram of the terminal 10 according to the embodiment. The terminal 10 is used as a CPU 101 (Central Processing Unit) for controlling the entire operation of the terminal 10, a ROM 102 (Read Only Memory) storing a program used for driving the CPU 101, such as an IPL (Initial Program Loader), and a work area for the CPU 101. A random access memory (RAM) 103, a flash memory 104 for storing various data such as a program for the terminal 10, image data, and sound data. Also, an SSD 105 (Solid State Drive) that controls reading or writing of various data from or to the flash memory 104 under the control of the CPU 101, and reading or writing (storage) of data from or to a recording medium 106 such as a flash memory or an IC card (Integrated Circuit Card). ) For controlling the media I / F 107 (Interface). Further, an operation button 108 operated when selecting a destination of the terminal 10, a power switch 109 for turning on / off the power of the terminal 10, and a network I for transmitting data using the communication network 2. / F111.

  Further, the terminal 10 includes a built-in camera 112 that obtains image data by capturing an image of a subject under the control of the CPU 101, an image sensor I / F 113 that controls driving of the camera 112, a built-in microphone 114 that inputs sound, It has a built-in speaker 115 for outputting sound, and a sound input / output I / F 116 for processing input / output of a sound signal between the microphone 114 and the speaker 115 under the control of the CPU 101. Also, a display I / F 117 for transmitting image data to an external display 120 under the control of the CPU 101, an external device connection I / F 118 for connecting various external devices, and an alarm lamp for notifying abnormalities of various functions of the terminal 10. 119, and bus lines 110 such as an address bus and a data bus for electrically connecting the above-mentioned components as shown in FIG. 3A.

  The display 120 is a display device that displays an image of a subject and the like. Examples of the display 120 include a liquid crystal and an organic EL (Organic Electroluminescence). The display 120 is connected to the display I / F 117 by a cable 120c. The cable 120c may be a cable for an analog RGB (VGA) signal, a cable for a component video, an HDMI (registered trademark) (High-Definition Multimedia Interface) or a DVI (Digital Video). (Interactive) signal cable.

  The camera 112 includes a lens and a solid-state imaging device that converts light into electric charge to convert an image (video) of a subject into an electron. As the solid-state imaging device, a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) is used. Are used.

  External devices such as an external camera, an external microphone, and an external speaker are electrically connected to the external device connection I / F 118 by a USB (Universal Serial Bus) cable inserted into the connection port 1132 of the housing 1100. Can be connected to When an external camera is connected, the external camera is driven under the control of the CPU 101. Similarly, when an external microphone is connected or an external speaker is connected, the external microphone and the external speaker are driven under the control of the CPU 101.

  Note that the recording medium 106 is detachable from the terminal 10. In addition, as long as it is a non-volatile memory that reads or writes data under the control of the CPU 101, an EEPROM (Electrically Erasable and Programmable ROM) may be used instead of the flash memory 104.

  FIG. 3B is a hardware configuration diagram of the terminal 70. As shown in FIG. 3B, the terminal 70 includes a CPU 701 for controlling the operation of the entire terminal 70, a ROM (Read Only Memory) 702 storing a program, a RAM 703 used as a work area of the CPU 701, and a CPU 701. Has an EEPROM (Electrically Erasable and Programmable ROM) 704 for reading or writing data in accordance with the above control. A media I / F 707 that controls reading or writing (storage) of data from or to a recording medium 706 such as a flash memory, and a CMOS (Complementary Metal Oxide Semiconductor) sensor 712 that captures an image of a subject and obtains image data under the control of the CPU 701. Have.

  Note that the EEPROM 704 stores an operating system (OS) executed by the CPU 701, other programs, and various data. In addition, the CMOS sensor 712 is a charge-coupled device that converts light into electric charge to digitize an image of a subject. If a subject can be imaged, the CMOS sensor can be replaced with a CCD (Charge Coupled Device) sensor.

  Further, the terminal 70 communicates with the nearest base station 2a by a wireless communication signal using the microphone 714 for converting sound to a sound signal, the speaker 715 for converting sound signal to sound, the antenna 711a, and the antenna 711a. A communication unit 711, a display 720 such as a liquid crystal display or an organic EL display that displays an image of a subject and various icons, and is mounted on the display 720 and is configured by a pressure-sensitive or electrostatic type panel. A touch panel 721 for detecting a touch position on the display 720 and a bus line 710 such as an address bus and a data bus for electrically connecting the above-described units are provided.

<< Hardware configuration example of management system >>
FIG. 4 is a hardware configuration diagram of the management system 50 according to the embodiment. The management system 50 includes a CPU 201 that controls the operation of the entire management system 50, a ROM 202 that stores a program such as an IPL used for driving the CPU 201, a RAM 203 that is used as a work area of the CPU 201, and various programs such as a program for the management system 50. It has an HD 204 for storing data. Also, an HDD 205 (Hard Disk Drive) that controls reading or writing of various data from or to the HD 204 under the control of the CPU 201, a media I / F 207 that controls reading or writing (storage) of data from or to a recording medium 206 such as a flash memory, , A display 208 for displaying various information such as menus, windows, characters, and images. Further, a network I / F 209 for performing data communication using the communication network 2, a keyboard 211 having a plurality of keys for inputting characters, numerical values, various instructions, etc., selection and execution of various instructions, execution of processing targets, It has a mouse 212 for selecting, moving a cursor, and the like, and a CD-ROM drive 214 for controlling reading or writing of various data to a CD-ROM 213 (Compact Disc Read Only Memory) as an example of a removable recording medium. Further, as shown in FIG. 4, a bus line 210 such as an address bus or a data bus for electrically connecting the above-mentioned components is provided.

  Since the relay device 30 has the same hardware configuration as the management system 50, the description thereof is omitted.

<Example of software configuration>
FIG. 5A is a software configuration diagram of the terminal 10 according to the embodiment. “Communication application A1” is installed in the terminal 10 as a client application. An application means application software. As shown in FIG. 5A, the OS 1020 and the communication application A1 operate on the work area 1010 of the RAM 103 of the terminal 10. Among these, the OS 1020 is basic software that provides basic functions and manages the entire terminal 10. The communication application A1 is an application that operates on the terminal 10 used by the teacher and communicates with another terminal 70.

  FIG. 5B is a software configuration diagram of the terminal 70 according to the embodiment. “Communication application A7” is installed in the terminal 70 as a client application. As shown in FIG. 5B, the OS 7020 and the communication application A7 operate on the work area 7010 of the RAM 703. Among them, the OS 7020 is basic software that provides basic functions and manages the entire terminal 70. The communication application A7 operates on the terminal 10 used by the student and is an application for communicating with the terminal 10.

  The communication protocols of the communication applications A1 and A7 include (1) SIP (Session Initiation Protocol), (2) H.323, (3) a protocol extended from SIP, (4) an instant messenger protocol, (5) Protocols using the MESSAGE method of SIP, (6) Internet Relay Chat protocol (IRC), (7) a protocol extended from the instant messenger protocol, and the like. Among them, (4) the instant messenger protocol is, for example, (4-1) XMPP (Extensible Messaging and Presence Protocol), or (4-2) ICQ (registered trademark), AIM (registered trademark), or Skype (registered trademark). Trademark). Also, (7) a protocol that is an extension of the instant messenger protocol is, for example, Jingle.

<About communication system functions>
Next, a functional configuration of the communication system 1 of the present embodiment will be described with reference to FIG. FIG. 6 is a functional block diagram of the terminals 10 and 70 and the management system 50 that constitute a part of the communication system 1. In FIG. 6, the terminals 10 and 70 and the management system 50 are connected so that data communication can be performed via the communication network 2.

<Functional configuration of terminal>
The terminals 10 and 70 include transmission / reception units 11 and 71, operation input reception units 12 and 72, activation units 13 and 73, output control units 14 and 74, and storage / readout units 19 and 79. These units are configured such that any one of the components shown in FIG. 3 is executed by the CPU 101 or 701 in accordance with the communication applications A1 and A7 (programs) developed on the RAM 103 and 703 from the flash memory 104 or the EEPROM 704. It is a function realized by operating with.

  In addition, the terminal 10 includes storage units 1000 and 7000 configured by the ROMs 102 and 702, the RAMs 103 and 703, the flash memory 104, or the EEPROM 704 illustrated in FIG.

  Each functional configuration in the terminal 10 will be described in detail. The transmission / reception units 11 and 71 are realized by commands from the CPUs 101 and 701 and the network I / F 111 or the communication unit 711, and communicate with the terminal, each device or system of the communication partner and various data (or information) via the communication network 2. ).

  The operation input receiving units 12 and 72 are realized by commands from the CPUs 101 and 701, the operation buttons 108 or the touch panel 721, and the power switch 109, and receive various inputs or various selections by the user.

  The activation units 13 and 73 are implemented by instructions from the CPUs 101 and 701, and activate the operations of the communication applications A1 and A7.

  The output control units 14 and 74 are realized by instructions from the CPUs 101 and 701, the display I / F 117 and the sound input / output I / F 116, and control output of image data and sound data.

  The storage / readout units 19 and 79 are realized by instructions from the CPUs 101 and 701, and perform processing of storing various data in the storage units 1000 and 7000 and reading various data stored in the storage units 1000 and 7000. .

<< Functional configuration of management system >>
The management system 50 includes a transmission / reception unit 51, an authentication unit 52, a management unit 53, a recording management unit 54, an analysis / result management unit 55, a session control unit 58, and a storage / readout unit 59. These units are functions realized by operating any of the components shown in FIG. 4 by a command from the CPU 201 in accordance with a program for the management system 50 developed on the RAM 203 from the HD 204. is there. The management system 50 has a storage unit 5000 constructed by the HD 204. Further, in the storage unit 5000, each DB is constructed by the following tables.

(Authentication management table)
FIG. 7A is a conceptual diagram illustrating an authentication management table. In the storage unit 5000, an authentication management DB 5001 is constructed by an authentication management table as shown in FIG. In this authentication management table, a password for authentication is managed in association with each communication ID of all terminals 10 and 70 managed by the management system 50.

(Terminal management table)
FIG. 7B is a conceptual diagram illustrating a terminal management table. In the storage unit 5000, a terminal management DB 5002 is constructed by a terminal management table as shown in FIG. In this terminal management table, the IP address and TYPE of each terminal 10, 70 are managed in association with each communication ID of each terminal 10, 70. The communication ID of each terminal 10, 70 is grasped by the login of each terminal 10, 70. The TYPE will be described later, but the TYPE may be registered at the time of login or the like, or may be registered in advance.

(Group management table)
FIG. 7C is a conceptual diagram showing a group management table. In the storage unit 5000, a group management DB 5005 is constructed by a group management table as shown in FIG. In this group management table, the communication IDs of the terminals 10 and 70 are managed in association with the group IDs. A group is a group of one teacher and a plurality of students led by the teacher. The group ID is identification information for identifying a group. One group participates in the same conference (an example of communication), and content data is mutually transmitted and received between the terminals 10 and 70 of the group.

  As a method of creating the group management table, for example, there is a method in which a teacher accesses the management system 50 and registers it before the meeting starts. Alternatively, as a method that does not require the creation of a group management table in advance, the management system 50 distributes the same invitation code by e-mail or the like to students participating in the same meeting, The students 50 may classify the students having the same invitation code into the same group (the group management table is created afterwards). Alternatively, the students may be classified in advance by language ability (the levels may be registered in the authentication management table), and the management system 50 may classify the students of the same level into the same group in the order of login. For example, if the number of students in one group is three, a group ID is assigned every time three students of the same level log in. In this case, a teacher is assigned to each group, and the conference starts.

(Recording management table)
FIG. 8A is a conceptual diagram showing a recording management table. In the storage unit 5000, a recording management DB 5003 is constructed by a recording management table as shown in FIG. The recording management table is a table in which information on recorded audio data is registered.

  Each item of the recording ID, the TYPE, the communication ID, and the recording file is associated with the recording management table. The recording ID is identification information for uniquely specifying or identifying a recording file of voice data of a teacher or a student, and is assigned by the management system 50 every time recording is performed. In FIG. 8A, a recording ID is a combination of a communication ID, a conference ID, a problem number (described later), and a retry number (described later).

  TYPE is an attribute of a recording file indicating whether the recording file is a voice uttered by a student (STUDENT), a voice uttered by a teacher (TEACHER), or a voice recorded in advance (preset). is there. For example, the determination is made by the communication applications A1 and A7 transmitting TYPE to the management system 50. The communication ID is the same as above, but the communication ID is NULL (meaning that there is no value) in the case of a voice that has been recorded in advance. The recording file name is the file name of the actually recorded audio data. The file name is automatically given by the management system 50. In FIG. 8A, the recording ID is used as the file name. One line of the recording management table is called a record.

(Result management table)
FIG. 8B is a conceptual diagram illustrating a result management table. In the storage unit 5000, a result management DB 5004 is constructed by a result management table as shown in FIG. The result management table is a table summarizing how the student's voice was evaluated compared to the model voice.

  In the result management table, items of sample recording ID, answer recording ID, conference ID, question number, retry number, sample recognition result, answer recognition result, and scoring result are associated. As the sample recording ID and the answer recording ID, the ID registered in the recording management DB 5003 is diverted. The type of the sample recording ID is either PRESET or TEACHER, and the type of the answer recording ID is STUDENT. The conference ID is identification information for uniquely determining which conference is a unit until the conference ends. The conference ID is assigned by the management system 50 at the start of the conference. The question number is identification information for identifying a question given by a teacher in one meeting. The question number is assigned by the management system 50 every time the question is asked. The retry number is a branch number for a question when the same question is re-issued. The model recognition result is a recognition result of the model voice data. In FIG. 8B, as an example, the result of recognition is converted to phonetic symbols. The answer recognition result is a recognition result of the voice data of the student. In FIG. 8B, as an example, the result of recognition is converted to phonetic symbols. In the scoring result, the comparison result between the model recognition result and the answer recognition result is indicated by “O” or “X” for each word. One row of the result management table is called a record.

<< Functional Configuration of Management System >>
Next, each functional configuration of the management system 50 will be described in detail. The transmission / reception unit 51 is realized by a command from the CPU 201 and the network I / F 209, and transmits / receives various data (or information) to / from each terminal, device, or system via the communication network 2.

  The authentication unit 52 is realized by a command from the CPU 201, searches the authentication management table using the communication ID and password received by the transmission / reception unit 51 as a search key, and the same communication ID and password are managed in the authentication management table. Authentication of the terminal by determining whether the terminal

  The management unit 53 is realized by an instruction from the CPU 201, and manages the terminals 10, 70 by registering the logged-in terminals 10, 70 in the terminal management table.

  The recording management unit 54 is realized by a command from the CPU 201, and starts recording upon an operation of a teacher, and ends recording upon an operation of a teacher or a student. Voice data from the start to the end is recorded, and each item is registered in the recording management DB 5003.

  The analysis / result management unit 55 is realized by an instruction from the CPU 201, compares and analyzes the recorded teacher and student voice data, and records the result in the result management DB 5004.

  The session control unit 58 controls a session for transmitting content data between the terminals 10 and 70 according to a command from the CPU 201. This control includes control for establishing a session, control for joining the terminals 10 and 70 to the established session, control for exiting from the session, and the like.

  The storage / readout unit 59 is realized by a command from the CPU 201 and the HDD 205, or is realized by a command from the CPU 201, and stores various data in the storage unit 5000 or extracts various data stored in the storage unit 5000. Is performed.

<Processing and operation of communication system>
Subsequently, processing and operation in the communication system 1 will be described.

<< Login processing >>
First, a process in which the terminals 10 and 70 log in to the management system 50 will be described with reference to FIG. FIG. 9 is an example of a sequence diagram illustrating a process in which the terminals 10 and 70 log in to the management system 50.

  When the user of the terminal 70 turns on the power switch, the operation input receiving unit 72 receives the power ON and activates the terminal 70 (step S1). When the terminal 70 is activated, the activation unit 73 activates the communication application A7 installed in the terminal 70 (Step S2). Hereinafter, the process in the terminal 70 is executed by a command of the communication application A7.

  The transmission / reception unit 71 of the terminal 70 transmits a login request to the management system 50 via the communication network 2 (Step S3). The login request includes a communication ID and a password for identifying the terminal that is the login request source.

  The transmission / reception unit 51 of the management system 50 receives the login request. By transmitting the login request from the terminal 70 to the management system 50, the management system 50 on the receiving side can acquire the IP address of the terminal 70 on the transmitting side.

  Next, the authentication unit 52 of the management system 50 searches the authentication management table of the storage unit 5000 using the communication ID and password included in the login request as a search key, and stores the same communication ID and password in the authentication management table. Authentication is performed by judging whether or not is managed (step S4).

  When the authentication unit 52 authenticates that the login request is from the terminal 70 having the right to use, the management unit 53 stores the communication ID and the IP address of the login request source terminal 70 in the terminal management table. Are stored in association with each other (step S5). As a result, information for accessing the logged-in terminal 70 is managed in the terminal management table. Further, the management unit 53 receives the information indicating the communication application A7 and registers the TYPE in the terminal management table.

  The transmission / reception unit 51 of the management system 50 transmits the authentication result information indicating the authentication result obtained by the authentication unit 52 to the terminal 70 as the login request source via the communication network 2 (step S6). Thereby, the transmission / reception unit 71 of the terminal 70 receives the authentication result information. Hereinafter, a case will be described in which the terminal 70 (the plurality of terminals) of the login request source operated by the student has been successfully authenticated and the terminal 70 has logged in to the management system 50.

  On the other hand, when the teacher of the terminal 10 on the teacher side turns on the power switch 109, the operation input receiving unit 12 receives the power ON and starts the terminal 10 (step S11). When the terminal 10 is activated, the activation unit 13 activates the communication application A1 installed on the terminal 10 (Step S12). Hereinafter, the process in the terminal 10 is executed by a command of the communication application A1.

  The terminal 10 transmits a login request to the management system 50, and logs in to the management system 50 (steps S13-2, S14, S15, S16). This processing is the same as the processing in steps S3, S4, S5, and S6 between the terminal 70 and the management system 50, and thus the description is omitted.

  As a method for the management system 50 to determine whether the user operating the terminals 10 and 70 is a teacher or a student, there is a method of identifying the communication applications A1 and A7 as described in step S5. In this method, for example, the terminals 10 and 70 may transmit information for distinguishing the communication applications A1 and A7 to the management system 50 at the time of login. Alternatively, there is a method of distributing the communication ID in advance. For example, a communication ID distributed to a teacher starts with T as “T001”, and a communication ID distributed to students starts with S as “S001”. With these methods, the management system 50 can easily determine whether it is a teacher or a student. Alternatively, when the above invitation code is used, it is possible to determine whether a student or a teacher by distributing a specific invitation code to a student.

  When the terminals 10 and 70 log in, the session control unit 58 determines, for example, the relay device 30 with a small load, and causes the relay device 30 to start relaying the content data transmitted and received by the terminals 10 and 70. The communication ID of the terminals 10 and 70 (the information may be any information that can identify the terminals 10 and 70) and the group ID are transmitted to the relay device 30 in association with each other. As a result, the content data transmitted by the terminals 10 and 70 together with the communication ID (the information may be any information that can identify the terminals 10 and 70) is transferred to the terminals 10 and 70 in the same group.

<< Evaluation of audio data >>
Next, a process in which the management system 50 evaluates student voice data will be described with reference to FIG. Here, I want to imagine a state where an online classroom for English conversation has been opened. In the following English conversation online classroom, it is assumed that the teacher gives questions as appropriate while talking with the students.

  FIG. 10 is an example of a sequence diagram illustrating a procedure in which the management system 50 evaluates student voice data. The process of FIG. 10 is described from the state where the terminals 10 and 70 log in and the relay device 30 is determined. In addition, description will be given with reference to examples of screens displayed by the teacher terminal 10 and the student terminal 10 shown in FIGS. Although the relay device 30 is not shown in FIG. 10, the management system 50 is regarded as integral with the relay device 30 in FIG. 10. The management system 50 or the relay device 30 may actually evaluate the voice.

  M1: The recording management unit 54 of the management system 50 assigns a conference ID when the conference starts. This enables management of audio data.

  M2-1, M2-2: The session control unit 58 of the management system 50 transmits a conference start request to the terminals 10 and 70 because the conference is ready. Thereby, the video conference is started. That is, transmission / reception of content data is started.

  T1, S1: The communication application A1 of the terminal 10 and the communication application A7 of the terminal 70 perform conference start processing. Specifically, the communication applications A1 and A7 display a screen, start communication with the relay device 30, and acquire necessary information from the management system 50.

  S2: The output control unit 74 of the student terminal 70 displays a student screen S1 (an example of a third screen). FIG. 14 shows the student screen S1. While the student screen S1 is displayed, the student listens to the teacher's voice in principle, but can also speak arbitrarily.

  M3, M4: On the other hand, the transmitting / receiving unit 51 of the management system 50 transmits a list of recorded files recorded in advance to the terminal 10 on the teacher side. That is, the TYPE transmits the recording ID and the recording file name of the recording file of PRESET from the recording management DB 5003. The terminal 10 on the teacher side is registered in the terminal management DB. The list is composed of a structured document, and examples of the structured document include HTML, XML (Extensible Markup Language), and the like.

  T2: The transmitting / receiving unit 11 of the terminal 10 receives a list of recording files recorded in advance. Thereby, the output control unit 14 of the terminal 10 displays the teacher screen T1 (an example of the second screen). FIG. 11 shows the teacher screen T1. The teacher screen T1 is a screen on which the teacher arbitrarily speaks and selects or inputs question sound data as needed.

  T3: In FIG. 10, it is assumed that the teacher has selected a pre-recorded voice (recorded file name) on the teacher screen T1. The process when the teacher inputs a voice will be described with reference to FIG. The operation input receiving unit 12 of the terminal 10 receives the operation, and the output control unit 14 displays the teacher screen T2a. The teacher screen T2a is shown in FIG. The teacher screen T2a is a screen in a state where the teacher has selected audio data.

  T4: The transmitting / receiving unit 11 of the terminal 10 transmits the recording ID of the recording file name selected by the teacher to the management system 50. This recording ID becomes a transmission request of the recording file.

  M5: The transmission / reception unit 51 of the management system 50 receives the recording ID. The recording management unit 54 acquires audio data corresponding to the recording ID from the recording management DB 5003.

  M6: The transmission / reception unit 51 of the management system 50 transmits the acquired audio data to the student terminal 70. In FIG. 10, there is one student terminal 70, but it is assumed that there are a plurality of terminals 70.

  S3: The transmission / reception unit 71 of the student terminal 70 receives the audio data, and upon receiving the audio data, the output control unit 74 displays the student screen S2 (an example of a fourth screen). The student screen S2 may be displayed at any timing from the start of reception of audio data, the completion of reception, or from the start to the end of reception. However, since the student screen S1 is a screen displayed during output of the teacher's utterance, The transition from the student screen S1 to the student screen S2 may be triggered by the completion of reception (also referred to as output completion because the data is output in real time upon reception). Note that the management system 50 may explicitly transmit a signal for transitioning from the student screen S1 to the student screen S2 to the terminal 70. FIG. 15 shows the student screen S2. The student screen S2 is a screen in which the audio data transmitted from the management system 50 is output from the speaker, and the student speaks using the audio data as an example. As described above, the screen of the student terminal 70 automatically changes when the teacher gives a question, so that the student does not need to operate the communication application A7 to display the student screen S2. In other words, the teacher has the initiative to control the screen of the student terminal 70.

  S4: The transmitting / receiving unit 71 of the terminal 70 displaying the student screen S2 transmits the communication ID and the TYPE to the management system 50. This is for recording voice. Since TYPE is registered in the terminal management DB 5002, it need not be transmitted.

  M7: The transmission / reception unit 51 of the management system 50 receives the communication ID and TYPE, and the recording management unit 54 starts recording.

  S5: Since the terminal 70 that has transitioned to the student screen S2 has started transmitting voice data, the transmitting / receiving unit 71 transmits the voice data uttered by the student to the management system 50 as needed (in real time). The recording management unit 54 of the management system 50 continues recording.

  S6: When the utterance ends, the student presses the end button 306 on the student screen S2.

  S7: The operation input receiving unit 72 receives the operation of the student, and the transmitting / receiving unit 71 transmits the communication ID again. This is for notifying the management system 50 that the utterance has ended.

  M8: The transmission / reception unit 51 of the management system 50 receives the communication ID, and the recording management unit 54 ends the recording of the audio data. The recording management unit 54 assigns a recording ID as an identifier of the recording file. As described above, the communication ID, the conference ID, the question number, and the retry number are the recording ID. As another pattern, the recording may be executed on the terminal 70 side, and the recording file may be sent to the management system 50 after the recording is completed.

  M9: The recording management unit 54 registers the recording ID, TYPE, communication ID, and recording file in the recording management DB 5003.

  M10: Then, the analysis / result management unit 55 compares and evaluates the audio data output (output means to reproduce) with the audio data transmitted from the terminal 70. Details will be described later.

  M11: The analysis / result management unit 55 registers the recording ID of the sampled recording file as the model recording ID and the recording ID of the student's recording file as the answer recording ID in the result management DB 5004. Similarly, the conference ID, the question of this conference (problem number), the retry number indicating the retry of the same question, the model recognition result, and the answer recognition result are registered in the result management DB 5004. I do. The retry will be described in detail on the teacher screen T3 (an example of the first screen).

  M12-1, M12-2: The transmitting / receiving unit 51 of the management system 50 transmits the evaluation result to the terminal 10 on the teacher side and the terminal 70 on the student side, respectively. In addition, the recording ID of the teacher's voice data (or the voice data itself) is transmitted to each of the student terminals 70, and the recording ID of the voice data uttered by the student (or the voice data itself) is transmitted to the terminal that uttered the voice data. 70 individually. The recording ID of the teacher's voice data (or the voice data itself) and the recording IDs of all the voice data uttered by each student (or the voice data itself) are transmitted to the terminal 10.

  T5: The transmitting / receiving unit 11 of the terminal 10 receives the evaluation result, and the output control unit 14 displays the teacher screen T3. FIG. 13 shows the teacher screen T3. The teacher screen T3 is a screen where the teacher checks the evaluation result of each student.

  S8: The transmitting / receiving unit 71 of the terminal 70 receives the evaluation result, and upon receiving the evaluation result, the output control unit 74 transits from the student screen S2 to the student screen S3 (an example of a fifth screen). Note that the management system 50 may explicitly transmit a signal for transitioning from the student screen S2 to the student screen S3 to the terminal 70. FIG. 16 shows the student screen S3. The student screen S3 is a screen on which the student confirms his / her own evaluation result. At the time of transition to the student screen S3, the student does not need to operate the communication application A7 to display the student screen S3.

<Screen example>
An example of a screen displayed by the teacher terminal 10 and the student terminal 70 will be described with reference to FIGS. FIG. 11 shows an example of the teacher screen T1. The teacher screen T1 is a screen displayed by the terminal 70 at the beginning of the conference. The teacher screen T1 has a teacher image display field 301, a student image display field 302, a pre-recorded data list field 303, an upload button 304, a start button 305, and an end button 306.

  The screen information of the teacher screen T1 includes at least HTML, and may be created by WebAPP (Web Application) including CSS (Cascade Style Sheet) and JavaScript (registered trademark). The same applies to teacher screens T2 and T3 and student screens S1 to S3 described later.

  The teacher image display column 301 is a column in which image data captured by the terminal 10 used by the teacher is displayed. The student image display column 302 is a column in which image data captured by the terminal 70 used by the student is displayed. In FIG. 11, there are four student image display columns 302, but the number of student image display columns 302 increases or decreases depending on the number of students.

  The pre-recorded data list column 303 is a column in which a list of recorded file names transmitted from the management system 50 to the terminal 10 is displayed. In FIG. 11, five recording file names are displayed, but the number is an example. If not, a scroll button or the like is displayed. Each recording file name can receive a press (selection) by the user. The upload button 304 is a button for the terminal 10 to transmit the recording ID of the recording file name selected by the teacher to the management system 50. The start button 305 and the end button 306 are used when the terminal 10 inputs voice data uttered in real time by the teacher instead of a recorded file recorded in advance, and the terminal 10 transmits the voice data to the management system 50. That is, the voice uttered by the teacher from when the teacher presses the start button 305 to when the end button 306 is pressed is transmitted to the management system 50. Details will be described with reference to FIG.

  FIG. 12A shows an example of the teacher screen T2a. The teacher screen T2a transits from the teacher screen T1. The difference between the teacher screen T2a and the teacher screen T1 will be described. The configuration of the teacher screen is the same as that of the teacher screen T1, but in the teacher screen T2a, one recording file name in the pre-recorded data list column 303 is selected. As described above, the recording file name selected by the teacher is emphasized, and the transmission of the recording ID to the management system 50 by the upload button 304 becomes possible.

  In the teacher screen T2a, if the teacher selects a recording file, the teacher can make the sample audio data available to the student, so that the same audio data can always be heard. The recording file need not be recorded by the teacher (same person), and may be reused.

  FIG. 12B shows an example of the teacher screen T2b. The teacher screen T2b transits from the teacher screen T1. The difference between the teacher screen T1 and the teacher screen T1 will be described. The configuration of the teacher screen is the same as that of the teacher screen T1, except that the start button 305 is pressed on the teacher screen T2b. Therefore, the teacher is producing a voice, and the voice data is transmitted to the management system 50 in real time.

  In the teacher screen T2b, the teacher can make the students listen to the sample audio data in real time, so that the students can practice arbitrary sentences (English sentences or sentences). Also, as is clear from the configuration of the teacher screens T1, T2a, and T2b, the teacher can teach while appropriately combining pre-recorded audio data and real-time audio data.

  FIG. 13 shows an example of the teacher screen T3. The teacher screen T3 has a correct answer voice play button 311, a retry button 312, an end button 313, a student image display field 314, a phonetic symbol field 316a, 316b, a detailed evaluation field 317, a score field 318, and an answer voice play button 315. . The correct sound reproduction button 311 is a button for outputting a model recording file on the terminals 10 and 70. The recording ID of the model recording file is transmitted to the management system 50. In this case, the retry number does not increase simply because the audio data flows. The retry button 312 is a button for transmitting the recording ID of the same model recording file to the management system 50 again and causing the students to speak.

  When the same question is asked again, the teacher presses the retry button 312 of the terminal 10, and the recording management unit 54 and the analysis / result management unit 55 re-execute Step T4 and subsequent steps in FIG. repeat. First, the recording ID of the model recording file and the fact that the retry is to be performed are transmitted to the management system 50. The recording management unit 54 transmits from the terminal screen 70 the retry to the terminal screen 70, thereby transitioning from the student screen S3 to the student screen S1 (this is a signal for transitioning from the student screen S3 to the student screen S1). Next, the recording management unit 54 transmits the recording file of the recording ID to the terminal 70, and the terminal 70 transitions from the student screen S1 to the student screen S2 when the reception of the recording file is completed, for example. The management system 50 may explicitly transmit a signal for transitioning from the student screen S1 to the student screen S2 to the terminal 70. The recording management unit 54 records the voice data from the terminal 70 again, assigns a recording ID, and registers the recording ID in the recording management DB 5003. Next, the analysis / result management unit 55 compares the sounds. The analysis / result management unit 55 searches the sample recording ID in the result management DB 5004 by the recording ID in step T4, specifies the record having the highest retry number from the hit records, and assigns the retry number of this record to one. Enlarge. The sample recording ID (recording ID transmitted in T4), the answer recording ID (recording ID of the voice data of the newly numbered terminal 70), the conference ID, and the problem number (the problem of the record hit by the recording ID) The retry number, the model analysis result, the answer recognition result, and the scoring result, which are increased by one, are registered in the result management DB 5004. The transmission / reception unit 51 transmits the evaluation result to the terminals 10 and 70. Thus, the teacher screen T3 of FIG. 13 is displayed on the terminal 10 again. The student screen S3 of FIG. 16 is displayed on the terminal 70 again.

  The end button 313 is a button for ending the question of the current model recording file. When the end button 313 is pressed, the teacher terminal 10 returns to the teacher screen T1, and the student terminal 10 returns to the student screen S1. Therefore, a normal video conference state is set. Also in this case, the student does not need to perform an operation for returning to the student screen S1.

  The student image display field 314 is the same as in FIG. The phonetic symbol column 316a shows phonetic symbols converted from teacher voice data, and the phonetic symbol column 316b shows phonetic symbols converted from student voice data. The detailed evaluation column 317 is a result of the evaluation by the management system 50 comparing the sample audio data and the student audio data. In FIG. 13, each word of the audio data is converted into a phonetic symbol, and "O" and "X" are attached to each phonetic symbol. “○” indicates that the pronunciation symbols of the sample voice data and the student's voice data match, and “x” indicates that the pronunciation symbols of the sample voice data and the student's voice data do not match. Further, points are given to the point column 318 according to the degree of coincidence. The analysis / result management unit 55 treats one question as one sentence, and sets a score of the proportion of words included in one sentence in which pronunciation symbols match. The scoring may be performed at the character level, and the scoring method is not limited to these.

  The answer voice reproduction button 315 is a button for outputting the voice data of the student. When the answer voice reproduction button 315 is pressed, the recording ID of the recording file is transmitted to the management system 50. The student's voice data may be transmitted by the management system 50 only to the teacher's terminal 10, may be transmitted only to the terminal 70 and the terminal 10 that uttered the voice data, or may be transmitted to all the terminals 10 and 70. May be. Since the communication ID is associated with the recording ID in the recording management DB 5003, the management system 50 can transmit only to the terminal 70 having this communication ID.

  As shown in FIG. 13, since the evaluation result of each student is displayed in a list on the teacher screen T3, even if each student utters at once, the voice data of each student can be individually evaluated. In addition, since the teacher can arbitrarily output the voice data of each student, the teacher can evaluate the voice data of the student.

  FIG. 14 shows an example of the student screen S1. The student screen S1 is a screen displayed by the terminal 10 at the beginning of the start of the conference. The student screen S1 has a teacher image display section 321 and a student image display section 322. The teacher image display section 321 and the student image display section 322 may be the same as the teacher screen T1. A message 323 "Listen in class" is displayed on the student screen S1. This means listening to the contents of the lesson, and the student screen S1 is displayed while the teacher's voice is being output. In the student screen S1, the teacher has a conversation with the student or the teacher gives an instruction to the student. Students can ask teachers questions.

  FIG. 15 shows an example of the student screen S2. The student screen S2 is a screen in which each student utters a voice by imitating model voice data. The student screen S2 has a teacher image display field 321, a student image display field 322, and an end button 325. The teacher image display section 321 and the student image display section 322 may be the same as the teacher screen T1. A message 324 “Repeat !!” is displayed on the student screen S2. This means that the model voice data is not uttered. When the student screen S2 is displayed, the terminal 70 outputs sample audio data from the speaker, and the management system 50 starts recording the audio data at the timing when the communication ID and the TYPE are transmitted. The end button 325 is a button for the student who has finished speaking to notify the management system 50 of the end of the recording.

  FIG. 16 shows an example of the student screen S3. The student screen S3 is a screen on which student evaluation results are displayed. The student screen S3 has a correct answer sound reproduction button 331, a teacher image display field 332, phonetic symbol fields 333a and 333b, a detailed evaluation field 334, a score field 336, and an answer sound reproduction button 335. These functions may be the same as those of the teacher screen 3.

  In the student screen S3, the evaluation result is displayed for each voice data retried and uttered, so that the student can grasp what is wrong with each voice data, whether the voice data has been improved, what is likely to be wrong, and the like. In addition, sample audio data can be arbitrarily output. In addition, the user can output his own voice data and check it.

<Processing procedure when the teacher utters voice instead of recorded voice data>
In FIG. 10, the flow of the process in which the student utters using the recorded voice data as an example has been described. However, the procedure of the process in which the student utters using the voice data uttered by the teacher as an example will be described with reference to FIG. FIG. 17 is an example of a sequence diagram showing a procedure in which the management system 50 evaluates student voice data. Note that the description of FIG. 17 mainly describes differences from FIG.

  First, steps T1 to T2, steps M1 to M4, and steps S1 to S3 may be the same as those in FIG.

  T3: The teacher presses the start button 305 on the teacher screen T1 to model the voice data uttered in real time.

  T4: The transmitting / receiving unit 11 of the terminal 10 transmits the communication ID and the TYPE to the management system 50 in order to notify the TYPE of the terminal 10 and the communication ID.

  M4: The transmission / reception unit 51 of the management system 50 receives the TYPE and the communication ID, whereby the recording management unit 54 starts recording the teacher's voice data.

  T5: When the teacher finishes speaking, he presses the end button 306 on the teacher screen T2b (FIG. 12B). The operation input receiving unit 12 receives the press.

  T6: The transmission / reception unit 11 of the teacher's terminal 10 transmits the communication ID to the management system 50 to end the recording.

  M5: The transmission / reception unit 51 of the management system 50 receives the communication ID, whereby the recording management unit 54 ends the recording and assigns the recording ID.

  M6: The recording management unit 54 of the management system 50 registers the recording ID, the TYPE, the communication ID, and the recording file in the recording management DB 5003.

  M7: The transmission / reception unit 51 of the management system 50 transmits the audio data registered in the recording management DB 5003 to the terminal 70. As a result, the terminal 70 displays the student screen S2, and the subsequent processing is the same as in FIG.

  In this manner, the communication system 1 of the present embodiment enables evaluation and detailed guidance in comparison with the voice data of many students, even if the teacher utters the voice data of the model.

<Comparison and evaluation of audio data>
In the present embodiment, the voice data of the teacher and the voice data of the student are compared using phonetic symbols, and the degree of pronunciation matching for each word is compared. The score was defined as the ratio of the number of words having the same phonetic symbol in one sentence. As a technique for converting speech data into phonetic symbols, a speech recognition technique can be used.

  FIG. 18 is an example of a diagram schematically illustrating speech recognition. FIG. 18 shows a neural network 401 that converts speech into phonetic symbols by machine learning. In particular, a deep layer of the neural network 401 is called deep learning. It is known that an RNN (Reccurent Neural Network) is suitable for classifying time-series data such as voice data.

  As an example, audio data sliced every 20 milliseconds is converted into MFCC (Mel-frequency cepstral coefficients) and input to each node of the input layer 402. Each node of the output layer 403 corresponds to each phonetic symbol.

  In the learning phase, voice data converted to MFCC is input to the input layer 402, and “1” is assigned to the node corresponding to the phonetic symbol of the voice included in the voice data to the output layer 403 (the other nodes are "0"). Then, the weight between the node of the input layer 402, the node of each layer of the neural network 401, and the node of the output layer 403 is learned by the back propagation method.

  In the recognition phase, a larger value (a value closer to 1) is output to a node of the output layer 403 closer to a phonetic symbol included in the input speech data. The analysis / result management unit 55 extracts a phonetic symbol corresponding to the node that has output a value equal to or greater than the threshold. If there are a plurality of phonetic symbols, refer to the dictionary data that indicates the ease of connection of phonetic symbols by probability, calculate the probability of easy connection for each combination with the preceding and following phonetic symbols, and determine the combination with the highest probability. Specify phonetic symbols so that

  Note that there are various voice recognition methods, and the description in FIG. 18 is merely an example. For example, speech recognition may be performed using a hidden Markov model. In the present embodiment, there is no problem even if other methods are adopted.

  In the present embodiment, the audio data is converted to phonetic symbols, but the audio data may be converted to words (alphabet). A known method can be used to convert the audio data into words (alphabets). In a general English dictionary, phonetic symbols for words are listed, so once a word is determined, phonetic symbols are uniquely determined. Therefore, voice data can be converted into phonetic symbols. However, when an excellent classifier that recognizes a voice with poor pronunciation as a correct word is generated, there is a possibility that a student may be judged to be a correct pronunciation symbol even if the student has poor pronunciation.

  For evaluation of sound data of musical instruments and the like, for example, there is a method of performing Fourier transform on sound data of a teacher and a student, comparing the difference in intensity for each frequency, and digitizing the difference.

<Summary>
As described above, in the communication system 1 of the present embodiment, the management system 50 compares the teacher's voice with each student's voice even in a mode in which a remote teacher instructs a plurality of students. It is possible for one teacher to teach a large number of students in parallel. Teaching that is as detailed as one-on-one lessons is possible.

<Other application examples>
The best mode for carrying out the present invention has been described above by using the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the gist of the present invention. And substitutions can be added.

<< Another example of system configuration >>
Although one management system is shown in FIG. 2, a plurality of management systems 50 may exist in the communication system 1. FIG. 19 is a schematic diagram of the communication system 1 according to the present embodiment. In FIG. 19, a plurality of management systems 50 are connected to a network. As described above, the management system 50 included in the communication system 1 may be plural, and any management system 50 may have a function.

  Further, the system configuration in which the terminal 10 and the management system 50 described in the present embodiment are connected is merely an example, and it goes without saying that there are various system configuration examples according to applications and purposes.

  As shown in FIG. 20, the management system 50 and the relay device 30 may exist as one server device 60. The server device 60 in FIG. 20 has both the function of the management system 50 and the function of the relay device 30. Further, a plurality of server devices 60 may exist.

<< User identification by email address >>
In the present embodiment, the terminals 10 and 70 are identified by the communication ID, but a user ID may be used as identification information for identifying the terminal 10. Also, a mail address may be used as the user ID. In this case, the user of the terminal 10 or 70 logs in to the management system 50 with the mail address.

  FIG. 21 is an example of a sequence diagram illustrating a process in which the terminals 10 and 70 log in to the management system 50. In the description of FIG. 21, differences from FIG. 9 will be mainly described. As shown in FIG. 21, in steps S3 and S13-2, a mail address is transmitted instead of a communication ID. The management system 50 authenticates the user with the mail address and the password.

  Similarly, in the procedure for evaluating student voice data described with reference to FIG. 10, a mail address is transmitted instead of a communication ID in steps S4 and S7. Similarly, in FIG. 17, a mail address is used instead of the communication ID.

<< Video conference using general browser application instead of application >>
In the embodiment described above, the communication applications A1 and A7 operate on the terminals 10 and 70, but the same processing can be realized by the Web application. The Web application operates by coordinating a program based on, for example, JavaScript (registered trademark) that operates on the Web browser and a program on the Web server side, and the user uses the Web application on the browser.

  FIG. 22 is a diagram illustrating a configuration example of a Web application in the terminals 10 and 70. The terminal 10 downloads the teacher program 1030 (HTML5 + JAVASCRIPT + CSS, etc.) from the management system 50 and executes it on the browser application 1040. The terminal 70 downloads the student program 7030 (HTML5 + JAVASCRIPT + CSS, etc.) from the management system 50 and executes it on the browser application 7040. In this case, for example, a communication ID is used to determine whether a teacher or a student.

  In FIG. 22, a user ID may be used as identification information for identifying the terminal 10 instead of the communication ID. A mail address may be used as the user ID.

  The terminals 10 and 70 can use services provided by the management system 50 by transmitting and receiving data to and from the management system 50 using a protocol such as HTTP or HTTPS. In such a usage form, there is no need to download the communication applications A1 and A7 to the terminals 10 and 70 in advance.

<Others>
For example, in the present embodiment, an example has been described in which the user participates in a conference room using a video conference terminal (communication terminal). However, the present invention can be applied to various devices having the function of a communication terminal. For example, the present invention can be suitably applied to an electronic blackboard having a function of a communication terminal.

  Some electronic blackboards have a large touch panel, and others have a projector that projects an image and detects the position of the electronic pen with a sound wave or the like and a camera.

  6 is divided according to main functions in order to facilitate understanding of the processing by the management system 50, the terminals 10, 70, and the relay device 30. The present invention is not limited by the way of dividing the processing unit or the name. The processing of the management system 50, the terminals 10, 70, and the relay device 30 can be divided into more processing units according to the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes.

  Each function of the embodiment described above can be realized by one or a plurality of processing circuits. Here, the “processing circuit” in the present specification refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), SOCs (System on a chip), GPUs (Graphics Processing Units), and devices such as conventional circuit modules.

  In the present embodiment, the teacher and the student have a conversation while the lesson is proceeding. However, the student downloads and outputs the recorded file by himself, and transmits the voice data based on the downloaded file to the management system 50. May be. The management system 50 evaluates the voice data of the student, and the teacher downloads the voice data at an arbitrary timing. This can save each other's time and allow students to do their homework.

  Further, not only audio data but also image data may be used for evaluation. For example, it can be similarly evaluated by converting mouth movements into phonetic symbols by machine learning.

  The transmission / reception unit 51 is an example of a communication unit or a second communication unit, the analysis / result management unit 55 is an example of a generation unit, and the sample audio data is an example of the first sound data. The uttered voice data is an example of second sound data, a teacher is an example of a first user, a student is an example of a second user, and the terminal 10 is an example of a first communication terminal. The terminal 70 is an example of a second communication terminal, the transmission / reception unit 11 is an example of a first communication unit, and the output control unit 14 is an example of a display control unit.

1 Communication System 10, 70 Terminal 30 Relay Device 50 Management System

JP-T-2009-511964

Claims (12)

A server capable of communicating with a plurality of communication terminals via a network,
The first sound data is transmitted to the plurality of second communication terminals according to the instruction of the first communication terminal operated by the first user, and the second users of the plurality of second communication terminals emit the respective sound data. Communication means for receiving the plurality of second sound data respectively;
Generating means for generating an evaluation for each of the plurality of second sound data based on the first sound data and the second sound data obtained from each of the plurality of second communication terminals; And
The server, wherein the communication unit transmits the evaluation obtained by the generation unit to the first communication terminal. The communication means may include, in the first communication terminal, identification information of the plurality of second sound data respectively issued by the second user of the plurality of second communication terminals or the second sound data, respectively. When,
2. The server according to claim 1, wherein screen information of a first screen having a button for individually receiving a plurality of the second sound data is transmitted. 3. The communication means transmits identification information of the first sound data or the first sound data to the first communication terminal,
The server according to claim 2, wherein the first screen further includes a button for receiving an output of the first sound data. The first screen has a retry button for outputting the first sound data again from the plurality of second communication terminals,
When the communication unit receives a notification that the retry button has been pressed,
The communication unit transmits the first sound data used by the generation unit to generate the evaluation to each of the plurality of second communication terminals, and a second user of the plurality of second communication terminals transmits Receiving a plurality of second sound data respectively emitted,
The generation unit compares the first sound data with the second sound data obtained from each of the plurality of second communication terminals, and evaluates each of the second sound data again. Generate
The server according to claim 3, wherein the communication unit transmits the evaluation obtained by the generation unit to the first communication terminal. The first sound data is sound data of a predetermined language,
The generating means converts the first sound data and the plurality of second sound data respectively emitted by the second users of the plurality of second communication terminals into phonetic symbols,
The phonetic symbol converted from the first sound data and the phonetic symbol converted from the second sound data are compared to evaluate each of the plurality of second sound data. Item 5. The server according to any one of Items 1 to 4. The communication means transmits, to the first communication terminal, screen information of a second screen including a list of the first sound data recorded in advance,
When receiving the identification information of the first sound data selected from the list displayed on the second screen,
The server according to any one of claims 1 to 5, wherein the first sound data specified by the identification information is transmitted to each of the plurality of second communication terminals. When a button for receiving the input is pressed on the second screen including a button for receiving an input of the first sound data displayed on the first communication terminal,
The communication unit transmits the first sound data input by the first user to the first communication terminal and received from the first communication terminal to the plurality of second communication terminals, respectively. The server according to claim 6, characterized in that: The communication means is provided to the second communication terminal,
The first sound data identified by the identification information of the first sound data received from the first communication terminal of the first user or the first sound data received from the first communication terminal When,
Screen information of a third screen indicating that the voice of the first user is being output during the output of the first sound data;
8. The server according to claim 7, further comprising: transmitting screen information of a fourth screen that is displayed when the output of the first sound data is completed and that receives an input of the second sound data. 9. . The communication unit transmits the second sound data so that the second communication terminal that transmitted the second sound data displays an evaluation of the plurality of second sound data obtained by the generation unit. Individually transmitted to said second communication terminal,
Further, the identification information of the first sound data or the first sound data, and the identification information of the second sound data or the second sound data, the second sound data transmitted the second sound data 2 individually transmitted to the communication terminal,
The evaluation that the second communication terminal receives the identification information of the first sound data or the first sound data, and the identification information of the second sound data or the second sound data. A fifth screen including a button for receiving the evaluation, the output of the first sound data, and a button for receiving the output of the second sound data, which is displayed at an opportunity, and transitioning from the fourth screen 9. The server according to claim 8, wherein screen information of the fifth screen to be transmitted is transmitted to the second communication terminal. A sound data evaluation method performed by a server capable of communicating with a plurality of communication terminals via a network,
The communication means transmits the first sound data to each of the plurality of second communication terminals according to an instruction of the first communication terminal operated by the first user, and transmits the first sound data to the second communication terminal of the plurality of second communication terminals. Receiving a plurality of second sound data respectively emitted by the user;
Generating means for generating an evaluation for each of the plurality of second sound data based on the first sound data and the second sound data obtained from each of the plurality of second communication terminals;
The communication unit transmitting the evaluation obtained by the generation unit to the first communication terminal;
A sound data evaluation method comprising: A server that can communicate with multiple communication terminals via a network,
The first sound data is transmitted to the plurality of second communication terminals according to the instruction of the first communication terminal operated by the first user, and the second users of the plurality of second communication terminals emit the respective sound data. Communication means for receiving the plurality of second sound data respectively;
The first sound data, based on the second sound data obtained from each of the plurality of second communication terminals, based on the second sound data, generating means for generating an evaluation for each of the plurality of second sound data,
The program, wherein the communication unit transmits the evaluation obtained by the generation unit to the first communication terminal. A communication system having a first communication terminal and a server capable of communicating via a network,
The first communication terminal includes:
First communication means for transmitting a transmission request for transmitting the first sound data to each of the plurality of second communication terminals to the server,
The server comprises:
The first sound data is transmitted to the plurality of second communication terminals in response to the transmission request, and a plurality of second sounds respectively emitted by a second user of the plurality of second communication terminals are transmitted. Second communication means for receiving data respectively;
Generating means for generating an evaluation for each of the plurality of second sound data based on the first sound data and the second sound data obtained from each of the plurality of second communication terminals; ,
The second communication unit transmits the evaluation obtained by the generation unit to the first communication terminal,
The communication system according to claim 1, wherein the first communication terminal includes display control means for displaying, on a display device, the plurality of evaluations received by the first communication means from the server.

Images