JP7119615B2 - Server, sound data evaluation method, program, communication system - Google Patents

Description

The present invention relates to a server, sound data evaluation method, program, and communication system.

In foreign language classes such as English conversation classes and announcement classes, there are many needs for students to actually speak out and receive feedback from teachers to learn correct pronunciation and vocalization. When learning, the teacher and students gather in the same place such as a classroom, and the teacher gives guidance to the sounds that the students make while giving examples.

In recent years, a form in which teachers and students in remote locations learn pronunciation and vocalization using the Internet has also been put to practical use. For example, in foreign language classrooms, there is already known a sound education system in which one-to-one instruction is provided using a communication system used for video conferences and the like.

Also known is a sound education system that assumes that a plurality of students receive instruction from one teacher at the same time (see, for example, Patent Document 1). Patent Literature 1 discloses a system in which a student can answer a set question via a terminal, and a teacher confirms the answer and instructs a plurality of students.

However, the conventional technology has the problem that it is difficult for one teacher to teach a plurality of students in parallel. If the teacher and the student are on a one-to-one basis, the teacher can check the student's voice and give detailed guidance. However, when there are a plurality of students, it takes a long time for the teacher to make the students speak one by one and give advice. On the other hand, if a plurality of students vocalize at the same time, the time can be shortened, but since the object of evaluation is sound, it is difficult for the teacher to simultaneously evaluate the vocalizations of a plurality of students, making detailed guidance difficult.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a server that allows one teacher to teach a plurality of students how to vocalize at the same time.

In view of the above problems, the present invention provides a server capable of communicating with a plurality of communication terminals via a network, wherein the first communication terminal operated by a first user instructs a first communication terminal to transmit first sound data to a plurality of first sound data. communication means for receiving a plurality of second sound data respectively transmitted to two communication terminals and respectively emitted by second users of the plurality of second communication terminals; the first sound data; generating means for generating an evaluation for each of the plurality of second sound data based on the second sound data obtained from each of the plurality of second communication terminals ; to the first communication terminal, the evaluation obtained by the generating means , the identification information of the plurality of second sound data respectively emitted by the second users of the plurality of second communication terminals, or the second sound data It is characterized by transmitting each sound data and screen information of a first screen having a button for individually accepting the output of the plurality of second sound data .

It is possible to provide a server that allows a single teacher to teach a plurality of students in parallel.

FIG. 2 is an example of a diagram illustrating an outline of processing in which a communication system evaluates a student's voice; 1 is an example of a schematic diagram of a communication system; FIG. 2 is a hardware configuration diagram of an example of a terminal; FIG. 1 is a hardware configuration diagram of an example of a management system; FIG. 3 is a software configuration diagram of an example of a terminal; FIG. 1 is an example of a functional block diagram of a terminal forming part of a communication system and a management system; FIG. 4A and 4B are diagrams showing examples of an authentication management table, a terminal management table, and a group management table; FIG. 4A and 4B are diagrams showing examples of a recording management table and a result management table; FIG. FIG. 10 is an example of a sequence diagram showing processing in which a terminal logs in to a management system; FIG. 10 is an example of a sequence diagram showing a procedure for the management system to evaluate voice data of students (output recording file); FIG. 10 is a diagram showing an example of a teacher screen T1; FIG. 10 is a diagram showing an example of teacher screens T2a and T2b; FIG. 10 is a diagram showing an example of a teacher screen T3; FIG. 10 is a diagram showing an example of a student screen S1; FIG. 10 is a diagram showing an example of a student screen S2; FIG. 10 is a diagram showing an example of a student screen S3; FIG. 10 is an example of a sequence diagram showing a procedure for the management system to evaluate the voice data of the students (uttered by the teacher); It is an example of the figure which illustrates speech recognition typically. Figure 2 is another example of a schematic diagram of a communication system; Figure 2 is another example of a schematic diagram of a communication system; FIG. 10 is an example of a sequence diagram showing processing in which a terminal logs in to a management system; FIG. 3 is a diagram illustrating a configuration example of a web application in a terminal;

Hereinafter, as an example of the embodiment of the present invention, a communication system and a sound data evaluation method performed by the communication system will be described with reference to the drawings.

<Overview of processing>
FIG. 1 is an example of a diagram illustrating an outline of processing for evaluating a student's voice by a communication system 1 according to this embodiment. In FIG. 1, a terminal 10 used by a teacher and terminals 70A to 70C (each of which corresponds to a communication terminal to be described later) used by students are connected to a management system 50 via a communication network so as to be able to communicate with each other.
(1) First, the teacher utters a model voice (or a pre-registered model voice may be used). The model voice is transmitted to the terminals 70A-70C via the management system 50, and can be listened to by the students.
(2) Next, using the teacher's speech as a model, the student utters speech almost simultaneously (overlapping at least part of the time period). The student's voice is sent to the management system 50 .
(3) The management system 50 compares the exemplar speech and the student's speech respectively to evaluate how close the student's speech is to the exemplar.
(4) The management system 50 sends the evaluation results of all students to the terminal 10 of the teacher. Since the terminal 10 displays the evaluation results of all the students, the teacher can check the evaluation results of each student and give detailed guidance to improve the vocalization of each student.

As described above, the communication system 1 of the present embodiment can be used even in a form in which a remote teacher instructs a plurality of students to vocalize, since the management system 50 compares the teacher's voice with each student's voice. becomes possible to teach the vocalization of many students in parallel.

<Terms>
A meeting originally means meeting and deliberating, or gathering and discussing. Meetings for sound evaluation and instruction are also forms of conferences.

The sound data may be data obtained by converting air vibrations into electric signals. In this embodiment, voice data uttered by a person will be described as an example. In addition, although the communication system 1 instructing vocalization in English as a predetermined language will be described, it can also be applied to predetermined languages other than English (Chinese, French, Spanish, Russian, German, Arabic, etc.). Moreover, the sound data is not limited to voice, and may be the sound emitted by a musical instrument in a music classroom or the like.

Also, sound refers to a state in which the air vibrates, and sound data is information or data, but the two are not strictly distinguished in this embodiment.

“Teaching in parallel” means that even if multiple students participate in one video conference and the time period for uttering sounds overlaps, it is possible to provide instruction equivalent to one-on-one lessons. . Teaching in parallel may be referred to as "teaching at the same time."

<System configuration example>
FIG. 2 is a schematic diagram of the communication system 1 according to this embodiment. As shown in FIG. 2, the communication system 1 manages a plurality of communication terminals 10a, 10b, and 10c, which are examples of video conference terminals, a plurality of communication terminals 70x and 70y, which are examples of smartphones, and each communication terminal. It is constructed by the communication management system 50 and the relay device 30 . Hereinafter, the “communication terminal” will be referred to as “terminal”, and the “communication management system” will be referred to as “management system”.

Although FIG. 2 shows three terminals 10a, 10b, 10c and two terminals 70x, 70y, the number of terminals is not limited to this. An arbitrary terminal among the plurality of terminals 70x and 70y is denoted as "terminal 70", and an arbitrary terminal among the plurality of terminals 10a, 10b and 10c is denoted as "terminal 10". The terminals 10 and 70 may be general-purpose computers having communication functions, electronic whiteboards, car navigation terminals, electronic signboards (digital signage), or the like.

The management system 50 is an information processing device having a server function, and is a so-called server (corresponding to the server in the claims). Terminals 10 and 70 are computers with corresponding client functionality. The terminals 10 and 70, the relay device 30, and the management system 50 can communicate via a communication network 2 such as the Internet, mobile phone network, LAN (Local Area Network), WiFi (Wireless Fidelity), or Bluetooth (registered trademark). . The communication network 2 also includes a base station 2a at the end of the mobile phone network. Although one base station 2a is shown in FIG. 2, the number is not limited to this.

The relay device 30 relays content data such as sound data (including audio data), video (image) data, and text data between the terminals 10 and 70 .

The terminal 10 is placed in the teacher's room (home, classroom, etc.) as an example of the base, and the terminal 70 is placed in each student's home as an example of the base. Teachers can be based anywhere, and students can be based anywhere. Also, the terminal 70 may be used by the teacher, and the terminal 10 may be used by the student. In this embodiment, for convenience of explanation, it is assumed that the terminal 10 is used by the teacher and the terminal 70 is used by the student.

The management system 50, the relay device 30, and the terminals 10 and 70 may be located in the same country or region, or may be located in different countries or regions. A user of the terminal 10 is, for example, a teacher, and a user of the terminal 70 is, for example, a student.

The communication system 1 includes a data providing system for unidirectionally transmitting content data from one terminal to the other via a management system 50, and a system for exchanging information, emotions, etc. between a plurality of terminals via the management system. Includes a communication system that communicates. This communication system is a system for mutually transmitting information, emotions, etc., between a plurality of terminals via a management system, and examples include a video conference system, a video telephone system, and the like.

<Hardware configuration example>
<<Example of terminal hardware configuration>>
FIG. 3A is a hardware configuration diagram of the terminal 10 according to one embodiment. The terminal 10 includes a CPU 101 (Central Processing Unit) that controls the overall operation of the terminal 10, a ROM 102 (Read Only Memory) that stores programs used to drive the CPU 101, such as an IPL (Initial Program Loader), and a work area for the CPU 101. and a flash memory 104 for storing various data such as programs for the terminal 10, image data, and sound data. In addition, an SSD 105 (Solid State Drive) that controls reading or writing of various data to or from a flash memory 104 under the control of the CPU 101, data reading or writing (storage) to or from a recording medium 106 such as a flash memory or an IC card (Integrated Circuit Card). ) to control the media I/F 107 (Interface). Also, an operation button 108 operated when selecting a destination of the terminal 10, a power switch 109 for switching ON/OFF the power of the terminal 10, and a network I for transmitting data using the communication network 2. /F111 is provided.

The terminal 10 also includes a built-in camera 112 that captures an image of a subject and obtains image data under the control of the CPU 101, an image sensor I/F 113 that controls driving of the camera 112, a built-in microphone 114 that inputs sound, It has a built-in speaker 115 that outputs sound, and a sound input/output I/F 116 that processes input/output of sound signals between the microphone 114 and the speaker 115 under the control of the CPU 101 . In addition, a display I/F 117 for transmitting image data to an external display 120 under the control of the CPU 101, an external device connection I/F 118 for connecting various external devices, and an alarm lamp for notifying an abnormality of various functions of the terminal 10. 119, and bus lines 110 such as an address bus and a data bus for electrically connecting the above components as shown in FIG. 3(a).

The display 120 is a display device that displays an image of a subject and the like. Examples of the display 120 include liquid crystal and organic EL (organic electroluminescence). Also, the display 120 is connected to the display I/F 117 by a cable 120c. This cable 120c may be an analog RGB (VGA) signal cable, a component video cable, HDMI (registered trademark) (High-Definition Multimedia Interface) or DVI (Digital Video Interface). Interactive) signal cable.

The camera 112 includes a lens and a solid-state imaging device that converts light into electric charge and digitizes an image (video) of a subject. etc. are used.

External devices such as an external camera, an external microphone, and an external speaker are electrically connected to the external device connection I/F 118 by a USB (Universal Serial Bus) cable or the like inserted into the connection port 1132 of the housing 1100. can be connected to When an external camera is connected, the external camera is driven under control of the CPU 101 . Similarly, when an external microphone is connected or an external speaker is connected, the external microphone and the external speaker are driven under the control of the CPU 101 .

Note that the recording medium 106 is detachable from the terminal 10 . In addition, as long as it is a non-volatile memory from which data is read or written under the control of the CPU 101, not only the flash memory 104 but also an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used.

FIG. 3B is a hardware configuration diagram of the terminal 70. As shown in FIG. As shown in FIG. 3B, the terminal 70 includes a CPU 701 that controls the overall operation of the terminal 70, a ROM (Read Only Memory) 702 that stores programs, a RAM 703 that is used as a work area for the CPU 701, a CPU 701 EEPROM (Electrically Erasable and Programmable ROM) 704 for reading or writing data under the control of . In addition, a media I/F 707 for controlling reading or writing (storage) of data to a recording medium 706 such as a flash memory, and a CMOS (Complementary Metal Oxide Semiconductor) sensor 712 for capturing an image of an object and obtaining image data under the control of the CPU 701 are provided. I have.

The EEPROM 704 stores an operating system (OS) executed by the CPU 701, other programs, and various data. Also, the CMOS sensor 712 is a charge-coupled device that converts light into electric charge and digitizes an image of an object. The CMOS sensor can be replaced with a CCD (Charge Coupled Device) sensor if it can capture an image of an object.

Further, the terminal 70 has a microphone 714 that converts sound into a sound signal, a speaker 715 that converts a sound signal into sound, an antenna 711a, and uses the antenna 711a to communicate with the nearest base station 2a using a wireless communication signal. A communication unit 711, a display 720 such as a liquid crystal or organic EL that displays an image of a subject, various icons, etc., is placed on the display 720, and is configured by a pressure-sensitive or electrostatic panel. A touch panel 721 for detecting a touch position on the display 720 and a bus line 710 such as an address bus and a data bus for electrically connecting the above units are provided.

<<Example of hardware configuration of management system>>
FIG. 4 is a hardware configuration diagram of the management system 50 according to one embodiment. The management system 50 includes a CPU 201 that controls the overall operation of the management system 50, a ROM 202 that stores programs used to drive the CPU 201 such as IPL, a RAM 203 that is used as a work area for the CPU 201, various programs for the management system 50, and the like. It has an HD 204 for storing data. In addition, an HDD 205 (Hard Disk Drive) that controls reading or writing of various data to or from the HD 204 under the control of the CPU 201, a media I/F 207 that controls reading or writing (storage) of data to or from a recording medium 206 such as a flash memory, a cursor , menus, windows, characters, or images. Also, a network I/F 209 for data communication using the communication network 2, a keyboard 211 having a plurality of keys for inputting characters, numerical values, various instructions, etc., selection and execution of various instructions, processing target It has a mouse 212 for selecting and moving a cursor, and a CD-ROM drive 214 for controlling reading and writing of various data to and from a CD-ROM 213 (Compact Disc Read Only Memory) as an example of a removable recording medium. Furthermore, as shown in FIG. 4, there are bus lines 210 such as an address bus and a data bus for electrically connecting the components described above.

Since the relay device 30 has the same hardware configuration as the management system 50, the description thereof will be omitted.

<Software configuration example>
FIG. 5A is a software configuration diagram of the terminal 10 according to one embodiment. A “communication application A1” is installed in the terminal 10 as a client application. Application means application software. As shown in FIG. 5A, the OS 1020 and communication application A1 operate on the work area 1010 of the RAM 103 of the terminal 10. FIG. Among these, the OS 1020 is basic software that provides basic functions and manages the terminal 10 as a whole. The communication application A1 is an application that operates on the terminal 10 used by the teacher and communicates with other terminals 70 .

FIG. 5B is a software configuration diagram of the terminal 70 according to one embodiment. A “communication application A7” is installed in the terminal 70 as a client application. As shown in FIG. 5B, the OS 7020 and communication application A7 operate on the work area 7010 of the RAM 703. FIG. Of these, the OS 7020 is basic software that provides basic functions and manages the terminal 70 as a whole. The communication application A7 is an application that operates on the terminal 10 used by the student and communicates with the terminal 10 .

The communication protocols of the communication applications A1 and A7 are (1) SIP (Session Initiation Protocol), (2) H.323, (3) an extended protocol of SIP, (4) instant messenger protocol, and (5) (6) Internet relay chat protocol (IRC (Internet Relay Chat)); Of these, (4) instant messenger protocol is, for example, (4-1) XMPP (Extensible Messaging and Presence Protocol), or (4-2) ICQ (registered trademark), AIM (registered trademark), or Skype (registered trademark) trademark), etc. (7) A protocol obtained by extending the instant messenger protocol is, for example, Jingle.

<Functions of communication system>
Next, the functional configuration of the communication system 1 of this embodiment will be described using FIG. FIG. 6 is a functional block diagram of the terminals 10 and 70 forming part of the communication system 1 and the management system 50. As shown in FIG. 6, the terminals 10 and 70 and the management system 50 are connected via the communication network 2 so as to be able to perform data communication.

<Functional configuration of terminal>
The terminals 10 and 70 have transmission/reception sections 11 and 71, operation input reception sections 12 and 72, activation sections 13 and 73, output control sections 14 and 74, and storage/read sections 19 and 79, respectively. Any one of the components shown in FIG. 3 executes commands from the CPUs 101 and 701 according to the communication applications A1 and A7 (programs) developed on the RAMs 103 and 703 from the flash memory 104 or EEPROM 704. It is a function realized by operating by

The terminal 10 also has storage units 1000 and 7000 constructed by the ROMs 102 and 702, the RAMs 103 and 703, the flash memory 104 or the EEPROM 704 shown in FIG.

Each functional configuration in the terminal 10 will be described in detail. The transmission/reception units 11 and 71 are implemented by commands from the CPUs 101 and 701 and the network I/F 111 or the communication unit 711, and transmit various data (or information ) are sent and received.

The operation input reception units 12 and 72 are implemented by commands from the CPUs 101 and 701, the operation buttons 108 or the touch panel 721, and the power switch 109, and receive various inputs or selections by the user.

The activation units 13 and 73 are realized by commands from the CPUs 101 and 701, and activate the operations of the communication applications A1 and A7.

Output control units 14 and 74 are realized by commands from CPUs 101 and 701, display I/F 117 and sound input/output I/F 116, and control output of image data and sound data.

The storage/reading units 19 and 79 are implemented by commands from the CPUs 101 and 701, and store various data in the storage units 1000 and 7000 and read various data stored in the storage units 1000 and 7000. .

<< Functional configuration of the management system >>
The management system 50 has a transmission/reception section 51 , an authentication section 52 , a management section 53 , a recording management section 54 , an analysis/result management section 55 , a session control section 58 , and a storage/readout section 59 . Each of these units is a function realized by the operation of any one of the components shown in FIG. be. The management system 50 also has a storage unit 5000 constructed by the HD 204 . Further, in the storage unit 5000, each DB is constructed by each table as shown below.

(Authentication management table)
FIG. 7A is a conceptual diagram showing an authentication management table. In the storage unit 5000, an authentication management DB 5001 is constructed by an authentication management table as shown in FIG. 7(a). In this authentication management table, each communication ID of all the terminals 10 and 70 managed by the management system 50 is associated with a password for authentication and managed.

(terminal management table)
FIG. 7B is a conceptual diagram showing a terminal management table. In the storage unit 5000, a terminal management DB 5002 is constructed by a terminal management table as shown in FIG. 7(b). In this terminal management table, the IP address and TYPE of each terminal 10, 70 are associated with each communication ID of each terminal 10, 70 and managed. The communication ID of each terminal 10,70 is grasped by logging in each terminal 10,70. The TYPE will be described later, but the TYPE may be registered at the time of login or may be registered in advance.

(group management table)
FIG. 7(c) is a conceptual diagram showing a group management table. In the storage unit 5000, a group management DB 5005 is built with a group management table as shown in FIG. 7(c). In this group management table, the communication IDs of the terminals 10 and 70 are managed in association with the group ID. A group is a collection of one teacher and several students supervised by this teacher. A group ID is identification information for identifying a group. One group participates in the same conference (an example of communication), and content data is mutually transmitted and received between the terminals 10 and 70 of the group.

As a method of creating the group management table, for example, a teacher accesses the management system 50 and registers before the meeting starts. Alternatively, as a method that eliminates the need to create a group management table in advance, the management system 50 distributes the same invitation code by e-mail or the like to students participating in the same conference, and the student enters the invitation code when logging in, and the management system 50 may classify students with the same invitation code into the same group (create a group management table after the fact). Alternatively, students may be divided into levels in advance according to their language ability (levels may be registered in the authentication management table), and the management system 50 may classify students of the same level into the same group in order of login. For example, if the number of students in one group is three, a group ID is assigned every time three students of the same level log in. In this case, each group is assigned a teacher and the meeting begins.

(recording management table)
FIG. 8(a) is a conceptual diagram showing a recording management table. In the storage unit 5000, a recording management DB 5003 is constructed by a recording management table as shown in FIG. 8(a). The recording management table is a table in which information about recorded voice data is registered.

The recording management table is associated with recording ID, TYPE, communication ID, and recording file items. A recording ID is identification information that uniquely identifies or identifies a recorded file of teacher's or student's voice data, and is numbered by the management system 50 for each recording. In FIG. 8A, the recording ID is a concatenation of the communication ID, meeting ID, question number (described later), and retry number (described later).

TYPE is an attribute of a recording file that indicates whether the recording file is a voice uttered by a student (STUDENT), a voice uttered by a teacher (TEACHER), or a pre-recorded (PRESET) voice. be. For example, communication applications A1 and A7 transmit TYPE to the management system 50 to determine. The communication ID is the same as above, but in the case of pre-recorded voice, the communication ID is NULL (meaning that there is no value). The recording file name is the file name of the voice data actually recorded. The file name is automatically assigned by the management system 50, and the recording ID is used as the file name in FIG. 8(a). One row of the recording management table is called a record.

(result management table)
FIG. 8B is a conceptual diagram showing the result management table. In the storage unit 5000, a result management DB 5004 is constructed with a result management table as shown in FIG. 8B. The result management table is a table summarizing how the student's speech was evaluated in comparison with the model speech.

The results management table is associated with each item of model recording ID, answer recording ID, meeting ID, problem number, retry number, model recognition result, answer recognition result, and scoring result. The IDs registered in the recording management DB 5003 are diverted to the sample recording ID and answer recording ID. The TYPE of the sample recording ID is either PRESET or TEACHER, and the TYPE of the answer recording ID is STUDENT. The conference ID is identification information that uniquely determines which conference is held until the end of the conference. The conference ID is numbered by the management system 50 at the start of the conference. A question number is identification information for identifying a question given by a teacher in one meeting. A question number is assigned by the management system 50 each time a question is given. A retry number is a branch number for a question when the same question is presented again. The model recognition result is the recognition result of the voice data of the model. In FIG. 8(b), as an example, the result of conversion into phonetic symbols is the recognition result. The answer recognition result is the recognition result of the student's speech data. In FIG. 8(b), as an example, the result of conversion into phonetic symbols is the recognition result. As the scoring results, the result of comparison between the model recognition result and the answer recognition result is indicated by "○" and "×" for each word. One row of the result management table is called a record.

<<Each functional configuration of the management system>>
Next, each functional configuration of the management system 50 will be described in detail. The transmission/reception unit 51 is realized by commands from the CPU 201 and the network I/F 209 , and transmits/receives various data (or information) to/from each terminal, device, or system via the communication network 2 .

The authentication unit 52 is realized by a command from the CPU 201, searches the authentication management table using the communication ID and password received by the transmission/reception unit 51 as a search key, and the same communication ID and password are managed in this authentication management table. The terminal is authenticated by determining whether the terminal is present.

The management unit 53 is realized by a command from the CPU 201, and manages the terminals 10 and 70 by registering the terminals 10 and 70 currently logged in to the terminal management table.

The recording management unit 54 is implemented by a command from the CPU 201, starts recording when triggered by the teacher's operation, and ends recording when triggered by the teacher's or student's operation. Voice data is recorded from the start to the end, and each item is registered in the recording management DB 5003 .

The analysis/result management unit 55 is realized by a command from the CPU 201 , compares/analyzes the recorded speech data of the teacher and the student, and records the result in the result management DB 5004 .

Session control unit 58 controls sessions for transmitting content data between terminals 10 and 70 according to commands from CPU 201 . This control includes control for establishing a session, control for joining the terminals 10 and 70 to the established session, control for leaving the session, and the like.

The storage/readout unit 59 is realized by a command from the CPU 201 and the HDD 205, or is realized by a command from the CPU 201, stores various data in the storage unit 5000, extracts various data stored in the storage unit 5000, and performs other operations. process.

<Processing and operation of communication system>
Next, processing and operations in the communication system 1 will be described.

<<Login process>>
First, the process for the terminals 10 and 70 to log in to the management system 50 will be described with reference to FIG. FIG. 9 is an example of a sequence diagram showing processing for the terminals 10 and 70 to log into the management system 50. As shown in FIG.

When the user of the terminal 70 turns on the power switch, the operation input receiving unit 72 receives the power ON and starts the terminal 70 (step S1). When the terminal 70 is activated, the activation unit 73 activates the communication application A7 installed on the terminal 70 (step S2). After that, the processing in the terminal 70 is executed by the command of the communication application A7.

The transmitter/receiver 71 of the terminal 70 transmits a login request to the management system 50 via the communication network 2 (step S3). This login request includes a communication ID for identifying the own terminal that is the login request source and a password.

The transmission/reception unit 51 of the management system 50 receives the login request. By transmitting a login request from the terminal 70 to the management system 50, the management system 50 on the receiving side can acquire the IP address of the terminal 70 on the transmitting side.

Next, the authentication unit 52 of the management system 50 searches the authentication management table of the storage unit 5000 using the communication ID and password included in the login request as search keys. is managed (step S4).

When the authentication unit 52 authenticates that the login request is from the terminal 70 having valid use authority, the management unit 53 stores the communication ID and the IP address of the terminal 70 that issued the login request in the terminal management table. are stored in association with each other (step S5). Accordingly, the terminal management table manages information for accessing the logged-in terminal 70 . Also, the management unit 53 receives information indicating that the communication application is A7, and registers TYPE in the terminal management table.

The transmitting/receiving unit 51 of the management system 50 transmits authentication result information indicating the authentication result obtained by the authentication unit 52 to the terminal 70 that requested the login via the communication network 2 (step S6). Thereby, the transmitting/receiving unit 71 of the terminal 70 receives the authentication result information. A case will be described below in which authentication of the terminal 70 (there are a plurality of terminals) operated by the student, which is the source of the login request, is successfully authenticated, and the terminal 70 logs into the management system 50 .

On the other hand, when the teacher of the terminal 10 on the teacher side turns on the power switch 109, the operation input receiving unit 12 receives the power ON and starts the terminal 10 (step S11). When the terminal 10 is activated, the activation unit 13 activates the communication application A1 installed on the terminal 10 (step S12). After that, the processing in the terminal 10 is executed by the command of the communication application A1.

The terminal 10 transmits a login request to the management system 50 and logs into the management system 50 (steps S13-2, S14, S15, S16). Since this processing is the same as the processing of steps S3, S4, S5, and S6 between the terminal 70 and the management system 50, the explanation is omitted.

As a method for the management system 50 to determine whether the user operating the terminals 10, 70 is a teacher or a student, there is a method of identifying the communication applications A1, A7 as described in step S5. In this method, for example, the terminals 10 and 70 may transmit information distinguishing the communication applications A1 and A7 to the management system 50 at the time of login. Alternatively, there is a method of distinguishing and distributing communication IDs in advance. For example, communication IDs distributed to teachers begin with T, such as "T001", and communication IDs distributed to students begin with S, such as "S001". In these ways, management system 50 can easily determine whether it is a teacher or a student. Alternatively, if the above invitation code is used, a specific invitation code may be distributed to the student to determine whether the student is a teacher or a student.

When the terminals 10 and 70 log in, the session control unit 58 determines, for example, the relay device 30 with a light load, and causes this relay device 30 to start relaying the content data transmitted/received by the terminals 10 and 70 . The communication IDs of the terminals 10 and 70 (any information that can identify the terminals 10 and 70 ) and the group ID are transmitted to the relay device 30 in association with each other. As a result, the content data transmitted by the terminals 10, 70 together with the communication ID (information that can identify the terminals 10, 70) is transferred to the terminals 10, 70 in the same group.

<<Evaluation of voice data>>
Next, with reference to FIG. 10, the process of evaluating the student's voice data by the management system 50 will be described. Here, imagine a state in which an online English conversation class has been opened. In the following online English conversation classroom, it is assumed that the teacher will set questions as appropriate while conversing with the students.

FIG. 10 is an example of a sequence diagram showing a procedure for the management system 50 to evaluate the student's voice data. The processing of FIG. 10 will be described from the state in which the terminals 10 and 70 have logged in and the relay device 30 has been determined. 11 to 16, examples of screens displayed by the teacher's terminal 10 and the student's terminal 10 will be described as appropriate. Although the relay device 30 is not shown in FIG. 10, the management system 50 is considered to be integrated with the relay device 30 in FIG. Either the management system 50 or the relay device 30 may actually evaluate the voice.

M1: The recording management unit 54 of the management system 50 numbers the conference ID when the conference starts. This makes it possible to manage voice data.

M2-1, M2-2: The session control unit 58 of the management system 50 sends a conference start request to the terminals 10 and 70 because preparations for the conference have been completed. This starts the teleconference. In other words, transmission/reception of content data is started.

T1, S1: The communication application A1 of the terminal 10 and the communication application A7 of the terminal 70 perform conference start processing. Specifically, the communication application A1 and the communication application A7 display screens, start communication with the relay device 30, and acquire necessary information from the management system 50. FIG.

S2: The output control unit 74 of the terminal 70 on the student side displays the student screen S1 (an example of the third screen). FIG. 14 shows the student screen S1. While the student screen S1 is being displayed, the student can basically hear the teacher's voice, but can speak arbitrarily.

M3, M4: On the other hand, the transmitting/receiving section 51 of the management system 50 transmits a list of prerecorded files to the terminal 10 of the teacher. That is, the recording management DB 5003 transmits the recording ID and recording file name of the recording file whose TYPE is PRESET. The terminal 10 on the teacher side is registered in the terminal management DB. The list is composed of structured documents, and examples of structured documents include HTML and XML (Extensible Markup Language).

T2: The transmitting/receiving unit 11 of the terminal 10 receives a list of recorded files recorded in advance. As a result, the output control unit 14 of the terminal 10 displays the teacher screen T1 (an example of the second screen). FIG. 11 shows the teacher screen T1. The teacher screen T1 is a screen where the teacher arbitrarily speaks and selects or inputs voice data for questions as necessary.

T3: In FIG. 10, it is assumed that the teacher has selected a pre-recorded voice (recorded file name) on the teacher screen T1. Processing when the teacher inputs voice will be described with reference to FIG. The operation input reception unit 12 of the terminal 10 receives the operation, and the output control unit 14 displays the teacher screen T2a. The teacher screen T2a is shown in FIG. 12(a). The teacher screen T2a is a screen in which the teacher has selected voice data.

T4: The transmitting/receiving unit 11 of the terminal 10 transmits the recording ID of the recording file name selected by the teacher to the management system 50 . This recording ID serves as a transmission request for the recording file.

M5: The transmission/reception unit 51 of the management system 50 receives the recording ID. The recording management unit 54 acquires voice data corresponding to the recording ID from the recording management DB 5003 .

M6: The transmission/reception unit 51 of the management system 50 transmits the acquired voice data to the terminal 70 of the student. In FIG. 10, there is one student's terminal 70, but it is assumed that there are a plurality of terminals 70. FIG.

S3: The transmitting/receiving unit 71 of the student's terminal 70 receives the voice data, and upon receiving the voice data, the output control unit 74 displays the student screen S2 (an example of the fourth screen). The student screen S2 may be displayed at any timing from the start of reception of voice data, the completion of reception, or from the start to the end of reception. It is preferable to switch from the student screen S1 to the student screen S2 upon completion of reception (also referred to as output completion because the information is output in real time along with reception). Note that the management system 50 may explicitly transmit to the terminal 70 a signal for transitioning from the student screen S1 to the student screen S2. FIG. 15 shows the student screen S2. The student screen S2 is a screen in which the voice data transmitted from the management system 50 is output from the speaker, and the student utters voice using this as a model. In this way, when the teacher sets a question, the screen of the student's terminal 70 automatically changes, so the student does not need to operate the communication application A7 to display the student screen S2. In other words, the teacher has the initiative to control the screen of the student's terminal 70 .

S4: The transmission/reception unit 71 of the terminal 70 displaying the student screen S2 transmits the communication ID and TYPE to the management system 50 . This is for recording audio. Since TYPE is registered in the terminal management DB 5002, it does not have to be transmitted.

M7: When the transmission/reception unit 51 of the management system 50 receives the communication ID and TYPE, the recording management unit 54 starts recording.

S5: Since the terminal 70 that has transitioned to the student screen S2 has started transmitting voice data, the transmitting/receiving unit 71 transmits voice data uttered by the student to the management system 50 at any time (in real time). Recording manager 54 of management system 50 continues to record.

S6: After finishing speaking, the student presses the end button 306 on the student screen S2.

S7: The operation input reception unit 72 receives the student's operation, and the transmission/reception unit 71 transmits the communication ID again. This is to inform the management system 50 that the utterance has ended.

M8: The transmission/reception unit 51 of the management system 50 receives the communication ID, and the recording management unit 54 finishes recording the voice data. The recording management unit 54 assigns a recording ID as an identifier of the recording file. As described above, the communication ID, meeting ID, problem number, and retry number are recording IDs. As another pattern, recording may be performed on the terminal 70 side, and the recorded file may be sent to the management system 50 after recording is completed.

M9: The recording management unit 54 registers the recording ID, TYPE, communication ID, and recording file in the recording management DB 5003 .

M10: Then, the analysis/result management unit 55 compares and evaluates the output (output means reproduction) audio data and the audio data transmitted from the terminal 70 . Details will be described later.

M11: The analysis/result management unit 55 registers the recording ID of the recorded file used as a model as a model recording ID and the recording ID of the student's recording file as an answer recording ID in the result management DB 5004 . Similarly, the meeting ID, the question number of this meeting (question number), the retry number indicating how many times the same question is retried, the model recognition result, and the answer recognition result are registered in the result management DB 5004. do. Retry will be described in detail in the teacher screen T3 (an example of the first screen).

M12-1, M12-2: The transmission/reception unit 51 of the management system 50 transmits the evaluation results to the terminal 10 on the teacher side and the terminal 70 on the student side, respectively. In addition, the recording ID of the teacher's voice data (or the voice data itself) is transmitted to each of the student terminals 70, and the recording ID of the voice data uttered by the student (or the voice data itself) is sent to the terminal that uttered the voice data. 70 separately. The recording ID of the teacher's voice data (or the voice data itself) and the recording IDs of all the voice data uttered by each student (or the voice data itself) are transmitted to the terminal 10 .

T5: The transmission/reception unit 11 of the terminal 10 receives the evaluation result, and the output control unit 14 displays the teacher screen T3. FIG. 13 shows the teacher screen T3. The teacher screen T3 is a screen for the teacher to confirm the evaluation result of each student.

S8: The transmitting/receiving unit 71 of the terminal 70 receives the evaluation result, and upon receiving the evaluation result, the output control unit 74 transitions from the student screen S2 to the student screen S3 (an example of the fifth screen). Note that the management system 50 may explicitly transmit to the terminal 70 a signal for transitioning from the student screen S2 to the student screen S3. FIG. 16 shows the student screen S3. The student screen S3 is a screen for the student to confirm his/her own evaluation result. The student does not need to operate the communication application A7 to display the student screen S3 when transitioning to the student screen S3.

<Screen example>
Examples of screens displayed by the teacher's terminal 10 and the student's terminal 70 will be described with reference to FIGS. 11 to 16. FIG. FIG. 11 shows an example of the teacher screen T1. The teacher screen T1 is a screen displayed by the terminal 70 at the beginning of the conference. The teacher screen T1 has a teacher image display field 301, a student image display field 302, a prerecorded data list field 303, an upload button 304, a start button 305, and an end button 306.

The screen information of the teacher screen T1 includes at least HTML and may be created by WebAPP (Web Application) including CSS (Cascade Style Sheet), JavaScript (registered trademark), and the like. The same applies to teacher screens T2 and T3 and student screens S1 to S3, which will be described later.

The teacher image display column 301 is a column for displaying image data captured by the terminal 10 used by the teacher. The student image display column 302 is a column in which image data captured by the terminal 70 used by the student is displayed. Although there are four student image display columns 302 in FIG. 11, the number of student image display columns 302 increases or decreases depending on the number of students.

The prerecorded data list field 303 is a field in which a list of recorded file names transmitted from the management system 50 to the terminal 10 is displayed. Although five recording file names are displayed in FIG. 11, the number is an example. If it cannot be displayed, a scroll button etc. will be displayed. Each recording file name can accept a user's press (selection). The upload button 304 is a button for the terminal 10 to send the recording ID of the recording file name selected by the teacher to the management system 50 . The start button 305 and the end button 306 are used when inputting voice data uttered by the teacher in real time instead of pre-recorded files and sending the data from the terminal 10 to the management system 50 . That is, the voice uttered by the teacher from when the teacher presses the start button 305 to when the end button 306 is pressed is transmitted to the management system 50 . Details will be described with reference to FIG.

FIG. 12(a) shows an example of the teacher screen T2a. The teacher screen T2a transitions from the teacher screen T1. Regarding the teacher screen T2a, differences from the teacher screen T1 will be explained. The configuration of the teacher screen is the same as that of the teacher screen T1, but one recording file name in the pre-recorded data list column 303 is selected in the teacher screen T2a. In this manner, the teacher-selected recording file name is highlighted, and upload button 304 enables transmission of the recording ID to management system 50 .

On the teacher screen T2a, if the teacher selects a recorded file, the teacher can have the students listen to the sample voice data, so the same voice data can always be heard. Note that the recording file does not have to be recorded by the teacher (same person), and can be reused.

FIG. 12(b) shows an example of the teacher screen T2b. The teacher screen T2b transitions from the teacher screen T1. Regarding the teacher screen T2b, differences from the teacher screen T1 will be explained. The configuration of the teacher screen is the same as that of the teacher screen T1, but the start button 305 is pressed on the teacher screen T2b. Therefore, the teacher is uttering a voice, and this voice data is transmitted to the management system 50 in real time.

On the teacher screen T2b, the teacher can make the students listen to the model voice data in real time, so that the students can practice arbitrary sentences (English sentences or sentences). Further, as is apparent from the configurations of the teacher screens T1, T2a, and T2b, the teacher can give lessons while appropriately combining pre-recorded voice data and real-time voice data.

FIG. 13 shows an example of the teacher screen T3. The teacher screen T3 has a correct answer voice playback button 311, a retry button 312, an end button 313, a student image display column 314, phonetic symbol columns 316a and 316b, a detailed evaluation column 317, a score column 318, and an answer voice playback button 315. . The correct voice reproduction button 311 is a button for outputting a model recorded file on the terminals 10 and 70 . The recording ID of the model recording file is sent to the management system 50 . In this case, the retry number is not incremented because the voice data simply flows. The retry button 312 is a button for sending the recording ID of the same model recording file again to the management system 50 and having the students vocalize it.

When the same question is asked again, the teacher presses the retry button 312 of the terminal 10, and the recording management unit 54 and the analysis/result management unit 55 re-execute step T4 and subsequent steps in FIG. repeat. First, the recording ID of the sample recording file and the fact that it is a retry are sent to the management system 50 . The recording management unit 54 causes a transition from the student screen S3 to the student screen S1 by transmitting a retry to the terminal 70 (this serves as a signal for transition from the student screen S3 to the student screen S1). Next, the recording management unit 54 transmits the recording file of the recording ID to the terminal 70, so that the terminal 70 transitions from the student screen S1 to the student screen S2 at the completion of receiving the recording file, for example. The management system 50 may explicitly transmit to the terminal 70 a signal for transitioning from the student screen S1 to the student screen S2. The recording management unit 54 again records the voice data from the terminal 70 , assigns a recording ID, and registers it in the recording management DB 5003 . Next, the analysis/result management unit 55 compares the voices. The analysis/result management unit 55 searches the sample recording ID in the result management DB 5004 with the recording ID of step T4, identifies the record with the largest retry number from the hit records, and increments the retry number of this record by one. Enlarge. Then, the sample recording ID (recording ID transmitted in T4), answer recording ID (recording ID of the voice data of the terminal 70 newly numbered), meeting ID, problem number (problem of the record hit by the recording ID number), the retry number increased by one, the model analysis result, the answer recognition result, and the scoring result are registered in the result management DB 5004 . The transmitter/receiver 51 transmits the evaluation result to the terminals 10 and 70 . As a result, the teacher screen T3 of FIG. 13 is displayed on the terminal 10 again. Further, the terminal 70 displays the student screen S3 of FIG. 16 again.

An end button 313 is a button for ending the presentation of the current model recorded file. When the end button 313 is pressed, the teacher's terminal 10 returns to the teacher's screen T1, and the student's terminal 10 returns to the student's screen S1. Therefore, it becomes a state of a normal video conference. In this case also, the student does not need to perform an operation to return to the student screen S1.

The student image display column 314 is the same as in FIG. The phonetic symbol column 316a shows the phonetic symbols converted from the teacher's voice data, and the phonetic symbol column 316b shows the phonetic symbols converted from the student's voice data. A detailed evaluation column 317 is the result of evaluation by the management system 50 by comparing the model speech data and the student's speech data. In FIG. 13, each word of the voice data is converted into a phonetic symbol, and "o" and "x" are attached to each phonetic symbol. "O" indicates that the phonetic symbols of the model voice data and the student's voice data match, and "X" indicates that the model voice data and the student's voice data do not match the phonetic symbols. Also, in the score column 318, scores are given according to the degree of matching. The analysis/result management unit 55 treats one question as one sentence, and assigns a score to the percentage of words included in one sentence whose phonetic symbols match. Scoring may be done at the character level, and scoring methods are not limited to these.

The answer voice reproduction button 315 is a button for outputting the voice data of the student. When the answer voice reproduction button 315 is pressed, the recording ID of the recording file is transmitted to the management system 50 . The voice data of the student may be transmitted by the management system 50 only to the terminal 10 of the teacher, may be transmitted only to the terminal 70 and the terminal 10 that uttered this voice data, or may be transmitted to all the terminals 10 and 70. You may Note that the recording ID is associated with the communication ID in the recording management DB 5003, so the management system 50 can transmit only to the terminal 70 having this communication ID.

As shown in FIG. 13, the evaluation results of each student are displayed in a list on the teacher screen T3, so even if each student speaks at once, the speech data of each student can be evaluated individually. Also, since the teacher can arbitrarily output the voice data of each student, the teacher can evaluate the voice data of the students himself.

FIG. 14 shows an example of the student screen S1. The student screen S1 is a screen displayed by the terminal 10 at the beginning of the meeting. The student screen S1 has a teacher image display field 321 and a student image display field 322 . The teacher image display column 321 and the student image display column 322 may be the same as those of the teacher screen T1. A message 323 of "Listen in class" is displayed on the student screen S1. This means listening to the content of the lesson, and the student screen S1 is displayed while the teacher's voice is being output. On the student screen S1, the teacher converses with the students, and the teacher gives instructions to the students. It is also possible for students to ask questions to the teacher.

FIG. 15 shows an example of the student screen S2. The student screen S2 is a screen for each student to imitate voice data of a model and utter a voice. The student screen S 2 has a teacher image display field 321 , a student image display field 322 and an end button 325 . The teacher image display column 321 and the student image display column 322 may be the same as those of the teacher screen T1. A message 324 of "Repeat!!" is displayed on the student screen S2. This means that it does not imitate the model voice data. When the student screen S2 is displayed, sample voice data is output from the speaker of the terminal 70, and the management system 50 starts recording the voice data at the timing when the communication ID and TYPE are transmitted. The end button 325 is a button for the student who has finished speaking to notify the management system 50 of the end of recording.

FIG. 16 shows an example of the student screen S3. The student screen S3 is a screen on which the student's evaluation result is displayed. The student screen S3 has a correct answer voice playback button 331, a teacher image display column 332, phonetic symbol columns 333a and 333b, a detailed evaluation column 334, a score column 336, and an answer voice playback button 335. These functions may be the same as those of the teacher screen 3.

On the student screen S3, evaluation results are displayed for each voice data uttered after retries, so that the student can grasp what is wrong with each voice data, what has been improved, and what is likely to be mistaken. In addition, model voice data can be arbitrarily output. You can also output and check your own voice data.

<Processing procedure when the teacher speaks instead of recorded voice data>
In FIG. 10, the flow of processing in which a student vocalizes using recorded voice data as a model has been described, but with reference to FIG. FIG. 17 is an example of a sequence diagram showing a procedure for the management system 50 to evaluate the student's voice data. 17 mainly explains the difference from FIG.

First, steps T1-T2, steps M1-M4, and steps S1-S3 may be the same as in FIG.

T3: The teacher presses the start button 305 on the teacher screen T1 in order to model the voice data uttered in real time.

T4: In order to notify the TYPE and communication ID of the terminal 10, the transmitting/receiving section 11 of the terminal 10 transmits the communication ID and TYPE to the management system 50.

M4: The transmission/reception unit 51 of the management system 50 receives the TYPE and the communication ID, and the recording management unit 54 starts recording the teacher's voice data.

T5: When the teacher finishes speaking, he presses the end button 306 on the teacher screen T2b (FIG. 12(b)). The operation input receiving unit 12 receives pressing.

T6: The transmitting/receiving section 11 of the teacher's terminal 10 transmits the communication ID to the management system 50 to end the recording.

M5: The transmission/reception unit 51 of the management system 50 receives the communication ID, whereby the recording management unit 54 terminates the recording and numbers the recording ID.

M6: The recording management unit 54 of the management system 50 registers the recording ID, TYPE, communication ID, and recording file in the recording management DB 5003 .

M7: The transmission/reception unit 51 of the management system 50 transmits the voice data registered in the recording management DB 5003 to the terminal 70 . As a result, the terminal 70 displays the student screen S2, and the subsequent processing is the same as in FIG.

As described above, the communication system 1 of the present embodiment enables evaluation and detailed instruction by comparing the voice data of many students even if the teacher utters the voice data of the model.

<Comparison and evaluation of voice data>
In this embodiment, the teacher's voice data and the student's voice data are compared using phonetic symbols to determine how much the pronunciations match each word. Also, the ratio of the number of words with the same phonetic symbols in one sentence was used as the score. A speech recognition technology can be used as a technology for converting voice data into phonetic symbols in this way.

FIG. 18 is an example of a diagram schematically explaining speech recognition. FIG. 18 shows a neural network 401 that converts speech into phonetic symbols by machine learning. In particular, a neural network 401 with a deep hierarchy is called deep learning. Also, it is known that an RNN (Reccurent Neural Network) is suitable for classifying time-series data such as voice data.

As an example, voice data sliced every 20 milliseconds is converted into MFCC (Mel-frequency cepstral coefficients) and input to each node of the input layer 402 . Each phonetic symbol corresponds to each node of the output layer 403 .

In the learning phase, speech data converted into MFCC is input to the input layer 402, and in the output layer 403, nodes corresponding to phonetic symbols of speech contained in the speech data are assigned "1" (other nodes are "0"). Then, the weights between the nodes of the input layer 402, the nodes of each layer of the neural network 401, and the nodes of the output layer 403 are learned by error backpropagation.

In the recognition phase, the node of the output layer 403 closer to the phonetic symbols included in the input speech data outputs a larger value (a value closer to 1). The analysis/result management unit 55 extracts the phonetic symbol corresponding to the node outputting a value equal to or greater than the threshold. If there are multiple phonetic symbols, refer to the dictionary data that indicates the probability of connection between the phonetic symbols, calculate the probability of easy connection for each combination of phonetic symbols before and after, and select the combination with the highest probability. Specify phonetic symbols so that

There are various speech recognition methods, and the description of FIG. 18 is only an example. For example, Hidden Markov Models may be used for speech recognition. Also, in this embodiment, there is no problem even if other methods are adopted.

Further, although the voice data is converted into phonetic symbols in this embodiment, the voice data may be converted into words (alphabet). A known method can be adopted for converting voice data into words (alphabet). Since the phonetic symbols of words are listed in general English dictionaries, the phonetic symbols are uniquely determined when a word is determined. Therefore, voice data can be converted into phonetic symbols. However, if an excellent discriminator is generated that recognizes a badly pronounced voice as a correct word, there is a risk that even if the student's pronunciation is bad, it will be judged to be a correct phonetic symbol.

In the evaluation of sound data of musical instruments, for example, there is a method of Fourier transforming the sound data of a teacher and a student respectively, comparing and quantifying the difference in intensity for each frequency.

<Summary>
As described above, in the communication system 1 of the present embodiment, the management system 50 compares the teacher's voice with each student's voice even in a form in which a remote teacher instructs a plurality of students to vocalize. It becomes possible for one teacher to teach the vocalization of many students in parallel. It is possible to provide detailed instruction that is equivalent to or better than one-on-one classes.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is by no means limited to such examples, and various modifications can be made without departing from the scope of the present invention. and substitutions can be added.

<<Another example of system configuration>>
Although there is one management system in FIG. 2 , a plurality of management systems 50 may exist within the communication system 1 . FIG. 19 is a schematic diagram of the communication system 1 according to this embodiment. In FIG. 19, multiple management systems 50 are connected to the network. As described above, a plurality of management systems 50 may be included in the communication system 1, and any management system 50 may be provided with the function.

Further, the system configuration in which the terminal 10 and the management system 50 are connected, which is described in this embodiment, is an example, and it goes without saying that there are various system configuration examples depending on the application and purpose.

Moreover, as shown in FIG. 20, the management system 50 and the relay device 30 may exist as a single server device 60 . The server device 60 of FIG. 20 has both the function of the management system 50 and the function of the relay device 30 . Also, a plurality of server devices 60 may exist.

<<User identification by email address>>
Although the terminals 10 and 70 are identified by the communication ID in this embodiment, a user ID may be used as identification information for identifying the terminal 10 . Also, an e-mail address may be used as this user ID. In this case, the users of the terminals 10 and 70 log in to the management system 50 with their e-mail addresses.

FIG. 21 is an example of a sequence diagram showing processing for the terminals 10 and 70 to log in to the management system 50. As shown in FIG. 21 mainly describes the difference from FIG. As shown in FIG. 21, in steps S3 and S13-2, not the communication ID but the mail address is sent. The management system 50 authenticates users by email addresses and passwords.

Similarly, in the procedure for evaluating the student's voice data explained with reference to FIG. 10, in steps S4 and S7, not the communication ID but the e-mail address is sent. Similarly, in FIG. 17, mail addresses are used instead of communication IDs.

＜＜Video conferencing using a general-purpose browser application instead of an application＞＞
In the above embodiment, the communication applications A1 and A7 operate on the terminals 10 and 70, but similar processing can also be realized by a web application. A web application operates through cooperation between, for example, a JavaScript (registered trademark) program operating on a web browser and a program on the web server side, and the user uses it on the browser.

FIG. 22 is a diagram showing a configuration example of a web application in terminals 10 and 70. As shown in FIG. The terminal 10 downloads a teacher program 1030 (HTML5+JAVASCRIPT+CSS, etc.) from the management system 50 and executes it on a browser application 1040 . The terminal 70 downloads a student program 7030 (HTML5+JAVASCRIPT+CSS etc.) from the management system 50 and executes it on a browser application 7040 . In this case, for example, the communication ID is used to determine whether the student is a teacher or a student.

Also in FIG. 22, the user ID may be used as identification information for identifying the terminal 10 instead of the communication ID. A mail address may be used as this user ID.

The terminals 10 and 70 can use the services provided by the management system 50 by transmitting and receiving data to and from the management system 50 using protocols such as HTTP or HTTPS. In such a form of use, it is not necessary to download the communication applications A1 and A7 to the terminals 10 and 70 in advance.

<Others>
For example, in the present embodiment, participation in a conference room using a teleconference terminal (communication terminal) has been described as an example, but various devices having communication terminal functions can also be applied. For example, it can be suitably applied to an electronic blackboard having a function of a communication terminal.

In addition to electronic blackboards having a large touch panel, there are also electronic blackboards in which a projector projects an image and the position of an electronic pen is detected by sound waves or the like and a camera.

Further, the configuration examples of FIG. 6 and the like are divided according to main functions in order to facilitate understanding of the processing by the management system 50, the terminals 10 and 70, and the relay device 30. FIG. The present invention is not limited by the division method or name of the unit of processing. The processing of the management system 50, the terminals 10 and 70, and the relay device 30 can also be divided into more processing units according to the content of the processing. Also, one processing unit can be divided to include more processing.

Each function of the embodiments described above may be implemented by one or more processing circuits. Here, the "processing circuit" in this specification means a processor programmed by software to perform each function, such as a processor implemented by an electronic circuit, or a processor designed to perform each function described above. devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array), SOC (System on a chip), GPU (Graphics Processing Unit) and conventional circuit modules.

In the present embodiment, the teacher and the student conduct the class while conversing with each other, but the student downloads and outputs the recorded file by himself, and transmits the voice data uttered based on this as a model to the management system 50. You may The management system 50 evaluates the student's voice data, and the teacher downloads it at an arbitrary timing. This saves each other time and allows students to complete their homework.

Also, not only audio data but also image data may be used for evaluation. For example, machine learning can be used to convert mouth movements into phonetic symbols for similar evaluation.

In addition, the transmission/reception unit 51 is an example of communication means or a second communication means, the analysis/result management unit 55 is an example of a generation means, and the sample voice data is an example of the first sound data. The uttered voice data is an example of the second sound data, the teacher is an example of the first user, the student is an example of the second user, the terminal 10 is an example of the first communication terminal, The terminal 70 is an example of a second communication terminal, the transmitting/receiving section 11 is an example of first communication means, and the output control section 14 is an example of display control means.

1 communication system 10, 70 terminal 30 relay device 50 management system

Japanese translation of PCT publication No. 2009-511964

Claims (11)

A server capable of communicating with a plurality of communication terminals via a network,
transmitting first sound data to each of a plurality of second communication terminals according to an instruction from a first communication terminal operated by a first user; a communication means for receiving each of the plurality of second sound data obtained by
generating means for generating an evaluation for each of the plurality of second sound data based on the first sound data and the second sound data acquired from each of the plurality of second communication terminals; death,
The communication means transmits to the first communication terminal the evaluation obtained by the generation means , identification of the plurality of second sound data respectively emitted by the second users of the plurality of second communication terminals. A server that transmits information or each of the second sound data, and screen information of a first screen having buttons for individually accepting outputs of a plurality of the second sound data . The communication means transmits identification information of the first sound data or the first sound data to the first communication terminal,
2. The server according to claim 1 , wherein said first screen further has a button for accepting output of said first sound data. the first screen has a retry button for the plurality of second communication terminals to output the first sound data again;
When the communication means receives that the retry button has been pressed,
The communication means transmits the first sound data used by the generation means to generate the evaluation to the plurality of second communication terminals, and the second users of the plurality of second communication terminals receiving a plurality of second sound data emitted respectively;
The generating means compares the first sound data with the second sound data acquired from each of the plurality of second communication terminals, and evaluates each of the second sound data again, generate and
3. The server according to claim 2 , wherein said communication means transmits the evaluation obtained by said generating means to said first communication terminal. the first sound data is voice data in a predetermined language;
The generating means converts the first sound data and a plurality of second sound data respectively uttered by the second users of the plurality of second communication terminals into phonetic symbols,
Each of the plurality of second sound data is evaluated by comparing phonetic symbols converted from the first sound data with phonetic symbols converted from the second sound data. 4. The server according to any one of items 1 to 3 . The communication means transmits screen information of a second screen including a list of the first sound data recorded in advance to the first communication terminal,
When receiving the identification information of the first sound data selected from the list displayed on the second screen,
5. The server according to any one of claims 1 to 4 , wherein said first sound data specified by said identification information is transmitted to each of said plurality of second communication terminals. When the button for accepting the input is pressed on the second screen including the button for accepting the input of the first sound data displayed on the first communication terminal,
The communication means transmits the first sound data input by the first user to the first communication terminal and received from the first communication terminal to each of the plurality of second communication terminals. 6. A server according to claim 5 . The communication means, to the second communication terminal,
Said first sound data identified by identification information of said first sound data received from said first communication terminal of said first user or said first sound data received from said first communication terminal When,
screen information of a third screen indicating that the first user's voice is being output while the first sound data is being output;
7. The server according to claim 6 , further transmitting screen information of a fourth screen for accepting input of said second sound data displayed when output of said first sound data is completed. . The communication means transmits the second sound data so that the second communication terminal that transmitted the second sound data displays an evaluation of the plurality of second sound data obtained by the generation means. individually transmitted to the second communication terminal,
Furthermore, the identification information of the first sound data or the first sound data, the identification information of the second sound data or the second sound data, and the second sound data that transmitted the second sound data Send to 2 communication terminals individually,
that the second communication terminal has received the evaluation, the identification information of the first sound data or the first sound data, and the identification information of the second sound data or the second sound data A fifth screen including the evaluation, a button for receiving the output of the first sound data, and a button for receiving the output of the second sound data, which is displayed as a trigger, and transitions from the fourth screen 8. The server according to claim 7 , wherein screen information of said fifth screen to be displayed is transmitted to said second communication terminal. A sound data evaluation method performed by a server capable of communicating with a plurality of communication terminals via a network,
communication means for transmitting first sound data to a plurality of second communication terminals according to an instruction from a first communication terminal operated by a first user; a step of respectively receiving a plurality of second sound data respectively emitted by a user;
generating means for generating an evaluation for each of the plurality of second sound data based on the first sound data and the second sound data obtained from each of the plurality of second communication terminals;
The communication means transmits to the first communication terminal the evaluation obtained by the generation means , identification of the plurality of second sound data respectively emitted by the second users of the plurality of second communication terminals. a step of transmitting information or each of the second sound data and screen information of a first screen having buttons for individually accepting outputs of a plurality of the second sound data;
A sound data evaluation method comprising: A server that can communicate with multiple communication terminals via a network,
transmitting first sound data to each of a plurality of second communication terminals according to an instruction from a first communication terminal operated by a first user; a communication means for receiving each of the plurality of second sound data obtained by
Functioning as generating means for generating an evaluation for each of the plurality of second sound data based on the first sound data and the second sound data acquired from each of the plurality of second communication terminals,
The communication means transmits to the first communication terminal the evaluation obtained by the generation means , identification of the plurality of second sound data respectively emitted by the second users of the plurality of second communication terminals. A program for transmitting information or each of the second sound data, and screen information of a first screen having buttons for individually accepting outputs of a plurality of the second sound data . A communication system having a first communication terminal and a server capable of communicating via a network,
The first communication terminal is
a first communication means for transmitting to the server a transmission request for transmitting the first sound data to each of the plurality of second communication terminals;
The server is
a plurality of second sounds emitted by second users of the plurality of second communication terminals by transmitting the first sound data to the plurality of second communication terminals in response to the transmission request; a second communication means for respectively receiving data;
generating means for generating an evaluation for each of the plurality of second sound data based on the first sound data and the second sound data acquired from each of the plurality of second communication terminals; ,
The second communication means transmits to the first communication terminal the evaluation obtained by the generating means, the plurality of second sounds respectively emitted by the second users of the plurality of second communication terminals. transmitting identification information of data or each of the second sound data, and screen information of a first screen having buttons for individually accepting outputs of a plurality of the second sound data;
A communication system, wherein the first communication terminal has display control means for displaying a plurality of the evaluations received from the server by the first communication means on a display device.

Images