KR102072237B1 - Method and apparatus for providing automatic speech translation service in face-to-face situation - Google Patents

Abstract

The present invention relates to an automatic interpretation service, the automatic interpretation service providing method according to the present invention, the automatic interpretation device of the user based on the strength of the signal for the wireless communication device for interpreting the peripheral; Exchanging information for an automatic interpretation with the retrieved automatic interpretation equipment; Generating a list of candidate devices for automatic interpretation using the automatic interpretation information and the signal strength; And connecting the candidate device having the largest change in signal strength among the devices on the generated list.

Description

METHOD AND APPARATUS FOR PROVIDING AUTOMATIC SPEECH TRANSLATION SERVICE IN FACE-TO-FACE SITUATION}

The present invention relates to an automatic interpretation, and more particularly, to a method for establishing or terminating a connection between devices supporting an automatic interpretation service in a face-to-face automatic interpretation situation.

When people meet different languages, communication barriers are caused by language barriers. They try to communicate by expressing limited expression by facial expressions or gestures or by using a text translation service of an Internet service provider. However, there is a limit to delivering information as much as that of a native language.

Automatic interpretation technology was developed to communicate with people of different languages.

The combination of smartphones and wearable devices provides a more convenient environment for using automatic interpretation. When using an automatic interpretation service using only a smartphone, a voice recognition is possible, that is, one hand cannot be free because the smartphone must be held around the face, but the use of the wearable device has the advantage of freeing both hands. Smartphones and wearable devices can be connected with protocols such as Bluetooth Low Energy (BLE) to control the connection and termination of automatic interpretation services.

Until now, in providing such an automatic interpretation service, a method of connecting by using personal information such as the other party's telephone number, subscriber identification number, ID (ID) of the interpretation service (first generation) or gestures, bump to bump when the other party is near There is a connection method (2nd generation) in which both terminals are connected to each other by contacting each other (Bump-to-bump), infrared pointing, etc., but both have to check the interpretation intention of the user and the other party and establish the connection manually according to the prescribed rules. There was an inconvenience.

In order to proceed with the manual connection, it is recommended that an interpreter doctor be required to start an interpreter service between users, or that an interpreter doctor must be contacted with each other after running the app. In order to do this, there is a problem of knowing at least the native language of the other party representing the interpreter. In addition, when ordering or serving in a restaurant, a taxi driver driving, if you do not use both hands freely, there is also a problem that can not proceed smoothly interpreting service by the conventional manual connection method.

The present invention has been made in the technical background as described above, and provides an automatic interpreter connection method that can automatically start an interpreting session without the cumbersome interpreter confirmation procedure between users with the automatic interpretation interface most similar to the human communication process For that purpose.

The object of the present invention is not limited to the above-mentioned object, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a method for providing an automatic interpretation service, the method comprising: searching, by an automatic interpretation device of a user, based on the strength of a signal for wireless communication; Exchanging information for an automatic interpretation with the retrieved automatic interpretation equipment; Generating a list of candidate devices for automatic interpretation using the automatic interpretation information and the signal strength; And connecting the candidate device having the largest change in signal strength among the devices on the generated list.

According to another aspect of the present invention, an automatic interpreter includes at least one processor, wherein the processor is configured to receive a signal for determining proximity information of the automatic interpreter around the automatic interpreter. Communication unit for transmitting and receiving automatic interpretation information with the automatic interpretation device; A controller configured to generate a list of candidate devices for automatic interpretation based on the received signal strength and the automatic interpretation information, and to start an automatic interpretation session by connecting to a device having the largest change in signal strength among the devices in the list; And an arithmetic unit configured to perform an automatic interpretation with the connected automatic interpreter.

According to the present invention, there is an effect that the user can proceed with the automatic interpretation connection and termination as if the user proceeds the normal conversation without the user's manual intervention, such as setting or attempting the automatic interpretation connection.

Also, as wearable devices such as hands-free earphones, smart watches, or glasses are becoming more common, users can automatically start interpretation sessions based on mutual proximity information with the other user after the wearable device checks the user's interpretation conditions without the operation of the smartphone. Therefore, it has the effect of enabling natural interpretation, and the termination of the interpretation service terminates the interpretation session when it is detected that the user is separated from each other by mutual proximity information. It is effective.

The wearable device-based automatic interpretation interface with such a user interface can use both hands freely, thereby maximizing the interpretation effect in various working environments where shopping, tourism, or both hands must be used.

1 is a flow chart of the interpretation mode progress method according to an embodiment of the present invention.
2 is a flowchart illustrating a method of starting an interpretation session according to an embodiment of the present invention.
3 is a view showing a method for selecting an automatic interpretation target according to an embodiment of the present invention.
4 is a structural diagram of a computer system according to another embodiment of the present invention.
5 is a flowchart illustrating a method for ending automatic interpretation service according to an embodiment of the present invention.
6 is a flowchart illustrating a method of connecting and terminating an automatic interpretation service according to an embodiment of the present invention.
7 is a structural diagram of an automatic interpreter according to another embodiment of the present invention.

Advantages and features of the present invention, and methods for achieving them will be apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various forms, and only the present embodiments make the disclosure of the present invention complete, and those of ordinary skill in the art to which the present invention belongs. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims. Meanwhile, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and / or “comprising” refers to a component, step, operation and / or device that is present in one or more other components, steps, operations and / or elements. Or does not exclude additions.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Figure 1 shows a flow chart of the interpretation session according to an embodiment of the present invention.

Interpretation mode setting step (S110) is a step in which the user sets the wearable device-based automatic interpretation device, such as his smartphone, can be set through the setting menu of the automatic interpretation device itself or the application of the smartphone.

In the interpretation mode setting step (S110), the automatic interpretation mode option and its own interpretation information and optional information may be set.

The Automatic Interpretation Mode option is used to set whether the user's automatic interpreter will automatically start and end the interpretation session. If the automatic interpretation mode is set to OFF, the automatic interpretation equipment starts and ends the interpretation session by the user's manual operation in the same way as before.

Interpretation information is information necessary to automatically search for an interpreter when the automatic interpretation mode is turned on. The first language required for interpretation and other languages that can communicate can be set. . In general, the first language may be set as the native language of the first language, and in case the other device does not support the automatic interpretation of the first language, a language capable of communicating other than the native language is used as the second and third language. It is to be set. The user only needs to set such interpretation information only once during the first execution of the automatic interpretation device or smartphone application, thereby avoiding inconvenience in automatic connection.

The selection information may include a gender, a name, a signal strength setting value, and the like of the user.

Signal strength can be set in up / down / down or 10 steps. If the signal level is set to phase, the device signal strength of the other party to be connected should be more than a certain intensity, so that it should be located not far from the user, and as a result, there will be fewer automatic interpretation targets. If you set the signal strength as low, as many auto-interpreters are searched as possible, there are too many targets leading to auto-interpretation, which can confuse the auto-interpretation conversation.

Interpretation session start step (S120) performs a process of searching for an interpreter target and automatically connecting to start the interpretation session when the user's automatic interpretation device is set to the automatic interpretation mode on. When connected with the other interpreter, the user can output a notification sound such as 'Ding dong' to indicate that the interpreter session has started, and the LED lights to indicate the status of the wearable device. You can inform other users of the current status of the automatic translator.

When the automatic interpretation session is started (S120) and the conversation using the automatic interpretation with the other users connected (S130), if the conditions for the termination of the interpretation is satisfied, the interpretation session ends automatically (S140).

Figure 2 is a flow chart illustrating in more detail the step of establishing a connection between the automatic interpretation equipment for the start of the interpretation session (S120) of the automatic interpretation.

The user's automatic interpreter searches for the surrounding automatic interpreter using BLE (Bluetooth Low Energy) signal (S210). As a signal for searching for a nearby device, various signals such as a Wi-Fi signal or a mobile communication signal such as 3G and LTE may be used as well as a BLE signal.

Since the signal strength of each BLE signal or Wi-Fi signal is different for each device, it is impossible to determine the exact distance between the devices due to the signal strength. You can also determine the more accurate distance between devices.

The searched devices exchange device IDs with each other (S220). ID information of the device refers to information for identifying a device such as a media access control (MAC) address unique to the automatic interpreter.

Next, the interpreted information including the first language information set in advance with the searched devices is exchanged (S230) to determine whether the language set in the counterpart device is a language capable of automatic interpretation service. In addition, if the device is set to the same language as the language set by the user is not included in the automatic interpretation candidate list because no interpretation service is required.

When the devices capable of automatic interpretation service are identified among the peripheral devices, the signal strengths of the peripheral devices are searched (S240), and the interpreter list is generated in order of signal strength (S250).

It searches for the adjacent devices in the created list and measures mutual proximity information to determine the other device to be connected.It is not simply connected to the device with the strongest signal or above a certain threshold in order of signal strength. You will decide which device to connect to.

Figure 3 (a) is a view showing a method for determining the automatic interpretation target in consideration of the user's movement path in the step S260 measuring the interpreter period.

라 하면 더 가까운 거리에 있는 A위치와의 신호강도 차이가 더 클 것이므로

>

의 관계이지만, 사용자가 X에서 X'로 이동하는 경우에 신호강도 차이값 자체가 아닌 신호강도 차이값의 변화량, 즉

와

를 비교하면

의 변화가 거의 없기 때문에

의 관계가 된다. 따라서 사용자의 기기는 A의 기기와 연결하지 않고 B의 기기와 연결하게 된다.Before the move, the user is in position X, and the difference in signal strength between the user's device and the other device in A and B

In this case, the difference in signal strength with the A-position closer to each other will be greater.

>

However, when the user moves from X to X ', the amount of change in the signal strength difference, not the signal strength difference itself,

Wow

If you compare

Because there is almost no change

Becomes a relationship. Therefore, the user's device does not connect with the A device but with the B device.

(n=1,2, … ,N) 중에서 변화량이 최대인 기기와 연결하는 것인데, 이는

이 가장 큰 기기와 가장 빠르게 가까워지고 있다는 의미이고 이는 사용자가 그 기기로 다가가고 있다고 볼 수 있기 때문이다. 따라서 이렇게

을 비교하는 방법은 N이 2이상인 경우에 의미가 있다.This connection method

to connect with the largest change in (n = 1,2,…, N),

This means you're getting closest to the biggest device, because you can see it coming to the device. So do this

The method of comparing is meaningful when N is 2 or more.

뿐만 아니라

의 변화량인

도 사용할 수 있고, 그런 경우 더 민감한 변화를 측정할 수 있기 때문에 더욱 정확한 연결이 가능하다.In addition, in order to increase the sensitivity to the user's movement,

As well as

Change of

Can be used, and more sensitive changes can be measured, allowing for more accurate connections.

Figure 3 (b) shows a method for determining the automatic interpretation target when there are several candidates for automatic interpretation targets in a similar position.

는 상대적으로 다른 대상들에 비해 작아서 연결후보에서 제외되지만,

,

는 비슷한 값을 가진다.In contrast to (a) of FIG. 3, when the user moves from X to X 'when three users of b1, b2, and b3 are gathered at the position of B,

Is relatively small compared to other targets and is therefore excluded from the consolidation candidate,

,

Has a similar value.

이 가장 큰 사용자와 연결할 수도 있고, b1, b2, b3의 세 사용자가 비슷한 장소에 모여있는 상황은 세 사용자가 동시에 대화를 진행하고 있는 것일 수 있으니 세 사용자와 동시에 연결을 시도할 수도 있다. 따라서

,

또는

,

가 미리 설정해 둔 임계값 이하로 차이가 난다면 그 중 어느 한 사용자만을 선택하여 자동통역 연결을 시도하는 것이 아니라 세 사용자 모두에게 연결을 시도함으로써 이미 진행중인 대화에 참여하는 자연스러운 자동통역 연결이 가능하다.In this case, like the old method

You can connect with this largest user, and the situation where three users b1, b2, and b3 are gathered in a similar place may mean that the three users are talking at the same time. therefore

,

or

,

If the difference is less than the threshold set in advance, it is possible to connect to all three users instead of selecting only one user for automatic interpretation.

3 (c) is a diagram illustrating a signal range for maintaining an automatic interpretation session after the user starts an automatic interpretation. When the user X enters into the signal range 310 of the counterpart A who is the target of interpretation and the automatic interpretation session is started, the user X is allowed to move in the automatic interpretation session maintenance range 320 to set an area for maintaining a natural conversation.

The user may proceed with the conversation closer or farther away from the other person in the automatic interpreter conversation.If the automatic interpreter ends automatically, the user may interrupt the natural conversation by repeating the connection and termination of the conversation, and the power consumption of the automatic interpreter is severe. This is to prevent the conversation from automatically ending unlike the user's intention.

또는 (-)

를 허용함으로써 사용자는 자연스럽게 상대방과 접근하거나 멀어지면서도 자동통역 세션을 유지할 수 있다.To this end, up to a preset threshold (-)

or (-)

By allowing the user, the user can naturally maintain the automatic interpretation session while approaching or away from the other party.

After the step of measuring mutual proximity information for the interpreter period for the automatic interpretation in the above-described manner (S260), it is determined whether to connect the interpreting session (S270), if there is a device to be connected (S280) by the automatic interpretation dialogue If there is no device to be connected, the process returns to the interpreter search step S210 to search for another interpreter.

When the interpreter session starts, the user's automatic interpreter will use the automatic interpreter system.The server type is used to communicate with the server separately from the interpreter server, and the interpreter is provided with the interpreter system itself. Two types of system can be used.

The automatic interpretation system recognizes the user's voice, converts it into a character, translates the converted character into the language of the counterpart, and then synthesizes it into a voice corresponding to the counterpart's language.

In the automatic interpreter which is being interpreted automatically, the volume of synthesized sound is adjusted according to the volume of the voice of the user or the counterpart. In addition, when a group of people conducts a conversation in a group, the voice of the far person far away from the user is heard smaller and the voice of the far far end is adjusted to be larger than that. Do it.

After the automatic interpreter period connection is approved and the conversation starts, the conversation partner may be moved to another place or terminated manually when the connection is made with an unwanted person. It is also possible to end the interpreting session automatically.

5 is a flowchart showing the end of an interpreting session.

Since the user device connected to the interpreter is continuously sending and receiving interpretation results, the user may check whether the interpreter is continued by searching for the connected device at a predetermined time interval (S510).

If the interpretation continues, it is checked whether there is a manual termination request from the user (S540), and if there is no manual termination request, the automatic interpretation continues and the connected devices are searched (S510). The manual termination request may be performed by the user by operating the smartphone or through the wearable device. For example, the manual termination request may be previously determined by briefly touching the touch sensor of the wearable device twice in a row.

If the interpreter is in a stopped state, the interpreter is not immediately terminated, and the signal strength is determined again (S530). If you stop talking for a while, such as checking your phone during the conversation, you can start the automatic interpreter conversation again.

If there is no decrease in signal strength, it is determined whether there is a manual termination request again (S540), and it is determined whether to automatically terminate the automatic interpretation.

If the signal strength decreases because the other party to the conversation away from the user can end the automatic interpretation when the signal strength falls below the preset threshold (S550).

If the end of the interpretation is approved automatically or manually, the automatic interpretation device resets the interpretation language (S560) and informs the user that the interpretation session is finished through the sound or LED of the automatic interpretation device (S570).

Figure 6 shows a flow diagram of the connection, interpretation, termination process when two people are using the smart phone and the wearable device for automatic interpretation.

When the automatic interpretation mode is set to ON in the user's smart phone (first and second wireless communication devices), the LED of the wearable device is displayed in blue and indicates that the automatic interpretation standby state (S110).

Using the BLE, the neighboring target wireless communication device is searched for (S210), and the ID information of the device and the interpreter information for interpretation are exchanged between the two devices under search (S220, S230).

When the signal strength is searched (S240) to generate a list of interpretation candidates (S250) and the user moves to measure mutual proximity information using the amount of change in the signal strength (S260), the measurement result is used to determine which user is approaching. And, after approving the connection with the device that is approaching (S280) starts the automatic interpretation conversation.

While generating the candidate list for interpretation (S250) and measuring mutual proximity information (S260), the wearable device blinks the LED in red to indicate that the connection is currently in progress, and after the connection is approved (S280), the connection is successful. The sound is ding dong 'and the LED turns green to indicate that an automatic interpretation conversation is in progress.

If there is no interpretation result (S510), if it is determined that the manual termination condition or the automatic termination condition is satisfied, the interpretation language is reset (S560). Quit.

When the connection is terminated, the wearable device announces the end of the automatic interpretation conversation with a 'beep' sound, and the blue LED indicates that it is in standby mode, waiting for another automatic interpretation session.

Based on the human behavior as described above, the automatic interpretation interface can be extended even when the N person proceeds to communicate in N languages. For example, when two users start an interpreting session and are talking, a third person joins the interpreting session and becomes a group of three or more.

If two persons are in the process of interpreting conversations by starting the first interpretation session, if the third party approaches the other party in conversation with one of the participants of the first interpretation session, the two persons proceed if the above-mentioned interpretation session connection conditions are met. The first interpreter session was changed to a group interpreter session of three persons.

Such a group interpreting session may be interpreted by automatically participating in the group interpreting session by satisfying the connection conditions if at least one of the participants of the interpreting session is in a different language.

In the case of a group interpretation session, even if the automatic interpretation device does not provide interpretation for some languages among the participating users, interpretation of the language can be provided through the interpretation session of another user.

For example, if user A and user B are in an automatic interpretation session in Korean and English, French user C wants to participate in the group interpretation session, but user C does not support Korean translation but only English. However, French for user C cannot be delivered directly to user A, but it is translated into English through user B and is delivered to user A. The user's automatic interpreter translates English into English to provide user C with Korean. It can be.

In the case of a group interpretation session, a similar termination method is used as in a two-person interpretation session, but in the case of a group interpretation session, it is not checked whether the interpretation is ongoing. This is because when three or more people are in a conversation, some members of the group may listen to the conversation without proceeding. Therefore, the group interpreting session uses a method of checking the end of the interpreting session only when the user moves away from the group.

7 shows a structural diagram of an automatic interpretation device 700 according to another embodiment of the present invention.

The automatic interpreter 700 includes a communication unit 710, a controller 720, and a calculator 730.

The communication unit 710 transmits and receives a signal to exchange information with another automatic interpreter through the antenna 711.

The controller 720 generates a list of candidate devices for automatic interpretation based on the strength of the signal received by the communication unit 710 and the automatic interpretation information. As described above, the connection with the device determined to be approached by the user based on the change amount of the signal strength is not performed with the device having the strongest signal strength among the generated lists.

The calculating unit 730 performs an automatic interpretation with the connected device and the automatic interpretation result is exchanged with the counterpart device through the communication unit 710.

On the other hand, the automatic interpretation service providing method according to an embodiment of the present invention may be implemented in a computer system or recorded on a recording medium. As shown in FIG. 4, a computer system includes at least one processor 421, a memory 423, a user input device 426, a data communication bus 422, a user output device 427, It may include a reservoir 428. Each of the components described above communicates data via a data communication bus 422.

The computer system can further include a network interface 429 coupled to the network. The processor 421 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 423 and / or the storage 428.

The memory 423 and the storage 428 may include various types of volatile or nonvolatile storage media. For example, the memory 423 may include a ROM 424 and a RAM 425.

Therefore, the automatic interpretation service providing method according to an embodiment of the present invention can be implemented in a computer executable method. When the automatic interpretation service providing method according to an embodiment of the present invention is performed in a computer device, computer readable instructions may perform the recognition method according to the present invention.

On the other hand, the automatic interpretation service providing method according to the present invention can be implemented as a computer-readable code on a computer-readable recording medium. Computer-readable recording media include all kinds of recording media having data stored thereon that can be decrypted by a computer system. For example, there may be a read only memory (ROM), a random access memory (RAM), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, and the like. The computer readable recording medium can also be distributed over computer systems connected over a computer network, stored and executed as readable code in a distributed fashion.

According to the present invention described above, unlike the conventional automatic interpretation method by which the user manually intervenes, the conversation partner can be automatically set up and connected using the change in signal strength without user intervention, and the ID information and the interpretation information between the devices in advance Because of the exchange, it is possible to talk with the automatic setting of the device without setting an interpreter language, and since the start and end of the conversation can be performed naturally based on human actions, the user can proceed without any dissatisfaction.

In the above, the configuration of the present invention has been described in detail with reference to the accompanying drawings, which are merely examples, and those skilled in the art to which the present invention pertains various modifications and changes within the scope of the technical idea of the present invention. Of course this is possible. Therefore, the protection scope of the present invention should not be limited to the above-described embodiment but should be defined by the following claims.

700: automatic interpreter
710: communication unit 711: antenna
720: control unit 730: arithmetic unit

Claims (10)

Searching, by the automatic interpreter device of the user, based on the strength of the signal for wireless communication;
Exchanging information for an automatic interpretation with the retrieved automatic interpretation equipment;
Generating a list of candidate devices for automatic interpretation using the automatic interpretation information and the signal strength; And
Connecting to the candidate device having the largest change in signal strength in consideration of the movement path of the automatic interpretation device among the devices on the generated list;
The connecting step
Group communication is performed by connecting both the device having the largest change in signal strength among the devices on the generated list and the device having a difference in signal intensity change between the device having the largest change in signal strength below a predetermined threshold value. How to provide automatic interpretation service.
delete The method of claim 1, wherein the searching
Searching for the surrounding automatic interpreter using the signal strength of the Bluetooth signal, the Wi-Fi signal or the ultrasonic signal of the automatic interpreter
To provide automatic interpretation services.
The method of claim 1, wherein after the step of connecting
An automatic interpretation step of controlling the volume of the other party's voice after the automatic interpretation is started in proportion to the distance from the other party determined using the signal strength of the other party's automatic interpreter; Containing more
To provide automatic interpretation services.
The method of claim 1, wherein after the step of connecting
Determining whether the interpretation is ongoing;
Determining signal strength of the other party's automatic interpreter if the interpretation is not performed while the interpretation is continued; And
Terminating the connection by reporting that the other party's automatic interpreter moves away when the signal strength determined as a result of the signal strength determination decreases; Containing more
To provide automatic interpretation services.
In the automatic interpretation device comprising one or more processors, the processor,
A communication unit configured to receive a signal for determining proximity information of an automatic interpreter device around the automatic interpreter device, and to transmit and receive automatic interpreter information with the surrounding auto interpreter device;
Create a list of candidate devices for automatic interpretation based on the received signal strength and the automatic interpretation information, and connect with the device having the largest change in signal strength in consideration of the movement path of the automatic interpretation equipment among the devices in the list. A control unit to start the automatic interpretation session; And
Performing an automatic interpretation, including;
The control unit
An automatic interpretation device for group conversation by connecting all devices having a difference in signal intensity change with a device having the largest change in signal intensity among the devices on the generated list below a predetermined threshold value.
delete The method of claim 6, wherein the communication unit
Using the signal strength of the Bluetooth signal, the Wi-Fi signal or the ultrasonic signal to determine the proximity information
Automatic interpreter.
The method of claim 6, wherein the operation unit
Adjusting the volume of the other party's voice in proportion to the distance to the other party determined using the signal strength of the other party's automatic interpreter
Automatic interpreter
The method of claim 6, wherein the operation unit
Determining whether the interpretation is ongoing;
Determining signal strength of the other party's automatic interpretation device if the interpretation is not performed while the interpretation is continued; And
Terminating the connection by reporting that the other party's automatic interpreter is farther away when the signal strength determined as a result of the signal strength determination decreases; To terminate automatic interpretation, including;
Automatic interpreter.

Images