KR20210004803A - Electronic apparatus and controlling method thereof - Google Patents

Abstract

Disclosed is an electronic device capable of quickly and accurately controlling devices through a user voice in a multi-device environment. The electronic device of the present invention comprises: a microphone; a communication unit; a memory in which a control command determination tool based on a control command determined by a voice recognition server for processing voice recognition of a user voice received from the electronic device is stored; and a processor obtaining user intention information by recognizing the user voice, receiving state information of an external device related to user intention information from a device control server for controlling a plurality of external devices, applying the user intention information and the state information of the external device to the control command determination tool to determine the control command for controlling a control target device among a plurality of external devices, and transmitting the determined control command to the device control server.

Description

Electronic device and its control method {ELECTRONIC APPARATUS AND CONTROLLING METHOD THEREOF}

The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device for controlling an external device through a user's voice, and a method for controlling the same.

With the recent development of network communication technology and voice recognition technology, users can control operations of various electronic devices connected through a network with voice.

For example, the user can control the operation of various electronic devices in the vicinity by firing a voice command to a hub device (eg, an AI speaker) in an Internet of Things (IoT) environment or a home network environment.

At this time, conventionally, the user's voice command received through the hub device was interpreted and processed through the cloud to control the device.In this case, a delay occurs in the interpretation and processing process, resulting in a problem that the reaction speed felt by the user is lowered. I can.

The present disclosure has been conceived in light of the above-described problems, and an object of the present disclosure is to provide an electronic device capable of quickly and accurately controlling devices through a user's voice in a multi-device environment, and a control method thereof.

An electronic device according to an embodiment of the present disclosure for achieving the above object is a control command determination tool based on a control command determined by a microphone, a communication unit, and a voice recognition server for voice recognition processing of a user's voice received from the electronic device. When the stored memory and the user voice are received through the microphone, the received user voice is voice-recognized to obtain user intention information, and state information of an external device related to the obtained user intention information is received through the communication unit. Received from a device control server for controlling an external device of, and applies the obtained user intention information and the received state information of the external device to the control command determination tool to control a control target device among the plurality of external devices And a processor that determines a control command to be performed and transmits the determined control command to the device control server through the communication unit.

In addition, the control command determination tool includes a rule DB including at least one rule in which user intention information, state information of an external device, and a control command determined by the voice recognition server are matched with each other, and the processor, When a rule corresponding to the acquired user intention information and the received state information of the external device exists in the rule DB, a control command matching the corresponding rule may be determined as a control command corresponding to the user voice. have.

In addition, the processor transmits the user voice to the voice recognition server through the communication unit when a rule corresponding to the obtained user intention information and the received state information of the external device does not exist in the rule DB. When the control command determined by the voice recognition server based on the user's voice is received through the communication unit, the received control command is matched with the acquired user intention information and the received state information of the external device to create a new rule. May be generated, and the generated new rule may be updated in the rule DB.

In addition, the control command determination tool includes an artificial intelligence model learned by inputting user intention information and state information of an external device as input, and outputting a control command determined by a voice recognition server, and the processor The acquired user intention information and the received state information of the external device may be input to the learned artificial intelligence model, and a control command output from the learned artificial intelligence model may be determined as a control command corresponding to the user voice. .

In addition, when the control command is not output from the learned artificial intelligence model in which the acquired user intention information and the received state information of the external device are input, the processor transmits the user voice through the communication unit. When the control command is transmitted to the recognition server and determined by the voice recognition server based on the user voice is received through the communication unit, the acquired user intention information, the received state information of the external device, and the received control command Based on, the learned artificial intelligence model may be retrained.

In addition, the processor may specify the operation of the control target device and the control target device only with the acquired user intention information based on the received state information of the external device, without using the control command determination tool, The control command may be determined based on the acquired user intention information.

In addition, the acquired user intention information includes information on an entity, and the processor, when state information of a plurality of external devices related to the entity is received from the device control server, the acquired user intention information It is determined that the control target device cannot be specified, and the control command may be determined using the control command determination tool.

In addition, the communication unit includes an IR (Infrared) communication module, and the processor, when the control target device is a device capable of controlling by an IR method, transmits the determined control command to the control target device through the IR communication module. Can be transferred to.

Meanwhile, a method for controlling an electronic device according to an exemplary embodiment of the present disclosure includes storing a control command determination tool based on a control command determined by a voice recognition server for voice recognition processing of a user voice received from the electronic device, When a user voice is received, the step of acquiring user intention information by speech recognition processing the received user voice, and status information of an external device related to the obtained user intention information from a device control server for controlling a plurality of external devices Receiving, applying the obtained user intention information and the received state information of the external device to a control command determination tool to determine a control command for controlling a control target device among the plurality of external devices, and the determination And transmitting the generated control command to the device control server.

In addition, the control command determination tool includes a rule DB including at least one rule in which user intention information, state information of an external device, and a control command determined by the voice recognition server are matched with each other, and the control command In the determining step, when a rule corresponding to the obtained user intention information and the received state information of the external device exists in the rule DB, a control command matched to the corresponding rule is controlled corresponding to the user voice. It can be judged by order.

In addition, when a rule corresponding to the acquired user intention information and the received state information of the external device does not exist in the rule DB, transmitting the user voice to the voice recognition server and based on the user voice When a control command determined by the voice recognition server is received, a new rule is generated by matching the received control command with the obtained user intention information and the received state information of the external device, and the generated new rule is It may further include the step of updating the rule DB.

In addition, the control command determination tool includes an artificial intelligence model learned by inputting user intention information and state information of an external device and outputting a control command determined by a voice recognition server, and determining the control command. In the step of, inputting the acquired user intention information and the received state information of the external device into the learned artificial intelligence model, and a control command output from the learned artificial intelligence model, a control command corresponding to the user voice. Can be judged as.

In addition, when the control command is not output from the learned artificial intelligence model in which the acquired user intention information and the received state information of the external device are input, transmitting the user voice to the voice recognition server, and the When a control command determined by the voice recognition server is received based on a user's voice, the learned artificial intelligence model is reconstructed based on the acquired user intention information, the received state information of the external device, and the received control command. It may further include the step of learning.

In addition, when the operation of the control target device and the control target device can be specified only with the acquired user intention information based on the received state information of the external device, the acquired user without using the control command determination tool It may further include determining the control command based on the intention information.

In addition, the acquired user intention information includes information on an entity, and when state information of a plurality of external devices related to the entity is received from the device control server, the control target device is controlled only by the acquired user intention information. It may further include determining that it cannot be specified, and determining the control command using the control command determination tool.

In addition, when the control target device is a device that can be controlled by the IR method, transmitting the determined control command to the control target device may be further included.

Meanwhile, the speech recognition server according to an embodiment of the present disclosure includes a communication unit, a memory in which a control command determination tool based on a control command determined by the speech recognition server based on a user voice received from an electronic device, and a user voice are stored in the communication unit. When received from the electronic device through the voice recognition process, the received user voice is processed to obtain user intention information, and state information of the external device related to the obtained user intention information is controlled through the communication unit. A control command for controlling a control target device among the plurality of external devices by applying the obtained user intention information and the received state information of the external device to the control command determination tool And a processor that determines and transmits the determined control command to the device control server through the communication unit.

In addition, the control command determination tool includes a rule DB including user intention information, state information of an external device, and at least one rule in which the control command determined by the voice recognition server is matched with each other, and user intention information and the state of the external device. It may include at least one of the artificial intelligence models learned by inputting information and outputting a control command determined by the voice recognition server.

According to various embodiments of the present disclosure as described above, it is possible to quickly and accurately control devices through a user's voice in a multi-device environment.

1 is a diagram illustrating a voice control system according to an embodiment of the present disclosure;
2 is a block diagram of an electronic device according to an embodiment of the present disclosure;
3 is a diagram illustrating a voice control system according to an embodiment of the present disclosure;
4 is a block diagram of an electronic device according to an embodiment of the present disclosure;
5 is a block diagram of a voice recognition server according to an embodiment of the present disclosure, and
6 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

In describing the present disclosure, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the present disclosure, a detailed description thereof will be omitted. In addition, redundant description of the same configuration will be omitted.

The suffix "unit" for the constituent elements used in the following description is given or used interchangeably in consideration of only the ease of writing the specification, and does not itself have a distinct meaning or role from each other.

Terms used in the present disclosure are used to describe embodiments, and are not intended to limit and/or limit the present disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise.

In the present disclosure, terms such as'include' or'have' are intended to designate the presence of features, numbers, steps, actions, components, parts, or a combination thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

Expressions such as "first," "second," "first," or "second," used in the present disclosure may modify various elements, regardless of order and/or importance, and one element It is used to distinguish it from other components and does not limit the components.

Some component (eg, a first component) is "(functionally or communicatively) coupled with/to)" to another component (eg, a second component) or " When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or may be connected through another component (eg, a third component). On the other hand, when it is mentioned that it is "directly connected" or "directly connected" to a certain component (eg, a first other component (eg, a second component)), between the certain component and the other component It may be understood that no other component (eg, a third component) exists in the.

Terms used in the embodiments of the present disclosure may be interpreted as meanings commonly known to those of ordinary skill in the art, unless otherwise defined.

Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram for describing a voice control system according to an embodiment of the present disclosure. Referring to FIG. 1, the voice control system 1000 may include an electronic device 100, external devices 200-1 and 200-2, a voice recognition server 400, and a device control server 500.

The electronic device 100 and the external devices 200-1 and 200-2 may configure a home network environment or an Internet of Things network environment in a premises 10 such as a home or an office.

The electronic device 100 may be implemented as various types of devices such as a smart speaker, an AI speaker, a smart TV, a smart refrigerator, a smart phone, an access point, a notebook, a desktop PC, and a tablet.

Meanwhile, since there is no limitation on the type of objects in the IoT environment, the types of external devices 200-1 and 200-2 are also not limited. In FIG. 1, TV 1 (200-1) and TV 2 (200-2) are shown, but these are only examples, and air conditioners, smart lights, fans, washing machines, microwaves, door locks, sound bars, home theaters, smartphones, There is no limitation on the type of device as long as it is a device that is connected in communication with the electronic device 100 such as a TV or refrigerator and can control an operation through the electronic device 100.

In this case, the electronic device 100 may control the operation of the external devices 200-1 and 200-2 according to the user's voice.

For example, when a user utters a user voice for controlling the operation of the external devices 200-1 and 200-2, the electronic device 100 receives the uttered user voice and transmits it to the voice recognition server 400 Accordingly, the operation of the external devices 200-1 and 200-2 may be controlled through the voice recognition server 400 and the device control server 500.

Specifically, the voice recognition server 400 may determine a control command corresponding to a user voice received from the electronic device 100 and transmit the determined control command to the device control server 500.

Here, the control command is a result of the speech recognition server 400 analyzing user intention information to be described later based on the state information of the external devices 200-1 and 200-2 received from the device control server 500 As an example, the control target device that the user wants to control through voice and information on the operation of the control target device may be included.

Specifically, when a user's voice is received from the electronic device 100, the voice recognition server 400 may perform voice recognition on the received user's voice to obtain user intention information corresponding to the user voice.

Here, the speech recognition processing performed by the speech recognition server 400 includes ASR (Auto Speech Recognition) processing for converting a user's speech into text, and an expression that a machine can understand the user's speech converted into text by the ASR processing. It includes NLU (Natural Language Understanding) processing that converts to.

Here, the user intention information is a result of the voice recognition processing of the user's voice before the voice recognition server 400 determines a control command using the state information of the external devices 200-1 and 200-2, It may include information about (entity) and an action of the entity.

The entity and the operation of the entity as a result of the speech recognition processing may vary according to the user's voice, and in some cases, the entity may not exist or the operation of the entity may not be specified. However, hereinafter, for convenience of description, assuming that the user uttered a user voice for controlling the operation of the external devices 200-1 and 200-2, the entity is the external devices 200-1 and 200-2. ), and the operation of the entity is specified as the operation of the external devices 200-1 and 200-2.

In this way, when the user intention information is obtained, the voice recognition server 400 requests the device control server 500 for status information of the entity related to the obtained user intention information, that is, the external devices 200-1 and 200-2. I can.

Here, the state information is information related to the current operating state of the external devices 200-1 and 200-2, for example, information on the power on/off state of the external devices 200-1 and 200-2 , Setting information related to functions of the external devices 200-1 and 200-2, location information of the premises 10 of the external devices 200-1 and 200-2, etc., but are not limited thereto.

When the requested state information is received from the device control server 500, the voice recognition server 400 interprets the user intention information based on the received state information of the external devices 200-1 and 200-2 to provide a control command. I can judge.

Specifically, the voice recognition server 400 determines whether the operation of the control target device and the control target device can be specified only with user intention information, based on the state information of the external devices 200-1 and 200-2, If specified, it is possible to determine a control command including information on the specified control target device and operation without further interpretation.

However, when the control target device and operation are not specified only with the user intention information, the voice recognition server 400 may determine a control command by additionally interpreting the user intention information according to its own policy of the voice recognition server 400. have. More details on this will be described later.

The voice recognition server 400 transmits the control command determined as above to the device control server 500, and the device control server 500 operates the control target device based on the control command received from the voice recognition server 400. You will be able to control it.

The device control server 500 may register devices connected to the device control server 500 through a network for each user account, and collect and manage state information of the registered devices. In the example of FIG. 1, the device control server 500 registers the electronic device 100 and the external devices 200-1 and 200-2, and the electronic device 100 and the external devices 200-1 and 200-2 ) Status information can be collected and managed.

Therefore, when the transmission of the state information of the external devices 200-1 and 200-2 is requested from the voice recognition server 400, the device control server 500 responds to the request to the external devices 200-1 and 200-2. ) Status information may be transmitted to the voice recognition server 400.

Meanwhile, the device control server 500 may store and manage a set of control signals for each of the registered devices. Here, the control signal set refers to a control code set capable of controlling various operations of the corresponding device.

Even though the control command for the same operation may have different control codes for each manufacturer or model for each device, the device control server 500 may store and manage a set of device-specific control signals to control the operation of registered devices. . In this case, the control signal set may be directly received from the device or may be acquired through a server managed by the device manufacturer.

Accordingly, when a control command is received from the voice recognition server 400, the device control server 500 checks the control signal corresponding to the control target device and the operation of the control target device in the control signal set, and controls the confirmed control signal. By transmitting to the target device, the operation of the controlled device can be controlled.

Hereinafter, an operation of the speech recognition system 100 will be described with a specific example. For example, as shown in FIG. 1, in the home 10, the electronic device 100, TV 1 (200-1), and TV 2 (200-2) constitute an IoT network, and TV 1 ( 200-1) and TV 2 200-2 are registered in the device control server 500 through a user account. In this case, the user may utter a user's voice such as "Turn on TV" to the electronic device 100 without specifying which TV it is.

The electronic device 100 transmits the user voice to the voice recognition server 400, and the voice recognition server 400 processes the received user voice ASR (Automatic Speech Recognition) to obtain an ASR processing result such as "Turn on TV". It is acquired, and user intention information such as "TV power-on" may be acquired by processing the ASR processing result by NLU (Natural Language Understanding). In this case, the entity included in the user intention information will be "TV", and the operation of the entity will be "power-on".

When the user intention information is obtained as described above, the voice recognition server 400 may request the device control server 500 for status information of an external device, that is, a TV, related to the user intention information.

At this time, since the TV of the user registered in the device control server 500 includes TV 1 (200-1) and TV 2 (200-2), the device control server 500 responds to the request of the voice recognition server 400 Accordingly, the state information of the TV 1 200-1 and the state information of the TV 2 200-2 at the time the state information transmission is requested may be transmitted to the voice recognition server 400.

Accordingly, the voice recognition server 400 may determine whether a control command is specified only with user intention information, based on the received state information of the TV 1 200-1 and TV 2 200-2.

In the above example, the entity included in the user intention information is "TV", but since status information for each of TV 1 (200-1) and TV 2 (200-2) has been received from the device control server 500, user intention information By itself, it cannot be specified which TV is TV 1 (200-1) or TV 2 (200-2).

That is, since the control target device is not specified using only the user intention information, the voice recognition server 400 may determine that the control command cannot be specified only with the user intention information, and may determine the control command according to its own policy.

Here, the self-policy is a means for the voice recognition server 400 to self-determine a control command, and may include, for example, a policy for determining a control command through a user's response to a query. It is not limited.

In the above example, the voice recognition server 400 transmits to the electronic device 100 a query asking which TV to turn on, among TV 1 (200-1) and TV 2 (200-2), and the electronic device 100 Can output the received query. Accordingly, when the user utters a response such as "TV 1", the electronic device 100 receives it and transmits it to the voice recognition server 400, and the voice recognition server 400 is based on the received voice. Accordingly, it is possible to determine the TV 1 (200-1) as a control target device.

Accordingly, the voice recognition server 400 may determine “TV 1 power-on” as a control command corresponding to “Turn on TV” and transmit the determined control command to the device control server 500. In this case, the device to be controlled will be "TV 1 (200-1)", and the operation of the device to be controlled will be "power-on".

Unlike the above example, if only one TV is registered in the user account, only status information about one registered TV will be received from the device control server 500, so the voice recognition server 400 provides user intention information. It is only possible to specify the device to be controlled. Therefore, the voice recognition server 400 is a control command "TV power-on" including a specified control target device "TV" and a specified operation "power-on" without additional interpretation such as using its own policy. Will be able to judge. This is the same even when a plurality of TVs are registered, even when a user specifies and utters a control target device such as "Turn on TV 1 (200-1)" from the beginning.

On the other hand, when a control command such as "TV 1 power-on" is received, the device control server 500 checks a control signal corresponding to "power-on" in a control signal set corresponding to TV 1 (200-1). And, the confirmed control signal can be transmitted to the TV 1 (200-1). Accordingly, TV 1 (200-1) is turned on, and a user can watch a broadcast program through TV 1 (200-1).

As described above, according to an embodiment of the present disclosure, the electronic device 100 transmits the received user's voice to the voice recognition server 400, so that the external device 200-through the control command determined by the voice recognition server 400 1, 200-2) can be controlled.

Meanwhile, according to an embodiment of the present disclosure, the electronic device 100 may control operations of the external devices 200-1 and 200-2 by directly determining a control command for a received user voice.

Specifically, the electronic device 100 generates a tool for determining a control command based on the history information that has controlled the external devices 200-1 and 200-2 through the voice recognition server 400, and generates The control command may be determined based on the control command determination tool.

Here, the control command determination tool includes 1) user intention information, a rule in which state information of an external device and a control command are matched with each other, or 2) inputs user intention information and state information of an external device, and outputs a control command. It can be a learned artificial intelligence model.

To this end, the electronic device 100 may have a voice recognition function, and may obtain user intention information by performing voice recognition on the received user voice.

Here, the voice recognition function of the electronic device 100 may include at least one of the aforementioned ASR and NLU processing functions.

For example, when the electronic device 100 includes both ASR and NLU processing functions, the electronic device 100 may obtain user intention information by performing speech recognition processing on the received user voice.

In addition, when the electronic device 100 includes only the ASR processing function, the electronic device 100 transmits the ASR processing result for the user's voice to the speech recognition server 400, and the NLU processing result from the speech recognition server 400 That is, user intention information can be obtained.

In addition, when the electronic device 100 includes only the NLU function, the electronic device 100 transmits the received user voice to the voice recognition server 400, and NLU processes the ASR processing result received from the voice recognition server 400. Thus, user intention information can be obtained.

At this time, when the electronic device 100 processes the user's voice using its own ASR processing function or NLU processing function, but the reliability of the processing result is less than a certain level, the electronic device 100 sends the voice recognition server 400 ASR processing and/or NLU processing for the user's voice may be requested, and the requested processing result may be obtained from the speech recognition server 400. Details on this will be described later.

Accordingly, the electronic device 100 requests and receives the status information of the external devices 200-1 and 200-2 related to the acquired user intention information to the device control server 500, and receives the received external device 200- The control command may be determined by applying the state information of 1, 200-2) and the acquired user intention information to the control command determination tool.

In addition, the electronic device 100 can directly transmit the determined control command to the device control server 500 to control the operation of the external devices 200-1 and 200-2 without going through the voice recognition server 400. have.

In this way, when the electronic device 100 directly processes the received user's voice to voice recognition to determine a control command or directly transmits the determined control command to the device control server 500, the voice recognition server 400 The operation of the external devices 200-1 and 200-2 can be controlled more quickly than when the control command is determined and transmitted.

On the other hand, when determining a control command using a control command determination tool, relatively simple operations such as comparing user intention information and status information of external devices 200-1 and 200-2 with rules or inputting into an artificial intelligence model. Since the control command is determined through the control command, the control command may be determined faster than when the control command is determined using the self-policy of the voice recognition server 400 described above. Accordingly, a delay that may occur when the voice recognition server 400 determines a control command through its own policy may be improved.

Hereinafter, an operation of the electronic device 100 according to an embodiment of the present disclosure will be described in detail with reference to FIG. 2.

2 is a block diagram of an electronic device according to an embodiment of the present disclosure. According to FIG. 2, the electronic device 100 includes a microphone 110, a processor 120, a communication unit 130, and a memory 140.

The microphone 110 receives sound in the form of a sound wave from the outside, converts it into an electrical signal, and provides the converted electrical signal to the processor 120. In particular, the microphone 110 may receive a user's voice and provide a corresponding electrical signal to the processor 120.

The communication unit 130 is a component for performing communication with various external servers or various devices.

In particular, the communication unit 130 communicates with the voice recognition server 400 or the device control server 500 to transmit various information or data such as user voice, user intention information, status information of external devices, control commands, etc. Can receive.

Also, the communication unit 130 may configure an IoT environment or a home network environment by performing communication with external devices 200-1 and 200-2 around the electronic device 100.

The memory 140 may store data or information in electric or magnetic form so that the processor 120 or the like can access the stored data or information.

In particular, the memory 140 includes a control command determination tool 141 that is a basis for determining a control command, a tool management module 125 for managing the control command determination tool, and performs a voice recognition function and determines a control command. A voice recognition module 123 for monitoring and a device control module 121 for monitoring state information of the external devices 200-1 and 200-2 and transmitting and receiving control commands may be stored.

The processor 120 controls the overall operation of the electronic device 100. In particular, the processor 120 loads the control command determination tool 141, the tool management module 125, the voice recognition module 123, the device control module 121, etc. stored in the memory 140 to perform functions of each module. Can be done.

Specifically, FIG. 2 shows a state in which the processor 120 loads the device control module 121, the voice recognition module 123, and the tool management module 125 stored in the memory 140 and performs the corresponding function. Is shown.

In the control command determination tool 141, the voice recognition module 123 (specifically, the NLU module 123-2) determines a control command by applying user intention information and state information of an external device, as described later. As a means for doing so, a rule generated based on history information in which the electronic device 100 previously controlled an external device, or artificial intelligence learned based on the history information in which the electronic device 100 previously controlled an external device. Can contain models.

Here, the history information includes user intention information obtained at the time of the previous control, state information of the external device at the time of the previous control, and a control command determined by the voice recognition server 400 at the time of the previous control (in particular, a control command determined by its own policy). ) May be included.

Specifically, the rule DB 141-1 is a database including a plurality of rules for determining a control command. The rule DB 141-1 may be in the form of a look-up table in which user intention information, status information of an external device, and control commands are matched, as shown in Table 1 below, but is not limited thereto.

User Intent Information External device status information Control command Rule 1 TV power-on TV 1(200-1): off
TV 2(200-2): on TV 1(200-1) power-on Rule 2 Air conditioner power-on Air conditioner 1: off Air conditioner 2: off Air conditioner 2 power-on

In each rule of the rule DB 141-1, the user intention information, the state information of the external device, and the control command are matched with each other, so the voice recognition module 123, as will be described later, It is possible to quickly determine a control command by acquiring information and state information of an external device, and checking a rule corresponding to the acquired user intention information and state information of an external device in the rule DB 141-1.

The artificial intelligence model 141-2 is learned by inputting user intention information and state information of an external device as input, and outputting a control command. Accordingly, as will be described later, the voice recognition module 123 may quickly obtain a control command by inputting the acquired user intention information and the state information of the external device into the learned artificial intelligence model.

The tool management module 125 may manage the control command determination tool 141 stored in the memory 140.

Specifically, the tool management module 125 accesses the memory 140 at an appropriate time so that the voice recognition module 123 can use the control command determination tool 141 when determining the control command. At least some of 141) may be loaded into the processor 120.

For example, when there is a request from the speech recognition module 123, the tool management module 125 processes at least some of the rules and/or the artificial intelligence model 141-2 included in the rule DB 141-1. It can be loaded at 120. However, the loading time is not limited thereto. For example, the tool management module 125 may be configured at various times before the voice recognition module 123 determines a control command, such as when a user's voice is received or when the electronic device 100 is turned on. The control command determination tool 141 may be loaded into the processor 120.

Meanwhile, the tool management module 125 may update the control command determination tool 141. This will be described later.

The device control module 121 may monitor status information of an external device through the device control server 500 and provide it to the voice recognition module 123.

Specifically, when the state information of the external device is requested from the voice recognition module 123, the device control module 121 requests the state information of the external device to the device control server 500 through the communication unit 130, and The current status information of the external device received from the control server 500 may be provided to the voice recognition module 123.

Meanwhile, the device control module 121 manages the current state information of the external device received from the device control server 500 when the state information of the external device is requested from the tool management module 125 as in an embodiment to be described later. It may also be provided as a module 125.

In addition, the device control module 121 may transmit a control command determined by the voice recognition module 123 to the device control server 500 to control the operation of the controlling device.

Specifically, when a control command is received from the voice recognition module 123, the device control module 121 may transmit the received control command to the device control server 500 through the communication unit 130. Accordingly, the device control server 500 transmits a control signal corresponding to the control command to the control target device, so that the operation of the control target device may be controlled.

The voice recognition module 123 may determine a control command corresponding to a user's voice received through the microphone 110.

Specifically, the voice recognition module 123 may perform a voice recognition function. To this end, the voice recognition module 123 may include an ASR module 123-1 and an NLU module 123-2.

The ASR module 123-1 may recognize a user voice and output the recognized user voice as text. For example, when a user voice such as "Turn on TV" is received through the microphone 110, the ASR module 123-1 recognizes this and sends a text such as "Turn on TV" to the NLU module 123-2. Can be printed as

To this end, the ASR module 123-1 may include an acoustic model and a language model. In this case, the acoustic model may include information related to speech, and the language model may include information on a combination of unit phoneme information and unit phoneme information. Accordingly, the ASR module 123-1 may convert the user's voice into text by using information related to vocalization and information on unit phoneme information.

The NLU module 123-2 may determine the meaning of the text received from the ASR module 123-1, that is, the user intention corresponding to the user's voice. For example, when a text such as "Turn on TV" is received from the ASR module 123-1, the NLU module 123-2 may analyze it and obtain user intention information such as "TV power-on". have.

To this end, the NLU module 123-2 performs tasks such as keyword matching, syntactic analysis, and semantic analysis with respect to the text received from the ASR module 123-1. To understand user intent.

In FIG. 2, it has been exemplified that the speech recognition module 123 includes both the ASR module 123-1 and the NLU module 123-2, but the embodiment is not limited thereto. That is, for example, the voice recognition module 123 may include only one of the ASR module 123-1 or the NLU module 123-2, and in this case, the voice recognition module 123 serves as the remaining modules. It is also possible to request the processing corresponding to the speech recognition server 400, and receive the processing result.

At this time, in any case, when the reliability of the speech recognition processing result, that is, the ASR processing and/or the NLU processing result, is less than a certain level, the speech recognition server 400 performs ASR processing and / Alternatively, as described above, it is possible to request NLU processing and obtain the processing result.

In this way, the voice recognition module 123 may perform voice recognition on the user voice received through the microphone 110 to obtain user intention information corresponding to the user voice.

Also, the voice recognition module 123 may obtain state information of an external device related to user intention information. Specifically, as described above, when the user intention information is obtained, the NLU module 123-2 (hereinafter, the operation of the NLU module 123-2, the voice recognition module 123 is the ASR module 123-1) In the case of an embodiment including only, the voice recognition module 123 may be regarded as being performed.) The device control server 500 through the device control module 121 transmits state information of the external device related to the acquired user intention information. ).

In this way, when the user intention information and the status information of the external device are obtained, the NLU module 123-2 transfers the loading of the rule DB 141-1 of the control command determination tool 141 to the tool management module 125. Can be requested.

Accordingly, when the rule DB 141-1 is loaded into the processor 120, the NLU module 123-2 includes the acquired user intention information and the state information of the external device among the loaded rule DB 141-1. Checks whether a rule in which the same user intention information and status information of an external device are matched exists in the rule DB 141-1, and if there is, a control command matching the rule is given to a control command corresponding to the user's voice. Can be judged as.

For example, based on the history of controlling the TV 1 (200-1) through the voice recognition server 400 as the user previously uttered the user's voice "Turn on the TV", rule 1 of Table 1 and Suppose that the same rule is created and stored in the rule DB 141-1, and the user utters a voice "Turn on TV" again later.

In this case, the voice recognition module 123 performs voice recognition on the user's voice to obtain user intention information of “TV power-on”, and the NLU module 123-2 controls the state information of the external device related to the TV. It may be received from the device control server 500 through the module 121.

In addition, the NLU module 123-2 requests the tool management module 125 to load the rule DB 141-1, and accordingly, the rule DB 141-1 including the rule 1 is transferred to the processor 120 Can be loaded into

At this time, when the status information of the external device received by the NLU module 123-2 is "TV 1 (200-1) off" and "TV 2 (200-2) on", the rule DB (141-1) Since the rule 1 in which the user intention information is "TV power-on" and the status information of the external device is "TV 1 (200-1) off" and "TV 2 (200-2) on" exists, the voice recognition module ( 123) may immediately determine “TV 1 (200-1) power-on”, which is a control command matched with Rule 1, as a control command for “Turn on TV” currently received.

In the above, it has been described as an example that the entire rule DB 141-1 stored in the memory 140 is loaded into the processor 140 in response to the request of the voice recognition module 123, but the embodiment is not limited thereto.

For example, only rules including external devices related to user intention information may be loaded into the processor 120. For example, if the rule DB 141-1 shown in Table 1 is stored in the memory 140, and user intention information “TV power-on” is obtained as in the above-described example, the speech recognition module ( 123) may request the tool management module 125 to load an external device related to user intention information, that is, a rule including a TV. Accordingly, the tool management module 125 may load 10,000 rules including a TV from the rule DB 141-1 into the processor 120. In this case, when the voice recognition module 123 checks the existence of a matching rule, the number of comparison targets decreases, so that the presence or absence of a control command may be more quickly determined.

Meanwhile, according to another embodiment of the present disclosure, when user intention information and status information of an external device are acquired, the NLU module 123-2 is an artificial intelligence model 141-2 of the control command determination tool 141. ) May be requested to be loaded from the tool management module 125.

Accordingly, when the artificial intelligence model 141-2 is loaded into the processor 120, the NLU module 123-2 transmits the acquired user intention information and the state information of the external device to the artificial intelligence model 141-2. A control command input and output accordingly (eg, a control command having a highest probability value among a plurality of control commands) may be determined as a control command corresponding to a user's voice.

For example, based on the history of controlling the TV 1 (200-1) through the voice recognition server 400 as the user previously uttered the user's voice "Turn on the TV", user intention information ("TV power -on") and the status information of the external device related to the user intention information ("TV 1(200-1) off", "TV 2(200-2) on"), a control command ("TV 1 (200 -1) Suppose that the artificial intelligence model 141-2 is trained by outputting "power-on") as the highest probability value.

In this case, assuming that the user utters the voice "Turn on TV" again later, the voice recognition module 123 performs voice recognition processing on the user's voice to obtain user intention information of "TV power-on". , The NLU module 123-2 may receive status information of an external device related to the TV from the device control server 500 through the device control module 121.

At this time, when the received status information of the external device is "TV 1 (200-1) off" or "TV 2 (200-2) on", the NLU module 123-2 is the currently acquired user intention information. "TV power-on" and "TV 1 (200-1) off" and "TV 2 (200-2) on", which are status information of the currently received external device, are added to the learned artificial intelligence model 141-2. You can enter. Accordingly, the NLU module 123-2 receives the control command "TV 1 (200-1) power-on" output from the artificial intelligence model 141-2 for the currently received "Turn on TV" Can be judged immediately.

On the other hand, according to an embodiment of the present disclosure, before using the control command determination tool 141, the NLU module 123-2, as in the operation of the voice recognition server 400 of FIG. Based on the information and the state information of the external device, it is determined first whether a control command is specified only with user intention information, and when the control command is specified only with user intention information, the acquired user without using the control command determination tool 141 The control command may be determined immediately based on the intention information.

For example, in the above example, when there is one TV in the user account registered in the device registration server 500 and the TV status information “TV off” is received, only the user intention information “TV power-on” , Since the control target device and the operation of the control target device are specified, the voice recognition module 123 does not request the control command determination tool 141 to the tool management module 125, and turns on "TV power-on" It could be judged right away with a control command for the user's voice, "Give me".

In this way, when a control command corresponding to the user's voice is determined, the voice recognition module 121 transmits the determined control command to the device control module 121 to control the operation of the controlling device.

Meanwhile, the tool management module 125 may update the rule DB 141-1 or the artificial intelligence model 141-2. Specifically, the voice recognition module 123 does not exist in the rule DB 141-1 or a rule corresponding to the acquired user intention information and the state information of the external device, or the acquired user intention information and the state information of the external device. When input to the artificial intelligence model 141-2 but a control command is not output (for example, when there is no control command having a predetermined probability value or more), the acquired user intention information and the status information of the external device are used as a tool. It transmits to the management module 125 and transmits the user's voice to the voice recognition server 400 through the communication unit 130.

Accordingly, as described above in FIG. 1, the voice recognition server 400 may determine a control command corresponding to the user's voice and transmit the determined control command to the device control server 500 and the electronic device 100. Alternatively, the voice recognition server 400 may transmit the determined control command only to the device control server 500, and the device control server 500 may transmit the control command to the electronic device 100.

At this time, the control command transmitted to the device control server 500 is transmitted to the control target device in the form of a control signal and is used to control the operation of the control target device, as described above in FIG. 1.

Meanwhile, the control command transmitted to the electronic device 100 is used to update the rule DB 141-1 or the artificial intelligence model 141-2.

Specifically, when the control command determined by the voice recognition server 400 is received through the communication unit 130, the tool management module 125 includes user intention information received from the voice recognition module 123 and state information of an external device. A new rule may be generated by matching with a control command received from the voice recognition server 400, and the generated new rule may be updated in the rule DB 141-1.

In addition, the tool management module 125 inputs user intention information received from the voice recognition module 123 and status information of an external device, and outputs a control command received from the voice recognition server 400 to produce artificial intelligence. By training the model, the artificial intelligence model 141-2 can be updated.

In this way, the updated rule DB 141-1 and the artificial intelligence model 141-2 may be used to determine a control command corresponding to the user's voice received through the microphone 110 afterwards.

Meanwhile, in the above, it has been exemplified that the tool management module 125 receives both user intention information and state information of an external device from the voice recognition module 123, but the embodiment is not limited thereto.

For example, the voice recognition module 123 may transmit only the acquired user intention information to the tool management module 125. Accordingly, the tool management module 125 requests the device control module 121 for status information of the external device related to the user intention information received from the voice recognition module 123, and the device control module 121 You can also receive status information directly.

Meanwhile, in the above, examples in which the NLU module 123-2 uses the rule DB 141-1 or the artificial intelligence model 142-2 to determine a control command have been described, respectively. However, according to an embodiment of the present disclosure, the NLU module 123-2 first attempts to determine a control command using the rule DB 141-1, and responds to the acquired user intention information and the state information of the external device. When the rule DB 141-1 does not exist, the control command may be determined using the artificial intelligence model 141-2.

Alternatively, according to another embodiment of the present disclosure, the NLU module 123-2 first attempts to determine a control command using the artificial intelligence model 141-2, and the control command having a predetermined probability value or more is artificial intelligence. When not output from the model 141-2, the control command may be determined based on the rule DB 141-1.

In the above two embodiments, when the control command is not determined regardless of whether the rule DB 141-1 and the artificial intelligence model 141-2 are used, the tool management module 125 uses the rule DB 141- 1) and the artificial intelligence model 141-2 can be updated.

Hereinafter, various embodiments of the present disclosure will be described with reference to FIGS. 3 to 5. In describing FIGS. 3 to 5, overlapping descriptions of the same contents as those described above will be omitted.

3 is a diagram illustrating a speech recognition system according to an embodiment of the present disclosure. According to FIG. 3, the voice recognition system 1000' includes an electronic device 100, a smart TV 200-3, a smart air conditioner 1 (200-4), a smart air conditioner 2 (200-5), and a legacy TV 200. -6), an access point 300, a voice recognition server 400, and a device control server 500.

The electronic device 100, smart TV (200-3), air conditioner 1 (200-4), air conditioner 2 (200-5) are devices that can connect to the Internet, and constitute an IoT network at home (10). It may be connected to the voice recognition server 400 and the device control server 500 through the access point 300. Meanwhile, in the case of the legacy TV 200-6, internet connection is not possible, and operation can be controlled only by using an IR communication method.

Meanwhile, the voice recognition server 400 and the device control server 500 may be cloud servers, but are not limited thereto.

In this situation, the user accesses the device control server 500 and stores the smart TV 200-3, air conditioner 1 (200-4), air conditioner 2 (200-5), and legacy TV 200-6 in his/her account. You can register at

At this time, the device control server 500 monitors and operates the status information only for the smart TV 200-3, air conditioner 1 (200-4), and air conditioner 2 (200-5) connected through the access point 300. It is possible to control, and it is impossible to monitor the status information and control the operation of the legacy TV 200-6. However, it can be known through the user account of the legacy TV 200-6.

Meanwhile, a rule DB as shown in Table 2 below may be stored in the memory 140 of the electronic device 100 based on history information that previously controlled the external device.

User Intent Information External device status information Control command Rule 1 TV power-on Smart TV (200-3): on
Legacy TV (200-6) Legacy TV (200-6) power-on Rule 2 Air conditioner power-on Air conditioner 1 (200-4): off
Air conditioner 2 (200-5): off Air conditioner 2(200-5) power-on

In such a situation, when a user's voice such as "Turn on TV" is received, the electronic device 100 acquires user intention information such as "TV power-on" by performing voice recognition processing on the user's voice, as described above. , TV status information may be requested from the device control server 500. Accordingly, when status information such as "Smart TV 200-3 on" and "Legacy TV 200-6" is received, the electronic device 100 is configured with the "Legacy TV 200" based on Rule 1 of Table 2. -6) "power-on" can be judged as a control command.

However, in this case, since it is impossible to control the operation of the legacy TV 200-6 through the device control server 500, the electronic device 100 directly transmits the determined control command to the legacy TV 200-6, The operation of the legacy TV 200-6 can be controlled. To this end, the electronic device 100 may include a configuration for transmitting and receiving an IR signal such as an IR blaster.

On the other hand, when the user utters a voice such as "Turn on the air conditioner" while the air conditioner 1 (200-4) and the air conditioner 2 (200-5) are both off, the electronic device 100 complies with Rule 2 of Table 2. Accordingly, a control command such as "air conditioner 2 (200-5) power-on" can be determined. In this case, since it is possible to control the operation of the air conditioner 2 (200-5) through the device control server 500, the electronic device 100 is controlled by the device control server 500 as described above in FIGS. 1 and 2 By sending a command, the operation of the air conditioner 2 (200-5) can be controlled.

However, the embodiment is not limited thereto. That is, even when operation control through the device control server 500 is possible, the electronic device 100 directly sends a control command to the air conditioner 2 (200-5) through a communication method between devices such as Wi-Fi Direct, according to an embodiment. By transmitting, it may be possible to control the operation of the air conditioner 2 (200-5).

4 is a block diagram of an electronic device according to an embodiment of the present disclosure. Referring to FIG. 4, the electronic device 100 ′ may include a microphone 110, a processor 120, a communication unit 130, a memory 140, a speaker 150, and a display 160.

The microphone 100 may be implemented as one or more microphones, may be implemented integrally with the electronic device 100 ′, or may be implemented separately. Here, the detachable microphone refers to a form in which the microphone is not included in the main body of the electronic device 100 ′, but is separately connected to the electronic device 100 ′ by wire or wirelessly.

The communication unit 130 may refer to hardware capable of transmitting and receiving various information (or data) by performing communication with various servers or devices.

The communication unit 130 includes Transmission Control Protocol/Internet Protocol (TCP/IP), User Datagram Protocol (UDP), Hyper Text Transfer Protocol (HTTP), Secure Hyper Text Transfer Protocol (HTTPS), File Transfer Protocol (FTP), and SFTP ( Secure File Transfer Protocol), MQTT (Message Queuing Telemetry Transport), and other communication protocols (protocols) can be used to transmit and receive various information with external servers (400, 500) or devices (200-1 to 200-6). have.

The communication unit 130 may be connected to various servers 400 and 500 and a plurality of external devices 200-1 to 200-6 through various networks. Here, the network includes a personal area network (PAN), a local area network (LAN), a wide area network (WAN), etc. according to an area or scale, and an intranet ( Intranet), an extranet, or the Internet.

The communication unit 130 may include at least one of a short-range wireless communication module (not shown) and a wireless LAN communication module (not shown). A short-range wireless communication module (not shown) is a communication module that performs wireless data communication with an external device located in a short distance, for example, a Bluetooth module, a ZigBee module, a NFC (Near Field Communication) module, It can be an infrared communication module, an IR (Infrared) communication module, a Wi-Fi module (when using the Wi-Fi P2P function), and the like. In addition, the wireless LAN communication module (not shown) is a module that is connected to an external network according to a wireless communication protocol such as WiFi and IEEE, and communicates with an external server or an external device.

In addition, the communication unit 130 according to various mobile communication standards such as 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), 5G (5th Generation mobile communications), etc. It may further include a mobile communication module that connects and performs communication, and HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), IEEE (Institute of Electrical and Eletronics Engineers) 1394, RS-232, RS-422 , RS-485, Ethernet, etc. may further include a wired communication module (not shown) according to the communication standard.

Meanwhile, the communication unit 130 may include a network interface or a network chip according to the wired/wireless communication method described above. In addition, the communication method is not limited to the above-described example, and may include a communication method newly emerging according to the development of technology.

The memory 140 may store an operating system (O/S) for the operation of the electronic device 100 ′ or the processor 120, various programs or applications, and data. Specifically, at least one instruction, module, or data necessary for the operation of the electronic device 100 ′ or the processor 120 may be stored in the memory 140.

Here, the instruction is a unit of code indicating the operation of the electronic device 100' or the processor 120, and may be written in a machine language, which is a language understandable by a computer. A module may be a set of instructions that perform a specific task in a unit of work. The data may be information in bits or bytes that can represent characters, numbers, images, and the like.

The memory 140 is accessed by the processor 120, and read/write/modify/delete/update instructions, modules, artificial intelligence models, or data by the processor 120 may be performed.

To this end, the memory 140 may include an internal memory or an external memory. The built-in memory may include at least one of a volatile memory or a non-volatile memory. The volatile memory may be, for example, dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), or the like. Non-volatile memory is, for example, one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, NAN flash memory, NOR flash. It could be memory, etc. Also, the internal memory may be a solid state drive (SSD). The external memory may include a flash drive, compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), or a memory stick. . The external memory may be functionally connected to the electronic device 100 ′ through various interfaces. Further, the electronic device 100 ′ may further include a storage device such as a hard drive.

The speaker 150 is a device that outputs an electrical signal in an audible form (eg, voice). The speaker may directly output a variety of notification sounds or voice messages as sound as well as various types of audio data on which various processing tasks such as decoding, amplification, and noise filtering have been performed by an audio processing unit (not shown). For example, when a query according to the policy of the voice recognition server 400 is received through the communication unit 130, the speaker 150 may output the received query as a voice.

The display 160 is a device that outputs information in a visual form (eg, graphic, text, image, etc.). The display 160 may display the image frame on the entire or partial area of the display area. For example, when a query according to a policy of the voice recognition server 400 is received through the communication unit 130, the display 160 may output the received query in text.

The processor 120 reads various programs (eg, at least one instruction, module, etc.) or data stored in the memory 140 to perform an operation of the electronic device 100 ′ according to various embodiments of the present disclosure. I can.

A rule DB, an artificial intelligence model, a voice recognition module, a device control module, and the like stored in the memory 140 may be loaded by the processor 120 to perform respective functions. To this end, the processor 120 may include an internal memory for loading at least a portion of various programs and data stored in the memory 140.

Meanwhile, the processor 120 is a central processing unit (CPU), a controller, an application processor (AP), a microprocessor unit (MPU), and a communication processor (CP). , GPU (Graphic Processing Unit), VPU (Vision Processing Unit), NPU (Neural Processing Unit), or may include one or more of the ARM processor.

Meanwhile, in FIG. 2, the rule corresponding to the acquired user intention information and the state information of the external device does not exist in the rule DB 141-1, or the acquired user intention information and the state information of the external device are used in an artificial intelligence model ( 141-2), but when a control command is not output, a user voice is transmitted to the voice recognition server 400, but the embodiment is not limited thereto.

That is, according to an embodiment, the processor 120 always sends the received user voice to the voice recognition server 400 when a user voice is received, regardless of whether a corresponding rule exists or whether a control command is output from the artificial intelligence model. It can also be transferred to.

In this embodiment, when a corresponding rule exists in the rule DB 141-1 or a control command is output from the artificial intelligence model, the control command determined by each of the electronic device 100 and the voice recognition server 400 is Since each is transmitted to the control server 500, the device control server 500 may receive the same control command from the electronic device 100 ′ and the voice recognition server 500, respectively.

In this case, the device control server 500 that has received the two control commands ignores the second control command received, or the control target device that has received the two control signals ignores the second control signal. By doing so, it is possible to prevent an error in which the controlling device performs the same operation twice.

At this time, the control command that the device control server 500 receives for the second time or the control signal that the control target device receives for the second time will be a control command transmitted by the voice recognition server 400 and a control signal according thereto. . This is because the control command determination by the control command determination tool 141 is faster than the control command determination by the voice recognition server 400.

Meanwhile, in FIGS. 1 and 2, a case in which an additional analysis of the voice recognition server 400 is required, and a case in which a control target device is not specified only with user intention information is exemplified, but is not limited thereto. Even if the operation of the device to be controlled is not specified, additional analysis of the voice recognition server 400 is required.

For example, if one TV 200-1 is registered in the user account, and the user utters a user voice such as "Play MBC", the electronic device 100' User intention information such as "Channel MBC" may be acquired, and state information of "TV" may be requested from the device control server 500.

In this case, when the TV 1 200-1 is turned on, the device control server 500 may transmit state information such as "TV 1 200-1 on" to the electronic device 100 ′. Meanwhile, the state information of the TV 1 200-1 is not limited to the on/off information as described above in the description of FIG. 1. For example, information on the current broadcasting channel being played on TV 1 (200-1), information on the volume of TV 1 (200-1) currently set may be further included in the status information according to embodiments. to be.

However, since the electronic device 100 ′ cannot know how many channels "MBC" means, the user's voice may be transmitted to the voice recognition server 400. (It is assumed that there are no rules related to "Play MBC" and the artificial intelligence model is not trained.)

Accordingly, the voice recognition server 400 also acquires user intention information such as “TV Channel MBC” through voice recognition processing, and a state such as “TV 1 (200-1) on” from the device control server 500 You can receive information.

Since the voice recognition server 400 also cannot know how many channels "MBC" means, it is determined that the operation of the control target device, that is, TV 1 200-1, is not specified only with the user intention information, and It is confirmed that MBC is channel 11 through additional interpretation according to its own policy.

Accordingly, the voice recognition server 400 determines a control command such as "TV 1 (200-1) Channel 11" and transmits it to the device control server 500, and accordingly, the channel of TV 1 (200-1) This is subject to change.

At this time, the voice recognition server 400 or the device control server 500 transmits the control command determined by the voice recognition server 400 to the electronic device 100', and accordingly, the processor of the electronic device 100' 120) may update the rule DB 141-1 by creating a rule as shown in Rule 1 in Table 3 below.

User Intent Information External device status information Control command Rule 1 TV Channel MBC TV 1(200-1): on TV 1(200-1) Channel 11 Rule 2 Air conditioner power-on Air conditioner 1: off
Air conditioner 2: off Air conditioner 2 power-on

Thereafter, when the user utters a user voice such as "Play MBC" again while the TV 1 (200-1) is turned on, the processor 120 can quickly determine a control command using Rule 1 of Table 3 above. You will be able to.

Meanwhile, various examples in which the electronic device 100 ′ uses the control command determination tool 141 to determine a control command corresponding to a user's voice have been described above. However, according to an embodiment of the present disclosure, the voice recognition server 400 may determine a control command corresponding to the user's voice using the control command determination tool 141.

5 is a block diagram of a voice recognition server according to an embodiment of the present disclosure. Referring to FIG. 5, the voice recognition server 400 may include a communication unit 410, a processor 420, and a memory 430.

The communication unit 410 is a component for performing communication with various external servers or various devices. In particular, the communication unit 410 communicates with the electronic device 100 and the device control server 500 to transmit and receive various information or data such as user voice, user intention information, status information of external devices, and control commands. can do.

The memory 430 may store data or information in electrical or magnetic form so that the processor 420 or the like can access the stored data or information. In particular, the memory 430 includes a control command determination tool 431 that is a basis for determining a control command, a tool management module 425 for managing its own policy, and the control command determination tool 431, and a voice recognition function. A voice recognition module 423 for performing and determining a control command, and a device control module 421 for monitoring status information of external devices 200-1 and 200-2 and transmitting and receiving control commands are stored. Can be.

The processor 420 controls the overall operation of the voice recognition server 400. In particular, the processor 420 loads a control command determination tool 431, a tool management module 425, a voice recognition module 423, a device control module 421, etc. stored in the memory 430 to perform functions of each module. Can be done.

Specifically, FIG. 5 shows a state in which the processor 420 loads the device control module 421, the voice recognition module 423, and the tool management module 425 stored in the memory 430 and performs a corresponding function. Is shown.

Among the components shown in FIG. 5, components having the same name as those shown in FIGS. 2 and 4 may have the same contents or operate in the same manner as the components shown in FIGS. 2 and 4. Hereinafter, redundant descriptions of the same contents as those described above with reference to FIGS. 2 and 4 will be omitted, and different contents will be mainly described.

Referring to FIG. 5, it can be seen that each configuration is the same as that of the electronic device 100 of FIG. 2 except that there is no microphone. In the electronic device 100, the user's voice is received through the microphone 110, but in the voice recognition server 400, it may be received from the electronic device 100 through the communication unit 410.

On the other hand, the voice recognition server 400, if the rule corresponding to the acquired user intention information and the state information of the external device does not exist in the rule DB 431-1, or the acquired user intention information and the state information of the external device are artificial. Even if the control command is input to the intelligent model 431-2 but the control command is not output, the control command is directly determined through its own policy, so the control command determined by the NLU module 123-2 is directly transmitted to the tool management module 425 It can be transmitted to and there is no need to receive it from the outside through the communication unit 410.

For example, when a user voice is received from the electronic device 100 through the communication unit 410, the voice recognition module 423 processes the received user voice ASR and NLU to obtain user intention information, and obtains user intention. Status information of the external device related to the information may be received from the device control server 500 through the device control module 421.

Accordingly, the NLU module 423-2 requests the loading of the control command determination tool 431 to the tool management module 425, and uses the loaded control command determination tool 431 as described above in FIG. A control command corresponding to the user's voice can be determined.

On the other hand, when the control command cannot be determined using the control command determination tool 431, the NLU module 423-2 transmits the acquired user intention information and the status information of the external device to the tool management module 425 do. Also, the NLU module 723-2 determines a control command using its own policy.

When a control command is determined through its own policy, the NLU module 423-2 transmits the determined control command to the tool management module 425, and the tool management module 425 is transferred from the NLU module 423-2. Based on the received user intention information, state information of an external device, and a control command, the rule DB 431-1 or the artificial intelligence model 431-2 may be updated as described above in FIG. 2.

In this way, the updated rule DB 125 and the artificial intelligence model 127 may be used to determine a control command corresponding to the user's voice received from the electronic device 100 through the communication unit 410 later.

Meanwhile, in general, a server device such as the voice recognition server 400 may have a larger storage space and a higher computing speed than a client device such as the electronic device 100.

Accordingly, the voice recognition server 400 includes a high-capacity, high-performance voice recognition module 423 (ASR module 423-1 and NLU module 423-2) capable of processing various voices input in various situations at a desired speed. Etc.) can be mounted.

However, in the case of the electronic device 100, there is a limit to the speech recognition module 123 (such as the ASR module 123-1 and the NLU module 123-2) that can be mounted due to the limitation of storage space or computational performance. have.

Accordingly, according to an embodiment of the present disclosure, when a voice recognition process (ASR process and/or NLU process) for a user's voice is requested from the electronic device 100, the voice recognition module 423 of the voice recognition server 400 ) Processes the received user voice according to the request of the electronic device 100, and transmits the result (text as a result of ASR processing or user intention information as a result of NLU processing) through the communication unit 410. ).

6 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 6, the electronic devices 100 and 100 ′ may store a control command determination tool 141 based on a control command determined by the voice recognition server 400 (S610 ).

In this case, the control command determination tool 141 includes: 1) a rule DB 141 including at least one rule in which user intention information, state information of an external device, and a control command determined by the voice recognition server 400 are matched with each other. -1), and 2) at least one of the artificial intelligence model 141-2 learned by inputting user intention information and state information of an external device and outputting a control command determined by the voice recognition server 400 It may include.

On the other hand, when a user voice is received, the electronic devices 100 and 100 ′ acquire user intention information by performing speech recognition on the received user voice (S620), and obtain state information of an external device related to the obtained user intention information. The device control server 500 may request and receive (S630).

Accordingly, the electronic devices 100 and 100 ′ apply the obtained user intention information and the received state information of the external device to the control command determination tool 141 to determine a control command for controlling the control target device. Then (S640), the determined control command may be transmitted to the device control server 500 (S650).

For example, if a rule corresponding to the acquired user intention information and the received state information of an external device exists in the rule DB 141-1, the electronic device 100 or 100 ′ may be matched to the corresponding rule. The control command may be determined as a control command corresponding to the user's voice.

If the corresponding rule does not exist in the rule DB 141-1, the electronic device 100, 100' transmits the user's voice to the voice recognition server 400, and accordingly, the voice recognition server 400 When the determined control command is received, a new rule is generated by matching the received control command with the acquired user intention information and the received state information of the external device, and the generated new rule is added to the rule DB 141-1. Can be updated.

Meanwhile, the electronic devices 100 and 100' input the acquired user intention information and the received state information of the external device to the artificial intelligence model 141-2, and control output from the artificial intelligence model 141-2. The command may be determined as a control command corresponding to the user's voice.

At this time, if the control command is not output from the artificial intelligence model 141-2, the electronic devices 100 and 100' transmit the user's voice to the voice recognition server 400, and accordingly, the voice recognition server 400 When the determined control command is received, the artificial intelligence model may be retrained based on the acquired user intention information, the received state information of the external device, and the received control command.

Meanwhile, according to an embodiment of the present disclosure, when the electronic device 100 or 100 ′ can specify the operation of the control target device and the control target device only with user intention information based on the state information of the external device, A control command may be determined based on user intention information without using the command determination tool 141.

At this time, when state information of a plurality of external devices related to an entity included in the user intention information is received from the device control server 500, the electronic devices 100 and 100 ′ may specify the control target device only with the user intention information. It is determined that there is no, and a control command may be determined using the control command determination tool 141.

Meanwhile, the electronic devices 100 and 100 ′ may directly transmit a control command to the control target device when the control target device is a device that can be controlled by the IR method.

According to various embodiments of the present disclosure as described above, it is possible to quickly and accurately control devices through a user's voice in a multi-device environment.

Meanwhile, various embodiments of the present disclosure may be implemented with software including instructions stored in a machine-readable storage media (eg, a computer). Here, the device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device 100 or 100 ′ or a voice recognition server 400 according to the disclosed embodiments. .

When the command is executed by a processor, the processor may perform a function corresponding to the command directly or by using other components under the control of the processor. Instructions may include code generated or executed by a compiler or interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here,'non-transient' means that the storage medium does not contain a signal and is tangible, but does not distinguish between semi-permanent or temporary storage of data in the storage medium.

According to an embodiment, a method according to various embodiments disclosed in the present disclosure may be provided by being included in a computer program product. Computer program products can be traded between sellers and buyers as commodities. The computer program product may be distributed online in the form of a device-readable storage medium (eg, compact disc read only memory (CD-ROM)) or through an application store (eg, Play StoreTM). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium such as a server of a manufacturer, a server of an application store, or a memory of a relay server.

Each of the constituent elements (eg, modules or programs) according to various embodiments may be composed of a singular or a plurality of entities, and some sub-elements of the aforementioned sub-elements are omitted, or other sub-elements are various. It may be further included in the embodiment. Alternatively or additionally, some constituent elements (eg, a module or a program) may be integrated into one entity, and functions performed by each corresponding constituent element prior to the consolidation may be performed identically or similarly. Operations performed by modules, programs, or other components according to various embodiments may be sequentially, parallel, repetitively or heuristically executed, or at least some operations may be executed in a different order, omitted, or other operations may be added. I can.

The above description is merely illustrative of the technical idea of the present disclosure, and those of ordinary skill in the art to which the present disclosure pertains will be able to make various modifications and variations without departing from the essential characteristics of the present disclosure. Further, the embodiments according to the present disclosure are not intended to limit the technical idea of the present disclosure, but are described, and the scope of the technical idea of the present disclosure is not limited by these embodiments. Accordingly, the scope of protection of the present disclosure should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present disclosure.

100: electronic device 110: microphone
120: processor 130: communication unit
140: memory
200-1 to 200-6: external device 300: access point
400: voice recognition server 500: device control server

Claims (18)

In the electronic device,
microphone;
Communication department;
A memory storing a control command determination tool based on a control command determined by a voice recognition server for voice recognition processing of the user's voice received from the electronic device; And
When a user voice is received through the microphone, the received user voice is voice-recognized to obtain user intention information, and status information of the external device related to the obtained user intention information is transmitted to a plurality of external devices through the communication unit. It receives from the device control server for controlling,
The obtained user intention information and the received state information of the external device are applied to the control command determination tool to determine a control command for controlling a control target device among the plurality of external devices, and the determined control command is And a processor that transmits to the device control server through the communication unit. The method of claim 1,
The control command determination tool,
And a rule DB including at least one rule in which user intention information, state information of an external device, and a control command determined by the voice recognition server are matched with each other,
The processor,
When a rule corresponding to the acquired user intention information and the received state information of the external device exists in the rule DB, determining a control command matched to the corresponding rule as a control command corresponding to the user voice, Electronic device. The method of claim 2,
The processor,
When a rule corresponding to the obtained user intention information and the received state information of the external device does not exist in the rule DB, the user voice is transmitted to the voice recognition server through the communication unit,
When a control command determined by the voice recognition server based on the user's voice is received through the communication unit, a new rule is established by matching the received control command with the acquired user intention information and the received state information of the external device. An electronic device that generates and updates the generated new rule to the rule DB. The method of claim 1,
The control command determination tool,
Including an artificial intelligence model learned by taking user intention information and state information of an external device as input, and outputting a control command determined by a voice recognition server,
The processor,
Inputting the acquired user intention information and the received state information of the external device into the learned artificial intelligence model, and determining a control command output from the learned artificial intelligence model as a control command corresponding to the user voice, Electronic device. The method of claim 4,
The processor,
When the control command is not output from the learned artificial intelligence model in which the acquired user intention information and the received state information of the external device are input, the user voice is transmitted to the voice recognition server through the communication unit,
When a control command determined by the voice recognition server based on the user voice is received through the communication unit, the learned user intention information, the received state information of the external device, and the received control command An electronic device that retrains an artificial intelligence model. The method of claim 1,
The processor,
When the operation of the control target device and the control target device can be specified only with the acquired user intention information based on the received state information of the external device, the acquired user intention information without using the control command determination tool The electronic device that determines the control command based on. The method of claim 6,
The obtained user intention information includes information on an entity,
The processor,
When status information of a plurality of external devices related to the entity is received from the device control server, it is determined that the control target device cannot be specified only with the obtained user intention information, and the control command is performed using the control command determination tool. To determine the electronic device. The method of claim 1,
The communication unit,
Including; IR (Infrared) communication module,
The processor,
When the control target device is a device that can be controlled by an IR method, the electronic device transmits the determined control command to the control target device through the IR communication module. In the control method of an electronic device,
Storing a control command determination tool based on a control command determined by a voice recognition server that performs voice recognition processing on the user's voice received from the electronic device;
When a user voice is received, performing voice recognition processing on the received user voice to obtain user intention information;
Receiving state information of an external device related to the acquired user intention information from a device control server for controlling a plurality of external devices;
Determining a control command for controlling a control target device among the plurality of external devices by applying the obtained user intention information and the received state information of the external device to a control command determination tool; And
And transmitting the determined control command to the device control server. The method of claim 9,
The control command determination tool,
And a rule DB including at least one rule in which user intention information, state information of an external device, and a control command determined by the voice recognition server are matched with each other,
The step of determining the control command,
When a rule corresponding to the acquired user intention information and the received state information of the external device exists in the rule DB, determining a control command matched to the corresponding rule as a control command corresponding to the user voice, Control method. The method of claim 10,
Transmitting the user voice to the voice recognition server when a rule corresponding to the obtained user intention information and the received state information of the external device does not exist in the rule DB; And
When a control command determined by the voice recognition server is received based on the user voice, a new rule is generated by matching the received control command with the obtained user intention information and the received state information of the external device, and the The method further comprising, updating the generated new rule to the rule DB. The method of claim 9,
The control command determination tool,
Including an artificial intelligence model learned by taking user intention information and state information of an external device as input, and outputting a control command determined by a voice recognition server,
The step of determining the control command,
Inputting the acquired user intention information and the received state information of the external device into the learned artificial intelligence model, and determining a control command output from the learned artificial intelligence model as a control command corresponding to the user voice, Control method. The method of claim 12,
Transmitting the user voice to the voice recognition server when the control command is not output from the learned artificial intelligence model in which the acquired user intention information and the received state information of the external device are input; And
When a control command determined by the voice recognition server is received based on the user voice, the learned artificial intelligence model is based on the acquired user intention information, the received state information of the external device, and the received control command. Re-learning; further comprising, the control method. The method of claim 9,
When the operation of the control target device and the control target device can be specified only with the acquired user intention information based on the received state information of the external device, the acquired user intention information without using the control command determination tool Determining the control command based on; further comprising, the control method. The method of claim 14,
The obtained user intention information includes information on an entity,
When status information of a plurality of external devices related to the entity is received from the device control server, it is determined that the control target device cannot be specified only with the obtained user intention information, and the control command is performed using the control command determination tool. Determining the; further comprising, the control method. The method of claim 1,
If the control target device is a device that can be controlled by the IR method, transmitting the determined control command to the control target device; further comprising, the control method. In the voice recognition server,
Communication department;
A memory storing a control command determination tool based on a control command determined by the voice recognition server based on a user voice received from an electronic device; And
When a user voice is received from the electronic device through the communication unit, the received user voice is voice-recognized to obtain user intention information, and a plurality of state information of an external device related to the obtained user intention information is received through the communication unit. Received from the device control server to control the external device of the
The obtained user intention information and the received state information of the external device are applied to the control command determination tool to determine a control command for controlling a control target device among the plurality of external devices, and the determined control command is A processor for transmitting to the device control server through the communication unit. The method of claim 17,
The control command determination tool,
A rule DB including at least one rule in which user intention information, state information of an external device, and a control command determined by the voice recognition server are matched with each other, and
A voice recognition server comprising at least one of an artificial intelligence model learned by inputting user intention information and state information of an external device as an input and outputting a control command determined by the voice recognition server.

Images