KR102701423B1 - Electronic device for performing speech recognition and the method for the same - Google Patents

Abstract

In various embodiments of the present invention, an electronic device and a method of controlling an operation of an electronic device include a network interface, at least one processor operatively connected to the network interface, and at least one memory operatively connected to the processor, wherein the at least one memory is configured to store a generic language model and a plurality of domain-based language models for automatic speech recognition (ASR), and the at least one memory is configured to, when executed, cause the processor to receive first speech data from an external device including a microphone through the network interface, use the generic language model but not the domain-based language models, process the first speech data with the ASR to generate first text data associated with at least one task to be performed by the system, determine a domain associated with the task based at least in part on the first text data, and generate one domain-based language model associated with the determined domain among the domain-based language models. The method may store instructions for selecting a language model, receiving second voice data from the external device through the network interface after receiving the first voice data and before performing the task, and processing the second voice data with the ARS using the generic language model and the selected one domain-based language model with the ASR to generate second text data.
Besides these, various other embodiments are possible.

Description

{ELECTRONIC DEVICE FOR PERFORMING SPEECH RECOGNITION AND THE METHOD FOR THE SAME}

Various embodiments of the present invention relate to an electronic device performing voice recognition and a method of operating the same.

Various electronic devices such as smart phones, tablet PCs, portable multimedia players (PMPs), personal digital assistants (PDAs), laptop personal computers, and wearable devices are becoming widespread.

Recently, voice recognition functions have been applied to electronic devices, and users of electronic devices can use their voices to execute various functions of the electronic devices.

Furthermore, as the Internet of Things, which connects devices used in the user's living environment to wired and wireless networks to share information, begins to be applied, an environment is being created in which various electronic devices, such as televisions and refrigerators, can perform voice commands to other electronic devices connected to the network.

Speech recognition technology can use an acoustic model and a language model to recognize the voice spoken by the user. An acoustic model can mean a database in which feature vectors obtained by processing voice data and candidate phonemes corresponding to the feature vectors are mapped and stored. A language model can mean a database in which information used to model the linguistic order relationship of words or syllables, etc., is stored using phonemes corresponding to voice data extracted from an acoustic model.

For words used in specific fields rather than general words, they are often not included in language models, so the accuracy of speech recognition for words used in specific fields may be lower than the accuracy of speech recognition for words used commonly.

A system according to various embodiments of the present invention comprises a network interface, at least one processor operatively connected to the network interface, and at least one memory operatively connected to the processor, wherein the at least one memory is configured to store a generic language model and a plurality of domain-based language models for automatic speech recognition (ASR), and wherein the at least one memory is configured to, when executed, cause the processor to receive first speech data from an external device including a microphone through the network interface, use the generic language model but not the domain-based language models, process the first speech data with the ASR to generate first text data associated with at least one task to be performed by the system, determine a domain associated with the task based at least in part on the first text data, select one domain-based language model associated with the determined domain from among the domain-based language models, and, after receiving the first speech data and performing the task, Before the execution, instructions can be stored for receiving second voice data from the external device through the network interface, and processing the second voice data with the ARS using the generic language model and the selected domain-based language model with the ASR to generate second text data.

A system according to various embodiments of the present invention comprises a network interface, at least one processor operatively connected to the network interface, and at least one memory operatively connected to the processor; wherein the at least one memory is configured to store a generic language model and a plurality of domain-based language models for automatic speech recognition (ASR), and wherein the at least one memory is configured to cause, when executed, the processor to receive first speech data from an external device including a microphone through the network interface, to process the first speech data using the generic language model but not using the domain-based language models, to generate first text data associated with at least one task to be performed by the system, to determine a domain associated with the task based at least in part on the first text data, to select one domain-based language model associated with the determined domain from among the domain-based language models, and after receiving the first speech data, Before performing the above task, instructions can be stored for receiving second voice data from the external device through the network interface, and processing the second voice data with the ARS using the selected domain-based language model with the ASR to generate second text data.

An operating method of an electronic device performing speech recognition according to various embodiments of the present invention comprises: receiving first speech data from an external device including a microphone through a network interface; using a generic language model for automatic speech recognition (ASR), but not using a plurality of domain-based language models, processing the first speech data using the ASR to generate first text data associated with at least one task to be performed by the electronic device; determining a domain associated with the task based at least in part on the first text data; selecting one domain-based language model associated with the determined domain among the domain-based language models; receiving second speech data from the external device through the network interface after receiving the first speech data and before performing the task; processing the second speech data with the ASR using the generic language model and the selected one domain-based language model to generate second text data. May include generating actions.

An electronic device performing speech recognition according to various embodiments of the present invention and an operating method of the electronic device determine a domain using text data converted from speech data, and perform speech recognition using both a content provider language model and a basic language model corresponding to the domain, thereby increasing the accuracy of speech recognition.

An electronic device performing speech recognition according to various embodiments of the present invention and an operating method of the electronic device can increase the accuracy of speech recognition for words used in a specific field by using a language model of a content provider specialized for a specific service or a specific field.

FIG. 1 is a block diagram of an electronic device according to various embodiments of the present invention.
FIG. 2 is a block diagram of a program operating on an electronic device according to various embodiments of the present invention.
FIG. 3 is a diagram illustrating an integrated intelligent system including an electronic device and a server performing voice recognition according to various embodiments of the present invention.
FIG. 4 is a block diagram illustrating an intelligent server performing voice recognition according to various embodiments of the present invention.
FIGS. 5A and 5B are block diagrams illustrating a language model and a speech recognition unit in an intelligent server according to various embodiments of the present invention.
FIGS. 6A to 6C are diagrams illustrating an example of selecting a language model and performing speech recognition using the selected language model in an intelligent server according to various embodiments of the present invention.
FIG. 7 is a block diagram illustrating an electronic device according to various embodiments of the present invention.
FIGS. 8A to 8D are diagrams illustrating examples of selecting a language model and performing speech recognition using the selected language model in an intelligent server and electronic device according to various embodiments of the present invention.
FIG. 9 is a diagram illustrating an example of selecting multiple language models and performing speech recognition using the selected language models in an intelligent server and electronic device according to various embodiments of the present invention.
FIGS. 10A to 12B are diagrams illustrating examples of outputting information about a session in an electronic device according to various embodiments of the present invention.
FIG. 13 is a flowchart illustrating an operation method of an electronic device performing voice recognition according to various embodiments of the present invention.

FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) through a first network (198) (e.g., a short-range wireless communication network), or may communicate with the electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input device (150), an audio output device (155), a display device (160), an audio module (170), a sensor module (176), an interface (177), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the display device (160) or the camera module (180)), or may include one or more other components. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, a sensor module (176) (e.g., a fingerprint sensor, an iris sensor, or a light sensor) may be implemented embedded in a display device (160) (e.g., a display).

The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may load a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) into the volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in the nonvolatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor), and a secondary processor (123) (e.g., a graphic processing unit, an image signal processor, a sensor hub processor, or a communication processor) that may operate independently or together therewith. Additionally or alternatively, the auxiliary processor (123) may be configured to use less power than the main processor (121), or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121), or as a part thereof.

The auxiliary processor (123) may control at least a portion of functions or states associated with at least one of the components of the electronic device (101) (e.g., the display device (160), the sensor module (176), or the communication module (190)), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)).

The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or nonvolatile memory (134).

The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input device (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input device (150) can include, for example, a microphone, a mouse, or a keyboard.

The audio output device (155) can output an audio signal to the outside of the electronic device (101). The audio output device (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive an incoming call. According to one embodiment, the receiver can be implemented separately from the speaker or as a part thereof.

The display device (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display device (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display device (160) can include touch circuitry configured to detect a touch, or a sensor circuitry configured to measure a strength of a force generated by the touch (e.g., a pressure sensor).

The audio module (170) can convert sound into an electric signal, or vice versa, convert an electric signal into sound. According to one embodiment, the audio module (170) can obtain sound through an input device (150), or output sound through an audio output device (155), or an external electronic device (e.g., an electronic device (102)) (e.g., a speaker or a headphone) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect an operating state (e.g., power or temperature) of the electronic device (101) or an external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) to an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., the electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that a user can perceive through a tactile or kinesthetic sense. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) can include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., the electronic device (102), the electronic device (104), or the server (108)), and performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., the application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module or a power line communication module). A corresponding communication module among these communication modules can communicate with an external electronic device via a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi direct, or IrDA (infrared data association)) or a second network (199) (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify and authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) can include one or more antennas, from which at least one antenna suitable for a communication method used in a communication network, such as the first network 198 or the second network 199, can be selected, for example, by the communication module (190). A signal or power can be transmitted or received between the communication module (190) and the external electronic device via the at least one selected antenna.

At least some of the above components may be connected to each other and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

According to one embodiment, a command or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the electronic devices (102, 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of executing the function or service itself or in addition, request one or more external electronic devices to perform at least a part of the function or service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

FIG. 2 is a block diagram (200) illustrating a program (140) according to various embodiments. According to one embodiment, the program (140) may include an operating system (142), middleware (144), or an application (146) executable in the operating system (142) for controlling one or more resources of the electronic device (101). The operating system (142) may include, for example, AndroidTM, iOSTM, WindowsTM, SymbianTM, TizenTM, or BadaTM. At least some of the programs (140) may be preloaded in the electronic device (101), for example, at the time of manufacturing, or may be downloaded or updated from an external electronic device (e.g., the electronic device (102 or 104), or a server (108)) when used by a user.

The operating system (142) may control the management (e.g., allocation or retrieval) of one or more system resources (e.g., processes, memory, or power) of the electronic device (101). The operating system (142) may additionally or alternatively include one or more driver programs for driving other hardware devices of the electronic device (101), such as an input device (150), an audio output device (155), a display device (160), an audio module (170), a sensor module (176), an interface (177), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197).

The middleware (144) may provide various functions to the application (146) so that functions or information provided from one or more resources of the electronic device (101) may be used by the application (146). The middleware (144) may include, for example, an application manager (201), a window manager (203), a multimedia manager (205), a resource manager (207), a power manager (209), a database manager (211), a package manager (213), a connectivity manager (215), a notification manager (217), a location manager (219), a graphics manager (221), a security manager (223), a call manager (225), or a voice recognition manager (227).

The application manager (201) can manage, for example, the life cycle of the application (146). The window manager (203) can manage, for example, one or more GUI resources used on the screen. The multimedia manager (205) can identify, for example, one or more formats required for playing media files, and perform encoding or decoding of a corresponding media file among the media files using a codec suitable for a corresponding format selected among the formats. The resource manager (207) can manage, for example, the source code of the application (146) or the memory space of the memory (130). The power manager (209) can manage, for example, the capacity, temperature, or power of the battery (189), and determine or provide related information required for the operation of the electronic device (101) using the corresponding information. According to one embodiment, the power manager (209) can be linked with a BIOS (basic input/output system) (not shown) of the electronic device (101).

The database manager (211) may, for example, create, search, or modify a database to be used by the application (146). The package manager (213) may, for example, manage the installation or update of an application distributed in the form of a package file. The connectivity manager (215) may, for example, manage a wireless connection or direct connection between the electronic device (101) and an external electronic device. The notification manager (217) may, for example, provide a function for notifying a user of the occurrence of a specified event (e.g., an incoming call, a message, or an alarm). The location manager (219) may, for example, manage location information of the electronic device (101). The graphics manager (221) may, for example, manage one or more graphic effects to be provided to the user or a user interface related thereto.

The security manager (223) may provide, for example, system security or user authentication. The telephony manager (225) may manage, for example, a voice call function or a video call function provided by the electronic device (101). The voice recognition manager (227) may, for example, transmit a user's voice data to the server (108) and receive, from the server (108), a command corresponding to a function to be performed in the electronic device (101) based at least in part on the voice data, or converted text data based at least in part on the voice data. In one embodiment, the middleware (244) may dynamically delete some of the existing components or add new components. In one embodiment, at least a portion of the middleware (144) may be included as a part of the operating system (142), or may be implemented as separate software different from the operating system (142).

The application (146) may include, for example, a home (251), a dialer (253), an SMS/MMS (255), an instant message (IM) (257), a browser (259), a camera (261), an alarm (263), a contact (265), voice recognition (267), an email (269), a calendar (271), a media player (273), an album (275), a watch (277), a health (279) (e.g., measuring biometric information such as the amount of exercise or blood sugar), or an environmental information (281) (e.g., measuring barometric pressure, humidity, or temperature information) application. According to one embodiment, the application (146) may further include an information exchange application (not shown) that can support information exchange between the electronic device (101) and an external electronic device. The information exchange application may include, for example, a notification relay application configured to transmit designated information (e.g., a call, a message, or an alarm) to an external electronic device, or a device management application configured to manage an external electronic device. The notification relay application may, for example, transmit notification information corresponding to a designated event (e.g., receipt of a mail) that occurred in another application (e.g., an email application (269)) of the electronic device (101) to the external electronic device. Additionally or alternatively, the notification relay application may receive notification information from the external electronic device and provide the information to a user of the electronic device (101).

The device management application may control, for example, the power (e.g., turning on or off) or the function (e.g., brightness, resolution, or focus of the display device (160) or the camera module (180)) of an external electronic device or a component thereof (e.g., a display device (160) or a camera module (180)) that communicates with the electronic device (101). The device management application may additionally or alternatively support installation, deletion, or updating of an application running on the external electronic device.

FIG. 3 is a diagram illustrating an integrated intelligent system including an electronic device and a server performing voice recognition according to various embodiments of the present invention.

Referring to FIG. 3, the integrated intelligence system (300) may include a user terminal (e.g., an electronic device (101), 310 of FIG. 1), an intelligent server (e.g., a server (108), 320 of FIG. 1), a cloud server (330), and a third-party server (340).

According to various embodiments of the present invention, the user terminal (310) can provide a service required by the user through an app (or application program) (e.g., an alarm app, a message app, or a photo (gallery) app, etc.) stored in the user terminal (310). For example, the user terminal (310) can execute and operate another app through an intelligent app (or a voice recognition app) stored in the user terminal (310). The user input for executing and operating the other app can be received through the intelligent app of the user terminal (310). The user input can be received through, for example, a physical button, a touch pad, a voice input, a remote input, etc. According to one embodiment, the user terminal (310) can be various terminal devices (or electronic devices) that can be connected to the Internet, such as a mobile phone, a smart phone, a PDA (personal digital assistant), a notebook computer, an electronic device to which IoT technology is applied (e.g., a smart TV, a smart refrigerator, an air purifier, etc.).

According to various embodiments of the present invention, the user terminal (310) can receive the user's speech as a user input. The user terminal (310) can receive the user's speech and generate a command for operating an app based on the user's speech. Accordingly, the user terminal (310) can operate the app using the command.

According to various embodiments of the present invention, the user terminal (310) can receive a user's speech and generate voice data corresponding to the user's speech. The generated voice data can be transmitted to an intelligent server (320).

According to various embodiments of the present invention, the user terminal (310) can preprocess the voice data before transmitting the voice data to the intelligent server (320). The user terminal (310) can perform preprocessing on the voice data by performing various operations, such as an operation of removing echo included in the voice data, an operation of removing background noise included in the voice data, and an operation of adjusting the volume of the voice included in the voice data, and transmit the preprocessed voice data to the intelligent server (320).

According to various embodiments of the present invention, an intelligent server (320) can receive voice data corresponding to a user's speech from an external device (e.g., a user terminal (310), etc.) through a communication network, and convert the voice data into text data.

According to various embodiments of the present invention, the intelligent server (320) can transmit converted text data to the user terminal (310). The user terminal (310) can output the received text data using a display or speaker.

According to various embodiments of the present invention, the operation of converting text data by the intelligent server (320) may be performed using an acoustic model and a language model stored in the memory of the intelligent server (320). The acoustic model may refer to a database in which feature vectors obtained by performing processing on speech data and candidate phonemes including phonemes corresponding to the feature vectors are mapped and stored. The language model may refer to a database in which information used to model the linguistic order relationship of words or syllables, etc., is stored using phonemes corresponding to speech data extracted from the acoustic model.

According to various embodiments of the present invention, the intelligent server (320) can extract phonemes (ㄴ/ㅏ/ㄴ/ㅡ/(ㄴ or ㅇ)/ㄱ/ㅏ/ㄴ/ㄷ/ㅏ or n/a/n/(un or ung)/g/an/d/a) corresponding to speech data (e.g., I go) using an acoustic model. In the acoustic model, a phoneme candidate group ("ㄴ" or "ㅇ" or "un" or "ung") corresponding to "ㄴ" in the speech data corresponding to "는" can be stored (in the speech data, "는" and "능" have similar pronunciations, so "ㄴ" and "ㅇ" can be stored as phoneme candidates).

According to various embodiments of the present invention, a language model may mean a database in which information used to model a linguistic order relationship such as words or syllables is stored using phonemes corresponding to speech data extracted from an acoustic model. In the language model, speech data and words including phonemes included in a phoneme candidate group may be respectively mapped and stored. Furthermore, in the language model, the probability that words including phonemes included in a phoneme candidate group will match speech data may be respectively mapped and stored. The intelligent server (320) may select a word corresponding to speech data based on the probability that each of the words including phonemes included in the phoneme candidate group will match speech data.

According to various embodiments of the present invention, the intelligent server (320) can convert voice data and transmit the generated text data to the cloud server (330).

According to various embodiments of the present invention, the cloud server (330) may perform natural language understanding (NLU) processing using text data received from the intelligent server (320). Natural language understanding processing may mean an operation of determining the user's speech intention using converted text data. The cloud server (330) may determine the user's intention by performing syntactic analysis or semantic analysis on the text data. The grammatical analysis may divide the user input into grammatical units (e.g., words, phrases, or morphemes, etc.) and determine which grammatical elements the divided units have. The semantic analysis may be performed using semantic matching, rule matching, formula matching, etc. The cloud server (330) can determine the user's intent and parameters by using matching rules divided into a domain, an intent, and a parameter (or slot) required to identify the intent for distinguishing a service matching the intent corresponding to the user input. For example, one domain (e.g., alarm) can include multiple intents (e.g., setting an alarm, releasing an alarm, etc.), and one intent can include multiple parameters (e.g., time, number of repetitions, alarm sound, etc.).

According to various embodiments of the present invention, the cloud server (330) may determine the meaning of words extracted from user input using linguistic features (e.g., grammatical elements) such as morphemes and phrases, and may determine the user's intention by matching the meaning of the identified words to domains and intentions. For example, the cloud server (330) may determine the user's intention by calculating how many words extracted from the user input are included in each domain and intention. According to one embodiment, the cloud server (330) may determine the parameter of the user input using the words that are the basis for determining the intention. According to one embodiment, the cloud server (330) may determine the user's intention using a natural language recognition database (not shown) in which linguistic features for determining the intention of the user input are stored. According to another embodiment, the cloud server (330) may determine the user's intention using a personal language model (PLM). For example, the cloud server (330) can determine the user's intention using personalized information (e.g., contact list, music list).

According to various embodiments of the present invention, various operations that can be performed in the cloud server (330) may also be performed in the intelligent server (320).

According to various embodiments of the present invention, the cloud server (330) can confirm the user's intention and the action appropriate for the user's intention determined through the natural language understanding action, and can confirm information on the parameters required to execute the action corresponding to the user's intention. If the cloud server (330) lacks the parameters required to execute the action corresponding to the user's intention, the cloud server (330) can request the parameters from the user terminal (310). If the parameters required to execute the action corresponding to the user's intention exist, the cloud server (330) can execute the action corresponding to the user's intention and transmit the execution result to the user terminal (310).

According to various embodiments of the present invention, the cloud server (330) may transmit a signal requesting execution of an operation corresponding to a user's intention to a third-party server (340). The third-party server (340) may mean a server operated by a different entity from the entity operating the intelligent server (320) and the cloud server (330). The third-party server (340) may execute an operation corresponding to the user's intention (e.g., ordering a pizza) and transmit the execution result to the user terminal (310).

According to various embodiments of the present invention, the intelligent server (320) may select a language model (or language model) to be used for additional speech recognition based on text data generated through conversion of speech data corresponding to a user's speech. The intelligent server (320) may include a basic language model (or generic language model) designated to be used basically in performing speech recognition and a plurality of domain-based language models (or domain-based language models) corresponding to domains distinguished by each of the services provided based on speech recognition. The intelligent server (320) may select one domain-based language model among the plurality of domain-based language models based on text data converted using the basic language model. The intelligent server (320) may perform speech recognition using the basic language model and the selected domain-based language model. The above contents are described in FIGS. 4 to 6c below.

FIG. 4 is a block diagram illustrating an intelligent server (320) that performs voice recognition according to various embodiments of the present invention.

Referring to FIG. 4, an intelligent server (e.g., intelligent server (320) of FIG. 3) that performs voice recognition according to various embodiments of the present invention may include a communication module (410), a processor (420), and a memory (430).

According to various embodiments of the present invention, the communication module (410) may receive first voice data corresponding to the user's speech from an external electronic device (e.g., the user terminal (310) of FIG. 3). The first voice data may mean data obtained by preprocessing voice data corresponding to the user's speech received using a microphone of the user terminal (310) (e.g., the input device (150) of FIG. 1).

According to various embodiments of the present invention, the communication module (410) can transmit text data or a voice recognition processing result obtained by performing a text conversion operation on voice data to an external electronic device (e.g., the user terminal (310) of FIG. 3 or the cloud server (340) of FIG. 3).

According to various embodiments of the present invention, the memory (430) may store an acoustic model, a basic language model designated to be used basically in performing speech recognition, and a domain-based language model corresponding to domains distinguished by services provided based on speech recognition. The acoustic model may mean a database in which feature vectors obtained by performing processing on speech data and phoneme candidate groups including phonemes corresponding to the feature vectors are mapped and stored. The language model may mean a database in which information used to model linguistic order relationships such as words or syllables is stored by using phonemes corresponding to speech data extracted from the acoustic model.

According to various embodiments of the present invention, the basic language model may mean a language model that can be basically used while performing speech recognition.

According to various embodiments of the present invention, a domain-based language model may mean a language model corresponding to each of the domains. A domain may be distinguished by a service provided based on speech recognition. For example, each of various services provided based on speech recognition (e.g., pizza ordering, accommodation reservation, traffic information provision) may have its own domain. For example, domain-based language models may include a language model related to pizza ordering, a language model related to accommodation reservation, a language model related to traffic information provision, etc. According to various embodiments of the present invention, a domain-based language model may be implemented by an operating side of a third-party server (e.g., the third-party server (340) of FIG. 3) differently from a basic language model. If the frequency of use of data included in a domain-based language model is equal to or greater than a set value (or exceeds), the data included in the domain-based language model may be updated to be included in the basic language model.

According to various embodiments of the present invention, the processor (420) may perform session management to manage a session. A session may mean an operation such as exchanging data (voice data, text data, parameter information required for a service, etc.) between an intelligent server (320) and a user terminal (e.g., the user terminal (310) of FIG. 3) to provide various services using voice recognition. A session may include a series of operations for performing various services that may be provided based on voice recognition. For example, a session may be defined as an operation from when a domain-based language model associated with a determined domain is selected and the selected domain-based language model is connected to an ASR (e.g., an ASR (510) of FIG. 5) until a service provided using voice recognition is terminated.

According to various embodiments of the present invention, the processor (420) can convert first voice data corresponding to a user's speech received through the communication module (410) into first text data. The processor (420) can generate the first text data using a basic language model. The processor (420) can determine a domain corresponding to the text data based on the generated text data.

According to various embodiments of the present invention, a domain may be an identifier that can be distinguished according to a service provided based on voice recognition. For example, each of various services provided based on voice recognition (e.g., pizza ordering, accommodation reservation, and transportation information provision) may have a unique domain.

According to various embodiments of the present invention, a domain may be created by a developer (or an operator) of a third-party server (340). The developer of the third-party server (340) may define at least one entity, which means parameter information that can be extracted from text data converted from voice data. For example, “pizza menu,” which is a parameter that can be extracted from voice data of a user who intends to “order pizza,” may be an entity name, and words included in the entity may be potato pizza, shrimp pizza, or pepperoni pizza. Words included in an entity added by the developer (or operator) of the third-party server (340) may be managed as a vocabulary. Entity information including an entity name and words included in the entity may be defined as a vocabulary. When a developer of a third-party server (340) makes a request for learning a language model based on vocabulary, the processor (420) converts words into phonemes and learns a language model through these, so that the language model can be added to a domain-based language model. According to another embodiment, when a developer of a third-party server (340) adds or deletes an entity, the processor (420) detects the change in the entity and learns the language model according to the changed entity even without a separate request for learning a language model.

According to various embodiments of the present invention, the processor (420) can create a session connected to the determined domain. The processor (420) can manage an operation of processing a service based on speech recognition through the created session. The processor (420) can select a domain-based language model corresponding to the determined domain among domain-based language models. For example, the processor (420) can determine a user's intention (e.g., order a pizza) based on first text data (e.g., order a pizza) and determine a domain (e.g., a pizza ordering service) that matches the user's intention. The processor (420) can select a domain-based language model (e.g., a language model specialized for ordering a pizza) associated with the determined domain.

According to various embodiments of the present invention, the processor (420) may transmit information about the session being used to an external electronic device (e.g., the user terminal (310) of FIG. 3). The external electronic device (310) may output information about the session using a display or speaker to inform the user of the information about the session.

According to various embodiments of the present invention, the processor (420) can check whether the first text data includes an identifier of a domain. For example, if a specified text format (e.g., (domain identifier), (user's intention)) is stored in the memory (430), the processor (420) can compare the first text data with the specified text format. The processor (420) can check the identifier of the domain based on the comparison result. As another example, the processor (420) can divide the first text data into specific units (e.g., words, phrases, etc.) and check whether the identifier of the domain exists in the divided data. If the identifier of the domain exists in the first text data, the processor (420) can determine the domain based on the result of the verification of the identifier of the domain.

According to various embodiments of the present invention, if a domain identifier does not exist in the first text data, the processor (420) may determine a user's intention using the first text data, and determine a task matching the user's intention. The processor (420) may determine a domain corresponding to the determined task. For example, the processor (420) may compare at least a portion (e.g., sentences, phrases, words, etc.) included in the first text data with comparison information (e.g., sentences, phrases, words, etc. corresponding to functions) stored in the memory (430), and if a portion of the first text data matches the comparison information, the processor (420) may determine a task matching the user's intention. The processor (420) may determine a domain corresponding to the task matching the user's intention. As another example, the processor (420) may divide the first text data into specific units (e.g., words, phrases, etc.) and determine whether the divided data (e.g., objects or verbs) includes comparison information stored in the memory (430). The processor (420) can determine a task matching the user's intention based on whether comparison information is included. The processor (420) can determine a domain corresponding to the task matching the user's intention.

According to various embodiments of the present invention, the processor (420) may receive an additional utterance of the user, perform speech recognition on second speech data corresponding to the additional utterance of the user using the base language model and the selected domain-based language model, and generate second text data corresponding to the second speech data. For example, the additional utterance of the user may be performed to input parameters (e.g., type, size, number of pizzas, etc.) for performing a service matching the intention corresponding to the user utterance. When performing text conversion on the second speech data corresponding to the additional utterance of the user, the processor (420) may utilize the base language model and the selected domain-based language model (e.g., a language model specialized for ordering pizza).

According to various embodiments of the present invention, when performing text conversion using a basic language model and a selected domain-based language model, the text conversion result when using the basic language model may be different from the text conversion result when using the selected domain-based language model. In this case, the processor (420) may perform text conversion based on the weight applied to the basic language model and the weight applied to the selected domain-based language model.

According to various embodiments of the present invention, the processor (420) may change the session by an additional user utterance while performing speech recognition using the generated session. For example, the processor (420) may receive third speech data corresponding to an additional user utterance (e.g., a request for bus route information) while performing speech recognition using a session related to a pizza order. The processor (420) may generate third text data corresponding to the third speech data based on a domain-based language model associated with a basic language model and a determined domain. The processor (420) may determine a domain corresponding to the third text data based on the third text data. If the domain corresponding to the first text data (e.g., a pizza order) and the domain corresponding to the third text data (e.g., a request for bus route information) are different from each other, the processor (420) may change the session related to the domain corresponding to the first text data to the session related to the domain corresponding to the third text data, and perform speech recognition using the session related to the domain corresponding to the third text data. According to various embodiments of the present invention, the processor (420) can select a domain-based language model related to a domain corresponding to third text data, and perform speech recognition based on the base language model and the domain-based language model related to the domain corresponding to the third text data.

According to various embodiments of the present invention, the processor (420) may determine a domain based on the first text data, and may perform text conversion on the first speech data again using a domain-based language model and a basic language model related to the determined domain. When conversion is performed based on the basic language model and the domain-based language model related to the determined domain, the conversion can be performed more accurately. When the first text data is changed, the processor (420) may transmit the changed text data to an external electronic device (e.g., the user terminal (310) of FIG. 3).

According to various embodiments of the present invention, the processor (420) may change text data (e.g., 1945 burger) into audio data. In this case, the processor (420) may generate audio data corresponding to words included in the domain-based language model and the basic language model related to the determined domain, and transmit the audio data to the user terminal (310). When performing audio conversion using the basic language model and the selected domain-based language model, the audio conversion result (e.g., one thousand nine hundred and forty-five burger or one thousand and nine hundred and forty-five burger) when using the basic language model may be different from the audio conversion result (e.g., one thousand nine hundred and forty-five burger or one thousand nine hundred and forty-five burger) when using the selected domain-based language model. In this case, the processor (420) may select one of the basic language model or the domain-based language model based on the weight applied to the basic language model and the weight applied to the selected domain-based language model, and perform audio conversion using the selected language model.

FIG. 5a is a block diagram illustrating a processor (420) in an intelligent server according to various embodiments of the present invention.

Referring to FIG. 5A, an intelligent server (e.g., a processor (420) of an intelligent server (320) of FIG. 3) may include an ASR module (510), a natural language understanding module (520), a base language model (530), a plurality of domain-based language models (541, 543, 545), and a session manager (550). According to various embodiments of the present invention, the ASR module (510), the natural language understanding module (520), the base language model (530), the plurality of domain-based language models (541, 543, 545), and the session manager (550) may be implemented in software on the processor (420).

According to various embodiments of the present invention, the ASR module (510) can convert voice data received from an external electronic device (e.g., the user terminal (310) of FIG. 3) into text data. The ASR module (510) can transmit the converted text data to the session manager (550).

According to various embodiments of the present invention, the session manager (550) can determine a domain using the converted text data. The session manager (550) can select a domain-based language model related to the determined domain among the domain-based language models (541, 543, 545). The session manager (550) can control the ASR module (510) to perform text conversion using the selected domain-based language model and the basic language model (530).

According to various embodiments of the present invention, the ASR module (510) can transmit converted text data to the NLU module (520).

According to various embodiments of the present invention, the natural language understanding module (520) can use linguistic features (e.g., grammatical elements) such as morphemes and phrases included in the text data to understand the meaning of words extracted from the text data and determine the user's intention. The natural language understanding module (520) can confirm the user's intention determined through the natural language understanding operation and an operation appropriate to the user's intention, and can confirm information on parameters required to execute the operation corresponding to the user's intention. When the session manager (550) receives a signal from the natural language understanding module (520) notifying that the parameters required to execute the operation corresponding to the user's intention are insufficient, the session manager (550) can request the parameters from the user terminal (310).

According to various embodiments of the present invention, the session manager (550) may receive information about the user's intention, an action suitable for the user's intention, and parameters from the natural language understanding module (520), and transmit the received information to a third party server (e.g., the third party server (340) of FIG. 3). The third party server (340) may receive information about the user's intention, an action suitable for the user's intention, and parameters, and perform a service for voice recognition using the received information. The third party server (340) may transmit the service execution result to the user terminal (310), and the user terminal (310) may output the service execution result using a display or a speaker.

According to various embodiments of the present invention, the natural language understanding module (520) may be implemented in the intelligent server (320), but may also be implemented in a cloud server (330) rather than the intelligent server (320). If the natural language understanding module (520) is implemented in the cloud server (330), text data generated by the ASR module (510) may be transmitted to the cloud server (330).

FIG. 5b is a block diagram illustrating a processor (420) in an intelligent server according to another embodiment of the present invention.

Referring to FIG. 5b, the processor (420) of the intelligent server (e.g., the intelligent server (320) of FIG. 3) may include an ASR module (510), a natural language understanding module (520), a base language model (530), a plurality of domain-based language models (541, 543, 545), a session manager (550), a language model generation module (560), and an integrated language model (570). Descriptions of components overlapping with those of FIG. 5a are omitted. According to various embodiments of the present invention, the ASR module (510), the natural language understanding module (520), the base language model (530), a plurality of domain-based language models (541, 543, 545), the session manager (550), the language model generation module (560), and the integrated language model (570) may be implemented in software on the processor (420).

According to various embodiments of the present invention, the language model generation module (560) can generate an integrated language model (570) using a base language model (530) and at least one language model associated with a currently running session among a plurality of domain-based language models (541, 543, 545).

According to various embodiments of the present invention, the ASR module (510) can convert voice data received from an external electronic device (e.g., the user terminal (310) of FIG. 3) into text data using the generated integrated language model.

According to various embodiments of the present invention, the ASR module (510) illustrated in FIG. 5a can transmit the received voice data to all language models (e.g., the base language model (530) and the domain-based language model (541)) connected to the ASR module (510). The ASR module (510) illustrated in FIG. 5b can transmit the received voice data to one integrated language model (570) connected to the ASR module (510) and convert the voice data into text data using the integrated language model (570).

FIGS. 6A to 6C are diagrams illustrating an example of selecting a language model and performing speech recognition using the selected language model in an intelligent server (320) according to various embodiments of the present invention.

Referring to FIGS. 6A to 6C, within the processor (420), an ASR module (510) implemented in software, a basic language model (e.g., the basic language model (530) of FIG. 5), and domain-based language models (541, 543, 545) corresponding to each of the domains divided into services provided based on speech recognition may be included.

FIG. 6A is a diagram illustrating an example of an ASR module (510) converting voice data into text data using a basic language model (530). According to various embodiments of the present invention, the ASR module (510) may convert first voice data transmitted by an external electronic device (e.g., user terminal (310) of FIG. 3) into first text data using a basic language model (530) before a session is created.

FIG. 6b is a diagram illustrating an embodiment in which the ASR module (510) converts second speech data into second text data using the base language model (530) and the first domain-based language model (541). According to various embodiments of the present invention, a session manager (e.g., the session manager (550) of FIG. 5) may determine a domain to be used for speech recognition using the first text data. The session manager (550) may select a domain-based language model (e.g., the first domain-based language model (541)) related to the determined domain. The session manager (550) may control the ASR module (510) to convert the second speech data into second text data using the selected domain-based language model (541) and the base language model (530).

FIG. 6c is a diagram illustrating an embodiment in which the ASR module (510) converts speech data into text data using the basic language model (530) and the second domain-based language model (543). According to various embodiments of the present invention, the session manager (550) may change the session by an additional user utterance while performing speech recognition using the generated session. For example, the session manager (550) may receive third speech data corresponding to an additional user utterance (e.g., a request for bus route information) while performing speech recognition using a session related to a pizza order. The session manager (550) may generate third text data corresponding to the third speech data based on the basic language model and a domain-based language model corresponding to a previously determined domain (e.g., the first domain-based language model (541) of FIG. 6b). The session manager (550) may determine a domain corresponding to the third text data based on the third text data. The session manager (550) may change a session related to the domain corresponding to the first text data (e.g., pizza order) and a domain corresponding to the third text data (e.g., bus route information request) from a session related to the domain corresponding to the first text data to a session related to the domain corresponding to the third text data, and perform speech recognition using the session related to the domain corresponding to the third text data, if the domain corresponding to the first text data and the domain-based language model related to the domain corresponding to the third text data are different from each other. According to various embodiments of the present invention, the session manager (550) may select a domain-based language model related to the domain corresponding to the third text data, and perform speech recognition based on a base language model and the domain-based language model related to the domain corresponding to the third text data.

FIG. 7 is a block diagram illustrating an electronic device (310) that performs voice recognition according to various embodiments of the present invention.

Referring to FIG. 7, an electronic device (310) performing voice recognition according to various embodiments of the present invention may include a communication module (710) (e.g., the communication module (190) of FIG. 1), a display (720) (e.g., the display device (160) of FIG. 1), a processor (730) (e.g., the processor (120) of FIG. 1), and a microphone (740) (e.g., the input device (150) of FIG. 1).

According to various embodiments of the present invention, the microphone (740) may receive a user's speech of the electronic device (310) and generate voice data corresponding to the speech. The generated voice data may be transmitted to an intelligent server (e.g., the intelligent server (320) of FIG. 3) by the communication module (710) after undergoing a preprocessing process. The processor (730) may perform preprocessing on the voice data by performing at least one of an operation of removing an echo included in the voice data, an operation of removing a noise included in the voice data, or an operation of adjusting the volume of a voice included in the voice data.

According to various embodiments of the present invention, the processor (730) may transmit preprocessed voice data to the intelligent server (320). As described in FIGS. 4 to 6C, the intelligent server (320) may receive the voice data and execute a series of operations related to voice recognition and a service based on voice recognition using the voice data. According to various embodiments of the present invention, the intelligent server (320) may determine a domain for performing voice recognition and generate a session associated with the determined domain. Information about the generated session and information related to the domain determined by the intelligent server (320) may be transmitted to the electronic device (310).

According to various embodiments of the present invention, the processor (730) may receive information about a session generated by an intelligent server (320) or a cloud server (e.g., the cloud server (330) of FIG. 3) and information related to a domain determined by the intelligent server (320).

According to various embodiments of the present invention, the processor (730) may display information about the generated session or information related to the domain on the display (720). The information about the generated session or information related to the domain may be displayed as a user interface implemented in a visual form such as an icon. For example, the information about the generated session or information related to the domain may be displayed in text form on the top of the display (720) on the top of the screen. As another example, the information about the generated session or information related to the domain may be displayed in the form of an icon on a part of the display (720). According to various embodiments of the present invention, when the generated session is terminated due to various causes such as a user's speech and another session is executed, the information about the generated session or information related to the domain may be displayed on the display (720). Various embodiments of outputting the information about the session or information related to the domain will be described later with reference to FIGS. 10, 11A, and 11B.

According to various embodiments of the present invention, the processor (730) may receive text data generated using a domain-based language model and a basic language model related to a determined domain from the intelligent server (320). The processor (730) may output the received text data through a display (720) or a speaker (not shown).

FIGS. 8A to 8D are diagrams illustrating examples of selecting a language model and performing speech recognition using the selected language model in an intelligent server (320) and an electronic device (310) according to various embodiments of the present invention.

The embodiments illustrated in FIGS. 8A to 8D are embodiments of a screen displayed on a display (e.g., a display (720) of FIG. 7) of a user terminal (e.g., a user terminal (310) of FIG. 3) in an operation of selecting a domain-based language model illustrated in FIGS. 6A to 6C and utilizing the selected domain-based language model.

Referring to FIG. 8A, a user may perform an utterance including a specific intention (order pizza), and the user terminal (310) may generate first voice data corresponding to the user's utterance (831) (e.g., "Order pizza from Domin"). The first voice data may undergo a preprocessing process and be transmitted to an intelligent server (e.g., the intelligent server (320) of FIG. 3). An ASR module (510) executed in a processor (e.g., the processor (420) of FIG. 4) of the intelligent server (320) may convert the received first voice data into first text data using a basic language model (530). A session manager (e.g., the session manager (550) of FIG. 5) executed in the processor (420) may determine a domain (e.g., pizza delivery) based on the first text data, and generate a session for providing a service (e.g., a pizza delivery-related service) based on voice recognition corresponding to the determined domain. The session manager (550) can determine a domain (e.g., pizza delivery) related to the session based on the generated first text data, and select a domain-based language model (e.g., a local pizza language model) (821) related to the determined domain among the domain-based language models (821, 823, 825).

According to various embodiments of the present invention, the session manager (550) transmits information about the created session to the user terminal (310), and the user terminal (310) can display information (833) related to the domain (e.g., calling Domino's Pizza) on a display (e.g., display (720) of FIG. 7).

According to various embodiments of the present invention, a processor (e.g., processor (730) of FIG. 7) may output domain-related information (833) on a display (720). The domain-related information (833) may include a name of the domain and a name of a third party providing the domain. The domain-related information (833) may be output in the form of a user interface.

Referring to FIG. 8b, the user terminal (310) may generate second voice data (841) corresponding to an additional user utterance (e.g., a specific utterance related to a pizza order). The second voice data (841) may be preprocessed and transmitted to an intelligent server (e.g., the intelligent server (320) of FIG. 3). An ASR module (e.g., the ASR module (510) of FIG. 5) running on a processor (e.g., the processor (420) of FIG. 4) of the intelligent server (320) may perform text conversion on the second voice data (841) based on a basic language model (530) and a domain-based language model (821) related to a determined domain, and generate second text data (843). NLU (e.g., NLU (520) of FIG. 5) can determine the user's intention using the second text data (843) and extract parameters necessary for performing a task matching the user's intention based on the second text data (843). The extracted parameters and second text data (843) can be transmitted to a third-party server (e.g., third-party server (340) of FIG. 3), and the third-party server (340) can perform operations related to a service related to voice recognition based on the received information.

According to various embodiments of the present invention, the session manager (550) can transmit second text data (843) to the user terminal (310). The user terminal (310) can display the second text data (843) on the display (720).

According to various embodiments of the present invention, the processor (730) can output domain-related information (845) on the display (720). The domain-related information (845) can include the name of the domain and the name of a third party providing the domain. The domain-related information (845) can be output in the form of a user interface.

According to various embodiments of the present invention, the session manager (550) may terminate a currently running session and change to (or create) another session based on a user input. Referring to FIG. 8c, the user terminal (310) may generate third voice data (851) corresponding to an additional user utterance (e.g., a request for information related to a bus route). The third voice data (851) may be preprocessed and transmitted to an intelligent server (e.g., the intelligent server (320) of FIG. 3). The ASR module (510) may convert the third voice data (851) into third text data using the basic language model (530) and the domain-based language model (821) related to the previously determined domain.

According to various embodiments of the present invention, the session manager (550) may determine a domain corresponding to the third text data based on the third text data. If the domain corresponding to the first text data (e.g., pizza order) and the domain corresponding to the third text data (e.g., bus route information request) are different from each other, the session manager (550) may change (or create) a session related to the domain corresponding to the first text data to a session related to the domain corresponding to the third text data, and perform speech recognition using the session related to the domain corresponding to the third text data.

According to various embodiments of the present invention, the processor (730) can output domain-related information (853) on the display (720). The domain-related information (853) can include the name of the domain and the name of a third party providing the domain. The domain-related information (853) can be output in the form of a user interface.

According to various embodiments of the present invention, the session manager (550) can terminate a previously running session and execute a session related to a changed domain. The session manager (550) can transmit information notifying a change in the session to the user terminal (310). The user terminal (310) can display information (853) related to the domain including the information notifying a change in the session on the display (720).

According to various embodiments of the present invention, the session manager (550) may select a domain-based language model (825) related to a changed domain among domain-based language models (821, 823, 25), and perform speech recognition using the selected domain-based language model (e.g., bus route language model) (825) and the base language model (530).

Referring to FIG. 8d, the user terminal (310) may receive fourth voice data (861) related to an additional utterance of the user (e.g., an utterance requesting bus route information). The fourth voice data (861) may be preprocessed and transmitted to an intelligent server (e.g., an intelligent server (320) of FIG. 3). The ASR module (510) may convert the fourth voice data (861) into fourth text data using a basic language model (530) and a domain-based language model (825) related to a determined domain. The session manager (550) may transmit the fourth text data to a third-party server (e.g., a third-party server (340) of FIG. 3), and the third-party server (340) may process a process related to voice recognition using the received data and transmit the processing result to the user terminal (310). The user terminal (310) can display the processing result on the display (720).

According to various embodiments of the present invention, the processor (730) can output domain-related information (863) on the display (720). The domain-related information (863) can include the name of the domain and the name of a third party providing the domain. The domain-related information (863) can be output in the form of a user interface.

FIG. 9 is a diagram illustrating an example of selecting multiple language models distinguished according to a service and performing speech recognition using the selected language models in an intelligent server and electronic device according to various embodiments of the present invention.

According to various embodiments of the present invention, an intelligent server (e.g., intelligent server (320) of FIG. 3) can perform speech recognition using a plurality of domain-based language models and a basic language model in one session that processes a series of operations related to speech recognition.

Referring to FIG. 9, a user terminal (310) (e.g., the electronic device (101) of FIG. 1) may generate first voice data corresponding to an utterance (931) including a specific intention of the user (e.g., travel plan). The first voice data may be transmitted to an intelligent server (e.g., the intelligent server (320) of FIG. 3) after going through a preprocessing process. An ASR module (510) implemented on a processor (e.g., the processor (420) of FIG. 4) of the intelligent server (320) may convert the received first voice data into first text data using a basic language model (530). A session manager (e.g., the session manager (550) of FIG. 5) implemented on the processor (420) may generate a session for providing a service based on voice recognition based on the first text data. The session manager (550) may identify a plurality of domains related to the session based on the first text data. For example, the session manager (550) can identify multiple domains (e.g., accommodation reservation, rental car reservation, airline ticket reservation) that match the user's intention (e.g., travel plan). The session manager (550) can select domain-based language models (921, 923, 925) corresponding to each of the multiple domains, and perform speech recognition using the multiple domain-based language models (921, 923, 925) and the base language model (530).

According to various embodiments of the present invention, a processor (e.g., processor (730) of FIG. 7) may output information related to a domain on a display (720). The information related to the domain may include a name of the domain and a name of a third party providing the domain. The information related to the domain may be output in the form of a user interface. Referring to FIG. 9, the processor (730) may display icons (941, 943, 945) representing each of the determined plurality of domains.

FIGS. 10A to 10B are diagrams illustrating an example of outputting information about a connected session in an electronic device (310) according to various embodiments of the present invention.

Referring to FIGS. 10A and 10B , an electronic device (310) performing voice recognition according to various embodiments of the present invention may receive information about a currently connected session from an intelligent server (320) or a cloud server (e.g., a cloud server (330) of FIG. 3) and output the received information using a display (e.g., a display (720) of FIG. 7).

Referring to FIG. 10A, the electronic device (310) may display information about the currently connected session in the form of an icon (1010) on a screen for providing a service (e.g., a service for searching and displaying a list of restaurants) using a voice input from a user of the electronic device (310). The service screen using a voice input may be implemented using a color similar to the color of the icon as the background color.

Referring to FIG. 10b, the electronic device (310) may display a screen for providing a service (e.g., a pizza ordering service) using a voice input of a user of the electronic device (310) in a pop-up form, while displaying information about the currently connected session in the form of an icon (1020).

According to various embodiments of the present invention, when a created session is terminated due to various reasons such as a user's speech and another session is executed, information about the changed session can be displayed on the display (720).

FIGS. 11A and 11B are diagrams illustrating an example of outputting information about a connected session in an electronic device (310) according to various embodiments of the present invention.

Referring to FIGS. 11A and 11B , an electronic device (310) according to various embodiments of the present invention may include a microphone for receiving a user's voice, and may be an intelligent speaker connected to an intelligent server (320) or a cloud server (330). The electronic device (310) may receive information about a currently connected session from an intelligent server (320) or a cloud server (e.g., the cloud server (330) of FIG. 3), and output the received information using an audio module (e.g., the audio module (170) of FIG. 1).

FIGS. 11A to 11D can be applied to both cases where the electronic device (310) according to various embodiments of the present invention does not have a display (e.g., the display (720) of FIG. 7) and thus cannot display information about a connected session, and cases where the electronic device (310) has a display (720).

According to various embodiments of the present invention, information about a connected session to be used for voice recognition may be output using an audio module (e.g., audio module (170) of FIG. 1) provided by an electronic device (310).

According to various embodiments of the present invention, when a generated session (e.g., a general voice recognition session) is terminated due to various causes such as a user's utterance, and another session (e.g., a pizza ordering session) is executed, information about the changed session may be output using an audio module (e.g., an audio module (170) of FIG. 1) provided to the electronic device (310). For example, a voice output in a session before the change and a voice output in a session after the change may have different voice characteristics including intonation, speed, velocity, gender of the speaker (male, female), or age of the speaker (child, elderly).

According to various embodiments of the present invention, the electronic device (310) can set the characteristics of the voice output using the audio module (170) differently so that the user of the electronic device (310) can recognize a change in the session.

Referring to FIG. 11a, the speaker corresponding to the currently connected session, which is a general voice session, may be a female. If the currently connected session is a general voice session, the electronic device (310) may output voice using a female speaker.

Referring to FIG. 11b, the speaker corresponding to the changed session, the pizza ordering session, may be male. If the currently connected session is the pizza ordering session, the electronic device (310) may output voice using a male speaker.

FIGS. 12A and 12B are diagrams illustrating an example of outputting information about a connected session in an electronic device (310) according to various embodiments of the present invention.

Referring to FIGS. 12A and 12B , an electronic device (310) according to various embodiments of the present invention may include a microphone for receiving a user's voice, and may be an intelligent speaker connected to an intelligent server (320) or a cloud server (330). The electronic device (310) may receive information about a currently connected session from an intelligent server (320) or a cloud server (e.g., a cloud server (330) of FIG. 3 ), and output the received information using an LED light (1210) included in the electronic device (310).

According to various embodiments of the present invention, the electronic device (310) can control the LED light (1210) to change the characteristics of the light output by the LED light (1210) according to the connected session. The characteristics of the light output by the LED light (1210) can include the degree of blinking of the light, the color of the light, and the intensity of the light.

According to various embodiments of the present invention, the electronic device (310) can control the LED light (1210) to output light differently for each of a plurality of sessions. The electronic device (310) can set the characteristics of the light output using the LED light (1210) differently so that the user of the electronic device (310) can recognize a change in the session.

Referring to FIG. 12a, the color of the light corresponding to the currently connected session, which is a general voice session, may be blue. If the currently connected session is a general voice session, the electronic device (310) may control the LED light (1210) to output blue light.

Referring to FIG. 12b, the color of the light corresponding to the currently connected session, which is a pizza ordering session, may be red. If the currently connected session is a pizza ordering session, the electronic device (310) may control the LED light (1210) to output red light.

A system according to various embodiments of the present invention comprises a network interface, at least one processor operatively connected to the network interface, and at least one memory operatively connected to the processor, wherein the at least one memory is configured to store a generic language model and a plurality of domain-based language models for automatic speech recognition (ASR), and wherein the at least one memory is configured to, when executed, cause the processor to receive first speech data from an external device including a microphone through the network interface, use the generic language model but not the domain-based language models, process the first speech data with the ASR to generate first text data associated with at least one task to be performed by the system, determine a domain associated with the task based at least in part on the first text data, select one domain-based language model associated with the determined domain from among the domain-based language models, and, after receiving the first speech data and performing the task, Before the execution, instructions can be stored for receiving second voice data from the external device through the network interface, and processing the second voice data with the ARS using the generic language model and the selected domain-based language model with the ASR to generate second text data.

In a system according to various embodiments of the present invention, the instructions may cause the processor to cause the external device to provide a user interface representing the determined domain.

In a system according to various embodiments of the present invention, the user interface may include at least one visual user interface or a voice-based user interface.

In a system according to various embodiments of the present invention, the instructions may cause the processor to provide the visual user interface on the display.

In a system according to various embodiments of the present invention, the external device includes at least one light emitting diode (LED), and the instructions may cause the processor to provide the visual user interface using the LED.

In a system according to various embodiments of the present invention, the external device includes a speaker, and the instructions may cause the processor to provide the voice-based user interface through the speaker.

In a system according to various embodiments of the present invention, the instructions may cause the processor to use a first text to speech model before the domain determination and to use a second TTS model after the domain determination to provide voice through the external device.

In a system according to various embodiments of the present invention, the instructions may cause the processor to maintain a combination of the generic language model and the selected domain-based language model until the task is performed.

In a system according to various embodiments of the present invention, the domain-based language models can be generated and classified according to the type of the service or the content provider.

In a system according to various embodiments of the present invention, the instructions may cause the processor to determine whether the first text data includes an identifier of the domain, and if the identifier of the domain is included in the first text data, select the generic language model corresponding to the included identifier.

In a system according to various embodiments of the present invention, the instructions may cause the processor to determine an intention of the user based on the first text data, if an identifier of the domain is not included in the first text data, and to determine at least one domain corresponding to the intention.

In a system according to various embodiments of the present invention, the instructions may cause the processor to receive third voice data corresponding to a user's speech through the communication module, generate text data corresponding to the third voice data, and change the domain based on the text data corresponding to the third voice data.

In a system according to various embodiments of the present invention, the instructions enable the processor to perform speech recognition using a domain-based language model and the generic language model corresponding to the changed domain when the domain is changed.

In a system according to various embodiments of the present invention, the instructions enable the processor to modify the first text data using the selected domain-based language model and the generic language model.

In a system according to various embodiments of the present invention, the instructions may cause the processor to transmit data notifying a change in the first text data to the external device when the first text data is changed.

According to various embodiments of the present invention, an electronic device includes: a communication module; at least one processor; and a memory storing a generic language model that is basically designated to be used in performing speech recognition, and domain-based language models that respectively correspond to domains distinguished by a service provided based on speech recognition, wherein the processor is configured to receive first speech data corresponding to a user's speech through the communication module, generate text data corresponding to the first speech data using the generic language model, determine a domain to be used for the speech recognition based on the generated text data, establish a session connected to the determined domain, select a domain-based language model corresponding to the determined domain, and perform speech recognition on second speech data corresponding to an additional speech of the user using the generic language model and the selected domain-based language model.

In an electronic device according to various embodiments of the present invention, domain-based language models can be generated separately according to content providers.

In an electronic device according to various embodiments of the present invention, the processor may transmit data notifying the established session to an external electronic device corresponding to the user.

A system according to various embodiments of the present invention comprises a network interface, at least one processor operatively connected to the network interface, and at least one memory operatively connected to the processor; wherein the at least one memory is configured to store a generic language model and a plurality of domain-based language models for automatic speech recognition (ASR), and wherein the at least one memory is configured to cause, when executed, the processor to receive first speech data from an external device including a microphone through the network interface, to process the first speech data using the generic language model but not using the domain-based language models, to generate first text data associated with at least one task to be performed by the system, to determine a domain associated with the task based at least in part on the first text data, to select one domain-based language model associated with the determined domain from among the domain-based language models, and after receiving the first speech data, Before performing the above task, instructions can be stored for receiving second voice data from the external device through the network interface, and processing the second voice data with the ARS using the selected domain-based language model with the ASR to generate second text data.

FIG. 13 is a flowchart illustrating an operation method of an electronic device (320) according to various embodiments of the present invention.

Referring to FIG. 13, in operation 1310, an electronic device (e.g., an intelligent server (320) of FIG. 3) according to various embodiments of the present invention may receive first voice data generated by a user's speech. For example, a communication module (410) of the electronic device (320) may receive first voice data corresponding to the user's speech from an external electronic device (e.g., a user terminal (310) of FIG. 3). The first voice data may mean data obtained by preprocessing voice data corresponding to the user's speech received using a microphone of the user terminal (310) (e.g., an input device (150) of FIG. 1).

According to various embodiments of the present invention, in operation 1320, the electronic device (320) may convert the first voice data using a basic language model (e.g., the basic language model (530) of FIG. 5) to generate the first text data. For example, the basic language model may mean a language model that can be basically used while performing voice recognition.

According to various embodiments of the present invention, a domain-based language model may mean a language model corresponding to each of the domains. A domain may be distinguished by a service provided based on speech recognition. For example, each of various services provided based on speech recognition (pizza ordering, accommodation reservation, traffic information provision) may have its own domain. For example, domain-based language models may include a language model related to pizza ordering, a language model related to accommodation reservation, a language model related to traffic information provision, etc. According to various embodiments of the present invention, a domain-based language model may be implemented by an operating side of a third-party server (e.g., the third-party server (340) of FIG. 3) differently from a basic language model. If the frequency of use of data included in a domain-based language model is equal to or greater than a set value (or exceeds), the data included in the domain-based language model may be updated to be included in the basic language model.

According to various embodiments of the present invention, in operation 1330, the electronic device (320) may determine a domain to be used for voice recognition based on the first text data. For example, the domain may be an identifier that can be distinguished by a service provided based on voice recognition. For example, each of various services provided based on voice recognition (pizza ordering, accommodation reservation, traffic information provision) may have a unique domain.

According to various embodiments of the present invention, a processor (e.g., processor (420) of FIG. 4) of an electronic device (320) may create a session connected to a determined domain. The created session may manage an operation for processing a service based on voice recognition. The processor (420) may select a domain-based language model corresponding to the determined domain among domain-based language models. For example, the processor (420) may determine a user's intention (e.g., pizza order) based on first text data (e.g., order pizza) and determine a domain (pizza order service) that matches the user's intention. The processor (420) may select a domain-based language model (e.g., a language model specialized for pizza ordering) corresponding to the determined domain.

In operation 1340, the electronic device (320) may select a domain-based language model (e.g., domain-based language model (541, 543, 545) of FIG. 5) corresponding to the determined domain.

In operation 1350, the electronic device (320) may perform speech recognition using the base language model (530) and the selected domain-based language model (e.g., the first domain-based language model (541)). For example, the processor (420) may receive an additional utterance of the user, and perform speech recognition on second speech data corresponding to the additional utterance of the user using the base language model and the selected domain-based language model. For example, the additional utterance of the user may be performed to input parameters (e.g., the type, size, number, etc. of pizzas) for performing a service matching an intention corresponding to the user utterance. When performing text conversion on the second speech data corresponding to the additional utterance of the user, the processor (420) may use the base language model and the selected domain-based language model (e.g., a language model specialized for ordering pizza).

According to various embodiments of the present invention, when performing text conversion using a basic language model and a selected domain-based language model, the text conversion result when using the basic language model may be different from the text conversion result when using the selected domain-based language model. In this case, the processor (420) may perform text conversion based on the weight applied to the basic language model and the weight applied to the selected domain-based language model.

An operating method of an electronic device performing speech recognition according to various embodiments of the present invention comprises: receiving first speech data from an external device including a microphone through a network interface; using a generic language model for automatic speech recognition (ASR), but not using a plurality of domain-based language models, processing the first speech data using the ASR to generate first text data associated with at least one task to be performed by the electronic device; determining a domain associated with the task based at least in part on the first text data; selecting one domain-based language model associated with the determined domain among the domain-based language models; receiving second speech data from the external device through the network interface after receiving the first speech data and before performing the task; processing the second speech data with the ASR using the generic language model and the selected one domain-based language model to generate second text data. May include generating actions.

The operating method of an electronic device according to various embodiments of the present invention may further include an operation of controlling the external device so that the external device provides a user interface indicating the determined domain.

In a method of operating an electronic device according to various embodiments of the present invention, the user interface may include at least one visual user interface or a voice-based user interface.

An operating method of an electronic device according to various embodiments of the present invention may further include an operation of controlling the external device to use a first text to speech model (TTS model) before determining the domain and to use a second TTS model after determining the domain, in order to provide voice through the external device.

The electronic devices according to various embodiments disclosed in this document may be devices of various forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliance devices. The electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to encompass various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can encompass all possible combinations of the items listed together in the corresponding phrase. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish the corresponding component from other corresponding components, and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" as used in this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the called at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, ‘non-transitory’ simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store ^TM ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities. According to various embodiments, one or more of the components or operations of the above-described components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the components of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. According to various embodiments, the operations performed by the module, program or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims (20)

In the system,
network interface;
At least one processor operatively connected to said network interface; and
At least one memory operatively connected to said processor;
wherein at least one memory is configured to store a generic language model for automatic speech recognition (ASR) and a plurality of domain-based language models,
The at least one memory, when executed, causes the processor to:
Receive first voice data from an external device including a microphone through the above network interface,
Using the above generic language model, but not using the domain-based language models, processing the first speech data with the ASR to generate first text data associated with at least one task to be performed by the system,
determining a domain associated with the task, based at least in part on the first text data;
Among the above domain-based language models, select one domain-based language model associated with the determined domain,
After receiving the first voice data and before performing the task, the second voice data is received from the external device through the network interface,
Store instructions for processing the second speech data using the ASR, the generic language model and the selected domain-based language model, to generate second text data;
The above memory is,
A system further storing an instruction for updating the generic language model so that data included in the plurality of domain-based language models, the usage frequency of which is greater than a set value, is included in the generic language model.
In paragraph 1,
The above instructions cause the processor to provide a user interface representing the determined domain to the external device.
In the second paragraph,
A system wherein the user interface comprises at least one visual user interface or a voice-based user interface.
In the third paragraph,
The above external device comprises a display,
The above instructions cause the system to cause the processor to provide the visual user interface on the display.
delete delete delete In paragraph 1,
The above instructions cause the processor to:
A system that maintains a combination of the generic language model and the selected domain-based language model until the above task is performed.
In paragraph 1,
The above domain-based language models
A system that is created by differentiating it according to the type of service or content provider.
In paragraph 1,
The above instructions cause the processor to
Verify that the above first text data contains the identifier of the above domain,
A system for selecting a domain-based language model corresponding to an identifier included in the first text data when the identifier of the domain is included in the first text data.
In Article 10,
The above instructions cause the processor to
If the identifier of the above domain is not included in the first text data, the user's intention is determined based on the first text data,
A system for determining at least one domain corresponding to the above intent.
In paragraph 1,
The above instructions, the processor,
Receive third voice data corresponding to the user's speech through the above network interface,
A system for generating text data corresponding to the third voice data and changing the domain based on the text data corresponding to the third voice data.
In Article 12,
The above instructions cause the processor to
A system that performs speech recognition using a domain-based language model corresponding to the changed domain and the generic language model when the above domain is changed.
delete delete delete delete delete delete delete

Images