KR20190037470A - Dialogue processing apparatus, vehicle having the same and dialogue processing method - Google Patents

Abstract

Provided are: a conversation system capable of providing a service matching the actual intention of a user or a service required by the user the most by accurately verifying the intention of the user based on various kinds of information including conversation with the user, vehicle status information, driving environment information, user information, and user reactions after utterance in a vehicle driving environment; a vehicle including the same; and a conversation processing method. According to an embodiment of the present invention, the conversation system includes: an input processor which receives the utterance of a user and a reaction after the utterance of the user, and extracts an action corresponding to the utterance; a storage unit which stores an information value of situation information including at least one of vehicle status information related to the status of a vehicle and driving environment information related to the driving environment of the vehicle; a conversation manager which forms at least one of a conditional determination factor and an action factor based on the reaction after utterance corresponding to the utterance, obtains a factor value of the conditional determination factor used to determine whether to perform an action corresponding to the utterance from the storage unit, determines an action to perform based on the factor value of the conditional determination factor, and obtains a factor value of the action factor used to perform the determined action from the storage unit; and a result processor which generates a response to perform the determined action by using the obtained factor value of the action factor.

Description

TECHNICAL FIELD [0001] The present invention relates to a dialog system, a vehicle including the same, and a dialog processing method.

The disclosed invention relates to a conversation system for grasping a user's intention through a conversation with a user and a response after a user's utterance and providing information or a service necessary for a user, a vehicle including the same, and a conversation processing method.

Car AVNs, or most mobile devices, can be inconvenient to provide visual information to users or receive user input due to small screens and small buttons.

In particular, shifting the gaze and releasing the steering wheel as a means for the user to visually identify information or operate the device during operation is a threat to safe driving.

Therefore, it is expected that a conversation system that grasps a user 's intention through a conversation with a user and provides a necessary service to the user can provide a safer and more convenient service when applied to a vehicle.

Korean Patent Publication No. 2013-0035983

By accurately grasping the intention of the user based on various information such as conversation with the user in the vehicle driving environment, vehicle condition information, driving environment information, user information, reaction after the user's utterance, etc., A conversation system capable of providing a service most necessary for a vehicle, a vehicle including the same, and a conversation processing method.

An interactive system according to an embodiment includes an input processor for inputting a user's utterance and a reaction after utterance of the user, and extracting an action corresponding to the utterance and the reaction after utterance; A storage unit for storing information values of status information including at least one of vehicle status information related to the status of the vehicle, driving environment information related to the driving environment of the vehicle, and feedback information related to the reaction after the user's utterance; Forming feedback information on the basis of the post-ignition reaction corresponding to the speech, forming at least one of the condition judgment factor and the action factor based on the feedback information, Determining an action to be performed based on an argument value of the condition determination factor and the feedback information, and performing the determined action A dialogue manager for obtaining an argument value of an action factor to be used from the storage; And a result processor for generating a response for performing the determined action using the obtained argument value of the action factor.

Wherein the storage unit stores the feedback information associated with the action for each action,

The input processor may obtain the feedback information related to the action corresponding to the utterance from the storage unit and transmit the obtained feedback information to the conversation manager.

The dialogue manager may change an argument value of the condition determination factor or an argument value of the action factor based on the feedback information derived using the post-utterance response transmitted from the input processor.

The dialogue manager may obtain from the storage at least one action associated with an action corresponding to the feedback information.

The dialogue manager may determine, based on the feedback information, an action corresponding to the utterance and a priority between the associated at least one action.

Wherein the conversation manager changes a parameter value of a condition judging factor used for judging whether or not to perform the associated action based on the feedback information, obtains the condition judging factor from the storing part rotor, Lt; RTI ID = 0.0 > of < / RTI > the associated at least one action.

The dialogue manager can perform an action corresponding to the utterance and at least one action associated therewith, and determine the action having the highest priority determined based on the feedback information as the action to be performed.

The interactive system according to one embodiment further includes a communication unit for communicating with an external server,

The dialogue manager may request the feedback information from the external server if the feedback information can not be obtained from the storage unit.

The result processor may generate an instruction for controlling the conversation response and the vehicle in response to the determined action.

And a communication unit for receiving the information value of the situation information from the vehicle and transmitting the response to the vehicle.

The interactive system may further include a communication unit that receives the information value of the status information from the mobile device connected to the vehicle, and transmits the response to the mobile device.

The conversation processing method according to an embodiment is a method for processing information on state information including at least one of vehicle state information related to a state of a vehicle, traveling environment information related to the traveling environment of the vehicle, and feedback information related to a reaction after utterance of the user Stores the value in the storage unit; Receiving a user's utterance and a reaction after the utterance of the user; Forming feedback information based on the post-ignition response corresponding to the speech, forming at least one of a condition determination factor and an action factor based on the feedback information, and generating at least one Acquiring at least one of an argument value of a condition judging factor used for judging whether or not the candidate action is to be performed and an argument value of an action factor used for performing the at least one candidate action; And determining an action to be performed among the at least one candidate action based on the feedback information and the feedback information; And generating a response to perform the determined action using an argument value of the obtained action factor.

According to an embodiment of the present invention, there is provided a dialog processing method comprising: storing, in the storage unit, the feedback information associated with an action; And obtaining the feedback information related to the action corresponding to the utterance from the storage unit and transmitting the obtained feedback information to the conversation manager.

The dialog processing method may further include changing an argument value of the condition determination factor or an argument value of the action factor based on the feedback information derived from the post-utterance response transmitted from the input processor; .

Obtaining at least one of the at least one factor value to the storage means,

And acquiring at least one action associated with the action corresponding to the feedback information from the storage unit

The dialog processing method according to an embodiment further includes determining a priority between the action corresponding to the utterance and the at least one action associated with the utterance based on the feedback information.

Determining an action to be performed among the at least one candidate action,

Wherein the control unit changes the parameter value of the condition judging factor used for judging whether the associated action is performed based on the feedback information, acquires the condition judging factor from the storage unit rotor, And determining whether to perform the associated at least one action.

Wherein the determining of the action to be performed among the at least one candidate action comprises: performing an action corresponding to the utterance and at least one action associated with the action; and determining the action having the highest priority determined based on the feedback information It can be decided by the action to be performed.

The method may further include communicating with an external server, and if the feedback information can not be obtained from the storage unit, requesting the feedback information to the external server .

Generating a response to perform the determined action comprises:

 And may generate an instruction for controlling the conversation response and the vehicle in response to the determined action.

The dialog processing method according to an embodiment further comprises receiving an information value of the situation information from the vehicle, and transmitting the response to the vehicle.

The method may further include receiving an information value of the status information from a mobile device connected to the vehicle, and transmitting the response to the mobile device.

The vehicle according to an embodiment includes an input processor for receiving a user's utterance and a user's feedback reaction and extracting an action corresponding to the utterance; A storage unit for storing information values of status information including at least one of vehicle status information related to the status of the vehicle and driving environment information related to the driving environment of the vehicle; Wherein the controller is configured to form at least one of the condition determining factor and the action factor on the basis of the feedback response corresponding to the utterance and to output a parameter value of a conditional judgment factor used for determining whether to perform the action corresponding to the utterance, A conversation manager for acquiring a rotor, determining an action to be performed based on a factor of the condition determining factor, and obtaining an argument value of an action factor used to perform the determined action; And a result processor for generating a response for performing the determined action using the obtained argument value of the action factor.

Wherein the storage unit stores the feedback information associated with the action for each action,

The input processor may obtain the feedback information related to the action corresponding to the utterance from the storage unit and transmit the obtained feedback information to the conversation manager.

The dialogue manager may change an argument value of the condition determination factor or an argument value of the action factor using the feedback information transmitted from the input processor.

The dialogue manager may obtain from the storage at least one action associated with the action corresponding to the utterance.

The dialogue manager may determine, based on the feedback information, an action corresponding to the utterance and a priority between the associated at least one action.

Wherein the conversation manager changes a parameter value of a condition judging factor used for judging whether or not to perform the associated action based on the feedback information, obtains the condition judging factor from the storing part rotor, Lt; RTI ID = 0.0 > of < / RTI > the associated at least one action.

The dialogue manager can perform an action corresponding to the utterance and at least one action associated therewith, and determine the action having the highest priority determined based on the feedback information as the action to be performed.

The vehicle according to an embodiment further includes a communication unit for communicating with an external server,

The dialogue manager may request the factor value of the condition determination factor or the factor of the action factor from the storage unit if the factor can not be obtained from the external server.

The result processor may generate an instruction for controlling the conversation response and the vehicle in response to the determined action.

The vehicle further includes a communication unit that receives the information value of the status information from the vehicle and transmits the response to the vehicle.

The vehicle further includes a communication unit that receives the information value of the status information from the mobile device connected to the vehicle, and transmits the response to the mobile device.

The conversation system according to one aspect, the vehicle including the conversation system, and the conversation processing method can be classified into a conversation system based on various information such as conversation with a user in a vehicle driving environment, vehicle state information, driving environment information, user information, By accurately grasping the intention, a service meeting the user's actual intention or a service most necessary to the user can be provided.

1 is a control block diagram of an interactive system according to one embodiment.
2 is a view showing a configuration inside the vehicle.
Figs. 3 to 5 are views showing examples of conversations that can be exchanged between the conversation system and the driver.
FIG. 6 is a diagram illustrating an example of a post-utterance response of a user corresponding to a utterance of a user according to an embodiment.
7 and 8 are control block diagrams schematically illustrating a connection relationship between the interactive system and the components of the vehicle.
Figs. 9 and 10 are simplified control block diagrams showing the connection relationship between the components of the interactive system and the components of the vehicle.
11 is a control block diagram of a vehicle-alone mode in which an interactive system is provided in a vehicle.
Figs. 12 and 13 are control block diagrams of a vehicle gateway system in which a conversation system is provided at a remote server and the vehicle serves as a gateway connecting the conversation system with the user. Fig.
14 is a control block diagram for a case where the vehicle can perform some input processing and output processing in the vehicle gateway system.
15 is a control block diagram for a hybrid method in which both a remote conversation system server and a vehicle can perform conversation processing.
16 and 17 are control block diagrams of a mobile gateway method in which a mobile device connected to a vehicle connects a user and a remote conversation system server.
18 is a control block diagram of a mobile single mode in which an interactive system is installed in a mobile device.
19, 20A and 20B are control block diagrams in which the configuration of the input processor is broken down in the configuration of the interactive system.
21A and 21B are views showing examples of information stored in the situation understanding table.
22 is a control block diagram in which the configuration of the dialogue manager is subdivided.
23 is a diagram showing an example of information stored in the association action DB.
24 is a diagram showing an example of information stored in the action execution condition DB.
25 is a diagram showing an example of information stored in the action factor DB.
26 is a diagram showing an example of information stored in the ambiguity resolution information DB.
FIGS. 27A and 27B are tables that summarize various examples in which the ambiguity resolver resolves ambiguities by referring to the ambiguity resolution information DB and extracts an action to control the vehicle.
28 is a control block diagram in which the configuration of the result processor is subdivided.
29 to 33 are diagrams showing specific examples in which the conversation system processes the input, manages the conversation, and outputs the result when the user inputs the utterance related to the guidance.
34 is a flowchart illustrating a method of processing user input in a dialog processing method according to an embodiment.
35 is a flowchart showing a method of managing conversation using the output of an input processor in the conversation processing method according to an embodiment.
36 is a flowchart showing a result processing method for generating a response corresponding to the result of conversation management in the conversation processing method according to the embodiment.
37 is a flowchart illustrating an operation of receiving feedback information of a user according to an embodiment.

Like reference numerals refer to like elements throughout the specification. The present specification does not describe all elements of the embodiments, and redundant description between general contents or embodiments in the technical field of the present invention will be omitted. The term 'part, module, member, or block' used in the specification may be embodied in software or hardware, and a plurality of 'part, module, member, and block' may be embodied as one component, It is also possible that a single 'part, module, member, block' includes a plurality of components.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only the case directly connected but also the case where the connection is indirectly connected, and the indirect connection includes connection through the wireless communication network do.

Also, when an element is referred to as "comprising ", it means that it can include other elements as well, without departing from the other elements unless specifically stated otherwise.

The singular forms " a " include plural referents unless the context clearly dictates otherwise.

The codes attached to each step are used to identify each step, and these codes do not indicate the order of the steps, and each step is different from the sequence specified unless explicitly stated in the context. .

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of an interactive system, a vehicle including the interactive system, and a method for processing a conversation will be described in detail with reference to the accompanying drawings.

An interactive system according to an exemplary embodiment of the present invention is an apparatus for identifying a user's intention using a user's voice and voice input and providing a service suitable for a user's intention or a service required for a user, The user can perform conversation with the user by outputting the system utterance as a means for clearly grasping the user.

In this embodiment, the service provided to the user includes all operations performed to meet the needs of the user, such as providing information, controlling the vehicle, executing audio / video / navigation functions, . ≪ / RTI >

In addition, the conversation system according to an embodiment can provide a conversation processing technique specific to the vehicle environment, thereby accurately grasping the intention of the user in a special environment of a vehicle.

The gateway connecting these dialog systems to the user may be a vehicle or a mobile device connected to the vehicle. As will be described later, the conversation system may be provided in a vehicle, or may be provided in a remote server outside the vehicle, and may exchange data through communication with a vehicle or a mobile device connected to the vehicle.

It is also possible that some of the components of the conversation system are provided in the vehicle and some are provided in the remote server so that the operation of the conversation system is partially performed in the vehicle and the remote server.

1 is a control block diagram of an interactive system according to one embodiment.

Referring to FIG. 1A, an interactive system 100 according to an embodiment includes an input processor 110 (FIG. 1A) that processes input including user input, vehicle-related information, A dialogue manager 120 for determining an intention of the user by using the processing result of the input processor 110 and determining an action corresponding to the intention of the user or the state of the vehicle, A result processor 130 for outputting a system utterance for providing a specific service or continuing a conversation, and a storage unit 140 for storing various information necessary for the conversation system 100 to perform an operation described later.

The input processor 110 can receive two kinds of input, that is, user voice and non-voice input. The non-voice input includes a user's gesture recognition, a non-voice input of the user input through the operation of the input device, vehicle status information indicating the status of the vehicle, driving environment information related to the driving environment of the vehicle, And the like. In addition to this information, information related to the vehicle and the user can be input to the input processor 110 if the information can be used to grasp the user's intention or provide the service to the user. The user may include both the driver and the passenger. The input processor 110 recognizes the input user speech, converts the input user speech into a textual speech, and grasps the user's intention by applying a Natural Language Understanding technique to the user's utterance.

In addition, the input processor 110 collects information related to the state of the vehicle and the driving environment in addition to the user's voice, and uses the collected information to understand the situation.

The input processor 110 transmits the information related to the intention of the user and the situation, etc., obtained through natural language understanding, to the dialogue manager 120. The input processor 110 may receive a user's utterance. The speech may be input through the speech input device 210 as described later. In addition, the input processor 110 can receive a response after the user's utterance after the utterance of the user. The user's post-utterance reaction may be a negative reaction, such as an ignition, and may be a negative response. If the response after the user's speech is negative, the user can input the response after the speech through the speech input device 210 as in the case of speech. Also, if the user's reaction after the speech is an extraneous response, the user can input the response after the speech input through the non-speech input device 220. The input processor 110 can transmit the post-ignition response to the reaction after the ignition.

On the other hand, the reaction after the user's utterance may be input by a reaction collecting apparatus provided separately from the voice input apparatus and the voice input apparatus. A detailed description related thereto will be described later.

The dialogue manager 120 determines an action corresponding to the user's intention and the current situation based on the intention of the user, information related to the situation, etc. transmitted from the input processor 110, and manages factors necessary for performing the action .

In this embodiment, an action may mean all actions performed to provide a particular service, and the type of action may be predefined. In some cases, provision of a service and performance of an action may have the same meaning. In particular, the conversation manager 120 learns the intention of the user by learning based on the post-utterance response inputted from the input processor 110. As described above, the post-utterance response of the user can also be input by a separate reaction collection apparatus. The reaction collection apparatus is connected to the function change learning unit provided separately from the dialogue manager 120, And derive the feedback information.

For example, an action such as a route guidance, a vehicle condition check, a petrol station recommendation or the like may be predefined in the domain / action reasoning rule DB 141 (see Fig. 19A), and an action corresponding to a user's utterance, That is, the user can extract an intended action from a predefined action. In addition, an action associated with an event that has occurred in the vehicle may be predefined and stored in the associated action DB 146b (see FIG. 21).

There is no restriction on the type of action, and the conversation system 100 is predefined as being executable through the vehicle 200 or the mobile device 400, and the relationship with the inference rule or other action / event is stored If it is, it can be an action.

The conversation manager 120 delivers the information about the determined action to the result processor 130. [

The result processor 130 generates and outputs an interactive response and a command necessary to perform the delivered action. The conversation response may be output as text, image, or audio, and when the command is output, services such as vehicle control and external content provision corresponding to the output command may be performed.

The storage unit 140 stores various kinds of information necessary for conversation processing and service provisioning. For example, information related to a domain, an action, an action, and an object name used for natural language understanding can be stored in advance, a situation understanding table used for understanding the situation can be stored from input information, One data, information related to the user, and information necessary for performing an action may be stored in advance. In particular, the storage unit 140 may store feedback information of a user corresponding to a user's utterance. A more detailed description of the information stored in the storage unit 140 will be described later.

As described above, the conversation system 100 provides a conversation processing technique specific to the vehicle environment. All of the components of the conversation system 100 may be included in the vehicle, or only a part thereof may be included. The conversation system 100 may be provided at a remote server and the vehicle may serve only as a gateway between the conversation system 100 and the user. In either case, the conversation system 100 may be connected to a user via a vehicle or a mobile device associated with the vehicle.

1B is a control block diagram of an interactive system according to another embodiment of the present invention.

Referring to FIG. 1B, the input processor may include a reaction collection device 110-1, and the conversation manager may include a function change learning device 120-1.

The reaction collecting apparatus can receive two kinds of inputs, such as a user voice and a voice input, as in the input processor described above. The non-voice input includes a user's gesture recognition, a non-voice input of the user input through the operation of the input device, vehicle status information indicating the status of the vehicle, driving environment information related to the driving environment of the vehicle, And the like.

However, the reaction collecting device can be implemented with a configuration in which the user's utterance is performed and the reaction is inputted after the user's utterance. That is, the reaction collecting device can grasp the user's utterance without unconditionally inputting the voice and the behavior of the user, and can input the reaction after the user's utterance.

In addition to this information, information related to the vehicle and the user can be input to the reaction collecting apparatus 110-1, if it is information that can be used to grasp the user's intention or provide the service to the user. The user may include both the driver and the passenger. The reaction collecting device 110-1 may be connected to the function change learning device 120-1 to derive the feedback information as described later, or may be connected to the dialogue manager to derive the feedback information.

The function change learning device 120-1 may be connected to the reaction collection device to derive the feedback information using the post-speech response of the user acquired by the reaction collection device. The function change learning device 120-1 may be configured to recognize an intention of the user's utterance based on the reaction after the utterance of the user and to derive the feedback information so that an action corresponding to the intention of the user can be performed only by the user's utterance have. 2 is a view showing a configuration inside the vehicle.

2, the center fascia 203, which is a central area of the dashboard 201 inside the vehicle 200, is required to perform control of the vehicle including an audio function, a video function, a navigation function, A display 231 for displaying a screen and an input button 221 for receiving a control command of the user may be provided.

An input button 223 may be provided on the steering wheel 207 for convenience of operation of the driver and may be provided on the center console area 202 between the driver's seat 254a and the passenger's seat 254b A jog shuttle 225 may be provided.

A module including a display 231, an input button 221 and a processor for controlling various functions may be referred to as an AVN (Audio Video Navigation) terminal or a head unit.

The display 231 may be implemented as one of various display devices such as a Liquid Crystal Display (LCD), a Light Emitting Diode (LED), a Plasma Display Panel (PDP), an Organic Light Emitting Diode (OLED), and a Cathode Ray Tube have.

The input button 221 may be provided in a hard key type in an area adjacent to the display 231 as shown in FIG. 2. When the display 231 is implemented as a touch screen type, It is also possible to perform the function of the control unit 221 as well.

The vehicle 1 can receive a voice command of the user through the voice input device 210. [ The audio input device 210 may include a microphone for receiving sound, converting the sound into an electrical signal, and outputting the electrical signal.

The input other than the voice during the user input may be input through the voice input device 220. The non-voice input device 220 may include input buttons 221 and 223 and a jog shuttle 225 for receiving a command through a user's operation.

The voice input device 210 may be provided in the head lining 205 as shown in FIG. 2 for the effective voice input, but the embodiment of the vehicle 200 is not limited thereto, and may be provided on the dashboard 201 Or may be provided on the steering wheel 207. In addition, there is no limit to the location where the user's voice can be input.

Inside the vehicle 200, a speaker 232 may be provided to output a sound necessary for conducting a conversation with a user or providing a desired service to the user. For example, the speaker 232 may be provided inside the driver's seat door 253a and the passenger-seat door 253b.

The speaker 232 outputs audio or voice included in the audio / video contents, voice for providing information or service desired by the user, system utterance generated as a response to the user's utterance, etc. .

The interactive system 100 according to an exemplary embodiment provides a service optimized for a lifestyle of a user by using a conversation processing technique specialized in a vehicle environment, and can provide services optimized for a connected car, an Internet (IoT), an artificial intelligence AI) can be configured. Also, the conversation system 100 can learn based on the feedback information based on the response after the user's utterance, and can grasp the user's intention based on the learning result. Further, the function change learning device can learn based on the feedback information based on the response after the user's utterance, and can grasp the intention of the user based on the result of learning.

In the case of applying the conversation processing technique specific to the vehicle environment such as the interactive system 100 according to the embodiment, it is easy to recognize and respond to the main context in the direct driving situation of the driver. In many cases, it is possible to provide information necessary for providing service such as travel time and destination information on the basis of the condition being moved to the destination in most cases. Can be obtained.

Further, it is possible to easily realize an intelligent service that grasps the intention of a driver and suggests a function. This is because priority is given to real-time information and action in the driver's direct driving situation. For example, if a driver searches for a gas station while driving, this can be interpreted as the intention of the driver to go to the gas station now. However, if a gas station is searched for in a non-vehicle environment, it can be interpreted as various other intentions such as location information inquiry, telephone number inquiry, and price inquiry besides the intention of going to a gas station now.

Further, although the vehicle is a limited space, various situations may exist therein. For example, the interactive system 100 can be utilized in a driving situation of an unfamiliar interface such as a rental car, a situation in which a surrogate operation is left, a vehicle management situation such as a car wash, a situation in which a child is boarded,

In addition, various service opportunities and conversation situations arise at each stage of the vehicle driving, including the vehicle inspection phase, the starting preparatory phase, the driving phase, and the parking phase. Especially, the situation of using the safety support functions such as the situation of not knowing how to cope with the vehicle problem, the situation of interlocking of the vehicle with various external devices, driving habit confirmation such as fuel consumption, Smart Cruise Control, The conversation system 100 can be utilized in a situation in which the user repeatedly travels the same route every day, or in a situation where it is necessary to confirm whether or not the parking space is available.

In addition, the vehicle 200 may include an air conditioner 222 to properly maintain the temperature inside the vehicle.

Figs. 3 to 5 are views showing examples of conversations that can be exchanged between the conversation system and the driver.

Referring to FIG. 3, when the driver inputs an utterance (U1: how much oil is left now) asking for the current amount of residual oil, the conversation system 100 determines whether or not the utterance (S1: 43 km can be traveled) can be output.

The driver can input a speech (U2: tell a nearby gas station) asking for a nearby gas station to receive a route guidance to the gas station, and the conversation system 100 can input an utterance (S2: There is a gas station near the location where you can output Aoyama Seongrim gas station, B oil pole gas station, and C oil Pacific gas station).

The operator can input an utterance (U3: Where is the cheapest?) For requesting additional information on the price of the oil, and the conversation system 100 can input the information about the price of the oil (S3: The lowest price is the B oil station gas station with a price of 1,294 won per liter, while the lowest gas price is the A Yong Sung lim gas station with a price of 985 won per liter).

The driver can input an utterance (U4) to guide to the B oil pole gas station, and the conversation system 100 outputs an utterance (S4: B starts at the oil station) to start the guide to the gas station selected by the driver can do.

That is, the driver can receive the route guidance to the nearby gas station, which sells the current type of the current vehicle at the lowest price through the conversation with the conversation system 100.

Meanwhile, in the example of FIG. 3, the conversation system 100 can shorten the conversation step and time by omitting some queries and directly providing information in selecting a gas station.

For example, the conversation system 100 can know in advance that the current type of the vehicle is gasoline, and the driver's gas station selection criteria is the price. Information on the type of the vehicle can be acquired from the vehicle, and the operator's selection criteria for the gas station can be obtained by pre-entering the driver or by learning the driver's conversation history or the gas station selection history. Such information may be stored in advance in the storage unit 140 of the interactive system 100. [

In this case, even if the driver does not input the utterance (U3) for requesting the information on the oil price (U3 is omitted), the conversation system 100 displays the information on the oil price, (S2 + S3 = S3 ') that provides information on the price of gasoline, which is a pollutant, in advance.

The driver may omit the utterance (U3) for requesting information on the oil price, and the response of the conversation system 100 may include the utterance (S2) for the guide to the nearby gas station and the utterance (S3) for the oil price guide By combining, conversation phase and time can be shortened.

In addition, the conversation system 100 may determine by itself that the intention of the driver is a gas station search on the basis of the fact that the driver has asked for the present amount of residual gas.

In this case, even if the driver does not input the utterance U2 for asking the nearby gas station (U2 is omitted), the conversation system 100 generates the utterance (S2 + S3 = S3 ") can be output in advance.

In addition, if there is one gas station having the lowest oil price, which is the closest to the current position in the utterance (S3 ") providing information on the oil price, it is possible to inquire about whether or not to guide the gas station to the gas station. Can input the utterance (U4 ': YES) agreeing with the inquiry of the conversation system 100 without inputting a specific utterance to be guided to a gas station, and can request the route guidance to the gas station.

In this way, the conversation system 100 can grasp the user's actual intention by taking into consideration the contents that the user has not uttered by using the previously obtained information, and can provide the corresponding information in advance. This can shorten the conversation step and time until the user provides the desired service.

FIG. 6 is a diagram illustrating an example of a post-utterance response of a user corresponding to a utterance of a user according to an embodiment.

Referring to FIG. 6A, the user can input the ignition (W1: hot) according to the temperature inside the vehicle. The conversation system 100 may be operable to reduce the temperature of the vehicle based on the user's utterance W1. According to one embodiment, the interactive system 100 may increase the wind strength of the air conditioner 222 provided inside the vehicle. The detailed operation in which the conversation system 100 controls the configuration inside the vehicle will be described later. When the user turns off the air conditioner (W2, 222) and opens the window (W3, 253a) corresponding to the operation in which the conversation system 100 controls the configuration of the vehicle, After the ignition, the reaction can be input. The dialog system 100 can derive the feedback information based on the response after the user's utterance. If the response after the user's utterance corresponding to the utterance of the user is repeated, To output the operation reflecting the intention of the user when the subsequent user inputs the same utterance. In the course of deriving such feedback information, the dialogue manager can learn the user's intention corresponding to the user's utterance.

Specifically, after the user has uttered "hot (W1)", the vehicle outputs an action for activating the air conditioner, but after the user's "warm (W1)" utterance, the user releases the window more than a predetermined number of times The dialogue manager can derive the feedback information that makes the action of bringing down the window correspond to "hot (W1) ". Through this operation, the user can learn the intention of the user to lower the window when the user has uttered the "hot (W1) ".

FIG. 6B is a diagram showing a case where the reaction after the user's utterance is negative.

Referring to FIG. 6B, the user can input the ignition (V1: hot) according to the temperature inside the vehicle. The conversation system 100 may be operable to reduce the temperature of the vehicle based on the user's utterance V1. According to one embodiment, the interactive system 100 may increase the wind strength of the air conditioner 222 provided inside the vehicle. In response to the operation of controlling the configuration of the vehicle by the conversation system 100, when the user inputs a reaction after uttering voice (V2: annoying), the voice input device 210 provided in the vehicle performs a reaction Can be input. The dialog system 100 can derive the feedback information based on the response after the user's utterance and can learn the intention of the user on the basis of the feedback information. When the reaction after the speech corresponding to the user's utterance is repeated, the action factor and the condition judgment factor are changed based on the feedback information of the user, so that when the next user inputs the same utterance, the operation reflecting the intention of the user can be outputted . 6B, since the user inputs an unsatisfactory response to the action output by the conversation system 100, the conversation system 100 determines whether or not the conversation system 100 receives the same speech (V1: hot) ) To open the window without increasing the wind. The operation shown in Fig. 6 is only an embodiment, and the operation in which the conversation system 100 learns the response after the user's utterance is not limited if the conversation system 100 controls the inside of the vehicle.

7 and 8 are control block diagrams briefly showing the connection relationship between the interactive system and the components of the vehicle.

Referring to FIG. 7, a voice of a user input to the conversation system 100 may be input through a voice input device 210 provided in the vehicle 200. As described above with reference to FIG. 2, the voice input device 210 may include a microphone provided inside the vehicle 200. The speech input apparatus 210 can receive a user's utterance, and can receive a user's utterance response corresponding to an action output by the conversation system 100.

The input other than the voice during the user input may be input through the voice input device 220. The non-voice input device 220 may include input buttons 221 and 223 and a jog shuttle 225 for receiving a command through a user's operation.

It is also possible that the voice input device 220 includes a camera for photographing a user. A gesture, a facial expression or a gaze direction of a user used as a means of inputting a command can be recognized through the image photographed by the camera. Alternatively, it is possible to grasp the state of the user (drowsiness state, etc.) through the image photographed by the camera.

Meanwhile, the non-voice input device 220 can receive the non-voice feedback operation of the user like the voice input device 210 described above. When the user operates the input buttons 221 and 223 and the jog shuttle 225 corresponding to the action output by the dialog system 100, the dialog system 100 can recognize the reaction after the user's utterance. Information about the vehicle can be input to the conversation system 100 via the vehicle controller 240. [ The information about the vehicle may include vehicle status information or surrounding status information obtained through various sensors provided in the vehicle 200, and may also include information basically stored in the vehicle 200, such as the type of the vehicle.

The conversation system 100 uses the user's voice inputted through the voice input device 210, the voice input from the user inputted through the voice input device 220 and various information inputted through the vehicle controller 240 It grasps the user's intention and situation, and outputs a response for performing an action corresponding to the user's intention. That is, the feedback information may be formed based on the response of the user after the speech, so that the action factors and the condition determination factors used in the liquid line outputted by the dialog system 100 may be newly formed or changed.

The talker output device 230 may include a display 231 and a speaker 232 provided in the vehicle 200 to provide a visual, auditory or tactile output to the talker. The display 231 and the speaker 232 can visually or audibly output a response to the user's utterance, a query to the user, or information requested by the user. Alternatively, it is also possible to mount a vibrator on the steering wheel 207 to output the vibration.

In addition, the vehicle controller 240 can control the vehicle 200 to perform an action corresponding to the user's intention or the current situation, in response to the response output from the interactive system 100. [

The vehicle 200 may receive not only data obtained through the sensors provided in the vehicle 200 but also information acquired from the external content server 300 or an external device through the communication device 280, , Temperature, passenger information, driver's personal information, and the like can also be collected and transmitted to the conversation system 100.

As shown in FIG. 8, information obtained from a sensor provided in the vehicle, such as a residual amount, a rainfall amount, a rainfall rate, surrounding obstacle information, a speed, an engine temperature, a tire air pressure, (100).

The traveling environment information obtained from the outside through the V2X (Vehicle to Everything) can be inputted to the conversation system 100 through the external signal controller 242. [ V2X means to exchange various useful information such as traffic situation while communicating with road infrastructure and other vehicles while the vehicle is traveling.

V2X communication includes communication between vehicle and infrastructure (V2I), vehicle-to-vehicle (V2V), and vehicle-to-nomadic devices (V2N) . Therefore, by using the V2X communication, vehicles can communicate directly with each other or communicate with infrastructures installed in the distances to inform the driver of information such as forward traffic information, accessibility or collision possibility of another vehicle, and the like.

Accordingly, the travel environment information input to the conversation system 100 through the external signal controller 242 may include traffic information in the forward direction, approach information of nearby vehicles, collision warning with other vehicles, real-time traffic situation, State, and the like.

Although not shown in the figure, a signal obtained via V2X may also be input to the vehicle 200 via the communication device 280. [

The vehicle controller 240 may include a memory in which a program for performing the above-described operations and the operation described later is stored, and a processor for executing the stored program. At least one memory and one processor may be provided, and if a plurality of memory and processors are provided, they may be integrated on one chip or physically separated.

Further, the internal signal controller 241 and the external signal controller 242 may be implemented by the same processor or by a separate processor.

Figs. 9 and 10 are simplified control block diagrams showing the connection relationship between the components of the interactive system and the components of the vehicle.

8, the user voice transmitted from the voice input device 210 is input to the voice input processor 111 in the input processor 110, and the voice input from the voice input device 220 And input to the situation information processor 112 in the input processor 110.

The situation information processor 112 understands the situation based on the vehicle condition information, the traveling environment information, the user information, and the like. Through this understanding of the situation, the conversation system 100 can more accurately grasp the intention of the user or effectively find the service required for the current user. The user information may include feedback information of the user corresponding to the user's utterance.

The response output from the result processor 130 is input to the talker output device 230 or the vehicle controller 240 to allow the vehicle 200 to provide the necessary service to the user. Or may be transmitted to the external content server 300 to request a service required by the user.

Vehicle condition information, driving environment information, user information, and the like transmitted from the vehicle controller 240 are stored in the storage unit 140.

Referring to FIG. 10, the storage unit 140 may include a long term memory 143 and a short term memory 144. The data stored in the storage unit 140 may be divided into short-term and long-term memories according to the importance and permanence of the data and the designer's intention.

The short term memory 144 may store a previously performed conversation. The previous conversation may be a conversation performed within the reference time from the current point in time. Alternatively, the conversation may be continuously stored until the capacity of the utterance content between the user and the conversation system 100 becomes a reference value.

For example, when the user utters "Tell me restaurant near Gangnam station", the conversation system 100 searches the restaurant near Gangnam station through the external contents server 300 and provides information about restaurants near the Gangnam station searched for . As an example of providing information, the restaurant list can be displayed on the display 231, and if the user says "first, " the conversation from inquiry to selection of the restaurant can be stored in the short term memory 144 .

Also, if the user utters "hot ", the conversation system 100 may lower the interior temperature of the vehicle through the air conditioning system 222. Then, when the user then utters "annoying", the dialog system 100 forms feedback information through the reaction after the user's utterance and stores the feedback information in the short-term memory 144. [

Alternatively, not only the entire conversation contents are stored, but also specific information included in the conversation contents can be stored. For example, it is possible to store the first restaurant of the restaurant list in the short-term memory 144 or the long-term memory 143 as a restaurant selected by the user.

When the user asks the conversation system 100 "What is the weather?" After the conversation with the restaurant near the Gangnam station, the conversation system 100 estimates from the conversation stored in the short-term memory 144 that the user's location of interest is the Gangnam station The user can output a response saying "It is raining in Gangnam Station."

Then, the conversation system 100 judges from the conversation stored in the short-term memory that "the restaurant" is a restaurant near the Gangnam station, The service obtains information on the recommendation menu of the restaurant and outputs a response such as "The restaurant is delicious. &Quot;

The long-term memory 143 may store data according to the presence or absence of data persistence. For example, the phone number of a family member or a friend, POI information such as a house or a company, a user's preference for a specific factor, etc., can be determined to guarantee the durability of the data and stored in the long- On the other hand, data judged not to guarantee the durability of the data can be stored in the short-term memory 144.

In one example, the current location of the user is temporary data and therefore may be stored in the short term memory 144, and the user's preference for the restaurant may be stored in the long term memory 144 since it may be viewed as persistent data that may be used thereafter.

As another example, the current temperature can be stored in the short term memory 144 since it is temporary data, and the user's preference for operating the air conditioner 222 can be stored in the long term memory 144. [

If the user speaks "no restaurants nearby?", The conversation system 100 can grasp the user's current location from the short-term memory 144 and know from the long-term memory 143 that the user's preferred restaurant is a Chinese restaurant . Therefore, it is possible to recommend a list of Chinese restaurants preferred by the user around the current position by using external contents.

In addition, the conversation system 100 may provide services or information to the user in advance by using data stored in the long-term memory 143 and the short-term memory 144. [

For example, the long-term memory 143 may store information about the user's home. The conversation system 100 may obtain information related to the user's home from the external content server 300 and provide the user with information such as "It is scheduled to sing in the apartment complex this Friday. &Quot;

Alternatively, information on the battery status of the vehicle may be stored in the short term memory 144. [ The conversation system 100 may analyze the battery status of the vehicle stored in the short term memory 1440 and provide information such as "The battery condition of the vehicle is not good.

11 is a control block diagram of a vehicle-alone mode in which an interactive system is provided in a vehicle.

In the vehicle alone mode, the conversation system 100 including the input processor 110, the conversation manager 120, the result processor 130, and the storage unit 140 is included in the vehicle 200 as shown in FIG. 11 .

When the conversation system 100 is included in the vehicle 200, the vehicle 200 itself can process the conversation with the user and provide the necessary services to the user. However, information necessary for dialog processing and service provision can be obtained from the external content server 300. [

The vehicle state information or the traveling environment information such as the remaining amount of the vehicle, the rainfall amount, the rainfall rate, the surrounding obstacle information, the tire inflation pressure, the current position, the engine temperature, (100).

The vehicle controller 240 also controls the air conditioner 251, the window 252, the door 253, the seat 254, or the AVN 255 provided in the vehicle 200 according to the response output from the interactive system 100. [ And so on.

For example, when the conversation system 100 determines that the intention of the user or the service required by the user is to lower the temperature inside the vehicle 200 and generates and outputs a corresponding command, the vehicle controller 240 controls the air conditioner The temperature inside the vehicle 200 can be lowered. However, if the feedback information based on the post-ignition response of the user does not want the user to operate the air conditioner 222, the conversation system 100 controls the window 252a, not the air conditioner 222, Can be lowered.

As another example, when the conversation system 100 determines that the intention of the user or the service required by the user is to raise the driver's seat window 252a and generates and outputs a corresponding instruction, the vehicle controller 240 displays the window 252 The driver's seat window 252a can be raised.

As another example, if the conversation system 100 determines that the service corresponding to the user's intention or the service required by the user is a route guidance to a specific destination, and generates and outputs a corresponding command, the vehicle controller 240 transmits AVN (255) to perform route guidance. If necessary, the communication device 280 can fetch map data, POI information, and the like from the external content server 300 and use it to provide services.

12 and 13 are control block diagrams of a vehicle gateway system in which a conversation system is provided at a remote server and the vehicle serves only as a gateway for connecting the user and the conversation system.

12, a remote conversation system server 1 is provided outside the vehicle 200, and the vehicle 200 is connected to the remote conversation system server 1 via the communication device 280 An interactive system client 270 is provided. The communication device 280 serves as a gateway for connecting the vehicle 200 and the remote conversation system server 1.

The conversation system client 270 functions as an interface connected to the input / output device, and can perform data collection and transmission / reception.

When the voice input device 210 and the voice input device 220 provided in the vehicle 200 receive the user input and transmit the voice input to the conversation system client 270, the conversation system client 270 transmits, via the communication device 280 It is possible to transmit the input data to the remote conversation system server 1.

The vehicle controller 240 may also communicate data sensed by the vehicle sensing unit 260 to the conversation system client 270 and the conversation system client 270 may communicate via the communication device 280 to the remote conversation system server 1 The vehicle sensing unit 260 can transmit the sensed data.

In the remote conversation system server 1, the above-described conversation system 100 is provided, and can perform both processing of input data, conversation processing based on processing results of input data, and result processing based on conversation processing results.

Also, the remote conversation system server 1 can fetch information and contents necessary for processing of input data, conversation management, or result processing from the external content server 300.

The vehicle 200 can also fetch the content necessary for providing the necessary service to the user from the external content server 300 according to the response transmitted from the remote interactive system server 1. [

13, the communication device 280 may include one or more communication modules that enable communication with an external device and may include, for example, a short-range communication module 281, a wired communication module 282, (283).

The short-range communication module 281 uses a wireless communication network at a short distance, such as a Bluetooth module, an infrared communication module, an RFID (Radio Frequency Identification) communication module, a WLAN (Wireless Local Access Network) communication module, an NFC communication module, and a Zigbee communication module. And various various local communication modules for transmitting and receiving signals.

The wired communication module 282 may be connected to various wired communication modules such as a local area network (LAN) module, a wide area network (WAN) module or a value added network (VAN) And may include at least one of a variety of cable communication modules such as a serial bus, a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), a recommended standard 232 (RS-232), a power line communication or a plain old telephone service have.

The wireless communication module 283 may be a wireless communication module such as a Global System for Mobile Communication (GSM), a Code Division Multiple Access (CDMA), a Wideband Code Division Multiple Access (WCDMA), a Universal Mobile Broadband (UMTS) A mobile communication system, a time division multiple access (TDMA), a long term evolution (LTE), a 4G, and a 5G.

Meanwhile, the communication device 280 may further include an internal communication module (not shown) for communication between the electronic devices inside the vehicle 200. A CAN (Controller Area Network), a LIN (Local Interconnection Network), a FlexRay, an Ethernet, or the like can be used as an internal communication protocol of the vehicle 200.

The conversation system 100 can exchange data with the external content server 300 or the remote conversation system server 1 using the wireless communication module 283. [ In addition, V2X communication can be performed using the wireless communication module 283. [ In addition, data can be exchanged with a mobile device connected to the vehicle 200 by using the short-distance communication module 281 or the wired communication module 282.

14 is a control block diagram for a case where the vehicle can perform some input processing and output processing in the vehicle gateway system.

As described above, although the conversation system client 270 of the vehicle 200 can perform only the function of collecting and transmitting / receiving data, the conversation system client 270 is provided with an input processor 271 ), A result processor 273, and a storage unit 274, and it is possible for the vehicle 200 to perform processing related to data input from a user or a vehicle and service provision determined to be necessary for the user. That is, the operations of the input processor 110 and the result processor 130 can be performed not only in the remote conversation system server 1 but also in the vehicle 200.

In this case, the interactive system client 270 may perform only a part of the operation of the input processor 110 described above, or may perform all of the operations. It is also possible for the interactive system client 270 to perform only a part of the operation of the result processor 130, or to perform all of the operations.

The task sharing between the remote conversation system server 1 and the conversation system client 270 can be determined in consideration of the capacity of the data to be processed and the data processing speed.

15 is a control block diagram for a hybrid method in which both a remote conversation system server and a vehicle can perform conversation processing.

14, the remote conversation system server 1 also includes an input processor 110, a conversation manager 120, a result processor 130, and a storage unit 140 to perform conversation processing The vehicle 200 is also provided with a terminal dialog system 290 including an input processor 291, a dialogue manager 292, a result processor 293 and a storage unit 294 to perform conversation processing.

However, the processor or the memory provided in the vehicle 200 may differ from the processor or the memory provided in the remote conversation system server 1 in terms of capacity and performance. Accordingly, if the terminal dialog system 290 processes all the input data and manages the dialog and outputs the result, the terminal dialog system 290 performs the entire process, otherwise, the remote dialog system server 1 ) To request processing.

Before performing the conversation processing in the terminal conversation system 290, it is determined whether the terminal conversation system 290 is capable of processing based on the type of the input data, and the processing is directly processed according to the determination result, .

Alternatively, when the terminal conversation system 290 performs a conversation process and a situation in which processing can not be performed occurs, it is also possible to request the remote conversation system server 1 for processing and send the result of the processing together with the request.

For example, when processing of high-performance computing power and long-term data is required, the terminal dialog system 290 can process the data in the remote conversation system server 1 and process it in real time. For example, when an instance requiring immediate processing occurs and data before synchronization needs to be processed, the terminal dialog system 290 can be set to preferentially process.

In addition, when there is an unregistered speaker in the vehicle and user confirmation is required, the remote conversation system server 1 can process the conversation.

When the terminal conversation system 290 can not complete the conversation processing itself in a situation where the connection with the remote conversation system server 1 through the communication device 280 is not possible, That the conversation process can not be performed.

The data stored in the terminal conversation system 290 and the data stored in the remote conversation system server 1 may be determined according to criteria such as the capacity of data or the type of data. For example, in the case of data that can be personally identified and may cause privacy intrusion, the data may be stored in the storage unit 294 of the terminal conversation system 290. Also, a large amount of data may be stored in the storage unit 140 of the remote conversation system server 1, and a small amount of data may be stored in the storage unit 294 of the terminal conversation system 290. Alternatively, it is also possible that a small amount of data is stored in both the storage unit 140 of the remote interactive system server 1 and the storage unit 294 of the terminal interactive system 290.

16 and 17 are control block diagrams of a mobile gateway method in which a mobile device connected to a vehicle connects a user and a remote conversation system server.

In the mobile gateway system, as shown in FIG. 16, the mobile device 400 receives vehicle status information and driving environment information from the vehicle 200, and transmits user input and vehicle status information to the remote conversation system server 1 do. That is, the mobile device 400 serves as a gateway for connecting the user with the remote interactive system server 1 or between the vehicle 200 and the remote interactive system server 1.

The mobile device 400 may be a portable device such as a smart phone, a smart watch, a smart glass, a PDA, or a tablet PC, and may be an electronic device capable of communicating data with an external server and a vehicle.

The mobile device 400 includes a voice input device 410 for receiving a user voice, a voice input device 420 for receiving a voice input from a user, an output device 430 for outputting a voice response in a visual, auditory or tactile sense A communication device 480 that communicates with the remote interactive system server 1 and the vehicle 200 to transmit and receive data and a remote interactive system server 480 that collects input data from the vehicle 200 and the user, And an interactive system client 470 that transmits the message to the client system 1.

The voice input device 410 may include a microphone for receiving sound, converting the sound into an electrical signal, and outputting the electrical signal.

The non-voice input device 420 may include an input button, a touch screen, or a camera provided in the mobile device 400.

The output device 430 may include a display, a speaker, or a vibrator provided in the mobile device 400.

It is possible to use not only the voice input device 410, the voice input device 420 and the output device 430 provided in the mobile device 400 as the input / output interface for the user but also the voice input device 210 ), The non-voice input device 220 and the talker output device 230 may be used.

When the vehicle 200 transmits data sensed by the vehicle sensing unit 260 and user input to the mobile device 400, the conversation system client 470 of the mobile device 400 transmits it to the remote conversation system server 1 send.

In addition, the conversation system client 470 may forward the response or command sent from the remote conversation system server 1 to the vehicle 200. When the talker output device 230 provided in the vehicle 200 is used as an input / output interface for the user, a response to the user's speech can be output through the talker output device 230. [ When the output device 430 provided in the mobile device 400 is used, a response to the user's utterance can be output through the output device 430 of the mobile device 400. [

A command for vehicle control is transmitted to the vehicle 200, and the vehicle controller 240 performs control corresponding to the transmitted command to provide the necessary service to the user.

Conversely, the conversation system client 470 not only collects and passes the input data to the remote conversation system server 1, but also some or all of the functions of the input processor 110 and result processor 130 of the conversation system 100, It is also possible to perform all of them.

17, the communication device 480 of the mobile device 400 may include one or more communication modules that enable communication with an external device. For example, the communication device 480 may include a short-range communication module 481, a wired communication module 482, and a wireless communication module 483.

The short-range communication module 481 uses a wireless communication network at a short distance, such as a Bluetooth module, an infrared communication module, a radio frequency identification (RFID) communication module, a wireless local access network (WLAN) communication module, an NFC communication module, and a Zigbee communication module. And various various local communication modules for transmitting and receiving signals.

The wired communication module 482 may be connected to various wired communication modules such as a local area network (LAN) module, a wide area network (WAN) module or a value added network (VAN) And may include at least one of a variety of cable communication modules such as a serial bus, a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), a recommended standard 232 (RS-232), a power line communication or a plain old telephone service have.

The wireless communication module 483 may be a wireless communication module such as a Global System for Mobile Communication (GSM), a Code Division Multiple Access (CDMA), a Wideband Code Division Multiple Access (WCDMA), a Universal Wireless Broadband (UMTS) A mobile communication system, a time division multiple access (TDMA), a long term evolution (LTE), a 4G, and a 5G.

For example, the mobile device 400 may be connected to the vehicle 200 via the short-range communication module 481 or the wired communication module 482 and may be connected to the remote conversation system server 1 or And may be connected to the external content server 300.

18 is a control block diagram of a mobile single mode in which an interactive system is installed in a mobile device.

In the mobile standalone system, as shown in FIG. 18, the interactive system 100 is provided in the mobile device 400.

Accordingly, it is not necessary to be connected to the remote conversation system server 1 in order to perform the conversation processing, and the mobile device 400 can process the conversation with the user and provide the necessary services to the user. However, information necessary for dialog processing and service provision can be obtained from the external content server 300. [

Hereinafter, the detailed configuration of the conversation system 100 and specific operations of the respective components will be described in detail. In the following embodiments, for convenience of explanation, it is assumed that the conversation system 100 is provided in the vehicle 200. FIG.

19, 20A and 20B are control block diagrams in which the configuration of the input processor is broken down in the configuration of the interactive system.

Referring to FIG. 19, the input processor 110 may include a speech input processor 111 for processing speech input and a context information processor 112 for processing context information.

The user's voice inputted through the voice input device 210 is transmitted to the voice input processor 111 and the voice other user input inputted through the voice input device 220 is transmitted to the situation information processor 112. [

The vehicle controller 240 transmits the vehicle state information, the traveling environment information, and the user information to the situation information processor 112. The driving environment information and the user information may be provided from the external content server 300 or the mobile device 400 connected to the vehicle 200. [

All non-speech inputs can be included in the context information. That is, the status information may include vehicle status information, driving environment information, and user information.

The vehicle state information may include information indicating the state of the vehicle as information acquired by a sensor provided in the vehicle 200, information stored in the vehicle as information related to the vehicle, such as the type of the vehicle, and the like.

The travel environment information is information obtained by a sensor provided in the vehicle 200 and is information acquired by sensors such as image information acquired by a front camera, a rear camera or a stereo camera, a radar, a lidar, and an ultrasonic sensor Obstacle information, rainfall / rainfall rate information obtained by the rainfall sensor, and the like.

In addition, the travel environment information may further include traffic situation information, traffic light information, approach or collision possibility information of neighboring vehicles, and the like, as information acquired through V2X.

The user information includes information related to the status of the user measured through the camera or the bio-signal measuring device provided in the vehicle, information related to the user directly inputting by using the input device provided in the vehicle, Information stored in the mobile device 400 connected to the vehicle, and the like. In particular, the user information may include feedback information of the user based on the response or action output in response to the speech input by the user in the dialog system 100.

The speech input processor 111 includes a speech recognizer 111a for recognizing the inputted user speech and outputting the speech as a text-based speech, a speech recognizer 111a for recognizing the user's speech by applying a Natural Language Understanding A natural language understanding device 111b for understanding the intention and a dialogue input manager 111c for transmitting the natural language understanding result and the situation information to the dialogue manager 120. [

The speech recognizer 111a includes a speech recognition engine. The speech recognition engine recognizes speech uttered by a user by applying a speech recognition algorithm to the input speech, and generates a recognition result.

At this time, the input voice can be converted into a more useful form for voice recognition, and the start point and the end point are detected from the voice signal and the actual voice interval included in the inputted voice is detected. This is called EPD (End Point Detection).

The characteristics such as cepstrum, linear predictive coding (LPC), mel-frequency cepstral coefficient (MFCC), or filter bank energy are detected within the detected interval. The feature vector of the input speech can be extracted by applying the vector extraction technique.

Then, the recognition result can be obtained by comparing the extracted feature vector with the trained reference pattern. For this purpose, an acoustic model for modeling and comparing the signal characteristics of speech and a language model for modeling the linguistic order relation of words or syllables corresponding to the recognition vocabulary can be used. To this end, the storage unit 140 may store an acoustic model / language model DB.

The acoustic model can be divided into a direct comparison method of setting a recognition object as a feature vector model and comparing it with a feature vector of voice data, and a statistical method of using the feature vector of the recognition object statistically.

The direct comparison method is a method of setting a unit of a recognition target word, a phoneme, etc. as a feature vector model and comparing how similar the input speech is, and a vector quantization method is typically used. According to the vector quantization method, a feature vector of input speech data is mapped to a codebook, which is a reference model, and is encoded into a representative value, thereby comparing the code values with each other.

The statistical model method is a method of constructing a unit of a recognition object as a state sequence and using the relation between the state strings. The state column may consist of a plurality of nodes. The method of using the relation between state columns is again Dynamic Time Warping (DTW), Hidden Markov Model (HMM), and a method using a neural network.

Dynamic time warping is a method of compensating for differences in the time axis when compared to a reference model, taking into account the dynamic characteristics of the speech, in which the length of the signal varies with time even if the same person pronounces the same. Hidden Markov models, Assuming a Markov process with probabilities and observation probabilities of nodes (output symbols) in each state, we estimate the state transition probability and observation probability of nodes through the learning data, and calculate the probability that the input speech is generated in the estimated model .

On the other hand, the language model modeling the linguistic order relations of words and syllables can reduce the acoustical ambiguity and reduce the errors of recognition by applying the order relation between the units constituting the language to the units obtained by speech recognition. There are statistical language models and Finite State Automata (FSA) based models for language models, and the chained probabilities of words such as Unigram, Bigram, and Trigram are used for statistical language models.

The voice recognizer 111a may use any of the above-described methods in recognizing the voice. For example, an acoustic model with a hidden Markov model can be used, or an N-best search method using an acoustic model and an audio model can be used. The N-best search method can improve the recognition performance by selecting up to N recognition result candidates using the acoustic model and the language model, and then re-evaluating the ranking of these candidates.

The speech recognizer 111a can calculate a confidence value to secure the reliability of the recognition result. The confidence value is a measure of how reliable the result is to speech recognition results. For example, a phoneme or word recognized as a result can be defined as a relative value of the probability that the word has been uttered from another phoneme or word. Therefore, the confidence value may be expressed as a value between 0 and 1, or as a value between 0 and 100.

When the confidence value exceeds a predetermined threshold value, the recognition result may be output to perform the operation corresponding to the recognition result. If the confidence value is equal to or less than the threshold value, the recognition result may be rejected have.

The text-type utterance, which is the recognition result of the voice recognizer 111a, is input to the natural language comprehension machine 111b.

The natural language understanding unit 111b can grasp the user's intention in the utterance by applying the natural language understanding technology. Accordingly, the user can input commands through a natural dialogue, and the conversation system 100 can also induce input of commands through conversation or provide services required by the user.

First, the natural language understanding unit 111b performs morphological analysis on the text-type utterance statement. A morpheme is the smallest unit of meaning and represents the smallest semantic element that can no longer be subdivided. Thus, morpheme analysis is the first step in understanding natural language, transforming input strings into morpheme strings.

The natural language understanding unit 111b extracts the domain from the utterance statement based on the morphological analysis result. The domain is a domain that is able to identify a subject of a language uttered by a user and represents various subjects such as route guidance, weather search, traffic search, schedule management, oiling guidance, air control, and the like.

The natural language understanding unit 111b can recognize the entity name from the utterance statement. An entity name is a proper name such as a name, a place name, an organization name, a time, a date, a money, etc., and an entity name recognition is a work for identifying an entity name in a sentence and determining the kind of the entity name. The meaning of sentences can be grasped by extracting important keywords from sentences through object name recognition.

The natural language understanding unit 111b can analyze an action of the utterance. The analysis is the analysis of the intention of the user utterance. It is to grasp the intention of the utterance about whether the user is asking a question, asking a request, responding, or simply expressing emotion.

The natural language understanding device 111b extracts an action corresponding to the user's utterance intention. The user's utterance intention is grasped based on information such as a domain, an object name, and a transcription corresponding to the utterance, and an action corresponding to the utterance intention is extracted. An action can be defined by an Object and an Operator.

In addition, the natural language understanding unit 111b can extract factors related to the action execution. The action associated with the action may be either an immediate or a non-valid argument used to perform the action.

For example, if the user's utterance is "hot ", the natural language understander 111b may extract" air conditioning control "from the domain corresponding to the utterance and extract" . Acts correspond to "expression of simple sentiment".

In addition, the natural language understanding unit 111b can extract means for expressing the expression relation of words, words, sentences and sentences such as a parse-tree.

Domain information, action information, speech information, extracted factor information, entity name information, parse tree, etc., which are the result of morphological analysis, which is the result of processing by the natural language understanding unit 111b, are transmitted to the dialogue inputting controller 111c.

The situation information processor 112 includes a situation information collector 112a for collecting information from the voice input device 220 and the vehicle controller 240, a situation information collection manager 112b for managing collection of situation information, And a situation understanding device 112c for understanding the situation based on the collected situation information.

The input processor 110 may include a memory in which a program for performing the above-described or later-described operations is stored, and a processor for executing the stored program. At least one memory and one processor may be provided, and if a plurality of memory and processors are provided, they may be integrated on one chip or physically separated.

In addition, the speech input processor 111 and the context information processor 112 included in the input processor 110 may be implemented by the same processor or by separate processors.

Hereinafter, how input data is processed by the elements of the input processor 110 using the information stored in the storage unit 140 will be described in detail with reference to FIGS. 20A and 20B.

Referring to FIG. 20A, the natural language understanding unit 111b may use a domain / action reasoning rule DB 141 for domain extraction, entity name recognition, action analysis, and action extraction.

The domain / action reasoning rule DB 141 may store a domain extraction rule, an action analysis rule, an object name conversion rule, an action extraction rule, and the like.

Other information such as non-voice user input, vehicle status information, driving environment information, and user information is input to the situation information collector 112a and stored in the situation information DB 142, long term memory 143 or short term memory 144 .

For example, in the situation information DB 142, raw data sensed by the vehicle sensing unit 260 may be stored as a sensor type and a sensor value.

The short-term memory 144 and the long-term memory 143 may store data meaningful to the user such as the current state of the user, the preference / propensity of the user, or data capable of determining the user's preference / tendency. As described above, the short-term memory 144 and the long-term memory 143 may store feedback information corresponding to the post-utterance response of the user who can grasp the user's utterance intention.

As described above, the long-term memory 143 can store long-term usable information such as a phone book, a schedule, a preference, education, personality, occupation, family information, etc. of the user. The short-term memory 144 may store information used in the short-term, such as the current / previous position, today's schedule, previous conversation contents, conversation participants, circumstance, domain, driver status, etc., whose persistence is not guaranteed or uncertain. Depending on the type of data, there may be data that is stored redundantly in two or more of the situation information DB 142, the short term memory 144 and the long term memory 143. [

In addition, information determined to be durable among the information stored in the short term memory 144 may be sent to the long term memory 143. [

It is also possible to acquire information to be stored in the long term memory 143 by using the information stored in the short term memory 144 or the situation information DB 142. [ For example, the destination information or the conversation contents accumulated for a predetermined period of time may be analyzed to acquire the preference of the user, and the acquired preference of the user may be stored in the long term memory 143.

It is possible to acquire information to be stored in the long term memory 143 using the information stored in the short term memory 144 or the situation information DB 142 in the interactive system 100, .

In the former case, it can be executed in the memory manager 135 of the result processor 130, which will be described later. In this case, among the data stored in the short-term memory 144 and the situation information DB 142, data used to acquire meaningful information or permanent information such as the user's preference or inclination is stored in the long-term memory 143 as a log file Lt; / RTI > The memory manager 135 analyzes the accumulated data for a predetermined period of time to acquire the persistent data and stores it in the long-term memory 143 again. The location where the persistent data is stored in the long term memory 143 may be different from the location where the data stored in the log file format is stored.

Alternatively, it is also possible for the memory manager 135 to judge the persistent data among the data stored in the short term memory 144, and to transfer the determined data to the long term memory 143 and store it.

A data management system 800 including a communication unit 810, a storage unit 820, and a control unit 830 may be used as shown in FIG. 19B.

The communication unit 810 receives the data stored in the situation information DB 142 or the short term memory 144. [ All of the stored data may be transmitted to the communication unit, and only the data used to acquire meaningful information or persistent information such as the user's preference or propensity may be selected and transmitted. The received data is stored in the storage unit 820.

The control unit 830 analyzes the accumulated data to acquire data having persistence, and transmits the acquired data to the conversation system 100 through the communication unit 810 again. The transmitted data is stored in the long term memory 143 of the interactive system 100.

The dialogue input managing unit 111c may transmit the output result of the natural language understanding unit 111b to the situation understanding unit 112c to obtain the situation information related to the action execution.

The situation understanding unit 112c can determine the status information related to the execution of the action corresponding to the user's utterance intention by referring to the context information per action stored in the context understanding table 145. [ For example, when the user utters "hot", the situation understanding unit 112c determines whether the user's utterance intention is an operation for increasing the intensity of the air conditioner wind or an operation for lowering the window based on feedback information corresponding to the utterance of the user Can be determined.

21A and 21B are views showing examples of information stored in the situation understanding table.

Referring to the example of FIG. 20A, in the situation understanding table 145, the situation information and the situation information type related to the execution of the action can be stored for each action.

For example, if the guidance is an action, the current position is required as the situation information, and the situation information type may be GPS information. When the vehicle condition check is an action, the moving distance is required for the situation information, and the situation information type can be an integer. When the gas station recommendation is an action, the residual information and the distance traveled distance (DTE) are required as the situation information, and the situation information type may be an integer.

According to another embodiment, in the case of an action for lowering the temperature in the vehicle, the temperature is required as the situation information, and the situation information type may be temperature, that is, an integer.

When the situation information related to the execution of the action corresponding to the user's utterance intention is already stored in the situation information DB 142, the long-term memory 143 or the short-term memory 144, the situation understanding unit 112c acquires the situation information DB 142 The long-term memory 143 or the short-term memory 144, and transfers the information to the dialog input manager 111c.

When the situation information related to the execution of the action corresponding to the user's utterance intention is not stored in the situation information DB 142, the long term memory 143 or the short term memory 144, 112b. The situation information collection manager 112b allows the situation information collector 112a to collect necessary information.

The situation information collector 112a may periodically collect data, collect data when a specific event occurs, periodically collect data, and collect additional data when a specific event occurs. In addition, data may be collected when a data collection request is input from the situation information collection manager 112b.

The situation information collector 112a collects necessary information and stores it in the situation information DB 142 or the short term memory 144 and transmits an acknowledgment signal to the situation information collection manager 112b.

The situation information collection management unit 112b also transmits an acknowledgment signal to the situation understanding unit 112c and the situation understanding unit 112c acquires necessary information from the situation information DB 142, the long term memory 143 or the short term memory 144 Fetch dialog input manager 111c.

As a specific example, when the action corresponding to the user's utterance intention is a route guidance, the situation understanding unit 112c searches the situation understanding table 145 to find that the situation information related to the route guidance is the current position.

As another example, when the action corresponding to the user's utterance intention is to lower the temperature of the vehicle, the situation understanding unit 112c may search the situation understanding table 145 to obtain the temperature to lower the temperature of the vehicle. In this case, when the user inputs the reaction after the ignition to lower the window rather than increasing the intensity of the air conditioner, the temperature of the vehicle can be lowered through the operation of lowering the window even when the temperature is high.

If the current position is already stored in the short term memory 144, the situation understanding unit 112c fetches the current position from the short term memory 144 and transfers it to the dialog input manager 111c.

If the current position is not stored in the short term memory 144, the status information collection manager 112b requests the current situation information collection manager 112b and the situation information collection manager 112b causes the situation information collector 112a to request the current position from the vehicle controller 240 To obtain the current position.

The situation information collector 112a acquires the current location, stores it in the short term memory 144, and sends an acknowledgment signal to the situation information collection manager 112b. The situation information collection management unit 112b also sends an acknowledgment signal to the situation understanding unit 112c and the situation understanding unit 112c fetches the current location information from the short term memory 144 and transfers it to the dialog input management unit 111c.

The dialogue input manager 111c may forward the output of the natural language understander 111b and the output of the situation understander 112c to the dialogue manager 120 to manage the redundant input to the dialogue manager 120 have. At this time, the output of the natural language understanding unit 111b and the output of the situation understanding unit 112c may be combined and transmitted to the dialogue manager 120, or may be independently transmitted.

On the other hand, the situation information collection management unit 112b can transmit an action trigger signal to the situation understanding unit 112c when the data collected by the situation information collector 112a satisfies predetermined conditions and a specific event is generated.

The situation understanding unit 112c searches the situation understanding table 145 to search for the situation information related to the event, and when the retrieved situation information is not stored, the situation understanding unit 112c sends a request signal of the situation information to the situation information collection management unit 112b again do.

As shown in FIG. 21B, in the situation understanding table 145, types of context information and context information related to an event can be stored for each event.

For example, if the generated event is an engine temperature warning, the integer type engine temperature may be stored as related status information. If the generated event is the driver's drowsiness detection, the driver's drowsiness step of the integral type can be stored as the related status information. When the generated event is tire air pressure shortage, the integral type tire air pressure can be stored as the related situation information. When the generated event is a fuel warning, the integer type traveling distance can be stored as the related state information. If the generated event is sensor abnormal, the sensor name of the character type can be stored in the related status information.

In addition, in the case where the event that occurred is a vehicle temperature increase, the temperature of the integral type vehicle can be stored as the related status information.

The situation information collection management unit 112b collects necessary situation information through the situation information collector 112a and transmits an acknowledgment signal to the situation understanding unit 112c. The situation understanding machine 112c fetches the necessary situation information from the situation information DB 142, the long-term memory 143 or the short-term memory 144, and transfers it to the dialog input manager 111c together with the action information.

The dialogue input manager 111c inputs the output of the situation understanding machine 112c to the dialogue manager 120. [

23 is a diagram showing an example of information stored in the association action DB, FIG. 24 is a diagram showing an example of information stored in the action execution condition DB, and FIG. Is an example of information stored in the action factor DB.

22, the conversation manager 120 includes a conversation flow manager 121 that requests to create / delete / update a conversation or an action, a create / delete / update dialog or action in response to a request from the conversation flow manager 121 An ambiguity resolver 123 for resolving the ambiguity of the situation and the ambiguity of the conversation to ultimately clarify the intention of the user, a factor manager 124 for managing the factors necessary for performing the action, An action priority determinator 125 for determining the action to be performed on the candidate actions of the action candidate, and determining the priorities of the actions, an external information manager for managing the external content list and related information, (126).

The dialogue manager 120 may include a memory in which a program for performing the above-described or later-described operations is stored, and a processor for executing the stored program. At least one memory and one processor may be provided, and if a plurality of memory and processors are provided, they may be integrated on one chip or physically separated.

It is also possible that each component included in the conversation manager 120 is implemented by a single processor or by a separate processor.

Also, the dialogue manager 120 and the input processor 110 may be implemented by a single processor or by a separate processor.

The natural language understanding result (the output of the natural language understanding device) and the situation information (the output of the situation understanding device), which are the outputs of the dialogue input management device 111a, The output of the natural language understanding unit 111b includes information on the user's utterance content itself such as the result of morphological analysis in addition to the information on domain, action, and the like. The output of the situation understanding unit 112c may include an event determined by the situation information collection management unit 112b in addition to the situation information.

The conversation flow manager 121 searches whether the conversation task or the action task corresponding to the input from the conversation input manager 111a exists in the conversation / action DB 147. [

The dialogue / action DB 147 is a storage space for managing the state of the conversation and the state of the action. The dialogue / action DB 147 can store the conversation state and the action state of the ongoing conversation, the actions, and the preliminary actions to be performed. For example, the status of the ended conversation / action, the stopped conversation / action, the ongoing conversation / action, and the conversation / action to be processed can be stored.

Also, it can store the switching status of the action, whether it is nested or not, the switched action index, the action change time, and the final output status such as screen / voice / command.

For example, when a domain and an action corresponding to a user utterance are extracted, if there is a dialog or action corresponding to the domain and the action among the most recently stored conversations, the dialog and the action corresponding to the input from the dialogue input manager 111a It can be judged as an interactive task or an action task.

If a domain and an action corresponding to a user utterance have not been extracted, an arbitrary task may be created or a dialog action generator 122 may be requested to refer to the most recently stored task.

If there is no dialog task or action task corresponding to the output of the input processor 110 in the dialog / action DB 147, the dialog flow manager 121 creates a new dialog task and an action task in the dialog action manager 122 .

The conversation flow manager 121 can refer to the conversation policy DB 148 when the conversation flow manager 121 manages the conversation flow. The conversation policy DB 148 stores a policy for developing a conversation, and specifically stores a policy for selecting / starting / proposing / stopping / terminating a conversation.

In addition, the conversation policy DB 148 may store a policy for the time when the system outputs a response and a methodology, and may include a policy for creating a response by linking a plurality of services, a policy for deleting an existing action, Lt; / RTI >

For example, if there are multiple candidate actions, or if there are multiple actions corresponding to a user's intent or situation (A action, B action), then you can select "Do A action and B action? After creating a response to an action, such as "Perform an A action" or "Do you want to perform an action B," a policy that generates a response at a time generates a separate response for the B action Policies are all possible.

The conversation policy DB 147 may also store a policy for determining the priority among the candidate actions. The priority determination policy will be described later.

The dialogue action manager 122 assigns a storage space to the dialogue / action DB 147 and creates an interactive task and an action task corresponding to the output of the input processor 110. [

On the other hand, when the domain and the action can not be extracted from the user's utterance, the dialogue action manager 122 can generate an arbitrary dialogue state. In this case, as described later, the ambiguity solver 123 can grasp the user's intention based on the user's utterance contents, the surrounding situation, the vehicle condition, the user information, and the like, and determine the appropriate action corresponding thereto.

If there is an interactive task and an action task corresponding to the output of the input processor 110 in the dialog / action DB 147, the dialogue flow manager 121 instructs the dialogue action manager 122 to refer to the dialogue task and the action task request.

The action priority decider 125 retrieves the action list included in the output of the input processor 110 or the action list associated with the event and extracts the candidate action from the association action DB 146b.

As shown in the example of FIG. 23, the associated action DB 146b may represent correlated actions, relationships between them, and actions associated with the events and relationships therebetween. For example, an action such as a route guidance, a vehicle condition check, or a gas station recommendation may be classified as an associated action, and their relationship may correspond to a mutual linkage.

Therefore, in performing the guiding action, actions such as vehicle condition check, gas station recommendation, etc. may be performed together. Here, being performed together may include both being performed before and after the guidance action and being performed during the execution of the guidance action (addition by way of way).

The warning light output event can be stored as an event-action associated with the workshop guidance action, and their relationship can also be correlated.

Therefore, when a warning light output event occurs, it is possible to perform an operation for guiding the workshop according to the type of warning light or the necessity of maintenance.

In the event of an air conditioner wind-up event, it may take an action to open a window or open a sunroof depending on temperature and weather, and may change these factors based on user feedback information.

In the case where not only the action corresponding to the user's utterance but also the event determined by the situation collection manager 112b are transmitted from the input processor 110 together with the action associated with the event corresponding to the user's utterance, . In determining such an action, the user can use the post-utterance reaction after utterance.

The extracted candidate action list is transmitted to the dialogue action manager 122, and the dialogue action manager 122 adds the candidate action list to update the action status of the dialogue / action status DB 147. [

The action priority decider 125 retrieves a condition for performing each candidate action in the action execution condition DB 146c.

As shown in the example of FIG. 24, the action execution condition DB 146c may store a necessary condition for performing an action and a factor used for determining whether the condition is satisfied or not, for each action.

For example, in the case of an action to lower the temperature of a vehicle, other actions may be output depending on temperature or weather. However, when the user operates the air conditioner to lower the temperature of the vehicle, when the user turns off the air conditioner and opens the window, the user inputs a reaction after the ignition. In this case, You can output an action to open a window without increasing the air conditioning wind. The action priority manager 125 delivers the execution condition of the candidate action to the dialogue action manager 122. The dialogue action manager 122 adds the action execution condition for each candidate action to the dialogue / action state DB 147 Update the action state. The user's feedback information can be used to update the action state.

The action priority decider 125 selects a factor necessary for determining an action execution condition in the situation information DB 142, the long-term memory 143, the short-term memory 144 or the dialogue / action state DB 147 ), And judge whether or not each candidate action can be performed using the retrieved factor

When the factor used for determining the action execution condition is not stored in the situation information DB 142, the long term memory 143, the short term memory 144, or the dialog / action status DB 147, the external information manager 126 The necessary parameters can be fetched from the external content server 300 through the Internet.

The action priority decider 125 may determine whether to perform each candidate action using a factor necessary for determining an action execution condition. In addition, the priority of each candidate action can be determined based on the performance of each candidate action and the priority determination rule stored in the conversation policy DB 148. [

Depending on the current situation, scores for each candidate action can be calculated. Higher priority is assigned in ascending order of the calculated score. For example, as shown in the following formula (1), the action corresponding to the user's utterance, the safety score, the score of the convenience, the processing time, the processing time (whether or not it should be processed immediately), the user's preference (Eg, the degree of acceptance of the vehicle or the pre-defined preference of the user), manager score, vehicle status association score, and action success rate (conversation success rate).

[Equation 1]

Priority score = w1 * User spoken action + w2 * Safety score + w3 * Number of convenience stores + w4 * Processing time + w5 * Processing time + w6 * User preference + w7 * Manager score + w8 * Action success rate * Action possible (1: Possible, Undefined 0: Not possible) * Action completed (1: Completed: 0) + w10 * Feedback information

As described above, the action priority decider 125 searches not only the action directly related to the user's utterance or the situation information but also the action list associated with the action, and determines the priority between them, thereby providing the most necessary service to the user .

The action priority manager 125 transfers the performance and the priority of each candidate action to the dialogue action manager 122. The dialogue action manager 122 adds the received information to the dialogue action manager 122, Update the action state.

The parameter manager 124 can search for an argument (hereinafter, referred to as an action argument) used to perform each candidate action in the action factor DB 146a.

As shown in the example of FIG. 25, the action factor DB 146a may store an essential factor, a selection factor, an initial value of a factor, and a reference position for obtaining an argument for each action. When the initial value of the argument is stored, the argument value corresponding to the argument is not included in the user's utterance or the status information output from the input processor 110, and the corresponding argument value is also stored in the context information DB 142 If not, the action may be performed according to the stored initial value, or the user may be asked to perform an action according to the stored initial value.

For example, the required parameters used to perform the guiding action may include the current location and the destination, and the selection factor may include the path type. A fast path can be stored as an initial value of the selection factor. The current position and the destination can be obtained by sequentially searching the dialogue / action state DB 147, the situation information DB 142, the short-term memory 144, or the long-term memory 143 sequentially.

The required parameters used to perform the vehicle condition check action include vehicle condition information, and the selection parameter may include a check part. The entire value can be stored as the initial value of the selection parameter. Vehicle condition information can be obtained from the situation information DB 142. [

The selection factor used to perform the service station recommendation action may include a preferred service station and the A oil may be stored as an initial value of the selection factor. The preferred gas station can be obtained from the long term memory 143. It is also possible that the optional factors include the vehicle type, the oil price, and so on.

As described above, the factor manager 124 retrieves the factor value of the factor retrieved from the action factor DB 146a at the reference position. The reference location from which the argument value may be obtained may be at least one of the situation information DB 142, the long term memory 143, the short term memory 144, the dialogue / action state DB 147 and the external contents server 300.

When the parameter manager 124 obtains an argument value from the external content server 300, the parameter manager 124 may access the external information manager 126. The external information manager 126 may refer to the external service set DB 146d to determine where to fetch the information.

The external service set DB 146d stores information on the external content server associated with the conversation system 100. [ For example, an external service name, a description of an external service, a type of information provided by the external service, a method of using the external service, and information on the provider of the external service can be stored.

The argument values obtained by the argument manager 124 are transmitted to the dialogue action manager 122. The dialogue action manager 122 adds the argument values for each candidate action to the action state to update the dialogue / action state DB 147 do.

The argument manager 124 may obtain the argument values of all the candidate actions and may obtain only the argument values of the candidate actions determined to be executable by the action priority decider 125. [

In addition, the parameter manager 124 can selectively use various types of parameter values indicating the same information. For example, a text station in the form of text representing a destination can be converted to a station in the form of POI using the navigation destination search service.

If there is no ambiguity in the conversation or the situation, it is possible to obtain necessary information according to the actions of the action priority decider 125, the factor manager 124 and the external information manager 126, and to manage the conversation and the action. However, when there is ambiguity in the dialogue or the situation, it is difficult to provide an appropriate service necessary for the user only by the actions of the action priority decider 125, the factor manager 124, and the external information manager 126.

In this case, the ambiguity solver 123 can resolve the ambiguity about the conversation or the ambiguity about the situation. For example, when the dialogue includes ambiguities such as the person, yesterday there, father, mother, grandmother, daughter-in-law, etc., the ambiguity solver 123 may not know what to call the situation information DB 142, 143 or the short term memory 144 to solve such ambiguities or to suggest a guide for solving the ambiguities.

For example, an ambiguous word included in a conversation, such as "Yesterday there," "A mart near a house," or "Yesterday's Seoul station," may be an argument of an action factor or an argument of a conditional factor. However, in this case, the word itself can not actually perform an action or judge an action execution condition due to the ambiguity.

The ambiguity resolver 123 can resolve the ambiguity of the argument value by referring to the information stored in the situation information DB 142, the long term memory 143 or the short term memory 144. [ Alternatively, it is possible to fetch necessary information from the external content server 300 using the external information manager 126 as necessary.

For example, by referring to the short term memory 144 and searching for the place where the user went yesterday, "yesterday there" can be converted into information usable as the destination of the route guidance action. Or, referring to the long-term memory 143, the user's home address is retrieved and the location information about the A-mart near the home address is fetched from the external content server 300, so that "A-mall near home" It can be converted into usable information.

Further, in a situation where an action (object, operator) is not clearly extracted in the input processor 110 or the user's intention is ambiguous, the ambiguity resolver 123 refers to the ambiguity resolution information DB 146e to grasp the intention of the user It is also possible to determine an action corresponding to the action.

26 is a diagram showing an example of information stored in the ambiguity resolution information DB.

The ambiguity resolution information DB 146e may store and store the utterance statement and the action corresponding thereto based on the vehicle status information and the surrounding situation information. The utterance statement stored in the ambiguity resolution information DB 146e may be a utterance which can not extract an action through natural language understanding. In the example of FIG. 25, the case where the contents of the utterance according to the result of the morpheme analysis is that the hand is cold or the hand is cold will be described.

The surrounding situation information may include a temperature outside the vehicle and rainfall, and the vehicle condition information may include whether the air conditioner / heater is on / off, the air flow rate, the wind direction, and whether the steering heating line is on or off.

As a specific example, if the outside temperature is more than 20 degrees and it is raining, if the air conditioner is on, the air conditioner temperature is set too low and it can be recognized that the hand is broken, and the corresponding vehicle control action can save the air conditioner temperature increase of 3 degrees have.

When the outside temperature exceeds 20 degrees and it is raining, if the air conditioner is off, it can be recognized that it is feeling cold due to rainfall, and the heater On can be stored by the corresponding vehicle control action.

If the air conditioner is turned on and the air conditioner direction is set to Up, the air conditioner wind can be regarded as the hand being turned toward the hand, and the air conditioner The change of the wind direction to Down (Down) can be stored.

If the outside temperature exceeds 20 degrees and it is not raining, if the air conditioner is on, the air conditioner direction is downward, and the air amount is set to middle or higher, the air conditioner direction is too strong to sense cold, Lowering the air conditioner air volume can be saved.

If the outside temperature is over 20 degrees and it is not raining, if the air conditioner is turned on, the air conditioner direction is downward, and the air conditioner air volume is set weakly (lower), it is possible to store the raising of the air conditioner temperature by 3 degrees by the corresponding vehicle control action.

If the heater is off while the outside temperature is lower than 20 degrees, it can be determined that the hand is blown due to the cold weather, and the turn-on can be saved by the corresponding vehicle control action.

If the heater is turned on while the outside temperature is lower than 20 degrees and the steering heating line is turned off, the warmth can not be transmitted to the hand, and it can be determined that the hand is broken and the corresponding vehicle control action can be stored as the steering heating wire.

When the outside temperature is lower than 20 degrees and both the heater and the steering heating wire are turned on and the heater wind direction is set downward, the heater wind can not be transmitted by hand and it can be recognized that the hand is blown. Directional change of < / RTI >

If the heater temperature is set lower than the maximum value, the corresponding vehicle control action can be saved as the heater temperature increase when the outside temperature is lower than 20 degrees and both the heater and the steering heating line are on and the heater direction is set upward. have.

When the heater and the steering heating line are all turned on while the outside temperature is lower than 20 degrees and the heater wind direction is set upward and the heater temperature is set to the maximum value and the heater air amount is not set to the maximum value, Control actions can be saved with increased heater airflow.

When both the heater and the steering heating wire are both turned on while the outside temperature is lower than 20 degrees and the heater wind direction is upward and both the heater temperature and the heater wind direction are set to the maximum values, You can save hot-line sheet-on by action.

The heater temperature and the heater air volume are both set to the maximum value with the heater temperature and the heater heating direction set to the maximum value while the heater and the steering heating line are both turned on while the external temperature is lower than 20 degrees, Since the heater is currently in full operation by action, you can output a message to wait for a while.

FIGS. 27A and 27B are tables that summarize various examples in which the ambiguity resolver resolves ambiguities by referring to the ambiguity resolution information DB and extracts an action to control the vehicle.

For example, as shown in FIGS. 26A and 26B, when the contents of the utterance according to the morphological analysis result indicate that the hand is cold or the hands are cold, the surroundings are in the summer and the vehicle condition is the head of the passenger (Upward), the air conditioner set temperature is 19 degrees, and in the case of the air volume, it can be judged that the hand is blown due to the air conditioner blowing toward the hand. Accordingly, the air conditioning control action for reducing the airflow intensity can be extracted as an action corresponding to the corresponding utterance while changing the wind direction to the foot side (downward), and the vehicle can be controlled according to the extracted action.

Also, in the same utterance, when the surrounding situation is winter, the vehicle condition is that the direction of the heater is directed toward the passenger's foot, the air conditioner set temperature is 25, and the warmth is not delivered to the user's hand I can grasp that the hand is broken. Accordingly, it is possible to extract the hot line on action of the steering wheel into an action corresponding to the corresponding utterance, and control the vehicle according to the extracted action.

If the contents of the morpheme analysis result indicates that the contents of the speech are frustrating, it can be determined that the traffic condition is a frustration when the vehicle condition is less than 30 km per hour and the distance between front and rear of the vehicle is less than 30 cm. Accordingly, an action for changing a route option (quick route guidance), playing multimedia contents such as music, or turning on a chat function is extracted as an action corresponding to the corresponding utterance in the route guidance action, It is possible to control the vehicle according to the vehicle speed.

If the contents of the morpheme analysis are drowsy and the vehicle condition is the bet mode, it can be considered that drowsiness is caused by the lack of air circulation. Therefore, an action to be changed to the open air mode can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

Further, in the case of the same utterance, when the vehicle is in the outside air mode and the heater is on, it can be determined that drowsiness has been caused by the warmth emitted from the heater. Therefore, the window opening can be extracted as an action corresponding to the corresponding utterance, and the vehicle can be controlled according to the extracted action.

 If the content of the utterance according to the morphological analysis result is sweaty or hot, if the surroundings are winter and the vehicle is in a state where the heater is on, it can be determined that the heat is caused by the warmth of the heater . Therefore, the temperature lowering of the heater or the reduction of the air flow rate can be stored as an action corresponding to the corresponding utterance.

Also, if the surrounding situation is winter and the heater is turned off in the same utterance, it can be determined that the heat is caused by the user's body heat. Accordingly, the proposal for opening a window or opening a window can be extracted as an action corresponding to the corresponding utterance, and the vehicle can be controlled according to the extracted action.

Also, if the surrounding situation is summer and the vehicle condition is the air conditioner is turned off in the same utterance, the internal temperature of the vehicle is high and it can be understood that the heat is generated. Therefore, it is possible to extract the air conditioner execution (On) as an action corresponding to the corresponding utterance, and control the vehicle according to the extracted action.

If the surrounding situation is summer and the air conditioner is on, the temperature of the air conditioner is set to be high, so that it can be determined that the heat is generated. Therefore, it is possible to extract the air conditioner temperature lowering or the air flow increase to an action corresponding to the corresponding utterance, and control the vehicle according to the extracted action.

If the contents are cold according to the result of morphological analysis, if the surrounding situation is summer and the condition of the car is on, the air conditioner temperature is set too low or the air conditioner wind is too strong, . Therefore, it is possible to extract the air conditioner temperature increase or air amount decrease as an action corresponding to the corresponding ignition, and control the vehicle according to the extracted action.

Also, if the surrounding situation is in the summer and the vehicle condition is the air conditioner is off in the same utterance, it can be understood that the cold condition is caused by the problem of the user's physical condition. Therefore, the execution of the heater or the user's biorhythm check can be extracted as an action corresponding to the corresponding utterance, and the vehicle can be controlled according to the extracted action.

Further, if the surrounding situation is winter and the vehicle is in the on state (On), the temperature of the heater is set low or the air volume is low, so that it can be determined that the cold has been caused. Therefore, it is possible to extract the heater temperature increase or the airflow increase as an action corresponding to the corresponding ignition, and control the vehicle according to the extracted action.

Also, if the surrounding situation is winter and the heater is off in the same utterance, it can be understood that the cold is caused by the non-execution of the heater. Therefore, the heater execution can be extracted as an action corresponding to the corresponding utterance, and the vehicle can be controlled according to the extracted action.

If the contents of the speech according to the morphological analysis result indicate that the head is sick, if the surrounding situation is winter and the vehicle condition is the heater, the air circulation does not occur and the headache can be determined to be caused. Accordingly, the outside air mode conversion or the window opening can be extracted as an action corresponding to the corresponding utterance, and the vehicle can be controlled according to the extracted action.

In addition, if the surrounding situation is winter and the vehicle condition is the heater is turned off in the same utterance, it can be understood that the headache is caused by the cold. Therefore, the heater execution can be extracted as an action corresponding to the corresponding utterance, and the vehicle can be controlled according to the extracted action.

Also, in the same situation, if the surrounding situation is in the summer and the vehicle condition is the air conditioner is off, it can be understood that the headache is caused by the heat. Accordingly, it is possible to extract the air conditioner execution as an action corresponding to the corresponding utterance, and control the vehicle according to the extracted action.

Also, if the surrounding situation is in the summer and the air conditioner is on (On) in the same contents, it can be understood that the headache is caused by the air conditioner. Therefore, it is possible to extract the change of the wind direction or the wind amount of the air conditioner as an action corresponding to the corresponding utterance, and to control the vehicle according to the extracted action.

If the content of the spoken word according to the result of the morphological analysis is insufficient, it can be understood that, when the surrounding situation is winter and it is rainy, it is caused by the high humidity. Therefore, execution of depot function or execution of dehumidification function can be extracted as an action corresponding to the corresponding utterance, and the vehicle can be controlled according to the extracted action.

In addition, if the surrounding situation is summer and the rain does not occur in the same utterance, the seasonal characteristic and the heat can be interpreted as unpleasant. Therefore, the execution of the air conditioner at the minimum temperature can be extracted as an action corresponding to the corresponding utterance, and the vehicle can be controlled according to the extracted action.

In addition, if the surrounding situation is summer and rainy in the same utterance, it can be interpreted that the unpleasantness is caused by the heat and the high humidity. Therefore, the execution of the air conditioner in the dehumidification mode can extract the action corresponding to the corresponding speech, and control the vehicle according to the extracted action.

According to the operation of the ambiguity resolver 123 described above, even when there is ambiguity in the user's utterance or the situation, the surrounding situation information and the vehicle condition information are integrally considered together with the utterance of the user, It is possible to precisely understand and provide necessary actions.

Information about the action determined by the ambiguity resolver 123 may be communicated to the dialogue action manager 122 and the dialogue action manager 122 may update the dialogue / action database 147 based on the transmitted information .

The action priority determiner 125 and the argument manager 124 determine the action execution condition, determine the priority, and obtain the argument value for the action determined by the ambiguity resolver 123 as described above do.

The dialogue action manager 122 transmits a signal to the conversation flow manager 121. When the conversation action manager 122 receives the current state of the argument values,

On the other hand, if an argument value required for performing an action or determining a condition does not exist in the dialog / action DB 147, the external content server 300, the long term memory 143, the short term memory 144, , It is also possible for the result handler 130 to generate an interactive response for querying the user for an argument value if it can only be obtained through the user.

The conversation flow manager 121 transmits information on the action corresponding to the first rank and the conversation state to the result processor 130. It is also possible to transmit information on a plurality of candidate actions according to the conversation policy.

28 is a control block diagram in which the configuration of the result processor is subdivided.

28, the result processor 130 includes a response generation manager 131 that manages the generation of a response required to perform an action input from the conversation manager 120, An interaction response generator 132 for generating a response, an image response or an audio response, an instruction generator 136 for generating an instruction for vehicle control or a service using external content at the request of the response generation manager 131 A service editor 134 for executing a plurality of services sequentially or sporadically and collecting result values to provide a service desired by the user, a text response, an image response or an audio response, or a command generator 136 An output manager 133 for outputting the generated command and determining the output order when the output is plural, a response generation manager 131, Based on the output of the output manager 133, and a memory manager 135 to manage the long-term memory 143 and the short-term memory 144.

The result processor 130 may include a memory in which a program for performing the above-described or later-described operations is stored, and a processor for executing the stored program. At least one memory and one processor may be provided, and if a plurality of memory and processors are provided, they may be integrated on one chip or physically separated.

It is also possible that each component included in the result processor 130 is implemented by a single processor or by a separate processor.

The result processor 130, the dialogue manager 120, and the input processor 110 may also be implemented by a single processor or by separate processors.

The response output in response to the user's utterance or situation may include an interactive response, vehicle control, provision of external content, and the like. The conversation response may have a format such as an initial conversation, an inquiry, an answer including information provision, and may be stored in the response template 149 in a database. Also, the user's feedback information can be stored in the agenda template 149. [

The response generation manager 131 requests the dialog response generator 132 and the command generator 136 to generate a response required to perform the action determined by the dialogue manager 120. [ To this end, information about the action to be performed may be sent to the dialog response generator 132 and the command generator 136, and the information about the action to be performed may include an action name, an argument value, and the like. In generating the response, the conversation response generator 132 and the instruction generator 136 may refer to the current conversation state, the action state, and the feedback information of the user.

The conversation response generator 132 can search the response template 149 to extract the conversation response form and fill in the required argument values in the extracted conversation response form to generate the conversation response. The generated dialog response is transmitted to the response generation manager 131. [ When the argument values necessary for generating the conversation response are not transmitted from the conversation manager 120 or when an instruction to use the external contents is received, the contents are received from the external contents server 300 or received from the long-term memory 143, the short- And can be searched from the situation information DB 142.

For example, when the action determined by the conversation manager 120 corresponds to a route guidance, the response template 149 is searched to find that it takes [Time: -] from "[present location: -] to [destination: -] Can I start the guidance? "Can be extracted as a dialog response form.

Among the factors to be filled in the conversation response form, the argument values of [current position] and [destination] are transmitted from the conversation manager 120, and the argument value of the [required time] may not be transmitted. In this case, the conversation response generator 132 may request the external content server 300 for the time from [current location] to [destination].

The command generator 136 generates a command for executing the vehicle control or external content provision if the response to the user's utterance or situation includes vehicle control or external content provision. For example, when the action determined by the conversation manager 120 is the control of the air conditioner, the window, the seat, the AVN, or the like, a command for executing the control is generated and transmitted to the response generation manager 131.

Alternatively, when the action determined by the dialogue manager 120 requires the provision of external content, a command for receiving the content from the external content server 300 is generated and transmitted to the response generation manager 131.

When there are a plurality of commands generated by the command generator 136, the service editor 134 determines the method and order in which a plurality of commands are executed and transmits them to the response generation manager 131.

The response generation manager 131 transfers the response received from the conversation response generator 132, the command generator 136 or the service editor 134 to the output manager 133.

The output manager 133 determines the output timing, the output order, and the output position of the dialog response generated by the dialog response generator 132 and the command generated by the command generator 136.

The output manager 133 outputs the response by sending the dialog response generated by the response generator 132 and the command generated by the instruction generator 36 to an appropriate output position in an appropriate order at an appropriate timing. A Text to Speech (TTS) response may be output via the speaker 232 and a text response may be output via the display 231. In the case of outputting the conversation response in the TTS format, the TTS module provided in the vehicle 200 may be used, or the output manager 133 may include the TTS module.

The command may be transmitted to the vehicle controller 240 according to the control object, or may be transmitted to the communication device 280 for communicating with the external content server 300.

The response generation manager 131 may also communicate the response received from the conversation response generator 132, the command generator 136 or the service editor 134 to the memory manager 135.

In addition, the output manager 133 can also transmit the response output by itself to the memory manager 135. [

The memory manager 135 manages the long-term memory 143 and the short-term memory 144 based on the contents received from the response generation manager 131 and the output manager 133. [ For example, the memory manager 135 may update the short-term memory 144 by storing the conversation between the user and the system based on the conversation response generated and output, and may store the user-related information The long-term memory 143 can be updated.

The long term memory 143 may also store information that is meaningful and persistent, such as a user's propensity or preference, stored in the short term memory 144, or information that can be used to obtain such information.

Also, it may update the user's preference, the vehicle control history, and the like stored in the long-term memory 143 based on the vehicle control or the external content request corresponding to the generated and outputted command.

According to the conversation system 100 according to the above-described embodiment, it is possible to provide an optimal service necessary for the user in consideration of various situations occurring in the vehicle. In particular, even if the user's utterance is not inputted, it is possible to determine the service required for the user based on the situation information or the driver information collected by the conversation system 100, and to preliminarily provide the service. In addition, the dialog system 100 can derive the user's most preferred action in a specific situation based on the feedback information of the user and provide it to the user.

For example, the evaluation criterion of the vehicle condition may be varied according to the situation at the start of the vehicle, and feedback on the vehicle condition may be provided in advance. The driving start time is defined as when the vehicle is started, EPB (Electronic Parking Brake) is released, or when a navigation destination is set. A vehicle condition evaluation system for calculating a possible score, and individual devices are weighted, and variable weights applied to individual devices are changed according to the situation factors. When it is judged that there is a problem in the vehicle condition, it is possible to provide a solution to the problem of the individual device such as the guide to the repair shop.

In addition, it is possible to determine whether the flow rate is insufficient relative to the destination when the vehicle is started, and if the flow rate is insufficient, the user can add the user's preferred gas station as the automatic route via the route to the destination, Also, depending on the user's response, the gas station added as an automatic stopover may be changed.

In addition, even if the present vehicle state does not indicate a shortage of flow rate, it is possible to preliminarily provide a fueling location or a viewpoint considering the user's future schedule, major movement record, remaining amount, and the like in a comprehensive manner.

Further, it is possible to acquire information on the driver's physical condition and sleep record, and conditionally authorize vehicle start based on the obtained information. For example, non-driving can be recommended if the drowsiness possibility is recognized by recognizing the physical condition and the sleep record outside the vehicle. Or may provide information on recommended operating hours based on physical condition or sleep history.

In addition, when a trigger indicating the possibility of drowsiness is repeatedly generated, it is possible to detect the drowsiness possibility and output a warning according to the degree of possibility of the drowsiness, or to provide feedback to automatically change the route (rest area). The trigger indicating the drowsiness driving possibility can be obtained by manually measuring the state of the driver or the state of the vehicle as in the case where the distance from the preceding vehicle is equal to or greater than the reference distance and the speed is equal to or lower than the reference speed, It is also possible to actively acquire a response through conversation, such as querying the driver and measuring the response speed of the driver.

In addition, when the user inputs an utterance expressing the emotion, the specific system can not extract a specific domain or action from the utterance of the user, but the conversation system 100 uses the surrounding state information, vehicle state information, You can identify your intentions and develop a conversation. This example can be performed by resolving the ambiguity of user utterance in ambiguity solver 230 as described above.

Hereinafter, an example of concrete conversation processing using the conversation system 100 according to an embodiment will be described in detail.

29 to 32 show specific examples in which the conversation system 100 processes inputs, manages conversations, and outputs results when a user inputs a speech related to a decrease in vehicle temperature.

As shown in Fig. 29, when the user inputs the utterance "hot ", the voice recognizer 111a outputs the voice of the user as a text-shaped utterance (hot).

The natural language understanding unit 111b analyzes the morpheme and refers to the domain / action reasoning rule DB 141 to determine the domain: air conditioning control, action: air conditioner wind increase, Speech: simple sentiment expression] can be extracted and input to the dialog input manager 111c.

Referring to FIG. 30, the dialogue input manager 111c transmits a natural language understanding result of the natural language understanding machine 111b to the situation understanding machine 112c, and requests that the additional information exists if there is additional information.

The situation understanding unit 112c searches the situation understanding table 145 to extract that the situation information related to [domain: air conditioning control], [action: air conditioner wind increase] is the current temperature and the situation information type is an integer value.

The situation understanding machine 112c searches the situation information DB 142 to extract the current temperature value. If the current temperature value is not stored in the situation information DB 142, the current temperature value is requested to the situation information collection manager 112b.

The situation information collection manager 112b sends a signal to the situation information collector 112a to collect the current temperature value and the situation information collector 112a collects the temperature value from the vehicle controller 240 and stores the temperature value in the situation information DB 142 And transmits a temperature value collection confirmation signal to the situation information collection management unit 112b while storing it. When the situation information collection management unit 112b transmits a temperature value collection confirmation signal to the situation understanding unit 112c, the situation understanding unit 112c extracts the temperature value of the current location from the situation information DB 142, 111c.

The dialog input manager 111c integrates [domain: air conditioning control], [action: air conditioner wind increase], [speech: simple emotional expression] and [situation information: current temperature: 30 degrees] 120.

Referring to FIG. 31, the conversation flow manager 121 searches the conversation / action state DB 147 to determine whether there is an ongoing conversation task or an action task. At this time, the conversation policy DB 148 can be referred to. In this example, it is assumed that there is no interactive task or action task in progress.

The conversation flow manager 121 requests the conversation action manager 122 to create an action task and an interactive task corresponding to the output of the input processor 110. [ Creating an action task and an interactive task means allocating storage space for storing and managing information about action state and conversation state.

Accordingly, the conversation action manager 122 allocates storage space in the conversation / action state DB 147 to store information about the action state and information about the conversation state.

The dialogue action manager 122 delivers the action state and the dialogue state to the action priority determiner 125. [

The action priority determiner 125 retrieves in the associated action DB 146b the actions associated with the vehicle temperature decrease, air conditioner wind increase, window down and sunroof opening. Air conditioner wind increases and conjoint actions become candidate actions.

The action priority decider 125 may determine the priority of each candidate action according to a rule stored in advance. The priority order may be determined before the execution condition for each candidate action is determined, or may be determined only for candidate actions that satisfy the execution condition after the execution condition is determined. At this time, the interactive system 100 can output the learned operation as an action when the user learns the preferred operation to lower the temperature of the vehicle based on the feedback information of the user. For example, if the conversation system 100 performs an operation to increase the wind of the air conditioner by an action factor, if the user repeatedly performs the operation of turning off the air conditioner and lowering the window, the conversation system 100 may allow the user to lower the temperature It is determined that the operation of lowering the window is preferred and the operation of lowering the window without increasing the wind of the air conditioner can be outputted.

The candidate action list is transmitted to the dialogue action manager 122 again, and the dialogue action manager 122 adds the searched action to update the action state.

Referring to FIG. 32, the action priority decider 125 retrieves a factor for determining an execution condition and an execution condition for each candidate action in the action execution condition DB 146c. In addition, the priority between each candidate action can also be determined.

For example, the action of reducing the temperature of the vehicle may be less than 30 degrees Celsius, and the factors required to determine the conditions of execution are temperature and weather. However, if the user inputs the action of lowering the window in the action of decreasing the temperature of the vehicle as feedback information, it may output an action of lowering the window rather than the candidate candidate line. The dialogue action manager 122 updates the action state by adding a condition for performing each candidate action and a factor necessary for condition determination to the dialogue / action state DB 147. [

The action priority determinator 125 stores an argument value necessary for judging whether or not the execution condition of each candidate action satisfies as an interaction / action state DB 147, a situation information DB 142, a long term memory 143 or a short term memory 144 ).

If the argument value is contained in previous conversation contents, included in the context information related to the conversation contents, or included in the reimbursement information related to the generated event, the argument value may be fetched from the conversation / action state DB 147.

It is possible to request the external information manager 126 when the argument value can not be fetched from the dialogue / action state DB 147, the situation information DB 142, the long term memory 143 or the short term memory 144. [

It is also possible to fetch the factor value from the mobile device 400 connected to the vehicle 200 as needed. For example, when user information such as contacts, schedules, and the like that are not stored in the long-term memory 143 is required as an argument value, the external information manager 126 requests information necessary for the mobile device 400 to acquire necessary factor values can do.

Also, when the parameter value can not be obtained through the storage unit 140, the external content server 300, and the mobile device 400, the user can inquire the user and obtain necessary parameter values.

The action priority decider 125 determines the execution condition of each candidate action using the argument value. Since the destination distance has not been retrieved, the judgment of the execution condition is suspended once for the vehicle condition check action and the gas station recommendation action.

As shown in FIG. 33, the dialogue action manager 122 updates the action state by adding to the dialogue / action state DB 147 the satisfaction of the action execution condition determined using the argument value and the corresponding argument value . User feedback information may be used in updating the action state.

The dialogue action manager 122 requests the argument manager 124 for a list of arguments used to perform each candidate action.

The factor manager 124 extracts the current position and the destination from the action factor 146a as an essential factor used for performing the guiding action, and extracts the feedback information as a selection factor.

The extracted argument list may be passed to the dialogue action manager 122 and used to update the action state.

The factor manager 124 includes a dialogue / action state DB 147, a situation information DB 142, a long-term memory 143, and a short-term memory 144 (or both) to obtain factor values corresponding to the required factors and selection factors of each candidate action. And if the argument value is to be provided through the external service, the external content server 126 may request the external content server 300 for the required argument value.

The factors used to determine the execution condition of the candidate action and the factors used to perform the candidate action may overlap each other. If there is a factor corresponding to the factor (essential factor, selection factor) used for performing the candidate action among the factor values acquired by the action priority determiner 125 and stored in the dialogue / action state DB 147, it can be used.

Hereinafter, a dialog processing method according to one embodiment will be described. The dialogue processing method according to one embodiment may be applied to the interactive system 100 described above or the vehicle 200 including the same. Accordingly, the description of FIGS. 1 to 33 may be applied to the dialog processing method according to the embodiment.

34 is a flowchart illustrating a method of processing user input in a dialog processing method according to an embodiment. A method of processing user input may be performed in the input processor 110 of the dialog system 100.

Referring to FIG. 34, when the user's utterance is input (YES in step 500), the voice recognizer 111a recognizes the entered utterance (510). The user's utterance may be input through the voice input device 210 provided in the vehicle 200 or the voice input device 410 provided in the mobile device 400. [

The voice recognizer 111a recognizes the input user's utterance and outputs it as a text-based utterance.

The natural language understanding unit 111b applies the natural language understanding technique to the textual utterance statement (520) and outputs the result.

Specifically, in the natural language understanding process 520, a morphological analysis 521 is performed on the basis of a speech utterance in the form of a text, a domain is extracted 522 based on a morphological analysis result, an entity name is recognized 523, Analyzing the speech (524), and extracting (525) the action.

The domain / action reasoning rule DB 141 can be referred to for domain extraction, object name recognition, and action extraction.

The output of the natural language understanding device 111b, that is, the natural language understanding result may include a domain, an action, an action, and a morphological analysis result corresponding to a user utterance.

And retrieves context information related to the extracted action (530). The context information related to the extracted action may be stored in the context understanding table 145. The context understanding unit 112c searches the context information related to the action extracted from the context understanding table 145, Long-term memory 143, or the short-term memory 144, as shown in FIG.

When there is context information that can not be fetched from the situation information DB 142, the long-term memory 143, or the short-term memory 144 in the case where additional situation information is required (YES in 540) 550). The non-voice input such as the vehicle status information, the surrounding information, and the driver information can be inputted through the situation information collector 111a independently of the user's utterance input.

Such information may be periodically input, input at the time of occurrence of a specific event, periodically input, and further input at the time of occurrence of a specific event. In any case, when information collection is requested, the information can be actively collected.

Accordingly, when the situation information related to the action is already collected, the corresponding information is fetched from the situation information DB 142, the long term memory 143 or the short term memory 144, and if not, the situation information collector 111a Collect the relevant information through.

When the situation information collector 111a requested to collect the situation information collects the situation information and stores the collected situation information in the situation information DB 142, the situation understanding device 112c can obtain the situation information from the situation information DB 142 have.

On the other hand, the situation information collection management unit 112b can transmit an action trigger signal to the situation understanding unit 112c when the data collected by the situation information collector 112a satisfies predetermined conditions and a specific event is generated.

The situation understanding unit 112c searches the situation understanding table 145 to search for the situation information related to the event, and when the retrieved situation information is not stored, the situation understanding unit 112c sends a request signal of the situation information to the situation information collection management unit 112b again do.

When the collection of the necessary situation information is completed, the natural language understanding result and the situation information are transmitted to the dialogue manager 120 (560). When an event is generated, information related to the event (which event has occurred) and status information related to the generated event can be transmitted together.

35 is a flowchart showing a method of managing conversation using the output of an input processor in the conversation processing method according to an embodiment. The method of managing the conversation can be performed in the conversation manager 120 of the conversation system 100.

Referring to FIG. 35, the conversation flow manager 121 searches 600 for a conversation history in the conversation / action status DB 147.

In this example, the domain and the action can be extracted from the user's utterance. However, the domain and the action can not be extracted from the utterance of the user because the utterance itself or the situation is ambiguous. In this case, the dialogue action manager 122 generates an arbitrary dialogue state, and the ambiguity solver 123 grasps the intention of the user based on the contents of the user's utterance, surrounding conditions, vehicle conditions, user information, The appropriate action can be determined. Feedback information corresponding to the user's utterance can be used to grasp the intention of the user.

If there is a related conversation history (600 example), it is referenced (690), and if not (600 no), a new conversation task and an action task are created (610).

In the association action DB 146b, an association action list associated with an action (hereinafter referred to as an input action) extracted from the user's utterance is searched to generate a candidate action list 620. The input action and the associated action become the candidate action list.

The execution condition for each candidate action is retrieved from the action execution condition DB 146c (620). The execution condition is a condition necessary for executing the action. If the condition is satisfied, it is determined that the action can be performed. If the condition is not satisfied, the execution condition is determined to be impossible. The action execution condition DB 146c also stores information about the type of the argument used for judging the action execution condition. The factor may include user feedback information.

And acquires an argument value used for determining an action execution condition (640). The factor used in judging the action execution condition is the condition judgment factor. The state information DB 142, the long-term memory 143, the short-term memory 144, or the dialogue / action state DB 147 to obtain the argument value of the condition determination factor. When the parameter value of the condition determination factor is to be provided through the external service, the external content server 300 can receive the required parameter value from the external content server 300.

In addition, when a situation or an utterance is ambiguous and a necessary factor value can not be obtained, ambiguity can be solved through the ambiguity solver 230 to obtain necessary factor values.

Also, even when the obtained factor is an ineffective factor that is difficult to be used in determining the condition for performing the action, the ambiguity resolver 230 can obtain the validity factor from the non-validity factor.

(650) and determines the priority of the candidate actions (660) based on the acquired condition determination factors. The rules for determining the priority of the candidate actions may be stored in advance. The action priority decider 125 may determine the priority by considering only the candidate actions that can be performed after the action is determined to be performed. Alternatively, after determining the priority irrespective of whether or not the actions are performed, the priority may be adjusted based on the action execution capability. Or may prioritize the user's preferred action based on the feedback information of the user.

In the action argument DB 146a, an argument list used for performing the candidate action is retrieved (670). The argument used to perform the action is called the action argument. Action arguments can contain required and optional arguments.

An argument value of an action factor used in performing the candidate action is obtained (680). Long-term memory 143, short-term memory 144, or conversation / action state DB 147 to obtain the argument value of the action factor. When the argument value of the action factor is to be provided through the external service, the external content server 300 can receive the required parameter value through the external information manager 126.

In addition, when a situation or an utterance is ambiguous and a necessary factor value can not be obtained, ambiguity can be solved through the ambiguity solver 230 to obtain necessary factor values.

Also, even if the obtained factor is a non-valid factor that is difficult to use in performing the action, the ambiguity resolver 230 can obtain the valid factor from the non-valid factor.

The conversation state and action state managed by the conversation action manager 122 can be updated each time the state is changed while the above-described steps are performed.

Once all the obtainable parameter values are obtained, the conversation flow manager 121 delivers the information about the candidate action and the conversation state to the result handler 130. According to the conversation policy, it is also possible to transmit only the information related to the action corresponding to the first rank, and it is also possible to transmit the information concerning the plurality of candidate actions.

On the other hand, if the required parameter value is not present in the external content server 300, the long term memory 143, the short term memory 144, and the situation information DB 142, It is also possible to output a dialog response to query.

36 is a flowchart showing a result processing method for generating a response corresponding to the result of conversation management in the conversation processing method according to the embodiment. The result processing method may be performed by the result processor 130 of the interactive system 100. [

Referring to FIG. 36, in the case where it is necessary to generate a conversation response (YES in step 700), the conversation response generator 132 searches for a response template 149 (step 710). In the response template 149, an interactive response form corresponding to the current dialog state and the action state is extracted, and the dialog response is generated by filling the argument values necessary for the dialog response form (720).

When the argument values necessary for generating the conversation response are not transmitted from the conversation manager 120 or when an instruction to use the external contents is received, the contents are received from the external contents server 300 or received from the long-term memory 143, the short- And can be searched from the situation information DB 142. If the required parameter value is not present in the external content server 300, the long term memory 143, the short term memory 144, and the situation information DB 142 and can be obtained only through the user, It is also possible to generate a conversation response for.

When command generation is required (760), the command generator 136 generates a command for vehicle control or external content utilization (770).

The generated conversation response or command is input to the output manager 133, and the output manager 133 may determine the output order between the conversation response and the command or the output order between the plurality of commands (730).

The memory is updated 740 based on the generated conversation response or command. The memory manager 135 can store the contents of the conversation between the user and the system on the basis of the generated and outputted conversation response, thereby updating the short-term memory 144, storing the user-related information acquired through the conversation with the user, The memory 143 can be updated. Also, based on the generated and outputted vehicle control or the external content request, the user's preference, the vehicle control history, and the like based on the feedback information of the user stored in the long-term memory 143 can be updated.

The output manager 133 sends the dialog response and the command to the appropriate output location and outputs a response (750). The TTS response may be output via the speaker 232 and the text response may be output via the display 231. [ The command may be sent to the vehicle controller 240, the external content server 300 or the communication device 280 for communicating with the external content server 300 according to the control object .

37 is a flowchart illustrating an operation of receiving feedback information of a user according to an embodiment.

Referring to FIG. 37, when a user inputs a speech (800), the dialog system may output a response based on a pre-stored action factor or condition determination factor (810). Based on the response output by the dialog system 100, the user may enter the response after the speech (820). The user's post-utterance reaction may be a negative reaction such as an ignition as described above or a negative reaction. Based on the post-utterance response of the user, the conversation system 100 may generate feedback information (830). In addition, condition decision factors and action factors may be formed based on the feedback information, or existing factors may be changed (840).

The method for processing a conversation according to an embodiment is not limited by the order in the above-described flowchart. The flow according to the flowcharts of FIGS. 34 to 37 is merely an example that can be applied to the dialogue processing method, and it is also possible that a plurality of steps are performed simultaneously, and the order of the steps is changed.

The foregoing description is merely illustrative of the technical idea of the present invention, and various modifications, alterations, and permutations thereof may be made without departing from the essential characteristics of the present invention. Therefore, the embodiments and the accompanying drawings described above are intended to illustrate and not limit the technical idea, and the scope of the technical idea is not limited by these embodiments and the accompanying drawings. The scope of which is to be construed in accordance with the following claims, and all technical ideas which are within the scope of the same shall be construed as being included in the scope of the right.

According to the conversation system described above, the vehicle and the conversation processing method including the conversation system, the conversation processing technique specialized for the vehicle environment can be used to provide the optimum service that meets the user's intention or the user needs.

In addition, it is possible to provide an optimal service required for a user in consideration of various situations occurring inside the vehicle. In particular, it is possible to judge the service required for the user based on the situation information or the driver information collected by the conversation system 100 regardless of the user's utterance, and to preliminarily provide the necessary service.

100: Conversation system
110: input processor
120: Conversation Manager
130: result handler
200: vehicle
210: Voice input device
220: Non-voice input device
230: talker output device
280: communication device

Claims (33)

An input processor for inputting a user's utterance and a reaction after the utterance of the user and extracting an action corresponding to the utterance and the reaction after utterance;
A storage unit for storing information values of status information including at least one of vehicle status information related to the status of the vehicle, driving environment information related to the driving environment of the vehicle, and feedback information related to the reaction after the user's utterance;
Forming the feedback information based on the post-ignition reaction corresponding to the ignition,
Forming at least one of the condition determination factor and the action factor based on the feedback information,
Acquiring an argument value of a conditional judgment factor used for judging whether or not the action corresponding to the utterance and the reaction after the utterance is to be performed is obtained from the storage unit rotor and the action to be performed based on the parameter value of the conditional judgment factor and the feedback information A dialogue manager for obtaining an argument value of an action factor used to perform the determined action from the storage; And
And a result processor for generating a response for performing the determined action using the argument value of the obtained action factor.
The method according to claim 1,
Wherein,
Storing the feedback information related to the action for each action,
Wherein the input processor comprises:
Acquiring the feedback information related to the action corresponding to the utterance from the storage unit, and transmitting the obtained feedback information to the conversation manager. 3. The method of claim 2,
The conversation manager includes:
And an argument value of the condition determination factor or an argument value of the action factor is changed based on the feedback information derived from the post-utterance response transmitted from the input processor. 3. The method of claim 2,
The conversation manager includes:
Wherein at least one action associated with an action corresponding to the feedback information is obtained from the storage. 5. The method of claim 4,
The conversation manager includes:
And determine a priority between the action corresponding to the utterance and the associated at least one action based on the feedback information. 6. The method of claim 5,
The conversation manager includes:
Wherein the control unit changes the parameter value of the condition judging factor used for judging whether the associated action is performed based on the feedback information, acquires the condition judging factor from the storage unit rotor, To determine whether to perform the associated at least one action. The method according to claim 6,
The conversation manager includes:
Determining an action corresponding to the utterance and at least one action associated therewith as an action to be performed, the action having the highest priority determined based on the feedback information. The method according to claim 1,
And a communication unit for communicating with an external server,
The conversation manager includes:
And requests the external server for the feedback information if the feedback information can not be acquired from the storage unit. The method according to claim 1,
The result processor comprises:
And generate a command for controlling the conversation response and the vehicle in response to performing the determined action. The method according to claim 1,
And a communication unit that receives the information value of the situation information from the vehicle and transmits the response to the vehicle. The method according to claim 1,
And a communication unit for receiving the information value of the status information from the mobile device connected to the vehicle, and transmitting the response to the mobile device.
Storing information values of status information including at least one of vehicle status information related to the status of the vehicle, driving environment information related to the driving environment of the vehicle, and feedback information related to the reaction after the user's utterance;
Receiving a user's utterance and a reaction after the utterance of the user;
Forming the feedback information based on the post-ignition reaction corresponding to the ignition,
Forming at least one of a condition determination factor and an action factor based on the feedback information,
At least one of an argument value of a condition judging factor used for judging whether or not to perform at least one candidate action including an action corresponding to the utterance and an argument value of an action factor used for performing the at least one candidate action, The rotor of the storage unit is acquired,
Determining an action to be performed among the at least one candidate action based on the obtained value of the condition judging factor and the feedback information;
And generating a response for performing the determined action using the argument value of the obtained action factor. 13. The method of claim 12,
Storing the feedback information related to the action in the storage unit for each action;
And obtaining the feedback information related to the action corresponding to the utterance from the storage unit and transmitting the obtained feedback information to the conversation manager. 14. The method of claim 13,
And changing an argument value of the condition determination factor or an argument value of the action factor based on the feedback information derived from the post-utterance response transmitted from the input processor. 14. The method of claim 13,
Obtaining at least one of the at least one factor value to the storage means,
And acquiring at least one action associated with the action corresponding to the feedback information from the storage unit 16. The method of claim 15,
And determining a priority between the action corresponding to the utterance and the associated at least one action based on the feedback information. 17. The method of claim 16,
Determining an action to be performed among the at least one candidate action,
Wherein the control unit changes the parameter value of the condition judging factor used for judging whether the associated action is performed based on the feedback information, acquires the condition judging factor from the storage unit rotor, And determining whether to perform the associated at least one action. The method according to claim 6,
Determining an action to be performed among the at least one candidate action,
Determining an action corresponding to the utterance and at least one action associated therewith as an action to be performed, the action having the highest priority determined based on the feedback information. The method according to claim 1,
Communicating with an external server,
And requesting the feedback information to the external server when the feedback information can not be obtained from the storage unit. The method according to claim 1,
Generating a response to perform the determined action comprises:
And generating a command for controlling the conversation response and the vehicle in response to the determined action. The method according to claim 1,
Receiving the information value of the situation information from the vehicle, and transmitting the response to the vehicle. The method according to claim 1,
Receiving an information value of the status information from a mobile device connected to the vehicle, and transmitting the response to the mobile device. An input processor for receiving a user's utterance and a user's feedback reaction and extracting an action corresponding to the utterance;
A storage unit for storing information values of status information including at least one of vehicle status information related to the status of the vehicle and driving environment information related to the driving environment of the vehicle;
Forming at least one of the condition determining factor and the action factor on the basis of the feedback reaction corresponding to the utterance,
Determining an action to be performed based on an argument value of the condition determination factor, performing an action to be performed on the action, A dialogue manager for acquiring an argument value of an action factor used for the action; And
And a result processor for generating a response to perform the determined action using the argument value of the obtained action factor. 24. The method of claim 23,
Wherein,
Storing the feedback information related to the action for each action,
Wherein the input processor comprises:
Acquiring the feedback information related to the action corresponding to the utterance from the storage unit and transmitting the obtained feedback information to the conversation manager 25. The method of claim 24,
The conversation manager includes:
And changes the factor of the condition determination factor or the factor of the action factor using the feedback information transmitted from the input processor. 25. The method of claim 24,
The conversation manager includes:
And acquiring at least one action associated with the action corresponding to the utterance from the storage. 27. The method of claim 26,
The conversation manager includes:
And determine a priority between the action corresponding to the utterance and the associated at least one action based on the feedback information. 28. The method of claim 27,
The conversation manager includes:
Wherein the control unit changes the parameter value of the condition judging factor used for judging whether the associated action is performed based on the feedback information, acquires the condition judging factor from the storage unit rotor, To determine whether to perform the associated at least one action. 29. The method of claim 28,
The conversation manager includes:
Determining an action corresponding to the utterance and at least one action associated therewith as an action to be performed, the action having the highest priority determined based on the feedback information. 24. The method of claim 23,
And a communication unit for communicating with an external server,
The conversation manager includes:
And requests the external server for the factor value if the factor of the condition factor or the factor of the action factor can not be obtained from the storage unit. 24. The method of claim 23,
The result processor comprises:
And generate a command for controlling the conversation response and the vehicle in response to performing the determined action. 24. The method of claim 23,
And a communication unit that receives the information value of the situation information from the vehicle and transmits the response to the vehicle. 24. The method of claim 23,
And a communication unit for receiving the information value of the status information from the mobile device connected to the vehicle, and transmitting the response to the mobile device.

Images