JP6513869B1 - Dialogue summary generation apparatus, dialogue summary generation method and program - Google Patents

Abstract

[PROBLEMS] To generate a highly accurate summary sentence which is sufficiently shortened and which sufficiently reflects emotions in a speaker's speech during dialogue from dialogue speech. A dialog speech summary generation apparatus separates dialog speech data into utterance units for each speaker identified by a speaker identification unit that identifies a speaker of dialog from conversation speech data. A speech separation unit, a speech recognition unit that generates speech speech text by speech recognition of speech speech data in units of speech separated by the speech separation unit, and speech speech text generated by the speech recognition unit is summarized. A summary sentence generation unit for generating a summary sentence text, analyzing dialogue speech data in units of speech to derive an emotional expression for each speaker, and adding the derived emotional expression to the summary text, or And an emotion analysis unit that replaces a part of the summary text or outputs it in association with the summary text. [Selected figure] Figure 3

Description

  The present invention relates to a dialogue summary generation apparatus, dialogue summary generation method and program. More particularly, the present invention provides a summary from recorded dialog speech available to, for example, a Customer Relationship Management (CRM) system that records, stores and manages customer and agent calls or face-to-face interactions. The present invention relates to a technique for creating and outputting a generated summary.

  Various techniques have been proposed for recording and managing on the business operator side dialogue voices made between the customer and the business operator. In recent years, for the purpose of compliance compliance of business operators, measures against customer complaints, evaluation and education of business operators, conversation contents in all situations, including not only telephone calls but also face-to-face dialogues It is requested that recordings of

  As an example, in a call recording system for digitizing, recording and searching the call contents of an operator in a call center which is a section handling a call from a customer, a public telephone generally operates a public telephone at a premises such as a call center operated by a business operator. A switch (PBX) is installed to concentrate call origination and reception from a switched switched telephone network (PSTN), and this switch distributes voice calls to fixed telephones in a call center site.

  Therefore, if a call recording server branching from the exchange is provided, it is possible to record and store a call in an audio data file. On the operator side, a terminal device such as a PC (Personal Computer) may be provided along with an extension telephone for voice response, and this operator terminal device is searched for customer information using, for example, a customer name told by a speaker as a key And a function to display the past call history of the customer.

The voice call between the customer and the operator recorded and stored in the voice data file in this way is recorded and maintained as a call history for each call, and the call history is viewed and reported after the call ends. It is required to enable output. In order to quickly verify and verify the contents of this response history, it is desirable to generate a textified summary from the recorded voice call.
As a technique for creating a summary text from such voice data, there is a technique for converting the voice in the voice data file into a character code by speech recognition processing and generating a summary text from the character coded voice text data. The creation of a text-based summary makes it easy to understand the contents of the response history, provides a list-like feature, and enables flexible collaboration with a computer, such as executing a search using words in the text as keywords. It becomes possible.

  For example, Patent Document 1 recognizes the voice recorded on a recording medium by a video tape recorder (VTR) as voice recognition and converts it into a character code string, and the importance of the component of the sentence in the character code string recognized as voice. Typically, it is judged by referring to the importance table registered in advance, classified according to parts of speech such as nouns, verbs, particles, adjectives, etc., and classified according to phrases such as subject, purpose case, predicate etc. Disclosed is a technique for automatically generating a summary sentence by combining sentence components judged to be of high importance.

  Further, Patent Document 2 extracts important sections from speech, detects topic boundaries using the appearance distribution of the extracted important sections, and semantically classifies the important sections included in each of the topic sections. Disclosed is a technology for generating a text summary divided into topics from speech.

Unexamined-Japanese-Patent No. 8-212228 gazette Unexamined-Japanese-Patent No. 2000-284793

  However, it is difficult to immediately apply the techniques disclosed in the above-mentioned patent documents to, for example, telephone service operations in a call center. Because voice communication between the customer and the operator usually involves acquisition / confirmation of customer information, acquisition / confirmation of inquiry content, acquisition / confirmation of response content to inquiry, presentation / confirmation of customer's understanding level and disclaimer content, etc. It is inevitably redundant because there are many steps, and as the same utterance content is repeated, the dialogue often takes a long time. In addition, the call recording data that is recorded and accumulated all day for a large number of operators becomes enormous, which makes it difficult to quickly confirm and check the response history.

  For this reason, there is a disadvantage that the generated summary also inevitably becomes redundant and long even if a known summary creation technology is applied to the voice call text obtained by speech recognition of the voice call as it is, The convenience was poor.

On the other hand, the speaker's emotion during the conversation is not uniform. For example, if the utterance "Yes" is recognized during the dialogue, whether the speaker of the utterance is "Yes" who uttered in a comfortable manner or "Yes" who was compelled to agree compelled, It may be an utterance based on different emotions.
However, according to the prior art, it was not possible to reflect the emotion in the utterance of the speaker during dialogue in the summary.

  The present invention has been made in view of the above problems, and an object thereof is a high-accuracy summary sentence sufficiently shortened from dialogue speech and sufficiently reflecting emotions in a speaker's speech during dialogue. A dialogue summary generation apparatus, a dialogue summary generation method, and a program capable of generating

  According to an aspect of the present invention, there is provided a speaker identifying unit for identifying a speaker of a dialog from dialog voice data, and the dialog voice for each speaker identified by the speaker identifying unit. A speech separation unit for separating data into speech units; a speech recognition unit for speech recognition of the dialogue speech data by the speech units separated by the speech separation unit to generate dialogue speech text; and the speech recognition unit A summary generation unit for summarizing the generated dialog voice text to generate a summary text, and analyzing the dialog voice data by the utterance unit to derive an emotional expression for each speaker, and the derived emotional expression An emotion analysis unit which appends to the summary sentence text or replaces a part of the summary sentence text with the emotion expression, or outputs it in association with the summary sentence text; Apparatus is provided.

  The emotion analysis unit further analyzes the dialogue speech data in units of the utterance to derive a transition of emotions in time series of each speaker in one dialogue, and the emotion analysis unit extracts the emotion derived for each speaker. A transition may be output in association with the summary text.

The dialogue summary generation apparatus further extracts an emotion word indicating an emotion of each speaker from the dialogue speech text generated by the speech recognition unit, and converts the extracted emotion word into a corresponding emotion expression. A second emotion analysis unit may be provided that replaces at least a portion of the abstract sentence text with the transformed emotion expression.
The dialog voice generation apparatus further determines whether the same or similar text appears multiple times in the dialog voice text of one dialog unit, and when the same or similar text appears multiple times. The method may further comprise a redundancy exclusion unit that deletes text appearing forward in time series.
The redundancy exclusion unit further refers to the keyword table which defines keywords in advance, extracts the text defined in the keyword table from the interactive speech text, and positions the text immediately before the extracted text. In addition, the second text may be searched for at least partially matching the reading of the extracted text, and the searched text may be deleted from the interactive speech text.

The dialogue summary generation apparatus further analyzes the dialogue speech text generated by the speech recognition unit to extract a number sentence, and adds a different unit and weight according to the type of the extracted number sentence, and generates the summary. A text correction unit for supplying to a unit may be provided.
The dialogue summary generation apparatus may further include a voice acquisition unit for recording a talk voice or a face-to-face dialogue voice to acquire the dialogue voice data.

  According to another aspect of the present invention, there is provided the steps of: identifying a speaker of a dialog from dialog voice data; separating the dialog voice data into utterance units for each identified speaker; and Speech recognition in the separated speech unit to generate dialogue speech text, summarizing the generated dialogue speech text to generate a summary text, and speech information in the speech unit The analysis is performed to derive an emotional expression for each speaker, and the derived emotional expression is added to the summary text, or a part of the summary text is replaced with the emotion expression, or the abstract text is associated with the summary text And outputting the dialog summary generation method.

  According to still another aspect of the present invention, there is provided an interactive speech summary generation program for causing a computer to execute a dialog summarizing process, wherein the program identifies the speaker of the dialogue from the spoken speech data in the computer. A speaker identification process, a speech separation process for separating the dialogue speech data into speech units for each identified speaker, and speech speech recognition by speech recognition of the dialogue speech data in the separated speech units A speech recognition process to be generated, a summary generation process of summarizing the generated dialog speech text to generate a summary text, and analysis of the dialog speech data on a speech basis to derive an emotional expression for each speaker Adding the derived emotion expression to the summary sentence text or replacing part of the summary sentence text with the emotion expression or corresponding to the summary sentence text And emotion analysis to only output is intended for causing a process including execution, interactive summarization program is provided.

  According to the dialog summary generation apparatus, the dialog summary generation method, and the program according to the present invention, a highly accurate summary sentence sufficiently shortened from the dialog speech and sufficiently reflecting the emotion in the speaker's speech during the dialog Can be generated. Therefore, it contributes to the improvement of the usefulness of the summary of dialogue speech.

It is a figure showing an example of the network configuration of the speech processing system concerning the embodiment of the present invention. It is a block diagram which shows an example of a function structure of the speech recognition server which comprises the speech processing system of FIG. It is a block diagram which shows an example of a function structure of the abstract generation server which comprises the audio processing system of FIG. It is a flowchart which shows an example of the processing flow of the speech recognition process which the speech recognition server of FIG. 2 performs. It is a figure explaining the speaker identification (S2) of FIG. 4 with respect to audio | voice data, and isolation | separation to the speech unit of an audio | voice (S3). It is a flowchart which shows an example of a processing flow of a summary production | generation process which the summary production | generation server of FIG. 3 performs. It is a flowchart which shows an example of a detailed processing flow of isolation | separation (S5) to conversion to a natural speech and a summary unit of FIG. It is a flowchart which shows an example of a detailed processing flow of isolation | separation (S53) to the summary unit of FIG. It is a figure which shows an example of the recognition result text by which speech recognition was carried out to the utterance unit by execution of S4 of FIG. It is a figure which shows an example of a syntactic analysis result of the recognition result text of FIG. It is a figure which shows an example of the morphological analysis result of the recognition result text of FIG. FIG. 10 is a diagram showing an example of recognition result text in which the recognition result text in FIG. 9 is separated into summary units by execution of S5 in FIG. 4. It is a flowchart which shows an example of a detailed processing flow of the competition analysis (S6) of FIG. FIG. 6 is a diagram for explaining the speech analysis (S6) of FIG. 4 for the audio data of FIG. 5; It is a figure which shows an example of the punctuation mark table applied with respect to a recognition result text. It is a figure which shows an example of the unit weight table applied with respect to a recognition result text. It is a figure which shows an example of the unnecessary word table applied with respect to a recognition result text. It is a figure which shows an example of the character substitution table applied with respect to a recognition result text. It is a figure which shows an example of the important word table applied with respect to a recognition result text. It is a figure which shows an example of the positive word table applied with respect to a recognition result text. It is a figure which shows an example of the negative term table applied with respect to a recognition result text. It is a figure which shows an example of the output of the emotion-analysis result with respect to a customer's dialogue speech. It is a figure which shows an example of the output of the emotion-analysis result with respect to an operator's dialogue speech. It is a figure which shows an example of the output of the emotion-analysis result with respect to the several conversational speech of an operator. It is a figure which shows an example of the emotion word table applied with respect to a recognition result text. It is a figure which shows an example of the summary text to which the emotion analysis result was added. It is a figure which shows the example of application to the summary text of the emotional word table of FIG. It is a figure which shows an example which isolate | separated the recognition result text of speech interaction into a summary unit for every speaker. It is a figure which shows the other example which isolate | separated the recognition result text of speech interaction into a summary unit per speaker. It is a figure which shows an example of the abstract sentence produced | generated from the recognition result text of FIG. It is a figure which shows an example of the user interface of the summary display of interactive speech. It is a figure which shows the example of a display of the emotion-analysis result which can be displayed with the summary of dialog speech. It is a figure which shows the example of a display of the speech recognition result of a dialog speech, a natural language processing result, and a corresponding summary result. It is a figure which shows an example of the hardware constitutions of each apparatus in this embodiment.

  Hereinafter, an embodiment for carrying out the present invention will be described in detail with reference to the attached drawings. The embodiment described below is an example as a realization means of the present invention, and should be appropriately corrected or changed according to the configuration of the apparatus to which the present invention is applied and various conditions. It is not necessarily limited to the embodiment. Moreover, not all combinations of the features described in the present embodiment are essential to the solution means of the present invention. In addition, about the same structure, the same code | symbol is attached | subjected and demonstrated.

<Network Configuration of Speech Processing System of This Embodiment>
The following describes an example of recording a call made between a customer and an operator of a call center via a telephone network, but the present embodiment is not limited to this. In the present embodiment, for example, instead of a call, a summary can be similarly generated also for dialogue voice in which a face-to-face dialogue is collected and recorded by a sound collection device such as a microphone.
FIG. 1 is a diagram showing a non-limiting example of the network configuration of the voice processing system according to the present embodiment. Referring to FIG. 1, the speech processing system includes a PBX (switching machine) 1, speech acquisition server 2, call recording server 3, control server 4, speech recognition server 5, emotion analysis server 6, summary generation server 7, and dialogue summary. It has a PC (Personal Computer) 9 available for inquiry. The PBX 1, the voice acquisition server 2, the call recording server 3, the control server 4, the speech recognition server 5, the emotion analysis server 6, the summary generation server 7, and all or part of the PC 9 are installed in a call center site and are LAN (Local Area) They may be interconnected by an IP (Internet Protocol) network such as an intranet 8 such as a Network) / WAN (Wide Area Network).

Alternatively, all or part of the voice acquisition server 2, the call recording server 3, the control server 4, the speech recognition server 5, the emotion analysis server 6, the summary generation server 7, and the PC 9 perform remote IP connection such as the Internet. It may be installed outside the call center as appropriate.
In particular, when an administrator other than the call center operator operates the dialogue summary query PC 9 to query or update the dialogue speech summary which is the response history in the summary sentence database, the dialogue summary query PC 9 is It is not necessary to be installed near the operator, and it is preferable to be installed outside the call center via remote IP connection as appropriate.

  The voice processing system may further comprise another PC 10 connected or incorporating a microphone, connected to the voice processing system via the intranet 8 or the Internet. According to this configuration, it is possible to input face-to-face dialogue speech collected by the microphone of the PC 10 into the speech processing system according to the present embodiment, and to generate a summary sentence of the face-to-face dialogue speech.

  The PBX 1 accommodates extension telephones in a call center, connects these extension telephones, and connects the telephone terminal 12 of each operator to the PSTN (public telephone network) 13 via the private lines 11a, 11b, 11c,. Circuit-switched connection is performed to realize a call between each operator's telephone terminal 12 and a customer's telephone terminal 14 connected to the PSTN 13.

Although the PBX 1 in FIG. 1 is connected to the customer's telephone terminal 14 via a public switched telephone network such as the PSTN 13, etc., instead of or in addition to this, it is provided with an IP network connection function. May be connected to a customer's IP call terminal equipped with an IP telephone function via a voice packet communication network such as a Voice Over Internet Protocol (VoIP) network, and in this case, the voice acquisition server 2 described later A voice call between the terminal and the operator's telephone terminal 12 can be obtained. The customer's telephone terminal 14 may be either a fixed telephone, a mobile telephone or a smart phone.
<Functional configuration of each server device>

  The voice acquisition server 2 is branch-connected to the PBX 1 and acquires a call voice between the telephone terminal 12 of each operator and the telephone terminal 14 of the customer, and identifies the acquired call voice as an identifier of the operator's telephone terminal 12 (for example, an extension number ) And supply to each server. Alternatively, the voice acquisition server 2 may be branch-connected to the circuit between the termination unit (DSU) of the PSTN 13 and the PBX 1.

The call recording server 3 compresses the call voice supplied from the voice acquisition server 2 after the incoming call as necessary under the control of the control server 4, and acquires the voice data acquired, for example, NAS (Network Appliance Storage) The dialogue speech file (dialogue speech file 31 in FIG. 2) composed of a large scale external storage device is stored and stored in the database.
Preferably, when analog voice is supplied from the voice acquisition server 2, the call recording server 3 samples the analog voice waveform represented by a voltage by sampling at a predetermined bit depth and a predetermined sampling frequency. , And stored in the dialog voice file 31.

  The digital audio data may be stored in the interactive audio file 31 after compression. Various known methods can be used for compression of recorded voice at various compression rates, and as a non-limiting example, recording is performed by monaural 1⁄5 compression, monaural 1/10 compression, stereo non-compression, etc. Audio is compressed. Alternatively, the call recording server 3 may store the voice data supplied from the voice acquisition server 2 in the call voice file 31 without converting and compressing it.

The call recording server 3 also writes call information acquired as call control information in a call information file (not shown) in association with the dialogue voice data for each call unit stored and stored in the dialogue voice file 31. This call information is provided by the PBX 1.
The call information acquired by the call recording server 3 includes, for example, incoming call start information (including incoming call start time stamp), outgoing call start information (including outgoing call start time stamp), call start information (including call start time stamp), Call control information such as call termination information (including call termination time stamp), source phone number, destination phone number, source channel number, sender number, incoming channel number, incoming phone number (destination extension number etc. And call identification information.

The call information further includes speaker identification information identifying the polarity of whether the speech in the recorded call is inbound, ie speech from the customer side or outbound, ie speech from the operator side. This speaker identification information can be acquired by the PBX 1, and can be grasped at session configuration in call generation, for example, in the case of SIP (Session Initiation Protocol), specifically, for example, at session configuration, In the Invite command sent from the calling party to the called party, specify the IP address and port number that the calling party uses for reception in SDP (Session Description Protocol) that describes the information necessary for session initiation, In response to this, the IP address and port number used for reception by the callee are specified in the SDP in the 200 OK message sent from the callee to the callee, and the specified IP address and port number are specified. RTP (Realtime Transport P) (rotocol) Voice data is transmitted and received on the protocol. Therefore, by acquiring the IP address and the port number used by the calling party and the called party for reception, it is possible to obtain the speaker identification information of each utterance in one call, and the customer in one call can The utterance and the operator's utterance can be distinguished or separated as needed.
In the case of ISDN, the speaker identification information can be obtained as a physical pin position of a digital service unit (DSU).

  The call information is preferably displayed in real time on the display device in cooperation with a CTI program running on the control server 4 implementing the Computer Telephony Integration (CTI) protocol or on the PC 9 of the operator. You may

  The call recording server 3 also has a customer information database (not shown) in which information of customers mainly of customers who already have a response history has been registered. This customer information includes personal information identifying the customer, and includes, for example, customer name, address, registered customer telephone number, date of birth, age group, gender, other customer attributes, product purchase history, response history, etc. And the terminal device that can be operated by the operator can be appropriately output according to the instruction input by the operator.

  Note that the call recording server 3 may be connected between, for example, the PSTN 13 and the PBX 1 instead of connecting to the local line 8, and if configured in this way, the call recording server 3 may Identification information can be obtained directly. Further alternatively, without separately installing the voice acquisition server 2, the call recording server 3 may be connected to the private line 8 and directly obtain the call voice supplied to the private line 8.

  The control server 4 performs processing based on data and control information supplied from the voice acquisition server 2, the call recording server 3, the speech recognition server 5, the emotion analysis server 6, and the summary generation server 7, Control transmission and reception of data traffic and control information between servers. Alternatively, the voice recognition server 5 and the summary generation server 7 may access the call voice file 31 and the call information file held by the call recording server 3 and interface to the PC 9 for dialog summary inquiry via the control server 4. May be provided directly. In this case, the voice processing system may not include the control server 4 separately.

The voice recognition server 5 reads the dialogue voice data accumulated and stored in the dialogue voice file 31 under the control of the control server 4 for every one call from off-hook to on-hook, and the dialogue voice for one call is divided into a plurality of utterance units. To separate. The separation into speech units is to identify a silent section and separate the dialog speech into the silent sections, which will be described later with reference to FIG.
In the present embodiment, the speech recognition server 5 analyzes dialogue speech data for each separated speech unit to extract feature quantities, and uses various recognition dictionaries such as a speech recognition dictionary (speech recognition dictionary 32 in FIG. 2). The speech recognition data is converted to a character code string by applying known speech recognition technology, and the converted character code string is output to a file as a dialog sound text. In the present embodiment, the interactive speech text output by the speech recognition server 5 includes text (summarized unit text in FIG. 2) divided into summary units. The process of dividing the dialogue speech text into summary units will be described later with reference to FIGS. 4, 7 and 8.

  The emotion analysis server 6 receives the dialogue voice data supplied from the call recording server 3 as an input, and for each speaker, a quantitative index indicating the speaker's emotion such as anger, satisfaction, stress, reliability, etc. Output as the speaker's emotion analysis result. The emotion analysis result can be output as a change in each emotion indicator in a predetermined period, such as within one call or all day. The details of the emotion analysis process executed by the emotion analysis server 6 will be described later with reference to FIGS. 6 and 22 to 24.

  The summary generation server 7 reads out the dialog speech text divided into summary units and stored in the dialog speech text file 33 for each call, executes a summary generation process, and generates the generated summary text of the dialogue. Output as text (abstract text 38 in FIG. 3). Details of this summary generation process will be described later with reference to FIG.

  The summary generation server 7 may read out interactive speech text of the speech of one of the speakers in one call, for example, the operator, and generate a summary sentence, and the answering part extracted from the speech of the other speaker, for example, the customer (Described later) may be added to the summary, or the summary may be created from the interactive speech texts of both speakers. In the latter case, it is preferable to associate the identification information of the speaker with the interactive voice text.

The abstract generated for each call can be output appropriately to an output device such as a display device such as a PC 9 for dialogue summary inquiry or a printer device in response to a query input, and preferably call information The call start time, the call end time, the caller identification information of the call (information for identifying a call received from a customer or a call sent from an operator) may be output in association with the decoding start time, the call end time, and the like.
Preferably, the summary displayed and output on the PC 9 or the like can be appropriately updated by the operator's correction input. By learning the update result and appropriately updating the important word table, unnecessary word table, various conversion tables, etc. to be referred to when generating the summary sentence, it is possible to generate a more accurate and clear summary sentence. It becomes.
In the present embodiment, the summary generation server 7 further receives the dialogue speech text supplied from the speech recognition server 5 as an input and refers to the emotion word table (the emotion word table 37 of FIG. 3) etc. Emotional expression parts are extracted and converted into emotional expression words to be included in the summary sentence.

  The configuration of the network and hardware shown in FIG. 1 is only a non-limiting example, and each server and database may be integrated as required, or each component may be integrated into an external facility such as ASP (Application Service Provide). You may install it.

<Example of Functional Configuration of Speech Recognition Server 5>
FIG. 2 is a diagram showing one non-limiting example of the functional configuration of the speech recognition server 5 according to the present embodiment.
Among the function modules of the speech recognition server 5 shown in FIG. 2, for the functions realized by software, a program for providing the function of each function module is stored in a memory such as a ROM and read into a RAM Is realized by executing. For the function realized by hardware, for example, a dedicated circuit may be automatically generated on the FPGA from a program for realizing the function of each functional module by using a predetermined compiler. FPGA is an abbreviation for Field Programmable Gate Array. Further, a Gate Array circuit may be formed in the same manner as an FPGA and realized as hardware. Also, it may be realized by an application specific integrated circuit (ASIC). Note that the configuration of the functional blocks shown in FIG. 2 is an example, and a plurality of functional blocks may constitute one functional block, or any functional block may be divided into blocks that perform a plurality of functions. It is also good. The same applies to the functional configuration of the summary generation server 7 shown in FIG. 3 and other server devices.
Referring to FIG. 2, the speech recognition server 5 includes a speech recognition pre-processing unit 51, a speech recognition unit 52, a speech recognition post-processing unit 53, and a harmony analysis unit 54.

  The voice recognition pre-processing unit 51 reads out a file of dialog voice for each call from the dialog voice file 31 accumulated and stored by the call recording server 3, and detects a silent section from the read dialog voice file of one call, The detected silence interval is used as a boundary to divide into utterance units in the dialogue. The speech recognition pre-processing unit 51 also supplies a plurality of speech units divided from the dialogue speech file of one call to the speech recognition unit 52 for each speech unit, and the speech recognition unit 52 performs speech recognition processing in speech units. Run

  The speech recognition unit 52 receives the dialogue speech for each utterance unit supplied from the speech recognition pre-processing unit 51, executes speech recognition processing, and supplies the dialogue speech text for each utterance unit to the speech recognition post-processing unit 53. . The speech recognition unit 52 can convert speech data of dialogue speech into dialogue speech text, for example, with reference to the speech recognition dictionary 32 that can define important terms and sentences that should be recognized correctly. The speech recognition unit 52 may be implemented in a known speech recognition engine, while the speech recognition pre-processing unit 51, the speech recognition post-processing unit 53, and the harmony analysis unit 54 may be implemented in the control server 4, for example.

  The speech recognition post-processing unit 53 executes syntactic analysis, morphological analysis, and the like on the dialogue speech text for each utterance unit output by the speech recognition unit 52 to divide the dialogue speech text into a summary unit and separate it into a summary unit. Output as the specified interactive speech text 33. The syntactic analysis result and the morphological analysis result may be associated with the call speech text divided into summary units. The summary unit is a unit of division further subdivided as a processing unit of summary generation processing so that summary generation can be easily and accurately performed from speech voice text of the utterance unit, and the details thereof are shown in FIG. Will be described later.

  The voice recognition post-processing unit 53 may also assign weights to the extracted summary units with reference to the voice recognition dictionary 32 that defines weights for each key word. For example, date, time, address, telephone number, etc. are often important words to be left in the summary, and erroneous conversion can be reduced by weighting these words by the speech recognition post-processing unit 53.

The speech analysis unit 54 detects and detects text presumed to be an answer such as “Yes” or “No” from the interactive speech text divided into the summary units supplied by the speech recognition post-processing unit 53. Determine if the text is a backstroke or a response. Based on the determination result, the competition analysis unit 54 deletes the text determined to be a competition from the interactive speech text 33 divided into summary units output by the speech recognition post-processing unit 53.
On the other hand, the competition analysis section 54 also includes the text judged to be an answer in the dialog speech text 33 so as to be included in the summary sentence generated by the summary generation server 7 and “answers” the text in the dialogue speech text. Tag that. The details of this competition analysis process will be described later with reference to FIGS. 13 and 14.

<Example of Functional Configuration of Summary Generation Server 7>
FIG. 3 is a diagram showing one non-limiting example of a functional configuration of the summary generation server 7 according to the present embodiment.
Referring to FIG. 3, summary generation server 7 includes text correction unit 71, redundancy elimination unit 72, summary generation unit 73, emotion analysis unit 74, and summary shortening unit 75.

  The text correction unit 71 reads out the dialog speech text 33 divided into summary units, and corrects the dialog speech text in order to facilitate the generation of a summary sentence based on the syntactic analysis result and the morphological analysis result. The interactive speech text is output to the redundancy elimination unit 72.

  The redundancy exclusion unit 72 eliminates redundancy of the corrected dialogue speech text supplied from the text correction unit 71. More specifically, the redundancy exclusion unit 72 refers to the unnecessary word table 35, for example, to delete unnecessary words and overlapping sentences from the dialogue speech text, and the dialogue speech to be supplied to the summary sentence generation unit 73. Shorten the text The redundancy removal unit 72 outputs the abbreviated dialogue speech text from which the redundancy is removed to the summary generation unit 73.

  The summary sentence generation unit 73 reads out the abbreviated dialogue speech text supplied from the redundancy exclusion unit 72, and generates a summary sentence text with reference to the important word table 34, the unnecessary word table 35, and the various conversion tables 36. . The summary sentence generation unit 73 may generate one summary sentence text for each call. The summary sentence output by the summary sentence generation unit 73 may be, for example, a brief style of a report style obtained by converting the spoken language of the call speech text, for example, an unspeakable style.

  In the present embodiment, the abstract sentence generation unit 73 acquires from the emotion analysis server 6 a quantitative index indicating the speaker's emotion during the dialog as a speaker's emotion analysis result, and acquires the speaker's emotion analysis The results can be included in the summary text to be generated, or can be displayed on the display simultaneously or in conjunction with the summary text. The emotion analysis result of the speaker supplied from the emotion analysis server 6 includes, for each speaker, quantitative indexes such as, for example, anger, satisfaction, stress, reliability, and the like.

The emotion analysis unit 74 refers to the emotion word table 37 from the summary sentence generated by the summary sentence generation unit 73, extracts the emotion expression part in the summary sentence text, and sets it as a simple emotion expression word to be included in the summary sentence. Convert and replace the emotion expression part extracted in the summary sentence text with the converted emotion expression word.
The summary sentence shortening unit 75 shortens the summary sentence so that the summary sentence length within the threshold is obtained when the summary sentence supplied from the summary sentence generation unit 73 exceeds a threshold of a predetermined length, for example, a predetermined number of characters. , The condensed summary sentence is output as a summary sentence text 38.

<Processing Procedure of Speech Recognition Processing in Speech Recognition Server 5>
FIG. 4 is a flowchart showing a non-limiting example of the processing procedure of the speech recognition process performed by each unit of the speech recognition server 5.
In step S1, the speech recognition pre-processing unit 51 of the speech recognition server 5 reads out the dialogue speech data converted into a file for each call from the dialogue speech file 31.
At S2, the speech recognition pre-processing unit 51 of the speech recognition server 5 identifies the speaker in the dialogue voice read at S1. Specifically, the speech recognition pre-processing unit 51 can identify the speaker, for example, the customer and the operator from the dialogue voice by referring to the speaker identification information of the call information associated with the dialogue voice file. it can.

Specifically, the speech recognition pre-processing unit 51 refers to the call information database (not shown) and determines the speaker identification information in one call so that each utterer of the utterance in one call is It can identify whether it is a customer or an operator.
In the speech recognition unit 52 in the latter stage, the dialogue speech data is speech-recognized for each identified speaker, and the summary generation unit 73 of the summary generation server 7 generates a summary from the speech speech-recognized dialogue speech text, The texts of the recognition results of both speakers can be associated with reference to the time stamp of the dialogue recording.

The speech recognition pre-processing unit 51 uses the dialogue speech data of one speaker, for example, the speech identified as the speech of the operator from the speech speech data of the utterance identified as the speech of the other speaker, for example, the customer. It may be supplied to the summary generation server 7 in priority. This is based on the finding that the utterance of one of the speakers, typically the operator, can more efficiently obtain information sufficient to summarize the response history as a summary text generation source.
Alternatively, the speech recognition pre-processing unit 51 may perform speech recognition on only one of the speakers, for example, speech of the speech identified as the speech of the operator and convert it into speech text. By limiting the target of speech recognition, hardware resources in the speech recognition server 5 performing high-load speech recognition can be reduced, and real-time performance of speech recognition processing and summary sentence generation processing is improved, and an interactive speech text file Resource capacity can also be reduced.

In S3, the speech recognition pre-processing unit 51 of the speech recognition server 5 separates the dialogue speech data in which the speaker read out for each call is separated into speech units, and divides the speech into speech units. Are supplied to the speech recognition unit 52.
Specifically, the speech recognition pre-processing unit 51 detects a certain silent section in the dialogue speech data, and cuts out a sound section by dividing the speech in the detected silent section, and sets it as the dialogue voice of the utterance unit. To separate.

As shown in FIG. 5, the interactive voice file for one call is composed of two channels CH1 and CH2. The sound of CH1 is, for example, the speech of a customer, and the sound of CH2 is, for example, the speech of an operator.
The speech recognition pre-processing unit 51 detects a silent section of a fixed length. The silence interval to be detected may be, for example, a silence interval of 1.5 seconds or more, and the lower limit may be adjusted, for example, between 1 second and 2 seconds. The lower limit value of this silent section is referred to as a first threshold. The lower limit value of this silent section can be set, for example, in consideration of the time required for breathing. In addition, it is preferable that the lower limit value of the silent section is set so as not to erroneously detect, as a silent section, a sound repellent "" in a speech of "I told you", for example.

Referring to FIG. 5, speech recognition pre-processing unit 51 detects and detects a silent section (SL11, SL12,..., SL16) having a length equal to or greater than a first threshold from the voice of the customer of CH1. A sound section (SP11, SP12,..., S17) existing between two silent sections is extracted. Each of the extracted voice segments (SP11, SP12,..., S17) is one utterance unit in the voice identified as the customer, and in the present embodiment, the voice recognition supplied to the voice recognition unit 52 It becomes a unit. Each of the sound segments can be regarded as a segment uttered without breath.
Similarly, referring to FIG. 5, speech recognition pre-processing unit 51 detects a silent interval (SL21, SL22,..., SL26) having the first threshold as the lower limit value from the voice of the operator of CH2. Extract a sound section (SP21, SP22,..., S27) between two detected silent sections. Each of the extracted voice segments (SP21, SP22,..., S27) is one utterance unit in the voice identified as the operator.

Returning to FIG. 4, in S4, the speech recognition unit 52 of the speech recognition server 5 performs speech recognition processing for each identified speaker on the dialogue speech data input from the speech recognition pre-processing unit 51 in units of speech. Execute to output interactive speech text, which is text-to-speech interactive speech.
In the present embodiment, the speech recognition process is performed on the basis of the dialogue speech data in this manner. The silent section above recognizes that the speaker has switched during the silent section or that the same speaker has changed the topic or content. Therefore, it can be estimated that the continuity in the uttered content is poor before and after the silent section, and improvement in the recognition accuracy can be expected by speech recognition of the dialog speech text on an utterance basis.

A known speech recognition engine can be applied to this speech recognition process.
As an example of conversion to a character code string in the speech recognition process performed by the speech recognition unit 52, for example, a feature amount extracted from speech waveforms which have been variously converted as necessary in interactive speech data is defined in advance The speech waveform data can be converted into a character code string by performing comparison processing with the reference sound pattern for each phoneme being processed.

  The speech recognition dictionary 32 referred to by the speech recognition unit 52 and the speech recognition post-processing unit 53 is an important word (or important sentence) including important information assumed to be a target of speech recognition in advance and to be included in a summary sentence. Since the data is defined, only the phoneme string of the interactive speech data corresponding to the key word defined in the speech recognition dictionary 32 may be extracted and made meaningful. In addition, important words (or important sentences) defined in the speech recognition dictionary 32 may be weighted. Of the dialogue speech data read out by the speech recognition unit 52, the dialogue speech data portion corresponding to the defined important word may be converted into the dialogue speech text and outputted as the speech recognition result.

  FIG. 9 shows a non-limiting example of interactive speech text which is a speech recognition result of a speech unit generated from the dialogue speech data, which is output from the speech recognition unit 52 in S4. In the example shown in Fig. 9, "Interaction summary processing consists of conversion from spoken language to written language in addition to unnecessary speech and expression deletion. It can be selected according to the characteristics of the data to be processed." It becomes an utterance unit extracted between two silent sections. As shown in FIG. 9, the speech recognition result of the speech unit output in S4 may include a plurality of sentences not separated by punctuation marks as one unit.

Returning to FIG. 4, in S5, the speech recognition post-processing unit 53 of the speech recognition server 5 converts the speech recognition result output by the speech recognition unit 52 in S4 into natural speech and divides it into a summary unit. The speech recognition post-processing unit 53 can also assign a type and a weight to the dialogue speech text of the summary unit divided in S5 based on the result of syntactic analysis and morphological analysis.
The details of the conversion process in S5 will be described later with reference to FIGS. 7 and 8.

In S6, the speech analysis server 54 of the speech recognition server 5 detects the text presumed to be an answer such as "Yes" or "No" from the interactive speech text divided into summary units, and the detected text is detected. It is determined whether it is a fight or a response.
Based on the determination result, the competition analysis unit 54 deletes the text determined to be a competition from the interactive speech text 33 divided into summary units output by the speech recognition post-processing unit 53. On the other hand, the competition analysis unit 54 includes the text determined to be the answer in the call voice text 33 so as to be included in the summary sentence generated by the summary generation server 7, and also "answers" the text element in the dialogue speech text. Tagging (classification) that it is. In addition, the details of the processing of analyzing the competition in S6 will be described later with reference to FIGS. 13 and 14.
In S7, the sumo analysis unit 54 outputs the dialog speech text divided into summary units to which the text determined as the answer is added.

<Detailed Processing Procedure of Speech Recognition Post Processing in Speech Recognition Post Processing Unit 53>
FIG. 7 is a flowchart showing an example of a detailed processing procedure of the speech recognition post-processing performed by the speech recognition post-processing unit 53 in S5 of FIG.
Referring to FIG. 7, in S51, the speech recognition post-processing unit 53 of the speech recognition server 5 refers to the speech recognition dictionary 32, and the dialogue of the utterance unit which is the speech recognition result output by the speech recognition unit 52 in S4. Perform parsing of speech text.
In step S52, the speech recognition post-processing unit 53 refers to the speech recognition dictionary 32 and executes morphological analysis of dialogue speech text in utterance units. Note that either the syntax analysis of S51 or the morphological analysis of S52 may be performed first, or may be performed concurrently.

FIG. 10 shows one non-limiting example of the parsing result obtained by performing the parsing process in S51 on the dialog voice text of the utterance unit shown in FIG. As shown in FIG. 10, in the parsing result output in S51, the relationship between morphemes in the text is structured.
FIG. 11 shows one non-limiting example of the morphological analysis result obtained by performing the morphological analysis processing in S52 on the dialog speech text of the utterance unit shown in FIG. As shown in FIG. 11, the morpheme analysis result may include, for each extracted morpheme, writing, reading, and acquired part-of-speech classification (major classification, middle classification, minor classification).

Referring back to FIG. 7, in step S53, the speech recognition post-processing unit 53 divides the dialogue unit speech text of the utterance unit into a summary unit based on the analysis results of the syntactic analysis and the morphological analysis of S51 and S52.
FIG. 8 is a flowchart showing an example of a detailed processing procedure of separation processing into summary units which is executed by the speech recognition post-processing unit 53 in S53 of FIG.
In step S531, the speech recognition post-processing unit 53 determines whether the part-of-speech classification obtained as a result of morphological analysis and syntax analysis is a noun. If the part-of-speech type of the delimiter unit obtained as a result of analysis is a noun, the process proceeds to S (S 531: Y) and S 532. On the other hand, when the part-of-speech type of the delimiter unit obtained as a result of analysis is other than the noun, the processing after S532 is skipped to end the processing, and the processing proceeds to S6.

  In step S532, the speech recognition post-processing unit 53 determines whether the beginning of the group (group) of delimiter units obtained as a result of morphological analysis and syntax analysis is other than a noun. If the beginning of the group of delimiter units is other than a noun (S532: Y), the processing after S533 is skipped to end the processing, and the processing proceeds to S6. On the other hand, if the beginning of the group of delimiter units is a noun (S5323: N), the process proceeds to S533.

  In S533, the speech recognition post-processing unit 53 determines whether the delimiter unit obtained as a result of morphological analysis and syntactic analysis is noun + α. If the delimiter unit is noun + α, that is, if the end includes particles other than nouns (S533: Y), the speech recognition post-processing unit 53 combines the delimiter unit with the immediately preceding delimiter unit in S534. The process ends, and the process proceeds to S6. On the other hand, if the delimiter unit is not noun + α, that is, only nouns (S533: N), the speech recognition post-processing unit 53 returns the processing to S532 after combining the delimiter unit with the preceding delimiter unit in S535. , S532 and S533 repeat the determination of the delimiting unit.

FIG. 12 receives as an input the dialogue speech text of the utterance unit shown in FIG. 9 and passes through the syntactic analysis result shown in FIG. 10 and the morphological analysis result shown in FIG. Is an example of interactive speech text.
Square symbols in FIG. 12 indicate the division of the summary unit. As shown in FIG. 12, by performing conversion to S5 into natural speech and separation processing into summary units, continuous "dialogue", "summary", "process" into one summary unit, or Consecutive "processes", "objects" and "data" are respectively combined into one other summary unit to constitute a summary unit.
The speech recognition post-processing unit 43 of the speech recognition server 5 may further add a type and a weight to each of the separated speech units in the summary unit by referring to the speech recognition dictionary 32. In FIG. 12, the summary unit "dialogue summary processing" and the summary unit "of data to be processed" are respectively weighted as important summary units to be included in the summary.

<Detailed Processing Procedure of Sumo Analysis Processing in Sumo Analysis Unit 54>
FIG. 13 is a flowchart showing a non-limiting example of the detailed processing procedure of the competition analysis process performed by the competition analysis unit 54 of the speech recognition server 5 in S6 of FIG.
Referring to FIG. 13, in S61, the speech analysis unit 54 of the speech recognition server 5 acquires dialogue speech of both speakers, for example, a customer and an operator from the dialogue speech file 31. Since the dialogue voice file 31 is provided with a time stamp capable of associating the two speakers for each call, the harmony analysis unit 44 sets the dialogue voices of both speakers constituting one talk unit. It can be acquired. Alternatively, the dialog voices of both speakers may be associated with each other by giving a common identifier to each of the dialog voices of the speakers constituting the call unit for each call unit. At S61, together with the acquired dialogue voice of both speakers, dialogue voice text obtained by speech recognition of the dialogue voice is input.

In S62, the competition analysis unit 54 compares the dialogue voices of both the customer and the operator, and determines whether a short utterance can be detected while the conversation partner is speaking.
Referring to FIG. 14A, since the short utterance (SP14) in the dialogue voice of the customer of CH1 is a short utterance made during the utterance (SP24) of the operator of CH2 which is the other party of the dialogue, S62. Is detected. The short utterance to be detected in S62 may be, for example, less than 2 seconds.
When a short utterance is not detected while the other party is speaking (S62: N), the processing of S63 to S68 is skipped and the processing is ended, and the processing proceeds to S7. On the other hand, if a short utterance is detected while the other party is speaking (S62: Y), the process proceeds to S63.

  In step S63, the harmony analysis unit 54 searches the dialog speech text of the speech recognition result having the same time stamp as the short speech detected in step S62, and whether or not the speech recognition result of the short speech can be estimated as an answer. That is, it is determined whether it becomes a candidate of an answer. For example, if the short utterance is a text such as “Yes”, “Yes”, “No”, “No”, etc., it can be judged as a candidate for an answer. For example, the answer candidate may be set in advance in the face analysis unit 54.

  If the speech recognition result of the short utterance is not the answer candidate (S63: N), the process proceeds to S64, determines that the short utterance is a compliment, and deletes it from the interactive speech text to be input for summary generation. That is, the short utterance determined to be a sumo wrestling in S64 is not a summary sentence generation source because it is a meaningless compliment in the summarization. On the other hand, if the speech recognition result of the short utterance is an answer candidate (S63: Y), the process proceeds to S65.

In S65, the competition analysis unit 54 further determines whether or not there is a short silent period in the voice of the other party in the dialogue during the utterance of the short utterance which is the answer candidate detected in S63.
Referring to FIG. 14A, in the utterance of the operator of CH2 corresponding to the short utterance (SP14) in the voice of the customer of CH1, the voice recognition pre-processing unit 41 in S3 of FIG. Since it does not include a silent section having a length of, it is detected as one utterance unit SP24. In S65, it is determined whether or not a short silent interval can be detected in the voice of the conversation partner using a second threshold smaller than the first threshold. The second threshold has a value smaller than the first threshold, and may be, for example, 1 second and adjusted between 0.5 and 1.5 seconds.

When a short silent interval having a length equal to or greater than the second threshold is detected in the utterance unit (speech zone) of the other party of the dialogue during the utterance of the short utterance which is the answer candidate in S65 (S65: Y It is determined at S66 that the short utterance is an answer, and the process proceeds to S67. On the other hand, when a short silent segment having a length equal to or greater than the second threshold is not detected in the utterance unit (speech segment) of the other party of the dialogue during the utterance of the short utterance which is the answer candidate (S65: N) Proceeding to S64, the short utterance that was the answer candidate is determined to be a compliment, and is deleted from the dialog speech text to be input for summary generation.
In S67, the harmony analysis unit 54 separates the voice of the other party of the dialogue into two utterance units before and after the short utterance determined as the answer in S66.

  Referring to FIG. 14B, assuming that the speech recognition result of the short utterance section (SP14) of the customer of CH1 is determined as the answer candidate, the utterance section of the operator of CH2 during this utterance (SP14) In SP24), a silent interval (SL24a) which is equal to or greater than the second threshold and smaller than the first threshold can be detected. In this case, the speech analysis unit 54 separates the speech section (SP24) of the operator before and after the detected silent section (SL24a), and the speech section (SP24a) immediately before the silent section (SL24a) and the silent section (SP24). SL24a) The speech section (SP24b) immediately after is acquired.

  In S68, the speech analysis section 54 determines that the dialogue speech text obtained by speech recognition of the speech section (SP24a) immediately before the short silent section (SL24a) separated in S67 is the speech text determined as the answer in S66. And the dialogue text to be paired with each other, and the text of the pair is associated with each other as the text of the answer and the text for specifying the answer to which the answer was prompted. And a summary unit interactive voice text file 33.

<Processing Procedure of Summary Generation Processing in Summary Generation Server 7>
FIG. 6 is a flowchart showing a non-limiting example of the processing procedure of the abstract generation process performed by each section of the abstract generation server 7.
Referring to FIG. 6, in S10, the text correction unit 71 of the summary generation server 7 reads out one dialogue unit speech text from the dialogue speech text 33 divided into summary units.

In S11, the text correction unit 71 corrects the dialogue voice text read out in S10. Specifically, as shown in FIG. 12, the text correction unit 71 inserts a punctuation mark into the interactive speech text of one utterance unit separated by a summary unit (processing unit of summary generation), and then the position of the punctuation mark Insert a line break with
FIG. 15 shows one non-limiting example of the punctuation mark table to which the text correction unit 71 refers. The punctuation table of FIG. 15 defines terms for which a punctuation mark or a reading point should be inserted immediately after. In FIG. 15, “1” indicates insertion of a reading point, and “0” indicates insertion of a punctuation mark. The text correction unit 71 refers to the punctuation table shown in FIG. 15 and is defined in the punctuation table in backward matching from the delimiter of the summary unit, “Mr.”, “Mr.”, “Mas”, “Yes” etc. The word is searched, and immediately after the searched word, punctuation marks or reading points are inserted according to the definition of the punctuation mark table. The text correction unit 71 may sequentially search the search terms defined in the punctuation mark table of FIG.

Furthermore, the text correction unit 71 searches for the numeral extracted by the morphological analysis, and analyzes the meaning of the searched numerical value. In the generation of a summary sentence for response history, the numeral is often an important word which is a key word in the summary. Therefore, the text correction unit 71 analyzes the meaning of the retrieved numeral, acquires the type according to the meaning of the analyzed numeral, and assigns a unit or weight according to the acquired type.
For example, “date”, “time”, “amount of money”, “telephone number”, “number of pieces” and the like may be given as the meaning of the numeral, but the present invention is not limited thereto.

FIG. 16 is a number classification type table that the text correction unit 71 refers to in order to assign types, units (notations), and weights to the elements of the analyzed number division. Referring to FIG. 16, for example, weights higher than the number (pieces) and the temperature (degrees) are given to the date, time, and money (yen).
On the other hand, when the dialogue speech text is searched for the number conjunction that is not related to the preceding and following words, the text correction unit 71 may determine that the speech is misrecognized and delete the number conjunction from the dialogue speech text. In addition, the text correction unit 71 may convert the retrieved numeral into half-width numerals in order to improve visibility and clarity in the summary text.

Returning to FIG. 6, in S12, the redundancy exclusion unit 72 of the summary generation server 7 outputs the simplified or simplified dialogue speech text by eliminating the redundancy in the speech speech dialogue speech text.
Specifically, the redundancy exclusion unit 72 refers to the unnecessary word table 35 and deletes the unnecessary words from the interactive speech text.
FIG. 17 shows one non-limiting example of the unnecessary word table 35 to which the redundancy exclusion unit 72 refers. Referring to FIG. 17, in the unnecessary word table 35, fixed phrases such as “Eh” interjections, “always being taken care of” and the like are defined as unnecessary words.

  The redundancy exclusion unit 72 further generates a sentence describing the same or similar contents (or a group having meaning such as a phrase or a word) multiple times from the dialogue speech text for one call. Duplicate sentences may be removed from the dialog speech text as appropriate. Preferably, when a sentence or the like describing the same or similar content is filed multiple times in the interactive voice text for one call, the redundancy exclusion unit 72 appears forward in time series from the call start to the end. You may delete the last sentence and leave the last one. This is because sentences near the end of the call are more likely to describe the final conclusion in the response. In addition, it is possible to estimate that the sentence that appears last is the repetition by the operator, and in this case, it can be expected that the read-out sentence is more likely to describe the exact content to be left in the summary as the response history. .

The redundancy exclusion unit 72 may further refer to the keyword table 34 and may delete wordings and repetitions of keywords registered in the keyword table 34.
For example, it is assumed that "eVoice" and "E-voice" are registered as keywords in the important word table 34.
In this case, if the recognition result is "I'm going to a good eVoice at 10 o'clock tomorrow", the redundancy exclusion unit 72 searches for a partial match of the reading from the beginning just before the registered keyword and searches Delete the word This makes it possible to delete the saying part from the dialogue speech text.
Similarly, assuming that the recognition result is "I ask eVoice to 10 o'clock tomorrow," the redundancy exclusion unit 72 deletes the repetition of the registered keyword forward as described above. In this way, repeated parts can be deleted from the interactive speech text.

  Returning to FIG. 6, in S13, the summary generation unit 73 of the summary generation server 7 generates a summary of the response history from the interactive speech text output by the redundancy removal unit 72. Specifically, the abstract sentence generation unit 73 shapes the interactive speech text described in the conversation body into a sentence. Preferably, the abstract sentence generation unit 73 shapes the dialogue speech text described in the conversation body into an unspeakable sentence style.

FIG. 18 shows one non-limiting example of the style conversion table 36 to which the abstract sentence generation unit 73 refers. Referring to FIG. 18, in the style conversion table 36, in the left column, the words (“Are you”, “I say“ I say ”,“ I told you ”, etc.) of the conversion source are shown. The words ("I", "I", "I was saying", etc.) of the text form of the conversion destination are respectively defined. The summary sentence generation unit 72 searches the dialogue speech text for the words of the conversion source conversation body defined in the style conversion table 36, and the retrieved conversation body words are defined in the style conversion table 36 as corresponding sentences. Convert to body words. As a result, polite words in the interactive speech text are converted into a concise report-like text.
It should be noted that in the sentence conversion table 36 of FIG. 19, the word of the sentence of the conversion destination corresponding to the word of “conversion” is not defined. In this case, the abstract sentence generation unit 72 may delete the conversion source word from the interactive speech text.

Returning to FIG. 6, in S13, the summary sentence generation unit 73 further searches the dialogue speech text for a previously defined key word, and includes the searched key word in the summary sentence to be output.
FIG. 19, FIG. 20 and FIG. 21 respectively show one non-limiting example of the important word table 34 to which the abstract sentence generation unit 73 refers. Referring to FIG. 17, in the key word table 34, the words "contact" and "confirmation" are defined as key words. In the key word table 34, key words may be defined with variable weights (points). In FIG. 19, the weight “1” is defined for the words “contact” and “confirmation”. In addition, the user may provide another important word table 34 that can be edited such as addition or deletion, so that proper nouns and the like can be defined as appropriate.
The summary sentence generation unit 73 searches the interactive speech text for the key words defined in the key word table 34, weights the searched key words according to the corresponding weights, and includes them in the summary sentence to be generated.

FIG. 20 shows a non-limiting example of the key word table 34 defining key words ("Yes", "OK", "Good", "OK", etc.) which are positive expressions, and FIG. One non-limiting example of the key word table 34 defining key words ("No", "Yadayo", "Any", "Do not approve", etc.) that are expressions is shown. The summary sentence generation unit 73 also refers to these key word tables 34 to search for key words from the interactive speech text, and weights the searched key words according to the corresponding weight to generate a summary sentence to be generated. include. The key words as positive or negative expressions included in FIG. 20 and FIG. 21 may be converted into sentences (“accept”, “reject”, etc.) as appropriate.
Note that, preferably, the summary generation unit 73 generates one sentence per one call unit, regardless of whether a plurality of sentences are supplied from the redundancy exclusion unit 72 or a single sentence is supplied. Summary sentences may be generated.

Returning to FIG. 6, in S14, when the abstract sentence generated by the abstract sentence generation unit 73 exceeds a threshold of a predetermined length, for example, a predetermined number of characters, the abstract sentence shortening unit 75 of the abstract generation server 7 Shorten the summary sentence so that it becomes the summary sentence length of.
Preferably, in the query result display screen on which the dialogue summary is displayed in a list, the summary shortening section 75 does not need to scroll the entire summary in an output field provided for displaying a summary of one call unit. The number of readable characters may be set as the threshold. This eliminates the need for an additional operation for summary confirmation, and enables quick visual recognition of the entire summary.

More specifically, the summary sentence shortening unit 75 may shorten the summary sentence based on the weight (importance degree point) given to the key words appearing in the summary sentence with reference to the various key word table 34. .
As an example, the summary sentence shortening unit 75 separates the dialogue speech text supplied from the redundancy exclusion unit 72 for each phrase point (“.”), And for each dialogue speech text sentence, an important word that appears in the sentence. Importance degree points may be added to preferentially select a call text sentence for which a high importance degree is calculated.
The summary sentence shortening unit 75 outputs the abbreviated summary sentence to the file of the summary sentence text 38.

In S15 of FIG. 6, in the present embodiment, the abstract sentence generation unit 73 outputs an abstract sentence to be output of the text of the pair to which the type of “Answer” is generated, generated by the speech analysis unit 54 of the speech recognition server 5 Add to
In the speech analysis process of FIG. 13 executed by the speech analysis unit 54 of the speech recognition server 5, the dialogue speech text uttered by one of the speakers (for example, the customer) determined as the answer and the speech immediately before the answer. A dialogue voice text uttered by the other speaker (e.g., an operator) that prompts the user to answer the question is specified as a pair, and a type of "answer" is given. It is included in the dialogue speech text as an answer form dialogue.

  The summary sentence generation unit 73 treats the pair of dialogue speech texts to which the type of “Answer” is given as an important word, refers to various conversion tables 36, converts it into a sentence style for summary sentences, and outputs it. Add to the abstract. For example, it is assumed that the dialogue speech text to which the type of "Answer" is assigned is a pair of "Should you like shipping within a few days (Operator's Question)" and "Yes (Customer's Answer)". In this case, the abstract sentence generation unit 73 converts the pair of dialogue speech texts into “accept delivery after 23 days” and the like, and outputs the converted text as an important word (important sentence) in the response history. Include in the statement.

  As another example, the dialogue voice text given the type of “Answer” is “Is the order item acceptable with the dialogue summary eV-Outline (operator question)” and “Yes, please (customer's answer It is assumed that it is a pair of "). In this case, the abstract sentence generation unit 73 converts the pair of dialogue speech texts into “confirm the order item is dialogue summary eV-Outline” or the like, and sets the converted text as the key word (important sentence) in the response history. Include in the summary to be output.

  In S16, the emotion analysis unit 74 of the summary generation server 7 executes an emotion analysis process of the dialog speaker based on the dialog speech text. Also, the emotion analysis unit 74 provides an interface from the summary generation unit 73 to the emotion analysis server 6, causes the emotion analysis server 6 to execute the emotion analysis process, and supplies the execution result of the emotion analysis process to the abstract sentence generation unit 73. You may Alternatively, without providing the emotion analysis server 6 separately, the emotion analysis unit 74 may execute the emotion analysis process of the speaker of the dialog to generate the summary sentence. Below, the example which performs an emotion analysis process using the former emotion analysis server 6 is demonstrated.

The emotion analysis process includes a non-verbal emotion analysis process using dialogue speech data and a linguistic emotion analysis process using dialogue speech text which is a speech recognition result.
In the emotion analysis process based on the former dialogue speech data, the emotion analysis server 6 called from the emotion analysis unit 74 takes the dialogue speech data supplied from the call recording server 3 as an input, for example, annoyance for each speaker, A quantitative index that quantifies the speaker's emotion such as the degree of satisfaction, stress, and reliability is output as the speaker's emotion analysis result.

The emotion analysis processing provided by the emotion analysis server 6 is such that the movement of the speaker's EEG and the movement of the vocal cords are linked, and in the process of speech, the human being can not control the emotion and the emotion appears in the voice. Based on the findings of the Therefore, the emotion analysis server 6 can digitize the speaker's emotion from the interactive speech data without depending on the language of the speaker's utterance.
In emotion analysis processing based on the latter dialog voice text, the emotion analysis unit 74 of the summary generation server 7 receives the dialog voice text supplied from the voice recognition server 5 as an input, and extracts emotion words in the dialog voice text. The word table 37 is referred to and converted into an emotional expression to be included in the summary sentence.

  FIG. 22 shows a non-limiting example output of the result of the emotion analysis processing performed by the emotion analysis server 6 on the interactive speech data of one speaker (customer) of one call unit. Referring to FIG. 22, the transition of the customer's emotion during one call of the customer (CS) is output in time series. FIG. 22 shows an example in which the operator convinces the customer during a call in response to a complaint from the customer. In FIG. 22, the emotion indexes of “Kiring” and “Satisfaction” increase in values from the middle to the second half, while the emotion indicators of “Stress level” decrease in values from the middle to the second half. In the middle to the second half of one talk unit, it is possible to read the transition of emotions that the customer's anger and stress are reduced and the dissatisfaction is converted to satisfaction.

Further, from the emotion analysis result of the customer illustrated in FIG. 22, it is possible to obtain an index for evaluating the quality of the response of the operator who is the other speaker.
For example, although the emotion index of "Kio" is negative and "Anger" is high from the beginning of the call, at the end of the call, the emotion index of "Kio" is converted to 0 or positive and tends to "Joy" In the case where the feeling level of "satisfaction" is also converted to 0 or positive and indicates a tendency of "satisfaction", the evaluation of the operator's response history may be "good for response" indicating an excellent response.
However, if, for example, the customer's “confidence level” sentiment indicator is negative and indicates a tendency of “distrust” at the end of the call, it can be evaluated that the credibility of the uttered content of the customer is low. You may note "Customer Caution", which indicates that the customer's remarks require caution.

On the other hand, when the emotion index of "Kiring" is largely converted to minus while the emotion index of "satisfaction" is also significantly converted to minus during the call, and the tendency of "anger" and "dissatisfaction" continues thereafter Since it can be evaluated that the statement of the operator immediately before the minus conversion induced the customer's anger and dissatisfaction, it may be considered as "response attention" indicating that it is necessary to confirm the operator's response.
Also in this case, for example, if the customer's "reliability" emotion indicator is negative and indicates a tendency of "distrust" at the end of the call, it can be evaluated that the credibility of the uttered content of the customer is low. , You may note "customer notice" to indicate that the customer's remarks require caution.
In addition, when the above tendency is not indicated, it may be set as "a response normal" indicating that the response is a reasonable response.

FIG. 23 shows a non-limiting output example of the result of the emotion analysis server 6 executing emotion analysis processing on the interactive speech data of the other speaker (operator) of one speech unit. FIG. 23 shows an example in which the operator feels stress in a call with a customer. In FIG. 23, the emotion index of “stress level” has a numerical value rising from the beginning to the end of the call, and it is possible to read the transition of emotion that the stress on the operator is increasing.
In this case, for example, when the stress in the current call is increased compared to the transition of the numerical value of the stress level to the previous time, the evaluation index of the operator is continuously monitored for the stress state of the operator. It may be a "careful attention" indicating that it should be done.

FIG. 24 shows the transition of emotions between multiple calls (15 times in FIG. 24) within a certain period (1 day, 1 week, etc.). In FIG. 24, as the number of calls increases, the average value of the operator's “stress level” emotional index gradually rises, and as the number of calls increases, transition of emotion that the operator's stress increases. Can read.
In this case, the evaluation index of the operator may be set as “cancellation of response” which indicates that the response of the operator is suspended and that the interview should be performed immediately.

  FIG. 25 shows a non-limiting example of the emotional word table 37 to which the emotion analysis unit 74 of the summary generation server 7 refers. Referring to FIG. 25, in the emotional word table 37, in the left column, the original emotional word (“Well?”, “Well, good. Thank you”, “I was disappointed”, “I'm okay”, "Somewhat I do", "I'm stupid", etc.) are the emotional expressions of the conversion destination in the right column ("Agreelessly", "Just", "Disappointing", "Disappointment", "Anxiety", "Discomfort", etc.) respectively It is defined. The emotion analysis unit 74 of the summary generation server 7 searches the dialogue speech text for the conversion-source emotion word defined in the emotion word table 37, and the searched emotion word is defined in the emotion word table 37. Convert to representation. As a result, the emotional words in the dialogue speech text are converted into brief emotional expressions.

  FIG. 27 shows a non-limiting example of generating a summary sentence incorporating emotion expression from dialogue speech text which is a speech recognition result by referring to the emotion word table 37 of FIG. Referring to FIG. 26, the emotion analysis unit 74 has the dialogue voice text “I replaced the device but I have broken it again and again” in the upper row of FIG. Convert to a summary sentence of The summary sentence to be output can include an emotional expression grasped from the dialogue speech text which is a speech recognition result. The converted "disappointment" word indicates the speaker's (customer's) emotional expression and is included in the output summary.

On the other hand, FIG. 26 shows a non-limiting example in which the emotion expression obtained by the emotion analysis server 7 executing the emotion analysis process from the dialogue voice data (voice color) is added to the summary text in parentheses. Referring to FIG. 26, the emotion analysis server 7 executes an emotion analysis process on the dialog speech data which is the basis of the dialogue speech text “food contains insects” in the upper stage of FIG. For example, when the emotion index of "reliability" of the voice data is negative and shows a tendency of "distrust", an emotion expression of "customer attention" indicating that attention is required for the remarks of the customer is generated, It interprets the emotion analysis unit 74 of the summary generation server 7 and supplies the summary sentence generation unit 73. The abstract sentence generation unit 73 of the abstract generation server 7 receives the “customer notice” supplied from the emotion analysis server 6 in the “absent food is mixed with food” in the abstract sentence in the lower part of FIG. In parentheses.
As described above, by reflecting the speaker's emotional expression in the generated summary text, it becomes possible to easily grasp the situation of the speaker's emotional transition and automatically extract the problem call for which measures should be taken.

Returning to FIG. 6, the summary generation unit 73 of the summary generation server 7 uses the emotion analysis result as described above in S17, and as shown in FIG. 27, the emotion more neatly categorized from the emotional words in the summary sentence Replace with a representation and, as shown in FIG. 26, add to the summary to be output.
In S18, the abstract sentence generation unit 73 or the abstract sentence shortening unit 75 outputs the finally generated abstract sentence to the file of the abstract sentence text 38.

An example of extraction and conversion processing up to the generation of a final output summary sentence from interactive speech text divided into summary units output by the speech recognition server 5 will be described with reference to FIGS. 28 to 30.
FIG. 28 shows a non-limiting example of interactive speech text of one call unit output from the speech recognition server 5 and input to the summary generation server 7. In the dialogue voice text of FIG. 28, the dialogue voice text of one utterance unit is shown in each line for each identified speaker (operator (OP) or customer (CS)), and the dialogue voice text of each line is A break of summary units indicated by a square is inserted.
FIG. 29 shows one non-limiting example of the summary text output in an intermediate manner by the summary generation section 73 of the summary generation server 7 from the interactive speech text shown in FIG. As shown in FIG. 29, the text of the sixth utterance unit (the third, sixth, ninth, eleventh, fourteenth, and fifteenth utterance units of the text) is extracted from the text of the twenty utterance units of FIG. And each of the extracted utterance unit texts is converted into more concise text for a summary sentence. The summary sentence generation unit 73 converts the interactive speech text of the entire one call of FIG. 28 into the intermediate summary text of FIG. 29 by referring to the key word table 34, the unnecessary word table 35, and the various conversion tables 36. .

  FIG. 30 shows one non-limiting example of the summary text finally output from the summary generator 73 to the summary shortening section 75 from the intermediate output text of FIG. As shown in FIG. 30, a five-line summary sentence is generated from the text of six utterance units extracted and converted in FIG. 29, and the end of each summary sentence is “hope”, “confirmation” etc. Has been converted to. In particular, the pair of the utterance (question) of the operator in the fifth line of FIG. 29 and the utterance (answer and answer) of the customer in the sixth line is “FIG. It is summarized in one summary sentence. The abstract sentence generation unit 73 generates the finalized abstract text of FIG. 30 functioning as the response history by referring to the keyword table 34 and the various conversion tables 36. The sentence end of the condensed sentence on the 5th line of FIG. 30 is converted to “hidden” reflecting the emotional expression of the speaker (customer) by applying the above-described emotion analysis processing.

FIG. 31 shows one non-limiting example of a user interface outputted to a result display device etc. in which the interactive voice text of FIG. 28 is inquired. Referring to FIG. 31, the user interface may include the identified speaker 311, the response content 312 of the utterance unit, the play button 313, and the speaker's emotion analysis result icon 314. By selecting the play button 313 corresponding to the desired speech, the speech file of the speech is reproduced.
FIG. 32 shows that the emotion analysis result obtained from the numerical value of the emotion index is “normal” for the emotion index, “normal” for the emotion index for each speaker for the speech unit inquired in FIG. The feeling is shown as "normal" to "somewhat high", stress as "no", "somewhat", etc. 31 and 32 may be displayed on the display device so as to be visible simultaneously.

FIG. 33 shows the speech recognition result of the dialog speech of the speech unit identified as the speaker for one speech unit (recording time 1.25.716), the natural language processing result with reference to the corresponding user dictionary, etc., and the link of the speech file It is a non-limiting display example showing the start and end times in a list. As shown in the lower left of FIG. 33, a summary generated for the call unit is displayed to facilitate cross reference between each processing result and the summary. After reproducing the voice file, the user interface of FIG. 33 may display the voice recognition result and the natural language processing result in an editable manner to cause the user to correct the error.
In addition, the generated summary in the lower left of Fig. 33 indicates that the dialogue "approved to see the prospectus of the Saxa Fund on the Internet" is finally shown in the dialogue. For example, the emotion analysis result obtained from the numerical value of a plurality of emotion indicators may be added in parenthesis as in “acknowledgement” or “acknowledgement”. "" May be replaced with an expression including emotion analysis results such as "favorable" or "faintly consent".
According to this embodiment, it is possible to integrate and output the dialogue recording data, the speech recognition result of the dialogue speech, the natural language processing result, the emotion analysis result, and the generated summary sentence.

(Example of hardware configuration of each device)
FIG. 34 is a diagram illustrating an example of a hardware configuration provided in each device in the speech processing system. The voice acquisition server 2, the call recording server 3, the control server 4, the voice recognition server 5, the emotion analysis server 6, the summary generation server 7, and the PCs 9 and 10 include all or part of the hardware components shown in FIG. Each device 100 illustrated in FIG. 34 may include a CPU 101, a ROM 102, a RAM 103, an external memory 104, an input unit 105, a display unit 106, a communication I / F 107, and a system bus 108.

  The CPU 101 centrally controls the operation of the apparatus, and controls each component (102 to 107) via the system bus 1088. The CPU 101 functions as a processing unit that executes each processing such as speech recognition processing, summary generation processing, and emotion analysis processing. The ROM 102 is a non-volatile memory that stores control programs and the like necessary for the CPU 101 to execute processing. The program may be stored in the external memory 104 or a removable storage medium (not shown). The RAM 103 functions as a main memory, a work area, and the like of the CPU 101. Therefore, the CPU 101 loads a necessary program and the like from the ROM 102 to the RAM 103 at the time of execution of processing, and realizes various functional operations by executing the program and the like.

  The external memory 104 stores, for example, various data, various information, and the like necessary when the CPU 101 performs processing using a program. Further, in the external memory 104, for example, various data, various information, and the like obtained by the CPU 101 performing processing using a program or the like are stored. The input unit 105 includes various input devices such as a keyboard and a tablet. The display unit 106 includes, for example, a liquid crystal display. The communication I / F 107 is an interface for communicating with an external device, and includes, for example, a wireless LAN (Wi-Fi) interface or a Bluetooth (registered trademark) interface. The system bus 108 communicably connects the CPU 101, the ROM 102, the RAM 103, the external memory 104, the input unit 105, the display unit 106, and the communication I / F 107.

As described above, according to this embodiment, it is possible to generate a highly accurate summary sentence which is sufficiently shortened and in which the emotion of the speaker's utterance in the dialogue is sufficiently reflected from the dialogue voice. . Therefore, it contributes to the improvement of the usefulness of the summary of dialogue speech.
In addition, it is possible to implement | achieve each embodiment mentioned above combining the plurality.
The present invention can also be realized by a program that implements part or one or more functions of the above-described embodiment. That is, it can be implemented by processing that the program is supplied to a system or apparatus via a network or storage medium, and one or more processors in a computer (or CPU, MPU, etc.) of the system or apparatus read and execute the program. is there. Alternatively, the program may be provided by being recorded on a computer readable recording medium.
Further, the functions of the embodiment are not limited to those realized by executing the program read by the computer. For example, an operating system (OS) or the like operating on a computer may perform part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiment may be realized by the processing.

  As mentioned above, although embodiment of this invention was described in detail, the said embodiment only showed the specific example in practicing this invention. The technical scope of the present invention is not limited to the above embodiment. The present invention can be modified in various ways without departing from the scope of the invention, which are also included in the technical scope of the present invention.

1 PBX
2 voice acquisition server 3 call recording server 4 control server 5 voice recognition server 6 emotion analysis server 7 summary generation server 8 private line 9, 10 PC
Reference Signs List 31 dialogue speech 32 speech recognition dictionary 33 summary unit text 34 double term table 35 unnecessary word table 36 conversion table 37 emotion word table 51 speech recognition pre-processing unit 52 speech recognition unit 53 speech recognition post-processing unit 54 speech analysis unit 71 text correction unit 72 redundancy exclusion unit 73 summary sentence generation unit 74 emotion analysis unit 75 summary sentence shortening unit

Claims (8)

A speaker identification unit for identifying a speaker of dialogue from dialogue speech data;
A speech separation unit for separating the dialogue speech data into speech units for each of the speakers identified by the speaker identification unit;
A speech recognition unit that generates speech speech text by speech recognition of the speech speech data in units of the speech separated by the speech separation unit;
A summary generation unit for summarizing the dialogue speech text generated by the speech recognition unit to generate a summary text;
The dialogue voice data is analyzed in the utterance unit to derive an emotional expression for each speaker, and the derived emotional expression is added to the abstract sentence text or output in association with the abstract sentence text . The emotion analysis department of
An emotion word indicating an emotion of each speaker is extracted from the dialogue speech text generated by the speech recognition unit, the extracted emotion word is converted into a corresponding emotion expression, and the converted emotion expression is obtained. A second emotion analysis unit that replaces at least a part of the summary sentence text;
An interactive summary generator comprising: The first emotion analysis unit further derives the transition of emotions in time series of each speaker in one dialogue by analyzing the dialogue speech data in the utterance unit, and is derived for each speaker Outputting the transition of the emotion in association with the summary text;
The dialogue abstract generation device according to claim 1, characterized in that: Determine whether the same or similar text appears multiple times in the dialogue speech text of one dialogue unit, and when the same or similar text appears multiple times, it appears forward in time series Further comprising a redundancy remover for deleting text,
The dialogue abstract generation device according to claim 1 or 2 characterized by things. The redundancy exclusion unit further refers to the keyword table which defines keywords in advance, extracts the text defined in the keyword table from the interactive speech text, and positions the text immediately before the extracted text. Searching for a second text at least partially matching the reading of the extracted text, and deleting the searched text from the interactive speech text,
The dialogue abstract generation device according to claim 3 , characterized in that: A text correction unit which analyzes the dialogue speech text generated by the speech recognition unit, extracts a number of speech, assigns different units and weights according to the type of the extracted number speech, and supplies the summarizing unit with the text correction unit; Furthermore,
The dialogue abstract generation device according to any one of claims 1 to 4 , characterized in that: It further comprises a voice acquisition unit for recording a talk voice or a face-to-face dialogue voice to acquire the dialogue voice data.
An interactive speech summary generator according to any one of claims 1 to 5 , characterized in that. Dialogue summary generation method executed by a dialogue summary generation apparatus comprising a speaker identification unit, a speech separation unit, a speech recognition unit, a summary generation unit, a first emotion analysis unit, and a second emotion analysis unit And
The speaker identification unit identifies a speaker of dialog from the dialog voice data;
The speech separation unit separating the dialogue speech data into speech units for each of the identified speakers;
The speech recognition unit performs speech recognition of the dialogue speech data by the separated speech unit to generate dialogue speech text;
The summary generation unit sums up the generated dialog voice text to generate summary text;
The first emotion analysis unit analyzes the dialogue speech data in the utterance unit to derive an emotion expression for each speaker, adds the derived emotion expression to the summary text , or the summary text Outputting in correspondence with the text;
The second emotion analysis unit extracts an emotion word indicating an emotion of each speaker from the generated dialogue speech text, converts the extracted emotion word into a corresponding emotion expression, and converts the emotion word. Replacing at least a portion of the abstract sentence text with an emotional expression;
A method of generating a dialog summary, comprising: A dialogue summary generation program for causing a computer to execute a dialogue summary generation process, said program comprising:
Speaker identification processing for identifying a speaker of a dialog from dialog voice data;
Speech separation processing for separating the dialogue speech data into speech units for each identified speaker;
Speech recognition processing for speech recognition of the dialogue speech data by the separated speech unit to generate dialogue speech text;
Summarizing processing for summarizing the generated dialogue speech text to generate a summary text;
The dialogue voice data is analyzed in the utterance unit to derive an emotional expression for each speaker, and the derived emotional expression is added to the abstract sentence text or output in association with the abstract sentence text . and emotion analysis processing of,
An emotional word indicating an emotion for each speaker is extracted from the generated dialogue voice text, the extracted emotional word is converted into a corresponding emotional expression, and the converted emotional expression is used for the summary text A second emotion analysis process of replacing at least a part of the second emotion analysis process;
To execute processing including
A dialogue summary generator characterized in that.

Images