WO2021092272A1

WO2021092272A1 - Qa-bots for information search in documents using paraphrases

Info

Publication number: WO2021092272A1
Application number: PCT/US2020/059236
Authority: WO
Inventors: Mitul Tiwari; Ravi Narasimhan Raj; Madhusudan Mathihalli; Kaushik Rangadurai; Srivatsava Daruru; Quaizar Vohra; Deepak Bobbarjung; Abhisaar Yadav
Original assignee: Rupert Labs Inc.
Priority date: 2019-11-06
Filing date: 2020-11-05
Publication date: 2021-05-14

Abstract

Example data processing systems and methods are described. In one implementation, a system accesses a corpus of data and analyzes the data contained in the corpus of data to identify multiple documents. The system generates vector indexes for the multiple documents such that the vector indexes allow a computing system to quickly access the plurality of documents and identify an answer to a question associated with the corpus of data.

Description

QA-BOTS FOR INFORMATION SEARCH IN DOCUMENTS USING

PARAPHRASES

TECHNICAL FIELD

[0001] The present disclosure relates to systems and methods that are capable of analyzing and processing data to create an index of information for answering questions and the like.

BACKGROUND

[0002] The use of hots in various systems, including systems with one or more computing devices, is growing rapidly. A hot (also referred to as an “Internet hot”, a “web robot”, and other terms) is a software application that executes various operations (such as automated tasks) via the Internet or other data communication network. For example, a hot may perform operations automatically that would otherwise require significant human involvement.

[0003] Example hots include chatbots that communicate with users via a messaging service, and voice assistants that communicate with users via voice data or other audio data. In some situations, chatbots simulate written or spoken human communications to replace a conversation with a real human person. In some implementations, a hot may interact with a user to provide information to the user in response to a question or other request for information.

BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various figures unless otherwise specified.

[0005] FIG. l is a block diagram illustrating an environment within which an example embodiment may be implemented.

[0006] FIG. 2 is a block diagram depicting an embodiment of a data processing system.

[0007] FIG. 3 is a block diagram depicting an embodiment of a run time processing system.

[0008] FIG. 4 is a process diagram depicting an embodiment of a method for processing data from one or more data sources.

[0009] FIG. 5 is a process diagram depicting an embodiment of a method for processing messages received from one or more users.

[0010] FIG. 6 illustrates an example article extraction from XML using PDF as a reference.

[0011] FIG. 7 is a process diagram depicting an embodiment of a method for generating paraphrases and selecting candidates.

[0012] FIG. 8 illustrates example paraphrases.

[0013] FIGs. 9 and 10 illustrate example algorithms for performing deduplication.

[0014] FIG. 11 is a block diagram illustrating an example computing device suitable for implementing the systems and methods described herein. DETAILED DESCRIPTION

[0015] In the following disclosure, reference is made to various figures and drawings which are shown as example implementations in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the concepts disclosed herein, and it is to be understood that modifications to the various disclosed embodiments may be made, and other embodiments may be utilized, without departing from the scope of the present disclosure. The following detailed description is, therefore, not to be taken in a limiting sense.

[0016] References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0017] Implementations of the systems, devices, and methods disclosed herein may comprise or utilize a special purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed herein. Implementations within the scope of the present disclosure may also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are computer storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, implementations of the disclosure can comprise at least two distinctly different kinds of computer-readable media: computer storage media (devices) and transmission media.

[0018] Computer storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

[0019] An implementation of the devices, systems, and methods disclosed herein may communicate over a computer network. A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links, which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media. [0020] Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter is described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described herein. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0021] Those skilled in the art will appreciate that the disclosure may be practiced in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like. The disclosure may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

[0022] Further, where appropriate, functions described herein can be performed in one or more of: hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) can be programmed to carry out one or more of the systems and procedures described herein. Certain terms are used throughout the description and claims to refer to particular system components. As one skilled in the art will appreciate, components may be referred to by different names. This document does not intend to distinguish between components that differ in name, but not function.

[0023] The systems and methods discussed herein provide a conversational interface that includes an ability to interact with a computing system in natural language and in a conversational way. The described systems and methods also include a hot building platform as described herein. The systems and methods described herein enable a computing system to understand natural language so it can interpret what the user means in terms of intent and identify information to generate a response back to the user. Intent identification is a part of natural language understanding to determine an intent from the natural language of a user.

[0024] The systems and methods described herein relate to data processing systems that analyze and process data to create an index of information for answering questions and the like. Additionally, run time processing systems receive messages from one or more users and generate responses to the messages based on the index of information. In particular embodiments, systems and methods are described that access information from a corpus of data and creates an index of documents (or data elements) that are used at run time for answering questions or requests for information from one or more users. For example, particular applications may access information from a corpus of data associated with a vehicle, such as a vehicle owner’s manual, vehicle operator’s manual, frequently asked questions and answers, and the like. The data associated with the vehicle is indexed to aid in future access to relevant information. At a future time, a vehicle owner or vehicle operator may ask a question related to the vehicle. The described systems and methods can accessed the indexed data to provide an answer to the question. A similar approach is used for any type of data associated with any product, service, topic, issue, and the like.

[0025] FIG. 1 is a block diagram illustrating an environment 100 within which an example embodiment may be implemented. Any number of data sources 102 represent a corpus of data associated with a particular topic, product, service, issue, and the like. Example data sources 102 include a knowledge base corpus 106, categories that are entered or extracted 108, categories made as intents 110, and words of importance made as utterances 112. The knowledge base corpus 106 includes, for example, operating manuals, user manuals, frequently asked questions and answers, articles, product support documents, catalogs, and the like. These items may be referred to herein as “documents”, “articles”, “information”, and “data”. The knowledge base corpus 106 includes data in any number of formats, such as CSV files, JSON files, PDF documents, html pages, and the like. In some embodiments, the knowledge base corpus 106 is accessed from websites, databases, and any other data sources.

[0026] In some embodiments, the knowledge base corpus 106 is accessed from multiple sources. These data sources may be normalized into a common format, such as CSV or JSON and mapped to certain fields in an index, as discussed herein. An example type of document may expect title, description, tags, category, and subcategory fields. If data cannot be mapped to an existing field, then a new type of document index is created. Some data sources may need further processing to join some sources.

[0027] The categories entered or extracted 108 include information associated with an optional field for the category of particular documents. The category of a document can be entered manually (e.g., by a user) or extracted automatically from a manual, where, for example, a chapter name could be the category of the document. In some embodiments, these categories are used for creating intents automatically, which can help in narrowing the search space when a user enters a message. The categories made as intents 110, as discussed herein, may be used in combination with the categories entered or extracted 108. The words of importance made as utterances 112 are important words and phrases that are extracted from each document in the knowledge base. These phrases or words of importance are added as utterances for the corresponding document. An automated utterance generation process (discussed herein) is part of extracting words and phrases of importance from each document.

[0028] As shown in FIG. 1, a runtime 104 processes various questions 114 and requests from any number of users 116 via any type of interface 118. For example a vehicle operator (a user 116) may submit a question 114 related to a particular vehicle feature or operation of the vehicle. Interface 118 includes any type of user interface, such as a voice interface, a text interface, communication tools, artificial intelligence hots, chatbots, and the like.

[0029] Run time 104 also includes vector space intent classification to a best category 120 which attempts to classify the intent of a particular question 114. A text and vector space similarity search within a category 122 includes a text similarity search and/or a vector space similarity search. A text similarity search includes traditional information retrieval methods to search for the presence of words (or synonyms) in a given query. A vector space similarity search is performed by converting a query to a vector using sentence embedding during run time and comparing the query vector to the document vectors in the index (computed offline) to find the most relevant document to the query. Run time 104 generates an answer to question 124, as discussed herein.

[0030] It will be appreciated that the embodiment of FIG. 1 is given by way of example only. Other embodiments may include fewer or additional components without departing from the scope of the disclosure. Additionally, illustrated components may be combined or included within other components without limitation.

[0031] FIG. 2 is a block diagram depicting an embodiment of a data processing system 200. As shown in FIG. 2, data processing system 200 includes a communication manager 202, a processor 204, and a memory 206. Communication manager 202 allows data processing system 200 to communicate with other systems, such as the various systems discussed herein. Processor 204 executes various instructions to implement the functionality provided by data processing system 200, as discussed herein. Memory 206 stores these instructions as well as other data used by processor 204 and other modules and components contained in data processing system 200.

[0032] Data processing system 200 also includes a parsing module 208 that parses information from various data sources 102, and a taxonomy mapping module 210 that maps a parsed article (or other information) to a node in a predefined taxonomy of topics. In some embodiments, taxonomy mapping module 210 specifies how different fields need to be treated, analyzed, and indexed. An index generator 212 creates one or more indexes, as discussed herein. An intent creation module 214 automatically creates intents based on the parsed articles and other information.

[0033] Data processing system 200 also includes a hot creation module 216 that manages the creation of one or more hots, such as conversational hots, as discussed herein. A hot training module 218 trains hots using various models to identify the intent of a message or information. An automated testing module 220 measures the accuracy of a particular hot and works with a tuning module 222 and a tagging module 224 to improve the accuracy and relevancy of the hot.

[0034] FIG. 3 is a block diagram depicting an embodiment of a run time processing system 300. As shown in FIG. 3, run time processing system 300 includes a communication manager 302, a processor 304, and a memory 306. Communication manager 302 allows run time processing system 300 to communicate with other systems, such as the various systems discussed herein. Processor 304 executes various instructions to implement the functionality provided by run time processing system 300, as discussed herein. Memory 306 stores these instructions as well as other data used by processor 304 and other modules and components contained in run time processing system 300.

[0035] Run time processing system 300 also includes a message processing module 308 that receives and manages the processing of messages (e.g., questions) from one or more users.

A user identification module 310 determines the identity of a particular user and an intent classification module 312 applies an intent classification model to a received message. A disambiguation module 314 presents various disambiguation options, if necessary, to a user. A search module 316 identifies one or more articles or other information to respond to the user’s message. A ranking module 318 ranks the one or more identified articles and a filtering module 320 applies one or more filters to identify the best articles or other information. A confidence threshold manager 322 determines if the articles or other information are above a confidence threshold and a response manager 324 returns an appropriate response to a user. In some embodiments, run time processing system 300 may be included in a vehicle, kiosk, computing system, or any other device or system.

[0036] FIG. 4 is a process diagram depicting an embodiment of a method 400 for processing data from one or more data sources. For example, method 400 accesses multiple data sources or knowledge bases and creates a conversational hot that can answer questions related to the data received from the multiple data sources or knowledge bases. [0037] As shown in FIG. 4, a parser (e.g., an XML parser) 404 receives a PDF/XML document 402 or other information/article as an input. Parser 404 includes an algorithm to parse the received document 402. For example, parser 404 may separate a document or other data item into multiple chapters, categories, subcategories, text pieces, and the like. Taxonomy mapping 406 includes mapping the parsed document to a node in a predefined taxonomy of topics, such as car functions or parts. A human review 408 is performed to determine if the taxonomy mapping was good (e.g., accurately mapped into the correct topic in the taxonomy). If the mapping was not good, human review 408 includes returning the document/article to the parser to perform the taxonomy mapping again. In some embodiments, the human review also includes a suggested topic and category to train/teach the automatic mapping (e.g., taxonomy mapping) algorithm.

[0038] If human review 408 determines that the mapping is good, the correct taxonomy mapping is stored in knowledge base 410. In some embodiments, a vector index 412 includes sentences in the knowledge base 410 embedded into vectors. An index is created for computing similarity, which results in the vector index 412. In some embodiments, sentence embedding is used to create vectors for the document (e.g., title, sentences, and utterances) that are stored in the vector index 412. An entity extraction 414 extracts meaningful entities from the knowledge base 410 and adds those entries to the index. In some embodiments, various entities such as user-specified information, topics of importance, noun phrases, and the like are extracted and stored 414. An elastic search index 416 is an information retrieval index that uses an inverted word-document index. In some embodiments, elastic search index 416 supports various types of analysis, mapping fields, querying, and ranking results. An elastic search system supports traditional information retrieval based searches. In some embodiments, the elastic search system is used for text indexing, traditional text searching, and the like. The indexes 412, 414, and 416 are used during run time to quickly identify answers to user questions and other user messages. For example, the indexes 412, 414, and 416 may be stored in knowledge base 410. In some embodiments, an analyzer specifies different ways that the data can be transformed both during indexing and querying time. For example, the analyzer may support different ways to specify stemming, stop words, casing, synonyms, and the like.

[0039] Method 400 continues with an automatic creation of intents 418. For each category of articles, the method automatically creates intents and adds important phrases from the articles as utterances for the intents. A hot is then created 420 and trained 422 using various models associated with intent identification and knowledge base ranking. In some embodiments, if a message does not return the right answer from the hot, then the systems and methods relabel the correct intent and knowledge base article for the message. The systems and methods may also retrain the intent classification and knowledge base ranking algorithm to return the correct answer. An automated testing process 424 is used to measure the accuracy of the hot. Blind testing 426 is performed to evaluate the accuracy of the hot without knowing the test set. If the blind testing 426 results are not satisfactory, the knowledge base weights are tuned 428 by changing the knowledge base ranking model weights to improve accuracy and relevancy. In some embodiments, the systems and methods include a machine learning algorithm that optimizes and tunes the weights for various feature scores (e.g., text score, vector score, title similarity, utterance similarity, etc.) to combine those score features. If, after the tuning 428, the results are not satisfactory, method 400 adds utterances and tags to the articles 430 to improve accuracy. In some implementations, if a message response is not the correct article, the systems and methods may add more utterances to improve the likelihood of returning the correct article for the message. [0040] FIG. 5 is a process diagram depicting an embodiment of a method 500 for processing messages (e.g., questions) received from one or more users. A user message 502 is received from a user or a system associated with a user. In some embodiments, the user message 502 is a verbal message spoken by the user. For example, a driver or occupant of a vehicle may ask a question about operation of the vehicle or other vehicle-related topics. The user message 502 may also be referred to as a “question”, “query”, and the like. A bot 504 receives the user message 502 and an identity of the user is identified 506. In some embodiments, a cookie-based method is used to give a unique user identifier to each user, and the unique user identifier identifies each user. The method 500 continues as an intent classification model is applied 508 to identify the category for the knowledge base (e.g., the category associated with the user message 502). The method 500 then determines whether disambiguation is necessary 510. For example, if a category is not identified, then the user is prompted to disambiguate 512. In some embodiments, in the case of ambiguity, disambiguation options are presented 512 to the user to select a particular category.

[0041] If disambiguation is not necessary at 510, the method 500 continues by attempting to find an appropriate articles associated with the user message 502. For example, the method 500 may include querying 514 an elastic search and sentence to a vector index. The elastic search and text similarity search includes traditional information retrieval methods to search for the presence of words (or synonyms) in a particular query. The vector space similarity search converts a query to a vector using sentence embedding during run time and compares the query vector to the document vectors in the index (computed offline) to find the most relevant document to the query. The method 500 ranks 516 the identified (and relevant) articles and applies filters 518 to determine the best articles. [0042] The method 500 continues by determining 520 whether a relevance score for each article is above a confidence threshold level. In some embodiments, the confidence threshold level is determined by a precision/recall accuracy measure. For example, for a set of messages (for various thresholds), the number of correct responses from the hot are measured. Based on the number of correct responses, the right confidence threshold is determined. If no article is determined to be above the confidence threshold level, then the top three articles 522 are presented or communicated to the user. If at least one article is determined to be above the confidence threshold level, then the top article (e.g., the highest ranked article) is returned 524 to the user.

[0043] In some embodiments, the described systems and methods provide a transformed query that removes stop words and intent keywords from the user message. In particular implementations, the systems and methods described herein use various approaches to match the indexed data with the query. When scoring, the systems and methods may specify a scoring function for each document-query combination. In some situations, this scoring may be performed in vector space. After scoring the documents, the systems and methods may rank the results of the scoring.

[0044] In some embodiments, the answer a user is expecting is only a portion of a particular article or document. In these situations, the described systems and methods may highlight just the portion that is of interest to the user, rather than providing an entire section of data that contains additional details not necessary to answer the user’s question. This approach provides a fast answer to the user and an improved user experience. In some implementations, the systems and methods use the functionality of an elastic search to highlight the specific words in the query that appear in the article and show (or communicate) only those portions. [0045] In some embodiments, the systems and methods use fields such as the title and description of the article to retrieve the fields. In some implementations, metadata is used to help the systems and methods determine which article to retrieve for a particular type of user query, thereby improving accuracy of retrieving the correct document. In particular embodiments, during the indexing phase, the data is indexed using a nested query. During the scoring phase, each tag is treated uniquely. With vector scoring, the systems and methods select the best matched tag for comparing two articles.

[0046] In some embodiments, the existence of some of the entities might make an article much more relevant to the query than other articles. For example, the presence of a "gift cards" entity in an article might be more important for queries concerning gift cards or gifts in general. The systems and methods handle this situation by using a query template that allows the process to specify the weight of the extracted entities. For example, during the index time (whenever an article is updated or a new article is created), the systems and methods automatically extract the entities present in the article (e.g., title, description etc.) and store it back in the index. During the configuration, the systems and methods specify the weight of this field. During the messaging, both the elastic search scoring and the vector scoring weighs the field appropriately when calculating the total score. In some embodiments, the systems and methods use noun chunks that are automatically extracted from the article or standard system entities (such as cities, colors, location entities, and the like) to score them.

[0047] In some situations, the knowledge base contains large articles. It is typically undesirable to display the entire article in text and undesirable to read the entire article in voice. In some embodiments, the systems and methods summarize the large article and present just the summary to the user (in text or audio format). For example, during the index time, if the systems and methods find the article to be too large, they automatically create a summary of the paragraph using an extractive summarizer. The summarizer picks the salient sentences from the large number of sentences and creates a summary. The summary is then stored back to the index. During the query time, if the systems and methods find the description to be too large, and if summarization is enabled in the hot, the systems and methods return just the summary from the index. In some embodiments, the summary is used as an index to be retrieved if the systems and methods determine that the original article is noisy. In some embodiments, if the article is too big and contains a significant amount of information (other than the relevant parts) that is not relevant to the given query, then a summary might be better to return to the user instead of the entire article.

[0048] FIG. 6 illustrates an example article extraction 600 from XML using PDF as a reference. As shown in FIG. 6, a source file 602 is analyzed and mapped to a taxonomy file 604.

[0049] Automated Utterance Generation

[0050] As mentioned above, conversational AI assistants and conversational hots are growing in popularity and their ability to answer questions is an important feature. Using relevant utterances as features in answering questions has shown to improve both the precision and recall for retrieving the right answer by a conversational hot. Therefore, utterance generation has become an important problem with the goal of generating relevant utterances (e.g., sentences or phrases) from a knowledge base article that consists of a title and a description. However, generating good utterances typically requires a significant amount of manual effort, creating the need for an automated utterance generation. The systems and methods discussed herein 1) use extractive summarization to extract important sentences from the description, 2) use multiple paraphrasing techniques to generate a diverse set of paraphrases of the title and summary sentences, and 3) select good candidate paraphrases with the help of a candidate selection algorithm.

[0051] Utterance generation is an important problem in Question-Answering,

Information Retrieval, and Conversational AI Assistants. Conversational skills developed for these devices need to understand various ways that an end user is asking a question, and be able to respond accurately. While voice assistants are becoming common, chatbots and conversational interfaces are being adopted for various conversational automation use cases such as website assistants, customer service automation and IT and enterprise service automation.

[0052] Question-answering is an important part of any conversational automation use case. It is important that a conversational assistant understands various paraphrases and utterances that could be used in asking the same question. Using relevant utterances as features in a question-answering system has shown to improve the accuracy both in terms of precision and recall to retrieve the right answer.

[0053] The systems and methods discussed herein address the problem of utterance generation in the context of a conversational virtual assistant and question-answering. As used herein, paraphrases are sentences/phrases that contain different words or different sequences of words but have the same meaning.

[0054] A knowledge base article (such as FAQs and manuals) usually consists of a title and an associated description. A user who needs help with a particular issue can frame the same issue in different ways. For example, a user who wants to pay their bill can use “How do I pay my bill?”, “I want to pay my bill”, or “I wish to settle my dues”. But, the article may be titled as “Pay your bill”. Existing information retrieval based models lack recall when the words chosen by a user are different from the article but are semantically related. Enriching articles with utterances which are semantically similar to their content can improve recall and precision of IR using the systems and methods described herein. As described below, the systems and methods automatically generate utterances for a given article, which can be further curated and used by human annotators to prepare a final list of reference utterances for the article.

[0055] In some embodiments, a conversational assistant user can refer to an article using an utterance which is either a paraphrase of the title or is related to the text present in the description. The former motivates the need for paraphrase generation. However, descriptions can be long and often contain sentences that users don’t refer to. Therefore, the systems and methods use extractive summarization to select important sentences from the description, following which the systems and methods generate paraphrases for each of the extracted sentences as well as the title of an article. The goal is to generate a diverse set of paraphrases for a sentence and, therefore, the systems and methods attempt to generate a large number of diverse paraphrases. The systems and methods first use multiple paraphrasing techniques to generate a large pool of paraphrases followed by implementation of a candidate selection algorithm to select useful and relevant paraphrases for each input sentence.

[0056] FIG. 7 is a process diagram depicting an embodiment of a method 700 for generating paraphrases and selecting candidates. Paraphrase generation uses many different methods for generating paraphrases, such as full backtranslation 702, noun/verb phrase backtranslation 704 using constituency parsing, synonym replacement 706, and phrase replacement 708.

[0057] The systems and methods perform full backtranslation 702 using neural machine translation models for generating paraphrases. In some embodiments, the systems and methods first generate multiple German translations of the input English sentence. For each of the German translations, the systems and methods generate multiple English translations. In order to generate multiple translations, the systems and methods use beam search at the time of decoding. In other embodiments, any other language may be used instead of German.

[0058] The described systems and methods use noun/verb phrase backtranslation 704 by generating paraphrases for a certain meaningful phrase from the input sentence. In some embodiments, the systems and methods use the Berkeley Neural Parser (Kitaev and Klein 2018) to perform constituency parsing and extract all noun and verb phrases from the input sentence. For each of these extracted phrases, the systems and methods generate backtranslations and replace the phrase in the original sentence with its respective backtranslations.

[0059] In some situations for implementing synonym replacement 706, paraphrasing involves replacing a single word with another having equivalent meaning in the context. To account for this, the systems and methods find synonyms for words in the input sentence from synsets obtained using WordNet (Miller 1995) and replace the word with its synonym. In some implementations, the systems and methods do not consider words which are stopwords, whose part-of-speech tag belongs to a manually curated list of tags, or which are less than three characters long.

[0060] WordNet typically contains synonyms for single words, whereas noun and verb phrase backtranslation generate paraphrases for only certain types of phrases. Phrase replacement 708 (PPDB) is a database of paraphrases of commonly occurring phrases, extracted from a bilingual corpus. The described systems and methods use this resource to replace all matching phrases from the input sentence with their paraphrases.

[0061] The multiple techniques for paraphrasing discussed above generates a large pool 710 of paraphrases that could potentially contain sentences which are semantically different from the input sentence or synonyms replaced in the wrong context as well as duplicates of the title and each other. This necessitates a method to select relevant candidate paraphrases. As part of the candidate selection algorithm, the systems and methods first remove the irrelevant sentences using a filtering mechanism 712, following which the systems and methods use a de-duplication method 714 (USE-Based de-duplication) and 716 (Token-Based de-duplication) to remove duplicates. The goal of filtering is to remove paraphrases that are not semantically equivalent.

As shown in FIG. 7, the filtering and de-duplication 712-716 is associated with candidate selection. The result of the processing in FIG. 7 is one or more final candidates 718.

[0062] In some embodiments, USE-based de-duplication 714 uses the Universal Sentence Encoder to get vector representations of the input sentence and the paraphrase, and compute the cosine similarity between them. If the cosine similarity between the representations is less than 0.5, the paraphrase is considered to be semantically different and is discarded. Analogously, if the similarity is greater than 0.95, the paraphrase is considered to be a duplicate of the input and, therefore, is also discarded.

[0063] In some embodiments, USE-based similarity does not explicitly tell if the paraphrase is semantically equivalent to the input sentence, but instead determines how similar the paraphrase is to the sentence. For similar sentences (e.g., sentences with high word overlap) which are not semantically equivalent, USE-based filtering may not work. To improve the precision of filtering, the systems and methods use a paraphrase detection model based on BERT (Devlin et al. 2019).

[0064] FIG. 8 illustrates multiple example paraphrases 800 generated by the systems and methods discussed herein. [0065] In some embodiments, to remove duplicates, the systems and methods run two algorithms sequentially after the filtering step. FIG. 9 illustrates a first algorithm 900 that uses similarity based on USE to de-duplicate the pool 710. In particular, algorithm 900, at every step, finds the paraphrase that has the highest cosine similarity with the original sentence and selects it if it does not have a high similarity with any of the paraphrases already selected.

[0066] FIG. 10 illustrates a second algorithm 1000 that diversifies the final set by selecting the paraphrase with the highest number of unique words at every step. The systems and methods only consider words that are not stopwords, have a character length of more than two and whose POS tags do not belong to a manually curated list of POS tags (such as prepositions, conjunction words, and forms of the verbs “be” and “have”).

[0067] When performing deduplication, many of the paraphrases generated may have just one or two keywords which are different and unique from the input sentence. It is important to select the sentences that are more related to the input sentence and which also are more probable as a sentence. Therefore, for each paraphrase, the systems and methods compute two scores, namely, the similarity between the USE encodings of the input sentence and the paraphrase, and a score computed using the cross entropy loss from the BERT model probabilities. The systems and methods normalize both of these scores across examples and use the average for tie-breaking.

[0068] In some embodiments, the systems and methods select the important sentences from the description by using extractive summarization. In particular implementations, the systems and methods only summarize a description if it is more than three sentences long. Otherwise, all sentences are selected as important sentences. [0069] As discussed above, given an article in a knowledge base consisting of a title and a description, the goal of the utterance generation process is to generate different utterances that potentially correspond to users’ utterances with that particular knowledge base article. These generated utterances can be further curated and used by human annotators to prepare a final list of utterances for the article. As discussed herein, the method proposed for utterance generation uses paraphrase generation and extractive summarization techniques to generate utterances. Paraphrase generation is used to generate multiple paraphrases of the title of an article, whereas extractive summarization is used to select the relevant sentences from the description of the article.

[0070] For paraphrase generation, to encourage diversity, the systems and methods use multiple techniques to over-generate paraphrases for a given input sentence. For example, the systems and methods may generate five best German translations of the input sentence (using beam search). For each of these translations, the systems and methods generate five best English translations, also using beam search, resulting in a total of 25 paraphrases for an input sentence. The systems and methods extract all noun and verb phrases from the input sentence using constituency parsing. Once noun and verb phrases are obtained, the systems and methods obtain 25 backtranslations for each of these phrases (using the same method as in the previous method) and replace the original phrase in the input sentence with the backtranslated phrase.

[0071] In some situations, a user might input different verbs/nouns for the same meaning, which the backtranslation might not capture. To account for this, the systems and methods replace nouns, verbs and adjectives (excluding variations of “be” and “have”) with their respective synonyms from synsets obtained using WordNet. The previous method for synonym replacement replaces only a single word. Because of this, the systems and methods use look-up into the PPDB database to replace longer phrases, typically of two to four words, with their paraphrases. The pool of generated paraphrases from the first step is very likely to contain irrelevant paraphrases as well as duplicates. Therefore, the systems and methods apply a filtering and de-duplication method to the paraphrase pool and restrict the final size of the paraphrase set.

[0072] The filtering step uses Universal Sentence Encoder embeddings to compute similarity between the original sentence and the generated sentence, and it removes very similar or very dissimilar paraphrases from the pool. The systems and methods run two algorithms sequentially for de-duplicating the pool of paraphrases obtained after filtering. The first algorithm uses similarity based on USE to de-duplicate the pool. For example, at every step, the algorithm finds the paraphrase that has the highest similarity with the original sentence and selects it if it does not have a high similarity with any of the paraphrases already selected. The second algorithm focuses on selecting a diverse set of paraphrases by selecting the paraphrase with the highest number of unique unigrams at every step. A word is allowed if it is not a stopword, has a character length of more than two and its POS tag does not belong to a manually curated list of POS tags (e.g., prepositions, conjunction words, forms of the verbs “be", “have", etc.)

[0073] For generating utterances from the description, the systems and methods use extractive summarization to select salient and important sentences from the description. For each of the sentence in the summary, the systems and methods generate paraphrases using the paraphrase generation method described above. The systems and methods also perform question generation (using syntactic rules based on dependency parsing) from the summary sentences. For example, from the sentence "If you want to disconnect your phone and use it again later, simply touch Disconnect on the Bluetooth settings screen", the systems and methods generate relevant questions such as "How can I disconnect my phone?", "How do I disconnect my phone?", and "What is the procedure to disconnect my phone?".

[0074] In many situations, hots are multilingual. Therefore, the described systems and methods support the transfer of features into different languages. To support different languages, the systems and methods use plugins for elastic search, and for vector search, the systems and methods use word segmenters in combination with a sentence encoder that is trained in different languages.

[0075] In some embodiments, knowledge bases may have a large number of articles belonging to potentially multiple non-overlapping categories. For example: Gift Cards, Returns, Rewards, Credit Cards, etc. are different categories for an ecommerce FAQ knowledge base. When a user queries a knowledge base about a question on gift cards, the systems and methods provide a better user experience (better related articles) and a higher accuracy if they search only on the articles in the gift card category.

[0076] When creating a knowledge base, the systems and methods also categorize the articles into various categories they belong to. For each of the categories, the systems and methods automatically create an intent with all the tags of all the articles in the category added to the intent. The systems and methods then train an intent classification engine. At prediction time, the intent classification engine identifies the category of the knowledge base to which the query belongs. The systems and methods will then only search the knowledge base with the specified category, hence having a better experience.

[0077] Although the described systems and methods use elastic search to search for documents that have words similar to the words in the query, it is desirable for users to be able to query the knowledge base in their words. Even if the user doesn't use the words mentioned in the articles, the systems and methods should retrieve the right article. In some embodiments, the systems and methods can map words/sentences to a higher dimensional space and search in this higher dimension space.

[0078] Given a document, the systems and methods create sentence embeddings for title, each sentence of description, and each of the tags/utterances using a sentence embedding technique. These embeddings are indexed separately (one for each field) using a nearest neighbor technique. The systems and methods load these in memory at run time. Given a query, the systems and methods convert the query to a vector using a sentence embedding technique. The systems and methods then search for the query vector in each of the field vectors. This is a search in the vector space and uses approximate nearest neighbor to optimize for performance. Given the scores and top vectors in each of the fields, the systems and methods learn the weights of these fields using a linear model optimized on user data.

[0079] In a knowledge base system, some conversations or dialogs are one-shot (e.g., the user asks a question and the hot responds with an article or document). In other situations, the hot may not have all the information to answer the question. The systems and methods described herein allow the hot to navigates the system (based on entities) to find the right answer by asking the right question. This is accomplished by getting the right articles for the query based on Elastic Search + Vector Search + Re-ranker. The systems then get the entities/metadata from the query and also get them for each article. If they all match, then the method continues. However, if they don't match, the systems and methods find the difference in the entities and get the priority from the missing entities and generate a question for the entity. The systems and methods then prompt the user with this question. The process is repeated until a valid article is identified.

[0080] In some embodiments, a reranker provides a second pass ranker on top of the initial results given by elastic search and vector search. The reranker is useful for the following reasons: elastic search and vector search are recall based (the systems and methods are optimized to retrieve the article if it exists in the top k results). However, in conversational scenarios, the systems and methods would need to be more precise on the top-1 or top-3 articles. This needs for a precision-based approach that is trained to retrieve the correct top-1 answer and penalizes wrong answers at top positions.

[0081] In some embodiments, the user can ask different types of questions. Sometimes there are multiple correct answers to the question and the systems and methods should be able to detect that situation and give the user a better experience and show multiple answers. In some situations, elastic search and vector search are not normalized and cannot be compared across different scoring mechanisms to know how similar or dissimilar the top articles are. The user can ask questions which have no answer in the knowledge base but do have some similar answers. Knowing when the systems and methods don't have an answer is crucial to give a better experience. Elastic search and vector search are recall based approaches and typically don't give a good uniform score that can be used to determine this.

[0082] In some embodiments, the systems and methods gather different sources of data from the knowledge base (the tags added for each article, the user messages that are labeled by analysts to mark the correct answers, and also click data of the users to map their queries to the correct article). All these give the data in the format (User Query Article ID). The systems and methods use each of these examples as a positive example and query the knowledge base to get the closest articles to the query as negative examples.

[0083] The systems and methods train a classifier that takes in all the article data (including the vector search and elastic search scores) and query as input for each of these examples and classify whether the article is correct or not. The output of this classifier is a number between 0 and 1. The closer to 1, the more confident the systems and methods are in saying it is the correct article.

[0084] If the re-ranker is enabled for the hot, the systems and methods get the results from vector search and elastic search, predict the score using the classifier and re-rank the results based on this score. If the first result's score is less than a minimum threshold, the systems and methods don't return any results. If it is less than a not confident threshold (> minimum threshold), the systems and methods tell the user that the system is not sure and show the user a few related articles that are above the minimum threshold. If the scores of the top few results are close and are greater than the not confident threshold, the systems and methods tell the users that it found several similar articles and return these similar articles that are all relevant.

[0085] In some embodiments, the user can provide a decision tree (if the user picks Apple, respond with this. If the user picks Orange, respond with that). This approach enables customers (e.g., system administrators) to easily configure flows and build chatbots very quickly. In the naive version of the system, the user has to exactly match what was configured by the customer. However, the end users should be able to say something similar and still be able to navigate the decision tree. To solve this problem, the described systems and methods integrate dialog acts into the knowledge base decision trees. Dialog Acts are types of speech acts that serve common actions with respect to navigating a decision tree. The following are examples of dialog acts:

[0086] Affirm - user has agreed to what the hot asked (typically a Yes/No question)

[0087] Negate - user has disagreed to what the hot asked (typically a Yes/No question) [0088] Show more - user wants to see more of what was asked.

[0089] Go Back - User wants to go back one step [0090] Skip - user wants to skip this step [0091] Hold - user wants to hold

[0092] Repeat - user didn't hear what was said by the hot.

[0093] Restart - user wants to restart the flow [0094] Abort - user wants to abort the flow

[0095] In some embodiments, the described systems and methods include a classifier to classify the user utterances to dialog acts. In one implementation, this is accomplished using bidirectional attention and context aware self attention.

[0096] FIG. 11 is a block diagram illustrating an example computing device 1100 suitable for implementing the systems and methods described herein. In some embodiments, a cluster of computing devices interconnected by a network may be used to implement any one or more components of the systems discussed herein.

[0097] Computing device 1100 may be used to perform various procedures, such as those discussed herein. Computing device 1100 can function as a server, a client, or any other computing entity. Computing device can perform various functions as discussed herein, and can execute one or more application programs, such as the application programs described herein. Computing device 1100 can be any of a wide variety of computing devices, such as a desktop computer, a notebook computer, a server computer, a handheld computer, tablet computer and the like.

[0098] Computing device 1100 includes one or more processor(s) 1102, one or more memory device(s) 1104, one or more interface(s) 1106, one or more mass storage device(s) 1108, one or more Input/Output (I/O) device(s) 1110, and a display device 1130 all of which are coupled to a bus 1112. Processor(s) 1102 include one or more processors or controllers that execute instructions stored in memory device(s) 1104 and/or mass storage device(s) 1108. Processor(s) 1102 may also include various types of computer-readable media, such as cache memory.

[0099] Memory device(s) 1104 include various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 1114) and/or nonvolatile memory (e.g., read-only memory (ROM) 1116). Memory device(s) 1104 may also include rewritable ROM, such as Flash memory.

[00100] Mass storage device(s) 1108 include various computer readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., Flash memory), and so forth. As shown in Fig. 11, a particular mass storage device is a hard disk drive 1124. Various drives may also be included in mass storage device(s) 1108 to enable reading from and/or writing to the various computer readable media. Mass storage device(s) 1108 include removable media 1126 and/or non-removable media.

[00101] I/O device(s) 1110 include various devices that allow data and/or other information to be input to or retrieved from computing device 1100. Example I/O device(s) 1110 include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCDs or other image capture devices, and the like.

[00102] Display device 1130 includes any type of device capable of displaying information to one or more users of computing device 1100. Examples of display device 1130 include a monitor, display terminal, video projection device, and the like.

[00103] Interface(s) 1106 include various interfaces that allow computing device 1100 to interact with other systems, devices, or computing environments. Example interface(s) 1106 include any number of different network interfaces 1120, such as interfaces to local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interface(s) include user interface 1118 and peripheral device interface 1122. The interface(s) 1106 may also include one or more user interface elements 1118. The interface(s) 1106 may also include one or more peripheral interfaces such as interfaces for printers, pointing devices (mice, track pad, etc.), keyboards, and the like.

[00104] Bus 1112 allows processor(s) 1102, memory device(s) 1104, interface(s) 1106, mass storage device(s) 1108, and I/O device(s) 1110 to communicate with one another, as well as other devices or components coupled to bus 1112. Bus 1112 represents one or more of several types of bus structures, such as a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.

[00105] For purposes of illustration, programs and other executable program components are shown herein as discrete blocks, although it is understood that such programs and components may reside at various times in different storage components of computing device 1100, and are executed by processor(s) 1102. Alternatively, the systems and procedures described herein can be implemented in hardware, or a combination of hardware, software, and/or firmware. For example, one or more application specific integrated circuits (ASICs) can be programmed to carry out one or more of the systems and procedures described herein.

[00106] While various embodiments of the present disclosure are described herein, it should be understood that they are presented by way of example only, and not limitation. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made therein without departing from the spirit and scope of the disclosure. Thus, the breadth and scope of the present disclosure should not be limited by any of the described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents. The description herein is presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Many modifications and variations are possible in light of the disclosed teaching. Further, it should be noted that any or all of the alternate implementations discussed herein may be used in any combination desired to form additional hybrid implementations of the disclosure.

Claims

1. A method of generating a knowledge base, the method comprising: accessing, by a data processing system, a corpus of data; analyzing, by the data processing system, data contained in the corpus of data to identify a plurality of documents; and generating, by the data processing system, a plurality of vector indexes for the plurality of documents, wherein the vector indexes allow a computing system to quickly access the plurality of documents and identify an answer to a question associated with the corpus of data.

2. The method of claim 1, wherein the vector indexes further identify specific portions of each of the plurality of documents.

3. The method of claim 2, wherein the specific portions of each of the plurality of documents are each associated with a specific type of information.

4. The method of claim 1, wherein analyzing data contained in the corpus of data includes parsing the data in the corpus of data.

5. The method of claim 1, comprising storing the plurality of vector indexes in a knowledge base for access by a run time processing system to respond to user queries.

6. The method of claim 1, comprising automatically creating a plurality of intents based on the data contained in the corpus of data.

7. The method of claim 1, comprising creating a conversational hot to respond to user queries.

8. The method of claim 7, comprising: training the conversational hot; and testing responses generated by the conversational hot.

9. The method of claim 8, comprising tuning a knowledge base weight if the conversational hot does not satisfy the test.

10. The method of claim 8, comprising tagging at least one article in the knowledge base if the conversational hot does not satisfy the test.

11. A data processing system comprising: a communication manager configured to access a corpus of data; a processor configured to analyze data contained in the corpus of data to identify a plurality of documents; and an index generator configured to generate a plurality of vector indexes for the plurality of documents, wherein the vector indexes allow a computing system to quickly access the plurality of documents and identify an answer to a question associated with the corpus of data.

12. The data processing system of claim 11, wherein the vector indexes further identify specific portions of each of the plurality of documents.

13. The data processing system of claim 12, wherein the specific portions of each of the plurality of documents are each associated with a particular type of information.

14. The data processing system of claim 11, comprising a parsing module configured to parse the data in the corpus of data.

15. The data processing system of claim 11, wherein the processor is further configured to store the plurality of vector indexes in a knowledge base for access by a run time processing system to respond to user queries.

16. The data processing system of claim 11, comprising an intent creation module configured to automatically create a plurality of intents based on the data contained in the corpus of data.

17. The data processing system of claim 11, comprising a hot creation module configured to create a conversational hot to respond to user queries.

18. The data processing system of claim 17, comprising: a hot training module configured to train the conversational hot; and an automated testing module configured to test responses generated by the conversational hot.

19. The data processing system of claim 18, comprising a tuning module configured to tune a knowledge base weight if the conversational hot does not satisfy the test.

20. The data processing system of claim 18, comprising a tagging module configured to tag at least one article in the knowledge base if the conversational hot does not satisfy the test.

21. A method of answering a query, the method comprising: accessing, by a computing system, a knowledge base containing a plurality of vector indexes, wherein each vector index is associated with a particular data element; receiving, by the computing system, a query associated with the knowledge base, wherein the query is received from a user; identifying, by the computing system, an answer to the query based on the plurality of vector indexes; and communicating, by the computing system, the answer to the user.

22. The method of claim 21, wherein identifying an answer to the query includes identifying an identity associated with the user.

23. The method of claim 21, wherein identifying an answer to the query includes classifying an intent associated with the query.

24. The method of claim 21, wherein identifying an answer to the query includes determining whether the answer needs disambiguation.

25. The method of claim 24, comprising prompting the user to disambiguate the answer responsive to determining that the answer needs disambiguation.

26. The method of claim 21, wherein identifying an answer to the query includes accessing a conversational hot trained to respond to user queries.

27. The method of claim 21, wherein identifying an answer to the query includes performing a search to identify a plurality of potential candidates as answers to the query.

28. The method of claim 27, comprising ranking the plurality of potential candidates.

29. The method of claim 28, comprising filtering the ranked candidates.

30. The method of claim 21, comprising determining whether a relevance score associated with each article is above a confidence threshold.

31. The method of claim 30, comprising providing multiple answers to the user if the relevance scores associated with the articles are not above the confidence threshold.

32. The method of claim 30, comprising providing a top ranked answer to the user if at least one relevance score is above the confidence threshold.

33. A run time processing system comprising: a communication manager configured to access a knowledge base containing a plurality of vector indexes, wherein each vector index is associated with a particular data element; a processor configured to receive a query associated with the knowledge base, wherein the query is received from a user; a search module configured to identify an answer to the query based on the plurality of vector indexes; and wherein the communication manager communicates the answer to the user.

34. The run time processing system of claim 33, comprising a user identification module configured to identify an identity associated with the user.

35. The run time processing system of claim 33, comprising an intent classification module configured to classify an intent associated with the query.

36. The run time processing system of claim 33, comprising a disambiguation module configured to determine whether the answer needs disambiguation.

37. The run time processing system of claim 36, wherein the communication manager is further configured to prompt the user to disambiguate the answer responsive to determining that the answer needs disambiguation.

38. The run time processing system of claim 33, wherein the search module is further configured to identify a plurality of potential candidates as answers to the query.

39. The run time processing system of claim 38, comprising a ranking module configured to rank the plurality of potential candidates.

40. The run time processing system of claim 39, comprising a filtering module configured to filter the ranked candidates.

41. A method of generating an utterance, the method comprising: paraphrasing, by a computing system, a document using at least two different paraphrasing techniques to generate a plurality of candidate paraphrases; selecting, by the computing system, relevant paraphrases from the plurality of candidate paraphrases by: filtering the plurality of candidate paraphrases; and de-duplicating the plurality of candidate paraphrases.

42. The method of claim 41, comprising applying extractive summarization to select best text portions.

43. The method of claim 41, wherein the at least two different paraphrasing techniques include full backtranslation using a neural machine translation model.

44. The method of claim 41, wherein the at least two different paraphrasing techniques include noun/verb phrase backtranslation.

45. The method of claim 41, wherein the at least two different paraphrasing techniques include synonym replacement.

46. The method of claim 41, wherein the at least two different paraphrasing techniques include phrase replacement.

47. The method of claim 41, wherein de-duplicating the plurality of candidate paraphrases includes use-based de-duplication.

48. The method of claim 47, wherein de-duplicating the use-based de-duplication: determines vector representations of an input sentence and the paraphrase; and computes a cosine similarity between the vector representations and the paraphrase.

49. The method of claim 48, comprising discarding a particular paraphrase if the cosine similarity is less than 0.5.

50. The method of claim 48, comprising discarding a particular paraphrase if the cosine similarity is greater than 0.95.

51. The method of claim 41, wherein de-duplicating the plurality of candidate paraphrases includes token-based de-duplication.

52. The method of claim 41, wherein de-duplicating the plurality of candidate paraphrases includes both use-based de-duplication and token-based de-duplication.

53. The method of claim 41, wherein filtering the plurality of candidate paraphrases includes removing irrelevant sentences.

54. A method of generating an utterance, the method comprising: paraphrasing, by a computing system, a document using at least two different paraphrasing techniques to generate a plurality of candidate paraphrases; selecting, by the computing system, relevant paraphrases from the plurality of candidate paraphrases by: filtering the plurality of candidate paraphrases; and de-duplicating the plurality of candidate paraphrases; applying extractive summarization to select best text portions.

55. The method of claim 54, wherein the at least two different paraphrasing techniques include at least two of full backtranslation, noun/verb phrase backtranslation, synonym replacement, and phrase replacement.

56. The method of claim 54, wherein de-duplicating the plurality of candidate paraphrases includes use-based de-duplication.

57. The method of claim 56, wherein de-duplicating the use-based de-duplication: determines vector representations of an input sentence and the paraphrase; and computes a cosine similarity between the vector representations and the paraphrase.

58. The method of claim 57, comprising discarding a particular paraphrase if the cosine similarity is less than 0.5.

59. The method of claim 57, comprising discarding a particular paraphrase if the cosine similarity is greater than 0.95.

60. The method of claim 54, wherein filtering the plurality of candidate paraphrases includes removing irrelevant sentences.