JP2017532684A - System and method for language detection - Google Patents

Abstract

Implementations of the present disclosure relate to a method, system and computer program storage device for detecting the language of a text message. A number of different language detection tests are performed on messages associated with the user. Each language detection test determines a score set representing the likelihood that the message is one of a plurality of different languages. One or more combinations of score sets are provided as input to one or more different classifiers. The output from each classifier includes an indication that the message is one of different languages. The language of the message may be identified as the language presented by one of the classifiers based on the credit score and / or the identified language domain.

Description

BACKGROUND This disclosure relates to language detection, and more particularly to a system and method for detecting the language of a short message.

  In general, language detection or identification is the process of automatically detecting the language present in the text body based on the content of the text. Language detection is useful for automatic language translation. In general, it is necessary to know the language of the text message before the message can be accurately translated into another language.

  Conventional language detection is usually performed on a collection of many phrases (ie, document level), and a field that is particularly difficult to deal with is the chat text field. In this field, messages often contain a small number of words (eg, no more than four), some or all of which may be informal words or misspelled words. In the case of the chat text field, given the lack of information and informal words present in the message, it can be seen that the language detection of existing approaches is inaccurate and / or slow.

Overview Embodiments of the systems and methods described in this disclosure include, for example, information about the content of a text message, information about the keyboard used to create the text message, and / or information about the language preference of the user who created the text message. Based on that, it detects the language of the message. Compared to conventional language detection techniques, the systems and methods described in this disclosure are generally faster and more accurate, especially for short (eg, 4 words or less) text messages.

  In various examples, the system and method use a plurality of language detection tests and classifiers to determine probabilities associated with possible languages in the text message. Each language detection test can output a probability set or probability vector associated with a potential language. The classifier can determine the most likely language of the message by combining the output from the language detection test. The selection of a particular language detection test and classifier for a message depends on the message's predictive accuracy, confidence score and / or language domain.

  In one aspect, the present invention relates to a computer-implemented method for identifying the language of a message. The method includes performing a plurality of different language detection tests on a message associated with the user, each language detection test determining a score set, each of which is a set of scores, The score represents the likelihood that the message is one language among a plurality of different languages. The method further includes providing one or more combinations of score sets as input to one or more different classifiers, and as an output from each of the one or more classifiers, the message is out of a plurality of different languages. Obtaining a presentation that is in one language, the presentation including a credit score. The method further includes identifying the language of the message to the language indicated by one of the one or more classifiers based on at least one of the credit score and the identified language field. .

  In particular examples, the particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation method. Identifying the language of the message can include selecting a presented language based on the confidence score. Identifying the language of the message can include selecting a classifier based on the identified language domain. In some examples, the language field is or includes video games, sports, news, agenda, politics, health, and / or travel.

  In some examples, the message includes two or more of letters, numbers, symbols, and emoticons. The plurality of different language detection tests may include at least two methods selected from the group consisting of byte n-gram methods, dictionary based methods, alphabet based methods, script based methods, and user language profile methods. . Multiple different language detection tests may be performed simultaneously (eg, by parallel processing). The one or more combinations can include a score set obtained from a byte n-gram method and a dictionary based method. The one or more combinations may further include a score set derived from a user language profile method and / or an alphabet based method.

  In another aspect, the invention relates to a system for identifying the language of a message. The system includes a computer storage device that stores instructions. The system also includes a data processing device configured to execute instructions and perform the following operations. These operations include performing a plurality of different language detection tests on messages associated with the user, each language detection test determining a score set, each of which is a set of scores, Each score represents the likelihood that the message is one of a plurality of different languages. These operations further provide one or more combinations of score sets as input to one or more different classifiers, and output from each of the one or more classifiers as a message in multiple different languages. Obtaining a presentation that is in one language, the presentation including a credit score. These actions further identify the language of the message as the language presented by one of the one or more classifiers based on the confidence score and / or the identified language field. Including.

  In certain examples, the particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation method. Identifying the language of the message can include selecting a presented language based on the confidence score. Identifying the language of the message can include selecting a classifier based on the identified language domain. In some examples, the language field is or includes video games, sports, news, agenda, politics, health, and / or travel.

  In some examples, the message includes two or more of letters, numbers, symbols, and emoticons. The plurality of different language detection tests may include at least two methods selected from the group consisting of byte n-gram methods, dictionary based methods, alphabet based methods, script based methods, and user language profile methods. . Multiple different language detection tests may be performed simultaneously (eg, by parallel processing). The one or more combinations can include a score set obtained from a byte n-gram method and a dictionary based method. The one or more combinations may further include a score set derived from a user language profile method and / or an alphabet based method.

  In another aspect, the invention relates to a computer program product for controlling a processing mode of a data processing device stored in one or more storage devices. When executed by a data processing device, the computer program product causes the data processing device to execute the following operations. These operations include performing a plurality of different language detection tests on messages associated with the user, each language detection test determining a score set, each of which is a set of scores, Each score represents the likelihood that the message is one of a plurality of different languages. These operations further provide one or more combinations of score sets as input to one or more different classifiers, and output from each of the one or more classifiers as a message in multiple different languages. Obtaining a presentation that is in one language, the presentation including a credit score. These actions further identify the language of the message as the language presented by one of the one or more classifiers based on the confidence score and / or the identified language field. Including.

  In certain examples, the particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation method. Identifying the language of the message can include selecting a presented language based on the confidence score. Identifying the language of the message can include selecting a classifier based on the identified language domain. In some examples, the language field is or includes video games, sports, news, agenda, politics, health, and / or travel.

  In some examples, the message includes two or more of letters, numbers, symbols, and emoticons. The plurality of different language detection tests may include at least two methods selected from the group consisting of byte n-gram methods, dictionary based methods, alphabet based methods, script based methods, and user language profile methods. . Multiple different language detection tests may be performed simultaneously (eg, by parallel processing). The one or more combinations can include a score set obtained from a byte n-gram method and a dictionary based method. The one or more combinations may further include a score set derived from a user language profile method and / or an alphabet based method.

  Elements of the embodiments described with respect to certain aspects of the invention can be used in various embodiments of other aspects of the invention. For example, features of a dependent claim that are dependent on one independent claim can be used in the apparatus and / or method of any other independent claim.

FIG. 2 illustrates an example system that performs language detection. 6 is a flowchart illustrating an exemplary method for detecting a language of a text message. FIG. 6 is a flowchart illustrating an exemplary n-gram method for detecting the language of a text message. FIG. 5 is a flowchart illustrating an exemplary dictionary-based method for detecting the language of a text message. FIG. 5 is a flowchart illustrating an exemplary alphabet-based method for detecting the language of a text message. FIG. 6 is a flowchart illustrating an example script-based method for detecting the language of a text message. FIG. 5 is a flowchart illustrating an exemplary user language profile method for detecting the language of a text message. FIG. 3 is a schematic diagram illustrating an exemplary language detection method module. FIG. 3 is a schematic diagram illustrating an exemplary classifier module. FIG. 9 is a flowchart illustrating an exemplary method for detecting the language of a text message using the language detection method module of FIG. 7 and the classifier module of FIG. 6 is a flowchart illustrating an exemplary method for detecting a language of a text message. 6 is a flowchart illustrating an exemplary method for detecting a language of a text message. 6 is a flowchart illustrating an exemplary method for detecting a language of a text message.

DETAILED DESCRIPTION Generally, if the language information of a message (eg, keyboard information from a client device) is missing, transformed or unreliable, the language detection system and language detection method described in this disclosure can be used to Language can be specified. The system and method of the present invention improves the accuracy of language translation methods used to translate text messages from one language to another. In general, language translation requires that the source language be specified accurately. Otherwise, the translation results may be inaccurate.

  FIG. 1A illustrates an exemplary system 10 for detecting a language from a message, such as a text message or a voice message. The server system 12 provides message analysis and language detection functions. Server system 12 includes software components and databases that can be deployed in one or more data centers 14 located, for example, in one or more geographic locations. The software components of the server system 12 include a detection method module 16, a classifier module 18, and a management module 20. Software components can include subcomponents that can be executed on the same data processing device or on different individual data processing devices. The database of server system 12 includes training data 22, dictionary 24, alphabet 26, script 28, and user profile information 30. A database can reside in one or more physical storage systems. Software parts and databases are further described below.

  By providing an application, such as a web application, as an end user application, a user can provide a message to the server system 12. Users of client devices, such as personal computer 34, smartphone 36, tablet computer 38, and laptop computer 40, can use end-user applications via network 32. Other client devices are possible. The message from the user may include information regarding the device used when composing the message, eg, information regarding the keyboard, client device and / or operating system used when composing the message.

  Although the classifier module 18 and management module 20 shown in FIG. 1A are connected to a database (ie, training data 22, dictionary 24, alphabet 26, script 28 and user profile information 30), the classifier module 18 and / or Or the management module 20 does not necessarily need to be connected to a part or all of the database. In general, the classifier module 18 receives input from the detection method module 16 and the management module 20 receives input from the classifier module 18. The classifier module 18 and / or the management module 20 need not receive other inputs.

  FIG. 1B illustrates an exemplary method 100 for detecting the language of a message using the system 10. The method 100 begins by receiving or obtaining a text message generated by a user (step 102). The text message is parsed using one or more language detection methods (eg, from detection method module 16) (step 104). Each language detection method presents one or more languages present in the message. One or more classifiers (eg, from classifier module 18) are then used to combine the output from the language detection method (step 106) to provide further presentation of the language present in the message. . The one or more classifiers can include, for example, a supervised learning model, a partially supervised learning model, an unsupervised learning model, and / or an interpolation method. The language of the message is then determined based on the output from the one or more classifiers (eg, using the management module 20) (step 108).

  In some implementations, language presentations from one or more classifiers are selected by the management module 20 according to the calculated confidence score and / or language domain. For example, the classifier can calculate a confidence score that indicates the confidence associated with the prediction of the language. Additionally or alternatively, output from a particular classifier may be selected according to the language domain associated with the user or message. For example, if the message comes from a computer game environment, the output from a particular classifier that provides the most accurate language prediction can be selected. Similarly, if the message comes from a sport (eg, a sporting event), an output from another classifier that is more appropriate for the sports language field can be selected. Other possible language areas include, for example, news, agenda, politics, health, travel, web pages, newspaper articles, and microblog messages. In general, certain language detection methods or combinations of language detection methods (eg, from a classifier) may be more accurate for certain language areas than for other language areas. In some implementations, the language domain may be determined based on terminology present in the message. For example, terminology for computer games can include common slang used by gamers.

  Language detection methods used in the detection method module 16 include, for example, n-gram methods (eg, byte n-gram methods), dictionary-based methods, alphabet-based methods, script-based methods, and user language profile methods. Can be included. Other language detection methods are possible. Any of these language detection methods can be used to detect the language present in the message. The output from each method may be, for example, a set of probabilities or probability vectors associated with each possible language in the message. In some examples, using parallel computation, two or more language detection methods can be executed simultaneously, which can significantly reduce computation time.

  In one implementation, the byte n-gram method detects a language using byte n-grams instead of word n-grams or character n-grams. Preferably, a byte n-gram method is trained on a mixture of byte n-grams (eg, 1 ≦ n ≦ 4) using a naive Bayes classifier with a polynomial event model. Preferably, the model is generalized for data from different linguistic fields. Thereby, the default configuration of the model is accurate across a variety of areas including newspaper articles, online games, web pages, and microblog messages. Information about the work of identifying languages can be aggregated from various fields.

  The task of achieving language identification with a high degree of accuracy is relatively simple compared to conventional text classification settings where training data in the field is available. If model parameters learned for one language field are used to classify or sort data from another language field, the task becomes more difficult. This issue can be addressed by placing emphasis on important functions related to language specific tasks. The coping method can be based, for example, on a concept called information gain. This information gain was first introduced as a decision tree splitting criterion, but later proved useful for feature selection in text classification. In some implementations, a detection score is calculated that represents the difference in information gain for the field and language. Features with a high detection score can provide information about the language without providing information about the field. For simplicity, extra features can be removed from the set of candidate features by selecting features based on term frequency before calculating the information gain.

  With reference to FIG. 2, an exemplary byte n-gram method 200 begins by training with training data 22. For example, a naive Bayes classifier with a multinomial event model can be used to train a method with a mixture of byte n-grams. Training data 22 is preferably collected for a large number and types of languages and adjusted so that the amount of data available for each language is uniform (step 202). A part of the training data 22 is extracted and set in a test set (step 204). After selecting the training data 22, a byte n-gram model is trained on the data 22 using appropriate smoothing and backoff techniques (step 206). Since the model input feature is a byte stream from each input sentence and the source language labels of these sentences are known, the model adjusts the parameters to learn a byte sequence specific to a given language. A language label is predicted based on the trained model using the initially divided test set (step 208). The accuracy of the prediction determines the language specific performance of this byte n-gram system. In some examples, it is difficult to train a byte n-gram system for each language field by collecting data across many language fields. The reason is that there is not enough data for each field. Thus, these byte n-gram systems are typically trained to address a common field rather than a specific field. The trained model may be compiled into a program along with intermediate machine parameters (step 210). This program can function as a general-purpose language specifying system.

  In general, dictionary-based language detection methods count the number of tokens or words belonging to a language by searching for words in a dictionary or word list associated with the language. The language with the most words in the message is selected as the most likely language. If there is more than one most likely language, select the more likely or more commonly used language from the most likely languages. The language dictionary can be stored in the dictionary database 24.

  FIG. 3 is a flowchart illustrating an exemplary dictionary-based language detection method 300. A text message is provided (step 302) and a possible language set of the text message is identified (step 304). A first possible language is then selected from the set (step 306). Count words in the text message that are present in the dictionary corresponding to the possible languages (step 308). If there are additional potential languages that have not been considered in the potential language set (step 310), a new potential language is selected (step 312) and step 308 is repeated. After considering all possible languages from the set, the language with the most words in the text message may be identified as the language of the message (step 314). Alternatively or additionally, this method can be used to calculate the probability that each language in the language set is present in the message. For example, the output from a dictionary-based method may be a probability vector for each language in the set.

  To ensure the accuracy of dictionary-based language detection methods, especially in the case of short sentences, preferably a dictionary containing non-formal words or chat words (abbreviations, acronyms, slang, profane words) in addition to formal words It is desirable to use Informal words are often used in short text communication and chat rooms. Preferably, the dictionary is continuously expanded to include informal words as they are newly created and used.

  Alphabet-based methods generally rely on the finding that many languages have unique alphabets or different character sets based on the alphabetic character count of each language. For example, Russian, English, Korean, and Japanese use different alphabets. Alphabet-based methods cannot accurately distinguish some languages (eg, languages that use similar alphabets such as Latin), but can generally quickly detect a particular language. In some cases, as described in this disclosure, it is preferable to use alphabet-based methods in combination with one or more other language detection methods (eg, using a classifier). The language alphabet is stored in the alphabet database 26.

  FIG. 4 is a flowchart illustrating an exemplary alphabet-based language detection method 400. A text message is provided (step 402) and a possible language set of the text message is identified (step 404). Next, the first possible language is selected from the set (step 406). The characters in the text message existing in the alphabet of possible languages are counted (step 408). If there are additional possible languages that have not been considered in the possible language set (step 410), a new possible language is selected (step 412) and step 408 is repeated. After considering all possible languages from the set, the language with the most characters in the text message may be identified as the language of the message (step 414). Alternatively or additionally, alphabet-based methods can be used to calculate the probability that each language in the language set is present in the message. For example, the output from an alphabet-based method may be a probability vector for each language in the set.

  In general, script-based language detection methods determine the character count of each script (eg, Latin script, CJK script, etc.) that may be present in a message. Script based methods rely on the observation that different languages may use different scripts (eg, Chinese and English). The method preferably uses a map that maps the script to a list of languages that use the script. For example, the map takes into account the Unicode values of characters or symbols present in the message. These Unicode values may be mapped to the language or possible language set corresponding to the message. Language scripts and Unicode values or range values may be stored in the script database 28.

  Referring to FIG. 5, in an example script-based method 500, a text message is provided (step 502) and a script present in the message is identified (504). Next, the number of characters in each script is counted (step 506). The script with the maximum number of characters is considered the most possible script (step 508) and the language corresponding to the most possible script is identified (step 510). If the most likely script corresponds to only one language, that language is considered the most likely language. If the most likely script corresponds to multiple languages, additional language detection methods can be used to perform further detection. In some implementations, the output from the script-based method is a set of probabilities (eg, vector form) for each possible language present in the message.

  Methods based on user language profiles use a user profile database 30 that stores past messages sent by various users. The language of these stored messages can be determined using, for example, one or more other language detection methods (eg, byte n-gram methods) that are described in this disclosure and that identify the language used by each user, Detected. For example, if all the user's past messages are in Spanish, the user's language profile can indicate that the user's preferred language is Spanish. Similarly, if the user's past message is a mixture of different languages, the user's language profile can indicate probabilities associated with different languages (eg, 80% English, 15% French, 5% Spanish). In general, methods based on user language profiles address language detection problems associated with very short messages. These messages often do not have enough information to make an accurate language decision. In this case, it is possible to predict the language of the user's message using the user's language preference, assuming that the user will continue to use the previously used language.

  Referring to FIG. 6, an exemplary user language profile detection method 600 stores a user's past message (step 602) and detects the language present in the stored message (step 604). The frequency at which different languages appear in the user's message is determined (606) and output (step 608).

  Referring to FIG. 7, the detection method module 16 can incorporate various language detection methods. A text message can be input to the detection method module 16 and one or more language detection methods can identify the language of the message. For example, each language detection method can provide a probability vector. Each probability in the vector is associated with a possible language in the message and represents the likelihood that the message will use a given language. Due to the different methods used and the information available for the messages, the probabilities from each language detection method may not match. The detection method module 16 includes, for example, an n-gram module 702 for executing an n-gram detection method (for example, byte n-gram detection method 200) and a dictionary-based module 704 for executing a dictionary-based method 300. An alphabet-based module 706 for executing the alphabet-based method 400, a script-based module 708 for executing the script-based method 500, and a user language profile module 710 for executing the user language profile method 600. Can be included or utilized. Additional language detection methods can be incorporated into the detection method module 16 as needed. Some known methods include the use of word level n-grams, Markov models and predictive modeling techniques.

  Outputs from various language detection methods within detection method module 16 can be combined using classifier module 18. With reference to FIG. 8, the classifier module 18 can include an interpolation module 802, a support vector machine (SVM) module 804, and a linear SVM module 806.

  Interpolation module 802 is used to perform linear interpolation of results from two or more language detection methods. For example, the language of the text message can be determined by interpolating the result of the byte n-gram method and the result of the dictionary based method. For the chat message “lol gtg”, the byte n-gram method has a probability of 0.3 in English, a probability of 0.4 in French, and a probability of 0.3 in Polish. (Ie, the output of the byte n-gram method is {en: 0.3, fr: 0.4, pl: 0.3}). A dictionary-based method can be determined to have a probability of 0.1 in English, a probability of 0.2 in French, and a probability of 0.7 in Polish (ie, The dictionary-based methods are {en: 0.1, fr: 0.2, pl: 0.7}). To interpolate the results of these two methods, the output from the byte n-gram is multiplied by a first weight, and the output from the dictionary based method is multiplied by the second weight. The sum of the first weight and the second weight is 1. The weighted outputs from the two methods are then added. For example, when a weight of 0.6 is given to the result of the byte n-gram, a weight of 0.4 is given to the result based on the dictionary. The two methods of interpolation are {en: 0.3, fr: 0.4, pl: 0.3} * 0.6 + {en: 0.1, fr: 0.2, pl: 0.7} * 0.4 = {en: 0.22, fr: 0.32, pl: 0.46}.

  In general, the optimal weight value for interpolating two or more values can be determined by trial and error. By trying different weights for a given set of messages, the most possible set of weights can be identified. In some cases, the weight may be a function of the number of words or characters in the message. Alternatively or additionally, the weight may depend on the language domain of the message. For example, the optimal weight for the game environment may be different from the optimal weight for the sport environment. For combinations of byte n-gram methods and dictionary-based methods, good results can be obtained using a weight of 0.1 for the byte n-gram method and a weight of 0.9 for the dictionary method. .

  The SVM module 804 may be or may include a supervised learning model that analyzes language data and recognizes language patterns. The SVM module 804 may be, for example, a multi-class SVM classifier. For the English SVM classifier, the feature vector is the concatenation of the above two distributions (ie {en: 0.3, fr: 0.4, pl: 0.3, en: 0.1, fr: 0.2, pl: 0.7}). May be. The SVM classifier is preferably trained on the labeled training data. The trained model functions as an input predictor. The feature selected in the case of language detection may be, for example, a sequence of bytes, words or phrases. The input training vector can be mapped to a multidimensional space. The SVM algorithm can then use the kernel to identify the optimal separation hyperplane between these dimensions, giving the algorithm a significant ability to predict the language (in this case). The kernel may be, for example, a linear kernel, a polynomial kernel, or a radial basis function (RBF) kernel. The preferred kernel for the SVM classifier is the RBF kernel. After training the SVM classifier using the training data, the classifier can be used to output the most likely language out of all possible languages.

  For example, training data for many messages having different message lengths, linguistic fields and / or languages may be and may include presentations representing output vectors and correct languages from different language detection methods. The training data can include many messages where the language of each message is known.

  The linear SVM module 806 may be or include a large linear classifier. An SVM classifier with a linear kernel can perform better than other linear classifiers such as linear regression. The linear SVM module 806 differs from the SVM module 804 at the kernel level. In some cases, the polynomial model performs better than the linear model and vice versa. The optimal kernel may depend on the language domain of the message data and / or the nature of the data.

  Other classifiers that can be used in the systems and methods described in this disclosure include, for example, decision tree learning, related rule learning, artificial neural networks, inductive theory programming, random forest, clustering, Bayesian networks, reinforcement learning, expression learning , Similarity and metric learning, and sparse dictionary learning. One or more of these classifiers or other classifiers can be incorporated into and / or form part of classifier module 18.

  With reference to FIG. 9, the exemplary method 900 uses the detection method module 16, the classifier module 18, and the management module 20 to detect the language of the message. The message is provided or provided to the detection method module 16 (step 902). The message may include information about the message and / or information about the user who created the message. The information may include, for example, a user identification number, information about the keyboard used by the user to create the message, and / or information about the operating system that controls the software used by the user to create the message. . For example, the message may include data indicating that the user has created a message using a French keyboard and that the user's operating system is English.

  The language of the message is detected using one or more language detection methods in detection method module 16 (step 904). Each method used by the detection method module 16 can output a prediction regarding the language present in the message. The prediction may be a vector containing the probabilities for each language that may be present in the message.

  The output from the detection method module 16 is then provided to a classifier module 18 that can combine results from two or more language detection methods (step 906). This allows for various combinations of language detection method results. In one example, the results from the byte n-gram method and the dictionary based method are combined in the classifier module 18 by interpolation. In another example, SVM combining or classification is performed on results from byte n-gram methods, dictionary-based methods, alphabetic methods, and user profile methods. Alternatively or additionally, the binding may include and take into account the results of script-based methods. Further examples include large linear combinations of byte n-gram methods, language profile methods and dictionary methods. In general, however, the classifier module 18 can combine the results of any two or more language detection methods.

  The method 900 uses the management module 20 to select an output from a particular classifier (step 908). The output may be selected based on, for example, the confidence score calculated by the classifier, the expected language detection accuracy, and / or the language field of the message. Next, the most likely language is determined from the output of the selected classifier (step 910).

  In some examples, the systems and methods described in this disclosure select a language detection method as a function of message length. For example, referring to FIG. 10, the method 1000 includes receiving or providing a message that includes information regarding the keyboard language used when composing the message (step 1002). If the message is longer than a threshold length (eg, 25 bytes or 25 characters) (step 1004), the byte n-gram method (or other method or combination of methods) can be used to detect the language. (Step 1006). The language of the message can then be selected based on the result from the byte n-gram method (step 1008). On the other hand, if the message is less than or equal to the threshold length, the system can determine whether a keyboard language is available (step 1010). If a keyboard language is available, the language of the message may be selected to be similar to the keyboard language (step 1012). Alternatively, if the keyboard language is not available, the method 1000 may again consider the message length. For example, if the message length is less than a second threshold (eg, 4 bytes or 4 characters) (step 1014), the language can be detected and selected using a dictionary based method (step 1016). If the message length is greater than the second threshold, the byte n-gram method (or other method or combination of methods) can be used to detect the language of the message (step 1018). The results from the byte n-gram method and the dictionary based method may be combined (eg, using an interpolator or other classifier) and the language of the message may be determined based on the combination ( Step 1020).

  FIG. 11 is an exemplary method 1100 for identifying the language of a text message. A text message created on the user's client device is received or provided (step 1102). An alphabet-based method and / or script-based method is used to determine an alphabet and / or script associated with the text message (step 1104). Identify candidate languages associated with the alphabet and / or script. If the candidate language is a language with a unique alphabet and / or script (eg, Russian, Arabic, Hebrew, Greek, Chinese, Taiwanese, Japanese or Korean) (step 1106), the candidate The language is determined as the language of the text message (step 1108).

  On the other hand, if the candidate language is not a language with a unique alphabet and / or script, the length of the text message is evaluated. If the message length is less than a threshold length (eg, 4 bytes or 4 characters) and the text message includes or comprises the keyboard language used by the client device (step 1110), the language of the message is selected as the keyboard language. (Step 1112).

  Alternatively, if the message length is longer than the threshold length or the keyboard language is not available, the first of the text message is processed by processing the message using an n-gram method (eg, byte n-gram method). A language set having the possibility of (1) is specified (step 1114). A second possible language set of the text message is then identified by processing the message using a dictionary based method (step 1116). If a user language profile exists (step 1118), the first possible language set, the second possible language set, and the user language profile (eg, using an SVM classifier or a large linear classifier). By combining (1120), a first possible language combination is obtained (step 1122). The language of the text message is then selected based on the first possible language combination (step 1124). On the other hand, if the user language profile is not available, the first possible language set and the second possible language set are combined (eg, using a linear interpolator or other classifier) to One possible language combination is obtained (step 1126). Finally, the language of the text message is selected based on the second possible language combination (step 1128).

  In some examples, language detection is performed by combining the output of multiple language detection methods in two or more steps. For example, the first step uses a method based on an alphabet script, and uses Chinese (cn), Japanese (ja), Korean (ko), Russian (ru), Hebrew (he), Greek (el ), Special languages that use a unique alphabet or script, such as Arabic (ar). Optionally, the second step uses a combination (eg, from a classifier) of multiple detection methods (eg, byte n-gram methods, user language profile based methods and dictionary based methods). , Other languages present in the message (eg, Latin, etc.) can be detected.

  In some examples, messages provided or received for language detection are not specific to a particular language and / or are specific numbers, letters or images (eg, that can be recognized by any user, regardless of language preference). , Emoticons or emoji). The systems and methods described in this disclosure can ignore such characters or images or messages containing only such characters or images when performing language detection.

  FIG. 12 is a flowchart illustrating an exemplary method 1200 for detecting the language of a message. The method uses the detection method module 16, the classifier module 18, and the manager module 20 to identify the most likely or best language 1202 for a given input message 1204. Input message 1204 may include information about the user or system used when composing the message. For example, input message 1204 may include a user identification number (or other user identifier), information about the keyboard used to create the message (eg, keyboard language), and / or the operating system used to create the message. Information about the system (eg, operating system language) may be included.

  In the illustrated exemplary method 1200, the detection method module 16 includes ten different language detection methods. The three language detection methods included in the detection method module 16 are byte n-gram A 1206), byte n-gram B 1208, and byte n-gram C 1210. These are all byte n-gram methods and can be configured to detect different sets or numbers of languages. For example, byte n-gram A 1206 is configured to detect 97 languages, byte n-gram B 1208 is configured to detect 27 languages, and byte n-gram C 1210 is It may be configured to detect 20 languages. Two of the language detection methods included in detection method module 16 are dictionary-based methods, dictionary A 1212 and dictionary B 1214 that may be configured to detect different sets or numbers of languages. For example, dictionary A 1212 may be configured to detect nine languages and dictionary B 1214 may be configured to detect ten languages. Two of the language detection methods included in detection method module 16 are user language profile methods, with language profile A 1216 and language profile B 1218 that can be configured to detect different sets or numbers of languages. is there. For example, language profile A 1216 may be configured to detect 20 languages, and language profile B 1218 may be configured to detect 27 languages. Two of the language detection methods included in detection method module 16 are alphabet-based methods, alphabet A 1220 and alphabet B 1222, configured to detect different sets or numbers of languages. For example, alphabet A 1220 may be configured to detect 20 languages, and alphabet B 1222 may be configured to detect 27 languages. The detection method module 16 further includes a language detection method 1224 based on a script.

  Outputs from different language detection methods in detection method module 16 are combined and processed by classifier module 18. For example, interpolation classifier 1226 combines the output from byte n-gram B 1208 and dictionary B 1214. The interpolation weight of byte n-gram B 1208 may be 0.1, for example, and the interpolation weight of dictionary B 1214 may be 0.9, for example. The classifier module 18 may also use an SVM classifier 1228 that combines the output from byte n-gram C 1210, dictionary B 1214, language profile B 1218, and alphabet B 1222. The classifier module 18 may also use a first combination 1230 of a script-based method 1224 and an SVM classifier combination of byte n-gram C 1210, dictionary A 1212, language profile A 1216 and alphabet A 1220. Further, the classifier module 18 may use a second combination 1232 of the script-based method 1224 and the linear SVM classifier combination of byte n-gram C 1210, dictionary A 1212 and language profile A 1216. FIG. 12 shows the combination of language detection test, classifier and / or detection test output used for a particular classifier module 18, but other language detection tests, classifiers and / or combinations are used. You can also

  Script-based methods 1224 and classifiers can be used in a step-wise manner for both the first combination 1230 and the second combination 1232. For example, a script-based method 1224 can be used to quickly identify languages with unique scripts. If the language of the message 1204 is specified, it is not necessary to use the first combination 1230 SVM classifier or the second combination linear SVM classifier.

  In general, management module 20 may identify the language in message 1204 by selecting a particular language detection method, classifier, and / or combination of detection method outputs. The management module 20 can make the above selection according to the language domain or according to the predicted language of the message. For example, the management module 20 can select a particular classifier according to the confidence score determined by the classifier. For example, the management module 20 can select the output with the highest predicted credit score from the classifier.

  In certain implementations, the systems and methods described in this disclosure are suitable for providing language detection to multiple users as a service. This service is enabled by the speed with which the system and method specify the language and / or enhanced by the ability of the system and method to handle multiple specific technologies at runtime based on service requests from various clients. .

  Embodiments of the subject matter and operations described in this disclosure are digital electronic circuits, computer software, firmware or hardware, or combinations of one or more thereof that include the structures disclosed in this disclosure and their structural equivalents. Can be realized. Embodiments of the subject matter described in this disclosure are one or more computer programs, ie, computer programs encoded on a computer storage medium and executed by a data processing device or for controlling the operation of the data processing device. It can be implemented as one or more modules of instructions. Alternatively or in addition, the program instructions are generated by encoding an artificially generated propagated signal, eg, information for transmission to an appropriate receiver for execution by a data processing device. It may be encoded on the machine-generated electrical signal. The computer storage medium may be or include a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or one or more combinations thereof. A computer storage medium may also be a source or destination of computer program instructions that are not propagation signals but are encoded in artificially generated propagation signals. A computer storage medium may be or be included in one or more separate physical elements or media (eg, multiple CDs, disks, or other storage devices).

  The operations described in this disclosure may be implemented as operations performed by data processing devices on data stored in one or more computer-readable storage devices or received from other sources.

  The term “data processing apparatus” includes all types of machines, devices and machines for processing data, such as programmable processors, computers, chip systems, or combinations thereof. The device may include dedicated logic circuitry such as, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). In addition to hardware, the device may generate code that generates an execution environment for the computer program involved, eg, processor firmware, protocol stack, database management system, operating system, cross-platform runtime environment, virtual machine, or one or more thereof Codes that make up the combination can be included. The devices and execution environments can implement a variety of different computing model infrastructures such as web services, distributed computing infrastructures and grid computing infrastructures.

  Computer programs (also known as programs, software, software applications, scripts or code) can be written in any programming language, including compiled or interpreted languages, declarative or procedural languages, and as stand-alone programs Or any form of module, component, subroutine, object or other unit suitable for use within a computing environment. A computer program can correspond to a file in a file system, but need not necessarily correspond. A program can be part of a file that holds other programs or data (eg, one or more scripts stored in a markup language document), a single file dedicated to the program involved, or multiple equivalent files (Eg, one or more modules, subprograms or files that store portions of code). A computer program can be implemented to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

  The processes and logic flows described in this disclosure can be implemented by one or more programmable processors executing one or more computer programs that operate by manipulating input data to produce output. The process and logic flow can also be performed by dedicated logic circuits such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuits.

  Processors suitable for the execution of computer programs include, by way of example, general purpose microprocessors, special purpose microprocessors, and any one or more processors of any digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing operations in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as a magnetic disk, magnetic optical disk or optical disk, and / or operates to send and receive data to and from these mass storage devices Combined as possible. However, the computer need not have these devices. In addition, the computer may be another device such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver, or portable storage device (eg, universal serial bus). (USB) flash drive). Devices suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, CD-ROMs and DVD-ROM disks. Including all types of non-volatile memory, media and memory devices. The processor and the memory may be supplemented by a dedicated logic circuit or may be incorporated in the dedicated logic circuit.

  In order to provide user interaction, embodiments of the subject matter described in this disclosure include a display device (eg, a CRT (cathode ray tube) monitor or LCD (liquid crystal display) monitor) for presenting information to the user, It can be implemented on a computer with a keyboard and pointing device (eg, a mouse or trackball) that allows the user to provide input to the computer. Other types of devices can also be used to provide user interaction. For example, the feedback provided to the user may be any type of sensory feedback, such as visual feedback, audio feedback or tactile feedback, and the input from the user may be in any form including acoustic input, audio input or tactile input. Can be received. In addition, the computer may send a web page to the web browser on the user's client device in response to a request received from the web browser, for example, by sending and receiving documents to and from the device used by the user. Can interact with the user.

  Embodiments of the subject matter described in this disclosure include a backend component such as a data server, or a middleware component such as an application server, a graphical user interface that allows a user to interact with an implementation of the subject matter described in this disclosure, or It can be implemented in a computing system that includes a front component, such as a client computer having a web browser, or a combination of one or more back-end components, middleware components, or front-end components. The components of the system can be interconnected with any form or medium of digital data communication, eg, a communication network. Examples of communication networks include a local area network (LAN), a wide area network (“WAN”), an inter-network (eg, the Internet), and a peer-to-peer network (eg, a temporary peer-to-peer network).

  The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. A client and a server are computer programs that are executed on corresponding computers and have a client-server relationship with each other. In some embodiments, the server sends data (eg, an HTML page) to the client device (eg, for displaying data to a user interacting with the client device and receiving input from the user). Data generated by the client device (eg, a result of user interaction) can be received from the client device on the server.

  This disclosure includes many specific implementation details, but these details are not intended to limit the scope of the invention or the claimable scope, but rather are descriptions of features specific to particular embodiments of a particular invention. Should be considered as. Certain features that are described in separate embodiments of the disclosure can also be implemented in combination in a single embodiment. Conversely, various features that are described in a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features have been described above as acting in particular combinations, in spite of such descriptions, one or more features may be deleted from the described combinations, and the described combinations are sub- It may be transformed into a combination.

  Similarly, operations are shown in a particular order in the drawings, but in order to achieve the desired result, these operations need to be performed according to the illustrated order or sequence, or all illustrated operations are performed. It should not be understood that you need to. In certain situations, multi-tasking and parallel processing can be advantageous. For example, multiple language detection methods can be executed simultaneously using parallel processing. Further, the separation of the various system elements in the above embodiments should not be construed as requiring such a separation in all embodiments, and the described program elements and systems are generally single It should be understood that it can be integrated into a single software product or packaged into multiple software products.

  Accordingly, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Moreover, to achieve the desired results, the processes shown in the accompanying drawings need not necessarily follow the specific order or sequence shown. In certain implementations, multi-tasking and parallel processing may be advantageous.

Claims (30)

A computer-implemented method for identifying the language of a message, comprising:
Performing a plurality of different language detection tests on messages associated with the user, each language detection test determining a score set, each of which is a set of scores, each score in the score set being The likelihood that the message is one of a plurality of different languages, the method further comprising:
Providing one or more combinations of the score sets as input to one or more different classifiers;
Obtaining an indication that the message is in one of the plurality of different languages as output from each of the one or more classifiers, the presentation comprising a confidence score, and the method Furthermore,
Identifying the language of the message as a language presented by one of the one or more classifiers based on at least one of the confidence score and the identified language field; Computer implementation method.   The method of claim 1, wherein the particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation method.   The method of claim 1, wherein identifying the language of the message comprises selecting the presented language based on the confidence score.   The method of claim 1, wherein identifying the language of the message comprises selecting the classifier based on the identified language domain.   The method of claim 1, wherein the language field is selected from the group consisting of video games, sports, news, agenda, politics, health, and travel.   The method of claim 1, wherein the message includes two or more of letters, numbers, symbols, and emoticons.   The plurality of different language detection tests includes at least two methods selected from the group consisting of byte n-gram methods, dictionary based methods, alphabet based methods, script based methods, and user language profile methods. Item 2. The method according to Item 1.   The method of claim 1, wherein the plurality of different language detection tests are performed simultaneously.   The method of claim 1, wherein the one or more combinations include a score set obtained from a byte n-gram method and a dictionary based method.   9. The method of claim 8, wherein the one or more combinations further comprises a score set obtained from at least one of a user language profile method and an alphabet based method. A system for identifying the language of a message,
A computer storage device for storing instructions;
A data processing device configured to execute the instructions and perform the following operations:
The operation is
Performing a plurality of different language detection tests on messages associated with the user, each language detection test determining a score set, each of which is a set of scores, each score in the score set being The likelihood that the message is one of a plurality of different languages, said action further comprising:
Providing one or more combinations of the score sets as input to one or more different classifiers;
Obtaining an indication that the message is in one of the plurality of different languages as output from each of the one or more classifiers, wherein the indication includes a confidence score and the action Furthermore,
Identifying the language of the message as a language presented by a classifier of the one or more classifiers based on at least one of the confidence score and the identified language field; system.   The system of claim 11, wherein the particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation method.   The system of claim 11, wherein identifying the language of the message includes selecting the presented language based on the confidence score.   The system of claim 11, wherein identifying the language of the message includes selecting the classifier based on the identified language domain.   The system of claim 11, wherein the language field is selected from the group consisting of video games, sports, news, agenda, politics, health, and travel.   The system of claim 11, wherein the message includes two or more of letters, numbers, symbols, and emoticons.   The plurality of different language detection tests includes at least two methods selected from the group consisting of byte n-gram methods, dictionary based methods, alphabet based methods, script based methods, and user language profile methods. Item 12. The system according to Item 11.   The system of claim 11, wherein the plurality of different language detection tests are performed simultaneously.   The system of claim 11, wherein the one or more combinations include a score set obtained from a byte n-gram method and a dictionary based method.   The system of claim 18, wherein the one or more combinations further include a score set obtained from at least one of a user language profile method and an alphabet based method. A computer program product stored in one or more storage devices for controlling a processing mode of a data processing device,
The computer program product, when executed by the data processing device, causes the data processing device to execute the following operations:
The operation is
Performing a plurality of different language detection tests on messages associated with the user, each language detection test determining a score set, each of which is a set of scores, each score in the score set being The likelihood that the message is one of a plurality of different languages, said action further comprising:
Providing one or more combinations of the score sets as input to one or more different classifiers;
Obtaining an indication that the message is in one of the plurality of different languages as output from each of the one or more classifiers, wherein the indication includes a confidence score and the action Furthermore,
Identifying the language of the message as a language presented by a classifier of the one or more classifiers based on at least one of the confidence score and the identified language field; Computer program product.   The computer program product of claim 21, wherein the particular classifier is a supervised learning model, a partially supervised learning model, an unsupervised learning model, or an interpolation method.   The computer program product of claim 21, wherein identifying the language of the message includes selecting the presented language based on the confidence score.   The computer program product of claim 21, wherein identifying the language of the message comprises selecting the classifier based on the identified language domain.   The computer program product of claim 21, wherein the language field is selected from the group consisting of video games, sports, news, agenda, politics, health, and travel.   The computer program product of claim 21, wherein the message includes two or more of letters, numbers, symbols, and emoticons.   The plurality of different language detection tests includes at least two methods selected from the group consisting of byte n-gram methods, dictionary based methods, alphabet based methods, script based methods, and user language profile methods. Item 22. The computer program product according to Item 21.   The computer program product of claim 21, wherein the plurality of different language detection tests are performed simultaneously.   The computer program product of claim 21, wherein the one or more combinations include a score set obtained from a byte n-gram method and a dictionary based method.   30. The computer program product of claim 28, wherein the one or more combinations further include a score set obtained from at least one of a user language profile method and an alphabet based method.