JP5094486B2 - Synonymity determination device, method, program, and recording medium - Google Patents

Description

The present invention is a technique for determining the synonymity of character string expressions included in text, and more specifically, synonym candidate expressions that are character string expressions as synonym candidates are extracted from text and consist of a pair of synonym candidate expressions. The present invention relates to a technique for generating a synonym candidate pair and determining whether or not synonym candidate expressions in the synonym candidate pair are synonymous (whether or not they indicate the same information).

  The synonym determination technique is used for detecting spelling errors in a text editor, creating a synonym dictionary by aggregating synonyms obtained by the technique, incorporating the dictionary into a search device, and using it for query expansion, etc. There is a usage.

  “PlayStation”, “PlayStation”, and “PlayStation” appearing in the specification and drawings are registered trademarks, and “Apple Inc.”, “Payeonjung”, “Takuya Kimura”, “Miki Ando”, “ “Megumi Yasu” and “Harry Potter” are names (names) of prominent companies, people, characters, etc., but this application is an invention of language processing, and the characters “(registered trademark)” are inserted, The meaning will change if the notation is changed.

  When expressing any one thing / event using a language, it is possible to select a variety of expressions, so the same information in the text consisting of a character string representing the language has multiple different expressions It exists in (character string representation). In addition, as the amount of text and the number of people who create it increase, the variation of expression for one piece of information increases. Therefore, in order to collect the same information in the text without omission, a synonymity determination method for determining whether or not two character string expressions indicate the same information is necessary. The granularity of the “character string expression indicating the same information” may vary from the document unit to the word unit, but the synonymity determination in the present invention is performed on the basis of nouns and compound noun units.

There are roughly two conventional methods for performing synonymity determination in the above-mentioned noun and compound noun unit granularity, one is an identification method, and the other is a generation method. The identification method is a method of extracting synonym candidate expressions from arbitrary text to generate synonym candidate pairs and determining whether or not they are synonymous (for example, see Non-Patent Document 1). The generation method is a method of generating all expressions that can be considered as synonym candidates for a certain character string expression, and the existence may be confirmed by using Web or the like after generation (for example, see Non-Patent Document 2).

In both methods, synonym candidate pairs are created and narrowed down to the correct answer (synonymity judgment is performed), the synonym coverage that can be acquired is the pair generation method, and accuracy is both hands of pair generation and narrowing down Depends on the law. First, if a variety of synonym candidate pairs such as character types, abbreviations, and notation fluctuations are generated, the coverage is widened, but the accuracy is lowered. To improve accuracy, limit the types of synonyms at the time of pair generation, narrow down the collection to more probable synonym candidate pairs, or use various synonyms as a method of narrowing down (synonymity judgment). It is necessary to adopt a method that can determine a word with high accuracy.

As a method for collecting synonym candidates in the identification method, a method using meta information such as a table structure, tags and symbols, and special expressions (XX and XX) (non-patent document 3) is used. There are some which are limited to specific types such as abbreviations and katakana notation and are collected by pattern matching with the features (Non-patent Document 1). In both methods, certain accuracy is ensured by providing constraints such as the type of meta information and notation when generating pairs.

The generation method is a method of generating all expressions that can be considered as synonym candidates for a certain text expression, and the existence may be confirmed using Web or the like after generation (for example, Non-Patent Document 2). In the generation method, synonym candidate pairs are generated using heuristic rules, probability models, and the like, and thus synonym candidates that can be generated are limited to specific types such as abbreviations and katakana different notations.
Hiroyuki Sakai, Shigeru Masuyama “Automatic Acquisition of Correspondence between Nouns and Abbreviations from Corpus” Proceedings of the 9th Annual Conference of the Language Processing Society of Japan, 2003, pp. 226-229 Norifumi Murayama, Manabu Okuyama “Abbreviations Automatic Estimation Using Noisy-channel model” Proceedings of the 12th Annual Conference of the Language Processing Society of Japan, 2006, pp. 763-766 Tsunehito Seki, Kazutaka Shimada, Tsutomu Endo “Synonym Extraction Using Table Structure” Proceedings of the 11th Annual Conference of the Language Processing Society of Japan, 2005, C1-6

In the conventional identification method, the method of collecting synonym candidate pairs using meta information cannot be used as synonym candidate pairs other than the notation written in a special description method. Since it specializes in (abbreviation and katakana different notation), the range that can be acquired is limited to the type, and there is a problem that the cover range is narrow.

In addition, in the generation method, synonym candidate pairs may be generated using heuristic rules as described above, and may be generated using a probabilistic model, but the former has a problem of high cost and narrow coverage. In the latter, there was a problem that the accuracy was extremely low.

  If synonyms are more likely to be synonymous as the notation is similar, such as fluctuations in notation, and if synonyms cannot be measured only by the similarity of the notation, such as abbreviations, then both of these properties are further The reason why the coverage of the conventional method is narrow is that the method has to be specialized for a specific type of synonym.

  However, paying attention to the generation process of synonyms with various types, there are three main types: (a) addition of a regular character string, (b) notation conversion by storing readings, and (c) omission. The cause can be narrowed down. The diversity of synonyms is increased when the three types occur individually and sometimes simultaneously (seven patterns of a, b, c, a + b, a + c, b + c, and a + b + c).

  The character string added in (a) is a fixed expression such as an affix expression such as “Chan” or “Tee” or a specific symbol. Therefore, if the added standard character string is deleted, it becomes the same as the notation before generation. In (b), “PlayStation” is converted to “PlayStation” and “PlayStation”, and the notation is converted while the readings are stored. Therefore, if readings are given to both synonyms, the readings are the same or very similar. In (c), characters are deleted while preserving the order of characters as in “United Nations” as in “United Nations”, so long characters include short ones, and there are some rules regarding which characters are deleted. There is sex.

  Therefore, first, the standard character string added in (a) is deleted to normalize the notation, and the notation converted in (b) is given a normalization, resulting in the notation. Or, if the readings are the same or very similar, it is determined to be synonymous. And if it is determined in the order of determining whether omission of (c) has occurred after absorbing the fluctuations generated in (a) and (b), all synonyms with diversity can be determined. It becomes possible to widen the cover range.

The present invention has been made in view of the above problems. When synonym candidates are collected, synonym candidate pairs can be used regardless of the type of synonyms in all nouns in one text (a group of sentences including at least one sentence). In order to expand the synonym candidate range of the generated diverse synonym candidate pairs to almost all types, normalization is performed from both notation and reading.

When synonym candidate pairs are generated, synonym candidate pairs are generated for all nouns, so that synonym candidate pairs can be generated regardless of the text description method. In addition, it is possible to prevent generation of synonym candidate pairs that exist across the text due to the restriction within one text but have similar notation but are irrelevant.

At the time of synonym determination, each generated synonym candidate pair is normalized from both notation and reading, and the synonym candidate pair is subjected to condition determination, and the synonym candidate pair is separated for each type. By performing the determination, a synonym determination method suitable for the type of the synonym candidate pair can be applied, and the accuracy can be improved.

In addition, since the synonym determination method of the present invention is compatible with almost all types of synonyms (abbreviations and katakana different notations), synonym determination is possible regardless of the method of collecting synonym candidates. It can also be used in combination with existing synonym candidate collection techniques.

The present invention, by entering the text, generates a synonym candidate pair from nouns and compound nouns contained therein, normalizing words each notation and reading pair readings were exactly the same notation or very similar in the process What is determined to be synonymous, and what is not determined to be synonymous and is in an inclusive relationship is determined using a classifier to determine whether it is a synonym.

  According to the present invention, synonymity of diverse character string expressions, including (a) addition of a regular character string, (b) notational conversion by storing readings, or (c) omission, etc. Can be determined with high accuracy.

  Hereinafter, the present invention will be described in detail with reference to the illustrated embodiments.

  FIG. 1 shows an example of an embodiment of a synonym determination device according to the present invention. The synonym determination device includes a synonym candidate pair generation unit 1, a synonym determination unit 2, an inverse conversion rule storage unit 3, and a syllable. It consists of a normalization rule storage unit 4, a syllable similarity table storage unit 5, an omission determination model storage unit 6, an analysis processing result table storage unit 7, and a synonym candidate pair list storage unit 8.

The synonym candidate pair generation unit 1 is a text input directly from a not-shown keyboard or the like, or read and input from a storage medium or input from another device or the like via a communication medium, here, one text (at least one text) Text data corresponding to a set of sentences including two sentences) is processed as a processing unit, and analysis processing such as well-known morphological analysis and specific expression extraction is performed, and synonym candidate expressions are extracted from the text based on the analysis results, The analysis result corresponding to the synonym candidate expression is stored in the analysis processing result table storage unit 7 and the reverse conversion rule stored in the reverse conversion rule storage unit 3 for each synonym candidate expression. Normalize the notation, normalize readings using the syllable normalization rules stored in the syllable normalization rule storage unit 4, and analyze the results And stores the result table storage section 7. Then, to create a synonym candidate paired with synonyms candidate expressions each other stored in the analysis result table storing unit 7 in total per stored synonym candidate pair list memory 8.

Synonymous determination unit 2 extracts a synonym candidate pair from synonym candidate pair list memory 8, whether synonyms candidates expressed together in the synonym candidate pairs interchangeably, the analysis result table storage section 7 Analysis results and normalization processing results corresponding to each synonym candidate expression in the stored synonym candidate pairs, and a syllable similarity table and an omission determination model storage unit 6 stored in the syllable similarity table storage unit 5 Judgment is made using the stored abbreviation judgment model as described below, and if it is synonymous, the synonym candidate pair is output as a synonym pair, which is all stored in the synonym candidate pair list storage unit 8 Similarly, the synonym pair list is output for the synonym candidate pairs.

  The determination of whether or not synonyms are made by first determining whether or not the normalized notation is exactly the same, and if it is not determined to be synonymous here, it is determined whether or not the normalized readings are similar, that is, Using the syllable similarity table stored in the syllable similarity table storage unit 5, the similarity of reading after normalization is obtained, and it is determined whether or not the obtained similarity is a predetermined value or more. If the notation or reading is in the inclusive relationship, the determination is made based on whether or not the abbreviation model is stored using the abbreviation determination model stored in the abbreviation determination model storage unit 6 (abbreviation).

Inverse conversion rule storage unit 3 deletes rules that delete prefixes and suffixes that are generally inserted when using nouns, and repetitive expressions that are used to create nicknames The reverse conversion rule for normalizing the notation of the synonym candidate expression including at least the rule to perform and normalizing the reading in accordance with the normalization of the notation is stored.

The syllable normalization rule storage unit 4 is applied to the vowel continuation, long sound, and prompting of the reading of the synonym candidate expression, so that the unit of reading length (mora and syllable) that is different between Japanese and foreign words, colloquial expression The syllable normalization rule for at least normalizing the fluctuation at the time of transliteration is stored.

Syllables similarity table storage unit 5, and a key to "notation syllable pairs differ for reading is similar", "distance (similarity)" was the value read synonyms candidate representation together in synonyms candidate pairs A syllable similarity table for determining the similarity of

Skip determination model storage unit 6 was produced in advance by machine learning, two words consisting of model determines whether the abbreviations relationship, the abbreviation relationship synonyms candidates expressed together in synonyms candidate pairs An abbreviated determination model for determining whether or not there is stored.

  Hereafter, each part mentioned above is demonstrated in detail. It should be noted that in the following description, the storage contents of the storage units 3 to 8 may be described with the reference numerals of the storage units as they are.

[Synonym candidate pair generation unit 1]
FIG. 2 shows details of the synonym candidate pair generation unit 1, which includes an analysis processing unit 11, a normalization processing unit 12, and a pair generation unit 13. In the synonym candidate pair generation unit 1, one text is input, and the analysis processing unit 11 performs text analysis processing such as morphological analysis and specific expression extraction, and extracts synonym candidate expressions based on the results. Then, after normalization processing unit 12 normalizes synonym candidate expressions, pair generation unit 13 generates synonym candidate pairs. In the present embodiment, it will be described as a case where the target-specific expression as a synonym candidate representation as an example.

The analysis processing unit 11 analyzes text using a morphological analysis technique, a specific expression extraction technique, and the like established as well-known techniques, and extracts synonym candidate expressions. In the morphological analysis, information such as morpheme (notation), reading, part of speech (including proper expression class), etc. is given to the text. At this time, simple normalization such as unification of half-width / full-width characters in the text is also completed. For the reading assignment, the most likely one may be used, or N-best may be used. In addition, if the notation is an unknown word such as an alphabet and the reading is not correctly given only by morphological analysis, the reading is given again. Regarding the provision of readings, a method of correctly estimating readings of unknown words such as alphabets (for example, Japanese Patent Application Laid-Open No. 2001-142877 (name of invention: alphabetic character / Japanese reading correspondence device and method and alphabetic word transliteration device and method) As well as a recording medium on which the processing program is recorded).

The synonym candidate expression is cut out by the unique expression extraction technique, and written to the synonym candidate expression column and the analysis result column of the analysis processing result table 7 together with the analysis result (information thereof) such as notation, reading, and part of speech. However, when synonym candidate expressions having exactly the same notation already exist in the table 7, writing is not performed so that records do not overlap. At this time, the morpheme segmentation information is stored in notation, reading, and part of speech using, for example, a symbol “/”. An example of the analysis processing result table 7 is shown in FIG. However, at this time, only the synonym candidate expression column and the analysis result column in the analysis processing result table 7 are filled, and the others are empty.

Here, for the sake of explanation, the cut-out target is a specific expression, but nouns and compound nouns may be used as synonym candidate expressions using the morphological analysis results. As a result, the created analysis processing result table 7 can have as many records as the number of different synonym candidate expressions existing in one text.

The flow of processing in the normalization processing unit 12 will be described with reference to FIG. The normalization processing unit 12 normalizes the extracted synonym candidate expression from notation and reading. The input is a list of analysis result columns of all records in the analysis processing result table 7 created by the analysis processing unit 11, and the following processing is repeated for each record in the list. When all the records have been processed, the processing in the normalization processing unit 12 ends.

  (Step s12-1) If the notation of the analysis result column in the analysis processing result table 7 is alphabet, uppercase / lowercase letters are unified to uppercase, and the same column is overwritten. Proceed to step s12-2.

  (Step s12-2) The notation rule and the reading rule (details will be described later) of the reverse conversion rule stored in the reverse conversion rule storage unit 3 are applied to the notation and the reading of the analysis result column in the analysis processing result table 7, respectively. The result is written to the analysis processing result table 7. For the export destination, out of the inverse conversion rules, the application result of the notation rule is the notation normalization column, and the application result of the reading rule is the notation + reading normalization column. The result column notation is written as it is to the notation normalization column, and the analysis result column reading is written as it is to the notation + reading normalization column.) Proceed to step s12-3.

  (Step s12-3) The syllable normalization stored in the syllable normalization rule storage unit 4 for the reading of the analysis result column of the analysis processing result table 7 and the reading of the notation + reading normalization column written in step s12-3 Apply the conversion rule (details will be described later), and write the result to the analysis processing result table 7. The result of normalizing the reading of the analysis result column is written to the reading normalization column, and the result of normalizing the reading after reading + normalization is overwritten to the notation + reading normalization column (There are no rules to apply) In this case, the reading of the analysis result column is read as it is and written to the normalization column, and the notation + reading normalization column is left as it is (not overwritten)).

In pairs producing formation unit 13 creates a synonym candidate paired with brute force all records of analysis results table 7 have been processed in the normalization processing unit 12, ID synonyms candidate representations of the pair (candidate ID) stored in the synonym candidate pair list memory 8 as a synonym candidate pair list. An example of the synonym candidate pair list is shown in FIG. With respect to the pair creation method, if the cover range is not regarded as important, it is possible to use the present invention thereafter even using a method other than brute force, for example, using meta information.

[Synonymity determination unit 2]
FIG. 6 shows details of the synonymity determination unit 2, which includes a notation similarity determination unit 21, a reading similarity determination unit 22, and an omission determination unit 23. FIG. 7 shows the flow of processing in the synonymity determination unit 2.

Synonymous determination unit 2 receives the analysis result table 7 and synonyms candidate pair list 8, the title similarity determination unit 21 reads the similarity determination unit 22 and the skip determination unit 23, a synonym candidate in synonym candidate pairs It is determined whether expressions are synonymous, and a synonym pair list is output.

That is, the notation similarity determination unit 21 determines synonymity from the normalized notation (step s21), and the reading similarity determination unit 22 determines synonymity from the normalized reading using the syllable similarity table 5 ( In step s22), the omission determination unit 23 determines whether or not the abbreviation word uses the omission determination model 6 from both normalized notation and reading, and determines synonymity (step s23). Repeating the processing of steps s21~s23 for each record synonym candidate pair list 8, recognized as synonymous synonyms candidate representations each other during its synonyms candidate pair when either is determined synonymous in the process (Step s24), the process proceeds to processing of the next synonym candidate pair. Pairs that have not been determined to be synonymous until the end are not recognized as synonyms (step s25). For information on each synonym candidate expression of the synonym candidate pair required for processing, such as notation, reading, part of speech, etc., refer to the corresponding information in the analysis processing result table 7 using the candidate ID of the synonym candidate pair list 8 To do. When the processing of all the records in the synonym candidate pair list 8 is completed, the synonym candidate pairs recognized as synonyms are output as a synonym pair list.

The flow of processing in the notation similarity determination unit 21 will be described with reference to FIG. The input is one record of the synonym candidate pair list 8. Here, the determination is made by looking at the normalized notation of each synonym candidate expression in the synonym candidate pair.

(Step s21-1) If the notation normalization column notation of each synonym candidate expression in the synonym candidate pair is exactly the same, go to step s21-2, otherwise in the notation similarity determination unit 21 The process ends, and the process proceeds to the reading similarity determination unit 22.

  (Step s21-2) It is determined to be synonymous, and the process in the synonymity determining unit 2 is terminated.

The flow of processing in the reading similarity determination unit 22 will be described with reference to FIG. A record of a synonym candidate pair that is not determined to be synonymous by the notation similarity determination unit 21 is input. Here, the determination is made by looking at each reading of the synonym candidate expression in the synonym candidate pair. There are three readings for one synonym candidate expression: reading of the analysis result column, reading of the reading normalization column, and reading of the notation + reading normalization column. Therefore, the following processing is repeated three times for each, If the synonym is determined to be synonymous in any process, the synonym is recognized and the process in the synonym determination unit 2 is terminated. If not, the process proceeds to the process in the omission determination unit 23. At this time, the reading of the analysis result column, the reading of the reading normalization column, and the reading of the notation + reading normalization column are simply described as “reading”. In the following processing, the morpheme delimiter information (“/”) is ignored during matching.

(Step S22-1) If the reading of synonyms candidate each synonym candidate expressed in the pair is identical to step S22-5, otherwise proceeds to step S22-2.

(Step s22-2) The number of syllables of reading of each synonym candidate expression in the synonym candidate pair is counted, and if it is the same, the process proceeds to step s22-3. In other cases, the process ends (if the number of repetitions of the process in the reading similarity determination unit 22 is 2 or less, the process returns to step s22-1 and proceeds to the process for the next reading of the synonym candidate pair. If there is, the process in the reading similarity determination unit 22 ends.)

(Step s22-3) The distance between syllables having the same syllable position but different readings in the reading of each synonym candidate expression in the synonym candidate pair is obtained using the syllable similarity table 5 (details will be described later). . When there are many syllables having the same syllable position but different readings, the sum of distances between syllables different between pairs is used. Proceed to step s22-4.

(Step s22-4) If the sum of the distances is smaller than a preset threshold value, the process proceeds to Step s22-5. In other cases, the process ends (if the number of repetitions of the process in the reading similarity determination unit 22 is 2 or less, the process returns to step s22-1 and proceeds to the process of the next reading of the synonym candidate pair. If there is, the process in the reading similarity determination unit 22 ends.)

  (Step s22-5) The synonymity determination unit 2 ends the process of determining synonymity.

The flow of processing in the omission determination unit 23 will be described with reference to FIG. A record of a synonym candidate pair that has not been determined to be synonymous by the reading similarity determination unit 22 is input. Here, for the synonym candidate pairs, the following processing is repeated for each of the four patterns of the analysis result column notations, notation normalization column notations, reading normalization column notations, notation + reading normalization column notations If it is determined to be synonymous in any of the processes, it is recognized as a synonym. Otherwise, it is determined as not a synonym and the process in the synonym determination unit 2 is terminated. At this time, the notation of the analysis result column and the notation normalization column are simply described as “notation”, and the reading normalization column reading and the notation + reading normalization column reading are simply described as “reading”. In the following processing, the morpheme delimiter information (“/”) is ignored during matching.

(Step s23-1) When notation is targeted, when notation is read, when reading is in an inclusive relationship, it progresses to step s23-2. In other cases, the process ends (if the number of repetitions of the process in the abbreviation determination unit 23 is 3 or less, the process proceeds to the next notation or reading of the synonym candidate pair. The process at is terminated.)

  (Step s23-2) Alignment is performed using a DP matching method (Richard E. Bellman, “Dynamic Programming”, 1957) or the like. Proceed to step s23-3.

  (Step s23-3) The feature for applying to the classifier based on the difference between before and after the omission, with the longer number of characters (number of syllables in the case of reading) of each pair before omission and after the shorter one is omitted. The details are described later. Proceed to step s23-4.

(Step S23-4) Synonym candidates expressed together in synonymous synonym candidate pair with skip determination model 6 is identity and classifier model extracted in step s23-3, it is determined whether the abbreviation relationship If it is determined that it is an abbreviation, the process proceeds to step s25-5. In other cases, the process ends (if the number of repetitions of the process in the omission determination unit 23 is 3 or less, the process proceeds to the next notation or reading process of the synonym candidate pair. The process at is terminated.)

  (Step s <b> 23-5) The synonym determination unit 2 terminates the process of determining synonymity.

As the feature used in the above-described (step s23-3), when performing extraction for each synonym candidate expression notation (notation of analysis result column or notation normalization column notation) in the synonym candidate pair,
<Synonym candidate expressions before and after omission>
-Before omission: morpheme number, number of characters, part of speech, proper expression class, character type-After omission: morpheme number, number of characters, part of speech, proper expression class, character type <Feature of morpheme unit>
・ If the entire morpheme is omitted: part of speech, notation, number of characters, character type, position information (whether it is the beginning or end or middle), whether the first morpheme remains ・ If the entire morpheme remains: part of speech, notation, number of characters, character type , Position information (whether it is the beginning, the end, or the middle), whether the morpheme at the end was omitted <Characteristic features in character units>
・ If omitted in character units: part of speech, notation, character type, position information (whether it is first or last or middle), or whether the first character in the notation is omitted ・ If left in character units: part of speech, notation, character type, Uses position information (whether it is the beginning, end, or middle) and whether the first character remains in the morpheme. However, in addition to those listed here, morphological analysis information, context information, and the like may be used. At this time, information such as part of speech and notation used uses information in the analysis processing result table 7.

Also, when extracting features for each synonym candidate expression reading (reading normalization column reading or notation + reading normalization column reading) in the synonym candidate pair, in the feature example described above, Reading is used as the feature of the notation, the number of syllables is used as the feature of the number of characters, and the number of syllables when counted by syllable is used as the feature of the position information. Also, the character type features are unified with “Katakana”.

[Inverse conversion rule (storage unit) 3]
FIG. 11 shows an example of the reverse conversion rule 3, which is used when creating a nickname or a rule for deleting a prefix form such as a prefix or suffix that is generally inserted when using a noun. Heuristic description of rules for deleting repeated expressions. All rules applicable to a set of synonym candidate pairs apply. This rule is a rule that deletes a typical character string that seems to be inserted for a more general notation in a synonym candidate expression from the notation, and corrects the reading as the character string is deleted. Therefore, it is different from a rule for creating an abbreviation (in the present invention, an abbreviation is not determined using an inverse conversion rule, but is determined using a classifier in an abbreviation determination unit). Therefore, the rule can be easily written using a regular expression or the like, and the rule is to delete a character string inserted in common with many synonyms. Note that the regular expression in FIG. 11 is described as being written in Perl. Therefore, when using other expressions, the method is followed.

  In the reverse conversion rule 3, a rule that applies to notation and a rule that applies to reading are paired. If a regular expression for notation is not applicable, the paired reading regular expression is not applied. However, on the contrary, if the regular expression for reading cannot be applied or does not exist, only the regular expression for notation may be applied (the reading may not change although the notation does not change, but the notation changes) Even if the reading doesn't change). The reverse conversion rules can be arbitrarily created and registered in addition to those shown in FIG. 11, and for example, “chan” or “Mr.” can be added as a suffix deletion rule.

[Syllable normalization rule (storage unit) 4]
FIG. 12 shows an example of the syllable normalization rule 4, in order to normalize the reading length unit (mora and syllable), colloquial expression, fluctuation in transliteration, etc. Consists of rules that apply to vowel continuations, long tones, and prompt sounds for reading synonym candidate expressions. The rules are applied in the order of rule numbers, and all applicable rules are applied.

  For synonymity determination by notation conversion, the distance between syllables having the same position but different readings is used. At that time, the syllable length is the same between synonyms by notation conversion. However, the unit of reading differs between Japanese words and mora, and foreign words and syllables. Furthermore, when giving Japanese readings to other languages (transliteration), there was a problem that the number of syllables was not the same between foreign words. Examples thereof include the following cases.

・ Japanese, colloquial "Yuko", "Yuko"
When counted by the number of mora, it is 3 mora, but when counted by the number of syllables, it becomes 3 syllables and 2 syllables.

・ Foreign words "spaghetti", "spaghetti", "spaghetti"
If counted by the number of syllables, all three are 4 syllables, but if counted by the number of mora, they become 4 mora, 5 mora, and 6 mora.

・ Japanese language reading "Wimbledon", "Wimbledon" in other languages
When counted by the number of syllables, 7 syllables and 6 syllables, and when counted by the number of mora, they become 7 mora and 6 mora.

  On the other hand, by using the syllable normalization rule as shown in FIG. 12, the unit of the reading length can be unified so that all can be counted by the number of syllables, and the alignment can be performed in (step s22-3). .

[Syllable similarity table (storage unit) 5]
FIG. 13 (a) shows a procedure for creating a syllable similarity table, and FIG. 13 (b) shows an example of a syllable similarity table. The syllable similarity table 5 is a syllable pair whose key: notation is different but reading is similar. , Value: distance (similarity). As shown in FIG. 13A, this table 5 collects words having the same standard notation and different pronunciation (reading) from the morphological analysis dictionary, and applies the syllable normalization rule 4 in the same manner as the reading normalization processing unit 12. Then, the unit of the reading length is unified into syllables, and alignment is performed when the number of syllables is equal. Then, syllable pairs with the same syllable position but different readings are extracted and counted, and the value obtained by dividing the number of occurrences of syllable pairs by the sum of the number of occurrences of each syllable constituting the syllable pair is the distance between syllables (similarity). To create.

[Omission determination model (storage unit) 6]
The abbreviation determination model 6 is a model for determining whether or not two words are in an abbreviation relationship. Notation and reading of each synonym candidate expression in the synonym candidate pair to be determined, morphological analysis information, position It consists of an identification function that inputs matching information and the like and makes a binary decision as to whether or not they are synonymous. As an identification function, for example, V.I. The discriminant function of Support Vector Machine (SVM) described in Vapnik, “The nature of statistical learning theory”, Springer, 1995, is used. Learning and determining with SVM. Here, although SVM was mentioned as a learning algorithm, you may utilize other learning algorithms, such as a decision tree and the maximum entropy method.

An example of specific processing in the above-described embodiment will be described in detail. Here, “0.9” is used as the threshold value used in the reading similarity determination unit 22 of the synonymity determination unit 2, and SVM is used as the classifier of the omission determination model 4. First, processing performed by the synonym candidate pair generation unit 1 and the synonymity determination unit 2 will be described, and then detailed application examples of the inverse transformation rule 3 and the syllable normalization rule 4, the syllable similarity table 5 and the omission determination model 6 An example of creating will be described.

[I] Process performed by synonym candidate pair generation unit 1 and synonymity determination unit 2 [synonym candidate pair generation unit 1]
If the input text to the analysis processing unit 11 is as shown in FIG. 14, synonym candidate expressions such as “Apple” and “eva” are extracted. Although the expression “Apple” appears twice in the input text, it is written once in the analysis processing result table 7. The following description will be made on the assumption that the analysis of one text has been completed as shown in FIG.

The input to the normalization processing unit 12 is a list of synonym candidate expression columns of all records in the analysis processing result table 7 shown in FIG. 15, and the following processing is repeated for each record in the list.

  First, processing is started from “Apple Inc.” of ID1.

  (Step s12-1) Since the notation “Apple / Company” in the analysis result column is not an alphabet, the process proceeds directly to Step s12-2 (example of processing here; record ID 2: eva is converted to uppercase EVA and overwritten) To do.)

  (Step s12-2) The notation regular expression and the reading regular expression of the reverse conversion rule 3 (details will be described later) are applied to the analysis result column notation “Apple / Company” and the analysis result column reading “Apple / Sha”, respectively. Then, “Apple” and “Apple” respectively. The result is written to the notation normalization column and the notation + reading normalization column of the analysis processing result table 7, and the process proceeds to step s12-3.

  (Step s12-3) The syllable normalization rule 4 (details will be described later) is applied to the analysis result column reading “Apple / Sha” and the notation + reading normalization column reading “Apple” written in Step s12-2. Apply. The former is “apple / sha” and is written to the reading normalization column. The latter becomes “apple” and is overwritten to the notation + reading normalization column.

  The above processing is repeated in the same manner after ID2 (in FIG. 15, ID14 is displayed). The result is shown in FIG.

Here, in (step s12-2), the reverse conversion rule 3 is applied because of the records in the processing result table 7 shown in FIG. 3 (the morpheme delimiter “/” is not necessary from the following example) Omitted as far as possible).
・ Record ID 1: Apple (→ notation: Apple, notation + reading: Apple)
・ Record ID 10: Shokotan (→ Notation: Choco, Notation + Reading: Choco)
・ Record ID 12: Yong (→ Notation: Yon, Notation + Reading: Yon)
Record ID 14: Mikiti (→ notation: Miki, notation + reading: Miki)
There are four.

In addition, the syllable normalization rule 4 is applied in (step s12-3) among the records in the processing result table 7 shown in FIG.
・ Record ID1: Apple Inc. (→ Reading: Apulcia, Notation + Reading: Apple)
・ Record ID2: Shoko (→ reading: sho, notation + reading: shoko)
・ Record ID4: Hakkeijima Sea Paradise (→ Reading: Hakeijimashi Paradise, Notation + Reading: Hakeijimashi Paradise)
・ Record ID8: Apple (→ reading: apple, notation + reading: apple)
-Record ID 11: Sea Para (→ Reading: Shipara, Notation + Reading: Shipara)
These are five.

Create a synonym candidate pairs in all records brute analysis processing result table 7 shown in FIG. 3 created in pairs producing formation unit 13 normalization processing unit 12, the output of the synonym candidate pair list 8 shown in FIG. 5 To do.

[Synonymity determination unit 2]
The input to the synonym determination unit 2 is the synonym candidate pair list 8 shown in FIG. 5 and the analysis processing result table 7 shown in FIG. Hereinafter, the processing from the notation similarity determination unit 21 to the omission determination unit 23 is repeated for each record in the list 8 of FIG. 5, and when the synonym candidate pair is determined to be synonymous in some process, the synonym candidate pair is recognized as synonymous. Then, the process proceeds to processing of the next synonym candidate pair. Pairs that have not been determined to be synonymous until the end are not recognized as synonyms. When all the records have been processed, synonym candidate pairs recognized as synonyms are output as a synonym pair list.

  First, processing is started from record ID 1 (“Apple”, “EVA”) in the synonym pair list 8.

When the notation similarity determination unit 21 refers to the analysis processing result table 7 from the candidate IDs 1 and 2 corresponding to the record ID 1 of the synonym pair list 8, the notation normalization columns are “Apple” and “EVA”. Therefore, the processing in the notation similarity determination unit 21 is terminated, and the process proceeds to the reading similarity determination unit 22.

The reading similarity determination unit 22 refers to the reading of the processing result table 7 from the candidate IDs 1 and 2 corresponding to the record ID 1 of the synonym pair list 8 and performs determination. The reading of this synonym candidate pair is the reading of the analysis result column: "Applesha", "Eva", the reading normalization column: "Apulcia", Eva ", the notation + the reading normalization column reading:" Apple " ”And“ Eve ”. The following process is repeated three times for each.

  First, the analysis result column readings: “Applesha” and “Eva” are processed.

(Step s22-1) Since reading of synonym candidate pairs is different, the process proceeds to Step s22-2.

  (Step s22-2) When the number of syllables to be read is counted, “Applesha” is different in four syllables and “Eva” is different in two syllables.

  The process returns to step s22-1 and repeats the processing “Apulcia”, “Eve”, “Apple”, and “Eve”, but the number of syllables differs from each other and is not synonymous.

  The omission determination unit 23 determines the following four types of information regarding the pair of record ID 1 in the synonym pair list 8 with reference to the analysis processing table 7. Analysis result column notation: “Apple”, “EVA”, notation normalization column notation: “Apple”, “EVA”, reading normalization column reading: “Aprsha”, “Eva”, notation + reading normalization Reading of the conversion column: There are 4 patterns of “apple” and “eva”. The following processing is repeated for each.

  First, the processing starts from the notation of the analysis result column: “Apple”, “EVA”.

  (Step s23-1) Since the notation is not in an inclusive relationship, the process proceeds to the next iteration. Subsequently, the processing of “Apple”, “EVA”, “Apulcia”, “Eva”, “Apple”, “Eva” is performed, but since all are not in an inclusion relationship, the processing in the omission determination unit 23 is terminated.

  Since the record ID 1 in the synonym pair list 8 has never been determined to be synonymous during the process of the synonym determination unit 2, the record ID 1 is not recognized as a synonym pair and the process proceeds to the next record, that is, ID2.

  Here, the record ID 2 (“Apple Inc.”, “Hakkeijima Sea Paradise”) and the record ID 3 (“Apple Inc.”, “Shoko”) are not synonymous with the record ID 1, so in the list 8 of FIG. Of the record ID7, ID22, ID45, ID63, ID78, ID90, and ID121 that are finally determined to be synonymous, the synonymity determination unit 2 narrows down to representative patterns ID7, ID22, ID45, ID63, and ID121. Processing will be described.

★ Record ID 7 ("Apple company", "Apple")
[Notation similarity determination unit 21]
(Step s21-1) When the analysis processing result table 7 is referred to from the candidate ID1 and ID8 of the synonym candidate pair list 8, the notation of the notation normalization column is “Apple” and “Apple”, which are exactly the same. Therefore, the process proceeds to step s21-2.

  (Step s21-2) It is determined to be synonymous, and the process in the synonymity determining unit 2 is terminated.

★ Record ID 22 ("EVA", "EVA")
[Notation similarity determination unit 21]
(Step s21-1) When the analysis processing result table 7 is referred to from the candidate ID2 and ID9 of the synonym candidate pair list 8, the notation similarity determination unit 21 is different because the notation of the notation normalization column differs between “EVA” and “EVA”. This process is terminated, and the process proceeds to the reading similarity determination unit 22.

[Reading similarity determination unit 22]
Refer to the analysis processing result table 7 from the candidate ID 2 and ID 9 of the synonym candidate pair list 8. Reading of the analysis result column: “Eva”, “Eva”
・ Reading of normalization column: “Eva”, “Eva”
・ Notation + Reading normalization column reading: "Eva", "Eva"
And repeat the process.

  First, the analysis result column is read: “eva” and “eva” are processed.

  (Step s22-1) Since the reading is different, the process proceeds to Step s22-2.

  (Step s22-2) The number of syllables to be read is counted, and since both are two syllables, the process proceeds to Step s22-3.

(Step s22-3) The syllables having the same syllable position but different readings are “B” and “V”, and the distance is obtained as 0.87 from the syllable similarity table 4 (FIG. 13B: details will be described later). . Since this synonym candidate pair has only one pair with the same syllable position and different readings, the distance between the pairs is 0.87. Proceed to step s22-4.

  (Step s22-4) Since the total sum of the distances is 0.87 and is smaller than the preset threshold value 0.9, the process proceeds to Step s22-5.

  (Step s22-5) The synonymity determination unit 2 ends the process of determining synonymity.

★ Record ID 45 (“Shoko”, “Shokotan”)
[Notation similarity determination unit 21]
(Step s21-1) When referring to the analysis processing result table 7 from the candidate ID3 and ID10 of the synonym candidate pair list 8, the notation similarity determination unit 21 is different because the notation of the notation normalization column differs between “Shoko” and “Shoko”. This process is terminated, and the process proceeds to the reading similarity determination unit 22.

[Reading similarity determination unit 22]
Refer to the analysis processing result table 7 from the candidate ID 3 and ID 10 of the synonym candidate pair list 8. Reading of the analysis result column: “shoko”, “shokotan”
・ Reading of reading normalization column: “Shoko”, “Shokotan”
・ Notation + Reading normalization column readings: “Shoko”, “Shoko”
And repeat the process.

  First, the analysis result column is read: “shoko” and “shokotan” are processed.

  (Step s22-1) Since the reading is different, the process proceeds to Step s22-2.

  (Step s22-2) The number of syllables to be read is counted, and “Shoko” is different in 3 syllables, and “Shochotan” is different in 4 syllables.

  Next, the reading normalization column readings: “chocolate” and “chocotan” are processed.

  (Step s22-1) Since the reading is different, the process proceeds to Step s22-2.

  (Step s22-2) The number of reading syllables is counted. Since “chocolate” is different in two syllables and “chocotan” is different in four syllables, the process proceeds to the next repetition process.

  Next, the notation + reading normalization column readings: “chocolate” and “chocolate” are processed.

  (Step s22-1) Since readings are equal, the flow proceeds to Step s22-5.

  (Step s22-5) The synonymity determination unit 2 ends the process of determining synonymity.

★ Record ID 63 ("Hakkeijima Sea Paradise", "Sea Para")
[Notation similarity determination unit 21]
(Step s21-1) When the analysis processing result table 7 is referred to from the candidate ID 4 and ID 11 of the synonym candidate pair list 8, the notation of the notation normalization column is expressed as “Hakkeijima Sea Paradise” and “Sea Para”. finished processing the similarity determination unit 21, proceeds non to read similarity determination unit 22.

[Reading similarity determination unit 22]
Refer to the analysis processing result table 7 from the candidate ID 4 and ID 11 of the synonym candidate pair list 8-Reading of the analysis result column: "Hackage Masi Paradise", "Sea Para"
-Reading normalization column readings: "Hakeijima paradise", "Shipara"
・ Notation + Reading normalization column readings: “Hakeijimashi Paradise”, “Shipara”
And repeat the process.

  As a result of repeating the processing in the same manner as described above for each of the three patterns, the number of syllables differs between the pairs and is not determined to be synonymous, so the processing proceeds to the omission determination unit 23.

[Omission determination unit 23]
Refer to the analysis processing result table 7 from the candidate ID 3 and ID 10 of the synonym candidate pair list 8. The notations in the analysis result column: “Hakkeijima Sea Paradise”, “Sea Para”
・ Notation of notation normalization column: "Hakkeijima Sea Paradise", "Sea Para"
-Readings of columns after reading normalization: "Hakeijimashi Paradise", "Shipara"
・ Notation + Reading normalization column readings: "Hakeijima paradise", "Shipara"
Ask for.

  The following processing is repeated for each of the four patterns.

  First, the notation of analysis result columns: “Hakkeijima Sea Paradise” and “Sea Para” are processed.

  (Step s23-1) Since the notation is in an inclusive relationship, the process proceeds to Step s23-2.

  (Step s23-2) When alignment is performed by the DP matching method, the result is as shown in the upper left of FIG. Proceed to step s23-3.

  (Step s23-3) Feature extraction is performed as shown in step s23-3 in FIG. The expression before deletion is “Hakkeijima Sea Paradise”, the expression after deletion is “Seapara”, the remaining morpheme is “Sea”, the deleted morpheme is “Hakkeijima”, the deleted characters are “Da”, “ “I” and “su” and the remaining characters are “pa” and “la”. With respect to the six, the features such as the number of morphemes, the number of characters, the part of speech, etc. are extracted as shown on the right side of FIG.

(Step s23-4) As a result of determining whether the synonym candidate pair is in an abbreviation relationship using the approximate determination model 6 that is a model of the classifier, the process proceeds to Step s23-5 because it is synonymous.

  (Step s <b> 23-5) The synonym determination unit 2 terminates the process of determining synonymity.

★ Record ID 121 ("Miki Ando", "Mikiti")
[Notation similarity determination unit 21]
(Step s21-1) When referring to the analysis processing result table 7 from the candidate ID 7 and ID 14 of the synonym candidate pair list 8, the notation of the notation normalization column is “Miki Ando” and “Miki”, so the notations are different. The process of the similarity determination unit 21 ends, and the process proceeds to the reading similarity determination unit 22.

[Reading similarity determination unit 22]
Refer to the analysis processing result table 7 from the candidate ID 7 and ID 14 of the synonym candidate pair list 8. Reading of the analysis result column: “Ando Miki”, “Mikiti”
-Reading normalization column readings: "Andumiki", "Mikiti"
・ Notation + Reading normalization column reading: “Ando Miki”, “Miki”
And repeat the process.

  As a result of repeating the processing in the same manner as described above for each of the three patterns, the number of syllables differs between the pairs and is not determined to be synonymous, so the processing proceeds to the omission determination unit 23.

[Omission determination unit 23]
Refer to the analysis processing result table 7 from the candidate IDs 7 and 14 of the synonym candidate pair list 8. The notations in the analysis result column: “Miki Ando”, “Mikiti”
・ Notation of notation normalization columns: “Miki Ando”, “Miki”
-Readings of columns after reading normalization: "Andumiki", "Mikity"
・ Notation + Reading normalization column readings: "Ando Miki", "Miki"
Ask for.

  The following processing is repeated for each of the four patterns.

  First, the notation of the analysis result column: “Miki Ando” and “Mikiti” are processed.

  (Step s23-1) Since the notation is not in an inclusive relationship, the process proceeds to the next iterative process.

  The following “Miki Ando”, “Miki”, “Ando Miki”, and “Mikiti” are sequentially processed, but since they are not in an inclusive relationship, they are not determined to be synonymous. The final “Andumiki” and “Miki” processes are as follows.

  (Step s23-1) Since reading is in an inclusive relationship, the process proceeds to Step s23-2.

  Thereafter, (step s23-2) to (step s23-5) are performed in the same manner as in the case of the record ID 63 (“Hakkeijima Sea Paradise”, “Sea Para”).

After finishing the synonym candidate all synonymous determination of synonym candidate pair list in 8 as described above, records determined synonymous ID7, ID22, ID45, ID63, ID78, ID90, ID121 as synonyms pairs Output.

[II] Detailed application example of reverse conversion rule 3 and syllable normalization rule 4, example of creation of syllable similarity table 5 and omission determination model 6 With respect to reverse conversion rule 3, the prefixes shown in FIG. , Deletion of suffix, deletion of reading kana, deletion of repeated expression, and deletion of ellipsis.

  When deleting a prefix, for example, when applied to a target “notation: o / kichi, reading: o / kichi”, the prefix “o” is deleted and becomes “notation: kichi, reading: kichi”.

  In the deletion of the suffix, for example, when applied to a target “notation: Apple / Company, reading: Apple / Sha”, the suffix “Company” is deleted to become “notation: Apple, reading: Apple”.

  In the case of deletion of a reading kana, for example, when applied to a target of “notation: cheap / (/ easy /) / Megumi, reading: Yasu / Yasu / Megumi”, it becomes “notation: Yasu / Megumi, reading: Yasu / Megumi”.

  When deleting repeated expressions, for example, when applied to an object “notation: Kyung Kyung, reading: Kyung Kyung”, it becomes “notation: Kyung, reading: Kyung”.

  When the ellipsis is deleted, it is applied to the object "notation: Harry Potter 3 // Fire / No /-, reading: Harry Potter Sun // Hoono / No", and "Notation: Harry Potter 3 // Fire / : Harry Potter Sun // Hoono / No ”.

  Regarding the syllable normalization rule 4, four examples of “Yuko”, “Wimbledon”, “Jerusalem”, and “Swing” shown in FIG. 12 will be described in this embodiment.

★ "Yuko"
Since the same vowel continues in “Yu” and “U”, Rule 1 is applied, and the continuous vowel part becomes longer and becomes “Yuko”. Next, rule 5 is applied to the part that has become a long sound according to rule 1, resulting in “Yuko”.

★ "Wimbledon"
Since the vowel “U” and the roaring “i” of another type of vowel are continuous, Rule 2 is applied, resulting in “Wimbledon”.

★ "Jerusalem"
Since the vowel “I” and the roaring “e” of another kind of vowel are continuous, Rule 2 is applied, resulting in “Jerusalem”. Furthermore, since the vowels that meet rule 2 and the stuttering of the vowels match the conditions of rule 3, the vowel “i” is deleted and becomes “Jerusalem”.

★ "Swing"
Since the vowel “U” and the roar “i” of another kind of vowel are continuous, Rule 2 is applied, resulting in “Swing”. Further, as a result of applying the rule 2, the vowels are continuous with “Uui”, and the condition of the rule 4 is satisfied. Therefore, one continuous vowel “U” of the same type is deleted and becomes “swing”.

  A method of creating the syllable similarity table 5 will be described with an example using “artist” and its different notation shown in FIG. The morphological analysis dictionary supports different notations and notation fluctuations, and uses a dictionary having information on the reading and standard notation. First, the words “Artist”, “Artist”, and “Artist” with the same standard notation and different pronunciation are collected from the morphological analysis dictionary. Next, the syllable normalization rule 4 is applied to each expression, and alignment is performed with the syllable number matching. As a result, three pairs “te and tee”, “chi and ti”, “te and ti”, and “te and ti” having the same syllable position and different readings are obtained as “Atist”, “Attest”, and “Atist”. Similarly, pairs of different notation collection, normalization, and reading are collected from the morphological analysis dictionary using the standard notation as a key. When all the dictionaries have been processed, the number of pairs with the same syllable position and different readings and the number of syllables that appear are counted for each type. Then, a table is created by calculating the distance using the formula of FIG.

As a method for creating the omission determination model 6, an example in which SVM is used as an identification function will be described. If possible, it is desirable that the text data used for learning is acquired from the same domain as the input text used in actual operation, and analysis processing is performed with the same analyzer as used in actual operation. After creating a synonym weather Hope A in the same manner as the embodiment 1 with respect to the text for learning, take out only what notation is in inclusion relation. Each entry is correctly answered whether it is synonymous or not. Then, a feature similar to that of the omission determination unit 23 is extracted, and an omission determination model is created by learning the parameters of the discrimination function. Further, synonyms climate Hope A created in synonym candidate pair generation unit 1 at this time, read column or representation normalization column, normalized column reading, if trained with data notation + readings normalized column, It is possible to create an omission judgment model corresponding to each column.

  Note that the description of the inverse conversion rule storage unit, syllable normalization rule storage unit, syllable similarity table storage unit, omission determination model storage unit, analysis processing result table storage unit, synonym candidate pair list storage unit in the embodiment This is an expression based on the functional difference of what kind of data is stored, and does not mean that an individual storage unit (storage device) is necessary in terms of hardware. In addition, the present invention can be applied to a program for realizing the functions shown in the configuration diagrams of FIGS. 1, 2, and 6 or the flowcharts of FIGS. 4, 7 to 10 via a medium or a communication line in a known computer. It can also be realized by installing a program having the indicated procedure.

The schematic block diagram which shows an example of embodiment of the synonym determination apparatus of this invention Block diagram showing details of synonym candidate pair generator Explanatory drawing which shows an example of an analysis process result table Process flow in normalization processing unit Explanatory drawing which shows an example of a synonym candidate pair list Block diagram showing details of synonymity determination unit Flow chart of processing in synonymity determination unit Flow chart of processing in the notation similarity determination unit Flow chart of processing in the reading similarity determination unit Flow chart of processing in the omission determination unit Explanatory drawing which shows an example of a reverse conversion rule Explanatory drawing which shows an example of a syllable normalization rule Explanatory drawing which shows the preparation procedure of the syllable similarity table and an example thereof Explanatory drawing showing an example of input text Explanatory drawing which shows an example of the analysis process result table at the time of an analysis process end Explanatory drawing which shows an example of the feature extraction process in an omission determination part

  1: synonym candidate pair generation unit, 2: synonymity determination unit, 3: inverse transformation rule storage unit, 4: syllable normalization rule storage unit, 5: syllable similarity table storage unit, 6: omission determination model storage unit, 7: analysis processing result table storage unit, 8: synonym candidate pair list storage unit, 11: analysis processing unit, 12: normalization processing unit, 13: pair generation unit, 21: notation similarity determination unit, 22: reading similarity determination Part, 23: omission determination part.

Claims (4)

It extracts synonyms candidate expression is a string representation of a synonym candidate generate synonyms candidate pair from the text, determining synonymy whether synonyms candidates expressed together in the synonym candidate pair synonymous A determination device,
It includes at least rules for deleting affix forms that are commonly inserted when using nouns, and rules for deleting repeated expressions used when creating nicknames. Each of these rules further deletes a string from the notation. A reverse conversion rule storage unit that stores a reverse conversion rule that includes a notation rule for reading and a reading rule for correcting reading in conjunction with deletion of the character string ;
Rule 1 for changing to a vowel and its extended sound when the same vowel continues, Rule 2 for changing the vowel to a normal vowel when the vowel and another type of vowel are continuous, and vowels that match rule 2 Apply rule 3 and rule 2 to delete vowel “I” when “i”, stuttering is “e”, and the vowel of the next two syllables changes from “d” higher to lower than “d” As a result, when two vowels of the same type become continuous + different types of vowels, it includes at least rule 4 for deleting one of the same vowels of the same type and rule 5 for deleting long vowels and prompt sounds, which are applied in the order of the rule numbers. a syllable normalization rule storage unit for storing syllable normalization rules that,
A syllable similarity table storage unit for storing a syllable similarity table with “syllabic pairs with different notations but similar readings” as keys, and “distance” as a value ;
An abbreviated determination model storage unit that stores an abbreviated determination model composed of a model for determining whether or not two words are preliminarily generated by machine learning .
At least morphological analysis processing is performed on the input text, and synonym candidate expressions are extracted from the text based on the analysis results, and corresponding analysis results are assigned, and the notation and reading in the analysis results of the synonym candidate expressions are reversed. Applying the reverse conversion rule stored in the conversion rule storage unit, and further adding the syllable normalization rule stored in the syllable normalization rule storage unit to the reading in the analysis result of the synonym candidate expression and the reading after applying the reverse conversion rule after applying, the synonym candidate pair generating means for generating a synonym candidate pair consisting of a pair of synonyms candidate expressed in combination with each other synonyms candidate expressions,
If the notation of each synonym candidate expression in the synonym candidate pair is exactly the same, it is determined as synonymous, or if the reading of each synonym candidate expression in the synonym candidate pair is exactly the same, it is determined as synonymous, Alternatively, the syllable similarity table in which the sum of the distances between syllables having the same number of syllables, the same syllable position, and different readings is stored in the syllable similarity table storage unit in the reading of each synonym candidate expression in the synonym candidate pair If the sum of the distances is smaller than a preset threshold, it is determined to be synonymous, or the omission determination model storage when the notation or reading of each synonym candidate expression in the synonym candidate pair is in an inclusive relationship The abbreviation determination model stored in the section is used to determine whether or not there is an abbreviation relationship. If there is an abbreviation relationship, it is determined to be synonymous, and the synonym candidate pair determined to be synonymous is output as a synonym pair. Synonymity judgment means , Synonymous determination apparatus characterized by comprising a. It extracts synonyms candidate expression is a string representation of a synonym candidate generate synonyms candidate pair from the text, determining synonymy whether synonyms candidates expressed together in the synonym candidate pair synonymous A determination method comprising:
It includes at least rules for deleting affix forms that are commonly inserted when using nouns, and rules for deleting repeated expressions used when creating nicknames. Each of these rules further deletes a string from the notation. A reverse conversion rule storage unit that stores a reverse conversion rule that includes a notation rule for reading and a reading rule for correcting reading in conjunction with deletion of the character string ;
Rule 1 for changing to a vowel and its extended sound when the same vowel continues, Rule 2 for changing the vowel to a normal vowel when the vowel and another type of vowel are continuous, and vowels that match rule 2 Apply rule 3 and rule 2 to delete vowel “I” when “i”, stuttering is “e”, and the vowel of the next two syllables changes from “d” higher to lower than “d” As a result, when two vowels of the same type become continuous + different types of vowels, it includes at least rule 4 for deleting one of the same vowels of the same type and rule 5 for deleting long vowels and prompt sounds, which are applied in the order of the rule numbers. a syllable normalization rule storage unit for storing syllable normalization rules that,
A syllable similarity table storage unit for storing a syllable similarity table with “syllabic pairs with different notations but similar readings” as keys, and “distance” as a value ;
Using an abbreviated determination model storage unit that stores an abbreviated determination model consisting of a model for determining whether or not two words are generated in advance by machine learning and are in an abbreviation relationship ;
The synonym candidate pair generation means performs at least morphological analysis on the input text, extracts a synonym candidate expression from the text based on the analysis result, and gives a corresponding analysis result to analyze the synonym candidate expression Apply the reverse conversion rule stored in the reverse conversion rule storage unit to the notation and reading in the result, and further to the syllable normalization rule storage unit in the reading in the analysis result of the synonym candidate expression and after the reverse conversion rule application After applying the stored syllable normalization rules, combining the synonym candidate expressions together to generate a synonym candidate pair consisting of a pair of synonym candidate expressions;
The synonym determining means determines that each synonym candidate expression in the synonym candidate pair is exactly the same, or determines that each synonym candidate expression in the synonym candidate pair is exactly the same. If the synonym candidate pairs are read as synonyms, the sum of distances between syllables with the same number of syllables, the same syllable position, and different readings is stored in the syllable similarity table storage unit. Is determined using the syllable similarity table, and is determined to be synonymous if the sum of the distances is smaller than a preset threshold, or the notation or reading of each synonym candidate expression in the synonym candidate pair is in an inclusive relationship Sometimes it is determined whether or not there is an abbreviation relationship using the abbreviation determination model stored in the abbreviation determination model storage unit, and if there is an abbreviation relationship, it is determined to be synonymous, and the synonym candidate pair determined to be synonymous As a synonym pair Synonymous determination method characterized by comprising the that step. The program for functioning a computer as each means of the synonym determination apparatus of Claim 1 . A computer-readable recording medium on which the program according to claim 3 is recorded.

Images