JP2000339310A - Method and device for classifying document and recording medium with program recorded thereon - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a document classifying method and a document classifying device which attach a sentence structure that correctly accompanies a semantic attribute and can perform classification including the intention of a document implementor even to a sentence that does not undergo proofreading and a sentence whose quality is low. SOLUTION: When an inputting means 11 inputs a document, a morpheme analyzing means 12 divides it into morphenes. A corpus learning means 13 learns the characteristic of a semantic attribute on the basis of the frequency of morphological information from a context in which the semantic attribute appears from a stored document which is stored in a corpus 18 and to which the morphological information and the semantic attribute are preliminarily attached. A semantics attaching means 14 attaches the semantic attribute having the characteristic on the basis of the frequency of the most analogous morphological information. A similarity calculating means 15 obtains similarity obtained by considering the semantic attribute of stored document stored in the corpus 18 and the inputted document. A classifying means 16 classifies the inputted document into categories existing in the stored document with the high similarity in the corpus 18.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document classification method and a document classification device, and more particularly to software for classifying and organizing an electronic document.

[0002]

2. Description of the Related Art In recent years, computer systems utilizing natural language processing techniques, such as a Japanese word processor and a machine translation system, have advanced, and demands for improvements in natural language processing techniques have been increasing.

[0003] In the natural language processing technology, a conventional method of obtaining a sentence structure is, for example, 14 of natural language processing (Iwanami Koza Software Science 15, edited by Makoto Nagao et al., Published by Iwanami Shoten).
The syntactic analysis and the semantic analysis were performed as described on pages 0 to 229. Further, the conventional classification method uses the frequency of appearance of nouns.

[0004]

In the conventional methods of syntactic analysis and semantic analysis, it often takes a long time or erroneously analyzes long sentences or sentences having a complicated structure. In particular, there is a problem that the accuracy of analysis is low for documents that have not been proofread, such as e-mails and homepage documents that have been increasing in recent years. Further, the classification method using the frequency of occurrence of nouns has a problem that classification cannot be performed in consideration of the detailed intention of the document creator.

SUMMARY OF THE INVENTION An object of the present invention is to provide a document structure with a correct semantic attribute to uncorrected text and low-quality text, and to provide a document creator's intention. It is an object of the present invention to provide a document classifying method and a document classifying device which can perform classification including a character string.

[0006]

SUMMARY OF THE INVENTION A document classification method according to the present invention is a method of acquiring a sentence structure with a semantic attribute from an electronic document and classifying the document. Analyzing the document and dividing it into morphemes with morpheme information including basic forms, parts of speech, inflected forms;
A morpheme appearing in the vicinity of a morpheme to which a semantic attribute tag is added from a corpus (a large collection of language data corpus) in which a plurality of documents composed of morphemes and to which a semantic attribute tag is added are stored in advance. Acquiring the frequency of the morpheme information of the input document, and learning the feature of the semantic attribute from the obtained frequency, the feature of the semantic attribute obtained by the learning result, and the position near the morpheme constituting the input document. Comparing the feature of the morpheme with the frequency feature of the morpheme to be performed, and assigning the semantic attribute having the most similar feature as the semantic attribute of the morpheme constituting the input document; The input document and the stored document stored in the corpus are compared with respect to the frequency of the morpheme arrangement obtained by adding the semantic attribute to the morpheme information. Calculating the similarity of the documents based on the frequency of the arrangement; and, when the similarity between the input document and the storage document to be compared with the corpus exceeds a threshold, the input document is classified into the category of the storage document as the comparison target. Classifying.

[0007] A step of acquiring, from the corpus, the frequency of morphological information of morphemes appearing in the vicinity of the morpheme to which the tag of the semantic attribute is attached, and learning the feature of the semantic attribute from the acquired frequency exists in the corpus. From the frequencies of the n morpheme information before and after the morpheme to which all the semantic attributes are assigned, the weight for the semantic attribute is calculated for all the morpheme information present in the corpus, and the weight is calculated from the morpheme information and the calculated weight. The feature vector may be a step of learning a feature in which the semantic attribute appears by obtaining a feature vector before and after the semantic attribute.

[0008] The feature of the semantic attribute obtained as a result of learning is compared with the feature of the frequency of the morpheme information of the morpheme located near the morpheme constituting the input document, and the feature having the most similar feature is obtained. The step of assigning the semantic attributes as the semantic attributes of the morphemes constituting the input document includes: a feature vector representing a feature before and after each semantic attribute created from the corpus by a predetermined method; By comparing the morpheme information of each morpheme with the feature vectors representing the preceding and following features created by a predetermined method, and calculating the distance between the feature vectors for the front and the back, the feature having the closest distance is obtained. The step of selecting a semantic attribute of the vector may be performed.

In the step of calculating the degree of similarity of a morpheme string obtained by adding a semantic attribute to morphological information, when comparing the input document with a stored document stored in the corpus, the morpheme to which the semantic attribute is added has a meaning. For the morphemes to which no semantic attribute is assigned using attributes, a sequence of morphemes is created using the basic form, and a feature vector is created for each document based on the frequency of the created morphemes, and each feature vector Comparing the documents by calculating the distance of the documents.

A document classification device of the present invention is a device for acquiring a sentence structure with a semantic attribute from an electronic document and classifying the document. The document classification device inputs a document, and analyzes the input document to form a basic form. Morphological analysis means for dividing into morphemes having morpheme information including word, part of speech, and inflected forms, a corpus in which a plurality of documents composed of morphemes and attached with semantic attribute tags are stored in advance, and a semantic attribute is obtained from the corpus. Corpus learning means for acquiring the frequency of morphological information of morphemes appearing in the vicinity of the morpheme to which the tag is attached, and learning the feature in which the semantic attribute appears from the acquired frequency, and the semantic attribute acquired by the learning result Is compared with a feature composed of the frequency of morpheme information of morphemes located near the morphemes constituting the input document, and the meaning having the most similar feature is compared. Means for assigning gender as a semantic attribute of a morpheme constituting the input document, and a morpheme obtained by adding a semantic attribute to morpheme information of an input document having a semantic attribute added to a morpheme and a stored document stored in a corpus. Means for calculating the similarity of a document based on the frequency of the morpheme arrangement, and comparing the similarity between the input document and the storage document to be compared with the corpus when a threshold exceeds the threshold. A classification unit for classifying an input document into a category of the stored document, an output unit for outputting a classification result to the outside, and a control unit for controlling processing of each unit.

[0011] Further, a recording medium is provided, and the operation of the control unit is as follows.
The control may be performed by an input document classification program recorded on a recording medium.

The recording medium on which the program of the present invention is recorded is characterized in that the semantic attribute appears based on the frequency characteristic of the morpheme information of the morpheme appearing near the morpheme to which the semantic attribute tag of the stored document stored in the corpus is attached. Learning
A control program for acquiring a semantic attribute of a morpheme obtained for a newly input document and further classifying the input document into a group of documents stored in the corpus which are similar to each other is recorded.

The features of the processing of the present invention will be described by taking as an example the case where received e-mails are classified. When a new received mail is input by the input means, the received mail is divided into morphemes by the morphological analysis means. The corpus learning means learns all the features of morphological information for each semantic attribute from documents stored in the corpus and to which morphological information and semantic attributes are added in advance. The meaning assigning means assigns a semantic attribute to the morpheme according to the morpheme information of the divided morpheme. The similarity calculating means obtains the similarity between the stored document stored in the corpus and the input document. The classification unit classifies the input document into a certain category of stored documents having a high degree of similarity in the corpus.

[0014]

Next, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a schematic block diagram of the document classification device according to the first embodiment of the present invention.
FIG. 2 is a flowchart illustrating a document classification method according to the first embodiment of this invention. You. The document classification device 10 includes an input unit 11 for inputting a document, a morphological analysis unit 12, a corpus learning unit 13, a meaning assignment unit 14, and a similarity calculation unit 1.
5, a classification unit 16, an output unit 17 for outputting a classification result of a document, a corpus 18, and a control unit 19 for controlling processing of each unit.

A corpus 18 is a group of linguistic data collected in large amounts (see Natural Language Processing, Iwanami Koza Software Science 15, Makoto Nagao et al., Published by Iwanami Shoten, page 253). It is assumed that a large number of documents composed of morphemes to which are added are stored in a predetermined format in advance.

FIG. 3 shows an example of a document in the corpus 18. FIG.
Is an example of a corpus-tagged stored document used in the document classification method according to the first embodiment. Two documents, "I cannot communicate with my mobile phone" and "No effective solution is written" It is described as an example.

The morphological analysis unit 12 divides the sentence of the input document input by the input unit 11 into morphemes having morphological information including basic forms, parts of speech, and inflected forms. Corpus learning means 14
Is a morpheme of a morpheme appearing in the vicinity of the morpheme to which the semantic attribute tag is attached, which is extracted from the corpus 18 in a predetermined form, and which is composed of the morphemes stored in the corpus 18 and to which the semantic attribute tag is attached. The frequency of the information is acquired, and the feature in which the semantic attribute appears is learned from the acquired frequency. The meaning providing means 13 compares the feature of the semantic attribute obtained as a result of learning with the feature of the frequency of the morpheme information of the morpheme located near the morpheme constituting the input document, and finds the most similar feature. Is assigned as a semantic attribute of a morpheme constituting the input document.

The similarity calculating means 15 compares the input document having the morpheme with the semantic attribute and the stored document stored in the corpus with respect to the frequency of the morphemes obtained by adding the semantic attribute to the morpheme information. The similarity of the document is calculated based on the arrangement frequency. When the similarity between the input document and the storage document to be compared with the corpus 18 exceeds the threshold, the classification unit 16 classifies the input document into the category of the storage document.

The operation of the document classification method according to the first embodiment of the present invention will be described with reference to FIG. When the document classification process is started (S11), a document is input by the input unit 11 (S12), and the input document is divided by the morphological analysis unit 12 into morphemes having morpheme information including basic forms, parts of speech, and inflected forms (S13). ). One of the morphemes to which the semantic attribute is assigned is selected by the meaning assigning unit 14 (S14), and the feature is calculated from the frequency of the morpheme information of the n morphemes before and after the morpheme (S15). The feature of the learned semantic attribute is compared (S16). If there is another candidate semantic attribute (S17N), the next candidate semantic attribute is selected (S18), and the process returns to step S16. Repeat the process. If no other candidate semantic attribute exists (S17N), a semantic attribute having the most similar feature is assigned (S19).

If the assignment of the entire semantic attribute of the morpheme of the input document is not completed (S20N), the morpheme to which the next semantic attribute of the input document is assigned is designated (S21), and step S21 is performed.
Returning to step 14, the process is repeated. If the assignment of the entire semantic attributes of the morphemes of the input document is completed (S20Y), one of the stored documents is selected by the similarity calculating means 15 (S2).
2) comparing the feature consisting of the frequency of morpheme arrangement including the semantic attribute of the input document with the feature consisting of the frequency of morpheme arrangement including the semantic attribute of the selected stored document (S23), and determining the similarity to a predetermined threshold value; If not (S24N), the next storage document is designated (S25), and the process returns to step S22 to repeat the processing. If the similarity is equal to or more than a predetermined threshold (S24
Y) Classifying means 17 classifies the input document into stored document categories whose similarity is equal to or greater than a predetermined threshold (S26),
The document classification process ends (S27). Next, step S1
The procedure for learning the semantic attributes described in 4 to S21 will be described in detail with an example. First, Corpus 1
The morpheme information of the sentence of the stored document stored in 8 is based on the part of speech, basic type, and inflected form of a word obtained by morphological analysis as shown in FIG. The vector is constructed based on this information.

For example, information such as Table 1 is given to a sentence “contents spoken in a lecture” of a document stored in the corpus 18.

[0022]

[Table 1] Here, when the document "You have confirmed the operation" is input and morphological analysis is performed, information as shown in Table 2 is obtained.

[0023]

[Table 2] In Table 1, the basic form “re” of “re” has semantic attributes of “respect”, “passive”, “possible”, and “self-motivated”, but here is “respect”. The purpose of this step is to learn the semantic attributes of the basic form "re" of Table 2 "re".

The documents stored in the corpus are referred to as n-grams (natural language processing, Iwanami Koza Software Science 15) from morphological information (basic form, part of speech, inflected form) of n morphemes before and after each semantic attribute. , Edited by Makoto Nagao et al., Published by Iwanami Shoten, p. 15), weighted the morpheme information of each co-occurred word from the entire corpus by a predetermined method, and determined the characteristics of each semantic attribute from the weight of all morpheme information. Vector is composed.

For an input document, a feature vector of morphological information of n pieces before and after "re" is created. In this case, the weights of the co-occurring morpheme information are all 1, and the other weights are 0.

The inner product of the feature vector of each semantic attribute of "re" that can be taken in the corpus and the feature vector of "re" of the input document is calculated, and the meaning (the angle of the vector) having the smallest distance is calculated. The attribute is obtained and adopted as the semantic attribute of the morpheme of the sentence of the input document. As a result, the semantic attribute of the morpheme used in the similar context is selected as the semantic attribute of the morpheme of the input document.

Next, the method of calculating the similarity between the document stored in the corpus and the input document, which is described in steps S22 to S26, will be described in detail with reference to an example. Here, a morpheme sequence is created using the morpheme sequence of the document, a morpheme with a semantic attribute is a semantic attribute, and a morpheme without a semantic attribute is created using the basic morpheme type. Create a vector.

A document composed of morpheme information shown in Table 3 and a morpheme having a semantic attribute will be described as an example.

[0029]

[Table 3] In this document, when n-gram is created with n = 3, (there, "restricted", operation, confirmation) ("restricted, operation, confirmation") (operation, confirmation, performed) (confirmation, performed, "respect") (performed) , "Respect", "assertion") ("respect", "assertion", "call") ("assertion", "call", "confirmation"), and these n-grams for the documents stored in the corpus. The feature vector of the document using the frequency of is created.

Similarly, for the input document having acquired the semantic attribute, a feature vector of the document is created using the frequency of n-gram, and a distance is calculated using an inner product with the feature vector of the stored document stored in the corpus. The input document is classified into the category of the stored document or the stored document group having the smallest distance (angle of the vector).

FIG. 4 shows an example of a sequence of morphemes in the corpus document example shown in FIG. 3 and a sequence of morphemes in a sentence of a newly input document. The sequence of morphemes uses a semantic attribute when a semantic attribute exists, and uses a basic morpheme when no semantic attribute exists.

Next, a document classification method and a document classification device according to a second embodiment of the present invention will be described with reference to the drawings. FIG. 5 is a schematic block diagram of a document classification device according to the second embodiment of this invention.

FIG. 5 shows a document classifying apparatus 100 according to the present invention.
It is shown as a computer that constitutes the device,
The computer includes an input unit 110 such as a modem, a keyboard, and a pointing device, an output unit 120 such as a modem, a printer, and a display, a data processing device 130, a storage unit 140, and a recording medium 150. Recording medium 150
Records a document classification system control program of the present invention capable of controlling the operation of each unit.
M, a semiconductor memory or the like is used.

The configuration of the document classifying apparatus and the document classifying method are the first.
The description is omitted because it is the same as that of the embodiment.

The feature of the semantic attribute is learned from the feature of the frequency of the morpheme information of the morpheme appearing near the morpheme to which the tag of the semantic attribute of the stored document stored in the corpus is added, and a newly input document is learned. A control program for acquiring the semantic attributes of the morphemes obtained with respect to, and further classifying the input document into similar stored documents in the corpus, is read from the recording medium 150 into the data processing device 130 and read from the data processing device 130. The operation of 130 is controlled. The data processing device 130 executes the following processing under the control of the control program.

That is, a process of newly inputting a document, a process of analyzing the input document and dividing it into morphemes having morpheme information including basic forms, parts of speech, and inflected forms, and adding tags of semantic attributes from the corpus. A process of acquiring the frequency of morpheme information of morphemes appearing in the vicinity of a morpheme that is present, learning the feature of the semantic attribute from the acquired frequency, the feature of the semantic attribute acquired by the learning result, and the input document A process of comparing a feature consisting of the frequency of morpheme information of morphemes located in the vicinity of morphemes constituting the morpheme and assigning a semantic attribute having the most similar feature as a semantic attribute of the morpheme constituting the input document; The input document in which the semantic attribute is added to the morpheme and the stored document stored in the corpus are compared with respect to the frequency of the arrangement of the morphemes in which the semantic attribute is added to the morpheme information. A process for calculating the similarity of documents based on the frequency of arrangement, and when the similarity between the input document and the stored document compared with the corpus exceeds a threshold, the input document is classified into the category of the stored document to be compared. And processing to be performed.

[0037]

As described above, the present invention supplements the semantic attributes of morphemes from the corpus and calculates the similarity with the stored document stored in the corpus by the arrangement of morphemes including the semantic attributes. Thus, there is an effect that more detailed classification of documents can be performed in consideration of the above.

[Brief description of the drawings]

FIG. 1 is a schematic block configuration diagram of a document classification device according to a first embodiment of this invention.

FIG. 2 is a flowchart illustrating a document classification method according to the first embodiment of this invention.

FIG. 3 is an example of a corpus-tagged stored document used in the document classification method according to the first embodiment;

FIG. 4 is an example of an arrangement of morphemes to be targeted by the similarity calculation means.

FIG. 5 is a schematic block diagram of a document classification device according to a second embodiment of the present invention.

[Explanation of symbols]

 10, 100 Document classification device 11, 111 Input unit 12, 112 Morphological analysis unit 13, 113 Corpus learning unit 14, 114 Meaning assignment unit 15, 115 Similarity calculation unit 16, 116 Classification unit 17, 117 Output unit 18, 118 Corpus 19, 119 control unit 110 input unit 120 output unit 130 data processing device 140 storage unit 150 recording medium S11 to S17 step

Claims (7)

[Claims] 1. A method of acquiring a sentence structure with a semantic attribute from an electronic document and classifying the document, comprising the steps of: inputting a new document; analyzing the input document to obtain a basic form, a part of speech, and an inflected form Dividing the document into morphemes having morpheme information including a morpheme, and extracting a semantic from a corpus (a large collection of linguistic data group corpus) in which a plurality of documents that are composed of morphemes and to which semantic attribute tags are added are stored in advance. Acquiring the frequency of the morphological information of the morpheme appearing in the vicinity of the morpheme to which the attribute tag is attached, learning the feature in which the semantic attribute appears from the acquired frequency; Compare the feature that appears and the feature that consists of the frequency of the morpheme information of the morphemes located near the morphemes that make up the input document and have the most similar feature Adding a semantic attribute as a semantic attribute of a morpheme constituting the input document; and adding the semantic attribute to the morpheme information of the input document having the semantic attribute added to the morpheme and the stored document stored in the corpus. Calculating the similarity of the document based on the frequency of the morpheme arrangement, and the similarity between the input document and the stored document to be compared with the corpus exceeds a threshold value. Categorizing the input document into a category of the stored document as a comparison target when the input document is compared. 2. A step of acquiring, from the corpus, the frequency of morpheme information of morphemes appearing in the vicinity of a morpheme to which a tag of a semantic attribute is attached, and learning a feature of the semantic attribute from the acquired frequency. From the frequencies of the n morpheme information before and after the morpheme to which all the semantic attributes present in the corpus are assigned, for all the morpheme information present in the corpus, a weight for the semantic attribute is calculated, and the morpheme information and the calculation are calculated. 2. The document classification method according to claim 1, further comprising the step of learning a feature in which the semantic attribute appears by obtaining a feature vector before and after the semantic attribute constituted by the determined weights. 3. A feature of a semantic attribute obtained as a result of learning and a feature of the frequency of morpheme information of a morpheme located near a morpheme constituting the input document are compared, and are most similar. The step of assigning a semantic attribute having a characteristic as a semantic attribute of a morpheme constituting the input document includes: a feature vector representing a feature before and after each semantic attribute created from the corpus by a predetermined method; By comparing feature vectors representing front and rear features created by a predetermined method from morpheme information of n morphemes before and after the morpheme, and calculating the distance of each feature vector for the front and the back, 2. A method for selecting a semantic attribute of a feature vector whose distance is the closest to the above.
Document classification method described in. 4. A step of calculating the similarity of a document based on the frequency of morpheme arrangement in which morpheme information is added with a semantic attribute, comprises: comparing the input document with the stored document stored in a corpus; For morphemes with semantic attributes, semantic attributes are used. For morphemes without semantic attributes, a morpheme sequence is created using the basic form, and for each document based on the frequency of the created morpheme sequence. 2. The method according to claim 1, further comprising the step of creating feature vectors and comparing documents by calculating a distance between the feature vectors. 5. An apparatus for acquiring a sentence structure with a semantic attribute from an electronic document and classifying the document, comprising: a document input unit for inputting a document; and analyzing the input document to determine a basic form, a part of speech, and an inflected form. Morphological analysis means for dividing into morphemes having morpheme information including, a corpus (a large amount of collected language data group corpus) in which a plurality of documents composed of morphemes and tagged with semantic attributes are stored in advance. A corpus learning means for acquiring, from the corpus, the frequency of morpheme information of morphemes appearing in the vicinity of the morpheme to which the tag of the semantic attribute is attached, and learning the feature of the semantic attribute from the acquired frequency; The feature obtained by the semantic attribute obtained by the above is compared with the feature consisting of the frequency of the morpheme information of the morpheme located near the morpheme constituting the input document. Means for assigning a semantic attribute having similar characteristics as a semantic attribute of a morpheme constituting the input document; and the input document having a semantic attribute added to the morpheme and the storage stored in the corpus. A similarity calculating means for comparing a document with a morpheme information obtained by adding a semantic attribute to the morpheme information and calculating a similarity of the document based on the morpheme arrangement frequency; and comparing the input document with the corpus. A classifying unit that classifies the input document into a category of the stored document when the similarity with the target stored document exceeds a threshold value; an output unit that outputs a classification result to the outside; and controls processing of each unit. And a control unit that performs the operation. 6. The document classification device according to claim 5, further comprising a recording medium, wherein the operation of the control unit can be controlled by an input document classification program recorded on the recording medium. 7. A method for determining semantic attributes based on the frequency characteristics of morpheme information of morphemes appearing near morphemes to which semantic attribute tags of stored documents stored in a corpus (a large amount of collected language data corpus) are attached. A control program for learning the appearing features, acquiring semantic attributes of morphemes obtained for a newly input document, and further classifying the input document into similar stored documents in the corpus was recorded. A recording medium, a procedure of newly inputting a document, a procedure of analyzing the input document and dividing the document into morphemes, and a morpheme of a morpheme appearing near a morpheme to which a semantic attribute tag is added from the corpus. A procedure of acquiring the frequency of the information and learning a feature of the semantic attribute from the acquired frequency; a feature of the semantic attribute acquired by the learned result; A morpheme located near a morpheme constituting the input document is compared with a feature composed of frequency of morpheme information, and a semantic attribute having the most similar feature is assigned as a semantic attribute of the morpheme constituting the input document. And comparing the input document in which the morpheme is given a semantic attribute and the stored document stored in the corpus with respect to the frequency of the morpheme sequence obtained by adding the semantic attribute to the morpheme information. Calculating the similarity of the document based on the input document, and when the similarity between the input document and the stored document to be compared with the corpus exceeds a threshold, the category of the stored document to be compared with the input document. Steps to classify
A machine-readable recording medium on which a program for executing the program is recorded.