JP4618083B2 - Document processing apparatus and document processing method - Google Patents

Description

  The present invention relates to a document processing apparatus and document processing method for automatically extracting important parts from a document. In particular, the present invention relates to a document processing apparatus and a document processing method for automatically extracting an important part of a document using a hash function.

  In recent years, with the rapid increase in the population using the network and remarkable progress in communication technology, there are many opportunities to view documents distributed from other devices via the network using electronic devices such as mobile terminals. It has become. In such a situation, a user of a mobile terminal can use his / her own document (text) that has not been created for the mobile terminal in advance, such as a Web page or e-mail with a large amount of information. There are so many opportunities to feel like browsing on a mobile device.

  However, a portable terminal is required to be small and lightweight due to the nature of being always carried by a user. Due to these requirements, portable devices that are becoming smaller and lighter in recent years have certain restrictions on the amount of information stored and the amount of processing due to spec issues, so documents such as e-mail can be displayed at once. May not be possible.

  To solve this problem, the natural text processing technology is used to reduce the amount of document information by automatically generating a summary sentence from the input document, and then send the summary sentence to the mobile terminal for display. Research is being sought. This research aims to automatically generate document summaries by computer using semantic understanding technology and context understanding technology. However, this technology has not yet been established to a practical level and is currently difficult to put into practical use.

  Therefore, as a method that is easier to put into practical use, a technique for extracting an important sentence from a document by matching a keyword has been proposed (see, for example, Patent Document 1). In this technology, first, specific words and phrases that are statistically included in important or unnecessary sentences are registered in advance in the computer by hand, and then registered in each sentence of the text first. It is checked whether words and phrases are included. As a result, the importance of each sentence is determined based on the number of specific words and phrases included in each sentence, and one or more sentences determined to have a high importance are extracted as summary sentences.

JP-A-6-259423

  However, the above technique has the following three problems. First, in the above technique, it is necessary to register a specific word or phrase that is a clue to calculate the importance in advance to a level that can sufficiently withstand practical use. It is that it takes. In addition, in order to appropriately cope with buzzwords that change with the times, it is necessary to constantly update the registration information by continuously performing the above registration work.

  Second, in the above technique, it is necessary to compare each sentence with the registered word or phrase in order to determine whether or not the registered word or phrase is included in each sentence. The processing load is very high. Specifically, to determine whether or not a registered word or phrase is included in a part of each sentence, it is necessary to compare the registered word with the character string included in each sentence by shifting one character at a time. There is. In other words, in this determination process, comparison of character strings between each sentence and a registered word or phrase must be repeated as many times as indicated by the number of characters in each sentence. For this reason, the processing amount for matching the character string becomes very large.

  Third, it is difficult for the above technique to cope with multiple languages. In other words, it is necessary to select words and phrases to be registered in each language. For this reason, the developer or registrant must be familiar with the target language. In particular, when targeting non-major languages, it is difficult to secure a person who is familiar with such a language, and even if you have searched for such a person, it will cost a lot to hire that person. End up.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is a new and improved automatic extraction of important parts from a document at high speed by reducing the processing load. Another object of the present invention is to provide a document processing apparatus and a document processing method.

  In order to solve the above-described problem, according to an aspect of the present invention, a storage unit that associates and stores a calculated value calculated based on a predetermined rule and an appearance frequency that represents the frequency at which the calculated value is calculated; A dividing unit that divides a document into a plurality of character strings based on a predetermined condition; a calculation unit that obtains a calculation value unique to each character string from each of the divided character strings based on the predetermined rule; A search unit for detecting the appearance frequency corresponding to each calculated value by comparing each calculated value obtained by the calculating unit and the calculated value stored in the storage unit, and the detected An extraction unit that selects one or more calculated values based on each appearance frequency corresponding to each calculated value, and extracts a character string for the selected one or more calculated values as a summary of the document; The document processing device It is subjected.

  Conventionally, when extracting a summary from a document, matching processing is performed by shifting whether or not a pre-registered word is included in a character string included in the document, one character at a time. This comparison of character strings actually required comparison as many times as indicated by the number of characters.

  However, according to the present invention, one data (calculated value) unique to each sentence is calculated from the character string of each divided sentence, and the calculated one data (calculated value) is stored in the storage unit. The stored calculated value is matched. That is, in the present invention, it is only necessary to compare numerical values once instead of comparing character strings. For this reason, when the summary is extracted from the document, the matching process can be completed very quickly. In other words, it is possible to dramatically reduce the processing load compared to the conventional method in which matching processing is performed while shifting whether characters registered in advance are included in character strings included in each sentence one character at a time. .

  In the present invention, as a result of such a high-speed matching process, it is possible to extract an important sentence that is a summary of the document at a high speed based on the appearance frequencies stored corresponding to the matched calculated values. As a result, even information that could not be displayed on a device having relatively poor specifications such as a mobile phone due to a large amount of document information can be quickly summarized and displayed. For this reason, the user can quickly grasp the important part of the document by the summary sentence of the document that could not be seen until now.

  Also, according to this, each time a document is summarized, the calculated value and information regarding the frequency of appearance of the calculated value are accumulated automatically in the storage unit. For this reason, it is not necessary to pre-register the specific word or phrase that serves as a clue to determine whether the sentence is an important sentence or an unnecessary sentence in the storage unit to the extent that it can sufficiently withstand practical use. .

  Furthermore, according to this, the character string of each sentence is converted into simple data independent of the language of the document. Therefore, it is possible to summarize the document without depending on the language of the document. Therefore, when building or operating a system using this document processing device, developers and registrants do not need to be familiar with each language, and can handle unknown languages. it can.

  The calculation unit may obtain a hash value as the calculation value from each character string using a hash function as the predetermined rule.

  In addition, the extraction unit compares the appearance frequencies corresponding to the searched calculation values, respectively, so that one or more calculations are sequentially performed from the calculation values stored in association with the appearance frequency having a lower value. A value may be selected.

  If it is determined as a result of the comparison by the search unit that the calculation value obtained by the calculation unit is stored in the storage unit, the search unit is stored in association with the calculated value. If it is determined that the calculated value is not stored in the storage unit, a new appearance frequency having a given value is stored together with the calculated value. May be.

  The dividing unit can divide the document into a plurality of character strings constituting any of a clause or a sentence or a paragraph.

  The document processing apparatus may further include a normalization unit that adjusts a format of a character string included in either the document or each divided sentence.

  According to this, for example, the format of the character string is adjusted, such as unifying full-width characters and half-width characters. As a result, it is possible to eliminate a calculation error due to a difference in notation, and to calculate a more accurate calculated value from a character string whose format is arranged. As a result, it is possible to extract a summary more accurately from a document using an accurate calculated value.

  Furthermore, the document processing apparatus may include a text classification unit that determines attributes of the document. At this time, the storage unit accumulates a plurality of the calculated appearance frequencies of the calculated values for each document attribute, and the extraction unit stores the attribute of the document and the attribute of the document stored in the storage unit. Each of the correlation values determined from the correlation with each other is used to weight each of the appearance frequencies stored for each attribute of the document in the storage unit, and one or more calculations are performed based on each weighted appearance frequency. A value may be selected.

  At this time, the correlation value is set to have a larger value as the relation between the document attribute determined by the text classification unit and the document attribute stored in the storage unit is smaller. Also good.

  Further, the extraction unit weights the calculated values using a plurality of appearance frequencies stored for each attribute of the document with respect to the calculated values, and calculates the sum of the weighted appearance frequencies. The importance corresponding to the value may be calculated, and one or two or more corresponding calculated values may be selected in descending order of the calculated importance.

  According to this, the correlation value regarding the attribute of the document is determined in advance, and each appearance frequency is weighted using the correlation value. Here, the correlation value is set to a higher value as the correlation degree is lower. For example, the correlation value for a word or sentence that appears frequently only in a specific field is highly related to the specific field, and can be set small in advance. As a result, the sentence that appears frequently only in a specific field is weighted so as to be more important than the sentence that frequently appears in other fields. As a result, the importance m for each sentence is more appropriately calculated, and a more detailed summary can be extracted based on the calculated importance.

  Further, the calculation unit obtains a calculation value unique to a part or all of the text as a total calculation value, and the search unit searches whether or not the total calculation value is stored in the storage unit. If the total calculated values are not stored in the storage unit, the total calculated values are stored in association with the character string extracted by the extracting unit as a summary of the document. When stored in the storage unit, the division unit, the calculation unit, the search unit, and the extraction unit are stored in the storage unit in association with all the calculated values without causing each unit to execute the operation of each unit. A character string corresponding to the calculated value may be extracted as a summary of the document.

  According to this, even when an email having the same content is input a plurality of times due to an operation error or an erroneous transmission, the value of the appearance frequency is set to take an appropriate value according to the actual situation. As a result, it is possible to prevent the importance of each sentence from decreasing more than necessary.

  Also, according to this, when the same text as the previously entered text is input, for example, when the same content mail is input multiple times, the calculation value with high processing load is again calculated. There is no need to execute matching processing. Therefore, it is possible to quickly provide the summary text to the user using the important sentence extracted previously while reducing the processing load.

  In order to solve the above-described problem, according to another aspect of the present invention, a calculated value calculated based on a predetermined rule and an appearance frequency representing the calculated frequency of the calculated value are associated with each other in the storage unit. Storing the document, dividing the document into a plurality of character strings based on a predetermined condition, obtaining a calculated value unique to each character string from each of the divided character strings based on the predetermined rule, By comparing each calculated value with the calculated value stored in the storage unit, the appearance frequency corresponding to each calculated value is searched, and each occurrence corresponding to each searched calculation value is searched. There is provided a document processing method for selecting one or more calculated values based on the frequency and extracting a character string corresponding to the selected one or more calculated values as a summary of the document.

  According to this, one calculated value corresponding to each sentence and the calculated value stored in the storage unit are matched at high speed. As a result, the importance of each sentence can be determined at high speed, and the summary of the input document can be extracted at high speed based on the importance. As a result, documents can be quickly summarized and provided to the user.

  As described above, according to the present invention, it is possible to provide a new and improved document processing apparatus and document processing method capable of automatically extracting important parts from a document at high speed by reducing the processing load.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and the accompanying drawings, components having the same configuration and function are denoted by the same reference numerals, and redundant description is omitted. In the following embodiments, an e-mail is taken as an example of text (document), and a document processing apparatus and method for automatically generating a summary thereof will be described.

(First embodiment)
(Hardware configuration of document processing apparatus 100)
First, the hardware configuration of the document processing apparatus according to the first embodiment will be described with reference to FIG. The document processing apparatus 100 includes an HDD (Hard Disk Drive) 105, a ROM (Read Only Memory) 110, a RAM (Random Access Memory) 115, a processor 120, an interface 125, and a bus 130.

  The HDD 105 stores various data and programs including information necessary for extracting important sentences from text. The HDD 105 is an example of a storage device, and may be a storage device such as an optical disk or a magneto-optical disk.

  The ROM 110 stores a basic program for operating the processor 120, a program that starts when the processor 120 is abnormal, and the like. The RAM 115 temporarily stores data such as text input from the outside, divided sentences (to be described later), and classification codes. The processor 120 executes a program stored in the HDD 105, the ROM 110, or the like in order to generate a summary from the input text.

  The interface 125 is configured to input text from input devices such as a keyboard 200, an OCR (Optical Character Reader) 205, a network card 210, and a voice input device 215, for example. Further, the interface 125 outputs the summary sentence extracted from the text to an output device such as a CRT (Cathode Ray Tube) 300, a printer 305, a network card 310, and an audio output device 315, for example.

  The bus 130 is a path for exchanging information among the devices such as the HDD 105, the ROM 110, the RAM 115, the processor 120, and the interface 125.

(Functional configuration of document processing apparatus 100)
Next, the functional configuration of the document processing apparatus 100 will be described with reference to FIG. The document processing apparatus 100 includes functions indicated by blocks of an input unit 150, a division unit 155, a normalization unit 160, a sentence hash calculation unit 165, a storage unit 170, a sentence hash search unit 175, an extraction unit 180, and an output unit 185. have.

  The input unit 150 is, for example, text entered from the keyboard 200 shown in FIG. 1, text taken from the OCR 205 or the network card 210, text inputted from the voice input device 215, or transmitted from the outside. Input texts such as web pages and e-mails for digitization.

  The dividing unit 155 divides the input document into a plurality of character strings based on a predetermined condition. In the following description, the dividing unit 155 divides a target electronic mail into sentence units. For example, the dividing unit 155 performs document division based on the condition that a document is divided when a punctuation point (when dividing each sentence separated by punctuation points) or a line feed (when each itemized line is divided) appears in the document. Is split into multiple strings.

  The normalization unit 160 unifies the difference in notation format. As an example, there is a case where a single-byte character or a single-byte character is unified. It should be noted that the normalization unit 160 may unify the format for each sentence after division, or may unify the format for the document before division. The normalization unit 160 is not an essential function for the document processing apparatus 100 according to the present embodiment. However, it is possible to eliminate the calculation error of the hash value due to the difference in the notation of the character string, and to calculate a more accurate calculated value from the character string whose format is arranged. As a result, the summary can be extracted more accurately from the input document.

  The sentence hash calculation unit 165 (corresponding to the calculation unit) obtains a unique calculation value for each character string from each divided character string based on a predetermined rule. The sentence hash calculation unit 165 calculates a hash value for each sentence divided by the division unit 155 using, for example, a hash function. The hash function is a function for converting an enumeration of character strings such as documents and numbers into data of a fixed length (hash value).

  The storage unit 170 has a divided sentence hash table 170a. In the divided sentence hash table 170a, as shown in FIG. 3, a hash value 170a1 calculated from each sentence of the text input so far and an output limit number 170a2 indicating the appearance frequency of the hash value are accumulated. ing. The appearance frequency 170a2 is an example of a value representing the frequency with which the hash value is calculated by the sentence hash calculation unit 165, that is, the frequency with which the character string of each sentence appears, and is a value representing the frequency with which each character string appears. If it is, it may be a value other than the number of times each character string appears, for example, the probability that each character string appears.

  The sentence hash search unit 175 (corresponding to the search unit) performs matching processing as to whether or not the hash value of each sentence calculated this time by the sentence hash calculation unit 165 is stored in the divided sentence hash table 170a. When the hash value calculated this time matches the hash value 170a1 stored in the divided sentence hash table 170a, the sentence hash search unit 175 detects the appearance frequency 170a2 stored corresponding to the matched hash value. .

  Thus, when the hash value calculated | required by the sentence hash calculation part 165 is memorize | stored in the division | segmentation sentence hash table 170a, the sentence hash search part 175 is the value of the appearance frequency 170a2 memorize | stored in association with the hash value 170a1. For example, “1” is increased.

  On the other hand, when the obtained hash value is not stored in the divided sentence hash table 170a, the sentence hash search unit 175, together with the hash value obtained by the sentence hash calculation unit 165, determines a predetermined value (for example, The appearance frequency having “1” indicating that the number of appearances is 1) is stored in the divided sentence hash table 170a.

  The extraction unit 180 selects one or more appearance frequencies from the appearance frequency values corresponding to the hash values detected by the sentence hash search unit 175, and associates them with the selected appearance frequency. Then, one or more original sentences corresponding to each hash value stored are extracted as a summary. At this time, when the extraction unit 180 extracts an important sentence, a sentence corresponding to a hash value having a high appearance frequency among hash values corresponding to each sentence included in the input text has low importance. A sentence corresponding to a hash value having a small appearance frequency uses the rule that the importance is high. Specifically, the extraction unit 180 selects a hash value 170a1 having a relatively low appearance frequency 170a2 by comparing the appearance frequency 170a2 of the corresponding hash value 170a1 stored in the divided sentence hash table 170a. A sentence corresponding to the hash value is extracted as an important sentence.

  As a specific method for determining one or more important sentences, for example, the extraction unit 180 extracts, as important sentences, a sentence corresponding to the hash value having the lowest appearance frequency among the sentences included in the input text. The method of doing is mentioned. Further, the extraction unit 180 may extract several important sentences from the sentences included in the input text in ascending order of appearance frequency. The output unit 185 outputs the important sentence extracted in this way as a text summary sentence, for example, on a screen of a mobile phone.

  Each function of the document processing apparatus 100 described above is actually executed by the processor 120 executing a program describing a processing procedure for realizing these functions, or for realizing any function. This is achieved by controlling the hardware and IC. For example, the functions of the input unit 150 and the output unit 185 may be realized by an IC that functions as the interface 125 of FIG. The functions of the division unit 155, normalization unit 160, sentence hash calculation unit 165, sentence hash search unit 175, and extraction unit 180 are executed by the processor 120 in FIG. It may be achieved by doing so. Further, the function of the storage unit 170 may be achieved using a storage area such as the HDD 105, the ROM 110, or the RAM 115.

(Operation of the document processing apparatus 100)
Next, a specific operation of the document processing apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 4 is a flowchart showing important sentence extraction processing executed by the document processing apparatus 100 according to the present embodiment. It is assumed that text is input by the input unit 150 and the text input by the storage unit 170 is stored in the RAM 115 or the HDD 105 before the important sentence extraction process is executed.

  When the important sentence extraction process is started from step 400 and the process proceeds to step 405, the dividing unit 155 divides the input text into sentence units. The sentence here includes each line in a bulleted list in addition to what is delimited by the punctuation marks.

  Next, the process proceeds to step 410, where the normalization unit 160 unifies the format of character strings such as punctuation marks, half-width characters, full-width characters, etc., and proceeds to step 415, where the sentence hash calculation unit 165 proceeds to the divided sentence. Calculate the hash value for each. More specifically, the sentence hash calculation unit 165 uses a hash function such as MD5 shown in RFC1321 or SHA-1 shown in RFC3174 to obtain a fixed length from a given original sentence (divided sentence). A hash value that is a pseudo-random number is calculated.

  As a result, the character string of each divided sentence is converted into a calculated value (hash value 170a1) having a fixed length and unique to each divided sentence, as shown in FIG. The hash value 170a1 converted in this way is stored in the divided sentence hash table 170a together with the appearance frequency 170a2 of the hash value using the function of the sentence hash search unit 175 described below. In this way, the hash value calculated each time by the sentence hash calculation unit 165 and the appearance frequency of the hash value are accumulated in the divided sentence hash table 170a.

Next, when proceeding to step 420, the sentence hash search unit 175 determines whether each hash value calculated corresponding to each sentence matches one of the hash values 170a1 stored in the divided sentence hash table 170a. Search for.

  As a result of the search, when it is determined that the hash value obtained by the sentence hash calculation unit 165 is stored in the divided sentence hash table 170a, the sentence hash search unit 175 stores the appearance frequency stored in association with the hash value. 170a2 is increased (for example, the appearance frequency 170a2 is increased by one). On the other hand, when it is determined that the hash value obtained by the sentence hash calculation unit 165 is not stored in the divided sentence hash table 170a, a predetermined value, for example, “ 1 ”is stored.

  In step 425, the extraction unit 180 determines an important sentence from the original sentence (original sentence) document (text), and the output unit 185 outputs the extracted important sentence as a summary text. Specifically, the extraction unit 180 extracts an important sentence as follows using the appearance frequency 170a2 stored in the divided sentence hash table 170a.

  As described above, the appearance frequency 170a2 stored in the divided sentence hash table 170a is an accumulation of the number of times the hash value of each sentence divided from all the texts calculated so far has appeared. It becomes an index that shows how many times it appeared. Therefore, when the appearance frequency 170a2 has a large value, it can be determined that the sentence appears in many texts such as electronic mail. Such a sentence that appears frequently has a high possibility of greetings (for example, “Good morning” or “Thank you”), and is generally estimated to be less important. From such a principle, the relationship between the application frequency n of the hash value corresponding to each sentence and the importance m of each sentence is expressed as follows using the function f.

m = f (n) + α
Where α: Importance determined by other factors (optional)
At this time, in m1 = f (n1) and m2 = f (n2),
If n1> n2, m1 ≦ m2.

  Based on the importance m for each hash value calculated in this way, the extraction unit 180 extracts a sentence corresponding to the hash value 170a1 having a large importance m as an important sentence.

  At this time, for example, the extraction unit 180 extracts, as an important sentence, a sentence corresponding to the hash value 170a1 having the lowest appearance frequency 170a2 among the hash values 170a1 corresponding to the sentences included in the input text. May be. The extraction unit 180 may extract several important sentences from the hash value 170a1 corresponding to each sentence included in the input text in ascending order of appearance frequency 170a2.

  Note that each sentence included in the input text and the hash value obtained from each sentence are stored in association with one of the storage areas of the storage unit 170 until the important sentence extraction process is completed. Therefore, the extraction unit 180 extracts a sentence corresponding to a hash value having a low appearance frequency 170a2 as an important sentence based on the data stored in the storage area. Thereafter, the extracted important sentence is displayed on a mobile phone or the like, and the process proceeds to step 495 to end the present process.

  Conventionally, when it is determined whether or not a registered word or phrase is included in a part of each sentence, the character strings included in each sentence must be compared while shifting one character at a time. As a result, a large amount of processing is required to determine whether or not a registered word or phrase is included in each sentence.

  However, as described above, according to the document processing apparatus 100 according to the present embodiment, each sentence is recognized as a value unique to the sentence, that is, a hash value, and one data (hash) corresponding to each sentence is identified. Value) and the hash value 170a1 stored in the divided sentence hash table 170a, and the importance of each sentence is determined based on the application frequency detected as a result of the matching. For this reason, the processing load can be drastically reduced as compared with the prior art. Thereby, the importance of each sentence can be determined at high speed, and the important sentence can be extracted from the input text at high speed based on the importance. As a result, even text that has a large amount of information and could not be displayed on a device such as a mobile phone, which has relatively poor specifications, can be quickly summarized and displayed. For this reason, the user can not only know the content of the text by the displayed summary sentence, but also can quickly grasp the important part of the text.

  Further, according to the document processing apparatus 100 according to the present embodiment, each time this apparatus is used, the hash value 170a1 corresponding to each sentence and the data of the appearance frequency 170a2 of the hash value are automatically converted into the divided sentence hash. Accumulated in the table 170a. For this reason, it is not necessary to pre-register a specific word or phrase that serves as a clue to determine whether the sentence is an important sentence or an unnecessary sentence in the computer to a level that can sufficiently withstand practical use.

  Further, since the hash value is calculated as simple data not related to the type (language) of each character, the document processing apparatus 100 according to the present embodiment is not dependent on the language of the text. Can be used to build or operate the system. For this reason, developers and registrants do not need to be familiar with each language, and can handle unknown languages.

(Second Embodiment)
Next, the document processing apparatus 100 according to the second embodiment will be described. As shown in FIG. 5, the document processing apparatus 100 according to the present embodiment is a document processing according to the first embodiment shown in FIG. 2 in that a classification coefficient table 170b and a text classification unit 190 are newly added. It differs from the device 100 in terms of functional configuration.

  In addition, the document processing apparatus 100 according to the present embodiment determines which classification the input text belongs to based on the classification information registered in the classification coefficient table 170b in advance, and the correlation between the appearance frequency and importance described above. In addition, it is different in operation from the document processing apparatus 100 according to the first embodiment in that an important sentence is extracted from the input text in consideration of the correlation between the determined text classification and importance. Therefore, the document processing apparatus 100 according to the present embodiment will be described focusing on these differences.

  The document processing apparatus 100 according to this embodiment includes an input unit 150, a dividing unit 155, a normalizing unit 160, a sentence hash calculating unit 165, a divided sentence hash table 170a in the storage unit 170, a sentence hash searching unit 175, and an extracting unit 180. In addition to the output unit 185, the text classification coefficient table 170b (in the storage unit 170) and the text classification unit 190 have the functions shown in the blocks.

  As shown in FIG. 6, in the divided sentence hash table 170a, in addition to the items of the hash value 170a1 and the appearance frequency 170a2, the item of the classification code 170a3 is newly stored. For example, the appearance frequency of the hash value “2753... A759” in FIG. 3 is “1020”. However, in this embodiment, the hash value “2753... A759” is represented by the hash value “2753. From the classification code 170a3 of the text including the sentence to be processed, the appearance frequency is classified into two. Specifically, the appearance frequency “1020” of the hash value “2753... A759” in FIG. 3 is the appearance frequency “621” when the classification code 170a3 is “20” and the classification code 170a3 is “24”. And the appearance frequency of “399”.

  The classification code is an example showing an attribute of a text including the original sentence for which the hash value 170a1 is calculated. For example, as shown in FIG. 7, the classification code is used to classify text such as baseball (20), soccer (24), and economy (06). Also, for example, as shown in FIG. 8, texts may be classified according to the recipient of the e-mail, such as the principal (01), the part to which the principal belongs (02), and the part to which the principal does not belong (03). .

  The classification code is an example representing text attributes, and may be any information as long as it can represent text attributes. For example, the text attribute may be represented by a document type such as e-mail or Web content, information on a transmission source that transmitted the text, and the like.

  In the classification coefficient table 170b, the correlation between two classification codes, the classification code stored in the divided sentence hash table 170a (horizontal axis in FIG. 7) and the classification code to which the input text belongs (vertical axis in FIG. 7). The numerical value (correlation value) determined by is stored in advance. This numerical value has a higher value as the correlation degree of each classification code is lower. For example, when the classification code to which the input text belongs is baseball, if each classification code stored in the divided sentence hash table 170a is baseball, the correlation value is “1”, if it is soccer, “1.5”, and if economic, Would be “4”. Therefore, it can be seen that the correlation between baseball and economy is the lowest, and the correlation increases in the order of soccer and baseball.

  FIG. 8 shows an example of other information stored in the classification coefficient table 170b. Specifically, in the classification coefficient table 170b, the classification code is set to one of the person (01), the part to which the person belongs (02), or the part to which the person does not belong (03) by the recipient of the e-mail. Correlation values for the classification codes are stored in advance.

The text classification unit 190 specifies a classification code indicating the classification of the input text.
For example, using the number of occurrences of words included in the text, as shown in FIG. 7, a method for specifying the classification code of the input text such as baseball (20), soccer (24), economy (06), As shown in FIG. 8, the classification code is specified by the recipient of the electronic mail.

(Operation of the document processing apparatus 100)
Next, a specific operation of the document processing apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 9 is a flowchart showing important sentence extraction processing executed by the document processing apparatus 100 according to the present embodiment.

  When the important sentence extraction process is started from step 900 and the process proceeds to step 905, the text classification unit 190 identifies a classification code indicating the classification of the input text. Next, in steps 405 to 415, each unit executes the same processing as in the first embodiment. That is, in step 405, the dividing unit 155 divides the input text into sentence units, in step 410, the normalizing unit 160 normalizes characters and the like, and in step 415, sentence hash calculation is performed. A unit 165 calculates a hash of each sentence.

  Next, in step 420, the sentence hash search unit 175 searches whether each hash value calculated from each sentence matches one of the hash values 170a1 stored in the divided sentence hash table 170a. To do. Here, even if the hash value 170a1 is the same, a plurality of appearance frequencies 170a2 with different classification codes 170a3 are stored in the divided sentence hash table 170a of the present embodiment. Therefore, in this embodiment, the sentence hash search unit 175 detects a plurality of appearance frequencies 170a2 that match each hash value corresponding to each sentence.

  When the hash value obtained from each sentence is stored in the divided sentence hash table 170a as a result of the search, the sentence hash search unit 175 classifies the text among the appearance frequencies 170a2 stored in association with the hash value. The appearance frequency 170a2 corresponding to the code is increased by one. On the other hand, if the hash value obtained from each sentence is not stored in the divided sentence hash table 170a, “1” is stored as the appearance frequency together with the obtained hash value and text classification code.

  Next, proceeding to step 425, the extraction unit 180 determines an important sentence from the text, and the output unit 185 outputs the extracted important sentence as a summary text. In the present embodiment, the extraction unit 180 extracts an important sentence using the appearance frequency 170a2 stored in the divided sentence hash table 170a and the correlation value stored in the classification coefficient table 170b.

  Specifically, the extraction unit 180 calculates the importance m of each sentence using the function f having as variables the application frequency n of the hash value corresponding to each sentence and the correlation value k obtained from the classification code. The function f is shown below.

m = Σf (ki · ni) + α
Where α: Importance determined by other factors (optional)
n = Σni (i = index of classification coefficient table)

At this time, in m1 = f (n1) and m2 = f (n2),
If n1> n2, m1 ≦ m2.

For example, when the text classification code is baseball (20), the extraction unit 180 uses the value of each item stored in the divided sentence hash table 170a and each correlation value stored in the classification coefficient table 170b. The importance m is calculated as follows.
m = f (1.621) + f (1.5.399) + α

  Based on the importance m for each hash value calculated in this way, the extraction unit 180 extracts a sentence corresponding to the hash value 170a1 having a large importance m as an important sentence. After the extracted important sentence is displayed on the mobile phone or the like, the process proceeds to step 995 and the process is terminated.

  As described above, according to the document processing apparatus 100 according to the present embodiment, the importance value m is obtained by obtaining the correlation value k from the text classification and weighting the appearance frequency using the correlation value k. It is done. Here, the correlation value k for words and sentences that appear frequently only in a specific field is set to be small in advance in the classification coefficient table 170b. Therefore, in the case of this embodiment, the function f is weighted so that a sentence that appears frequently only in a specific field is more important than a sentence that frequently appears in other fields. In this way, the importance m for each sentence is appropriately calculated, and a more appropriate summary text can be extracted based on each calculated importance m.

(Third embodiment)
Next, a document processing apparatus 100 according to the third embodiment will be described. As shown in FIG. 10, the document processing apparatus 100 according to the present embodiment is the same as that shown in FIG. 2 in that a full-text hash table 170c, a full-hash calculation unit 195, and a full-hash search unit 199 are newly added. This is different from the document processing apparatus 100 according to the embodiment in terms of functional configuration.

  In the document processing apparatus 100 according to the present embodiment, a hash value (hereinafter referred to as a total hash value) for a character string of the entire input text is obtained, and the obtained total hash value is stored in the full text hash table 170c in advance. If the hash value matches the registered hash value, the sentence registered in advance in the full-text hash table 170c corresponding to the corresponding hash value is not executed without executing the important sentence extraction processing according to the first embodiment of FIG. It is different in operation from the document processing apparatus 100 according to the first embodiment in that it is an important sentence. Therefore, the document processing apparatus 100 according to the present embodiment will be described focusing on these differences.

  The document processing apparatus 100 according to this embodiment includes an input unit 150, a dividing unit 155, a normalizing unit 160, a sentence hash calculating unit 165, a divided sentence hash table 170a in the storage unit 170, a sentence hash searching unit 175, and an extracting unit 180. In addition to the output unit 185, the full-text hash table 170c (in the storage unit 170), the full-hash calculation unit 195, and the full-hash search unit 199 have functions shown in the respective blocks.

  Although not shown in the full-text hash table 170c, all hash values (corresponding to all calculated values) for all character strings of each input text calculated so far by the all-hash calculation unit 195 described later are the input text. On the other hand, it is accumulated in association with important sentences previously extracted. Note that the all hash calculation unit 195 may calculate the entire hash value for all the character strings of the specific part, with the text excluding the address information and the transmission source information in the input text as the specific part.

  The all hash calculation unit 195 calculates a hash value (all hash values) for all character strings of the input text. The all hash search unit 199 searches whether each all hash value obtained by the all hash calculation unit 195 matches any one of the hash values stored in the full text hash table 170c.

(Operation of the document processing apparatus 100)
Next, a specific operation of the document processing apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 11 is a flowchart showing important sentence extraction processing executed by the document processing apparatus 100 according to the present embodiment.

  When the important sentence extraction process starts from step 1100 and proceeds to step 1105, the all hash calculation unit 195 calculates a hash value (all hash values) for the entire input text. Next, proceeding to step 1110, the all-hash search unit 199 determines whether or not all hash values obtained by the all-hash calculation unit 195 match any hash value stored in the full-text hash table 170c. Perform the matching process.

  If it is determined as a result of the matching process by the all-hash search unit 199 that all hash values obtained by the all-hash calculation unit 195 are stored in the full-text hash table 170c, the all-hash search unit 199 performs step 1115. "Yes" is determined, and the process immediately proceeds to step 1120. In step 1120, the extraction unit 180 extracts the important sentence stored in the full-text hash table 170c corresponding to the full hash value as a summary text. The summary text is displayed to the user using the mobile phone by the output unit 185, and then the process proceeds to step 1195 to end the present process.

  On the other hand, as a result of the matching process by the all-hash search unit 199, if it is determined that the all-hash value obtained by the all-hash calculation unit 195 is not stored in the full-text hash table 170c, the all-hash search unit 199 In step 1115, “No” is determined, the process proceeds to step 405, and in steps 405 to 420, the same processing as that of the first embodiment is executed, so that the frequency of appearance of the hash value of each sentence with respect to the input text is obtained. 170a2 is detected.

  Next, it progresses to step 1120 and the extraction part 180 extracts an important sentence based on the appearance frequency 170a2 by performing the process similar to 1st Embodiment. The extraction unit 180 associates the extracted important sentences with the all hash values obtained by the all hash calculation unit 195 and stores them in the full text hash table 170c. In step 1120, the important sentence extracted by the output unit 185 is output as a summary text. Then, the process proceeds to step 1195, and the process ends.

  As described above, according to the document processing apparatus 100 according to the present embodiment, even when a mail with the same content is input a plurality of times due to an operation error or erroneous transmission, the value of the appearance frequency becomes higher than necessary. First, it is set to take an appropriate value that matches the actual situation. As a result, it is possible to prevent the importance of each sentence from decreasing more than necessary.

  Further, according to the document processing apparatus 100 according to the present embodiment, when the same text as the previously input text is input, for example, when the same content mail is input a plurality of times, step 405 is performed. It is not necessary to execute the processing shown in step 420. That is, while reducing the processing load, summary text can be quickly provided to the user using important sentences extracted previously.

  In all the embodiments described above, e-mail has been described as an example of input text. However, the present invention is not limited to this, and the document processing apparatus 100 targets a plurality of types of text such as text created by itself. can do.

  Further, in all the embodiments described above, the document processing apparatus 100 has been described by giving an example of summarizing received electronic mail. However, the present invention is not limited to this example, and the document processing apparatus 100 may be used, for example, for summarizing a document input with a keyboard or the like or a document stored in a storage area. Further, the document processing apparatus 100 may be used when a sentence created by itself is summarized and transmitted.

  Further, the appearance frequency 170a2 of each hash value described in the above embodiment is an example of the appearance frequency of each hash value, and the appearance frequency of each hash value may be a value indicating the rate at which each hash value appears. Any value can be used. For example, as another example of the appearance frequency of the hash value, the appearance rate of the corresponding hash value with respect to all the hash values stored in the divided sentence hash table 170a and the corresponding hash value with respect to the average appearance frequency of all the hash values Deviations are mentioned.

  In the above, the dividing unit 155 divides the text into sentence units. However, the dividing unit 155 only needs to be able to divide the text into a plurality of character strings based on a predetermined condition, and is not necessarily divided into sentence units. For example, the dividing unit 155 may divide the text into phrase units or paragraph units. More specifically, the dividing unit 155 may divide the text into paragraphs based on a condition that a line break occurs in the text. In addition, based on the condition that one of the comma or period is divided After the appearance, "Hello, this is ○○." "Hello," a document that may be divided into two character string "is ○○" . In this case, the document processing device 100, by deleting "Hello, this is ○○." "Hello, this is △△." The line containing the string or "Hello" in the high frequency of occurrence "Hello" from, An important sentence (that is, a summary sentence consisting of “XX” and “△△”) can be extracted from the text.

  In the above description, the output unit 185 displays the summary text on the mobile phone display. However, the present invention is not limited to this. For example, the output unit 185 outputs summary text to the CRT 300 or printer 305 of another device as shown in FIG. 1 or stores the summary text in the network card 310 or the like. Alternatively, the summary text may be output to the voice output device 315 as voice information.

  In the above embodiment, the operations of the respective units are related to each other, and can be replaced as a series of operations in consideration of the relationship between each other. And by replacing in this way, the embodiment of the invention of the document processing apparatus can be made an embodiment of the document processing method.

  Further, by replacing the operation of each unit with the processing of each unit, a program embodiment can be obtained. Further, by storing the program in a computer-readable recording medium in which the program is recorded, the embodiment of the program can be an embodiment of a computer-readable recording medium in which the program is recorded.

  Therefore, the embodiment of the document processing method includes a process of associating a calculated value calculated based on a predetermined rule and an appearance frequency indicating the frequency at which the calculated value is calculated in association with each other, and storing the document in a predetermined condition. A process of dividing into a plurality of character strings based on the above, a process of determining a calculated value specific to each character string from each of the divided character strings based on the predetermined rule, By comparing the calculated values stored in the storage unit to search for the appearance frequencies corresponding to the calculated values, and based on the appearance frequencies corresponding to the calculated values. Implementation of a document processing program that causes a computer to select one or more calculated values and extract a character string corresponding to the selected one or more calculated values as a summary of the document. It can be a state.

  Further, the embodiment of the document processing method includes a process of associating a calculated value calculated based on a predetermined rule and an appearance frequency indicating the frequency at which the calculated value is calculated in association with each other, and storing the document in a predetermined condition. A process of dividing into a plurality of character strings based on the above, a process of determining a calculated value specific to each character string from each of the divided character strings based on the predetermined rule, By comparing the calculated values stored in the storage unit to search for the appearance frequencies corresponding to the calculated values, and based on the appearance frequencies corresponding to the calculated values. , Selecting one or more calculated values, and extracting a character string for the selected one or more calculated values as a summary of the document. It can be used as an embodiment of the computer readable recording medium.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to the example which concerns. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

  For example, in the document processing apparatus 100 according to each of the embodiments described above, it has been described that each unit exists in the document processing apparatus 100. However, the present invention is not limited to this, and some of the functions of each unit are It may be included in another control means connected by a network, and there may be a plurality of means and functions of each part for load distribution and ensuring safety.

  The present invention is applicable to a document processing apparatus and a document processing method for automatically extracting important parts from a document at high speed.

It is a hardware block diagram of the document processing apparatus concerning 1st Embodiment of this invention. It is a functional block diagram of the document processing apparatus concerning the embodiment. It is the figure which showed an example of the data structure of a division | segmentation sentence hash table. It is the flowchart which showed the important sentence extraction processing routine which a document processing apparatus performs in the same embodiment. It is a functional block diagram of the document processing apparatus concerning 2nd Embodiment of this invention. It is the figure which showed another example of the data structure of a division | segmentation sentence hash table. It is the block diagram which showed an example of the data structure of a classification coefficient table. It is the block diagram which showed another example of the data structure of a classification coefficient table. It is the flowchart which showed the important sentence extraction processing routine which a document processing apparatus performs in the same embodiment. It is a functional block diagram of the document processing apparatus concerning 3rd Embodiment of this invention. It is the flowchart which showed the important sentence extraction processing routine which a document processing apparatus performs in the same embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Document processing apparatus 120 Processor 150 Input part 155 Division | segmentation part 160 Normalization part 165 Sentence hash calculation part 170 Storage part 170a Division | segmentation sentence hash table 170b Classification coefficient table 170c Whole sentence hash table 175 Sentence hash search part 180 Extraction part 185 Output part 190 Text Classification part 195 All hash calculation part 199 All hash search part

Claims (11)

A storage unit that associates and stores a calculated value calculated based on a predetermined rule and an appearance frequency indicating a frequency at which the calculated value is calculated;
A dividing unit for dividing the document into a plurality of character strings based on a predetermined condition;
A calculation unit for obtaining a calculated value unique to each character string from each of the divided character strings based on the predetermined rule;
A search unit that detects each occurrence frequency corresponding to each calculated value by comparing each calculated value obtained by the calculating unit and the calculated value stored in the storage unit;
An extraction unit that selects one or more calculated values based on the appearance frequency corresponding to each detected calculated value and extracts a character string corresponding to the selected one or more calculated values as a summary of the document And a document processing apparatus. The calculation unit is
The document processing apparatus according to claim 1, wherein a hash value as the calculated value is obtained from each character string using a hash function as the predetermined rule. The extraction unit
The one or two or more calculation values are selected in order from the calculation value stored in association with the appearance frequency having a lower value by comparing the appearance frequencies corresponding to the detected calculation values. The document processing apparatus according to claim 1 or 2. The search unit
As a result of the comparison, if it is determined that the calculated value obtained by the calculating unit is stored in the storage unit, the appearance frequency stored in association with the calculated value is increased, 4. If it is determined that the calculated value is not stored in the storage unit, an appearance frequency having a given value is newly stored together with the calculated value. Document processing apparatus described in the above. The dividing unit is
The document processing apparatus according to claim 1, wherein the document is divided into a plurality of character strings constituting any of a clause or a sentence or a paragraph. Said document processing device, further comprising:
The document processing apparatus according to claim 1, further comprising: a normalization unit that adjusts a format of a character string included in either the document or each divided sentence. Said document processing device, further comprising:
A text classification unit for defining the attributes of the document;
The storage unit
Storing a plurality of appearance frequencies of the calculated values for each attribute of the document;
The extraction unit
A plurality of appearance frequencies stored for each attribute of the document in the storage unit are respectively used by using a correlation value determined from a correlation between the determined attribute of the document and the attribute of the document stored in the storage unit. The document processing apparatus according to claim 1, wherein weighting is performed, and one or two or more calculated values are selected based on each weighted appearance frequency. The extraction unit
Using the correlation values, the calculated values are respectively weighted to a plurality of appearance frequencies stored for each attribute of the document, and the sum of the weighted appearance frequencies is assigned as the importance corresponding to the calculated values. 8. The document processing apparatus according to claim 7, wherein one or two or more corresponding calculated values are selected in descending order of the calculated importance. The correlation value is
9. The apparatus according to claim 7, wherein the smaller the association between the attribute of the document determined by the text classification unit and the attribute of the document stored in the storage unit is, the larger the value is set. Document processing device described. The calculation unit is
Find the calculated value specific to a specific part or all part of the document as the total calculated value,
The search unit
Search whether all the calculated values are stored in the storage unit ,
When all the calculated values are not stored in the storage unit,
Processing of the dividing unit that divides a document into a plurality of character strings based on a predetermined condition; and a calculation unit that obtains a calculation value specific to each character string from each of the divided character strings based on the predetermined rule. Processing, and processing of the search unit for detecting appearance frequencies corresponding to the calculated values by comparing the calculated values calculated with the calculated values stored in the storage unit Thereafter, the extraction unit extracts a character string for one or more calculated values selected based on the appearance frequency corresponding to each detected calculated value as a summary of the document,
When all the calculated values are stored in the storage unit ,
The extraction section, the document processing apparatus according to a character string corresponding to the total calculated value stored immediately before term memory unit to any one of claims 1 to 9 for extracting as a summary of the document. Storing the calculated value calculated based on a predetermined rule and the appearance frequency indicating the frequency at which the calculated value is calculated in association with each other in the storage unit;
Dividing the document into a plurality of character strings based on predetermined conditions;
Obtaining a calculated value unique to each character string from each of the divided character strings based on the predetermined rule;
Searching for the appearance frequency corresponding to each of the calculated values by comparing the calculated values and the calculated values stored in the storage unit;
Document processing for selecting one or more calculated values based on each appearance frequency corresponding to each searched calculated value and extracting a character string corresponding to the selected one or more calculated values as a summary of the document Method.

Images