JP2008197952A - Text segmentation method, its device, its program and computer readable recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To achieve a text segmentation corresponding to various fields without making it necessary to provide any data base for learning in the case of dividing an input text into the blocks of a semantically consistent sentence aggregate. <P>SOLUTION: An input text is divided into sentences in order to divide one text with consistent content, and the nouns of each sentence are defined as retrieval words (S1 to S3). Then, Web retrieval is performed by using the retrieval words. Nouns whose appearance frequency is high are extracted from the text retrieved by Web retrieval, and defined as relevant words (S4). The retrieval words+relevant words are used as the keyword group of the sentence, and when the number of the keywords of the adjacent sentences is predetermined or more and overlapped, they are gathered as one block, and when they are not overlapped, they are defined as the division of a text, and the division result of the text is output (S5, S6). <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a method for dividing a sentence in a text into a sentence or a group of sentences according to the description content of the text in a field where the text is handled by a computer such as a personal computer. The present invention relates to a text segmentation method that does not use a learning database.

  In recent years, with the rapid performance improvement of computers, it has become possible to build a database by accumulating enormous text (here, a set of sentences consisting only of character strings). However, it is generally difficult to manually organize and manage stored text. The technique of dividing a given text according to its content is called text segmentation, and it is being applied to automatically classify and organize text databases with a computer. For example, in the following Patent Document 1 (topic boundary determination method and apparatus and topic boundary determination program), a technique for performing text segmentation using information called a concept base is proposed.

In this technology, a plurality of concept vectors obtained by converting a certain word and a pattern co-occurring into it into numerical vectors are created from a learning database accumulated in advance. Then, text segmentation is performed using a concept base which is a collection of concept vectors. In the learning database, many texts related to one field (for example, text related only to the field of “politics”) are accumulated.
JP 2004-234512 A

  However, in order to increase the accuracy of the conventional text segmentation method, a large-scale learning database must be prepared, and enormous effort is required to prepare the learning database. If the learning database is small, the concept base cannot be created properly, which reduces the accuracy of text segmentation. In addition, since the learning database prepared in advance supports only a specific field, text segmentation cannot be performed on texts in different fields.

  An object of the present invention is to provide a text segmentation technique that does not require a learning database and can handle texts in various fields.

  In general, it is desirable to be able to perform text segmentation without preparing a learning database. Therefore, the present invention solves the above-mentioned problems by creating a new text segmentation technique that focuses on the concept of searching on the web.

  In the present invention, for example, in order to divide a single text into coherent contents, first, the text of the document to be processed is divided into sentences, and nouns in each sentence are used as search words. When nouns are used as search terms, unnecessary nouns are removed in advance from search term candidates as necessary. Next, a web search is performed using the search terms. A noun with high appearance frequency is extracted from the text searched by the web search and used as a related word. The search word + related word is set as a keyword set of the sentence, and if there are no more than a predetermined number of keywords between adjacent sentences, the sentence is separated.

  Here, the web means a set of texts described in a structured language such as HTML or XML that can be accessed via a network such as the Internet. Currently, a huge amount of information is accumulated on the web, and the latest topics are always provided. In other words, the web can be understood as a dictionary with various information. In fact, when we look into a certain thing, we enter a search word on a search site and search the web to find out the meaning of the word and the contents of things. From that point of view, if the information on the web is used appropriately without using a learning database, the concept of "sports" and "balls" can be obtained for "soccer" and "baseball". I can say that. As a result, words corresponding to the description contents of the text can be acquired from various information on the web, and the contents of sentences can be tracked by comparing the relevance of sentences widely. Therefore, it is not necessary to prepare a learning database in advance or to maintain and manage it, and text segmentation corresponding to various fields can be realized.

  As described above, in the present invention, by performing a web search using a search term, text described in a plurality of structured languages related to the description content of the text is acquired, and a plurality of items highly related to the description content are acquired. The focus is on the ability to extract words. As a result, it is possible to perform text segmentation that can deal with various fields without requiring the learning database that is the object of the present invention.

  Specifically, the present invention is a text segmentation method in which a text, which is a set of electronically converted sentences, is divided into blocks made up of one or more sentences by a computer, and the text to be divided is input. The input text is divided into sentence units, morphological analysis is performed on each divided sentence, a search word is extracted for each sentence, and the search word extracted for each sentence is used on the web. Perform a search, obtain related words from the obtained search results, create keyword sets that are pairs of search words and related words in each sentence, compare those keyword sets for each adjacent sentence, and share them Based on the number of keywords, a decision is made as to whether or not the sentences in the keyword set to be compared are combined into a single block, and a block is generated. It is output as a result.

  Thereby, without using a learning database, the input text can be accurately divided into a sentence group that is semantically grouped or a sentence group block that refers to the same content.

  In addition, in the acquisition of related words from the search results obtained by web search, the noun word is extracted from the multiple texts described in the structured language that is the search result, the appearance frequency of the extracted word is calculated, A predetermined number of words in descending order of appearance frequency are selected as related words. The number of related words may be selected so that the total number of search words and related words in one sentence becomes a constant value, for example. This makes it possible to select words that are closely related to the search word as related words, and to guarantee a certain degree of uniformity regarding the strength of the semantic unit of each block into which the text is divided. It becomes like this.

  In addition, when extracting a search word from each sentence or acquiring a related word from a search result obtained by web search, a noun word included in each sentence or the search result is registered in an unnecessary word list in advance. Search words or related words are excluded. As a result, it is possible to avoid the case where words cannot be divided by words having poor semantic content.

  According to the present invention, it is possible to realize a text segmentation technique that does not require a learning database to be prepared in advance by using the concept of searching on the web. In addition, there is an advantage that there is no restriction in the field of description content for the target text for text segmentation because various information accumulated on the web is used.

  The present invention can be applied as support technology for automatically organizing and updating text data in the field of handling enormous text data and the field of distributing news articles.

  FIG. 1 shows an outline of the processing procedure of the present invention. In FIG. 1, in step S1, a process for inputting text is executed. In step S2, a process of dividing the input text into sentence units is executed. In step S3, a process for extracting a search term from the sentence is executed. In step S4, a search is performed on the web using the search word, and a process of acquiring a related word from the search result is executed. In step S5, a process of dividing the text using a keyword set made up of a set of search terms and related terms is executed. In step S6, a process for outputting the text segmentation result of the text divided in step S5 is executed.

  FIG. 2 shows a configuration diagram of a text segmentation processing apparatus that does not use a learning database using web search in the embodiment of the present invention. In FIG. 2, a computer 1 includes a text decomposition processing unit 11, a search word extraction processing unit 12, a related word acquisition processing unit 13, a text division processing unit 14, and a control unit configured by software programs, storage devices, and the like. 15, an input unit 16, an output unit 17, a decomposed text storage unit 20, a search word storage unit 30, a related word storage unit 40, and a divided block storage unit 50.

  The computer 1 is connected to a network 3 and can access the web 4. The web 4 stores a plurality of texts 5 written in a structured language such as HTML or XML. Text 6 is text that is input to the input unit 16 of the computer 1. The display unit 2 is a device for displaying the result output from the control unit 15 through the output unit 17.

  FIG. 3 shows an example of the text 6 in the embodiment of the present invention. A text 6 shown in FIG. 3 is an example of text input by the input unit 16 to be segmented for explaining the embodiment of the present invention.

  FIG. 4 shows an example of sentences stored in the decomposed sentence storage unit 20 in the embodiment of the present invention. In FIG. 4, 21 is the first sentence of text 6, 22 is the second sentence of text 6, 23 is the third sentence of text 6, 24 is the fourth sentence of text 6, and 25 is 5 of text 6 The sixth sentence, 26 is the sixth sentence of text 6, 27 is the seventh sentence of text 6, 28 is the eighth sentence of text 6, and 29 is the ninth sentence of text 6.

  FIG. 5 shows an example of an unnecessary word list in the embodiment of the present invention. In FIG. 5, reference numeral 60 denotes an unnecessary word list. In the unnecessary word list 60, words to be ignored in the segmentation process are registered in advance in a storage unit (not shown).

  FIG. 6 shows an example of a search word stored in the search word storage unit 30 in the embodiment of the present invention. In FIG. 6, 31 is a search word corresponding to sentence 21, 32 is a search word corresponding to sentence 22, 33 is a search word corresponding to sentence 23, 34 is a search word corresponding to sentence 24, and 35 is equivalent to sentence 25. , 36 is a search word corresponding to the sentence 27, 37 is a search word corresponding to the sentence 27, 38 is a search word corresponding to the sentence 28, and 39 is a search word corresponding to the sentence 29.

  In FIG. 7, an example of the related word stored in the related word memory | storage part 40 in embodiment of this invention is shown. In FIG. 7, 41 is a related word corresponding to the search word 31, 42 is a related word corresponding to the search word 32, 43 is a related word corresponding to the search word 33, 44 is a related word corresponding to the search word 34, 45 is Related words corresponding to the search word 35, 46 related words corresponding to the search word 36, 47 related words corresponding to the search word 37, 48 related words corresponding to the search word 38, and 49 corresponding to the search word 39. Represents a related term.

  FIG. 8 shows an example of a keyword set created by the text division processing unit 14 in the embodiment of the present invention. In FIG. 8, 71 is a keyword set generated from a set of a search word 31 and a related word 41, 72 is a keyword set generated from a set of a search word 32 and a related word 42, and 73 is a search word 33 and a related word 43. A keyword set generated from the set, 74 is a keyword set generated from the set of search terms 34 and related terms 44, 75 is a keyword set generated from a set of search terms 35 and related terms 45, and 76 is a search term 36. A keyword set generated from a set of related terms 46, 77 is a set of keywords generated from a set of search terms 37 and related terms 47, 78 is a set of keywords generated from a set of search terms 38 and related terms 48, and 79 is a set of keywords. A keyword set generated from a set of search terms 39 and related terms 49 is represented.

  FIG. 9 shows an example of sentence numbers belonging to each block stored in the divided block storage unit in the embodiment of the present invention. In FIG. 9, 51 represents a sentence number belonging to the first block, and 52 represents a sentence number belonging to the second block.

  A text segmentation processing procedure according to an embodiment of the present invention will be described in detail with specific examples. First, when the text 6 is input through the input unit 16, the text decomposition processing unit 11 is called from the control unit 15.

  The text decomposition processing unit 11 reads the text 6 character by character and cuts it out in sentence units. And the process which stores the cut-out several sentence in the decomposition | disassembly sentence memory | storage part 20 via the control part 15 is performed. Here, a sentence refers to a sentence delimited by the punctuation mark “.”. Here, for example, punctuation marks appearing in a conversation sentence surrounded by parenthesis symbols such as ““ ”and“ ”” are ignored, and the number of generated sentences varies depending on the input text 6. Example of Text 6 When the text decomposition processing unit 11 is executed on the text 6 shown in FIG. 3, a sentence 29 is generated from the nine sentences 21 decomposed into sentence units as shown in FIG. 4, and is decomposed via the control unit 15. It is stored in the text storage unit 20.

  Next, the control unit 15 executes processing by the search word extraction processing unit 12 for each sentence stored in the decomposed text storage unit 20. Here, the search term refers to one or more words that are input when searching on the web. First, the search word extraction processing unit 12 performs morphological analysis on each inputted sentence. Then, a plurality of words determined as nouns are taken out as search words and stored in the search word storage unit 30 via the control unit 15.

  However, by simply extracting the noun word, commonly used words such as “month” and “day” are also extracted. Therefore, an unnecessary word list 60 such as “month” or “day” is created in advance, and noun words that are not registered in the unnecessary word list 60 are handled as search words. FIG. 5 shows an example of such an unnecessary word list 60.

  On the other hand, when all the words extracted as nouns such as sentence 27 are unnecessary words or when no word that becomes a noun exists in the sentence like sentence 23, the search word extraction processing unit 12 selects the search word. Not extracted. In that case, the search word extraction processing unit 12 does not store the word in the search word storage unit 30 like the search word 33 in FIG.

In addition, there are cases where a small number of search terms are extracted. However, there is no problem even if such a sentence does not have any particularly meaningful content. Therefore, in the search word extraction processing unit 12, the number of the extracted search terms in the case of less than a predetermined threshold value S _T is treated as no search word sentence does not store the word in the search word storage unit 30. For example, if S _T = 1, the noun extracted in the sentence 22 is one “Kurihama”, so it is treated as a sentence having no search word, and the search word is searched like the search word 32 in the search word storage unit 30. Do not store. In FIG. 6, examples of search words from sentence 21 to sentence 29 when S _T = 1 are shown as search word 31 to search word 39.

  After the search word 39 is created from the search word 31 corresponding to the sentence 21 to the sentence 29, the related word acquisition processing unit 13 is called from the control unit 15. In the related word acquisition processing unit 13, the search word first extracted by the search word extraction processing unit 12 is extracted from the search word storage unit 30 through the control unit 15 and input. Next, a search is performed on the web 4 connected by the network 3 using the input search word. Then, a predetermined number P of texts 5 described in a structured language such as HTML or XML referenced in the search result are acquired from the web 4 via the network 3, and the text is obtained from the acquired texts. Extract the contents of.

  Note that the web search for acquiring related terms is based on searching for a plurality of search terms under the and condition. In other words, a web page where all search terms appear is basically searched. The reason is that when multiple search terms are searched using the or condition, a web page in which only some of the search terms appear appears in the search results, and words that are weakly related to the entire search terms may be extracted as related terms. Because it is expensive.

  By analyzing a tag composed of a character string surrounded by “<” and “>” in a text described in a structured language, P body texts describing the content of the body text are obtained. After extracting P body texts, the related word acquisition processing unit 13 performs morphological analysis on the texts to extract noun words. And the appearance frequency of the word of the extracted noun is investigated, and a plurality of words are stored in the related word storage unit 40 as related words in descending order of frequency.

  However, if a noun word is directly used as a related word, a universally used word such as “month” or “day” may be treated as a related word, as in the search word extraction processing unit 12. Therefore, in the related word acquisition processing unit 13, similarly to the search word extraction processing unit 12, the unnecessary word list 60 as shown in FIG. 5 is referred to and only words that are not registered in the unnecessary word list 60 are used as related words. Stored in the related word storage unit 40.

  As an example of the related word, a related word 41 when the search word 31 is input is shown in FIG. At this time, in the related word acquisition processing unit 13, the number of related words obtained differs depending on the input search word. Therefore, in order to adjust the number of related words to be acquired, the number of related words to be collected is set using the threshold value T of the total number of search words and related words. More specifically, if S search words are extracted by the search word extraction processing unit 12 for a certain sentence, the related word acquisition processing unit 13 obtains only T−S related words by a web search. Extract from text. If the number of search terms exceeds the total number T, no related terms are acquired, and only T search terms selected at random are left.

Furthermore, in the related word acquisition processing unit 13, the number of web pages obtained differs depending on the input search word, but it is desirable that as many web pages as possible can be obtained in the present invention. Therefore, a threshold value P _T is set for the number of web pages obtained by web search. When the obtained number of web pages does not exceed the threshold value P _T , it is determined that the search cannot be performed sufficiently, the related words are not extracted, and the words are not stored in the related word storage unit 40. For example, when a web search is performed using the search word 38 for the sentence 28, the number of searches is four. Therefore, if the threshold value P _{T is set} to P _T = 5, a related word is not extracted from the web page acquired by the related word acquisition processing unit 13 for the search word 38, and a related word storage unit like the related word 48 is used. 40 does not store any words. FIG. 7 shows an example of related words 41 to 49 corresponding to sentences 21 to 29 when P _T = 5.

  Finally, when the processing by the search word extraction processing unit 12 and the related word acquisition processing unit 13 is completed for all the sentences stored in the decomposed text storage unit 20, the control unit 15 performs the text division processing unit 14 Processing is executed. In the text division processing unit 14, first, the search terms and the related terms stored in the search term storage unit 30 and the related term storage unit 40 are sequentially extracted via the control unit 15, and a keyword set including these as elements is created. Repeat for each sentence.

  For example, for the sentence 21, the keyword set 71 of FIG. 8 is created from the search word 31 and the related word 41. Here, when there is no search word as in the sentence 22 (32 in FIG. 6), there is no corresponding keyword, so there is no corresponding keyword set. On the other hand, when there is a search word as in the sentence 28 and there is no related word, a keyword set is created using only the search word 38.

When the creation of the keyword set is completed, the text division processing unit 14 compares the generated keyword set of each sentence two by two from the top and examines the presence of the common word 6 Split. In general, sentences are often written in order from the beginning. Therefore, the present invention analyzes the keyword set for the two statements in order from the beginning, the number of common words without dividing if more than a predetermined threshold value C _T, performs the division is less than C _T. This comparison process is repeated until the last two sets of text are compared. Then, a block composed of a plurality of or one sentence obtained by the text division processing unit 14 is stored in the divided block storage unit 50 through the control unit 15.

Here, when the two keyword sets are compared in the text division processing unit 14, the one that always includes a word is used. A specific processing procedure will be described using an example in which the keyword set and C _T = 1 are set in FIG.

First, comparison between the keyword set 71 and the keyword set 72 in FIG. 8 is attempted. However, since there is no word in the keyword set 72, the comparison process is not performed and the keyword set in which the word exists is found. As a result, the keyword set 71 and the keyword set 74 having words in the keyword set are first compared. When the common words in the keyword set 71 and the keyword set 74 are examined, one common word “travel” is extracted. Since this number is equal to or greater than the threshold C _T = 1, sentences 21 through 24 are set as one block.

  Next, the keyword set 74 and the keyword set 75 are compared. As a result, since two common words “hot spring” and “travel” are extracted, sentences 21 to 25 are assumed to be one block. Then, since there is no common word when comparing the keyword set 75 and the keyword set 76, the first block is determined to be the sentence 21 and the sentence 25, and the control unit 15 determines the number of the sentence belonging to the first block. Thus, the data is stored in the divided block storage unit 50 as shown by the sentence number 51 in FIG.

  When the same processing is repeated for the keyword sets subsequent to the keyword set 76, the second block becomes sentence 26 to sentence 29, and the sentence numbers belonging to the second block are assigned sentence numbers 52 in FIG. As shown in FIG. From the result of FIG. 9, the given text 6 is divided into two, the first to fifth sentences belong to the first block, and the sixth to ninth sentences belong to the second block. I understand that.

  On the other hand, when there is no common word in the i-th and j-th (where i <j) keyword sets, and there are no words in the i + 1-th to j-1-th keyword sets, the i + 1-th to j-1 The second is not assigned to any block. In this case, the i-th sentence is divided into the i-th sentence, the i + 1-th sentence to the j-1-th sentence are divided into one block (referred to as an empty block in the present invention), and the divided block storage unit 50 stores the divided sentences. Store.

  Finally, the sentence number of each block stored in the divided block storage unit 50 is output to the output unit 17 through the control unit 15. For example, as shown in FIG. 9, there is a method in which the number of each block and the number of a sentence belonging to it are output as a set.

In the above embodiments, with respect to fineness of computation time or divided to perform text segmentation, the parameter S _T from the outside, P _T, C _T, T, by providing a means for specifying a P, optionally by these parameters Can be adjusted. As a method of comparing keyword sets, a method of extracting common words by weighting frequently used words for each field can be considered. Further, the output unit 17 refers to the search word storage unit 30 and the related word storage unit 40, and uses the information that the word does not exist in the keyword set as shown in FIG. 8 to remove the sentence assigned to the empty block. Can also be output.

  The above text segmentation processing can be realized by a computer and a software program. The program can be provided by being recorded on a computer-readable recording medium or provided through a network.

It is a figure which shows the outline | summary of the process sequence in this invention. It is a block diagram of the text segmentation processing apparatus in embodiment of this invention. It is a figure which shows an example of the text in embodiment of this invention. It is a figure which shows an example of the sentence stored in the decomposition | disassembly text storage part in embodiment of this invention. It is a figure which shows an example of the unnecessary word list | wrist in embodiment of this invention. It is a figure which shows an example of the search word stored in the search word memory | storage part in embodiment of this invention. It is a figure which shows an example of the related word stored in the related word memory | storage part in embodiment of this invention. It is a figure which shows an example of the keyword set produced by the text division | segmentation process part in embodiment of this invention. It is a figure which shows an example of the sentence number stored in the division | segmentation block memory | storage part in embodiment to this invention.

Explanation of symbols

S1 Process for inputting text S2 Process for dividing text into sentences S3 Process for extracting words as search terms for each sentence S4 Process for acquiring related terms using search terms S5 Keyword set (with search terms and S6 Process for Dividing Text by Set of Related Words S6 Process for Outputting Division Results 1 Computer 2 Display Unit 3 Network 4 Web 5 Multiple Texts Described in Structured Language 6 Input Text 11 Text Decomposition Processing Unit 12 Search Term Extraction processing unit 13 Related word acquisition processing unit 14 Text division processing unit 15 Control unit 16 Input unit 17 Output unit 20 Decomposed sentence storage unit 21-29 1 to 9th sentence stored in decomposed sentence storage unit 30 Search word storage 31 to 39 are stored in the search word storage corresponding to the first to ninth sentences registered in the decomposed sentence storage Word 40 related word storage unit 41-49 word stored in related word storage unit corresponding to first to ninth words registered in search word storage unit 50 divided block storage unit 51, 52 in divided block storage unit Sentence number belonging to the 1st and 2nd blocks stored 60 Unnecessary word list 71 to 79 1st to 9th words registered in the search word storage unit and 1 to 9th stored in the related word storage unit Keyword set created by pairing words

Claims (6)

In a text segmentation method for dividing text, which is a set of electronic information sentences, into blocks consisting of one or more sentences by a computer,
The computer is
The process of entering the text to be split,
The process of dividing the input text into sentences,
A process of performing morphological analysis on each divided sentence, extracting a search word for each sentence, and storing it in the search word storage means;
Performing a search on the web using the search terms extracted for each sentence, obtaining related terms from the obtained search results, and storing them in the related term storage means;
By referring to the search word storage means and the related word storage means, a keyword set which is a set of search words and related words in each sentence is created, and the keyword sets are compared for each adjacent sentence. The process of generating a block by deciding whether or not to combine the sentences of the keyword set to be compared into one block according to the number of common keywords,
A text segmentation method, comprising: outputting a generated block as a result of text division. The text segmentation method of claim 1,
In the acquisition of related words from the search results obtained by the web search, the words of nouns are extracted from a plurality of texts described in the structured language as the search results, and the appearance frequency of the extracted words is calculated. A text segmentation method, wherein a predetermined number of words in order of frequency are selected as related words. The text segmentation method according to claim 1 or 2,
In the extraction of search terms from each sentence or the acquisition of related words from the search results obtained by the web search, the noun words included in the respective sentences or the search results are registered in the unnecessary word list in advance. A text segmentation method characterized in that a search word or related word is excluded from a word. In a text segmentation processing apparatus that divides text, which is a set of electronically converted sentences, into blocks consisting of one or more sentences,
An input means for inputting the text to be divided;
Text decomposition processing means for dividing input text into sentence units;
Search word extraction processing means for performing morphological analysis on each divided sentence and extracting a search word for each sentence;
Search term storage means for storing the extracted search terms;
A related word acquisition processing means for performing a search on the web using the search word extracted for each sentence, and acquiring a related word from the obtained search result;
A related word storage means for storing the acquired related words;
By referring to the search word storage means and the related word storage means, a keyword set which is a set of search words and related words in each sentence is created, and the keyword sets are compared for each adjacent sentence. , A text division processing means for determining whether or not to combine the sentences of the keyword set to be compared into one block according to the number of common keywords, and generating a block;
An output means for outputting the generated block as a result of text division. A text segmentation processing apparatus, comprising:   A text segmentation processing program for causing a computer to execute the text segmentation method according to claim 1, claim 2 or claim 3.   A computer-readable recording medium on which a text segmentation processing program for causing a computer to execute the text segmentation method according to claim 1, 2 or 3 is recorded.

Images