JP2009015795A - Text segmentation apparatus, text segmentation method, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a text segmentation method, an apparatus therefor, a program, and a recording medium which dispense with learning data. <P>SOLUTION: A text segmentation apparatus includes: a text decomposition section for decomposing input text into sentence units; a retrieval word extraction section for performing morphological analysis on a sentence decomposed with the text decomposition section to extract a noun as a retrieval word from among analyzed morphemes; a related word acquisition section for performing web retrieval with a retrieval word extracted with the retrieval word extraction section, and performing morphological analysis on the retrieved text to acquire a noun as a related word from among the analyzed morphemes; and a connectivity determination section for using a keyword set as a combination of retrieval word and related word to determine connectivity among the plurality of sentences obtained by decomposing the input text, and then segmenting the input text by extracting semantic paragraphs as sentences each having connectivity higher than a predetermined threshold. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

The present invention relates to a technique for automatically dividing a sentence in a text according to a plurality of contents described in the text in a field where the text is used on a computer.

  In recent years, with the rapid improvement in computer performance, it is possible to accumulate a huge amount of text (here, a set of sentences composed only of character strings) and construct a database.

  However, it is generally difficult to manually organize and manage stored text. Therefore, a technique called “text segmentation” has been developed in which the stored text database is analyzed and the inputted text is divided into semantic paragraphs according to the contents. When a computer performs classification and organization of a text database, the above text segmentation is being applied. As one typical method of text segmentation, a technique for performing text segmentation using information called a concept base has been proposed (for example, see Patent Document 1).

  In this proposed technique, a plurality of concept vectors obtained by converting a certain word and a co-occurrence pattern into numerical vectors are created from previously stored learning data. Then, text segmentation is performed using a concept base which is a collection of concept vectors. In the learning data, many texts related to one field (for example, text related only to the field of “politics”) are accumulated.

It is also known to evaluate the semantic continuity between sentences based on the degree of connectivity for one type of block (see Non-Patent Document 1, for example). In this conventional example, when the block length is short, there is a high possibility that semantic paragraphs are excessively estimated instead of easily following local changes in semantic content. On the other hand, if the length of the block is long, it is not easy to catch a change in global semantic content, but it is not possible to deal with text that changes its semantic content slowly.
JP 2002-342324 A Hearst, MA,: Multi-Paragraph Segmentation of Expository Text, 32nd Annual Meeting of the Association for Computational Linguistics, pp. 9-16 (1994)

  However, in order to improve the accuracy of the conventional text segmentation technique, large-scale learning data must be prepared. Therefore, when the learning data is small, the concept base cannot be appropriately created, and there is a problem that the accuracy of text segmentation is lowered.

  Moreover, since it corresponds to the field | area contained in the learning data prepared beforehand, text segmentation cannot be performed about the text of a different field | area. For example, if only information related to “politics” and “economy” is accumulated in the learning data, it is difficult to text segment the text in the field of “sports”.

An object of the present invention is to provide a text segmentation method that does not require learning data, an apparatus thereof, a program, and a recording medium.

The present invention provides a text decomposition unit that decomposes input text into sentence units, a morphological analysis of a sentence decomposed by the text decomposition unit, and a search word extraction that extracts a noun as a search word from the analyzed morphemes And a web search using the search terms extracted by the search term extraction unit, morphological analysis of the searched text, and among the analyzed morphemes, a related word acquisition unit that acquires nouns as related terms, and Meaning that, by using a keyword set that is a combination of a search term and the related word, the connectivity between the plurality of sentences obtained by decomposing the input text is determined, and sentences having a connectivity higher than a predetermined threshold value. A text segmentation apparatus comprising: a connectivity determination unit that divides the input text by extracting a paragraph.

According to the present invention, when text segmentation is performed, the concept of searching on the web is used, so that it is not necessary to prepare learning data in advance.

  The best mode for carrying out the invention is the following examples.

  FIG. 1 is a block diagram showing a text segmentation apparatus 100 that is Embodiment 1 of the present invention.

  The text segmentation device 100 is a device that performs text segmentation on input text without using learning data. The text segmentation device 100 performs a web search based on the input text, analyzes the web searched text, and analyzes the analyzed text. A device that divides the input text according to the content of the text, and is configured by a computer.

  The text segmentation device 100 includes a text decomposition unit 1, a search word extraction unit 2, a related word acquisition unit 3, a connectivity determination unit 4, a decomposed sentence storage unit M1, a search word storage unit M2, and a related word storage. The unit M3, the semantic paragraph storage unit M4, the input unit 5, the control unit 6, and the output unit 7 are included.

  The text decomposition unit 1 decomposes the input text T1 into sentence units.

  The search word extraction unit 2 performs morphological analysis on the sentence decomposed by the text decomposition unit 1 and extracts nouns as search words from the analyzed morphemes.

  The related word acquisition unit 3 searches the web using the search word extracted by the search word extraction unit 2, performs morphological analysis on the searched text, and acquires a noun as a related word among the analyzed morphemes.

  The connectivity determination unit 4 determines the connectivity of the plurality of sentences obtained by decomposing the input text T1 using a keyword set that is a combination of the search word and the related word, and the connectivity is a predetermined threshold value. The input text T1 is divided by extracting semantic paragraphs that are higher sentences.

  The input unit 5 inputs text T1. The control unit 6 controls the entire text segmentation apparatus 100.

  The output unit 7 outputs the text divided by the connectivity determination unit 4.

  The decomposed sentence storage unit M1 stores the sentence divided by the text dividing unit 1.

  The search term storage unit M2 stores the search terms extracted by the search term extraction unit 2.

  The related word storage unit M3 stores the related words acquired by the related word acquisition unit 3.

  The semantic paragraph storage unit M4 stores the semantic paragraph extracted by the connectivity determination unit 4.

  A network NW1 is connected to the text segmentation device 100, and the web W1 can be accessed. The web W1 is text that is input as a search result to the input unit 5 of the text segmentation device 100. The display unit D1 is a device that outputs the control unit 6 through the output unit 7 and displays the result.

  Next, the operation of the text segmentation device 100 will be described.

  FIG. 2 is a diagram showing an outline of a processing procedure in the text segmentation apparatus 100.

  In S1, the text T1 to be text segmented is input. This text is “input text” T1. In S2, the text decomposition unit 1 divides the input text T1 into sentence units. In S3, nouns are extracted from the divided sentences, and the extracted nouns are used as search terms. In S4, a search is performed on the web using the search word, and the noun contained in the searched text T2 is used as a related word. In S5, a keyword set is created by combining the search term and the related word, and among the plurality of sentences obtained by disassembling the input text T1 based on the keyword set, sentences whose contents are coherent are defined. Extract as a paragraph. In S6, the text segmentation result is output.

  Next, the text segmentation processing procedure will be described more specifically.

  First, when the text T1 is input through the input unit 5, the control unit 6 calls the text decomposition unit 1, and the text decomposition unit 1 reads the text T1 character by character and cuts it out in sentence units to obtain N sentences. . Then, the control unit 6 stores the extracted N sentences in the decomposed sentence storage unit M1. Here, “sentence” is one sentence delimited by a period “.”.

  FIG. 3 is a diagram illustrating an example of the input text T1 in the first embodiment of the present invention.

  FIG. 4 is a diagram illustrating an example of sentences stored in the decomposed sentence storage unit M1 according to the first embodiment.

  The text T1 includes the first sentence 11, the second sentence 12, the third sentence 13, the fourth sentence 14, the fifth sentence 15, the sixth sentence 16, and the seventh sentence. 17, an eighth sentence 18, and a ninth sentence 19.

  The text decomposition unit 1 decomposes the text T1 shown in FIG. 3, and as shown in FIG. 4, nine sentences 11 to 19 decomposed into sentence units are generated and stored in the decomposed sentence storage unit M1.

  The number of sentences generated by the text decomposition unit 1 varies depending on the input text T1. In addition, when there are two or more sentences that are semantically continued (a plurality of sentences continues without using the phrase “.”), Or when there is an input error of the phrase “.”, A plurality of sentences are Treated as a sentence.

  Next, the control unit 6 causes the search word extraction unit 2 to extract a search word from each sentence stored in the decomposed text storage unit M1. Here, the “search word” is one or more nouns that are input when searching on the web. The search word extraction unit 2 performs morphological analysis on the first input sentence. Then, a plurality of words classified as nouns by morphological analysis are extracted as search terms. The control unit 6 stores the retrieved search term in the search term storage unit M2. However, just extracting a noun will also extract commonly used words (words that have no meaning as search terms), such as “year” and “time”. Therefore, an unnecessary word list L1 that is a list of unnecessary words such as “year” and “time” is created in advance, and nouns that are not registered in the unnecessary word list L1 are handled as search words.

  FIG. 5 is a diagram illustrating an example of the unnecessary word list L1 according to the first embodiment.

  Unnecessary words are registered in the unnecessary word list L1.

  The search terms stored in the search term storage unit M2 vary depending on the unnecessary word list L1.

Further, when performing a web search, it is preferable to perform an AND search with an appropriate number of words. Therefore, the number of words extracted as noun, is less than the threshold value S _T, the search word extraction unit 2, without extracting a search word, the search word storage unit M2 does not store the word nothing.

Conversely, when the number S of nouns is equal to or greater than the threshold T, T search terms are randomly selected from the S search terms and stored in the search term storage unit M2. When T = 10 and S _T = 2, when the search word extraction unit 2 extracts a search word, the search word is stored in the search word storage unit M2.

  FIG. 6 is a diagram illustrating an example of search terms stored in the search term storage unit M2 in the first embodiment.

  The search terms stored in the search term storage unit M2 are a search term 21 corresponding to the sentence 11, a search word 22 corresponding to the sentence 12, a search word 23 corresponding to the sentence 13, and a search corresponding to the sentence 14. A search word 25, a search word 25 corresponding to the sentence 15, a search word 26 corresponding to the sentence 16, a search word 27 corresponding to the sentence 17, a search word 28 corresponding to the sentence 18, and a search corresponding to the sentence 19. Word 29.

  After creating the search terms 21 to 29 corresponding to the sentences 11 to 18, the control unit 6 calls the related term acquisition unit 3. The search term extracted by the search term extraction unit 2 is extracted from the search term storage unit M2 and input to the related term acquisition unit 3 via the control unit 6.

  An AND search is performed on the web W1 connected via the network NW1, using the search word input to the related word acquisition unit 3. By performing an AND search, it is possible to search the web W1 for texts that contain all of the search terms. Here, when a search is performed on the web W1, the search result does not affect the order in which the search terms are input.

  Then, P pieces of text T2 are acquired from the searched texts in the order of small difference between the time when the searched text was created and the time when the input text T1 was created. Since the date on which the text W1 is created is normally recorded in the text existing on the web W1, a time difference from the date on which the text T1 is created can be obtained. By acquiring the text with a small time difference, articles (text) more strongly related to the text T1 can be collected from the web W1.

  Here, if the search word does not exist in the search word storage unit M2, the related word acquisition unit 3 does not perform web search and stores nothing in the related word storage unit M3. Further, when the number S of search words is the same as T, the web search is not performed and the related words are not stored in the related word storage unit M3.

  Next, body text is extracted from the P texts T2 collected in time order. Then, the body text is extracted from the P texts T2 collected in time order. Since the text T2 is described in a structured language such as HTML or XML, a tag indicating the location of the body in the text T2 (the tags are “<”, “>”, and a character string surrounded by these. Is made up of). Therefore, the text of the body can be extracted from the text T2 by analyzing the tag for the obtained text T2.

  Then, the related word acquisition unit 3 performs morphological analysis on the extracted text of the body, and extracts nouns. A noun included in the text T2 searched for on the web is a related word. The number of related terms varies depending on the search terms when the web search is performed and the number of collected texts T2.

  Moreover, when all the nouns extracted by morphological analysis are used as related words, as in the case where the search word extraction unit 2 extracts, universally used words such as “year” and “month” are used as related words. May be treated. Therefore, the related word acquisition unit 3 refers to the unnecessary word list L1 and removes unnecessary words, similarly to the processing in the search word extraction unit 2. When the number of search words is S, among the words extracted from the P body texts, TS words are set as related words in descending order of appearance frequency.

  T is a number given in advance. S is the number of search terms. The TS related words are stored in the related word storage unit M3 via the control unit 6. That is, the total number of search words and related words extracted in each sentence is set to a value T given in advance. In this way, the total number of search words and related words extracted in each sentence is set to a predetermined value T in order to evaluate the connectivity equally for any sentence.

In order to obtain appropriate related terms, it is desirable that the number of texts T2 obtained by web search is as large as possible. Therefore, if the number P of the texts T2 obtained by the web search is less than the target number P _T , the search word is corrected, the AND search is performed again on the web, and the text T2 is collected.

  If there are S search terms, a combination of S-1 search terms is made, search is performed with the search terms of each combination, and P texts T2 are searched again with the search term combination with the largest number of searches, collect. For example, if the search terms are “Mount Fuji”, “Drive”, and “Sports”, S = 3, and if S−1 = 2 search terms are selected, a combination of “Drive” and “Sports” A combination of “Mt. Fuji”, “Sport”, “Mt. Fuji”, and “Drive” can be combined, for a total of three combinations. An AND search is performed again on the web W1 using the combination of these three that has the largest number of searches.

Then, the correction of the search word and the web search are repeated until the number P of the text T2 becomes _{equal to} or greater than _PT , and when P ≧ P _T , TS related words are extracted from the text T2.

If the number of collected texts T2 does not exceed P _T even if the search terms are corrected, the original S search terms are left in the search term storage unit M2, and the related term storage unit M3 contains related items. Do not store anything as a word.

For the search terms 21 to 29 shown in FIG. 6, assuming that a predetermined value T = 10, the number of searched texts T2 P = 20, and the target number of searches for text T2 by web search P _T = 20, The related word acquisition unit 3 acquires related words.

  FIG. 7 is a diagram illustrating an example of related words stored in the related word storage unit M3 in the first embodiment.

  The related words stored in the related word storage unit M3 shown in FIG. 7 are related words 31 corresponding to the search word 21, related words 32 corresponding to the search word 22, and related words 33 corresponding to the search word 23. , A related word 34 corresponding to the search word 24, a related word 35 corresponding to the search word 25, a related word 36 corresponding to the search word 26, a related word 37 corresponding to the search word 27, and a search word 28 Related words 38 and related words 39 corresponding to the search word 29.

  When the search word extraction by the search word extraction unit 2 and the related word acquisition by the related word acquisition unit 3 are completed for all sentences stored in the decomposed text storage unit M1, the control unit 6 connects The sex determination unit 4 determines the connectivity of the divided sentences. That is, the connectivity determination unit 4 reads the search word stored in the search word storage unit M2 and the related word stored in the related word storage unit M3 via the control unit 6, and combines them. To create a keyword set. That is, the keyword set is a combination of a search word and a related word.

  FIG. 8 is a diagram illustrating an example of a keyword set created by the connectivity determination unit 4 in the first embodiment.

  The keyword sets shown in FIG. 8 are keyword sets 51, 52, 53, 54, 55, 56, 57, 58, 59. The keyword set 51 is a keyword set generated by a set of the search word 21 and the related word 31. The keyword set 52 is a keyword set generated by a set of the search word 22 and the related word 32. The keyword set 53 is a keyword set generated by a set of the search word 23 and the related word 33.

  The keyword set 54 is a keyword set generated by a set of the search word 24 and the related word 34. The keyword set 55 is a keyword set generated by a set of the search word 25 and the related word 35. The keyword set 56 is a keyword set generated by a set of the search word 26 and the related word 36. The keyword set 57 is a keyword set generated by a set of search terms 27 and related terms 37. The keyword set 58 is a keyword set generated by a combination of the search word 28 and the related word 38. The keyword set 59 is a keyword set generated by a set of the search word 29 and the related word 39.

  If there is no search word, there is no related word corresponding to the search word, so the connectivity determination unit 4 does not create a keyword set. If there is a search word but no related word, a keyword set is created using only the search word.

  The keyword set is a word that reflects the content of the text. In the text T1, the words included in each keyword set change from the first sentence 11 to the sentences 12, 13,..., And by examining these words, Capturing changes in content.

  Therefore, the connectivity determination unit 4 compares the generated keyword sets with each other, and detects one sentence or a plurality of sentences that are grouped (similar) in content, that is, the detected contents. Extract semantic paragraphs that are composed of one or more sentences. The extracted semantic paragraph is stored in the semantic paragraph storage unit M4 through the control unit 6.

  When comparing keyword sets with each other, it is common for text to be written in order from the beginning, so a plurality of keyword sets are compared with each other in order from the beginning of the text. Specifically, with reference to the i-th keyword set, the block B1 including the i + 1-b-th (1 ≦ b) to i-th keyword sets and the i + 1-th to i + b-th keyword sets are included. Block B2 is created. Here, b is the number of keyword sets included in a block (a block having at least one sentence), that is, the width of the block.

  That is, the contents of the body are analyzed by looking at a keyword set corresponding to a plurality of sentences before and after the i-th sentence that is a reference sentence.

  Here, when the blocks B1 and B2 are created, a keyword set in which no word exists is not included in the block. Instead of doing this, a sentence number is added to a variable H that stores information on an empty keyword set. For example, in FIG. 8, since the keyword set for the second sentence is empty, the number of the second sentence is substituted into the variable H, so that H = {2}.

After creating the two blocks B1 and B2, the frequency w _{t of the} word t included in the keyword set is calculated for each block. Then, after obtaining the frequencies of all the words included in the two blocks, the degree of connectivity (degree of similarity) C _i ^b between the i-th sentence and the i + 1-th sentence is expressed as the frequency w of the word t. Evaluation is made by the following formula (1) using _t .

ｗ_ｔ ^Ｂ１は、ブロックＢ１における単語ｔの出現頻度であり、ｗ_ｔ ^Ｂ２は、ブロックＢ２における単語ｔの出現頻度である。また、ｆ^Ｂ１、ｆ^Ｂ２は、それぞれブロックＢ１、Ｂ２に含まれている空でないキーワード集合の個数である。ｉ番目の文とｉ＋１番目の文との連結度Ｃ_ｉ ^ｂは、０以上１以下の値を取り、１に近いほど、ブロックＢ１とＢ２とに含まれている単語が同じである。つまり、連結度Ｃ_ｉ ^ｂが１に近いほど、ブロックＢ１とＢ２とに含まれている単語のうちで、互いに同じ単語の割合が多い。共通単語抽出部３１は、ｉ＝｛１，２，…，Ｎ｝と変化させ、Ｃ_１ ^ｂ，Ｃ_２ ^ｂ，…，Ｃ_Ｎ ^ｂを計算する。さらに、ブロックの幅ｂが

w _t ^B1 is the appearance frequency of the word t in the block B1, and w _t ^B2 is the appearance frequency of the word t in the block B2. F ^B1 and f ^B2 are the numbers of non-empty keyword sets included in the blocks B1 and B2, respectively. The connectivity C _i ^b between the i-th sentence and the i + 1-th sentence takes a value of 0 or more and 1 or less, and the closer to 1, the more words are included in the blocks B1 and B2. In other words, the closer the degree of connectivity C _i ^b is to 1, the larger the percentage of the same word among the words included in the blocks B1 and B2. Common word extraction unit 31, i = {1,2, ..., N} is ^{_{changed, C 1 b, C 2 b}} , ..., to calculate the _C ^{N b.} Furthermore, the width b of the block is

であると設定し、各ブロック幅に対して、連結度Ｃ_ｉ ^ｂを計算し、この計算された平均値が、ｉ番目の文と、ｉ＋１番目の文とにおける平均連結度Ｃ_ｉであるとする。

The connectivity C _i ^b is calculated for each block width, and the calculated average value is the average connectivity C _i in the i-th sentence and the i + 1-th sentence. To do.

＃ｂは、ブロック幅のパラメータｂの個数であり、パラメータｂに含まれている要素数である。たとえば、Ｎ＝１０である場合、式（２）に記載されているｂ_ｍａｘの条件式から、ｂ_ｍａｘ＝２である。この結果、上記「数２」に示す式から、ｂ＝｛１，２，４｝であり、パラメータｂには、３個のパラメータが設定される。したがって、＃ｂ＝３になる。

#B is the number of parameters b of the block width, and is the number of elements included in the parameter b. For example, if it is N = 10, the condition of _{b max} listed in equation _(2), a _{b max} = 2. As a result, b = {1, 2, 4} from the equation shown in the above “Expression 2”, and three parameters are set as the parameter b. Therefore, # b = 3.

Here, in each of the blocks B1 and B2, when the number of non-empty keyword sets included in the block is less than b, the frequency w _t ^B is adjusted by the weight (b / f ^B ). . Specifically, for example, when i = 1 and b = 2, the block B1 includes only one non-empty keyword set, so the word frequency is doubled.

The average connectivity C _i (where i = {1, 2,..., N}) is used to check whether or not the average connectivity exceeds the threshold C _T in order from C ₁ and is described in the input text T1. Analyze changes in the contents. Specifically, if the average connectivity C _i is equal to or greater than the threshold C _T , it is determined that the contents of the i-th sentence and the i + 1-th sentence are the same, and the sentence number i is added to the variable G. To do. On the other hand, if it is less than the threshold value C _T, in the i-th sentence, the i + 1 th sentence, it is determined that the content has changed.

  At that time, the sentence number stored in the variable G is stored in the semantic paragraph storage unit M4 via the control unit 6 as a sentence belonging to the jth semantic paragraph. If there is a sentence number included in the variable H within the range of the j-th semantic paragraph, the sentence number is also assigned as the j-th semantic paragraph, Store in the storage unit M4.

Then, the contents of the variable G, initializing, the i + 1 th average connectivity as j = j + 1, again starts the processing, examined until the value of the average connectivity C _N, the information meaning paragraph, means paragraph storage unit When stored in M4, the connectivity determination unit 4 ends the comparison process.

  Here, the semantic paragraph number j, the variable G, and the variable H are initialized when the connectivity determination unit 4 is executed.

  FIG. 9 is a diagram illustrating an example of semantic paragraphs stored in the semantic paragraph storage unit M4 in the first embodiment.

That is, FIG. 9 is a diagram illustrating a result of the connectivity determination unit 4 determining connectivity to the keyword sets 51 to 59 illustrated in FIG. 8 with C _T = 0.1. In FIG. 9, reference numeral 41 indicates the number of a sentence belonging to the first semantic paragraph (meaning paragraph of paragraph number 1) detected by the connectivity determination unit 4. What is indicated by reference numeral 42 is the number of a sentence belonging to the second semantic paragraph (the semantic paragraph of paragraph number 2) detected by the connectivity determination unit 4. Here, the second keyword set is empty, but since the second sentence is within the range of the first semantic paragraph, it is added to the first semantic paragraph. On the other hand, each sentence in the variable H that is not assigned to any semantic paragraph is treated as one semantic paragraph.

  When the connectivity determination unit 4 ends the process, the control unit 6 calls the output unit 7 and displays the result of text segmentation for the text T1 on the display unit D1. Specifically, the semantic paragraph storage unit M4 is referred to and the sentence number stored in the semantic paragraph storage unit M4 is displayed. For example, sentence numbers indicated by reference numerals 41 and 42 may be displayed, and sentence numbers indicated by reference numerals 41 and 42 may be displayed together with paragraph numbers.

Parameter _{T, S T, P, P} T, by changing the _{C T,} it is possible to adjust the calculation time and accuracy in the above embodiment. That is, by adjusting the parameter T and S _T , P _T , and P, it is possible to adjust the extraction accuracy and calculation time of the related words obtained by the web search. Further, the parameter C _T, the susceptibility to capture changes in the contents of text, can be adjusted.

  In addition, it may be difficult to grasp the semantic content of the sentence corresponding to the sentence number assigned to the variable H. Thus, by ignoring the sentence number assigned to the variable H, it is possible to perform text segmentation for only sentences that are clear in terms of content.

  In the above embodiment, when web search is performed, the texts are collected in the order of little difference from the creation time of the input text T1, so that words that are highly relevant to the contents of the input text T1 can be collected. It should be noted that related words can be collected with a certain degree of accuracy by using the P texts T2 referred to in the obtained search results without considering the time difference.

If the number P of texts T2 obtained by web search is less than P _T , the search term is corrected and re-search is performed, so a keyword set consisting of a combination of the search term and the related term is assigned to each sentence. be able to.

Further, according to the above embodiment, when the number P of the texts T2 obtained by the web search is less than _PT , the keyword set may be created using only the search words without performing the re-search. In this case, since a plurality of predetermined keyword sets are taken into consideration before and after the reference sentence, the contents of the body can be analyzed and text segmented with practical accuracy in a short calculation time. .

  According to the above embodiment, since the concept of searching on the web is used in text segmentation, it is not necessary to prepare learning data in advance.

  Further, the above embodiment can be applied as a support measure for automatically managing / updating a database in the field of handling enormous text data and the field of distributing news articles.

  In addition, the above-described embodiment has an advantage that there are few restrictions on description contents and creation time because related words in a wide range of fields can be collected by web search without using learning data for the text to be analyzed.

  Further, since the above embodiment is divided into contents, it can be used for a technique for collecting only sentences that include predetermined keywords and are related in contents.

  Here, the web is a set of texts described in a structured language such as HTML or XML that can be accessed via a network such as the Internet. Currently, a huge amount of information is accumulated on the web, and the latest topics are always provided. In other words, the web can be understood as a dictionary having various information. In fact, when we look into a certain thing, we enter a search word on a search site, search the web, and look up the meaning of the word and the contents of things. From this point of view, the concept corresponding to “soccer” and “baseball” is “sports” and “ball” if the information on the web is used appropriately without using learning data. I can say that.

  As a result, it is possible to acquire words according to the description content of the text of interest from various information existing on the web, and compare the relevance of sentences widely and track the contents of the sentences. The contents can be divided into content groups (referred to as “meaning paragraphs” in the above embodiment). Therefore, it is not necessary to prepare learning data in advance, and text segmentation corresponding to various fields can be realized.

  In other words, in the above embodiment, a text decomposition unit that decomposes input text into sentence units, and a morphological analysis of a sentence decomposed by the text decomposition unit, and a noun is extracted as a search word from the analyzed morphemes. A search word extraction unit and a related word acquisition unit that performs a web search using the search words extracted by the search word extraction unit, analyzes the searched text, and acquires a noun as a related word among the analyzed morphemes And using a keyword set that is a combination of the search word and the related word, the connectivity of the plurality of sentences obtained by decomposing the input text is determined, and sentences whose connectivity is higher than a predetermined threshold A connectivity determination unit that divides the input text by extracting the meaning paragraph.

  In this case, the text segmentation apparatus is characterized in that the texts are collected in the order of small difference from the input text creation time.

  When the number of search terms is S, a combination of S-1 search terms is created, the search is performed using the search terms of each combination, and the text is searched and collected by the combination of search terms having the largest number of searches.

  In addition, the text segmentation device excludes the search unnecessary word list and the same search word candidate as the term included in the search unnecessary word list from the search word, and is included in the search unnecessary word list. Means for excluding the same related word candidate as the term from the related word.

  Further, a web segmentation method for performing a web search on the input text and dividing the input text according to a result of the web search, wherein the text decomposition unit decomposes the input text into sentence units, The text decomposition process stored in the storage device and the sentence decomposed in the text decomposition process are subjected to morphological analysis, and a noun is extracted as a search word from the analyzed morphemes and stored in the storage device. In accordance with the search word extraction process to be stored and the search word extracted in the search word extraction process, the related word acquisition unit performs a web search, analyzes the searched text, and among the analyzed morphemes , Using a keyword collection that is a combination of a related word acquisition step of acquiring a noun as a related word and storing it in a storage device, and the search word and the related word, The result determination unit determines connectivity between a plurality of the sentences obtained by decomposing the input text, and extracts a semantic paragraph that is a sentence having a connectivity higher than a predetermined threshold, thereby inputting the input text. Is a text segmentation method including a connectivity determination step of dividing.

  Text is collected in ascending order of difference from the input text creation time.

  In addition, when there are S search terms, a combination of S-1 search terms is created, the search is performed using the search terms of each combination, and the text is searched and collected by the search term combination having the largest number of searches. .

  Then, the search term candidates and the same search term candidates as the terms included in the search unnecessary word list are excluded from the search terms, and the same related word candidates as the terms included in the search unnecessary word list Are excluded from related terms.

  Moreover, the said Example is a program which makes a computer perform any one method of Claims 5-7.

Furthermore, the said Example is a computer-readable recording medium which recorded the program of Claim 8.

It is a block diagram which shows the text segmentation apparatus 100 which is Example 1 of this invention. 3 is a diagram showing an outline of a processing procedure in the text segmentation apparatus 100. FIG. In Example 1 of this invention, it is a figure which shows an example of the input text T1. It is a figure which shows an example of the sentence stored in the decomposition | disassembly text storage part M1 in Example 1. FIG. It is a figure which shows an example of the unnecessary word list | wrist L1 in Example 1. FIG. In Example 1, it is a figure which shows an example of the search word stored in the search word memory | storage part M2. In Example 1, it is a figure which shows an example of the related word stored in the related word memory | storage part M3. In Example 1, it is a figure which shows an example of the keyword set which the connectivity determination part 4 produced. In Example 1, it is a figure which shows an example of the meaning paragraph stored in the meaning paragraph memory | storage part M4.

Explanation of symbols

100 ... text segmentation device,
1 ... Text decomposition part,
2 ... Search term extraction unit,
3 ... related word acquisition part,
4 ... connectivity determination unit,
M1 ... decomposition sentence storage part,
M2 ... Search word storage unit,
M3 ... related word storage unit,
M4 ... semantic paragraph storage unit,
5 ... Input section,
6 ... control part,
7 ... Output section,
NW1 ... Network,
W1 ... Web,
T1 ... text input from the input unit,
T2: Text obtained from web search,
L1 ... unnecessary word list,
T ... Number given in advance,
S: Number of search terms.

Claims (10)

A text decomposition unit that decomposes the input text into sentence units;
A search word extraction unit that performs morphological analysis on the sentence decomposed by the text decomposition unit and extracts a noun as a search word from the analyzed morphemes;
A related word acquisition unit that performs a web search using the search terms extracted by the search term extraction unit, performs a morphological analysis on the searched text, and acquires a noun as a related word among the analyzed morphemes;
Using a keyword set that is a combination of the search word and the related word, the connectivity of the plurality of sentences obtained by decomposing the input text is determined, and the sentences having a connectivity higher than a predetermined threshold value. A connectivity determination unit that divides the input text by extracting a block;
A text segmentation device comprising: In claim 1,
A text segmentation device that collects text in the order of little difference from the input text creation time. In claim 1,
When there are S search terms, a combination of S-1 search terms is created, search is performed with the search terms of each combination, and text is searched and collected with the search term combination with the largest number of searches. Feature text segmentation device. In claim 1,
Search unnecessary word list;
The same search term candidate as the term included in the search unnecessary word list is excluded from the search term, and the same related word candidate as the term included in the search unnecessary word list is excluded from the related word. A text segmentation device. A text segmentation method that performs a web search on input text and divides the input text according to the result of the web search,
A text decomposition process in which the text decomposition unit decomposes the input text into sentence units and stores them in a storage device;
A search word extraction step in which a search word extraction unit analyzes the sentence decomposed in the text decomposition step, extracts a noun as a search word from the analyzed morphemes, and stores it in a storage device;
In accordance with the search word extracted in the search word extraction step, the related word acquisition unit performs a web search, analyzes the searched text, and acquires a noun as a related word from the analyzed morphemes. A related word acquisition step of storing in the storage device;
Using the keyword set that is a combination of the search term and the related word, the connectivity determination unit determines connectivity between the plurality of sentences obtained by decomposing the input text, and the connectivity is more than a predetermined threshold. A connectivity determination step of dividing the input text by extracting blocks that are higher sentences;
A text segmentation method characterized by comprising: In claim 5,
A text segmentation method characterized by collecting texts in the order of little difference from the input text creation time. In claim 5,
When there are S search terms, a combination of S-1 search terms is created, search is performed with the search terms of each combination, and text is searched and collected with the search term combination with the largest number of searches. A featured text segmentation method. In claim 5,
Search unnecessary word list;
The same search term candidate as the term included in the search unnecessary word list is excluded from the search term, and the same related word candidate as the term included in the search unnecessary word list is excluded from the related word. A text segmentation method characterized by   The program which makes a computer perform any one method of Claim 5-7.   A computer-readable recording medium on which the program according to claim 8 is recorded.

Images