CN107357777A - The method and apparatus for extracting label information - Google Patents
The method and apparatus for extracting label information Download PDFInfo
- Publication number
- CN107357777A CN107357777A CN201710459445.XA CN201710459445A CN107357777A CN 107357777 A CN107357777 A CN 107357777A CN 201710459445 A CN201710459445 A CN 201710459445A CN 107357777 A CN107357777 A CN 107357777A
- Authority
- CN
- China
- Prior art keywords
- candidate
- candidate phrase
- keyword
- score
- sentence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 239000000203 mixture Substances 0.000 claims abstract description 6
- 230000011218 segmentation Effects 0.000 claims description 15
- 238000012545 processing Methods 0.000 description 9
- 239000000284 extract Substances 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 239000013585 weight reducing agent Substances 0.000 description 2
- 238000012937 correction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
- G06F16/353—Clustering; Classification into predefined classes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Machine Translation (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method and apparatus for extracting label information, belong to Internet technical field.Method includes：Text message is segmented, obtains candidate phrase set, candidate phrase set includes at least one candidate phrase, and each candidate phrase includes at least one keyword；For each candidate phrase in candidate phrase set, the scoring of the position of sentence where determining the scoring of candidate phrase place sentence and determining the candidate phrase, and determine the first scoring of each keyword that the candidate phrase includes, first scoring of each keyword that the scoring of the position of sentence and the candidate phrase where the scoring of sentence, the candidate phrase according to where the candidate phrase include, determines the scoring of the candidate phrase；Based on the scoring of each candidate phrase, the selection scoring highest preset number candidate phrase from candidate phrase set；By the label information of preset number candidate phrase composition text information.The present invention improves the accuracy of label information.
Description
Technical Field
The invention relates to the technical field of internet, in particular to a method and a device for extracting label information.
Background
With the development of internet technology, the number of network citizens in China is increasing at a high speed, and text information on the network is also increasing explosively. However, the amount of information of the text information on the network is redundant, and the user can only read the entire content of the text information to know the subject of the text information. From the perspective of the user, the user can quickly select the interested text information to save much reading time. Therefore, the server needs to extract the tag information from the text information, and the user selects the text information which is interested in the user according to the tag information to read, so that the reading efficiency is improved.
In the prior art, when a server extracts tag information from text information, the server extracts a special type word corresponding to a regular expression from the text information through the regular expression. The server carries out word segmentation processing on the text information to obtain a first keyword set; and randomly combining the first keywords in the first keyword set to obtain a second keyword set. And the server screens each second keyword in the second keyword set based on a preset rule to obtain a third keyword set, and adds a special type word corresponding to the regular expression as the third keyword to the third keyword set. The server calculates the characteristic value of each third key word in the third key word set; and calculating the score of each third key word in the third key word set based on the characteristic value of each third key word in the third key word set. And the server extracts target keywords from the third keyword set based on the score of each third keyword in the third keyword set, and forms the extracted target keywords into label information.
In the process of implementing the invention, the inventor finds that the prior art has at least the following problems:
in the method, the third key words are scored based on the characteristic values of the third key words, however, some third key words which are not semantically recognized have high scores with a high probability, so that the extracted label information cannot accurately express the gist of the text information, and the accuracy is low.
Disclosure of Invention
In order to solve the problems of the prior art, the invention provides a method and a device for extracting tag information. The technical scheme is as follows:
in a first aspect, the present invention provides a method for extracting tag information, the method including:
segmenting words of text information to obtain a candidate phrase set, wherein the candidate phrase set comprises at least one candidate phrase, and each candidate phrase comprises at least one keyword;
for each candidate phrase in the candidate phrase set, determining the score of the sentence in which the candidate phrase is located, determining the score of the position of the sentence in which the candidate phrase is located, determining the first score of each keyword included in the candidate phrase, and determining the score of the candidate phrase according to the score of the sentence in which the candidate phrase is located, the score of the position of the sentence in which the candidate phrase is located and the first score of each keyword included in the candidate phrase;
selecting a preset number of candidate phrases with highest scores from the candidate phrase set based on the score of each candidate phrase;
and forming the label information of the text information by the preset number of candidate phrases.
In one possible implementation, the determining the first score of each keyword included in the candidate phrase includes:
determining a first occurrence number and a second occurrence number for each keyword, wherein the first occurrence number is the occurrence number of the keyword in the text information, and the second occurrence number is the total occurrence number of each keyword included in the text information;
determining the word frequency of the keyword according to the first occurrence frequency and the second occurrence frequency;
determining a first quantity and a second quantity, wherein the first quantity is the quantity of sample text information included in a sample text information base, and the second quantity is the quantity of text information including the keywords in the sample text information base;
determining the reverse file frequency of the keywords according to the first quantity and the second quantity;
and determining a first score of the keyword according to the word frequency and the reverse file frequency.
In one possible implementation manner, the determining the score of the position of the sentence in which the candidate phrase is located includes:
determining a first position of a paragraph of a sentence in which the candidate phrase is located in the text information and a second position of the candidate phrase in the paragraph;
and determining the grade of the position of the sentence in which the candidate phrase is positioned according to the first position and the second position.
In one possible implementation manner, the determining the score of the candidate phrase according to the score of the sentence in which the candidate phrase is located, the score of the position of the sentence in which the candidate phrase is located, and the first score of each keyword included in the candidate phrase includes:
determining a first weight corresponding to a sentence in which the candidate phrase is located, a second weight corresponding to the position of the sentence in which the candidate phrase is located, and a third weight corresponding to each keyword included in the candidate phrase;
for each keyword included in the candidate phrase, multiplying the score of the sentence in which the candidate phrase is located by the first weight to obtain a first numerical value, multiplying the score of the position of the sentence in which the candidate phrase is located by the second weight to obtain a second numerical value, multiplying the first score of the keyword by a third weight corresponding to the keyword to obtain a third numerical value, and adding the first numerical value, the second numerical value and the third numerical value to obtain a second score of the keyword;
and determining the score of the candidate phrase according to the second score of each keyword.
In a possible implementation manner, the adding the first numerical value, the second numerical value, and the third numerical value to obtain a second score of the keyword includes:
determining the contribution degree of the keyword, and determining a fourth weight corresponding to the keyword according to the contribution degree of the keyword;
adding the first numerical value, the second numerical value and the third numerical value to obtain a fourth numerical value;
and multiplying the fourth numerical value and the fourth weight to obtain a second score of the keyword.
In one possible implementation manner, the forming the preset number of candidate phrases into the tag information of the text information includes:
selecting candidate phrases of concept types from the preset number of candidate phrases to form concept label information; and/or the presence of a gas in the gas,
and selecting candidate phrases of the event type from the preset number of candidate phrases to form event tag information.
In one possible implementation, the method further includes:
moving candidate phrases ending with keywords with preset parts of speech in the concept label information to the event label information;
and moving the candidate phrases which do not contain the keywords of the preset part of speech in the event label information to the concept label information.
In a possible implementation manner, the segmenting the text information to obtain a candidate phrase set includes:
sentence breaking is carried out on the text information to obtain at least one candidate sentence, and the at least one candidate sentence is formed into a candidate sentence set;
segmenting each candidate sentence in the candidate sentence set to obtain at least one keyword, and forming the at least one keyword into a keyword set;
generating at least one candidate phrase from the keywords in the keyword set based on a syntax tree algorithm;
and forming the at least one candidate phrase into the candidate phrase set.
In a possible implementation manner, before performing word segmentation on each candidate sentence in the candidate sentence set to obtain at least one keyword, the method further includes:
determining a sentence component of each candidate sentence in the set of candidate sentences;
and deleting the candidate sentences of which the sentence components are preset components in the candidate sentence set according to the sentence components of each candidate sentence.
In a second aspect, the present invention provides an apparatus for extracting tag information, the apparatus comprising:
the word segmentation module is used for segmenting words of the text information to obtain a candidate phrase set, wherein the candidate phrase set comprises at least one candidate phrase, and each candidate phrase comprises at least one keyword;
the scoring module is used for determining the score of a sentence where the candidate phrase is located and the score of the position of the sentence where the candidate phrase is located and the first score of each keyword included in the candidate phrase for each candidate phrase in the candidate phrase set, and determining the score of the candidate phrase according to the score of the sentence where the candidate phrase is located, the score of the position of the sentence where the candidate phrase is located and the first score of each keyword included in the candidate phrase;
a selection module, configured to select a preset number of candidate phrases with highest scores from the candidate phrase set based on the score of each candidate phrase;
and the composition module is used for composing the preset number of candidate phrases into label information of the text information.
In a possible implementation manner, the scoring module is further configured to determine, for each keyword, a first occurrence number and a second occurrence number, where the first occurrence number is the occurrence number of the keyword in the text information, and the second occurrence number is a total occurrence number of each keyword included in the text information; determining the word frequency of the keyword according to the first occurrence frequency and the second occurrence frequency; determining a first quantity and a second quantity, wherein the first quantity is the quantity of sample text information included in a sample text information base, and the second quantity is the quantity of text information including the keywords in the sample text information base; determining the reverse file frequency of the keywords according to the first quantity and the second quantity; and determining a first score of the keyword according to the word frequency and the reverse file frequency.
In one possible implementation manner, the scoring module is further configured to determine a first position of a paragraph in the text information where the sentence in which the candidate phrase is located, and a second position of the candidate phrase in the paragraph; and determining the grade of the position of the sentence in which the candidate phrase is positioned according to the first position and the second position.
In a possible implementation manner, the scoring module is further configured to determine a first weight corresponding to a sentence in which the candidate phrase is located, a second weight corresponding to a position of the sentence in which the candidate phrase is located, and a third weight corresponding to each keyword included in the candidate phrase; for each keyword included in the candidate phrase, multiplying the score of the sentence in which the candidate phrase is located by the first weight to obtain a first numerical value, multiplying the score of the position of the sentence in which the candidate phrase is located by the second weight to obtain a second numerical value, multiplying the first score of the keyword by a third weight corresponding to the keyword to obtain a third numerical value, and adding the first numerical value, the second numerical value and the third numerical value to obtain a second score of the keyword; and determining the score of the candidate phrase according to the second score of each keyword.
In a possible implementation manner, the scoring module is further configured to determine a contribution degree of the keyword, and determine a fourth weight corresponding to the keyword according to the contribution degree of the keyword; adding the first numerical value, the second numerical value and the third numerical value to obtain a fourth numerical value; and multiplying the fourth numerical value and the fourth weight to obtain a second score of the keyword.
In a possible implementation manner, the composition module is further configured to select candidate phrases of concept types from the preset number of candidate phrases to compose concept tag information; and selecting candidate phrases of the event type from the preset number of candidate phrases to form event tag information.
In one possible implementation, the apparatus further includes:
a moving module, configured to move a candidate phrase ending with a keyword with a preset part of speech in the concept tag information to the event tag information; and/or the presence of a gas in the gas,
the moving module is further configured to move a candidate phrase, which does not include the keyword of the preset part of speech, in the event tag information to the concept tag information.
In a possible implementation manner, the word segmentation module is further configured to perform sentence segmentation on the text information to obtain at least one candidate sentence, and form the at least one candidate sentence into a candidate sentence set; segmenting each candidate sentence in the candidate sentence set to obtain at least one keyword, and forming the at least one keyword into a keyword set; generating at least one candidate phrase from the keywords in the keyword set based on a syntax tree algorithm; and forming the at least one candidate phrase into the candidate phrase set.
In one possible implementation, the word segmentation module is further configured to determine a sentence component of each candidate sentence in the candidate sentence set; and deleting the candidate sentences of which the sentence components are preset components in the candidate sentence set according to the sentence components of each candidate sentence.
In the embodiment of the invention, the text information is segmented to obtain a candidate phrase set, the candidate phrase set comprises at least one candidate phrase, and label information is extracted based on the candidate phrases in the candidate phrase set, so that multi-element label information can be extracted. And for each candidate phrase in the candidate phrase set, determining the score of the sentence in which the candidate phrase is positioned, determining the score of the position of the sentence in which the candidate phrase is positioned, determining the first score of each keyword included in the candidate phrase, and determining the score of the candidate phrase according to the score of the sentence in which the candidate phrase is positioned, the score of the position of the sentence in which the candidate phrase is positioned and the first score of each keyword included in the candidate phrase. Due to the combination of sentence scoring, position scoring and keyword scoring, the scoring accuracy of the candidate phrases is improved, and the accuracy of extracting the tag information is improved.
Drawings
Fig. 1 is a flowchart of a method for extracting tag information according to an embodiment of the present invention;
fig. 2 is a flowchart of a method for extracting tag information according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an apparatus for extracting tag information according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a server for extracting tag information according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The embodiment of the invention provides a method for extracting tag information, and referring to fig. 1, the method comprises the following steps:
step 101: the method comprises the steps of segmenting words of text information to obtain a candidate phrase set, wherein the candidate phrase set comprises at least one candidate phrase, and each candidate phrase comprises at least one keyword.
Step 102: for each candidate phrase in the candidate phrase set, determining the score of the sentence in which the candidate phrase is located, determining the score of the position of the sentence in which the candidate phrase is located, determining the first score of each keyword included in the candidate phrase, and determining the score of the candidate phrase according to the score of the sentence in which the candidate phrase is located, the score of the position of the sentence in which the candidate phrase is located and the first score of each keyword included in the candidate phrase.
Step 103: based on the score of each candidate phrase, a preset number of candidate phrases with the highest scores are selected from the set of candidate phrases.
Step 104: and forming the label information of the text information by the preset number of candidate phrases.
In one possible implementation, determining a first score for each keyword included in the candidate phrase includes:
for each keyword, determining a first occurrence number and a second occurrence number, wherein the first occurrence number is the occurrence number of the keyword in the text information, and the second occurrence number is the total occurrence number of each keyword included in the text information;
determining the word frequency of the keyword according to the first occurrence frequency and the second occurrence frequency;
determining a first quantity and a second quantity, wherein the first quantity is the quantity of sample text information included in a sample text information base, and the second quantity is the quantity of text information including the keyword in the sample text information base;
determining the reverse file frequency of the keyword according to the first quantity and the second quantity;
and determining a first score of the keyword according to the word frequency and the reverse file frequency.
In one possible implementation, determining a score of a position of a sentence in which the candidate phrase is located includes:
determining a first position of a paragraph in which the sentence of the candidate phrase is located in the text information and a second position of the candidate phrase in the paragraph;
and determining the score of the position of the sentence in which the candidate phrase is positioned according to the first position and the second position.
In one possible implementation manner, determining the score of the candidate phrase according to the score of the sentence in which the candidate phrase is located, the score of the position of the sentence in which the candidate phrase is located, and the first score of each keyword included in the candidate phrase includes:
determining a first weight corresponding to a sentence in which the candidate phrase is located, a second weight corresponding to the position of the sentence in which the candidate phrase is located, and a third weight corresponding to each keyword included in the candidate phrase;
for each keyword included in the candidate phrase, multiplying the score of the sentence in which the candidate phrase is located by the first weight to obtain a first numerical value, multiplying the score of the position of the sentence in which the candidate phrase is located by the second weight to obtain a second numerical value, multiplying the first score of the keyword by a third weight corresponding to the keyword to obtain a third numerical value, and adding the first numerical value, the second numerical value and the third numerical value to obtain a second score of the keyword;
and determining the score of the candidate phrase according to the second score of each keyword.
In one possible implementation manner, adding the first value, the second value, and the third value to obtain a second score of the keyword, includes:
determining the contribution degree of the keyword, and determining a fourth weight corresponding to the keyword according to the contribution degree of the keyword;
adding the first numerical value, the second numerical value and the third numerical value to obtain a fourth numerical value;
and multiplying the fourth numerical value and the fourth weight to obtain a second score of the keyword.
In one possible implementation manner, the forming the preset number of candidate phrases into the tag information of the text information includes:
selecting candidate phrases of concept types from the preset number of candidate phrases to form concept label information; and/or the presence of a gas in the gas,
selecting candidate phrases of the event type from the preset number of candidate phrases to form event tag information.
In one possible implementation, the method further includes:
moving candidate phrases ending with keywords with preset parts of speech in the concept label information to the event label information;
and moving the candidate phrases which do not contain the keywords of the preset part of speech in the event label information to the concept label information.
In one possible implementation, segmenting the text information to obtain a candidate phrase set includes:
carrying out sentence breaking on the text information to obtain at least one candidate sentence, and forming the at least one candidate sentence into a candidate sentence set;
segmenting each candidate sentence in the candidate sentence set to obtain at least one keyword, and forming the at least one keyword into a keyword set;
generating at least one candidate phrase from the keywords in the keyword set based on a syntactic tree algorithm;
at least one candidate phrase is grouped into the set of candidate phrases.
In a possible implementation manner, before performing word segmentation on each candidate sentence in the candidate sentence set to obtain at least one keyword, the method further includes:
determining a sentence component of each candidate sentence in the set of candidate sentences;
and deleting the candidate sentences of which the sentence components are preset components in the candidate sentence set according to the sentence components of each candidate sentence.
In the embodiment of the invention, the text information is segmented to obtain a candidate phrase set, the candidate phrase set comprises at least one candidate phrase, and label information is extracted based on the candidate phrases in the candidate phrase set, so that multi-element label information can be extracted. And for each candidate phrase in the candidate phrase set, determining the score of the sentence in which the candidate phrase is positioned, determining the score of the position of the sentence in which the candidate phrase is positioned, determining the first score of each keyword included in the candidate phrase, and determining the score of the candidate phrase according to the score of the sentence in which the candidate phrase is positioned, the score of the position of the sentence in which the candidate phrase is positioned and the first score of each keyword included in the candidate phrase. Due to the combination of sentence scoring, position scoring and keyword scoring, the scoring accuracy of the candidate phrases is improved, and the accuracy of extracting the tag information is improved.
The embodiment of the invention provides a method for extracting tag information, which is applied to a server and is shown in fig. 2, and the method comprises the following steps:
step 201: the server divides words of the text information to obtain a candidate phrase set, wherein the candidate phrase set comprises at least one candidate phrase, and each candidate phrase comprises at least one keyword.
In order to improve reading efficiency of a user, before the user acquires text information from a server through a terminal, the server extracts label information from the text information, wherein the label information is used for indicating the theme of the text information. When a user acquires text information from the server through the terminal, the server sends label information of the text information to the terminal. The terminal receives the label information of the text information sent by the server and displays the label information of the text information, so that the user can quickly know the gist of the text information according to the label information. The text information can be any text information including character information; for example, the text information may be electronic news information, social network information, commodity review information, web page information, mail information, and the like; in the embodiment of the present invention, the text information is not particularly limited.
When the server performs word segmentation on the text information to obtain a candidate phrase set, the server can perform word segmentation on the text information to obtain a keyword set, and based on the keyword set, a candidate phrase set is generated; accordingly, this step can be realized by the following steps (1) to (4), including:
(1): and the server breaks the sentence of the text information to obtain at least one candidate sentence, and the candidate sentence is formed into a candidate sentence set.
As the contribution of the main sentence is far greater than the effect of the clauses for understanding an article; moreover, when candidate phrases are extracted through a syntax tree algorithm subsequently, the extraction speed is greatly influenced by the length of the sentence, and in order to improve the operation speed of extracting the candidate phrases, the server filters out the number-of-words clauses; correspondingly, after the server composes the at least one candidate sentence into the candidate sentence set, the method further includes:
the server determines a sentence component of each candidate sentence in the candidate sentence set; and deleting the candidate sentences of which the sentence components are preset components in the candidate sentence set according to the sentence components of each candidate sentence.
The preset component may be a idiom or a fixed language, etc. In the embodiment of the present invention, the predetermined composition is not particularly limited.
(2): the server carries out word segmentation on each candidate sentence in the candidate sentence set to obtain at least one keyword, and the at least one keyword forms a keyword set.
Since the keywords such as "of", "bar", "o" have a small degree of contribution to the text information. Therefore, in order to reduce the amount of computation and improve the accuracy, the server may remove the keywords such as "the", "bar", and "o" in this step. Therefore, after the server makes at least one keyword into the keyword set, the method further includes:
the server marks the part of speech of each keyword in the keyword set; and searching keywords with a first preset part of speech from the keyword set according to the part of speech of each keyword, and removing the keywords with the first preset part of speech from the keyword set. The first preset part-of-speech keywords may be auxiliary words, prepositions, language-atmosphere words or numerals. In the embodiment of the present invention, the first predetermined part of speech is not specifically limited.
(3): and the server generates at least one candidate phrase from the keywords in the keyword set based on a syntactic tree algorithm.
The server inputs each keyword in the keyword set into a syntax tree model, wherein the syntax tree model comprises a syntax tree algorithm; and generating a keyword tree from the keywords in the keyword set through the syntax tree algorithm. The keyword tree comprises a plurality of nodes and relations among the plurality of nodes; one node in the keyword tree is a keyword. The server generates at least one candidate phrase based on the keyword tree.
Since the keyword tree includes keywords and relationships between the keywords; only one or more keywords having a relationship can constitute a candidate phrase. Correspondingly, the step of generating, by the server, at least one candidate phrase based on the keyword tree may be:
for each leaf node in the keyword tree, the server selects keywords of the leaf node and keywords of a parent node of the leaf node from the keyword tree to form a candidate phrase.
In this step, the server may also generate a candidate phrase according to the candidate keyword of the parent node of the leaf node, and correspondingly, the step of generating at least one candidate phrase by the server based on the keyword tree may be:
for each leaf node in the keyword tree, the server selects the keyword of the leaf node from the keyword tree, acquires the keyword of the father node of the leaf node, and acquires the keyword of the father node until acquiring the keyword of the root node; and forming the acquired keywords into candidate phrases.
In the embodiment of the invention, the candidate phrases are extracted through the syntax tree algorithm, so that the extracted candidate phrases are multiple, and the keywords included in the candidate phrases have strong semantic consistency, thereby improving the accuracy of candidate label information generation based on the candidate phrases.
(4): the server composes the at least one candidate phrase into a set of candidate phrases.
After the server obtains the set of candidate phrases, a score is determined for each candidate phrase in the set of candidate phrases by the following step 202.
Step 202: for each candidate phrase in the candidate phrase set, the server determines the score of the sentence in which the candidate phrase is located, the score of the position of the sentence in which the candidate phrase is located and the first score of each keyword included in the candidate phrase, and determines the score of the candidate phrase according to the score of the sentence in which the candidate phrase is located, the score of the position of the sentence in which the candidate phrase is located and the first score of each keyword included in the candidate phrase.
For each candidate phrase in the set of candidate phrases, the server may determine a score for the candidate phrase by the following steps (1) to (4).
(1): the server determines a score for the sentence in which the candidate phrase is located.
The server acquires the sentences of the candidate phrases; the score of the sentence is determined by the BM25 algorithm.
(2): the server determines a score for the position of the sentence in which the candidate phrase is located.
Because the grades of the sentences in different positions are different, for example, the sentences in the abstract can better reflect the gist of text information, and the sentences in the abstract correspond to higher grades; sentences in the body correspond to lower scores. Therefore, before this step, the correspondence between the position and the score is stored in the server. The location may be a summary or a body. Correspondingly, the steps can be as follows:
and the server determines the position of the sentence in which the candidate phrase is positioned in the text information, and acquires the score of the position of the sentence in which the candidate phrase is positioned from the corresponding relation between the position and the score according to the position.
In this step, the server may further calculate a score of the position of the sentence where the candidate phrase is located according to the position of the sentence where the candidate phrase is located. And accordingly. The method comprises the following steps:
the server determines a first position of a sentence where the candidate phrase is located in the text information, and a second position of the candidate phrase in the paragraph; and determining the score of the position of the sentence in which the candidate phrase is positioned according to the first position and the second position.
Wherein the first location may be a title, abstract or body text. The second position may be the first paragraph first sentence, the first paragraph is not the first sentence, the non-first paragraph first sentence or the non-first paragraph is not the first sentence. Correspondingly, the step of determining, by the server, the score of the position of the sentence in which the candidate phrase is located according to the first position and the second position may be:
the server acquires a first sub-score corresponding to the first position from the corresponding relation between the first position and the sub-score according to the first position; and according to the second position, acquiring a second sub-score corresponding to the second position from the corresponding relation between the second position and the sub-score, and multiplying the first sub-score and the second sub-score to obtain the score of the position of the sentence where the candidate phrase is located.
For example, when the first position is title, the first sub-score is 0.9; when the first position is the abstract, the first sub-score is 0.6; the first sub-score is 0.3 when the first position is text. When the second position is the first sentence of the first paragraph, the second sub-score is 0.3; when the second position is not the first sentence, the second sub-score is 0.3 x 0.4-0.12; when the second position is a non-first-segment first sentence, the second sub-score is 0.1; when the second position is a non-first sentence, the second sub-score is 0.1 × 0.4 ═ 0.04.
(3): the server determines a first score for each keyword that the candidate phrase includes.
For each keyword included in the candidate phrase, the server determines a first score for the keyword by the following steps (3-1) to (3-5), including:
(3-1): the server determines a first occurrence number and a second occurrence number, wherein the first occurrence number is the occurrence number of the keyword in the text information, and the second occurrence number is the total occurrence number of each keyword included in the text information.
(3-2): and the server determines the word frequency of the keyword according to the first occurrence frequency and the second occurrence frequency.
And the server determines the ratio of the first occurrence frequency to the second occurrence frequency as the word frequency of the keyword.
(3-3): the server determines a first quantity and a second quantity, wherein the first quantity is the quantity of the sample text information included in the sample text information base, and the second quantity is the quantity of the text information including the keyword in the sample text information base.
The server stores a sample text information base in advance, and the sample text information base comprises at least one sample text information. In this step, the server counts the number of sample text messages included in the sample text message library, which is referred to as a first number for convenience of description. The server counts the number of text messages including the keyword in the sample text information base, and the number is called a second number.
(3-4): and the server determines the reverse file frequency of the keyword according to the first quantity and the second quantity.
The server determines the reverse file frequency of the keyword according to the first quantity and the second quantity through the following formula I:
the formula I is as follows:wherein idf is the reverse file frequency of the keyword, D is a first number, and J is a second number.
Since there may be no text information in the sample text information base that includes the keyword, the second number may be zero. Therefore, in this step, the following steps may be performed:
the server determines the reverse file frequency of the keyword according to the first quantity and the second quantity through the following formula II:
the formula II is as follows:wherein idf is the reverse file frequency of the keyword, D is a first number, and J is a second number.
(3-4): and the server determines a first score of the keyword according to the word frequency and the reverse file frequency of the keyword.
And the server determines a first score of the keyword through a first preset algorithm according to the word frequency and the reverse file frequency of the keyword.
The first preset algorithm may be set and changed as needed, and in the embodiment of the present invention, the first preset algorithm is not specifically limited. For example, the first preset algorithm may be multiplication, addition, subtraction, division, weighted multiplication, or weighted division.
When the first predetermined algorithm is multiplication, the step may be:
and the server multiplies the word frequency of the keyword by the reverse file frequency to obtain a first score of the keyword.
(4): and the server determines the score of the candidate phrase according to the score of the sentence in which the candidate phrase is positioned, the score of the position of the sentence in which the candidate phrase is positioned and the first score of each keyword included in the candidate phrase.
In this step, the server may determine the score of the candidate phrase directly according to the score of the sentence in which the candidate phrase is located, the score of the position of the sentence in which the candidate phrase is located, and the first score of each keyword included in the candidate phrase, that is, the following first implementation manner. The server may also set a first weight, a second weight, and a third weight for the sentence where the candidate phrase is located, the position of the matrix where the candidate phrase is located, and each keyword included in the candidate phrase, respectively, and determine the score of the candidate phrase based on the score of the sentence where the candidate phrase is located, the score of the position of the sentence where the candidate phrase is located, the first score of each keyword included in the candidate phrase, and the first weight, the second weight, and the third weight, that is, the following second implementation manner.
For the first implementation, this step may be implemented by the following steps (4-1) to (4-2), including:
(4-1): for each keyword included in the candidate phrase, the server determines a second score of the keyword through a second preset algorithm according to the score of the sentence in which the candidate phrase is located, the score of the position of the sentence in which the candidate phrase is located and the first score of the keyword.
The second preset algorithm may be set and changed as needed, and in the embodiment of the present invention, the second preset algorithm is not specifically limited. For example, the second predetermined algorithm may be multiplication, addition, subtraction, division, weighted multiplication, or weighted division.
When the second predetermined algorithm is addition, the step may be:
and the server adds the score of the sentence in which the candidate phrase is positioned, the score of the position of the sentence in which the candidate phrase is positioned and the first score of the keyword to obtain a second score of the keyword.
It should be noted that the contribution degree of the single word to the understanding of the article is small. Therefore, when the server determines the second score of the keyword of the single word, the server performs the weight reduction processing on the keyword. Words with quotation marks that appear in the text information many times contribute greatly to understanding the article. Therefore, when the server determines that the second score of the keyword with quotation marks appears for a plurality of times, the server performs weighting processing on the keyword. Correspondingly, the steps can be as follows:
for each keyword included in the candidate phrase, the server determines the contribution degree of the keyword; determining a fourth weight corresponding to the keyword according to the contribution degree; obtaining a fifth numerical value through a second preset algorithm according to the score of the sentence where the candidate phrase is located, the score of the position of the sentence where the candidate phrase is located and the first score of the keyword; and multiplying the fifth numerical value by the fourth weight to obtain a second score of the keyword.
The server determines the contribution degree of the keyword according to the word number of the keyword and/or the importance degree of the keyword, and the specific process may be as follows:
the server takes the word number of the keyword as the contribution degree of the keyword; and, the more words, the higher the contribution. Or,
the server takes the importance of the keyword as the contribution of the keyword; and, the higher the importance, the higher the contribution. Or,
and the server performs weighting processing on the word number and the importance of the keyword to obtain the contribution degree of the keyword.
Wherein, the importance of the keyword can be used for representing the occurrence number of the keyword and/or representing the keyword by highlighting and the like.
Storing the corresponding relation between the contribution degree and the weight in the server; correspondingly, the step of determining, by the server, the fourth weight corresponding to the keyword according to the contribution degree may be:
and the server acquires a fourth weight corresponding to the keyword from the corresponding relation between the contribution degree and the weight according to the contribution degree.
(4-2): the server determines the score of the candidate phrase according to the second score of each keyword included in the candidate phrase.
And the server determines the score of the candidate phrase through a third preset algorithm according to the second score of each keyword included in the candidate phrase.
The third preset algorithm may be set and changed as needed, and in the embodiment of the present invention, the third preset algorithm is not specifically limited. For example, the third preset algorithm may be multiplication, addition, subtraction, division, weighted multiplication, weighted division, or maximum value.
When the third preset algorithm is addition, the step may be:
and the server adds the second scores of each keyword included in the candidate phrase to obtain the score of the candidate phrase.
When the third preset algorithm is to find the maximum value, the step may be:
the server selects the maximum score from the second scores of each keyword included in the candidate phrase, and takes the maximum score as the score of the candidate phrase.
For the second implementation, this step may be implemented by the following steps (4-a) to (4-c), including:
(4-a): the server determines a first weight corresponding to a sentence in which the candidate phrase is located, a second weight corresponding to the position of the sentence in which the candidate phrase is located, and a third weight corresponding to each keyword included in the candidate phrase.
The server prestores a first weight corresponding to a sentence where the candidate phrase is located and a second weight of the position of the sentence where the candidate phrase is located; in this step, the server obtains a first weight corresponding to a sentence where the stored candidate phrase is located and a second weight corresponding to a position of the sentence where the candidate phrase is located.
Storing the corresponding relation between each keyword and the third weight in the server; correspondingly, the step of the server obtaining the third weight corresponding to each keyword included in the candidate phrase may be:
and the server acquires the third weight corresponding to each keyword included in the candidate phrase from the corresponding relation between the keyword and the third weight according to each keyword included in the candidate phrase.
It should be noted that the third weights corresponding to each keyword may be the same or different. For example, the first weight a1 of the sentence in which the candidate phrase is located is 0.1, the second weight a2 of the sentence in which the candidate phrase is located is 0.55, and the third weight a3 of each keyword included in the candidate phrase is 0.35.
(4-b): for each keyword included in the candidate phrase, the server multiplies the score of the sentence in which the candidate phrase is located by the first weight to obtain a first numerical value, multiplies the score of the position of the sentence in which the candidate phrase is located by the second weight to obtain a second numerical value, multiplies the first score of the keyword by the third weight corresponding to the keyword to obtain a third numerical value, and adds the first numerical value, the second numerical value and the third numerical value to obtain a second score of the keyword.
Also, the contribution degree of the single word to the understanding of the article is small. Therefore, when the server determines the second score of the keyword of the single word, the server performs the weight reduction processing on the keyword. Words with quotation marks that appear in the text information many times contribute greatly to understanding the article. Therefore, when the server determines that the second score of the keyword with quotation marks appears for a plurality of times, the server performs weighting processing on the keyword. Correspondingly, the step of adding the first numerical value, the second numerical value and the third numerical value by the server to obtain the second score of the keyword may be:
the server determines the contribution degree of the keyword; determining a fourth weight corresponding to the keyword according to the contribution degree; and adding the first numerical value, the second numerical value and the third numerical value to obtain a fourth numerical value, and multiplying the fourth numerical value by a fourth weight to obtain a second score of the keyword.
(4-c): and the server determines the score of the candidate phrase according to the second score of each keyword.
This step is the same as step (4-2), and will not be described herein again.
Step 203: the server selects a preset number of candidate phrases with the highest scores from the candidate phrase set based on the score of each candidate phrase.
And the server sorts each candidate phrase according to the grade of the candidate phrase from high to low, and outputs a preset number of candidate phrases sorted at the top.
The preset number may be set and changed as needed, and in the embodiment of the present invention, the preset number is not specifically limited. For example, the preset number may be 8 or 10, etc.
In the embodiment of the present invention, since the candidate phrases may include keywords having no meaning, such as auxiliary words, prepositions, discourse words, and number words; therefore, after the server selects the preset number of candidate phrases, the server filters out keywords of a second preset part of speech in the preset number of candidate phrases.
The second preset part of speech and the first preset part of speech can be the same or different; this is not particularly limited in the embodiments of the present invention. For example, the keywords of the second predetermined part of speech may be auxiliary words, prepositions, ambiguities, or numerals.
In the embodiment of the invention, the server can set concept tag information and event tag information; the concept label information comprises the most core concept phrase of the text information, and the event label information comprises the core event in the text information. After step 203 is performed, the server generates conceptual tag information by the following step 204 and event tag information by the following step 205.
Step 204: the server selects candidate phrases of concept types from a plurality of preset candidate phrases to form concept label information.
Wherein the candidate phrase of the concept type refers to a candidate phrase containing a noun.
Step 205: the server selects candidate phrases of the event type from a preset number of candidate phrases to form event tag information.
Wherein the candidate phrase of the event type refers to a candidate phrase containing a verb.
It should be noted that step 204 and step 205 do not have a time sequence, and step 204 may be executed first, and then step 205 may be executed; step 205 may be performed first, and then step 204 may be performed.
Since the server may make an error in classifying the concept tag information and the event tag information; therefore, the server can also perform phrase correction, and the specific process can be as follows:
the server moves candidate phrases ending with keywords of a third preset part of speech in the concept label information to the event label information; and/or moving candidate phrases which do not contain the keywords of the third preset part of speech in the event label information into the concept label information.
The third preset part of speech may be set and changed as needed, and in the embodiment of the present invention, the third preset part of speech is not specifically limited. For example, the third preset part of speech may be a verb.
After the server extracts the label information, the server stores the corresponding relation between the identification of the text information and the label information of the text information. The terminal can acquire the label information from the server; the specific process can be as follows:
and the terminal sends an acquisition request to the server, wherein the acquisition request carries the identifier of the text information to be acquired. The server receives an acquisition request sent by the terminal, acquires the label information of the text information from the corresponding relation between the identification and the label information according to the identification of the text information, and sends the label information of the text information to the terminal. And the terminal receives the label information of the text information sent by the server and displays the label information of the text information. So that the user can quickly understand the subject matter of the text information based on the tag information of the text information. The identification of the text information may be a name, a URL, a storage path or a number of the text information, or the like.
In the embodiment of the present invention, tag information extracted by LDA (document topic Allocation) in the existing method is a unitary tag, whereas tag information extracted based on a syntax tree in the embodiment of the present invention is multi-element tag information, and concept tag information and event tag information are extracted.
In the embodiment of the invention, the text information is segmented to obtain a candidate phrase set, the candidate phrase set comprises at least one candidate phrase, and label information is extracted based on the candidate phrases in the candidate phrase set, so that multi-element label information can be extracted. And for each candidate phrase in the candidate phrase set, determining the score of the sentence in which the candidate phrase is positioned, determining the score of the position of the sentence in which the candidate phrase is positioned, determining the first score of each keyword included in the candidate phrase, and determining the score of the candidate phrase according to the score of the sentence in which the candidate phrase is positioned, the score of the position of the sentence in which the candidate phrase is positioned and the first score of each keyword included in the candidate phrase. Due to the combination of sentence scoring, position scoring and keyword scoring, the scoring accuracy of the candidate phrases is improved, and the accuracy of extracting the tag information is improved.
The invention provides a device for extracting tag information, which is applied to a server and used for executing the server in the method for extracting the tag information. Referring to fig. 3, the apparatus includes:
a word segmentation module 301, configured to segment words of text information to obtain a candidate phrase set, where the candidate phrase set includes at least one candidate phrase, and each candidate phrase includes at least one keyword;
a scoring module 302, configured to determine, for each candidate phrase in the candidate phrase set, a score of a sentence in which the candidate phrase is located, a score of a position of the sentence in which the candidate phrase is located, and a first score of each keyword included in the candidate phrase, and determine a score of the candidate phrase according to the score of the sentence in which the candidate phrase is located, the score of the position of the sentence in which the candidate phrase is located, and the first score of each keyword included in the candidate phrase;
a selecting module 303, configured to select, based on the score of each candidate phrase, a preset number of candidate phrases with the highest scores from the candidate phrase set;
a composing module 304, configured to compose the preset number of candidate phrases into tag information of the text information.
In a possible implementation manner, the scoring module 302 is further configured to determine, for each keyword, a first occurrence number and a second occurrence number, where the first occurrence number is the occurrence number of the keyword in the text information, and the second occurrence number is a total occurrence number of each keyword included in the text information; determining the word frequency of the keyword according to the first occurrence frequency and the second occurrence frequency; determining a first quantity and a second quantity, wherein the first quantity is the quantity of sample text information included in a sample text information base, and the second quantity is the quantity of text information including the keywords in the sample text information base; determining the reverse file frequency of the keywords according to the first quantity and the second quantity; and determining a first score of the keyword according to the word frequency and the reverse file frequency.
In a possible implementation manner, the scoring module 302 is further configured to determine a first position of a paragraph in the text information where the sentence in which the candidate phrase is located, and a second position of the candidate phrase in the paragraph; and determining the grade of the position of the sentence in which the candidate phrase is positioned according to the first position and the second position.
In a possible implementation manner, the scoring module 302 is further configured to determine a first weight corresponding to a sentence in which the candidate phrase is located, a second weight corresponding to a position of the sentence in which the candidate phrase is located, and a third weight corresponding to each keyword included in the candidate phrase; for each keyword included in the candidate phrase, multiplying the score of the sentence in which the candidate phrase is located by the first weight to obtain a first numerical value, multiplying the score of the position of the sentence in which the candidate phrase is located by the second weight to obtain a second numerical value, multiplying the first score of the keyword by a third weight corresponding to the keyword to obtain a third numerical value, and adding the first numerical value, the second numerical value and the third numerical value to obtain a second score of the keyword; and determining the score of the candidate phrase according to the second score of each keyword.
In a possible implementation manner, the scoring module 302 is further configured to determine a contribution degree of the keyword, and determine a fourth weight corresponding to the keyword according to the contribution degree of the keyword; adding the first numerical value, the second numerical value and the third numerical value to obtain a fourth numerical value; and multiplying the fourth numerical value and the fourth weight to obtain a second score of the keyword.
In a possible implementation manner, the composition module 304 is further configured to select candidate phrases of concept types from the preset number of candidate phrases to compose concept tag information; and selecting candidate phrases of the event type from the preset number of candidate phrases to form event tag information.
In one possible implementation, the apparatus further includes:
a moving module, configured to move a candidate phrase ending with a keyword with a preset part of speech in the concept tag information to the event tag information; and/or the presence of a gas in the gas,
the moving module is further configured to move a candidate phrase, which does not include the keyword of the preset part of speech, in the event tag information to the concept tag information.
In a possible implementation manner, the word segmentation module 301 is further configured to perform sentence segmentation on the text information to obtain at least one candidate sentence, and form the at least one candidate sentence into a candidate sentence set; segmenting each candidate sentence in the candidate sentence set to obtain at least one keyword, and forming the at least one keyword into a keyword set; generating at least one candidate phrase from the keywords in the keyword set based on a syntax tree algorithm; and forming the at least one candidate phrase into the candidate phrase set.
In one possible implementation, the word segmentation module 301 is further configured to determine a sentence component of each candidate sentence in the candidate sentence set; and deleting the candidate sentences of which the sentence components are preset components in the candidate sentence set according to the sentence components of each candidate sentence.
In the embodiment of the invention, the text information is segmented to obtain a candidate phrase set, the candidate phrase set comprises at least one candidate phrase, and label information is extracted based on the candidate phrases in the candidate phrase set, so that multi-element label information can be extracted. And for each candidate phrase in the candidate phrase set, determining the score of the sentence in which the candidate phrase is positioned, determining the score of the position of the sentence in which the candidate phrase is positioned, determining the first score of each keyword included in the candidate phrase, and determining the score of the candidate phrase according to the score of the sentence in which the candidate phrase is positioned, the score of the position of the sentence in which the candidate phrase is positioned and the first score of each keyword included in the candidate phrase. Due to the combination of sentence scoring, position scoring and keyword scoring, the scoring accuracy of the candidate phrases is improved, and the accuracy of extracting the tag information is improved.
It should be noted that: in the above embodiment, when extracting the tag information, the apparatus for extracting the tag information is illustrated by only dividing the functional modules, and in practical applications, the function distribution may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus for extracting tag information and the method for extracting tag information provided by the above embodiments belong to the same concept, and specific implementation processes thereof are detailed in the method embodiments and are not described herein again.
Fig. 4 illustrates a server for extracting tag information according to an example embodiment. Referring to fig. 4, server 400 includes a processing component 422, which further includes one or more processors, and memory resources, represented by memory 432, for storing instructions, such as applications, that are executable by processing component 422. The application programs stored in memory 432 may include one or more modules that each correspond to a set of instructions. Further, the processing component 422 is configured to execute instructions to perform the functions performed by the server in the above-described method of extracting tag information.
The server 400 may also include a power component 426 configured to perform power management of the server 400, a wired or wireless network interface 450 configured to connect the server 400 to a network, and an input/output (I/O) interface 458. The Server 400 may operate based on an operating system, such as Windows Server, stored in the memory 432TM，Mac OSXTM，UnixTM,LinuxTM，FreeBSDTMOr the like.
An embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium may be a computer-readable storage medium contained in the memory in the foregoing embodiment; or it may be a computer-readable storage medium that exists separately and is not assembled into a server. The computer readable storage medium stores one or more programs for use by one or more processors in performing a method of extracting tag information.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A method of extracting tag information, the method comprising:
segmenting words of text information to obtain a candidate phrase set, wherein the candidate phrase set comprises at least one candidate phrase, and each candidate phrase comprises at least one keyword;
for each candidate phrase in the candidate phrase set, determining the score of the sentence in which the candidate phrase is located, determining the score of the position of the sentence in which the candidate phrase is located, determining the first score of each keyword included in the candidate phrase, and determining the score of the candidate phrase according to the score of the sentence in which the candidate phrase is located, the score of the position of the sentence in which the candidate phrase is located and the first score of each keyword included in the candidate phrase;
selecting a preset number of candidate phrases with highest scores from the candidate phrase set based on the score of each candidate phrase;
and forming the label information of the text information by the preset number of candidate phrases.
2. The method of claim 1, wherein said determining a first score for each keyword included in said candidate phrase comprises:
determining a first occurrence number and a second occurrence number for each keyword, wherein the first occurrence number is the occurrence number of the keyword in the text information, and the second occurrence number is the total occurrence number of each keyword included in the text information;
determining the word frequency of the keyword according to the first occurrence frequency and the second occurrence frequency;
determining a first quantity and a second quantity, wherein the first quantity is the quantity of sample text information included in a sample text information base, and the second quantity is the quantity of text information including the keywords in the sample text information base;
determining the reverse file frequency of the keywords according to the first quantity and the second quantity;
and determining a first score of the keyword according to the word frequency and the reverse file frequency.
3. The method of claim 1, wherein the determining the score of the position of the sentence in which the candidate phrase is located comprises:
determining a first position of a paragraph of a sentence in which the candidate phrase is located in the text information and a second position of the candidate phrase in the paragraph;
and determining the grade of the position of the sentence in which the candidate phrase is positioned according to the first position and the second position.
4. The method of claim 1, wherein determining the score of the candidate phrase according to the score of the sentence in which the candidate phrase is located, the score of the position of the sentence in which the candidate phrase is located, and the first score of each keyword included in the candidate phrase comprises:
determining a first weight corresponding to a sentence in which the candidate phrase is located, a second weight corresponding to the position of the sentence in which the candidate phrase is located, and a third weight corresponding to each keyword included in the candidate phrase;
for each keyword included in the candidate phrase, multiplying the score of the sentence in which the candidate phrase is located by the first weight to obtain a first numerical value, multiplying the score of the position of the sentence in which the candidate phrase is located by the second weight to obtain a second numerical value, multiplying the first score of the keyword by a third weight corresponding to the keyword to obtain a third numerical value, and adding the first numerical value, the second numerical value and the third numerical value to obtain a second score of the keyword;
and determining the score of the candidate phrase according to the second score of each keyword.
5. The method of claim 4, wherein said adding the first value, the second value, and the third value to obtain a second score for the keyword comprises:
determining the contribution degree of the keyword, and determining a fourth weight corresponding to the keyword according to the contribution degree of the keyword;
adding the first numerical value, the second numerical value and the third numerical value to obtain a fourth numerical value;
and multiplying the fourth numerical value and the fourth weight to obtain a second score of the keyword.
6. The method of claim 1, wherein the grouping the preset number of candidate phrases into tag information of the text information comprises:
selecting candidate phrases of concept types from the preset number of candidate phrases to form concept label information; and/or the presence of a gas in the gas,
and selecting candidate phrases of the event type from the preset number of candidate phrases to form event tag information.
7. The method of claim 6, further comprising:
moving candidate phrases ending with keywords with preset parts of speech in the concept label information to the event label information;
and moving the candidate phrases which do not contain the keywords of the preset part of speech in the event label information to the concept label information.
8. The method according to any one of claims 1-7, wherein the segmenting the text information into candidate phrases comprises:
sentence breaking is carried out on the text information to obtain at least one candidate sentence, and the at least one candidate sentence is formed into a candidate sentence set;
segmenting each candidate sentence in the candidate sentence set to obtain at least one keyword, and forming the at least one keyword into a keyword set;
generating at least one candidate phrase from the keywords in the keyword set based on a syntax tree algorithm;
and forming the at least one candidate phrase into the candidate phrase set.
9. The method of claim 8, wherein before the tokenizing each candidate sentence in the set of candidate sentences to obtain at least one keyword, the method further comprises:
determining a sentence component of each candidate sentence in the set of candidate sentences;
and deleting the candidate sentences of which the sentence components are preset components in the candidate sentence set according to the sentence components of each candidate sentence.
10. An apparatus for extracting tag information, the apparatus comprising:
the word segmentation module is used for segmenting words of the text information to obtain a candidate phrase set, wherein the candidate phrase set comprises at least one candidate phrase, and each candidate phrase comprises at least one keyword;
the scoring module is used for determining the score of a sentence where the candidate phrase is located and the score of the position of the sentence where the candidate phrase is located and the first score of each keyword included in the candidate phrase for each candidate phrase in the candidate phrase set, and determining the score of the candidate phrase according to the score of the sentence where the candidate phrase is located, the score of the position of the sentence where the candidate phrase is located and the first score of each keyword included in the candidate phrase;
a selection module, configured to select a preset number of candidate phrases with highest scores from the candidate phrase set based on the score of each candidate phrase;
and the composition module is used for composing the preset number of candidate phrases into label information of the text information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710459445.XA CN107357777B (en) | 2017-06-16 | 2017-06-16 | Method and device for extracting label information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710459445.XA CN107357777B (en) | 2017-06-16 | 2017-06-16 | Method and device for extracting label information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107357777A true CN107357777A (en) | 2017-11-17 |
| CN107357777B CN107357777B (en) | 2020-07-07 |
Family
ID=60272421
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710459445.XA Active CN107357777B (en) | 2017-06-16 | 2017-06-16 | Method and device for extracting label information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107357777B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108519970A (en) * | 2018-02-06 | 2018-09-11 | 平安科技（深圳）有限公司 | The identification method of sensitive information, electronic device and readable storage medium storing program for executing in text |
| CN110008309A (en) * | 2019-03-21 | 2019-07-12 | 腾讯科技（深圳）有限公司 | A kind of short phrase picking method and device |
| CN110059312A (en) * | 2019-03-27 | 2019-07-26 | 阿里巴巴集团控股有限公司 | Short phrase picking method, apparatus and electronic equipment |
| CN111125355A (en) * | 2018-10-31 | 2020-05-08 | 北京国双科技有限公司 | Information processing method and related equipment |
| CN111858938A (en) * | 2020-07-23 | 2020-10-30 | 鼎富智能科技有限公司 | Extraction method and device of referee document label |
| TWI727624B (en) * | 2020-01-21 | 2021-05-11 | 兆豐國際商業銀行股份有限公司 | News filtering device and news filtering method |
| CN113836257A (en) * | 2021-10-13 | 2021-12-24 | 科大讯飞股份有限公司 | Hot word mining method, device, equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140337257A1 (en) * | 2013-05-09 | 2014-11-13 | Metavana, Inc. | Hybrid human machine learning system and method |
| CN106021234A (en) * | 2016-05-31 | 2016-10-12 | 徐子涵 | Label extraction method and system |
-
2017
- 2017-06-16 CN CN201710459445.XA patent/CN107357777B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140337257A1 (en) * | 2013-05-09 | 2014-11-13 | Metavana, Inc. | Hybrid human machine learning system and method |
| CN106021234A (en) * | 2016-05-31 | 2016-10-12 | 徐子涵 | Label extraction method and system |
Non-Patent Citations (2)
| Title |
|---|
| JENQ-HAUR WANG ET AL.: "Unsupervised Opinion Phrase Extraction and Rating in Chinese Blog Posts", 《2011 IEEE THIRD INTERNATIONAL CONFERENCE ON PRIVACY, SECURITY, RISK AND TRUST AND 2011 IEEE THIRD INTERNATIONAL CONFERENCE ON SOCIAL COMPUTING》 * |
| 寇宛秋 等: "基于种子词汇的话题标签抽取研究", 《中文信息学报》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108519970A (en) * | 2018-02-06 | 2018-09-11 | 平安科技（深圳）有限公司 | The identification method of sensitive information, electronic device and readable storage medium storing program for executing in text |
| CN108519970B (en) * | 2018-02-06 | 2021-08-31 | 平安科技（深圳）有限公司 | Method for identifying sensitive information in text, electronic device and readable storage medium |
| CN111125355A (en) * | 2018-10-31 | 2020-05-08 | 北京国双科技有限公司 | Information processing method and related equipment |
| CN110008309A (en) * | 2019-03-21 | 2019-07-12 | 腾讯科技（深圳）有限公司 | A kind of short phrase picking method and device |
| CN110008309B (en) * | 2019-03-21 | 2021-03-30 | 腾讯科技（深圳）有限公司 | Phrase mining method and device |
| CN110059312A (en) * | 2019-03-27 | 2019-07-26 | 阿里巴巴集团控股有限公司 | Short phrase picking method, apparatus and electronic equipment |
| TWI727624B (en) * | 2020-01-21 | 2021-05-11 | 兆豐國際商業銀行股份有限公司 | News filtering device and news filtering method |
| CN111858938A (en) * | 2020-07-23 | 2020-10-30 | 鼎富智能科技有限公司 | Extraction method and device of referee document label |
| CN111858938B (en) * | 2020-07-23 | 2024-05-24 | 鼎富智能科技有限公司 | Method and device for extracting referee document tag |
| CN113836257A (en) * | 2021-10-13 | 2021-12-24 | 科大讯飞股份有限公司 | Hot word mining method, device, equipment and storage medium |
| CN113836257B (en) * | 2021-10-13 | 2024-04-30 | 科大讯飞股份有限公司 | Hot word mining method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107357777B (en) | 2020-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107357777B (en) | Method and device for extracting label information | |
| TWI536181B (en) | Language identification in multilingual text | |
| US7269544B2 (en) | System and method for identifying special word usage in a document | |
| CN101872351B (en) | Method, device for identifying synonyms, and method and device for searching by using same | |
| CN104850574B (en) | A kind of filtering sensitive words method of text-oriented information | |
| JP5710581B2 (en) | Question answering apparatus, method, and program | |
| Ramisch et al. | mwetoolkit: A framework for multiword expression identification. | |
| Gupta et al. | Text summarization of Hindi documents using rule based approach | |
| JP4534666B2 (en) | Text sentence search device and text sentence search program | |
| US8204736B2 (en) | Access to multilingual textual resources | |
| JP2011118689A (en) | Retrieval method and system | |
| CN118210908B (en) | Retrieval enhancement method and device, electronic equipment and storage medium | |
| Selamat et al. | Word-length algorithm for language identification of under-resourced languages | |
| Bagalkotkar et al. | A novel technique for efficient text document summarization as a service | |
| CN113468339B (en) | Label extraction method and system based on knowledge graph, electronic equipment and medium | |
| CN115794995A (en) | Target answer obtaining method and related device, electronic equipment and storage medium | |
| Ghosh et al. | A rule based extractive text summarization technique for Bangla news documents | |
| CN110705285B (en) | Government affair text subject word library construction method, device, server and readable storage medium | |
| CN111428031B (en) | Graph model filtering method integrating shallow semantic information | |
| CN107239455B (en) | Core word recognition method and device | |
| Gupta et al. | Text analysis and information retrieval of text data | |
| CN107665442B (en) | Method and device for acquiring target user | |
| CN113934910A (en) | Automatic optimization and updating theme library construction method and hot event real-time updating method | |
| JP2004030202A (en) | System for sampling characteristic word | |
| Shah et al. | An automatic text summarization on Naive Bayes classifier using latent semantic analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20190904 Address after: 100089 Unit 6, Floor 3, 25 Shangdi East Road, Haidian District, Beijing Applicant after: China Science and Technology (Beijing) Co., Ltd. Address before: Room 601, Block A, Wanliu Xingui Building, 28 Wanquanzhuang Road, Haidian District, Beijing Applicant before: Beijing Shenzhou Taiyue Software Co., Ltd. Applicant before: China Science and Technology (Beijing) Co., Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: 230000 zone B, 19th floor, building A1, 3333 Xiyou Road, hi tech Zone, Hefei City, Anhui Province Patentee after: Dingfu Intelligent Technology Co., Ltd Address before: 100089 Haidian District East Road, No. three, floor 6, unit 25, Patentee before: DINFO (BEIJING) SCIENCE DEVELOPMENT Co.,Ltd. |
|
| CP03 | Change of name, title or address |