CN115618022B - Low-resource relation extraction method based on data synthesis and two-stage self-training - Google Patents

Abstract

The invention relates to the field of data synthesis, and discloses a low-resource relationship extraction method based on data synthesis and two-stage self-training, including a data synthesis method and a two-stage self-training framework, and data synthesis effectively alleviates the lack of labeled data in the current relationship extraction task , the problem of high labeling cost. The two-stage self-training framework sequentially uses unlabeled generated data and labeled data to train the model in each iteration. On the one hand, it can promote the collaborative learning of the model from unlabeled generated data and labeled data, and on the other hand, it also effectively reduces the generated data. the effect of noise. The present invention fits the low-resource situation in the real scene, and can utilize the existing pre-trained language model more effectively.

Description

A low-resource relation extraction method based on data synthesis and two-stage self-training

technical field

The invention relates to the field of data synthesis, in particular to a low-resource relationship extraction method based on data synthesis and two-stage self-training.

Background technique

As an important technical support for knowledge extraction and graph construction, the relationship extraction system aims to mine the relationship between entities in unstructured text, and has become a research hotspot in the field of natural language processing in recent years. Although neural network models, especially pretrained language models, have achieved remarkable breakthroughs in relation extraction tasks, training these models requires a large amount of labeled data. However, in many real-world scenarios, obtaining high-quality labeled data is usually time-consuming and labor-intensive. Therefore, how to build a relationship extraction system with excellent performance under limited resources and data has become a major challenge.

Distant supervision automatically labels relation extraction data by aligning entities in text with existing knowledge bases, which has been widely studied as an effective method for building large-scale relation extraction datasets. However, due to differences in relational patterns and text corpora, the data annotated by distant supervision may be quite different from downstream tasks, which inhibits further optimization of model performance. For example, due to the dependence on the existing knowledge base, most of the current distant supervision uses Wikidata as the source of relational triples, and Wikipedia as the corpus of distant supervision. This limits the schema and text of the labeled data to everyday knowledge, but downstream tasks may involve other domain-specific knowledge, such as semantic relationships between names, chemical protein interactions.

Considering that the large-scale language model has demonstrated its powerful language generation capabilities in several fields such as news articles, commodity categories, and daily dialogues, the present invention uses the large-scale language model to synthesize data for relational extraction tasks to solve low-resource scenarios Under the training data scarcity problem and the domain difference problem of distant supervision. On the other hand, in order to make better use of generated data, the present invention proposes a two-stage self-learning framework.

Contents of the invention

In order to solve the above technical problems, the present invention provides a low-resource relationship extraction method based on data synthesis and two-stage self-training. The self-learning framework usually iteratively labels and learns pseudo-labels of unlabeled data to guide the continuous improvement of model performance. But in each iteration of the two-stage self-learning framework of the present invention, the generated data is used for training in the first stage and the labeled data is used for training in the second stage. Since the labeled data is introduced in the later stage of the training process, this sequential form of the training process increases the attention of the model to the labeled data. Furthermore, the present invention formulates generative data training as a knowledge distillation process using soft pseudo-labels instead of assigning them hard labels. In general, the two-stage self-training framework uses generated data to solve the problem of less labeled data under low resources, and further improves the performance of the knowledge extraction system.

In order to solve the problems of the technologies described above, the present invention adopts the following technical solutions:

A low-resource relationship extraction method based on data synthesis and two-stage self-training, comprising the following steps:

Step 1. Data synthesis based on the marked training data: by adding placeholders to the training data, the training data is converted into a linear natural language sequence; a data synthesis model based on a large-scale language model is constructed, and the training data is paired with The data synthesis model is fine-tuned; the data synthesis process is iteratively performed using polynomial sampling until an unlabeled generated dataset meeting the preset criteria is obtained , there are placeholders in the generated data; To generate a sequence of words in the data, , respectively subject and object of for the amount of generated data;

Step 2, two-stage self-learning: in the training data set Train the self-encoding language model η on , and then add the generated data set with placemarks Use the self-encoding language model η classification to realize soft pseudo-label annotation: ;

make soft pseudo label set , is the soft pseudo-label; use K different random seeds to train multiple self-encoding language models, denoted as teacher model η, and the soft-pseudo-label set marked by the kth teacher model is denoted as ;Initialize a new self-encoding language model, denoted as the student model , for the student model Apply a two-stage training strategy: In the first stage of training, distillation training is performed using generated data with soft pseudo-labels: ; the student model Optimized for student models , to calculate the distillation loss :

;

represents the KL divergence; in the second stage of training, the student model in the training dataset Train on: ; is the standard cross-entropy loss function, is the student model obtained after the second stage of training iterations; the next time the two-stage training strategy is executed, the student model As a teacher model η; repeat the two-stage training strategy until a dataset is generated Each generated data in is marked with a soft pseudo-label;

Step 3. Relation extraction: build a relation extraction model based on the self-encoded language model; the generated data and training data are collectively referred to as training instances, and the relation labels of training instances are referred to as real labels , input the training instance into the relationship extraction model, and calculate the relationship label predicted by the relationship extraction model and the true labels of the training instances The cross-entropy loss to train the relation extraction model.

Further, in step 1, the fine-tuning loss of the data synthesis model is ; The fine-tuning method of the data synthesis model is the same as the pre-training method of the data synthesis model, where Represents a probability function, LLM represents a large-scale language model, is the sequence of words in the training data with placeholders added words in is a special start character, after fine-tuning, in the Add marker prompts before generation.

Further, in step 2, the two-stage training strategy will be repeated T times; in each iteration, the data set is generated from Sampling 1/T of generated data, after T iterations, generate a data set All generated data in is annotated with soft pseudo-labels.

Further, when adding placeholders to the training data in step 1:

;

in, Indicates the word sequence after adding placeholders in the training data, and is used to label the training data word sequence body the placeholder for the position, and is used to label the training data word sequence object The placeholder for the location.

Further, when the relationship label predicted by the relationship extraction model is predicted in step 3, the vector representation h of the corresponding position of the word sequence in the training instance is concatenated for classification, and the relationship label predicted by the relationship extraction model is obtained :

;

in is the activation function, is a fully connected network, is the vector representation of the subject, is the vector representation of the object, [ ; ] represents the splicing operation.

Compared with the prior art, the beneficial technical effect of the present invention is:

The present invention proposes a low-resource relationship extraction method based on data synthesis and two-stage self-training, including a data synthesis method and a two-stage self-training framework; data synthesis effectively alleviates the problem of less labeled data and high labeling costs in the current relationship extraction task question. The two-stage self-training framework sequentially uses unlabeled generated data and labeled data to train the self-encoded language model in each iteration. It also effectively reduces the impact of generated data noise. This relation extraction system fits the low-resource situation in the real scene, can make more effective use of the existing pre-trained language model, tap its potential, and has broad application prospects.

Description of drawings

Fig. 1 is an overall model architecture diagram of the present invention.

Detailed ways

A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

In the present invention, the training data set is set , the label set corresponding to the training data set , for the training data the corresponding relation label, and , is the set of all labeled relationship labels in the training dataset, is the number of training data in the training data set. here is the sequence of words in the i-th training data of the training data set , represent a sequence of words in the first words, L is the sequence of words length, and sequence of words subject and object in represents the subject in a sequence of words The starting position in , ending at , represents the object in the sequence of words The starting position in , ending at .

generate dataset , the corresponding soft-pseudo-label set , For the soft pseudo-label, To generate a sequence of words in the data, , respectively subject and object of The amount of data generated for the generated dataset.

generate data and training data collectively referred to as training instances , the relation label of the training instance is called the ground truth label , that is, the soft pseudo-label and relationship labels collectively referred to as ground truth labels .

The goal of relation extraction is to learn a function , the function pass to predict the true label .

The relation extraction system proposed by the present invention is shown in Fig. 1, and includes the following three parts: 1. a relation extraction model; 2. a data synthesis model for low-resource scenarios; 3. a two-stage self-learning framework.

1. Relationship extraction model

The main body of the relationship extraction model uses a BERT-like self-encoding language model instead of an autoregressive language model, because the self-encoding language model usually performs better on language understanding tasks. Considering the previous research on the extraction model, the present invention has added some special placemarks to the word sequence of the training example of the input relation extraction model to emphasize the position of the subject and object in the word sequence.

Formula one: ;

in Represents the training instance word sequence after adding special placeholders, and is used to mark the subject special placeholders for positions, and is used to mark the object A special placeholder for a position.

After being encoded by the relationship extraction model, the vector representation h of the corresponding position of the word sequence is spliced for classification:

;

in is the activation function, is a fully connected network, is the vector representation of the subject, is the vector representation of the object, [ ; ] represents the splicing operation, is the relationship label distribution predicted by the relationship extraction model, and the present invention calculates and The cross-entropy loss function to train the relation extraction model.

The method of generating training examples will be introduced later.

2. Data synthesis model for low-resource scenarios

This embodiment adopts a data synthesis model based on a large-scale language model (LLM). Considering the powerful language generation ability of large-scale language models, the present invention solves the problem of scarcity of training data in low-resource scenarios and the problem of domain differences in remote supervision by synthesizing the data required for the task of relation extraction through large-scale language models based on labeled data. . A training instance for a relation extraction model has a specific structure: , that is, a relational fact consists of a text (i.e. sequence of words) and Subjects contained in and object Sure. First, use a method similar to formula 1 to convert the training data into a linear natural language sequence:

;

Data synthesis models can be based on arbitrary large-scale language models, such as GPT-2. Then fine-tune the large-scale language model based on the labeled training data. The fine-tuning method is the same as the pre-training method of the data synthesis model. The fine-tuning loss of the data synthesis model for:

;

in Represents a probability function, LLM represents a large-scale language model, is a sequence of words words in , and the special start character <bos> (i.e. beginning of sentence) as is added before the sequence. Note that the relationship labels in the training data are ignored here , so this is an unconditional generating process. This can remove the noise due to label-semantic inconsistency and enable the data synthesis model to model itself, which helps the data synthesis model to learn from unlabeled generated data. After fine-tuning is done, just add a tag before the special start character <bos> to prompt the generation, and use polynomial sampling to iteratively perform the generation process until you get a generated data set that meets expectations .

3. Two-stage self-learning framework

Self-learning is a learning algorithm widely used in semi-supervised learning. Usually, to jointly learn from an unlabeled dataset and a labeled dataset, one needs to iteratively sample from the unlabeled dataset, assign pseudo-labels to the sampled data, then merge them with the labeled dataset, re- Train the model. But this naive pooling design is based on a strong assumption that the unlabeled dataset must have exactly the same distribution as the labeled dataset, which cannot be strictly satisfied by the generated data.

To this end, the present invention proposes a different two-stage self-learning framework: sequentially using the unlabeled generated dataset and the labeled training dataset to train the model separately, instead of combining them for training. First in the labeled training dataset Train a self-encoded language model (such as the BERT model) η on , and then add the unlabeled generated data with placemarks Use the self-encoding language model η classification to realize soft pseudo-label annotation:

;

make soft pseudo label set , note that the superscript "^" here means "soft pseudo-label", that is, only keep the distribution of label categories instead of further taking their argmax value. In order to further eliminate the fluctuation of pseudo-labels, K different random seeds are used to train multiple self-encoding language models, denoted as teacher model η, and the soft pseudo-label set marked by the kth teacher model is denoted as .

The soft pseudo-label is a soft pseudo-label, and the soft label is a concept corresponding to the hard label. The hard label will directly label the sample with discrete labels such as 0 and 1 to represent positive and negative samples, while the soft label retains the distribution of the sample label category, and labels the sample as 0~1. In contrast, hard labels are easy to annotate, but cannot be differentiable when optimized; soft labels are smoother and have better expressive power. The soft pseudo-label in the present invention can enhance the expression ability of the generated data label.

Then reinitialize a new student model , and apply a two-stage training strategy. In the first stage of training, the generated data with soft pseudo-labels is used for training:

;

This can be seen as a distillation process, where the dataset With the help of η, the knowledge is transferred from the teacher model η to the student model , optimize it to the student model , to calculate the distillation loss :

;

stands for KL divergence. Then in the second stage of training, the student model In the labeled training dataset Train on:

;

here is the standard cross-entropy loss function, is the student model obtained after the second stage of training iterations. Then in the next iteration, the student model Relabeled as teacher model η .

The above whole process will be repeated T times. According to the standard self-learning setting, in each iteration from Sampling 1/T of generated data, after T iterations, all The generated data in will be marked with a soft pseudo-label.

After this two-stage self-learning method, the semantic gap between the existing general-purpose knowledge base and the downstream task data can be well bridged, and the interference caused by the noise of the generated data can also be effectively reduced.

It will be apparent to those skilled in the art that the invention is not limited to the details of the above-described exemplary embodiments, but that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be regarded as exemplary and non-restrictive in all points of view. The scope of the present invention is defined by the appended claims rather than the above description, and it is therefore intended that the scope of the present invention be defined by the appended claims rather than by the foregoing description. All changes within the meaning and range of equivalent elements are embraced in the present invention, and any reference sign in a claim shall not be construed as limiting the claim concerned.

In addition, it should be understood that although this specification is described according to implementation modes, not each implementation mode includes only one independent technical solution, and this description in the specification is only for clarity, and those skilled in the art should take the specification as a whole, The technical solutions in the various embodiments can also be properly combined to form other implementations that can be understood by those skilled in the art.

Claims (5)

1. A low-resource relationship extraction method based on data synthesis and two-stage self-training, comprising the following steps: Step 1. Data synthesis based on the marked training data: by adding placeholders to the training data, the training data is converted into a linear natural language sequence; a data synthesis model based on a large-scale language model is constructed, and the training data is paired with The data synthesis model is fine-tuned; the data synthesis process is iteratively performed using polynomial sampling until an unlabeled generated dataset meeting the preset criteria is obtained , there are placeholders in the generated data; To generate a sequence of words in the data, , respectively subject and object of for the amount of generated data; Step 2, two-stage self-learning: in the training data set Train the self-encoding language model η on , and then add the generated data set with placemarks Use the self-encoding language model η classification to realize soft pseudo-label annotation: ; make soft pseudo label set , is the soft pseudo-label; use K different random seeds to train multiple self-encoding language models, denoted as teacher model η, and the soft-pseudo-label set marked by the kth teacher model is denoted as ;Initialize a new self-encoding language model, denoted as the student model , for the student model Apply a two-stage training strategy: In the first stage of training, distillation training is performed using generated data with soft pseudo-labels: ; the student model Optimized for student models , to calculate the distillation loss : ; represents the KL divergence; in the second stage of training, the student model in the training dataset Train on: ; is the standard cross-entropy loss function, for the corresponding label set, is the student model obtained after the second stage of training iterations; the next time the two-stage training strategy is executed, the student model As a teacher model η; repeat the two-stage training strategy until a dataset is generated Each generated data in is marked with a soft pseudo-label; Step 3. Relation extraction: build a relation extraction model based on the self-encoded language model; the generated data and training data are collectively referred to as training instances, and the relation labels of training instances are referred to as real labels , input the training instance into the relationship extraction model, and calculate the relationship label predicted by the relationship extraction model and the true labels of the training instances The cross-entropy loss to train the relation extraction model. 2. The low-resource relationship extraction method based on data synthesis and two-stage self-training according to claim 1, wherein in step 1, the fine-tuning loss of the data synthesis model is ; The fine-tuning method of the data synthesis model is the same as the pre-training method of the data synthesis model, where Represents a probability function, LLM represents a large-scale language model, is the sequence of words in the training data with placeholders added words in is a special start character, after fine-tuning, in the Add marker prompts before generation. 3. the method for extracting low-resource relations based on data synthesis and two-stage self-training according to claim 1, characterized in that, in step 2, the two-stage training strategy is repeatedly executed T times; in each iteration, from the generated data set Sampling 1/T of generated data, after T iterations, generate a data set All generated data in is annotated with soft pseudo-labels. 4. the method for extracting low-resource relations based on data synthesis and two-stage self-training according to claim 1, wherein, in step 1, when adding placemarks in the training data: ; in, Indicates the word sequence after adding placeholders in the training data, and is used to label the training data word sequence body the placeholder for the position, and is used to label the training data word sequence object The placeholder for the location. 5. The low-resource relationship extraction method based on data synthesis and two-stage self-training according to claim 1, characterized in that: when the relationship label predicted by the relationship extraction model in step 3, the vector of the corresponding position of the word sequence in the training example Represents h splicing for classification, and obtains the relationship label predicted by the relationship extraction model : ; in is the activation function, is a fully connected network, is the vector representation of the subject, is the vector representation of the object, [ ; ] represents the splicing operation.

Images