• SynPL: a zero-shot prompt language model to process multiple-choice questions on synonyms
  • Overview
    • Background
    • Approaches
    • Results
    • Findings
  • Datasets
  • Question structure
  • AutoModelForMultipleChoice model
    • Three input patterns
    • Finetune
    • Results
  • Fill-mask prompt model
    • Background
    • Prompt idea
    • Finetune
    • Definition of accuracy
    • Data processing
    • Results
  • MultiOpt VS Prompt
    • Results
  • Requirements
  • Model files

Multiple-choice questions are a classic section in exams. When taking a language test such as TOEFL®, some synonym questions need you to select the "best" choice among a set of four options of words or phrases that is the closest meaning to a "keyword" in the context of a reading passage.

Here my objective is to build language models to process this kind of questions automatically.

I developed 2 kinds of language models to solve this problem.

The first is to use the AutoModelForMultipleChoice pre-trained model from 🤗TRANSFORMERS. This is a generic model with a multiple-choice head, which yields scores of inputs in a given selection. We select ID with max scores as the most plausible inputs.

The second is to build a prompt for the classic fill-mask model, so that this multiple-choice task can be formulated as a masked language modeling problem, which is what pre-trained models like BERT are designed for in the first place.

Figure 1 shows the schematic. Where:

• MultiOpt_KO model is finetuned AutoModelForMultipleChoice pre-trained BERT model with Keyword [SEP] Options pattern as input
• PromptBi model is the classic fill-mask model with prompt

Fig 1. Schematic for 2 models

Figure 2 is the final result of these 2 models. Where "Raw BERT" is un-finetuned AutoModelForMultipleChoice pre-trained BERT model.

Fig 2. The Final result of the 2 models

• Same as humans, models don't need contexts for this kind of synonym questions
• Prompt has the best performance in terms of accuracy, input conciseness, and the amount of training data needed (Because it can be finetuned on other datasets first).
• To my best knowledge, prompt models are a novel way of language models for these kinds of problems.

Two Datasets are used in this project.

• A total of 454 synonym multiple-choice questions from the TOEFL® test
• WordNet

Due to copyright issues, I can't share the TOEFL® test dataset here.

A typical example of synonym multiple-choice questions from the TOEFL® test would be like this:

45. Coincident with concerns about the accelerating loss of species and habitats has been a growing appreciation of the importance of biological diversity, the number of species in a particular ecosystem, to the health of the Earth and human well-being.

Q: The word "appreciation" is closest in meaning to which of following?

A. ignorance
B. recognition
C. tolerance
D. forgiveness

Ans: B

It consists of 4 parts: Context, Question, 4 Options, and the Answer. We can transform those strings via Python's re library into a structured data frame.

This is a generic model class built in 🤗TRANSFORMERS that will be instantiated as a model class with a multiple-choice head. Read the document here.

I used BERT as the base model. Its input is a set of four arrays loaded in an array, and the output/label is the numeric version of the answer — namely encode [A, B, C, D] as [0, 1, 2, 3]. So we can try to build the input in 3 different patterns and make a comparison of which input pattern is better. See an illustration below.

There are 3 input patterns we can try to build and then feed into the model:

1. Context [SEP] Keyword [SEP] Options - MultiOpt_CKO
2. Keyword [SEP] Options - MultiOpt_KO
3. Keyword sentence [SEP] Option sentences - MultiOpt_KsOs

Fig 3. Diagram of 3 input patterns. Where C stands for Context, K stands for Keyword, O stands for Options, Ks stands for Keyword sentence and Os stands for Option sentences.

I divided each dataset 70% as train data and 30% as test data with the same random seed. Each model was finetuned 5 times with different random initial values.

Table 1 and Figure 4 show the mean accuracies for 3 models.

Table 1. Evaluation Metrics for 3 AutoModelForMultipleChoice Models

MultiOpt_KO (the Keyword [SEP] Options pattern) got the best result. Not only has it the highest accuracy but also is the most robust one that achieved almost consistent prediction performance when trained from different initializations.

Fig 4. Accuracy(%) of 3 Models

It actually adheres to our intuitions that when doing TOEFL® synonym multiple-choice questions, most of the time we don't have to read the context. By only examining the keyword and options, we can still select the best choice out.

This 70% of accuracy seems mediocre, but we can't raise its performance to the next level due to the lack of enough data for finetuning. However, as declared above, giving the context is kind of unnecessary, we may try to build a prompt to improve our model.

We can design a template that contains relationships among synonyms as a prompt. In this way, we can use other data to finetune language models and perform a zero-shot on this TOEFL® dataset.

Prompts are some small templates inserted into the inputs, with the aim that tasks such as text classification can be transformed back into a pre-trained language model's original form — a fill-mask problem.

For example, if you get a text classification problem to grade a movie review "The drama discloses nothing", then you can insert a text "It was ____" into the end of this review. After executing your model, you just need to examine logit scores of "terrible" and "great", so that you can determine whether this review is positive or negative.

Fig 5. A typical example of using a prompt

You can read Gao's post for more information.

A simple idea is to build a template like this: "A is close in meaning to B.", where A is an English word that appears in any synonym datasets, and B is one of A's synonyms. Here I used the WordNet dataset.

This prompt can be performed like this on the TOEFL® dataset:

• First, we let the keyword in the question be A
• Second, we mask B
• Last, execute the model, extract the logit scores for 4 options, and then decided the best choice with the max score

We still need to finetune the model with this prompt, because we should let the model know that it's facing a synonym problem. See Figure 6 and Table 2 for how this step can boost its performance.

Fig 6. An example of before and after finetuning. Here you can see after finetuning, its output words are more like synonyms.

I re-defined the definition of accuracy during this finetuning. Since many words have different meanings and each meaning can have multiple synonyms, so I gathered all synonyms of one word as a list. as long as any of top N (N=1, 2, 3, 5, 10) words predicted by the model falls into this list, I count it as one successful prediction.

I eliminated data where all phrases along with words that the tokenizer doesn't know. This is because there is only one [MASK] token in my designed template. Those words or phrases may take more than one masked token positions, which would have a negative influence on models. And the model can never predict them successfully.

Last but not least, synonyms are a mutual thing. A is close in meaning to B also means B is close in meaning to A. This is obvious to humans, but maybe not to a language model. Switching the A and B can make models better understand this synonym task. See Table 2 for how this step can boost its performance.

In conclusion, there are 3 models in this part.

1. Raw_BERT - un-finetuned raw BERT model with prompt
2. Prompt_Uni - finetuned prompt model without switching the position of A and B
3. Prompt_Bi - finetuned prompt model with switching the position of A and B

Table 2. Accuracy of prompt models

Fig 7. Accuracy of prompt models. Check definition of accuracy here.

Below are some other metrics.

Finally, let's see how those two kinds perform on the TOEFL® dataset.

We select 3 models to compare.

1. Raw_BERT - un-finetuned raw BERT prompt model
2. MultiOpt_KO - finetuned AutoModelForMultipleChoice model with Keyword [SEP] Options as input pattern
3. Prompt_Bi - finetuned prompt model with switching the position of A and B

These models all use the same test data, which means data where all phrases along with words that the tokenizer doesn't know are eliminated.

Table 3 and Figure 2 show how these models perform on the TOEFL® dataset.

Table 3. Metrics of 3 final models

🤗 transformers==4.10.1
🤗 datasets==1.12.1
nltk==3.6.3

• src/MultiOpt_CKO.py: Trainer for MultiOpt_CKO model.
• src/MultiOpt_KO.py: Trainer for MultiOpt_KO model.
• src/MultiOpt_KsOs.py: Trainer for MultiOpt_KsOs model.
• src/MultiOpt_KO_PickVal.py: Trainer for MultiOpt_KO model but with test data for final comparison (others are train data).
• src/Prompt_Uni.py: Trainer for Prompt_Uni model.
• src/Prompt_Bi.py: Trainer for Prompt_Bi model.
• src/Prompt_Bi.ipynb: Notebook for a example of training the Prompt_Bi model.