Interpreting Pretrained Language Models via Concept Bottlenecks

Pretrained language models (PLMs) have made significant strides in various natural language processing tasks. However, the lack of interpretability due to their “black-box” nature poses challenges for responsible implementation. Although previous studies have attempted to improve interpretability by using, e.g., attention weights in self-attention layers, these weights often lack clarity, readability, and intuitiveness. In this research, we propose a novel approach to interpreting PLMs by employing high-level, meaningful concepts that are easily understandable for humans. For example, we learn the concept of “Food” and investigate how it influences the prediction of a model’s sentiment towards a restaurant review. We introduce C\(^3\)M, which combines human-annotated and machine-generated concepts to extract hidden neurons designed to encapsulate semantically meaningful and task-specific concepts. Through empirical evaluations on real-world datasets, we show that our approach offers valuable insights to interpret PLM behavior, helps diagnose model failures, and enhances model robustness amidst noisy concept labels.

1. 1.

   https://www.kaggle.com/datasets/omkarsabnis/yelp-reviews-dataset.

2. 2.

   https://github.com/Zhen-Tan-dmml/CBM_NLP.git.

This work is supported by the National Science Foundation (NSF) under grants IIS-2229461.

Authors and Affiliations

1. Arizona State University, Tempe, AZ, USA

   Zhen Tan & Huan Liu

2. University of Illinois Chicago, Chicago, IL, USA

   Lu Cheng

3. University of Virginia, Charlottesville, VA, USA

   Song Wang & Jundong Li

4. Zhejiang University, Zhejiang, China

   Bo Yuan

Corresponding author

Correspondence to Zhen Tan .

Editors and Affiliations

1. Academia Sinica, Taipei, Taiwan

   De-Nian Yang

2. Microsoft Research Asia, Beijing, China

   Xing Xie

3. National Yang Ming Chiao Tung University, Hsinchu, Taiwan

   Vincent S. Tseng

4. Duke University, Durham, NC, USA

   Jian Pei

5. National Cheng Kung University, Tainan, Taiwan

   Jen-Wei Huang

6. Silesian University of Technology, Gliwice, Poland

   Jerry Chun-Wei Lin

A Definitions of Training Strategies

Given a text input \(x \in \mathbb {R}^d\), concepts \(c\in \mathbb {R}^k\) and its label y, the strategies for fine-tuning the text encoder \(f_\theta \), the projector \(p_\psi \) and the label predictor \(g_\phi \) are defined as follows:

i) Vanilla fine-tuning a PLM: The concept labels are ignored, and then the text encoder \(f_\theta \) and the label predictor \(g_\phi \) are fine-tuned either as follows:

or as follows (frozen text encoder \(f_\theta \)):

where \(L_{CE}\) indicates the cross-entropy loss. In this work we only consider the former option for its significant better performance.

ii) Independently training PLM with the concept and task labels: The text encoder \(f_\theta \), the projector \(p_\psi \) and the label predictor \(g_\phi \) are trained separately with ground truth concepts labels and task labels as follows:

During inference, the label predictor will use the output from the projector rather than the ground-truth concepts.

iii) Sequentially training PLM with the concept and task labels: We first learn the concept encoder as the independent training strategy above, and then use its output to train the label predictor:

iv) Jointly training PLM with the concept and task labels: Learn the concept encoder and label predictor via a weighted sum \(L_{joint}\) of the two objectives described above:

It’s worth noting that the CBE-PLMs trained jointly are sensitive to the loss weight \(\gamma \). We report the most effective results here, tested value for \(\gamma \) are given in Table 2 in Appendix D.

B Details of the Manual Concept Annotation for the IMDB Dataset

Our annotation policy is following a previous work [5] for NLP datasets annotating. For the IMDB-C dataset, we annotate the four concepts (Acting, Stroyline, Emotional Arousal, Cinematography) manually. Even though the concepts are naturally understandable by humans, two Master students familiar with sentiment analysis are selected as annotators for independent annotation with the annotation tool introduced by [30]. The strict quadruple matching F1 score between two annotators is \(85.74\%\), which indicates a consistent agreement between the two annotators [13]. In case of disagreement, a third expert will be asked to make the final decision.

C Implementation Detail

In this section, we provide more details on the implementation settings of our experiments. Specifically, we implement our framework with PyTorch [24] and HuggingFace [29] and train our framework on a single 80 GB Nvidia A100 GPU. We follow a prior work [1] for backbone implementation. All backbone models have a maximum token number of 512 and a batch size of 8. We use the Adam optimizer to update the backbone, projector, and label predictor according to Sect. 3.1. The values of other hyperparameters (Table 2 in Appendix D) for each specific PLM type are determined through grid search. We run all the experiments on an Nvidia A100 GPU with 80 GB RAM.

D Parameters and Notations

In this section, we provide used notations in this paper along with their descriptions for comprehensive understanding. We also list their experimented values and optimal ones, as shown in Table 2.

E Statistics of Data Splits

The Statistics and split policies of the experimented datasets, including the source concept dataset \(\mathcal {D}_s\), the unlabeled concept dataset \(\mathcal {D}_u\), and their augmented versions. The specific details are presented in Table 3.

F Statistics of Concepts in Transformed Datasets

The Statistics and split policies of the transformed datasets of experimented datasets are presented in Table 4.

G More Results on Explainable Predictions

Case studies on explainable predictions for both CEBaB and IMDB-C datasets are given in Fig. 5 and Fig. 6 respectively.

Illustration of the explanable prediction for an example from the CEBaB dataset.

Full size image

Illustration of the explanable prediction for an example from the IMDB-C dataset.

Full size image

H A Case Study on Test-Time Intervention

We present a case study of Test-time Intervention using an example from the transformed unlabeled concept data \(\tilde{\mathcal {D}}_u\) of the CEBaB dataset, as shown in Fig. 7. The first row displays the target concept labels generated by ChatGPT. The second row shows the predictions from the trained CBE-PLM model, which mispredicts two concepts ("Waiting time" and "Waiting area"). The third row demonstrates test-time intervention using ChatGPT as the oracle, which corrects the predicted task labels. Finally, the fourth row implements test-time intervention with a human oracle, rectifying the concept that ChatGPT originally mislabeled.

Illustration of the explanable prediction for an example from the transformed unlabeled concept data \(\tilde{\mathcal {D}}_{u}\) of the CEBaB dataset. The brown box with dash lines indicates the test-time intervention on corresponding concepts. (Color figure online)

Full size image

I Examples of Querying ChatGPT

In this paper, we query ChatGPT for 1) augmenting the concept set, and 2) annotate missing concept labels. Note that in practice, we query ChatGPT (GPT4) via OpenAI API. Here we demonstrate examples from the ChatGPT (GPT4) GUI for better illustration. The illustrations are given in Fig. 8 and Fig. 9.

The illustration of querying ChatGPT for additional concepts for the IMDB-C dataset.

Full size image

The illustration of querying ChatGPT for annotating a missing concept label for the IMDB-C dataset.

Full size image

Reprints and permissions

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

Policies and ethics