%0 Conference Proceedings
%T MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
%A Pfeiffer, Jonas
%A Vulić, Ivan
%A Gurevych, Iryna
%A Ruder, Sebastian
%Y Webber, Bonnie
%Y Cohn, Trevor
%Y He, Yulan
%Y Liu, Yang
%S Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
%D 2020
%8 November
%I Association for Computational Linguistics
%C Online
%F pfeiffer-etal-2020-mad
%X The main goal behind state-of-the-art pre-trained multilingual models such as multilingual BERT and XLM-R is enabling and bootstrapping NLP applications in low-resource languages through zero-shot or few-shot cross-lingual transfer. However, due to limited model capacity, their transfer performance is the weakest exactly on such low-resource languages and languages unseen during pre-training. We propose MAD-X, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations. In addition, we introduce a novel invertible adapter architecture and a strong baseline method for adapting a pre-trained multilingual model to a new language. MAD-X outperforms the state of the art in cross lingual transfer across a representative set of typologically diverse languages on named entity recognition and causal commonsense reasoning, and achieves competitive results on question answering. Our code and adapters are available at AdapterHub.ml.
%R 10.18653/v1/2020.emnlp-main.617
%U https://aclanthology.org/2020.emnlp-main.617
%U https://doi.org/10.18653/v1/2020.emnlp-main.617
%P 7654-7673