Speech and Language Recognition with Low-rank Adaptation of Pretrained Models

parameter reduction, language identification, speech recognition, wav2vec2.0

Finetuning large pretrained models demands considerable computational resources, posing practical constraints. Major- ity of the total number of parameters in these models are used by fully connected layers. In this work, we consider applying a semi-orthogonal constraint, followed by full finetuning to the fully connected layers reduces model parameters significantly without sacrificing efficacy in downstream tasks. Specifically, we consider wav2vec2.0 XLS-R and Whisper models for Auto- matic Speech Recognition and Language Recognition. Our re- sults show that we can reduce the model size by approximately 24% during both training and inference time with 0.7% absolute drop in performance for XLS-R and no drop in performance for Whisper for ASR. In combination with performance-efficient training with low-rank adapters, the resource requirements for training can be further reduced by up to 90%.