Abstract
Language models operating on discrete audio representations are increasingly becoming the go-to framework for many speech-processing tasks. Recently, discrete embeddings of the fundamental frequency (F0), have been shown to improve performance across a variety of tasks. However, the benefits of using F0 embeddings can only be as good as the embeddings themselves. Therefore, in this paper, we present an exhaustive study on using the Vector-Quantized Variational Autoencoder (VQ-VAE) to generate high-quality embeddings of the F0 curve. We experiment with various input transformations that focus on handling unvoiced regions of the F0, which are regions where F0 is not defined. For each transformation, we perform an exhaustive grid search over the embedding size and codebook size parameters, in order to achieve highest possible embedding quality. Our experiments are conducted on two different-sized datasets, LJSpeech and LibriTTS, and, in total, comprise over 140 different experiment settings. We reach results ranging from 0.53% to 4.29% F0 Frame Error (FFE), depending on the dataset and preprocessing strategy used, and we publish our best models on the HuggingFace website.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
We use the train-clean-360 subset containing 191 h of clean speech.
- 4.
We omit the comparison on LibriTTS, as [8] did not use this dataset.
References
Chu, W., Alwan, A.: Reducing F0 frame error of F0 tracking algorithms under noisy conditions with an unvoiced/voiced classification frontend. In: 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3969–3972 (2009). https://api.semanticscholar.org/CorpusID:5637941
Ito, K., Johnson, L.: The LJ speech dataset (2017). https://keithito.com/LJ-Speech-Dataset/
Kasi, K., Zahorian, S.A.: Yet another algorithm for pitch tracking. In: Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing, ICASSP 2002, pp. I–361–I–364. IEEE, Orlando, FL, USA (2002). https://doi.org/10.1109/ICASSP.2002.5743729, http://ieeexplore.ieee.org/document/5743729/
Kharitonov, E., et al.: Text-free prosody-aware generative spoken language modeling. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8666–8681. Association for Computational Linguistics, Dublin, Ireland (2022). https://doi.org/10.18653/v1/2022.acl-long.593, https://aclanthology.org/2022.acl-long.593
Kreuk, F., et al.: Textless speech emotion conversion using discrete and decomposed representations (2022). http://arxiv.org/abs/2111.07402
Nakatani, T., Amano, S., Irino, T., Ishizuka, K., Kondo, T.: A method for fundamental frequency estimation and voicing decision: application to infant utterances recorded in real acoustical environments. Speech Commun. 50(3), 203–214 (2008). https://doi.org/10.1016/j.specom.2007.09.003
Oord, A.v.d., Vinyals, O., Kavukcuoglu, K.: Neural discrete representation learning (2018). http://arxiv.org/abs/1711.00937
Polyak, A., et al.: Speech resynthesis from discrete disentangled self-supervised representations (2021). http://arxiv.org/abs/2104.00355
. Polyak, A., Wolf, L., Taigman, Y.: TTS Skins: speaker conversion via ASR (2020) http://arxiv.org/abs/1904.08983
Tran, T., Toshniwal, S., Bansal, M., Gimpel, K., Livescu, K., Ostendorf, M.: Parsing Speech: a neural approach to integrating lexical and acoustic-prosodic information. In: Walker, M., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 69–81. Association for Computational Linguistics, New Orleans, Louisiana (2018). https://doi.org/10.18653/v1/N18-1007, https://aclanthology.org/N18-1007
Triantafyllopoulos, A., et al.: An overview of affective speech synthesis and conversion in the deep learning era. Proc. IEEE 111(10), 1355–1381 (2023). https://doi.org/10.1109/JPROC.2023.3250266
Zen, H., et al.: LibriTTS: a corpus derived from LibriSpeech for text-to-speech (2019). http://arxiv.org/abs/1904.02882
Zhao, Y., Li, H., Lai, C.I., Williams, J., Cooper, E., Yamagishi, J.: Improved prosody from learned F0 codebook representations for VQ-VAE speech waveform reconstruction (2020). http://arxiv.org/abs/2005.07884
Acknowledgement
Computational resources were provided by the e-INFRA CZ project (ID:90254), supported by the Ministry of Education, Youth and Sports of the Czech Republic.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Porteš, D., Horák, A. (2024). Generating High-Quality F0 Embeddings Using the Vector-Quantized Variational Autoencoder. In: Nöth, E., Horák, A., Sojka, P. (eds) Text, Speech, and Dialogue. TSD 2024. Lecture Notes in Computer Science(), vol 15049. Springer, Cham. https://doi.org/10.1007/978-3-031-70566-3_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-70566-3_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70565-6
Online ISBN: 978-3-031-70566-3
eBook Packages: Computer ScienceComputer Science (R0)