Adversarial Generation of Time-Frequency Features with application in audio synthesis

Andrés Marafioti, Nathanaël Perraudin, Nicki Holighaus, Piotr Majdak
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:4352-4362, 2019.

Abstract

Time-frequency (TF) representations provide powerful and intuitive features for the analysis of time series such as audio. But still, generative modeling of audio in the TF domain is a subtle matter. Consequently, neural audio synthesis widely relies on directly modeling the waveform and previous attempts at unconditionally synthesizing audio from neurally generated invertible TF features still struggle to produce audio at satisfying quality. In this article, focusing on the short-time Fourier transform, we discuss the challenges that arise in audio synthesis based on generated invertible TF features and how to overcome them. We demonstrate the potential of deliberate generative TF modeling by training a generative adversarial network (GAN) on short-time Fourier features. We show that by applying our guidelines, our TF-based network was able to outperform a state-of-the-art GAN generating waveforms directly, despite the similar architecture in the two networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-marafioti19a, title = {Adversarial Generation of Time-Frequency Features with application in audio synthesis}, author = {Marafioti, Andr{\'e}s and Perraudin, Nathana{\"e}l and Holighaus, Nicki and Majdak, Piotr}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {4352--4362}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/marafioti19a/marafioti19a.pdf}, url = {https://proceedings.mlr.press/v97/marafioti19a.html}, abstract = {Time-frequency (TF) representations provide powerful and intuitive features for the analysis of time series such as audio. But still, generative modeling of audio in the TF domain is a subtle matter. Consequently, neural audio synthesis widely relies on directly modeling the waveform and previous attempts at unconditionally synthesizing audio from neurally generated invertible TF features still struggle to produce audio at satisfying quality. In this article, focusing on the short-time Fourier transform, we discuss the challenges that arise in audio synthesis based on generated invertible TF features and how to overcome them. We demonstrate the potential of deliberate generative TF modeling by training a generative adversarial network (GAN) on short-time Fourier features. We show that by applying our guidelines, our TF-based network was able to outperform a state-of-the-art GAN generating waveforms directly, despite the similar architecture in the two networks.} }
Endnote
%0 Conference Paper %T Adversarial Generation of Time-Frequency Features with application in audio synthesis %A Andrés Marafioti %A Nathanaël Perraudin %A Nicki Holighaus %A Piotr Majdak %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-marafioti19a %I PMLR %P 4352--4362 %U https://proceedings.mlr.press/v97/marafioti19a.html %V 97 %X Time-frequency (TF) representations provide powerful and intuitive features for the analysis of time series such as audio. But still, generative modeling of audio in the TF domain is a subtle matter. Consequently, neural audio synthesis widely relies on directly modeling the waveform and previous attempts at unconditionally synthesizing audio from neurally generated invertible TF features still struggle to produce audio at satisfying quality. In this article, focusing on the short-time Fourier transform, we discuss the challenges that arise in audio synthesis based on generated invertible TF features and how to overcome them. We demonstrate the potential of deliberate generative TF modeling by training a generative adversarial network (GAN) on short-time Fourier features. We show that by applying our guidelines, our TF-based network was able to outperform a state-of-the-art GAN generating waveforms directly, despite the similar architecture in the two networks.
APA
Marafioti, A., Perraudin, N., Holighaus, N. & Majdak, P.. (2019). Adversarial Generation of Time-Frequency Features with application in audio synthesis. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:4352-4362 Available from https://proceedings.mlr.press/v97/marafioti19a.html.

Related Material