loading page

A Flow-Based Deep Latent Variable Model for Speech Spectrogram Modeling and Enhancement
  • Aditya Arie Nugraha ,
  • Kouhei Sekiguchi ,
  • Kazuyoshi Yoshii
Aditya Arie Nugraha
Author Profile
Kouhei Sekiguchi
Author Profile
Kazuyoshi Yoshii
Author Profile


This paper describes a deep latent variable model of speech power spectrograms and its application to semi-supervised speech enhancement with a deep speech prior. By integrating two major deep generative models, a variational autoencoder (VAE) and a normalizing flow (NF), in a mutually-beneficial manner, we formulate a flexible latent variable model called the NF-VAE that can extract low-dimensional latent representations from high-dimensional observations, akin to the VAE, and does not need to explicitly represent the distribution of the observations, akin to the NF. In this paper, we consider a variant of NF called the generative flow (GF a.k.a. Glow) and formulate a latent variable model called the GF-VAE. We experimentally show that the proposed GF-VAE is better than the standard VAE at capturing fine-structured harmonics of speech spectrograms, especially in the high-frequency range. A similar finding is also obtained when the GF-VAE and the VAE are used to generate speech spectrograms from latent variables randomly sampled from the standard Gaussian distribution. Lastly, when these models are used as speech priors for statistical multichannel speech enhancement, the GF-VAE outperforms the VAE and the GF.
2020Published in IEEE/ACM Transactions on Audio, Speech, and Language Processing volume 28 on pages 1104-1117. 10.1109/TASLP.2020.2979603