TechRxiv
Artificial_Vocal_Learning_Guided_by_Visual_Information_Manuscript.pdf (709.2 kB)
Download file

Artificial Vocal Learning guided by Phoneme Recognition and Visual Information

Download (709.2 kB)
preprint
posted on 2022-08-23, 04:48 authored by Paul KrugPaul Krug, Peter Birkholz, Branislav Gerazov, Daniel Rudolph van Niekerk, Anqi Xu, Yi XuYi Xu

This paper introduces a paradigm shift regarding vocal learning simulations, in which the communicative function of speech acquisition determines the learning process and intelligibility is considered the main measure of learning success. Thereby, a novel approach for artificial early vocal learning is presented that utilizes deep neural network-based phoneme recognition in order to calculate the speech acquisition objective function. This function guides a learning framework that involves the state-of-the-art articulatory speech synthesizer VocalTractLab as the motor-to-acoustic forward model. It is shown that in this way an extensive set of German phonemes consisting of most German consonants and all stressed vowels can be produced successfully. The synthetic phonemes were rated as highly intelligible by human listeners in a listening experiment. Furthermore, it is shown that visual speech information, such as lip and jaw movements can be extracted from video recordings and be incorporated into the learning framework as an additional loss component during the optimization process. It was observed that this visual loss did not increase the overall intelligibility of phonemes. Instead, the visual loss acted as a regularization mechanism that facilitated the finding of more biologically plausible solutions in the articulatory domain.

Funding

Leverhulme Trust Research Project Grant No. RPG-2019-241

History

Email Address of Submitting Author

paul_konstantin.krug@tu-drseden.de

ORCID of Submitting Author

https://orcid.org/0000-0001-8518-8142

Submitting Author's Institution

Technische Universit├Ąt Dresden

Submitting Author's Country

  • Germany

Usage metrics

    Licence

    Exports