Artificial Vocal Learning guided by Phoneme Recognition and Visual Information

This paper introduces a paradigm shift regarding vocal learning simulations, in which the communicative function of speech acquisition determines the learning process and intelligibility is considered the main measure of learning success. Thereby, a novel approach for artificial early vocal learning is presented that utilizes deep neural network-based phoneme recognition in order to calculate the speech acquisition objective function. This function guides a learning framework that involves the state-of-the-art articulatory speech synthesizer VocalTractLab as the motor-to-acoustic forward model. It is shown that in this way an extensive set of German phonemes consisting of most German consonants and all stressed vowels can be produced successfully. The synthetic phonemes were rated as highly intelligible by human listeners in a listening experiment. Furthermore, it is shown that visual speech information, such as lip and jaw movements can be extracted from video recordings and be incorporated into the learning framework as an additional loss component during the optimization process. It was observed that this visual loss did not increase the overall intelligibility of phonemes. Instead, the visual loss acted as a regularization mechanism that facilitated the finding of more biologically plausible solutions in the articulatory domain.