Speech-conditioned face generation using Generative Adversarial Networks

Fork me on GitHub

upc-logo dcu-logo

Universitat Politècnica de Catalunya

Dublin City University


Speech is a rich biometric signal that contains information about the identity, gender and emotional state of the speaker. In this work, we explore its potential to generate face images of a speaker by conditioning a Generative Adversarial Network (GAN) with raw speech input. We propose a deep neural network that is trained from scratch in an end-to-end fashion, generating a face directly from the raw speech waveform without any additional identity information (e.g reference image or one-hot encoding). Our model is trained in a self-supervised fashion by exploiting the audio and visual signals naturally aligned in videos. With the purpose of training from video data, we present a novel dataset collected for this work, with high-quality videos of ten youtubers with notable expressiveness in both the speech and visual signals.


Our model is divided in three modules trained end-to-end: a speech encoder, a generator network and a discriminator network. We have used SEGAN decoder as the encoder. Besides, both the generator and discriminator modules were inspired by this network using, however, a Least Squares GAN.


Some results from Jaime Altozano

This project was developed with Python 2.7 and PyTorch 0.4.0. To download and install PyTorch, please follow the official guide. You can also fork or download the project from [here] (https://github.com/miqueltubau/Wav2Pix.git).



We especially want to thank our technical support team:

Amanda Duarte Phd grant is funded by ”la Caixa” Foundation through the MSCA actions in the the Horizon 2020 FP of the European Commission. logo-LaCaixa
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GeForce GTX Titan Z and Titan X used in this work. logo-nvidia
The Image Processing Group at the UPC is a SGR17 Consolidated Research Group recognized by the Government of Catalonia (Generalitat de Catalunya) through its AGAUR office. logo-catalonia
This work has been developed in the framework of projects TEC2015-69266-P and TEC2016-75976-R, financed by the Spanish Ministerio de Economía y Competitividad and the European Regional Development Fund (ERDF). logo-spain