GestSync: Determining who is speaking without a talking head
Sindhu B Hegde (University of Oxford),* Andrew Zisserman (University of Oxford)
The 34th British Machine Vision Conference
Abstract
In this paper we introduce a new synchronisation task, {\em Gesture-Sync}: determining if a person's gestures are correlated with their speech or not. In comparison to Lip-Sync, Gesture-Sync is far more challenging as there is a far looser relationship between the voice and body movement than there is between voice and lip motion. We introduce a dual-encoder model for this task, and compare a number of input representations including RGB frames, keypoint images, and keypoint vectors, assessing their performance and advantages. We show that the model can be trained using self-supervised learning alone, and evaluate its performance on the LRS3 dataset. Finally, we demonstrate applications of Gesture-Sync for audio-visual synchronisation, and in determining who is the speaker in a crowd, without seeing their faces. The code, datasets and pre-trained models can be found at: \url{https://www.robots.ox.ac.uk/~vgg/research/gestsync}.Video
Citation
@inproceedings{Hegde_2023_BMVC,
author = {Sindhu B Hegde and Andrew Zisserman},
title = {GestSync: Determining who is speaking without a talking head},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0506.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by