Cross-attention Masked Auto-Encoder for Human 3D Motion Infilling and Denoising
David Björkstrand (KTH Royal Institute of Technology / Tracab),* Josephine Sullivan (KTH Royal Institute of Technology), Lars M C Bretzner (Tracab AB), Gareth Loy (TRACAB), Tiesheng Wang (Tracab)
The 34th British Machine Vision Conference
Abstract
Human 3D pose and motion capture have numerous applications in fields such as augmented and virtual reality, animation, robotics and sports. However, even the best capturing methods suffer from artifacts such as missed joints and noisy or inaccurate joint positions. To address this we propose the Cross-attention Masked Auto-Encoder (XMAE) for human 3D motion infilling and denoising. XMAE extends the original Masked Auto-Encoder design by introducing cross-attention in the decoder to deal with the train-test gap common in methods utilizing masking and mask tokens. Furthermore, we introduce joint displacement as an additional noise source during training, enabling XMAE to learn to correct incorrect joint positions. Through extensive experiments, we show XMAE's effectiveness compared to state-of-the-art approaches across three public datasets and its ability to denoise real-world data, reducing limb length standard deviation by 28\% when applied on our in-the-wild professional soccer dataset.Video
Citation
@inproceedings{Björkstrand_2023_BMVC,
author = {David Björkstrand and Josephine Sullivan and Lars M C Bretzner and Gareth Loy and Tiesheng Wang},
title = {Cross-attention Masked Auto-Encoder for Human 3D Motion Infilling and Denoising},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0609.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by