Functional Hand Type Prior for 3D Hand Pose Estimation and Action Recognition from Egocentric View Monocular Videos
WONSEOK ROH (Korea University), Seung Hyun Lee (Korea University), Won Jeong Ryoo (Korea University), Jakyung Lee (Korea University), Gyeongrok Oh (Korea University), Sooyeon Hwang (Korea University Sejong), Hyung-gun Chi (Purdue University), Sangpil Kim (Korea University)*
The 34th British Machine Vision Conference
Abstract
Current methods for egocentric view action recognition often face challenges in perceiving dynamic hand movements relying solely on geometrical or physical information. In this work, we effectively address this problem by gaining insights into the correlation between functional hand configurations and objects, which improves the detailed interpretation of real-world scenarios. To this end, we introduce a practical taxonomy of hand types based on the functioning perspective and utilize it for per-frame hand type labeling on existing datasets. We also propose a novel hand action recognition framework considering semantic details of the hand type as prior. This approach boosts the network's understanding of the continuous hand interaction throughout the action sequence. Our whole pipeline consists of three main modules: (1) Feature Extraction, (2) Egocentric Knowledge Module, which estimates 3D hand pose, object category, and hand type leveraging short-term cues, and (2) Egocentric Action Module, which aggregates per-frame knowledge, including text embeddings of hand type, over a longer time. In our extensive experiments with large-scale benchmarks, FPHA and H2O, our model outperforms current state-of-the-art methods, demonstrating its superior performance.Video
Citation
@inproceedings{ROH_2023_BMVC,
author = {WONSEOK ROH and Seung Hyun Lee and Won Jeong Ryoo and Jakyung Lee and Gyeongrok Oh and Sooyeon Hwang and Hyung-gun Chi and Sangpil Kim},
title = {Functional Hand Type Prior for 3D Hand Pose Estimation and Action Recognition from Egocentric View Monocular Videos},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0193.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by