Efficient Vision Transformer for Human Pose Estimation via Patch Selection
Kaleab A Kinfu (Johns Hopkins University),* Rene Vidal (University of Pennsylvania)
The 34th British Machine Vision Conference
Abstract
While Convolutional Neural Networks (CNNs) have been widely successful in 2D human pose estimation, Vision Transformers (ViTs) have emerged as a promising alternative to CNNs, boosting state-of-the-art performance. However, the quadratic computational complexity of ViTs has limited their applicability for processing high-resolution images. In this paper, we propose three methods for reducing ViT's computational complexity, which are based on selecting and processing a small number of most informative patches while disregarding others. The first two methods leverage a lightweight pose estimation network to guide the patch selection process, while the third method utilizes a set of learnable joint tokens to ensure that the selected patches contain the most important information about body joints. Experiments across six benchmarks show that our proposed methods achieve a significant reduction in computational complexity, ranging from 30% to 44%, with only a minimal drop in accuracy between 0% and 3.5%.Video
Citation
@inproceedings{Kinfu_2023_BMVC,
author = {Kaleab A Kinfu and Rene Vidal},
title = {Efficient Vision Transformer for Human Pose Estimation via Patch Selection},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0167.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by