Video-adverb retrieval with compositional adverb-action embeddings
Thomas Hummel (University of Tübingen),* Otniel-Bogdan Mercea (University of Tübingen), A. Sophia Koepke (University of Tübingen), Zeynep Akata (University of Tübingen)
The 34th British Machine Vision Conference
Abstract
Retrieving adverbs that describe an action in a video poses a crucial step towards fine-grained video understanding. We propose a framework for video-to-adverb retrieval (and vice versa) that aligns video embeddings with their matching compositional adverb-action text embedding in a joint embedding space. The compositional adverb-action text embedding is learned using a residual gating mechanism, along with a novel training objective consisting of triplet losses and a regression target. Our method achieves state-of-the-art performance on five recent benchmarks for video-adverb retrieval. Furthermore, we introduce dataset splits to benchmark video-adverb retrieval for unseen adverb-action compositions on subsets of the MSR-VTT Adverbs and ActivityNet Adverbs datasets. Our proposed framework outperforms all prior works for the generalisation task of retrieving adverbs from videos for unseen adverb-action compositions. Code and dataset splits are available at https://hummelth.github.io/ReGaDa/.Video
Citation
@inproceedings{Hummel_2023_BMVC,
author = {Thomas Hummel and Otniel-Bogdan Mercea and A. Sophia Koepke and Zeynep Akata},
title = {Video-adverb retrieval with compositional adverb-action embeddings},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0581.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by