Learning Temporal Sentence Grounding From Narrated EgoVideos
Kevin Flanagan (University of Bristol),* Dima Damen (University of Bristol), Michael Wray (University of Bristol)
The 34th British Machine Vision Conference
Abstract
The onset of long-form egocentric datasets such as Ego4D and EPIC-Kitchens presents a new challenge for the task of Temporal Sentence Grounding (TSG). Compared to traditional benchmarks on which this task is evaluated, these datasets offer finer-grained sentences to ground in notably longer videos. In this paper, we develop an approach for learning to ground sentences in these datasets using only narrations and their corresponding rough narration timestamps. We propose to artificially merge clips to train for temporal grounding in a contrastive manner using text-conditioning attention. This Clip Merging (CliMer) approach is shown to be an effective approach when compared with a high performing TSG method—e.g. mean R@1 improves from 3.9 to 5.7 on Ego4D and from 10.7 to 13.0 on EPIC-Kitchens. Code and data splits available from: https://github.com/keflanagan/CliMerVideo
Citation
@inproceedings{Flanagan_2023_BMVC,
author = {Kevin Flanagan and Dima Damen and Michael Wray},
title = {Learning Temporal Sentence Grounding From Narrated EgoVideos},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0332.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by