Weakly-Supervised Visual-Textual Grounding with Semantic Prior Refinement
Davide Rigoni (University of Padova), Luca Parolari (University of Padova), Luciano Serafini (Fondazione Bruno Kessler), Alessandro Sperduti (Università di Padova (IT)), Lamberto Ballan (University of Padova)*
The 34th British Machine Vision Conference
Abstract
Using only image-sentence pairs, weakly-supervised visual-textual grounding aims to learn region-phrase correspondences of the respective entity mentions. Compared to the supervised approach, learning is more difficult since bounding boxes and textual phrases correspondences are unavailable. In light of this, we propose the Semantic Prior Refinement Model (SPRM), whose predictions are obtained by combining the output of two main modules. The first untrained module aims to return a rough alignment between textual phrases and bounding boxes. The second trained module is composed of two sub-components that refine the rough alignment to improve the accuracy of the final phrase-bounding box alignments. The model is trained to maximize the multimodal similarity between an image and a sentence, while minimizing the multimodal similarity of the same sentence and a new unrelated image, carefully selected to help the most during training. Our approach shows state-of-the-art results on two popular datasets, Flickr30k Entities and ReferIt, shining especially on ReferIt with a 9.6% absolute improvement. Moreover, thanks to the untrained component, it reaches competitive performances just using a small fraction of training examples.Video
Citation
@inproceedings{Rigoni_2023_BMVC,
author = {Davide Rigoni and Luca Parolari and Luciano Serafini and Alessandro Sperduti and Lamberto Ballan},
title = {Weakly-Supervised Visual-Textual Grounding with Semantic Prior Refinement},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0229.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by