Generating Context-Aware Natural Answers for Questions in 3D Scenes
Mohammed Munzer Dwedari (Technical University of Munich),* Matthias Niessner (Technical University of Munich), Zhenyu Chen (Technical University of Munich)
The 34th British Machine Vision Conference
Abstract
3D question answering is a young field in 3D vision-language that is yet to be explored. Previous methods are limited to a pre-defined answer space and cannot generate answers naturally. In this work, we pivot the question answering task to a sequence generation task to generate free-form natural answers for questions in 3D scenes (Gen3DQA). To this end, we optimize our model directly on the language rewards to secure the global sentence semantics. Here, we also adapt a pragmatic language understanding reward to further improve the sentence quality. Our method sets a new SOTA on the ScanQA benchmark (CIDEr score 72.22/66.57 on the test sets).Video
Citation
@inproceedings{Dwedari_2023_BMVC,
author = {Mohammed Munzer Dwedari and Matthias Niessner and Zhenyu Chen},
title = {Generating Context-Aware Natural Answers for Questions in 3D Scenes},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0596.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by