Text and Click inputs for unambiguous open vocabulary instance segmentation
Vighnesh N Birodkar (Google),* Jonathan Huang (Google), Meera Hahn (Google), Irfan Essa (Georgia Institute of Technology), Nikolai Warner (Georgia Tech)
The 34th British Machine Vision Conference
Abstract
Segmentation localizes objects in an image on a fine-grained per-pixel scale. Segmentation benefits by humans-in-the-loop to provide additional input of objects to segment using a combination of foreground or background clicks. Tasks include photoediting or novel dataset annotation, where human annotators leverage an existing segmentation model instead of drawing raw pixel level annotations. We propose a new segmentation process, Text + Click segmentation, where a model takes as input an image, a text phrase describing a class to segment, and a single foreground click specifying the instance to segment. Compared to previous approaches, we leverage open-vocabulary image-text models to support a wide-range of text prompts. Conditioning segmentations on text prompts improves the accuracy of segmentations on novel or unseen classes. We demonstrate that the combination of a single user-specified foreground click and a text prompt allows a model to better disambiguate overlapping or co-occurring semantic categories, such as “tie”, “suit”, and “person”. We study these results across common segmentation datasets such as refCOCO, COCO, VOC, and OpenImagesVideo
Citation
@inproceedings{Birodkar_2023_BMVC,
author = {Vighnesh N Birodkar and Jonathan Huang and Meera Hahn and Irfan Essa and Nikolai Warner},
title = {Text and Click inputs for unambiguous open vocabulary instance segmentation},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0815.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by