Superpixel Positional Encoding to Improve ViT-based Semantic Segmentation Models
Roberto Amoroso (University of Modena and Reggio Emilia),* Matteo Tomei (Prometeia), Lorenzo Baraldi (University of Modena and Reggio Emilia), Rita Cucchiara (Università di Modena e Reggio Emilia)
The 34th British Machine Vision Conference
Abstract
In this paper, we present a novel superpixel-based positional encoding technique that combines Vision Transformer (ViT) features with superpixels priors to improve the performance of semantic segmentation architectures. Recently proposed ViT-based segmentation approaches employ a Transformer backbone and exploit self-attentive features as an input to a convolutional decoder, achieving state-of-the-art performance in dense prediction tasks. Our proposed technique is plug-and-play, model-agnostic, and operates by computing superpixels over the input image. It determines a positional encoding based on the centroids and shapes of the superpixels, and then unifies this semantic-aware information with the self-attentive features extracted by the ViT-based backbone. Our results demonstrate that this simple positional encoding strategy, when applied to the decoder of ViT-based architectures, leads to a significant improvement in performance without increasing the number of parameters and with negligible impact on the training time. We evaluate our approach on different backbones and architectures and observe a significant improvement in terms of mIoU on the ADE20K and Cityscapes datasets. Notably, our approach provides improved performance on classes with low occurrence in the dataset while mitigating overfitting on classes with higher representation, ensuring a good balance between generalization and specificity.Video
Citation
@inproceedings{Amoroso_2023_BMVC,
author = {Roberto Amoroso and Matteo Tomei and Lorenzo Baraldi and Rita Cucchiara},
title = {Superpixel Positional Encoding to Improve ViT-based Semantic Segmentation Models},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0623.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by