Label-guided Real-time Fusion Network forRGB-T Semantic Segmentation


Zengrong Lin (Sun Yat-sen University), Baihong Lin (University of Electronic Science and Technology of China),* Yulan Guo (Sun Yat-sen University)
The 34th British Machine Vision Conference

Abstract

RGB-T semantic segmentation has emerged as a promising solution to handle hard scenes with poor lighting conditions by fusing a pair of RGB and thermal images. Although various deep-learning-based fusion networks have been proposed with excellent performance, most of them are not suitable for real-time applications due to high computational overhead and latency. To realize high-accuracy RGB-T real-time semantic segmentation, this paper proposes a novel Label-guided Real-time Fusion Network which fuses multi-level features of RGB and thermal images extracted from double two-pathway lightweight backbones based on the proposed Label-guided Fusion Module (LFM). The proposed LFM realizes efficient multi-modal feature fusion by spatial weighted summation, in which a spatial attention map is generated with the guidance of semantic label in the training phase to accurately indicate the contributions of different modalities. Exhaustive experiments on the MFNet and PST900 datasets demonstrate that the proposed method simultaneously achieves higher speed and accuracy compared with other state-of-the-art methods.

Video



Citation

@inproceedings{Lin_2023_BMVC,
author    = {Zengrong Lin and Baihong Lin and Yulan Guo},
title     = {Label-guided Real-time Fusion Network forRGB-T Semantic Segmentation},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year      = {2023},
url       = {https://papers.bmvc2023.org/0767.pdf}
}


Copyright © 2023 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection