Lips-SpecFormer: Non-Linear Interpolable Transformer for Spectral Reconstruction using Adjacent Channel Coupling
Abhishek Kumar Sinha (Indian Space Research Organization),* Manthira Moorthi S (ISRO)
The 34th British Machine Vision Conference
Abstract
Spectral Recovery from RGB images is a challenging yet interesting domain to learn end-to-end spectral mapping using neural nets. Transformers have recently gained popularity due to their ability to learn long range dependencies through self-attention. In our work, we show that spectral feature learning with self-attention is prone to instability. We propose a transformer based network for spectral reconstruction from RGB images with a Non-Linear Interpolable Spectral Attention (N-LISA) to learn the spectral features. We further analyse the stability of the N-LISA using the theory of Lipschitz constant. The method is evaluated and compared with different state-of-the-art methods on multiple standard datasets. In addition, ablation analysis is performed to analyse the effectiveness of the proposed spectral attention, and other modules.Video
Citation
@inproceedings{Sinha_2023_BMVC,
author = {Abhishek Kumar Sinha and Manthira Moorthi S},
title = {Lips-SpecFormer: Non-Linear Interpolable Transformer for Spectral Reconstruction using Adjacent Channel Coupling},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0268.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by