Attentive Contractive Flow with Lipschitz Constrained Self-Attention
Avideep Mukherjee (Indian Institute of Technology Kanpur),* Badri N Patro (KU Leuven), Vinay Namboodiri (University of Bath)
The 34th British Machine Vision Conference
Abstract
Normalizing flows provide an elegant method for obtaining tractable density estimates from distributions using invertible transformations. The main challenge is improving the models' expressivity while keeping the invertibility constraints intact. We propose to do so via the incorporation of localized self-attention. However, conventional self-attention mechanisms do not satisfy the requirements to obtain invertible flows and cannot be naively incorporated into normalizing flows. To address this, we introduce a novel approach called Attentive Contractive Flow (ACF) which utilizes a special category of flow-based generative models - contractive flows. We demonstrate that ACF can be introduced into various state-of-the-art flow models in a plug-and-play manner. This is demonstrated to improve the representation power of these models (improving on the bits per dim metric) and result in significantly faster convergence in training them. Qualitative results, including interpolations between test images, demonstrate that samples are more realistic and capture local correlations in the data well. We evaluate the results further by performing perturbation analysis using AWGN demonstrating that ACF models (especially the dot-product variant) show better and more consistent resilience to additive noise.Video
Citation
@inproceedings{Mukherjee_2023_BMVC,
author = {Avideep Mukherjee and Badri N Patro and Vinay Namboodiri},
title = {Attentive Contractive Flow with Lipschitz Constrained Self-Attention},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0276.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by