Attentive Contractive Flow with Lipschitz Constrained Self-Attention

Avideep Mukherjee (Indian Institute of Technology Kanpur),* Badri N Patro (KU Leuven), Vinay Namboodiri (University of Bath)
The 34th British Machine Vision Conference


Normalizing flows provide an elegant method for obtaining tractable density estimates from distributions using invertible transformations. The main challenge is improving the models' expressivity while keeping the invertibility constraints intact. We propose to do so via the incorporation of localized self-attention. However, conventional self-attention mechanisms do not satisfy the requirements to obtain invertible flows and cannot be naively incorporated into normalizing flows. To address this, we introduce a novel approach called Attentive Contractive Flow (ACF) which utilizes a special category of flow-based generative models - contractive flows. We demonstrate that ACF can be introduced into various state-of-the-art flow models in a plug-and-play manner. This is demonstrated to improve the representation power of these models (improving on the bits per dim metric) and result in significantly faster convergence in training them. Qualitative results, including interpolations between test images, demonstrate that samples are more realistic and capture local correlations in the data well. We evaluate the results further by performing perturbation analysis using AWGN demonstrating that ACF models (especially the dot-product variant) show better and more consistent resilience to additive noise.



author    = {Avideep Mukherjee and Badri N Patro and Vinay Namboodiri},
title     = {Attentive Contractive Flow with Lipschitz Constrained Self-Attention},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year      = {2023},
url       = {}

Copyright © 2023 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection