Conditional Generation from Pre-Trained Diffusion Models using Denoiser Representations
Alexandros Graikos (Stony Brook University),* Srikar Yellapragada (Stony Brook University), Dimitris Samaras (Stony Brook University)
The 34th British Machine Vision Conference
Abstract
Denoising diffusion models have gained popularity as a generative modeling technique for producing high-quality and diverse images. Applying these models to downstream tasks requires conditioning, which can take the form of text, class labels, or other forms of guidance. However, providing conditioning information to these models can be challenging, particularly when annotations are scarce or imprecise. In this paper, we propose adapting pre-trained unconditional diffusion models to new conditions using the learned internal representations of the denoiser network. We demonstrate the effectiveness of our approach on various conditional generation tasks, including attribute-conditioned generation and mask-conditioned generation. Additionally, we show that augmenting the Tiny ImageNet training set with synthetic images generated by our approach improves the classification accuracy of ResNet baselines by up to 8\%. Our approach provides a powerful and flexible way to adapt diffusion models to new conditions and generate high-quality augmented data for various conditional generation tasks.Video
Citation
@inproceedings{Graikos_2023_BMVC,
author = {Alexandros Graikos and Srikar Yellapragada and Dimitris Samaras},
title = {Conditional Generation from Pre-Trained Diffusion Models using Denoiser Representations},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0478.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by