A Structure-Guided Diffusion Model for Large-Hole Image Completion
Daichi Horita (The University of Tokyo),* Jiaolong Yang (Microsoft Research), Dong Chen (Microsoft Research Asia), Yuki Koyama (National Institute of Advanced Industrial Science and Technology (AIST)), Kiyoharu Aizawa (The University of Tokyo), Nicu Sebe (University of Trento)
The 34th British Machine Vision Conference
Abstract
Image completion techniques have made significant progress in filling missing regions (i.e., holes) in images. However, large-hole completion remains challenging due to limited structural information. In this paper, we address this problem by incorporating explicit structural guidance with a structure-guided diffusion model (SGDM). Our proposed SGDM consists of two cascaded diffusion probabilistic models: structure and texture generators. The structure generator first generates an edge image representing plausible structures within the holes, which later guides the texture generation process. To train both generators jointly, we devise a novel strategy that leverages optimal Bayesian denoising, which denoises the output of the structure generator in a single step and thus allows backpropagation. Our diffusion-based approach enables a diversity of plausible completions, while the editable edges allow for editing parts of an image. Our experiments on natural scene (Places) and face (CelebA-HQ) datasets demonstrate that our method achieves a superior or comparable visual quality performance compared to state-of-the-art approaches.Video
Citation
@inproceedings{Horita_2023_BMVC,
author = {Daichi Horita and Jiaolong Yang and Dong Chen and Yuki Koyama and Kiyoharu Aizawa and Nicu Sebe},
title = {A Structure-Guided Diffusion Model for Large-Hole Image Completion},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0258.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by