A Structure-Guided Diffusion Model for Large-Hole Image Completion

Daichi Horita (The University of Tokyo),* Jiaolong Yang (Microsoft Research), Dong Chen (Microsoft Research Asia), Yuki Koyama (National Institute of Advanced Industrial Science and Technology (AIST)), Kiyoharu Aizawa (The University of Tokyo), Nicu Sebe (University of Trento)
The 34th British Machine Vision Conference


Image completion techniques have made significant progress in filling missing regions (i.e., holes) in images. However, large-hole completion remains challenging due to limited structural information. In this paper, we address this problem by incorporating explicit structural guidance with a structure-guided diffusion model (SGDM). Our proposed SGDM consists of two cascaded diffusion probabilistic models: structure and texture generators. The structure generator first generates an edge image representing plausible structures within the holes, which later guides the texture generation process. To train both generators jointly, we devise a novel strategy that leverages optimal Bayesian denoising, which denoises the output of the structure generator in a single step and thus allows backpropagation. Our diffusion-based approach enables a diversity of plausible completions, while the editable edges allow for editing parts of an image. Our experiments on natural scene (Places) and face (CelebA-HQ) datasets demonstrate that our method achieves a superior or comparable visual quality performance compared to state-of-the-art approaches.



author    = {Daichi Horita and Jiaolong Yang and Dong Chen and Yuki Koyama and Kiyoharu Aizawa and Nicu Sebe},
title     = {A Structure-Guided Diffusion Model for Large-Hole Image Completion},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year      = {2023},
url       = {https://papers.bmvc2023.org/0258.pdf}

Copyright © 2023 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection