Complex Scene Image Editing by Scene Graph Comprehension
Zhongping Zhang (Boston University),* Huiwen He (Boston University), Bryan Plummer (Boston University), Zhenyu Liao (Kwai Inc), Huayan Wang (Kuaishou Technology)
The 34th British Machine Vision Conference
Abstract
Conditional diffusion models have demonstrated impressive performance on various tasks like text-guided semantic image editing. Prior work requires image regions to be identified manually by human users or use an object detector that only perform well for object-centric manipulations. For example, if an input image contains multiple objects with the same semantic meaning (such as a group of birds), object detectors may struggle to recognize and localize the target object, let alone accurately manipulate it. To address these challenges, we propose a two-stage method for achieving complex scene image editing by Scene Graph Comprehension (SGC-Net). In the first stage, we train a Region of Interest (RoI) prediction network that uses scene graphs and predict the locations of the target objects. Unlike object detection methods based solely on object category, our method can accurately recognize the target object by comprehending the objects and their semantic relationships within a complex scene. The second stage uses a conditional diffusion model to edit the image based on our RoI predictions. We evaluate the effectiveness of our approach on the CLEVR and Visual Genome datasets. We report an 8 point improvement in SSIM on CLEVR and our edited images were preferred by human users by 9-33% over prior work on Visual Genome, validating the effectiveness of our proposed method. Code is available at github.com/Zhongping-Zhang/SGC_Net.Video
Citation
@inproceedings{Zhang_2023_BMVC,
author = {Zhongping Zhang and Huiwen He and Bryan Plummer and Zhenyu Liao and Huayan Wang},
title = {Complex Scene Image Editing by Scene Graph Comprehension},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0451.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by