Breathing New Life into 3D Assets with Generative Repainting
Tianfu Wang (ETH Zurich), Menelaos Kanakis (ETH Zurich), Konrad Schindler (ETH Zurich), Luc Van Gool (ETH Zurich), Anton Obukhov (ETH Zurich)*
The 34th British Machine Vision Conference
Abstract
Diffusion-based text-to-image models ignited immense attention from the vision community, artists, and content creators. Broad adoption of these models is due to significant improvement in the quality of generations and efficient conditioning on various modalities, not just text. However, lifting the rich generative priors of these 2D models into 3D is challenging. Recent works have proposed various pipelines powered by the entanglement of diffusion models and neural fields. We explore the power of pretrained 2D diffusion models and standard 3D neural radiance fields as independent, standalone tools and demonstrate their ability to work together in a non-learned fashion. Such modularity has the intrinsic advantage of eased partial upgrades, which became an important property in such a fast-paced domain. Our pipeline accepts any legacy renderable geometry, such as textured or untextured meshes, orchestrates the interaction between 2D generative refinement and 3D consistency enforcement tools, and outputs a painted input geometry in several formats. We conduct a large-scale study on a wide range of objects and categories from the ShapeNetSem dataset and demonstrate the advantages of our approach, both qualitatively and quantitatively. Project page: https://www.obukhov.ai/repainting_3d_assetsVideo
Citation
@inproceedings{Wang_2023_BMVC,
author = {Tianfu Wang and Menelaos Kanakis and Konrad Schindler and Luc Van Gool and Anton Obukhov},
title = {Breathing New Life into 3D Assets with Generative Repainting},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0405.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by