Neural Canvas: Supporting Scenic Design Prototyping by Integrating 3D Sketching and Generative AI
HKUST(GZ)1
Yale University2
HKUST3
teaser
Abstract

We propose Neural Canvas, a lightweight 3D platform that integrates sketching and a collection of generative AI models to facilitate scenic design prototyping. Compared with traditional 3D tools, sketching in a 3D environment helps designers quickly express spatial ideas, but it does not facilitate the rapid prototyping of scene appearance or atmosphere. Neural Canvas integrates generative AI models into a 3D sketching interface and incorporates four types of projection operations to facilitate 2D-to-3D content creation. Our user study shows that Neural Canvas is an effective creativity support tool, enabling users to rapidly explore visual ideas and iterate 3D scenic designs. It also expedites the creative process for both novices and artists who wish to leverage generative AI technology, resulting in attractive and detailed 3D designs created more efficiently than using traditional modeling tools or individual generative AI platforms.

Gallery
Preview Video
Trial
If you are interested and want to experience our platform, please contact us and we'll consider offer access.
Bibtex
inproceedings{Shen:2024:NeuralCanvas,
    author = {Shen, Yulin and Shen, Yifei and Cheng, Jiawen and Jiang, Chutian and Fan, Mingming and Wang, Zeyu},
    title = {{Neural Canvas: Supporting Scenic Design Prototyping by Integrating 3D Sketching and Generative AI}},
    year = {2024},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3613904.3642096},
    doi = {10.1145/3613904.3642096},
    booktitle = {Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems},
    series = {CHI '24}
}