Interactive Object Insertion with Differentiable Rendering

Abstract

Compositing virtual objects into real-world imagery, referred to as object insertion, has a number of applications across film visual effects, augmented reality, and even interior design. Creating physically realistic composites can be a tedious process requiring an artist to perform manual editing of illumination effects for a given object and background scene. This manual process lacks interactive feedback, making it difficult to finetune aspects of the composite, such as object location, size, and orientation. In this work, we propose a modern framework and accompanying user interface to bring recent advancements in computational photography to artists and designers in an accessible and extensible manner. Specifically, we follow the paper Physically Controllable Relighting of Photographs and leverage state-of-the-art mid-level vision estimations to build a virtual 3D scene from a single image. We then use differentiable rendering and optional user constraints to determine the lighting conditions in the scene. Finally, we allow the user to place 3D objects into the scene and render them using the estimated illumination, resulting in a final realistic composite. Our method brings together ideas from the past decade of inverse rendering research to create an open-source tool for artists and designers.

This work was developed by Weikun and Sota as a class project for CMPT 461/769 - Computational Photography at SFU.

Implementation

GitHub Repository for CMPT 461/769 - Computational Photography

Paper

Video

BibTeX

Related Publications

Physically Controllable Relighting of Photographs Chris Careaga and Yağız Aksoy Proc. SIGGRAPH, 2025 Abstract We present a self-supervised approach to in-the-wild image relighting that enables fully controllable, physically based illumination editing. We achieve this by combining the physical accuracy of traditional rendering with the photorealistic appearance made possible by neural rendering. Our pipeline works by inferring a colored mesh representation of a given scene using monocular estimates of geometry and intrinsic components. This representation allows users to define their desired illumination configuration in 3D. The scene under the new lighting can then be rendered using a path-tracing engine. We send this approximate rendering of the scene through a feed-forward neural renderer to predict the final photorealistic relighting result. We develop a differentiable rendering process to reconstruct in-the-wild scene illumination, enabling self-supervised training of our neural renderer on raw image collections. Our method represents a significant step in bringing the explicit physical control over lights available in typical 3D computer graphics tools, such as Blender, to in-the-wild relighting. Manuscript & more BibTeX @INPROCEEDINGS{careagaRelighting, author={Chris Careaga and Ya\u{g}{\i}z Aksoy}, title={Physically Controllable Relighting of Photographs}, booktitle={Proc. SIGGRAPH}, year={2025}, }

Intrinsic Harmonization for Illumination-Aware Compositing Chris Careaga, S. Mahdi H. Miangoleh, and Yağız Aksoy SIGGRAPH Asia, 2023 Abstract Despite significant advancements in network-based image harmonization techniques, there still exists a domain disparity between typical training pairs and real-world composites encountered during inference. Most existing methods are trained to reverse global edits made on segmented image regions, which fail to accurately capture the lighting inconsistencies between the foreground and background found in composited images. In this work, we introduce a self-supervised illumination harmonization approach formulated in the intrinsic image domain. First, we estimate a simple global lighting model from mid-level vision representations to generate a rough shading for the foreground region. A network then refines this inferred shading to generate a harmonious re-shading that aligns with the background scene. In order to match the color appearance of the foreground and background, we utilize ideas from prior harmonization approaches to perform parameterized image edits in the albedo domain. To validate the effectiveness of our approach, we present results from challenging real-world composites and conduct a user study to objectively measure the enhanced realism achieved compared to state-of-the-art harmonization methods. Manuscript & more BibTeX @INPROCEEDINGS{careagaCompositing, author={Chris Careaga and S. Mahdi H. Miangoleh and Ya\u{g}{\i}z Aksoy}, title={Intrinsic Harmonization for Illumination-Aware Compositing}, booktitle={Proc. SIGGRAPH Asia}, year={2023}, }

Physically-Based Compositing of 2D Graphics Tyrus Tracey, Stefan Diaconu, Sebastian Dille, S. Mahdi H. Miangoleh, and Yağız Aksoy SIGGRAPH Posters, 2025 Abstract We propose an interactive pipeline that enables the seamless integration of a 2D logo into a target image, adapting to the surface geometry and lighting conditions of the scene to ensure realistic appearance. Manuscript & more BibTeX @INPROCEEDINGS{traceyDiaconuCompositing, author={Tyrus Tracey and Stefan Diaconu and Sebastian Dille and S. Mahdi H. Miangoleh and Ya\u{g}{\i}z Aksoy}, title={Physically-Based Compositing of {2D} Graphics}, booktitle={SIGGRAPH Posters}, year={2025}, }

More posters from CMPT 461/769: Computational Photography

Interactive RGB+NIR Photo Editing Samuel Antunes Miranda*, Shahrzad Mirzaei*, Mariam Bebawy*, Sebastian Dille, and Yağız Aksoy SIGGRAPH Posters, 2024 Abstract Near-infrared imagery offers great possibilities for creative image editing. Lying outside the visual spectrum, the NIR information can effectively serve as a fourth color channel to common RGB. Compared to the latter, it shows interesting and complementary behavior: its intensity strongly varies with the surface materials in the scene and is less affected by atmospheric perturbations. For these reasons, NIR imaging has been a long-standing topic of interest in research and its integration has been proven successful for applications like false coloring, contrast enhancement, image dehazing, and purification of low-light images. Recent developments in smartphone technology have simplified the capturing process, making NIR data readily available for broader use outside the research community. At the same time, existing tools for NIR processing and manipulation are rare and still limited in functionality. With many solutions lacking specialized features, the editing process is inefficient and cumbersome, making them prone to generate suboptimal results. To tackle this issue, we introduce a simple and intuitive photo editing tool that combines RGB and NIR properties, offering functions tailored specifically for the RGB+NIR combination, and granting the user the ability to edit and refine images more creatively. Manuscript & more BibTeX @INPROCEEDINGS{NIREditing, author={Samuel Antunes Miranda and Shahrzad Mirzaei and Mariam Bebawy and Sebastian Dille and Ya\u{g}{\i}z Aksoy}, title={Interactive RGB+NIR Photo Editing}, booktitle={SIGGRAPH Posters}, year={2024}, }

Datamoshing with Optical Flow Chris Careaga, Mahesh Kumar Krishna Reddy, and Yağız Aksoy SIGGRAPH Asia Posters, 2023 Abstract We propose a simple method for emulating the effect of data moshing, without relying on the corruption of encoded video, and explore its use in different application scenarios. Like traditional data moshing, we apply motion information to mismatched visual data. Our approach uses off-the-shelf optical flow estimation to generate motion vectors for each pixel. Our core algorithm can be implemented in a handful of lines but unlocks multiple video editing effects. The use of accurate optical flow rather than compression data also creates a more natural transition without block artifacts. We hope our method provides artists and content creators with more creative freedom over the process of data moshing. Manuscript & more BibTeX @INPROCEEDINGS{datamosh, author={Chris Careaga and Mahesh Kumar Krishna Reddy and Ya\u{g}{\i}z Aksoy}, title={Datamoshing with Optical Flow}, booktitle={SIGGRAPH Asia Posters}, year={2023}, }

Parallax Background Texture Generation Brigham Okano, Shao Yu Shen, Sebastian Dille, and Yağız Aksoy SIGGRAPH Posters, 2022 Abstract Art assets for games can be time intensive to produce. Whether it is a full 3D world, or simpler 2D background, creating good looking assets takes time and skills that are not always readily available. Time can be saved by using repeating assets, but visible repetition hurts immersion. Procedural generation techniques can help make repetition less uniform, but do not remove it entirely. Both approaches leave noticeable levels of repetition in the image, and require significant time and skill investments to produce. Video game developers in hobby, game jam, or early prototyping situations may not have access to the required time and skill. We propose a framework to produce layered 2D backgrounds without the need for significant artist time or skill. In our pipeline, the user provides segmented photographic input, instead of creating traditional art, and receives game-ready assets. By utilizing photographs as input, we can achieve both a high level of realism for the resulting background texture as well as a shift from manual work away towards computational run-time which frees up developers for other work. Manuscript & more BibTeX @INPROCEEDINGS{parallaxBG, author={Brigham Okano and Shao Yu Shen and Sebastian Dille and Ya\u{g}{\i}z Aksoy}, title={Parallax Background Texture Generation}, booktitle={SIGGRAPH Posters}, year={2022}, }

DynaPix: Normal Map Pixelization for Dynamic Lighting Gerardo Gandeaga, Denys Iliash, Chris Careaga, and Yağız Aksoy SIGGRAPH Posters, 2022 Abstract This work introduces DynaPix, a Krita extension that automatically generates pixelated images and surface normals from an input image. DynaPix is a tool that aids pixel artists and game developers more efficiently develop 8-bit style games and bring them to life with dynamic lighting through normal maps that can be used in modern game engines such as Unity. The extension offers artists a degree of flexibility as well as allows for further refinements to generated artwork. Powered by out of the box solutions, DynaPix is a tool that seamlessly integrates in the artistic workflow. Manuscript & more BibTeX @INPROCEEDINGS{dynapix, author={Gerardo Gandeaga and Denys Iliash and Chris Careaga and Ya\u{g}{\i}z Aksoy}, title={Dyna{P}ix: Normal Map Pixelization for Dynamic Lighting}, booktitle={SIGGRAPH Posters}, year={2022}, }