Intrinsic Single-Image HDR Reconstruction

Natural scenes contain a wide range of intensities spanning multiple orders of magnitude. This high dynamic range (HDR) can be perceived by our visual system without problem. However, the low dynamic range (LDR) of common cameras fails to capture the rich contrast, and most casual photos have lost colors and details in saturated pixels. Dynamic range extension aims to revert the clipping. Restoring the lost information in the highlights not only results in more realistic images, but also enables the physical modeling of light and opens up many downstream applications, such as image-based lighting and illumination-aware image editing. In this thesis, we approach this task using three novel formulations. Leveraging the physical properties of the scene, our models tame the long-tailed distribution of HDR radiance and relax the dependency on large HDR datasets by separating the problem into simpler sub-tasks. Looking into the image formation process, we observe important differences between the intrinsic components. For one, separating illumination from reflectance enables us to extend the dynamic range of the former without compressing the latter. Analogously, we can adapt to different material properties by separating moderate diffuse reflectance from the bright directed light of active light sources. Or, we take inspiration from the exposure mechanisms within the camera and progressively extend the dynamic range of an input image by one exposure value at a time. Our novel formulations allow for dedicated networks with a more precise supervision. We can drastically extend the limited diversity of HDR data by learning diffuse reflectance or a single-EV extension from widely available RAW image collections. This provides our models with a strong prior over natural images, combined with a realistic highlight distribution learned from full HDR data. Our novel perspectives improve performance and enable generalization to everyday photographs. We evaluate our approaches extensively on various benchmarks, including a novel multi-illuminant dataset. Our models show state-of-the-art results against a wide set of baselines, generate vivid colors, and reproduce bright highlights with a wide dynamic range. We outperform heavy diffusion-based networks with our streamlined architectures and create 4K results in less than a second.

We present an approach to progressively extend the highlights of an image. Instead of reconstructing the full dynamic range of a complex scene directly, we learn a simpler task first: We extend the dynamic range of an input image by a single exposure value. Once this is mastered, we retrieve the full HDR image for the scene by executing our network recurrently, progressively increasing the dynamic range of the input. Our formulation is agnostic to the input dynamic range and targets a bounded output domain. This enables us to use widely available RAW images for the reconstruction task and adapt adversarial losses to construct realistic images. By incorporating Memory Replay for backpropagation, we can train our network recurrently over multiple inference stages and reduce reconstruction errors. As a consequence, our system reconstructs challenging long-tailed HDR scenes robustly and shows powerful recovery of bright light sources and highlights.

Intrinsic decomposition is a fundamental mid-level vision problem that plays a crucial role in various inverse rendering and computational photography pipelines. Generating highly accurate intrinsic decompositions is an inherently under-constrained task that requires precisely estimating continuous-valued shading and albedo. In this work, we achieve high-resolution intrinsic decomposition by breaking the problem into two parts. First, we present a dense ordinal shading formulation using a shift- and scale-invariant loss in order to estimate ordinal shading cues without restricting the predictions to obey the intrinsic model. We then combine low- and high-resolution ordinal estimations using a second network to generate a shading estimate with both global coherency and local details. We encourage the model to learn an accurate decomposition by computing losses on the estimated shading as well as the albedo implied by the intrinsic model. We develop a straightforward method for generating dense pseudo ground truth using our models predictions and multi-illumination data, enabling generalization to in-the-wild imagery. We present exhaustive qualitative and quantitative analysis of our predicted intrinsic components against state-of-the-art methods. Finally, we demonstrate the real-world applicability of our estimations by performing otherwise difficult editing tasks such as recoloring and relighting.

Despite significant advancements in network-based image harmonization techniques, there still exists a domain disparity between typical training pairs and real-world composites encountered during inference. Most existing methods are trained to reverse global edits made on segmented image regions, which fail to accurately capture the lighting inconsistencies between the foreground and background found in composited images. In this work, we introduce a self-supervised illumination harmonization approach formulated in the intrinsic image domain. First, we estimate a simple global lighting model from mid-level vision representations to generate a rough shading for the foreground region. A network then refines this inferred shading to generate a harmonious re-shading that aligns with the background scene. In order to match the color appearance of the foreground and background, we utilize ideas from prior harmonization approaches to perform parameterized image edits in the albedo domain. To validate the effectiveness of our approach, we present results from challenging real-world composites and conduct a user study to objectively measure the enhanced realism achieved compared to state-of-the-art harmonization methods.

Flash is an essential tool as it often serves as the sole controllable light source in everyday photography. However, the use of flash is a binary decision at the time a photograph is captured with limited control over its characteristics such as strength or color. In this work, we study the computational control of the flash light in photographs taken with or without flash. We present a physically motivated intrinsic formulation for flash photograph formation and develop flash decomposition and generation methods for flash and no-flash photographs, respectively. We demonstrate that our intrinsic formulation outperforms alternatives in the literature and allows us to computationally control flash in in-the-wild images.