DANI-Net: Uncalibrated Photometric Stereo by Differentiable Shadow Handling, Anisotropic Reflectance Modeling, and Neural Inverse Rendering

Zongrui Li1, Qian Zheng2, Boxin Shi3, Gang Pan2, Xudong Jiang1,
1Nanyang Technological University, 2Zhejiang University, 3Peking University
CVPR 2023
Paper Code arXiv

Abstract

Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguity brought by the unknown light. Although the ambiguity is alleviated on non-Lambertian objects, the problem is still difficult to solve for more general objects with complex shapes introducing irregular shadows and general materials with complex reflectance like anisotropic reflectance. To exploit cues from shadow and reflectance to solve UPS and improve performance on general materials, we propose DANI-Net, an inverse rendering framework with differentiable shadow handling and anisotropic reflectance modeling. Unlike most previous methods that use non-differentiable shadow maps and assume isotropic material, our network benefits from cues of shadow and anisotropic reflectance through two differentiable paths. Experiments on multiple real-world datasets demonstrate our superior and robust performance.

Overview

The proposed DANI-Net differs from other PS or UPS methods in two aspects:

  1. Shadow Handling. The path in the sequence of inverse rendering errors, shadow maps, and surface normal maps (or light conditions) of the DANI-Net is differentiable;
  2. Reflectance Modeling. DANI-Net adopts an anisotropic reflectance model.
The proposed DANI-Net produces a smoother and more realistic shadow map, thanks to the differentiable shadow handling, and renders a more realistic image and Bidirectional Reflectance Distribution Function (BRDF) sphere (of the reference point) due to the anisotropic reflectance modeling.

Teaser



Soft Shadow Handling

DANI-Net implements a differentiable shadow handling method that fully exploits shadow cues to solve UPS. DANI-Net can generate realistic soft shaodw maps given arbitrary light directions.

Teaser



Anisotropic Reflectance Modeling

DANI-Net represents the spatially varying anisotropic specularity as a weighted sum of Anisotropic Spherical Gaussian (ASG). Thanks to ASG, DANI-Net can generate realistic appearance given arbitrary light conditions.

Teaser



Relighting

Given arbitrary directional lights, DANI-Net can relight the object with realistic appearance.

Teaser



Video Presentation




BibTeX

@inproceedings{li2023dani,
      title={DANI-Net: Uncalibrated Photometric Stereo by Differentiable Shadow Handling, Anisotropic Reflectance Modeling, and Neural Inverse Rendering},
      author={Li, Zongrui and Zheng, Qian and Shi, Boxin and Pan, Gang and Jiang, Xudong},
      booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
      year={2023}}