Rao, P;
Mallikarjun, BR;
Fox, G;
Weyrich, T;
Bickel, B;
Pfister, H;
Matusik, W;
... Elgharib, M; + view all
(2022)
VoRF: Volumetric Relightable Faces.
In:
BMVC 2022 - 33rd British Machine Vision Conference Proceedings.
British Machine Vision Association (BMVA)
Preview |
PDF
rao2022vorf.pdf - Published Version Download (5MB) | Preview |
Abstract
Portrait viewpoint and illumination editing is an important problem with several applications in VR/AR, movies, and photography. Comprehensive knowledge of geometry and illumination is critical for obtaining photorealistic results. Current methods are unable to explicitly model in 3D while handing both viewpoint and illumination editing from a single image. In this paper, we propose VoRF, a novel approach that can take even a single portrait image as input and relight human heads under novel illuminations that can be viewed from arbitrary viewpoints. VoRF represents a human head as a continuous volumetric field and learns a prior model of human heads using a coordinate-based MLP with individual latent spaces for identity and illumination. The prior model is learnt in an auto-decoder manner over a diverse class of head shapes and appearances, allowing VoRF to generalize to novel test identities from a single input image. Additionally, VoRF has a reflectance MLP that uses the intermediate features of the prior model for rendering One-Light-at-A-Time (OLAT) images under novel views. We synthesize novel illuminations by combining these OLAT images with target environment maps. Qualitative and quantitative evaluations demonstrate the effectiveness of VoRF for relighting and novel view synthesis, even when applied to unseen subjects under uncontrolled illuminations. Figure 1: We present VoRF, a learning framework that synthesizes novel views and relighting under any lighting conditions given a single image or a few posed images. VoRF has explicit control over the direction of a point light source and that allows the rendering of a basis of one-light-at-a-time (OLAT) images (c). Finally, given an environment map (see d, insets) VoRF can relight the input (d) by linearly combining the OLAT images.
| Type: | Proceedings paper |
|---|---|
| Title: | VoRF: Volumetric Relightable Faces |
| Event: | BMVC 2022 - 33rd British Machine Vision Conference |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://bmvc2022.mpi-inf.mpg.de/708/ |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10181905 |
Archive Staff Only
 |
View Item |
