research-article

Practical dynamic facial appearance modeling and acquisition

Authors:

Jérémy Riviere,

Abhijeet Ghosh,

Thabo BeelerAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 37, Issue 6

Article No.: 232, Pages 1 - 13

https://doi.org/10.1145/3272127.3275073

Published: 04 December 2018 Publication History

Abstract

We present a method to acquire dynamic properties of facial skin appearance, including dynamic diffuse albedo encoding blood flow, dynamic specular intensity, and per-frame high resolution normal maps for a facial performance sequence. The method reconstructs these maps from a purely passive multi-camera setup, without the need for polarization or requiring temporally multiplexed illumination. Hence, it is very well suited for integration with existing passive systems for facial performance capture. To solve this seemingly underconstrained problem, we demonstrate that albedo dynamics during a facial performance can be modeled as a combination of: (1) a static, high-resolution base albedo map, modeling full skin pigmentation; and (2) a dynamic, one-dimensional component in the CIE L*a*b* color space, which explains changes in hemoglobin concentration due to blood flow. We leverage this albedo subspace and additional constraints on appearance and surface geometry to also estimate specular reflection parameters and resolve high-resolution normal maps with unprecedented detail in a passive capture system. These constraints are built into an inverse rendering framework that minimizes the difference of the rendered face to the captured images, incorporating constraints from multiple views for every texel on the face. The presented method is the first system capable of capturing high-quality dynamic appearance maps at full resolution and video framerates, providing a major step forward in the area of facial appearance acquisition.

References

[1]

Sameer Agarwal, Keir Mierle, and Others. 2016. Ceres Solver. http://ceres-solver.org.

[2]

Oleg Alexander, Mike Rogers, William Lambeth, Jen-Yuan Chiang, Wan-Chun Ma, Chuan-Chang Wang, and Paul Debevec. 2010. The digital emily project: Achieving a photorealistic digital actor. Computer Graphics and Applications, IEEE 30, 4 (2010), 20--31.

Digital Library

[3]

R Rox Anderson and John A Parrish. 1981. The optics of human skin. Journal of investigative dermatology 77, 1 (1981), 13--19.

[4]

Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross. 2010. High-Quality Single-Shot Capture of Facial Geometry. ACM Transactions on Graphics (TOG) 29, 3 (2010), 40:1--40:9.

Digital Library

[5]

Thabo Beeler and Derek Bradley. 2014. Rigid stabilization of facial expressions. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1--9.

Digital Library

[6]

Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul Beardsley, Craig Gotsman, Robert W. Sumner, and Markus Gross. 2011. High-quality passive facial performance capture using anchor frames. ACM Transactions on Graphics (ACM) 30, Article 75 (August 2011), 10 pages. Issue 4.

Digital Library

[7]

Derek Bradley, Wolfgang Heidrich, Tiberiu Popa, and Alla Sheffer. 2010. High resolution passive facial performance capture. ACM Transactions on Graphics (TOG) 29, 4 (2010), 41.

Digital Library

[8]

Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert. 2004. High accuracy optical flow estimation based on a theory for warping. In ECCV. 25--36.

[9]

Chen Cao, Derek Bradley, Kun Zhou, and Thabo Beeler. 2015. Real-time High-fidelity Facial Performance Capture. ACM Transactions on Graphics (TOG) 34, 4, Article 46 (July 2015), 9 pages.

Digital Library

[10]

A Chardon, I Cretois, and C Hourseau. 1991. Skin colour typology and suntanning pathways. International journal of cosmetic science 13, 4 (1991), 191--208.

[11]

Robert Cook and Kenneth E. Torrance. 1981. A reflectance model for computer graphics. Computer Graphics (SIGGRAPH '81 Proceedings) 15, 3 (1981), 301--316.

Digital Library

[12]

Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 145--156.

Digital Library

[13]

Eugene d'Eon, David Luebke, and Eric Enderton. 2007. Efficient Rendering of Human Skin. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR'07). 147--157.

Digital Library

[14]

Craig Donner and Henrik Wann Jensen. 2005. Light Diffusion in Multi-Layered Translucent Materials. ACM Transactions on Graphics (TOG) 24, 3 (2005), 1032--1039.

Digital Library

[15]

Craig Donner and Henrik Wann Jensen. 2006. A Spectral BSSRDF for Shading Human Skin. In Proceedings of the 17th Eurographics Conference on Rendering Techniques (EGSR '06). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 409--417.

Digital Library

[16]

Craig Donner, Tim Weyrich, Eugene d'Eon, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2008. A Layered, Heterogeneous Reflectance Model for Acquiring and Rendering Human Skin. ACM Transactions on Graphics (TOG) 27, 5 (Dec. 2008), 140:1--140:12.

Digital Library

[17]

Martin Fuchs, Volker Blanz, Hendrik Lensch, and Hans-Peter Seidel. 2005. Reflectance from Images: A Model-Based Approach for Human Faces. IEEE Transactions on Visualization and Computer Graphics 11, 3 (May 2005), 296--305.

Digital Library

[18]

Graham Fyffe and Paul Debevec. 2015. Single-Shot Reflectance Measurement from Polarized Color Gradient Illumination. In International Conference on Computational Photography (ICCP).

[19]

Graham Fyffe, Paull Graham, Borom Tunwattanapong, Abhijeet Ghosh, and Paul Debevec. 2016. Near-Instant Capture of High-Resolution Facial Geometry and Reflectance. Computer Graphics Forum (CGF) 35, 2 (2016), 353--363.

[20]

Graham Fyffe, Tim Hawkins, Chris Watts, Wan-Chun Ma, and Paul Debevec. 2011. Comprehensive Facial Performance Capture. Computer Graphics Forum (CGF) 30, 2 (2011).

[21]

Graham Fyffe, Andrew Jones, Oleg Alexander, Ryosuke Ichikari, and Paul Debevec. 2014. Driving High-Resolution Facial Scans with Video Performance Capture. ACM Transactions on Graphics (TOG) 34, 1, Article 8 (Dec. 2014), 14 pages.

Digital Library

[22]

Pablo Garrido, Levi Valgaerts, Chenglei Wu, and Christian Theobalt. 2013. Reconstructing Detailed Dynamic Face Geometry from Monocular Video. ACM Transactions on Graphics (TOG) 32, 6 (November 2013), 158:1--158:10.

Digital Library

[23]

Sergio Garrido-Jurado, Rafael Mu noz Salinas, F.J. Madrid-Cuevas, and Manuel Jesus Marín-Jiménez. 2014. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition 47, 6 (2014), 2280 -- 2292.

Digital Library

[24]

Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Transactions on Graphics (TOG) 30, 6 (2011), 129.

Digital Library

[25]

Abhijeet Ghosh, Tim Hawkins, Pieter Peers, Sune Frederiksen, and Paul Debevec. 2008. Practical Modeling and Acquisition of Layered Facial Reflectance. ACM Trans. Graph. 27, 5 (Dec. 2008), 139:1--139:10.

Digital Library

[26]

Paulo Gotardo, Tomas Simon, Yaser Sheikh, and Iain Matthews. 2015. Photogeometric Scene Flow for High-Detail Dynamic 3D Reconstruction. In IEEE International Journal of Computer Vision. 846--854.

Digital Library

[27]

Paul Graham, Borom Tunwattanapong, Jay Busch, Xueming Yu, Andrew Jones, Paul Debevec, and Abhijeet Ghosh. 2013. Measurement-Based Synthesis of Facial Microgeometry. Computer Graphics Forum (CGF) 32, 2 (2013), 335--344.

[28]

Tim Hawkins, Andreas Wenger, Chris Tchou, Andrew Gardner, Fredrik Göransson, and Paul Debevec. 2004. Animatable Facial Reflectance Fields. In Proceedings of the Fifteenth Eurographics Conference on Rendering Techniques (EGSR'04). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 309--319.

Digital Library

[29]

Loc Huynh, Weikai Chen, Shunsuke Saito, Jun Xing, Koki Nagano, Andrew Jones, Paul Debevec, and Hao Li. 2018. Mesoscopic Facial Geometry Inference Using Deep Neural Networks. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR).

[30]

Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly. 2015. Dynamic 3D Avatar Creation from Hand-held Video Input. ACM Transactions on Graphics (TOG) 34, 4 (2015), 45:1--45:14.

Digital Library

[31]

Henrik Wann Jensen, Steve Marschner, Marc Levoy, and Pat Hanrahan. 2001. A practical model for subsurface light transport. In In Proceedings of ACM SIGGRAPH. 511--518.

Digital Library

[32]

Jorge Jimenez, Timothy Scully, Nuno Barbosa, Craig Donner, Xenxo Alvarez, Teresa Vieira, Paul Matts, Verónica Orvalho, Diego Gutierrez, and Tim Weyrich. 2010. A Practical Appearance Model for Dynamic Facial Color. ACM Transactions on Graphics (TOG) 29, 6, Article 141 (Dec. 2010), 10 pages.

Digital Library

[33]

Jorge Jimenez, Veronica Sundstedt, and Diego Gutierrez. 2009. Screen-space perceptual rendering of human skin. ACM Transactions on Applied Perception 6, 4 (2009), 23:1--23:15.

Digital Library

[34]

Oliver Klehm, Fabrice Rousselle, Marios Papas, Derek Bradley, Christophe Hery, Bernd Bickel, Wojciech Jarosz, and Thabo Beeler. 2015. Recent Advances in Facial Appearance Capture. Computer Graphics Forum (CGF) 34, 2 (May 2015), 709--733.

Digital Library

[35]

Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec. 2007. Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR'07). Eurographics Association, 183--194.

Digital Library

[36]

Wan-Chun Ma, Andrew Jones, Jen-Yuan Chiang, Tim Hawkins, Sune Frederiksen, Pieter Peers, Marko Vukovic, Ming Ouhyoung, and Paul Debevec. 2008. Facial Performance Synthesis Using Deformation-driven Polynomial Displacement Maps. ACM Transactions on Graphics (TOG) 27, 5, Article 121 (Dec. 2008), 10 pages.

Digital Library

[37]

Stephen R. Marschner, Stephen H. Westin, Eric P. F. Lafortune, Kenneth E. Torrance, and Donald P. Greenberg. 1999. Image-based BRDF Measurement Including Human Skin. In Proceedings of the 10th Eurographics Conference on Rendering (EGWR'99). 131--144.

Digital Library

[38]

Koki Nagano, Graham Fyffe, Oleg Alexander, Jernej Barbic, Hao Li, Abhijeet Ghosh, and Paul E Debevec. 2015. Skin microstructure deformation with displacement map convolution. ACM Transactions on Graphics (TOG) 34, 4 (2015), 109.

Digital Library

[39]

Ruth Onn and Alfred Bruckstein. 1990. Integrability disambiguates surface recovery in two-image photometric stereo. International Journal of Computer Vision 5, 1 (1990), 105--113.

Digital Library

[40]

Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, and Hao Li. 2017. Photorealistic Facial Texture Inference Using Deep Neural Networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]

Christoph Schlick. 1994. An inexpensive BRDF model for physically-based rendering. Computer Graphics Forum (CGF) 13, 3 (1994), 233--246.

[42]

Scott Shafer. 1985. Using Color to Separate Reflection Components. Color Research & Application 10, 4 (1985), 210--218.

Digital Library

[43]

Gaurav Sharma and Raja Bala. 2002. Digital color imaging handbook. CRC press.

Digital Library

[44]

Fuhao Shi, Hsiang-Tao Wu, Xin Tong, and Jinxiang Chai. 2014. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Transactions on Graphics (TOG) 33, 6 (2014), 222.

Digital Library

[45]

Levi Valgaerts, Chenglei Wu, Andrés Bruhn, Hans-Peter Seidel, and Christian Theobalt. 2012. Lightweight Binocular Facial Performance Capture under Uncontrolled Lighting. ACM Transactions on Graphics (TOG) 31, 6 (November 2012), 187:1--187:11.

Digital Library

[46]

Javier von der Pahlen, Jorge Jimenez, Etienne Danvoye, Paul Debevec, Graham Fyffe, and Oleg Alexander. 2014. Digital Ira and Beyond: Creating Real-time Photoreal Digital Actors. In ACM SIGGRAPH 2014 Courses (SIGGRAPH '14). ACM, New York, NY, USA, Article 1, 384 pages.

Digital Library

[47]

Andreas Wenger, Andrew Gardner, Chris Tchou, Jonas Unger, Tim Hawkins, and Paul Debevec. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. 24, 3 (2005), 756--764.

Digital Library

[48]

Tim Weyrich, Jason Lawrence, Hendrik P. A. Lensch, Szymon Rusinkiewicz, and Todd Zickler. 2009. Principles of Appearance Acquisition and Representation. Found. Trends. Comput. Graph. Vis. 4, 2 (Feb. 2009), 75--191.

Digital Library

[49]

Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, and Markus Gross. 2006. Analysis of Human Faces Using a Measurement-based Skin Reflectance Model. ACM Transactions on Graphics (TOG) 25, 3 (July 2006), 1013--1024.

Digital Library

[50]

Cyrus A Wilson, Abhijeet Ghosh, Pieter Peers, Jen-Yuan Chiang, Jay Busch, and Paul Debevec. 2010. Temporal upsampling of performance geometry using photometric alignment. ACM Transactions on Graphics (TOG) 29, 2 (2010), 17.

Digital Library

[51]

Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. An anatomically-constrained local deformation model for monocular face capture. ACM Transactions on Graphics (TOG) 35, 4 (2016), 115.

Digital Library

[52]

Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, and Hao Li. 2018. High-Fidelity Facial Reflectance and Geometry Inference From an Unconstrained Image. ACM Transactions on Graphics (TOG) 37, 4 (2018).

Digital Library

Cited By

Bridgeman LRainer GGhosh A(2024)High-Quality Facial Geometry from Sparse Heterogeneous Devices under Active IlluminationProceedings of 21st ACM SIGGRAPH Conference on Visual Media Production10.1145/3697294.3697296(1-10)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3697294.3697296
Ha HHwang IMonzon NCho JKim DBaek SMuñoz AGutierrez DKim M(2024)Polarimetric BSSRDF Acquisition of Dynamic FacesACM Transactions on Graphics10.1145/368776743:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687767
Medin SLi GDu RGarbin SDavidson PWornell GBeeler TMeka A(2024)FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic FacesProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36513047:1(1-17)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3651304
Show More Cited By

Index Terms

Practical dynamic facial appearance modeling and acquisition
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Appearance and texture representations
      2. Image and video acquisition
        3D imaging
  2. Computer graphics
    1. Rendering
      1. Reflectance modeling

Recommendations

Single-shot high-quality facial geometry and skin appearance capture

We propose a new light-weight face capture system capable of reconstructing both high-quality geometry and detailed appearance maps from a single exposure. Unlike currently employed appearance acquisition systems, the proposed technology does not require ...
Image-based rendering of diffuse, specular and glossy surfaces from a single image
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques

In this paper, we present a new method to recover an approximation of the bidirectional reflectance distribution function (BRDF) of the surfaces present in a real scene. This is done from a single photograph and a 3D geometric model of the scene. The ...
Practical modeling and acquisition of layered facial reflectance
SIGGRAPH Asia '08: ACM SIGGRAPH Asia 2008 papers

We present a practical method for modeling layered facial reflectance consisting of specular reflectance, single scattering, and shallow and deep subsurface scattering. We estimate parameters of appropriate reflectance models for each of these layers ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 37, Issue 6

December 2018

1401 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3272127

Editor:
Takeo Igarashi
The University of Tokyo, Japan

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2018

Published in TOG Volume 37, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

50
Total Citations
View Citations
843
Total Downloads

Downloads (Last 12 months)65
Downloads (Last 6 weeks)7

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bridgeman LRainer GGhosh A(2024)High-Quality Facial Geometry from Sparse Heterogeneous Devices under Active IlluminationProceedings of 21st ACM SIGGRAPH Conference on Visual Media Production10.1145/3697294.3697296(1-10)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3697294.3697296
Ha HHwang IMonzon NCho JKim DBaek SMuñoz AGutierrez DKim M(2024)Polarimetric BSSRDF Acquisition of Dynamic FacesACM Transactions on Graphics10.1145/368776743:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687767
Medin SLi GDu RGarbin SDavidson PWornell GBeeler TMeka A(2024)FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic FacesProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36513047:1(1-17)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3651304
Rao PFox GMeka AB R MZhan FWeyrich TBickel BPfister HMatusik WElgharib MTheobalt C(2024)Lite2Relight: 3D-aware Single Image Portrait RelightingACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657470(1-12)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657470
Kavoosighafi BHajisharif SMiandji EBaravdish GCao WUnger J(2024)Deep SVBRDF Acquisition and Modelling: A SurveyComputer Graphics Forum10.1111/cgf.1519943:6Online publication date: 16-Sep-2024
https://doi.org/10.1111/cgf.15199
Xu YChandran PWeiss SGross MZoss GBradley D(2024)Artist-Friendly Relightable and Animatable Neural Heads2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00238(2457-2467)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00238
Han YLyu JXu F(2024)High-Quality Facial Geometry and Appearance Capture at Home2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00073(697-707)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00073
Li XGuarnera GLin AGhosh A(2024)Practical Measurement and Neural Encoding of Hyperspectral Skin Reflectance2024 International Conference on 3D Vision (3DV)10.1109/3DV62453.2024.00116(1301-1309)Online publication date: 18-Mar-2024
https://doi.org/10.1109/3DV62453.2024.00116
Kim JYoo T(2023)A Study on the Digital Twin Pipeline for Facial Digitizing System OptimizationJOURNAL OF BROADCAST ENGINEERING10.5909/JBE.2023.28.5.53028:5(530-544)Online publication date: 30-Sep-2023
https://doi.org/10.5909/JBE.2023.28.5.530
Li YWeng DChen J(2023)Non-Rigid Point Cloud Matching Based on Invariant Structure for Face DeformationElectronics10.3390/electronics1204082812:4(828)Online publication date: 6-Feb-2023
https://doi.org/10.3390/electronics12040828
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents