research-article

High resolution passive facial performance capture

Authors:

Wolfgang Heidrich,

Alla ShefferAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 29, Issue 4

Article No.: 41, Pages 1 - 10

https://doi.org/10.1145/1778765.1778778

Published: 26 July 2010 Publication History

Abstract

We introduce a purely passive facial capture approach that uses only an array of video cameras, but requires no template facial geometry, no special makeup or markers, and no active lighting. We obtain initial geometry using multi-view stereo, and then use a novel approach for automatically tracking texture detail across the frames. As a result, we obtain a high-resolution sequence of compatibly triangulated and parameterized meshes. The resulting sequence can be rendered with dynamically captured textures, while also consistently applying texture changes such as virtual makeup.

Supplementary Material

JPG File (tp049-10.jpg)

Download
9.88 KB

Supplemental material. (041.zip)

Final-MainVideo.mp4 - the main paper video. Final-Additional.mp4 - Video of additional material.

Download
87.29 MB

MP4 File (tp049-10.mp4)

Download
157.82 MB

References

[1]

Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital emily project: photoreal facial modeling and animation. In ACM SIGGRAPH Courses, 1--15.

Digital Library

[2]

Beeler, T., Bickel, B., Sumner, R., Beardsley, P., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH).

Digital Library

[3]

Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graphics (Proc. SIGGRAPH), 33.

Digital Library

[4]

Blanz, V., Basso, C., Vetter, T., and Poggio, T. 2003. Reanimating faces in images and video. Computer Graphics Forum (Proc. Eurographics) 22, 3, 641--650.

[5]

Borshukov, G., Piponi, D., Larsen, O., Lewis, J. P., and Tempelaar-Lietz, C. 2003. Universal capture: image-based facial animation for "the matrix reloaded". In ACM SIGGRAPH Sketches & Applications.

Digital Library

[6]

Bouguet, J.-Y. 1999. Pyramidal implementation of the lucas kanade feature tracker: Description of the algorithm. Tech. rep., Intel Corporation, Microprocessor Research Labs.

[7]

Bradley, D., and Heidrich, W. 2010. Binocular camera calibration using rectification error. In Canadian Conference on Computer and Robot Vision.

Digital Library

[8]

Bradley, D., Boubekeur, T., and Heidrich, W. 2008. Accurate multi-view reconstruction using robust binocular stereo and surface meshing. In Proc. CVPR.

[9]

Bradley, D., Popa, T., Sheffer, A., Heidrich, W., and Boubekeur, T. 2008. Markerless garment capture. ACM Trans. Graphics (Proc. SIGGRAPH) 27, 3, 99.

Digital Library

[10]

Bradley, D., Atcheson, B., Ihrke, I., and Heidrich, W. 2009. Synchronization and rolling shutter compensation for consumer video camera arrays. In International Workshop on Projector-Camera Systems (PROCAMS 2009).

[11]

Canny, J. 1986. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 6, 679--698.

Digital Library

[12]

de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., eidel, H.-P. S., and Thrun, S. 2008. Performance capture from sparse multi-view video. ACM Trans. Graphics (Proc. SIGGRAPH), 98.

Digital Library

[13]

DeCarlo, D., and Metaxas, D. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In CVPR, 231.

Digital Library

[14]

DeCarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. Journal of Computer Vision 38, 2, 99--127.

Digital Library

[15]

Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads using input from video. In Proceedings of Computer Animation, 68.

Digital Library

[16]

Fiala, M., and Shu, C. 2008. Self-identifying patterns for plane-based camera calibration. Machine Vision and Applications 19, 4, 209--216.

Digital Library

[17]

Furukawa, Y., and Ponce, J. 2009. Dense 3d motion capture for human faces. In CVPR.

[18]

Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In SIGGRAPH '98: Proceedings of the 25th annual conference on Computer graphics and interactive techniques, 55--66.

Digital Library

[19]

Kadir, T., and Brady, M. 2001. Saliency, scale and image description. Int. J. Comput. Vision 45, 2, 83--105.

Digital Library

[20]

Lévy, B., Petitjean, S., Ray, N., and Maillot, J. 2002. Least squares conformal maps for automatic texture atlas generation. ACM Trans. Graph. 21, 3, 362--371.

Digital Library

[21]

Li, H., Roivainen, P., and Forcheimer, R. 1993. 3-d motion estimation in model-based facial image coding. IEEE Trans. Pattern Anal. Mach. Intell. 15, 6, 545--555.

Digital Library

[22]

Lin, I.-C., and Ouhyoung, M. 2005. Mirror mocap: Automatic and efficient capture of dense 3d facial motion parameters from video. The Visual Computer 21, 6, 355--372.

[23]

Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Eurographics Symposium on Rendering, 183--194.

Digital Library

[24]

Ma, W.-C., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., and Debevec, P. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 27, 5, 121.

Digital Library

[25]

Mohammed, U., Prince, S. J. D., and Kautz, J. 2009. Visiolization: generating novel facial images. ACM Trans. Graph. 28, 3, 57.

Digital Library

[26]

Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3, 313--318.

Digital Library

[27]

Pighin, F. H., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3d model-based tracking. In ICCV, 143--150.

[28]

Shewchuk, J. 1996. Triangle: Engineering a 2D Quality Mesh Generator and Delaunay Triangulator. In Applied Computational Geometry: Towards Geometric Engineering, M. C. Lin and D. Manocha, Eds., vol. 1148 of Lecture Notes in Computer Science. Springer-Verlag, 203--222.

Digital Library

[29]

Wang, Y., Huang, X., Lee, C.-S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dyanmic 3-d facial expressions. In Computer Graphics Forum, 677--686.

[30]

Williams, L. 1990. Performance-driven facial animation. SIGGRAPH Comput. Graph. 24, 4, 235--242.

Digital Library

[31]

Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Trans. Graphics (Proc. SIGGRAPH), 548--558.

Digital Library

Cited By

Tian ZLiang BFang HWeng D(2024)Dynamic 4D facial capture pipeline with appearance driven progressive retopology based on optical flowOptics Express10.1364/OE.52984632:18(31830)Online publication date: 19-Aug-2024
https://doi.org/10.1364/OE.529846
Ha HHwang IMonzon NCho JKim DBaek SMuñoz AGutierrez DKim M(2024)Polarimetric BSSRDF Acquisition of Dynamic FacesACM Transactions on Graphics10.1145/368776743:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687767
Bai SWang TLi CVenkatesh ASimon TCao CSchwartz GSaragih JSheikh YWei S(2024)Universal Facial Encoding of Codec Avatars from VR HeadsetsACM Transactions on Graphics10.1145/365823443:4(1-22)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658234
Show More Cited By

Index Terms

High resolution passive facial performance capture

Recommendations

High resolution passive facial performance capture
SIGGRAPH '10: ACM SIGGRAPH 2010 papers

We introduce a purely passive facial capture approach that uses only an array of video cameras, but requires no template facial geometry, no special makeup or markers, and no active lighting. We obtain initial geometry using multi-view stereo, and then ...
High-quality passive facial performance capture using anchor frames

We present a new technique for passive and markerless facial performance capture based on anchor frames. Our method starts with high resolution per-frame geometry acquisition using state-of-the-art stereo reconstruction, and proceeds to establish a ...
Facial hair tracking for high fidelity performance capture

Facial hair is a largely overlooked topic in facial performance capture. Most production pipelines in the entertainment industry do not have a way to automatically capture facial hair or track the skin underneath it. Thus, actors are asked to shave clean ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 29, Issue 4

July 2010

942 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1778765

Issue’s Table of Contents

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 July 2010

Published in TOG Volume 29, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

194
Total Citations
View Citations
2,607
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)4

Reflects downloads up to 30 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tian ZLiang BFang HWeng D(2024)Dynamic 4D facial capture pipeline with appearance driven progressive retopology based on optical flowOptics Express10.1364/OE.52984632:18(31830)Online publication date: 19-Aug-2024
https://doi.org/10.1364/OE.529846
Ha HHwang IMonzon NCho JKim DBaek SMuñoz AGutierrez DKim M(2024)Polarimetric BSSRDF Acquisition of Dynamic FacesACM Transactions on Graphics10.1145/368776743:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687767
Bai SWang TLi CVenkatesh ASimon TCao CSchwartz GSaragih JSheikh YWei S(2024)Universal Facial Encoding of Codec Avatars from VR HeadsetsACM Transactions on Graphics10.1145/365823443:4(1-22)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658234
Chen YWang LLi QXiao HZhang SYao HLiu Y(2024)MonoGaussianAvatar: Monocular Gaussian Point-based Head AvatarACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657499(1-9)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657499
Xu YChen BLi ZZhang HWang LZheng ZLiu Y(2024)Gaussian Head Avatar: Ultra High-Fidelity Head Avatar via Dynamic Gaussians2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00189(1931-1941)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00189
Peng SZhu XYi DQian CLei Z(2024)Formulating facial mesh tracking as a differentiable optimization problem: a backpropagation-based solutionVisual Intelligence10.1007/s44267-024-00054-x2:1Online publication date: 19-Jul-2024
https://doi.org/10.1007/s44267-024-00054-x
Ming XLi JLing JZhang LXu F(2024)High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse RenderingComputer Vision – ECCV 202410.1007/978-3-031-72897-6_7(106-125)Online publication date: 2-Dec-2024
https://doi.org/10.1007/978-3-031-72897-6_7
Kim JYoo T(2023)A Study on the Digital Twin Pipeline for Facial Digitizing System OptimizationJOURNAL OF BROADCAST ENGINEERING10.5909/JBE.2023.28.5.53028:5(530-544)Online publication date: 30-Sep-2023
https://doi.org/10.5909/JBE.2023.28.5.530
Li MHuang HZheng YLi MSang NMa C(2023)Implicit Neural Deformation for Sparse‐View Face ReconstructionComputer Graphics Forum10.1111/cgf.1470441:7(601-610)Online publication date: 20-Mar-2023
https://doi.org/10.1111/cgf.14704
Chen SLai YXia SRosin PGao L(2023)3D Face Reconstruction and Gaze Tracking in the HMD for Virtual InteractionIEEE Transactions on Multimedia10.1109/TMM.2022.315682025(3166-3179)Online publication date: 2023
https://doi.org/10.1109/TMM.2022.3156820
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents