research-article

Detailed spatio-temporal reconstruction of eyelids

Authors:

Markus GrossAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 34, Issue 4

Article No.: 44, Pages 1 - 11

https://doi.org/10.1145/2766924

Published: 27 July 2015 Publication History

Abstract

In recent years we have seen numerous improvements on 3D scanning and tracking of human faces, greatly advancing the creation of digital doubles for film and video games. However, despite the high-resolution quality of the reconstruction approaches available, current methods are unable to capture one of the most important regions of the face - the eye region. In this work we present the first method for detailed spatio-temporal reconstruction of eyelids. Tracking and reconstructing eyelids is extremely challenging, as this region exhibits very complex and unique skin deformation where skin is folded under while opening the eye. Furthermore, eyelids are often only partially visible and obstructed due to self-occlusion and eyelashes. Our approach is to combine a geometric deformation model with image data, leveraging multi-view stereo, optical flow, contour tracking and wrinkle detection from local skin appearance. Our deformation model serves as a prior that enables reconstruction of eyelids even under strong self-occlusions caused by rolling and folding skin as the eye opens and closes. The output is a person-specific, time-varying eyelid reconstruction with anatomically plausible deformations. Our high-resolution detailed eyelids couple naturally with current facial performance capture approaches. As a result, our method can largely increase the fidelity of facial capture and the creation of digital doubles.

Supplementary Material

ZIP File (a44-bermano.zip)

Supplemental files

Download
116.13 MB

References

[1]

Alexander, O., Fyffe, G., Busch, J., Yu, X., Ichikari, R., Jones, A., Debevec, P., Jimenez, J., Danvoye, E., Antionazzi, B., et al. 2013. Digital ira: creating a real-time photoreal digital actor. In ACM SIGGRAPH 2013 Posters.

Digital Library

[2]

Bando, Y., Kuratate, T., and Nishita, T. 2002. A simple method for modeling wrinkles on human skin. In Proc. Pacific Graphics.

Digital Library

[3]

Beeler, T., and Bradley, D. 2014. Rigid stabilization of facial expressions. ACM Trans. Graph. 33, 4 (July), 44:1--44:9.

Digital Library

[4]

Beeler, T., Bickel, B., Sumner, R., Beardsley, P., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH).

Digital Library

[5]

Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 75:1--75:10.

Digital Library

[6]

Beeler, T., Bickel, B., Noris, G., Beardsley, P., Marschner, S., Sumner, R. W., and Gross, M. 2012. Coupled 3d reconstruction of sparse facial hair and skin. ACM Trans. Graph. 31, 4 (July), 117:1--117:10.

Digital Library

[7]

Bérard, P., Bradley, D., Nitti, M., Beeler, T., and Gross, M. 2014. High-quality capture of eyes. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33, 6.

Digital Library

[8]

Bermano, A. H., Bradley, D., Beeler, T., Zund, F., Nowrouzezahrai, D., Baran, I., Sorkine-Hornung, O., Pfister, H., Sumner, R. W., Bickel, B., and Gross, M. 2014. Facial performance enhancement using dynamic shape space analysis. ACM Trans. Graphics 33, 2.

Digital Library

[9]

Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graphics (Proc. SIGGRAPH), 33.

Digital Library

[10]

Bickel, B., Lang, M., Botsch, M., Otaduy, M. A., and Gross, M. 2008. Pose-space animation and transfer of facial details. In Proc. SCA, 57--66.

Digital Library

[11]

Botsch, M., and Sorkine, O. 2008. On linear variational surface deformation methods. Visualization and Computer Graphics, IEEE Transactions on 14, 1, 213--230.

Digital Library

[12]

Bouaziz, S., Wang, Y., and Pauly, M. 2013. Online modeling for realtime facial animation. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 40:1--40:10.

Digital Library

[13]

Bradley, D., Popa, T., Sheffer, A., Heidrich, W., and Boubekeur, T. 2008. Markerless garment capture. ACM Trans. Graphics (Proc. SIGGRAPH), 99.

Digital Library

[14]

Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 41:1--41:10.

Digital Library

[15]

Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In ECCV. Springer, 25--36.

[16]

Canny, J. 1986. A computational approach to edge detection. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 6, 679--698.

Digital Library

[17]

Cao, X., Wei, Y., Wen, F., and Sun, J. 2012. Face alignment by explicit shape regression. In IEEE CVPR, 2887--2894.

Digital Library

[18]

Cao, C., Weng, Y., Lin, S., and Zhou, K. 2013. 3d shape regression for real-time facial animation. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 41:1--41:10.

Digital Library

[19]

Cao, C., Hou, Q., and Zhou, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans. Graphics (Proc. SIGGRAPH) 33, 4, 43:1--43:10.

Digital Library

[20]

Cootes, T. F., Edwards, G. J., and Taylor, C. J. 2001. Active appearance models. IEEE Transactions on pattern analysis and machine intelligence 23, 6, 681--685.

Digital Library

[21]

Deng, Z., Lewis, J., and Neumann, U. 2005. Automated eye motion using texture synthesis. CGA 25, 2.

Digital Library

[22]

Dutreve, L., Meyer, A., and Bouakaz, S. 2011. Easy acquisition and real-time animation of facial wrinkles. Comput. Animat. Virtual Worlds 22, 2-3, 169--176.

Digital Library

[23]

Feng, W.-W., Yu, Y., and Kim, B.-U. 2010. A deformation transformer for real-time cloth animation. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, 108:1--108:9.

Digital Library

[24]

François, G., Gautron, P., Breton, G., and Bouatouch, K. 2009. Image-based modeling of the human eye. IEEE TVCG 15, 5, 815--827.

Digital Library

[25]

Garrido, P., Valgaerts, L., Wu, C., and Theobalt, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. In ACM Trans. Graphics (Proc. SIGGRAPH Asia), vol. 32, 158:1--158:10.

Digital Library

[26]

Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 6, 129:1--129:10.

Digital Library

[27]

Huang, H., Chai, J., Tong, X., and Wu, H.-T. 2011. Leveraging motion capture and 3d scanning for high-fidelity facial performance acquisition. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, 74:1--74:10.

Digital Library

[28]

Kähler, K., Haber, J., Yamauchi, H., and Seidel, H.-P. 2002. Head shop: Generating animated head models with anatomical structure. In Proc. SCA, 55--63.

Digital Library

[29]

Kim, D., Koh, W., Narain, R., Fatahalian, K., Treuille, A., and O'Brien, J. F. 2013. Near-exhaustive precomputation of secondary cloth effects. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 87:1--87:8.

Digital Library

[30]

Klaudiny, M., and Hilton, A. 2012. High-detail 3d capture and non-sequential alignment of facial performance. In 3DIM-PVT.

Digital Library

[31]

Larboulette, C., and Cani, M.-P. 2004. Real-time dynamic wrinkles. In Proc. Computer Graphics Int., 522--525.

Digital Library

[32]

Le, B. H., Ma, X., and Deng, Z. 2012. Live speech driven head-and-eye motion generators. IEEE TVCG 18, 11, 1902--1914.

Digital Library

[33]

Lee, S. P., Badler, J. B., and Badler, N. I. 2002. Eyes alive. ACM Trans. Graphics (Proc. SIGGRAPH) 21, 3, 637--644.

Digital Library

[34]

Li, H., Yu, J., Ye, Y., and Bregler, C. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 42:1--42:10.

Digital Library

[35]

Li, J., Xu, W., Cheng, Z., Xu, K., and Klein, R. 2015. Lightweight wrinkle synthesis for 3d facial modeling and animation. Computer-Aided Design 58, 0, 117--122.

Digital Library

[36]

Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Eurographics Symposium on Rendering, 183--194.

Digital Library

[37]

Ma, W.-C., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., and Debevec, P. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 27, 5, 121.

Digital Library

[38]

Magnenat-Thalmann, N., Kalra, P., Luc Leveque, J., Bazin, R., Batisse, D., and Querleux, B. 2002. A computational skin model: Fold and wrinkle formation. Trans. Info. Tech. Biomed. 6, 4, 317--323.

Digital Library

[39]

Müller, M., and Chentanez, N. 2010. Wrinkle meshes. In Proc. SCA, 85--92.

Digital Library

[40]

Perona, P., and Malik, J. 1990. Scale-space and edge detection using anisotropic diffusion. Pattern Analysis and Machine Intelligence, IEEE Transactions on 12, 7 (Jul), 629--639.

Digital Library

[41]

Peterson, M. F., and Eckstein, M. P. 2012. Looking just below the eyes is optimal across face recognition tasks. Proceedings of the National Academy of Sciences 109, 48.

[42]

Popa, T., Zhou, Q., Bradley, D., Kraevoy, V., Fu, H., Sheffer, A., and Heidrich, W. 2009. Wrinkling captured garments using space-time data-driven deformation. Computer Graphics Forum (Proc. Eurographics) 28, 2, 427--435.

[43]

Rhee, T., Hwang, Y., Kim, J. D., and Kim, C. 2011. Real-time facial animation from live video tracking. In Proc. SCA.

Digital Library

[44]

Rohmer, D., Popa, T., Cani, M.-P., Hahmann, S., and Sheffer, A. 2010. Animation wrinkling: Augmenting coarse cloth simulations with realistic-looking wrinkles. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 29, 6, 157:1--157:8.

Digital Library

[45]

Ruhland, K., Andrist, S., Badler, J., Peters, C., Badler, N., Gleicher, M., Mutlu, B., and McDonnell, R. 2014. Look me in the eyes: A survey of eye and gaze animation for virtual agents and artificial systems. In Eurographics State of the Art Reports, 69--91.

[46]

Sagar, M. A., Bullivant, D., Mallinson, G. D., and Hunter, P. J. 1994. A virtual environment and model of the eye for surgical simulation. In Proceedings of Computer Graphics and Interactive Techniques, 205--212.

Digital Library

[47]

Seiler, M., Spillmann, J., and Harders, M. 2012. Enriching coarse interactive elastic objects with high-resolution data-driven deformations. In Proc. SCA, 9--17.

Digital Library

[48]

Shi, F., Wu, H.-T., Tong, X., and Chai, J. 2014. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33.

Digital Library

[49]

Smith, M. L., Cottrell, G. W., Gosselin, F., and Schyns, P. G. 2005. Transmitting and decoding facial expressions. Psychological Science 16, 3, 184--189.

[50]

Sorkine, O., and Alexa, M. 2007. As-rigid-as-possible surface modeling. In Symposium on Geometry processing, vol. 4.

Digital Library

[51]

Suwajanakorn, S., Kemelmacher-Shlizerman, I., and Seitz, S. M. 2014. Total moving face reconstruction. In ECCV.

[52]

Trutoiu, L. C., Carter, E. J., Matthews, I., and Hodgins, J. K. 2011. Modeling and animating eye blinks. ACM Trans. Appl. Percept. 8, 3.

Digital Library

[53]

Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 31, 6.

Digital Library

[54]

Wang, H., Hecht, F., Ramamoorthi, R., and O'Brien, J. F. 2010. Example-based wrinkle synthesis for clothing animation. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, 107:1--107:8.

Digital Library

[55]

Warburton, M., and Maddock, S. 2014. Physically-based forehead animation including wrinkles. Comput. Animat. Virtual Worlds.

Digital Library

[56]

Weise, T., Bouaziz, S., Li, H., and Pauly, M. 2011. Real-time performance-based facial animation. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, 77:1--77:10.

Digital Library

[57]

Weissenfeld, A., Liu, K., and Ostermann, J. 2010. Video-realistic image-based eye animation via statistically driven state machines. The Visual Computer 26, 9, 1201--1216.

Digital Library

[58]

Wu, C., Varanasi, K., Liu, Y., Seidel, H.-P., and Theobalt, C. 2011. Shading-based dynamic shape refinement from multi-view video under general illumination. In ICCV.

Digital Library

[59]

Zhang, Y., Sim, T., and Tan, C. 2005. Simulating wrinkles in facial expressions on an anatomy-based face. In Proc. ICCS, 207--215.

Digital Library

[60]

Zurdo, J. S., Brito, J. P., and Otaduy, M. A. 2013. Animating wrinkles by example on non-skinned cloth. IEEE TVCG 19, 1, 149--158.

Digital Library

Cited By

Li GSarkar KMeka ABuehler MMueller FGotardo PHilliges OBeeler T(2024)ShellNeRF: Learning a Controllable High‐resolution Model of the Eye and Periocular RegionComputer Graphics Forum10.1111/cgf.1504143:2Online publication date: 24-Apr-2024
https://doi.org/10.1111/cgf.15041
Li CJin LZheng YYu YHan X(2023)EMS: 3D Eyebrow Modeling from Single-View ImagesACM Transactions on Graphics10.1145/361832342:6(1-19)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618323
Juncen ZCao JYang YRen WHan H(2023)mmDrive: Fine-grained Fatigue Driving Detection Using mmWave RadarACM Transactions on Internet of Things10.1145/36144374:4(1-30)Online publication date: 22-Nov-2023
https://dl.acm.org/doi/10.1145/3614437
Show More Cited By

Index Terms

Detailed spatio-temporal reconstruction of eyelids
1. Applied computing
  1. Document management and text processing
    1. Document capture
      1. Document scanning
      2. Graphics recognition and interpretation
2. Computing methodologies
  1. Computer graphics
    1. Image manipulation

Recommendations

Real-time 3D eyelids tracking from semantic edges

State-of-the-art real-time face tracking systems still lack the ability to realistically portray subtle details of various aspects of the face, particularly the region surrounding the eyes. To improve this situation, we propose a technique to ...
Realtime 3D eye gaze animation using a single RGB camera

This paper presents the first realtime 3D eye gaze capture method that simultaneously captures the coordinated movement of 3D eye gaze, head poses and facial expression deformation using a single RGB camera. Our key idea is to complement a realtime 3D ...
Reconstructing detailed dynamic face geometry from monocular video

Detailed facial performance geometry can be reconstructed using dense camera and light setups in controlled studios. However, a wide range of important applications cannot employ these approaches, including all movie productions shot from a single ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 34, Issue 4

August 2015

1307 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2809654

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2015

Published in TOG Volume 34, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
817
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)6

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li GSarkar KMeka ABuehler MMueller FGotardo PHilliges OBeeler T(2024)ShellNeRF: Learning a Controllable High‐resolution Model of the Eye and Periocular RegionComputer Graphics Forum10.1111/cgf.1504143:2Online publication date: 24-Apr-2024
https://doi.org/10.1111/cgf.15041
Li CJin LZheng YYu YHan X(2023)EMS: 3D Eyebrow Modeling from Single-View ImagesACM Transactions on Graphics10.1145/361832342:6(1-19)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618323
Juncen ZCao JYang YRen WHan H(2023)mmDrive: Fine-grained Fatigue Driving Detection Using mmWave RadarACM Transactions on Internet of Things10.1145/36144374:4(1-30)Online publication date: 22-Nov-2023
https://dl.acm.org/doi/10.1145/3614437
Kerbiriou GAvril QDanieau FMarchal M(2023)Detailed Eye Region Capture and AnimationComputer Graphics Forum10.1111/cgf.1464241:8(279-282)Online publication date: 20-Mar-2023
https://doi.org/10.1111/cgf.14642
Qiu ZLi YHe DZhang QZhang LZhang YWang JXu LWang XZhang YYu J(2022)SCULPTORACM Transactions on Graphics10.1145/3550454.355546241:6(1-17)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555462
Ma WGhifary MLewis JChoi BEom H(2022)FDLS: A Deep Learning Approach to Production Quality, Controllable, and Retargetable Facial Performances.Proceedings of the 2022 Digital Production Symposium10.1145/3543664.3543672(1-9)Online publication date: 7-Aug-2022
https://dl.acm.org/doi/10.1145/3543664.3543672
Winberg SZoss GChandran PGotardo PBradley D(2022)Facial hair tracking for high fidelity performance captureACM Transactions on Graphics10.1145/3528223.353011641:4(1-12)Online publication date: 22-Jul-2022
https://dl.acm.org/doi/10.1145/3528223.3530116
Dong YJörg SJain E(2022)Is the avatar scared? Pupil as a perceptual cueComputer Animation and Virtual Worlds10.1002/cav.204033:2Online publication date: 3-Feb-2022
https://doi.org/10.1002/cav.2040
Xiao QZhang HZhang ZWu YWang LJin XJiang XYang YShao TZhou K(2021)EyelashNetACM Transactions on Graphics10.1145/3478513.348054040:6(1-17)Online publication date: 10-Dec-2021
https://dl.acm.org/doi/10.1145/3478513.3480540
Liu YZheng CXu FTong XGuo B(2021)Data-Driven 3D Neck Modeling and AnimationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.296703627:7(3226-3237)Online publication date: 1-Jul-2021
https://doi.org/10.1109/TVCG.2020.2967036
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents