Article

Automatic determination of facial muscle activations from sparse motion capture marker data

Authors:

Eftychios Sifakis,

Ronald FedkiwAuthors Info & Claims

SIGGRAPH '05: ACM SIGGRAPH 2005 Papers

Pages 417 - 425

https://doi.org/10.1145/1186822.1073208

Published: 01 July 2005 Publication History

Abstract

We built an anatomically accurate model of facial musculature, passive tissue and underlying skeletal structure using volumetric data acquired from a living male subject. The tissues are endowed with a highly nonlinear constitutive model including controllable anisotropic muscle activations based on fiber directions. Detailed models of this sort can be difficult to animate requiring complex coordinated stimulation of the underlying musculature. We propose a solution to this problem automatically determining muscle activations that track a sparse set of surface landmarks, e.g. acquired from motion capture marker data. Since the resulting animation is obtained via a three dimensional nonlinear finite element method, we obtain visually plausible and anatomically correct deformations with spatial and temporal coherence that provides robustness against outliers in the motion capture data. Moreover, the obtained muscle activations can be used in a robust simulation framework including contact and collision of the face with external objects.

Supplementary Material

MP4 File (pps003.mp4)

Download
36.15 MB

References

[1]

Basu, S., Oliver, N., and Pentland, A. 1998. 3D lip shapes from video: a combined physical-statistical model. Speech Communication 26, 131--148.

Digital Library

[2]

Basu, S., Oliver, N. and Pentland, A. 1998. 3D modeling and tracking of human lip motions. IEEE Computer Society, 337--343.

Digital Library

[3]

Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3D faces. In Proc. of ACM SIGGRAPH, ACM Press, 187--194.

Digital Library

[4]

Blanz, V., and Vetter, T. 2003. Face recognition based on fitting a 3D morphable model. IEEE Trans. on Pattern Analysis and Machine Intelligence 29, 9, 1063--1074.

Digital Library

[5]

Blanz, V., Basso, C., Poggio, T., and Vetter, T. 2003. Reanimating faces in images and video. In Proc. of Eurographics, vol. 22.

[6]

Blanz, V., Scherbaum, K., Vetter, T., and Seidel, H. P. 2004. Exchanging faces in images. In Proc. of Eurographics, vol. 23.

[7]

Borshukov, G., Piponi, D., Larsen, O., Lewis, J. P., and Tempelaar-Lietz, C. 2003. Universal Capture - image-based facial animation for "The Matrix Reloaded". In ACM SIGGRAPH 2003 Sketches & Applications, ACM Press, 1--1.

Digital Library

[8]

Brand, M. 1999. Voice puppetry. In Proc. of ACM SIGGRAPH, 21--28.

Digital Library

[9]

Bregler, C., Covell, M., and Slaney, M. 1997. Video Rewrite: driving visual speech with audio. In Proc. of ACM SIGGRAPH, 353--360.

Digital Library

[10]

Byun, M., and Badler, N. I. 2002. FacEMOTE: Qualitative parametric modifiers for facial animations. In Proc. of ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., ACM Press, 65--71.

Digital Library

[11]

Cao, Y., Faloutsos, P., and Pighin, F. 2003. Unsupervised learning for speech motion editing. In Proc. of the ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 225--231.

Digital Library

[12]

Cao, Y., Faloutsos, P., Kohler, E., and Pighin, F. 2004. Real-time speech motion synthesis from recorded motions. In Proc. of 2003 ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 347--355.

Digital Library

[13]

Cassell, J., Pelachaud, C., Badler, N., Steedman, M., Achorn, B., Becket, T., Doubille, B., Prevost, S., and Stone, M. 1994. Animated conversation: Rule-based generation of facial expression, gesture and spoken intonation for multiple conversational agents. In Proc. of ACM SIGGRAPH, ACM Press, 413--420.

Digital Library

[14]

Cassell, J., Vilhjálmsson, H. H., and Bickmore, T. 2001. BEAT: the Behavior Expression Animation Toolkit. In Proc. of ACM SIGGRAPH, 477--486.

Digital Library

[15]

Chai, J., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3D facial animation. In Proc. of ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 193--206.

Digital Library

[16]

Choe, B., and Ko, H.-S. 2001. Analysis and synthesis of facial expressions with hand-generated muscle actuation basis. In Proc. of Comput. Anim., 12--19.

Digital Library

[17]

Choe, B., Lee, H., and Ko, H.-S. 2001. Performance-driven muscle-based facial animation. J. Vis. and Comput. Anim. 12, 67--79.

[18]

DeCarlo, D., Metaxas, D., and Stone, M. 1998. An anthropometric face model using variational techniques. In Proc. of ACM SIGGRAPH, ACM Press, 67--74.

Digital Library

[19]

Ekman, P., and Friesen, W. V. 1978. Facial Action Coding System. Consulting Psychologist Press, Palo Alto.

[20]

Essa, I., and Pentland, A. 1997. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. on Pattern Analysis and Machine Intelligence 19, 7, 757-763.

Digital Library

[21]

Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads using input from video. In Proc. of Computer Animation, IEEE Computer Society, 68--79.

Digital Library

[22]

Ezzat, T., Geiger, G., and Poggio, T. 2002. Trainable videorealistic speech animation. In ACM Transactions on Graphics, ACM Press, vol. 21, 388--398.

Digital Library

[23]

Gill, P. E., Murray, W., and Wright, M. H. 1981. Practical Optimization. Academic Press, San Diego, USA.

[24]

Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Proc. ACM SIGGRAPH. ACM Press, 55--66.

Digital Library

[25]

Heidelberger, B., Teschner, M., Keiser, R., Müller, M., and Gross, M. 2004. Consistent penetration depth estimation for deformable collision response. In Proc. of Vision, Modeling, Visualization (VMV), 339--346.

[26]

Joshi, P., Tien, W. C., Desbrun, M., and Pighin, F. 2003. Learning controls for blend shape based realistic facial animation. In Proc. ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 365--373.

Digital Library

[27]

Kahler, K., Haber, J., and Seidel, H.-P. 2001. Geometry-based muscle modeling for facial animation. In Proc. of Graphics Interface, 37--46.

Digital Library

[28]

Kahler, K., Haber, J., Yamauchi, H., and Seidel, H.-P. 2002. Head shop: Generating animated head models with anatomical structure. In Proc. of ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 55--63.

Digital Library

[29]

Kahler, K., Haber, J., and Seidel, H.-P. 2003. Reanimating the dead: Reconstruction of expressive faces from skull data. In ACM Trans. on Graphics, vol. 22, 554--561.

Digital Library

[30]

Kalra, P., Mangili, A., Magnetat-Thalmann, N., and Thalmann, D. 1992. Simulation of facial muscle actions based on rational free form deformations. In Proc. of Eurographics, 59--69.

[31]

Keeve, E., Girod, S., Pfeifle, P., and Girod, B. 1996. Anatomy-based facial tissue modeling using the finite element method. In Proc. of Visualization, 21--28.

Digital Library

[32]

Koch, R. M., Gross, M. H., Carls, F. R., von Buren, D. F., Fankhauser, G., and Parish, Y. I. H. 1996. Simulating facial surgery using finite element models. Computer Graphics 30, Annual Conference Series, 421--428.

Digital Library

[33]

Koch, R., Gross, M., and Bosshard, A. 1998. Emotion editing using finite elements. Proceedings of Eurographics 1998 17, 3.

[34]

Kshirsagar, S., and Magnenat-Thalmann, N. 2003. Visyllable based speech animation. In Proc. of Eurographics, vol. 22.

[35]

Lee, Y., Terzopoulos, D., and Waters, K. 1995. Realistic modeling for facial animation. Comput. Graph. (SIGGRAPH Proc.), 55--62.

Digital Library

[36]

Lee, S. P., Badler, J. B., and Badler, N. I. 2002. Eyes alive. In Proc. of ACM SIGGRAPH, ACM Press, 637--644.

Digital Library

[37]

Magnenat-Thalmann, N., Primeau, E., and Thalmann, D. 1988. Abstract muscle action procedures for human face animation. The Visual Computer 3, 5, 290--297.

[38]

Morishima, S., Ishikawa, T., and Terzopoulos, D. 1998. Facial muscle parameter decision from 2D frontal image. In Proc. of the Int. Conf. on Pattern Recognition, vol. 1, 160--162.

Digital Library

[39]

Moving Picture Experts Group, 1998. Information technology - coding of audio-visual objects part 2: Visual. Final draft of international standard ISO/IEC JTC1/SC29/WG11 N2501 14496--2.

[40]

Na, K., and Jung, M. 2004. Hierarchical retargetting of fine facial motions. In Proc. of Eurographics, vol. 23.

[41]

Noh, J., and Neumann, U. 2001. Expression cloning. In Proc. of ACM SIGGRAPH, ACM Press, E. Fiume, Ed., 277--288.

Digital Library

[42]

Parke, F. I., and Waters, K. 1996. Computer Facial Animation. AK Peters, Ltd.

Digital Library

[43]

Parke, F. I. 1972. Computer generated animation of faces. In Proc. of ACM Conference, ACM Press, 451--457.

Digital Library

[44]

Pieper, S., Rosen, J., and Zeltzer, D. 1992. Interactive graphics for plastic surgery: A task-level analysis and implementation. In Proc. of Symp. on Interactive 3D graphics, ACM Press, 127--134.

Digital Library

[45]

Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. H. 1998. Synthesizing realistic facial expressions from photographs. In Proc. of ACM SIGGRAPH, ACM Press, 75--84.

Digital Library

[46]

Pighin, F., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3D model-based tracking. In Proc. of Int. Conf. on Comput. Vision, 143--150.

[47]

Platt, S. M., and Badler, N. I. 1981. Animating facial expressions. Comput. Graph. (SIGGRAPH Proc.), 245--252.

Digital Library

[48]

Pyun, H., Kim, Y., Chae, W., Kang, H. W., and Shin, S. Y. 2003. An example-based approach for facial expression cloning. In Proc. of ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 167--176.

Digital Library

[49]

Roth, S. H., Gross, M., Turello, M. H., and Carls, S. 1998. A Bernstein-Bézier based approach to soft tissue simulation. In Proc. of Eurographics, vol. 17, 285--294.

[50]

Sumner, R., and Popović, J. 2004. Deformation transfer for triangle meshes. In Proc. of ACM SIGGRAPH, vol. 23, 32--39.

Digital Library

[51]

Teran, J., Blemker, S., Ng, V., and Fedkiw, R. 2003. Finite volume methods for the simulation of skeletal muscle. In Proc. of the 2003 ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., 68--74.

Digital Library

[52]

Teran, J., Sifakis, E., Irving, G., and Fedkiw, R. 2005. Robust quasistatic finite elements and flesh simulation. ACM Trans. on Graphics (to appear).

[53]

Teran, J., Sifakis, E., Salinas-Blemker, S., Ng-Thow-Hing, V., Lau, C., and Fedkiw, R. 2005. Creating and simulating skeletal muscle from the visible human data set. IEEE Trans. on Vis. and Comput. Graph. 11, 3, 317--328.

Digital Library

[54]

Terzopoulos, D., and Waters, K. 1993. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Trans. on Pattern Analysis and Machine Intelligence 15, 6.

Digital Library

[55]

Teschner, M., Girod, S., and Girod, B. 2000. Direct computation of nonlinear soft-tissue deformation. In Proc. of Vision, Modeling, and Visualization, 383--390.

[56]

Teschner, M., Heidelberger, B., Müller, M., Pomeranets, D., and Gross, M. 2003. Optimized spatial hashing for collision detection of deformable objects. In Proc. of Vision, Modeling, Visualization (VMV), 47--54.

[57]

U.S. NATIONAL LIBRARY OF MEDICINE, 1994. The visible human project. http://www.nlm.nih.gov/research/visible/.

[58]

Wang, Y., Huang, X., Lee, C. S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dynamic 3-D facial expressions. In Proc. of Eurographics, 677--686.

[59]

Waters, K., and Frisbie, J. 1995. A coordinated muscle model for speech animation. In Proc. of Graphics Interface, 163--170.

[60]

Waters, K. 1987. A muscle model for animating three-dimensional facial expressions. Comput. Graph. (SIGGRAPH Proc.), 17--24.

Digital Library

[61]

Williams, L. 1990. Performance-driven facial animation. In Computer Graphics (Proc. of Int. Conf. on Computer Graphics and Interactive Techniques), ACM Press, 235--242.

Digital Library

[62]

Wu, Y., Kalra, P., Magnenat-Thalmann, N., and Thalmann, D. 1999. Simulating wrinkles and skin aging. The Visual Computer 15, 4, 183--198.

[63]

Zajac, F. 1989. Muscle and tendon: Properties, models, scaling, and application to biomechanics and motor control. Critical Reviews in Biomed. Eng. 17, 4, 359--411.

[64]

Zhang, Q., Liu, Z., Guo, B., and Shum, H. 2003. Geometry-driven photorealistic facial expression synthesis. In Proc. of ACM SIGGRAPH/Eurographics Symp. on Comput. Anim., ACM Press, 16--22.

Digital Library

[65]

Zhang, L., Snavely, N., Curless, B., and Seitz, S. 2004. Spacetime faces: High resolution capture for modeling and animation. In Proc. of ACM SIGGRAPH, ACM Press, vol. 23, 548--558.

Digital Library

Cited By

Zhu CJoslin C(2024)Fabrig: A Cloth-Simulated Transferable 3D Face ParameterizationSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687695(1-10)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687695
Racković SJakovetić DSoares C(2024)Refined Inverse Rigging: A Balanced Approach to High-fidelity Blendshape AnimationSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687670(1-9)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687670
Park HGrama Srinivasan SCong MKim DKim BSwartz JMuseth KSifakis E(2024)Near-realtime Facial Animation by Deep 3D Simulation Super-ResolutionACM Transactions on Graphics10.1145/367068743:5(1-20)Online publication date: 9-Aug-2024
https://dl.acm.org/doi/10.1145/3670687
Show More Cited By

Index Terms

Automatic determination of facial muscle activations from sparse motion capture marker data
1. Computing methodologies
  1. Computer graphics
    1. Animation
      1. Physical simulation
    2. Shape modeling
2. Mathematics of computing
  1. Continuous mathematics
    1. Topology
      1. Geometric topology

Recommendations

Automatic determination of facial muscle activations from sparse motion capture marker data

We built an anatomically accurate model of facial musculature, passive tissue and underlying skeletal structure using volumetric data acquired from a living male subject. The tissues are endowed with a highly nonlinear constitutive model including ...
Muscle-based facial retargeting with anatomical constraints
SIGGRAPH '19: ACM SIGGRAPH 2019 Talks

We present a physically based facial retargeting algorithm that is suitable for use in high-end production. Given an actor's facial performance, we first run a targeted muscle simulation on the actor in order to determine the actor blendshape muscles ...
Muscle simulation for facial animation in Kong: Skull Island
SIGGRAPH '17: ACM SIGGRAPH 2017 Talks

For Kong: Skull Island, Industrial Light & Magic created an anatomically motivated facial simulation model for Kong that includes the facial skeleton and musculature. We applied a muscle simulation framework that allowed us to target facial shapes while ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '05: ACM SIGGRAPH 2005 Papers

July 2005

826 pages

ISBN:9781450378253

DOI:10.1145/1186822

Editor:
Markus Gross
Eidgenössische Technische Hochschule Zürich

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGGRAPH05

Sponsor:

SIGGRAPH

SIGGRAPH05: Special Interest Group on Computer Graphics and Interactive Techniques Conference

July 31 - August 4, 2005

California, Los Angeles

Acceptance Rates

SIGGRAPH '05 Paper Acceptance Rate 98 of 461 submissions, 21%;

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

69
Total Citations
View Citations
3,675
Total Downloads

Downloads (Last 12 months)66
Downloads (Last 6 weeks)5

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhu CJoslin C(2024)Fabrig: A Cloth-Simulated Transferable 3D Face ParameterizationSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687695(1-10)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687695
Racković SJakovetić DSoares C(2024)Refined Inverse Rigging: A Balanced Approach to High-fidelity Blendshape AnimationSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687670(1-9)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687670
Park HGrama Srinivasan SCong MKim DKim BSwartz JMuseth KSifakis E(2024)Near-realtime Facial Animation by Deep 3D Simulation Super-ResolutionACM Transactions on Graphics10.1145/367068743:5(1-20)Online publication date: 9-Aug-2024
https://dl.acm.org/doi/10.1145/3670687
Kim JLee JKim Y(2024)Motion Generation and Analyzing the User’s Arm Muscles via Leap Motion and Its Data-Driven RepresentationsIEEE Access10.1109/ACCESS.2024.338331812(47787-47796)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3383318
Zhang YSu RYu JLi R(2024)3D facial modeling, animation, and rendering for digital humans: A surveyNeurocomputing10.1016/j.neucom.2024.128168598(128168)Online publication date: Sep-2024
https://doi.org/10.1016/j.neucom.2024.128168
Zhu CJoslin C(2024)A review of motion retargeting techniques for 3D character facial animationComputers & Graphics10.1016/j.cag.2024.104037123(104037)Online publication date: Oct-2024
https://doi.org/10.1016/j.cag.2024.104037
Sun LWang YQin W(2024)A language‐directed virtual human motion generation approach based on musculoskeletal modelsComputer Animation and Virtual Worlds10.1002/cav.225735:3Online publication date: 5-Jun-2024
https://doi.org/10.1002/cav.2257
Wagner NBotsch MSchwanecke U(2023)SoftDECA: Computationally Efficient Physics-Based Facial AnimationsProceedings of the 16th ACM SIGGRAPH Conference on Motion, Interaction and Games10.1145/3623264.3624439(1-11)Online publication date: 15-Nov-2023
https://dl.acm.org/doi/10.1145/3623264.3624439
Lee SJiang YLiu C(2023)Anatomically Detailed Simulation of Human TorsoACM Transactions on Graphics10.1145/359242542:4(1-11)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592425
Vahdati DDuc Nguyen TStamm M(2023)Defending Low-Bandwidth Talking Head Videoconferencing Systems From Real-Time Puppeteering Attacks2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW59228.2023.00105(983-992)Online publication date: Jun-2023
https://doi.org/10.1109/CVPRW59228.2023.00105
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten