research-article

Real-time facial animation from live video tracking

Authors:

Youngkyoo Hwang,

James Dokyoon Kim,

Changyeong KimAuthors Info & Claims

SCA '11: Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation

Pages 215 - 224

https://doi.org/10.1145/2019406.2019435

Published: 05 August 2011 Publication History

Abstract

This paper describes a complete pipe-line of a practical system for producing real-time facial expressions of a 3D virtual avatar controlled by an actor's live performances. The system handles practical challenges arising from markerless expression captures from a single conventional video camera. For robust tracking, a localized algorithm constrained by belief propagation is applied to the upper face, and an appearance matching technique using a parameterized generic face model is exploited for lower face and head pose tracking. The captured expression features then transferred to high dimensional 3D animation controls using our facial expression space which is a structure-preserving map between two algebraic structures. The transferred animation controls drive facial animation of a 3D avatar while optimizing the smoothness of the face mesh. An example-based face deformation technique produces non-linear local detail deformations on the avatar that are not captured in the movement of the animation controls.

Supplementary Material

MOV File (p215-rhee.mov)

Download
24.32 MB

References

[1]

{ACP02} Allen B., Curless B., Popović; Z.: Articulated body deformation from range scan data. In ACM SIGGRAPH 2002 papers (2002), pp. 612--619. 3

Digital Library

[2]

{Ahl01} Ahlberg J.: CANDIDE-3 - An Updated Parameterised Face. Tech. rep., 2001. 3

[3]

{BBA*07} Bickel B., Botsch M., Angst R., Matusik W., Otaduy M., Pfister H., Gross M.: Multi-scale capture of facial geometry and motion. In ACM SIGGRAPH 2007 papers (2007), p. 33. 3

Digital Library

[4]

{BHPS10} Bradley D., Heidrich W., Popa T., Sheffer A.: High resolution passive facial performance capture. In ACM SIGGRAPH 2010 papers (2010), pp. 41:1--41:10. 2

Digital Library

[5]

{BLB*08} Bickel B., Lang M., Botsch M., Otaduy M. A., Gross M.: Pose-space animation and transfer of facial details. In SCA '08: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation (2008), pp. 57--66. 3, 6, 7

Digital Library

[6]

{BLF*06} Bartlett M. S., Littlewort G., Frank M. G., Lainscsek C., Fasel I. R., Movellan J. R.: Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia 1, 6 (2006), 22--35. 2, 5

[7]

{CXH03} Chai J.-X., Xiao J., Hodgins J.: Vision-based control of 3d facial animation. In SCA '03: Proceedings of the ACM SIGGRAPH/Eurographics symposium on Computer animation (2003), pp. 193--206. 2, 3

Digital Library

[8]

{DCFN06} Deng Z., Chiang P.-Y., Fox P., Neumann U.: Animating blendshape faces by cross-mapping motion capture data. In I3D '06: Proceedings of the 2006 symposium on Interactive 3D graphics and games (2006), pp. 43--48. 2, 3

Digital Library

[9]

{DD06} Dornaika F., Davoine F.: On appearance based face and facial action tracking. IEEE Transactions on Circuits and Systems for Video Technology 16, 9 (2006), 1107--1124. 2, 4

Digital Library

[10]

{EFH} Ekman P., Friesen W. V., Hager J. C.: Facial antion coding system. http://www.cs.cmu.edu/afs/cs/project/face/www/facs.htm.5

[11]

{FKY08} Feng W.-W., Kim B.-U., Yu Y.: Real-time data driven deformation using kernel canonical correlation analysis. In ACM SIGGRAPH 2008 papers (2008), pp. 91:1--91:9. 3

Digital Library

[12]

{GGW*98} Guenter B., Grimm C., Wood D., Malvar H., Pighin F.: Making faces. In ACM SIGGRAPH 1998 papers (1998), pp. 55--66. 2

Digital Library

[13]

{JTDP06} Joshi P., Tien W. C., Desbrun M., Pighin F.: Learning controls for blend shape based realistic facial animation. In ACM SIGGRAPH 2006 Courses (2006), p. 17. 3

Digital Library

[14]

{KM04} Kurihara T., Miyata N.: Modeling deformable human hands from medical images. In Proc. Symposium on Computer Animation (SCA-04) (2004), pp. 357--366. 3, 7

Digital Library

[15]

{KMTT92} Kalra P., Mangili A., Thalmann N. M., Thalmann D.: Simulation of facial muscle actions based on rational free form deformations. Computer Graphics Forum 2, 3 (1992), 59--C69. 3

[16]

{LCF00} Lewis J. P., Cordner M., Fong N.: Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. In SIGGRAPH '00 (2000), pp. 165--172. 3, 7

Digital Library

[17]

{LK81} Lucas B. D., Kanade T.: An iterative image registration technique with an application to stereo vision. In IJCAI'81: Proceedings of the 7th international joint conference on Artificial intelligence (1981), pp. 674--679. 4

Digital Library

[18]

{LMDN05} Lewis J. P., Mooser J., Deng Z., Neumann U.: Reducing blendshape interference by selected motion attenuation. In I3D '05: Proceedings of the 2005 symposium on Interactive 3D graphics and games (2005), pp. 25--29. 3

Digital Library

[19]

{LWP10} Li H., Weise T., Pauly M.: Example-based facial rigging. ACM Transactions on Graphics (Proceedings SIGGRAPH 2010) 29, 3 (July 2010). 3

Digital Library

[20]

{Mar63} Marquardt D. W.: An algorithm for least-squares estimation of nonlinear parameters. SIAM Journal on Applied Mathematics 11 (1963), 431--441. 6

[21]

{MB04} Matthews I., Baker S.: Active appearance models revisited. Int. J. Comput. Vision 60, 2 (2004), 135--164. 3

Digital Library

[22]

{MJC*08} Ma W.-C., Jones A., Chiang J.-Y., Hawkins T., Frederiksen S., Peers P., Vukovic M., Ouhyoung M., Debevec P.: Facial performance synthesis using deformation-driven polynomial displacement maps. In SIGGRAPH Asia '08 (2008), pp. 1--10. 3

Digital Library

[23]

{MTLT88} Magnenat-Thalmann N., Laperrière R., Thalmann D.: Joint--dependent local deformations for hand animation and object grasping. In Graphics Interface '88 (June 1988), pp. 26--33. 3, 7

Digital Library

[24]

{NFN00} Noh J.-Y., Fidaleo D., Neumann U.: Animated deformations with radial basis functions. In VRST '00: Proceedings of the ACM symposium on Virtual reality software and technology (2000), pp. 166--174. 3, 6

Digital Library

[25]

{Par72} Parke F. I.: Computer generated animation of faces. In ACM '72: Proceedings of the ACM annual conference (1972), pp. 451--457. 3

Digital Library

[26]

{PHL*98} Pighin F., Hecker J., Lischinski D., Szeliski R., Salesin D. H.: Synthesizing realistic facial expressions from photographs. In ACM SIGGRAPH 1998 papers (1998), pp. 75--84. 3

Digital Library

[27]

{PL06} Pighin F., Lewis J.: Performance-driven facial animation. In SIGGRAPH '06 Courses (2006). 2

Digital Library

[28]

{RLN06} Rhee T., Lewis J., Neumann U.: Real-time weighted pose-space deformation on the GPUs. Computer Graphics Forum, Proc. of Eurographics 2006 25, 3 (2006), 439--448. 3, 7

[29]

{SSK*05} Surazhsky V., Surazhsky T., Kirsanov D., Gortler S. J., Hoppe H.: Fast exact and approximate geodesics on meshes. ACM Trans. Graph. 24, 3 (2005), 553--560. 6

Digital Library

[30]

{Wil90} Williams L.: Performance-driven facial animation. In ACM SIGGRAPH 1990 papers (1990), pp. 235--242. 2

Digital Library

[31]

{WLVGP09} Weise T., Li H., Van Gool L., Pauly M.: Face/off: live facial puppetry. In SCA '09: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation (2009), pp. 7--16. 2

Digital Library

Cited By

Prof. Shivparasd BJ Dhanraj S H Naveen Hiremath Jagadish Gowda Darshan R Shetty (2024)A Review on Modern Techniques in 3D Character AnimationInternational Journal of Advanced Research in Science, Communication and Technology10.48175/IJARSCT-19272(463-468)Online publication date: 4-Aug-2024
https://doi.org/10.48175/IJARSCT-19272
Mazala DEsperança CMarroquim R(2022)Laplacian face blendingComputer Animation and Virtual Worlds10.1002/cav.204434:2Online publication date: 22-Mar-2022
https://doi.org/10.1002/cav.2044
Gong SBahri MBronstein MZafeiriou S(2020)Geometrically Principled Connections in Graph Neural Networks2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR42600.2020.01143(11412-11421)Online publication date: Jun-2020
https://doi.org/10.1109/CVPR42600.2020.01143
Show More Cited By

Index Terms

Real-time facial animation from live video tracking
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Motion capture
  2. Computer graphics
    1. Animation
      1. Motion capture
      2. Motion processing

Recommendations

Real-time 3D neural facial animation from binocular video

We present a method for performing real-time facial animation of a 3D avatar from binocular video. Existing facial animation methods fail to automatically capture precise and subtle facial motions for driving a photo-realistic 3D avatar "in-the-wild" (...
3D shape regression for real-time facial animation

We present a real-time performance-driven facial animation system based on 3D shape regression. In this system, the 3D positions of facial landmark points are inferred by a regressor from 2D video frames of an ordinary web camera. From these 3D points, ...
Easy acquisition and real-time animation of facial wrinkles

Facial animation details like wrinkles or bulges are very useful for the analysis and the interpretation of facial emotions and expressions. However, outfitting a virtual face with expression details for real-time applications is a difficult task. In ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SCA '11: Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation

August 2011

297 pages

ISBN:9781450309233

DOI:10.1145/2019406

Conference Chairs:
Karen Liu
Georgia Institute of Technology
,
Robert Bridson
University of British Columbia
,
Editor:
Stephen N. Spencer
University of Washington
,
Program Chairs:
Adam Bargteil
University of Utah
,
Michiel van de Panne
University of British Columbia

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques
EUROGRAPHICS: The European Association for Computer Graphics

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 August 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

SCA '11

Sponsor:

SIGGRAPH
EUROGRAPHICS

SCA '11: The ACM SIGGRAPH / Eurographics Symposium on Computer Animation 2011

August 5 - 7, 2011

British Columbia, Vancouver, Canada

Acceptance Rates

Overall Acceptance Rate 183 of 487 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
1,338
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)1

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Prof. Shivparasd BJ Dhanraj S H Naveen Hiremath Jagadish Gowda Darshan R Shetty (2024)A Review on Modern Techniques in 3D Character AnimationInternational Journal of Advanced Research in Science, Communication and Technology10.48175/IJARSCT-19272(463-468)Online publication date: 4-Aug-2024
https://doi.org/10.48175/IJARSCT-19272
Mazala DEsperança CMarroquim R(2022)Laplacian face blendingComputer Animation and Virtual Worlds10.1002/cav.204434:2Online publication date: 22-Mar-2022
https://doi.org/10.1002/cav.2044
Gong SBahri MBronstein MZafeiriou S(2020)Geometrically Principled Connections in Graph Neural Networks2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR42600.2020.01143(11412-11421)Online publication date: Jun-2020
https://doi.org/10.1109/CVPR42600.2020.01143
Cao CChai MWoodford OLuo L(2018)Stabilized real-time face tracking via a learned dynamic rigidity priorACM Transactions on Graphics10.1145/3272127.327509337:6(1-11)Online publication date: 4-Dec-2018
https://dl.acm.org/doi/10.1145/3272127.3275093
Xiao LWang Z(2018)Dense Convolutional Recurrent Neural Network for Generalized Speech Animation2018 24th International Conference on Pattern Recognition (ICPR)10.1109/ICPR.2018.8545744(633-638)Online publication date: Aug-2018
https://doi.org/10.1109/ICPR.2018.8545744
Asano NMasai KSugiura YSugimoto MBillinghurst MSmith RLindeman RBruder GIwai D(2017)Facial performance capture by embedded photo reflective sensors on a smart eyewearProceedings of the 27th International Conference on Artificial Reality and Telexistence and 22nd Eurographics Symposium on Virtual Environments10.5555/3298830.3298835(21-28)Online publication date: 22-Nov-2017
https://dl.acm.org/doi/10.5555/3298830.3298835
Wu CBradley DGross MBeeler T(2016)An anatomically-constrained local deformation model for monocular face captureACM Transactions on Graphics10.1145/2897824.292588235:4(1-12)Online publication date: 11-Jul-2016
https://dl.acm.org/doi/10.1145/2897824.2925882
Ma WRhee TYoshiyasu Y(2015)Making digital charactersSIGGRAPH Asia 2015 Courses10.1145/2818143.2818172(1-79)Online publication date: 2-Nov-2015
https://dl.acm.org/doi/10.1145/2818143.2818172
Cao CBradley DZhou KBeeler T(2015)Real-time high-fidelity facial performance captureACM Transactions on Graphics10.1145/276694334:4(1-9)Online publication date: 27-Jul-2015
https://dl.acm.org/doi/10.1145/2766943
Bermano ABeeler TKozlov YBradley DBickel BGross M(2015)Detailed spatio-temporal reconstruction of eyelidsACM Transactions on Graphics10.1145/276692434:4(1-11)Online publication date: 27-Jul-2015
https://dl.acm.org/doi/10.1145/2766924
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents