research-article

Performance capture from sparse multi-view video

Authors:

Edilson de Aguiar,

Christian Theobalt,

Hans-Peter Seidel,

Sebastian ThrunAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 27, Issue 3

Pages 1 - 10

https://doi.org/10.1145/1360612.1360697

Published: 01 August 2008 Publication History

Abstract

This paper proposes a new marker-less approach to capturing human performances from multi-view video. Our algorithm can jointly reconstruct spatio-temporally coherent geometry, motion and textural surface appearance of actors that perform complex and rapid moves. Furthermore, since our algorithm is purely meshbased and makes as few as possible prior assumptions about the type of subject being tracked, it can even capture performances of people wearing wide apparel, such as a dancer wearing a skirt. To serve this purpose our method efficiently and effectively combines the power of surface- and volume-based shape deformation techniques with a new mesh-based analysis-through-synthesis framework. This framework extracts motion constraints from video and makes the laser-scan of the tracked subject mimic the recorded performance. Also small-scale time-varying shape detail is recovered by applying model-guided multi-view stereo to refine the model surface. Our method delivers captured performance data at high level of detail, is highly versatile, and is applicable to many complex types of scenes that could not be handled by alternative marker-based or marker-free recording techniques.

Supplementary Material

FLV File (23.flv)

Download
108.82 MB

MOV File (a98-de_aguilar.mov)

Download
33.46 MB

References

[1]

Allen, B., Curless, B., and Popović, Z. 2002. Articulated body deformation from range scan data. ACM Trans. Graph. 21, 3, 612--619.

Digital Library

[2]

Balan, A. O., Sigal, L., Black, M. J., Davis, J. E., and Haussecker, H. W. 2007. Detailed human shape and pose from images. In Proc. CVPR.

[3]

Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. In Proc. of SIGGRAPH, 33.

Digital Library

[4]

Botsch, M., and Sorkine, O. 2008. On linear variational surface deformation methods. IEEE TVCG 14, 1, 213--230.

Digital Library

[5]

Botsch, M., Pauly, M., Wicke, M., and Gross, M. 2007. Adaptive space deformations based on rigid cells. Computer Graphics Forum 26, 3, 339--347.

[6]

Byrd, R., Lu, P., Nocedal, J., and Zhu, C. 1995. A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comp. 16, 5, 1190--1208.

Digital Library

[7]

Carranza, J., Theobalt, C., Magnor, M., and Seidel, H.-P. 2003. Free-viewpoint video of human actors. In Proc. SIGGRAPH, 569--577.

Digital Library

[8]

de Aguiar, E., Theobalt, C., Stoll, C., and Seidel, H.-P. 2007. Marker-less deformable mesh tracking for human shape and motion capture. In Proc. CVPR, IEEE, 1--8.

[9]

de Aguiar, E., Theobalt, C., Stoll, C., and Seidel, H. 2007. Marker-less 3d feature tracking for mesh-based human motion capture. In Proc. ICCV HUMO07, 1--15.

Digital Library

[10]

de Aguiar, E., Theobalt, C., Thrun, S., and Seidel, H.-P. 2008. Automatic conversion of mesh animations into skeleton-based animations. Computer Graphics Forum (Proc. Eurographics EG'08) 27, 2 (4), 389--397.

[11]

Einarsson, P., Chabert, C.-F., Jones, A., Ma, W.-C., Lamond, B., im Hawkins, Bolas, M., Sylwan, S., and Debevec, P. 2006. Relighting human locomotion with flowed reflectance fields. In Proc. EGSR, 183--194.

[12]

Goesele, M., Curless, B., and Seitz, S. M. 2006. Multiview stereo revisited. In Proc. CVPR, 2402--2409.

Digital Library

[13]

Gross, M., Würmlin, S., Näf, M., Lamboray, E., Spagno, C., Kunz, A., Koller-Meier, E., Svoboda, T., Gool, L. V., Lang, S., Strehlke, K., Moere, A. V., and Staadt, O. 2003. blue-c: a spatially immersive display and 3d video portal for telepresence. ACM TOG 22, 3, 819--827.

Digital Library

[14]

Kanade, T., Rander, P., and Narayanan, P. J. 1997. Virtualized reality: Constructing virtual worlds from real scenes. IEEE MultiMedia 4, 1, 34--47.

Digital Library

[15]

Kazhdan, M., Bolitho, M., and Hoppe, H. 2006. Poisson surface reconstruction. In Proc. SGP, 61--70.

Digital Library

[16]

Leordeanu, M., and Hebert, M. 2005. A spectral technique for correspondence problems using pairwise constraints. In Proc. ICCV.

Digital Library

[17]

Lowe, D. G. 1999. Object recognition from local scale-invariant features. In Proc. ICCV, vol. 2, 1150ff.

Digital Library

[18]

Matusik, W., Buehler, C., Raskar, R., Gortler, S., and McMillan, L. 2000. Image-based visual hulls. In Proc. SIGGRAPH, 369--374.

Digital Library

[19]

Menache, A., and Manache, A. 1999. Understanding Motion Capture for Computer Animation and Video Games. Morgan Kaufmann.

Digital Library

[20]

Mitra, N. J., Flory, S., Ovsjanikov, M., Gelfand, N., as, L. G., and Pottmann, H. 2007. Dynamic geometry registration. In Proc. SGP, 173--182.

Digital Library

[21]

Moeslund, T. B., Hilton, A., and Krüger, V. 2006. A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Underst. 104, 2, 90--126.

Digital Library

[22]

Müller, M., Dorsey, J., McMillan, L., Jagnow, R., and Cutler, B. 2002. Stable real-time deformations. In Proc. of SCA, ACM, 49--54.

Digital Library

[23]

Paramount, 2007. Beowulf movie page. http://www.beowulfmovie.com/.

[24]

Park, S. I., and Hodgins, J. K. 2006. Capturing and animating skin deformation in human motion. ACM TOG (SIGGRAPH 2006) 25, 3 (Aug.).

Digital Library

[25]

Poppe, R. 2007. Vision-based human motion analysis: An overview. CVIU 108, 1.

Digital Library

[26]

Rosenhahn, B., Kersting, U., Powel, K., and Seidel, H.-P. 2006. Cloth x-ray: Mocap of people wearing textiles. In LNCS 4174: Proc. DAGM, 495--504.

Digital Library

[27]

Sand, P., McMillan, L., and Popović, J. 2003. Continuous capture of skin deformation. ACM TOG 22, 3.

Digital Library

[28]

Scholz, V., Stich, T., Keckeisen, M., Wacker, M., and Magnor, M. 2005. Garment motion capture using colorcoded patterns. Computer Graphics Forum (Proc. Eurographics EG'05) 24, 3 (Aug.), 439--448.

Digital Library

[29]

Shinya, M. 2004. Unifying measured point sequences of deforming objects. In Proc. of 3DPVT, 904--911.

[30]

Sorkine, O., and Alexa, M. 2007. As-rigid-as-possible surface modeling. In Proc. SGP, 109--116.

Digital Library

[31]

Starck, J., and Hilton, A. 2007. Surface capture for performance based animation. IEEE CGAA 27(3), 21--31.

Digital Library

[32]

Stoll, C., Karni, Z., Rössl, C., Yamauchi, H., and Seidel, H.-P. 2006. Template deformation for point cloud fitting. In Proc. SGP, 27--35.

[33]

Sumner, R. W., and Popović, J. 2004. Deformation transfer for triangle meshes. In SIGGRAPH '04, 399--405.

Digital Library

[34]

Vedula, S., Baker, S., and Kanade, T. 2005. Image-based spatio-temporal modeling and view interpolation of dynamic events. ACM Trans. Graph. 24, 2, 240--261.

Digital Library

[35]

Wand, M., Jenke, P., Huang, Q., Bokeloh, M., Guibas, L., and Schilling, A. 2007. Reconstruction of deforming geometry from time-varying point clouds. In Proc. SGP, 49--58.

Digital Library

[36]

Waschbüsch, M., Würmlin, S., Cotting, D., Sadlo, F., and Gross, M. 2005. Scalable 3D video of dynamic scenes. In Proc. Pacific Graphics, 629--638.

[37]

White, R., Crane, K., and Forsyth, D. 2007. Capturing and animating occluded cloth. In ACM TOG (Proc. SIGGRAPH).

Digital Library

[38]

Wilburn, B., Joshi, N., Vaish, V., Talvala, E., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM TOG 24, 3, 765--776.

Digital Library

[39]

Xu, W., Zhou, K., Yu, Y., Tan, Q., Peng, Q., and Guo, B. 2007. Gradient domain editing of deforming mesh sequences. In Proc. SIGGRAPH, ACM, 84ff.

Digital Library

[40]

Yamauchi, H., Gumhold, S., Zayer, R., and Seidel, H.-P. 2005. Mesh segmentation driven by gaussian curvature. Visual Computer 21, 8--10, 649--658.

[41]

Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM TOG 23, 3, 600--608.

Digital Library

Cited By

Kyriakou Tde la Campa Crespo MPanayiotou AChrysanthou YCharalambous PAristidou A(2024)Virtual Instrument Performances (VIP): A Comprehensive ReviewComputer Graphics Forum10.1111/cgf.1506543:2Online publication date: 30-Apr-2024
https://doi.org/10.1111/cgf.15065
Hong JYoung Noh SLee HSik Cheong WChang J(2024)3D Clothed Human Reconstruction from Sparse Multi-View Images2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00072(677-687)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00072
Ami-Williams TSerghides CAristidou A(2024)Digitizing traditional dances under extreme clothing: The case study of EyoJournal of Cultural Heritage10.1016/j.culher.2024.02.01167(145-157)Online publication date: May-2024
https://doi.org/10.1016/j.culher.2024.02.011
Show More Cited By

Index Terms

Performance capture from sparse multi-view video
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Animation

Recommendations

Multi-view Performance Capture of Surface Details

This paper presents a novel approach to recover true fine surface detail of deforming meshes reconstructed from multi-view video. Template-based methods for performance capture usually produce a coarse-to-medium scale detail 4D surface reconstruction ...
On-set performance capture of multiple actors with a stereo camera

State-of-the-art marker-less performance capture algorithms reconstruct detailed human skeletal motion and space-time coherent surface geometry. Despite being a big improvement over marker-based motion capture methods, they are still rarely applied in ...
Facial hair tracking for high fidelity performance capture

Facial hair is a largely overlooked topic in facial performance capture. Most production pipelines in the entertainment industry do not have a way to automatically capture facial hair or track the skin underneath it. Thus, actors are asked to shave clean ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 27, Issue 3

August 2008

844 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1360612

Issue’s Table of Contents

Copyright © 2008 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 August 2008

Published in TOG Volume 27, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

371
Total Citations
View Citations
2,403
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)2

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kyriakou Tde la Campa Crespo MPanayiotou AChrysanthou YCharalambous PAristidou A(2024)Virtual Instrument Performances (VIP): A Comprehensive ReviewComputer Graphics Forum10.1111/cgf.1506543:2Online publication date: 30-Apr-2024
https://doi.org/10.1111/cgf.15065
Hong JYoung Noh SLee HSik Cheong WChang J(2024)3D Clothed Human Reconstruction from Sparse Multi-View Images2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00072(677-687)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00072
Ami-Williams TSerghides CAristidou A(2024)Digitizing traditional dances under extreme clothing: The case study of EyoJournal of Cultural Heritage10.1016/j.culher.2024.02.01167(145-157)Online publication date: May-2024
https://doi.org/10.1016/j.culher.2024.02.011
Huang YTaheri OBlack MTzionas D(2024)InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction from Multi-view RGB-D ImagesInternational Journal of Computer Vision10.1007/s11263-024-01984-1132:7(2551-2566)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s11263-024-01984-1
Sun KLitvak DZhang YLi HWu JWu S(2024)Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online VideosComputer Vision – ECCV 202410.1007/978-3-031-73232-4_6(100-119)Online publication date: 30-Sep-2024
https://doi.org/10.1007/978-3-031-73232-4_6
Mollyn VArakawa RGoel MHarrison CAhuja K(2023)IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and EarbudsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581392(1-12)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581392
Jang DYang DJang DChoi BJin TLee S(2023)MOVIN: Real‐time Motion Capture using a Single LiDARComputer Graphics Forum10.1111/cgf.1496142:7Online publication date: 5-Nov-2023
https://doi.org/10.1111/cgf.14961
Wang KPeng SZhou XYang JZhang G(2023)NerfCap: Human Performance Capture With Dynamic Neural Radiance FieldsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.320250329:12(5097-5110)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TVCG.2022.3202503
Kim BKwon PLee KLee MHan SKim DJoo H(2023)Chupa: Carving 3D Clothed Humans from Skinned Shape Priors using 2D Diffusion Probabilistic Models2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01463(15919-15930)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01463
Wang YHan QHabermann MDaniilidis KTheobalt CLiu L(2023)NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00305(3272-3283)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00305
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents