research-article

Too Stiff, Too Strong, Too Smart: Evaluating Fundamental Problems with Motion Control Policies

Authors:

Sheldon Andrews,

Victor B. Zordan,

Paul G. KryAuthors Info & Claims

Proceedings of the ACM on Computer Graphics and Interactive Techniques, Volume 6, Issue 3

Article No.: 34, Pages 1 - 17

https://doi.org/10.1145/3606935

Published: 24 August 2023 Publication History

Abstract

Deep reinforcement learning (DRL) methods have demonstrated impressive results for skilled motion synthesis of physically based characters, and while these methods perform well in terms of tracking reference motions or achieving complex tasks, several concerns arise when evaluating the naturalness of the motion. In this paper, we conduct a preliminary study of specific quantitative metrics for measuring the naturalness of motion produced by DRL control policies beyond their visual appearance. Namely, we propose to study the stiffness of the control policy, in anticipation that it will influence how the character behaves in the presence of external perturbation. Second, we establish two baselines for strength that allow evaluating the use of joint torques in comparison to human performance. Third, we propose the study of variability to reveal the unnatural precision of control policies and how they compare to real human motion. In sum, we aim to establish repeatable measures to assess the naturalness of control policies produced by DRL methods, and we present a set of comparisons from state-of-the-art systems. Finally, we propose simple modifications to improve realism on these axes.

Supplemental Material

ZIP File - xie

Supplemental movie, appendix, image and software files for, Too Stiff, Too Strong, Too Smart: Evaluating Fundamental Problems with Motion Control Policies

Download
176.92 MB

References

[1]

Ijaz Akhter and Michael J. Black. 2015. Pose-conditioned joint angle limits for 3D human pose reconstruction. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1446--1455. https://doi.org/10.1109/CVPR.2015.7298751

[2]

Mazen Al Borno, Martin De Lasa, and Aaron Hertzmann. 2012. Trajectory optimization for full-body movements with complex contacts. IEEE transactions on visualization and computer graphics 19, 8 (2012), 1405--1414.

[3]

Dennis Anderson, Michael Madigan, and Maury Nussbaum. 2007. Maximum voluntary joint torque as a function of joint angle and angular velocity: Model development and application to the lower limb. Journal of biomechanics 40 (02 2007), 3105--13. https://doi.org/10.1016/j.jbiomech.2007.03.022

[4]

Arash Arami, Edwin van Asseldonk, Herman van der Kooij, and Etienne Burdet. 2020. A Clustering-Based Approach to Identify Joint Impedance During Walking. IEEE Transactions on Neural Systems and Rehabilitation Engineering 28, 8 (2020), 1808--1816. https://doi.org/10.1109/TNSRE.2020.3005389

[5]

D.J. Bennett, J.M. Hollerbach, and Y. Xu. 1992. Time-varying stiffness of human elbow joint during cyclic voluntary movement. Experimental Brain Research 88, 2 (1992).

[6]

David J Bennett. 1993. Torques generated at the human elbow joint in response to constant position errors imposed during voluntary movements. Experimental Brain Research 95, 3 (1993), 488--498.

[7]

Kevin Bergamin, Simon Clavet, Daniel Holden, and James Richard Forbes. 2019. DReCon: Data-Driven Responsive Control of Physics-Based Characters. ACM Trans. Graph. 38, 6, Article 206 (nov 2019), 11 pages. https://doi.org/10.1145/3355089.3356536

Digital Library

[8]

Steven H. Collins and Arthur D. Kuo. 2013. Two Independent Contributions to Step Variability during Over-Ground Human Walking. PLOS ONE 8, 8 (08 2013), null. https://doi.org/10.1371/journal.pone.0073597

[9]

Stelian Coros, Philippe Beaudoin, and Michiel Van de Panne. 2010. Generalized biped walking control. ACM Transactions On Graphics (TOG) 29, 4 (2010), 1--9.

Digital Library

[10]

Shannon M. Danforth, Xinyi Liu, Martin J. Ward, Patrick D. Holmes, and Ram Vasudevan. 2022. Predicting Sagittal-Plane Swing Hip Kinematics in Response to Trips. In 2022 9th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob). 1--8. https://doi.org/10.1109/BioRob52689.2022.9925354

Digital Library

[11]

Hiroaki Gomi and Mitsuo Kawato. 1997. Human arm stiffness and equilibrium-point trajectory during multi-joint movement. Biological Cybernetics 76, 3 (1997), 163--171.

[12]

Paul L Gribble, David J Ostry, Vittorio Sanguineti, and Rafael Laboissière. 1998. Are complex control signals required for human arm movement? Journal of neurophysiology 79, 3 (1998), 1409--1424.

[13]

Ikhsanul Habibie, Daniel Holden, Jonathan Schwarz, Joe Yearsley, and Taku Komura. 2017. A recurrent variational autoencoder for human motion synthesis. In 28th British Machine Vision Conference.

[14]

Samuel R Hamner, Ajay Seth, and Scott L Delp. 2010. Muscle contributions to propulsion and support during running. Journal of biomechanics 43, 14 (2010), 2709--2716.

[15]

Yifeng Jiang and C. Karen Liu. 2018. Data-Driven Approach to Simulating Realistic Human Joint Constraints. In 2018 IEEE International Conference on Robotics and Automation (ICRA). 1098--1103. https://doi.org/10.1109/ICRA.2018.8461010

Digital Library

[16]

Yifeng Jiang, Tom Van Wouwe, Friedl De Groote, and C. Karen Liu. 2019. Synthesis of Biologically Realistic Human Motion Using Joint Torque Actuation. ACM Trans. Graph. 38, 4, Article 72 (jul 2019), 12 pages. https://doi.org/10.1145/3306346.3322966

Digital Library

[17]

Paul G. Kry and Dinesh K. Pai. 2006. Interaction Capture and Synthesis. ACM Trans. Graph. 25, 3 (jul 2006), 872--880. https://doi.org/10.1145/1141911.1141969

Digital Library

[18]

Ariel Kwiatkowski, Eduardo Alvarado, Vicky Kalogeiton, C. Karen Liu, Julien Pettré, Michiel van de Panne, and Marie-Paule Cani. 2022. A Survey on Reinforcement Learning Methods in Character Animation. Computer Graphics Forum 41, 2 (2022), 613--639. https://doi.org/10.1111/cgf.14504 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14504

[19]

Seunghwan Lee, Phil Sik Chang, and Jehee Lee. 2022. Deep Compliant Control. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH '22). Association for Computing Machinery, New York, NY, USA, Article 23, 9 pages. https://doi.org/10.1145/3528233.3530719

Digital Library

[20]

Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. 2020. Character Controllers Using Motion VAEs. ACM Trans. Graph. 39, 4, Article 40 (aug 2020), 12 pages. https://doi.org/10.1145/3386569.3392422

Digital Library

[21]

C. Karen Liu, Aaron Hertzmann, and Zoran Popović. 2005. Learning Physics-Based Motion Style with Nonlinear Inverse Optimization. ACM Trans. Graph. 24, 3 (jul 2005), 1071--1081. https://doi.org/10.1145/1073204.1073314

Digital Library

[22]

Libin Liu, Michiel Van De Panne, and KangKang Yin. 2016. Guided learning of control graphs for physics-based characters. ACM Transactions on Graphics (TOG) 35, 3 (2016), 1--14.

Digital Library

[23]

Libin Liu, KangKang Yin, Michiel Van de Panne, Tianjia Shao, and Weiwei Xu. 2010. Sampling-based contact-rich motion control. In ACM SIGGRAPH 2010 papers. 1--10.

Digital Library

[24]

Daniel Ludvig, Serge Pfeifer, Xiao Hu, and Eric J. Perreault. 2012a. Time-Varying System Identification for Understanding the Control of Human Knee Impedance. IFAC Proceedings Volumes 45, 16 (2012), 1306--1310. https://doi.org/10.3182/20120711-3-BE-2027.00410 16th IFAC Symposium on System Identification.

[25]

Daniel Ludvig, Serge Pfeifer, Xiao Hu, and Eric J. Perreault. 2012b. Time-Varying System Identification for Understanding the Control of Human Knee Impedance. IFAC Proceedings Volumes 45, 16 (2012), 1306--1310. https://doi.org/10.3182/20120711-3-BE-2027.00410 16th IFAC Symposium on System Identification.

[26]

N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. Black. 2019. AMASS: Archive of Motion Capture As Surface Shapes. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE Computer Society, Los Alamitos, CA, USA, 5441--5450. https://doi.org/10.1109/ICCV.2019.00554

[27]

Lucas Mourot, Ludovic Hoyet, François Le Clerc, François Schnitzler, and Pierre Hellier. 2022. A Survey on Deep Learning for Skeleton-Based Human Animation. Computer Graphics Forum 41, 1 (2022), 122--157. https://doi.org/10.1111/cgf.14426 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14426

[28]

Hwangpil Park, Ri Yu, Yoonsang Lee, Kyungho Lee, and Jehee Lee. 2022. Understanding the stability of deep control policies for biped locomotion. The Visual Computer (2022), 1--15.

[29]

Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018a. DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills. ACM Trans. Graph. 37, 4, Article 143 (jul 2018), 14 pages. https://doi.org/10.1145/3197517.3201311

Digital Library

[30]

Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. 2018b. SFV: Reinforcement Learning of Physical Skills from Videos. ACM Trans. Graph. 37, 6, Article 178 (dec 2018), 14 pages. https://doi.org/10.1145/3272127.3275014

Digital Library

[31]

Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. 2021. AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control. ACM Trans. Graph. 40, 4, Article 144 (jul 2021), 20 pages. https://doi.org/10.1145/3450626.3459670

Digital Library

[32]

Xue Bin Peng and Michiel van de Panne. 2017. Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?. In Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation (Los Angeles, California) (SCA '17). Association for Computing Machinery, New York, NY, USA, Article 12, 13 pages. https://doi.org/10.1145/3099564.3099567

Digital Library

[33]

D. Piovesan, Alberto Pierobon, Paul Dizio, and James Lackner. 2012. Measuring Multi-Joint Stiffness during Single Movements: Numerical Validation of a Novel Time-Frequency Approach. PLoS ONE 7, 3 (08 2012). https://doi.org/10.1371/journal.pone.0033086

[34]

D. Piovesan, Alberto Pierobon, Paul Dizio, and James Lackner. 2013. Experimental Measure of Arm Stiffness During Single Reaching Movements with a Time-Frequency Analysis. Journal of neurophysiology 110, 10 (08 2013). https://doi.org/10.1152/jn.01013.2012

[35]

Alla Safonova, Jessica K. Hodgins, and Nancy S. Pollard. 2004. Synthesizing Physically Realistic Human Motion in Low-Dimensional, Behavior-Specific Spaces. In ACM SIGGRAPH 2004 Papers (Los Angeles, California) (SIGGRAPH '04). Association for Computing Machinery, New York, NY, USA, 514--521. https://doi.org/10.1145/1186562.1015754

Digital Library

[36]

Stephen H Scott. 2002. Optimal strategies for movement: success with variability. nature neuroscience 5, 11 (2002), 1110--1111.

[37]

Takaaki Shiratori, Brooke Coley, Rakié Cham, and Jessica K. Hodgins. 2009. Simulating Balance Recovery Responses to Trips Based on Biomechanical Principles. In Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (New Orleans, Louisiana) (SCA '09). Association for Computing Machinery, New York, NY, USA, 37--46. https://doi.org/10.1145/1599470.1599475

Digital Library

[38]

Gabor Stepan. 2009. Delay effects in the human sensory system during balancing. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 367, 1891 (2009), 1195--1212.

[39]

Jie Tan, Karen Liu, and Greg Turk. 2011. Stable Proportional-Derivative Controllers. IEEE Comput. Graph. Appl. 31, 4 (jul 2011), 34--44. https://doi.org/10.1109/MCG.2011.30

Digital Library

[40]

Tianxin Tao, Matthew Wilson, Ruiyu Gou, and Michiel van de Panne. 2022. Learning to Get Up. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH '22). Association for Computing Machinery, New York, NY, USA, Article 47, 10 pages. https://doi.org/10.1145/3528233.3530697

Digital Library

[41]

Emanuel Todorov and Michael I Jordan. 2002. Optimal feedback control as a theory of motor coordination. Nature neuroscience 5, 11 (2002), 1226--1235.

[42]

Tim J. van der Zee, Emily M. Mundinger, and Arthur D. Kuo. 2022. A biomechanics dataset of healthy human walking at various speeds, step lengths and step widths. Scientific Data 9, 1 (2022).

[43]

Pei Xu and Ioannis Karamouzas. 2021. A GAN-Like Approach for Physics-Based Imitation Learning and Interactive Character Control. Proc. ACM Comput. Graph. Interact. Tech. 4, 3, Article 44 (sep 2021), 22 pages. https://doi.org/10.1145/3480148

Digital Library

[44]

Sang Hoon Yeo, Martin Lesmana, Debanga R. Neog, and Dinesh K. Pai. 2012. Eyecatch: Simulating Visuomotor Coordination for Object Interception. ACM Trans. Graph. 31, 4, Article 42 (jul 2012), 10 pages. https://doi.org/10.1145/2185520.2185538

Digital Library

[45]

KangKang Yin, Kevin Loken, and Michiel Van de Panne. 2007. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics (TOG) 26, 3 (2007), 105--es.

Digital Library

[46]

Zhiqi Yin, Zeshi Yang, Michiel Van De Panne, and Kangkang Yin. 2021. Discovering Diverse Athletic Jumping Strategies. ACM Trans. Graph. 40, 4, Article 91 (jul 2021), 17 pages. https://doi.org/10.1145/3450626.3459817

Digital Library

[47]

Wenhao Yu, Greg Turk, and C. Karen Liu. 2018. Learning Symmetric and Low-Energy Locomotion. ACM Trans. Graph. 37, 4, Article 144 (jul 2018), 12 pages. https://doi.org/10.1145/3197517.3201397

Digital Library

[48]

Victor B Zordan, Anna Majkowska, Bill Chiu, and Matthew Fast. 2005. Dynamic response for motion capture animation. ACM Transactions on Graphics (TOG) 24, 3 (2005), 697--701.

Digital Library

Cited By

Boursin PKedadry YZordan VKry PCani M(2024)ReGAIL: Toward Agile Character Control From a Single Reference MotionProceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction, and Games10.1145/3677388.3696330(1-10)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3677388.3696330
Wang ZWang YYang J(2024)EarSlideProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435158:1(1-29)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643515
Chen SWang XLi WZhang JQi YTeng JZeng Z(2024)Silent Delivery: Practices and Challenges of Delivering Among Deaf or Hard of Hearing CouriersProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642801(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642801
Show More Cited By

Recommendations

Physics-based Motion Retargeting from Sparse Inputs

Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very ...
DeepMimic: example-guided deep reinforcement learning of physics-based character skills

A longstanding goal in character animation is to combine data-driven specification of behavior with a system that can execute a similar behavior in a physical simulation, thus enabling realistic responses to perturbations and environmental variation. We ...
Animation of dynamic legged locomotion

This paper is about the use of control algorithms to animate dynamic legged locomotion. Control could free the animator from specifying the details of joint and limb motion while producing both physically realistic and natural looking results. We ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques

Proceedings of the ACM on Computer Graphics and Interactive Techniques Volume 6, Issue 3

August 2023

403 pages

EISSN:2577-6193

DOI:10.1145/3617582

Issue’s Table of Contents

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2023

Published in PACMCGIT Volume 6, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
164
Total Downloads

Downloads (Last 12 months)127
Downloads (Last 6 weeks)17

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Boursin PKedadry YZordan VKry PCani M(2024)ReGAIL: Toward Agile Character Control From a Single Reference MotionProceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction, and Games10.1145/3677388.3696330(1-10)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3677388.3696330
Wang ZWang YYang J(2024)EarSlideProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435158:1(1-29)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643515
Chen SWang XLi WZhang JQi YTeng JZeng Z(2024)Silent Delivery: Practices and Challenges of Delivering Among Deaf or Hard of Hearing CouriersProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642801(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642801
Hartmann AKang DZargarbashi FZamora MCoros S(2024)Deep Compliant Control for Legged Robots2024 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA57147.2024.10611209(11421-11427)Online publication date: 13-May-2024
https://doi.org/10.1109/ICRA57147.2024.10611209
Liu JWang HLi YLi P(2023)Research on Common Structure of Motion Data2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)10.1109/BIBM58861.2023.10385523(2639-2646)Online publication date: 5-Dec-2023
https://doi.org/10.1109/BIBM58861.2023.10385523

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents