research-article

Near-contact Person-to-3D Character Dance Training: Comparing AR and VR for Interactive Entertainment

Authors:

Salva Kirakosian,

Grigoris Daskalogrigorakis,

Emmanuel Maravelakis,

Katerina ManiaAuthors Info & Claims

2021 IEEE Conference on Games (CoG)

Pages 1 - 5

https://doi.org/10.1109/CoG52621.2021.9619037

Published: 17 August 2021 Publication History

Abstract

This paper presents an innovative, near-contact, person-to-3D character, real-time dance training framework for leaders developed in Augmented Reality (AR) for interactive entertainment. Rather than mimicking pre-recorded dance animations as in previous work, the user is trained to lead a 3D partner while dancing, employing mid-air gestures. Real and virtual hands are joined and dance movements are initiated to which the 3D character responds to, through Unity's Inverse Kinematics solvers, in a non-predetermined manner following Latin dance rules. We evaluate task workload and usability of our AR dance training system compared with the Virtual Reality (VR) equivalent. Perceived training performance, accuracy of actions and fun were lower in AR, evoking increased training effort in AR compared to playfully training in VR. Design recommendations include increasing AR's Field of View, enhancing high quality graphics in AR, removing obtrusive cabling in VR and improving finger tracking in VR.

References

[1]

G. A. Koulieris, K. Akşit, M. Stengel, R. K Mantiuk, K. Mania, and C. Richardt. Near-eye display and tracking technologies for virtual and augmented reality. In Computer Graphics Forum, vol 38, pp 493–519.

[2]

S. Kirakosian, E. Maravelakis, and K. Mania. Immersive simulation and training of person-to-3d character dance in real-time. In 2019 11th Intl. Conf. on Virtual Worlds and Games for Serious Applications, pp 1–4.

[3]

K. Jeonghyun and Y. Ungyeon. Implementation and evaluation of “just follow me”: An immersive, vr-based, motion-training system. Presence: Teleoperators and Virtual Environments, vol 11, pp 304–323.

[4]

K. Hachimura, H. Kato, and H. Tamura. A prototype dance training support system with motion capture and mixed reality technologies. In 13th IEEE Intl. Workshop on Robot and Human Interactive Communication pp 217–222.

[5]

D. Alexiadis, P. Kelly, P. Daras, N. O'Connor, T. Boubekeur, and M. B. Moussa. Evaluating a dancer's performance using kinect-based skeleton tracking. In Proc. 19th ACM International Conference on Multimedia, MM '11, pp 659–662.

[6]

J. C. P. Chan, H. Leung, J. K. T. Tang, and T. Komura. A virtual reality dance training system using motion capture technology. IEEE Trans. on Learning Technologies, vol 4, pp 187–195.

[7]

M. Raptis, D. Kirovski, and H. Hoppe. Real-time classification of dance gestures from skeleton animation. In Proc. 2011 ACM SIG-GRAPH/Eurographics Symp. on Computer Animation, pages 147–156.

[8]

L. Deng, H. Leung, N. Gu, and Y. Yang. Real-time mocap dance recognition for an interactive dancing game. Computer Animation and Virtual Worlds, 22(2–3):229–237, 2011.

Digital Library

[9]

Y. Yang, H. Leung, L. Yue, and L. Deng. Automatic dance lesson generation. IEEE Trans. Learn. Technol., 5(3):191–198, January2012.

Digital Library

[10]

M. Kyan, G. Sun, H. Li, L. Zhong, P. Muneesawang, Nan N. Dong, B. Elder, and L. Guan. An approach to ballet dance training through ms kinect and visualization in a cave virtual reality environment. ACM Trans. Intell. Syst. Technol., 6(2):23:1–23:37, 2015.

[11]

J. Tang, Jacky J. C. P. Chan, and H. Leung. Interactive dancing game with real-time recognition of continuous dance moves from 3d human motion capture. In Proc. of 5th ACM Int. Conf. on Ubiquitous Information Management and Communication, pages 50:1–50:9.

[12]

I. Rallis, A. Langis, I. Georgoulas, A. Voulodimos, N. Doulamis, and A. Doulamis. An embodied learning game using kinect and labanotation for analysis and visualization of dance kinesiology. In 10th Int. Conf. on Virtual Worlds and Games for Serious Appl.2018, pages 1–8.

[13]

C. Mousas. Performance-driven dance motion control of a virtual partner character. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 57–64, March2018.

[14]

T. Takeda, Y. Hirata, and K. Kosuge. Dance step estimation method based on hmm for dance partner robot. IEEE Transactions on Industrial Electronics, 54(2):699–706, April2007.

[15]

E. Ho, J. C. P. Chan, T. Komura, and H. Leung. Interactive partner control in close interactions for real-time applications. ACM Trans. Multimedia Comput. Commun. Appl., 9(3):21:1–21:19, July2013.

[16]

V. Saxena, T. Feldt, and M. Goel. Augmented telepresence as a tool for immersive simulated dancing in experience and learning. In Proc. of ACM India HCI 2014 Conference on Human Computer Interaction, IndiaHCI'14, pages 86:86–86:89. ACM, 2014.

[17]

I. Kico and F. Liarokapis. A mobile augmented reality interface for teaching folk dances. In 25th ACM Symp. on Virtual Reality Software and Technology, 47:1–2.

[18]

F. Anderson, T. Grossman, J. Matejka, and G. Fitzmaurice. Youmove: Enhancing movement training with an augmented reality mirror. In Proc. 26th Annual ACM Symposium on User Interface Software and Technology, pp 311–320.

[19]

S. Yan, G. Ding, Z. Guan, N. Sun, H. Li, and L. Zhang. Outsideme: Augmenting dancer's external self-image by using a mixed reality system. In Proc. 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA'15, pp 965–970.

[20]

L. Merino, M. Schwarzl, M. Kraus, M. Sedlmair, D. Schmalstieg, and D. Weiskopf. Evaluating mixed and augmented reality: A systematic literature review (2009-2019). In 2020 IEEE Intl. Symp. on Mixed and Augmented Reality (ISMAR), pp 438–451.

[21]

M. Billinghurst. Grand challenges for augmented reality. Frontiers in Virtual Reality, vol 2, pp 12.

[22]

A. Dey, M. Billinghurst, R. W. Lindeman, and J Swan. A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Frontiers in Robotics and AI, 5:37.

[23]

F. Pallavicini, A. Pepe, and M. E. Minissi. Gaming in virtual reality: what changes in terms of usability, emotional response and sense of presence compared to non-immersive video games?Simulation & Gaming, vol 50, pp 136–159.

[24]

G. A. Koulieris, G. Drettakis, D. Cunningham, and K. Mania. Gaze prediction using machine learning for dynamic stereo manipulation in games. In 2016 IEEE Virtual Reality (VR), pp 113–120.

[25]

K. Kim, M. Billinghurst, G. Bruder, H. B.-L. Duh, and G. F. Welch. Revisiting trends in augmented reality research: A review of the 2nd decade of ismar (2008-2017). IEEE trans. on visualization and computer graphics, vol 24, pp 2947–2962.

[26]

JJ-W. Lin, H. B.-L. Duh, D. E. Parker, H. Abi-Rached, and T. A. Furness. Effects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment. In Proc. ieee virtual reality 2002, pp 164–171.

[27]

Y. Ling, H. T. Nefs, W.-P. Brinkman, C. Qu, and I. Heynderickx. The effect of perspective on presence and space perception. PloS one, vol 8.

[28]

S. G. Hart. Nasa-task load index (nasa-tlx); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, vol 50, pp 904–908.

[29]

T. Piumsomboon, D. Altimira, H. Kim, A. Clark, G. Lee, and M. Billinghurst. Grasp-shell vs gesture-speech: A comparison of direct and indirect natural interaction techniques in augmented reality. In 2014 IEEE Intl. Symp. on Mixed and Augmented Reality (ISMAR), pp 73–82.

Cited By

Pegia MDiplaris SVrochidis SSchuldt HSpiess FArnold RBailer WGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Multimedia Retrieval in and for XRProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658421(1324-1325)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658421
Hassan SDe Lacerda Pataca CNourian LTigwell GDavis BSilver Wagman W(2024)Designing and Evaluating an Advanced Dance Video Comprehension Tool with In-situ Move Identification CapabilitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642710(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642710
Wang YZhao SChen WShi ZXie LYan YYin E(2024)Trajectory-based Calibration for Optical See-Through Head-Mounted Displays Without AlignmentPattern Recognition and Computer Vision10.1007/978-981-97-8508-7_16(226-240)Online publication date: 18-Oct-2024
https://dl.acm.org/doi/10.1007/978-981-97-8508-7_16
Show More Cited By

Index Terms

Near-contact Person-to-3D Character Dance Training: Comparing AR and VR for Interactive Entertainment

Index terms have been assigned to the content through auto-classification.

Recommendations

Exhibition approach using an AR and VR pillar
SA '17: SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications

This demonstration presents a development of an Augmented Reality (AR) and Virtual Reality (AR) pillar, a novel approach for showing AR and VR content in a public setting. A pillar in a public exhibition venue was converted to a four-sided AR and VR ...
Rapid Prototyping for AR/VR Experiences
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

This course will provide an introduction to techniques for rapid prototyping for Augmented Reality (AR) and Virtual Reality (VR) experiences. With the rise of consumer head mounted displays and powerful mobile phones, AR and VR are becoming increasingly ...
From VR to AR: Adding AR Functionality to an Existing VR Software Framework
SVR '12: Proceedings of the 2012 14th Symposium on Virtual and Augmented Reality

We present our approach to extend a Virtual Reality software framework towards the use for Augmented Reality applications. Although VR and AR applications have very similar requirements in terms of abstract components (like 6DOF input, stereoscopic ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

2021 IEEE Conference on Games (CoG)

Aug 2021

1113 pages

Copyright © 2021.

Publisher

IEEE Press

Publication History

Published: 17 August 2021

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pegia MDiplaris SVrochidis SSchuldt HSpiess FArnold RBailer WGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Multimedia Retrieval in and for XRProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658421(1324-1325)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658421
Hassan SDe Lacerda Pataca CNourian LTigwell GDavis BSilver Wagman W(2024)Designing and Evaluating an Advanced Dance Video Comprehension Tool with In-situ Move Identification CapabilitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642710(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642710
Wang YZhao SChen WShi ZXie LYan YYin E(2024)Trajectory-based Calibration for Optical See-Through Head-Mounted Displays Without AlignmentPattern Recognition and Computer Vision10.1007/978-981-97-8508-7_16(226-240)Online publication date: 18-Oct-2024
https://dl.acm.org/doi/10.1007/978-981-97-8508-7_16
Gavgiotaki DNtoa SMargetis GApostolakis KStephanidis C(2023)Gesture-based Interaction for AR Systems: A Short ReviewProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3594806.3594815(284-292)Online publication date: 5-Jul-2023
https://dl.acm.org/doi/10.1145/3594806.3594815
Kritikos YGiariskanis FProtopapadaki EPapanastasiou APapadopoulou EMania K(2023)Audio Augmented Reality OutdoorsProceedings of the 2023 ACM International Conference on Interactive Media Experiences10.1145/3573381.3597028(199-204)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3573381.3597028
Giariskanis FKritikos YProtopapadaki EPapanastasiou APapadopoulou EMania K(2022)The Augmented Museum: A Multimodal, Game-Based, Augmented Reality Narrative for Cultural HeritageProceedings of the 2022 ACM International Conference on Interactive Media Experiences10.1145/3505284.3532967(281-286)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1145/3505284.3532967

View Options

View options

Media

Figures

Other

Tables

View Table of Contents