Work in Progress

Virtual Triplets: A Mixed Modal Synchronous and Asynchronous Collaboration with Human-Agent Interaction in Virtual Reality

Authors:

Jingjing Zhang,

Simon Hoermann,

Thammathip PiumsomboonAuthors Info & Claims

CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

Article No.: 366, Pages 1 - 8

https://doi.org/10.1145/3613905.3650766

Published: 11 May 2024 Publication History

Abstract

We introduce the Virtual Triplets (VTs), a Virtual Reality collaborative system designed for both synchronous and asynchronous interaction through collaboration between humans and human-agent. When a single instructor supervises two students in a showcased virtual cooking class scenario, the system enables the instructor to switch control between avatars in two separate environments, and a virtual agent takes over in the instructor’s absence, ensuring continuous support. VTs facilitate the recording and playback of the instructor’s demonstrations for students, coupled with a feature that allows the instructor to employ a bird’s-eye view for effective classroom management. Analysis of observational data and interviews from our pilot study indicates good system usability, ease of avatar management for instructors, and a strong sense of continuous support among students. We discuss potential improvements and broader applications of VTs, aiming to enhance user experience in multitasking scenarios involving multiuser human-human and human-agent collaboration.

Supplemental Material

MP4 File - Video Preview

Video Preview

Transcript for: Video Preview

MP4 File

Talk Video

Transcript for: Talk Video

MP4 File

Video Figure

Transcript for: Video Figure

References

[1]

Keith Anderson, Elisabeth André, T. Baur, Sara Bernardini, M. Chollet, E. Chryssafidou, I. Damian, C. Ennis, A. Egges, P. Gebhard, H. Jones, M. Ochs, C. Pelachaud, Kaśka Porayska-Pomsta, P. Rizzo, and Nicolas Sabouret. 2013. The TARDIS Framework: Intelligent Virtual Agents for Social Coaching in Job Interviews. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8253 LNCS. 476–491. https://doi.org/10.1007/978-3-319-03161-3_35

[2]

Zahra Ashktorab, Casey Dugan, James Johnson, Qian Pan, Wei Zhang, Sadhana Kumaravel, and Murray Campbell. 2021. Effects of Communication Directionality and AI Agent Differences in Human-AI Interaction. 1–15. https://doi.org/10.1145/3411764.3445256

Digital Library

[3]

A Bangor, P Kortum, J Miller Journal of usability studies, and undefined 2009. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. uxpajournal.org 4 (2009), 114–123. https://uxpajournal.org/wp-content/uploads/sites/7/pdf/JUS_Bangor_May2009.pdf

[4]

Timothy W. Bickmore, Daniel Schulman, and Candace Sidner. 2013. Automated interventions for multiple health behaviors using conversational agents. Patient Education and Counseling 92, 2 (2013). https://doi.org/10.1016/j.pec.2013.05.011

[5]

John Brooke. 1996. SUS -A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996).

[6]

Zoraida Callejas, Brian Ravenet, Magalie Ochs, and Catherine Pelachaud. 2014. A computational model of social attitudes for a virtual recruiter. In 13th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2014, Vol. 1.

[7]

Yuanzhi Cao, Xun Qian, Tianyi Wang, Rachel Lee, Ke Huo, and Karthik Ramani. 2020. An Exploratory Study of Augmented Reality Presence for Tutoring Machine Tasks. In Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376688

Digital Library

[8]

Alan Cheng, Lei Yang, and Erik Andersen. 2017. Teaching language and culture with a virtual reality game. In Conference on Human Factors in Computing Systems - Proceedings, Vol. 2017-May. https://doi.org/10.1145/3025453.3025857

Digital Library

[9]

Hyunwoo Cho, Bowen Yuan, Jonathon Derek Hart, Eunhee Chang, Zhuang Chang, Jiashuo Cao, Gun A. Lee, Thammathip Piumsomboon, and Mark Billinghurst. 2023. An Asynchronous Hybrid Cross Reality Collaborative System. 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) (10 2023), 70–73. https://doi.org/10.1109/ISMAR-ADJUNCT60411.2023.00022

[10]

Andre Doucette, Carl Gutwin, Regan Mandryk, Miguel Nacenta, and Sunny Sharma. 2013. Sometimes when we touch: How arm embodiments change reaching and coordination on digital tables. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW. https://doi.org/10.1145/2441776.2441799

Digital Library

[11]

Volodymyr Dziubak, Ben Lafreniere, Tovi Grossman, Andrea Bunt, and George Fitzmaurice. 2018. Maestro: Designing a System for Real-Time Orchestration of 3D Modeling Workshops. (2018). https://doi.org/10.1145/3242587.3242606

Digital Library

[12]

Pak Ming Fan, Santawat Thanyadit, and Ting Chuen Pong. 2023. VisTA-LIVE: A Visualization Tool for Assessment of Laboratories in Virtual Environments. IEEE Transactions on Visualization and Computer Graphics (2023). https://doi.org/10.1109/TVCG.2023.3341079

Digital Library

[13]

Tiziano Furlanetto, Cesare Bertone, and Cristina Becchio. 2013. The bilocated mind: new perspectives on self-localization and self-identification. Frontiers in Human Neuroscience 7 (2013). https://doi.org/10.3389/fnhum.2013.00071

[14]

S. Greenberg, C. Gutwin, and M. Roseman. 1996. Semantic telepointers for groupware. In Proceedings - 6th Australian Conference on Computer-Human Interaction, OzCHI 1996. https://doi.org/10.1109/OZCHI.1996.559988

[15]

Chad Harms and Frank Biocca. 2004. Internal Consistency and Reliability of the Networked Minds Measure of Social Presence. Seventh Annual International Workshop: Presence 2004 (2004).

[16]

Jonathon Derek Hart, Thammathip Piumsomboon, Gun A. Lee, Ross T. Smith, and Mark Billinghurst. 2021. Manipulating Avatars for Enhanced Communication in Extended Reality. In Proceedings - 2021 IEEE International Conference on Intelligent Reality, ICIR 2021. https://doi.org/10.1109/ICIR51845.2021.00011

[17]

P. Heidicker, E. Langbehn, and F. Steinicke. 2017. Influence of avatar appearance on presence in social VR. In 2017 IEEE Symposium on 3D User Interfaces, 3DUI 2017 - Proceedings. 233–234. https://doi.org/10.1109/3DUI.2017.7893357

[18]

D. A. Kolb. 2015. Experiential Learning: Experience as the source of Learning and Development Second Edition. Pearson Education, Inc. Upper Saddle River (2015), 41. https://doi.org/10.1002/job.4030080408

[19]

B.T. Kumaravel, F. Anderson, G. Fitzmaurice, B. Hartmann, and T. Grossman. 2019. Loki: Facilitating remote instruction of physical tasks using Bi-directional mixed-reality telepresence. In UIST 2019 - Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 161–174. https://doi.org/10.1145/3332165.3347872

Digital Library

[20]

B.T. Kumaravel, C. Nguyen, S. DiVerdi, and B. Hartmann. 2019. TutoriVR: A video-based tutorial system for design applications in virtual reality. In Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3290605.3300514

Digital Library

[21]

Marc Erich Latoschik, Daniel Roth, Dominik Gall, Jascha Achenbach, Thomas Waltemate, and Mario Botsch. 2017. The effect of avatar realism in immersive social virtual realities. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST, Vol. Part F131944. https://doi.org/10.1145/3139131.3139156

Digital Library

[22]

Gun Lee, Seungjun Ahn, William Hoff, and Mark Billinghurst. 2019. AR Tips: Augmented first-person view task instruction videos. Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019 (10 2019), 34–36. https://doi.org/10.1109/ISMAR-ADJUNCT.2019.00024

[23]

Zahira Merchant, Ernest T. Goetz, Lauren Cifuentes, Wendy Keeney-Kennicutt, and Trina J. Davis. 2014. Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Computers & Education 70 (1 2014), 29–40. https://doi.org/10.1016/J.COMPEDU.2013.07.033

[24]

Mikhail Morozov, Alexey Gerasimov, Mikhail Fominykh, and Andrey Smorkalov. 2013. Asynchronous immersive classes in a 3D virtual world: Extended description of vAcademia. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 7848. https://doi.org/10.1007/978-3-642-38803-3_5

[25]

Michael Nebeling, Shwetha Rajaram, Liwei Wu, Yifei Cheng, and Jaylin Herskovitz. 2021. XRStudio: A Virtual Production and Live Streaming System for Immersive Instructional Experiences. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–12. https://doi.org/10.1145/3411764.3445323

Digital Library

[26]

Thammathip Piumsomboon, Gun A. Lee, Jonathon D. Hart, Barrett Ens, Robert W. Lindeman, Bruce H. Thomas, and Mark Billinghurst. 2018. Mini-me: An adaptive avatar for Mixed Reality remote collaboration. In Conference on Human Factors in Computing Systems - Proceedings, Vol. 2018-April. https://doi.org/10.1145/3173574.3173620

Digital Library

[27]

Thammathip Piumsomboon, Gun A. Lee, Andrew Irlitti, Barrett Ens, Bruce H. Thomas, and Mark Billinghurst. 2019. On the shoulder of the giant: A multi-scale mixed reality collaboration with 360 video sharing and tangible interaction. In Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3290605.3300458

Digital Library

[28]

Jaziar Radianti, Tim A. Majchrzak, Jennifer Fromm, and Isabell Wohlgenannt. 2020. A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers and Education 147 (2020). https://doi.org/10.1016/j.compedu.2019.103778

Digital Library

[29]

Albert Rizzo, Stefan Scherer, David Devault, Jonathan Gratch, Ronald Artstein, Arno Hartholt, Gale Lucas, Stacey Marsella, Fabrizio Morbini, Angela Nazarian, Giota Stratou, David Traum, Rachel Wood, Jill Boberg, and Louis Philippe Morency. 2016. Detection and computational analysis of psychological signals using a virtual human interviewing agent. Journal of Pain Management 9, 3 (2016).

[30]

Kyoungwon Seo, Joice Tang, Ido Roll, Sidney Fels, and Dongwook Yoon. 2021. The impact of artificial intelligence on learner–instructor interaction in online learning. International Journal of Educational Technology in Higher Education 18, 1 (2021). https://doi.org/10.1186/s41239-021-00292-9

[31]

Matthew J. Smith, Emily J. Ginger, Katherine Wright, Michael A. Wright, Julie Lounds Taylor, Laura Boteler Humm, Dale E. Olsen, Morris D. Bell, and Michael F. Fleming. 2014. Virtual reality job interview training in adults with autism spectrum disorder. Journal of Autism and Developmental Disorders 44, 10 (2014). https://doi.org/10.1007/s10803-014-2113-y

[32]

William Swartout, Benjamin D. Nye, Arno Hartholt, Adam Reilly, Arthur C. Graesser, Kurt VanLehn, Jon Wetzel, Matt Liewer, Fabrizio Morbini, Brent Morgan, Lijia Wang, Grace Benn, and Milton Rosenberg. 2016. Designing a personal assistant for life-long learning (PAL3). In Proceedings of the 29th International Florida Artificial Intelligence Research Society Conference, FLAIRS 2016.

[33]

Anthony Tang, Carman Neustaedter, and Saul Greenberg. 2007. VideoArms: Embodiments for Mixed Presence Groupware. In People and Computers XX — Engage. Springer London, London, 85–102. https://doi.org/10.1007/978-1-84628-664-3_8

[34]

Santawat Thanyadit, Matthias Heintz, and Effie L.C. Law. 2023. Tutor In-sight: Guiding and Visualizing Students’ Attention with Mixed Reality Avatar Presentation Tools. In Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3544548.3581069

Digital Library

[35]

S. Thanyadit, P. Punpongsanon, T. Piumsomboon, and T.-C. Pong. 2022. XR-LIVE: Enhancing Asynchronous Shared-Space Demonstrations with Spatial-temporal Assistive Toolsets for Effective Learning in Immersive Virtual Laboratories. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022). https://doi.org/10.1145/3512983

Digital Library

[36]

Santawat Thanyadit, Parinya Punpongsanon, and Ting Chuen Pong. 2019. ObserVAR: Visualization system for observing virtual reality users using augmented reality. In Proceedings - 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2019. https://doi.org/10.1109/ISMAR.2019.00023

[37]

T.-Y. Wang, Y. Sato, M. Otsuki, H. Kuzuoka, and Y. Suzuki. 2019. Effect of full body avatar in augmented reality remote collaboration. In 26th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2019 - Proceedings. 1221–1222. https://doi.org/10.1109/VR.2019.8798044

[38]

Boram Yoon, Hyung Il Kim, Gun A. Lee, Mark Billinqhurst, and Woontack Woo. 2019. The effect of avatar appearance on social presence in an augmented reality remote collaboration. In 26th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2019 - Proceedings. https://doi.org/10.1109/VR.2019.8797719

[39]

K. Zibrek, E. Kokkinara, and R. McDonnell. 2018. The effect of realistic appearance of virtual characters in immersive environments - Does the character’s personality play a role?IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1681–1690. https://doi.org/10.1109/TVCG.2018.2794638

Digital Library

Cited By

Bimberg PZielasko DWeyers B(2024)The Impact of Task-Responsibility on User Experience and Behaviour under Asymmetric Knowledge ConditionsProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687721(1-11)Online publication date: 9-Oct-2024
https://dl.acm.org/doi/10.1145/3641825.3687721

Index Terms

Virtual Triplets: A Mixed Modal Synchronous and Asynchronous Collaboration with Human-Agent Interaction in Virtual Reality
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Virtual reality

Recommendations

Multi-sensorial virtual reality and augmented human food interaction
MHFI '16: Proceedings of the 1st Workshop on Multi-sensorial Approaches to Human-Food Interaction

In the field of virtual reality (VR) research, media technologies to create a realistic feeling of being present in a real/virtual world by duplicating multi-sensory information have been studied over a long period. Recently, technologies for multi-...
Headset removal for virtual and mixed reality
SIGGRAPH '17: ACM SIGGRAPH 2017 Talks

Virtual Reality (VR) has advanced significantly in recent years and allows users to explore novel environments (both real and imaginary), play games, and engage with media in a way that is unprecedentedly immersive. However, compared to physical reality,...
Virtual reality and mixed reality for virtual learning environments

This paper explores educational uses of virtual learning environment (VLE) concerned with issues of learning, training and entertainment. We analyze the state-of-art research of VLE based on virtual reality and augmented reality. Some examples for the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

May 2024

4761 pages

ISBN:9798400703317

DOI:10.1145/3613905

Editors:
Florian Floyd Mueller
Monash University
,
Penny Kyburz
The Australian National University
,
Julie R. Williamson
University of Glasgow
,
Corina Sas
Lancaster University

Copyright © 2024 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 May 2024

Check for updates

Author Tags

Qualifiers

Work in progress
Research
Refereed limited

Funding Sources

China Scholarship Council

Conference

CHI '24

Sponsor:

CHI '24: CHI Conference on Human Factors in Computing Systems

May 11 - 16, 2024

HI, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
584
Total Downloads

Downloads (Last 12 months)584
Downloads (Last 6 weeks)85

Reflects downloads up to 30 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bimberg PZielasko DWeyers B(2024)The Impact of Task-Responsibility on User Experience and Behaviour under Asymmetric Knowledge ConditionsProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687721(1-11)Online publication date: 9-Oct-2024
https://dl.acm.org/doi/10.1145/3641825.3687721

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Table of Conten