abstract

AvatarMeeting: An Augmented Reality Remote Interaction System With Personalized Avatars

Authors:

Xuanyu Wang,

Yang Wang,

Yan Shi,

Weizhan Zhang,

Qinghua ZhengAuthors Info & Claims

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Pages 4533 - 4535

https://doi.org/10.1145/3394171.3414449

Published: 12 October 2020 Publication History

Get Access

Abstract

To further enhance the immersion perception of remote interaction, avatars can be involved harnessing Head Mounted Display (HMD) based Augmented Reality (AR). In our demonstration, we present an avatar based remote interaction system AvatarMeeting, enabling users to meet with remote peers through interactive personalized avatars just like face to face. Specifically, we propose a novel framework including a consumer-grade set-up, a complete transmission scheme and a processing pipeline, which consists of prescan modeling, pose detection and action reconstruction. And an angle based reconstruction approach is introduced to empower the AR avatars to perform the same actions as each remote real person do in real time smoothly while keeping a good avatar shape.

Supplementary Material

MP4 File (3394171.3414449.mp4)

To further enhance the immersion perception of remote interaction, avatars can be involved harnessing Head Mounted Display (HMD) based Augmented Reality (AR). In our demonstration, we present an avatar based remote interaction system AvatarMeeting, enabling users to meet with remote peers through interactive personalized avatars just like face to face. Specifically, we propose a novel framework including a consumer-grade set-up, a complete transmission scheme and a processing pipeline, which consists of pre scan modeling, pose detection and action reconstruction. And an angle based reconstruction approach is introduced to empower the AR avatars to perform the same actions as each remote real person do in real time smoothly while keeping a good avatar shape. We see our demo as a first step into the complete usable avatar based remote interaction system, and in the future, will optimize it from the aspects of: real time 3D pose estimation methods, 3D human reconstruction and mobile deployment.

Download
41.35 MB

References

[1]

Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. 2018. Video based reconstruction of 3d people models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8387--8397.

Crossref

Google Scholar

[2]

A. Best, S. Narang, and D. Manocha. 2020. SPA: Verbal Interactions between Agents and Avatars in Shared Virtual Environments using Propositional Planning. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 117--126.

Google Scholar

[3]

Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2018. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. arXiv preprint arXiv:1812.08008 (2018).

Google Scholar

[4]

Young-Woon Cha, True Price, Zhen Wei, Xinran Lu, Nicholas Rewkowski, Rohan Chabra, Zihe Qin, Hyounghun Kim, Zhaoqi Su, Yebin Liu, et al. 2018. Towards fully mobile 3D face, body, and environment capture using only head-worn cameras. IEEE transactions on visualization and computer graphics, Vol. 24, 11 (2018), 2993--3004.

Google Scholar

[5]

Youjin Choi, Jeongmi Lee, and Sung-Hee Lee. 2020. Effects of Locomotion Style and Body Visibility of a Telepresence Avatar. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 1--9.

Crossref

Google Scholar

[6]

Allen J Fairchild, Simon P Campion, Arturo S Garcia, Robin Wolff, Terrence Fernando, and David J Roberts. 2016. A mixed reality telepresence system for collaborative space operation. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 27, 4 (2016), 814--827.

Digital Library

Google Scholar

[7]

Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, et al. 2016. Holoportation: Virtual 3d teleportation in real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 741--754.

Digital Library

Google Scholar

[8]

Tanmay V Randhavane, Aniket Bera, Emily Kubin, Kurt Gray, and Dinesh Manocha. 2019. Modeling data-driven dominance traits for virtual characters using gait analysis. IEEE transactions on visualization and computer graphics (2019).

Google Scholar

[9]

Patrick Stotko, Stefan Krumpen, Matthias B Hullin, Michael Weinmann, and Reinhard Klein. 2019. SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence. IEEE transactions on visualization and computer graphics, Vol. 25, 5 (2019), 2102--2112.

Google Scholar

[10]

P. Stotko, S. Krumpen, M. Weinmann, and R. Klein. 2019. Efficient 3D Reconstruction and Streaming for Group-Scale Multi-Client Live Telepresence. In 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 19--25.

Google Scholar

[11]

Ramanarayan Vasudevan, Gregorij Kurillo, Edgar Lobaton, Tony Bernardin, Oliver Kreylos, Ruzena Bajcsy, and Klara Nahrstedt. 2011. High-quality visualization for geographically distributed 3-d teleimmersive applications. IEEE Transactions on Multimedia, Vol. 13, 3 (2011), 573--584.

Digital Library

Google Scholar

Cited By

View all

Zawish MDharejo FKhowaja SRaza SDavy SDev KBellavista P(2024)AI and 6G Into the Metaverse: Fundamentals, Challenges and Future Research TrendsIEEE Open Journal of the Communications Society10.1109/OJCOMS.2024.33494655(730-778)Online publication date: 2024
https://doi.org/10.1109/OJCOMS.2024.3349465
Abdari AFalcon ASerra G(2024)A Language-Based Solution to Enable Metaverse RetrievalMultiMedia Modeling10.1007/978-3-031-53311-2_35(477-488)Online publication date: 28-Jan-2024
https://doi.org/10.1007/978-3-031-53311-2_35
Franzò MPascucci SMarinozzi FBini F(2024)Challenges and Outlook to Designing Cutting-Edge Mixed Reality Technologies of Human Pose EstimationMEDICON’23 and CMBEBIH’2310.1007/978-3-031-49062-0_78(743-754)Online publication date: 4-Jan-2024
https://doi.org/10.1007/978-3-031-49062-0_78
Show More Cited By

Index Terms

AvatarMeeting: An Augmented Reality Remote Interaction System With Personalized Avatars
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Mixed / augmented reality

Recommendations

Mini-Me: An Adaptive Avatar for Mixed Reality Remote Collaboration
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

We present Mini-Me, an adaptive avatar for enhancing Mixed Reality (MR) remote collaboration between a local Augmented Reality (AR) user and a remote Virtual Reality (VR) user. The Mini-Me avatar represents the VR user's gaze direction and body gestures ...
Holoportation: Virtual 3D Teleportation in Real-time
UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and Technology

We present an end-to-end system for augmented and virtual reality telepresence, called Holoportation. Our system demonstrates high-quality, real-time 3D reconstructions of an entire space, including people, furniture and objects, using a set of new ...
The effect of avatar realism in immersive social virtual realities
VRST '17: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology

This paper investigates the effect of avatar realism on embodiment and social interactions in Virtual Reality (VR). We compared abstract avatar representations based on a wooden mannequin with high fidelity avatars generated from photogrammetry 3D scan ...

Comments

Information & Contributors

Information

Published In

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

October 2020

4889 pages

ISBN:9781450379885

DOI:10.1145/3394171

General Chairs:
Chang Wen Chen
Chinese University of Hong Kong, Shenzhen, China
,
Rita Cucchiara
UNIMORE, Italy
,
Xian-Sheng Hua
Alibaba Group, China
,
Program Chairs:
Guo-Jun Qi
Futurewei Technologies, USA
,
Elisa Ricci
UNITN & Fondazione Bruno Kessler, Italy
,
Zhengyou Zhang
Tencent, China
,
Roger Zimmermann
National University of Singapore, Singapore

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2020

Check for updates

Author Tags

Qualifiers

Abstract

Funding Sources

NSFC
National Key R&D Program of China

Conference

MM '20

Sponsor:

SIGMM

MM '20: The 28th ACM International Conference on Multimedia

October 12 - 16, 2020

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
349
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)4

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zawish MDharejo FKhowaja SRaza SDavy SDev KBellavista P(2024)AI and 6G Into the Metaverse: Fundamentals, Challenges and Future Research TrendsIEEE Open Journal of the Communications Society10.1109/OJCOMS.2024.33494655(730-778)Online publication date: 2024
https://doi.org/10.1109/OJCOMS.2024.3349465
Abdari AFalcon ASerra G(2024)A Language-Based Solution to Enable Metaverse RetrievalMultiMedia Modeling10.1007/978-3-031-53311-2_35(477-488)Online publication date: 28-Jan-2024
https://doi.org/10.1007/978-3-031-53311-2_35
Franzò MPascucci SMarinozzi FBini F(2024)Challenges and Outlook to Designing Cutting-Edge Mixed Reality Technologies of Human Pose EstimationMEDICON’23 and CMBEBIH’2310.1007/978-3-031-49062-0_78(743-754)Online publication date: 4-Jan-2024
https://doi.org/10.1007/978-3-031-49062-0_78
Abdari AFalcon ASerra GJi WWei YZheng ZFei HChua T(2023)Metaverse Retrieval: Finding the Best Metaverse Environment via LanguageProceedings of the 1st International Workshop on Deep Multimodal Learning for Information Retrieval10.1145/3606040.3617445(1-9)Online publication date: 2-Nov-2023
https://dl.acm.org/doi/10.1145/3606040.3617445
Venugopal JSubramanian APeatchimuthu J(2023)The realm of metaverse: A surveyComputer Animation and Virtual Worlds10.1002/cav.215034:5Online publication date: 2-Mar-2023
https://doi.org/10.1002/cav.2150
Peng S(2022)Excitement Projector: Augmenting Excitement-Perception and Arousal through Bio-signal-based Haptic Feedback in Remote-Sport WatchingCHI Conference on Human Factors in Computing Systems Extended Abstracts10.1145/3491101.3519619(1-6)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519619
Kida YMatsuoka KOgi T(2022)Necessary Requirements of Avatars for Remote Communication in Real SpaceAdvances in Network-Based Information Systems10.1007/978-3-031-14314-4_36(354-364)Online publication date: 12-Aug-2022
https://doi.org/10.1007/978-3-031-14314-4_36
Yoo STanaka J(2022)Increasing Motivation of Walking Exercise Using 3D Personalized Avatar in Augmented RealityHuman Interface and the Management of Information: Applications in Complex Technological Environments10.1007/978-3-031-06509-5_13(172-191)Online publication date: 16-Jun-2022
https://doi.org/10.1007/978-3-031-06509-5_13

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Mini-Me: An Adaptive Avatar for Mixed Reality Remote Collaboration

Holoportation: Virtual 3D Teleportation in Real-time

The effect of avatar realism in immersive social virtual realities