abstract

"Megereality": Leveraging Physical Affordances for Multi-Device Gestural Interaction in Augmented Reality

Authors:

Shengzhi Wu,

Daragh Byrne,

Molly Wright SteensonAuthors Info & Claims

CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

Pages 1 - 4

https://doi.org/10.1145/3334480.3383170

Published: 25 April 2020 Publication History

Get Access

Abstract

We present a novel gestural interaction strategy for multi-device interactions in augmented reality (AR), in which we leverage existing physical affordances of everyday products and spaces for intuitive interactions in AR. To explore this concept, we designed and prototyped three demo scenarios: pulling virtual sticky notes from a tablet, pulling a 3D model from a computer display, and 'slurping' color from the real-world environment to smart lights with a virtual eyedropper. By merging the boundary of digital and physical, utilizing metaphors in AR and embodying the abstract process, we demonstrate an interaction strategy that harnesses the physical affordances to assist digital interaction in AR with hand gestures.

Supplementary Material

JPG File (int008.jpg)

Video figure captions

Download
173.01 KB

MP4 File (int008pv.mp4)

Preview video

Download
14.44 MB

MP4 File (dein1113vf.mp4)

Supplemental video

Download
89.20 MB

References

[1]

Feuchtner, T., & Müller, J. (2017, May). Extending the body for interaction with reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 5145--5157). ACM.

Digital Library

Google Scholar

[2]

Henderson, S. J., & Feiner, S. (2008, October). Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality. In Proceedings of the 2008 ACM symposium on Virtual reality software and technology(pp. 211--218). ACM.

Digital Library

Google Scholar

[3]

Heun, V., Hobin, J., & Maes, P. (2013, September). Reality editor: programming smarter objects. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication (pp. 307--310). ACM.

Digital Library

Google Scholar

[4]

Huang, Z., Li, W., & Hui, P. (2015, October). Ubii: Towards seamless interaction between digital and physical worlds. In Proceedings of the 23rd ACM international conference on Multimedia (pp. 341--350). ACM.

Digital Library

Google Scholar

[5]

Ishii, H. and Ullmer, B., Tangible bits: towards seamless interfaces between people, bits and atoms. in CHI, 1997.

Digital Library

Google Scholar

[6]

Jamie Zigelbaum, Adam Kumpf, Alejandro Vazquez, Hiroshi Ishii, Slurp: tangibility spatiality and an eyedropper, CHI '08 Extended Abstracts on Human Factors in Computing Systems, April 05--10, 2008, Florence, Italy [doi>10.1145/1358628.1358713]

Digital Library

Google Scholar

[7]

Martin Schubert, Barrett Fox, Summoning and Superpowers: Designing VR Interactions at a Distance, http://blog.leapmotion.com/summoningsuperpowers-designing-vr-interactions-distance/

Google Scholar

[8]

Norman, D. A. (1988). The psychology of everyday things(Vol. 5). New York: Basic books.

Google Scholar

[9]

Pannuto, P., Kempke, B., Chuo, L. X., Blaauw, D., & Dutta, P. (2018). Harmonium: Ultra wideband pulse generation with bandstitched recovery for fast, accurate, and robust indoor localization. ACM Transactions on Sensor Networks (TOSN), 14(2), 11.

Digital Library

Google Scholar

[10]

Terrenghi, L., Kirk, D., Sellen, A., & Izadi, S. (2007, April). Affordances for manipulation of physical versus digital media on interactive surfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 1157--1166). ACM. Figure 4: Deme 3: slurping color to a smart light with pinch gesture. Figure 5: Deluminator from Harry Potter movie.

Digital Library

Google Scholar

Cited By

View all

Zhao LIsenberg TXie FLiang HYu L(2025)SpatialTouch: Exploring Spatial Data Visualizations in Cross-RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345636831:1(897-907)Online publication date: Jan-2025
https://doi.org/10.1109/TVCG.2024.3456368
Schenkluhn MKnierim MKiss FWeinhardt C(2024)Connecting Home: Human-Centric Setup Automation in the Augmented Smart HomeProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642862(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642862
Wang NMaurer F(2024)A Cross Reality Application for a Hydrogen Pipeline Digital Twin2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct64951.2024.00038(141-145)Online publication date: 21-Oct-2024
https://doi.org/10.1109/ISMAR-Adjunct64951.2024.00038
Show More Cited By

Index Terms

"Megereality": Leveraging Physical Affordances for Multi-Device Gestural Interaction in Augmented Reality
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques
      1. Gestural input

Recommendations

Multi-scale gestural interaction for augmented reality
SA '17: SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications

We present a multi-scale gestural interface for augmented reality applications. With virtual objects, gestural interactions such as pointing and grasping can be convenient and intuitive, however they are imprecise, socially awkward, and susceptible to ...
Tangible Meets Gestural: Comparing and Blending Post-WIMP Interaction Paradigms
TEI '15: Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction

More and more objects of our everyday environment are becoming smart and connected, offering us new interaction possibilities. Tangible interaction and gestural interaction are promising communication means with these objects in this post-WIMP ...
A Review of Augmented Reality-Based Human-Computer Interaction Applications of Gesture-Based Interaction
HCI International 2019 – Late Breaking Papers
Abstract
In recent years, augmented reality (AR) is an extremely growing field in information technology, computer science, and computer engineering. Although there are many recent works that use augmented reality for different purposes, most of the ...

Comments

Information & Contributors

Information

Published In

CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

April 2020

4474 pages

ISBN:9781450368193

DOI:10.1145/3334480

General Chairs:
Regina Bernhaupt
Eindhoven University of Technology, Netherlands
,
Florian 'Floyd' Mueller
Monash University, Australia
,
David Verweij
Newcastle University, UK
,
Josh Andres
RMIT, Australia
,
Program Chairs:
Joanna McGrenere
University of British Columbia, Canada
,
Andy Cockburn
University of Canterbury, New Zealand
,
Ignacio Avellino
University of Maryland Baltimore County, USA
,
Alix Goguey
Grenoble Alpes University, France
,
Pernille Bjørn
University of Copenhagen, Denmark
,
Shengdong (Shen) Zhao
National University of Singapore, Singapore
,
Briane Paul Samson
Future University Hakodate, Japan & De La Salle University, Philippines
,
Rafal Kocielnik
University of Washington, USA

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2020

Check for updates

Author Tags

Qualifiers

Abstract

Conference

CHI '20

Sponsor:

SIGCHI

CHI '20: CHI Conference on Human Factors in Computing Systems

April 25 - 30, 2020

HI, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
1,196
Total Downloads

Downloads (Last 12 months)128
Downloads (Last 6 weeks)15

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zhao LIsenberg TXie FLiang HYu L(2025)SpatialTouch: Exploring Spatial Data Visualizations in Cross-RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345636831:1(897-907)Online publication date: Jan-2025
https://doi.org/10.1109/TVCG.2024.3456368
Schenkluhn MKnierim MKiss FWeinhardt C(2024)Connecting Home: Human-Centric Setup Automation in the Augmented Smart HomeProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642862(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642862
Wang NMaurer F(2024)A Cross Reality Application for a Hydrogen Pipeline Digital Twin2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct64951.2024.00038(141-145)Online publication date: 21-Oct-2024
https://doi.org/10.1109/ISMAR-Adjunct64951.2024.00038
Gavgiotaki DNtoa SMargetis GApostolakis KStephanidis C(2023)Gesture-based Interaction for AR Systems: A Short ReviewProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3594806.3594815(284-292)Online publication date: 5-Jul-2023
https://dl.acm.org/doi/10.1145/3594806.3594815
Lee JJin TLee SLee SBae S(2023)Stereoscopic Viewing and Monoscopic Touching: Selecting Distant Objects in VR Through a Mobile DeviceProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606809(1-7)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606809
Luo WYu ZRzayev RSatkowski MGumhold SMcGinity MDachselt R(2023)Pearl: Physical Environment based Augmented Reality Lenses for In-Situ Human Movement AnalysisProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580715(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580715
Aigner DWang NKielmayer DSteiner JHochpöchler JHeinzl CRoth DMaurer FAnthes C(2023)Cardiac Visualisation Along the RV-Continuum - A High-Fidelity Pilot Study2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00145(675-680)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00145
Li TWu SJin YShi HLiu S(2023)X-Space: Interaction design of extending mixed reality space from Web2D visualizationVisual Informatics10.1016/j.visinf.2023.10.0017:4(73-83)Online publication date: Dec-2023
https://doi.org/10.1016/j.visinf.2023.10.001
Byrne DLockton DHu MLuong MRanade AEscarcha KGiesa KHuang YYochum CRobertson GYeung LCruz MDanner CWang EKhurana MChen ZHeyison AFu Y(2022)Spooky TechnologyProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533547(759-775)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533547
Wu SByrne DDu RSteenson M(2022)“Slurp” Revisited: Using ‘system re-presencing’ to look back on, encounter, and design with the history of spatial interactivity and locative mediaProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533464(263-276)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533464
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

Supplementary Material

References

Cited By

Index Terms

Recommendations

Multi-scale gestural interaction for augmented reality

Tangible Meets Gestural: Comparing and Blending Post-WIMP Interaction Paradigms

A Review of Augmented Reality-Based Human-Computer Interaction Applications of Gesture-Based Interaction

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations