research-article

Extended Mid-air Ultrasound Haptics for Virtual Reality

Authors:

Jess Hartcher-O'Brien,

Albrecht Schmidt,

Tonja-Katrin MachullaAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 6, Issue ISS

Article No.: 578, Pages 500 - 524

https://doi.org/10.1145/3567731

Published: 14 November 2022 Publication History

Abstract

Mid-air haptics allow bare-hand tactile stimulation; however, it has a constrained workspace, making it unsuitable for room-scale haptics. We present a novel approach to rendering mid-air haptic sensations in a large rendering volume by turning a static array into a dynamic array following the user's hand. We used a 6DOF robot to drive a haptic ultrasound array over a large 3D space. Our system enables rendering room-scale mid-air experiences while preserving bare-hand interaction, thus, providing tangibility for virtual environments. To evaluate our approach, we performed three evaluations. First, we performed a technical system evaluation, showcasing the feasibility of such a system. Next, we conducted three psychophysical experiments, showing that the motion does not affect the user's perception with high likelihood. Lastly, we explored seven use cases that showcase our system's potential using a user study. We discuss challenges and opportunities in how large-scale mid-air haptics can contribute toward room-scale haptic feedback. Thus, with our system, we contribute to general haptic mid-air feedback on a large scale.

Supplementary Material

Teaser (iss22main-id5096-p-teaser.mp4)

Teaser Video

Download
143.85 MB

References

[1]

Ahmad Abiri, Yen-Yi Juo, Anna Tao, Syed J Askari, Jake Pensa, James W Bisley, Erik P Dutson, and Warren S Grundfest. 2019. Artificial palpation in robotic surgery using haptic feedback. Surgical Endoscopy 33, 4 ( 2019 ), 1252-1259. https://doi.org/10.1007/s00464-018-6405-8

[2]

Nathaniel Agharese, Tyler Cloyd, Laura H Blumenschein, Michael Raitor, Elliot W Hawkes, Heather Culbertson, and Allison M Okamura. 2018. HapWRAP: Soft growing wearable haptic device. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, New York, NY, USA, 5466-5472. https://doi.org/10.1109/ICRA. 2018.8460891

[3]

Kentaro Ariga, Masahiro Fujiwara, Yasutoshi Makino, and Hiroyuki Shinoda. 2021. Workspace Evaluation of LongDistance Midair Haptic Display Using Curved Reflector. In 2021 IEEE World Haptics Conference (WHC). IEEE, New York, NY, USA, 85-90. https://doi.org/10.1109/WHC49131. 2021.9517193

[4]

Jonas Auda, Nils Verheyen, Sven Mayer, and Stefan Schneegass. 2021. Flyables: Haptic Input Devices for Virtual Reality Using Quadcopters. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology (Osaka, Japan) (VRST '21). Association for Computing Machinery, New York, NY, USA, Article 40, 11 pages. https: //doi.org/10.1145/3489849.3489855

Digital Library

[5]

Héctor Barreiro, Stephen Sinclair, and Miguel A Otaduy. 2019. Ultrasound rendering of tactile interaction with fluids. In 2019 IEEE World Haptics Conference (WHC '19). IEEE, New York, NY, USA, 521-526. https://doi.org/10.1109/WHC. 2019.8816137

[6]

Daniel Brice, Thomas McRoberts, and Karen Raferty. 2019. A proof of concept integrated multi-systems approach for large scale tactile feedback in VR. In International Conference on Augmented Reality, Virtual Reality and Computer Graphics. Springer International Publishing, Cham, 120-137. https://doi.org/10.1007/978-3-030-25965-5_10

Digital Library

[7]

Tom Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, and Sriram Subramanian. 2013. UltraHaptics: MultiPoint Mid-Air Haptic Feedback for Touch Surfaces. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST '13). Association for Computing Machinery, New York, NY, USA, 505-514. https://doi.org/10.1145/2501988.2502018

Digital Library

[8]

Xiaojun Chen and Junlei Hu. 2018. A review of haptic simulator for oral and maxillofacial surgery based on virtual reality. Expert Review of Medical Devices 15, 6 ( 2018 ), 435-444. https://doi.org/10.1080/17434440. 2018.1484727

[9]

Inrak Choi, Eyal Ofek, Hrvoje Benko, Mike Sinclair, and Christian Holz. 2018. CLAW: A Multifunctional Handheld Haptic Controller for Grasping, Touching, and Triggering in Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1-13. https://doi.org/10.1145/3173574.3174228

Digital Library

[10]

Giulia Corniani and Hannes P. Saal. 2020. Tactile innervation densities across the whole body. Journal of Neurophysiology 124, 4 ( 2020 ), 1229-1240. https://doi.org/10.1152/jn.00313.2020

[11]

Richard C. Dorf and Robert H. Bishop. 2011. Modern control systems. Pearson, London, England, UK.

[12]

Cathy Fang, Yang Zhang, Matthew Dworman, and Chris Harrison. 2020. Wireality: Enabling Complex Tangible Geometries in Virtual Reality with Worn Multi-String Haptics. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) ( CHI '20). Association for Computing Machinery, New York, NY, USA, 1-10. https://doi.org/10.1145/3313831.3376470

Digital Library

[13]

Euan Freeman, Ross Anderson, Julie Williamson, Graham Wilson, and Stephen A. Brewster. 2017. Textured Surfaces for Ultrasound Haptic Displays. In Proceedings of the 19th ACM International Conference on Multimodal Interaction (Glasgow, UK) ( ICMI '17). Association for Computing Machinery, New York, NY, USA, 491-492. https://doi.org/10. 1145/3136755.3143020 and Games (HAVE). IEEE, New York, NY, USA, 1-6. https://doi.org/10.1109/HAVE. 2019.8921211

[14]

Meta. 2021. Oculus Quest 2. https://www.oculus.com/quest-2/ Accessed: 2022-08-16.

[15]

Kouta Minamizawa, Souichiro Fukamachi, Hiroyuki Kajimoto, Naoki Kawakami, and Susumu Tachi. 2007. Gravity Grabber: Wearable Haptic Display to Present Virtual Mass Sensation. In ACM SIGGRAPH 2007 Emerging Technologies. Association for Computing Machinery, New York, NY, USA, 8-es. https://doi.org/10.1145/1278280.1278289

Digital Library

[16]

Marianna Obrist, Sriram Subramanian, Elia Gatti, Benjamin Long, and Thomas Carter. 2015. Emotions Mediated Through Mid-Air Haptics. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 2053-2062. https: //doi.org/10.1145/2702123.2702361

Digital Library

[17]

Ryan M Peters, Monica D McKeown, Mark G Carpenter, and J Timothy Inglis. 2016. Losing touch: age-related changes in plantar skin sensitivity, lower limb cutaneous reflex strength, and postural stability in older adults. Journal of Neurophysiology 116, 4 ( 2016 ), 1848-1858. https://doi.org/10.1152/jn.00339.2016

[18]

Ismo Rakkolainen, Euan Freeman, Antti Sand, Roope Raisamo, and Stephen Brewster. 2021. A Survey of Mid-Air Ultrasound Haptics and Its Applications. IEEE Transactions on Haptics 14, 1 ( 2021 ), 2-19. https://doi.org/10.1109/TOH. 2020.3018754

[19]

Holger T. Regenbrecht, Thomas W. Schubert, and Frank Friedmann. 1998. Measuring the Sense of Presence and its Relations to Fear of Heights in Virtual Environments. International Journal of Human-Computer Interaction 10, 3 ( 1998 ), 233-249. https://doi.org/10.1207/s15327590ijhc1003_2

[20]

Martin Schrepp, Andreas Hinderks, and Jörg Thomaschewski. 2017. Construction of a Benchmark for the User Experience Questionnaire (UEQ). International Journal of Interactive Multimedia and Artificial Intelligence 4, 4 ( 2017 ), 40-44. https://doi.org/10.9781/ijimai. 2017.445

[21]

Heiko H Schütt, Stefan Harmeling, Jakob H Macke, and Felix A Wichmann. 2016. Painfree and accurate Bayesian estimation of psychometric functions for (potentially) overdispersed data. Vision research 122 ( 2016 ), 105-123. https: //doi.org/10.1016/j.visres. 2016. 02.002

[22]

Mike Sinclair, Eyal Ofek, Mar Gonzalez-Franco, and Christian Holz. 2019. CapstanCrunch: A Haptic VR Controller with User-Supplied Force Feedback. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) ( UIST '19). Association for Computing Machinery, New York, NY, USA, 815-829. https://doi.org/10.1145/3332165.3347891

Digital Library

[23]

Shun Suzuki, Masahiro Fujiwara, Yasutoshi Makino, and Hiroyuki Shinoda. 2019. Midair Ultrasound Haptic Display with Large Workspace. In Haptic Interaction. Springer Singapore, Singapore, 3-5. https://doi.org/10.1007/ 978-981-13-3194-7_1

[24]

Natalya Tatarchuk. 2006. Artist-Directable Real-Time Rain Rendering in City Environments. In ACM SIGGRAPH 2006 Courses (Boston, Massachusetts) ( SIGGRAPH '06). Association for Computing Machinery, New York, NY, USA, 23-64. https://doi.org/10.1145/1185657.1185828

Digital Library

[25]

UltraLeap. 2022. Ultrahaptics Developer Information: Interaction zones and form factors. https://developer.ultrahaptics. com/knowledgebase/interaction-zones-form-factors/ Accessed: 2022-08-16.

[26]

Valve. 2019. Valve Index. https://store.steampowered.com/valveindex Accessed: 2022-08-16.

[27]

Steeven Villa and Sven Mayer. 2022. Cobity: A Plug-And-Play Toolbox to Deliver Haptics in Virtual Reality. In Proceedings of Mensch Und Computer 2022 (Darmstadt, Germany) (MuC '22). Association for Computing Machinery, New York, NY, USA, 78-84. https://doi.org/10.1145/3543758.3543775

Digital Library

[28]

Chyanna Wee, Kian Meng Yap, and Woan Ning Lim. 2021. Haptic Interfaces for Virtual Reality: Challenges and Research Directions. IEEE Access 9 ( 2021 ), 112145-112162. https://doi.org/10.1109/ACCESS. 2021.3103598

[29]

Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) ( CHI '11). Association for Computing Machinery, New York, NY, USA, 143-146. https://doi.org/10.1145/1978942.1978963

Digital Library

[30]

Andre Zenner and Antonio Krüger. 2017. Shifty: A weight-shifting dynamic passive haptic proxy to enhance object perception in virtual reality. IEEE transactions on visualization and computer graphics 23, 4 ( 2017 ), 1285-1294. https: //doi.org/10.1109/TVCG. 2017.2656978

Digital Library

[31]

Yang Zhang, Wolf Kienzle, Yanjun Ma, Shiu S. Ng, Hrvoje Benko, and Chris Harrison. 2019. ActiTouch: Robust Touch Detection for On-Skin AR/VR Interfaces. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) ( UIST '19). Association for Computing Machinery, New York, NY, USA, 1151-1159. https://doi.org/10.1145/3332165.3347869

Digital Library

Cited By

Villa SWeiss YHirsch NWiethoff A(2024)An Examination of Ultrasound Mid-air Haptics for Enhanced Material and Temperature Perception in Virtual EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36764888:MHCI(1-21)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676488
Dietz DVilla SZiarko MBonfert MMüller FButz AKostakos VKay JHoang T(2024)Embracer: A Wearable Encountered-Type Haptic Controller for 3 DoF Input and FeedbackProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676626(140-143)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676626
Lim CSeifi HPark G(2024)An Interactive Tool for Simulating Mid-Air Ultrasound Tactons on the SkinExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650981(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650981
Show More Cited By

Index Terms

Extended Mid-air Ultrasound Haptics for Virtual Reality
1. Computer systems organization
  1. Embedded and cyber-physical systems
    1. Robotics
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Virtual reality

Recommendations

Mid-air Contactless Haptics to augment VR experiences
Web3D '23: Proceedings of the 28th International ACM Conference on 3D Web Technology

We present four technologies to deliver contactless haptic stimuli for enriching Virtual Reality (VR) experiences. The technologies are electrostatic piloerection, focused light-induced heat, electric plasma, and ultrasound; the user does not require ...
FurAir: Non-contact Presentation of Soft Fur Texture by Psuedo-haptics and Mid-air Ultrasound Haptic Feedback
SA '23: SIGGRAPH Asia 2023 Emerging Technologies

We propose a non-contact method to present the tactile sensation of soft fur texture using ultrasound haptic feedback and pseudo-haptics. By responsively adjusting haptic and visual feedback according to user interaction, our approach effectively ...
Haptics in Augmented Reality
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2

An augmented reality system merges synthetic sensory information into a user's perception of a three-dimensional environment. An important performance goal for an augmented reality system is that the user perceives a single seamless environment. In most ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 6, Issue ISS

December 2022

746 pages

EISSN:2573-0142

DOI:10.1145/3554337

Editor:
Jeffrey Nichols
Apple, USA

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 November 2022

Published in PACMHCI Volume 6, Issue ISS

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
540
Total Downloads

Downloads (Last 12 months)167
Downloads (Last 6 weeks)12

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Villa SWeiss YHirsch NWiethoff A(2024)An Examination of Ultrasound Mid-air Haptics for Enhanced Material and Temperature Perception in Virtual EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36764888:MHCI(1-21)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676488
Dietz DVilla SZiarko MBonfert MMüller FButz AKostakos VKay JHoang T(2024)Embracer: A Wearable Encountered-Type Haptic Controller for 3 DoF Input and FeedbackProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676626(140-143)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676626
Lim CSeifi HPark G(2024)An Interactive Tool for Simulating Mid-Air Ultrasound Tactons on the SkinExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650981(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650981
Song KTung SKim APaulos E(2024)Füpop: "Real Food" Flavor Delivery via Focused UltrasoundProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642709(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642709
Katins CWoźniak PChen ATumay ILe LUschold JKosch T(2024)Assessing User Apprehensions About Mixed Reality Artifacts and Applications: The Mixed Reality Concerns (MRC) QuestionnaireProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642631(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642631
John KLi YSeifi H(2024)AdapTics: A Toolkit for Creative Design and Integration of Real-Time Adaptive Mid-Air Ultrasound TactonsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642090(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642090
Morisaki TKamigaki TFujiwara MMakino YShinoda H(2024)UltLever: Ultrasound-Driven Passive Haptic Actuator Based on Amplifying Radiation Force Using a Simple Lever MechanismIEEE Transactions on Haptics10.1109/TOH.2024.336376417:3(471-482)Online publication date: Jul-2024
https://doi.org/10.1109/TOH.2024.3363764
Restrepo GPrakash EDashti SCastillo S. AGómez JOviedo LFloyd JAycardi JTrejos JGonzález JSierra MNavarro-Newball A(2024)Extended Realities for Sensorially Diverse ChildrenIEEE Computer Graphics and Applications10.1109/MCG.2024.341969944:4(26-39)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/MCG.2024.3419699
Matsubayashi AMakino YShinoda H(2024)Latency Compensation in Ultrasound Tactile Presentation by Linear Prediction of Hand PostureHaptics: Understanding Touch; Technology and Systems; Applications and Interaction10.1007/978-3-031-70058-3_30(365-377)Online publication date: 3-Nov-2024
https://doi.org/10.1007/978-3-031-70058-3_30
Dalsgaard TSchneider O(undefined)A Unified Model for Haptic ExperienceACM Transactions on Computer-Human Interaction10.1145/3711842
https://dl.acm.org/doi/10.1145/3711842

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents