skip to main content
10.1145/3173574.3174020acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation

Published: 21 April 2018 Publication History

Abstract

We present a mobile system that enhances mixed reality experiences and games with force feedback by means of electrical muscle stimulation (EMS). The benefit of our approach is that it adds physical forces while keeping the users' hands free to interact unencumbered-not only with virtual objects, but also with physical objects, such as props and appliances. We demonstrate how this supports three classes of applications along the mixed-reality continuum: (1) entirely virtual objects, such as furniture with EMS friction when pushed or an EMS-based catapult game. (2) Virtual objects augmented via passive props with EMS-constraints, such as a light control panel made tangible by means of a physical cup or a balance-the-marble game with an actuated tray. (3) Augmented appliances with virtual behaviors, such as a physical thermostat dial with EMS-detents or an escape-room that repurposes lamps as levers with detents. We present a user-study in which participants rated the EMS-feedback as significantly more realistic than a no-EMS baseline.

Supplementary Material

suppl.mov (pn3702-file3.mp4)
Supplemental video
suppl.mov (pn3702-file5.mp4)
Supplemental video

References

[1]
Hideyuki Ando, Takeshi Miki, Masahiko Inami, and Taro Maeda. 2002. The nail-mounted tactile display for the behavior modeling. In Proceedings of ACM SIGGRAPH 2002 conference abstracts and applications, 264--264.
[2]
Hideyuki Ando, Eisuke Kusachi, and Junji Watanabe. 2007. Nail-mounted tactile display for boundary/texture augmentation. In Proceedings of the international conference on Advances in computer entertainment technology, 292--293.
[3]
Mohamed Benali-Khoudja, Moustapha Hafez, JeanMarc Alexandre, and Abderrahmane Kheddar. 2004. Tactile interfaces: a state-of-the-art survey. In Int. Symposium on Robotics, vol. 31, pp. 23--26.
[4]
Matteo Bianchi, Edoardo Battaglia, Mattia Poggiani, Simone Ciotti, and Antonio Bicchi. 2016. A Wearable Fabric-based display for haptic multi-cue delivery. In Proceedings of Haptics Symposium (HAPTICS 2016), 277--283.
[5]
James Biggs and Mandayam A. Srinivasan. 2002. Tangential Versus Normal Displacements of Skin: Relative Effectiveness for Producing Tactile Sensations. In Proceedings of the 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS 2002), 121--128.
[6]
Christoph W. Borst and Richard A Volz. 2005. Evaluation of a Haptic Mixed Reality System for Interactions with a Virtual Control Panel. Presence, 14(6), 677--696.
[7]
Grigore C. Burdea. 2000. Haptic Issues in Virtual Environments. In Proceedings of Computer Graphics International (GCI '00).
[8]
Tom Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, and Sriram Subramanian. 2013. Ultrahaptics: Multi-Point Mid-Air Haptic Feedback for Touch Surfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST '13), 505--514.
[9]
Daniel K. Y. Chen, Iain A. Anderson, Cameron G. Walker and Thor F. Besier. 2016. Lower Extremity Lateral Skin Stretch Perception for Haptic Feedback. Haptics IEEE Transactions, 9, 62--68, 2016.
[10]
Ulrich Eck and Christian Sandor. 2013. HARP: A framework for visuo-haptic augmented reality. In IEEE Virtual Reality.
[11]
Heather Culbertson, Juan José López Delgado, and Katherine J. Kuchenbecker. 2014. One hundred datadriven haptic texture models and open-source methods for rendering on 3D objects. In Proceedings of Haptics Symposium (HAPTICS 2014), IEEE, 319--325.
[12]
Ayaka Ebisu, Satoshi Hashizume, Kenta Suzuki, Akira Ishii, Mose Sakashita, and Yoichi Ochiai. 2016. Stimulated percussions: techniques for controlling human as percussive musical instrument by using electrical muscle stimulation. In SIGGRAPH ASIA 2016 Posters (SA '16), Article 37.
[13]
Arthur Elsenaar and Remko Scha, Electric Body Manipulation as Performance Art: A Historical Perspective, in Leonardo Music Journal, December 2002, Vol. 12, Pages: 17--28.
[14]
Escape Room, https://en.wikipedia.org/wiki/Escape_room, last accessed on 3/10/2017.
[15]
Farzam Farbiz, Zhou Hao Yu, Corey Manders, and Waqas Ahmad. 2007. An electrical muscle stimulation haptic feedback for mixed reality tennis game. In Proceedings of ACM SIGGRAPH 2007 posters, p. 140.
[16]
Steven Feiner. 2002. Augmented reality: A new way of seeing. Scientific American, 286(4), 2002, 34--41.
[17]
Xiaochi Gu, Yifei Zhang, Weize Sun, Yuanzhe Bian, Dao Zhou, and Per Ola Kristensson. 2016. Dexmo: An Inexpensive and Lightweight Mechanical Exoskeleton for Motion Capture and Force Feedback in VR. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 1991--1995.
[18]
Blake Hannaford and Allison M. Okamura, Haptics (chapter), Handbook of Robotics, 2008, Oussama Khatib, Bruno Siciliano(eds.), Springer, pp 719--739.
[19]
Steven J. Henderson and Steven Feiner. 2008. Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality. In Proceedings of the 2008 ACM symposium on Virtual reality software and technology (VRST '08), 211--218.
[20]
Steven J. Henderson and Steven Feiner. 2009. Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality (ISMAR '09), 135--144.
[21]
Ken Hinckley, Randy Pausch, J.C. Goble and N.F. Kassell. 1994. Passive Real-World Interface Props for Neurosurgical Visualization, in Proceedings of CHI '94, 452--458.
[22]
Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1957--1967.
[23]
John M. Hollerbach, and Stephen C. Jacobsen. 1995. Haptic interfaces for teleoperation and virtual environments. In Proceedings of the First Workshop on Simulation and Interaction in Virtual Environments.
[24]
HoloLens, https://www.microsoft.com/microsofthololens/en-us/developers
[25]
John N. Howell, Robert L. Williams, Robert R. Conatser, Janet M. Burns, and David C. Eland. 2005. The Virtual Haptic Back (VHB): a virtual reality simulation of the human back for palpatory diagnostic training. SAE Technical Paper, No. 2005-01--2679.
[26]
Wolfgang Hürst, Nina Rosa, and Jean-Paul van Bommel. 2016. Vibrotactile experiences for augmented reality. In Proceedings of the 2016 ACM on Multimedia Conference, 744--745.
[27]
Brent Edward Insko. 2001. Passive Haptics Significantly Enhances Virtual Environments. Ph.D. Dissertation. The University of North Carolina at Chapel Hill. Advisor(s) Frederick P. Brooks, Jr. AAI3007820.
[28]
Hiroshi Ishii and Brygg Ullmer. 1997. Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (CHI '97), 234241.
[29]
Seungzoo Jeong, Naoki Hashimoto, and Sato Makoto. 2004. A novel interaction system with force feedback between real-and virtual human: an entertainment system: virtual catch ball. In Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology, 61--66.
[30]
Hwan Kim, Minhwan Kim, and Woohun Lee. 2016. HapThimble: A Wearable Haptic Device towards Usable Virtual Touch Screen. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 3694--3705.
[31]
Benjamin Knoerlein, Gábor Székely, and Matthias Harders. 2007. Visuo-haptic collaborative augmented reality ping-pong. In Proceedings of the international conference on Advances in computer entertainment technology (ACE '07), 91--94.
[32]
Ernst Kruijff, Dieter Schmalstieg, and Steffi Beckhaus. 2006. Using neuromuscular electrical stimulation for pseudo-haptic feedback. In Proceedings of the ACM symposium on Virtual reality software and technology (VRST '06), 316--319.
[33]
Joshua Lifton and Joseph A. Paradiso, Dual Reality: Merging the Real and Virtual, 2017, International Conference on Facets of Virtual Environments FaVE 2009: pp. 12--28.
[34]
Robert W., Lindeman, John L. Sibert and James K. Hahn. 1999. Hand-Held Windows: Towards Effective 2D Interaction in Immersive Virtual Environments. In Proceedings of the IEEE Virtual Reality (VR '99), 205.
[35]
Robert W. Lindeman, Yasuyuki Yanagida, Haruo Noma, and Kenichi Hosaka. 2006. Wearable vibrotactile systems for virtual contact and information display. Virtual Reality 9, no. 2--3 (2006): 203--213.
[36]
Pedro Lopes and Patrick Baudisch. 2013. Musclepropelled force feedback: bringing force feedback to mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 2577--2580.
[37]
Pedro Lopes, Alexandra Ion, and Patrick Baudisch. 2015. Impacto: Simulating Physical Impact by Combining Tactile Stimulation with Electrical Muscle Stimulation. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15), 11--19.
[38]
Pedro Lopes, Alexandra Ion, Willi Mueller, Daniel Hoffmann, Patrik Jonell, and Patrick Baudisch. 2015. Proprioceptive Interaction. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 939--948.
[39]
Pedro Lopes, Patrik Jonell, and Patrick Baudisch. 2015. Affordance++: Allowing Objects to Communicate Dynamic Use. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 2515--2524.
[40]
Pedro Lopes, Doğa Yüksel, François Guimbretière and Patrick Baudisch,.2016. Muscle plotter: An Interactive System based on Electrical Muscle Stimulation that Produces Spatial Output. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), 207--217.
[41]
Pedro Lopes, Sijing Young, Lung-pan Cheng, Sebastian Marwecki, and Patrick Baudisch, Providing Haptics to Walls and Other Heavy Objects in Virtual Reality by Means of Electrical Muscle Stimulation. In Proceedings of the 35th Annual ACM Conference on Human Factors in Computing Systems (CHI '17).
[42]
Cornelio Martinez, Patricia Ivette, Silvana De Pirro, Chi Thanh Vi, and Sriram Subramanian. 2017. Agency in mid-air interfaces. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17).
[43]
Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino. 1995. Augmented reality: A class of displays on the reality-virtuality continuum. In Proceedings of SPIE 2351, Telemanipulator and Telepresence Technologies, 282--292.
[44]
Paul Milgram and Fumio Kishino. 1994. A Taxonomyof Mixed Reality Visual Displays. IEICE Transactions on Information and Systems E77-D,1321--1329.
[45]
Jun Murayama, Laroussi Bougrila, YanLin Luo, Katsuhito Akahane, Shoichi Hasegawa, Béat Hirsbrunner, and Makoto Sato. 2004. SPIDAR G&G: a two-handed haptic interface for bimanual VR interaction. In Proceedings of EuroHaptics, vol. 2004, 138--146.
[46]
Kazuki Nagai, Soma Tanoue, Katsuhito Akahane, and Makoto Sato. 2015. Wearable 6-DoF wrist haptic device SPIDAR-W. In Proceedings of SIGGRAPH Asia 2015 Haptic Media And Contents Design, p. 19.
[47]
Geoff Norman. 2010. Likert scales, levels of measurement and the "laws" of statistics. Advances in health sciences education 15, no. 5 (2010): 625--632.
[48]
Ohan Oda and Steven Feiner. 2010. Rolling and shooting: two augmented reality games. CHI EA '10, 30413044.
[49]
Ohan Oda, Carmine Elvezio, Mengu Sukan, Steven Feiner, and Barbara Tversky. 2015. Virtual Replicas for Remote Assistance in Virtual and Augmented Reality. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15), 405415.
[50]
Max Pfeiffer, Tim Dünte, Stefan Schneegass, Florian Alt, and Michael Rohs. 2015. Cruise Control for Pedestrians: Controlling Walking Direction using Electrical Muscle Stimulation. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 2505--2514.
[51]
Max Pfeiffer, Stefan Schneegass, Florian Alt, and Michael Rohs. 2014. Let me grab this: a comparison of EMS and vibration for haptic feedback in free-hand interaction. In Proceedings of the 5th Augmented Human International Conference (AH '14). ACM, New York, NY, USA, Article 48, 8 pages.
[52]
Raf Ramakers, Fraser Anderson, Tovi Grossman, and George Fitzmaurice. 2016. RetroFab: A Design Tool for Retrofitting Physical Interfaces using Actuators, Sensors and 3D Printing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 409--419.
[53]
Rehastim by Hasomed, https://www.hasomed.de/en/products.html
[54]
Joan Sol Roo, Martin Hachet, One Reality: Augmenting How the Physical World is Experienced by combining Multiple Mixed Reality Modalities, to appear, UIST 2017.
[55]
Antti Sand, Ismo Rakkolainen, Poika Isokoski, Jari Kangas, Roope Raisamo, and Karri Palovuori. 2015. Head-mounted display with mid-air tactile feedback. In Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology (VRST '15), 51--58.
[56]
Stefano Scheggi, Gionata Salvietti, and Domenico Prattichizzo. 2010. Shape and weight rendering for haptic augmented reality. In Proceedings of the 19th International Symposium in Robot and Human Interactive Communication, 44--49.
[57]
William R. Sherman, and Alan B. Craig. 2002. Understanding virtual reality: Interface, application, and design. Elsevier, 2002.
[58]
Rajinder Sodhi, Ivan Poupyrev, Matthew Glisson, and Ali Israr. 2013. AIREAL: interactive tactile experiences in free air. ACM Transactoions on Graphics. 32, 4, Article 134 (July 2013), 10 pages.
[59]
Primož Strojnik, Alojz Kralj, and I. Ursic. 1979. Programmed six-channel electrical stimulator for complex stimulation of leg muscles during walking. IEEE Transactions on Biomedical Engineering 2 (1979): 112116.
[60]
Emi Tamaki, Takashi Miyaki, and Jun Rekimoto. 2011. PossessedHand: techniques for controlling human hands using electrical muscles stimuli. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 543--552.
[61]
Dzmitry Tsetserukou, Katsunari Sato, and Susumu Tachi. 2010. ExoInterfaces: novel exosceleton haptic interfaces for virtual reality, augmented sport and rehabilitation. In Proceedings of the 1st Augmented Human International Conference (AH '10), 1--6.
[62]
Vuforia, https://developer.vuforia.com
[63]
Sean White and Steven Feiner. 2009. SiteLens: situated visualization techniques for urban site visits. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09), 1117--1120.
[64]
Yasuyoshi Yokokohji, Ralph L. Hollis, and Takeo Kanade. 1996. What you can see is what you can feeldevelopment of a visual/haptic interface to virtual environment." In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium, 46--53.

Cited By

View all

Index Terms

  1. Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
    April 2018
    8489 pages
    ISBN:9781450356206
    DOI:10.1145/3173574
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 21 April 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. ar
    2. augmented reality
    3. body i/o
    4. electrical muscle stimulation
    5. haptics
    6. hololens
    7. mixed reality
    8. mr
    9. wearable

    Qualifiers

    • Research-article

    Conference

    CHI '18
    Sponsor:

    Acceptance Rates

    CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)223
    • Downloads (Last 6 weeks)30
    Reflects downloads up to 23 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Electrical Muscle Stimulation for Kinesthetic Feedback in AR/VR: A Systematic Literature ReviewMultimodal Technologies and Interaction10.3390/mti80200078:2(7)Online publication date: 25-Jan-2024
    • (2024)Augmenting perceived stickiness of physical objects through tactile feedback after finger lift-offFrontiers in Robotics and AI10.3389/frobt.2024.141546411Online publication date: 18-Sep-2024
    • (2024)Resonant Sticker Buttons: The Effect of Button Size and Feedback Latency on Perceived Button Weight and Vibrotactile Feedback StrengthProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701591(210-227)Online publication date: 1-Dec-2024
    • (2024)Towards Trustworthy MetaShopping: Studying Manipulative Audiovisual Designs in Virtual-Physical Commercial PlatformsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681679(68-77)Online publication date: 28-Oct-2024
    • (2024)PneuExtensio: Designing Pneumatic-based Bodily Extensions to Facilitate Embodiment across Everyday Life ExperiencesCompanion Publication of the 2024 ACM Designing Interactive Systems Conference10.1145/3656156.3665136(19-23)Online publication date: 1-Jul-2024
    • (2024)Can a Smartwatch Move Your Fingers? Compact and Practical Electrical Muscle Stimulation in a SmartwatchProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676373(1-15)Online publication date: 13-Oct-2024
    • (2024)Shared Bodily Fusion: Leveraging Inter-Body Electrical Muscle Stimulation for Social PlayProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660723(2088-2106)Online publication date: 1-Jul-2024
    • (2024)Exploring Human Values in Mixed Reality FuturesProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660712(197-213)Online publication date: 1-Jul-2024
    • (2024)PneuMa: Designing Pneumatic Bodily Extensions for Supporting Movement in Everyday LifeProceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3623509.3633349(1-16)Online publication date: 11-Feb-2024
    • (2024)PneuMa: Designing Pneumatic Bodily Extensions for Supporting Movement in Everyday LifeExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3649125(1-4)Online publication date: 11-May-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media