research-article

Open access

Interferi: Gesture Sensing using On-Body Acoustic Interferometry

Authors:

Yasha Iravantchi,

Chris HarrisonAuthors Info & Claims

CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Paper No.: 276, Pages 1 - 13

https://doi.org/10.1145/3290605.3300506

Published: 02 May 2019 Publication History

All formats PDF

Abstract

Interferi is an on-body gesture sensing technique using acoustic interferometry. We use ultrasonic transducers resting on the skin to create acoustic interference patterns inside the wearer's body, which interact with anatomical features in complex, yet characteristic ways. We focus on two areas of the body with great expressive power: the hands and face. For each, we built and tested a series of worn sensor configurations, which we used to identify useful transducer arrangements and machine learning fea-tures. We created final prototypes for the hand and face, which our study results show can support eleven- and nine-class gestures sets at 93.4% and 89.0% accuracy, re-spectively. We also evaluated our system in four continu-ous tracking tasks, including smile intensity and weight estimation, which never exceed 9.5% error. We believe these results show great promise and illuminate an inter-esting sensing technique for HCI applications.

Supplementary Material

MP4 File (paper276.mp4)

Supplemental video

Download
393.69 MB

MP4 File (paper276p.mp4)

Preview video

Download
4.49 MB

References

[1]

Karan Ahuja, Rahul Islam, Varun Parashar, Kuntal Dey, Chris Harrison, and Mayank Goel. 2018. EyeSpyVR: Interactive Eye Sensing Using Off-the-Shelf, Smartphone-Based VR Headsets. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 2, Article 57 (July 2018), 10 pages.

Digital Library

[2]

Brian Amento, Will Hill, and Loren Terveen. 2002. The sound of one hand: a wrist-mounted bio-acoustic fingertip gesture interface. In CHI '02 Extended Abstracts on Human Factors in Computing Systems (CHI EA '02). ACM, New York, NY, USA, 724--725.

Digital Library

[3]

Toshiyuki Ando, Yuki Kubo, Buntarou Shizuki, and Shin Takahashi. 2017. CanalSense: Face-Related Movement Recognition System based on Sensing Air Pressure in Ear Canals. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 679--689.

Digital Library

[4]

E. H. Brandt (2001). Acoustic physics: Suspended by Sound. Nature, 413(6855), 474--475.

[5]

Laura A. Brooks and Peter Gerstoft. Ocean acoustic interferometry. The Journal of the Acoustical Society of America 121, no. 6 (2007): 3377--3385.

[6]

Tom Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, and Sriram Subramanian. UltraHaptics: multi-point mid-air haptic feedback for touch surfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology, pp. 505--514. ACM, 2013.

Digital Library

[7]

Ciprian Adrian Corneanu, Marc Oliu Simón, Jeffrey F. Cohn, and Sergio Escalera Guerrero. Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications. IEEE transactions on pattern analysis and machine intelligence 38, no. 8 (2016): 1548--1568.

[8]

Artem Dementyev and Joseph A. Paradiso. 2014. WristFlex: lowpower gesture input with wrist-worn pressure sensors. In Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST '14). ACM, New York, NY, USA, 161--166.

Digital Library

[9]

Travis Deyle, Szabolcs Palinko, Erika Shehan Poole, and Thad Starner. 2007. Hambone: A Bio-Acoustic Gesture Interface. In Proceedings of the 2007 11th IEEE International Symposium on Wearable Computers (ISWC '07). IEEE Computer Society, Washington, DC, USA, 1--8.

Digital Library

[10]

EMCO SIP100 DC-DC Converter, http://www.eie-ic.com/Images/EMCO/EMCO/sipseries.pdf

[11]

Jun Gong, Xing-Dong Yang, and Pourang Irani. 2016. WristWhirl: One-handed Continuous Smartwatch Input using Wrist Gestures. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 861--872.

Digital Library

[12]

Anna Gruebler, and Kenji Suzuki. Measurement of distal EMG signals using a wearable device for reading facial expressions. In Engineering in Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE, pp. 4594--4597. IEEE, 2010.

[13]

Anna Gruebler, and Kenji Suzuki. Design of a wearable device for reading positive expressions from facial emg signals. IEEE Transactions on affective computing 5, no. 3 (2014): 227--237.

[14]

Chris Harrison, Desney Tan, and Dan Morris. Skinput: appropriating the body as an input surface. In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 453--462. ACM, 2010.

Digital Library

[15]

Claude F. Harbarger, Paul M. Weinberger, Jack C. Borders, and Charles A. Hughes. "Prenatal ultrasound exposure and association Figure 21. Interferi could be integrated into future smartwatch bands and AR/VR headset liners, as seen in these mock-ups. with postnatal hearing outcomes." Journal of Otolaryngology-Head & Neck Surgery 42, no. 1 (2013): 3.

[16]

Reli Hershkovitz, Eyal Sheiner, and Moshe Mazor. "Ultrasound in obstetrics: a review of safety." European Journal of Obstetrics & Gynecology and Reproductive Biology101, no. 1 (2002): 15--18.

[17]

Takeshi Ide, James Friend, Kentaro Nakamura, and Sadayuki Ueha. A non-contact linear bearing and actuator via ultrasonic levitation. Sensors and Actuators A: Physical 135, no. 2 (2007): 740--747.

[18]

N. Inoue, et al. A new ultrasonic interferometer for velocity measurement in liquids using optical diffraction 1986 J. Phys. D: Appl. Phys. 1

[19]

Pyeong-Gook Jung, Gukchan Lim, Seonghyok Kim, and Kyoungchul Kong. A wearable gesture recognition device for detecting muscular activities based on air-pressure sensors. IEEE Transactions on Industrial Informatics 11, no. 2 (2015): 485--494.

[20]

Frederic Kerber, Michael Puhl, and Antonio Krüger. 2017. User-independent real-time hand gesture recognition based on surface electromyography. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, Article 36, 7 pages.

Digital Library

[21]

David Kim, Otmar Hilliges, Shahram Izadi, Alex D. Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12). ACM, New York, NY, USA, 167176.

Digital Library

[22]

Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: HighFidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 321--333.

Digital Library

[23]

Hao Li, Laura Trutoiu, Kyle Olszewski, Lingyu Wei, Tristan Trutna, Pei-Lun Hsieh, Aaron Nicholls, and Chongyang Ma. 2015. Facial performance sensing head-mounted display. ACM Trans. Graph. 34, 4, Article 47 (July 2015), 9 pages.

Digital Library

[24]

Jhe-Wei Lin, Chiuan Wang, Yi Yao Huang, Kuan-Ting Chou, HsuanYu Chen, Wei-Luan Tseng, and Mike Y. Chen. 2015. BackHand: Sensing Hand Gestures via Back of the Hand. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 557--564.

Digital Library

[25]

Katsutoshi Masai, Yuta Sugiura, Masa Ogata, Kai Kunze, Masahiko Inami, and Maki Sugimoto. 2016. Facial Expression Recognition in Daily Life by Embedded Photo Reflective Sensors on Smart Eyewear. In Proceedings of the 21st International Conference on Intelligent User Interfaces (IUI '16). ACM, New York, NY, USA, 317--326.

Digital Library

[26]

Katsutoshi Masai, Kai Kunze, Yuta Sugiura, Masa Ogata, Masahiko Inami, and Maki Sugimoto. 2017. Evaluation of Facial Expression Recognition by a Smart Eyewear for Facial Direction Changes, Repeatability, and Positional Drift. ACM Trans. Interact. Intell. Syst. 7, 4, Article 15 (December 2017), 23 pages.

Digital Library

[27]

Denys J. C. Matthies, Bernhard A. Strecker, and Bodo Urban. 2017. EarFieldSensing: A Novel In-Ear Electric Field Sensing to Enrich Wearable Gesture Input through Facial Expressions. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1911--1922.

Digital Library

[28]

Jess McIntosh, Asier Marzo, Mike Fraser, and Carol Phillips. 2017. EchoFlex: Hand Gesture Recognition using Ultrasound Imaging. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1923--1934.

Digital Library

[29]

Adiyan Mujibiya, Xiang Cao, Desney S. Tan, Dan Morris, Shwetak N. Patel, and Jun Rekimoto. 2013. The sound of touch: on-body touch and gesture sensing based on transdermal ultrasound propagation. In Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces (ITS '13). ACM, New York, NY, USA, 189--198.

Digital Library

[30]

Masa Ogata, and Michita Imai. SkinWatch: skin gesture interaction for smart watch. In Proceedings of the 6th Augmented Human International Conference, pp. 21--24. ACM, 2015.

Digital Library

[31]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel et al. Scikit-learn: Machine learning in Python. Journal of machine learning research 12, no. Oct (2011): 2825--2830.

Digital Library

[32]

John Kangchun Perng, Brian Fisher, Seth Hollar, and Kristofer SJ Pister. Acceleration sensing glove (ASG). In Wearable Computers, 1999. Digest of Papers. The Third International Symposium on, pp. 178--180. IEEE, 1999.

Digital Library

[33]

PUI Audio 40 kHz Ultrasonic Transducer, http://www.puiaudio.com/pdf/UTR-1440K-TT-R.pdf

[34]

Jun Rekimoto. Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In Wearable Computers, 2001. Proceedings. Fifth International Symposium on, pp. 21--27. IEEE, 2001.

Digital Library

[35]

T. Scott Saponas, Desney S. Tan, Dan Morris, and Ravin Balakrishnan. 2008. Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 515--524.

Digital Library

[36]

T. Scott Saponas, Desney S. Tan, Dan Morris, Ravin Balakrishnan, Jim Turner, and James A. Landay. 2009. Enabling always-available input with muscle-computer interfaces. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST '09). ACM, New York, NY, USA, 167--176.

Digital Library

[37]

Jocelyn Scheirer, Raul Fernandez, and Rosalind W. Picard. 1999. Expression glasses: a wearable device for facial expression recognition. In CHI '99 Extended Abstracts on Human Factors in Computing Systems (CHI EA '99). ACM, New York, NY, USA, 262--263.

Digital Library

[38]

Katsuhiro Suzuki, Fumihiko Nakamura, Jiu Otsuka, Katsutoshi Masai, Yuta Itoh, Yuta Sugiura, and Maki Sugimoto. 2016. Facial Expression Mapping inside Head Mounted Display by Embedded Optical Sensors. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16 Adjunct). ACM, New York, NY, USA, 91--92.

Digital Library

[39]

Teensy 3.6 Microcontroller, PJRC, https://www.pjrc.com/store/teensy36.html

[40]

Thalmic Lab, Inc. http://www.thalmic.com/myo/

[41]

Hsin-Ruey Tsai, Cheng-Yuan Wu, Lee-Ting Huang, and Yi-Ping Hung. 2016. ThumbRing: private interactions using one-handed thumb motion input on finger segments. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI '16). ACM, New York, NY, USA, 791--798.

Digital Library

[42]

David Way and Joseph Paradiso. 2014. A Usability User Study Concerning Free-Hand Microgesture and Wrist-Worn Sensors. In Proceedings of the 2014 11th International Conference on Wearable and Implantable Body Sensor Networks (BSN '14). IEEE Computer Society, Washington, DC, USA, 138--142.

Digital Library

[43]

Eric Whitmire, Mohit Jain, Divye Jain, Greg Nelson, Ravi Karkar, Shwetak Patel, and Mayank Goel. 2017. DigiTouch: Reconfigurable Thumb-to-Finger Input and Text Entry on Head-mounted Displays. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 113 (September 2017), 21 pages.

Digital Library

[44]

Chao Xu, Parth H. Pathak, and Prasant Mohapatra. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications, pp. 9--14. ACM, 2015.

Digital Library

[45]

Yang Zhang and Chris Harrison. 2015. Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 167173.

Digital Library

[46]

Yang Zhang, Robert Xiao, and Chris Harrison. 2016. Advancing Hand Gesture Recognition with High Resolution Electrical Impedance Tomography. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 843--850.

Digital Library

[47]

Cheng Zhang, AbdelKareem Bedri, Gabriel Reyes, Bailey Bercik, Omer T. Inan, Thad E. Starner, and Gregory D. Abowd. 2016. TapSkin: Recognizing On-Skin Input for Smartwatches. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (ISS '16). ACM, New York, NY, USA, 13--22.

Digital Library

[48]

Cheng Zhang, Qiuyue Xue, Anandghan Waghmare, Ruichen Meng, Sumeet Jain, Yizeng Han, Xinyu Li, Kenneth Cunefare, Thomas Ploetz, Thad Starner, Omer Inan, and Gregory D. Abowd. 2018. FingerPing: Recognizing Fine-grained Hand Poses using Active Acoustic On-body Sensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Paper 437, 10 pages.

Digital Library

Cited By

Iravantchi YSample A(2024)T4Train: Rapid Prototyping of ML-Driven Interactive ApplicationsExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636269(1-4)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3636269
Song KTung SKim APaulos E(2024)Füpop: "Real Food" Flavor Delivery via Focused UltrasoundProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642709(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642709
Kyu AMao HZhu JGoel MAhuja K(2024)EITPose: Wearable and Practical Electrical Impedance Tomography for Continuous Hand Pose EstimationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642663(1-10)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642663
Show More Cited By

Index Terms

Interferi: Gesture Sensing using On-Body Acoustic Interferometry
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques
      1. Gestural input

Recommendations

BeamBand: Hand Gesture Sensing with Ultrasonic Beamforming
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

BeamBand is a wrist-worn system that uses ultrasonic beamforming for hand gesture sensing. Using an array of small transducers, arranged on the wrist, we can ensem-ble acoustic wavefronts to project acoustic energy at spec-ified angles and focal ...
Hand Gesture Interaction with a Low-Resolution Infrared Image Sensor on an Inner Wrist
AVI '20: Proceedings of the 2020 International Conference on Advanced Visual Interfaces

We propose a hand gesture interaction method using a low-resolution infrared image sensor on an inner wrist. We attach the sensor to the strap of a wrist-worn device, on the palmar side, and apply machine-learning techniques to recognize the gestures ...
Itchy nose: discreet gesture interaction using EOG sensors in smart eyewear
ISWC '17: Proceedings of the 2017 ACM International Symposium on Wearable Computers

We propose a sensing technique for detecting finger movements on the nose, using EOG sensors embedded in the frame of a pair of eyeglasses. Eyeglasses wearers can use their fingers to exert different types of movement on the nose, such as flicking, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

May 2019

9077 pages

ISBN:9781450359702

DOI:10.1145/3290605

General Chairs:
Stephen Brewster
University of Glasgow, Scotland, UK
,
Geraldine Fitzpatrick
TU Wien, Austria
,
Program Chairs:
Anna Cox
University College London, UK
,
Vassilis Kostakos
University of Melbourne, Australia

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article

Conference

CHI '19

Sponsor:

SIGCHI

CHI '19: CHI Conference on Human Factors in Computing Systems

May 4 - 9, 2019

Glasgow, Scotland Uk

Acceptance Rates

CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

58
Total Citations
View Citations
2,505
Total Downloads

Downloads (Last 12 months)508
Downloads (Last 6 weeks)49

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Iravantchi YSample A(2024)T4Train: Rapid Prototyping of ML-Driven Interactive ApplicationsExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636269(1-4)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3636269
Song KTung SKim APaulos E(2024)Füpop: "Real Food" Flavor Delivery via Focused UltrasoundProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642709(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642709
Kyu AMao HZhu JGoel MAhuja K(2024)EITPose: Wearable and Practical Electrical Impedance Tomography for Continuous Hand Pose EstimationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642663(1-10)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642663
Li KZhang RChen SChen BSakashita MGuimbretiere FZhang C(2024)EyeEcho: Continuous and Low-power Facial Expression Tracking on GlassesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642613(1-24)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642613
Yi CWei BZhu JChen CWang YChen ZHuang YJiang F(2024)Mordo2: A Personalization Framework for Silent Command RecognitionIEEE Transactions on Neural Systems and Rehabilitation Engineering10.1109/TNSRE.2023.334206832(133-143)Online publication date: 2024
https://doi.org/10.1109/TNSRE.2023.3342068
Kishi FIto KFujita KItoh Y(2024)Recognizing object localization using acoustic markers with active acoustic sensingQuality and User Experience10.1007/s41233-024-00066-x9:1Online publication date: 11-Mar-2024
https://doi.org/10.1007/s41233-024-00066-x
Kim JKim MLee WYoon S(2023)VibAware: Context-Aware Tap and Swipe Gestures Using Bio-Acoustic SensingProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614544(1-12)Online publication date: 13-Oct-2023
https://dl.acm.org/doi/10.1145/3607822.3614544
Waghmare ABoldu RWhitmire EKienzle W(2023)OptiRing: Low-Resolution Optical Sensing for Subtle Thumb-to-Index Micro-InteractionsProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614538(1-13)Online publication date: 13-Oct-2023
https://dl.acm.org/doi/10.1145/3607822.3614538
Azim MRahman AHeo S(2023)UnifiedSense: Enabling Without-Device Gesture Interactions Using Over-the-shoulder Training Between Redundant Wearable SensorsProceedings of the ACM on Human-Computer Interaction10.1145/36042777:MHCI(1-25)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604277
Kim DHarrison C(2023)Pantœnna: Mouth pose estimation for ar/vr headsets using low-profile antenna and impedance characteristic sensingProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606805(1-12)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606805
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents