research-article

Toffee: enabling ad hoc, around-device interaction with acoustic time-of-arrival correlation

Authors:

James Marsanico,

Divya Hariharan,

Chris HarrisonAuthors Info & Claims

MobileHCI '14: Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services

Pages 67 - 76

https://doi.org/10.1145/2628363.2628383

Published: 23 September 2014 Publication History

Abstract

The simple fact that human fingers are large and mobile devices are small has led to the perennial issue of limited surface area for touch-based interactive tasks. In response, we have developed Toffee, a sensing approach that extends touch interaction beyond the small confines of a mobile device and onto ad hoc adjacent surfaces, most notably tabletops. This is achieved using a novel application of acoustic time differences of arrival (TDOA) correlation. Previous time-of-arrival based systems have required semi-permanent instrumentation of the surface and were too large for use in mobile devices. Our approach requires only a hard tabletop and gravity -- the latter acoustically couples mobile devices to surfaces. We conducted an evaluation, which shows that Toffee can accurately resolve the bearings of touch events (mean error of 4.3° with a laptop prototype). This enables radial interactions in an area many times larger than a mobile device; for example, virtual buttons that lie above, below and to the left and right.

References

[1]

Ashbrook, D., Lyons, K., and Starner, T. An investigation into round touchscreen wristwatch interaction. In Proc. MobileHCI '08. 311--314.

Digital Library

[2]

Avrahami, D., Wobbrock, J., and Izadi, S. Portico: tangible interaction on and around a tablet. In Proc. UIST '11. 347--356.

Digital Library

[3]

Bai, M. R. and Tsai Y. K. Impact localization combined with haptic feedback for touch panel applications based on the time-reversal approach. Journal of the Acoustical Society of America, vol. 129, no. 3, pp. 1297--1305, March 2011.

[4]

Bancroft, S. An algebraic solution of the GPS equations. IEEE Trans. Aerospace and Electronic Systems, vol. AES-21, pp. 56--59, Jan. 1985.

[5]

Blanchard, W. F. Hyperbolic Airborne Radio Navigation Aids - A Navigator's View of their History and Development. The Journal Of Navigation, vol. 44, no. 3, pp. 285--315, Sept 1991.

[6]

Bornand, C., Camurri, A., Castellano, G., Catheline, S., Crevoisier, A., Roesch, E. B., Scherer, K. R., and Volpe, G. Usability evaluation and comparison of prototypes of tangible acoustic interfaces. In Proc. ENACTIVE '05.

[7]

Butler, A., Izadi, S., and Hodges, S. SideSight: multi-"touch" interaction around small devices. In Proc. UIST '08. 201--204.

Digital Library

[8]

Caffery, J. and Stuber, G. L. Overview of radiolocation in CDMA cellular systems. IEEE Commun. Mag., 36(4), pp. 38--45, Apr. 1998.

Digital Library

[9]

Carter, G. C. Time delay estimation for passive sonar signal processing. IEEE Trans. on Acoustics, Speech and Signal Processing, vol. ASSP-29, pp. 463--470, June 1981.

[10]

Chan, Y. T. A Simple and Efficient Estimator for Hyperbolic Location. IEEE Trans. on Signal Processing, vol. 42, no. 8, pp. 1905--1915, August 1994.

Digital Library

[11]

Fahy, F. and Gardonio, P. (2007). Sound and Structural Vibration, 2nd Ed. Academic Press. Oxford, UK.

[12]

Harrison, C. and Hudson, S. Abracadabra: wireless, high-precision, and unpowered finger input for very small mobile devices. In Proc. UIST '09. 121--124.

Digital Library

[13]

Harrison, C. and Hudson, S. Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces. In Proc. UIST '08. 205--208.

Digital Library

[14]

Harrison, C. Appropriated Interaction Surfaces. IEEE Computer Magazine, June 2010, 43(6). 86--89.

Digital Library

[15]

Harrison, C., Schwarz, J., and Hudson, S. E. TapSense: enhancing finger interaction on touch surfaces. In Proc. UIST '11, 627--636.

Digital Library

[16]

Ishii, H., Wisneski, C., Orbanes, J., Chun, C., and Paradiso, J. PingPongPlus: design of an athletic-tangible interface for computer-supported cooperative play. In Proc. CHI '99. 394--401.

Digital Library

[17]

Jones, B., Sodhi, R., Forsyth, D., Bailey, B. and Maciocci, G. Around device interaction for multiscale navigation. In Proc. MobileHCI '12. 83--92.

Digital Library

[18]

Kane, S., Avrahami, D., Wobbrock, J., Harrison, B., Rea, A., Philipose, M., and LaMarca, A. Bonfire: a nomadic system for hybrid laptop-tabletop interaction. In Proc. UIST '09. 129--138.

Digital Library

[19]

Ketabdar, H., Roshandel, M. and Yüksel, K. Towards using embedded magnetic field sensor for around mobile device 3D interaction. In Proc. MobileHCI '10. 153--156.

Digital Library

[20]

Kratz, S. and Rohs, M. HoverFlow: expanding the design space of around-device interaction. In Proc. MobileHCI '09. Article 4, 8 pages.

Digital Library

[21]

Kratz, S., Rohs, M., Guse, D., Müller, J., Bailly, G. and Nischt, M. PalmSpace: continuous around-device gestures vs. multitouch for 3D rotation tasks on mobile devices. In Proc. AVI '12. 181--188.

Digital Library

[22]

LeafLabs. Maple Mini. http://leaflabs.com/devices

[23]

Leo, C. K. Contact and Free-Gesture Tracking for Large Interactive Surfaces. MEng Thesis, MIT Dept. of EECS and MIT Media Lab, May 2002.

[24]

Paradiso, J. and Leo, C. Tracking and Characterizing Knocks Atop Large Interactive Displays. Sensor Review, vol. 25, no. 2, pp. 134--143, 2005.

[25]

Paradiso, J., Leo, C., Checka, N., and Hsiao, K. Passive acoustic sensing for tracking knocks atop large interactive displays. In Proc. IEEE Sensors '02. 521--527.

[26]

Patel, S. N. and Abowd, G. D. Blui: low-cost localized blowable user interfaces. In Proc. UIST '07. 217--220.

Digital Library

[27]

Pham, D. T., Al-Kutubi, M., Ji, Z., Yang, M., Wang, Z. and Catheline, S. Tangible acoustic interface approaches. In Proc. IPROMS '05. 497--502.

[28]

Pham, D. T., Ji, Z., Peyroutet, O., Yang, M., Wang, Z. and Al-Kutubi, M. Localisation of impacts on solid objects using the wavelet transform and maximum likelihood estimation. In Proc. IPROMS '06. 541--547.

[29]

Pham, D. T., Ji, Z., Yang, M., Wang, Z., and Al-Kutubi, M. A novel human-computer interface based on passive acoustic localisation. In Proc. HCI '07. 901--909.

Digital Library

[30]

Priyantha, N. B. Chakraborty, A., and Balakrishnan, H. The Cricket Location-Support system. In Proc. MOBICOM '00. 32--43.

Digital Library

[31]

Prosser, W. H., Gorman, M. R., and Humes, D. H. Acoustic emission signals in thin plates produced by impact damage. Journal of Acoustic Emission, vol. 17, no. 1-2, pp. 29--36, 1999.

[32]

Sato, M., Poupyrev, I., and Harrison, C. Touché: enhancing touch interaction on humans, screens, liquids, and everyday objects. In Proc. CHI '12. 483--492.

Digital Library

[33]

Seniuk, A. and Blostein, D. Pen Acoustic Emissions for Text and Gesture Recognition. In Proc. ICDAR '09. 872--876.

Digital Library

[34]

Urashima, A. and Toriyama, T. Ubiquitous Character Input Device Using Multiple Acoustic Sensors on a Flat Surface. In Proc. ICAT '10. 39--43.

[35]

Yang, M. Al-Kutubi, M., and Pham, D. T. In-Solid Acoustic Source Localization Using Likelihood Mapping Algorithm. Open Journal of Acoustics, vol. 1, no. 2, pp. 34--40, Sept. 2011.

[36]

Zimmerman, T., Smith, J., Paradiso, J., Allport, D., and Gershenfeld, N. Applying electric field sensing to human-computer interfaces. In Proc. CHI '95. 280--287.

Digital Library

Cited By

Wattanaparinton RKitada KTakemura K(2024)RFTIRTouch: Touch Sensing Device for Dual-sided Transparent Plane Based on Repropagated Frustrated Total Internal ReflectionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676428(1-10)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676428
Yang YChen THuang YGuo XShangguan L(2024)MAF: Exploring Mobile Acoustic Field for Hand-to-Face Gesture InteractionsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642437(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642437
Su YZhang TFeng JShi YYang XWu T(2024)Tagnoo: Enabling Smart Room-Scale Environments with RFID-Augmented PlywoodProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642356(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642356
Show More Cited By

Index Terms

Toffee: enabling ad hoc, around-device interaction with acoustic time-of-arrival correlation
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Skin buttons: cheap, small, low-powered and clickable fixed-icon laser projectors
UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology

Smartwatches are a promising new interactive platform, but their small size makes even basic actions cumbersome. Hence, there is a great need for approaches that expand the interactive envelope around smartwatches, allowing human input to escape the ...
Bezel swipe: conflict-free scrolling and multiple selection on mobile touch screen devices
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Zooming user interfaces are increasingly popular on mobile devices with touch screens. Swiping and pinching finger gestures anywhere on the screen manipulate the displayed portion of a page, and taps open objects within the page. This makes navigation ...
A study on touch & hover based interaction for zooming
CHI EA '12: CHI '12 Extended Abstracts on Human Factors in Computing Systems

Proximity is a useful medium for interaction with high interactive digital contents. It can be used in different contexts such as for navigation through depth in 3D space in zoomable interfaces. In this paper, we propose hover-based zoom interaction as ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MobileHCI '14: Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services

September 2014

664 pages

ISBN:9781450330046

DOI:10.1145/2628363

Conference Chairs:
Aaron Quigley
University of St. Andrews, UK
,
Sara Diamond
OCAD University, Canada
,
Program Chairs:
Pourang Irani
University of Manitoba, Canada
,
Sriram Subramanian
University of Bristol, UK

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 September 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

MobileHCI '14

Sponsor:

SIGCHI

MobileHCI '14: 16th International Conference on Human-Computer Interaction with Mobile Devices and Services

September 23 - 26, 2014

ON, Toronto, Canada

Acceptance Rates

MobileHCI '14 Paper Acceptance Rate 35 of 124 submissions, 28%;

Overall Acceptance Rate 202 of 906 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

61
Total Citations
View Citations
886
Total Downloads

Downloads (Last 12 months)51
Downloads (Last 6 weeks)4

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wattanaparinton RKitada KTakemura K(2024)RFTIRTouch: Touch Sensing Device for Dual-sided Transparent Plane Based on Repropagated Frustrated Total Internal ReflectionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676428(1-10)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676428
Yang YChen THuang YGuo XShangguan L(2024)MAF: Exploring Mobile Acoustic Field for Hand-to-Face Gesture InteractionsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642437(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642437
Su YZhang TFeng JShi YYang XWu T(2024)Tagnoo: Enabling Smart Room-Scale Environments with RFID-Augmented PlywoodProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642356(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642356
Shi YLi CSu YYang XWu T(2024)WooDowel: Electrode Isolation for Electromagnetic Shielding in Triboelectric Plywood SensorsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642304(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642304
Rateau HRekik YLank E(2023)Ether-Mark: An Off-Screen Marking Menu For Mobile DevicesProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614150(224-233)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614150
Li ZLiang CWang YQin YYu CYan YFan MShi Y(2023)Enabling Voice-Accompanying Hand-to-Face Gesture Recognition with Cross-Device SensingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581008(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581008
Iravantchi YZhao YKin KSample A(2023)SAWSense: Using Surface Acoustic Waves for Surface-bound Event RecognitionProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580991(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580991
Byanjankar ALiu YShu YShin IChoi MKim H(2023)S-UbiTap: Leveraging Acoustic Dispersion for Ubiquitous and Scalable Touch Interface on Solid SurfacesIEEE Transactions on Mobile Computing10.1109/TMC.2022.319522622:11(6800-6816)Online publication date: 1-Nov-2023
https://doi.org/10.1109/TMC.2022.3195226
Grubert J(2023)Mixed Reality Interaction TechniquesSpringer Handbook of Augmented Reality10.1007/978-3-030-67822-7_5(109-129)Online publication date: 1-Jan-2023
https://doi.org/10.1007/978-3-030-67822-7_5
Amesaka TWatanabe HSugimoto MShizuki B(2022)Gesture Recognition Method Using Acoustic Sensing on Usual GarmentProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35345796:2(1-27)Online publication date: 7-Jul-2022
https://dl.acm.org/doi/10.1145/3534579
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents