skip to main content
10.1145/2642918.2647373acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

In-air gestures around unmodified mobile devices

Published: 05 October 2014 Publication History

Abstract

We present a novel machine learning based algorithm extending the interaction space around mobile devices. The technique uses only the RGB camera now commonplace on off-the-shelf mobile devices. Our algorithm robustly recognizes a wide range of in-air gestures, supporting user variation, and varying lighting conditions. We demonstrate that our algorithm runs in real-time on unmodified mobile devices, including resource-constrained smartphones and smartwatches. Our goal is not to replace the touchscreen as primary input device, but rather to augment and enrich the existing interaction vocabulary using gestures. While touch input works well for many scenarios, we demonstrate numerous interaction tasks such as mode switches, application and task management, menu selection and certain types of navigation, where such input can be either complemented or better served by in-air gestures. This removes screen real-estate issues on small touchscreens, and allows input to be expanded to the 3D space around the device. We present results for recognition accuracy (93% test and 98% train), impact of memory footprint and other model parameters. Finally, we report results from preliminary user evaluations, discuss advantages and limitations and conclude with directions for future work.

Supplementary Material

ZIP File (uistf2149-file5.zip)
The supplementary material contains additional information on the random forest evaluation and in particular aspects pertaining to the implementation on mobile phones. Furthermore, we report additional results from experiments with an extended gesture set.
suppl.mov (uistf2149-file3.mp4)
Supplemental video

References

[1]
Ashbrook, D., Baudisch, P., and White, S. Nenya : Subtle and Eyes-Free Mobile Input with a Magnetically-tracked Finger Ring. In Proc. ACM SIGCHI (2011), 2043--2046.
[2]
Breiman, L. Random Forests. Mach. Learn. 45, 1 (Oct. 2001), 5--32.
[3]
Butler, A., Izadi, S., and Hodges, S. SideSight: multi-touch interaction around small devices. In Proc. ACM UIST, UIST '08 (2008), 201--204.
[4]
Chen, K.-Y., Lyons, K., White, S., and Patel, S. uTrack: 3D input using two magnetic sensors. In Proc. ACM UIST (2013).
[5]
Chen, X. A., Grossman, T., Wigdor, D., and Fitzmaurice, G. Duet: Exploring Joint Interactions on a Smart Phone and a Smart Watch. In Proc. ACM CHI'14 (2014).
[6]
Erol, A., Bebis, G., Nicolescu, M., Boyle, R. D., and Twombly, X. Vision-based hand pose estimation: A review. Comput. Vis. Image Underst. (2007).
[7]
Goc, M. L., Taylor, S., Izadi, S., and Keskin, C. A Low-cost Transparent EF Sensor for 3D Interaction on Mobile Devices. In Proc. ACM CHI (2014).
[8]
Guiard, Y. Asymmetric Division of Labor in Human Skilled Bimanual Action. J. Mot. Behav. 19, 4 (1987).
[9]
Gustafson, S., Bierwirth, D., and Baudisch, P. Imaginary interfaces. In Proc. ACM UIST (2010), 3--12.
[10]
Harrison, C., Benko, H., and Wilson, A. D. OmniTouch: Wearable Multitouch Interaction Everywhere. In Proc. ACM UIST, ACM (2011), 441--450.
[11]
Harrison, C., Tan, D., and Morris, D. Skinput: Appropriating the Body As an Input Surface. In Proc. ACM CHI (2010), 453--462.
[12]
Hilliges, O., Kim, D., Izadi, S., and Weiss, M. HoloDesk: Direct 3D Interactions with a Situated See-Through Display. In Proc. ACM CHI (2012).
[13]
Hinckley, K., and Song, H. Sensor synaesthesia: touch in motion, and motion in touch. In Proc. ACM CHI (2011).
[14]
Hinckley, K., Yatani, K., Pahud, M., Coddington, N., Rodenhouse, J., Wilson, A., Benko, H., and Buxton, B. Pen + touch = new tools. In Proc. ACM UIST (2010).
[15]
Hwang, S., Ahn, M., and Wohn, K.-y. MagGetz: customizable passive tangible controllers on and around mobile devices. In Proc. ACM UIST (2013).
[16]
Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., and Fitzgibbon, A. KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera. In Proc. ACM UIST (2011), 559--568.
[17]
Jones, B., Sodhi, R., Forsyth, D., Bailey, B., and Maciocci, G. Around device interaction for multiscale navigation. In Proc. of MobileHCI (2012).
[18]
Kakumanu, P., Makrogiannis, S., and Bourbakis, N. A survey of skin-color modeling and detection methods. Pattern Recognit. 40, 3 (2007), 1106--1122.
[19]
Keskin, C., Kirac, F., Kara, Y. E., Akarun, L., and Kiracç, F. Hand Pose Estimation and Hand Shape Classification Using Multi-layered Randomized Decision Forests. In Proc. ECCV (Berlin, Heidelberg, 2012).
[20]
Ketabdar, H., Roshandel, M., and Yuksel, K. A. Towards using embedded magnetic field sensor for around mobile device 3D interaction. In Proc. MobileHCI (2010).
[21]
Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., and Olivier, P. Digits: Freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proc. ACM UIST (2012).
[22]
Kim, D., Izadi, S., Dostal, J., Rhemann, C., Keskin, C., Zach, C., Shotton, J., Large, T., Bathiche, S., Nieß ner, M., Butler, A., Fanello, S., and Pradeep, V. RetroDepth: 3D Silhouette Sensing for High-Precision Input On and Above Physical Surfaces. In Proc. ACM CHI (2014).
[23]
Kim, J., He, J., Lyons, K., and Starner, T. The Gesture Watch: A Wireless Contact-free Gesture based Wrist Interface. In IEEE ISWC (2007), 1--8.
[24]
Kratz, S., and Rohs, M. HoverFlow: expanding the design space of around-device interaction. In Proc. MobileHCI, ACM (2009), 4.
[25]
Lee, S. C., Li, B., and Starner, T. AirTouch: Synchronizing In-air Hand Gesture and On-body Tactile Feedback. In IEEE ISWC (2011), 3--10.
[26]
Mistry, P., and Maes, P. SixthSense: a wearable gestural interface. In ACM SIGGRAPH ASIA Sketches (2009).
[27]
Molyneaux, D., Izadi, S., Kim, D., Hilliges, O., Hodges, S., Cao, X., Butler, A., and Gellersen, H. Interactive environment-aware handheld projectors for pervasive computing spaces. In Pervasive Comput. (2012).
[28]
Niikura, T., Hirobe, Y., Cassinelli, A., Watanabe, Y., Komuro, T., and Ishikawa, M. In-air typing interface for mobile devices with vibration feedback. In ACM SIGGRAPH Emerg. Technol. (2010).
[29]
Ogata, M., Sugiura, Y., Makino, Y., Inami, M., and Imai, M. SenSkin: adapting skin as a soft interface. In Proc. ACM UIST (New York, New York, USA, 2013).
[30]
Oikonomidis, I., Kyriazis, N., and Argyros, A. A. Tracking the Articulated Motion of Two Strongly Interacting Hands. In IEEE CVPR (Providence, 2012).
[31]
Rekimoto, J. GestureWrist and GesturePad: unobtrusive wearable interaction devices. In IEEE ISWC (2001).
[32]
Saponas, S., Tan, D., Morris, D., Balakrishnan, R., Turner, J., and Landay, J. Enabling always-available input with muscle-computer interfaces. In Proc. ACM UIST (2009).
[33]
Shotton, J., Fitzgibbon, A. W., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. Real-time human pose recognition in parts from single depth images. Commun. ACM 56, 1 (2013), 116--124.
[34]
Song, H., Guimbretiere, F., Grossman, T., and Fitzmaurice, G. MouseLight: Bimanual Interactions on Digital Paper Using a Pen and a Spatially-aware Mobile Projector. In Proc. ACM CHI (2010).
[35]
Sridhar, S., Oulasvirta, A., and Theobalt, C. Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data. In IEEE ICCV (2013).
[36]
Starner, T., Auxier, J., Ashbrook, D., and Gandy, M. The gesture pendant: a self-illuminating, wearable, infrared computer vision system for home automation control and medical monitoring. In IEEE ISWC (2000), 87--94.
[37]
Starner, T., Weaver, J., and Pentland, A. A wearable computer based American sign language recognizer. In IEEE ISWC, ISWC '97 (1997).
[38]
Tang, D., Yu, T., and Kim, T. Real-time Articulated Hand Pose Estimation using Semi-supervised Transductive Regression Forests. In IEEE ICCV (2013).
[39]
Taylor, S., Keskin, C., Hilliges, O., Izadi, S., and Helmes, J. Type-Hover-Swipe in 96 Bytes: A Motion Sensing Mechanical Keyboard. In Proc. ACM CHI (2014).
[40]
Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J., and Shen, C. LucidTouch: A See-Through Mobile Device. In Proc. ACM UIST, ACM Press (2007).

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology
October 2014
722 pages
ISBN:9781450330695
DOI:10.1145/2642918
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 October 2014

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. HCI
  2. gesture recognition
  3. mobile computing
  4. mobile gestures
  5. mobile interaction
  6. random forests

Qualifiers

  • Research-article

Conference

UIST '14

Acceptance Rates

UIST '14 Paper Acceptance Rate 74 of 333 submissions, 22%;
Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25
The 38th Annual ACM Symposium on User Interface Software and Technology
September 28 - October 1, 2025
Busan , Republic of Korea

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)45
  • Downloads (Last 6 weeks)8
Reflects downloads up to 03 Jan 2025

Other Metrics

Citations

Cited By

View all

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media