research-article

User-defined interaction for smart homes: voice, touch, or mid-air gestures?

Authors:

Fabian Hoffmann,

Miriam-Ida Tyroller,

Niels HenzeAuthors Info & Claims

MUM '19: Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia

Article No.: 33, Pages 1 - 7

https://doi.org/10.1145/3365610.3365624

Published: 26 November 2019 Publication History

Abstract

Smart home appliances and smart homes, in general, are on the verge of ubiquity. Research and industry proposed a range of modalities, including speech, mid-air gestures, and touch displays, to control smart homes. While previous work designed for the individual modalities, it is unclear how they compare from a user-centered perspective. Therefore, we conducted an elicitation study that asked participants to propose commands using speech, mid-air gestures, and a touch display. Also, we asked participants to rate their suggestions and the modalities. The results show that using voice commands or a touch display is clearly preferred compared to the use of mid-air gestures. As we found high agreement scores for voice commands, our results also highlight the potential of elicitation studies for voice interfaces.

References

[1]

Christopher Ackad, Andrew Clayphan, Martin Tomitsch, and Judy Kay. 2015. An In-the-wild Study of Learning Mid-air Gestures to Browse Hierarchical Information at a Large Interactive Public Display. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 1227--1238.

Digital Library

[2]

Shaikh Shawon Arefin Shimon, Courtney Lutton, Zichun Xu, Sarah Morrison-Smith, Christina Boucher, and Jaime Ruiz. 2016. Exploring Non-touchscreen Gestures for Smartwatches. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3822--3833.

Digital Library

[3]

Andrea Bellucci, Andrea Vianello, Yves Florack, Luana Micallef, and Giulio Jacucci. 2019. Augmenting objects at home through programmable sensor tokens: A design journey. International Journal of Human-Computer Studies 122 (2019), 211--231.

[4]

Joanna Bergstrom-Lehtovirta and Antti Oulasvirta. 2014. Modeling the Functional Area of the Thumb on Mobile Touchscreen Surfaces. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 1991--2000.

Digital Library

[5]

Richard A. Bolt. 1980. Put-that-there: Voice and Gesture at the Graphics Interface. SIGGRAPH Comput. Graph. 14, 3 (July 1980), 262--270.

Digital Library

[6]

Tilman Dingler, Rufat Rzayev, Alireza Sahami Shirazi, and Niels Henze. 2018. Designing Consistent Gestures Across Device Types. In Engage with CHI, Regan Mandryk and Mark Hancock (Eds.). The Association for Computing Machinery, New York, New York, 1--12.

Digital Library

[7]

Yasmin Felberbaum and Joel Lanir. 2018. Better Understanding of Foot Gestures. In Engage with CHI, Regan Mandryk and Mark Hancock (Eds.). The Association for Computing Machinery, New York, New York, 1--12.

Digital Library

[8]

Florian Habler, Marco Peisker, and Niels Henze. 2019. Differences Between Smart Speakers and Graphical User Interfaces for Music Search Considering Gender Effects. In Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia (MUM 2019). ACM, New York, NY, USA, 7.

Digital Library

[9]

Florian Habler, Valentin Schwind, and Niels Henze. 2019. Effects of Smart Virtual Assistants' Gender and Language. In Proceedings of Mensch Und Computer 2019 (MuC'19). ACM, New York, NY, USA, 469--473.

Digital Library

[10]

Charles Hannon. 2016. Gender and Status in Voice User Interfaces. Interactions 23, 3 (April 2016), 34--37.

Digital Library

[11]

Niels Henze, Andreas Löcken, Susanne Boll, Tobias Hesselmann, and Martin Pielot. 2010. Free-hand Gestures for Music Playback: Deriving Gestures with a User-centred Process. In Proceedings of the 9th International Conference on Mobile and Ubiquitous Multimedia (MUM '10). ACM, New York, NY, USA, Article 16, 10 pages.

Digital Library

[12]

Lynn Hoff, Eva Hornecker, and Sven Bertel. 2016. Modifying Gesture Elicitation: Do Kinaesthetic Priming and Increased Production Reduce Legacy Bias?. In Proceedings of the TEI '16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '16). ACM, New York, NY, USA, 86--91.

Digital Library

[13]

Alison Duncan Kerr. 2018. Alexa and the Promotion of Oppression. In Proceedings of the 2018 ACM Celebration of Women in Computing (womENcourage '18). ACM, New York, NY, USA. https://womencourage.acm.org/2018/wp-content/uploads/2018/07/womENcourage_2018_paper_54.pdf

[14]

Christine Kühnel, Tilo Westermann, Fabian Hemmert, Sven Kratz, Alexander Müller, and Sebastian Möller. 2011. I'm home: Defining and evaluating a gesture set for smart-home control. International Journal of Human-Computer Studies 69, 11 (2011), 693--704.

[15]

Anne Köpsel and Nikola Bubalo. 2015. Benefiting from Legacy Bias. interactions 22, 5 (Aug. 2015), 44--47.

Digital Library

[16]

Huy Viet Le, Sven Mayer, Benedict Steuerlein, and Niels Henze. 2019. Investigating Unintended Inputs for One-Handed Touch Interaction Beyond the Touchscreen. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '19). ACM, New York, NY, USA, Article 34, 14 pages.

Digital Library

[17]

Andreas Löcken, Tobias Hesselmann, Martin Pielot, Niels Henze, and Susanne Boll. 2011. User-centred process for the definition of free-hand gestures applied to controlling music playback. Multimedia Systems 18 (2011), 15--31.

[18]

Sven Mayer, Perihan Gad, Katrin Wolf, Paweł W. Woźniak, and Niels Henze. 2017. Understanding the Ergonomic Constraints in Designing for Touch Surfaces. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, Article 33, 9 pages.

Digital Library

[19]

Sven Mayer, Lars Lischke, Adrian Lanksweirt, Huy Viet Le, and Niels Henze. 2018. How to Communicate New Input Techniques. In Proceedings of the 10th Nordic Conference on Human-Computer Interaction (NordiCHI '18). ACM, New York, NY, USA, 460--472.

Digital Library

[20]

Sven Mayer, Lars Lischke, Paweł W. Woźniak, and Niels Henze. 2018. Evaluating the Disruptiveness of Mobile Interactions: A Mixed-Method Approach. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 406, 14 pages.

Digital Library

[21]

Meredith Ringel Morris, Andreea Danielescu, Steven Drucker, Danyel Fisher, Bongshin Lee, m. c. schraefel, and Jacob O. Wobbrock. 2014. Reducing Legacy Bias in Gesture Elicitation Studies. interactions 21, 3 (May 2014), 40--45.

Digital Library

[22]

Meredith Ringel Morris, Jacob O. Wobbrock, and Andrew D. Wilson. 2010. Understanding Users' Preferences for Surface Gestures. In Proceedings of Graphics Interface 2010 (GI '10). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 261--268. http://dl.acm.org/citation.cfm?id=1839214.1839260

Digital Library

[23]

Miguel A. Nacenta, Yemliha Kamber, Yizhou Qiang, and Per Ola Kristensson. 2013. Memorability of Pre-designed and User-defined Gesture Sets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1099--1108.

Digital Library

[24]

Michael Nielsen, Moritz Störring, Thomas B Moeslund, and Erik Granum. 2003. A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In International gesture workshop. Springer, 409--420.

[25]

Chidera Obinali. 2019. The Perception of Gender in Voice Assistants. In Proceedings of the Southern Association for Information Systems Conference. 6. https://aisel.aisnet.org/sais2019/39

[26]

Julie Rico and Stephen Brewster. 2010. Gesture and Voice Prototyping for Early Evaluations of Social Acceptability in Multimodal Interfaces. In International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction (ICMI-MLMI '10). ACM, New York, NY, USA, Article 16, 9 pages.

Digital Library

[27]

Jaime Ruiz, Yang Li, and Edward Lank. 2011. User-defined motion gestures for mobile interaction. In Conference proceedings and extended abstracts / the 29th Annual CHI Conference on Human Factors in Computing Systems, Desney Tan, Geraldine Fitzpatrick, Carl Gutwin, Bo Begole, and Wendy A. Kellogg (Eds.). ACM, New York, NY, 197.

Digital Library

[28]

Jaime Ruiz and Daniel Vogel. 2015. Soft-Constraints to Reduce Legacy and Performance Bias to Elicit Whole-body Gestures with Low Arm Fatigue. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3347--3350.

Digital Library

[29]

Thomas Schlömer, Benjamin Poppinga, Niels Henze, and Susanne Boll. 2008. Gesture Recognition with a Wii Controller. In Proceedings of the 2Nd International Conference on Tangible and Embedded Interaction (TEI '08). ACM, New York, NY, USA, 11--14.

Digital Library

[30]

Valentin Schwind, Niklas Deierlein, Romina Poguntke, and Niels Henze. 2019. Understanding the Social Acceptability of Mobile Devices Using the Stereotype Content Model. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Article 361, 12 pages.

Digital Library

[31]

Radu-Daniel Vatavu. 2012. User-defined Gestures for Free-hand TV Control. In Proceedings of the 10th European Conference on Interactive TV and Video (EuroITV '12). ACM, New York, NY, USA, 45--48.

Digital Library

[32]

Radu-Daniel Vatavu. 2013. A Comparative Study of User-defined Handheld vs. Freehand Gestures for Home Entertainment Environments. Journal of Ambient Intelligence and Smart Environments, 5, 2 (March 2013), 187--211.

[33]

Radu-Daniel Vatavu and Jacob O. Wobbrock. 2015. Formalizing Agreement Analysis for Elicitation Studies. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15, Bo Begole, Jinwoo Kim, Kori Inkpen, and Woontack Woo (Eds.). ACM Press, New York, New York, USA, 1325--1334.

Digital Library

[34]

Panagiotis Vogiatzidakis and Panayiotis Koutsabasis. 2018. Gesture Elicitation Studies for Mid-Air Interaction: A Review. Multimodal Technologies and Interaction 2, 4 (2018).

[35]

Jacob O. Wobbrock, Htet Htet Aung, Brandon Rothrock, and Brad A. Myers. 2005. Maximizing the Guessability of Symbolic Input. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (CHI EA '05). ACM, New York, NY, USA, 1869--1872.

Digital Library

[36]

Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-defined gestures for surface computing. In CHI 2009 - digital life, new world, Dan R. Olsen, Richard B. Arthur, Ken Hinckley, Meredith Ringel Morris, Scott Hudson, and Saul Greenberg (Eds.). ACM, New York, NY, 1083.

Digital Library

[37]

Ionuţ-Alexandru Zaiţi, Ştefan-Gheorghe Pentiuc, and Radu-Daniel Vatavu. 2015. On free-hand TV control: experimental results on user-elicited gestures with Leap Motion. Personal and Ubiquitous Computing 19, 5--6 (2015), 821--838.

Cited By

Wu LLafreniere BGrossman TWhite TSantosa S(2024)Body Language for VUIs: Exploring Gestures to Enhance Interactions with Voice User InterfacesProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660691(133-150)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660691
Hosseini MMueller HBoll S(2024)Controlling the Rooms: How People Prefer Using Gestures to Control Their Smart HomesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642687(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642687
Graichen LGraichen M(2024)Do users desire gestures for in-vehicle interaction? Towards the subjective assessment of gestures in a high-fidelity driving simulatorComputers in Human Behavior10.1016/j.chb.2024.108189156:COnline publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1016/j.chb.2024.108189
Show More Cited By

Index Terms

User-defined interaction for smart homes: voice, touch, or mid-air gestures?
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces
      2. Natural language interfaces
  2. Ubiquitous and mobile computing

Recommendations

Eliciting Mid-Air Gestures for Wall-Display Interaction
NordiCHI '16: Proceedings of the 9th Nordic Conference on Human-Computer Interaction

Freehand mid-air gestures are a promising input method for interacting with wall displays. However, work on mid-air gestures for wall-display interaction has mainly explored what is technically possible, which might not result in gestures that users ...
Mimetic interaction spaces: controlling distant displays in pervasive environments
IUI '14: Proceedings of the 19th international conference on Intelligent User Interfaces

Pervasive computing is a vision that has been an inspiring long-term target for many years now. Interaction techniques that allow one user to efficiently control many screens, or that allow several users to collaborate on one distant screen, are still ...
Investigating Cross-Device Interaction between a Handheld Device and a Large Display
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

There is a growing interest in HCI research to explore cross-device interaction, giving rise to an interest in different approaches facilitating interaction between handheld devices and large displays. Contributing to this, we have investigated the use ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MUM '19: Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia

November 2019

462 pages

ISBN:9781450376242

DOI:10.1145/3365610

General Chair:
Fabio Paternò
CNR-ISTI, Italy
,
Program Chairs:
Giulio Jacucci
University of Helsinki, Finland
,
Michael Rohs
University of Hannover, Germany
,
Carmen Santoro
CNR-ISTI, Italy

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 November 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MUM 2019

MUM 2019: 18th International Conference on Mobile and Ubiquitous Multimedia

November 26 - 29, 2019

Pisa, Italy

Acceptance Rates

Overall Acceptance Rate 190 of 465 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
683
Total Downloads

Downloads (Last 12 months)54
Downloads (Last 6 weeks)11

Reflects downloads up to 07 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu LLafreniere BGrossman TWhite TSantosa S(2024)Body Language for VUIs: Exploring Gestures to Enhance Interactions with Voice User InterfacesProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660691(133-150)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660691
Hosseini MMueller HBoll S(2024)Controlling the Rooms: How People Prefer Using Gestures to Control Their Smart HomesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642687(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642687
Graichen LGraichen M(2024)Do users desire gestures for in-vehicle interaction? Towards the subjective assessment of gestures in a high-fidelity driving simulatorComputers in Human Behavior10.1016/j.chb.2024.108189156:COnline publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1016/j.chb.2024.108189
Kostic ZDumas CPratt SBeyer J(2023)Exploring Mid-Air Hand Interaction in Data VisualizationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.333264730:9(6347-6364)Online publication date: 14-Nov-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3332647
Shin HKim MCho J(2023)Towards Developing Novel and Intuitive Brain-based Control Interface via On-skin Input2023 11th International Winter Conference on Brain-Computer Interface (BCI)10.1109/BCI57258.2023.10078660(1-5)Online publication date: 20-Feb-2023
https://doi.org/10.1109/BCI57258.2023.10078660
Hou WGuo GCheng Y(2023)Multimodal Interaction Grammar Analysis Based on Two-Stage User-Based Elicitation in 3D ModelingInternational Journal of Human–Computer Interaction10.1080/10447318.2023.2265735(1-22)Online publication date: 16-Oct-2023
https://doi.org/10.1080/10447318.2023.2265735
Cha MJi Y(2023)Context Matters: Understanding the Effect of Usage Contexts on Users’ Modality Selection in Multimodal SystemsInternational Journal of Human–Computer Interaction10.1080/10447318.2023.225060640:20(6287-6302)Online publication date: 29-Aug-2023
https://doi.org/10.1080/10447318.2023.2250606
Ch NTosca DCrump TAnsah AKun AShaer OJi YJeon M(2022)Gesture and Voice Commands to Interact With AR Windshield Display in Automated Vehicle: A Remote Elicitation StudyProceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3543174.3545257(171-182)Online publication date: 17-Sep-2022
https://dl.acm.org/doi/10.1145/3543174.3545257
Heck MShon SBecker C(2022)Does Using Voice Authentication in Multimodal Systems Correlate With Increased Speech Interaction During Non-critical Routine Tasks?Proceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511129(868-877)Online publication date: 22-Mar-2022
https://dl.acm.org/doi/10.1145/3490099.3511129
Beşevli CGenç HCoşkun AGöksun TYemez YÖzcan O(2022)Gestural interaction in the kitchen: Insights into designing an interactive display controlled by hand specific on-skin gesturesThe Design Journal10.1080/14606925.2022.205844425:3(353-373)Online publication date: 14-Apr-2022
https://doi.org/10.1080/14606925.2022.2058444
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents