research-article

Improving Finger Stroke Recognition Rate for Eyes-Free Mid-Air Typing in VR

Authors:

Yatharth Singhal,

Richard Huynh Noeske,

Ayush Bhardwaj,

Jin Ryong KimAuthors Info & Claims

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Article No.: 346, Pages 1 - 9

https://doi.org/10.1145/3491102.3502100

Published: 29 April 2022 Publication History

Abstract

We examine mid-air typing data collected from touch typists to evaluate the features and classification models for recognizing finger stroke. A large number of finger movement traces have been collected using finger motion capture systems, labeled into individual finger strokes, and classified into several key features. We test finger kinematic features, including 3D position, velocity, acceleration, and temporal features, including previous fingers and keys. Based on this analysis, we assess the performance of various classifiers, including Naive Bayes, Random Forest, Support Vector Machines, and Deep Neural Networks, in terms of the accuracy for correctly classifying the keystroke. We finally incorporate a linguistic heuristic to explore the effectiveness of the character prediction model and improve the total accuracy.

Supplementary Material

MP4 File (3491102.3502100-video-figure.mp4)

Video Figure

Download
23.93 MB

MP4 File (3491102.3502100-video-preview.mp4)

Video Preview

Download
17.05 MB

MP4 File (3491102.3502100-talk-video.mp4)

Talk Video

Download
166.04 MB

References

[1]

Jiban Adhikary and Keith Vertanen. 2021. Typing on Midair Virtual Keyboards: Exploring Visual Designs and Interaction Styles. In IFIP Conference on Human-Computer Interaction. Springer, 132–151.

[2]

Costas Boletsis and Stian Kongsvik. 2019. Text input in virtual reality: A preliminary evaluation of the drum-like vr keyboard. Technologies 7, 2 (2019), 31.

[3]

Steven J Castellucci and I Scott MacKenzie. 2008. Graffiti vs. unistrokes: an empirical comparison. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 305–308.

Digital Library

[4]

Sibo Chen, Junce Wang, Santiago Guerra, Neha Mittal, and Soravis Prakkamakul. 2019. Exploring word-gesture text entry techniques in virtual reality. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–6.

Digital Library

[5]

Tafadzwa Joseph Dube and Ahmed Sabbir Arif. 2019. Text entry in virtual reality: A comprehensive review of the literature. In International Conference on Human-Computer Interaction. Springer, 419–437.

Digital Library

[6]

John Dudley, Hrvoje Benko, Daniel Wigdor, and Per Ola Kristensson. 2019. Performance envelopes of virtual keyboard text input strategies in virtual reality. In 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 289–300.

[7]

John J Dudley, Keith Vertanen, and Per Ola Kristensson. 2018. Fast and precise touch-based text entry for head-mounted augmented reality with variable occlusion. ACM Transactions on Computer-Human Interaction (TOCHI) 25, 6(2018), 1–40.

Digital Library

[8]

Anna Maria Feit, Srinath Sridhar, Christian Theobalt, and Antti Oulasvirta. 2015. Investigating multi-finger gestures for mid-air text entry. ACM womENcourage (2015).

[9]

Anna Maria Feit, Daryl Weir, and Antti Oulasvirta. 2016. How we type: Movement strategies and performance in everyday typing. In Proceedings of the 2016 chi conference on human factors in computing systems. 4262–4273.

Digital Library

[10]

Leah Findlater, Jacob O Wobbrock, and Daniel Wigdor. 2011. Typing on flat glass: examining ten-finger expert typing patterns on touch surfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems. 2453–2462.

Digital Library

[11]

Conor R Foy, John J Dudley, Aakar Gupta, Hrvoje Benko, and Per Ola Kristensson. 2021. Understanding, Detecting and Mitigating the Effects of Coactivations in Ten-Finger Mid-Air Typing in Virtual Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–11.

Digital Library

[12]

Hyunjae Gil, Yonghwan Shin, Hyungki Son, Inwook Hwang, Ian Oakley, and Jin Ryong Kim. 2020. Characterizing In-Air Eyes-Free Typing Movements in VR. In 26th ACM Symposium on Virtual Reality Software and Technology. 1–10.

[13]

Joshua Goodman, Gina Venolia, Keith Steury, and Chauncey Parker. 2002. Language modeling for soft keyboards. In Proceedings of the 7th international conference on Intelligent user interfaces. 194–195.

Digital Library

[14]

Jens Grubert, Lukas Witzani, Eyal Ofek, Michel Pahud, Matthias Kranz, and Per Ola Kristensson. 2018. Text entry in immersive head-mounted display-based virtual reality using standard keyboards. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 159–166.

[15]

Jan Gugenheimer, David Dobbelstein, Christian Winkler, Gabriel Haas, and Enrico Rukzio. 2016. FaceTouch: Touch interaction for mobile virtual reality. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 3679–3682.

Digital Library

[16]

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning. PMLR, 1321–1330.

[17]

Aakar Gupta, Cheng Ji, Hui-Shyong Yeo, Aaron Quigley, and Daniel Vogel. 2019. Rotoswype: Word-gesture typing using a ring. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.

Digital Library

[18]

Ramin Hedeshy, Chandan Kumar, Raphael Menges, and Steffen Staab. 2021. Hummer: Text Entry by Gaze and Hum. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–11.

Digital Library

[19]

Haiyan Jiang and Dongdong Weng. 2020. HiPad: Text entry for head-mounted displays using circular touchpad. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 692–703.

[20]

Jin Ryong Kim and Hong Z Tan. 2014. Haptic feedback intensity affects touch typing performance on a flat keyboard. In International Conference on Human Haptic Sensing and Touch Enabled Computer Applications. Springer, 369–375.

[21]

Jin Ryong Kim and Hong Z Tan. 2014. A study of touch typing performance with keyclick feedback. In 2014 IEEE Haptics Symposium (HAPTICS). IEEE, 227–233.

[22]

Jin Ryong Kim and Hong Z Tan. 2015. Effect of information content in sensory feedback on typing performance using a flat keyboard. In 2015 IEEE World Haptics Conference (WHC). IEEE, 228–234.

[23]

Pascal Knierim, Valentin Schwind, Anna Maria Feit, Florian Nieuwenhuizen, and Niels Henze. 2018. Physical keyboards in virtual reality: Analysis of typing performance and effects of avatar hands. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–9.

Digital Library

[24]

Per-Ola Kristensson and Shumin Zhai. 2004. SHARK2: a large vocabulary shorthand writing system for pen-based computers. In Proceedings of the 17th annual ACM symposium on User interface software and technology. 43–52.

Digital Library

[25]

Minkyung Lee, Woontack Woo, 2003. ARKB: 3D vision-based Augmented Reality Keyboard. In ICAT.

[26]

Yongjae Lee and Gerard J Kim. 2017. Vitty: Virtual touch typing interface with added finger buttons. In International Conference on Virtual, Augmented and Mixed Reality. Springer, 111–119.

[27]

I Scott MacKenzie and R William Soukoreff. 2003. Phrase sets for evaluating text entry techniques. In CHI’03 extended abstracts on Human factors in computing systems. 754–755.

[28]

Jennifer Mankoff and Gregory D Abowd. 1998. Cirrin: A word-level unistroke keyboard for pen input. In Proceedings of the 11th annual ACM symposium on User interface software and technology. 213–214.

Digital Library

[29]

Anders Markussen, Mikkel Rønne Jakobsen, and Kasper Hornbæk. 2014. Vulture: a mid-air word-gesture keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1073–1082.

Digital Library

[30]

Mark McGill, Daniel Boland, Roderick Murray-Smith, and Stephen Brewster. 2015. A dose of reality: Overcoming usability challenges in vr head-mounted displays. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2143–2152.

[31]

Tao Ni, Doug Bowman, and Chris North. 2011. AirStroke: bringing unistroke text entry to freehand gesture interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2473–2476.

Digital Library

[32]

Ken Perlin. 1998. Quikwriting: continuous stylus-based text entry. In Proceedings of the 11th annual ACM symposium on User interface software and technology. 215–216.

Digital Library

[33]

Duc-Minh Pham and Wolfgang Stuerzlinger. 2019. Hawkey: Efficient and versatile text entry for virtual reality. In 25th ACM Symposium on Virtual Reality Software and Technology. 1–11.

Digital Library

[34]

Samuel Reese, Gemma Boleda, Montse Cuadros, Lluís Padró, and German Rigau. 2010. Wikicorpus: A word-sense disambiguated multilingual wikipedia corpus. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10).

[35]

Junxiao Shen, John Dudley, and Per Ola Kristensson. 2021. Simulating Realistic Human Motion Trajectories of Mid-Air Gesture Typing. In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 393–402.

[36]

Marco Speicher, Anna Maria Feit, Pascal Ziegler, and Antonio Krüger. 2018. Selection-based text entry in virtual reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[37]

Srinath Sridhar, Anna Maria Feit, Christian Theobalt, and Antti Oulasvirta. 2015. Investigating the dexterity of multi-finger input for mid-air text entry. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 3643–3652.

Digital Library

[38]

James Walker, Bochao Li, Keith Vertanen, and Scott Kuhl. 2017. Efficient typing on a visually occluded physical keyboard. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 5457–5461.

Digital Library

[39]

Eric Whitmire, Mohit Jain, Divye Jain, Greg Nelson, Ravi Karkar, Shwetak Patel, and Mayank Goel. 2017. Digitouch: Reconfigurable thumb-to-finger input and text entry on head-mounted displays. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1–21.

Digital Library

[40]

Jacob O Wobbrock and Brad A Myers. 2006. Analyzing the input stream for character-level errors in unconstrained text entry evaluations. ACM Transactions on Computer-Human Interaction (TOCHI) 13, 4(2006), 458–489.

Digital Library

[41]

Jacob O Wobbrock, Brad A Myers, and Duen Horng Chau. 2006. In-stroke word completion. In Proceedings of the 19th annual ACM symposium on User interface software and technology. 333–336.

Digital Library

[42]

Naoki Yanagihara, Buntarou Shizuki, and Shin Takahashi. 2019. Text Entry Method for Immersive Virtual Environments Using Curved Keyboard. In 25th ACM Symposium on Virtual Reality Software and Technology. 1–2.

[43]

Hui-Shyong Yeo, Xiao-Shen Phang, Taejin Ha, Woontack Woo, and Aaron Quigley. 2017. TiTAN: exploring midair text entry using freehand input. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 3041–3049.

Digital Library

[44]

Xin Yi, Chun Yu, Mingrui Zhang, Sida Gao, Ke Sun, and Yuanchun Shi. 2015. ATK: Enabling ten-finger freehand typing in air based on 3d hand tracking data. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. 539–548.

Digital Library

[45]

Chun Yu, Yizheng Gu, Zhican Yang, Xin Yi, Hengliang Luo, and Yuanchun Shi. 2017. Tap, dwell or gesture? Exploring head-based text entry techniques for HMDs. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 4479–4488.

Digital Library

[46]

Difeng Yu, Kaixuan Fan, Heng Zhang, Diego Monteiro, Wenge Xu, and Hai-Ning Liang. 2018. PizzaText: Text entry for virtual reality systems using dual thumbsticks. IEEE transactions on visualization and computer graphics 24, 11(2018), 2927–2935.

Cited By

Zhang HLiang XLei YChen YHe ZZhang YChen LLin HHan TTian F(2024)Understanding the Effects of Restraining Finger Coactivation in Mid-Air Typing: from a Neuromechanical PerspectiveProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676441(1-18)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676441
Wu JWang ZWang LDuan YLi J(2024)FanPad: A Fan Layout Touchpad Keyboard for Text Entry in VR2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00045(222-232)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VR58804.2024.00045
Krasner AGabbard J(2024)MusiKeys: Exploring Haptic-to-Auditory Sensory Substitution to Improve Mid-Air Text-EntryIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337206530:5(2247-2256)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372065
Show More Cited By

Index Terms

Improving Finger Stroke Recognition Rate for Eyes-Free Mid-Air Typing in VR
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques
      1. Text input

Recommendations

Characterizing In-Air Eyes-Free Typing Movements in VR
VRST '20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology

We empirically explore fundamental requirements for achieving VR in-air typing by observing the unconstrained eyes-free in-air typing of touch typists. We show that unconstrained typing movements differ substantively from previously observed constrained ...
The potential of dwell-free eye-typing for fast assistive gaze communication
ETRA '12: Proceedings of the Symposium on Eye Tracking Research and Applications

We propose a new research direction for eye-typing which is potentially much faster: dwell-free eye-typing. Dwell-free eye-typing is in principle possible because we can exploit the high redundancy of natural languages to allow users to simply look at ...
Design and evaluation of a dwell-free eye typing technique
CHI EA '14: CHI '14 Extended Abstracts on Human Factors in Computing Systems

Dwelling, activated through gaze fixation for a prolonged time, is an essential task to be performed to select keys from on-screen keyboard present in the eye typing interface. Normally fixation on a key takes sufficient time which slows down eye typing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

10459 pages

ISBN:9781450391573

DOI:10.1145/3491102

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA
,
Steven Drucker
Microsoft Research, USA
,
Julie Williamson
University of Glasgow, UK
,
Koji Yatani
University of Tokyo, Japan

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
519
Total Downloads

Downloads (Last 12 months)105
Downloads (Last 6 weeks)12

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang HLiang XLei YChen YHe ZZhang YChen LLin HHan TTian F(2024)Understanding the Effects of Restraining Finger Coactivation in Mid-Air Typing: from a Neuromechanical PerspectiveProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676441(1-18)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676441
Wu JWang ZWang LDuan YLi J(2024)FanPad: A Fan Layout Touchpad Keyboard for Text Entry in VR2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00045(222-232)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VR58804.2024.00045
Krasner AGabbard J(2024)MusiKeys: Exploring Haptic-to-Auditory Sensory Substitution to Improve Mid-Air Text-EntryIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337206530:5(2247-2256)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372065
Singh AKoundal D(2023)A State-of-Art in Mid-Air Handwriting Recognition techniques2023 Seventh International Conference on Image Information Processing (ICIIP)10.1109/ICIIP61524.2023.10537774(695-701)Online publication date: 22-Nov-2023
https://doi.org/10.1109/ICIIP61524.2023.10537774

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents