How Do People with Limited Movement Personalize Upper-Body Gestures? Considerations for the Design of Personalized and Accessible Gesture Interfaces

A corrigendum was issued for this paper on February 13, 2024. You can download the corrigendum from the Supplemental Material section of this citation page

Always-on, upper-body input from sensors like accelerometers, infrared cameras, and electromyography hold promise to enable accessible gesture input for people with upper-body motor impairments. When these sensors are distributed across the person’s body, they can enable the use of varied body parts and gestures for device interaction. Personalized upper-body gestures that enable input from diverse body parts including the head, neck, shoulders, arms, hands and fingers and match the abilities of each user, could be useful for ensuring that gesture systems are accessible. In this work, we characterize the personalized gesture sets designed by 25 participants with upper-body motor impairments and develop design recommendations for upper-body personalized gesture interfaces. We found that the personalized gesture sets that participants designed were highly ability-specific. Even within a specific type of disability, there were significant differences in what muscles participants used to perform upper-body gestures, with some predominantly using shoulder and upper-arm muscles, and others solely using their finger muscles. Eight percent of gestures that participants designed were with their head, neck, and shoulders, rather than their hands and fingers, demonstrating the importance of tracking the whole upper-body. To combat fatigue, participants performed 51% of gestures with their hands resting on or barely coming off of their armrest, highlighting the importance of using sensing mechanisms that are agnostic to the location and orientation of the body. Lastly, participants activated their muscles but did not visibly move during 10% of the gestures, demonstrating the need for using sensors that can sense muscle activations without movement. Both inertial measurement unit (IMU) and electromyography (EMG) wearable sensors proved to be promising sensors to differentiate between personalized gestures. Personalized upper-body gesture interfaces that take advantage of each person’s abilities are critical for enabling accessible upper-body gestures for people with upper-body motor impairments.