A Systematic Review of User Studies as a Basis for the Design of Systems for Automatic Sign Language Processing

Deaf persons, whether or not they are sign language users, make up one of various existing marginalized populations that historically have been socially and politically underrepresented. Unfortunately, this also happens in technology design. Conducting user studies in which marginalized populations are represented is a step towards guaranteeing their right to participate in choices and decisions that are made for, with, and by them. This article presents and discusses results from a Systematic Literature Review (SLR) of user studies in the design of systems for Automatic Sign Language Processing (ASLP). Following our SLR protocol, from 2,486 papers initially found, we applied inclusion and exclusion criteria to finally select 37 papers in our review. We excluded publications that were not full papers, were not related to our main topic of interest, or that reported results that had been updated by more recent papers. All the selected papers focus on user studies as a basis for the design of three major aspects of ASLP: generation (ASLG), recognition (ASLR), and translation (ASLT). With regard to our specific area of interest, we analyzed four areas related to our research questions: goals and research methods, types of user involvement in the interaction design life cycle, cultural and collaborative aspects, and other lessons learned from the primary studies under review. Salient findings from our analysis show that numerical scale questionnaires are the most frequently used research instruments, co-designing ASLP systems with sign language users is not a common practice (as potential users are included mostly in the evaluation phase), and only seldom are Deaf persons who are sign language users included as members of research teams. These findings point to the need of conducting more inclusive and qualitative research for, with and by Deaf persons who are sign language users.