research-article

Towards Automated Sign Language Production: A Pipeline for Creating Inclusive Virtual Humans

PETRA '22: Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments

Pages 260 - 268

https://doi.org/10.1145/3529190.3529202

Published: 11 July 2022 Publication History

Abstract

In everyday life, Deaf People face barriers because information is often only available in spoken or written language. Producing sign language videos showing a human interpreter is often not feasible due to the amount of data required or because the information changes frequently. The ongoing AVASAG project addresses this issue by developing a 3D sign language avatar for the automatic translation of texts into sign language for public services. The avatar is trained using recordings of human interpreters translating text into sign language. For this purpose, we create a corpus with video and motion capture data and an annotation scheme that allows for real-time translation and subsequent correction without requiring to correct the animation frames manually. This paper presents the general translation pipeline focusing on innovative points, such as adjusting an existing annotation system to the specific requirements of sign language and making it usable to annotators from the Deaf communities.

References

[1]

Melissa L Anderson, Timothy Riker, Stephanie Hakulin, Jonah Meehan, Kurt Gagne, Todd Higgins, Elizabeth Stout, Emma Pici-D’Ottavio, Kelsey Cappetta, and Kelly S Wolf Craig. 2020. Deaf ACCESS: Adapting Consent Through Community Engagement and State-of-the-Art Simulation. The Journal of Deaf Studies and Deaf Education 25, 1 (2020), 115–125.

[2]

AVASAG consortium. 2021. AVASAG project web page. Charamel GmbH. Retrieved August 20, 2021 from https://avasag.de

[3]

Tobias Baur, Alexander Heimerl, Florian Lingenfelser, Johannes Wagner, Michel F. Valstar, Björn Schuller, and Elisabeth André. 2020. eXplainable Cooperative Machine Learning with NOVA. KI - Künstliche Intelligenz 34, 2 (2020), 143–164.

[4]

Penny Braem. 1995. Einführung in die Gebärdensprache und ihre Erforschung. Signum-Verlag, Hamburg, Germany.

[5]

Charamel GmbH. 2021. Structure of StoryBuilder. Charamel GmbH. Retrieved December 23, 2021 from https://vuppetmaster.de/documentation/docs/storybuilder/2_structure/

[6]

Charamel GmbH. 2021. VuppetMaster web page. Charamel GmbH. Retrieved August 20, 2021 from https://vuppetmaster.de

[7]

Torkil Clemmensen. 2004. Four approaches to user modelling—a qualitative research interview study of HCI professionals’ practice. Interacting with Computers 16, 4 (2004), 799–829.

[8]

Alan Cooper. 2004. The inmates are running the asylum: Why high-tech products drive us crazy and how to restore the sanity(2 ed.). Sams Publishing, Indianapolis.

[9]

Arie Croitoru, Peggy Agouris, and Anthony Stefanidis. 2005. 3D trajectory matching by pose normalization. In Proceedings of the 2005 international workshop on Geographic information systems - GIS ’05. ACM Press, Bremen, Germany, 153.

Digital Library

[10]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.

[11]

Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kruengkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si, and Chunyan Miao. 2020. DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 6045–6057. https://doi.org/10.18653/v1/2020.emnlp-main.488

[12]

Sarah Ebling and John Glauert. 2016. Building a Swiss German Sign Language avatar with JASigning and evaluating it among the Deaf community. Universal Access in the Information Society 15, 4 (2016), 577–587.

Digital Library

[13]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems 27 (2014), 2672–2680.

[14]

Thomas Hanke. 2004. HamNoSys—Representing sign language data in language resources and language processing contexts. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC’04), Vol. 4. ELRA, Lisbon, Portugal, 1–6.

[15]

Tharon Howard. 2014. Journey mapping: A brief overview. Communication Design Quarterly Review 2, 3 (2014), 10–13.

Digital Library

[16]

Matt Huenerfauth and Vicki Hanson. 2009. Sign language in the interface: access for deaf signers. Universal Access Handbook. NJ: Erlbaum 38 (2009), 14.

[17]

Vince Jennings, Ralph Elliott, Richard Kennaway, and John Glauert. 2010. Requirements for a signing avatar. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC’04). ELRA, Valletta, Malta, 133–136.

[18]

Trevor Johnston. 2010. From archive to corpus: Transcription and annotation in the creation of signed language corpora. International journal of corpus linguistics 15, 1 (2010), 106–131.

[19]

Plinio Thomaz Aquino Junior and Lucia Vilela Leite Filgueiras. 2005. User modeling with personas. In CLIHC ’05: Proceedings of the 2005 Latin American conference on Human-computer interaction. ACM Press, Cuernavaca, 277–282.

Digital Library

[20]

Michael Kipp, Quan Nguyen, Alexis Heloir, and Silke Matthes. 2011. Assessing the deaf user perspective on sign language avatars. In The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility. ACM, ACM Press, New York, USA, 107–114.

Digital Library

[21]

Vincenzo Lombardo, Cristina Battaglino, Rossana Damiano, and Fabrizio Nunnari. 2011. An Avatar-based Interface for the Italian Sign Language. In 2011 International Conference on Complex, Intelligent, and Software Intensive Systems. IEEE Computer Society, Washington, DC, USA, 589–594. https://doi.org/10.1109/CISIS.2011.97

Digital Library

[22]

Ceil Lucas, Clayton Valli, and Robert Bayley. 2002. Sociolinguistic Variation in American Sign Language. Bibliovault OAI Repository, the University of Chicago Press 24, 4(2002).

[23]

Michael McKee, Deirdre Schlehofer, and Denise Thew. 2013. Ethical Issues in Conducting Research With Deaf Populations. American Journal of Public Health 103, 12 (2013), 2174–2178.

[24]

George A Miller. 1995. WordNet: a lexical database for English. Commun. ACM 38, 11 (1995), 39–41.

Digital Library

[25]

MongoDB, Inc.2021. MongoDB. MongoDB, Inc. Retrieved December 23, 2021 from https://www.mongodb.com/

[26]

NaturalPoint. 2021. OptiTrack. NaturalPoint. Retrieved December 23, 2021 from https://optitrack.com/

[27]

John Pruitt and Tamara Adlin. 2006. The Persona Lifecycle. Elsevier, San Francisco.

[28]

Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2020. Progressive Transformers for End-to-End Sign Language Production. In European Conference on Computer Vision. Springer, Berlin/Heidelberg, Germany, 687–705.

[29]

Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2021. Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks. International Journal of Computer Vision 129, 7 (2021), 2113–2135.

Digital Library

[30]

Stephanie Stoll, Necati Cihan Camgoz, Simon Hadfield, and Richard Bowden. 2020. Text2Sign: Towards sign language production using neural machine translation and generative adversarial networks. International Journal of Computer Vision 128, 4 (2020), 891–908.

Digital Library

[31]

Amelie Unger, Dieter Wallach, and Nicole Jochems. 2021. Lost in Translation: Challenges and Barriers to Sign Language-Accessible User Research. (2021). To be presented at The 23rd International ACM SIGACCESS Conference on Computers and Accessibility.

[32]

Amelie Unger, Dieter P. Wallach, and Nicole Jochems. 2021. Lost in Translation: Challenges and Barriers to Sign Language-Accessible User Research. In The 23rd International ACM SIGACCESS Conference on Computers and Accessibility. Association for Computing Machinery, New York, NY, USA, 1–5. https://doi.org/10.1145/3441852.3476473

Digital Library

[33]

Oliver van Kaick, Hao Zhang, Ghassan Hamarneh, and Daniel Cohen-Or. 2011. A Survey on Shape Correspondence. Computer Graphics Forum 30, 6 (2011), 1681–1707.

[34]

Jason Wei and Kai Zou. 2019. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 6382–6388.

[35]

World Federation of the Deaf. 2021. Our Work. World Federation of the Deaf. Retrieved September 27, 2021 from https://wfdeaf.org/our-work/

[36]

World Health Organization. 2021. Deafness and hearing loss. World Health Organization. Retrieved September 27, 2021 from https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss

Cited By

Waldow KFuhrmann ARoth D(2024)Deep Neural Labeling: Hybrid Hand Pose Estimation Using Unlabeled Motion Capture Data With Color Gloves in Context of German Sign Language2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR59861.2024.00009(1-10)Online publication date: 17-Jan-2024
https://doi.org/10.1109/AIxVR59861.2024.00009
Nolte ALueneburg KWallach DJochems N(2022)Creating Personas for Signing User Populations: An Ability-Based Approach to User Modelling in HCIProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3550364(1-6)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3550364

Index Terms

Towards Automated Sign Language Production: A Pipeline for Creating Inclusive Virtual Humans

Index terms have been assigned to the content through auto-classification.

Recommendations

JUMLA-QSL-22: Creation and Annotation of a Qatari Sign Language Corpus for Sign Language Processing
PETRA '23: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments

Sign language processing (SLP) is essential to creating digital assistive technology for deaf people and requires the availability of large-scale corpus for accurate systems. However, large-scale publicly available datasets for sign languages, ...
A survey on recent advances in Sign Language Production
Abstract
Sign Language is the dominant form of communication language used in the Deaf and hearing-impaired community. To make easy and mutual communication between the hearing-impaired and the hearing communities, building a robust system capable of ...
Graphical abstract

Display Omitted
Highlights
- Vision and language modalities are mostly used input modalities in SLP.
- The lack of a large annotated dataset is a major challenge in SLP.
- The proposed works in SLP can be categorized into five categories.
- One limitation in SLP ...
Sign language animation using TVML
VRCAI '10: Proceedings of the 9th ACM SIGGRAPH Conference on Virtual-Reality Continuum and its Applications in Industry

The deaf communities in Japan use a visual language called Japanese Sign Language (JSL) as their first language. Learning to read Japanese can be difficult for some deaf people, because JSL has a totally different grammar from Japanese. NHK broadcasts ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PETRA '22: Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments

June 2022

704 pages

ISBN:9781450396318

DOI:10.1145/3529190

Conference Chair:
Fillia Makedon

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Bundesministerium für Bildung und Forschung

Conference

PETRA '22

PETRA '22: The15th International Conference on PErvasive Technologies Related to Assistive Environments

June 29 - July 1, 2022

Corfu, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
272
Total Downloads

Downloads (Last 12 months)84
Downloads (Last 6 weeks)6

Reflects downloads up to 30 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Waldow KFuhrmann ARoth D(2024)Deep Neural Labeling: Hybrid Hand Pose Estimation Using Unlabeled Motion Capture Data With Color Gloves in Context of German Sign Language2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR59861.2024.00009(1-10)Online publication date: 17-Jan-2024
https://doi.org/10.1109/AIxVR59861.2024.00009
Nolte ALueneburg KWallach DJochems N(2022)Creating Personas for Signing User Populations: An Ability-Based Approach to User Modelling in HCIProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3550364(1-6)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3550364

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents