We present NEMSI, a cloud-based multimodal dialog system designed to have naturalistic interactions with individuals for the purpose of screening neurological or mental conditions. The system has been used by thousands of people capturing audio and video responses to open-ended questions and structured health surveys.

References

[1]

Andrea Bandini, Jordan Green, Brian Richburg, and Yana Yunusova. 2018. Automatic Detection of Orofacial Impairment in Stroke. Proc. Interspeech 2018 (2018), 1711--1715.

Crossref

Google Scholar

[2]

Anne Case and Angus Deaton. 2017. Mortality and morbidity in the 21st century. Brookings papers on economic activity 2017 (2017), 397.

Google Scholar

[3]

David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et al. 2014. SimSensei Kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 2014 international conference on Autonomous agents and multiagent systems. International Foundation for Autonomous Agents and Multiagent Systems, 1061--1068.

Digital Library

Google Scholar

[4]

Statista Dossier. 2017. Depression in the U.S. -- Statistics & Facts. Retrieved March 8, 2019.

Google Scholar

[5]

SG Gauthier. 2005. Alzheimer's disease: the benefits of early treatment. European Journal of Neurology 12 (2005), 11--16.

Crossref

Google Scholar

[6]

Jordan R Green, Kristen M Allison, Claire Cordella, Brian D Richburg, Gary L Pattee, James D Berry, Eric A Macklin, Erik P Pioro, and Richard A Smith. 2018. Additional evidence for a therapeutic effect of dextromethorphan/quinidine on bulbar motor function in patients with amyotrophic lateral sclerosis: A quantitative speech analysis. British journal of clinical pharmacology 84, 12 (2018), 2849--2856.

Google Scholar

[7]

Aron Halfin. 2007. Depression: the benefits of early and appropriate treatment. The American journal of managed care 13, 4 Suppl (2007), S92--7.

Google Scholar

[8]

Christine Lisetti, Reza Amini, and Ugan Yasavur. 2015. Now all together: overview of virtual health assistants emulating face-to-face health interview experience. KI-Künstliche Intelligenz 29, 2 (2015), 161--172.

Crossref

Google Scholar

[9]

Oliver Roesler and David Suendermann-Oeft. 2019. Towards visual behavior detection in human-machine conversations. In Proceedings of the 2019 international conference on Activity and behavior computing.

Crossref

Google Scholar

[10]

Christopher A Taylor, Sujay F Greenlund, Lisa C McGuire, Hua Lu, and Janet B Croft. 2017. Deaths from Alzheimer's Disease--United States, 1999--2014. MMWR. Morbidity and mortality weekly report 66, 20 (2017), 521.

Google Scholar

Cited By

View all

Ramanarayanan V(2024)Multimodal Technologies for Remote Assessment of Neurological and Mental HealthJournal of Speech, Language, and Hearing Research10.1044/2024_JSLHR-24-00142(1-13)Online publication date: 10-Jul-2024
https://doi.org/10.1044/2024_JSLHR-24-00142
Kothare HRamanarayanan VNeumann MLiscombe JRichter VLampinen LBai APreciado CBrogan KDemopoulos C(2024)Vocal and Facial Behavior During Affect Production in Autism Spectrum DisorderJournal of Speech, Language, and Hearing Research10.1044/2024_JSLHR-23-00080(1-16)Online publication date: 19-Dec-2024
https://doi.org/10.1044/2024_JSLHR-23-00080
Otero-González IPacheco-Lorenzo MFernández-Iglesias MAnido-Rifón L(2024)Conversational agents for depression screening: A systematic reviewInternational Journal of Medical Informatics10.1016/j.ijmedinf.2023.105272181(105272)Online publication date: Jan-2024
https://doi.org/10.1016/j.ijmedinf.2023.105272
Show More Cited By

Index Terms

NEMSI: A Multimodal Dialog System for Screening of Neurological or Mental Conditions
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Natural language interfaces

Recommendations

From vocal to multimodal dialogue management
ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces

Multimodal, speech-enabled systems pose different research problems when compared to unimodal, voice-only dialogue systems. One of the important issues is the question of how a multimodal interface should look like in order to make the multimodal ...
Facilitating multiparty dialog with gaze, gesture, and speech
ICMI-MLMI '10: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction

We study how synchronized gaze, gesture and speech rendered by an embodied conversational agent can influence the flow of conversations in multiparty settings. We begin by reviewing a computational framework for turn-taking that provides the foundation ...
Towards Conversationally Intelligent Dialog Systems
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

Spoken dialog systems, lacking the means to address the complex phenomena of spontaneous speech and conversational dynamics, force users into a constrained mode of dialog that resembles text-based interaction more closely than spoken conversation. Turn-...

Comments

Information & Contributors

Information

Published In

IVA '19: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents

July 2019

282 pages

ISBN:9781450366724

DOI:10.1145/3308532

General Chairs:
Catherine Pelachaud
CNRS-ISIR, Sorbonne Universite, France
,
Jean-Claude Martin
CNRS-LIMSI, Universite Paris Saclay, France
,
Program Chairs:
Hendrik Buschmeier
Bielefeld University, Germany
,
Gale Lucas
University of Southern California, USA
,
Stefan Kopp
Bielefeld University, Germany

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2019

Check for updates

Author Tags

Qualifiers

Demonstration

Conference

IVA '19

Sponsor:

SIGAI

IVA '19: ACM International Conference on Intelligent Virtual Agents

July 2 - 5, 2019

Paris, France

Acceptance Rates

IVA '19 Paper Acceptance Rate 15 of 63 submissions, 24%;

Overall Acceptance Rate 53 of 196 submissions, 27%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
184
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)3

Reflects downloads up to 23 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ramanarayanan V(2024)Multimodal Technologies for Remote Assessment of Neurological and Mental HealthJournal of Speech, Language, and Hearing Research10.1044/2024_JSLHR-24-00142(1-13)Online publication date: 10-Jul-2024
https://doi.org/10.1044/2024_JSLHR-24-00142
Kothare HRamanarayanan VNeumann MLiscombe JRichter VLampinen LBai APreciado CBrogan KDemopoulos C(2024)Vocal and Facial Behavior During Affect Production in Autism Spectrum DisorderJournal of Speech, Language, and Hearing Research10.1044/2024_JSLHR-23-00080(1-16)Online publication date: 19-Dec-2024
https://doi.org/10.1044/2024_JSLHR-23-00080
Otero-González IPacheco-Lorenzo MFernández-Iglesias MAnido-Rifón L(2024)Conversational agents for depression screening: A systematic reviewInternational Journal of Medical Informatics10.1016/j.ijmedinf.2023.105272181(105272)Online publication date: Jan-2024
https://doi.org/10.1016/j.ijmedinf.2023.105272
Neumann MKothare HRamanarayanan V(2024)Multimodal speech biomarkers for remote monitoring of ALS disease progressionComputers in Biology and Medicine10.1016/j.compbiomed.2024.108949180(108949)Online publication date: Sep-2024
https://doi.org/10.1016/j.compbiomed.2024.108949
Richter VNeumann MKothare HRoesler OLiscombe JSuendermann-Oeft DProkop SKhan AYavorsky CLindenmayer JRamanarayanan V(2022)Towards Multimodal Dialog-Based Speech & Facial Biomarkers of SchizophreniaCompanion Publication of the 2022 International Conference on Multimodal Interaction10.1145/3536220.3558075(171-176)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536220.3558075
Tanqueray LPaulsson TZhong MLarsson SCastellano G(2022)Gender Fairness in Social Robotics: Exploring a Future Care of Peripartum Depression2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)10.1109/HRI53351.2022.9889468(598-607)Online publication date: 7-Mar-2022
https://doi.org/10.1109/HRI53351.2022.9889468
Kothare HRoesler OBurke WNeumann MLiscombe JExner ASnyder SCornish AHabberstad DPautler DSuendermann-Oeft DHuber JRamanarayanan V(2022)Speech, Facial and Fine Motor Features for Conversation-Based Remote Assessment and Monitoring of Parkinson's Disease2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)10.1109/EMBC48229.2022.9871375(3464-3467)Online publication date: 11-Jul-2022
https://doi.org/10.1109/EMBC48229.2022.9871375
Zhong MBilal APapadopoulos FCastellano G(2021)Psychiatrists’ Views on Robot-Assisted Diagnostics of Peripartum DepressionSocial Robotics10.1007/978-3-030-90525-5_40(464-474)Online publication date: 2-Nov-2021
https://doi.org/10.1007/978-3-030-90525-5_40

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

From vocal to multimodal dialogue management

Facilitating multiparty dialog with gaze, gesture, and speech

Towards Conversationally Intelligent Dialog Systems