article

Synthesizing multimodal utterances for conversational agents: Research Articles

Authors:

Stefan Kopp,

Ipke WachsmuthAuthors Info & Claims

Computer Animation and Virtual Worlds, Volume 15, Issue 1

Pages 39 - 52

Published: 01 March 2004 Publication History

Abstract

Conversational agents are supposed to combine speech with non-verbal modalities for intelligible multimodal utterances. In this paper, we focus on the generation of gesture and speech from XML-based descriptions of their overt form. An incremental production model is presented that combines the synthesis of synchronized gestural, verbal, and facial behaviors with mechanisms for linking them in fluent utterances with natural co-articulation and transition effects. In particular, an efficient kinematic approach for animating hand gestures from shape specifications is presented, which provides fine adaptation to temporal constraints that are imposed by cross-modal synchrony. Copyright © 2004 John Wiley & Sons, Ltd.

Cited By

View all

Wang ZZhang WYe LZeng DMei T(2024)Cross-Modal Quantization for Co-Speech Gesture GenerationIEEE Transactions on Multimedia10.1109/TMM.2024.340574326(10251-10263)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3405743
Oralbayeva NAly ASandygulova ABelpaeme T(2023)Data-driven Communicative Behaviour Generation: A SurveyACM Transactions on Human-Robot Interaction10.1145/360923513:1(1-39)Online publication date: 16-Aug-2023
https://dl.acm.org/doi/10.1145/3609235
de Wit JVogt PKrahmer E(2022)The Design and Observed Effects of Robot-performed Manual Gestures: A Systematic ReviewACM Transactions on Human-Robot Interaction10.1145/354953012:1(1-62)Online publication date: 19-Jul-2022
https://dl.acm.org/doi/10.1145/3549530
Show More Cited By

Recommendations

Expressive multimodal conversational acts for SAIBA agents
IVA'11: Proceedings of the 10th international conference on Intelligent virtual agents

We discuss here the need to define what we call an agent conversational language, a language for Embodied Conversational Agents (ECA) to have conversations with a human. We propose a set of Expressive Multimodal Conversation Acts (EMCA), which is based ...
Multimodal backchannels for embodied conversational agents
IVA'10: Proceedings of the 10th international conference on Intelligent virtual agents

One of the most desirable characteristics of an Embodied Conversational Agent (ECA) is the capability of interacting with users in a human-like manner. While listening to a user, an ECA should be able to provide backchannel signals through visual and ...
Children's and adults' multimodal interaction with 2D conversational agents
CHI EA '05: CHI '05 Extended Abstracts on Human Factors in Computing Systems

Few systems combine both Embodied Conversational Agents (ECAs) and multimodal input. This research aims at modeling the behavior of adults and children during their multimodal interaction with ECAs. A Wizard-of-Oz setup was used and users were video-...

Comments

Information & Contributors

Information

Published In

cover image Computer Animation and Virtual Worlds

Computer Animation and Virtual Worlds Volume 15, Issue 1

March 2004

60 pages

ISSN:1546-4261

EISSN:1546-427X

Issue’s Table of Contents

Publisher

John Wiley and Sons Ltd.

United Kingdom

Publication History

Published: 01 March 2004

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

63
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Wang ZZhang WYe LZeng DMei T(2024)Cross-Modal Quantization for Co-Speech Gesture GenerationIEEE Transactions on Multimedia10.1109/TMM.2024.340574326(10251-10263)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3405743
Oralbayeva NAly ASandygulova ABelpaeme T(2023)Data-driven Communicative Behaviour Generation: A SurveyACM Transactions on Human-Robot Interaction10.1145/360923513:1(1-39)Online publication date: 16-Aug-2023
https://dl.acm.org/doi/10.1145/3609235
de Wit JVogt PKrahmer E(2022)The Design and Observed Effects of Robot-performed Manual Gestures: A Systematic ReviewACM Transactions on Human-Robot Interaction10.1145/354953012:1(1-62)Online publication date: 19-Jul-2022
https://dl.acm.org/doi/10.1145/3549530
Pelachaud CBusso CHeylen D(2021)Multimodal Behavior Modeling for Socially Interactive AgentsThe Handbook on Socially Interactive Agents10.1145/3477322.3477331(259-310)Online publication date: 10-Sep-2021
https://dl.acm.org/doi/10.1145/3477322.3477331
Kucherenko TNagy RJonell PNeff MKjellström HHenter G(2021)Speech2Properties2GesturesProceedings of the 21st ACM International Conference on Intelligent Virtual Agents10.1145/3472306.3478333(145-147)Online publication date: 14-Sep-2021
https://dl.acm.org/doi/10.1145/3472306.3478333
Wang SAlexanderson SGustafson JBeskow JHenter GSzékely É(2021)Integrated Speech and Gesture SynthesisProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479914(177-185)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3462244.3479914
Cafaro APelachaud CMarsella S(2019)Nonverbal behavior in multimodal performancesThe Handbook of Multimodal-Multisensor Interfaces10.1145/3233795.3233803(219-262)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3233795.3233803
Pons Lelardeux CPanzoli DLubrano VMinville VLagarrigue PJessel J(2017)Communication system and team situation awareness in a multiplayer real-time learning environmentThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-016-1280-633:4(489-515)Online publication date: 1-Apr-2017
https://dl.acm.org/doi/10.1007/s00371-016-1280-6
Wang YTree JWalker MNeff M(2016)Assessing the Impact of Hand Motion on Virtual Character PersonalityACM Transactions on Applied Perception10.1145/287435713:2(1-23)Online publication date: 3-Mar-2016
https://dl.acm.org/doi/10.1145/2874357
Bozkurt EYemez YErzin E(2016)Multimodal analysis of speech and arm motion for prosody-driven synthesis of beat gesturesSpeech Communication10.1016/j.specom.2016.10.00485:C(29-42)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1016/j.specom.2016.10.004
Show More Cited By

Abstract

Cited By

Recommendations

Expressive multimodal conversational acts for SAIBA agents

Multimodal backchannels for embodied conversational agents

Children's and adults' multimodal interaction with 2D conversational agents

Comments

Information

Published In

Publisher

Publication History

Author Tags

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations