abstract

Integration of DNN Generated Spontaneous Reactions with a Generic Multimodal Framework for Embodied Conversational Agents

Authors:

Hung-Hsuan Huang,

Masato Fukuda,

Stef van der Struijk,

Toyoaki NishidaAuthors Info & Claims

HAI '18: Proceedings of the 6th International Conference on Human-Agent Interaction

Pages 371 - 373

https://doi.org/10.1145/3284432.3287190

Published: 04 December 2018 Publication History

Get Access

Abstract

This paper describes the recent extensions of our previously proposed GECA Framework in incorporating the interconnection with GECA original networking library and ZeroMQ message passing library, the connection with Python based DNN developing API, Keras, and a character animator supporting FACS based detailed facial animations and lip-syncing with real-time generated voice tracks.

References

[1]

Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2017. Multimodal Machine Learning: A Survey and Taxonomy. CoRR, Vol. abs/1705.09406 (2017). arxiv: 1705.09406 http://arxiv.org/abs/1705.09406

Google Scholar

[2]

Paul Ekman, Wallace V. Friesen, and Joseph C. Hager. 2002. Facial Action Coding System (FACS). Website. http://www.face-and-emotion.com/dataface/facs/description.jsp

Google Scholar

[3]

Hung-Hsuan Huang, Aleksandra Cerekovic, Yukiko Nakano, Igor S. Pandzic, and Toyoaki Nishida. 2008a. The Design of a Generic Framework for Integrating ECA Components. In The 7th International Conference of Autonomous Agents and Multiagent Systems (AAMAS'08), Lin Padgham, David Parkes, and Jorg P. Muller (Eds.). ACM, IFAAMAS, Estorial, Portugal, 128--135.

Digital Library

Google Scholar

[4]

Hung-Hsuan Huang, Aleksandra Cerekovic, Kateryna Tarasenko, Vjekoslav Levacic, Goranka Zoric, Igor S. Pandzic, Yukiko Nakano, and Toyoaki Nishida. 2008b. Integrating Embodied Conversational Agent Components with a Generic Framework. Multiagent and Grid Systems, Vol. 4, 4 (Dec. 2008), 371--386.

Digital Library

Google Scholar

[5]

David R. Traum and Staffan Larsson. 2003. The information state approach to dialogue management. Kluwer, Dordrecht, 325--353.

Google Scholar

[6]

Stef van der Struijk, Hung-Hsuan Huang, Maryam Sadat Mirzaei, and Toyoaki Nishida. 2018. FACSvatar: An Open Source Modular Framework for Real-Time FACS based Facial Animation. In 18th ACM International Conference on Intelligent Virtual Agents (IVA 2018). Sydney, Australia.

Digital Library

Google Scholar

Cited By

View all

Huang HFukuda MNishida TOka NKoda TObaid MNakanishi HMubin OTanaka K(2019)An Investigation on the Effectiveness of Multimodal Fusion and Temporal Feature Extraction in Reactive and Spontaneous Behavior Generative RNN Models for Listener AgentsProceedings of the 7th International Conference on Human-Agent Interaction10.1145/3349537.3351908(89-96)Online publication date: 25-Sep-2019
https://dl.acm.org/doi/10.1145/3349537.3351908
Huang HFukuda MNishida TPelachaud CMartin JBuschmeier HLucas GKopp S(2019)Development of a Platform for RNN Driven Multimodal Interaction with Embodied Conversational AgentsProceedings of the 19th ACM International Conference on Intelligent Virtual Agents10.1145/3308532.3329448(200-202)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3308532.3329448
Huang HFukuda MNishida T(2019)Toward RNN Based Micro Non-verbal Behavior Generation for Virtual Listener AgentsSocial Computing and Social Media. Design, Human Behavior and Analytics10.1007/978-3-030-21902-4_5(53-63)Online publication date: 8-Jun-2019
https://doi.org/10.1007/978-3-030-21902-4_5

Index Terms

Integration of DNN Generated Spontaneous Reactions with a Generic Multimodal Framework for Embodied Conversational Agents
1. Computer systems organization
  1. Real-time systems
    1. Real-time system architecture
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools
      1. User interface toolkits

Recommendations

Development of a Platform for RNN Driven Multimodal Interaction with Embodied Conversational Agents
IVA '19: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents

This paper describes our ongoing project to build a platform that enables real-time multimodal interaction with embodied conversational agents. All of the components are in modular design and can be switched to other models easily. A prototype listener ...
Embodied conversational agents: computing and rendering realistic gaze patterns
PCM'06: Proceedings of the 7th Pacific Rim conference on Advances in Multimedia Information Processing

We describe here our efforts for modeling multimodal signals exchanged by interlocutors when interacting face-to-face. This data is then used to control embodied conversational agents able to engage into a realistic face-to-face interaction with human ...
Personality Analysis of Embodied Conversational Agents
IVA '18: Proceedings of the 18th International Conference on Intelligent Virtual Agents

People tend to personify machines. Giving machines the ability to actually produce social information can help improve human-machine interactions. Embodied Conversational Agents (ECAs) are virtual software agents that can process and produce speech, ...

Comments

Information & Contributors

Information

Published In

HAI '18: Proceedings of the 6th International Conference on Human-Agent Interaction

December 2018

402 pages

ISBN:9781450359535

DOI:10.1145/3284432

General Chairs:
Michita Imai
Keio University, Japan
,
Tim Norman
University of Southampton, UK
,
Program Chairs:
Elizabeth Sklar
King's College London, UK
,
Takanori Komatsu
Meiji University, Japan

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2018

Check for updates

Author Tags

Qualifiers

Abstract

Conference

HAI '18

Sponsor:

SIGCHI

HAI '18: 6th International Conference on Human-Agent Interaction

December 15 - 18, 2018

Southampton, United Kingdom

Acceptance Rates

HAI '18 Paper Acceptance Rate 40 of 92 submissions, 43%;

Overall Acceptance Rate 121 of 404 submissions, 30%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
77
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Huang HFukuda MNishida TOka NKoda TObaid MNakanishi HMubin OTanaka K(2019)An Investigation on the Effectiveness of Multimodal Fusion and Temporal Feature Extraction in Reactive and Spontaneous Behavior Generative RNN Models for Listener AgentsProceedings of the 7th International Conference on Human-Agent Interaction10.1145/3349537.3351908(89-96)Online publication date: 25-Sep-2019
https://dl.acm.org/doi/10.1145/3349537.3351908
Huang HFukuda MNishida TPelachaud CMartin JBuschmeier HLucas GKopp S(2019)Development of a Platform for RNN Driven Multimodal Interaction with Embodied Conversational AgentsProceedings of the 19th ACM International Conference on Intelligent Virtual Agents10.1145/3308532.3329448(200-202)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3308532.3329448
Huang HFukuda MNishida T(2019)Toward RNN Based Micro Non-verbal Behavior Generation for Virtual Listener AgentsSocial Computing and Social Media. Design, Human Behavior and Analytics10.1007/978-3-030-21902-4_5(53-63)Online publication date: 8-Jun-2019
https://doi.org/10.1007/978-3-030-21902-4_5

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Development of a Platform for RNN Driven Multimodal Interaction with Embodied Conversational Agents

Embodied conversational agents: computing and rendering realistic gaze patterns

Personality Analysis of Embodied Conversational Agents

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations