abstract

Automatic Generation System of Virtual Agent's Motion using Natural Language

Authors:

Ryo Ishii,

Taichi Katayama,

Ryuichiro Higashinaka,

Junji TomitaAuthors Info & Claims

IVA '18: Proceedings of the 18th International Conference on Intelligent Virtual Agents

Pages 357 - 358

https://doi.org/10.1145/3267851.3267869

Published: 05 November 2018 Publication History

Get Access

Abstract

A virtual agent in a dialogue system should express appropriate body motions according to utterances and effectively communicate with a user. We previously proposed a generation model of whole body motions such as head direction, nodding, facial expressions, hand gestures, and upper-body posture accompanying utterances at appropriate times similar to humans by using various types of natural-language-analysis information obtained from spoken language. As an attempt to promote this model, we constructed an API that can easily generate motions by using the generation model and constructed a demonstration system that can automatically control a virtual agent from only the spoken language. When inputting an arbitrary utterance language, synthesized sound and motion information are acquired from the speech synthesizer and motion-generation API, and the vocalization of the virtual agent and animated motion are generated. A dialog agent that is more attractive and can communicate smoothly by automatically generating natural motions is expected.

References

[1]

Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. 2016. Open-Face: A general-purpose face recognition library with mobile applications. Technical Report. CMU-CS-16-118, CMU School of Computer Science.

Google Scholar

[2]

Justine Cassell, Hannes Högni Vilhjálmsson, and Timothy Bickmore. 2001. BEAT: The Behavior Expression Animation Toolkit. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01). ACM, New York, NY, USA, 477--486.

Digital Library

Google Scholar

[3]

C.-C Chiu, L.-P Morency, and Stacy Marsella. 2015. Predicting Co-verbal Gestures: A Deep and Temporal Modeling Approach., 152--166 pages.

Google Scholar

[4]

NTT TechnoCross Corporation. 2018. FutureVoice. http://www.v-series.jp/futurevoice/

Google Scholar

[5]

Kenji Imamura. 2007. Analysis of Japanese dependency analysis of semi-spoken words by series labeling. In Proceedings of the Annual Meeting of the Association for Natural Language Processing. 518--521.

Google Scholar

[6]

Ryo Ishii, Taichi Katayama, Ryuichiro Higashinaka, and Junji Tomita. 2018. Generating Body Motions using Spoken Language in Dialogue. In Intelligent Virtual Agents (IVA'18).

Digital Library

Google Scholar

[7]

Ryo Ishii, Toshimitsu Miyajima, Kinya Fujita, and Yukiko I. Nakano. 2006. Avatar's Gaze Control to Facilitate Conversational Turn-Taking in Virtual-Space Multi-user Voice Chat System. In Intelligent Virtual Agents, 6th International Conference (IVA'06). 458.

Digital Library

Google Scholar

[8]

Manja Lohse, Reinier Rothuis, Jorge Gallego-Pérez, Daphne E. Karreman, and Vanessa Evers. 2014. Robot Gestures Make Difficult Tasks Easier: The Impact of Gestures on Perceived Workload and Task Performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 1459--1466.

Digital Library

Google Scholar

[9]

Toyomi Meguro, Ryuichiro Higashinaka, Yasuhiro Minami, and Kohji Dohsaka. 2010. Controlling listening-oriented dialogue using partially observable Markov decision processes. In International conference on computational linguistics. 761--769.

Digital Library

Google Scholar

[10]

Yukiko I. Nakano, Masashi Okamoto, Daisuke Kawahara, Qing Li, and Toyoaki Nishida. 2004. Converting Text into Agent Animations: Assigning Gestures to Text., 4 pages.

Google Scholar

[11]

Petra Wagner, Zofia Malisz, and Stefan Kopp. 2014. Gesture and speech in interaction: An overview. 57 (2014), 209--232.

Digital Library

Google Scholar

Cited By

View all

Ishii RMorikawa AEitoku SFukayama ANakamura TLugrin BLatoschik Mvon Mammen SKopp SPécune FPelachaud C(2023)How Far ahead Can Model Predict Gesture Pose from Speech and Spoken Text?Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents10.1145/3570945.3607336(1-3)Online publication date: 19-Sep-2023
https://dl.acm.org/doi/10.1145/3570945.3607336
Ishii RKatayama THigashinaka RTomita J(2021)Automatic Head-Nod Generation Using Utterance Text Considering Personality TraitsIncreasing Naturalness and Flexibility in Spoken Dialogue Interaction10.1007/978-981-15-9323-9_26(299-306)Online publication date: 11-Mar-2021
https://doi.org/10.1007/978-981-15-9323-9_26
Ishii RAhuja CNakano YMorency L(2020)Impact of Personality on Nonverbal Behavior GenerationProceedings of the 20th ACM International Conference on Intelligent Virtual Agents10.1145/3383652.3423908(1-8)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3383652.3423908

Index Terms

Automatic Generation System of Virtual Agent's Motion using Natural Language
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
  2. Human computer interaction (HCI)
    1. HCI theory, concepts and models
    2. Interaction paradigms
      1. Collaborative interaction

Recommendations

Generating Body Motions using Spoken Language in Dialogue
IVA '18: Proceedings of the 18th International Conference on Intelligent Virtual Agents

We propose a model to automatically generate whole body motions accompanying utterances at appropriate times, similar to humans, by using various types of natural-language-analysis information obtained from spoken language. Specifically, we focus on the ...
Mode and modal transfer effects on performance and discourse organization with an information retrieval dialogue system in natural language

This paper describes an experiment on the effects of learning, mode of interaction (written vs. spoken) and transfer mode on user performance and discourse organization during interaction with a natural language dialogue system. Forty-eight participants ...
A third modality of natural language?
Special issue on integration of natural language and vision processing: intelligent multimedia

Comments

Information & Contributors

Information

Published In

IVA '18: Proceedings of the 18th International Conference on Intelligent Virtual Agents

November 2018

381 pages

ISBN:9781450360135

DOI:10.1145/3267851

Conference Chairs:
Anton Bogdanovych
Western Sydney University
,
Deborah Richards
Macquarie University
,
Simeon Simoff
Western Sydney University
,
Program Chairs:
Catherine Pelachaud
CNRS - ISIR, Université Pierre et Marie Curie
,
Dirk Heylen
University of Twente
,
Tomas Trescak
Western Sydney University

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 November 2018

Check for updates

Author Tags

Qualifiers

Abstract
Research
Refereed limited

Conference

IVA '18

Sponsor:

SIGAI

IVA '18: International Conference on Intelligent Virtual Agents

November 5 - 8, 2018

NSW, Sydney, Australia

Acceptance Rates

IVA '18 Paper Acceptance Rate 17 of 82 submissions, 21%;

Overall Acceptance Rate 53 of 196 submissions, 27%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
136
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 24 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ishii RMorikawa AEitoku SFukayama ANakamura TLugrin BLatoschik Mvon Mammen SKopp SPécune FPelachaud C(2023)How Far ahead Can Model Predict Gesture Pose from Speech and Spoken Text?Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents10.1145/3570945.3607336(1-3)Online publication date: 19-Sep-2023
https://dl.acm.org/doi/10.1145/3570945.3607336
Ishii RKatayama THigashinaka RTomita J(2021)Automatic Head-Nod Generation Using Utterance Text Considering Personality TraitsIncreasing Naturalness and Flexibility in Spoken Dialogue Interaction10.1007/978-981-15-9323-9_26(299-306)Online publication date: 11-Mar-2021
https://doi.org/10.1007/978-981-15-9323-9_26
Ishii RAhuja CNakano YMorency L(2020)Impact of Personality on Nonverbal Behavior GenerationProceedings of the 20th ACM International Conference on Intelligent Virtual Agents10.1145/3383652.3423908(1-8)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3383652.3423908

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Generating Body Motions using Spoken Language in Dialogue

Mode and modal transfer effects on performance and discourse organization with an information retrieval dialogue system in natural language

A third modality of natural language?

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations