research-article

Multimodal analysis of vocal collaborative search: a public corpus and results

Authors:

Mary Czerwinski,

Nick CraswellAuthors Info & Claims

ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

Pages 456 - 463

https://doi.org/10.1145/3136755.3136813

Published: 03 November 2017 Publication History

Abstract

Intelligent agents have the potential to help with many tasks. Information seeking and voice-enabled search assistants are becoming very common. However, there remain questions as to the extent by which these agents should sense and respond to emotional signals. We designed a set of information seeking tasks and recruited participants to complete them using a human intermediary. In total we collected data from 22 pairs of individuals, each completing five search tasks. The participants could communicate only using voice, over a VoIP service. Using automated methods we extracted facial action, voice prosody and linguistic features from the audio-visual recordings. We analyzed the characteristics of these interactions that correlated with successful communication and understanding between the pairs. We found that those who were expressive in channels that were missing from the communication channel (e.g., facial actions and gaze) were rated as communicating poorly, being less helpful and understanding. Having a way of reinstating nonverbal cues into these interactions would improve the experience, even when the tasks are purely information seeking exercises. The dataset used for this analysis contains over 15 hours of video, audio and transcripts and reported ratings. It is publicly available for researchers at: http://aka.ms/MISCv1.

References

[1]

Irman Abdic, Lex Fridman, Daniel McDuff, Erik Marchi, Bryan Reimer, and Björn Schuller. 2016. Driver Frustration Detection from Audio and Video in the Wild. KI 2016: Advances in Artificial Intelligence (2016), 237.

Digital Library

[2]

Ioannis Arapakis, Joemon M Jose, and Philip D Gray. 2008. Affective feedback: an investigation into the role of emotions in the information seeking process. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 395–402.

Digital Library

[3]

Ioannis Arapakis, Ioannis Konstas, and Joemon M Jose. 2009. Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance. In Proceedings of the 17th ACM international conference on Multimedia. ACM, 461–470.

Digital Library

[4]

Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2017. Multimodal Machine Learning: A Survey and Taxonomy. arXiv preprint arXiv:1705.09406 (2017).

[5]

Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on. IEEE, 1–10.

[6]

Justine Cassell and Kristinn R Thorisson. 1999. The power of a nod and a glance: Envelope vs. emotional feedback in animated conversational agents. Applied Artificial Intelligence 13, 4-5 (1999), 519–538.

[7]

Sidney D’Mello and Jacqueline Kory. 2012. Consistent but modest: a metaanalysis on unimodal and multimodal affect detection accuracies from 30 studies. In Proceedings of the 14th ACM international conference on Multimodal interaction. ACM, 31–38.

Digital Library

[8]

Paul Ekman, Wallace V Friesen, and John Hager. 2002. Facial action coding system: A technique for the measurement of facial movement. Research Nexus, Salt Lake City, UT. ICMI’17, November 13–17, 2017, Glasgow, UK Daniel McDuff, Paul Thomas, Mary Czerwinski, and Nick Craswell

[9]

Paul Ekman and Erika L Rosenberg. 1997. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA.

[10]

Florian Eyben, Felix Weninger, Florian Gross, and Björn Schuller. 2013. Recent developments in opensmile, the munich open-source multimedia feature extractor. In Proceedings of the 21st ACM international conference on Multimedia. ACM, 835– 838.

Digital Library

[11]

Jonathan Gratch, Ning Wang, Jillian Gerten, Edward Fast, and Robin Duffy. 2007. Creating rapport with virtual agents. In International Workshop on Intelligent Virtual Agents. Springer, 125–138.

Digital Library

[12]

Jing Han, Zixing Zhang, Fabien Ringeval, and Björn Schuller. 2017. Predictionbased learning for continuous emotion recognition in speech. In 42nd IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017.

[13]

Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology 52 (1988), 139–183.

[14]

Natasha Jaques, Daniel McDuff, Yoo Lim Kim, and Rosalind Picard. 2016. Understanding and Predicting Bonding in Conversations Using Thin Slices of Facial Expressions and Body Language. In International Conference on Intelligent Virtual Agents. Springer, 64–74.

[15]

Tom Johnstone and Klaus R Scherer. 2000. Vocal communication of emotion. Handbook of emotions 2 (2000), 220–235.

[16]

Ashish Kapoor, Winslow Burleson, and Rosalind W Picard. 2007. Automatic prediction of frustration. International journal of human-computer studies 65, 8 (2007), 724–736.

Digital Library

[17]

Karim Sadik Kassam. 2010. Assessment of emotional experience through facial expression. Harvard University.

[18]

Daniel Jonathan McDuff. 2014. Crowdsourcing affective responses for predicting media effectiveness. Ph.D. Dissertation. Massachusetts Institute of Technology.

[19]

Kate G Niederhoffer and James W Pennebaker. 2002. Linguistic style matching in social interaction. Journal of Language and Social Psychology 21, 4 (2002), 337–360.

[20]

James W Pennebaker, Roger J Booth, and Martha E Francis. 2007. Linguistic inquiry and word count: LIWC {Computer software}. Austin, TX: liwc. net (2007).

[21]

Rosalind W Picard. 1997. Affective computing. Vol. 252. MIT press Cambridge.

Digital Library

[22]

Fabien Ringeval, Andreas Sonderegger, Juergen Sauer, and Denis Lalanne. 2013. Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions. In Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on. IEEE, 1–8.

[23]

Stefan Scherer, Giota Stratou, Marwa Mahmoud, Jill Boberg, Jonathan Gratch, Albert Rizzo, and Louis-Philippe Morency. 2013. Automatic behavior descriptors for psychological disorder analysis. In Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on. IEEE, 1–8.

[24]

Paul Thomas, Daniel McDuff, Mary Czerwinski, and Nick Craswell. 2017. MISC: A data set of information-seeking conversations. In Proc. Int. W’shop on Conversational Approaches to Information Retrieval.

[25]

Matthew Turk. 2014. Multimodal interaction: A review. Pattern Recognition Letters 36 (2014), 189–195.

Digital Library

[26]

David Watson, Lee A Clark, and Auke Tellegen. 1988. Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of personality and social psychology 54, 6 (1988), 1063.

Cited By

Schneider BSung GChng EYang S(2021)How Can High-Frequency Sensors Capture Collaboration? A Review of the Empirical Links between Multimodal Metrics and Collaborative ConstructsSensors10.3390/s2124818521:24(8185)Online publication date: 8-Dec-2021
https://doi.org/10.3390/s21248185
Spina DTrippas JThomas PJoho HByström KClark LCraswell NCzerwinski MElsweiler DFrummet AGhosh SKiesel JLopatovska IMcDuff DMeyer SMourad AOwoicho PCherumanal SRussell DSitbon L(2021)Report on the future conversations workshop at CHIIR 2021ACM SIGIR Forum10.1145/3476415.347642155:1(1-22)Online publication date: 16-Jul-2021
https://dl.acm.org/doi/10.1145/3476415.3476421
Thomas PCzerwinksi MMcduff DCraswell N(2021)Theories of Conversation for Conversational IRACM Transactions on Information Systems10.1145/343986939:4(1-23)Online publication date: 16-Aug-2021
https://dl.acm.org/doi/10.1145/3439869
Show More Cited By

Index Terms

Multimodal analysis of vocal collaborative search: a public corpus and results
1. Computing methodologies
  1. Artificial intelligence
    1. Distributed artificial intelligence
      1. Intelligent agents
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Location grounding in multimodal local search
ICMI-MLMI '10: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction

Computational models of dialog context have often focused on unimodal spoken dialog or text, using the language itself as the primary locus of contextual information. But as we move from spoken interaction to situated multimodal interaction on mobile ...
Multimodal interaction patterns in mobile local search
IUI '12: Proceedings of the 2012 ACM international conference on Intelligent User Interfaces

Speak4it™ is a mobile search application that leverages multimodal input and integration to allow users to search for and act on local business information. We present an initial empirical analysis of user interaction with a multimodal local search ...
Multimodal dialogue in mobile local search
ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

Speak4it^SM is a multimodal, mobile search application that provides information about local businesses. Users can combine speech and touch input simultaneously to make search queries or commands to the application. For example, a user might say, "gas ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

November 2017

676 pages

ISBN:9781450355438

DOI:10.1145/3136755

General Chairs:
Edward Lank
University of Waterloo, Canada
,
Alessandro Vinciarelli
University of Glasgow, UK
,
Program Chairs:
Eve Hoggan
Aarhus University, Denmark
,
Sriram Subramanian
University of Sussex, UK
,
Stephen A. Brewster
University of Glasgow, UK

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '17

Sponsor:

SIGCHI

ICMI '17: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 13 - 17, 2017

Glasgow, UK

Acceptance Rates

ICMI '17 Paper Acceptance Rate 65 of 149 submissions, 44%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
195
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)3

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schneider BSung GChng EYang S(2021)How Can High-Frequency Sensors Capture Collaboration? A Review of the Empirical Links between Multimodal Metrics and Collaborative ConstructsSensors10.3390/s2124818521:24(8185)Online publication date: 8-Dec-2021
https://doi.org/10.3390/s21248185
Spina DTrippas JThomas PJoho HByström KClark LCraswell NCzerwinski MElsweiler DFrummet AGhosh SKiesel JLopatovska IMcDuff DMeyer SMourad AOwoicho PCherumanal SRussell DSitbon L(2021)Report on the future conversations workshop at CHIIR 2021ACM SIGIR Forum10.1145/3476415.347642155:1(1-22)Online publication date: 16-Jul-2021
https://dl.acm.org/doi/10.1145/3476415.3476421
Thomas PCzerwinksi MMcduff DCraswell N(2021)Theories of Conversation for Conversational IRACM Transactions on Information Systems10.1145/343986939:4(1-23)Online publication date: 16-Aug-2021
https://dl.acm.org/doi/10.1145/3439869
Ghosh SGhosh SScholer FThomas PElsweiler DJoho HKando NSmith C(2021)Classifying Speech Acts using Multi-channel Deep Attention Network for Task-oriented Conversational Search AgentsProceedings of the 2021 Conference on Human Information Interaction and Retrieval10.1145/3406522.3446057(267-272)Online publication date: 14-Mar-2021
https://dl.acm.org/doi/10.1145/3406522.3446057
Xing ZYuan XWu DHuang YMostafa J(2020)Understanding Voice Search Behavior: Review and Synthesis of ResearchHCI International 2020 - Late Breaking Papers: Multimodality and Intelligence10.1007/978-3-030-60117-1_23(305-320)Online publication date: 17-Oct-2020
https://doi.org/10.1007/978-3-030-60117-1_23
Liao QGeyer WMuller MKhazaen Y(2020)Conversational Interfaces for Information SearchUnderstanding and Improving Information Search10.1007/978-3-030-38825-6_13(267-287)Online publication date: 30-May-2020
https://doi.org/10.1007/978-3-030-38825-6_13
Hasan MSen TYang YBaten RHaut KHoque M(2019)LIWC into the Eyes: Using Facial Features to Contextualize Linguistic Analysis in Multimodal Communication2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)10.1109/ACII.2019.8925467(1-7)Online publication date: Sep-2019
https://doi.org/10.1109/ACII.2019.8925467

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents