Article

Affective multimodal mirror: sensing and eliciting laughter

Authors:

Willem A. Melder,

Khiet P. Truong,

Marten Den Uyl,

David A. Van Leeuwen,

Mark A. Neerincx,

Lodewijk R. Loos,

B. Stock PlumAuthors Info & Claims

HCM '07: Proceedings of the international workshop on Human-centered multimedia

Pages 31 - 40

https://doi.org/10.1145/1290128.1290134

Published: 28 September 2007 Publication History

Abstract

In this paper, we present a multimodal affective mirror that senses and elicits laughter. Currently, the mirror contains a vocal and a facial affect-sensing module, a component that fuses the output of these two modules to achieve a user-state assessment, a user state transition model, and a component to present audiovisual affective feedback that should keep or bring the user in the intended state. Interaction with this intelligent interface involves a full cyclic process of sensing, interpreting, reacting, sensing (of the reaction effects), interpreting & The intention of the mirror is to evoke positive emotions, to make people laugh and to increase the laughter. The first user experiences tests showed that users show cooperative behavior, resulting in mutual user-mirror action-reaction cycles. Most users enjoyed the interaction with the mirror and immersed in an excellent user experience.

References

[1]

Aarts, E. Ambient intelligence: a multimedia perspective. IEEE Multimedia 11 (1), 2004, 12--19.

Digital Library

[2]

Andrés del Valle, A.C. and Opalach, A. The Persuasive Mirror. In Proceedings of Persuasive, 2006.

[3]

Bachorowski, J.A., amd Owren, M.J. (2001). Not all laugs are alike: Voiced but not unvoiced laughter readily elicits positive affect. Psychological Science, Vol. 12(3), 252--257.

[4]

Bradley, M.M. and Lang, P.J. Affective norms for English words (ANEW). Gainesville, FL. The NIMH Center for the Study of Emotion and Attention, University of Florida.

[5]

Brøndsted, T., Nielsen, T.D., and Ortega, S. Affective multimodal interaction with a 3d agent. In Proceedings of the 8th international workshop on the cognitive science of natural language processing, 1999, 102--109.

[6]

Clavel, C., Ehrette, T., and Richard, G. Events detection for an audio-based surveillance system. In Proceedings of IEEE International Conference on Multimedia and Expo (ICME 2005), 2005, 1306--1309.

[7]

Collobert, R., Bengio, S., and Mariéthoz, J. Torch: a modular machine learning software library. Technical Report IDIAP-RR 02-46, IDIAP, 2002.

[8]

Darrell, T., Gordon, G., Woodfill, J., and Harville, M. A virtual mirror interface using real-time robust face tracking. In Proceedings of the 3th International Conference on Face and Gesture Recognition, 1998.

Digital Library

[9]

Den Uyl, M.J. and Van Kuilenburg, H. The FaceReader: online facial expression recognition. In Proceedings of Measuring Behavior, 2005.

[10]

Duric, Z., Gray, W.D., Heishman, R., Li, F., Rosenfeld, A., Schoelles, M.J., Schunn, C., and Wechsler, H. Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction. In Proceedings of the IEEE, Vol.90 (7), 2002, 1272--1289.

[11]

Ex'ovision EyeCatcher: http://www.exovision.nl/download/EyeCatcher-leaflet.pdf

[12]

Grootjen, M., Neerincx, M.A., Weert, J.C.M., and Truong, K.P. Measuring cognitive task load on a naval ship: implications of a real world environment. In Proceedings of International Conference on Human-Computer Interaction (HCII'07), 2007.

[13]

Hauptmann, A.G. Speech and gestures for graphic image manipulation. In Proceedings of ACM conference on Human Factors in Computing Systems (CHI'89), 1989, 241--245.

Digital Library

[14]

Janin, A., Baron, D., Edwards, J., Ellis, D., Gelbart, D., Morgan, N., Peskin, B., Pfau, T., Shriberg, E., Stolcke, A., and Wooters, C. The ICSI meeting corpus. In Proceedings of IEEE International conference on acoustics, speech and signal processing (ICASSP'03), 2003, 364--367.

[15]

Kapoor, A. and Picard, R.W. Multimodal affect recognition in learning environments. In Proceedings of the 13th annual ACM International Conference on Multimedia, 2005, 677--682.

Digital Library

[16]

Kim, J., André, E., Rehm, M., Vogt, T., and Wagner, J.allIntegrating information from speech and physiological signals to achieve emotional sensitivity. In Proceedings of Interspeech, 2005.

[17]

Leatherby, J.H., and Pausch, R. Voice input as a replacement for keyboard accelerators in a mouse-based graphical editor: an empirical study. In Journal of the American Voice Input/Output Society 11 (2), 1992.

[18]

Lisetti, C.L. and Nasoz, F. MAUI: a multimodal affective user interface. In Proceedings of ACM Multimedia Interaction Conference 2002, 161--170.

Digital Library

[19]

Litman, D.J. and Forbes-Riley, K. Recognizing student emotions and attitudes on the basis of utterances in spoken tutoring dialogues with both human and computer tutors. Speech Communication 48 (5), 2006, 559--590.

[20]

Lundqvist, D., Flykt, A., and Öhman, A. The Karolinska Directed Emotional Faces-KDEF, Department of Clinical Neuroscience, Psychology section, Karolinska Institute.

[21]

Maat, L. and Pantic, M. Gaze-X: adaptive affective multimodal interface for single-user office scenarios. In Artificial Intelligence for Human Computing, Vol.4451, 2007, 251--271.

Digital Library

[22]

Merkx, P.A.B., Truong, K.P. and Neerincx, M.A. (2007). Inducing and measuring emotion through a multiplayer first-person shooter computer game. In: H.J. van den Herik, J.W.H.M. Uiterwijk, M.H.M. Winands, and M.P.D. Schadd (Eds.). Proceedings of the Computer Games Workshop 2007, Amsterdam, The Netherlands.

[23]

Neerincx, M.A. and Lindenberg, J. Situated cognitive engineering for complex task environments. In: Schraagen, J.M.(Ed.), Natural Decision Making & Macro cognition. Ashley.

[24]

Olivier, M., Benoit, M., Irène, K., Arman, S., and Jordi, A. Multimodal caricatural mirror. In Proceedings of ENTERFACE'05, 1st Summer Workshop on Multimodal Interfaces, 2005, 13--20.

[25]

Oviatt, S. User-centered modeling and evaluation of multimodal interfaces. In Proceedings of the IEEE, Vol.91 (9), 2003, 1457--1468.

[26]

Pantic, M. and Rothkrantz, L.J.M. Towards an affect-sensitive multimodal human-computer interaction. In Proceedings of the IEEE, Vol.91, 2003.

[27]

Pantic, M., Sebe, N., Cohn, J.F. and Huang, T. Affective multimodal human-computer interaction, In Proceedings of the 13th annual ACM international conference on multimedia, 2006.

Digital Library

[28]

Pellom, B., SONIC: The university of Colorado Continuous Speech Recognizer, University of Colorado, tech report#TR-CSLR-2001-01, Boulder, Colorado, March, 2001.

[29]

Picard, R.W. Affective computing. MIT Press, Cambridge, MA, 1997.

Digital Library

[30]

Prendinger, H., Mayer, S., Mori, J., and Ishizuka, M. Persona effect revisited: using bio-signal to measure and reflect the impact of character-based interfaces. In Proceedings of the Intelligent Virtual Agents 4th International Workshop (IVA'03), 2003.

[31]

Sebe, N., Cohen, I., Gevers, T., and Huang, T. Emotion recognition based on joint visual and audio cues. In Proceedings of the International Conference on Pattern Recognition (ICPR 2006), 2006, 1136--1139.

Digital Library

[32]

SPRACHcore. http://www.icsi.berkeley.edu/~dpwe/projects/sprach/

[33]

Sung, K.K. and Poggio, T. Example-based learning for view-based human face detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 20 (1), 1998, 39--51.

Digital Library

[34]

Truong, K.P. and Van Leeuwen, D.A. Automatic discrimination between laughter and speech, Speech Communication 49 (2), 2007, 144--158.

Digital Library

[35]

Truong, K.P, van Leeuwen, D.A., and Neerincx, M.A. (in press). Unobtrusive Multimodal Emotion Detection in Adaptive Interfaces: Speech and Facial Expressions. In: D.D. Schmorrow & L.M. Reeves (Eds), Foundations of Augmented Cognition, 3rd Edition, LNAI 4565 proceedings (10 pages), ISBN 978-3-540-73215-0.

Digital Library

[36]

V2_ institute for the unstable media, MUSH device v3: http://multimedian.v2.nl/mush/mushv3_documentation.pdf

[37]

Van Kuilenburg, H., Wiering, Marco, Den Uyl, M. A model based method for automatic facial expression recognition. In Proceedings of the European Conference on Machine Learning (ECML' 05), 2005, 194--205.

Digital Library

[38]

Vidrascu, L., and Devillers, L. Detection of real-life emotions in dialogs recorded in a call center. In Proceedings of Interspeech, 2005, 1841--1844.

[39]

Vogt, T. and André, E. Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. In Proceedings of IEEE International Conference on Multimedia & Expo (ICME'05), 2005.

[40]

Waag Society, KeyWorx: http://www.keyworx.org, Realizer: http://kwlive.dev.waag.org/realizer/doc/doxygen/html/

[41]

Wilting, J., Krahmer, E., and Swerts, M. Real vs. acted emotional speech. In Proceedings of Interspeech, 2006.

[42]

Yacoub, S., Simske, S.,Lin, X., and Burns, J. Recognition of emotions in interactive voice response systems. In Proceedings of Eurospeech, 2003, 729--732.

[43]

Yang, G., Lin, Y., and Bhattacharya, P. A driver fatigue recognition model using fusion of multiple features. In IEEE SMC, Vol.2, 2005, 1777--1784.

[44]

Zeng, Z., Hu, Y., Fu, Y., Huang, T.S., Roisman, Z.W., and Wen, Z. Audio-visual emotion recognition in adult attachment interview. In Poceedings of the 8th International Conference on Multimodal Interfaces (ICMI '06), 2006, 139--145.

Digital Library

[45]

Zhai, J. and Barreto, A. Stress recognition using non-invasive technology. In Proceedings of FLAIRS Conference, 2006, 395--401.

Cited By

Li YInoue KTian LFu CIshi CIshiguro HKawahara TLai C(2023)I Know Your Feelings Before You Do: Predicting Future Affective Reactions in Human-Computer DialogueExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585869(1-7)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585869
Stephanidis CAntona MNtoa S(2021)HUMAN FACTORS IN AMBIENT INTELLIGENCE ENVIRONMENTSHANDBOOK OF HUMAN FACTORS AND ERGONOMICS10.1002/9781119636113.ch41(1058-1084)Online publication date: 13-Aug-2021
https://doi.org/10.1002/9781119636113.ch41
Pietrowicz MAgurto CCasebeer JHasegawa-Johnson MKarahalios KCecchi G(2019)Dimensional Analysis of Laughter in Female Conversational SpeechICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2019.8683566(6600-6604)Online publication date: May-2019
https://doi.org/10.1109/ICASSP.2019.8683566
Show More Cited By

Recommendations

Toward multimodal fusion of affective cues
HCM '06: Proceedings of the 1st ACM international workshop on Human-centered multimedia

During face to face communication, it has been suggested that as much as 70% of what people communicate when talking directly with others is through paralanguage involving multiple modalities combined together (e.g. voice tone and volume, body language)...
Modeling Multimodal Expression of User's Affective Subjective Experience

With the growing importance of information technology in our everyday life, new types of applications are appearing that require the understanding of information in a broad sense. Information that includes affective and subjective content plays a major ...
Affective Learning: Empathetic Agents with Emotional Facial and Tone of Voice Expressions

Empathetic behavior has been suggested to be one effective way for Embodied Conversational Agents (ECAs) to provide feedback to learners' emotions. An issue that has been raised is the effective integration of parallel and reactive empathy. The aim of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

HCM '07: Proceedings of the international workshop on Human-centered multimedia

September 2007

112 pages

ISBN:9781595937810

DOI:10.1145/1290128

General Chairs:
Alejandro Jaimes
IDIAP, Switzerland
,
Nicu Sebe
University of Amsterdam, The Netherlands

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 September 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

MM07

Sponsor:

MM07: The 15th ACM International Conference on Multimedia 2007

September 28, 2007

Bavaria, Augsburg, Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
678
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li YInoue KTian LFu CIshi CIshiguro HKawahara TLai C(2023)I Know Your Feelings Before You Do: Predicting Future Affective Reactions in Human-Computer DialogueExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585869(1-7)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585869
Stephanidis CAntona MNtoa S(2021)HUMAN FACTORS IN AMBIENT INTELLIGENCE ENVIRONMENTSHANDBOOK OF HUMAN FACTORS AND ERGONOMICS10.1002/9781119636113.ch41(1058-1084)Online publication date: 13-Aug-2021
https://doi.org/10.1002/9781119636113.ch41
Pietrowicz MAgurto CCasebeer JHasegawa-Johnson MKarahalios KCecchi G(2019)Dimensional Analysis of Laughter in Female Conversational SpeechICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2019.8683566(6600-6604)Online publication date: May-2019
https://doi.org/10.1109/ICASSP.2019.8683566
Portalés CGimeno JCasas SOlanda RGiner Martínez F(2018)Interacting With Augmented Reality MirrorsVirtual and Augmented Reality10.4018/978-1-5225-5469-1.ch002(18-46)Online publication date: 2018
https://doi.org/10.4018/978-1-5225-5469-1.ch002
Schuller B(2018)Multimodal user state and trait recognitionThe Handbook of Multimodal-Multisensor Interfaces10.1145/3107990.3107997(129-165)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1145/3107990.3107997
Seo SGriffin KYoung JBunt APrentice SLoureiro-Rodríguez V(2017)Investigating People’s Rapport Building and Hindering Behaviors When Working with a Collaborative RobotInternational Journal of Social Robotics10.1007/s12369-017-0441-810:1(147-161)Online publication date: 2-Nov-2017
https://doi.org/10.1007/s12369-017-0441-8
Portalés CGimeno JCasas SOlanda RGiner Martínez F(2016)Interacting with Augmented Reality MirrorsHandbook of Research on Human-Computer Interfaces, Developments, and Applications10.4018/978-1-5225-0435-1.ch009(216-244)Online publication date: 2016
https://doi.org/10.4018/978-1-5225-0435-1.ch009
Wagner AKilincsoy ÜReitmeir MVink P(2016)Functional customization: Value creation by individual storage elements in the car interiorWork10.3233/WOR-16236054:4(873-885)Online publication date: 1-Sep-2016
https://doi.org/10.3233/WOR-162360
El Haddad KÇakmak HGilmartin EDupont SDutoit TNakano YAndré ENishida TMorency LBusso CPelachaud C(2016)Towards a listening agent: a system generating audiovisual laughs and smiles to show interestProceedings of the 18th ACM International Conference on Multimodal Interaction10.1145/2993148.2993182(248-255)Online publication date: 31-Oct-2016
https://dl.acm.org/doi/10.1145/2993148.2993182
Cosentino SSessa STakanishi A(2016)Quantitative Laughter Detection, Measurement, and Classification—A Critical SurveyIEEE Reviews in Biomedical Engineering10.1109/RBME.2016.25276389(148-162)Online publication date: 2016
https://doi.org/10.1109/RBME.2016.2527638
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents