demonstration

Multimodal affective feedback: combining thermal, vibrotactile, audio and visual signals

Authors:

Graham Wilson,

Euan Freeman,

Stephen BrewsterAuthors Info & Claims

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

Pages 400 - 401

https://doi.org/10.1145/2993148.2998522

Published: 31 October 2016 Publication History

Get Access

Abstract

In this paper we describe a demonstration of our multimodal affective feedback designs, used in research to expand the emotional expressivity of interfaces. The feedback leverages inherent associations and reactions to thermal, vibrotactile, auditory and abstract visual designs to convey a range of affective states without any need for learning feedback encoding. All combinations of the different feedback channels can be utilised, depending on which combination best conveys a given state. All the signals are generated from a mobile phone augmented with thermal and vibrotactile stimulators, which will be available to conference visitors to see, touch, hear and, importantly, feel.

References

[1]

Belin, B., Fillion-Bilodeau, S. and Gosselin, F. 2008. The Montreal Affective Voices: a validated set of nonverbal affect bursts for research on auditory affective processing. Behav res methods 40, 2: 531–539. http://doi.org/10.3758/BRM.40.2.531

Crossref

Google Scholar

[2]

Lakoff, G. and Johnson, M. 1999. Philosophy in the Flesh: The embodied mind and its challenge to Western thought. Harper-Collins, New York.

Google Scholar

[3]

Miller, H, Thebault-Spieker, J., Chang, S., Johnson, I., Terveen, L. and Hecht, B. 2016. “Blissfully happy” or “ready to fight”: Varying Interpretations of Emoji. Proceedings of ICWSM 2016.

Google Scholar

[4]

Paquette, S., Peretz, I. and Belin, P. 2013. The “Musical Emotional Bursts”: A validated set of musical affect bursts to investigate auditory affective processing. Frontiers in Psychology 4, 1–7. http://doi.org/10.3389/fpsyg.2013.00509

Crossref

Google Scholar

[5]

Seifi, H. and Maclean, K. 2013. A first look at individuals’ affective ratings of vibrations. Proceedings of World Haptics 2013, 605–610. http://doi.org/10.1109/WHC.2013.6548477

Crossref

Google Scholar

[6]

Valtchanov, D. and Hancock, M. 2015. EnviroPulse : Providing Feedback about the Expected Affective Valence of the Environment. Proceedings of ACM CHI 2015, 2073–2082.

Digital Library

Google Scholar

[7]

Wilson, G., Dobrev, D. and Brewster, S. 2016. Hot Under the Collar: Mapping Thermal Feedback to Dimensional Models of Emotion. Proceedings of ACM CHI 2016, 4838–4849.

Digital Library

Google Scholar

[8]

Wilson, G., Romeo, P. and Brewster, S. 2016. Mapping Abstract Visual Feedback to a Dimensional Model of Emotion. Proceedings of ACM CHI 2016 Extended Abstracts.

Digital Library

Google Scholar

[9]

Yoo, Y., Yoo, T., Kong, J. and Choi, S. 2015. Emotional Responses of Tactile Icons: Effects of Amplitude, Frequency, Duration, and Envelope. Proceedings of World Haptics 2015, 235–240. http://dx.doi.org/10.1109/WHC.2015.7177719

Crossref

Google Scholar

Cited By

View all

Günther SSkogseide ABuhlmann RMühlhäuser M(2024)Assessing the Influence of Visual Cues in Virtual Reality on the Spatial Perception of Physical Thermal StimuliProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642154(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642154
Jung KKim SOh SYoon S(2024)HapMotion: motion-to-tactile framework with wearable haptic devices for immersive VR performance experienceVirtual Reality10.1007/s10055-023-00910-z28:1Online publication date: 9-Jan-2024
https://dl.acm.org/doi/10.1007/s10055-023-00910-z
Macdonald SPollick FBrewster S(2022)The Impact of Thermal Cues on Affective Responses to Emotionally Resonant VibrationsProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3556572(259-269)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536221.3556572
Show More Cited By

Index Terms

Multimodal affective feedback: combining thermal, vibrotactile, audio and visual signals
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Haptic devices
  2. Interaction design
    1. Interaction design process and methods
      1. User interface design

Recommendations

Haptic Empathy: Conveying Emotional Meaning through Vibrotactile Feedback
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

Touch plays an essential role in communicating emotions and intensifying interpersonal communication[4]. A lot of research focuses on how to create or improve haptic interfaces looking into challenges and possibilities that the haptic technology can ...
Multi-moji: Combining Thermal, Vibrotactile & Visual Stimuli to Expand the Affective Range of Feedback
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

This paper explores the combination of multiple concurrent modalities for conveying emotional information in HCI: temperature, vibration and abstract visual displays. Each modality has been studied individually, but can only convey a limited range of ...
The impact of system feedback on learners' affective and physiological states
ITS'10: Proceedings of the 10th international conference on Intelligent Tutoring Systems - Volume Part I

We investigate how positive, neutral and negative feedback responses from an Intelligent Tutoring System (ITS) influences learners' affect and physiology. AutoTutor, an ITS with conversational dialogues, was used by learners (n=16) while their ...

Comments

Information & Contributors

Information

Published In

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

October 2016

605 pages

ISBN:9781450345569

DOI:10.1145/2993148

General Chairs:
Yukiko I. Nakano
Seikei University, Japan
,
Elisabeth André
Augsburg University, Germany
,
Toyoaki Nishida
Kyoto University, Japan
,
Program Chairs:
Louis-Philippe Morency
Carnegie Mellon University, USA
,
Carlos Busso
University of Texas at Dallas, USA
,
Catherine Pelachaud
ISIR, France / University of Paris6, France

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 October 2016

Check for updates

Author Tags

Qualifiers

Demonstration

Funding Sources

Seventh Framework Programme

Conference

ICMI '16

Sponsor:

SIGCHI

ICMI '16: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 12 - 16, 2016

Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
515
Total Downloads

Downloads (Last 12 months)53
Downloads (Last 6 weeks)5

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Günther SSkogseide ABuhlmann RMühlhäuser M(2024)Assessing the Influence of Visual Cues in Virtual Reality on the Spatial Perception of Physical Thermal StimuliProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642154(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642154
Jung KKim SOh SYoon S(2024)HapMotion: motion-to-tactile framework with wearable haptic devices for immersive VR performance experienceVirtual Reality10.1007/s10055-023-00910-z28:1Online publication date: 9-Jan-2024
https://dl.acm.org/doi/10.1007/s10055-023-00910-z
Macdonald SPollick FBrewster S(2022)The Impact of Thermal Cues on Affective Responses to Emotionally Resonant VibrationsProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3556572(259-269)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536221.3556572
Jiang MNanjappan Vten Bhömer MLiang H(2021)On the Use of Movement-Based Interaction with Smart Textiles for Emotion RegulationSensors10.3390/s2103099021:3(990)Online publication date: 2-Feb-2021
https://doi.org/10.3390/s21030990
Shetty YMehta STran DSoni BMcDaniel T(2021)Emotional Response to Vibrothermal StimuliApplied Sciences10.3390/app1119890511:19(8905)Online publication date: 24-Sep-2021
https://doi.org/10.3390/app11198905
Gharat SShetty YMcDaniel T(2021)A Vibrothermal Haptic Display for Socio-emotional CommunicationHCI International 2021 - Late Breaking Papers: Multimodality, eXtended Reality, and Artificial Intelligence10.1007/978-3-030-90963-5_2(17-30)Online publication date: 11-Nov-2021
https://doi.org/10.1007/978-3-030-90963-5_2
Günther SMüller FSchön DElmoghazy OMühlhäuser MSchmitz MBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Therminator: Understanding the Interdependency of Visual and On-Body Thermal Feedback in Virtual RealityProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376195(1-14)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376195
Yan XMadier KPark SNewman MHarrison SBardzell SNeustaedter CTatar D(2019)Towards Low-burden In-situ Self-reportingCompanion Publication of the 2019 on Designing Interactive Systems Conference 2019 Companion10.1145/3301019.3323905(337-346)Online publication date: 18-Jun-2019
https://dl.acm.org/doi/10.1145/3301019.3323905
Poguntke RKiss FKaplan ASchmidt ASchneegass S(2018)RainSenseProceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct10.1145/3236112.3236114(9-15)Online publication date: 3-Sep-2018
https://dl.acm.org/doi/10.1145/3236112.3236114
Frey JGrabli MSlyper RCauchard JMandryk RHancock MPerry MCox A(2018)BreezeProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3174219(1-12)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3174219
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Haptic Empathy: Conveying Emotional Meaning through Vibrotactile Feedback

Multi-moji: Combining Thermal, Vibrotactile & Visual Stimuli to Expand the Affective Range of Feedback

The impact of system feedback on learners' affective and physiological states