skip to main content
10.1145/3025453.3025614acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Multi-moji: Combining Thermal, Vibrotactile & Visual Stimuli to Expand the Affective Range of Feedback

Published: 02 May 2017 Publication History

Abstract

This paper explores the combination of multiple concurrent modalities for conveying emotional information in HCI: temperature, vibration and abstract visual displays. Each modality has been studied individually, but can only convey a limited range of emotions within two-dimensional valence-arousal space. This paper is the first to systematically combine multiple modalities to expand the available affective range. Three studies were conducted: Study 1 measured the emotionality of vibrotactile feedback by itself; Study 2 measured the perceived emotional content of three bimodal combinations: vibrotactile + thermal, vibrotactile + visual and visual + thermal. Study 3 then combined all three modalities. Results show that combining modalities increases the available range of emotional states, particularly in the problematic top-right and bottom-left quadrants of the dimensional model. We also provide a novel lookup resource for designers to identify stimuli to convey a range of emotions

References

[1]
Francis Adams and Charles Osgood. 1973. A CrossCultural Study of the Affective Meanings of Color. Journal of Cross-Cultural Psychology 4, 2: 135--156. http://doi.org/0803973233
[2]
Moses Akazue, Martin Halvey, Lynne Baillie, and Stephen Brewster. 2016. The Effect of Thermal Stimuli on the Emotional Perception of Images. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16), 4401--4412. http://doi.org/10.1145/2858036.2858307
[3]
Akshita, Harini Sampath, Bipin Indurkhya, Eunhwa Lee, and Yudong Bae. 2015. Towards Multimodal Affective Feedback?: Interaction between Visual and Haptic Modalities. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '15), 2043--2052. http://doi.acm.org/10.1145/2702123.2702288
[4]
Betsy App, Daniel N McIntosh, Catherine L Reed, and Matthew J Hertenstein. 2011. Nonverbal channel use in communication of emotion: how may depend on why. Emotion 11, 3: 603--617. http://doi.org/10.1037/a0023164
[5]
Solomon E Asch. 1946. Forming impressions of personality. Journal of abnormal psychology 41: 258-- 290. http://doi.org/10.1037/h0060423
[6]
Jeremy N Bailenson, Nick Yee, Scott Brave, Dan Merget, and David Koslow. 2007. Virtual interpersonal touch: expressing and recognizing emotions through haptic devices. Human-Computer Interaction 22: 325-- 353. http://doi.org/10.1080/07370020701493509
[7]
Moshe Bar and Maital Neta. 2006. Humans prefer curved visual objects. Psychological Science 17, 8: 645-- 648. http://doi.org/10.1111/j.1467--9280.2006.01759.x
[8]
Moshe Bar and Maital Neta. 2007. Visual elements of subjective preference modulate amygdala activation. Neuropsychologia 45, 10: 2191--200. http://doi.org/10.1016/j.neuropsychologia.2007.03.008
[9]
Pascal Belin, Sarah Fillion-Bilodeau, and Frédéric Gosselin. 2008. The Montreal Affective Voices: a validated set of nonverbal affect bursts for research on auditory affective processing. Behavior research methods 40, 2: 531--539. http://doi.org/10.3758/BRM.40.2.531
[10]
John Bowlby. 1969. Attachment and Loss. Hogarth Press, London.
[11]
Stephen A. Brewster, Peter C. Wright, and Alistair D.N. Edwards. 1995. Experimentally derived guidelines for the creation of earcons. Adjunct Proceedings of HCI 1995, 155--159.
[12]
Lorna Brown, Stephen Brewster, and Helen Purchase. 2006. Multidimensional Tactons for Non-Visual Information Presentation in Mobile Devices. Proceedings of the International Conference on HumanComputer Interaction with Mobile Devices and Services (MobileHCI '06), 231--238. papers://c80d98e4--9a964487--8d06--8e1acc780d86/Paper/p79
[13]
Jessica Dawson, Oliver Schneider, Joel Ferstay, et al. 2013. It's alive!: exploring the design space of a gesturing phone. Proceedings of Graphics Interface 2013, 205--212. http://dl.acm.org/citation.cfm?id=2532164
[14]
Paul Ekman. 1993. Facial Expression and Emotion. American Psychologist 48, 4: 376--379.
[15]
Susan T. Fiske, Amy J.C. Cuddy, and Peter Glick. 2007. Universal dimensions of social cognition: warmth and competence. Trends in Cognitive Sciences 11, 2: 77--83. http://doi.org/10.1016/j.tics.2006.11.005
[16]
Martin Halvey, Michael Henderson, Stephen A. Brewster, Graham Wilson, and Stephen A. Hughes. 2012. Augmenting Media with Thermal Stimulation. Proceedings of International Workshop on Haptic and Audio Interaction Design (HAID '12), 91--100. http://dx.doi.org/10.1007/978--3--642--32796--4_10
[17]
Harry F. Harlow. 1958. The Nature of Love. American Psychologist 13: 673--685.
[18]
Fabian Hemmert, Matthias Löwe, Anne Wohlauf, and Gesche Joost. 2013. Animate mobiles. Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction - TEI '13: 267. http://doi.org/10.1145/2460625.2460669
[19]
Matthew J Hertenstein, Rachel Holmes, Margaret McCullough, and Dacher Keltner. 2009. The communication of emotion via touch. Emotion 9, 4: 566--573. http://dx.doi.org/10.1037/a0016108
[20]
Inwook Hwang, Jongman Seo, Myongchan Kim, and Seungmoon Choi. 2013. Vibrotactile perceived intensity for mobile devices as a function of direction, amplitude, and frequency. IEEE Transactions on Haptics 6, 3: 352-- 362. http://doi.org/10.1109/TOH.2013.2
[21]
L Jones and M Berris. 2002. The Psychophysics of Temperature Perception and Thermal-Interface Design. Proceedings of the International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS), 137--142. papers://c80d98e4--9a964487--8d06--8e1acc780d86/Paper/p591
[22]
Peter J Lang, Margaret M Bradley, and BN Cuthbert. 2008. International Affective Picture System (IAPS): Technical Manual and Affective Ratings. Gainesville, FL.
[23]
Jung Min Lee, SY Jeong, and Ju DY. 2015. Emotional Interaction and Notification of Flexible Handheld Devices. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '15), 2025-- 2030.
[24]
Wonjun Lee and Youn-kyung Lim. 2012. Explorative research on the heat as an expression medium: focused on interpersonal communication. Personal and Ubiquitous Computing 16: 1039--1049. http://dx.doi.org/10.1007/s00779-011-0424-y
[25]
Markus Löchtefeld, Nadine Lautemann, Sven Gehring, and Antonio Krüger. 2014. ambiPad -- Enriching Mobile Digital Media With Ambient Feedback. Proceedings of the International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '14), 295--298.
[26]
Hannah Miller, Jacob Thebault-Spieker, Shuo Chang, Isaac Johnson, Loren Terveen, and Brent Hecht. 2016. "Blissfully happy" or "ready to fight": Varying Interpretations of Emoji. Proceedings of ICWSM 2016.
[27]
Marianna Obrist, Sriram Subramanian, Elia Gatti, Benjamin Long, and Thomas Carter. 2015. Emotions Mediated Through Mid-Air Haptics. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '15), 2053--2062. http://doi.acm.org/10.1145/2702123.2702361
[28]
Ofcom. 2015. Time spent online doubles in a decade. Media Use and Attitudes 2015. Retrieved September 18, 2016 from http://consumers.ofcom.org.uk/news/timespent-online-doubles/
[29]
Esben W Pedersen, Sriram Subramanian, and Kasper Hornbæk. 2014. Is My Phone Alive?: A Large-scale Study of Shape Change in Handheld Devices Using Videos. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14): 2579-- 2588. http://doi.org/10.1145/2556288.2557018
[30]
AT Poffenberger and BE Barrows. 1924. The Feeling Value of Lines. Journal of Applied Psychology 8, 2: 187--205.
[31]
James A Russell. 1980. A Circumplex Model of Affect. Journal of Personality and Social Psychology 39, 6: 1161--1178. http://dx.doi.org/10.1037/h0077714
[32]
Katri Salminen, Veikko Surakka, Jani Lylykangas, et al. 2012. Tactile modulation of emotional speech samples. Advances in Human-Computer Interaction 2012. http://doi.org/10.1155/2012/741304
[33]
Katri Salminen, Veikko Surakka, Jukka Raisamo, et al. 2011. Emotional Responses to Thermal Stimuli. Proceedings of ICMI 2011, 193--196. http://doi.acm.org/10.1145/2070481.2070513
[34]
Katri Salminen, Veikko Surakka, Jukka Raisamo, et al. 2013. Cold or hot? How thermal stimuli are related to human emotional system? Proceedings of International Workshop on Haptic and Audio Interaction Design (HAID '13) 7989: 20--29. http://doi.org/10.1007/978--3642--41068-0_3
[35]
Hasti Seifi, Chamila Anthonypillai, and Karon E. Maclean. 2014. End-user customization of affective tactile messages: A qualitative examination of tool parameters. IEEE Haptics Symposium, HAPTICS: 251-- 256. http://doi.org/10.1109/HAPTICS.2014.6775463
[36]
Hasti Seifi and Karon E. Maclean. 2013. A first look at individuals? affective ratings of vibrations. Proceedings of World Haptics 2013, 605--610. http://doi.org/10.1109/WHC.2013.6548477
[37]
Jocelyn Smith and Karon MacLean. 2007. Communicating emotion through a haptic link: Design space and methodology. International Journal of Human-Computer Studies 65: 376--387. http://www.sciencedirect.com/science/article/pii/S10715 81906001911
[38]
Paul Strohmeier, Juan Pablo Carrascal, Bernard Cheng, Margaret Meban, and Roel Vertegaal. 2016. An Evaluation of Shape Changes for Conveying Emotions. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16), 3781--3792. http://doi.org/10.1145/2858036.2858537
[39]
Katja Suhonen, Sebastian Muller, Jussi Rantala, Kaisa Vaananen-Vainio-Mattila, Roope Raisamo, and Vuokko Lantz. 2012. Haptically Augmented Remote Speech Communication: A Study of User Practices and Experiences. Proceedings of the Nordic Conference on Human-Computer Interaction (NordiCHI '12), 361-- 369. http://doi.acm.org/10.1145/2399016.2399073
[40]
Katja Suhonen, Kaisa Väänänen-Vainio-Mattila, and Kalle Mäkelä. 2012. User experiences and expectations of vibrotactile, thermal and squeeze feedback in interpersonal communication. Proceedings of the Annual BCS Interaction Specialist Group Conference on People and Computers (BCS-HCI '12), 205--214. http://dl.acm.org/citation.cfm?id=2377916.2377939
[41]
TactileLabs. Haptuator Mark II Specification. Retrieved August 16, 2016 from http://tactilelabs.com/wpcontent/uploads/2012/07/TL002-09-A_v1.01.pdf
[42]
Patricia Valdez and Albert Mehrabian. 1994. Effects of color on emotions. Journal of Experimental Psychology 123, 4: 394--409. http://doi.org/10.1037/00963445.123.4.394
[43]
Deltcho Valtchanov and Mark Hancock. 2015. EnviroPulse?: Providing Feedback about the Expected Affective Valence of the Environment. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '15), 2073--2082.
[44]
Hanna Venesvirta, Veikko Surakka, Yulia Gizatdinova, et al. 2015. Emotional Reactions to Point-Light Display Animations. Interacting with Computers, 1973: iwv028. http://doi.org/10.1093/iwc/iwv028
[45]
T W Whitfield and T J Wiltshire. 1990. Color psychology: a critical review. Genetic, social, and general psychology monographs 116, 4: 385--411. http://www.ncbi.nlm.nih.gov/ /2289687
[46]
Graham Wilson, Stephen Brewster, Martin Halvey, and Stephen Hughes. 2012. Thermal Icons: Evaluating Structured Thermal Feedback for Mobile Interaction. Proceedings of the International Conference on HumanComputer Interaction with Mobile Devices and Services (MobileHCI '12), 309--312. papers://c80d98e4--9a964487--8d06--8e1acc780d86/Paper/p9328
[47]
Graham Wilson, Stephen Brewster, Martin Halvey, and Stephen Hughes. 2013. Thermal Feedback Identification in a Mobile Environment. Proceedings of International Workshop on Haptic and Audio Interaction Design (HAID '13), Article 2. papers://c80d98e4--9a96--44878d06--8e1acc780d86/Paper/p9329
[48]
Graham Wilson, Gavin Davidson, and Stephen Brewster. 2015. In the Heat of the Moment?: Subjective Interpretations of Thermal Feedback During Interaction. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '15), 2063--2072. http://doi.acm.org/10.1145/2702123.2702219
[49]
Graham Wilson, Dobromir Dobrev, and Stephen A Brewster. 2016. Hot Under the Collar: Mapping Thermal Feedback to Dimensional Models of Emotion. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16), 4838--4849.
[50]
Graham Wilson, Martin Halvey, Stephen A Brewster, and Stephen A Hughes. 2011. Some Like it Hot? Thermal Feedback for Mobile Devices. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 2555--2564. http://doi.acm.org/10.1145/1978942.1979316
[51]
Graham Wilson, Pietro Romeo, and Stephen A Brewster. 2016. Mapping Abstract Visual Feedback to a Dimensional Model of Emotion. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16) Extended Abstracts.
[52]
Steve Yohanan and Karon E. MacLean. 2011. Design and assessment of the haptic creature's affect display. Proceedings of Human-robot interaction 2011, 473. http://doi.org/10.1145/1957656.1957820
[53]
Yongjae Yoo, Taekbeom Yoo, Jihyun Kong, and Seungmoon Choi. 2015. Emotional Responses of Tactile Icons?: Effects of Amplitude, Frequency, Duration, and Envelope. Proceedings of World Haptics 2015, 235--240. http://dx.doi.org/10.1109/WHC.2015.7177719

Cited By

View all

Index Terms

  1. Multi-moji: Combining Thermal, Vibrotactile & Visual Stimuli to Expand the Affective Range of Feedback

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. emotion
    2. thermal feedback
    3. vibration
    4. visual feedback

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)138
    • Downloads (Last 6 weeks)14
    Reflects downloads up to 22 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media