research-article

Thermal imaging for affect detection

Authors:

Verónica Pérez-Rosas,

Alexis Narvaez,

Rada MihalceaAuthors Info & Claims

PETRA '13: Proceedings of the 6th International Conference on PErvasive Technologies Related to Assistive Environments

Article No.: 36, Pages 1 - 4

https://doi.org/10.1145/2504335.2504374

Published: 29 May 2013 Publication History

Abstract

In this paper, we explore a thermal imaging approach to sensing affective state. Using features extracted from a thermal map of the face, obtained from a dataset consisting of 70 recordings of positive, negative, or neutral states, we show that we can effectively predict the presence of affect, with an error reduction of up to 50% as compared to a majority class baseline.

References

[1]

J. Allen. Photoplethysmography and its application in clinical physiological measurement. Physiological measurement, 28:R1, 2007.

[2]

C. Alm, D. Roth, and R. Sproat. Emotions from text: Machine learning for text-based emotion prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 347--354, Vancouver, Canada, 2005.

Digital Library

[3]

V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH '99: Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187--194, New York, NY, USA, 1999. ACM Press/Addison-Wesley Publishing Co.

Digital Library

[4]

T. Cootes, G. Edwards, and C. Taylor. Active appearance models. Pattern Analysis and Machine Intel ligence, IEEE Transactions on, 23(6):681--685, Jun 2001.

Digital Library

[5]

T. F. Cootes and C. J. Taylor. Active shape models - 'smart snakes'. In Proceedings of the British Machine Vision Conference, 1992.

[6]

D. Cristinacce and T. Cootes. Feature detection and tracking with constrained local models. In Brittish Machine Vision Conference, 2006.

[7]

P. Ekman and W. Friesen. Manual for the Facial Action Coding System. Palo Alto: Consulting Psychologists Press, 1977.

[8]

J. Fei and I. Pavlidis. Thermistor at a distance: Unobtrusive measurement of breathing. Biomedical Engineering, IEEE Transactions on, 57(4):988--998, 2010.

[9]

M. Garbey, N. Sun, A. Merla, and I. Pavlidis. Contact-free measurement of cardiac pulse based on the analysis of thermal imagery. Biomedical Engineering, IEEE Transactions on, 54(8):1418--1426, 2007.

[10]

E. Greneker. Radar sensing of heartbeat and respiration at a distance with applications of the technology. In Radar 97 (Conf. Publ. No. 449), pages 150--154. IET, 1997.

[11]

H. Gunes and M. Pantic. Automatic, dimensional and continuous emotion recognition. Int'l Journal of Synthetic Emotion, 1(1):68--99, 2010.

Digital Library

[12]

M. Khan, R. Ward, and M. Ingleby. Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature. ACM Transactions on Applied Perception, 6(1), 2009.

Digital Library

[13]

A. Maas, R. Daly, P. Pham, D. Huang, A. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the Association for Computational Linguistics (ACL 2011), Portland, OR, 2011.

Digital Library

[14]

G. Mishne. Experiments with mood classification in blog posts. In Proceedings of the 1st Workshop on Stylistic Analysis Of Text For Information Access (Style 2005), Brazile, 2005.

[15]

L. Morency, J. Whitehill, and J. Movellan. Generalized adaptive view-based appearance model: Integrated framework for monocular head pose estimation. In Automatic Face and Gesture Recognition, pages 1--8, 2008.

[16]

E. Murphy-Chutorian and M. Trivedi. Head pose estimation in computer vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intel ligence, 31:607--626, 2009.

Digital Library

[17]

M. Nicolaou, H. Gunes, and M. Pantic. Output-associative rvm regression for dimensional and continuous emotion prediction. In IEEE FG'11, 2011.

[18]

B. Pang and L. Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics, Barcelona, Spain, July 2004.

Digital Library

[19]

M. Pantic and M. S. Bartlett. Machine analysis of facial expressions. In K. Delac and M. Grgic, editors, Face Recognition, pages 377--416. I-Tech Education and Publishing, Vienna, Austria, July 2007.

[20]

I. Pavlidis and J. Levine. Thermal image analysis for polygraph testing. IEEE Engineering in Medicine and Biology Magazine, 21(6), 2002.

[21]

R. Picard. Affective computing. MIT Press, Cambridge, MA, USA, 1997.

Digital Library

[22]

C. Puri, L. Olson, I. Pavlidis, J. Levine, and J. Starren. Stresscam: non-contact measurement of users' emotional states through thermal imaging. In Proceedings of the 2005 ACM Conference on Human Factors in Computing Systems (CHI), 2005.

Digital Library

[23]

E. Riloff and J. Wiebe. Learning extraction patterns for subjective expressions. In Conference on Empirical Methods in Natural Language Processing (EMNLP-03), pages 105--112, 2003.

Digital Library

[24]

C. Strapparava and R. Mihalcea. Semeval-2007 task 14: Affective text. In Proceedings of the 4th International Workshop on the Semantic Evaluations (SemEval 2007), Prague, Czech Republic, 2007.

Digital Library

[25]

C. Strapparava and R. Mihalcea. Learning to identify emotions in text. In Proceedings of the ACM Conference on Applied Computing ACM-SAC 2008, Fortaleza, Brazile, 2008.

Digital Library

[26]

C. Strapparava and A. Valitutti. Wordnet-affect: an affective extension of wordnet. In Proceedings of the 4th International Conference on Language Resources and Evaluation, Lisbon, 2004.

[27]

P. Turney. Thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002), pages 417--424, Philadelphia, 2002.

Digital Library

[28]

S. Ulyanov and V. Tuchin. Pulse-wave monitoring by means of focused laser beams scattered by skin surface and membranes. In Proceedings of SPIE, 1993.

[29]

D. Watson and L. Clark. THE PANAS-X Manual for the Positive and Negative Affect Schedule - Expanded Form. University of Iowa, 1994.

[30]

J. Wiebe. Learning subjective adjectives from corpora. In Proceedings of the American Association for Artificial Intel ligence (AAAI 2000), pages 735--740, Austin, Texas, 2000.

Digital Library

[31]

T. Wilson, J. Wiebe, and P. Hoffmann. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of Human Language Technologies Conference/Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005), Vancouver, Canada, 2005.

Digital Library

[32]

M. Wöllmer, F. Eyben, S. Reiter, B. Schuller, C. Cox, E. Douglas-Cowie, and R. Cowie. Abandoning emotion classes - towards continuous emotion recognition with modelling of long-range dependencies. In INTERSPEECH. ISCA, 2008.

[33]

Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. volume 31, Los Alamitos, CA, USA, 2009. IEEE Computer Society.

Digital Library

Cited By

Küster D(2025)The Face Behind the Mask: Thermography of the FaceBody Language Communication10.1007/978-3-031-70064-4_12(285-313)Online publication date: 2-Jan-2025
https://doi.org/10.1007/978-3-031-70064-4_12
Xiao YSharma HZhang ZBergen-Cico DRahman TSalekin A(2024)Reading Between the HeatProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314417:4(1-30)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631441
Al Qudah MMohamed ALutfi S(2023)Analysis of Facial Occlusion Challenge in Thermal Images for Human Affective State RecognitionSensors10.3390/s2307351323:7(3513)Online publication date: 27-Mar-2023
https://doi.org/10.3390/s23073513
Show More Cited By

Index Terms

Thermal imaging for affect detection
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Discourse, dialogue and pragmatics

Recommendations

Automatic Frustration Detection Using Thermal Imaging
HRI '22: Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction

To achieve seamless interactions, robots have to be capable of reliably detecting affective states in real time. One of the possible states that humans go through while interacting with robots is frustration. Detecting frustration from RGB images can be ...
Skin Temperature Extraction Using Facial Landmark Detection and Thermal Imaging for Comfort Assessment
BuildSys '19: Proceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation

Despite the large share of energy consumption, current HVAC systems in buildings fail to meet their primary purpose of maintaining comfortable indoor conditions. Current "one size fits all" approach to control the thermal conditions in an environment ...
Lock-in thermal imaging for the early-stage detection of cutaneous melanoma

This paper theoretically evaluates lock-in thermal imaging for the early-stage detection of cutaneous melanoma. Lock-in thermal imaging is based on the periodic thermal excitation of the specimen under test. Resulting surface temperature oscillations ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PETRA '13: Proceedings of the 6th International Conference on PErvasive Technologies Related to Assistive Environments

May 2013

413 pages

ISBN:9781450319737

DOI:10.1145/2504335

Conference Chair:
Fillia Makedon
University of Texas at Arlington
,
Program Chairs:
Margrit Betke
Boston University
,
Magy Self El-Nasr
Northeastern University
,
Ilias Maglogiannis
University of Central Greece, Greece

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

NSF: National Science Foundation
FORTH: Foundation for Research and Technology - Hellas
HERACLEIA: HERACLEIA Human-Centered Computing Laboratory at UTA
U of Tex at Arlington: U of Tex at Arlington
TEI: Technological Educational Institution of Athens
UCG: University of Central Greece
NCRS: Demokritos National Center for Scientific Research
Fulbrigh, Greece: Fulbright Foundation, Greece
Ionian: Ionian University, GREECE

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 May 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

PETRA '13

Sponsor:

NSF
FORTH
HERACLEIA
U of Tex at Arlington
TEI
UCG
NCRS
Fulbrigh, Greece
Ionian

PETRA '13: The 6th International Conference on PErvasive Technologies Related to Assistive Environments

May 29 - 31, 2013

Rhodes, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
208
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Küster D(2025)The Face Behind the Mask: Thermography of the FaceBody Language Communication10.1007/978-3-031-70064-4_12(285-313)Online publication date: 2-Jan-2025
https://doi.org/10.1007/978-3-031-70064-4_12
Xiao YSharma HZhang ZBergen-Cico DRahman TSalekin A(2024)Reading Between the HeatProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314417:4(1-30)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631441
Al Qudah MMohamed ALutfi S(2023)Analysis of Facial Occlusion Challenge in Thermal Images for Human Affective State RecognitionSensors10.3390/s2307351323:7(3513)Online publication date: 27-Mar-2023
https://doi.org/10.3390/s23073513
Rooj SRoutray AMandal M(2023)Feature based analysis of thermal images for emotion recognitionEngineering Applications of Artificial Intelligence10.1016/j.engappai.2022.105809120:COnline publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1016/j.engappai.2022.105809
Johansen AHenriksen JHaque MSabet Jahromi MNasrollahi KMoeslund T(2019)Multimodal heartbeat rate estimation from the fusion of facial RGB and thermal videosEleventh International Conference on Machine Vision (ICMV 2018)10.1117/12.2523385(114)Online publication date: 15-Mar-2019
https://doi.org/10.1117/12.2523385
Wang SPan BChen HJi Q(2018)Thermal Augmented Expression RecognitionIEEE Transactions on Cybernetics10.1109/TCYB.2017.278630948:7(2203-2214)Online publication date: Jul-2018
https://doi.org/10.1109/TCYB.2017.2786309
Wu CWang SPan BChen HHanjalic ASnoek CWorring MBulterman DHuet BKelliher AKompatsiaris YLi J(2016)Facial Expression Recognition with Deep two-view Support Vector MachineProceedings of the 24th ACM international conference on Multimedia10.1145/2964284.2967295(616-620)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1145/2964284.2967295

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents