skip to main content
research-article

Automated Detection of Stressful Conversations Using Wearable Physiological and Inertial Sensors

Published: 18 December 2020 Publication History

Abstract

Stressful conversation is a frequently occurring stressor in our daily life. Stressors not only adversely affect our physical and mental health but also our relationships with family, friends, and coworkers. In this paper, we present a model to automatically detect stressful conversations using wearable physiological and inertial sensors. We conducted a lab and a field study with cohabiting couples to collect ecologically valid sensor data with temporally-precise labels of stressors. We introduce the concept of stress cycles, i.e., the physiological arousal and recovery, within a stress event. We identify several novel features from stress cycles and show that they exhibit distinguishing patterns during stressful conversations when compared to physiological response due to other stressors. We observe that hand gestures also show a distinct pattern when stress occurs due to stressful conversations. We train and test our model using field data collected from 38 participants. Our model can determine whether a detected stress event is due to a stressful conversation with an F1-score of 0.83, using features obtained from only one stress cycle, facilitating intervention delivery within 3.9 minutes since the start of a stressful conversation.

References

[1]
2019. LENA Research Foundation. http://www.lenafoundation.org//. (Accessed: February 2019).
[2]
Mustafa al'Absi, Stephan Bongard, Tony Buchanan, Gwendolyn A Pincomb, Julio Licinio, and William R Lovallo. 1997. Cardiovascular and neuroendocrine adjustment to public speaking and mental arithmetic stressors. Psychophysiology 34, 3 (1997), 266--275.
[3]
David M Almeida and Ronald C Kessler. 1998. Everyday stressors and gender differences in daily distress. Journal of personality and social psychology 75, 3 (1998), 670.
[4]
David M Almeida, Elaine Wethington, and Ronald C Kessler. 2002. The daily inventory of stressful events: An interview-based approach for measuring daily stressors. Assessment 9, 1 (2002), 41--55.
[5]
Anne H Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. 1991. The HCRC map task corpus. Journal of Language and speech (1991).
[6]
Rummana Bari, Roy J Adams, Md Mahbubur Rahman, Megan Battles Parsons, Eugene H Buder, and Santosh Kumar. 2018. rConverse: Moment by Moment Conversation Detection Using a Mobile Respiration Sensor. ACM IMWUT 2, 1 (2018), 2.
[7]
Javad Birjandtalab, Diana Cogan, Maziyar Baran Pouyan, and Mehrdad Nourani. 2016. A non-EEG biosignals dataset for assessment and visualization of neurological status. In 2016 IEEE International Workshop on Signal Processing Systems (SiPS). 110--114.
[8]
Richard W Bohannon. 1997. Comfortable and maximum walking speed of adults aged 20-79 years: reference values and determinants. Age and Aging (1997), 15--19.
[9]
Daniel Bone, Julia Mertens, Emily Zane, Sungbok Lee, Shrikanth S Narayanan, and Ruth B Grossman. 2017. Acoustic-Prosodic and Physiological Response to Stressful Interactions in Children with Autism Spectrum Disorder. In INTERSPEECH. 147--151.
[10]
Yekta Said Can, Niaz Chalabianloo, Deniz Ekiz, and Cem Ersoy. 2019. Continuous stress detection using wearable sensors in real life: Algorithmic programming contest case study. Sensors 19, 8 (2019), 1849.
[11]
Jongyoon Choi, Beena Ahmed, and Ricardo Gutierrez-Osuna. 2011. Development and evaluation of an ambulatory stress monitor based on wearable sensors. IEEE transactions on information technology in biomedicine 16, 2 (2011), 279--286.
[12]
Lester Coleman, Jan Mitcheson, and Gareth Lloyd. 2013. Couple relationships: Why are they important for health and wellbeing? Journal of Health Visiting 1, 3 (2013), 168--172.
[13]
Eco J de Geus, Lorenz J Van Doornen, and Jacob F Orlebeke. 1993. Regular exercise and aerobic fitness in relation to psychological make-up and physiological stress reactivity. Psychosomatic medicine 55, 4 (1993), 347--363.
[14]
Agnete Skovlund Dissing, Tobias Bornakke Jørgensen, Thomas Alexander Gerds, Naja Hulvej Rod, and Rikke Lund. 2019. High perceived stress and social interaction behaviour among young adults. A study based on objective measures of face-to-face and smartphone interactions. PloS one 14, 7 (2019), e0218429.
[15]
Andreas Ejupi and Carlo Menon. 2018. Detection of talking in respiratory signals: A feasibility study using machine learning and wearable textile-based sensors. Sensors 18, 8 (2018), 2474.
[16]
E Ertin, N Stohs, S Kumar, A Raij, M al'Absi, T Kwon, S Mitra, S Shah, and J Jeong. 2011. AutoSense: Unobtrusively Wearable Sensor Suite for Inferencing of Onset, Causality, and Consequences of Stress in the Field. In ACM SenSys.
[17]
Andrew J Fuligni, Eva H Telzer, Julienne Bower, Steve W Cole, Lisa Kiang, and Michael R Irwin. 2009. A preliminary study of daily interpersonal stress and C-reactive protein levels among adolescents from Latin American and European backgrounds. Psychosomatic Medicine (2009).
[18]
Aditya Gujral, Theodora Chaspari, Adela C Timmons, Yehsong Kim, Sarah Barrett, and Gayla Margolin. 2018. Population-specific Detection of Couples' Interpersonal Conflict using Multi-task Learning. In Proceedings of the 2018 on International Conference on Multimodal Interaction. ACM.
[19]
Jennifer A Healey and Rosalind W Picard. 2005. Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems (2005), 156--166.
[20]
Karen Hovsepian, Mustafa al'Absi, Emre Ertin, Thomas Kamarck, Motohiro Nakajima, and Santosh Kumar. 2015. cStress: towards a gold standard for continuous stress assessment in the mobile environment. In ACM UbiComp.
[21]
Ryo Ishii, Xutong Ren, Michal Muszynski, and Louis-Philippe Morency. 2020. Can Prediction of Turn-management Willingness Improve Turn-changing Modeling?. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents. 1--8.
[22]
Chelsy Jungbluth, Ian M MacFarlane, Patricia McCarthy Veach, and Bonnie S LeRoy. 2011. Why is everyone so anxious?: an exploration of stress and anxiety in genetic counseling graduate students. Journal of Genetic Counseling 20, 3 (2011), 270--286.
[23]
Robert-Paul Juster, Bruce S McEwen, and Sonia J Lupien. 2010. Allostatic load biomarkers of chronic stress and impact on health and cognition. Neuroscience & Biobehavioral Reviews 35, 1 (2010), 2--16.
[24]
Daniel Kahneman, Alan B Krueger, David A Schkade, Norbert Schwarz, and Arthur A Stone. 2004. A survey method for characterizing daily life experience: The day reconstruction method. Science (2004), 1776--1780.
[25]
Janice K Kiecolt-Glaser, Jean-Philippe Gouin, and Liisa Hantsoo. 2010. Close relationships, inflammation, and health. Neuroscience & Biobehavioral Reviews 35, 1 (2010), 33--38.
[26]
Bishal Lamichhane, Ulf Großekathöfer, Giuseppina Schiavone, and Pierluigi Casale. 2017. Towards stress detection in real-life scenarios using wearable sensors: normalization factor to reduce variability in stress physiology. In eHealth 360°. 259--270.
[27]
Munhee Lee, Junhyung Moon, Dongmi Cheon, Juneil Lee, and Kyoungwoo Lee. 2020. Respiration signal based two layer stress recognition across non-verbal and verbal situations. In Proceedings of the 35th Annual ACM Symposium on Applied Computing. 638--645.
[28]
Youngki Lee, Chulhong Min, Chanyou Hwang, Jaeung Lee, Inseok Hwang, Younghyun Ju, Chungkuk Yoo, Miri Moon, Uichin Lee, and Junehwa Song. 2013. Sociophone: Everyday face-to-face interaction monitoring platform using multi-phone sensor fusion. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services. 375--388.
[29]
Iulia Lefter, Gertjan J Burghouts, and Léon JM Rothkrantz. 2015. Recognizing stress using semantics and modulation of speech and gestures. IEEE Transactions on Affective Computing 7, 2 (2015), 162--175.
[30]
Hong Lu, Denise Frauendorfer, Mashfiqui Rabbi, Marianne Schmid Mast, Gokul T Chittaranjan, Andrew T Campbell, Daniel Gatica-Perez, and Tanzeem Choudhury. 2012. Stresssense: Detecting stress in unconstrained acoustic environments using smartphones. In Proceedings of the 2012 ACM conference on ubiquitous computing. 351--360.
[31]
Raul Montoliu, Jan Blom, and Daniel Gatica-Perez. 2013. Discovering places of interest in everyday life from smartphone data. Springer Multimedia Tools and Applications (2013), 179--207.
[32]
Amir Muaremi, Bert Arnrich, and Gerhard Tröster. 2013. Towards measuring stress with smartphones and wearable devices during workday and sleep. BioNanoScience (2013).
[33]
Tamara Munzner. 2014. Visualization analysis and design. AK Peters/CRC Press.
[34]
Michael Myrtek and Georg Brügner. 1996. Perception of emotions in everyday life: studies with patients and normals. Biological psychology 42, 1-2 (1996), 147--164.
[35]
Annamalai Natarajan, Deepak Ganesan, and Benjamin M Marlin. 2019. Hierarchical Active Learning for Model Personalization in the Presence of Label Scarcity. In 2019 IEEE 16th International Conference on Wearable and Implantable Body Sensor Networks (BSN). 1--4.
[36]
Rajdeep Kumar Nath, Himanshu Thapliyal, and Allison Caban-Holt. 2020. Validating Physiological Stress Detection Model Using Cortisol as Stress Bio Marker. In 2020 IEEE International Conference on Consumer Electronics (ICCE). 1--5.
[37]
Kurt Plarre, Andrew Raij, Syed Monowar Hossain, Amin Ahsan Ali, Motohiro Nakajima, Mustafa al'absi, Emre Ertin, Thomas Kamarck, Santosh Kumar, Marcia Scott, et al. 2011. Continuous inference of psychological stress from sensory measurements collected in the natural environment. In ACM IPSN.
[38]
Sally I Powers, Paula R Pietromonaco, Meredith Gunlicks, and Aline Sayer. 2006. Dating couples' attachment styles and patterns of cortisol reactivity and recovery in response to a relationship conflict. Journal of personality and social psychology 90, 4 (2006), 613.
[39]
Md Mahbubur Rahman, Amin Ahsan Ali, Kurt Plarre, Mustafa Al'Absi, Emre Ertin, and Santosh Kumar. 2011. mconverse: Inferring conversation episodes from respiratory measurements collected in the field. In Proceedings of the 2nd Conference on Wireless Health. 1--10.
[40]
Md. Mahbubur Rahman, Amin Ahsan Ali, Kurt Plarre, Mustafa al'Absi, Emre Ertin, and Santosh Kumar. 2011. mConverse: Inferring Conversation Episodes from Respiratory Measurements Collected in the Field. In ACM Wireless Health.
[41]
Md Mahbubur Rahman, Rummana Bari, Amin Ahsan Ali, Moushumi Sharmin, Andrew Raij, Karen Hovsepian, Syed Monowar Hossain, Emre Ertin, Ashley Kennedy, David H Epstein, and Others. 2014. Are we there yet?: feasibility of continuous stress assessment via wireless physiological sensors. In Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics. ACM, 479--488.
[42]
G Senthil Raja and S Dandapat. 2010. Speaker recognition under stressed condition. International Journal of Speech Technology 13, 3 (2010), 141--161.
[43]
Raul I Ramos-Garcia, Stephen Tiffany, and Edward Sazonov. 2016. Using respiratory signals for the recognition of human activities. In 2016 38th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, 173--176.
[44]
Hillol Sarker, Karen Hovsepian, Soujanya Chatterjee, Inbal Nahum-Shani, Susan A Murphy, Bonnie Spring, Emre Ertin, Mustafa Al'Absi, Motohiro Nakajima, and Santosh Kumar. 2017. From markers to interventions: The case of just-in-time stress intervention. In Mobile health. 411--433.
[45]
Hillol Sarker, Matthew Tyburski, Md Mahbubur Rahman, Karen Hovsepian, Moushumi Sharmin, David H Epstein, Kenzie L Preston, C Debra Furr-Holden, Adam Milam, Inbal Nahum-Shani, et al. 2016. Finding significant stress episodes in a discontinuous time series of rapidly varying mobile sensor data. In Proceedings of the 2016 CHI conference on human factors in computing systems. 4489--4501.
[46]
Corina Sas, Scott Challioner, Christopher Clarke, Ross Wilson, Alina Coman, Sarah Clinch, Mike Harding, and Nigel Davies. 2015. Self-Defining Memory Cues: Creative Expression and Emotional Meaning. In ACM CHI Extended Abstract.
[47]
Philip Schmidt, Attila Reiss, Robert Duerichen, and Kristof Van Laerhoven. 2018. Wearable affect and stress recognition: A review. arXiv preprint arXiv:1811.08854 (2018).
[48]
Moushumi Sharmin, Andrew Raij, David Epstien, Inbal Nahum-Shani, J Gayle Beck, Sudip Vhaduri, Kenzie Preston, and Santosh Kumar. 2015. Visualization of time-series sensor data to inform the design of just-in-time adaptive stress interventions. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 505--516.
[49]
Douglas K Snyder and W Kim Halford. 2012. Evidence-based couple therapy: Current status and future directions. Journal of Family Therapy 34, 3 (2012), 229--249.
[50]
Graham B Spanier. 1976. Measuring dyadic adjustment: New scales for assessing the quality of marriage and similar dyads. Journal of Marriage and the Family (1976).
[51]
Gertraud Stadler, Kenzie A Snyder, Andrea B Horn, Patrick E Shrout, and Niall P Bolger. 2012. Close relationships and health in daily life: A review and empirical data on intimacy and somatic symptoms. Psychosomatic Medicine (2012).
[52]
Anna Ståhl, Kristina Höök, and Elsa Kosmack-Vaara. 2011. Reflecting on the Design Process of Affective Health. In IASDR.
[53]
Joachim Taelman, Steven Vandeput, Elke Vlemincx, Arthur Spaepen, and Sabine Van Huffel. 2011. Instantaneous changes in heart rate regulation due to mental load in simulated office work. European journal of applied physiology 111, 7 (2011), 1497--1505.
[54]
Karen P Tang, Jason I Hong, and Daniel P Siewiorek. 2011. Understanding how visual representations of location feeds affect end-user privacy concerns. In ACM UbiComp.
[55]
Adela C Timmons, Theodora Chaspari, Sohyun C Han, Laura Perrone, Shrikanth S Narayanan, and Gayla Margolin. 2017. Using multimodal wearable technology to detect conflict among couples. Computer 3 (2017), 50--59.
[56]
Sudip Vhaduri, Amin Ali, Moushumi Sharmin, Karen Hovsepian, and Santosh Kumar. 2014. Estimating drivers' stress from GPS traces. In Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. 1--8.
[57]
Stephanie J Wilson, Brittney E Bailey, Lisa M Jaremka, Christopher P Fagundes, Rebecca Andridge, William B Malarkey, Kathleen M Gates, and Janice K Kiecolt-Glaser. 2018. When couples' hearts beat together: Synchrony in heart rate variability during conflict predicts heightened inflammation throughout the day. Psychoneuroendocrinology 93 (2018), 107--116.
[58]
Danny Wyatt, Tanzeem Choudhury, and Henry Kautz. 2007. Capturing spontaneous conversation and social dynamics: A privacy-sensitive data collection effort. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP'07, Vol. 4. IEEE, IV-213.
[59]
Chenren Xu, Sugang Li, Gang Liu, Yanyong Zhang, Emiliano Miluzzo, Yih-Farn Chen, Jun Li, and Bernhard Firner. 2013. Crowd++ unsupervised speaker count with smartphones. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. 43--52.

Cited By

View all

Index Terms

  1. Automated Detection of Stressful Conversations Using Wearable Physiological and Inertial Sensors

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
      Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 4, Issue 4
      December 2020
      1356 pages
      EISSN:2474-9567
      DOI:10.1145/3444864
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 18 December 2020
      Published in IMWUT Volume 4, Issue 4

      Permissions

      Request permissions for this article.

      Check for updates

      Qualifiers

      • Research-article
      • Research
      • Refereed

      Funding Sources

      • NIH

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)175
      • Downloads (Last 6 weeks)11
      Reflects downloads up to 01 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media