skip to main content
10.1145/3581783.3611754acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Sensing Micro-Motion Human Patterns using Multimodal mmRadar and Video Signal for Affective and Psychological Intelligence

Published: 27 October 2023 Publication History

Abstract

Affective and psychological perception are pivotal in human-machine interaction and essential domains within artificial intelligence. Existing physiological signal-based affective and psychological datasets primarily rely on contact-based sensors, potentially introducing extraneous affectives during the measurement process. Consequently, creating accurate non-contact affective and psychological perception datasets is crucial for overcoming these limitations and advancing affective intelligence. In this paper, we introduce the Remote Multimodal Affective and Psychological (ReMAP) dataset, for the first time, apply head micro-tremor (HMT) signals for affective and psychological perception. ReMAP features 68 participants and comprises two sub-datasets. The stimuli videos utilized for affective perception undergo rigorous screening to ensure the efficacy and universality of affective elicitation. Additionally, we propose a novel remote affective and psychological perception framework, leveraging multimodal complementarity and interrelationships to enhance affective and psychological perception capabilities. Extensive experiments demonstrate HMT as a "small yet powerful" physiological signal in psychological perception. Our method outperforms existing state-of-the-art approaches in remote affective recognition and psychological perception. The ReMAP dataset is publicly accessible at https://remap-dataset.github.io/ReMAP.

References

[1]
Mojtaba Khomami Abadi, Ramanathan Subramanian, Seyed Mostafa Kia, Paolo Avesani, Ioannis Patras, and Nicu Sebe. 2015. DECAF: MEG-based multimodal database for decoding affective physiological responses. IEEE Trans. Affect. Comput., Vol. 6, 3 (2015), 209--222.
[2]
Foteini Agrafioti, Dimitrios Hatzinakos, and Adam K. Anderson. 2012. ECG Pattern Analysis for Emotion Detection. IEEE Trans. Affect. Comput., Vol. 3, 1 (2012), 102--115. https://doi.org/10.1109/T-AFFC.2011.28
[3]
Yuri Agrawal, Daniel M Merfeld, Fay B Horak, Mark S Redfern, Brad Manor, Kelly P Westlake, Gay R Holstein, Paul F Smith, Tanvi Bhatt, Nicolaas I Bohnen, et al. 2020. Aging, vestibular function, and balance: proceedings of a national institute on aging/national institute on deafness and other communication disorders workshop. The Journals of Gerontology: Series A, Vol. 75, 12 (2020), 2471--2480.
[4]
Md. Shad Akhtar, Deepanway Ghosal, Asif Ekbal, Pushpak Bhattacharyya, and Sadao Kurohashi. 2022. All-in-One: Emotion, Sentiment and Intensity Prediction Using a Multi-Task Ensemble Framework. IEEE Trans. Affect. Comput., Vol. 13, 1 (2022), 285--297. https://doi.org/10.1109/TAFFC.2019.2926724
[5]
Mark S Allen and Sylvain Laborde. 2014. The role of personality in sport and physical activity. Current directions in psychological science, Vol. 23, 6 (2014), 460--465.
[6]
Oscar Araque, Lorenzo Gatti, Jacopo Staiano, and Marco Guerini. 2022. DepecheMood: A Bilingual Emotion Lexicon Built Through Simple Yet Powerful Techniques. IEEE Trans. Affect. Comput., Vol. 13, 1 (2022), 496--507. https://doi.org/10.1109/TAFFC.2019.2934444
[7]
Carey D Balaban. 2004. Projections from the parabrachial nucleus to the vestibular nuclei: potential substrates for autonomic and limbic influences on vestibular responses. Brain research, Vol. 996, 1 (2004), 126--137.
[8]
Carey D Balaban and Julian F Thayer. 2001. Neurological bases for balance--anxiety links. Journal of anxiety disorders, Vol. 15, 1--2 (2001), 53--79.
[9]
S Marc Breedlove and Neil V Watson. 2013. Biological psychology: An introduction to behavioral, cognitive, and clinical neuroscience. Sinauer Associates.
[10]
Thomas Brown. 2012. Physiological psychology. Elsevier.
[11]
Joseph E Carmona, Alissa K Holland, and David W Harrison. 2009. Extending the functional cerebral systems theory of emotion to the vestibular modality: a systematic and integrative approach. Psychological bulletin, Vol. 135, 2 (2009), 286.
[12]
Mohamed Dahmane, Jahangir Alam, Pierre-Luc St-Charles, Marc Lalonde, Kevin Heffner, and Samuel Foucher. 2022. A Multimodal Non-Intrusive Stress Monitoring From the Pleasure-Arousal Emotional Dimensions. IEEE Trans. Affect. Comput., Vol. 13, 2 (2022), 1044--1056. https://doi.org/10.1109/TAFFC.2020.2988455
[13]
Marianne Dieterich and Thomas Brandt. 2019. Perception of verticality and vestibular disorders of balance and falls. Frontiers in neurology, Vol. 10 (2019), 172.
[14]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
[15]
Paul Ekman and Wallace V Friesen. 2003. Unmasking the face: A guide to recognizing emotions from facial clues. Vol. 10. Ishk.
[16]
Christoph Feichtenhofer. 2020. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 203--213.
[17]
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision. 6202--6211.
[18]
Patrick A Forbes, Gunter P Siegmund, Alfred C Schouten, and Jean-Sébastien Blouin. 2015. Task, muscle and frequency dependent vestibular control of posture. Frontiers in integrative neuroscience, Vol. 8 (2015), 94.
[19]
Rahma Fourati, Boudour Ammar, Javier J. Sánchez Medina, and Adel M. Alimi. 2022. Unsupervised Learning in Reservoir Computing for EEG-Based Emotion Recognition. IEEE Trans. Affect. Comput., Vol. 13, 2 (2022), 972--984. https://doi.org/10.1109/TAFFC.2020.2982143
[20]
Songcheng Gao, Wenzhong Li, Lynda J. Song, Xiao Zhang, Mingkai Lin, and Sanglu Lu. 2020. PersonalitySensing: A Multi-View Multi-Task Learning Approach for Personality Detection based on Smartphone Usage. In MM '20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann (Eds.). ACM, 2862--2870. https://doi.org/10.1145/3394171.3413591
[21]
Caroline Gurvich, Jerome J Maller, Brian Lithgow, Saman Haghgooie, and Jayashri Kulkarni. 2013. Vestibular insights into cognition and psychiatry. Brain research, Vol. 1537 (2013), 244--259.
[22]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[23]
Zhenliang He, Meina Kan, Jie Zhang, Xilin Chen, and Shiguang Shan. 2017. A fully end-to-end cascaded cnn for facial landmark detection. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017). IEEE, 200--207.
[24]
Cheng-Ju Hsieh, Wei-Hao Chung, and Chiou-Ting Hsu. 2022. Augmentation of rPPG Benchmark Datasets: Learning to Remove and Embed rPPG Signals via Double Cycle Consistent Learning from Unpaired Facial Videos. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI (Lecture Notes in Computer Science, Vol. 13676), Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer, 372--387. https://doi.org/10.1007/978-3-031-19787-1_21
[25]
Joshua Zhexue Huang, Michael K Ng, Hongqiang Rong, and Zichen Li. 2005. Automated variable weighting in k-means type clustering. IEEE transactions on pattern analysis and machine intelligence, Vol. 27, 5 (2005), 657--668.
[26]
Ziyu Jia, Youfang Lin, Xiyang Cai, Haobin Chen, Haijun Gou, and Jing Wang. 2020. Sst-emotionnet: Spatial-spectral-temporal based attention 3d dense network for eeg emotion recognition. In Proceedings of the 28th ACM international conference on multimedia. 2909--2917.
[27]
Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. 2020. DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild. In MM '20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann (Eds.). ACM, 2881--2889. https://doi.org/10.1145/3394171.3413620
[28]
Júlio C. S. Jacques Júnior, Yagmur Güçlütürk, Marc Pérez, Umut Güşlü, Carlos Andú jar, Xavier Baró, Hugo Jair Escalante, Isabelle Guyon, Marcel A. J. van Gerven, Rob van Lier, and Sergio Escalera. 2022. First Impressions: A Survey on Vision-Based Apparent Personality Trait Analysis. IEEE Trans. Affect. Comput., Vol. 13, 1 (2022), 75--95. https://doi.org/10.1109/TAFFC.2019.2930058
[29]
Nick Kanas, Gro Mjeldheim Sandal, Jennifer E Boyd, Vadim I Gushin, Dietrich Manzey, Regina North, Gloria R Leon, Peter Suedfeld, Sheryl L Bishop, Edna R Fiedler, et al. 2013. Psychology and culture during long-duration space missions. Springer.
[30]
Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. 2011. Deap: A database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput., Vol. 3, 1 (2011), 18--31.
[31]
Benjamin Kossack, Eric L. Wisotzky, Peter Eisert, Sebastian P. Schraven, Brigitta Globke, and Anna Hilsmann. 2022. Perfusion assessment via local remote photoplethysmography (rPPG). In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2022, New Orleans, LA, USA, June 19-20, 2022. IEEE, 2191--2200. https://doi.org/10.1109/CVPRW56347.2022.00238
[32]
Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn. 2019. Context-Aware Emotion Recognition Networks. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 10142--10151. https://doi.org/10.1109/ICCV.2019.01024
[33]
Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. 2021. Improved multiscale vision transformers for classification and detection. arXiv preprint arXiv:2112.01526 (2021).
[34]
Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. 2022. MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild. CoRR, Vol. abs/2208.00847 (2022). https://doi.org/10.48550/arXiv.2208.00847 showeprint[arXiv]2208.00847
[35]
Anna J Matheson, Cynthia L Darlington, and Paul F Smith. 1999. Dizziness in the elderly and age-related degeneration of the vestibular system. New Zealand journal of psychology, Vol. 28, 1 (1999), 10.
[36]
Seyed Mohammad Mavadati, Mohammad H. Mahoor, Kevin Bartlett, Philip Trinh, and Jeffrey F. Cohn. 2013. DISFA: A Spontaneous Facial Action Intensity Database. IEEE Trans. Affect. Comput., Vol. 4, 2 (2013), 151--160. https://doi.org/10.1109/T-AFFC.2013.4
[37]
Juan Abdon Miranda-Correa, Mojtaba Khomami Abadi, Nicu Sebe, and Ioannis Patras. 2018. Amigos: A dataset for affect, personality and mood research on individuals and groups. IEEE Trans. Affect. Comput., Vol. 12, 2 (2018), 479--493.
[38]
Diana E Mitchell and Kathleen E Cullen. 2016. Vestibular system. The Curated Reference Collection in Neuroscience and Biobehavioral Psychology (2016), 127--132.
[39]
Gelareh Mohammadi and Patrik Vuilleumier. 2022. A Multi-Componential Approach to Emotion Recognition and the Effect of Personality. IEEE Trans. Affect. Comput., Vol. 13, 3 (2022), 1127--1139. https://doi.org/10.1109/TAFFC.2020.3028109
[40]
Laura P Naumann, Simine Vazire, Peter J Rentfrow, and Samuel D Gosling. 2009. Personality judgments based on physical appearance. Personality and social psychology bulletin, Vol. 35, 12 (2009), 1661--1671.
[41]
Marco Perugini and Lisa Di Blas. 2002. Analyzing personality related adjectives from an eticemic perspective: the big five marker scales (BFMS) and the Italian AB5C taxonomy. Big Five Assessment (2002), 281--304.
[42]
Anh Viet Phan, Minh Le Nguyen, Yen Lam Hoang Nguyen, and Lam Thu Bui. 2018. Dgcnn: A convolutional neural network over large-scale labeled graphs. Neural Networks, Vol. 108 (2018), 533--543.
[43]
AM Soza Ried and M Aviles. 2007. Asymmetries of vestibular dysfunction in major depression. Neuroscience, Vol. 144, 1 (2007), 128--134.
[44]
James A Russell. 1980. A circumplex model of affect. Journal of personality and social psychology, Vol. 39, 6 (1980), 1161.
[45]
Pritam Sarkar and Ali Etemad. 2022. Self-Supervised ECG Representation Learning for Emotion Recognition. IEEE Trans. Affect. Comput., Vol. 13, 3 (2022), 1541--1554. https://doi.org/10.1109/TAFFC.2020.3014842
[46]
Zilong Shao, Siyang Song, Shashank Jaiswal, Linlin Shen, Michel F. Valstar, and Hatice Gunes. 2021. Personality Recognition by Modelling Person-specific Cognitive Processes using Graph Representation. In MM '21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021, Heng Tao Shen, Yueting Zhuang, John R. Smith, Yang Yang, Pablo César, Florian Metze, and Balakrishnan Prabhakaran (Eds.). ACM, 357--366. https://doi.org/10.1145/3474085.3475460
[47]
Mohammad Soleymani, Jeroen Lichtenauer, Thierry Pun, and Maja Pantic. 2012. A Multimodal Database for Affect Recognition and Implicit Tagging. IEEE Trans. Affect. Comput., Vol. 3, 1 (2012), 42--55. https://doi.org/10.1109/T-AFFC.2011.25
[48]
Tengfei Song, Suyuan Liu, Wenming Zheng, Yuan Zong, Zhen Cui, Yang Li, and Xiaoyan Zhou. 2023. Variational Instance-Adaptive Graph for EEG Emotion Recognition. IEEE Trans. Affect. Comput., Vol. 14, 1 (2023), 343--356. https://doi.org/10.1109/TAFFC.2021.3064940
[49]
Ramanathan Subramanian, Julia Wache, Mojtaba Khomami Abadi, Radu L Vieriu, Stefan Winkler, and Nicu Sebe. 2016. ASCERTAIN: Emotion and personality recognition using commercial sensors. IEEE Trans. Affect. Comput., Vol. 9, 2 (2016), 147--160.
[50]
Yao Wang, Tao Gu, Tom H. Luan, Minjie Lyu, and Yue Li. 2022. HeartPrint: Exploring a Heartbeat-Based Multiuser Authentication With Single mmWave Radar. IEEE Internet Things J., Vol. 9, 24 (2022), 25324--25336. https://doi.org/10.1109/JIOT.2022.3196143
[51]
Lynn Williams and Amanda Wingate. 2012. Type D personality, physical symptoms and subjective stress: The mediating effects of coping and social support. Psychology & health, Vol. 27, 9 (2012), 1075--1085.
[52]
Kathryn E Wilson and Rodney K Dishman. 2015. Personality and physical activity: A systematic review and meta-analysis. Personality and individual differences, Vol. 72 (2015), 230--242.
[53]
Lotta Winter, Tillmann HC Kruger, Jean Laurens, Harald Engler, Manfred Schedlowski, Dominik Straumann, and M Axel Wollmer. 2012. Vestibular stimulation on a motion-simulator impacts on mood states. Frontiers in psychology, Vol. 3 (2012), 499.
[54]
Xueyuan Xu, Tianyuan Jia, Qing Li, Fulin Wei, Long Ye, and Xia Wu. 2023. EEG Feature Selection via Global Redundancy Minimization for Emotion Recognition. IEEE Trans. Affect. Comput., Vol. 14, 1 (2023), 421--435. https://doi.org/10.1109/TAFFC.2021.3068496
[55]
Qi Yang, Sergey Nikolenko, Alfred Huang, and Aleksandr Farseev. 2022. Personality-Driven Social Multimedia Content Recommendation. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022, Jo a o Magalh a es, Alberto Del Bimbo, Shin'ichi Satoh, Nicu Sebe, Xavier Alameda-Pineda, Qin Jin, Vincent Oria, and Laura Toni (Eds.). ACM, 7290--7299. https://doi.org/10.1145/3503161.3548769
[56]
Le Zhang, Songyou Peng, and Stefan Winkler. 2022. PersEmoN: A Deep Network for Joint Analysis of Apparent Personality, Emotion and Their Relationship. IEEE Trans. Affect. Comput., Vol. 13, 1 (2022), 298--305. https://doi.org/10.1109/TAFFC.2019.2951656
[57]
Wei-Long Zheng and Bao-Liang Lu. 2015. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on autonomous mental development, Vol. 7, 3 (2015), 162--175.

Cited By

View all
  • (2024)Body-Area Capacitive or Electric Field Sensing for Human Activity Recognition and Human-Computer InteractionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435558:1(1-49)Online publication date: 6-Mar-2024

Index Terms

  1. Sensing Micro-Motion Human Patterns using Multimodal mmRadar and Video Signal for Affective and Psychological Intelligence

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      MM '23: Proceedings of the 31st ACM International Conference on Multimedia
      October 2023
      9913 pages
      ISBN:9798400701085
      DOI:10.1145/3581783
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 27 October 2023

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. affective
      2. head micro-tremor
      3. personality
      4. physiological signal

      Qualifiers

      • Research-article

      Funding Sources

      • Beijing Nova Program
      • Natural Science Foundation of China
      • National Key R&D Program of China
      • China Postdoctoral Science Foundation

      Conference

      MM '23
      Sponsor:
      MM '23: The 31st ACM International Conference on Multimedia
      October 29 - November 3, 2023
      Ottawa ON, Canada

      Acceptance Rates

      Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)191
      • Downloads (Last 6 weeks)7
      Reflects downloads up to 07 Nov 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Body-Area Capacitive or Electric Field Sensing for Human Activity Recognition and Human-Computer InteractionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435558:1(1-49)Online publication date: 6-Mar-2024

      View Options

      Get Access

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media