Cited By
View all- Pan HZhao XHe LShi YLin X(2024)A survey of multimodal federated learning: background, applications, and perspectivesMultimedia Systems10.1007/s00530-024-01422-930:4Online publication date: 29-Jul-2024
Just Accepted
FedUSL: A Federated Annotation Method for Driving Fatigue Detection based on Multimodal Sensing Data
Accepted on 14 March 2024
Abstract
Single-modal data has a limitation on fatigue detection, while the shortage of labeled data is pervasive in multimodal sensing data. Besides, it is a time-consuming task for board-certified experts to manually annotate the physiological signals, especially hard for EEG sensor data. To solve this problem, we propose FedUSL (Federated Unified Space Learning), a federated annotation method for multimodal sensing data in the driving fatigue detection scenario, which has the innate ability to exploit more than four multimodal data simultaneously for correlations and complementary with low complexity. To validate the efficiency of the proposed method, we first collect the multimodal data (aka, camera, physiological sensor) through simulated fatigue driving. The data is then preprocessed and features are extracted to form a usable multimodal dataset. Based on the dataset, we analyze the performance of the proposed method. The experimental results demonstrate that FedUSL outperforms other approaches for driver fatigue detection with carefully selected modal combinations, especially when a modality contains only \(10\%\) labeled data.
References
[1]
Stockholm Declaration. 2020. Third global ministerial conference on road safety: achieving global goals 2030. (2020). https://www.roadsafetysweden.com/contentassets/b37f0951c837443eb9661668d5be439e/stockholm-declaration-english.pdf
[2]
Paul M Salmon, Gemma JM Read, Vanessa Beanland, Jason Thompson, Ashleigh J Filtness, Adam Hulme, Rod McClure, and Ian Johnston. 2019. Bad behaviour or societal failure? Perceptions of the factors contributing to drivers’ engagement in the fatal five driving behaviours. Applied ergonomics 74(2019), 162–171.
[3]
Ingrid van Schagen. 2020. European Road Safety Observatory Road Safety Thematic Report-Fatigue. (2020). https://road-safety.transport.ec.europa.eu/system/files/2021-07/road_safety_thematic_report_fatigue_tc_final.pdf.
[4]
SWOV. 2019. Fatigue. SWOV Fact sheet, September 2019. (2019). https://swov.nl/sites/default/files/bestanden/downloads/FS%20Fatigue.pdf.
[5]
Muneeb Ahmed, Sarfaraz Masood, Musheer Ahmad, and Ahmed A Abd El-Latif. 2021. Intelligent driver drowsiness detection for traffic safety based on multi CNN deep model and facial subsampling. IEEE Transactions on Intelligent Transportation Systems 23, 10(2021), 19743–19752.
[6]
Gulbadan Sikander and Shahzad Anwar. 2018. Driver fatigue detection systems: A review. IEEE Transactions on Intelligent Transportation Systems 20, 6(2018), 2339–2352.
[7]
Abdulhamit Subasi, Aditya Saikia, Kholoud Bagedo, Amarprit Singh, and Anil Hazarika. 2022. EEG-based driver fatigue detection using FAWT and multiboosting approaches. IEEE Transactions on Industrial Informatics 18, 10(2022), 6602–6609.
[8]
Rahul Bhardwaj and Venkatesh Balasubramanian. 2019. Viability of cardiac parameters measured unobtrusively using capacitive coupled electrocardiography (cECG) to estimate driver performance. IEEE Sensors Journal 19, 11 (2019), 4321–4330.
[9]
Hafeez Ur Rehman Siddiqui, Adil Ali Saleem, Robert Brown, Bahattin Bademci, Ernesto Lee, Furqan Rustam, and Sandra Dudley. 2021. Non-invasive driver drowsiness detection system. Sensors 21, 14 (2021), 4833.
[10]
Gang Wu, Zhangwei Liu, Xiaodong Pan, Feng Chen, Meng Xu, Deshan Feng, and Zhiguang Xia. 2017. Fatigue driving influence research and assessment. In Advances in Human Aspects of Transportation: Proceedings of the AHFE 2016 International Conference on Human Factors in Transportation, July 27-31, 2016, Walt Disney World®, Florida, USA. Springer, 677–688.
[11]
Mika Sunagawa, Shin-ichi Shikii, Wataru Nakai, Makoto Mochizuki, Koichi Kusukame, and Hiroki Kitajima. 2019. Comprehensive drowsiness level detection model combining multimodal information. IEEE Sensors Journal 20, 7 (2019), 3709–3717.
[12]
Boon-Giin Lee and Wan-Young Chung. 2012. Driver alertness monitoring using fusion of facial features and bio-signals. IEEE Sensors journal 12, 7 (2012), 2416–2422.
[13]
Luis Alfredo Moctezuma and Marta Molinas. 2019. Event-related potential from eeg for a two-step identity authentication system. In 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), Vol. 1. IEEE, 392–399.
[14]
Kaili Wang and Xiaohui Chen. 2019. Ppg signal identification method based on csasvm. In 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS). IEEE, 1–4.
[15]
Shekh MM Islam, Abraham Sylvester, George Orpilla, and Victor M Lubecke. 2020. Respiratory feature extraction for radar-based continuous identity authentication. In 2020 IEEE Radio and Wireless Symposium (RWS). IEEE, 119–122.
[16]
Vincenzo Catrambone, Alberto Greco, Giuseppe Averta, Matteo Bianchi, Gaetano Valenza, and Enzo Pasquale Scilingo. 2019. Predicting object-mediated gestures from brain activity: an EEG study on gender differences. IEEE Transactions on Neural Systems and Rehabilitation Engineering 27, 3(2019), 411–418.
[17]
Bushra Fatima, Aasim Raheel, Aamir Arsalan, Muhammad Majid, Muhammad Ehatisham-Ul-Haq, and Syed Muhammad Anwar. 2021. Gender Recognition using EEG during Mobile Game Play. In 2021 International Conference on Information Technology (ICIT). IEEE, 634–639.
[18]
Duo Chen, Haihong Zhang, Perumpadappil Thomas Kavitha, Fong Ling Loy, Soon Huat Ng, Chuanchu Wang, Kok Soon Phua, Soon Yin Tjan, Su-Yin Yang, and Cuntai Guan. 2022. Scalp EEG-based pain detection using convolutional neural network. IEEE Transactions on Neural Systems and Rehabilitation Engineering 30 (2022), 274–285.
[19]
Pedro M Rodrigues, Bruno C Bispo, Carolina Garrett, Dílio Alves, Joao P Teixeira, and Diamantino Freitas. 2021. Lacsogram: A new EEG tool to diagnose Alzheimer’s disease. IEEE Journal of Biomedical and Health Informatics 25, 9(2021), 3384–3395.
[20]
Baochen Xiong, Xiaoshan Yang, Fan Qi, and Changsheng Xu. 2022. A unified framework for multi-modal federated learning. Neurocomputing 480(2022), 110–118.
[21]
Yuxin Peng, Xiaohua Zhai, Yunzhen Zhao, and Xin Huang. 2015. Semi-supervised cross-media feature learning with unified patch graph regularization. IEEE transactions on circuits and systems for video technology 26, 3(2015), 583–596.
[22]
Xiaoshan Yang, Baochen Xiong, Yi Huang, and Changsheng Xu. 2022. Cross-Modal Federated Human Activity Recognition via Modality-Agnostic and Modality-Specific Representation Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 3063–3071.
[23]
Jianliang Min, Chen Xiong, Yonggang Zhang, and Ming Cai. 2021. Driver fatigue detection based on prefrontal EEG using multi-entropy measures and hybrid model. Biomedical Signal Processing and Control 69 (2021), 102857.
[24]
Chaojie Fan, Yong Peng, Shuangling Peng, Honghao Zhang, Yuankai Wu, and Sam Kwong. 2021. Detection of train driver fatigue and distraction based on forehead EEG: a time-series ensemble learning method. IEEE Transactions on Intelligent Transportation Systems 23, 8(2021), 13559–13569.
[25]
Sobhan Sheykhivand, Tohid Yousefi Rezaii, Zohreh Mousavi, Saeed Meshgini, Somaye Makouei, Ali Farzamnia, Sebelan Danishvar, and Kenneth Teo Tze Kin. 2022. Automatic Detection of Driver Fatigue Based on EEG Signals Using a Developed Deep Neural Network. Electronics 11, 14 (2022), 2169.
[26]
Andrea Nemcova, Veronika Svozilova, Kateřina Bucsuházy, Radovan Smíšek, Martin Mezl, Branislav Hesko, Michal Belak, Martin Bilik, Pavel Maxera, Martin Seitl, et al. 2021. Multimodal features for detection of driver stress and fatigue. IEEE Transactions on Intelligent Transportation Systems 22, 6(2021).
[27]
Kapotaksha Das, Salem Sharak, Kais Riani, Mohamed Abouelenien, Mihai Burzo, and Michalis Papakostas. 2021. Multimodal Detection of Drivers Drowsiness and Distraction. In Proceedings of the 2021 International Conference on Multimodal Interaction. 416–424.
[28]
Guangyi Zhang and Ali Etemad. 2021. Capsule attention for multimodal EEG-EOG representation learning with application to driver vigilance estimation. IEEE Transactions on Neural Systems and Rehabilitation Engineering 29 (2021), 1138–1149.
[29]
Mohammad Naim Rastgoo, Bahareh Nakisa, Frederic Maire, Andry Rakotonirainy, and Vinod Chandran. 2019. Automatic driver stress level classification using multimodal deep learning. Expert Systems with Applications 138 (2019), 112793.
[30]
Soo-Yeon Han, No-Sang Kwak, Taegeun Oh, and Seong-Whan Lee. 2020. Classification of pilots’ mental states using a multimodal deep learning network. Biocybernetics and Biomedical Engineering 40, 1 (2020), 324–336.
[31]
Imen Jegham, Anouar Ben Khalifa, Ihsen Alouani, and Mohamed Ali Mahjoub. 2020. Soft spatial attention-based multimodal driver action recognition using deep learning. IEEE Sensors Journal 21, 2 (2020), 1918–1925.
[32]
Fahad Sohrab, Jenni Raitoharju, Alexandros Iosifidis, and Moncef Gabbouj. 2021. Multimodal subspace support vector data description. Pattern Recognition 110(2021), 107648.
[33]
Xinghao Yang, Weifeng Liu, Wei Liu, and Dacheng Tao. 2019. A survey on canonical correlation analysis. IEEE Transactions on Knowledge and Data Engineering 33, 6(2019), 2349–2368.
[34]
Giuseppe Lisanti, Iacopo Masi, and Alberto Del Bimbo. 2014. Matching people across camera views using kernel canonical correlation analysis. In Proceedings of the international conference on distributed smart cameras. 1–6.
[35]
Yuhua Jia, Liang Bai, Shuang Liu, Peng Wang, Jinlin Guo, and Yuxiang Xie. 2019. Semantically-enhanced kernel canonical correlation analysis: a multi-label cross-modal retrieval. Multimedia Tools and Applications 78 (2019), 13169–13188.
[36]
Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik. 2014. A multi-view embedding space for modeling internet images, tags, and their semantics. International journal of computer vision 106 (2014), 210–233.
[37]
Yong Luo, Dacheng Tao, Kotagiri Ramamohanarao, Chao Xu, and Yonggang Wen. 2015. Tensor canonical correlation analysis for multi-view dimension reduction. IEEE transactions on Knowledge and Data Engineering 27, 11(2015), 3111–3124.
[38]
Xiaohua Zhai, Yuxin Peng, and Jianguo Xiao. 2013. Learning cross-media joint representation with sparse and semisupervised regularization. IEEE Transactions on Circuits and Systems for Video Technology 24, 6(2013), 965–978.
[39]
Feng Tian, Quge Wang, Xin Li, and Ning Sun. 2019. Heterogeneous multimedia cooperative annotation based on multimodal correlation learning. Journal of Visual Communication and Image Representation 58 (2019), 544–553.
[40]
Atiqa Zafar, Christian Prehofer, and Chih-Hong Cheng. 2021. Federated learning for driver status monitoring. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). IEEE, 1463–1469.
[41]
Xiaokang Zhou, Xuzhe Zheng, Xuesong Cui, Jiashuai Shi, Wei Liang, Zheng Yan, Laurance T Yang, Shohei Shimizu, I Kevin, and Kai Wang. 2023. Digital twin enhanced federated reinforcement learning with lightweight knowledge distillation in mobile networks. IEEE Journal on Selected Areas in Communications (2023).
[42]
Chen Zhao, Zhipeng Gao, Qian Wang, Kaile Xiao, Zijia Mo, and M Jamal Deen. 2023. FedSup: A communication-efficient federated learning fatigue driving behaviors supervision approach. Future Generation Computer Systems 138 (2023), 52–60.
[43]
Yuchen Zhao, Payam Barnaghi, and Hamed Haddadi. 2022. Multimodal federated learning on iot data. In 2022 IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementation (IoTDI). IEEE, 43–54.
[44]
Arijit Nandi and Fatos Xhafa. 2022. A federated learning method for real-time emotion state classification from multi-modal streaming. Methods 204(2022), 340–347.
[45]
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. 2022. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 8432–8440.
[46]
Feng Tian, Xianmei Liu, Zhuoxuan Liu, Ning Sun, Mei Wang, Haochang Wang, and Fengquan Zhang. 2019. Multimedia integrated annotation based on common space learning. Multimedia Tools and Applications 78 (2019), 437–456.
[47]
Liangli Zhen, Peng Hu, Xi Peng, Rick Siow Mong Goh, and Joey Tianyi Zhou. 2022. Deep Multimodal Transfer Learning for Cross-Modal Retrieval. IEEE Transactions on Neural Networks and Learning Systems 33, 2(2022), 798–810.
[48]
Jun Yin and Shiliang Sun. 2019. Multiview uncorrelated locality preserving projection. IEEE transactions on neural networks and learning systems 31, 9(2019), 3442–3455.
[49]
Hongjie Zhang, Junyan Tan, Jinxin Zhang, Yingyi Chen, and Ling Jing. 2022. Double information preserving canonical correlation analysis. Engineering Applications of Artificial Intelligence 112 (2022), 104870.
[50]
Anna Åkerstedt Miley, Göran Kecklund, and Torbjörn Åkerstedt. 2016. Comparing two versions of the Karolinska Sleepiness Scale (KSS). Sleep and biological rhythms 14 (2016), 257–260.
[51]
Mona Sazgar, Michael G Young, Mona Sazgar, and Michael G Young. 2019. Overview of EEG, electrode placement, and montages. Absolute Epilepsy and EEG Rotation Review: Essentials for Trainees (2019), 117–125.
[52]
Xucheng Liu, Gang Li, Sujie Wang, Feng Wan, Yi Sun, Hongtao Wang, Anastasios Bezerianos, Chuantao Li, and Yu Sun. 2021. Toward practical driving fatigue detection using three frontal EEG channels: A proof-of-concept study. Physiological Measurement 42, 4 (2021), 044003.
[53]
Arnaud Delorme and Scott Makeig. 2004. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of neuroscience methods 134, 1 (2004), 9–21.
[54]
Xiaokang Zhou, Xiaozhou Ye, I Kevin, Kai Wang, Wei Liang, Nirmal Kumar C Nair, Shohei Shimizu, Zheng Yan, and Qun Jin. 2023. Hierarchical federated learning with social context clustering-based participant selection for internet of medical things applications. IEEE Transactions on Computational Social Systems (2023).
[55]
Junnan Li and Qingsheng Zhu. 2020. A boosting self-training framework based on instance generation with natural neighbors for K nearest neighbor. Applied Intelligence 50, 11 (2020), 3535–3553.
[56]
Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. 2011. Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing 3, 1 (2011), 18–31.
[57]
Xiaokang Zhou, Wei Liang, I Kevin, Kai Wang, Zheng Yan, Laurence T Yang, Wei Wei, Jianhua Ma, and Qun Jin. 2023. Decentralized P2P Federated Learning for Privacy-Preserving and Resilient Mobile Robotic Systems. IEEE Wireless Communications 30, 2 (2023), 82–89.
Index Terms
- FedUSL: A Federated Annotation Method for Driving Fatigue Detection based on Multimodal Sensing Data
Recommendations
Client-Adaptive Cross-Model Reconstruction Network for Modality-Incomplete Multimodal Federated Learning
MM '23: Proceedings of the 31st ACM International Conference on MultimediaMultimodal federated learning (MFL) is an emerging field that allows many distributed clients, each with multimodal data, to work together to train models targeting multimodal tasks without sharing local data. Whereas, existing methods assume that all ...
Multimodal federated learning: Concept, methods, applications and future directions
AbstractMultimodal learning mines and analyzes multimodal data in reality to better understand and appreciate the world around people. However, how to exploit this rich multimodal data without violating user privacy is a key issue. Federated learning is ...
Highlights- The three different modes in the multimodal federated learning model are summarized.
- Multimodal fusion based on the federated learning framework is also specified.
- The difficulties and some ideas of multimodal federated learning ...
A Multimodal Contrastive Federated Learning for Digital Healthcare
AbstractDigital healthcare applications have gained enormous global interest due to the rapid development of the internet of medical things (IoMT), which helps access massive amounts of multimodal healthcare data. Using this rich multimodal data without ...
Comments
Information & Contributors
Information
Published In
EISSN:1550-4867
Table of ContentsCopyright © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Journal Family
Publication History
Online AM: 10 April 2024
Accepted: 14 March 2024
Revised: 12 January 2024
Received: 19 December 2022
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 371Total Downloads
- Downloads (Last 12 months)371
- Downloads (Last 6 weeks)56
Reflects downloads up to 24 Dec 2024
Other Metrics
Citations
Cited By
View all- Pan HZhao XHe LShi YLin X(2024)A survey of multimodal federated learning: background, applications, and perspectivesMultimedia Systems10.1007/s00530-024-01422-930:4Online publication date: 29-Jul-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in