research-article

AI-Assisted Food Intake Activity Recognition Using 3D mmWave Radars

Authors:

Shervin Shirmohammadi,

Cheng-Hsin HsuAuthors Info & Claims

MADiMa '22: Proceedings of the 7th International Workshop on Multimedia Assisted Dietary Management

Pages 81 - 89

https://doi.org/10.1145/3552484.3555753

Published: 24 October 2022 Publication History

Abstract

The automatic recognition of when and for how long a person is eating a certain food or drinking has applications in telecare, smarthome data monetization, and diet control. Existing food recognition systems either recognize the type of the food, but not when and for how long the person was eating and drinking, or use invasive sensors or privacy-intruding cameras which users are hesitant to install in their homes. In this paper, we propose a non-invasive system, using Artificial Intelligence to process 3D point cloud data collected from a 3D mmWave radar, that can distinguish a person's eating and drinking activities from other daily activities. This is challenging because eating and drinking activities are much more fine-grained than activities that existing systems can detect, such as sitting, running, walking, etc. Performance evaluations show that our proposed system significantly outperforms a representative state-of-the-art activity recognition system, RadHAR, by at least 27% and can reach 96.56% and 96.73% accuracy for two different training/testing split setups.

References

[1]

N. Ahmed, J. Rafiq, and M. Islam. 2020. Enhanced human activity recognition based on smartphone sensor data using hybrid feature selection model. Sensors 20, 1 (2020), 317:1--317:19.

[2]

S. An and U. Ogras. 2021. MARS: mmWave-based Assistive Rehabilitation System for Smart Healthcare. ACM Transactions on Embedded Computing Systems 20, 5s (2021), 1--22.

[3]

S. Balli, E. Saugbacs, and M. Peker. 2019. Human activity recognition from smart watch sensor data using a hybrid of principal component analysis and random forest algorithm. Measurement and Control 52, 1--2 (2019), 37--45.

[4]

F. Baradel, C. Wolf, and J. Mille. 2018. Human activity recognition with pose driven attention to RGB. In Proc. of British Machine Vision Conference (BMVC). 1--14.

[5]

S. Bhalla, M. Goel, and R. Khurana. 2021. IMU2Doppler: Cross-Modal Domain Adaptation for Doppler-based Activity Recognition Using IMU Data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 145:1--145:20.

[6]

D. Cook, K. Feuz, and N. Krishnan. 2013. Transfer learning for activity recognition: A survey. Knowledge and Information Systems 36, 3 (2013), 537 --556.

Digital Library

[7]

Dfintech. 2022. Cisco Visual Networking Index: Forecast and Methodology, 2016- 2021. Retrieved March 19, 2022 from https://dfintech.ch/en/

[8]

M. Farooq and E. Sazonov. 2018. Accelerometer-based detection of food intake in free-living individuals. IEEE Sensors Journal 18, 9 (2018), 3752--3758.

[9]

A. Franco, A. Magnani, and D. Maio. 2020. A multimodal approach for human activity recognition based on skeleton and RGB data. Pattern Recognition Letters 131 (2020), 293--299.

[10]

F. Fuchs, D. Worrall, V. Fischer, and M. Welling. 2020. SE(3)-Transformers: 3D rototranslation equivariant attention networks. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Vol. 33. 1970--1981.

[11]

P. Gong, C. Wang, and L. Zhang. 2021. Mmpoint-GNN: Graph neural network with dynamic edges for human activity recognition through a millimeter-wave radar. In Proc. of International Joint Conference on Neural Networks (IJCNN). 1--7.

[12]

L. Harnack, L. Steffen, D. Arnett, S. Gao, and R. Luepker. 2004. Accuracy of estimation of large food portions. Journal of the American Dietetic Association 104, 5 (2004), 804--806.

[13]

M. Hassan, M. Uddin, A. Mohamed, and A. Almogren. 2018. A robust human activity recognition system using smartphone sensors and deep learning. Future Generation Computer Systems 81 (2018), 307--313.

Digital Library

[14]

S. He, S. Li, A. Nag, S. Feng, T. Han, S. Mukhopadhyay, and W. Powel. 2020. A comprehensive review of the use of sensors for food intake detection. Sensors and Actuators A: Physical 315 (2020), 112318:1--112318:16.

[15]

J. Hu, W. Zheng, J. Lai, and J. Zhang. 2015. Jointly learning heterogeneous features for RGB-D activity recognition. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5344--5352.

[16]

A. Iosifidis, E. Marami, A. Tefas, and I. Pitas. 2012. Eating and drinking activity recognition based on discriminant analysis of fuzzy distances and activity volumes. In Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2201--2204.

[17]

H. Liu and T. Schultz. 2019. A wearable real-time human activity recognition system using biosensors integrated into a knee bandage. In Proc. of International Conference on Biomedical Electronics and Devices. 47--55.

[18]

S. Mekruksavanich and A. Jitpattanakul. 2020. Smartwatch-based human activity recognition using hybrid lstm network. (2020), 1--4.

[19]

W. Min, S. Jiang, L. Liu, Y. Rui, and R. Jain. 2019. A survey on food computing. Comput. Surveys 52, 5 (2019), 1--36.

[20]

W. Min, L. Liu, Z. Luo, and S. Jiang. 2019. Ingredient-guided cascaded multi attention network for food recognition. In Proc. ACM International Conference on Multimedia (MM). 1331--1339.

[21]

A. Moin, A. Zhou, A. Rahimi, A. Menon, S. Benatti, G. Alexandrov, S. Tamakloe, J. Ting, N. Yamamoto, Y. Khan, et al. 2021. A wearable biosensing system with in sensor adaptive machine learning for hand gesture recognition. Nature Electronics 4, 1 (2021), 54--63.

[22]

C. Qi, L. Yi, H. Su, and L. Guibas. 2017. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems 30 (2017).

[23]

J. Qi, G. Jiang, G. Li, Y. Sun, and B. Tao. 2019. Intelligent human-computer interaction based on surface EMG gesture recognition. IEEE Access 7 (2019), 61378--61387.

[24]

N. Rashid, M. Dautta, P. Tseng, and M. Faruque. 2020. HEAR: Fog-enabled energy aware online human eating activity recognition. IEEE Internet of Things Journal 8, 2 (2020), 860--868.

[25]

G. Riegler, A. Osman Ulusoy, and A. Geiger. 2017. Octnet: Learning deep 3D representations at high resolutions. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3577--3586.

[26]

A. Salehzadeh, A. Calitz, and J. Greyling. 2020. Human activity recognition using deep electroencephalography learning. Biomedical Signal Processing and Control 62 (2020), 102094.

[27]

N. Selamat and S. Ali. 2020. Automatic food intake monitoring based on chewing activity: A survey. IEEE Access 8 (2020), 48846--48869.

[28]

A. Singh, S. Sandha, L. Garcia, and M. Srivastava. 2019. Radhar: Human activity recognition from point clouds generated through a millimeter-wave radar. In Proc. of the ACM Workshop on Millimeter-wave Networks and Sensing Systems (mmNets). 51--56.

[29]

T. Singh and D. Vishwakarma. 2021. A deeply coupled ConvNet for human activity recognition using dynamic and RGB images. Neural Computing and Applications 33, 1 (2021), 469--485.

Digital Library

[30]

A. Stisen, H. Blunck, S. Bhattacharya, T. Prentow, M. Kjaergaard, A. Dey, T. Sonne, and M. Jensen. 2015. Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In Proc. of ACM Conference on Embedded Networked Sensor Systems (SenSys). 127--140.

[31]

K. Verma and B. Singh. 2021. Deep multi-Model fusion for human activity recognition using evolutionary algorithms. International Journal of Interactive Multimedia & Artificial Intelligence 7, 2 (2021).

[32]

C. Wang, Z. Lin, Y. Xie, X. Guo, Y. Ren, and Y. Chen. 2020. WiEat: Fine-grained device-free eating monitoring leveraging Wi-Fi signals. (2020), 1--9.

[33]

Y. Wang, H. Liu, K. Cui, A. Zhou, W. Li, and H. Ma. 2021. m-Activity: Accurate and Real-Time Human Activity Recognition Via Millimeter Wave Radar. In ICASSP 2021--2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 8298--8302.

[34]

G. Weiss, K. Yoneda, and T. Hayajneh. 2019. Smartphone and smartwatch-based biometrics using activities of daily living. IEEE Access 7 (2019), 133190--133202.

[35]

A. Wellnitz, J. Wolff, C. Haubelt, and T. Kirste. 2019. Fluid intake recognition using inertial sensors. In Proc. of international Workshop on Sensor-based Activity Recognition and Interaction (iWOAR). 1--7.

[36]

Z. Wharton, A. Behera, Y. Liu, and N. Bessis. 2021. Coarse temporal attention network (cta-net) for driver's activity recognition. In Proc. of IEEE Winter Conference on Applications of Computer Vision (WACV). 1279--1289.

[37]

Y. Xie, R. Jiang, X. Guo, Y. Wang, J. Cheng, and Y. Chen. 2022. mmEat: Millimeter wave-enabled environment-invariant eating behavior monitoring. Smart Health 23 (2022), 10023:1--10023:8.

[38]

K. Yatani and K. Truong. 2012. Bodyscope: a wearable acoustic sensor for activity recognition. In Proc. of ACM Conference on Ubiquitous Computing (UbiComp). 341--350

Cited By

Ghosian Moghaddam MAsghar Nazari Shirehjini AShirmohammadi S(2025)Device-Free Human Activity Recognition: A Systematic Literature ReviewIEEE Open Journal of Instrumentation and Measurement10.1109/OJIM.2024.35028854(1-34)Online publication date: 2025
https://doi.org/10.1109/OJIM.2024.3502885
Raeis HKazemi MShirmohammadi S(2024)CAE-MAS: Convolutional Autoencoder Interference Cancellation for Multiperson Activity Sensing With FMCW Microwave RadarIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.336657573(1-10)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3366575
Wang CKumar TDe Raedt WCamps GHallez HVanrumste B(2024)Evaluation Metrics for Food Intake Activity Recognition Using Segment-Wise IoU2024 IEEE International Symposium on Medical Measurements and Applications (MeMeA)10.1109/MeMeA60663.2024.10596709(1-6)Online publication date: 26-Jun-2024
https://doi.org/10.1109/MeMeA60663.2024.10596709
Show More Cited By

Index Terms

AI-Assisted Food Intake Activity Recognition Using 3D mmWave Radars
1. Information systems
  1. Information systems applications
    1. Multimedia information systems

Recommendations

Memory-Efficient High-Accuracy Food Intake Activity Recognition with 3D mmWave Radars
MADiMa '23: Proceedings of the 8th International Workshop on Multimedia Assisted Dietary Management

Non-invasive and privacy-preserving recognition of food intake activities has applications in diet management, telecare, and smarthome data monetization. In this paper, we tackle the challenging problem of recognizing food intake activity using sparse ...
A Dataset of Food Intake Activities Using Sensors with Heterogeneous Privacy Sensitivity Levels
MMSys '23: Proceedings of the 14th ACM Multimedia Systems Conference

Human activity recognition, which involves recognizing human activities from sensor data, has drawn a lot of interest from researchers and practitioners as a result of the advent of smart homes, smart cities, and smart systems. Existing studies on ...
A Hierarchical Approach in Food and Drink Intake Recognition Using Wearable Inertial Sensors
PETRA '18: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference

Despite the increasing attention given to inertial sensors for Human Activity Recognition (HAR), efforts are principally focused on fitness applications where quasi-periodic activities like walking or running are studied. In contrast, activities like ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MADiMa '22: Proceedings of the 7th International Workshop on Multimedia Assisted Dietary Management

October 2022

97 pages

ISBN:9781450395021

DOI:10.1145/3552484

General Chairs:
Stavroula Mougiakakou
University of Bern, Switzerland
,
Giovanni Maria Farinella
University of Catania, Italy
,
Keiji Yanai
The University of Electro-Communications, Tokyo, Japan
,
Dario Allegra
University of Catania, Italy

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 10, 2022

Lisboa, Portugal

Acceptance Rates

MADiMa '22 Paper Acceptance Rate 9 of 10 submissions, 90%;

Overall Acceptance Rate 16 of 24 submissions, 67%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
373
Total Downloads

Downloads (Last 12 months)121
Downloads (Last 6 weeks)15

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ghosian Moghaddam MAsghar Nazari Shirehjini AShirmohammadi S(2025)Device-Free Human Activity Recognition: A Systematic Literature ReviewIEEE Open Journal of Instrumentation and Measurement10.1109/OJIM.2024.35028854(1-34)Online publication date: 2025
https://doi.org/10.1109/OJIM.2024.3502885
Raeis HKazemi MShirmohammadi S(2024)CAE-MAS: Convolutional Autoencoder Interference Cancellation for Multiperson Activity Sensing With FMCW Microwave RadarIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.336657573(1-10)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3366575
Wang CKumar TDe Raedt WCamps GHallez HVanrumste B(2024)Evaluation Metrics for Food Intake Activity Recognition Using Segment-Wise IoU2024 IEEE International Symposium on Medical Measurements and Applications (MeMeA)10.1109/MeMeA60663.2024.10596709(1-6)Online publication date: 26-Jun-2024
https://doi.org/10.1109/MeMeA60663.2024.10596709
Ahmed SAbdullah SCho S(2024)Advancements in Radar Point Cloud Processing for Macro Human Movements in Healthcare and Assisted Living Domains: A ReviewIEEE Sensors Journal10.1109/JSEN.2024.345211024:22(36287-36305)Online publication date: 15-Nov-2024
https://doi.org/10.1109/JSEN.2024.3452110
Wang CKumar TDe Raedt WCamps GHallez HVanrumste B(2024)Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network With AttentionIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2023.333970328:2(1000-1011)Online publication date: Feb-2024
https://doi.org/10.1109/JBHI.2023.3339703
Diraco GRescio GSiciliano PLeone A(2023)Review on Human Action Recognition in Smart Living: Sensing Technology, Multimodality, Real-Time Processing, Interoperability, and Resource-Constrained ProcessingSensors10.3390/s2311528123:11(5281)Online publication date: 2-Jun-2023
https://doi.org/10.3390/s23115281
Moghaddam MNazari Shirehjini AShirmohammadi S(2023)Device-Free Fine-Grained Dining Activity Sensing2023 IEEE Sensors Applications Symposium (SAS)10.1109/SAS58821.2023.10254010(1-6)Online publication date: 18-Jul-2023
https://doi.org/10.1109/SAS58821.2023.10254010
Wang CKumar TDe Raedt WCamps GHallez HVanrumste B(2023)Evaluation of Different Radar Placements for Food Intake Monitoring Using Deep Learning2023 IEEE Radar Conference (RadarConf23)10.1109/RadarConf2351548.2023.10149626(1-6)Online publication date: 1-May-2023
https://doi.org/10.1109/RadarConf2351548.2023.10149626

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents