research-article

Experiences in Building a Real-World Eating Recogniser

Authors:

Vigneshwaran Subbaraju,

Rajesh Krishna Balan,

Youngki LeeAuthors Info & Claims

WPA '17: Proceedings of the 4th International on Workshop on Physical Analytics

Pages 7 - 12

https://doi.org/10.1145/3092305.3092306

Published: 19 June 2017 Publication History

Abstract

In this paper, we describe the progressive design of the gesture recognition module of an automated food journaling system -- Annapurna. Annapurna runs on a smartwatch and utilises data from the inertial sensors to first identify eating gestures, and then captures food images which are presented to the user in the form of a food journal. We detail the lessons we learnt from multiple in-the-wild studies, and show how eating recognizer is refined to tackle challenges such as (i) high gestural diversity, and (ii) non-eating activities with similar gestural signatures. Annapurna is finally robust (identifying eating across a wide diversity in food content, eating styles and environments) and accurate (false-positive and false-negative rates of 6.5% and 3.3% respectively)

References

[1]

Amft, O., Junker, H., and Troster, G. Detection of eating and drinking arm gestures using inertial body-worn sensors. Wearable Computers. International Symposium on, 2005.

Digital Library

[2]

Dong, Y., Hoover, A., Scisco, J., and Muth, E. A new method for measuring meal intake in humans via wrist motion tracking. Applied psychophysiology and biofeedback, 2012.

[3]

Lee, J., Banerjee, A., and Gupta, S. K. Mt-diet: Automated smartphone based diet assessment with infrared images. Pervasive Computing and Communications (PerCom), 2016.

[4]

Liu, J., Johns, E., Atallah, L., Pettitt, C., Lo, B., Frost, G., and Yang, G.-Z. An intelligent food-intake monitoring system using wearable sensors. Wearable and Implantable Body Sensor Networks (BSN), 9th International Conference on, 2012.

Digital Library

[5]

Liu, L., Karatas, C., Li, H., Tan, S., Gruteser, M., Yang, J., Chen, Y., and Martin, R. P. Toward detection of unsafe driving with wearables. Proceedings of the 2015 Workshop on Wearable Systems and Applications, 2015.

Digital Library

[6]

Mirtchouk, M., Merck, C., and Kleinberg, S. Automated estimation of food type and amount consumed from body-worn audio and motion sensors. Proceedings of the ACM Conference on Pervasive and Ubiquitous Computing, 2016.

Digital Library

[7]

Parate, A., Chiu, M.-C., Chadowitz, C., Ganesan, D., and Kalogerakis, E. Risq: Recognizing smoking gestures with inertial sensors on a wristband. 12th International Conference on Mobile systems, applications, and services, 2014.

Digital Library

[8]

Rahman, T. et al. Bodybeat: A mobile system for sensing non-speech body sounds. International Conference on Mobile Systems, Applications, and Services, 2014.

Digital Library

[9]

Reddy, S., Parker, A., Hyman, J., Burke, J., Estrin, D., and Hansen, M. Image browsing, processing, and clustering for participatory sensing: lessons from a dietsense prototype. 4th workshop on Embedded networked sensors, 2007.

Digital Library

[10]

Sen, S., Grover, K., Subbaraju, V., and Misra, A. Inferring smartphone keypress via smartwatch inertial sensing. Pervasive Computing and Communication Workshops (PerCom Workshops), IEEE International Conference on, 2017.

[11]

Sen, S., Subbaraju, V., Misra, A., Balan, R. K., and Lee, Y. The case for smartwatch-based diet monitoring. Pervasive Computing and Communication Workshops (PerCom Workshops), IEEE International Conference on, 2015.

[12]

Thomaz, E., Essa, I., and Abowd, G. D. A practical approach for recognizing eating moments with wrist-mounted inertial sensing. Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing, 2015.

Digital Library

[13]

Wang, C., Guo, X., Wang, Y., Chen, Y., and Liu, B. Friend or foe?: Your wearable devices reveal your personal pin. Proceedings of the 11th ACM on Asia Conference on Computer and Communications Security, 2016.

Digital Library

[14]

Xu, C., Pathak, P. H., and Mohapatra, P. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch. 16th International Workshop on Mobile Computing Systems and Applications, 2015.

Digital Library

[15]

Yan, Z., Subbaraju, V., Chakraborty, D., Misra, A., and Aberer, K. Energy-efficient continuous activity recognition on mobile phones: An activity-adaptive approach. Wearable Computers (ISWC), 16th International Symposium on, 2012.

Digital Library

Cited By

Chen XKamavuako E(2023)Vision-Based Methods for Food and Fluid Intake Monitoring: A Literature ReviewSensors10.3390/s2313613723:13(6137)Online publication date: 4-Jul-2023
https://doi.org/10.3390/s23136137
Lu XThomaz EEpstein D(2022)Understanding People's Perceptions of Approaches to Semi-Automated Dietary MonitoringProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35502886:3(1-27)Online publication date: 7-Sep-2022
https://dl.acm.org/doi/10.1145/3550288
Rantala EBalatsas-Lekkas ASozer NPennanen K(2022)Overview of objective measurement technologies for nutrition research, food-related consumer and marketing researchTrends in Food Science & Technology10.1016/j.tifs.2022.05.006125(100-113)Online publication date: Jul-2022
https://doi.org/10.1016/j.tifs.2022.05.006
Show More Cited By

Index Terms

Experiences in Building a Real-World Eating Recogniser
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Empirical studies in ubiquitous and mobile computing

Recommendations

Assisting Food Journaling with Automatic Eating Detection
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems

In this work we study the feasibility and usability of an assistive food journaling system that sends users just-in-time reminders when unique hand gestures during food consumption are detected using a smartwatch. Our study shows that participants were ...
A practical approach for recognizing eating moments with wrist-mounted inertial sensing
UbiComp '15: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing

Recognizing when eating activities take place is one of the key challenges in automated food intake monitoring. Despite progress over the years, most proposed approaches have been largely impractical for everyday usage, requiring multiple on-body ...
Inferring Meal Eating Activities in Real World Settings from Ambient Sounds: A Feasibility Study
IUI '15: Proceedings of the 20th International Conference on Intelligent User Interfaces

Dietary self-monitoring has been shown to be an effective method for weight-loss, but it remains an onerous task despite recent advances in food journaling systems. Semi-automated food journaling can reduce the effort of logging, but often requires that ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WPA '17: Proceedings of the 4th International on Workshop on Physical Analytics

June 2017

50 pages

ISBN:9781450349581

DOI:10.1145/3092305

Program Chairs:
Robert LiKamWa
Arizona State University, USA
,
Rajesh Krishna Balan
Singapore Management University, Singapore

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

In-Cooperation

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 June 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Research Foundation, Prime Minister's Office, Singapore
Singapore Ministry of Education Academic Research Fund Tier 2

Conference

MobiSys'17

Sponsor:

SIGMOBILE

MobiSys'17: The 15th Annual International Conference on Mobile Systems, Applications, and Services

June 19, 2017

New York, Niagara Falls, USA

Acceptance Rates

Overall Acceptance Rate 11 of 17 submissions, 65%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
214
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen XKamavuako E(2023)Vision-Based Methods for Food and Fluid Intake Monitoring: A Literature ReviewSensors10.3390/s2313613723:13(6137)Online publication date: 4-Jul-2023
https://doi.org/10.3390/s23136137
Lu XThomaz EEpstein D(2022)Understanding People's Perceptions of Approaches to Semi-Automated Dietary MonitoringProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35502886:3(1-27)Online publication date: 7-Sep-2022
https://dl.acm.org/doi/10.1145/3550288
Rantala EBalatsas-Lekkas ASozer NPennanen K(2022)Overview of objective measurement technologies for nutrition research, food-related consumer and marketing researchTrends in Food Science & Technology10.1016/j.tifs.2022.05.006125(100-113)Online publication date: Jul-2022
https://doi.org/10.1016/j.tifs.2022.05.006
Bi SLu YTobias NRyan EMasterson TSen SHalter RSorber JGilbert-Diamond DKotz D(2020)Measuring children’s eating behavior with a wearable device2020 IEEE International Conference on Healthcare Informatics (ICHI)10.1109/ICHI48887.2020.9374304(1-11)Online publication date: Nov-2020
https://doi.org/10.1109/ICHI48887.2020.9374304
Selamat NAli S(2020)Automatic Food Intake Monitoring Based on Chewing Activity: A SurveyIEEE Access10.1109/ACCESS.2020.29782608(48846-48869)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2978260
Bell BAlam RAlshurafa NThomaz EMondol Ade la Haye KStankovic JLach JSpruijt-Metz D(2020)Automatic, wearable-based, in-field eating detection approaches for public health research: a scoping reviewnpj Digital Medicine10.1038/s41746-020-0246-23:1Online publication date: 13-Mar-2020
https://doi.org/10.1038/s41746-020-0246-2
Heydarian HAdam MBurrows TCollins CRollo M(2019)Assessing Eating Behaviour Using Upper Limb Mounted Motion Sensors: A Systematic ReviewNutrients10.3390/nu1105116811:5(1168)Online publication date: 24-May-2019
https://doi.org/10.3390/nu11051168
Lee JPaudyal PBanerjee AGupta S(2019)A User-adaptive Modeling for Eating Action Identification from Wristband Time SeriesACM Transactions on Interactive Intelligent Systems10.1145/33001499:4(1-35)Online publication date: 10-Oct-2019
https://dl.acm.org/doi/10.1145/3300149
Jaiswal DGhose AMuralidharan K(2018)Continuous Activity Recognition Using Smart WatchesProceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers10.1145/3267305.3267681(662-665)Online publication date: 8-Oct-2018
https://dl.acm.org/doi/10.1145/3267305.3267681
Bi SWang TTobias NNordrum JWang SHalvorsen GSen SPeterson ROdame KCaine KHalter RSorber JKotz D(2018)AuracleProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/32649022:3(1-27)Online publication date: 18-Sep-2018
https://dl.acm.org/doi/10.1145/3264902
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents