research-article

An Activity Recognition System for Taking Medicine Using In-The-Wild Data to Promote Medication Adherence

Authors:

Tracy HammondAuthors Info & Claims

IUI '21: Proceedings of the 26th International Conference on Intelligent User Interfaces

Pages 575 - 584

https://doi.org/10.1145/3397481.3450673

Published: 14 April 2021 Publication History

Abstract

Nearly half of people prescribed medication to treat chronic or short-term conditions do not take their medicine as prescribed. This leads to worse treatment outcomes, higher hospital admission rates, increased healthcare costs, and increased morbidity and mortality rates. While some instances of medication non-adherence are a result of problems with the treatment plan or barriers caused by the health care provider, many are instances caused by patient-related factors such as forgetting, running out of medication, and not understanding the required dosages. This presents a clear need for patient-centered systems that can reliably increase medication adherence. To that end, in this work we describe an activity recognition system capable of recognizing when individuals take medication in an unconstrained, real-world environment. Our methodology uses a modified version of the Bagging ensemble method to suit unbalanced data and a classifier trained on the prediction probabilities of the Bagging classifier to identify when individuals took medication during a full-day study. Using this methodology we are able to recognize when individuals took medication with an F-measure of 0.77. Our system is a first step towards developing personal health interfaces that are capable of providing personalized medication adherence interventions.

References

[1]

Zahraa S Abdallah, Mohamed Medhat Gaber, Bala Srinivasan, and Shonali Krishnaswamy. 2018. Activity recognition with evolving data streams: A review. ACM Computing Surveys (CSUR) 51, 4 (2018), 71.

Digital Library

[2]

Folami Alamudun, Hong-Jun Yoon, Kathleen B Hudson, Garnetta Morin-Ducote, Tracy Hammond, and Georgia D Tourassi. 2017. Fractal analysis of visual search activity for mass detection during mammographic screening. Medical physics 44, 3 (2017), 832–846.

[3]

Murtadha Aldeer, Jorge Ortiz, Richard E Howard, and Richard P Martin. 2019. PatientSense: patient discrimination from in-bottle sensors data. In Proceedings of the 16th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services. ACM, Houston, Texas, USA, 143–152.

Digital Library

[4]

Parviz Asghari, Elnaz Soleimani, and Ehsan Nazerfard. 2020. Online human activity recognition employing hierarchical hidden Markov models. Journal of Ambient Intelligence and Humanized Computing 11, 3 (2020), 1141–1152.

[5]

Oresti Banos, Miguel Damas, Alberto Guillen, Luis-Javier Herrera, Hector Pomares, Ignacio Rojas, and Claudia Villalonga. 2015. Multi-sensor fusion based on asymmetric decision weighting for robust activity recognition. Neural Processing Letters 42, 1 (2015), 5–26.

Digital Library

[6]

Ling Bao and Stephen S Intille. 2004. Activity recognition from user-annotated acceleration data. In Pervasive Computing. Springer, Linz and Vienna, Austria, 1–17.

[7]

James S. Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems 24, J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger(Eds.). Curran Associates, Inc., Granada, Spain, 2546–2554. http://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf

[8]

Leo Breiman. 1996. Bagging predictors. Machine learning 24, 2 (1996), 123–140.

[9]

Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5–32.

Digital Library

[10]

Kam Capoccia, Peggy S Odegard, and Nancy Letassy. 2016. Medication adherence with diabetes medication: a systematic review of the literature. The Diabetes Educator 42, 1 (2016), 34–71.

[11]

Kyle D Checchi, Krista F Huybrechts, Jerry Avorn, and Aaron S Kesselheim. 2014. Electronic medication packaging devices and medication adherence: a systematic review. Jama 312, 12 (2014), 1237–1247.

[12]

Chen Chen, Nasser Kehtarnavaz, and Roozbeh Jafari. 2014. A medication adherence monitoring system for pill bottles based on a wearable inertial sensor. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, IEEE, New York, NY, USA, 4983–4986.

[13]

Liming Chen, Chris D Nugent, and Hui Wang. 2011. A knowledge-driven approach to activity recognition in smart homes. IEEE Transactions on Knowledge and Data Engineering 24, 6(2011), 961–974.

Digital Library

[14]

Josh Cherian, Vijay Rajanna, Daniel Goldberg, and Tracy Hammond. 2017. Did you Remember To Brush? : A Noninvasive Wearable Approach to Recognizing Brushing Teeth for Elderly Care. In 11th EAI International Conference on Pervasive Computing Technologies for Healthcare. ACM, Barcelona, Spain, 48–57. https://dl.acm.org/citation.cfm?id=3154866.

Digital Library

[15]

Lindsey E Dayer, Rebecca Shilling, Madalyn Van Valkenburg, Bradley C Martin, Paul O Gubbins, Kristie Hadden, and Seth Heldenbrand. 2017. Assessing the medication adherence app marketplace from the health professional and consumer vantage points. JMIR mHealth and uHealth 5, 4 (2017), e45.

[16]

Stefan Dernbach, Barnan Das, Narayanan C Krishnan, Brian L Thomas, and Diane J Cook. 2012. Simple and complex activity recognition through smart phones. In Intelligent Environments (IE), 2012 8th International Conference on. IEEE, Guanajuato, Mexico, 214–221.

Digital Library

[17]

Iram Fatima, Muhammad Fahim, Young-Koo Lee, and Sungyoung Lee. 2013. A Genetic Algorithm-based Classifier Ensemble Optimization for Activity Recognition in Smart Homes.KSII Transactions on Internet & Information Systems 7, 11 (2013), 21 pages.

[18]

Donya Fozoonmayeh, Hai Vu Le, Ekaterina Wittfoth, Chong Geng, Natalie Ha, Jingjue Wang, Maria Vasilenko, Yewon Ahn, and Diane Myung-kyung Woodbridge. 2020. A Scalable Smartwatch-Based Medication Intake Detection System Using Distributed Machine Learning. Journal of Medical Systems 44, 4 (2020), 1–14.

[19]

Enrique Garcia-Ceja, Carlos E Galván-Tejada, and Ramon Brena. 2018. Multi-view stacking for activity recognition with sound and accelerometer data. Information Fusion 40(2018), 45–56.

Digital Library

[20]

Anil K Gehi, Sadia Ali, Beeya Na, and Mary A Whooley. 2007. Self-reported medication adherence and cardiovascular events in patients with stable coronary heart disease: the heart and soul study. Archives of Internal Medicine 167, 16 (2007), 1798–1803.

[21]

Katherine A Grosset, Ian Bone, and Donald G Grosset. 2005. Suboptimal medication adherence in Parkinson’s disease. Movement disorders: official journal of the Movement Disorder Society 20, 11(2005), 1502–1507.

[22]

Donghai Guan, Weiwei Yuan, Young-Koo Lee, Andrey Gavrilov, and Sungyoung Lee. 2007. Activity recognition based on semi-supervised learning. In 13th IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA 2007). IEEE, IEEE, New York, NY, USA, 469–475.

Digital Library

[23]

Niloofar Hezarjaribi, Ramin Fallahzadeh, and Hassan Ghasemzadeh. 2016. A machine learning approach for medication adherence monitoring using body-worn sensors. In 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, IEEE, New York, NY, USA, 842–845.

[24]

Tâm Huynh and Bernt Schiele. 2006. Towards less supervision in activity recognition from wearable sensors. In 2006 10th IEEE International Symposium on Wearable Computers. IEEE, IEEE, New York, NY, USA, 3–10.

[25]

Aurel O Iuga and Maura J McGuire. 2014. Adherence and health care costs. Risk management and healthcare policy 7 (2014), 35.

[26]

Haik Kalantarian, Nabil Alshurafa, Ebrahim Nemati, Tuan Le, and Majid Sarrafzadeh. 2015. A smartwatch-based medication adherence system. In 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN). IEEE, IEEE, New York, NY, USA, 1–6.

[27]

H. Kalantarian, N. Alshurafa, and M. Sarrafzadeh. 2016. Detection of Gestures Associated With Medication Adherence Using Smartwatch-Based Inertial Sensors. IEEE Sensors Journal 16, 4 (Feb 2016), 1054–1061. https://doi.org/10.1109/JSEN.2015.2497279

[28]

Haik Kalantarian, Babak Motamed, Nabil Alshurafa, and Majid Sarrafzadeh. 2016. A wearable sensor system for medication adherence prediction. Artificial intelligence in medicine 69 (2016), 43–52.

[29]

Jennifer R Kwapisz, Gary M Weiss, and Samuel A Moore. 2011. Activity recognition using cell phone accelerometers. ACM SigKDD Explorations Newsletter 12, 2 (2011), 74–82.

Digital Library

[30]

Gierad Laput and Chris Harrison. 2019. Sensing Fine-Grained Hand Activity with Smartwatches. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow, Scotland, 1–13.

Digital Library

[31]

Oscar D Lara and Miguel A Labrador. 2013. A survey on human activity recognition using wearable sensors. Communications Surveys & Tutorials, IEEE 15, 3 (2013), 1192–1209.

[32]

Young-Seol Lee and Sung-Bae Cho. 2014. Activity recognition with android phone using mixture-of-experts co-trained with labeled and unlabeled data. Neurocomputing 126(2014), 106–115.

Digital Library

[33]

Brent Longstaff, Sasank Reddy, and Deborah Estrin. 2010. Improving activity classification for health applications on mobile devices using active and semi-supervised learning. In 2010 4th International Conference on Pervasive Computing Technologies for Healthcare. IEEE, IEEE, New York, NY, USA, 1–7.

[34]

Jinxin Ma, Anaelia Ovalle, and Diane Myung-kyung Woodbridge. 2018. Medhere: A smartwatch-based medication adherence monitoring system using machine learning and distributed computing. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, Honolulu, Hawaii, USA, 4945–4948.

[35]

Victor Mendiola, Abnob Doss, Will Adams, Jose Ramos, Matthew Bruns, Josh Cherian, Puneet Kohli, Daniel Goldberg, and Tracy Hammond. 2019. Automatic exercise recognition with machine learning. In International Workshop on Health Intelligence. Springer, Honolulu, Hawaii, USA, 33–44.

[36]

Alexander Mertens, Christopher Brandl, Talya Miron-Shatz, Christopher Schlick, Till Neumann, Andreas Kribben, Sven Meister, Clarissa Jonas Diamantidis, Urs-Vito Albrecht, Peter Horn, 2016. A mobile application improves therapy-adherence rates in elderly patients undergoing rehabilitation: a crossover design study comparing documentation via iPad with paper-based control. Medicine 95, 36 (2016), 1–8.

[37]

Bobak Jack Mortazavi, Mohammad Pourhomayoun, Gabriel Alsheikh, Nabil Alshurafa, Sunghoon Ivan Lee, and Majid Sarrafzadeh. 2014. Determining the single best axis for exercise repetition recognition and counting on smartwatches. In 2014 11th International Conference on Wearable and Implantable Body Sensor Networks. IEEE, Zurich, Switzerland, 33–38.

Digital Library

[38]

Vasily Moshnyaga, Masaki Koyanagi, Fumiyuki Hirayama, Akihisa Takahama, and Koji Hashimoto. 2016. A medication adherence monitoring system for people with dementia. In 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, Budapest, Hungary, 000194–000199.

Digital Library

[39]

Andrea B Neiman, Todd Ruppar, Michael Ho, Larry Garber, Paul J Weidle, Yuling Hong, Mary G George, and Phoebe G Thorpe. 2017. CDC grand rounds: improving medication adherence for chronic disease management—innovations and opportunities. MMWR. Morbidity and mortality weekly report 66, 45 (2017), 1248.

[40]

Lars Osterberg and Terrence Blaschke. 2005. Adherence to medication. New England Journal of Medicine 353, 5 (2005), 487–497.

[41]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.

Digital Library

[42]

Nishkam Ravi, Nikhil Dandekar, Preetham Mysore, and Michael L Littman. 2005. Activity recognition from accelerometer data. In AAAI, Vol. 5. AAAI, Palo Alto, CA, USA, 1541–1546.

[43]

Bernardino Romera-Paredes, Min SH Aung, and Nadia Bianchi-Berthouze. 2013. A One-Vs-One Classifier Ensemble With Majority Voting for Activity Recognition. In ESANN. ESANN, Bruges, Belgium, 6 pages.

[44]

Charissa Ann Ronao and Sung-Bae Cho. 2016. Human activity recognition with smartphone sensors using deep learning neural networks. Expert Systems with Applications 59 (2016), 235–244.

Digital Library

[45]

Loren J Schleiden, Olufunmilola K Odukoya, and Michelle A Chui. 2015. Older adults’ perceptions of e-prescribing: impact on patient care. Perspectives in health information management 12, Winter(2015), 15 pages.

[46]

Linda Simoni-Wastila, Yu-Jung Wei, Jingjing Qian, Ilene H Zuckerman, Bruce Stuart, Thomas Shaffer, Anand A Dalal, and Lynda Bryant-Comstock. 2012. Association of chronic obstructive pulmonary disease maintenance medication adherence with all-cause hospitalization and spending in a Medicare population. The American journal of geriatric pharmacotherapy 10, 3(2012), 201–210.

[47]

Maja Stikic, Kristof Van Laerhoven, and Bernt Schiele. 2008. Exploring semi-supervised and active learning for activity recognition. In 2008 12th IEEE International Symposium on Wearable Computers. IEEE, IEEE, New York, NY, USA, 81–88.

Digital Library

[48]

Surescripts. 2011. The National Progress Report on e-prescribing and interoperable health care.

[49]

Christina Tarantola. 2017. The Top Reminder Apps for Patients. https://www.pharmacytimes.com/contributor/christina-tarantola/2017/12/the-top-medication-reminder-apps-for-patients Last Accessed Sept. 29, 2020.

[50]

Edison Thomaz, Irfan Essa, and Gregory D Abowd. 2015. A practical approach for recognizing eating moments with wrist-mounted inertial sensing. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, Osaka, Japan, 1029–1040.

Digital Library

[51]

George Vavoulas, Charikleia Chatzaki, Thodoris Malliotakis, Matthew Pediaditis, and Manolis Tsiknakis. 2016. The MobiAct Dataset: Recognition of Activities of Daily Living using Smartphones. In ICT4AgeingWell. SCITEPRESS, Setúbal, Portugal, 143–151.

[52]

Rui Wang, Zdeňka Sitová, Xiaoqing Jia, Xiang He, Tobi Abramson, Paolo Gasti, Kiran S Balagani, and Aydin Farajidavar. 2014. Automatic identification of solid-phase medication intake using wireless wearable accelerometers. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, IEEE, New York, NY, USA, 4168–4171.

Cited By

Cherian JRay STaele PKoh JHammond T(2024)Exploring the Impact of the NULL Class on In-the-Wild Human Activity RecognitionSensors10.3390/s2412389824:12(3898)Online publication date: 16-Jun-2024
https://doi.org/10.3390/s24123898
Cherian JRay SMernar TTaele PMach HKoh JYe PHammond T(2024)A Step Toward Better Care: Understanding What Caregivers and Residents in Assisted Living Facilities Value in Health Monitoring SystemsProceedings of the ACM on Human-Computer Interaction10.1145/36372908:CSCW1(1-29)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637290
Bolesnikov ACochrane KGirouard A(2023)Wearable Identities: Understanding Wearables’ Potential for Supporting the Expression of Queer IdentitiesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581327(1-19)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581327

Index Terms

An Activity Recognition System for Taking Medicine Using In-The-Wild Data to Promote Medication Adherence

Index terms have been assigned to the content through auto-classification.

Recommendations

A personalized medication management platform (PMMP) to improve medication adherence

Addressed medication non-adherence, an important aspect in healthcare which significantly influences patients health and expenses to patients, healthcare providers and insurance companies.The PMMP platform implemented and tested through a randomized ...
Smart medication management system and multiple interventions for medication adherence

To keep healthcare costs under control, a high-level of medication adherence, or compliance with medication regimen, must be achieved. The multifaceted nature of medication adherence, due to a large number of underlying factors, presents several ...
A Framework for Studying Patterns of Effective Medication Adherence

Medication adherence has been studied extensively in the healthcare literature. Most of the studies focus on improving medication adherence using interventions, including those based on wireless and mobile technologies, and measure average medication ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '21: Proceedings of the 26th International Conference on Intelligent User Interfaces

April 2021

618 pages

ISBN:9781450380171

DOI:10.1145/3397481

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 April 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

IUI '21

Sponsor:

IUI '21: 26th International Conference on Intelligent User Interfaces

April 14 - 17, 2021

TX, College Station, USA

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
242
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cherian JRay STaele PKoh JHammond T(2024)Exploring the Impact of the NULL Class on In-the-Wild Human Activity RecognitionSensors10.3390/s2412389824:12(3898)Online publication date: 16-Jun-2024
https://doi.org/10.3390/s24123898
Cherian JRay SMernar TTaele PMach HKoh JYe PHammond T(2024)A Step Toward Better Care: Understanding What Caregivers and Residents in Assisted Living Facilities Value in Health Monitoring SystemsProceedings of the ACM on Human-Computer Interaction10.1145/36372908:CSCW1(1-29)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637290
Bolesnikov ACochrane KGirouard A(2023)Wearable Identities: Understanding Wearables’ Potential for Supporting the Expression of Queer IdentitiesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581327(1-19)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581327

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents