research-article

ExtraSensory App: Data Collection In-the-Wild with Rich User Interface to Self-Report Behavior

Authors:

Yonatan Vaizman,

Katherine Ellis,

Gert Lanckriet,

Nadir WeibelAuthors Info & Claims

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Paper No.: 554, Pages 1 - 12

https://doi.org/10.1145/3173574.3174128

Published: 21 April 2018 Publication History

Abstract

We introduce a mobile app for collecting in-the-wild data, including sensor measurements and self-reported labels describing people's behavioral context (e.g., driving, eating, in class, shower). Labeled data is necessary for developing context-recognition systems that serve health monitoring, aging care, and more. Acquiring labels without observers is challenging and previous solutions compromised ecological validity, range of behaviors, or amount of data. Our user interface combines past and near-future self-reporting of combinations of relevant context-labels. We deployed the app on the personal smartphones of 60 users and analyzed quantitative data collected in-the-wild and qualitative user-experience reports. The interface's flexibility was important to gain frequent, detailed labels, support diverse behavioral situations, and engage different users: most preferred reporting their past behavior through a daily journal, but some preferred reporting what they're about to do. We integrated insights from this work back into the app, which we make available to researchers for conducting in-the-wild studies.

References

[1]

Frank Bentley and Konrad Tollmar. 2013. The Power of Mobile Notifications to Increase Wellbeing Logging Behavior. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1095--1098.

Digital Library

[2]

Daniel Castro, Steven Hickson, Vinay Bettadapura, Edison Thomaz, Gregory Abowd, Henrik Christensen, and Irfan Essa. 2015. Predicting daily activities from egocentric images using deep learning. In Proceedings of the 2015 ACM International Symposium on Wearable Computers. ACM, 75--82.

Digital Library

[3]

Tanzeem Choudhury, Sunny Consolvo, Beverly Harrison, Jeffrey Hightower, Anthony LaMarca, Louis LeGrand, Ali Rahimi, Adam Rea, G Bordello, Bruce Hemingway, and others. 2008. The mobile sensing platform: An embedded activity recognition system. IEEE Pervasive Computing 7, 2 (2008).

Digital Library

[4]

Sunny Consolvo, David W McDonald, Tammy Toscos, Mike Y Chen, Jon Froehlich, Beverly Harrison, Predrag Klasnja, Anthony LaMarca, Louis LeGrand, Ryan Libby, and others. 2008. Activity sensing in the wild: a field trial of ubifit garden. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1797--1806.

Digital Library

[5]

Mihaly Csikszentmihalyi and Reed Larson. 2014. Validity and reliability of the experience-sampling method. In Flow and the foundations of positive psychology. Springer, 35--54.

[6]

Yujie Dong, Jenna Scisco, Mike Wilson, Eric Muth, and Adam Hoover. 2014. Detecting periods of eating during free-living by tracking wrist motion. IEEE journal of biomedical and health informatics 18, 4 (2014), 1253--1260.

[7]

Katherine Ellis, Suneeta Godbole, Jacqueline Kerr, and Gert Lanckriet. 2014. Multi-Sensor physical activity recognition in free-living. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. ACM, 431--440.

Digital Library

[8]

Raghu Kiran Ganti, Soundararajan Srinivasan, and Aca Gacic. 2010. Multisensor fusion in smartphones for lifestyle monitoring. In 2010 International Conference on Body Sensor Networks. IEEE, 36--43.

Digital Library

[9]

John J Guiry, Pepijn van de Ven, and John Nelson. 2014. Multi-Sensor Fusion for Enhanced Contextual Awareness of Everyday Activities with Ubiquitous Devices. Sensors 14 (2014), 5687--5701.

[10]

Erik B Hekler, Predrag Klasnja, Vicente Traver, and Monique Hendriks. 2013. Realizing effective behavioral management of health: the metamorphosis of behavioral science methods. IEEE pulse 4, 5 (2013), 29--34.

[11]

Daniel Kahneman, Alan B Krueger, David A Schkade, Norbert Schwarz, and Arthur A Stone. 2004. A survey method for characterizing daily life experience: The day reconstruction method. Science 306, 5702 (2004), 1776--1780.

[12]

Jacqueline Kerr, Ruth E Patterson, Katherine Ellis, Suneeta Godbole, Eileen Johnson, Gert Lanckriet, and John Staudenmayer. 2016. Objective Assessment of Physical Activity: Classifiers for Public Health. Medicine and science in sports and exercise 48, 5 (2016), 951--957.

[13]

Adil Mehmood Khan, Ali Tufail, Asad Masood Khattak, and Teemu H Laine. 2014. Activity recognition on smartphones via sensor-fusion and kda-based svms. International Journal of Distributed Sensor Networks 2014 (2014).

[14]

Abby C King, Eric B Hekler, Lauren A Grieco, Sandra J Winter, Jylana L Sheats, Matthew P Buman, Banny Banerjee, Thomas N Robinson, and Jesse Cirimele. 2013. Harnessing different motivational frames via mobile phones to promote daily physical activity and reduce sedentary behavior in aging adults. PloS one 8, 4 (2013), e62613.

[15]

Kai Kunze and Paul Lukowicz. 2014. Sensor placement variations in wearable activity recognition. IEEE Pervasive Computing 13, 4 (2014), 32--41.

[16]

Matthew L Lee and Anind K Dey. 2015. Sensor-based observations of daily living for aging in place. Personal and Ubiquitous Computing 19, 1 (2015), 27--43.

Digital Library

[17]

Beth Logan and others. 2000. Mel Frequency Cepstral Coefficients for Music Modeling. In ISMIR.

[18]

Gloria Mark, Shamsi T. Iqbal, Mary Czerwinski, Paul Johns, and Akane Sano. 2016. Neurotics Can'T Focus: An in Situ Study of Online Multitasking in the Workplace. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1739--1744.

Digital Library

[19]

Abhinav Mehrotra, Veljko Pejovic, Jo Vermeulen, Robert Hendley, and Mirco Musolesi. 2016. My Phone and Me: Understanding People's Receptivity to Mobile Notifications. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1021--1032.

Digital Library

[20]

Hamed Pirsiavash and Deva Ramanan. 2012. Detecting activities of daily living in first-person camera views. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2847--2854.

Digital Library

[21]

Susanna Pirttikangas, Kaori Fujinami, and Tatsuo Nakajima. 2006. Feature selection and activity recognition from wearable sensors. In International Symposium on Ubiquitious Computing Systems. Springer, 516--527.

Digital Library

[22]

Tauhidur Rahman, Mi Zhang, Stephen Voida, and Tanzeem Choudhury. 2014. Towards Accurate Non-Intrusive Recollection of Stress Levels Using Mobile Sensing and Contextual Recall. In International Conference on Pervasive Computing Technologies for Healthcare.

Digital Library

[23]

Christie Napa Scollon, Chu-Kim Prieto, and Ed Diener. 2009. Experience sampling: promises and pitfalls, strength and weaknesses. In Assessing well-being. Springer, 157--180.

[24]

Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjærgaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. 2015. Smart Devices are Different: Assessing and MitigatingMobile Sensing Heterogeneities for Activity Recognition. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems. ACM, 127--140.

Digital Library

[25]

Edison Thomaz, Irfan Essa, and Gregory D Abowd. 2015a. A practical approach for recognizing eating moments with wrist-mounted inertial sensing. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 1029--1040.

Digital Library

[26]

Edison Thomaz, Cheng Zhang, Irfan Essa, and Gregory D Abowd. 2015b. Inferring Meal Eating Activities in Real World Settings from Ambient Sounds: A Feasibility Study. In Proceedings of the 20th International Conference on Intelligent User Interfaces. ACM, 427--431.

Digital Library

[27]

Yonatan Vaizman, Katherine Ellis, and Gert Lanckriet. 2017a. Recognizing Detailed Human Context In-the-Wild from Smartphones and Smartwatches. IEEE Pervasive Computing 16, 4 (2017).

[28]

Yonatan Vaizman, Nadir Weibel, and Gert Lanckriet. 2017b. Behavioral Context Recognition In-the-Wild: Unified Model for Multi-Modal Sensors and Multi-Label Classification. Under review for Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2017).

Digital Library

[29]

Evan Welbourne and Emmanuel Munguia Tapia. 2014. CrowdSignals: a call to crowdfund the community's largest mobile dataset. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. ACM, 873--877.

Digital Library

[30]

Hao Yan and Ted Selker. 2000. Context-aware office assistant. In Proceedings of the 5th international conference on Intelligent user interfaces. ACM, 276--279.

Digital Library

[31]

Yi-Hsuan Yang and Yuan-Ching Teng. 2015. Quantitative Study of Music Listening Behavior in a Smartphone Context. ACM Trans. Interact. Intell. Syst. 5, 3, Article 14 (Sept. 2015), 30 pages.

Digital Library

Cited By

Hoelzemann AVan Laerhoven K(2024)A matter of annotation: an empirical study on in situ and self-recall activity annotations from wearable sensorsFrontiers in Computer Science10.3389/fcomp.2024.13797886Online publication date: 18-Jul-2024
https://doi.org/10.3389/fcomp.2024.1379788
Chen HGouin-Vallerand CBouchard KGaboury SNgankam HLussier MCouture MBier NGiroux S(2024)Utilizing Self-Supervised Learning for Recognizing Human Activity in Older Adults through Labeling Applications in Real-World Smart HomesProceedings of the 2024 International Conference on Information Technology for Social Good10.1145/3677525.3678671(275-283)Online publication date: 4-Sep-2024
https://dl.acm.org/doi/10.1145/3677525.3678671
Wang YLi MKim YLee BDanilovich MLazar AConroy DKacorri HChoe E(2024)Redefining Activity Tracking Through Older Adults' Reflections on Meaningful ActivitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642170(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642170
Show More Cited By

Index Terms

ExtraSensory App: Data Collection In-the-Wild with Rich User Interface to Self-Report Behavior
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Redefining Activity Tracking Through Older Adults' Reflections on Meaningful Activities
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

Activity tracking has the potential to promote active lifestyles among older adults. However, current activity tracking technologies may inadvertently perpetuate ageism by focusing on age-related health risks. Advocating for a personalized approach in ...
Investigating self-reporting behavior in long-term studies
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Self-reporting techniques, such as data logging or a diary, are frequently used in long-term studies, but prone to subjects' forgetfulness and other sources of inaccuracy. We conducted a six-week self-reporting study on smartphone usage in order to ...
What's going on? Age, distraction, and multitasking during online survey taking

Participants in an online survey revealed what other activities they engaged in while taking it.Younger people were more likely than older people to multitask.The relationship between age, multitasking, and sense of distraction was curvilinear.Most ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

April 2018

8489 pages

ISBN:9781450356206

DOI:10.1145/3173574

General Chairs:
Regan Mandryk
University of Saskatchewan, Canada
,
Mark Hancock
University of Waterloo, Canada
,
Program Chairs:
Mark Perry
Brunel University London, UK
,
Anna Cox
University College London, UK

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article

Conference

CHI '18

Sponsor:

SIGCHI

CHI '18: CHI Conference on Human Factors in Computing Systems

April 21 - 26, 2018

Montreal QC, Canada

Acceptance Rates

CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

93
Total Citations
View Citations
1,577
Total Downloads

Downloads (Last 12 months)141
Downloads (Last 6 weeks)6

Reflects downloads up to 28 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hoelzemann AVan Laerhoven K(2024)A matter of annotation: an empirical study on in situ and self-recall activity annotations from wearable sensorsFrontiers in Computer Science10.3389/fcomp.2024.13797886Online publication date: 18-Jul-2024
https://doi.org/10.3389/fcomp.2024.1379788
Chen HGouin-Vallerand CBouchard KGaboury SNgankam HLussier MCouture MBier NGiroux S(2024)Utilizing Self-Supervised Learning for Recognizing Human Activity in Older Adults through Labeling Applications in Real-World Smart HomesProceedings of the 2024 International Conference on Information Technology for Social Good10.1145/3677525.3678671(275-283)Online publication date: 4-Sep-2024
https://dl.acm.org/doi/10.1145/3677525.3678671
Wang YLi MKim YLee BDanilovich MLazar AConroy DKacorri HChoe E(2024)Redefining Activity Tracking Through Older Adults' Reflections on Meaningful ActivitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642170(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642170
Arrotta LBettini CCivitarese GFiori M(2024)ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition Models2024 IEEE International Conference on Smart Computing (SMARTCOMP)10.1109/SMARTCOMP61445.2024.00029(55-62)Online publication date: 29-Jun-2024
https://doi.org/10.1109/SMARTCOMP61445.2024.00029
Ji GWoo JLee GMsigwa CBernard DYun J(2024)AIoT-Based Smart Healthcare in Everyday Lives: Data Collection and Standardization From Smartphones and SmartwatchesIEEE Internet of Things Journal10.1109/JIOT.2024.340050911:16(27597-27619)Online publication date: 15-Aug-2024
https://doi.org/10.1109/JIOT.2024.3400509
Rahin SHui BLi W(2024)Location-Aware Context Detection Based-On Behavior Sensors2024 6th International Conference on Computer Communication and the Internet (ICCCI)10.1109/ICCCI62159.2024.10674531(83-93)Online publication date: 14-Jun-2024
https://doi.org/10.1109/ICCCI62159.2024.10674531
Feng TNarayanan S(2024)Understanding Stress, Burnout, and Behavioral Patterns in Medical Residents Using Large-scale Longitudinal Wearable Recordings2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)10.1109/EMBC53108.2024.10782414(1-7)Online publication date: 15-Jul-2024
https://doi.org/10.1109/EMBC53108.2024.10782414
D’Aniello GFortino GGaeta MRehman Z(2024)A data-driven approach for context definition in situation-aware wearable computing systems2024 IEEE 20th International Conference on Automation Science and Engineering (CASE)10.1109/CASE59546.2024.10711334(1525-1531)Online publication date: 28-Aug-2024
https://doi.org/10.1109/CASE59546.2024.10711334
Khan DAlshahrani AAlmjally AAl Mudawi NAlgarni AAlnowaiser KJalal A(2024)Advanced IoT-Based Human Activity Recognition and Localization Using Deep Polynomial Neural NetworkIEEE Access10.1109/ACCESS.2024.342075212(94337-94353)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3420752
Alam UAhmad Farhan AKanwal SAllheeib N(2024)Entropy and Memory Aware Active Transfer Learning in Smart Sensing SystemsIEEE Access10.1109/ACCESS.2024.341265312(88841-88861)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3412653
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents