research-article

Designing Videogames to Crowdsource Accelerometer Data Annotation for Activity Recognition Research

Authors:

Aditya Ponnada,

Binod Thapa-Chhetry,

Josh Aaron Miller,

Stephen IntilleAuthors Info & Claims

CHI PLAY '19: Proceedings of the Annual Symposium on Computer-Human Interaction in Play

Pages 135 - 147

https://doi.org/10.1145/3311350.3347153

Published: 17 October 2019 Publication History

Abstract

Human activity recognition using wearable accelerometers can enable in-situ detection of physical activities to support novel human-computer interfaces and interventions. However, developing valid algorithms that use accelerometer data to detect everyday activities often requires large amounts of training datasets, precisely labeled with the start and end times of the activities of interest. Acquiring annotated data is challenging and time-consuming. Applied games, such as human computation games (HCGs) have been used to annotate images, sounds, and videos to support advances in machine learning using the collective effort of "non-expert game players." However, their potential to annotate accelerometer data has not been formally explored. In this paper, we present two proof-of-concept, web-based HCGs aimed at enabling game players to annotate accelerometer data. Using results from pilot studies with Amazon Mechanical Turk players, we discuss key challenges, opportunities, and, more generally, the potential of using applied videogames for annotating raw accelerometer data to support activity recognition research.

Supplementary Material

MP4 File (fp2112.mp4)

Supplemental video

Download
18.91 MB

MP4 File (p135-ponnada.mp4)

Download
93.71 MB

References

[1]

Accelerometer Technologies, Specifications, and Limitations: A presentation by ActiGraph at ICAMPAM 2013 ActiGraph.

[2]

Zooniverse. 2013. The Bat Detective. (Accessed on 2018 12/05); Available from: https://www.batdetective.org/#!/home.

[3]

RedOctane and Harmonix. 2005. Guitar Hero. (Accessed on 2018 12/27); Available from: https://www.guitarhero.com/game.

[4]

Zooniverse. 2018. Planet Hunters. (Accessed on 2018 12/10); Available from: https://www.zooniverse.org/projects/nora-doteisner/planet-hunters-tess.

[5]

Zooniverse. 2011. Whale FM. (Accessed on 2018 12/19); Available from: https://whale.fm/.

[6]

Luis von Ahn and Laura Dabbish. 2008. Designing games with a purpose. Commun. ACM, 51(8): p. 58--67.

Digital Library

[7]

B. E. Ainsworth, W. L. Haskell, S. D. Herrmann, N. Meckes, D. R. Bassett, Jr., C. Tudor-Locke, J. L. Greer, J. Vezina, M. C. Whitt-Glover, and A. S. Leon. 2011. 2011 Compendium of physical activities: A second update of codes and MET values. Med Sci Sports Exerc, 43(8): p. 1575--81.

[8]

Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz. 2013. A public domain dataset for human activity recognition using smartphones. in ESANN.

[9]

L. Bao and S.S. Intille.2004. Activity recognition from user-annotated acceleration data, in Pervasive Computing, A. Ferscha and F. Mattern, Editors., Springer-Verlag: Berlin. p. 1--17.

[10]

Michael Barz, Mohammad Mehdi Moniri, Markus Weber, and Daniel Sonntag, Multimodal multisensor activity annotation tool, in Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. 2016, ACM: Heidelberg, Germany. p. 17--20.

Digital Library

[11]

E. J. Bassey, H. M. Dallosso, P. H. Fentem, J. M. Irving, and J. M. Patrick. 1987. Validation of a simple mechanical accelerometer (pedometer) for the estimation of walking activity. Eur J Appl Physiol Occup Physiol, 56(3): p. 323--30.

[12]

C. V. Bouten, K. T. Koekkoek, M. Verduin, R. Kodde, and J. D. Janssen. 1997. A triaxial accelerometer and portable data processing unit for the assessment of daily physical activity. IEEE Trans. on Bio-Medical Engineering, 44(3): p. 13647.

[13]

C. V. Bouten, K. R. Westerterp, M. Verduin, and J. D. Janssen. 1994. Assessment of energy expenditure for physical activity using a triaxial accelerometer. Med. Sci. Sports Exerc., 26(12): p. 1516--1523.

[14]

William Callaghan, Joslin Goh, Michael Mohareb, Andrew Lim, and Edith Law. 2018. MechanicalHeart: A Human-Machine Framework for the Classification of Phonocardiograms. Proc. ACM Hum.-Comput. Interact., 2(CSCW): p. 1--17.

Digital Library

[15]

U.S. Department of Health and Human Services. 2018. NHANES - National Health and Nutrition Examination Survey Homepage. [Website] (Accessed on 2018 Nov 24); Available from: http://www.cdc.gov/nchs/nhanes.htm.

[16]

T. Choudhury, S. Consolvo, B. Harrison, J. Hightower, A. LaMarca, L. LeGrand, A. Rahimi, and A. Rea. 2008. The mobile sensing platform: An embedded activity recognition system. Pervasive Comp., 7(2): p. 32--41.

Digital Library

[17]

S. Cooper, F. Khatib, A. Treuille, J. Barbero, J. Lee, M. Beenen, A. Leaver-Fay, D. Baker, Z. Popovic, and Foldit players. 2010. Predicting protein structures with a multiplayer online game. Nature, 466(7307): p. 756--60.

[18]

Seth Cooper, A Framework for Scientific Discovery through Video Games. 2014: Association for Computing Machinery and Morgan &; Claypool. 133.

[19]

Seth Cooper, Amy L. R. Sterling, Robert Kleffner, William M. Silversmith, and Justin B. Siegel, Repurposing citizen science games as software tools for professional scientists, in Proceedings of the 13th International Conference on the Foundations of Digital Games. 2018, ACM: Malmö, Sweden. p. 1--6.

Digital Library

[20]

Seth Cooper, Adrien Treuille, Janos Barbero, Andrew Leaver-Fay, Kathleen Tuite, Firas Khatib, Alex Cho Snyder, Michael Beenen, David Salesin, David Baker, and Zoran Popovic, The challenge of designing scientific discovery games, in Proceedings of the Fifth International Conference on the Foundations of Digital Games. 2010, ACM: Monterey, California. p. 40--47.

Digital Library

[21]

J. Costa, C. Silva, M. Antunes, and B. Ribeiro. 2011. On using crowdsourcing and active learning to improve classification performance. in 2011 11th International Conference on Intelligent Systems Design and Applications.

[22]

Mihaly Csikszentmihalyi, Finding flow: The psychology of engagement with everyday life. 1997: Basic Books.

[23]

E. De la Hoz, P. Ariza, J. Medina, and M. Espinilla, Sensor-based datasets for human activity recognition -- A systematic review of literature. Vol. PP. 2018. 1--1.

[24]

R.W. DeVaul and S. Dunn, Real-Time Motion Classification for Wearable Computing Applications. 2001, MIT Media Laboratory.

[25]

A. Diete, T. Sztyler, and H. Stuckenschmidt. 2017. A smart data annotation tool for multisensor activity recognition. in 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[26]

D. W. Esliger, A. Probert, S. C. Gorber, S. Bryan, M. Laviolette, and M. S. Tremblay. 2007. Validity of the Actical accelerometer step-count function. Med. Sci. Sports Exerc., 39(7): p. 1200--4.

[27]

P.S. Freedson, E. Melanson, and J. Sirard. 1998. Calibration of the Computer Science and Applications (CSA), Inc. accelerometer. Med. Sci. Sports Exerc., 30(5): p. 777--781.

[28]

Kelsie M. Full, Jacqueline Kerr, Michael A. Grandner, Atul Malhotra, Kevin Moran, Suneeta Godoble, Loki Natarajan, and Xavier Soler. 2018. Validation of a physical activity accelerometer device worn on the hip and wrist against polysomnography. Sleep Health: Journal Nat. Sleep Found., 4(2): p. 209--216.

[29]

Chien-Ju Ho, Tao-Hsuan Chang, Jong-Chuan Lee, Jane Yung-jen Hsu, and Kuan-Ta Chen. 2010. KissKissBan: a competitive human computation game for image annotation. SIGKDD Explor. Newsl., 12(1): p. 21--24.

Digital Library

[30]

Ioanna Iacovides, Charlene Jennett, Cassandra Cornish-Trestrail, and Anna L Cox. 2013. Do games attract or sustain engagement in citizen science?: a study of volunteer motivations. in CHI'13 Extended Abstracts on Human Factors in Computing Systems. ACM.

Digital Library

[31]

Charlene Jennett, Laure Kloetzer, Daniel Schneider, Ioanna Iacovides, Anna Cox, Margaret Gold, Brian Fuchs, Alexandra Eveleigh, Kathleen Methieu, and Zoya Ajani. 2016. Motivations, learning and creativity in online citizen science. Journal of Sci. Comm., 15(3).

[32]

A. Kawrykow, G. Roumanis, A. Kam, D. Kwak, C. Leung, C. Wu, E. Zarour, L. Sarmenta, M. Blanchette, and J. Waldispuhl. 2012. Phylo: a citizen science approach for improving multiple sequence alignment. PLoS One, 7(3): p. e31362.

[33]

F. Khatib, F. DiMaio, S. Cooper, M. Kazmierczyk, M. Gilski, S. Krzywda, H. Zabranska, I. Pichova, J. Thompson, Z. Popovic, M. Jaskolski, and D. Baker. 2011. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nat Struct Mol Biol, 18(10): p. 1175--7.

[34]

J. S. Kim, M. J. Greene, A. Zlateski, K. Lee, M. Richardson, S. C. Turaga, M. Purcaro, M. Balkam, A. Robinson, B. F. Behabadi, M. Campos, W. Denk, and H. S. Seung. 2014. Space-time wiring specificity supports direction selectivity in the retina. Nature, 509(7500): p. 331--6.

[35]

A. Kosmadopoulos, D. Darwent, and G. D. Roach. 2016. Is it on? An algorithm for discerning wristaccelerometer non-wear times from sleep/wake activity. Chronobiology International: p. 1--5.

[36]

F. Krüger, C. Heine, S. Bader, A. Hein, S. Teipel, and T. Kirste. 2017. On the applicability of clinical observation tools for human activity annotation. in 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[37]

Oscar D. Lara and Miguel A. Labrador. 2013. A survey on human activity recognition using wearable sensors. IEEE Communications Surveys and Tutorials, 15(3): p. 1192--1209.

[38]

Edith LM Law, Luis Von Ahn, Roger B Dannenberg, and Mike Crawford. 2007. TagATune: A Game for Music and Sound Annotation. in ISMIR.

[39]

J. Lee, W. Kladwang, M. Lee, D. Cantu, M. Azizyan, H. Kim, A. Limpaecher, S. Yoon, A. Treuille, R. Das, and R. N. A. Participants Ete. 2014. RNA design rules from a massive open laboratory. Proc Natl Acad Sci U S A, 111(6): p. 2122--7.

[40]

L. Litman, J. Robinson, and T. Abberbock. 2017. TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behav Res Methods, 49(2): p. 433--442.

[41]

S. Loguercio, B. M. Good, and A. I. Su. 2013. Dizeez: an online game for human gene-disease annotation. PLoS One, 8(8): p. e71171.

[42]

A. Mannini, M. Rosenberger, W. L. Haskell, A. M. Sabatini, and S. S. Intille. 2017. Activity recognition in youth using single accelerometer placed at wrist or ankle. Med. Sci. Sports Exerc., 49(4): p. 801--812.

[43]

G. A. Meijer, K. R. Westerterp, F. M. Verhoeven, H. B. Koper, and F. ten Hoor. 1991. Methods to assess physical activity with special reference to motion sensors and accelerometers. IEEE Trans Biomed Eng, 38(3): p. 221--9.

[44]

Ana Elisa Méndez Méndez, Mark Cartwright, and Juan Pablo Bello, Machine-crowd-expert model for increasing user engagement and annotation quality, in CHI'19 Extended Abstracts. 2019, ACM Press.

Digital Library

[45]

Jeff Miller, Accelerometer technologies, specfications, and limitations. Actigraph, LLC.

[46]

Nathan Prestopnik and Dania Souid, Forgotten island: a story-driven citizen science adventure, in CHI '13 Extended Abstracts on Human Factors in Computing Systems. 2013, ACM: Paris, France. p. 2643--2646.

Digital Library

[47]

Alexander J Quinn and Benjamin B Bederson. 2011. Human computation: a survey and taxonomy of a growing field. in Proceedings of the SIGCHI conference on human factors in computing systems. ACM.

Digital Library

[48]

G. Rallapalli, Players Fraxinus, D. G. Saunders, K. Yoshida, A. Edwards, C. A. Lugo, S. Collin, B. Clavijo, M. Corpas, D. Swarbreck, M. Clark, J. A. Downie, S. Kamoun, Cooper Team, and D. MacLean. 2015. Lessons from Fraxinus, a crowdsourced citizen science game in genomics. Elife, 4: p. e07460.

[49]

Nishkam Ravi, Nikhil Dandekar, Preetham Mysore, and Michael Littman.2005. Activity recognition from accelerometer data, in Proceedings of Innovative Applications of Artificial Intelligence, N. Jacobstein and B. Porter, Editors., AAAI Press: Menlo Park, CA. p. 15411546.

[50]

M. E. Rosenberger, M. P. Buman, W. L. Haskell, M. V. McConnell, and L. L. Carstensen. 2016. 24 hours of sleep, sedentary behavior, and physical activity with nine wearable devices. Med Sci Sports Exerc, 48(3): p. 457--65.

[51]

Thomas A. Rowles. 2013. Power to the people: Does Eterna signal the arrival of a new wave of crowd-sourced projects? BMC Biochem., 14(1): p. 26.

[52]

Anurag Sarkar and Seth Cooper, Comparing paid and volunteer recruitment in human computation games, in Proceedings of the 13th International Conference on the Foundations of Digital Games. 2018, ACM: Malmö, Sweden. p. 1--9.

Digital Library

[53]

Anna Shcherbina, C. Mikael Mattsson, Daryl Waggott, Heidi Salisbury, Jeffrey W. Christle, Trevor Hastie, Matthew T. Wheeler, and Euan A. Ashley. 2017. Accuracy in Wrist-Worn, SensorBased Measurements of Heart Rate and Energy Expenditure in a Diverse Cohort. Journal of Pers. Med., 7(2): p. 3.

[54]

Team Psycho Sisters, Exoplanets: The next phase of project discovery, in Dev-Blogs. 2017: Eve Online.

[55]

Jinhua Song, Hao Wang, Yang Gao, and Bo An. 2018. Active learning with confidence-based answers for crowdsourcing labeling tasks. Knowledge-Based Systems, 159: p. 244--258.

[56]

Maja Stikic and Bernt Schiele, Activity Recognition from Sparsely Labeled Data Using Multi-Instance Learning, in Location and Context Awareness. 2009, Springer, Berlin, Heidelberg. p. 156--173.

Digital Library

[57]

Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, Jane Green, Martin Landray, Bette Liu, Paul Matthews, Giok Ong, Jill Pell, Alan Silman, Alan Young, Tim Sprosen, Tim Peakman, and Rory Collins. 2015. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med., 12(3): p. e1001779.

[58]

Chuxuan Tong, Jinglan Zhang, Alok Chowdhury, and Stewart G. Trost, An Interactive Visualization Tool for Sensor-based Physical Activity Data Analysis, in Proceedings of the Australasian Computer Science Week Multiconference. 2019, ACM: Sydney, NSW, Australia. p. 1--4.

Digital Library

[59]

L. von Ahn. 2006. Games with a purpose. Computer, 39(6): p. 92--94.

Digital Library

[60]

L. von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum. 2008. reCAPTCHA: human-based character recognition via Web security measures. Science, 321(5895): p. 1465--8.

[61]

Luis von Ahn. 2008. Human computation. in Proceedings of the 2008 IEEE 24th International Conference on Data Engineering. IEEE Computer Society.

Digital Library

[62]

Luis von Ahn and Laura Dabbish, Labeling images with a computer game, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2004, ACM: Vienna, Austria. p. 319--326.

Digital Library

[63]

Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. 2019. Deep learning for sensor-based activity recognition: A survey. Pattern Recognition Letters, 119: p. 3--11.

Digital Library

[64]

Kristina Yordanova, Adeline Paiement, Max Schröder, Emma Tonkin, Przemyslaw Woznowski, Carl Magnus Olsson, Joseph Rafferty, and Timo Sztyler. 2018. Challenges in annotation of user data for ubiquitous systems: Results from the 1st ARDUOUS workshop. arXiv preprint arXiv:1803.05843.

Cited By

Pearce EMajid ABrown TShepherd RRising CWilsnack CThompson AGilkey MRibisl KLazard AHan PWerner-Lin AHutson SSavage S(2024)“Crying in the Wilderness”—The Use of Web-Based Support in Telomere Biology Disorders: Thematic AnalysisJMIR Formative Research10.2196/643438(e64343)Online publication date: 16-Dec-2024
https://doi.org/10.2196/64343
Hoelzemann ARomero JBock MLaerhoven KLv Q(2023)Hang-Time HAR: A Benchmark Dataset for Basketball Activity Recognition Using Wrist-Worn Inertial SensorsSensors10.3390/s2313587923:13(5879)Online publication date: 25-Jun-2023
https://doi.org/10.3390/s23135879
Kougioumtzian LEl Raheb KKatifori ARoussou M(2022)Blazing fire or breezy wind? A story-driven playful experience for annotating dance movementFrontiers in Computer Science10.3389/fcomp.2022.9572744Online publication date: 23-Sep-2022
https://doi.org/10.3389/fcomp.2022.957274
Show More Cited By

Index Terms

Designing Videogames to Crowdsource Accelerometer Data Annotation for Activity Recognition Research
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Facilitating Human Activity Data Annotation via Context-Aware Change Detection on Smartwatches

Annotating activities of daily living (ADL) is vital for developing machine learning models for activity recognition. In addition, it is critical for self-reporting purposes such as in assisted living where the users are asked to log their ADLs. However,...
Annotating smart environment sensor data for activity learning
Smart Environments: Technology to Support Healthcare

The pervasive sensing technologies found in smart homes offer unprecedented opportunities for providing health monitoring and assistance to individuals experiencing difficulties living independently at home. In order to monitor the functional health of ...
Accelerometer Placement for Posture Recognition and Fall Detection
IE '11: Proceedings of the 2011 Seventh International Conference on Intelligent Environments

This paper presents an approach to fall detection with accelerometers that exploits posture recognition to identify postures that may be the result of a fall. Posture recognition as a standalone task was also studied. Nine placements of up to four ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI PLAY '19: Proceedings of the Annual Symposium on Computer-Human Interaction in Play

October 2019

680 pages

ISBN:9781450366885

DOI:10.1145/3311350

General Chairs:
Joan Arnedo
Universitat Oberta de Catalunya, Barcelona, Spain
,
Lennart E. Nacke
University of Waterloo, Waterloo, Canada
,
Program Chairs:
Vero Vanden Abeele
KU Leuven, Leuven, Belgium
,
Phoebe O. Toups Dugas
New Mexico State University, New Mexico, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health (NIH)

Conference

CHI PLAY '19

Sponsor:

SIGCHI

CHI PLAY '19: The Annual Symposium on Computer-Human Interaction in Play

October 22 - 25, 2019

Barcelona, Spain

Acceptance Rates

CHI PLAY '19 Paper Acceptance Rate 51 of 181 submissions, 28%;

Overall Acceptance Rate 421 of 1,386 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
292
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pearce EMajid ABrown TShepherd RRising CWilsnack CThompson AGilkey MRibisl KLazard AHan PWerner-Lin AHutson SSavage S(2024)“Crying in the Wilderness”—The Use of Web-Based Support in Telomere Biology Disorders: Thematic AnalysisJMIR Formative Research10.2196/643438(e64343)Online publication date: 16-Dec-2024
https://doi.org/10.2196/64343
Hoelzemann ARomero JBock MLaerhoven KLv Q(2023)Hang-Time HAR: A Benchmark Dataset for Basketball Activity Recognition Using Wrist-Worn Inertial SensorsSensors10.3390/s2313587923:13(5879)Online publication date: 25-Jun-2023
https://doi.org/10.3390/s23135879
Kougioumtzian LEl Raheb KKatifori ARoussou M(2022)Blazing fire or breezy wind? A story-driven playful experience for annotating dance movementFrontiers in Computer Science10.3389/fcomp.2022.9572744Online publication date: 23-Sep-2022
https://doi.org/10.3389/fcomp.2022.957274
Ponnada ACooper STang QThapa-Chhetry BMiller JJohn DIntille S(2021)Signaligner Pro: A Tool to Explore and Annotate Multi-day Raw Accelerometer Data2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)10.1109/PerComWorkshops51409.2021.9431110(475-480)Online publication date: 22-Mar-2021
https://doi.org/10.1109/PerComWorkshops51409.2021.9431110

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten