research-article

MCoMat: a new performance metric for imbalanced multi-layer activity recognition dataset

Authors:

Sayeda Shamma Alia,

Sozo InoueAuthors Info & Claims

UbiComp/ISWC '20 Adjunct: Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers

Pages 232 - 237

https://doi.org/10.1145/3410530.3414364

Published: 12 September 2020 Publication History

Abstract

Existing performance metrics assess classifiers on single granularity layer. Having multi-layer labels is also possible such as activity recognition datasets. Semantic annotations could be given with multiple granularity layers in these datasets e.g., activity and the current step within that activity like: cooking and taking ingredients from fridge. Recognizing both layers is important i.e., remote monitoring of patients with dementia. To evaluate a classifier for both layers concurrently, a new performance metric is required. However, it is not easy to design as there are many underlying issues: the relation between the layers and the impact of class imbalance. This work proposes a new metric for evaluating multi-layer labeled dataset considering the mentioned factors and is applied on two datasets. It is found that it can assess the performance of a model classifying activities at two different granularity layers and give more insightful results i.e. reflecting performance for each layer.

References

[1]

Andreas Bulling, U Blanke, and Bernt Schiele. 2014. A tutorial on human activity recognition using body-worn inertial sensors. ACM Computing Surveys (CSUR) 1, June (2014), 1--33.

Digital Library

[2]

Stefan Dernbach, Barnan Das, Narayanan C. Krishnan, Brian L. Thomas, and Diane J. Cook. 2012. Simple and Complex Activity Recognition Through Smart Phones. In Proceedings of the International Conference on Intelligent Environments. IEEE, Guanajuato, Mexico.

[3]

Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. 2018. AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions. In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 6047--6056.

[4]

Robert M. Haralick, K. Shanmugam, and Its'Hak Dinstein. 1973. Textural Features for Image Classification. IEEE Transactions on Systems, Man, and Cybernetics SMC-3, November (1973), 610--621.

[5]

Rim Helaoui, Daniele Riboni, Mathias Niepert, Claudio Bettini, and Heiner Stuckenschmidt. 2012. Towards activity recognition using probabilistic description logics. Technical Report.

[6]

Rim Helaoui, Daniele Riboni, and Heiner Stuckenschmidt. 2013. A Probabilistic Ontological Framework for the Recognition of Multilevel Human Activities. In Proceedings of the ACM international joint conference on Pervasive and ubiquitous computings. ACM, Switzerland, 345--354.

Digital Library

[7]

Sang-Hack Jung, Yanlin Guo, Harpreet S Sawhney, and Rankesh Kumar. 2008. Action Video Retrieval based on Atomic Action Vocabulary. In Proceedings of the ACM international conference on Multimedia information retrieval (MIR). ACM, 245--252.

Digital Library

[8]

Md Eusha Kadir, Pritom Saha Akash, Sadia Sharmin, Amin Ahsan Ali, and Mohammad Shoyaib. 2019. Can a simple approach identify complex nurse care activity?. In Adjunct Proceedings of UbiComp/ISWC '19. 736--740.

Digital Library

[9]

Paula Lago, Shingo Takeda, Kohei Adachi, Sayeda Shamma Alia, Moe Matsuki, Brahim Benaissa, Sozo Inoue, and Francois Charpillet. 2020. Cooking Activity Dataset with Macro and Micro Activities.

[10]

Young-Seol Lee and Sung-Bae Cho. 2011. Activity Recognition Using Hierarchical Hidden Markov Models on a Smartphone with 3D Accelerometer. In Proceedings of the TInternational Conference on Hybrid Artificial Intelligence Systems. Springer, 460--467.

[11]

Li Liu, Li Cheng, Ye Liu, Yongpo Jia, and David S. Rosenblum. 2016. Recognizing Complex Activities by a Probabilistic Interval-Based Model. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI, USA, 1266--1272.

[12]

Alexandrin Popescul, Lyle H. Ungar, David M Pennock, and Steve Lawrence. 2013. Probabilistic Models for Unified Collaborative and Content-Based Recommendation in Sparse-Data Environments. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (UAI2001). arXiv:1301.2303

[13]

Philipp Probst, Quay Au, Giuseppe Casalicchio, Clemens Stachl, and Bernd Bischl. 2017. Classification with R Package mlr. The R Journal 9/1 (2017), 352--369.

[14]

Nishkam Ravi, Nikhil Dandekar, Preetham Mysore, and Michael L. Littman. 2005. Activity Recognition from Accelerometer Data. In American Association for Artificial Intelligence. 6.

[15]

D. Roggen, A. Calatroni, M. Rossi, T. Holleczek, K. Förster, G. Tröster, P. Lukowicz, D. Bannach, G. Pirkl, A. Ferscha, J. Doppler, C. Holzmann, M. Kurz, G. Holl, R. Chavarriaga, H. Sagha, H. Bayati, M. Creatura, and J. d. R. Millàn. 2010. Collecting complex activity datasets in highly rich networked sensor environments. In 2010 Seventh International Conference on Networked Sensing Systems (INSS). 233--240.

[16]

Saguna Saguna, Arkady Zaslavsky, and Dipanjan Chakraborty. 2013. Complex activity recognition using context-driven activity theory and activity signatures. ACM Transactions on Computer-Human Interaction 20 (2013). Issue 6.

[17]

Sadia Sharmin, Mohammad Shoyaib, Amin Ahsan Ali, Muhammad Asif Hossain Khan, and Oksam Chae. 2019. Simultaneous feature selection and discretization based on mutual information. Pattern Recognition 91 (2019), 162--174.

Digital Library

[18]

Chen Sun and Ram Nevatia. 2013. ACTIVE: Activity Concept Transitions in Video Event Classification. In Proceedings of The IEEE International Conference on Computer Vision (ICCV). IEEE, 913--920.

Digital Library

Cited By

Xu SZhang LHuang WWu HSong A(2022)Deformable Convolutional Networks for Multimodal Human Activity Recognition Using Wearable SensorsIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2022.315842771(1-14)Online publication date: 2022
https://doi.org/10.1109/TIM.2022.3158427
Cheng XZhang LTang YLiu YWu HHe J(2022)Real-Time Human Activity Recognition Using Conditionally Parametrized Convolutions on Mobile and Wearable DevicesIEEE Sensors Journal10.1109/JSEN.2022.314933722:6(5889-5901)Online publication date: 15-Mar-2022
https://doi.org/10.1109/JSEN.2022.3149337
Alia SHossain TInoue S(2021)PerMML: A Performance Metric for Multi-layer DatasetAdjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers10.1145/3460418.3479352(285-290)Online publication date: 21-Sep-2021
https://dl.acm.org/doi/10.1145/3460418.3479352

Index Terms

MCoMat: a new performance metric for imbalanced multi-layer activity recognition dataset
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing design and evaluation methods

Recommendations

PerMML: A Performance Metric for Multi-layer Dataset
UbiComp/ISWC '21 Adjunct: Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers
Classifier Loss Under Metric Uncertainty
ECML '07: Proceedings of the 18th European conference on Machine Learning

Classifiers that are deployed in the field can be used and evaluated in ways that were not anticipated when the model was trained. The final evaluation metric may not have been known at training time, additional performance criteria may have been added, ...
Weighted kappa measures for ordinal multi-class classification performance
Abstract
Assessing the classification performance of ordinal classifiers is a challenging problem under imbalanced data compositions. Considering the critical impact of the metrics on the choice of classifiers, employing a metric with the highest ...
Highlights
- Evaluation metrics for ordinal classifiers are studied.
- Agreement metrics are considered to be used for classification performance.
- 37 metrics are assessed under different compositions of confusion matrix.
- Weighted Scott’s pi ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp/ISWC '20 Adjunct: Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers

September 2020

732 pages

ISBN:9781450380768

DOI:10.1145/3410530

General Chairs:
Monica Tentori
CICESE, Mexico
,
Nadir Weibel
UC San Diego
,
Kristof Van Laerhoven
University of Siegen, Germany
,
Program Chairs:
Gregory Abowd
Georgia Tech
,
Flora Salim
RMIT, Australia

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UbiComp/ISWC '20

Sponsor:

UbiComp/ISWC '20: 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and 2020 ACM International Symposium on Wearable Computers

September 12 - 17, 2020

Virtual Event, Mexico

Acceptance Rates

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
115
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)1

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xu SZhang LHuang WWu HSong A(2022)Deformable Convolutional Networks for Multimodal Human Activity Recognition Using Wearable SensorsIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2022.315842771(1-14)Online publication date: 2022
https://doi.org/10.1109/TIM.2022.3158427
Cheng XZhang LTang YLiu YWu HHe J(2022)Real-Time Human Activity Recognition Using Conditionally Parametrized Convolutions on Mobile and Wearable DevicesIEEE Sensors Journal10.1109/JSEN.2022.314933722:6(5889-5901)Online publication date: 15-Mar-2022
https://doi.org/10.1109/JSEN.2022.3149337
Alia SHossain TInoue S(2021)PerMML: A Performance Metric for Multi-layer DatasetAdjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers10.1145/3460418.3479352(285-290)Online publication date: 21-Sep-2021
https://dl.acm.org/doi/10.1145/3460418.3479352

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents