research-article

Activity Recognition: Translation across Sensor Modalities Using Deep Learning

Authors:

Tsuyoshi Okita,

Sozo InoueAuthors Info & Claims

UbiComp '18: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers

Pages 1462 - 1471

https://doi.org/10.1145/3267305.3267512

Published: 08 October 2018 Publication History

Abstract

We propose a method to translate between multi-modalities using an RNN encoder-decoder model. Based on such a model allowing to translate between modalities, we built an activity recognition system. The idea of equivalence of modality was investigated by Banos et al. This paper replaces this with deep learning. We compare the performance of translation with/without clustering and sliding window. We show the preliminary performance of activity recognition attained the F1 score of 0.78.

References

[1]

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, Neural Machine Translation by Jointly Learning to Align and Translate, arXiv, 2014.

[2]

Banos, Oresti and Calatroni, Alberto and Damas, Miguel and Pomares, Hector and Rojas, Ignacio and Sagha, Hesam and del R. Millán, Jose and Troster, Gerhard and Chavarriaga, Ricardo and Roggen, Daniel, "Kinect=IMU? Learning MIMO Signal Mappings to Automatically Translate Activity Recognition Systems Across Sensor Modalities", ISWC12, pp. 92--99, 2012.

Digital Library

[3]

Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", CVPR, 2017.

[4]

Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, Yoshua Bengio, Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, EMNLP, 2014.

[5]

Brendan J. Frey, Delbert Dueck, Clustering by Passing Messages Between Data Points, Science, 315(5814), pp. 972--976, 2007.

[6]

Zelun Luo, Boya Peng, De-An Huang, Alexandre Alahi, Li Fei-Fei, "Unsupervised Learning of Long-Term Motion Dynamics for Videos", CVPR, 2017

[7]

Minh-Thang Luong, Hieu Pham, Christopher D. Manning, Effective Approaches to Attention-based Neural Machine Translation, EMNLP, 2015.

[8]

Tittaya Mairittha, Tsuyoshi Okita and Sozo Inoue, Pre-Consulting Dialogue Systems for Telemedicine: Yes/No Intent Classification, WellComp workshop (collocated at UbiComp18), 2018.

Digital Library

[9]

Laurens van der Maaten and Geoffrey E. Hinton. Visualizing Non-Metric Similarities in Multiple Maps. Machine Learning 87(1):33--55, 2012.

Digital Library

[10]

Natalia Neverova, Christian Wolf, Graham W.Taylor, and Florian Nebout, "Multi-scale deep learning for gesture detection and localization", Workshop on Looking at People (ECCV), 2014.

[11]

Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, Rene Vidal and Ruzena Bajcsy. Berkeley MHAD: A Comprehensive Multimodal Human Action Database. In Proceedings of the IEEE Workshop on Applications on Computer Vision (WACV), 2013.

Digital Library

[12]

Tsuyoshi Okita, Sozo Inoue, "Recognition of Multiple Overlapping Activities Using Compositional CNN-LSTM Model", Ubicomp Adjunct, Sep, 2017.

Digital Library

[13]

Francisco Ordonez, Daniel Roggen. "Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors 16:115, 2016.

[14]

Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, Douglas Eck, Online and Linear-Time Attention by Enforcing Monotonic Alignments, arXiv, 2017.

[15]

Daniel Roggen, Gerhard Troster, Paul Lukowicz, Alois Ferscha, Jose del R. Millan, Ricardo Chavarriaga, Opportunistic human activity and context recognition, Computer, 46(2), pp. 36--45, 2013.

Digital Library

[16]

Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, Wang-chun Woo, "Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting", arXiv, 2015.

[17]

Ilya Sutskever, Oriol Vinyals, and Quoc Le, Sequence to Sequence Learning with Neural Networks, NIPS, 2014.

Digital Library

[18]

Shih-En Wei, Varun Ramakrishna, Takeo Kanade, Yaser Sheikh, "Convolutional pose machines, CVPR, 2016.

[19]

Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, Wenyu Liu, Revisiting Multiple Instance Neural Networks, arXiv, 2016.

[20]

Jiajun Wu, Yinan Yu, Chang Huang, Kai Yu, Deep Multiple Instance Learning for Image Classification and Auto-Annotation, arXiv, 2015.

[21]

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio, "Show, Attend and Tell: Neural Image Caption Generation with Visual Attention" arXiv, 2015.

[22]

Xiaowei Zhou, Menglong Zhu, Georgios Pavlakos, Spyridon Leonardos, Konstantinos G.Derpanis, and Kostas Daniilidis, "MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior", PAMI, 2018.

Digital Library

Cited By

Sah RGhasemzadeh H(2024)Adversarial Transferability in Embedded Sensor Systems: An Activity Recognition PerspectiveACM Transactions on Embedded Computing Systems10.1145/364186123:2(1-31)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1145/3641861
Okita T(2021)Activity Simulation from SignalsAdjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers10.1145/3460418.3479275(59-60)Online publication date: 21-Sep-2021
https://dl.acm.org/doi/10.1145/3460418.3479275
Mairittha NMairittha TInoue STentori MWeibel NVan Laerhoven KAbowd GSalim F(2020)Improving activity data collection with on-device personalization using fine-tuningAdjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers10.1145/3410530.3414370(255-260)Online publication date: 10-Sep-2020
https://dl.acm.org/doi/10.1145/3410530.3414370
Show More Cited By

Index Terms

Activity Recognition: Translation across Sensor Modalities Using Deep Learning
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

A CNN Based Transfer Learning Model for Automatic Activity Recognition from Accelerometer Sensors
Machine Learning and Data Mining in Pattern Recognition
Abstract
Accelerometers are become ubiquitous and available in several devices such as smartphones, smartwaches, fitness trackers, and wearable devices. Accelerometers are increasingly used to monitor human activities of daily living in different contexts ...
Deep appearance and motion learning for egocentric activity recognition

Egocentric activity recognition has recently generated great popularity in computer vision due to its widespread applications in egocentric video analysis. However, it poses new challenges comparing to the conventional third-person activity recognition ...
Deep learning for sensor-based activity recognition: A survey
Highlights
- We survey deep learning based HAR in sensor modality, deep model, and application.
Abstract
Sensor-based activity recognition seeks the profound high-level knowledge about human activities from multitudes of low-level sensor readings. Conventional pattern recognition approaches have made tremendous progress in the past years. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp '18: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers

October 2018

1881 pages

ISBN:9781450359665

DOI:10.1145/3267305

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing
University of Florida: University of Florida
SIGCHI: ACM Special Interest Group on Computer-Human Interaction

In-Cooperation

SIGSPATIAL: ACM Special Interest Group on Spatial Information

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

UbiComp '18

Sponsor:

SIGMOBILE
University of Florida
SIGCHI

UbiComp '18: The 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing

October 8 - 12, 2018

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
179
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sah RGhasemzadeh H(2024)Adversarial Transferability in Embedded Sensor Systems: An Activity Recognition PerspectiveACM Transactions on Embedded Computing Systems10.1145/364186123:2(1-31)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1145/3641861
Okita T(2021)Activity Simulation from SignalsAdjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers10.1145/3460418.3479275(59-60)Online publication date: 21-Sep-2021
https://dl.acm.org/doi/10.1145/3460418.3479275
Mairittha NMairittha TInoue STentori MWeibel NVan Laerhoven KAbowd GSalim F(2020)Improving activity data collection with on-device personalization using fine-tuningAdjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers10.1145/3410530.3414370(255-260)Online publication date: 10-Sep-2020
https://dl.acm.org/doi/10.1145/3410530.3414370
Adachi KLago POkita TInoue S(2020)Improvement of Human Action Recognition Using 3D Pose EstimationActivity and Behavior Computing10.1007/978-981-15-8944-7_2(21-37)Online publication date: 24-Dec-2020
https://doi.org/10.1007/978-981-15-8944-7_2
Mairittha NMairittha TInoue S(2019)On-Device Deep Learning Inference for Efficient Activity Data CollectionSensors10.3390/s1915343419:15(3434)Online publication date: 5-Aug-2019
https://doi.org/10.3390/s19153434

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents