research-article

Cost-effective On-device Continual Learning over Memory Hierarchy with Miro

Authors:

Myeongjae JeonAuthors Info & Claims

ACM MobiCom '23: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking

Article No.: 83, Pages 1 - 15

https://doi.org/10.1145/3570361.3613297

Published: 02 October 2023 Publication History

Abstract

Continual learning (CL) trains NN models incrementally from a continuous stream of tasks. To remember previously learned knowledge, prior studies store old samples over a memory hierarchy and replay them when new tasks arrive. Edge devices that adopt CL to preserve data privacy are typically energy-sensitive and thus require high model accuracy while not compromising energy efficiency, i.e., cost-effectiveness. Our work is the first to explore the design space of hierarchical memory replay-based CL to gain insights into achieving cost-effectiveness on edge devices. We present Miro, a novel system runtime that carefully integrates our insights into the CL framework by enabling it to dynamically configure the CL system based on resource states for the best cost-effectiveness. To reach this goal, Miro also performs online profiling on parameters with clear accuracy-energy trade-offs and adapts to optimal values with low overhead. Extensive evaluations show that Miro significantly outperforms baseline systems we build for comparison, consistently achieving higher cost-effectiveness.

References

[1]

Carousel Memory (CarM) - Official Pytorch Implementation. https://github.com/omnia-unist/CarM.

[2]

Convenient Power Measurements on the Jetson TX2/Tegra X2 Board. https://embeddeddl.wordpress.com/2018/04/25/convenient-power-measurements-on-the-jetson-tx2-tegra-x2-board/.

[3]

Jetson TX2 Module. https://developer.nvidia.com/embedded/jetson-tx2.

[4]

Jetson Xavier NX Series. https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx.

[5]

OEM PRODUCT DESIGN GUIDE: NVIDIA Jetson TX2. https://usermanual.wiki/Pdf/jetsontx2oemproductdesignguide.2134990230.pdf.

[6]

Python Shared Memory. https://docs.python.org/3/library/multiprocessing.html.

[7]

Tiny ImageNet Visual Recognition Challenge. https://tiny-imagenet.herokuapp.com.

[8]

J. Bang, H. Kim, Y. Yoo, J.-W. Ha, and J. Choi. Rainbow Memory: Continual Learning with a Memory of Diverse Samples. In CVPR, 2021.

[9]

B. Barshan and K. Altun. Daily and Sports Activities. UCI Machine Learning Repository, 2013.

[10]

R. Bhardwaj, Z. Xia, G. Ananthanarayanan, J. Jiang, N. Karianakis, Y. Shu, K. Hsieh, V. Bahl, and I. Stoica. Ekya: Continuous Learning of Video Analytics Models on Edge Compute Servers. In NSDI, 2022.

[11]

Z. Borsos, M. Mutnỳ, and A. Krause. Coresets via Bilevel Optimization for Continual Learning and Streaming. In NeurIPS, 2020.

[12]

P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. CALDERARA. Dark Experience for General Continual Learning: a Strong, Simple Baseline. In NeurIPS, 2020.

[13]

F. M. Castro, M. J. Marin-Jimenez, N. Guil, C. Schmid, and K. Alahari. End-to-End Incremental Learning. In ECCV, 2018.

Digital Library

[14]

A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. S. Torr. Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence. In ECCV, 2018.

Digital Library

[15]

A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny. Efficient Lifelong Learning with A-GEM. In ICLR, 2019.

[16]

A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato. On Tiny Episodic Memories in Continual Learning. arXiv:1902.10486, 2019.

[17]

J. Chauhan, Y. D. Kwon, P. Hui, and C. Mascolo. ContAuth: Continual Learning Framework for Behavioral-based User Authentication. IMWUT, 4(4):1--23, 2020.

[18]

T. Chen, B. Xu, C. Zhang, and C. Guestrin. Training Deep Nets with Sublinear Memory Cost. arXiv preprint arXiv:1604.06174, 2016.

[19]

Y. Cong, M. Zhao, J. Li, S. Wang, and L. Carin. GAN Memory with No Forgetting. In NeurIPS, 2020.

[20]

A. Diwan, C.-F. Yeh, W.-N. Hsu, P. Tomasello, E. Choi, D. Harwath, and A. Mohamed. Continual Learning for On-Device Speech Recognition using Disentangled Conformers. arXiv preprint arXiv:2212.01393, 2022.

[21]

K. Fall and S. Floyd. Simulation-Based Comparisons of Tahoe, Reno and SACK TCP. SIGCOMM Comput. Commun. Rev., 26(3):5--21, jul 1996.

Digital Library

[22]

A. Gepperth and B. Hammer. Incremental Learning Algorithms and Applications. In ESANN, 2016.

[23]

I. Gim and J. Ko. Memory-Efficient DNN Training on Mobile Devices. In MobiSys, 2022.

Digital Library

[24]

A. Griewank and A. Walther. Algorithm 799: Revolve: An Implementation of Checkpointing for the Reverse or Adjoint Mode of Computational Differentiation. ACM Trans. Math. Softw., 26(1):19--45, mar 2000.

Digital Library

[25]

S. Ha, I. Rhee, and L. Xu. CUBIC: A New TCP-Friendly High-Speed TCP Variant. SIGOPS Oper. Syst. Rev., 42(5):64--74, jul 2008.

Digital Library

[26]

T. L. Hayes and C. Kanan. Online Continual Learning for Embedded Devices. In Conference on Lifelong Learning Agents, 2022.

[27]

K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770--778, 2016.

[28]

W. Hu, Z. Lin, B. Liu, C. Tao, Z. Tao, J. Ma, D. Zhao, and R. Yan. Overcoming Catastrophic Forgetting via Model Adaptation. In ICLR, 2019.

[29]

G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep Networks with Stochastic Depth. In European Conference on Computer Vision, 2016.

[30]

Y. Huang, Y. Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism. In NeurIPS, 2019.

[31]

M. Khani, G. Ananthanarayanan, K. Hsieh, J. Jiang, R. Netravali, Y. Shu, M. Alizadeh, and V. Bahl. RECL: Responsive Resource-Efficient Continuous Learning for Video Analytics. In NSDI, 2023.

[32]

A. Krizhevsky, G. Hinton, et al. Learning Multiple Layers of Features from Tiny Images. 2009.

[33]

Y. D. Kwon, J. Chauhan, A. Kumar, P. H. HKUST, and C. Mascolo. Exploring System Performance of Continual Learning for Mobile and Embedded Sensing Applications. In IEEE/ACM SEC, 2021.

[34]

S. Lee, M. Weerakoon, J. Choi, M. Zhang, D. Wang, and M. Jeon. CarM: Hierarchical Episodic Memory for Continual Learning. In DAC, 2022.

Digital Library

[35]

C. F. S. Leite and Y. Xiao. Resource-Efficient Continual Learning for Sensor-Based Human Activity Recognition. ACM Transactions on Embedded Computing Systems, 21(6):1--25, 2022.

Digital Library

[36]

Y. Liu, Y. Su, A.-A. Liu, B. Schiele, and Q. Sun. Mnemonics Training: Multi-Class Incremental Learning Without Forgetting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.

[37]

D. Lopez-Paz and M. Ranzato. Gradient Episodic Memory for Continual Learning. In NeurIPS, 2017.

[38]

M. McCloskey and Neal. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. In Psychology of Learning and Motivation, 1989.

[39]

L. Pellegrini, V. Lomonaco, G. Graffieti, and D. Maltoni. Continual Learning at the Edge: Real-Time Training on Smartphone Devices. arXiv preprint arXiv:2105.13127, 2021.

[40]

A. Prabhu, H. A. Al Kader Hammoud, P. K. Dokania, P. H. Torr, S.-N. Lim, B. Ghanem, and A. Bibi. Computationally Budgeted Continual Learning: What Does Matter? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3698--3707, June 2023.

[41]

A. Prabhu, P. H. Torr, and P. K. Dokania. GDumb: A Simple Approach that Questions Our Progress in Continual Learning. In ECCV, 2020.

Digital Library

[42]

S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. iCaRL: Incremental Classifier and Representation Learning. In CVPR, 2017.

[43]

D. Rolnick, A. Ahuja, J. Schwarz, T. P. Lillicrap, and G. Wayne. Experience Replay for Continual Learning. 2019.

[44]

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. ImageNet Large Scale Visual Recognition Challenge. Int J Comput Vis, 115:211--252, 2015.

Digital Library

[45]

J. Salamon and J. P. Bello. Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification. IEEE Signal Processing Letters, 24:279--283, 2016.

[46]

J. Salamon, C. Jacoby, and J. P. Bello. A Dataset and Taxonomy for Urban Sound Research. In 22nd ACM International Conference on Multimedia (ACM-MM'14), pages 1041--1044, Orlando, FL, USA, Nov. 2014.

Digital Library

[47]

M. Schiemer, L. Fang, S. Dobson, and J. Ye. Online Continual Learning for Human Activity Recognition. Available at SSRN 4357622.

[48]

A. Seff, A. Beatson, D. Suo, and H. Liu. Continual Learning in Generative Adversarial Nets. arXiv preprint arxiv:1705.08395, 2017.

[49]

K. Shaheen, M. A. Hanif, O. Hasan, and M. Shafique. Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks. Journal of Intelligent & Robotic Systems, 105(1):9, 2022.

Digital Library

[50]

D. Shim, Z. Mai, J. Jeong, S. Sanner, H. Kim, and J. Jang. Online Class-Incremental Continual Learning with Adversarial Shapley Value. In AAAI, 2021.

[51]

H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual Learning with Deep Generative Replay. In NeurIPS, 2017.

[52]

G. Sokar, D. C. Mocanu, and M. Pechenizkiy. SpaceNet: Make Free Space For Continual Learning. Neurocomputing, 439:1--11, 2020.

[53]

K. N. Vokinger, S. Feuerriegel, and A. S. Kesselheim. Continual Learning in Medical Devices: FDA's Action Plan and Beyond. The Lancet Digital Health, 3(6):e337--e338, 2021.

[54]

Q. Wang, M. Xu, C. Jin, X. Dong, J. Yuan, X. Jin, G. Huang, Y. Liu, and X. Liu. Melon: Breaking the Memory Wall for Resource-Efficient on-Device Machine Learning. In MobiSys, 2022.

Digital Library

[55]

Y. Wang, Z. Jiang, X. Chen, P. Xu, Y. Zhao, Y. Lin, and Z. Wang. E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings. In NeurIPS, 2019.

[56]

Z. Wang, Z. Zhan, Y. Gong, G. Yuan, W. Niu, T. Jian, B. Ren, S. Ioannidis, Y. Wang, and J. Dy. SparCL: Sparse Continual Learning on the Edge. In NeurIPS, 2022.

[57]

C. Wu, L. Herranz, X. Liu, y. wang, J. van de Weijer, and B. Raducanu. Memory Replay GANs: Learning to Generate New Categories without Forgetting. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.

[58]

Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu. Large Scale Incremental Learning. In CVPR, 2019.

[59]

L. Yang, S. Lin, F. Zhang, J. Zhang, and D. Fan. Efficient Self-supervised Continual Learning with Progressive Task-correlated Layer Freezing. ArXiv, abs/2303.07477, 2023.

[60]

J. You, J.-W. Chung, and M. Chowdhury. Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training. In NSDI, 2023.

[61]

Y. Zhao, D. Saxena, and J. Cao. Memory-Efficient Domain Incremental Learning for Internet of Things. In SenSys, 2023.

Cited By

Yongwei WTao SShengyu ZFan WZhou ZHaibin CChengfei LLizhuang MChenglei YFei W(2024)Advances in edge-cloud collaboration and evolution for large-small modelsJournal of Image and Graphics10.11834/jig.24001129:6(1510-1534)Online publication date: 2024
https://doi.org/10.11834/jig.240011
Zhu SVoigt TRahimian FKo J(2024)On-device Training: A First Overview on Existing SystemsACM Transactions on Sensor Networks10.1145/369600320:6(1-39)Online publication date: 14-Sep-2024
https://dl.acm.org/doi/10.1145/3696003
Shen LYang QCui KZheng YWei XLiu JHan JOkoshi TKo JLiKamWa R(2024)FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated ClientsProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661880(398-411)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661880
Show More Cited By

Index Terms

Cost-effective On-device Continual Learning over Memory Hierarchy with Miro
1. Computing methodologies
  1. Machine learning
2. Human-centered computing
  1. Ubiquitous and mobile computing

Recommendations

Semi-supervised Continual Learning with Meta Self-training
CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management

Continual learning (CL) aims to enhance sequential learning by alleviating the forgetting of previously acquired knowledge. Recent advances in CL lack consideration of the real-world scenarios, where labeled data are scarce and unlabeled data are ...
Delta: A Cloud-assisted Data Enrichment Framework for On-Device Continual Learning
ACM MobiCom '24: Proceedings of the 30th Annual International Conference on Mobile Computing and Networking

In modern mobile applications, users frequently encounter various new contexts, necessitating on-device continual learning (CL) to ensure consistent model performance. While existing research predominantly focused on developing lightweight CL frameworks, ...
LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms
SenSys '23: Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems

Continual Learning (CL) allows applications such as user personalization and household robots to learn on the fly and adapt to context. This is an important feature when context, actions, and users change. However, enabling CL on resource-constrained ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ACM MobiCom '23: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking

October 2023

1605 pages

ISBN:9781450399906

DOI:10.1145/3570361

Chairs:
Xavier Costa,
Joerg Widmer,
Co-chairs:
Diego Perino,
Domenico Giustiniano,
Program Chair:
Haitham Al Hassanieh,
Program Co-chairs:
Arash Asadi,
Landon Cox

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ACM MobiCom '23

Sponsor:

SIGMOBILE

ACM MobiCom '23: 29th Annual International Conference on Mobile Computing and Networking

October 2 - 6, 2023

Madrid, Spain

Acceptance Rates

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
640
Total Downloads

Downloads (Last 12 months)403
Downloads (Last 6 weeks)29

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yongwei WTao SShengyu ZFan WZhou ZHaibin CChengfei LLizhuang MChenglei YFei W(2024)Advances in edge-cloud collaboration and evolution for large-small modelsJournal of Image and Graphics10.11834/jig.24001129:6(1510-1534)Online publication date: 2024
https://doi.org/10.11834/jig.240011
Zhu SVoigt TRahimian FKo J(2024)On-device Training: A First Overview on Existing SystemsACM Transactions on Sensor Networks10.1145/369600320:6(1-39)Online publication date: 14-Sep-2024
https://dl.acm.org/doi/10.1145/3696003
Shen LYang QCui KZheng YWei XLiu JHan JOkoshi TKo JLiKamWa R(2024)FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated ClientsProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661880(398-411)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661880
Gong CZheng ZWu FJia XChen GGanesan DLane NShi W(2024)Delta: A Cloud-assisted Data Enrichment Framework for On-Device Continual LearningProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3690701(1408-1423)Online publication date: 4-Dec-2024
https://dl.acm.org/doi/10.1145/3636534.3690701
Wu YLi LTian CChang TLin CWang CXu C(2024)Heterogeneity-Aware Memory Efficient Federated Learning via Progressive Layer Freezing2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS)10.1109/IWQoS61813.2024.10682916(1-10)Online publication date: 19-Jun-2024
https://doi.org/10.1109/IWQoS61813.2024.10682916
Soltoggio ABen-Iwhiwhu EBraverman VEaton EEpstein BGe YHalperin LHow JItti LJacobs MKantharaju PLe LLee SLiu XMonteiro SMusliner DNath SPanda PPeridis CPirsiavash HParekh VRoy KShperberg SSiegelmann HStone PVedder KWu JYang LZheng GKolouri S(2024)A collective AI via lifelong learning and sharing at the edgeNature Machine Intelligence10.1038/s42256-024-00800-26:3(251-264)Online publication date: 22-Mar-2024
https://doi.org/10.1038/s42256-024-00800-2

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents