research-article

Long live TIME: improving lifetime for training-in-memory engines by structured gradient sparsification

Authors:

Huazhong YangAuthors Info & Claims

DAC '18: Proceedings of the 55th Annual Design Automation Conference

Article No.: 107, Pages 1 - 6

https://doi.org/10.1145/3195970.3196071

Published: 24 June 2018 Publication History

Abstract

Deeper and larger Neural Networks (NNs) have made breakthroughs in many fields. While conventional CMOS-based computing platforms are hard to achieve higher energy efficiency. RRAM-based systems provide a promising solution to build efficient Training-In-Memory Engines (TIME). While the endurance of RRAM cells is limited, it's a severe issue as the weights of NN always need to be updated for thousands to millions of times during training. Gradient sparsification can address this problem by dropping off most of the smaller gradients but introduce unacceptable computation cost. We proposed an effective framework, SGS-ARS, including Structured Gradient Sparsification (SGS) and Aging-aware Row Swapping (ARS) scheme, to guarantee write balance across whole RRAM crossbars and prolong the lifetime of TIME. Our experiments demonstrate that 356× lifetime extension is achieved when TIME is programmed to train ResNet-50 on Imagenet dataset with our SGS-ARS framework.

References

[1]

Aji et al. 2017. Sparse Communication for Distributed Gradient Descent. In Empirical Methods in Natural Language Processing (EMNLP).

[2]

Karsten Beckmann et al. 2016. Nanoscale Hafnium Oxide RRAM Devices Exhibit Pulse Dependent Behavior and Multi-level Resistance Capability. Mrs Advances 1 (2016), 1--6.

[3]

G. W. Burr et al. 2015. Large-scale neural networks implemented with nonvolatile memory as the synaptic weight element: Comparative performance analysis (accuracy, speed, and power). In IEEE International Electron Devices Meeting. 4.4.1--4.4.4.

[4]

C. H Cheng et al. 2010. Novel Ultra-low power RRAM with good endurance and retention. In VLSI Technology. 85--86.

[5]

Ming Cheng et al. 2017. TIME: A Training-in-memory Architecture for Memristor-based Deep Neural Networks. In Proceedings of the 54th Annual Design Automation Conference 2017. ACM, 26.

Digital Library

[6]

Kaiming He et al. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[7]

Kevin Hsieh et al. 2017. Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, Boston, MA, 629--647.

Digital Library

[8]

Hsu et al. 2013. Self-rectifying bipolar TaOx/TiO2 RRAM with superior endurance over 1012 cycles for 3D high-density storage-class memory. In VLSI Technology. T166--T167.

[9]

Andrej Karpathy et al. 2015. Deep visual-semantic alignments for generating image descriptions. In Computer Vision and Pattern Recognition.

[10]

Yujun Lin et al. 2018. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training. International Conference on Learning Representations (2018).

[11]

Wei Liu et al. 2016. SSD: Single Shot MultiBox Detector. In European Conference on Computer Vision. 21--37.

[12]

Olga Russakovsky et al. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115, 3 (2015), 211--252.

Digital Library

[13]

Linghao Song et al. 2017. PipeLayer: A Pipelined ReRAM-Based Accelerator for Deep Learning. In IEEE International Symposium on High PERFORMANCE Computer Architecture. 541--552.

[14]

Christian Szegedy et al. 2015. Going deeper with convolutions. In Computer Vision and Pattern Recognition. 1--9.

[15]

M. M. Waldrop. 2016. The chips are down for Moore's law. Nature 530, 7589 (2016), 144.

[16]

Yu Wang et al. 2016. Low power Convolutional Neural Networks on a chip. In IEEE International Symposium on Circuits and Systems. 129--132.

[17]

Lixue Xia et al. 2017. Fault-Tolerant Training with On-Line Fault Detection for RRAM-Based Neural Computing Systems. In Design Automation Conference. 1--6.

Digital Library

Cited By

Zhou HWu BCheng HZhao WWei XLiu JFeng DTong W(2023)ODLPIM: A Write-Optimized and Long-Lifetime ReRAM-Based Accelerator for Online Deep Learning2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE56975.2023.10137130(1-6)Online publication date: Apr-2023
https://doi.org/10.23919/DATE56975.2023.10137130
Yang XYang HDoppa JPande PChakrabartys KLi H(2023)ESSENCE: Exploiting Structured Stochastic Gradient Pruning for Endurance-Aware ReRAM-Based In-Memory Training SystemsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2022.321654642:7(2187-2199)Online publication date: Jul-2023
https://doi.org/10.1109/TCAD.2022.3216546
Zhu HGu JFeng CLiu MJiang ZChen RPan D(2023)ELight: Toward Efficient and Aging-Resilient Photonic In-Memory NeurocomputingIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2022.318096942:3(820-833)Online publication date: Mar-2023
https://doi.org/10.1109/TCAD.2022.3180969
Show More Cited By

Recommendations

ERIS live: a NUMA-aware in-memory storage engine for tera-scale multiprocessor systems
SIGMOD '14: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data

The ever-growing demand for more computing power forces hardware vendors to put an increasing number of multiprocessors into a single server system, which usually exhibits a non-uniform memory access (NUMA). In-memory database systems running on NUMA ...
Long Live TIME: Improving Lifetime for Training-In-Memory Engines by Structured Gradient Sparsification
2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC)
Deeper and larger Neural Networks (NNs) have made breakthroughs in many fields. While conventional CMOS-based computing platforms are hard to achieve higher energy efficiency. RRAM-based systems provide a promising solution to build efficient Training-In-...
Multiple neural networks for a long term time series forecast

The artificial neural network (ANN) methodology has been used in various time series prediction applications. However, the accuracy of a neural network model may be seriously compromised when it is used recursively for making long-term multi-step ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

DAC '18: Proceedings of the 55th Annual Design Automation Conference

June 2018

1089 pages

ISBN:9781450357005

DOI:10.1145/3195970

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

EDAC: Electronic Design Automation Consortium
SIGDA: ACM Special Interest Group on Design Automation
IEEE Council on Electronic Design Automation (CEDA)

In-Cooperation

SIGBED: ACM Special Interest Group on Embedded Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

National Key R&D Program of China
Joint Fund of Equipment pre-Research and Ministry of Education
National Natural Science Foundation of China
Beijing National Research Center for Information Science and Technology

Conference

DAC '18

Sponsor:

EDAC
SIGDA

DAC '18: The 55th Annual Design Automation Conference 2018

June 24 - 29, 2018

California, San Francisco

Acceptance Rates

Overall Acceptance Rate 1,770 of 5,499 submissions, 32%

Upcoming Conference

DAC '25

Sponsor:
sigda

62nd ACM/IEEE Design Automation Conference

June 22 - 26, 2025

San Francisco , CA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
418
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)2

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou HWu BCheng HZhao WWei XLiu JFeng DTong W(2023)ODLPIM: A Write-Optimized and Long-Lifetime ReRAM-Based Accelerator for Online Deep Learning2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE56975.2023.10137130(1-6)Online publication date: Apr-2023
https://doi.org/10.23919/DATE56975.2023.10137130
Yang XYang HDoppa JPande PChakrabartys KLi H(2023)ESSENCE: Exploiting Structured Stochastic Gradient Pruning for Endurance-Aware ReRAM-Based In-Memory Training SystemsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2022.321654642:7(2187-2199)Online publication date: Jul-2023
https://doi.org/10.1109/TCAD.2022.3216546
Zhu HGu JFeng CLiu MJiang ZChen RPan D(2023)ELight: Toward Efficient and Aging-Resilient Photonic In-Memory NeurocomputingIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2022.318096942:3(820-833)Online publication date: Mar-2023
https://doi.org/10.1109/TCAD.2022.3180969
Lee SFouda MLee JEltawil AKurdahi F(2023)Offline Training-Based Mitigation of IR Drop for ReRAM-Based Deep Neural Network AcceleratorsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2022.317700242:2(521-532)Online publication date: Feb-2023
https://doi.org/10.1109/TCAD.2022.3177002
Smagulova KFouda MKurdahi FSalama KEltawil A(2023)Resistive Neural Hardware AcceleratorsProceedings of the IEEE10.1109/JPROC.2023.3268092111:5(500-527)Online publication date: May-2023
https://doi.org/10.1109/JPROC.2023.3268092
Ogbogu CAbernot MDelacour CTodri-Sanial APasricha SPande P(2023)Energy-Efficient Machine Learning Acceleration: From Technologies to Circuits and Systems2023 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED)10.1109/ISLPED58423.2023.10244360(1-8)Online publication date: 7-Aug-2023
https://doi.org/10.1109/ISLPED58423.2023.10244360
Zhang JWang CZhu ZKline DJones AYang HWang Y(2023)Realizing Extreme Endurance Through Fault-aware Wear Leveling and Improved Tolerance2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)10.1109/HPCA56546.2023.10071093(964-976)Online publication date: Feb-2023
https://doi.org/10.1109/HPCA56546.2023.10071093
Zhao SWang FLiu BFeng DLiu Y(2023)LayCO: Achieving Least Lossy Accuracy for Most Efficient RRAM-Based Deep Neural Network Accelerator via Layer-Centric Co-OptimizationJournal of Computer Science and Technology10.1007/s11390-023-2545-y38:2(328-347)Online publication date: 30-Mar-2023
https://doi.org/10.1007/s11390-023-2545-y
Li SHoefler TLee JAgrawal KSpear M(2022)Near-optimal sparse allreduce for distributed deep learningProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming10.1145/3503221.3508399(135-149)Online publication date: 2-Apr-2022
https://dl.acm.org/doi/10.1145/3503221.3508399
Meng ZSun YQian WOshana R(2022)Write or notProceedings of the 59th ACM/IEEE Design Automation Conference10.1145/3489517.3530558(985-990)Online publication date: 10-Jul-2022
https://dl.acm.org/doi/10.1145/3489517.3530558
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents