poster

Fast Distributed Deep Learning via Worker-adaptive Batch Sizing

Authors:

Bo LiAuthors Info & Claims

SoCC '18: Proceedings of the ACM Symposium on Cloud Computing

Page 521

https://doi.org/10.1145/3267809.3275463

Published: 11 October 2018 Publication History

Get Access

Abstract

In heterogeneous or shared clusters, distributed learning processes are slowed down by straggling workers. In this work, we propose LB-BSP, a new synchronization scheme that eliminates stragglers by adapting each worker's training load (batch size) to its processing capability. For training in shared production clusters, a prerequisite for deciding the workers' batch sizes is to know their processing speeds before each iteration starts. To this end, we adopt NARX, an extended recurrent neural network that accounts for both the historical speeds and the driving factors such as CPU and memory in prediction.

References

[1]

Eugen Diaconescu. 2008. The use of NARX neural networks to predict chaotic time series. Wseas Transactions on computer research 3, 3 (2008), 182--191.

Digital Library

Google Scholar

[2]

Aaron Harlap, Henggang Cui, Wei Dai, Jinliang Wei, Gregory R Ganger, Phillip B Gibbons, Garth A Gibson, and Eric P Xing. 2016. Addressing the straggler problem for iterative convergent parallel ML. In ACM SoCC.

Digital Library

Google Scholar

[3]

Jiawei Jiang, Bin Cui, Ce Zhang, and Lele Yu. 2017. Heterogeneity-aware distributed parameter servers. In ACM SIGMOD.

Digital Library

Google Scholar

[4]

Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. 2014. Scaling Distributed Machine Learning with the Parameter Server. In USENIX OSDI.

Digital Library

Google Scholar

Cited By

View all

Ni CDu H(2023)Baileys: An Efficient Distributed Machine Learning Framework by Dynamic GroupingProceedings of the 2023 15th International Conference on Machine Learning and Computing10.1145/3587716.3587731(92-96)Online publication date: 17-Feb-2023
https://dl.acm.org/doi/10.1145/3587716.3587731
Ye QZhou YShi MSun YLv J(2023)DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural NetworksIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2022.32202247:4(1217-1227)Online publication date: Aug-2023
https://doi.org/10.1109/TETCI.2022.3220224
Wang HGuo STang BLi RYang YQu ZWang Y(2022)Heterogeneity-Aware Gradient Coding for Tolerating and Leveraging StragglersIEEE Transactions on Computers10.1109/TC.2021.306318071:4(779-794)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TC.2021.3063180
Show More Cited By

Index Terms

Fast Distributed Deep Learning via Worker-adaptive Batch Sizing
1. Computing methodologies
  1. Artificial intelligence
    1. Distributed artificial intelligence
      1. Cooperation and coordination
  2. Distributed computing methodologies
    1. Distributed algorithms

Recommendations

LBB: load-balanced batching for efficient distributed learning on heterogeneous GPU cluster
Abstract
As the cost of deep learning training increases, using heterogeneous GPU clusters is a reasonable way to scale cluster resources to support distributed deep learning (DDL) tasks. However, the commonly used synchronous stochastic gradient descent (...
BOA: batch orchestration algorithm for straggler mitigation of distributed DL training in heterogeneous GPU cluster
Abstract
Training deep learning model is a time-consuming job since it usually uses a large amount of data. To reduce the training time, most practitioners train the models in a GPU cluster environment in a distributed way. The synchronous stochastic ...
Fast Training of Deep LSTM Networks
Advances in Neural Networks – ISNN 2019
Abstract
Deep recurrent neural networks (RNN), such as LSTM, have many advantages over forward networks. However, the LSTM training method, such as backward propagation through time (BPTT), is really slow.
In this paper, by separating the LSTM cell into ...

Comments

Information & Contributors

Information

Published In

SoCC '18: Proceedings of the ACM Symposium on Cloud Computing

October 2018

546 pages

ISBN:9781450360111

DOI:10.1145/3267809

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2018

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

SoCC '18

Sponsor:

SoCC '18: ACM Symposium on Cloud Computing

October 11 - 13, 2018

CA, Carlsbad, USA

Acceptance Rates

Overall Acceptance Rate 169 of 722 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
435
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)2

Reflects downloads up to 15 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ni CDu H(2023)Baileys: An Efficient Distributed Machine Learning Framework by Dynamic GroupingProceedings of the 2023 15th International Conference on Machine Learning and Computing10.1145/3587716.3587731(92-96)Online publication date: 17-Feb-2023
https://dl.acm.org/doi/10.1145/3587716.3587731
Ye QZhou YShi MSun YLv J(2023)DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural NetworksIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2022.32202247:4(1217-1227)Online publication date: Aug-2023
https://doi.org/10.1109/TETCI.2022.3220224
Wang HGuo STang BLi RYang YQu ZWang Y(2022)Heterogeneity-Aware Gradient Coding for Tolerating and Leveraging StragglersIEEE Transactions on Computers10.1109/TC.2021.306318071:4(779-794)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TC.2021.3063180
Moreno-Alvarez SPaoletti MCavallaro GRico JHaut J(2022)Remote Sensing Image Classification Using CNNs With Balanced Gradient for Distributed Heterogeneous ComputingIEEE Geoscience and Remote Sensing Letters10.1109/LGRS.2022.317305219(1-5)Online publication date: 2022
https://doi.org/10.1109/LGRS.2022.3173052
Ji ZZhang XLi JWei JWei Z(2022)EP4DDL: addressing straggler problem in heterogeneous distributed deep learningThe Journal of Supercomputing10.1007/s11227-022-04466-878:13(15663-15680)Online publication date: 21-Apr-2022
https://dl.acm.org/doi/10.1007/s11227-022-04466-8
Zheng MMao DYang LWei YHu Z(2022)DOSP: an optimal synchronization of parameter server for distributed machine learningThe Journal of Supercomputing10.1007/s11227-022-04422-678:12(13865-13892)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.1007/s11227-022-04422-6
Xie YHe MMa TTian W(2022)Optimal distributed parallel algorithms for deep learning framework TensorflowApplied Intelligence10.1007/s10489-021-02588-952:4(3880-3900)Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1007/s10489-021-02588-9
Zhang JQu ZChen CWang HZhan YYe BGuo S(2021)Edge LearningACM Computing Surveys10.1145/346441954:7(1-36)Online publication date: 18-Jul-2021
https://dl.acm.org/doi/10.1145/3464419
Zhou QGuo SQu ZLi PLi LGuo MWang K(2021)Petrel: Heterogeneity-Aware Distributed Deep Learning Via Hybrid SynchronizationIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2020.304060132:5(1030-1043)Online publication date: 1-May-2021
https://doi.org/10.1109/TPDS.2020.3040601
Zhou QWang KLu HXu WSun YGuo S(2021)Canary: Decentralized Distributed Deep Learning Via Gradient Sketch and Partition in Multi-Interface NetworksIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2020.303673832:4(900-917)Online publication date: 1-Apr-2021
https://doi.org/10.1109/TPDS.2020.3036738
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

LBB: load-balanced batching for efficient distributed learning on heterogeneous GPU cluster

BOA: batch orchestration algorithm for straggler mitigation of distributed DL training in heterogeneous GPU cluster

Fast Training of Deep LSTM Networks

Comments

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

LBB: load-balanced batching for efficient distributed learning on heterogeneous GPU cluster

BOA: batch orchestration algorithm for straggler mitigation of distributed DL training in heterogeneous GPU cluster

Fast Training of Deep LSTM Networks

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations