GSPO: A Graph Substitution and Parallelization Joint Optimization Framework for DNN Inference
Authors: 
Zheng Xu, Xu Dai, Shaojun Wei, Shouyi Yin, Yang HuAuthors Info & Claims
Zheng Xu, Xu Dai, Shaojun Wei, Shouyi Yin, Yang HuAuthors Info & ClaimsArticle No.: 214, Pages 1 - 6
Abstract
This work proposes GSPO, an automatic unified framework that jointly applies graph substitution and parallelization for DNN inference. GSPO uses a joint optimization computation graph (JOCG) to represent graph substitution and parallelization at the operator level. Then, a novel cost model customized for joint optimization is used to evaluate the computation graph execution time quickly. With the graph partition and backtracking search algorithm, GSPO can find the optimal joint optimization solution within an acceptable search time. Compared to existing frameworks applying graph substitution or parallelization, GSPO can achieve up to 27.1% end-to-end performance improvement and reduce search time by up to 94.3%.
References
[1]
M. Abadi, P. Barham, J. Chen, Z. Chen, and et al. Davis, A. 2016. TensorFlow: A System for Large-Scale Machine Learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI) (2016).
[2]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR (2014).
[3]
Yuri Boykov and Vladimir Kolmogorov. 2004. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE transactions on pattern analysis and machine intelligence (2004).
[4]
T.; Jiang Z.; Zheng L. Yan E. Shen H. Cowan M. Wang L. et al. Chen, T.; Moreau. 2018. TVM: An automated end-to-end optimizing compiler for deep learning. 13th USENIX Symposium on Operating Systems Design and Implementation (2018).
[5]
CUDA. 2022. CUDA C++ Programming Guide. https://docs.nvidia.com/cuda.
[6]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR (2018).
[7]
Yaoyao Ding, Ligeng Zhu, Zhihao Jia, Gennady Pekhimenko, and Song Han. 2021. Ios: Inter-operator scheduler for cnn acceleration. Proceedings of Machine Learning and Systems (2021).
[8]
Jingzhi Fang, Yanyan Shen, Yue Wang, and Lei Chen. 2020. Optimizing DNN Computation Graph Using Graph Substitutions. Proc. VLDB Endow. (2020).
[9]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition (2016).
[10]
Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5 MB model size. arXiv preprint arXiv:1602.07360 (2016).
[11]
Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken. 2019. TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions. In Proceedings of the 27th ACM Symposium on Operating Systems Principles.
[12]
Zhihao Jia, James Thomas, Todd Warszawski, Mingyu Gao, Matei Zaharia, and Alex Aiken. 2019. Optimizing DNN Computation with Relaxed Graph Substitutions. In Proceedings of Machine Learning and Systems.
[13]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems (2012).
[14]
Woosuk Kwon, Gyeong-In Yu, Eunji Jeong, and Byung-Gon Chun. 2020. Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learning. In Advances in Neural Information Processing Systems.
[15]
PyTorch. 2022. PyTorch CUDA Semantics. https://pytorch.org.
[16]
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. Proceedings of the IEEE conference on computer vision and pattern recognition (2016).
[17]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. Advances in Neural Information Processing Systems (2017).
[18]
Yuxuan Zhao, Qi Sun, Zhuolun He, Yang Bai, and Bei Yu. 2023. AutoGraph: Optimizing DNN Computation Graph for Parallel GPU Kernel Execution. Association for the Advancement of Artificial Intelligence (2023).
[19]
Barret Zoph and Quoc V Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016).
[20]
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. 2018. Learning transferable architectures for scalable image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition (2018).
Index Terms
- GSPO: A Graph Substitution and Parallelization Joint Optimization Framework for DNN Inference
Index terms have been assigned to the content through auto-classification.
Recommendations
Compact hashing with joint optimization of search accuracy and time
CVPR '11: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern RecognitionSimilarity search, namely, finding approximate nearest neighborhoods, is the core of many large scale machine learning or vision applications. Recently, many research results demonstrate that hashing with compact codes can achieve promising performance ...
Vertex-substitution framework verifies the reconstruction conjecture for finite undirected graphs
Graphical abstractLeft: A vertex-substitution framework is proposed to prove the reconstruction conjecture. The vertex-deleted deck is composed of the subgraphs induced by removing each vertex in turn. The vertex-substituted deck is constructed by ...
Highlights- An algebraic proof is provided for the reconstruction conjecture in undirected graphs.
- Hypomorphism is reframed under a vertex-substitution operation.
- Bijections mapping cards between hypomorphic decks map vertices between parent ...
Abstract BackgroundThe Kelly-Ulam reconstruction conjecture proposes that a graph’s isomorphism class is determined by the classes of its multiset of vertex-deleted subgraphs. Although the conjecture has been verified for many families of undirected ...
Hierarchical Parallelization and Runtime Scheduling for Pregel-Like Graph Processing Systems
CLOUDCOM '14: Proceedings of the 2014 IEEE 6th International Conference on Cloud Computing Technology and ScienceGraph processing has become popular for various big data analytic applications. Google's Pregel framework enables vertex-centric graph processing in distributed environment based on Bulk Synchronous Parallel (BSP) model. However, the BSP model is ...
Comments
Information & Contributors
Information
Published In
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 November 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
DAC '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,770 of 5,499 submissions, 32%
Upcoming Conference
DAC '25
- Sponsor:
- sigda
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 84Total Downloads
- Downloads (Last 12 months)84
- Downloads (Last 6 weeks)47
Reflects downloads up to 06 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in