research-article

PRE-NAS: predictor-assisted evolutionary neural architecture search

Authors:

Vic Ciesielski,

Haytham M. Fayek,

Xiaojun ChangAuthors Info & Claims

GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference

Pages 1066 - 1074

https://doi.org/10.1145/3512290.3528727

Published: 08 July 2022 Publication History

Abstract

Neural architecture search (NAS) aims to automate architecture engineering in neural networks. This often requires a high computational overhead to evaluate a number of candidate networks from the set of all possible networks in the search space during the search. Prediction of the networks' performance can alleviate this high computational overhead by mitigating the need for evaluating every candidate network. Developing such a predictor typically requires a large number of evaluated architectures which may be difficult to obtain. We address this challenge by proposing a novel evolutionary-based NAS strategy, Predictor-assisted E-NAS (PRE-NAS), which can perform well even with an extremely small number of evaluated architectures. PRE-NAS leverages new evolutionary search strategies and integrates high-fidelity weight inheritance over generations. Unlike one-shot strategies, which may suffer from bias in the evaluation due to weight sharing, offspring candidates in PRE-NAS are topologically homogeneous, which circumvents bias and leads to more accurate predictions. Extensive experiments on NAS-Bench-201 and DARTS search spaces show that PRE-NAS can outperform state-of-the-art NAS methods. With only a single GPU searching for 0.6 days, competitive architecture can be found by PRE-NAS which achieves 2.40% and 24% test error rates on CIFAR-10 and ImageNet respectively.

References

[1]

Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. 2017. Practical Neural Network Performance Prediction for Early Stopping. CoRR abs/1705.10823 (2017).

[2]

Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. 2018. Understanding and Simplifying One-Shot Architecture Search. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 550--559.

[3]

James Bergstra and Yoshua Bengio. 2012. Random Search for Hyper-Parameter Optimization. J. Mach. Learn. Res. 13 (2012), 281--305.

[4]

Han Cai, Ligeng Zhu, and Song Han. 2018. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. CoRR abs/1812.00332 (2018). arXiv:1812.00332

[5]

Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. 2019. Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).

[6]

Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. 2020. Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XV, Vol. 12360. Springer, 465--480.

[7]

Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6, 2 (2002), 182--197.

Digital Library

[8]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171--4186.

[9]

Xuanyi Dong and Yi Yang. 2019. One-Shot Neural Architecture Search via Self-Evaluated Template Network. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 3680--3689.

[10]

Xuanyi Dong and Yi Yang. 2019. Searching for a Robust Neural Architecture in Four GPU Hours. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16--20, 2019. Computer Vision Foundation / IEEE, 1761--1770.

[11]

Xuanyi Dong and Yi Yang. 2020. NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26--30, 2020.

[12]

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6--9, 2019.

[13]

Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and Efficient Hyperparameter Optimization at Scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10--15, 2018 (Proceedings of Machine Learning Research, Vol. 80), Jennifer G. Dy and Andreas Krause (Eds.). PMLR, 1436--1445.

[14]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. CoRR abs/1502.01852 (2015). arXiv:1502.01852

[15]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]

Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely Connected Convolutional Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21--26, 2017. IEEE Computer Society, 2261--2269.

[17]

Andrew Hundt, Varun Jain, and Gregory D. Hager. 2019. sharpDARTS: Faster and More Accurate Differentiable Architecture Search. CoRR abs/1903.09900 (2019).

[18]

Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. 2017. Learning curve prediction with Bayesian neural networks. International Conference on Learning Representations (2017).

[19]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25. 1097--1105.

[20]

Liam Li and Ameet Talwalkar. 2020. Random Search and Reproducibility for Neural Architecture Search. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference (Proceedings of Machine Learning Research, Vol. 115). 367--377.

[21]

Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. 2018. Progressive Neural Architecture Search. In Proceedings of the European Conference on Computer Vision (ECCV).

Digital Library

[22]

Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019. DARTS: Differentiable Architecture Search. In International Conference on Learning Representations (ICLR).

[23]

Zhichao Lu, Kalyanmoy Deb, Erik D. Goodman, Wolfgang Banzhaf, and Vishnu Naresh Boddeti. 2020. NSGANetV2: Evolutionary Multi-objective Surrogate-Assisted Neural Architecture Search. In Proceedings of the European Conference on Computer Vision (ECCV), Vol. 12346. Springer, 35--51.

Digital Library

[24]

Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh D. Dhebar, Kalyanmoy Deb, Erik D. Goodman, and Wolfgang Banzhaf. 2019. NSGA-Net: neural architecture search using multi-objective genetic algorithm. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2019, Prague, Czech Republic, July 13--17, 2019, Anne Auger and Thomas Stützle (Eds.). ACM, 419--427.

Digital Library

[25]

Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. 2018. Efficient Neural Architecture Search via Parameters Sharing. In Proceedings of the 35th International Conference on Machine Learning (ICML). 4095--4104.

[26]

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. 2019. Regularized Evolution for Image Classifier Architecture Search. In AAAI Conference on Artificial Intelligence, Vol. 33. 4780--4789.

[27]

Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. 2017. Large-Scale Evolution of Image Classifiers. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70). International Convention Centre, Sydney, Australia, 2902--2911.

[28]

Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James T. Kwok, and Tong Zhang. 2020. Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS. In Advances in Neural Information Processing Systems, Vol. 33.

[29]

Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. 2020. NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search. CoRR abs/2008.09777 (2020).

[30]

Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations (ICLR).

[31]

Nilotpal Sinha and Kuan-Wen Chen. 2021. Evolving neural architecture using one shot model. In GECCO '21: Genetic and Evolutionary Computation Conference, Lille, France, July 10--14, 2021, Francisco Chicano and Krzysztof Krawiec (Eds.). ACM, 910--918.

Digital Library

[32]

Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao. 2018. A Genetic Programming Approach to Designing Convolutional Neural Network Architectures. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18. International Joint Conferences on Artificial Intelligence Organization, 5369--5373.

[33]

Yanan Sun, Handing Wang, Bing Xue, Yaochu Jin, Gary G. Yen, and Mengjie Zhang. 2020. Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor. IEEE Trans. Evol. Comput. 24, 2 (2020), 350--364.

[34]

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. 2019. MnasNet: Platform-Aware Neural Architecture Search for Mobile. In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR). 2820--2828.

[35]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, Vol. 30. 5998--6008.

[36]

Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. 2016. Network Morphism. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19--24, 2016 (JMLR Workshop and Conference Proceedings, Vol. 48), Maria-Florina Balcan and Kilian Q. Weinberger (Eds.). 564--572.

[37]

Wei Wen, Hanxiao Liu, Yiran Chen, Hai Helen Li, Gabriel Bender, and Pieter-Jan Kindermans. 2020. Neural Predictor for Neural Architecture Search. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XXIX, Vol. 12374. Springer, 660--676.

[38]

Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. 2020. CARS: Continuous Evolution for Efficient Neural Architecture Search. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13--19, 2020. IEEE, 1826--1835.

[39]

Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. 2020. Evaluating The Search Phase of Neural Architecture Search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26--30, 2020.

[40]

Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. 2018. Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search. In ICML 2018 AutoML Workshop.

[41]

Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. 2020. EcoNAS: Finding Proxies for Economical Neural Architecture Search. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13--19, 2020. IEEE, 11393--11401.

[42]

Hui Zhu, Zhulin An, Chuanguang Yang, Kaiqiang Xu, Erhu Zhao, and Yongjun Xu. 2019. EENA: Efficient Evolution of Neural Architecture. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27--28, 2019. IEEE, 1891--1899.

[43]

Barret Zoph and Quoc V. Le. 2017. Neural Architecture Search with Reinforcement Learning. In International Conference on Learning Representations (ICLR).

[44]

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2018. Learning Transferable Architectures for Scalable Image Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Cited By

Vo ALuong NLi XHandl J(2024)Efficient Multi-Objective Neural Architecture Search via Pareto Dominance-based Novelty SearchProceedings of the Genetic and Evolutionary Computation Conference10.1145/3638529.3654064(1146-1155)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638529.3654064
Guo BXu LChen TYe PHe SLiu HChen J(2024)Latency-Aware Neural Architecture Performance Predictor With Query-to-Tier TechniqueIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.328768434:7(5868-5883)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TCSVT.2023.3287684
Song XXie XLv ZYen GDing WLv JSun Y(2024)Efficient Evaluation Methods for Neural Architecture Search: A SurveyIEEE Transactions on Artificial Intelligence10.1109/TAI.2024.34774575:12(5990-6011)Online publication date: Dec-2024
https://doi.org/10.1109/TAI.2024.3477457
Show More Cited By

Index Terms

PRE-NAS: predictor-assisted evolutionary neural architecture search
1. Computing methodologies
  1. Artificial intelligence
    1. Search methodologies
      1. Discrete space search
2. Networks
  1. Network performance evaluation
    1. Network performance modeling

Recommendations

Fast Evolutionary Neural Architecture Search by Contrastive Predictor with Linear Regions
GECCO '23: Proceedings of the Genetic and Evolutionary Computation Conference

Evolutionary neural architecture search (ENAS) has emerged as a promising approach to finding high-performance neural architectures. However, widespread application has been limited by the expensive computational costs due to the nature of ...
How predictors affect the RL-based search strategy in Neural Architecture Search?
Abstract
Predictor-based Neural Architecture Search is an important topic since it can efficiently reduce the computational cost of evaluating candidate architectures. Most existing predictor-based NAS algorithms aim to design different predictors to ...
Highlights
- Theoretically analyze predictor’s impact on RL search strategy for the first time.
- Perform comprehensive experiments to investigate RL-Predictor based NAS algorithms.
- Propose RL-Predictor-based NAS framework to enhance search ...
Multi-population evolutionary neural architecture search with stacked generalization
Abstract
In recent years, neural architecture search (NAS) algorithms based on Evolutionary Computation (EC) have demonstrated immense potential in the automated design of deep neural network architectures, garnering widespread attention in the field of ...
Highlights
- A novel multi-population evolutionary search strategy is proposed.
- A performance predictor was designed using stacked generalization.
- The multi-head attention mechanism was employed.

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference

July 2022

1472 pages

ISBN:9781450392372

DOI:10.1145/3512290

Editor:
Jonathan E. Fieldsend
University of Exeter
,
General Chair:
Markus Wagner
The University of Adelaide

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GECCO '22

Sponsor:

SIGEVO

GECCO '22: Genetic and Evolutionary Computation Conference

July 9 - 13, 2022

Massachusetts, Boston

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
156
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)3

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Vo ALuong NLi XHandl J(2024)Efficient Multi-Objective Neural Architecture Search via Pareto Dominance-based Novelty SearchProceedings of the Genetic and Evolutionary Computation Conference10.1145/3638529.3654064(1146-1155)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638529.3654064
Guo BXu LChen TYe PHe SLiu HChen J(2024)Latency-Aware Neural Architecture Performance Predictor With Query-to-Tier TechniqueIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.328768434:7(5868-5883)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TCSVT.2023.3287684
Song XXie XLv ZYen GDing WLv JSun Y(2024)Efficient Evaluation Methods for Neural Architecture Search: A SurveyIEEE Transactions on Artificial Intelligence10.1109/TAI.2024.34774575:12(5990-6011)Online publication date: Dec-2024
https://doi.org/10.1109/TAI.2024.3477457
Nie JYang YZhu QLin Q(2024)NAS-SW: Efficient Neural Architecture Search with Stage-Wise Search Strategy2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650420(1-9)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650420
Ma BZhang JXia YTao D(2024)VNAS: Variational Neural Architecture SearchInternational Journal of Computer Vision10.1007/s11263-024-02014-wOnline publication date: 23-Apr-2024
https://doi.org/10.1007/s11263-024-02014-w
Liang JCao HLu YSu M(2024)Architecture search of accurate and lightweight CNNs using genetic algorithmGenetic Programming and Evolvable Machines10.1007/s10710-024-09484-425:1Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1007/s10710-024-09484-4
Luo GLi HChen ZZhou Y(2024)Pareto-Informed Multi-objective Neural Architecture SearchParallel Problem Solving from Nature – PPSN XVIII10.1007/978-3-031-70071-2_23(369-385)Online publication date: 7-Sep-2024
https://doi.org/10.1007/978-3-031-70071-2_23
Cao BZhou ZLiu XHossain MLv ZChen JWang WJeon G(2023)Adaptive Multiobjective Evolutionary Neural Architecture Search for GANs based on Two-Factor Cooperative Mutation MechanismProceedings of the 2023 Workshop on Advanced Multimedia Computing for Smart Manufacturing and Engineering10.1145/3606042.3616463(71-76)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3606042.3616463
Wang STang HOuyang JSilva SPaquete L(2023)A Neural Architecture Search Method using Auxiliary Evaluation Metric based on ResNet ArchitectureProceedings of the Companion Conference on Genetic and Evolutionary Computation10.1145/3583133.3590618(687-690)Online publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1145/3583133.3590618
Peng YSong ACiesielski VFayek HChang XSilva SPaquete L(2023)Fast Evolutionary Neural Architecture Search by Contrastive Predictor with Linear RegionsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3583131.3590452(1257-1266)Online publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1145/3583131.3590452
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents