Eventually, all you need is a simple evolutionary algorithm (for neuroevolution of continuous control policies)
Pages 1904 - 1913
Abstract
Artificial neural networks (ANNs) are a popular choice for tackling continuous control tasks due to their approximation abilities. When the ANN architecture is fixed, finding optimal weights becomes a numerical optimization problem, suitable for evolutionary algorithms (EAs), i.e., a form of neuroevolution. Here, we compare the performance of well-established EAs in solving neuroevolution problems, focusing on continuous control. We evaluate them on a set of navigation problems and a set of control problems based on modular soft robots. As a reference, we compare the same EAs on regression problems and classic numerical optimization benchmarks. Our findings suggest that simple EAs like genetic algorithm (GA) and differential evolution (DE) achieve good performance on control problems, even if they are surpassed by more sophisticated algorithms on benchmark problems. We hypothesize that the effectiveness of these simpler EAs stems from their use of crossover, which can be advantageous in the rugged fitness landscapes encountered in complex control tasks.
References
[1]
David Ackley. 2012. A connectionist machine for genetic hillclimbing. Vol. 28. Springer science & business media.
[2]
Marco Baioletti, Gabriele Di Bari, Alfredo Milani, and Valentina Poggioni. 2020. Differential evolution for neural networks optimization. Mathematics 8, 1 (2020). 69.
[3]
Alberto Bartoli, Marco Catto, Andrea De Lorenzo, Eric Medvet, and Jacopo Talamini. 2020. Mechanisms of social learning in evolved artificial life. In Artificial Life Conference Proceedings 32. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info ..., 190--198.
[4]
Lucian Buşoniu, Tim De Bruin, Domagoj Tolić, Jens Kober, and Ivana Palunko. 2018. Reinforcement learning for control: Performance, stability, and deep approximators. Annual Reviews in Control 46 (2018), 8--28.
[5]
Maurice Clerc. 2012. Beyond standard particle swarm optimisation. In Innovations and Developments of Swarm Intelligence Applications. IGI Global, 1--19.
[6]
Swagatam Das, Sayan Maity, Bo-Yang Qu, and Ponnuthurai Nagaratnam Suganthan. 2011. Real-parameter evolutionary multimodal optimization---A survey of the state-of-the-art. Swarm and Evolutionary Computation 1, 2 (2011), 71--88.
[7]
Joaquín Derrac, Salvador García, Daniel Molina, and Francisco Herrera. 2011. A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm and Evolutionary Computation 1, 1 (2011), 3--18.
[8]
Andrea Ferigo, Eric Medvet, and Giovanni Iacca. 2022. Optimizing the sensory apparatus of voxel-based soft robots through evolution and babbling. SN Computer Science 3, 2 (2022), 109.
[9]
Nikolaus Hansen. 2016. The CMA evolution strategy: A tutorial. arXiv preprint arXiv:1604.00772 (2016).
[10]
Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. 2021. COCO: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software 36, 1 (2021), 114--144.
[11]
Matthew Hausknecht, Joel Lehman, Risto Miikkulainen, and Peter Stone. 2014. A neuroevolution approach to general atari game playing. IEEE Transactions on Computational Intelligence and AI in Games 6, 4 (2014), 355--366.
[12]
Yifan He and Claus Aranha. 2024. Evolving Benchmark Functions to Compare Evolutionary Algorithms via Genetic Programming. arXiv preprint arXiv:2403.14146 (2024).
[13]
Verena Heidrich-Meisner and Christian Igel. 2009. Neuroevolution strategies for episodic reinforcement learning. Journal of Algorithms 64, 4 (2009), 152--168.
[14]
Daniel Hein, Alexander Hentschel, Thomas A Runkler, and Steffen Udluft. 2016. Reinforcement learning with particle swarm optimization policy (PSO-P) in continuous state and action spaces. International Journal of Swarm Intelligence Research (IJSIR) 7, 3 (2016), 23--42.
[15]
Masahiko Higashi, Reiji Suzuki, and Takaya Arita. 2018. The role of social learning in the evolution on a rugged fitness landscape. Frontiers in Physics 6 (2018), 88.
[16]
Jonathan Hiller and Hod Lipson. 2011. Automatic design and manufacture of soft robots. IEEE Transactions on Robotics 28, 2 (2011), 457--466.
[17]
Seyed Mohammad Jafar Jalali, Sajad Ahmadian, Abbas Khosravi, Seyedali Mirjalili, Mohammad Reza Mahmoudi, and Saeid Nahavandi. 2020. Neuroevolution-based autonomous robot navigation: A comparative study. Cognitive Systems Research 62 (2020), 35--43.
[18]
Momin Jamil and Xin-She Yang. 2013. A literature survey of benchmark functions for global optimisation problems. International Journal of Mathematical Modelling and Numerical Optimisation 4, 2 (2013), 150--194.
[19]
Krasimir Kolarov. 1997. Landscape ruggedness in evolutionary algorithms. In Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC'97). IEEE, 19--24.
[20]
Julie Legrand, Seppe Terryn, Ellen Roels, and Bram Vanderborght. 2023. Reconfigurable, multi-material, voxel-based soft robots. IEEE Robotics and Automation Letters 8, 3 (2023), 1255--1262.
[21]
Horia Mania, Aurelia Guy, and Benjamin Recht. 2018. Simple random search of static linear policies is competitive for reinforcement learning. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2018/file/7634ea65a4e6d9041cfd3f7de18e334a-Paper.pdf
[22]
Eric Medvet, Alberto Bartoli, Andrea De Lorenzo, and Giulio Fidel. 2020. Evolution of distributed neural controllers for voxel-based soft robots. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference. 112--120.
[23]
Eric Medvet, Alberto Bartoli, Andrea De Lorenzo, and Stefano Seriani. 2020. 2D-VSR-Sim: A simulation tool for the optimization of 2-D voxel-based soft robots. SoftwareX 12 (2020), 100573.
[24]
Eric Medvet, Alberto Bartoli, Federico Pigozzi, and Marco Rochelli. 2021. Biodiversity in evolved voxel-based soft robots. In Proceedings of the Genetic and Evolutionary Computation Conference. 129--137.
[25]
Eric Medvet and Giorgia Nadizar. 2024. GP for Continuous Control: Teacher or Learner? The Case of Simulated Modular Soft Robots. In Genetic Programming Theory and Practice XX. Springer, 203--224.
[26]
Eric Medvet, Giorgia Nadizar, and Luca Manzoni. 2022. JGEA: a Modular Java Framework for Experimenting with Evolutionary Computation. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 2009--2018.
[27]
Eric Medvet, Stefano Seriani, Alberto Bartoli, and Paolo Gallina. 2020. Evolutionary optimization of sliding contact positions in powered floor systems for mobile robots. at-Automatisierungstechnik 68, 2 (2020), 97--109.
[28]
Alican Mertan and Nick Cheney. 2024. Investigating Premature Convergence in Co-optimization of Morphology and Control in Evolved Virtual Soft Robots. In European Conference on Genetic Programming (Part of EvoStar). Springer.
[29]
David E Moriarty, Alan C Schultz, and John J Grefenstette. 1999. Evolutionary algorithms for reinforcement learning. Journal of Artificial Intelligence Research 11 (1999), 241--276.
[30]
Heinz Mühlenbein, M Schomisch, and Joachim Born. 1991. The parallel genetic algorithm as function optimizer. Parallel computing 17, 6--7 (1991), 619--632.
[31]
Jessica Mégane, Eric Medvet, Nuno Lourenço, and Penousal Machado. 2024. Grammar-based Evolution of Polyominoes. In European Conference on Genetic Programming (Part of EvoStar). Springer.
[32]
Giorgia Nadizar, Eric Medvet, Stefano Nichele, and Sidney Pontes-Filho. 2023. An experimental comparison of evolved neural network models for controlling simulated modular soft robots. Applied Soft Computing (2023), 110610.
[33]
Giorgia Nadizar, Eric Medvet, Kathryn Walker, and Sebastian Risi. 2023. A Fully-Distributed Shape-Aware Neural Controller for Modular Robots. In Proceedings of the Genetic and Evolutionary Computation Conference (Lisbon, Portugal) (GECCO '23). Association for Computing Machinery, New York, NY, USA, 184--192.
[34]
Simon Naumov. 2023. Empirical Analysis of Crossover-Based Evolutionary Algorithms on Rugged Landscape. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation. 2370--2373.
[35]
Paolo Pagliuca, Nicola Milano, and Stefano Nolfi. 2020. Efficacy of modern neuroevolutionary strategies for continuous control optimization. Frontiers in Robotics and AI 7 (2020), 98.
[36]
Felice Andrea Pellegrino, Franco Blanchini, Gianfranco Fenu, and Erica Salvato. 2022. Closed-loop Control from Data-Driven Open-Loop Optimal Control Trajectories. In 2022 European Control Conference (ECC). 1379--1384.
[37]
Felice A. Pellegrino, Franco Blanchini, Gianfranco Fenu, and Erica Salvato. 2023. Data-driven dynamic relatively optimal control. European Journal of Control 74 (2023), 100839. 2023 European Control Conference Special Issue.
[38]
Federico Pigozzi, Yujin Tang, Eric Medvet, and David Ha. 2022. Evolving modular soft robots without explicit inter-module communication using local self-attention. In Proceedings of the Genetic and Evolutionary Computation Conference. 148--157.
[39]
Christiaan J Pretorius, Mathys C Du Plessis, and John W Gonsalves. 2017. Neuroevolution of inverted pendulum control: a comparative study of simulation techniques. Journal of Intelligent & Robotic Systems 86 (2017), 419--445.
[40]
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864 (2017).
[41]
Rainer Storn and Kenneth Price. 1997. Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. Journal of global optimization 11 (1997), 341--359.
[42]
Jacopo Talamini, Eric Medvet, Alberto Bartoli, and Andrea De Lorenzo. 2019. Evolutionary synthesis of sensing controllers for voxel-based soft robots. In Artificial life conference proceedings. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info ..., 574--581.
[43]
Matthew E Taylor, Shimon Whiteson, and Peter Stone. 2006. Comparing evolutionary and temporal difference methods in a reinforcement learning domain. In Proceedings of the 8th annual conference on Genetic and evolutionary computation. 1321--1328.
[44]
Sarah L Thomson, Léni K Le Goff, Emma Hart, and Edgar Buchanan. 2024. Understanding fitness landscapes in morpho-evolution via local optima networks. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '24). Association for Computing Machinery, New York, NY, USA.
[45]
David R White, James McDermott, Mauro Castelli, Luca Manzoni, Brian W Goldman, Gabriel Kronberger, Wojciech Jaśkowski, Una-May O'Reilly, and Sean Luke. 2013. Better GP benchmarks: community survey results and proposals. Genetic Programming and Evolvable Machines 14 (2013), 3--29.
[46]
Darrell Whitley, Stephen Dominic, Rajarshi Das, and Charles W Anderson. 1993. Genetic reinforcement learning for neurocontrol problems. Machine Learning 13 (1993), 259--284.
[47]
Bo Zhao, Fangchao Luo, Haowei Lin, and Derong Liu. 2021. Particle swarm optimized neural networks based local tracking control scheme of unknown nonlinear interconnected systems. Neural Networks 134 (2021), 54--63.
Index Terms
- Eventually, all you need is a simple evolutionary algorithm (for neuroevolution of continuous control policies)
Recommendations
Stable and Sample-Efficient Policy Search for Continuous Control via Hybridizing Phenotypic Evolutionary Algorithm with the Double Actors Regularized Critics
GECCO '23: Proceedings of the Genetic and Evolutionary Computation ConferenceEvolutionary Reinforcement Learning arises from hybridizing the sample efficiency of policy gradient with the stability of evolutionary computation. Proximal Distilled Evolutionary Reinforcement Learning (PDERL) implements the hybridization by having ...
Attraction basin sphere estimating genetic algorithm for neuroevolution problems
In our previous work, we proposed a niching genetic algorithm: attraction basin sphere estimating genetic algorithm (ABSEGA). It can detect and recognize attraction basin spheres on a fitness landscape. Attraction basin spheres are used to improve the ...
Improving Distributed Neuroevolution Using Island Extinction and Repopulation
Applications of Evolutionary ComputationAbstractNeuroevolution commonly uses speciation strategies to better explore the search space of neural network architectures. One such speciation strategy is the use of islands, which are also popular in improving the performance of distributed ...
Comments
Information & Contributors
Information
Published In
July 2024
2187 pages
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Italian government
Conference
GECCO '24 Companion
Sponsor:
GECCO '24 Companion: Genetic and Evolutionary Computation Conference Companion
July 14 - 18, 2024
VIC, Melbourne, Australia
Acceptance Rates
Overall Acceptance Rate 1,669 of 4,410 submissions, 38%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 56Total Downloads
- Downloads (Last 12 months)56
- Downloads (Last 6 weeks)8
Reflects downloads up to 23 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in