research-article

Replicable Self-Documenting Experiments with Arbitrary Search Spaces and Algorithms

Authors:

Zhize WuAuthors Info & Claims

GECCO '23 Companion: Proceedings of the Companion Conference on Genetic and Evolutionary Computation

Pages 1891 - 1899

https://doi.org/10.1145/3583133.3596306

Published: 24 July 2023 Publication History

Abstract

We introduce moptipy, a toolbox for implementing, experimenting with, and applying optimization algorithms. It features mechanisms for executing fully reproducible experiments. Our seeding procedure for random number generators makes our experiments deterministic. Our system creates self-documenting log files that store the algorithm setup, the system configuration, the random seed, the final solutions, and the progress of the optimization algorithm over time. The parallelization and distribution of experiments works on most operating systems and requires no additional synchronization, inter-process-communication, or libraries. moptipy supports both single- and multi-objective optimization with arbitrary search spaces. Time measurements and computational budgets can be based both on wall clock time and objective function evaluations. moptipy is also an educational platform with comprehensive documentation and an accompanying free electronic book introducing the basic concepts of metaheuristics, discussing the implemented algorithms, and showing their performance on basis of actual experimental results.

References

[1]

ACM. 2020. Artifact Review and Badging, Version 1.1. Retrieved 2023-01-21 from https://www.acm.org/publications/policies/artifact-review-and-badging-current

[2]

John Edward Beasley. 1990. OR-Library: Distributing Test Problems by Electronic Mail. The Journal of the Operational Research Society 41 (1990), 1069--1072.

[3]

Mark S. Boddy and Thomas L. Dean. 1989. Solving Time-Dependent Planning Problems. Technical Report CS-89-03. Brown University, Department of Computer Science, Providence, RI, USA. Retrieved 2023-01-21 from ftp://ftp.cs.brown.edu/pub/techreports/89/cs89-03.pdf

[4]

NumPy Community. 2022. Permuted Congruential Generator (64-bit, PCG64). In NumPy Reference, Release 1.23.0. NumFOCUS, Inc., Austin, TX, USA. Retrieved 2023-01-21 from https://numpy.org/doc/1.23/numpy-ref.pdf

[5]

Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. 2000. A Fast Elitist Non-Dominated Sorting Genetic Algorithm for Multi-Objective Optimization: NSGA-II. In Intl. Conf. on Parallel Problem Solving from Nature (PPSN VI), Sept. 18--20, 2000, Paris, France (LNCS, Vol. 1917). Springer, Berlin, Heidelberg, 849--858.

[6]

Carola Doerr, Hao Wang, Furong Ye, Sander van Rijn, and Thomas Bäck. 2018. IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics. Cornell University, New York, NY, USA. Retrieved 2023-01-21 from https://arxiv.org/pdf/1810.05281.pdf arXiv:1810.05281v1 [cs.NE] 11 Oct 2018.

[7]

Carola Doerr, Furong Ye, Naama Horesh, Hao Wang, Ofer M. Shir, and Thomas Bäck. 2020. Benchmarking Discrete Optimization Heuristics with IOHprofiler. Applied Soft Computing 88, 106027 (2020), 1--21.

Digital Library

[8]

Marco Dorigo and Thomas Stützle. 2004. Ant Colony Optimization. MIT Press, Cambridge, MA, USA.

[9]

Ingy döt Net, Tina Müller, Pantelis Antoniou, Eemeli Aro, and Thomas Smith. 2021. YAML Ain't Markup Language (YAML®) version 1.2, Revision 1.2.2 (2021-10-01). YAML Language Development Team, Seattle, WA, USA. Retrieved 2023-01-21 from https://yaml.org/spec/1.2.2/

[10]

Johann Dreo, Arnaud Liefooghe, Sébastien Verel, Marc Schoenauer, Juan J. Merelo, Alexandre Quemy, Benjamin Bouvier, and Jan Gmys. 2021. Paradiseo: From a Modular Framework for Evolutionary Computation to the Automated Design of Metaheuristics: 22 Years of Paradiseo. In Genetic and Evolutionary Computation Conf. Companion, July 10--14, 2021, Lille, France, Francisco Chicano (Ed.). ACM, New York, NY, USA, 1522--1530.

Digital Library

[11]

Olive Jean Dunn. 1961. Multiple Comparisons Among Means. J. Amer. Statist. Assoc. 56, 293 (1961), 52--64.

[12]

David Edward Goldberg. 1989. Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley Longman, Boston, MA, USA.

Digital Library

[13]

Odd Erik Gundersen, Yolanda Gil, and David W. Aha. 2018. On Reproducible AI: Towards Reproducible Research, Open Science, and Digital Scholarship in AI Publications. AI Magazine 39, 3 (2018), 56--68.

Digital Library

[14]

Nikolaus Hansen, Anne Auger, Steffen Finck, and Raymond Ros. 2010. Real-Parameter Black-Box Optimization Benchmarking 2010: Experimental Setup. Rapports de Recherche RR-7215. Institut National de Recherche en Informatique et en Automatique (INRIA). Retrieved 2023-01-21 from http://hal.inria.fr/inria-00462481 inria-00462481.

[15]

Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. 2021. COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting. Optimization Methods and Software 36 (2021), 114--144. Issue 1. also arXiv:1603.08785v4 [cs.AI] 9 Sep 2020.

[16]

Nikolaus Hansen and Andreas Ostermeier. 2001. A Completely Derandomized Self-Adaptation in Evolution Strategies. Evolutionary Computation 9, 2 (2001), 159--195.

Digital Library

[17]

Nikolaus Hansen, Tea Tušar, Olaf Mersmann, Anne Auger, and Dimo Brockhoff. 2016. COCO: The Experimental Procedure. Cornell University, New York, NY, USA. Retrieved 2023-01-21 from https://arxiv.org/abs/1603.08776 arXiv:1603.08776v2 [cs.AI] 19 May 2016.

[18]

Myles Hollander and Douglas Alan Wolfe. 1973. Nonparametric Statistical Methods. John Wiley and Sons Ltd., New York, NY, USA.

[19]

Holger H. Hoos and Thomas Stützle. 1998. Evaluating Las Vegas Algorithms - Pitfalls and Remedies. In 14^th Conf. on Uncertainty in Artificial Intelligence (UAI'98), July 24--26, 1998, Madison, WI, USA, Gregory F. Cooper and Serafín Moral (Eds.). Morgan Kaufmann, San Francisco, CA, USA, 238--245.

[20]

Holger H. Hoos and Thomas Stützle. 2005. Stochastic Local Search: Foundations and Applications. Elsevier, Amsterdam, The Netherlands.

[21]

IEEE and The Open Group. 2018. open, openat - open file. In The Open Group Base Specifications Issue 7, 2018 Edition, IEEE Std 1003.1-2017 (Revision of IEEE Std 1003.1-2008). IEEE and The Open Group, Burlington, MA, USA. https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html

[22]

Yan Jiang, Thomas Weise, Jörg Lässig, Raymond Chiong, and Rukshan Athauda. 2014. Comparing a Hybrid Branch and Bound Algorithm with Evolutionary Computation Methods, Local Search and their Hybrids on the TSP. In IEEE Symposium on Computational Intelligence in Production and Logistics Systems (CIPLS'14), part of the IEEE Symposium Series on Computational Intelligence (SSCI'14), Dec. 9--12, 2014, Orlando, FL, USA. IEEE Computer Society Press, Los Alamitos, CA, USA, 148--155.

[23]

Tianyu Liang, Zhize Wu, Jörg Lässig, Daan van den Berg, and Thomas Weise. 2022. Solving the Traveling Salesperson Problem using Frequency Fitness Assignment. In IEEE Symposium on Foundations of Computational Intelligence. IEEE, Los Alamitos, USA, 360--367.

[24]

Laurent Meunier, Herilalaina Rakotoarison, Pak Kan Wong, Baptiste Roziere, Jeremy Rapin, Olivier Teytaud, Antoine Moreau, and Carola Doerr. 2021. Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking. Cornell University, New York, NY, USA. Retrieved 2023-01-21 from https://arxiv.org/abs/2010.04542v3 arXiv:2010.04542v3 [cs.LG] 23 Feb 2021.

[25]

Heinz Mühlenbein. 1992. How Genetic Algorithms Really Work: Mutation and Hillclimbing. In Proceedings of Parallel Problem Solving from Nature 2 (PPSN-II), Sept. 28--30, 1992, Brussels, Belgium. Elsevier, Amsterdam, The Netherlands, 15--26.

[26]

Melissa E. O'Neill. 2014. PCG: A Family of Simple Fast Space-Efficient Statistically Good Algorithms for Random Number Generation. Technical Report HMC-CS-2014-0905. Harvey Mudd College, Computer Science Department, Claremont, CA, USA. Retrieved 2023-01-21 from https://www.cs.hmc.edu/tr/hmc-cs-2014-0905.pdf

[27]

Michael James David Powell. 2009. The BOBYQA Algorithm for Bound Constrained Optimization without Derivatives. Technical Report DAMTP 2009/NA06. Department of Applied Mathematics and Theoretical Physics, Cambridge University, Cambridge, UK. Retrieved 2023-01-21 from https://www.damtp.cam.ac.uk/user/na/NA_papers/NA2009_06.pdf

[28]

Penny Pritzker and Willie E. May (Eds.). 2015. Secure Hash Standard (SHS). Federal Information Processing Standards Publication, Vol. FIPS PUB 180-4. National Institute of Standards and Technology, Information Technology Laboratory, Gaithersburg, MD, USA.

[29]

Jeremy Rapin and Olivier Teytaud. 2018. Nevergrad - A Gradient-Free Optimization Platform. GitHub, San Francisco, CA, USA. Retrieved 2023-01-21 from https://gitHub.com/FacebookResearch/Nevergrad

[30]

Keivan Tafakkori. 2023. List of optimization packages in Python. GitHub, San Francisco, CA, USA. Retrieved on 2023-01-15 from https://ktafakkori.github.io/optimization-packages-in-python-list/

[31]

Dave Andrew Douglas Tompkins and Holger H. Hoos. 2004. UBCSAT: An Implementation and Experimentation Environment for SLS Algorithms for SAT and MAX-SAT. In Revised Selected Papers from the Seventh Intl. Conf. on Theory and Applications of Satisfiability Testing (SAT'04), May 10--13, 2004, Vancouver, BC, Canada (LNCS, Vol. 3542). Springer-Verlag GmbH, Berlin, Germany, 306--320.

Digital Library

[32]

Unicode Consortium (Ed.). 2022. The Unicode® Standard, Version 15.0 - Core Specification. Unicode, Inc., Mountain View, CA, USA. Retrieved 2023-01-21 from https://www.unicode.org/versions/Unicode15.0.0/

[33]

Pauli Virtanen et al. and SciPy 1.0 Contributors. 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods 17 (2020), 261--272.

[34]

Hao Wang, Diederick Vermetten, Furong Ye, Carola Doerr, and Thomas Bäck. 2022. IOHanalyzer: Detailed Performance Analyses for Iterative Optimization Heuristics. ACM Transactions on Evolutionary Learning and Optimization 2 (2022), 1--29. Issue 1[3].

Digital Library

[35]

Thomas Weise. 2009. Global Optimization Algorithms - Theory and Application. Institute of Applied Optimization, Hefei University, Hefei, Anhui, China. Retrieved 2023-01-21 from http://iao.hfuu.edu.cn/images/publications/W2009GOEB.pdf

[36]

Thomas Weise. 2017. From Standardized Data Formats to Standardized Tools for Optimization Algorithm Benchmarking. In 16^th IEEE Conf. on Cognitive Informatics & Cognitive Computing (ICCI*CC'17), July 26--28, 2017, Oxford, UK. IEEE Computer Society Press, Los Alamitos, CA, USA, 490--497.

[37]

Thomas Weise. 2020--2023. Optimization Algorithms. Institute of Applied Optimization, School of Artificial Intelligence and Big Data, Hefei University, Hefei, Anhui, China. Retrieved 2023-01-21 from https://thomasweise.github.io/oa

[38]

Thomas Weise, Yan Chen, Xinlu Li, and Zhize Wu. 2020. Selecting a diverse set of benchmark instances from a tunable model problem for black-box discrete optimization algorithms. Applied Soft Computing 92 (2020), 106269.

[39]

Thomas Weise, Raymond Chiong, Ke Tang, Jörg Lässig, Shigeyoshi Tsutsui, Wenxiang Chen, Zbigniew Michalewicz, and Xin Yao. 2014. Benchmarking Optimization Algorithms: An Open Source Framework for the Traveling Salesman Problem. IEEE Computational Intelligence Magazine 9, 3 (2014), 40--52.

Digital Library

[40]

Thomas Weise, Li Niu, and Ke Tang. 2010. AOAB - Automated Optimization Algorithm Benchmarking. In 12^th Annual Conf. Companion on Genetic and Evolutionary Computation (GECCO'10), July 7--11, 2010, Portland, OR, USA. ACM Press, New York, NY, USA, 1479--1486.

Digital Library

[41]

Thomas Weise, Xiaofeng Wang, Qi Qi, Bin Li, and Ke Tang. 2018. Automatically discovering clusters of algorithm and problem instance behaviors as well as their causes from experimental data, algorithm setups, and instance features. Applied Soft Computing 73 (2018), 366--382.

[42]

Thomas Weise, Yuezhong Wu, Raymond Chiong, Ke Tang, and Jörg Lässig. 2016. Global versus Local Search: The Impact of Population Sizes on Evolutionary Algorithm Performance. Journal of Global Optimization 66 (2016), 511--534. Issue 3.

Digital Library

[43]

Weixiong Zhang. 1999. Truncated and Anytime Depth-First Branch and Bound: A Case Study on the Asymmetric Traveling Salesman Problem. In AAAI Spring Symposium Series: Search Techniques for Problem Solving Under Uncertainty and Incomplete Information (AAAI Technical Report, Vol. SS-99-07). AAAI, Menlo Park, CA, USA, 148--155.

Cited By

Acharya DKuppan KDivya B(2025)Agentic AI: Autonomous Intelligence for Complex Goals—A Comprehensive SurveyIEEE Access10.1109/ACCESS.2025.353285313(18912-18936)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2025.3532853
Weise T(2023)Software - motipy: the Metaheuristic Optimization in Python LibraryACM SIGEVOlution10.1145/3638461.363846416:4(1-2)Online publication date: 21-Dec-2023
https://dl.acm.org/doi/10.1145/3638461.3638464

Index Terms

Replicable Self-Documenting Experiments with Arbitrary Search Spaces and Algorithms

Recommendations

Reproducible Online Search Experiments
Advances in Information Retrieval
Abstract
In the empirical sciences, the evidence is commonly manifested by experimental results. However, very often, these findings are not reproducible, hindering scientific progress. Innovations in the field of information retrieval (IR) are mainly ...
MLXP: A framework for conducting replicable experiments in Python
ACM REP '24: Proceedings of the 2nd ACM Conference on Reproducibility and Replicability

Replicability in machine learning (ML) research is increasingly concerning due to the utilization of complex non-deterministic algorithms and the dependence on numerous hyper-parameter choices, such as model architecture and training datasets. Ensuring ...
Search bias in ant colony optimization: on the role of competition-balanced systems

One of the problems encountered when applying ant colony optimization (ACO) to combinatorial optimization problems is that the search process is sometimes biased by algorithm features such as the pheromone model and the solution construction process. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '23 Companion: Proceedings of the Companion Conference on Genetic and Evolutionary Computation

July 2023

2519 pages

ISBN:9798400701207

DOI:10.1145/3583133

Chair:
Sara Silva,
Program Chair:
Luís Paquete

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

University Natural Sciences Research Project of Anhui Province
National Natural Science Foundation of China
Key Research Plan of Anhui
Hefei Specially Recruited Foreign Expert program
Hefei Foreign Expert Office program
Youth Project of the Provincial Natural Science Foundation of Anhui

Conference

GECCO '23 Companion

Sponsor:

SIGEVO

GECCO '23 Companion: Companion Conference on Genetic and Evolutionary Computation

July 15 - 19, 2023

Lisbon, Portugal

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
27
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Acharya DKuppan KDivya B(2025)Agentic AI: Autonomous Intelligence for Complex Goals—A Comprehensive SurveyIEEE Access10.1109/ACCESS.2025.353285313(18912-18936)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2025.3532853
Weise T(2023)Software - motipy: the Metaheuristic Optimization in Python LibraryACM SIGEVOlution10.1145/3638461.363846416:4(1-2)Online publication date: 21-Dec-2023
https://dl.acm.org/doi/10.1145/3638461.3638464

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten