research-article

On the performance of the Bayesian optimization algorithm with combined scenarios of search algorithms and scoring metrics

Authors:

Ciniro A. L. Nametala,

Wandry R. Faria,

Benvindo R. Pereira JúniorAuthors Info & Claims

Genetic Programming and Evolvable Machines, Volume 23, Issue 2

Pages 193 - 223

https://doi.org/10.1007/s10710-022-09430-2

Published: 01 June 2022 Publication History

Abstract

The Bayesian Optimization Algorithm (BOA) is one of the most prominent Estimation of Distribution Algorithms. It can detect the correlation between multiple variables and extract knowledge on regular patterns in solutions. Bayesian Networks (BNs) are used in BOA to represent the probability distributions of the best individuals. The BN’s construction is challenging since there is a trade-off between acuity and computational cost to generate it. This trade-off is determined by combining a search algorithm (SA) and a scoring metric (SM). The SA is responsible for generating a promising BN and the SM assesses the quality of such networks. Some studies have already analyzed how this relationship affects the learning process of a BN. However, such investigation had not yet been performed to determine the bond linking the selection of SA and SM and the BOA’s output quality. Acting on this research gap, a detailed comparative analysis involving two constructive heuristics and four scoring metrics is presented in this work. The classic version of BOA was applied to discrete and continuous optimization problems using binary and floating-point representations. The scenarios were compared through graphical analyses, statistical metrics, and difference detection tests. The results showed that the selection of SA and SM affects the quality of the BOA results since scoring metrics that penalize complex BN models perform better than metrics that do not consider the complexity of the networks. This study contributes to a discussion on this metaheuristic’s practical use, assisting users with implementation decisions.

References

[1]

Goldberg DE and Holland JH Genetic algorithms and machine learning Mach. Learn. 1988 3 2–3 95-99

[2]

Gaspar Cunha A, Takahashi R, and Antunes CH Manual de computação evolutiva e metaheurística 2012 Coimbra Imprensa da Universidade de Coimbra

[3]

Zhang Y, Wang S, and Ji G A comprehensive survey on particle swarm optimization algorithm and its applications Math. Probl. Eng. 2015 2015 1

[4]

Mahdavi S, Shiri ME, and Rahnamayan S Metaheuristics in large-scale global continues optimization: a survey Inf. Sci. 2015 295 407-428

[5]

Pelikan M, Goldberg DE, and Cantu-Paz E Linkage problem, distribution estimation, and bayesian networks Evol. Comput. 2000 8 3 311-340

[6]

Tanev I Genetic programming incorporating biased mutation for evolution and adaptation of snakebot Genet. Programm. Evolvable Mach. 2007 8 1 39-59

[7]

Mühlenbein H The equation for response to selection and its use for prediction Evol. Comput. 1997 5 3 303-346

[8]

S. Baluja, Population-based incremental learning: a method for integrating genetic search based function optimization and competitive learning. Technical report, Carnegie Mellon University (1994).

[9]

Smith J On appropriate adaptation levels for the learning of gene linkage Genet. Program Evolvable Mach. 2002 3 2 129-155

[10]

Przewozniczek MW and Komarnicki MM Empirical linkage learning IEEE Trans. Evol. Comput. 2020 24 6 1097-1111

[11]

G.R. Harik, F.G. Lobo, K. Sastry, Linkage learning via probabilistic modeling in the extended compact genetic algorithm (ECGA), in Scalable Optimization via Probabilistic Modeling (Springer, Berlin, Heidelberg, 2006), pp. 39–61.

[12]

Goldberg DE The Design of Innovation: Lessons from and for Competent Genetic Algorithms 2013 Berlin Springer

[13]

M. Pelikan, D.E. Goldberg, E. Cantú-Paz, et al., Boa: the bayesian optimization algorithm, in Proceedings of the Genetic and Evolutionary Computation Conference GECCO-99, vol. 1 (1999), pp. 525–532.

[14]

M.K. Crocomo, Algoritmo de otimizaçao bayesiano com detecçao de comunidades. PhD thesis, Universidade de São Paulo (2012).

[15]

Torun HM, Swaminathan M, Davis AK, and Bellaredj MLF A global bayesian optimization algorithm and its application to integrated system design IEEE Trans. Very Large Scale Integr. VLSI Syst. 2018 26 4 792-802

[16]

He F, Zhou J, Feng Z, Liu G, and Yang Y A hybrid short-term load forecasting model based on variational mode decomposition and long short-term memory networks considering relevant factors with bayesian optimization algorithm Appl. Energy 2019 237 103-116

[17]

R. Ding, W. Zhou, H. Cheng, A novel hybrid model of wind speed forecasting based on EWT, BiLSTM, SVR optimized by BOA in inner Mongolia, China, in Lecture Notes in Electrical Engineering (Springer, Singapore, 2019), pp. 183–191.

[18]

B. Huang, Q. Ding, G. Sun, H. Li, Stock prediction based on bayesian-lstm, in Proceedings of the 2018 10th International Conference on Machine Learning and Computing (2018), pp. 128–133.

[19]

Tanaka R and Iwata H Bayesian optimization for genomic selection: a method for discovering the best genotype among a large number of candidates Theor. Appl. Genet. 2017 131 1 93-105

[20]

Chan L, Hutchison GR, and Morris GM BOKEI: bayesian optimization using knowledge of correlated torsions and expected improvement for conformer generation Phys. Chem. Chem. Phys. 2020 22 9 5211-5219

[21]

C.W. Ahn, R.S. Ramakrishna, D.E. Goldberg, Real-coded bayesian optimization algorithm: bringing the strength of boa into the continuous world, in Genetic and Evolutionary Computation Conference (Springer, 2004), pp. 840–851.

[22]

M. Pelikan, D.E. Goldberg, Hierarchical bayesian optimization algorithm, in Scalable Optimization via Probabilistic Modeling (Springer, 2006), pp. 63–90.

[23]

J. Očenášek, J. Schwarz, The parallel bayesian optimization algorithm, in The State of the Art in Computational Intelligence (Springer, 2000), pp. 61–67.

[24]

N. Khan, D.E. Goldberg, M. Pelikan, Multi-objective bayesian optimization algorithm, in Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation (Citeseer, 2002), pp. 684–684

[25]

Scanagatta M, Salmerón A, and Stella F A survey on bayesian network structure learning from data Prog. Artif. Intell. 2019 8 4 425-439

[26]

Tsamardinos I, Brown LE, and Aliferis CF The max-min hill-climbing bayesian network structure learning algorithm Mach. Learn. 2006 65 1 31-78

[27]

Scutari M, Graafland CE, and Gutiérrez JM Who learns better bayesian network structures: Accuracy and speed of structure learning algorithms Int. J. Approx. Reason. 2019 115 235-253

[28]

M. Scutari, An empirical-Bayes score for discrete Bayesian networks, in Proceedings of the Eighth International Conference on Probabilistic Graphical Models ed. by A. Antonucci, G. Corani, C.P. Campos (2016), pp. 438–448

[29]

Beretta S, Castelli M, Gonçalves I, Henriques R, and Ramazzotti D Learning the structure of bayesian networks: a quantitative assessment of the effect of different algorithmic schemes Complexity 2018

[30]

Pelikan M, Goldberg DE, and Lobo FG A survey of optimization by building and using probabilistic models Comput. Optim. Appl. 2002 21 1 5-20

[31]

M. Pelikan, D.E Goldberg, K. Sastry, et al., Bayesian optimization algorithm, decision graphs, and occam’s razor, in Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001), vol. 519526 (2001).

[32]

H. Karshenas, A. Nikanjam, B.H. Helmi, A.T. Rahmani, Combinatorial effects of local structures and scoring metrics in bayesian optimization algorithm, in Proceedings of the first ACM/SIGEVO Summit on Genetic and Evolutionary Computation - GEC ’09 (ACM Press, 2009).

[33]

Gheisari S and Meybodi MR BNC-PSO: structure learning of bayesian networks by particle swarm optimization Inf. Sci. 2016 348 272-289

[34]

A.H.M. Soares, Algoritmos de estimação de distribuição baseados em árvores filogenéticas. PhD thesis, Universidade de São Paulo (2014).

[35]

J. Martins, Analysis of Linkage Learning in Evolutionary Optimization. PhD thesis, Universidade de São Paulo, 05 (2015).

[36]

Russell S Artificial Intelligence : A Modern Approach 2010 3 Upper Saddle River Prentice Hall 0136042597

[37]

Glover F Heuristics for integer programming using surrogate constraints Decis. Sci. 1977 8 1 156-166

[38]

K.B. Korb, A.E. Nicholson, Bayesian Artificial Intelligence (CRC Press, Boca Raton, 2010). ISBN 9781439815915

[39]

Akaike H A new look at the statistical model identification IEEE Trans. Autom. Control 1974 19 6 716-723

[40]

David Maxwell Chickering Learning equivalence classes of bayesian-network structures J. Mach. Learn. Res. 2002 2 445-498

[41]

C. Echegoyen, J.A. Lozano, R. Santana, P. Larranaga, Exact bayesian network learning in estimation of distribution algorithms, in 2007 IEEE Congress on Evolutionary Computation (IEEE, 2007).

[42]

Scutari M Learning bayesian networks with the bnlearn r package J. Stat. Softw. 2010 35 3 1

[43]

A. Ankan, A. Panda, pgmpy: probabilistic graphical models using python, in Proceedings of the 14th Python in Science Conference (SCIPY 2015) (Citeseer, 2015).

[44]

Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, and Witten IH The WEKA data mining software ACM SIGKDD Explor. Newsl. 2009 11 1 10

[45]

Y. Lavinas, C. Aranha, T. Sakurai, M. Ladeira, Experimental analysis of the tournament size on genetic algorithms, in 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (2018). pp. 3647–3653.

[46]

Wang Y, Chen W, and Tellambura C Genetic algorithm based nearly optimal peak reduction tone set selection for adaptive amplitude clipping papr reduction IEEE Trans. Broadcast. 2012 58 3 462-471

[47]

Zhang H, Liu F, Zhou Y, and Zhang Z A hybrid method integrating an elite genetic algorithm with tabu search for the quadratic assignment problem Inf. Sci. 2020 539 347-374

[48]

Cooper GF and Herskovits E A bayesian method for the induction of probabilistic networks from data Mach. Learn. 1992 9 4 309-347

[49]

Heckerman D, Geiger D, and Chickering DM Learning bayesian networks: the combination of knowledge and statistical data Mach. Learn. 1995 20 3 197-243

[50]

C. Aparecido L. Nametala, W.R. Faria, B.R. Pereira Júnior, On the Performance of the Bayesian Optimization Algorithm with Combined Scenarios of Search Algorithms and Scoring Metrics: R Source Code and Experiment Data (2021).

[51]

Royston P Remark AS r94: a remark on algorithm AS 181: the w-test for normality Appl. Stat. 1995 44 4 547

[52]

Montgomery DC and Runger GC Applied Statistics and Probability for Engineers 2010 Hoboken Wiley

[53]

Hollander M, Chicken E, and Wolfe D Nonparametric Statistical Methods 2013 Hoboken Wiley 0470387378

[54]

Doerr C, Ye F, Horesh N, Wang H, Shir OM, and Back T Benchmarking discrete optimization heuristics with iohprofiler Appl. Soft Comput. 2020 88

[55]

Harik GR, Lobo FG, and Goldberg DE The compact genetic algorithm IEEE Trans. Evol. Comput. 1999 3 4 287-297

[56]

X. Li, K. Deb, Comparing lbest pso niching algorithms using different position update rules, in IEEE Congress on Evolutionary Computation (2010), pp. 1–8.

[57]

M. Kronfeld, A. Zell, Towards scalability in niching methods, in IEEE Congress on Evolutionary Computation (2010), pp. 1–8.

[58]

Soares A, Râbelo R, and Delbem A Optimization based on phylogram analysis Expert Syst. Appl. 2017 78 32-50

[59]

D.E Goldberg, A design approach to problem difficulty, in The Design of Innovation (Springer, 2002), pp. 71–100

[60]

Qian C, Bian C, Jiang W, and Tang K Running time analysis of the (1+1)-ea for onemax and leadingones under bit-wise noise Algorithmica 2019 81 2 749-795

[61]

N. Buskulic, C. Doerr, Maximizing drift is not optimal for solving onemax, in Proceedings of the Genetic and Evolutionary Computation Conference Companion (2019), pp. 425–426.

[62]

S. Strasser, J.W Sheppard, Evaluating factored evolutionary algorithm performance on binary deceptive functions, in 2017 IEEE Symposium Series on Computational Intelligence (SSCI) (IEEE, 2017), pp. 1–8.

[63]

Tinós R and Yang S A self-organizing random immigrants genetic algorithm for dynamic optimization problems Genet. Program. Evolvable Mach. 2007 8 3 255-286

[64]

Shakya S, Santana R, and Lozano JA A markovianity based optimisation algorithm Genet. Program. Evolvable Mach. 2012 13 2 159-195

[65]

Complex Systems Design Lab (CSDL), Comparing Continuous Optimizers (coco) (CSDL, 2021). https://coco.gforge.inria.fr

[66]

GECCO, Gecco: Genetic and Evolutionary Computation Conference (GECCO, 2021). https://dl.acm.org/conference/gecco

[67]

Hellwig M and Beyer H-G Benchmarking evolutionary algorithms for single objective real-valued constrained optimization—a critical review Swarm Evol. Comput. 2019 44 927-944

Cited By

Yuan XLi D(2022)An Intelligent Recognition and Diagnosis System for English in Radio Land-Air Calls Based on Optimization AlgorithmProceedings of the 7th International Conference on Cyber Security and Information Engineering10.1145/3558819.3563735(154-159)Online publication date: 23-Sep-2022
https://dl.acm.org/doi/10.1145/3558819.3563735

Index Terms

On the performance of the Bayesian optimization algorithm with combined scenarios of search algorithms and scoring metrics

Index terms have been assigned to the content through auto-classification.

Recommendations

Combinatorial effects of local structures and scoring metrics in bayesian optimization algorithm
GEC '09: Proceedings of the first ACM/SIGEVO Summit on Genetic and Evolutionary Computation

Bayesian Optimization Algorithm (BOA) has been used with different local structures to represent more complex models and a variety of scoring metrics to evaluate Bayesian network. But the combinatorial effects of these elements on the performance of BOA ...
Biasing Bayesian optimization algorithm using case based reasoning

Studies show that application of the prior knowledge in biasing the Estimation of Distribution Algorithms (EDAs), such as Bayesian Optimization Algorithm (BOA), increases the efficiency of these algorithms significantly. One of the main advantages of ...
A discrete gravitational search algorithm for solving combinatorial optimization problems

Metaheuristics are general search strategies that, at the exploitation stage, intensively exploit areas of the solution space with high quality solutions and, at the exploration stage, move to unexplored areas of the solution space when necessary. The ...

Comments

Information & Contributors

Information

Published In

cover image Genetic Programming and Evolvable Machines

Genetic Programming and Evolvable Machines Volume 23, Issue 2

Jun 2022

143 pages

ISSN:1389-2576

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 June 2022

Accepted: 07 March 2022

Revision received: 24 February 2022

Received: 21 July 2020

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yuan XLi D(2022)An Intelligent Recognition and Diagnosis System for English in Radio Land-Air Calls Based on Optimization AlgorithmProceedings of the 7th International Conference on Cyber Security and Information Engineering10.1145/3558819.3563735(154-159)Online publication date: 23-Sep-2022
https://dl.acm.org/doi/10.1145/3558819.3563735

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents