research-article

Ranking and Selection: A New Sequential Bayesian Procedure for Use with Common Random Numbers

Authors:

Bjürn Görder,

Michael KolonkoAuthors Info & Claims

ACM Transactions on Modeling and Computer Simulation (TOMACS), Volume 29, Issue 1

Article No.: 2, Pages 1 - 24

https://doi.org/10.1145/3241042

Published: 24 January 2019 Publication History

Abstract

We introduce a new sampling scheme for selecting the best alternative out of a given set of systems that are evaluated with respect to their expected performances. We assume that the systems are simulated on a computer and that a joint observation of all systems has a multivariate normal distribution with unknown mean and unknown covariance matrix. In particular, the observations of the systems may be stochastically dependent as is the case if common random numbers are used for simulation.

In each iteration of the algorithm, we allocate a fixed budget of simulation runs to the alternatives. We use a Bayesian set-up with a noninformative prior distribution and derive a new closed-form approximation for the posterior distributions that allows provision of a lower bound for the posterior probability of a correct selection (PCS). Iterations are continued until this lower bound is greater than 1−α for a given α. We also introduce a new allocation strategy that allocates the available budget according to posterior error probabilities. Our procedure needs no additional prior parameters and can cope with different types of ranking and selection tasks.

Our numerical experiments show that our strategy is superior to other procedures from the literature, namely, KN++ and Pluck. In all of our test scenarios, these procedures needed more observation and/or had an empirical PCS below the required 1−α. Our procedure always had its empirical PCS above 1−α, underlining the practicability of our approximation of the posterior distribution.

References

[1]

Jürgen Branke, Stephen E. Chick, and Christian Schmidt. 2007. Selecting a selection procedure. Management Science 53, 12 (2007), 1916--1932.

Digital Library

[2]

Chun-Hung Chen, Hsiao-Chang Chen, and Liyi Dai. 1996. A gradient approach for smartly allocating computing budget for discrete event simulation. In Proceedings of the 1996 Winter Simulation Conference, John M. Charnes, Douglas J. Morrice, Daniel T. Brunner, and James J. Swain (Eds.). IEEE, 398--405.

Digital Library

[3]

Chun-Hung Chen, Donghai He, Michael Fu, and Loo Hay Lee. 2008. Efficient simulation budget allocation for selecting an optimal subset. INFORMS Journal on Computing 20, 4 (2008), 579--595.

[4]

Chun-Hung Chen and Loo Hay Lee. 2010. Stochastic Simulation Optimization: An Optimal Computing Budget Allocation. World Scientific Publishing Company.

Digital Library

[5]

Chun-Hung Chen, Jianwu Lin, Enver Yücesan, and Stephen E. Chick. 2000. Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Discrete Event Dynamic Systems 10, 3 (2000), 251--270.

Digital Library

[6]

Chun-Hung Chen and Enver Yucesan. 2005. An alternative simulation budget allocation scheme for efficient simulation. International Journal of Simulation and Process Modelling 1, 1 (2005), 49--57.

[7]

Hsiao-Chang Chen, Liyi Dai, Chun-Hung Chen, and Enver Yücesan. 1997. New development of optimal computing budget allocation for discrete event simulation. In Proceedings of the 1997 Winter Simulation Conference, Sigrun Andradottir, Kevin J. Healy, David H. Withers, and Barry L. Nelson (Eds.). IEEE, 334--341.

Digital Library

[8]

Stephen E. Chick and Koichiro Inoue. 2001a. New procedures to select the best simulated system using common random numbers. Management Science 47, 8 (2001), 1133--1149.

Digital Library

[9]

Stephen E. Chick and Koichiro Inoue. 2001b. New two-stage and sequential procedures for selecting the best simulated system. Operations Research 49, 5 (2001), 732--743.

Digital Library

[10]

Morris H. DeGroot. 2004. Optimal Statistical Decisions. John Wiley 8 Sons.

[11]

Francesca Dominici, Giovanni Parmigiani, and Merlise Clyde. 2000. Conjugate analysis of multivariate normal data with incomplete observations. Canadian Journal of Statistics 28, 3 (2000), 533--550.

[12]

Marco Dorigo and Thomas Stützle. 2010. Ant colony optimization: Overview and recent advances. In Handbook of Metaheuristics. Springer, 227--263.

[13]

Richard L. Dykstra. 1970. Establishing the positive definiteness of the sample covariance matrix. The Annals of Mathematical Statistics 41, 6 (1970), 2153--2154.

[14]

Peter I. Frazier, Jing Xie, and Stephen E. Chick. 2011. Value of information methods for pairwise sampling with correlations. In Proceedings of the Winter Simulation Conference. Winter Simulation Conference, 3979--3991.

Digital Library

[15]

Michael C. Fu, Jian-Qiang Hu, Chun-Hung Chen, and Xiaoping Xiong. 2007. Simulation allocation for determining the best design in the presence of correlated sampling. INFORMS Journal on Computing 19, 1 (2007), 101--111.

Digital Library

[16]

Siyang Gao, Hui Xiao, Enlu Zhou, and Weiwei Chen. 2016. Optimal computing budget allocation with input uncertainty. In Proceedings of the 2016 Winter Simulation Conference. IEEE Press, 839--846.

Digital Library

[17]

Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. 2004. Bayesian Data Analysis (2nd ed.). Chapman 8 Hall/CRC.

[18]

Paul Glasserman and David D. Yao. 1992. Some guidelines and guarantees for common random numbers. Management Science 38, 6 (1992), 884--908.

Digital Library

[19]

Peter Glynn and Sandeep Juneja. 2004. A large deviations perspective on ordinal optimization. In Proceedings of the 2004 Winter Simulation Conference, Ricki G. Ingalls, Manuel D. Rossetti, Jeffrey S. Smith, and Brett A. Peters (Eds.). Vol. 1. IEEE, 577--585.

Digital Library

[20]

Peter Glynn and Sandeep Juneja. 2015. Ordinal optimization-empirical large deviations rate estimators, and stochastic multi-armed bandits. arXiv preprint arXiv:1507.04564 (2015).

[21]

Björn Görder. 2012. Simulationsbasierte Optimierung mit statistischen Ranking- und Selektionsverfahren. Ph.D. Dissertation (in German). TU Clausthal, Lower Saxony, Germany.

[22]

S.-H. Kim and B. L. Nelson. 2006a. Selecting the best system. In Simulation Handbooks in Operations Research and Management Science, S. G. Henderson and B. L. Nelson (Eds.). Elsevier, 501--532.

[23]

Seong-Hee Kim and Barry L. Nelson. 2006b. On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Operations Research 54, 3 (2006), 475--488.

Digital Library

[24]

Soonhui Lee and Barry L. Nelson. 2014. Bootstrap ranking 8 selection revisited. In Proceedings of the 2014 Winter Simulation Conference. IEEE Press, 3857--3868.

Digital Library

[25]

Soonhui Lee and Barry L. Nelson. 2016. General-purpose ranking and selection for computer simulation. IIE Transactions 48, 6 (2016), 555--564.

[26]

Jun Luo, L. Jeff Hong, Barry L. Nelson, and Yang Wu. 2015. Fully sequential procedures for large-scale ranking-and-selection problems in parallel computing environments. Operations Research 63, 5 (2015), 1177--1194.

Digital Library

[27]

Barry L. Nelson and Frank J. Matejcik. 1995. Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Management Science 41, 12 (1995), 1935--1945.

[28]

Eric C. Ni, Shane G. Henderson, and Susan R. Hunter. 2014. A comparison of two parallel ranking and selection procedures. In Proceedings of the 2014 Winter Simulation Conference. IEEE Press, 3761--3772.

Digital Library

[29]

Yijie Peng, Chun-Hung Chen, Michael Fu, and Jian-Qiang Hu. 2013. Efficient simulation resource sharing and allocation for selecting the best. IEEE Trans. Automat. Control 58 (2013), 1017--1023.

[30]

Y. Peng, E. K. P. Chong, C.-H. Chen, and M. C. Fu. 2017. Ranking and selection as stochastic control. ArXiv E-prints (Oct. 2017). arxiv:cs.LG/1710.02619

[31]

Huashuai Qu, Ilya O. Ryzhov, and Michael C. Fu. 2012. Ranking and selection with unknown correlation structures. In Proceedings of the Winter Simulation Conference. Winter Simulation Conference, 12.

Digital Library

[32]

Huashuai Qu, Ilya O. Ryzhov, Michael C. Fu, and Zi Ding. 2015. Sequential selection with unknown correlation structures. Operations Research 63, 4 (2015), 931--948.

Digital Library

[33]

Colin R. Reeves. 2010. Genetic algorithms. In Handbook of Metaheuristics. Springer, 109--139.

[34]

Daniel Russo. 2016. Simple Bayesian algorithms for best arm identification. In 29th Annual Conference on Learning Theory (Proceedings of Machine Learning Research), Vitaly Feldman, Alexander Rakhlin, and Ohad Shamir (Eds.), Vol. 49. PMLR, Columbia University, New York, New York, 1417--1418. http://proceedings.mlr.press/v49/russo16.html.

[35]

Ilya O. Ryzhov. 2015. Approximate Bayesian inference for simulation and optimization. In Modeling and Optimization: Theory and Applications. Springer, 1--28.

[36]

Joseph L. Schafer. 1997. Analysis of Incomplete Multivariate Data. CRC Press.

[37]

Christian Schmidt, Jürgen Branke, and Stephen E. Chick. 2006. Integrating techniques from statistical ranking into evolutionary algorithms. In Applications of Evolutionary Computing, Franz Rothlauf et al. (Eds.). 752--763.

Digital Library

[38]

James R. Swisher, Sheldon H. Jacobson, and Enver Yücesan. 2003. Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey. ACM Transactions on Modeling and Computer Simulation 13, 2 (2003), 134--154.

Digital Library

[39]

Zijun Wu and Michael Kolonko. 2014. Asymptotic properties of a generalized cross-entropy optimization algorithm. IEEE Transactions on Evolutionary Computation 18, 5 (2014), 658--673.

[40]

Jing Xie, Peter I. Frazier, and Stephen E. Chick. 2016. Bayesian optimization via simulation with pairwise sampling and correlated prior beliefs. Operations Research 64, 2 (2016), 542--559.

Digital Library

[41]

Qiong Zhang and Yongjia Song. 2016. Moment matching based conjugacy approximation for Bayesian ranking and selection. arXiv preprint arXiv:1610.09400 (2016).

Cited By

Pearce MPoloczek MBranke J(2022)Bayesian Optimization Allowing for Common Random NumbersOperations Research10.1287/opre.2021.220870:6(3457-3472)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1287/opre.2021.2208
Pearce MPoloczek MBranke JSon YHaas P(2019)Bayesian simulation optimization with common random numbersProceedings of the Winter Simulation Conference10.5555/3400397.3400680(3492-3503)Online publication date: 8-Dec-2019
https://dl.acm.org/doi/10.5555/3400397.3400680
Pearce MPoloczek MBranke J(2019)Bayesian Simulation Optimization with Common Random Numbers2019 Winter Simulation Conference (WSC)10.1109/WSC40007.2019.9004680(3492-3503)Online publication date: Dec-2019
https://doi.org/10.1109/WSC40007.2019.9004680

Index Terms

Ranking and Selection: A New Sequential Bayesian Procedure for Use with Common Random Numbers

Recommendations

Moment-Matching-Based Conjugacy Approximation for Bayesian Ranking and Selection

We study the conjugacy approximation models in the context of Bayesian ranking and selection with unknown correlations. Under the assumption of normal-inverse-Wishart prior distribution, the posterior distribution remains a normal-inverse-Wishart ...
Convergence rate analysis for optimal computing budget allocation algorithms
Abstract
Ordinal optimization (OO) is a widely-studied technique for optimizing discrete-event dynamic systems (DEDS). It evaluates the performance of the system designs in a finite set by sampling and aims to correctly make ordinal comparison of the ...
Application of a predictive distribution formula to Bayesian computation for incomplete data models

We consider exact and approximate Bayesian computation in the presence of latent variables or missing data. Specifically we explore the application of a posterior predictive distribution formula derived in Sweeting And Kharroubi (2003), which is a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation

ACM Transactions on Modeling and Computer Simulation Volume 29, Issue 1

January 2019

149 pages

ISSN:1049-3301

EISSN:1558-1195

DOI:10.1145/3309768

Editor:
Adelinde M. Uhrmacher
University of Rostock, Germany

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 January 2019

Accepted: 01 July 2018

Revised: 01 June 2018

Received: 01 May 2017

Published in TOMACS Volume 29, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
153
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pearce MPoloczek MBranke J(2022)Bayesian Optimization Allowing for Common Random NumbersOperations Research10.1287/opre.2021.220870:6(3457-3472)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1287/opre.2021.2208
Pearce MPoloczek MBranke JSon YHaas P(2019)Bayesian simulation optimization with common random numbersProceedings of the Winter Simulation Conference10.5555/3400397.3400680(3492-3503)Online publication date: 8-Dec-2019
https://dl.acm.org/doi/10.5555/3400397.3400680
Pearce MPoloczek MBranke J(2019)Bayesian Simulation Optimization with Common Random Numbers2019 Winter Simulation Conference (WSC)10.1109/WSC40007.2019.9004680(3492-3503)Online publication date: Dec-2019
https://doi.org/10.1109/WSC40007.2019.9004680

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents