article

ASAP3: a batch means procedure for steady-state simulation analysis

Authors:

Natalie M. Steiger,

James R. Wilson,

Jeffrey A. Joines,

Christos Alexopoulos,

David GoldsmanAuthors Info & Claims

ACM Transactions on Modeling and Computer Simulation (TOMACS), Volume 15, Issue 1

Pages 39 - 73

https://doi.org/10.1145/1044322.1044325

Published: 01 January 2005 Publication History

Abstract

We introduce ASAP3, a refinement of the batch means algorithms ASAP and ASAP2, that delivers point and confidence-interval estimators for the expected response of a steady-state simulation. ASAP3 is a sequential procedure designed to produce a confidence-interval estimator that satisfies user-specified requirements on absolute or relative precision as well as coverage probability. ASAP3 operates as follows: the batch size is progressively increased until the batch means pass the Shapiro-Wilk test for multivariate normality; and then ASAP3 fits a first-order autoregressive (AR(1)) time series model to the batch means. If necessary, the batch size is further increased until the autoregressive parameter in the AR(1) model does not significantly exceed 0.8. Next, ASAP3 computes the terms of an inverse Cornish-Fisher expansion for the classical batch means t-ratio based on the AR(1) parameter estimates; and finally ASAP3 delivers a correlation-adjusted confidence interval based on this expansion. Regarding not only conformance to the precision and coverage-probability requirements but also the mean and variance of the half-length of the delivered confidence interval, ASAP3 compared favorably to other batch means procedures (namely, ABATCH, ASAP, ASAP2, and LBATCH) in an extensive experimental performance evaluation.

References

[1]

Alexopoulos, C. and Goldsman, D. 2004. To batch or not to batch? ACM Trans. Model. Comput. Simul. 14, 1 (Jan.), 76--114.

[2]

Alexopoulos, C. and Seila, A. F. 1998. Output data analysis. In Handbook of Simulation: Principles, Methodology, Advances, Applications, and Practice}, J. Banks, Ed. John Wiley & Sons, New York, NY, 225--272.

[3]

Amemiya, T. and Wu, R. Y. 1972. The effect of aggregation on prediction in the autoregressive model. J. Amer. Statist. Assoc. 67, 339 (Sept.), 628--632.

[4]

Bickel, P. J. and Doksum, K. A. 1977. Mathematical Statistics: Basic Ideas and Selected Topics. Holden-Day, San Francisco, CA.

[5]

Box, G. E. P. 1954. Some theorems on quadratic forms applied in the study of analysis of variance problems, I. Effect of inequality of variance in the one-way classification. Ann. Math. Stat. 25, 290--302.

[6]

Box, G. E. P., Jenkins, G. M., and Reinsel, G. C. 1994. Time Series Analysis: Forecasting and Control. 3rd Ed. Prentice Hall, Englewood Cliffs, NJ.

[7]

Chow, Y. S. and Robbins, H. 1965. On the asymptotic theory of fixed-width sequential confidence intervals for the mean. Ann. Math. Stat. 36, 457--462.

[8]

Fishman, G. S. 1996. Monte Carlo: Concepts, Algorithms, and Applications. Springer-Verlag, New York, NY.

[9]

Fishman, G. S. 1998. LABATCH.2 for analyzing sample path data {online}. Department of Operations Research, University of North Carolina, Chapel Hill, NC. Available at <http://www.unc.edu/depts/or/downloads/tech_reports/fishman/uncor97-04.ps>.

[10]

Fishman, G. S. and Yarberry, L. S. 1997. An implementation of the batch means method. INFORMS J. Comput. 9, 3, 296--310.

[11]

Forsythe, G. E., Malcolm, M. A., and Moler, C. B. 1977. Computer Methods for Mathematical Computations. Prentice-Hall, Englewood Cliffs, NJ.

[12]

Fox, B. L., Goldsman, D., and Swain, J. J. 1991. Spaced batch means. Oper. Res. Lett. 10, 5 (July), 255--263.

[13]

Fuller, W. A. 1996. Introduction to Statistical Time Series. 2nd Ed. John Wiley & Sons, New York, NY.

[14]

Jenkins, G. M. 1954. An angular transformation of the serial correlation coefficient. Biometrika 41, 1/2, 261--265.

[15]

Johnson, N. L., Kotz, S., and Balakrishnan, N. 1994. Continuous Univariate Distributions, Vol. 1. 2nd Ed. John Wiley & Sons, New York, NY.

[16]

Kang, K. and Schmeiser, B. W. 1987. Properties of batch means from stationary ARMA time series. Oper. Res. Lett. 6, 1 (March), 19--24.

[17]

Lada, E. K., Wilson, J. R., and Steiger, N. M. 2003. A wavelet-based spectral method for steady-state simulation analysis. In Proceedings of the 2003 Winter Simulation Conference, S. Chick, P. J. Sánchez, D. Ferrin, and D. J. Morrice, Eds. Institute of Electrical and Electronics Engineers, Piscataway, NJ, 422--430. Available at <http://www.informs-cs.org/wsc03papers/052.pdf>.

[18]

Lada, E. K., Wilson, J. R., Steiger, N. M., and Joines, J. A. 2004a. Performance evaluation of a wavelet-based spectral method for steady-state simulation analysis. In Proceedings of the 2004 Winter Simulation Conference, R. G. Ingalls, M. D. Rossetti, J. S. Smith, and B. A. Peters, Eds. Institute of Electrical and Electronics Engineers, Piscataway, NJ, 694--702. Available at <www.informs-sim.org/wsc04papers/084.pdf>.

[19]

Lada, E. K., Wilson, J. R., Steiger, N. M., and Joines, J. A. 2004b. Performance of a wavelet-based spectral procedure for steady-state simulation analysis. INFORMS J. Comput. to appear. Available at <ftp.ncsu.edu/pub/eos/pub/jwilson/lada04joc.pdf>.

[20]

Malkovich, J. F. and Afifi, A. A. 1973. On tests for multivariate normality. J. Amer. Statist. Assoc. 68, 341 (March), 176--179.

[21]

Nádas, A. 1969. An extension of a theorem of Chow and Robbins on sequential confidence intervals for the mean. The Ann. Math. Stat. 40, 2, 667--671.

[22]

Royston, J. P. 1982a. An extension of Shapiro and Wilk's W test for normality to large samples. Appl. Stat. 31, 2, 115--124.

[23]

Royston, J. P. 1982b. Algorithm AS 181. The W test for normality. Appl. Stat. 31, 176--180.

[24]

Satterthwaite, F. E. 1941. Synthesis of variance. Psychometrika 6, 5, 309--316.

[25]

Satterthwaite, F. E. 1946. An approximate distribution of estimates of variance components. Biometrics Bull. 2, 6, 110--114.

[26]

Searle, S. R. 1982. Matrix Algebra Useful for Statistics. John Wiley & Sons, New York, NY.

[27]

Steiger, N. M. 1999. Improved batching for confidence interval construction in steady state simulation. PhD thesis, Department of Industrial Engineering, North Carolina State University, Raleigh, NC. Available at <http://www.lib.ncsu.edu/etd/public/etd-19231992992670/etd.pdf>.

[28]

Steiger, N. M. and Wilson, J. R. 1999. Improved batching for confidence interval construction in steady-state simulation. In Proceedings of the 1999 Winter Simulation Conference, P. A. Farrington, H. B. Nembhard, D. T. Sturrock, and G. W. Evans, Eds. Institute of Electrical and Electronics Engineers, Piscataway, NJ, 442--451. Available at <http://www.informs-cs.org/wsc99papers/061.PDF>.

[29]

Steiger, N. M. and Wilson, J. R. 2000. Experimental performance evaluation of batch-means procedures for simulation output analysis. In Proceedings of the 2000 Winter Simulation Conference, R. R. Barton, J. A. Joines, K. Kang, and P. A. Fishwick, Eds. Institute of Electrical and Electronics Engineers, Piscataway, NJ, 627--636. Available at <http://www.informs-cs.org/wsc00papers/084.PDF>.

[30]

Steiger, N. M. and Wilson, J. R. 2001. Convergence properties of the batch means method for simulation output analysis. INFORMS J. Comput. 13, 4, 277--293.

[31]

Steiger, N. M. and Wilson, J. R. 2002a. An improved batch means procedure for simulation output analysis. Manage. Sci. 48 12, 1569--1586.

[32]

Steiger, N. M. and Wilson, J. R. 2002b. ASAP software and user's manual. Department of Industrial Engineering, North Carolina State University, Raleigh, NC. Available at <ftp.ncsu.edu/pub/eos/pub/jwilson/installasap.exe>.

[33]

Steiger, N. M., Lada, E. K., Wilson, J. R., Alexopoulos, C., Goldsman, D., and Zouaoui, F. 2002. ASAP2: An improved batch means procedure for simulation output analysis. In Proceedings of the 2002 Winter Simulation Conference, E. Yücesan, C.-H. Chen, J. L. Snowdon, and J. M. Charnes, Eds. Institute of Electrical and Electronics Engineers, Piscataway, NJ, 336--344. Available at <http://www.informs-cs.org/wsc02papers/043.pdf>.

[34]

Steiger, N. M., Lada, E. K., Wilson, J. R., Joines, J. A., Alexopoulos, C., and Goldsman, D. 2003. ASAP3 software and user's manual. Department of Industrial Engineering, North Carolina State University, Raleigh, NC. Available at <ftp.ncsu.edu/pub/eos/pub/jwilson/installasap3.exe>.

[35]

Steiger, N. M., Lada, E. K., Wilson, J. R., Joines, J. A., Alexopoulos, C., and Goldsman, D. 2004. Steady-state simulation analysis using ASAP3. In Proceedings of the 2004 Winter Simulation Conference, R. G. Ingalls, M. D. Rossetti, J. S. Smith, and B. A. Peters, Eds. Institute of Electrical and Electronics Engineers, Piscataway, NJ. 672--680. Available at <www.informs-sim.org/wsc04papers/081.pdf>.

[36]

Stuart, A. and Ord, J. K. 1994. Kendall's Advanced Theory of Statistics, Volume 1: Distribution Theory. 6th. Ed. Edward Arnold, London.

[37]

Suárez-González, A., López-Ardao, J. C., López-García, C., Rodríguez-Pérez, M., Fernández-Veiga, M., and Sousa-Vieira, M. E. 2002. A batch means procedure for mean value estimation of processes exhibiting long range dependence. In Proceedings of the 2002 Winter Simulation Conference, E. Yücesan, C.-H. Chen, J. L. Snowdon, and J. M. Charnes, Eds. Institute of Electrical and Electronics Engineers, Piscataway, NJ, 456--464. Available at <http://www.informs-cs.org/wsc02papers/057.pdf>.

[38]

Tew, J. D. and Wilson, J. R. 1992. Validation of simulation analysis methods for the Schruben-Margolin correlation-induction strategy. Oper. Res. 40, 1, 87--103.

[39]

Welch, B. L. 1956. On linear combinations of several variances. J. Amer. Stat. Assoc. 51 273, 132--148.

[40]

Welch, P. D. 1983. The statistical analysis of simulation results. In Computer Performance Modeling Handbook, S. S. Lavenberg, Ed. Academic Press, New York NY, 268--329.

Cited By

He SLam H(2024)Higher-order coverage errors of batching methods via Edgeworth expansions on t-statisticsThe Annals of Statistics10.1214/24-AOS237752:4Online publication date: 1-Aug-2024
https://doi.org/10.1214/24-AOS2377
Vandin A(2024)Statistical Model Checking of Python Agent-Based Models: An Integration of MultiVeStA and MesaBridging the Gap Between AI and Reality10.1007/978-3-031-75434-0_26(398-419)Online publication date: 30-Dec-2024
https://doi.org/10.1007/978-3-031-75434-0_26
Lam H(2023)Statistical Uncertainty Quantification for Expensive Black-Box Models: Methodologies and Input Uncertainty ApplicationsProceedings of the Winter Simulation Conference10.5555/3643142.3643266(1501-1515)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3643142.3643266
Show More Cited By

Index Terms

ASAP3: a batch means procedure for steady-state simulation analysis
1. Computing methodologies
  1. Modeling and simulation
    1. Simulation evaluation
2. Mathematics of computing
  1. Probability and statistics
    1. Statistical paradigms
      1. Time series analysis

Recommendations

To batch or not to batch?

When designing steady-state computer simulation experiments, one may be faced with the choice of batching observations in one long run or replicating a number of smaller runs. Both methods are potentially useful in the course of undertaking simulation ...
A Batch Means Methodology for Estimation of a Nonlinear Function of a Steady-State Mean

We study the estimation of steady-state performance measures from an â ^d-valued stochastic process Y = { Y t : t â 0} representing the output of a simulation. In many applications, we may be interested in the estimation of a steady-state ...
Steady-state simulation analysis using ASAP3
WSC '04: Proceedings of the 36th conference on Winter simulation

We discuss ASAP3, a refinement of the batch means algorithms ASAP and ASAP2. ASAP3 is a sequential procedure designed to produce a confidence-interval estimator for the expected response of a steady-state simulation that satisfies user-specified ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation

ACM Transactions on Modeling and Computer Simulation Volume 15, Issue 1

January 2005

107 pages

ISSN:1049-3301

EISSN:1558-1195

DOI:10.1145/1044322

Issue’s Table of Contents

Copyright © 2005 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 January 2005

Published in TOMACS Volume 15, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

89
Total Citations
View Citations
739
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)2

Reflects downloads up to 27 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

He SLam H(2024)Higher-order coverage errors of batching methods via Edgeworth expansions on t-statisticsThe Annals of Statistics10.1214/24-AOS237752:4Online publication date: 1-Aug-2024
https://doi.org/10.1214/24-AOS2377
Vandin A(2024)Statistical Model Checking of Python Agent-Based Models: An Integration of MultiVeStA and MesaBridging the Gap Between AI and Reality10.1007/978-3-031-75434-0_26(398-419)Online publication date: 30-Dec-2024
https://doi.org/10.1007/978-3-031-75434-0_26
Lam H(2023)Statistical Uncertainty Quantification for Expensive Black-Box Models: Methodologies and Input Uncertainty ApplicationsProceedings of the Winter Simulation Conference10.5555/3643142.3643266(1501-1515)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3643142.3643266
Lam H(2023)Statistical Uncertainty Quantification for Expensive Black-Box Models: Methodologies and Input Uncertainty Applications2023 Winter Simulation Conference (WSC)10.1109/WSC60868.2023.10407847(1501-1515)Online publication date: 10-Dec-2023
https://doi.org/10.1109/WSC60868.2023.10407847
Vandin AGiachini DLamperti FChiaromonte F(2022)Automated and distributed statistical analysis of economic agent-based modelsJournal of Economic Dynamics and Control10.1016/j.jedc.2022.104458143(104458)Online publication date: Oct-2022
https://doi.org/10.1016/j.jedc.2022.104458
Morgan LBarton R(2022)Fourier trajectory analysis for system discriminationEuropean Journal of Operational Research10.1016/j.ejor.2021.05.052296:1(203-217)Online publication date: Jan-2022
https://doi.org/10.1016/j.ejor.2021.05.052
Walitsarangkul AKittithreerapronchai O(2020)Storage Allocation For Stocked and Buffer BasketsIOP Conference Series: Materials Science and Engineering10.1088/1757-899X/893/1/012009893(012009)Online publication date: 29-Jul-2020
https://doi.org/10.1088/1757-899X/893/1/012009
Dong JGlynn PSon YHaas P(2019)The asymptotic validity of sequential stopping rules for confidence interval construction using standardized time seriesProceedings of the Winter Simulation Conference10.5555/3400397.3400424(332-343)Online publication date: 8-Dec-2019
https://dl.acm.org/doi/10.5555/3400397.3400424
Otunuga OLadde GLadde N(2019)Local Lagged Adapted Generalized Method of Moments: An Innovative Estimation and Forecasting Approach and its ApplicationsJournal of Time Series Econometrics10.1515/jtse-2016-002411:1Online publication date: 25-Feb-2019
https://doi.org/10.1515/jtse-2016-0024
Alexopoulos CGoldsman DMokashi ATien KWilson J(2019)SequestOperations Research10.1287/opre.2018.182967:4(1162-1183)Online publication date: 28-Jun-2019
https://dl.acm.org/doi/10.1287/opre.2018.1829
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents