research-article

HYpothesis-driven experiment design in computer simulation studies

Authors:

Daniel S. Lebherz,

Jan Ole Berndt,

Ingo J. TimmAuthors Info & Claims

WSC '17: Proceedings of the 2017 Winter Simulation Conference

Article No.: 103, Pages 1 - 12

Published: 03 December 2017 Publication History

Abstract

In simulation studies, the goal specifies the objective or purpose of the study and thus drives the entire experimentation process. Relevant experiments and respective experiment hypotheses are derived from the study's goal and the model's observed behavior provides evidence whether these hypotheses hold. Current assistance systems do not integrate research hypotheses. Thus, the researcher has to make important design decisions which limits both replicability and reproducibility of the results. In this paper, the process of simulation studies is systematized based on a formally specified hypothesis. By this means, the research hypothesis becomes the key element of the study and guides the entire process. Hypothesis-driven simulation studies allow for the automated design, execution, and evaluation of experiments based on specific research questions. This facilitates documentation and execution as well as replication of simulation studies.

References

[1]

Banks, J. 1998. Handbook of Simulation. John Wiley & Sons.

[2]

Better, M., F. Glover, and M. Laguna. 2007. "Advances in Analytics: Integrating Dynamic Data Mining with Simulation Optimization". IBM Journal of Research and Development 51 (3.4): 477--487.

Digital Library

[3]

Bettonvil, B., and J. P. Kleijnen. 1997. "Searching for Important Factors in Simulation Models with many Factors: Sequential Bifurcation". European Journal of Operational Research 96 (1): 180--194.

[4]

Conway, R. W., and J. O. McClain. 2003. "The Conduct of an Effective Simulation Study". INFORMS Transactions on Education 3 (3): 13--22.

Digital Library

[5]

Ewald, R., and A. M. Uhrmacher. 2014. "SESSL: A Domain-Specific Language for Simulation Experiments". ACM Transactions on Modeling and Computer Simulation (TOMACS) 24 (2): 11.

Digital Library

[6]

Fujimoto, R. 2015. "Parallel and Distributed Simulation". In Proceedings of the 2015 Winter Simulation Conference, edited by C. M. Macal, M. D. Rossetti, L. Yilmaz, I.-C. Moon, W. K. Chan, and T. Roeder, 45--59. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc.

Digital Library

[7]

Gil, A. 2012. Artificial Supply Chain. www.modelingcommons.org/browse/one_model/3378, accessed 03/17.

[8]

Griffin, T. G., S. Petrovic, A. Poplawski, and B. Premore. 2002. SOS. www.ssfnet.org/sos, accessed 03/17.

[9]

Himmelspach, J., and A. M. Uhrmacher. 2007. "Plug'n Simulate". In ANSS, 137--143. IEEE.

Digital Library

[10]

Hoad, K., S. Robinson, and R. Davies. 2010. "Automated Selection of the Number of Replications for a Discrete-Event Simulation". Journal of the Operational Research Society 61 (11): 1632--1644.

[11]

King, R. D., J. Rowland, S. G. Oliver, M. Young, W. Aubrey, E. Byrne, M. Liakata, M. Markham, P. Pir, L. N. Soldatova et al. 2009. "The Automation of Science". Science 324 (5923): 85--89.

[12]

Kleijnen, J. P. 2008. Design and Analysis of Simulation Experiments, Volume 20. Springer.

[13]

Kleijnen, J. P., B. Bettonvil, and F. Persson. 2003. "Finding the Important Factors in Large Discrete-Event Simulation: Sequential Bifurcation and its Applications". CentER Discussion Paper; Vol. 2003--104.

[14]

Köhn, D., and N. Le Novère. 2008. "SED-ML-an XML Format for the Implementation of the MIASE Guidelines". In Computational Methods in Systems Biology, 176--190. Springer.

Digital Library

[15]

Lattner, A. D., T. Bogon, and I. J. Timm. 2011. "An Approach to Significance Estimation for Simulation Studies". In Agents and Artificial Intelligence, 177--186. ICAART.

[16]

Lattner, A. D., H. Pitsch, I. J. Timm, S. Spieckermann, and S. Wenzel. 2011. "AssistSim-Towards Automation of Simulation Studies in Logistics". Simulation Notes Europe 21 (3--4): 119.

[17]

Law, A. M., and W. D. Kelton. 1991. Simulation Modeling and Analysis, Volume 2. McGraw-Hill.

Digital Library

[18]

Lorig, F., C. A. Becker, and I. J. Timm. 2017. "Formal Specification of Hypotheses for Assisting Computer Simulation Studies". In TMS/DEVS at SpringSim Multiconferenece, 1180--1191. SCS.

Digital Library

[19]

Maria, A. 1997. "Introduction to Modeling and Simulation". In Proceedings of the 1997 Winter Simulation Conference, edited by D. H. Withers, B. L. Nelson, S. Andradóttir, and K. J. Healy, 7--13. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc.

Digital Library

[20]

Mill, J. S. 1868. System of Logic: Vol. II. Longmans, Green, Reader, and Dyer.

[21]

Montgomery, D. C. 2012. Design and Analysis of Experiments, Volume 8. John Wiley & Sons.

[22]

Morris, M. 1991. "Factorial Sampling Plans for Preliminary Experiments". Technometrics 33 (2): 161--174.

Digital Library

[23]

Ören, T. I., B. P. Zeigler, and M. S. Elzas. 1984. Simulation and Model-Based Methodologies: An Integrative View, Volume 10. NATO ASI Series F: Computer and System Sciences.

[24]

Peng, D. 2017. Reusing Simulation Experiments for Model Composition and Extension. Rostock University.

[25]

Perrone, L. F., C. S. Main, and B. C. Ward. 2012. "SAFE: Simulation Automation Framework for Experiments". In Proceedings of the 2012 Winter Simulation Conference, edited by O. Rose, A. Uhrmacher, M. Rabe, C. Laroque, R. Pasupathy, and J. Himmelspach, 2825--2836. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc.

Digital Library

[26]

Robinson, S. 2005. "Automated Analysis of Simulation Output Data". In Proceedings of the 2005 Winter Simulation Conference, edited by F. Armstrong, J. Joines, N. Steiger, and M. Kuhl, 763--770. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc.

Digital Library

[27]

Sanchez, S. M. 2005. "Work Smarter, not Harder". In Proceedings of the 2005 Winter Simulation Conference, edited by F. Armstrong, J. A. Joines, N. Steiger, and M. E. Kuhl, 69--82. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc.

Digital Library

[28]

Soldatova, L. N., and R. D. King. 2006. "An Ontology of Scientific Experiments". Journal of the Royal Society Interface 3 (11): 795--803.

[29]

Teran-Somohano, A., A. E. Smith, J. Ledet, L. Yilmaz, and H. Oĝuztüzün. 2015. "A Model-Driven Engineering Approach to Simulation Experiment Design and Execution". In Proceedings of the 2015 Winter Simulation Conference, edited by C. M. Macal, M. D. Rossetti, L. Yilmaz, I.-C. Moon, W. K. Chan, and T. Roeder, 2632--2643. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc.

Digital Library

[30]

Timm, I. J., and F. Lorig. 2015. "A Survey on Methodological Aspects of Computer Simulation as Research Technique". In Proceedings of the 2015 Winter Simulation Conference, edited by C. M. Macal, M. D. Rossetti, L. Yilmaz, I.-C. Moon, W. K. Chan, and T. Roeder, 2704--2715. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc.

Digital Library

[31]

Tolk, A., B. L. Heath, M. Ihrig, J. J. Padilla, E. H. Page, E. D. Suarez, C. Szabo, P. Weirich, and L. Yilmaz. 2013. "Epistemology of Modeling and Simulation". In Proceedings of the 2013 Winter Simulation Conference, edited by R. Hill, M. Kuhn, R. Pasupathy, S.-H. Kim, and A. Tolk, 1152--1166. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc.

Digital Library

[32]

Uhrmacher, A., S. Brailsford, J. Liu, M. Rabe, and A. Tolk. 2016. "Reproducible Research in Discrete Event Simulation". In Proceedings of the 2016 Winter Simulation Conference, edited by T. Huschka, S. Chick, J. Jiminez, P. Frazier, T. Roeder, R. Szechtman, and E. Zhou, 1301--1315. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc.

Digital Library

[33]

Waltemath, D., R. Adams, D. A. Beard, F. T. Bergmann, U. S. Bhalla, R. Britten et al. 2011. "Minimum Information about a Simulation Experiment (MIASE)". PLoS Comput Biol 7 (4): e1001122.

[34]

Waltz, D., and B. G. Buchanan. 2009. "Automating Science". Science 324 (5923): 43--44.

[35]

Whewell, W. 1847. The Philosophy of the Inductive Sciences. J. W. Parker.

[36]

Winsberg, E. 2010. Science in the Age of Computer Simulation. University of Chicago Press.

Digital Library

[37]

Yilmaz, L., S. Chakladar, and K. Doud. 2016. "The Goal-Hypothesis-Experiment Framework". In Proceedings of the 2016 Winter Simulation Conference, 1001--1012. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc.

Digital Library

[38]

Yilmaz, L., S. J. Taylor, R. Fujimoto, and F. Darema. 2014. "Panel: The Future of Research in Modeling & Simulation". In Proceedings of the 2014 Winter Simulation Conference, edited by S. J. Buckley, J. A. Miller, A. Tolk, L. Yilmaz, S. Y. Diallo, and I. O. Ryzhov, 2797--2811. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc.

Digital Library

[39]

Zeigler, B. P., H. Praehofer, and T. G. Kim. 2000. Theory of Modeling and Simulation. Academic press.

Digital Library

Cited By

Meyer TRuscheinski AWilsdorf PUhrmacher A(2021)Simulation-supported engineering of self-adaptive software systemsProceedings of the Winter Simulation Conference10.5555/3522802.3522833(1-12)Online publication date: 13-Dec-2021
https://dl.acm.org/doi/10.5555/3522802.3522833
Berndt JBecker CTimm IJohansson BJain S(2018)Inequality, famine, and counter-measuresProceedings of the 2018 Winter Simulation Conference10.5555/3320516.3321075(4202-4203)Online publication date: 9-Dec-2018
https://dl.acm.org/doi/10.5555/3320516.3321075
Berndt JRodermund STimm IJohansson BJain S(2018)Social contagion of fertilityProceedings of the 2018 Winter Simulation Conference10.5555/3320516.3320638(953-964)Online publication date: 9-Dec-2018
https://dl.acm.org/doi/10.5555/3320516.3320638
Show More Cited By

Recommendations

Hypothesis-Driven Simulation Studies: Assistance for the Systematic Design and Conducting of Computer Simulation Experiments
The goal-hypothesis-experiment framework: a generative cognitive domain architecture for simulation experiment management
WSC '16: Proceedings of the 2016 Winter Simulation Conference

Simulation experiments are not conducted in a vacuum. They are performed to address specific research questions that require evaluation of testable hypotheses. However, the connections among goals, hypotheses, and experiments are often characterized in ...
A model-driven engineering approach to simulation experiment design and execution
WSC '15: Proceedings of the 2015 Winter Simulation Conference

There is an increasing awareness of the added value that efficient design of experiments can provide to simulation studies. However, many practitioners lack the proper training required to design, execute and analyze a well-designed simulation ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WSC '17: Proceedings of the 2017 Winter Simulation Conference

December 2017

4389 pages

ISBN:9781538634271

Sponsors

SIGSIM: ACM Special Interest Group on Simulation and Modeling

Publisher

IEEE Press

Publication History

Published: 03 December 2017

Check for updates

Qualifiers

Research-article

Conference

WSC '17

Sponsor:

SIGSIM

WSC '17: Winter Simulation Conference

December 3 - 6, 2017

Nevada, Las Vegas

Acceptance Rates

Overall Acceptance Rate 3,413 of 5,075 submissions, 67%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
43
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Meyer TRuscheinski AWilsdorf PUhrmacher A(2021)Simulation-supported engineering of self-adaptive software systemsProceedings of the Winter Simulation Conference10.5555/3522802.3522833(1-12)Online publication date: 13-Dec-2021
https://dl.acm.org/doi/10.5555/3522802.3522833
Berndt JBecker CTimm IJohansson BJain S(2018)Inequality, famine, and counter-measuresProceedings of the 2018 Winter Simulation Conference10.5555/3320516.3321075(4202-4203)Online publication date: 9-Dec-2018
https://dl.acm.org/doi/10.5555/3320516.3321075
Berndt JRodermund STimm IJohansson BJain S(2018)Social contagion of fertilityProceedings of the 2018 Winter Simulation Conference10.5555/3320516.3320638(953-964)Online publication date: 9-Dec-2018
https://dl.acm.org/doi/10.5555/3320516.3320638
Taylor SEldabi TMonks TRabe MUhrmacher AJohansson BJain S(2018)Crisis, what crisisProceedings of the 2018 Winter Simulation Conference10.5555/3320516.3320615(749-762)Online publication date: 9-Dec-2018
https://dl.acm.org/doi/10.5555/3320516.3320615

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents