article

Scoring Rules and Decision Analysis Education

Author:

J. Eric BickelAuthors Info & Claims

Decision Analysis, Volume 7, Issue 4

Pages 346 - 357

https://doi.org/10.1287/deca.1100.0184

Published: 01 December 2010 Publication History

Abstract

Experiential learning is perhaps the most effective way to teach. One example is the scoring procedure used for exams in some decision analysis programs. Under this grading scheme, students take a multiple-choice exam, but rather than simply marking which answer they think is correct, they must assign a probability to each possible answer. The exam is then scored with a special scoring rule, under which students' best strategy is to avoid guessing and instead assign their true beliefs. Such a scoring function is known as a strictly proper scoring rule. In this paper, we discuss several different scoring rules and demonstrate how their use in testing situations provides insights for both students and instructors.

References

[1]

Bernardo, J. M. and Smith, A. F. M., Bayesian Theory, John Wiley & Sons, West Sussex, UK, 2000.

[2]

Bickel, J. E., "Some comparisons between quadratic, spherical, and logarithmic scoring rules," Decision Anal., v4, pp. 49-65, 2007.

Digital Library

[3]

Bickel, J. E., "Experiential learning and strictly proper scoring rules," ASEE Annual Conf.: Paper AC 2009-2249, 2009.

[4]

Cover, T. M. and Thomas, J. A., Elements of Information Theory, John Wiley & Sons, New York, 1991.

Digital Library

[5]

Dawid, A. P., "The well-calibrated Bayesian," J. Amer. Statist. Assoc., v77, pp. 605-610, 1982.

[6]

DeGroot, M. H., Fienberg, S. E., Gupta, S. S. and Berger, J. O., "Assessing probability assessors: Calibration and refinement," Statistical Decision Theory and Related Topics III, Academic Press, New York, pp. 291-314, 1982.

[7]

Epstein, E. S., "A scoring system for probability forecasts of ranked categories," J. Appl. Meteorology, v8, pp. 985-987, 1969.

[8]

Jose, V. R. R., Nau, R. F. and Winkler, R. L., "Sensitivity to distance and baseline distributions in forecast evaluation," Management Sci., v55, pp. 582-590, 2009.

Digital Library

[9]

Kerridge, D. F., "Inaccuracy and inference," J. Roy. Statist. Soc., v23, pp. 184-194, 1961.

[10]

Kullback, S. and Leibler, R. A., "On information sufficiency," Ann. Math. Statist., v22, pp. 79-86, 1951.

[11]

Lichtendahl, K. C. and Winkler, R. L., "Probability elicitation, scoring rules, and competition among forecasters," Management Sci., v53, pp. 1745-1755, 2007.

Digital Library

[12]

Lindley, D. V., Tiago de Oliveira, J. and Epstein, B., "The Bayesian approach to statistics," Some Recent Advances in Statistics, Academic Press, New York, pp. 65-87, 1982.

[13]

Roberts, H. V., "Probabilistic prediction," J. Amer. Statist. Assoc., v60, pp. 50-62, 1965.

[14]

Roby, T. B., "Belief states: A preliminary empirical study," Behavioral Sci., v10, pp. 255-270, 1965.

[15]

Shannon, C. E., "A mathematical theory of communication," Bell System Tech. J., v37, pp. 379-423, 1948.

[16]

Shuford, E. H., Albert, A. and Massengill, H. E., "Admissible probability measurement procedures," Psychometrika, v31, pp. 125-145, 1966.

[17]

Staël von Holstein, C.-A. S., "A family of strictly proper scoring rules which are sensitive to distance," J. Appl. Meteorology, v9, pp. 360-364, 1970.

[18]

Toda, M., "Measurement of subjective probability distributions," 1963.

[19]

Winkler, R. L., “Good” probability assessors, J. Appl. Meteorology, v7, pp. 751-758, 1968.

[20]

Winkler, R. L., "Scoring rules and the evaluation of probability assessors," J. Amer. Statist. Assoc., v64, pp. 1073-1078, 1969.

[21]

Winkler, R. L., "Scoring rules and the evaluation of probabilities," Test, v5, pp. 1-60, 1996.

Cited By

Smith ZBickel J(2022)Weighted Scoring Rules and Convex Risk MeasuresOperations Research10.1287/opre.2021.219070:6(3371-3385)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1287/opre.2021.2190
Werner T(2022)Elicitability of Instance and Object RankingDecision Analysis10.1287/deca.2021.044619:2(123-140)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1287/deca.2021.0446
Bier VFrench S(2020)From the EditorsDecision Analysis10.1287/deca.2020.040817:1(1-8)Online publication date: 1-Mar-2020
https://dl.acm.org/doi/10.1287/deca.2020.0408
Show More Cited By

Recommendations

Some Comparisons among Quadratic, Spherical, and Logarithmic Scoring Rules

Strictly proper scoring rules continue to play an important role in probability assessment. Although many such rules have been developed, relatively little guidance exists as to which rule is the most appropriate. In this paper, we discuss two important ...
Optimization of Scoring Rules
EC '22: Proceedings of the 23rd ACM Conference on Economics and Computation

This paper introduces an objective for optimizing proper scoring rules. The objective is to maximize the increase in payoff of a forecaster who exerts a binary level of effort to refine a posterior belief from a prior belief. In this framework we ...
Tailored Scoring Rules for Probabilities

When scoring rules were first widely used, they were developed as a way to measure the accuracy of probability forecasts ex post. Ex ante, proper scoring rules encourage honestly reported and sharper probabilities, both of which increase the forecaster'...

Comments

Information & Contributors

Information

Published In

cover image Decision Analysis

Decision Analysis Volume 7, Issue 4

December 2010

86 pages

ISSN:1545-8490

Issue’s Table of Contents

Publisher

INFORMS

Linthicum, MD, United States

Publication History

Published: 01 December 2010

Accepted: 28 June 2010

Received: 07 December 2009

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Smith ZBickel J(2022)Weighted Scoring Rules and Convex Risk MeasuresOperations Research10.1287/opre.2021.219070:6(3371-3385)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1287/opre.2021.2190
Werner T(2022)Elicitability of Instance and Object RankingDecision Analysis10.1287/deca.2021.044619:2(123-140)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1287/deca.2021.0446
Bier VFrench S(2020)From the EditorsDecision Analysis10.1287/deca.2020.040817:1(1-8)Online publication date: 1-Mar-2020
https://dl.acm.org/doi/10.1287/deca.2020.0408
Smith ZBickel J(2020)Additive Scoring Rules for Discrete Sample SpacesDecision Analysis10.1287/deca.2019.039817:2(115-133)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1287/deca.2019.0398
Carvalho A(2016)An Overview of Applications of Proper Scoring RulesDecision Analysis10.1287/deca.2016.033713:4(223-242)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1287/deca.2016.0337
(2013)From the EditorsDecision Analysis10.5555/2736042.273604310:1(1-5)Online publication date: 1-Mar-2013
https://dl.acm.org/doi/10.5555/2736042.2736043
(2013)From the EditorMedian Aggregation, Scoring Rules, Expert Forecasts, Choices with Binary Attributes, Portfolio with Dependent Projects, and Information SecurityDecision Analysis10.5555/2700014.270001510:4(277-278)Online publication date: 1-Dec-2013
https://dl.acm.org/doi/10.5555/2700014.2700015
Keller LAbbas ABickel JBier VBudescu DButler JDiecidue EDillon-Merrill RHämäläinen RLichtendahl KMerrick JSimon JWu G(2012)From the Editors---Brainstorming, Multiplicative Utilities, Partial Information on Probabilities or Outcomes, and Regulatory FocusDecision Analysis10.1287/deca.1120.02559:4(297-302)Online publication date: 1-Dec-2012
https://dl.acm.org/doi/10.1287/deca.1120.0255
Keller LKophazi K(2012)From the Editors---Copulas, Group Preferences, Multilevel Defenders, Sharing Rewards, and Communicating AnalyticsDecision Analysis10.1287/deca.1120.02509:3(213-218)Online publication date: 1-Sep-2012
https://dl.acm.org/doi/10.1287/deca.1120.0250
Keller LAbbas ABickel JBier VBudescu DButler JDelquié PLichtendahl KMerrick JSalo AWu G(2011)From the Editors---Probability Scoring Rules, Ambiguity, Multiattribute Terrorist Utility, and Sensitivity AnalysisDecision Analysis10.1287/deca.1110.02228:4(251-255)Online publication date: 1-Dec-2011
https://dl.acm.org/doi/10.1287/deca.1110.0222
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents