Recent Trends in Information Elicitation
Pages 122 - 134
Abstract
This note provides a survey for the Economics and Computation community of some recent trends in the field of information elicitation. At its core, the field concerns the design of incentives for strategic agents to provide accurate and truthful information. Such incentives are formalized as proper scoring rules, and turn out to be the same object as loss functions in machine-learning settings, providing many connections. More broadly, the field concerns the design of mechanisms to obtain information from groups of agents and aggregate it or use it for decision making. Recently, work on information elicitation has expanded and been connected to online no-regret learning, mechanism design, fair division, and more.
References
[1]
Abernethy, J., Chen, Y., and Vaughan, J. W. 2013. Efficient market making via convex optimization, and a connection to online learning. ACM Transactions on Economics and Computation 1, 2, 12.
[2]
Agarwal, A. and Agarwal, S. 2015. On consistent surrogate risk minimization and property elicitation. In Conference on Learning Theory. 4--22.
[3]
Angeris, G., Agrawal, A., Evans, A., Chitra, T., and Boyd, S. 2022. Constant function market makers: Multi-asset trades via convex optimization. In Handbook on Blockchain. Springer, 415--444.
[4]
Angeris, G. and Chitra, T. 2020. Improved price oracles: Constant function market makers. In Proceedings of the 2nd ACM Conference on Advances in Financial Technologies. 80--91.
[5]
Arieli, I., Babichenko, Y., and Smorodinsky, R. 2018. Robust forecast aggregation. Proceedings of the National Academy of Sciences 115, 52, E12135--E12143.
[6]
Bartlett, P. L., Jordan, M. I., and McAuliffe, J. D. 2006. Convexity, classification, and risk bounds. Journal of the American Statistical Association 101, 473, 138--156.
[7]
Bellini, F., Fadina, T., Wang, R., and Wei, Y. 2022. Parametric measures of variability induced by risk measures. Insurance: Mathematics and Economics 106, 270--284.
[8]
Brier, G. 1950. Verification of forecasts expressed in terms of probability. Monthly weather review 78, 1, 1--3.
[9]
Carroll, G. 2019. Robust incentives for information acquisition. Journal of Economic Theory 181, 382--420.
[10]
Casalaina-Martin, S., Frongillo, R., Morgan, T., and Waggoner, B. 2017. Multi-observation elicitation. In Conference on Learning Theory. PMLR, 449--464.
[11]
Chen, Y. and Pennock, D. 2007. A utility framework for bounded-loss market makers. In Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence. 49--56.
[12]
Chen, Y. and Yu, F.-Y. 2023. Optimal scoring rule design under partial knowledge.
[13]
Dasgupta, A. and Ghosh, A. 2013. Crowdsourced judgement elicitation with endogenous proficiency. In Proceedings of the 22nd International Conference on World Wide Web. 319--330.
[14]
De Oliveira, H., Ishii, Y., and Lin, X. 2021. Robust merging of information. In Proceedings of the 22nd ACM Conference on Economics and Computation. EC '21. Association for Computing Machinery, New York, NY, USA, 341--342.
[15]
Dearborn, K. and Frongillo, R. 2020. On the indirect elicitability of the mode and modal interval. Annals of the Institute of Statistical Mathematics 72, 5, 1095--1108.
[16]
Dimitriadis, T., Patton, A. J., and Schmidt, P. W. 2019. Testing forecast rationality for measures of central tendency. arXiv preprint arXiv:1910.12545.
[17]
Dudík, M., Lahaie, S., Pennock, D. M., and Rothschild, D. 2013. A combinatorial prediction market for the us elections. In Proceedings of the Fourteenth ACM Conference on Electronic Commerce. 341--358.
[18]
Ehm, W., Gneiting, T., Jordan, A., and Krüger, F. 2016. Of quantiles and expectiles: consistent scoring functions, choquet representations and forecast rankings. Journal of the Royal Statistical Society Series B: Statistical Methodology 78, 3, 505--562.
[19]
Embrechts, P., Puccetti, G., Rüschendorf, L., Wang, R., and Beleraj, A. 2014. An Academic Response to Basel 3.5. Risks 2, 1 (Feb.), 25--48.
[20]
Emmer, S., Kratz, M., and Tasche, D. 2015. What is the best risk measure in practice? a comparison of standard measures. Journal of Risk 18, 2, 31--60.
[21]
Faltings, B. 2023. Game-theoretic mechanisms for eliciting accurate information. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. IJCAI '23.
[22]
Fiat, A., Karlin, A., Koutsoupias, E., and Vidali, A. 2013. Approaching Utopia: strong truthfulness and externality-resistant mechanisms. In Proceedings of the 4th conference on Innovations in Theoretical Computer Science. 221--230.
[23]
Finocchiaro, J. and Frongillo, R. 2018. Convex elicitation of continuous properties. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. Curran Associates, Inc., 10425--10434.
[24]
Finocchiaro, J., Frongillo, R., and Waggoner, B. 2019. An embedding framework for consistent polyhedral surrogates. In Proceedings of Advances In Neural Information Processing Systems (NeurIPS). 10780--10790.
[25]
Finocchiaro, J., Frongillo, R., and Waggoner, B. 2020. Embedding dimension of polyhedral losses. In Proceedings of Conference on Learning Theory (COLT).
[26]
Finocchiaro, J., Frongillo, R. M., and Waggoner, B. 2024. An embedding framework for the design and analysis of consistent polyhedral surrogates. Journal of Machine Learning Research 25, 63, 1--60.
[27]
Fissler, T., Frongillo, R., Hlavinová, J., and Rudloff, B. 2021. Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals. Electronic Journal of Statistics 15, 1034--1084.
[28]
Fissler, T. and Hoga, Y. 2024. Backtesting systemic risk forecasts using multi-objective elicitability. Journal of Business & Economic Statistics 42, 2, 485--498.
[29]
Fissler, T., Liu, F., Wang, R., and Wei, L. 2024. Elicitability and identifiability of tail risk measures. arXiv preprint arXiv:2404.14136.
[30]
Fissler, T., Ziegel, J., and Gneiting, T. 2016. Expected Shortfall is jointly elicitable with Value at Risk-Implications for backtesting. Risk Magazine.
[31]
Fissler, T. and Ziegel, J. F. 2016. Higher order elicitability and Osband's principle. The Annals of Statistics 44, 4, 1680--1707.
[32]
Fissler, T. and Ziegel, J. F. 2021. On the elicitability of range value at risk. Statistics & risk modeling 38, 1-2, 25--46.
[33]
Freeman, R., Pennock, D., Podimata, C., and Vaughan, J. W. 2020. No-regret and incentive-compatible online learning. In International Conference on Machine Learning. PMLR, 3270--3279.
[34]
Freeman, R. and Pennock, D. M. 2018. An axiomatic view of the parimutuel consensus wagering mechanism. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 1936--1938.
[35]
Freeman, R., Pennock, D. M., and Vaughan, J. W. 2019. An equivalence between wagering and fair-division mechanisms. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 1957--1964.
[36]
Freeman, R., Witkowski, J., Wortman Vaughan, J., and Pennock, D. M. 2023. An equivalence between fair division and wagering mechanisms. Management Science 0, 0.
[37]
Frongillo, R., Gomez, R., Thilagar, A., and Waggoner, B. 2021. Efficient competitions and online learning with strategic forecasters. In Proceedings of the 22nd ACM Conference on Economics and Computation. 479--496.
[38]
Frongillo, R. and Kash, I. 2014. General truthfulness characterizations via convex analysis. In Web and Internet Economics. Springer, 354--370.
[39]
Frongillo, R. and Kash, I. 2015a. Vector-Valued Property Elicitation. In Proceedings of the 28th Conference on Learning Theory. 1--18.
[40]
Frongillo, R. and Kash, I. A. 2015b. On Elicitation Complexity and Conditional Elicitation. arXiv preprint arXiv:1506.07212.
[41]
Frongillo, R., Lladser, M., Thilagar, A., and Waggoner, B. 2023. Forecasting competitions with correlated events. arXiv preprint arXiv:2303.13793.
[42]
Frongillo, R., Mehta, N. A., Morgan, T., and Waggoner, B. 2019. Multi-Observation Regression. In The 22nd International Conference on Artificial Intelligence and Statistics. 2691--2700.
[43]
Frongillo, R., Neyman, E., and Waggoner, B. 2023. Agreement implies accuracy for substitutable signals. In Proceedings of the 2023 ACM Conference on Economics and Computation. EC. ACM.
[44]
Frongillo, R., Papireddygari, M., and Waggoner, B. 2024. An Axiomatic Characterization of CFMMs and Equivalence to Prediction Markets. In 15th Innovations in Theoretical Computer Science Conference (ITCS 2024). Vol. 287. 51:1--51:21.
[45]
Frongillo, R. and Waggoner, B. 2018. An axiomatic study of scoring rule markets. In Proceedings of the 9th Conference on Innovations in Theoretical Computer Science (ITCS). 15:1--15:20.
[46]
Frongillo, R. M. and Kash, I. A. 2021a. Elicitation complexity of statistical properties. Biometrika 108, 4, 857--879.
[47]
Frongillo, R. M. and Kash, I. A. 2021b. General truthfulness characterizations via convex analysis. Games and Economic Behavior 130, 636--662.
[48]
Gneiting, T. 2011. Making and Evaluating Point Forecasts. Journal of the American Statistical Association 106, 494, 746--762.
[49]
Gneiting, T. and Raftery, A. E. 2007. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102, 477, 359--378.
[50]
Good, I. J. 1952. Rational decisions. Journal of the Royal Statistical Society. Series B (Methodological), 107--114.
[51]
Gopalan, P., Kalai, A. T., Reingold, O., Sharan, V., and Wieder, U. 2022. Omnipredictors. In 13th Innovations in Theoretical Computer Science Conference (ITCS 2022), M. Braverman, Ed. Leibniz International Proceedings in Informatics (LIPIcs), vol. 215. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 79:1--79:21.
[52]
Guo, Y., Hartline, J. D., Huang, Z., Kong, Y., Shah, A., and Yu, F.-Y. 2024. Algorithmic robust forecast aggregation.
[53]
Hanson, R. 2003. Combinatorial Information Market Design. Information Systems Frontiers 5, 1, 107--119.
[54]
Hartline, J. D., Shan, L., Li, Y., and Wu, Y. 2023. Optimal scoring rules for multi-dimensional effort. In Proceedings of Thirty Sixth Conference on Learning Theory, G. Neu and L. Rosasco, Eds. Proceedings of Machine Learning Research, vol. 195. PMLR, 2624--2650.
[55]
Hebert-Johnson, U., Kim, M., Reingold, O., and Rothblum, G. 2018. Multicalibration: Calibration for the (Computationally-identifiable) masses. In Proceedings of the 35th International Conference on Machine Learning, J. Dy and A. Krause, Eds. Proceedings of Machine Learning Research, vol. 80. PMLR, 1939--1948.
[56]
Heinrich, C. 2014. The mode functional is not elicitable. Biometrika 101, 1, 245--251.
[57]
Heinrich-Mertsching, C. and Fissler, T. 2022. Is the mode elicitable relative to unimodal distributions? Biometrika 109, 4, 1157--1164.
[58]
Hu, L., Navon, I. R. L., Reingold, O., and Yang, C. 2023. Omnipredictors for constrained optimization. In International Conference on Machine Learning. PMLR, 13497--13527.
[59]
Jung, C., Lee, C., Pai, M., Roth, A., and Vohra, R. 2021. Moment multicalibration for uncertainty estimation. In Conference on Learning Theory. PMLR, 2634--2678.
[60]
Kleinberg, B., Leme, R. P., Schneider, J., and Teng, Y. 2023. U-calibration: Forecasting for an unknown agent. In The Thirty Sixth Annual Conference on Learning Theory. PMLR, 5143--5145.
[61]
Kong, Y. 2020. Dominantly truthful multi-task peer prediction with a constant number of tasks. In Proceedings of the Fourteenth annual ACM-SIAM Symposium on Discrete Algorithms. SIAM, 2398--2411.
[62]
Kong, Y. 2024. Dominantly truthful peer prediction mechanisms with a finite number of tasks. J. ACM 71, 2 (apr).
[63]
Kong, Y. and Schoenebeck, G. 2023. False Consensus, Information Theory, and Prediction Markets. In 14th Innovations in Theoretical Computer Science Conference (ITCS 2023), Y. Tauman Kalai, Ed. Leibniz International Proceedings in Informatics (LIPIcs), vol. 251. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 81:1--81:23.
[64]
Lambert, N. 2018. Elicitation and Evaluation of Statistical Forecasts. Preprint.
[65]
Lambert, N. S., Langford, J., Vaughan, J. W., Chen, Y., Reeves, D. M., Shoham, Y., and Pennock, D. M. 2015. An axiomatic characterization of wagering mechanisms. Journal of Economic Theory 156, 389--416.
[66]
Lambert, N. S., Langford, J., Wortman, J., Chen, Y., Reeves, D., Shoham, Y., and Pennock, D. 2008. Self-financed wagering mechanisms for forecasting. In Proceedings of the 9th ACM conference on Electronic commerce. ACM, 170--179.
[67]
Lambert, N. S., Pennock, D. M., and Shoham, Y. 2008. Eliciting properties of probability distributions. In Proceedings of the 9th ACM Conference on Electronic Commerce. 129--138.
[68]
Levy, G. and Razin, R. 2021. A maximum likelihood approach to combining forecasts. Theoretical Economics 16, 1, 49--71.
[69]
Levy, G. and Razin, R. 2022. Combining forecasts in the presence of ambiguity over correlation structures. Journal of Economic Theory 199, 105075. Symposium Issue on Ambiguity, Robustness, and Model Uncertainty.
[70]
Li, Y., Hartline, J. D., Shan, L., and Wu, Y. 2022. Optimization of scoring rules. In Proceedings of the 23rd ACM Conference on Economics and Computation. EC '22. Association for Computing Machinery, New York, NY, USA, 988--989.
[71]
Li, Y. and Libgober, J. 2023. Optimal scoring for dynamic information acquisition.
[72]
Lin, T. and Chen, Y. 2023. Sample complexity of forecast aggregation. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds. Vol. 36. Curran Associates, Inc., 36053--36093.
[73]
Liu, Y., Wang, J., and Chen, Y. 2023. Surrogate scoring rules. ACM Trans. Econ. Comput. 10, 3 (feb).
[74]
Miller, N., Resnick, P., and Zeckhauser, R. 2005. Eliciting informative feedback: The peer-prediction method. Management Science 51, 9, 1359--1373.
[75]
Mortazavi, A., Lin, J., and Mehta, N. 2024. On the price of exact truthfulness in incentive-compatible online learning with bandit feedback: a regret lower bound for WSU-UX. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, S. Das-gupta, S. Mandt, and Y. Li, Eds. Proceedings of Machine Learning Research, vol. 238. PMLR, 4681--4689.
[76]
Neyman, E., Noarov, G., and Weinberg, S. M. 2021. Binary scoring rules that incentivize precision. In Proceedings of the 22nd ACM Conference on Economics and Computation. 718--733.
[77]
Neyman, E. and Roughgarden, T. 2022. Are you smarter than a random expert? the robust aggregation of substitutable signals. In Proceedings of the 23rd ACM Conference on Economics and Computation. EC '22. Association for Computing Machinery, New York, NY, USA, 990--1012.
[78]
Osband, K. H. 1985. Providing Incentives for Better Cost Forecasting. University of California, Berkeley.
[79]
Osokin, A., Bach, F., and Lacoste-Julien, S. 2017. On structured prediction theory with calibrated convex surrogate losses. In Advances in Neural Information Processing Systems. 302--313.
[80]
Papireddygari, M. and Waggoner, B. 2022. Contracts with information acquisition, via scoring rules. In Proceedings of the 2022 ACM Conference on Economics and Computation. EC. ACM.
[81]
Prelec, D. 2004. A Bayesian Truth Serum for Subjective Data. Science 306, 5695 (Oct.), 462--466.
[82]
Prelec, D. 2021. Bilateral bayesian truth serum: The nxm signals case. Available at SSRN 3908446.
[83]
Ramaswamy, H. G. and Agarwal, S. 2016. Convex calibration dimension for multiclass loss matrices. The Journal of Machine Learning Research 17, 1, 397--441.
[84]
Ramaswamy, H. G., Tewari, A., and Agarwal, S. 2018. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics 12, 1, 530--554.
[85]
Roughgarden, T. and Schrijvers, O. 2017. Online prediction with selfish experts. Advances in Neural Information Processing Systems 30.
[86]
Sadeghi, O. and Fazel, M. 2023. No-regret online prediction with strategic experts. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds. Vol. 36. Curran Associates, Inc., 54696--54715.
[87]
Savage, L. 1971. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association 66, 336, 783--801.
[88]
Schoenebeck, G., Yu, C., and Yu, F.-Y. 2021. Timely information from prediction markets. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems. AAMAS '21. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1145--1153.
[89]
Schoenebeck, G. and Yu, F.-Y. 2023. Two strongly truthful mechanisms for three heterogeneous agents answering one question. ACM Trans. Econ. Comput. 10, 4 (feb).
[90]
Steinwart, I., Pasin, C., Williamson, R., and Zhang, S. 2014. Elicitation and Identification of Properties. In Proceedings of The 27th Conference on Learning Theory. 482--526.
[91]
Tewari, A. and Bartlett, P. L. 2007. On the consistency of multiclass classification methods. The Journal of Machine Learning Research 8, 1007--1025.
[92]
Wang, J., Liu, Y., and Chen, Y. 2021. Forecast aggregation via peer prediction. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 9, 1 (Oct.), 131--142.
[93]
Wang, R. and Wei, Y. 2018. Risk functionals with convex level sets. Available at SSRN 3292661.
[94]
Wang, R. and Ziegel, J. F. 2015. Elicitable distortion risk measures: A concise proof. Statistics & Probability Letters 100, 172--175.
[95]
Wang, X., Pennock, D. M., Devanur, N. R., Rothschild, D. M., Tao, B., and Wellman, M. P. 2021. Designing a combinatorial financial options market. In Proceedings of the 22nd ACM Conference on Economics and Computation. 864--883.
[96]
Witkowski, J., Freeman, R., Vaughan, J. W., Pennock, D. M., and Krause, A. 2018. Incentive-compatible forecasting competitions. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. AAAI.
[97]
Witkowski, J., Freeman, R., Vaughan, J. W., Pennock, D. M., and Krause, A. 2023. Incentive-compatible forecasting competitions. Management Science 69, 3, 1354--1374.
[98]
Zhang, Y. and Schoenebeck, G. 2023. High-effort crowds: Limited liability via tournaments. In Proceedings of the ACM Web Conference 2023. WWW '23. Association for Computing Machinery, New York, NY, USA, 3467--3477.
[99]
Zheng, S., Yu, F.-Y., and Chen, Y. 2021. The limits of multi-task peer prediction. In Proceedings of the 22nd ACM Conference on Economics and Computation. EC '21. Association for Computing Machinery, New York, NY, USA, 907--926.
Index Terms
- Recent Trends in Information Elicitation
Index terms have been assigned to the content through auto-classification.
Recommendations
Preference elicitation in combinatorial auctions
EC '01: Proceedings of the 3rd ACM conference on Electronic CommerceCombinatorial auctions (CAs) where bidders can bid on bundles of items can be very desirable market mechanisms when the items sold exhibit complementarity and/or substitutability, so the bidder's valuations for bundles are not additive. However, in a ...
Incentive compatible budget elicitation in multi-unit auctions
SODA '10: Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete algorithmsIn this paper, we consider the problem of designing incentive compatible auctions for multiple (homogeneous) units of a good, when bidders have private valuations and private budget constraints. When only the valuations are private and the budgets are ...
Mechanisms for partial information elicitation: the truth, but not the whole truth
AAAI'06: Proceedings of the 21st national conference on Artificial intelligence - Volume 1We examine a setting in which a buyer wishes to purchase probabilistic information from some agent. The seller must invest effort in order to gain access to the information, and must therefore be compensated appropriately. However, the information being ...
Comments
Information & Contributors
Information
Published In
June 2024
180 pages
Copyright © 2024 Copyright is held by the owner/author(s).
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 08 October 2024
Published in SIGECOM Volume 22, Issue 1
Check for updates
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 29Total Downloads
- Downloads (Last 12 months)29
- Downloads (Last 6 weeks)6
Reflects downloads up to 18 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in