article

The kappa statistic: a second look

Authors:

Barbara Di Eugenio,

Michael GlassAuthors Info & Claims

Computational Linguistics, Volume 30, Issue 1

Pages 95 - 101

https://doi.org/10.1162/089120104773633402

Published: 01 March 2004 Publication History

Abstract

In recent years, the kappa coefficient of agreement has become the de facto standard for evaluating intercoder agreement for tagging tasks. In this squib, we highlight issues that affect κ and that the community has largely neglected. First, we discuss the assumptions underlying different computations of the expected agreement component of κ. Second, we discuss how prevalence and bias affect the κ measure.

References

[1]

Allen, James and Mark Core. 1997. DAMSL: Dialog act markup in several layers; Coding scheme developed by the participants at two discourse tagging workshops, University of Pennsylvania, March 1996, and Schloß Dagstuhl, February 1997. Draft.

[2]

Bartko, John J. and William T. Carpenter. 1976. On the methods and theory of reliability. Journal of Nervous and Mental Disease, 163(5):307-317.

[3]

Berry, Charles C. 1992. The κ statistic {letter to the editor}. Journal of the American Medical Association, 268(18):2513-2514.

[4]

Byrt, Ted, Janet Bishop, and John B. Carlin. 1993. Bias, prevalence, and kappa. Journal of Clinical Epidemiology, 46(5):423-429.

[5]

Carletta, Jean. 1996. Assessing agreement on classification tasks: The Kappa statistic. Computational Linguistics, 22(2):249-254.

Digital Library

[6]

Carletta, Jean, Amy Isard, Stephen Isard, Jacqueline C. Kowtko, Gwyneth Doherty-Sneddon, and Anne H. Anderson. 1997. The reliability of a dialogue structure coding scheme. Computational Lingustics, 23(1):13-31.

Digital Library

[7]

Cicchetti, Domenic V. and Alvan R. Feinstein. 1990. High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6):551-558.

[8]

Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37-46.

[9]

Di Eugenio, Barbara. 2000. On the usage of Kappa to evaluate agreement on coding tasks. In LREC2000: Proceedings of the Second International Conference on Language Resources and Evaluation, pages 441-444, Athens.

[10]

Di Eugenio, Barbara, Pamela W. Jordan, Richmond H. Thomason, and Johanna D. Moore. 2000. The agreement process: An empirical investigation of human-human computer-mediated collaborative dialogues. International Journal of Human Computer Studies, 53(6):1017-1076.

Digital Library

[11]

Fleiss, Joseph L. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378-382.

[12]

Goldman, Ronald L. 1992. The κ statistic {letter to the editor (in reply)}. Journal of the American Medical Association, 268(18):2513-2514.

[13]

Grove, William M., Nancy C. Andreasen, Patricia McDonald-Scott, Martin B. Keller, and Robert W. Shapiro. 1981. Reliability studies of psychiatric diagnosis: Theory and practice. Archives of General Psychiatry, 38:408-413.

[14]

Krippendorff, Klaus. 1980. Content Analysis: An Introduction to Its Methodology. Sage Publications, Beverly Hills, CA.

[15]

Rietveld, Toni and Roeland van Hout. 1993. Statistical Techniques for the Study of Language and Language Behaviour. Mouton de Gruyter, Berlin.

[16]

Scott, William A. 1955. Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19:127-141.

[17]

Siegel, Sidney and N. John Castellan, Jr. 1988. Nonparametric statistics for the behavioral sciences. McGraw Hill, Boston.

[18]

Wiebe, Janyce M., Rebecca F. Bruce, and Thomas P. O'Hara. 1999. Development and use of a gold-standard data set for subjectivity classifications. In ACL99: Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 246-253, College Park, MD.

Digital Library

Cited By

Taulé MNofre MBargiela VBonet X(2024)NewsCom-TOX: a corpus of comments on news articles annotated for toxicity in SpanishLanguage Resources and Evaluation10.1007/s10579-023-09711-x58:4(1115-1155)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s10579-023-09711-x
Trajano DBordini RVieira R(2024)OLID-BR: offensive language identification dataset for Brazilian PortugueseLanguage Resources and Evaluation10.1007/s10579-023-09657-058:4(1263-1289)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s10579-023-09657-0
Scotti VSbattella LTedesco R(2023)A Primer on Seq2Seq Models for Generative ChatbotsACM Computing Surveys10.1145/360428156:3(1-58)Online publication date: 6-Oct-2023
https://dl.acm.org/doi/10.1145/3604281
Show More Cited By

Index Terms

The kappa statistic: a second look
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Language resources

Recommendations

An improved Fuzzy Kappa statistic that accounts for spatial autocorrelation

The Fuzzy Kappa statistic expresses the agreement between two categorical raster maps. The statistic goes beyond cell-by-cell comparison and gives partial credit to cells based on the categories found in the neighborhood. When matching categories are ...
Justification for the Use of Cohen’s Kappa Statistic in Experimental Studies of NLP and Text Mining
Abstract
Modern metrics for evaluating agreement coefficients between the experimental results and expert opinion are compared, and the possibility of using these metrics in experimental research in automatic text processing by machine learning methods is ...
Benchmarking Kappa: Interrater Agreement in Software ProcessAssessments

Software process assessments are by now a prevalent tool for process improvement and contract risk assessment in the software industry. Given that scores are assigned to processes during an assessment, a process assessment can be considered a subjective ...

Comments

Information & Contributors

Information

Published In

cover image Computational Linguistics

Computational Linguistics Volume 30, Issue 1

March 2004

149 pages

ISSN:0891-2017

EISSN:1530-9312

Issue’s Table of Contents

Publisher

MIT Press

Cambridge, MA, United States

Publication History

Published: 01 March 2004

Published in COLI Volume 30, Issue 1

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

80
Total Citations
View Citations
2,719
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)3

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Taulé MNofre MBargiela VBonet X(2024)NewsCom-TOX: a corpus of comments on news articles annotated for toxicity in SpanishLanguage Resources and Evaluation10.1007/s10579-023-09711-x58:4(1115-1155)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s10579-023-09711-x
Trajano DBordini RVieira R(2024)OLID-BR: offensive language identification dataset for Brazilian PortugueseLanguage Resources and Evaluation10.1007/s10579-023-09657-058:4(1263-1289)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s10579-023-09657-0
Scotti VSbattella LTedesco R(2023)A Primer on Seq2Seq Models for Generative ChatbotsACM Computing Surveys10.1145/360428156:3(1-58)Online publication date: 6-Oct-2023
https://dl.acm.org/doi/10.1145/3604281
Arco JOrtiz ARamírez JMartínez-Murcia FZhang YGórriz J(2023)Uncertainty-driven ensembles of multi-scale deep architectures for image classificationInformation Fusion10.1016/j.inffus.2022.08.01089:C(53-65)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.inffus.2022.08.010
Wu CGuo X(2023)Adaptive enhanced interval type-2 possibilistic fuzzy local information clustering with dual-distance for land cover classificationEngineering Applications of Artificial Intelligence10.1016/j.engappai.2022.105806119:COnline publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.engappai.2022.105806
Jin SLian XJung HPark JSuh J(2023)Building a deep learning-based QA system from a CQA datasetDecision Support Systems10.1016/j.dss.2023.114038175:COnline publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1016/j.dss.2023.114038
Bejarano AReig SSenapati PWilliams TSakamoto DWeiss AHiatt LShiomi M(2022)You Had Me at Hello: The Impact of Robot Group Presentation Strategies on Mental Model FormationProceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3523760.3523810(363-371)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.5555/3523760.3523810
Wu CGuo X(2022)A novel interval-valued data driven type-2 possibilistic local information c-means clustering for land cover classificationInternational Journal of Approximate Reasoning10.1016/j.ijar.2022.05.007148:C(80-116)Online publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1016/j.ijar.2022.05.007
Papangelis KLykourentzou IKhan VChamberlain ACao TSaker MLalone N(2021)Locating Identities in Time: An Examination of the Formation and Impact of Temporality on Presentations of the Self through Location-Based Social NetworksACM Transactions on Social Computing10.1145/34730434:3(1-23)Online publication date: 8-Oct-2021
https://dl.acm.org/doi/10.1145/3473043
Novielli NCalefato FLanubile FSerebrenik A(2021)Assessment of off-the-shelf SE-specific sentiment analysis tools: An extended replication studyEmpirical Software Engineering10.1007/s10664-021-09960-w26:4Online publication date: 1-Jul-2021
https://dl.acm.org/doi/10.1007/s10664-021-09960-w
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents