research-article

CrowdGrader: a tool for crowdsourcing the evaluation of homework assignments

Authors:

Luca de Alfaro,

Michael ShavlovskyAuthors Info & Claims

SIGCSE '14: Proceedings of the 45th ACM technical symposium on Computer science education

Pages 415 - 420

https://doi.org/10.1145/2538862.2538900

Published: 05 March 2014 Publication History

Get Access

Abstract

CrowdGrader is a system that lets students submit and collaboratively review and grade homework. We describe the techniques and ideas used in CrowdGrader, and report on the experience of using CrowdGrader in disciplines ranging from Computer Science to Economics, Writing, and Technology. In CrowdGrader, students receive an overall crowd-grade that reflects both the quality of their homework, and the quality of their work as reviewers. This creates an incentive for students to provide accurate grades and helpful reviews of other students' work. Instructors can use the crowd-grades as final grades, or fine-tune the grades according to their wishes. Our results on seven classes show that students actively participate in the grading and write reviews that are generally helpful to the submissions' authors. The results also show that grades computed by CrowdGrader are sufficiently precise to be used as the homework component of class grades. Students report that the main benefits in using CrowdGrader are the quality of the reviews they receive, and the ability to learn from reviewing their peers' work. Instructors can leverage peer learning in their classes, and easily handle homework evaluation in large classes.

References

[1]

J. Bartholdi, C. Tovey, and M. Trick. Voting schemes for which it can be difficult to tell who won the election. Social Choice and Welfare, 6(2):157--165, 1989.

Crossref

Google Scholar

[2]

D. J. Boud, R. Cohen, and J. Sampson. Peer learning in higher education: learning from & with each other. Psychology Press, 2001.

Google Scholar

[3]

R. Bradley and M. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):pp. 324--345, 1952.

Crossref

Google Scholar

[4]

K. Cho, T. Chung, W. King, and C. Schunn. Peer-based computer-supported knowledge refinement: An empirical investigation. Communications of the ACM, 51(3):83--88, 2008.

Digital Library

Google Scholar

[5]

C. Davidson. How to crowdsource grading, 2009. http://www.hastac.org/blogs/cathy-davidson/how-crowdsource-grading.

Google Scholar

[6]

L. de Alfaro and M. Shavlovsky. Crowdgrader: Crowdsourcing the evaluation of homework assignments. Technical Report UCSC-SOE-13-11, UC Santa Cruz, 2013. arXiv:1308.5273.

Google Scholar

[7]

J.-C. de Borda. Memoire sur les Elections au Scrutin. 1781.

Google Scholar

[8]

C. Dwork, R. Kumar, M. Naor, and D. Sivakumar. Rank aggregation methods for the web. In Proceedings of the 10th international conference on World Wide Web, pages 613--622. ACM, 2001.

Digital Library

Google Scholar

[9]

A. Elo. The Rating of Chess Players Past and Present. New York, Arco, 1978.

Google Scholar

[10]

M. Glickman. Paired Comparison Models with Time-varying Parameters. Harvard University, 1993.

Crossref

Google Scholar

[11]

R. Luce. Individual choice behavior : a theoretical analysis. Wiley N.Y, 1959.

Google Scholar

[12]

M. Merrifield and D. Saari. Telescope time without tears: A distributed approach to peer review. Astronomy & Geophysics, 50(4):4--16, 2009.

Crossref

Google Scholar

[13]

P. Naghizadeh and M. Liu. Incentives, quality, and risk: A look into the NSF proposal review pilot. Arxiv, 1307.6528v1, 2013.

Google Scholar

[14]

National Science Foundation. Dear colleague letter: Information to principal investigators (PIs) planning to submit proposals to the sensors and sensing systems (sss) program October 1, 2013 deadline, 2013.

Google Scholar

Cited By

View all

Ma WLiu XYu JYu KWang X(2024)A Collusion Attack Resistance Data Aggregation Scheme in Internet of ThingsIEEE Transactions on Industrial Informatics10.1109/TII.2024.343551320:11(13352-13362)Online publication date: Nov-2024
https://doi.org/10.1109/TII.2024.3435513
Olckers MWalsh T(2024)Manipulation and Peer Mechanisms: A SurveyArtificial Intelligence10.1016/j.artint.2024.104196(104196)Online publication date: Aug-2024
https://doi.org/10.1016/j.artint.2024.104196
Babik DGehringer EKidd JSunday KTinapple DGilbert S(2024)A systematic review of educational online peer-review and assessment systems: charting the landscapeEducational technology research and development10.1007/s11423-024-10349-x72:3(1653-1689)Online publication date: 19-Mar-2024
https://doi.org/10.1007/s11423-024-10349-x
Show More Cited By

Index Terms

CrowdGrader: a tool for crowdsourcing the evaluation of homework assignments
1. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

Survey on computer program grading practices. Part I: analytic grading
Rubricking Like a Boss: Writing and Using Rubrics For Faster, Fairer Grading of Student Assignments (Abstract Only)
SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science Education

Educators responsible for coordinating a staff to grade a large volume of exams or projects have learned, sometimes the hard way, the importance of writing good rubrics: they keep the graders on track, grading equitably, consistently, and quickly. But ...
Peer Grading Without Protest: The SPARK Approach to Summative Peer Assessment
SIGCSE 2022: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1

As Computer Science Professors, we strive to construct courses that maximally support and contribute to student learning through carefully crafted in-class and out-of-class activities. There is evidence that homework enhances student learning, and that ...

Comments

Information & Contributors

Information

Published In

SIGCSE '14: Proceedings of the 45th ACM technical symposium on Computer science education

March 2014

800 pages

ISBN:9781450326056

DOI:10.1145/2538862

Conference Chairs:
J. D. Dougherty
J.D. Dougherty, Haverford College
,
Kris Nagel
Georgia Gwinnett College
,
Program Chairs:
Adrienne Decker
Rochester Institute of Technology
,
Kurt Eiselt
University of British Columbia

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 March 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGCSE '14

Sponsor:

SIGCSE

SIGCSE '14: The 45th ACM Technical Symposium on Computer Science Education

March 5 - 8, 2014

Georgia, Atlanta, USA

Acceptance Rates

SIGCSE '14 Paper Acceptance Rate 108 of 274 submissions, 39%;

Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE TS 2025

Sponsor:
sigcse

The 56th ACM Technical Symposium on Computer Science Education

February 26 - March 1, 2025

Pittsburgh , PA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

61
Total Citations
View Citations
488
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)2

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ma WLiu XYu JYu KWang X(2024)A Collusion Attack Resistance Data Aggregation Scheme in Internet of ThingsIEEE Transactions on Industrial Informatics10.1109/TII.2024.343551320:11(13352-13362)Online publication date: Nov-2024
https://doi.org/10.1109/TII.2024.3435513
Olckers MWalsh T(2024)Manipulation and Peer Mechanisms: A SurveyArtificial Intelligence10.1016/j.artint.2024.104196(104196)Online publication date: Aug-2024
https://doi.org/10.1016/j.artint.2024.104196
Babik DGehringer EKidd JSunday KTinapple DGilbert S(2024)A systematic review of educational online peer-review and assessment systems: charting the landscapeEducational technology research and development10.1007/s11423-024-10349-x72:3(1653-1689)Online publication date: 19-Mar-2024
https://doi.org/10.1007/s11423-024-10349-x
Li BDu X(2024)A Strategy-Proof and Collusion-Proof Peer Grading MechanismMechanism Design in Social Networks10.1007/978-981-96-0214-8_7(86-94)Online publication date: 22-Dec-2024
https://doi.org/10.1007/978-981-96-0214-8_7
Burrell NSchoenebeck GLeyton-Brown KSamuelson LHartline J(2023)Measurement Integrity in Peer Prediction: A Peer Assessment Case StudyProceedings of the 24th ACM Conference on Economics and Computation10.1145/3580507.3597744(369-389)Online publication date: 9-Jul-2023
https://dl.acm.org/doi/10.1145/3580507.3597744
Darvishi AKhosravi HRahimi ASadiq SGasevic D(2023)Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review ModelsIEEE Transactions on Learning Technologies10.1109/TLT.2022.322902216:1(106-120)Online publication date: 1-Feb-2023
https://doi.org/10.1109/TLT.2022.3229022
Han YWu WLiang YZhang L(2023)Peer Grading Eliciting Truthfulness Based on AutograderIEEE Transactions on Learning Technologies10.1109/TLT.2022.321694616:3(353-363)Online publication date: 1-Jun-2023
https://doi.org/10.1109/TLT.2022.3216946
Khan SBlessing LNdiaye Y(2023)Artificial Intelligence for Competency Assessment in Design Education: A Review of LiteratureDesign in the Era of Industry 4.0, Volume 310.1007/978-981-99-0428-0_85(1047-1058)Online publication date: 25-Jul-2023
https://doi.org/10.1007/978-981-99-0428-0_85
López de Aberasturi Gómez ASabater-Mir JSierra C(2022)Probabilistic Models for Competence Assessment in EducationApplied Sciences10.3390/app1205236812:5(2368)Online publication date: 24-Feb-2022
https://doi.org/10.3390/app12052368
Alabdulaziz MHassan HSoliman M(2022)The effect of the interaction between crowdsourced style and cognitive style on developing research and scientific thinking skillsEurasia Journal of Mathematics, Science and Technology Education10.29333/ejmste/1242818:10(em2162)Online publication date: 2022
https://doi.org/10.29333/ejmste/12428
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Survey on computer program grading practices. Part I: analytic grading

Rubricking Like a Boss: Writing and Using Rubrics For Faster, Fairer Grading of Student Assignments (Abstract Only)

Peer Grading Without Protest: The SPARK Approach to Summative Peer Assessment