A Peer Grading Approach for Open-ended Programming Projects Based on Binary System and Swiss System
Pages 1484 - 1490
Abstract
Peer grading is widely used in high education as effective active learning but still faces challenges. We present the peer grading approach for Open-ended Programming Projects based on the binary and Swiss systems. First, we design a grading specification to improve the accuracy of scoring. Second, to make grading easier for inexperienced students, we utilize a pairwise comparison system based on the binary system. Third, we propose a score calculation algorithm based on Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) to improve grading accuracy. We developed an online peer review tool called Peer Review Studio (PRS) based on the approach. We carry out the method in the undergraduate programming course of 2023. We collect and analyze the learning data between 2022 and 2023. When measured by Krippendorff's alpha, the inter-rater reliability between instructor and peer grading is in good agreement. When measured by Kruskal-Wallis, students' project performance and learning engagement significantly improve in the first year of peer grading. The course questionnaire 2023 reveals that most students hold a positive attitude toward peer grading and have benefited significantly from this approach.
References
[1]
Dmytro Babik, Scott Paul Stevens, Andrew E. Waters, and David Tinapple. 2020. The Effects of Dispersion and Reciprocity on Assessment Fidelity in Peer-Review Systems: A Simulation Study. IEEE Trans. Learn. Technol., Vol. 13, 3 (2020), 580--592. https://doi.org/10.1109/TLT.2020.2971495
[2]
Gabriel Badea and Elvira Popescu. 2019. Using LearnEval Peer Assessment Platform in Project-Based Learning Settings: A First Experience Report. In 18th International Conference on Information Technology Based Higher Education and Training, ITHET 2019. IEEE, Magdeburg, Germany, 1--5. https://doi.org/10.1109/ITHET46829.2019.8937359
[3]
Gabriel Badea and Elvira Popescu. 2022. LearnEval Peer Assessment Platform: Iterative Development Process and Evaluation. IEEE Trans. Learn. Technol., Vol. 15, 3 (2022), 421--433. https://doi.org/10.1109/TLT.2022.3185423
[4]
Andrew Berns. 2020. Scored out of 10: Experiences with Binary Grading Across the Curriculum. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education, SIGCSE 2020. ACM, Portland OR, USA, 1152--1157. https://doi.org/10.1145/3328778.3366956
[5]
Bhavya, Jinfeng Xiao, and Chengxiang Zhai. 2021. Scaling Up Data Science Course Projects: A Case Study. In L@S'21: Eighth ACM Conference on Learning @ Scale. ACM, Virtual Event, Germany, 311--314. https://doi.org/10.1145/3430895.3460168
[6]
Nicola Capuano, Vincenzo Loia, and Francesco Orciuoli. 2017. A Fuzzy Group Decision Making Model for Ordinal Peer Assessment. IEEE Trans. Learn. Technol., Vol. 10, 2 (2017), 247--259. https://doi.org/10.1109/TLT.2016.2565476
[7]
Binglin Chen, Matthew West, and Craig B. Zilles. 2022. Peer-grading "Explain in Plain English": A Bayesian Calibration Method for Categorical Answers. In SIGCSE 2022: The 53rd ACM Technical Symposium on Computer Science Education. ACM, Providence, RI, USA, 133--139. https://doi.org/10.1145/3478431.3499409
[8]
John Hamer, Helen C. Purchase, Paul Denny, and Andrew Luxton-Reilly. 2009. Quality of peer assessment in CS1. In Proceedings of the Fifth International Workshop on Computing Education Research, ICER 2009. ACM, Berkeley, CA, USA, 27--36. https://doi.org/10.1145/1584322.1584327
[9]
Theresia Devi Indriasari, Andrew Luxton-Reilly, and Paul Denny. 2021. Investigating Accuracy and Perceived Value of Feedback in Peer Code Review Using Gamification. In ITiCSE '21: Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education. ACM, Virtual Event, Germany, 199--205. https://doi.org/10.1145/3430665.3456338
[10]
Valen E. Johnson and James H. Albert. 2000. Ordinal Data Modeling. Journal of the American Statistical Association., Vol. 95, 451 (2000), 1011--1012. https://doi.org/10.2307/2669495
[11]
David A. Joyner, Wade Ashby, Liam Irish, Yeeling Lam, Jacob Langson, Isabel Lupiani, Mike Lustig, Paige Pettoruto, Dana Sheahen, Angela Smiley, Amy S. Bruckman, and Ashok K. Goel. 2016. Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms. In Proceedings of the Third ACM Conference on Learning @ Scale, L@S 2016. ACM, Edinburgh, Scotland, UK, 399--408. https://doi.org/10.1145/2876034.2876044
[12]
Jennifer S. Kay. 2022. Peer Grading Without Protest: The SPARK Approach to Summative Peer Assessment. In SIGCSE 2022: The 53rd ACM Technical Symposium on Computer Science Education. ACM, Providence, RI, USA, 119--125. https://doi.org/10.1145/3478431.3499284
[13]
Chinmay Kulkarni, Michael S. Bernstein, and Scott R. Klemmer. 2015. PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings of the Second ACM Conference on Learning @ Scale, L@S 2015. ACM, Vancouver, BC, Canada, 75--84. https://doi.org/10.1145/2724660.2724670
[14]
Chinmay Eishan Kulkarni, Richard Socher, Michael S. Bernstein, and Scott R. Klemmer. 2014. Scaling short-answer grading by combining peer assessment with algorithmic scoring. In First (2014) ACM Conference on Learning @ Scale, L@S 2014. ACM, Atlanta, GA, USA, 99--108. https://doi.org/10.1145/2556325.2566238
[15]
Chinmay Eishan Kulkarni, Pang Wei Wei, Huy Le, Daniel Jin hao Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R. Klemmer. 2013. Peer and self assessment in massive online classes. ACM Trans. Comput. Hum. Interact., Vol. 20, 6 (2013), 33:1--33:31. https://doi.org/10.1145/2505057
[16]
Lin Ling and Chee Wei Tan. 2021. Peer-Grading at Scale with Rank Aggregation. In L@S'21: Eighth ACM Conference on Learning @ Scale. ACM, Virtual Event, Germany, 327--330. https://doi.org/10.1145/3430895.3460980
[17]
Heng Luo, Anthony C. Robinson, and Jae-Young Park. 2014. Peer grading in a MOOC: Reliability, validity, and perceived effects. Journal of Asynchronous Learning Network, Vol. 18, 2 (2014), 1--14. https://doi.org/10.24059/olj.v18i2.429
[18]
David J. Nicol. 2014. Unlocking learners' evaluative skills: a peer review perspective. In International Computing Education Research Conference, ICER 2014. ACM, Glasgow, United Kingdom, 1--2. https://doi.org/10.1145/2632320.2632359
[19]
Joonsuk Park and Kimberley Williams. 2016. The Effects of Peer- and Self-assessment on the Assessors. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, SIGCSE 2016. ACM, Memphis, TN, USA, 249--254. https://doi.org/10.1145/2839509.2844602
[20]
Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong B. Do, Andrew Y. Ng, and Daphne Koller. 2013. Tuned Models of Peer Assessment in MOOCs. In Proceedings of the 6th International Conference on Educational Data Mining. International Educational Data Mining Society, Memphis, Tennessee, USA, 153--160. http://www.educationaldatamining.org/EDM2013/papers/rn_paper_23.pdf
[21]
Nea Pirttinen and Juho Leinonen. 2022. Can Students Review Their Peers?: Comparison of Peer and Instructor Reviews. In ITiCSE 2022: Innovation and Technology in Computer Science Education. ACM, Dublin, Ireland, 12--18. https://doi.org/10.1145/3502718.3524762
[22]
Joe Gibbs Politz, Joseph M. Collard, Arjun Guha, Kathi Fisler, and Shriram Krishnamurthi. 2016. The Sweep: Essential Examples for In-Flow Peer Review. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, SIGCSE 2016. ACM, Memphis, TN, USA, 243--248. https://doi.org/10.1145/2839509.2844626
[23]
Joe Gibbs Politz, Shriram Krishnamurthi, and Kathi Fisler. 2014. In-flow peer-review of tests in test-first programming. In International Computing Education Research Conference, ICER 2014. ACM, Glasgow, United Kingdom, 11--18. https://doi.org/10.1145/2632320.2632347
[24]
Karthik Raman and Thorsten Joachims. 2015. Bayesian Ordinal Peer Grading. In Proceedings of the Second ACM Conference on Learning @ Scale, L@S 2015. ACM, Vancouver, BC, Canada, 149--156. https://doi.org/10.1145/2724660.2724678
[25]
Mehdi S. M. Sajjadi, Morteza Alamgir, and Ulrike von Luxburg. 2016. Peer Grading in a Course on Algorithms and Data Structures: Machine Learning Algorithms do not Improve over Simple Baselines. In Proceedings of the Third ACM Conference on Learning @ Scale, L@S 2016. ACM, Edinburgh, Scotland, UK, 369--378. https://doi.org/10.1145/2876034.2876036
[26]
Anshul Shah, Jonathan Liu, Kristin Stephens-Martinez, and Susan H. Rodger. 2021. The CS1 Reviewer App: Choose Your Own Adventure or Choose for Me!. In ITiCSE '21: Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education. ACM, Virtual Event, Germany, 331--337. https://doi.org/10.1145/3430665.3456333
[27]
Joanna Smith, Joe Tessler, Elliot Kramer, and Calvin Lin. 2012. Using peer review to teach software testing. In International Computing Education Research Conference, ICER '12. ACM, Auckland, New Zealand, 93--98. https://doi.org/10.1145/2361276.2361295
[28]
Xiangyu Song, Seth Copen Goldstein, and Majd Sakr. 2020. Using Peer Code Review as an Educational Tool. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE 2020. ACM, Trondheim, Norway, 173--179. https://doi.org/10.1145/3341525.3387370
[29]
Scott A. Turner, Manuel A. Pé rez-Qui n ones, Stephen H. Edwards, and Joseph Chase. 2011. Student attitudes and motivation for peer review in CS2. In Proceedings of the 42nd ACM technical symposium on Computer science education, SIGCSE 2011. ACM, Dallas, TX, USA, 347--352. https://doi.org/10.1145/1953163.1953268
[30]
Ella Tuson and Tim Hickey. 2022. Mastery Learning and Specs Grading in Discrete Math. In ITiCSE 2022: Innovation and Technology in Computer Science Education. ACM, Dublin, Ireland, 19--25. https://doi.org/10.1145/3502718.3524766
[31]
Maya Usher and Miri Barak. 2018. Peer assessment in a project-based engineering course: comparing between on-campus and online learning environments. Assessment & Evaluation in Higher Education, Vol. 43, 5 (2018), 745--759. https://doi.org/10.1080/02602938.2017.1405238
[32]
Masaki Uto, Nguyen Duc Thien, and Maomi Ueno. 2020. Group Optimization to Maximize Peer Assessment Accuracy Using Item Response Theory and Integer Programming. IEEE Trans. Learn. Technol., Vol. 13, 1 (2020), 91--106. https://doi.org/10.1109/TLT.2019.2896966
[33]
Andrew E. Waters, David Tinapple, and Richard G. Baraniuk. 2015. BayesRank: A Bayesian Approach to Ranked Peer Grading. In Proceedings of the Second ACM Conference on Learning @ Scale, L@S 2015. ACM, Vancouver, BC, Canada, 177--183. https://doi.org/10.1145/2724660.2724672
[34]
James R. Wright, Chris Thornton, and Kevin Leyton-Brown. 2015. Mechanical TA: Partially Automated High-Stakes Peer Grading. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education, SIGCSE 2015. ACM, Kansas City, MO, USA, 96--101. https://doi.org/10.1145/2676723.2677278
[35]
William Wu, Constantinos Daskalakis, Nicolaas Kaashoek, Christos Tzamos, and S. Matthew Weinberg. 2015. Game Theory based Peer Grading Mechanisms for MOOCs. In Proceedings of the Second ACM Conference on Learning @ Scale. ACM, Vancouver, BC, Canada, 281--286. https://doi.org/10.1145/2724660.2728676 io
Index Terms
- A Peer Grading Approach for Open-ended Programming Projects Based on Binary System and Swiss System
Recommendations
On the validity of peer grading and a cloud teaching assistant system
LAK '15: Proceedings of the Fifth International Conference on Learning Analytics And KnowledgeWe introduce a new grading system, the Cloud Teaching Assistant System (CTAS), as an additional element to instructor grading, peer grading and automated validation in massive open online courses (MOOCs). The grading distributions of the different ...
Implementation and Experience of the Online Peer Grading System for Our Real Class
SIGUCCS '15: Proceedings of the 2015 ACM SIGUCCS Annual ConferenceIn an online learning course like MOOC (Massive Open Online Course), peer grading is useful for scaling the grades of assignments to the large number of students. Peer grading is also attractive because it reduces the burden of teachers and gives rich ...
Peer Grading Without Protest: The SPARK Approach to Summative Peer Assessment
SIGCSE 2022: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1As Computer Science Professors, we strive to construct courses that maximally support and contribute to student learning through carefully crafted in-class and out-of-class activities. There is evidence that homework enhances student learning, and that ...
Comments
Information & Contributors
Information
Published In
March 2024
1583 pages
ISBN:9798400704239
DOI:10.1145/3626252
- General Chairs:
- Ben Stephenson,
- Jeffrey A. Stone,
- Program Chairs:
- Lina Battestilli,
- Samuel A. Rebelsky,
- Libby Shoop
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 March 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
SIGCSE 2024
Sponsor:
SIGCSE 2024: The 55th ACM Technical Symposium on Computer Science Education
March 20 - 23, 2024
OR, Portland, USA
Acceptance Rates
Overall Acceptance Rate 1,595 of 4,542 submissions, 35%
Upcoming Conference
SIGCSE TS 2025
- Sponsor:
- sigcse
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 67Total Downloads
- Downloads (Last 12 months)67
- Downloads (Last 6 weeks)3
Reflects downloads up to 09 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in