short-paper

Public Access

Long-Term Peer Reviewing Effort is Anti-Reciprocal

Authors:

Yasmine Kotturi,

Chinmay KulkarniAuthors Info & Claims

L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

Pages 279 - 282

https://doi.org/10.1145/3051457.3054004

Published: 12 April 2017 Publication History

Abstract

Many studies demonstrate that peer reviewing provides pedagogical benefits such as inspiration and developing expert vision, and changes classroom culture by encouraging reciprocity. However, much large-scale research in peer assessment has focused on MOOCs, where students have short tenures, and is unable to describe how reciprocity-oriented classroom cultures evolve over time. This short paper presents the first long-term analysis of peer reviewing with 304 students, conducted in three large physical classes in a year-long undergraduate series. Surprisingly, this analysis reveals that when students receive better reviews on their work, they write worse reviews in the future. This suggests that while students believe in the reciprocal nature of peer review, they act anti-reciprocally. Therefore, battling the emergent norm of anti-reciprocity is crucial both for system designers and practitioners who use peer assessment.

References

[1]

David Boud, Ruth Cohen, and Jane Sampson. 2014. Peer learning in higher education: Learning from and with each other. Routledge.

[2]

Kwangsu Cho and Christian D Schunn. 2007. Scaffolded writing and rewriting in the discipline: A web based reciprocal peer review system. Computers & Education 48, 3 (2007), 409--426.

Digital Library

[3]

Derrick Coetzee, Seongtaek Lim, Armando Fox, Bjorn Hartmann, and Marti A Hearst. 2015. Structuring interactions for large-scale synchronous peer learning. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 1139--1152.

Digital Library

[4]

Catherine H Crouch and Eric Mazur. 2001. Peer instruction: Ten years of experience and results. American journal of physics 69, 9 (2001), 970--977.

[5]

Thomas Erickson and Wendy A Kellogg. 2000. Social translucence: an approach to designing systems that support social processes. ACM transactions on computer-human interaction (TOCHI) 7, 1 (2000), 59--83.

[6]

K Anders Ericsson and Paul Ward. 2007. Capturing the naturally occurring superior performance of experts in the laboratory toward a science of expert and exceptional performance. Current Directions in Psychological Science 16, 6 (2007), 346--350.

[7]

Alvin W Gouldner. 1960. The norm of reciprocity: A preliminary statement. American sociological review (1960), 161--178.

[8]

Yasmine Kotturi, Chinmay E Kulkarni, Michael S Bernstein, and Scott Klemmer. 2015. Structure and messaging techniques for online peer learning systems that increase stickiness. In Proceedings of the Second= (2015) ACM Conference on Learning@ Scale. ACM, 31--38.

Digital Library

[9]

Robert E Kraut, Paul Resnick, Sara Kiesler, Moira Burke, Yan Chen, Niki Kittur, Joseph Konstan, Yuqing Ren, and John Riedl. 2012. Building successful online communities: Evidence-based social design. Mit Press.

[10]

Chinmay Kulkarni, Julia Cambre, Yasmine Kotturi, Michael S Bernstein, and Scott R Klemmer. 2015b. Talkabout: Making distance matter with small groups in massive classes. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 1116--1128.

Digital Library

[11]

Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R Klemmer. 2015c. Peer and self assessment in massive online classes. In Design thinking research. Springer, 131--168.

[12]

Chinmay E Kulkarni, Michael S Bernstein, and Scott R Klemmer. 2015a. PeerStudio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale. ACM, 75--84.

Digital Library

[13]

Tricia Ngoon, Alexander Gamero-Garrido, and Scott Klemmer. 2016. Supporting Peer Instruction with Evidence-Based Online Instructional Templates. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 301--304.

Digital Library

[14]

Leo Porter, Cynthia Bailey Lee, and Beth Simon. 2013. Halving fail rates using peer instruction: a study of four computer science courses. In Proceeding of the 44th ACM technical symposium on Computer science education. ACM, 177--182.

Digital Library

[15]

Todd Rogers and Avi Feller. 2016. Discouraged by peer excellence: Exposure to exemplary peer performance causes quitting. Psychological science 27, 3 (2016), 365--374.

[16]

Donald Schön. 1987. Educating the reflective practitioner. (1987).

[17]

Michelle K Smith, William B Wood, Wendy K Adams, Carl Wieman, Jennifer K Knight, Nancy Guild, and Tin Tin Su. 2009. Why peer discussion improves student performance on in-class concept questions. Science 323, 5910 (2009), 122--124.

[18]

Thomas Staubitz, Dominic Petrick, Matthias Bauer, Jan Renz, and Christoph Meinel. 2016. Improving the Peer Assessment Experience on MOOC Platforms. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 389--398.

Digital Library

[19]

Milena Tsvetkova and Michael W Macy. 2014. The social contagion of generosity. PloS one 9, 2 (2014), e87275.

Cited By

Bucur CKuhn TCeolin Dvan Ossenbruggen J(2023)Nanopublication-based semantic publishing and reviewing: a field study with formalization papersPeerJ Computer Science10.7717/peerj-cs.11599(e1159)Online publication date: 21-Feb-2023
https://doi.org/10.7717/peerj-cs.1159
Morales-Trujillo MGalster M(2023)On Evidence-based Feedback Practices in Software Engineering for Continuous People Improvement2023 IEEE 35th International Conference on Software Engineering Education and Training (CSEE&T)10.1109/CSEET58097.2023.00033(158-162)Online publication date: Aug-2023
https://doi.org/10.1109/CSEET58097.2023.00033
Bawabe SWilson LZhou TMarks EHuang J(2021)The UX Factor: Using Comparative Peer Review to Evaluate Designs through User PreferencesProceedings of the ACM on Human-Computer Interaction10.1145/34798635:CSCW2(1-23)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479863
Show More Cited By

Index Terms

Long-Term Peer Reviewing Effort is Anti-Reciprocal
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education

Recommendations

Improving Assessment on MOOCs Through Peer Identification and Aligned Incentives
L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

Massive Open Online Courses (MOOCs) use peer assessment to grade open ended questions at scale, allowing students to provide feedback. Relative to teacher based grading, peer assessment on MOOCs traditionally delivers lower quality feedback and fewer ...
Peer Grading Without Protest: The SPARK Approach to Summative Peer Assessment
SIGCSE 2022: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1

As Computer Science Professors, we strive to construct courses that maximally support and contribute to student learning through carefully crafted in-class and out-of-class activities. There is evidence that homework enhances student learning, and that ...
Long term study of peer behavior in the KAD DHT

Distributed hash tables (DHTs) have been actively studied in literature and many different proposals have been made on how to organize peers in a DHT. However, very few DHTs have been implemented in real systems and deployed on a large scale. One ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

April 2017

352 pages

ISBN:9781450344500

DOI:10.1145/3051457

General Chair:
Claudia Urrea
Massachusetts Institute of Technology, USA
,
Program Chairs:
Justin Reich
Massachusetts Institute of Technology, USA
,
Candace Thille
Stanford University, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

NSF

Conference

L@S 2017

Sponsor:

ACM

L@S 2017: Fourth (2017) ACM Conference on Learning @ Scale

April 20 - 21, 2017

Massachusetts, Cambridge, USA

Acceptance Rates

L@S '17 Paper Acceptance Rate 14 of 105 submissions, 13%;

Overall Acceptance Rate 117 of 440 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
393
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)1

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bucur CKuhn TCeolin Dvan Ossenbruggen J(2023)Nanopublication-based semantic publishing and reviewing: a field study with formalization papersPeerJ Computer Science10.7717/peerj-cs.11599(e1159)Online publication date: 21-Feb-2023
https://doi.org/10.7717/peerj-cs.1159
Morales-Trujillo MGalster M(2023)On Evidence-based Feedback Practices in Software Engineering for Continuous People Improvement2023 IEEE 35th International Conference on Software Engineering Education and Training (CSEE&T)10.1109/CSEET58097.2023.00033(158-162)Online publication date: Aug-2023
https://doi.org/10.1109/CSEET58097.2023.00033
Bawabe SWilson LZhou TMarks EHuang J(2021)The UX Factor: Using Comparative Peer Review to Evaluate Designs through User PreferencesProceedings of the ACM on Human-Computer Interaction10.1145/34798635:CSCW2(1-23)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479863
Froelich NLiu AShang RXiao ZNeils TFrens JAragon C(2021)Reciprocity in Reviewing on Fanfiction.NetHCI International 2021 - Posters10.1007/978-3-030-78645-8_5(39-44)Online publication date: 3-Jul-2021
https://doi.org/10.1007/978-3-030-78645-8_5
Kulkarni C(2019)Design Perspectives of Learning at ScaleProceedings of the Sixth (2019) ACM Conference on Learning @ Scale10.1145/3330430.3333620(1-11)Online publication date: 24-Jun-2019
https://dl.acm.org/doi/10.1145/3330430.3333620
Seering JMayol RHarpstead EChen TCook AHammer JArnedo JNacke LVanden Abeele VToups Dugas P(2019)Peer Feedback Processes in the Game IndustryProceedings of the Annual Symposium on Computer-Human Interaction in Play10.1145/3311350.3347176(427-438)Online publication date: 17-Oct-2019
https://dl.acm.org/doi/10.1145/3311350.3347176

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents