research-article

Open access

Juxtapeer: Comparative Peer Review Yields Higher Quality Feedback and Promotes Deeper Reflection

Authors:

Chinmay KulkarniAuthors Info & Claims

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Paper No.: 294, Pages 1 - 13

https://doi.org/10.1145/3173574.3173868

Published: 21 April 2018 Publication History

Abstract

Peer review asks novices to take on an evaluator's role, yet novices often lack the perspective to accurately assess the quality of others' work. To help learners give feedback on their peers' work through an expert lens, we present the Juxtapeer peer review system for structured comparisons. We build on theories of learning through contrasting cases, and contribute the first systematic evaluation of comparative peer review. In a controlled experiment, 476 consenting learners across four courses submitted 1,297 submissions, 4,102 reviews, and 846 self assessments. Learners assigned to compare submissions wrote reviews and self-reflections that were longer and received higher ratings from experts than those who evaluated submissions one at a time. A second study found that a ranking of submissions derived from learners' comparisons correlates well with staff ranking. These results demonstrate that comparing algorithmically-curated pairs of submissions helps learners write better feedback.

Supplementary Material

suppl.mov (pn2815-file5.mp4)

Supplemental video

Download
8.53 MB

References

[1]

Louis Alfieri, Timothy J. Nokes-Malach, and Christian D. Schunn. 2013. Learning Through Case Comparisons: A Meta-Analytic Review. Educational Psychologist 48, 2: 87--113.

[2]

J.R Anderson and G. H Bower. 1972. Recognition and Retrieval Process in Free recall. Psychological review 79, 2: 97--123.

[3]

Anish Athalye. 2016. Gavel. Retrieved from https://github.com/anishathalye/gavel

[4]

Elizabeth L Bjork and Robert A Bjork. 2011. Making Things Hard on Yourself, But in a Good Way: Creating Desirable Difficulties to Enhance Learning. Psychology and the Real World: Essays Illustrating Fundamental Contributions to Society: 55--64.

[5]

David Boud. 1995. Enhancing learning through self-assessment. Routledge.

[6]

David Boud, Ruth Cohen, and Jane Sampson. 2014. Peer learning in higher education: Learning from and with each other. Routledge.

[7]

P A Carlson and F C Berry. 2003. Calibrated Peer Review and assessing learning outcomes. Frontiers in Education Conference 2: 1--6.

[8]

Xi Chen, Paul N. Bennett, Kevyn CollinsThompson, and Eric Horvitz. 2013. Pairwise ranking aggregation in a crowdsourced setting. Proceedings of the Sixth ACM International Conference on Web Search and Data Mining: 193-- 202.

Digital Library

[9]

Michelene T. H. Chi, Paul J. Feltovich, and Robert Glaser. 1981. Categorization and Representation of Physics Problems by Experts and Novices*. Cognitive Science 5, 2: 121--152.

[10]

Kwangsu Cho and Christian D. Schunn. 2007. Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system. Computers and Education 48, 3: 409--426.

Digital Library

[11]

Robert B Cialdini. 2001. Influence: Science and Practice. Book 3rd: 262.

[12]

Steven P. Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L. Schwartz, and Scott R. Klemmer. 2010. Parallel prototyping leads to better design results, more divergence, and increased selfefficacy. ACM Transactions on Computer-Human Interaction 17, 4: 1--24.

Digital Library

[13]

Dedre Gentner, Jeffrey Loewenstein, and Leigh Thompson. 2003. Learning and transfer: A general role for analogical encoding. Journal of Educational Psychology 95, 2: 393--408.

[14]

Dedre Gentner and Arthur B Markman. 1994. Structural Alignment in Comparison: No Difference Without Similarity. Psychological Science 5, 3: 152--159.

[15]

Eleanor Jack Gibson. 1969. Principles of perceptual learning and development. AppletonCentury-Crofts.

[16]

Michael D Greenberg, Matthew W Easterday, and Elizabeth M Gerber. 2015. Critiki: A scaffolded approach to gathering design feedback from paid crowdworkers. C&C 2015: 1--10.

Digital Library

[17]

Brian Greer. 1997. Modelling reality in mathematics classrooms: The case of word problems. Learning and Instruction 7, 4: 293--307.

[18]

Adriaan D De Groot, Fernand Gobet, and Riekent W Jongman. 1996. Perception and memory in chess: Studies in the heuristics of the professional eye. Van Gorcum & Co.

[19]

Catherine M Hicks, Vineet Pandey, C Ailie Fraser, and Scott Klemmer. 2016. Framing Feedback. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16: 458--469.

Digital Library

[20]

David A. Joyner. 2017. Scaling Expert Feedback. Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale - L@S '17: 71--80.

Digital Library

[21]

Philip J. Kellman, Christine M. Massey, and Ji Y. Son. 2010. Perceptual learning modules in mathematics: Enhancing students' pattern recognition, structure extraction, and fluency. Topics in Cognitive Science 2, 2: 285--305.

[22]

Avraham N. Kluger and Angelo DeNisi. 1996. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin 119, 2: 254--284.

[23]

Pushkar Kolhe, Michael L. Littman, and Charles L. Isbell. 2016. Peer Reviewing Short Answers using Comparative Judgement. Proceedings of the Third ACM Conference on Learning @ Scale: 241--244.

Digital Library

[24]

Markus Krause, Tom Garnack, and Steven P. Dow. 2017. Critique Style Guide: Improving Crowdsourced Design Critiques using a Natural Language Model. CHI '17 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, May.

Digital Library

[25]

Chinmay E. Kulkarni, MS Michael S Bernstein, and Scott R. Klemmer. 2015. PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale - L@S '15, 75--84.

Digital Library

[26]

Chinmay E. Kulkarni, Richard Socher, Michael S. Bernstein, and Scott R. Klemmer. 2014. Scaling short-answer grading by combining peer assessment with algorithmic scoring. Proceedings of the first ACM conference on Learning @ scale conference L@S '14: 99--108.

Digital Library

[27]

Chinmay Kulkarni, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R Klemmer. 2013. Peer and Self Assessment in Massive Online Classes. 20, 6.

[28]

Kenneth J. Kurtz, Chun-Hui Miao, and Dedre Gentner. 2001. Learning by Analogical Bootstrapping. Journal of the Learning Sciences 10, 4: 417--446.

[29]

Marianne LaFrance. 1989. The quality of expertise: implications of expert-novice differences for knowledge acquisition. ACM SIGART Bulletin, 108: 6--14.

Digital Library

[30]

Penelope Lockwood and Ziva Kunda. 1997. Superstars and me: Predicting the impact of role models on the self. Journal of Personality and Social Psychology 73, 1: 91--103.

[31]

Jeffrey Loewenstein, Leigh Thompson, and Dedre Gentner. 1999. Analogical encoding facilitates knowledge transfer in negotiation. Psychonomic Bulletin & Review 6, 4: 586--597.

[32]

Kristi Lundstrom and Wendy Baker. 2009. To give is better than to receive: The benefits of peer review to the reviewer's own writing. Journal of Second Language Writing 18, 1: 30--43.

[33]

Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P. Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P. Dow. 2015. Structuring, Aggregating, and Evaluating Crowdsourced Design Critique. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW '15: 473--485.

Digital Library

[34]

Thomas Mussweiler. 2003. Comparison processes in social judgment: Mechanisms and consequences. Psychological Review 110, 3: 472--489.

[35]

Duyen T Nguyen, Thomas R Garncarz, Felicia Ng, Laura Dabbish, and Steven Dow. 2016. Fruitful Feedback: Positive affective language and source anonymity improve critique reception and work outcomes. Manuscript submitted for publication: 1024--1034.

Digital Library

[36]

Laura R Novick. 1988. Analogical transfer, problem similarity, and expertise. Journal of Experimental Psychology: Learning, Memory, and Cognition 14, 3: 510.

[37]

Tiffany Potter, Letitia Englund, James Charbonneau, Mark Thomson MacLean, Jonathan Newell, and Ido Roll. 2017. ComPAIR: A New Online Tool Using Adaptive Comparative Judgement to Support Learning with Peer Feedback. Teaching & Learning Inquiry 5, 2: 89-- 113.

[38]

K Raman and T Joachims. 2014. Methods for ordinal peer grading. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining: 1037--1046.

Digital Library

[39]

Bethany Rittle-Johnson and Jon R. Star. 2007. Does comparing solution methods facilitate conceptual and procedural knowledge? An experimental study on learning to solve equations. Journal of Educational Psychology 99, 3: 561--574.

[40]

T. Rogers and A. Feller. 2016. Discouraged by Peer Excellence: Exposure to Exemplary Peer Performance Causes Quitting. Psychological Science 27, 3: 365--374.

[41]

D.Royce Sadler. 1989. Formative assessment and the design of instructional systems. Instructional Studies 18, 2: 119--144.

[42]

Donald A Schön. 1984. The reflective practitioner: How professionals think in action. Basic books.

[43]

Daniel L. Schwartz, Catherine C. Chase, Marily A. Oppezzo, and Doris B. Chin. 2011. Practicing versus inventing with contrasting cases: The effects of telling first on learning and transfer. Journal of Educational Psychology 103, 4: 759--775.

[44]

Daniel L Schwartz, Jessica M Tsang, and Kristen P Blair. 2016. The ABCs of How We Learn: 26 Scientifically Proven Approaches, How They Work, and When to Use Them. WW Norton & Company.

[45]

Nihar B Shah, Joseph K Bradley, Abhay Parekh, Martin Wainwright, and Kannan Ramchandran. 2013. A Case for Ordinal Peer-evaluation in MOOCs. NIPS Workshop on Data Driven Education: 1--8. Retrieved from http://lytics.stanford.edu/datadriveneducation/paper s/shahetal.pdf

[46]

Nancy Sommers. 1982. Responding to student writing. College composition and communication 33, 2: 148--156.

[47]

Leigh Thompson, Dedre Gentner, and Jeffrey Loewenstein. 2000. Avoiding Missed Opportunities in Managerial Life: Analogical Training More Powerful Than Individual Case Training. Organizational Behavior and Human Decision Processes 82, 1: 60--75.

[48]

David Tinapple, Loren Olson, and John Sadauskas. 2013. CritViz: Web-Based Software Supporting Peer Critique in Large Creative Classrooms. Bulletin of the IEEE Technical Committee on Learning Technology 15, 1: 29--35. Retrieved from http://lttf.ieee.org/issues/january2013/Tinapple.pdf

[49]

Maryam Tohidi, William Buxton, Ronald Baecker, and Abigail Sellen. 2006. Getting the right design and the design right. Proceedings of ACM CHI 2006 Conference on Human Factors in Computing Systems 1: 1243--1252.

Digital Library

[50]

Amos Tversky and Itamar Simonson. 1993. Context-dependent preferences. Management science 39, 10: 1179--1189.

[51]

A.E.a Waters, D.b Tinapple, and R.G.a Baraniuk. 2015. BayesRank: A bayesian approach to ranked peer grading. L@S 2015 - 2nd ACM Conference on Learning at Scale: 177--183.

Digital Library

[52]

David Scott Yeager, Valerie Purdie-Vaughns, Julio Garcia, Nancy Apfel, Patti Brzustoski, Allison Master, William T. Hessert, Matthew E. Williams, and Geoffrey L. Cohen. 2014. Breaking the cycle of mistrust: Wise interventions to provide critical feedback across the racial divide. Journal of Experimental Psychology: General 143, 2: 804-- 824.

[53]

A Yuan, K Luther, and M Krause. 2016. Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques. (CSCW'16) Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing: 1005-- 1017.

Digital Library

Cited By

Nguyen HHa Van L(2024)Challenges and Implications of Blended Learning Adoption in the Age of Communication Technology: A Case Study of Vietnamese Universities During the COVID-19 PandemicrEFLections10.61508/refl.v31i3.27677731:3Online publication date: 11-Nov-2024
https://doi.org/10.61508/refl.v31i3.276777
Palea DVadhul GLee D(2024)Annota: Peer-based AI Hints Towards Learning Qualitative Coding at ScaleProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645168(455-470)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645168
Yen YE JJin HLi MLin GPan IDow S(2024)ProcessGallery: Contrasting Early and Late Iterations for Design Principle LearningProceedings of the ACM on Human-Computer Interaction10.1145/36373898:CSCW1(1-35)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637389
Show More Cited By

Index Terms

Juxtapeer: Comparative Peer Review Yields Higher Quality Feedback and Promotes Deeper Reflection
1. Applied computing
  1. Education

Recommendations

Enhance Students' Computing Skills via Web-Mediated Self-Regulated Learning with Feedback in Blended Environment

The vocational colleges in Taiwan regard professional certifications as a badge of skills achievement. To enhance student learning in this specific context, the authors conducted a quasi-experiment to explore effects of web-mediated self-regulated ...
Understanding the Effect of In-Video Prompting on Learners and Instructors
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Online instructional videos are ubiquitous, but it is difficult for instructors to gauge learners' experience and their level of comprehension or confusion regarding the lecture video. Moreover, learners watching the videos may become disengaged or fail ...
A hybrid model for an e-learning system which develops metacognitive skills at students
HCI'13: Proceedings of the 15th international conference on Human Interface and the Management of Information: information and interaction for learning, culture, collaboration and business - Volume Part III

One of the goals of academic education is to train students in view of acquiring metacognitive skills, thus becoming self-regulated learners prepared for for lifelong learning. The research in this area has proved that metacognition can be taught and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

April 2018

8489 pages

ISBN:9781450356206

DOI:10.1145/3173574

General Chairs:
Regan Mandryk
University of Saskatchewan, Canada
,
Mark Hancock
University of Waterloo, Canada
,
Program Chairs:
Mark Perry
Brunel University London, UK
,
Anna Cox
University College London, UK

Copyright © 2018 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 April 2018

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '18

Sponsor:

SIGCHI

CHI '18: CHI Conference on Human Factors in Computing Systems

April 21 - 26, 2018

Montreal QC, Canada

Acceptance Rates

CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

46
Total Citations
View Citations
1,927
Total Downloads

Downloads (Last 12 months)257
Downloads (Last 6 weeks)17

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Nguyen HHa Van L(2024)Challenges and Implications of Blended Learning Adoption in the Age of Communication Technology: A Case Study of Vietnamese Universities During the COVID-19 PandemicrEFLections10.61508/refl.v31i3.27677731:3Online publication date: 11-Nov-2024
https://doi.org/10.61508/refl.v31i3.276777
Palea DVadhul GLee D(2024)Annota: Peer-based AI Hints Towards Learning Qualitative Coding at ScaleProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645168(455-470)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645168
Yen YE JJin HLi MLin GPan IDow S(2024)ProcessGallery: Contrasting Early and Late Iterations for Design Principle LearningProceedings of the ACM on Human-Computer Interaction10.1145/36373898:CSCW1(1-35)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637389
Zhou THuang JChan G(2024)Epigraphics: Message-Driven Infographics AuthoringProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642172(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642172
Beaudouin-Lafon ME JXia H(2023)Color Field: Developing Professional Vision by Visualizing the Effects of Color FiltersProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606828(1-16)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606828
Crain PLee JYen YKim JAiello ABailey B(2023)Visualizing Topics and Opinions Helps Students Interpret Large Collections of Peer Feedback for Creative ProjectsACM Transactions on Computer-Human Interaction10.1145/357181730:3(1-30)Online publication date: 10-Jun-2023
https://dl.acm.org/doi/10.1145/3571817
Chen TLiu MDing EO'Neil EAgarwal MKraut RDabbish L(2023)Facilitating Counselor Reflective Learning with a Real-time Annotation toolProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581551(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581551
Sato ASramek ZYatani K(2023)Groupnamics: Designing an Interface for Overviewing and Managing Parallel Group Discussions in an Online ClassroomProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581322(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581322
Zhang YSchunn C(2023)Self-regulation of peer feedback quality aspects through different dimensions of experience within prior peer feedback assignmentsContemporary Educational Psychology10.1016/j.cedpsych.2023.10221074(102210)Online publication date: Jul-2023
https://doi.org/10.1016/j.cedpsych.2023.102210
Khosravi HDenny PMoore SStamper J(2023)Learnersourcing in the age of AI: Student, educator and machine partnerships for content creationComputers and Education: Artificial Intelligence10.1016/j.caeai.2023.1001515(100151)Online publication date: 2023
https://doi.org/10.1016/j.caeai.2023.100151
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents