research-article

Interactive Guidance Techniques for Improving Creative Feedback

Authors:

Tricia J. Ngoon,

C. Ailie Fraser,

Ariel S. Weingarten,

Mira Dontcheva,

Scott KlemmerAuthors Info & Claims

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Paper No.: 55, Pages 1 - 11

https://doi.org/10.1145/3173574.3173629

Published: 19 April 2018 Publication History

Abstract

Good feedback is critical to creativity and learning, yet rare. Many people do not know how to actually provide effective feedback. There is increasing demand for quality feedback -- and thus feedback givers -- in learning and professional settings. This paper contributes empirical evidence that two interactive techniques -- reusable suggestions and adaptive guidance -- can improve feedback on creative work. We present these techniques embodied in the CritiqueKit system to help reviewers give specific, actionable, and justified feedback. Two real-world deployment studies and two controlled experiments with CritiqueKit found that adaptively-presented suggestions improve the quality of feedback from novice reviewers. Reviewers also reported that suggestions and guidance helped them describe their thoughts and reminded them to provide effective feedback.

Supplementary Material

suppl.mov (pn1362-file3.mp4)

Supplemental video

Download
32.73 MB

References

[1]

Heidi Andrade. 2005. Teaching with Rubrics: The Good, the Bad, and the Ugly. College Teaching 53, 1: 27--30.

[2]

Benjamin S. Bloom. 1984. The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Educational Researcher 13, 6:4-- 16.

[3]

Michael Brooks, Sumit Basu, Charles Jacobs, and Lucy Vanderwende. 2014. Divide and Correct: Using Clusters to Grade Short Answers at Scale. In Proceedings of the first ACM conference on Learning@ scale conference.

Digital Library

[4]

Kwangsu Cho, Christian D Schunn, and Davida Charney. 2006. Commenting on Writing Typology and Perceived Helpfulness of Comments from Novice Peer Reviewers and Subject Matter Experts. Written Communication 23, 3: 260--294.

[5]

Steven P. Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L. Schwartz, and Scott Klemmer. 2010. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction 17, 4: 1--24.

Digital Library

[6]

Steven P Dow, Anand Kulkarni, Scott R Klemmer, and Björn Hartmann. 2012. Shepherding the Crowd Yields Better Work. In Proceedings of the ACM conference on Computer Supported Cooperative Work, 1013--1022.

Digital Library

[7]

Ethan Fast, Colleen Lee, Alex Aiken, Michael S Bernstein, Daphne Koller, and Eric Smith. 2013. Crowdscale Interactive Formal Reasoning and Analytics. In Proceedings of the 26th annual ACM symposium on User interface software and technology.

Digital Library

[8]

Graham Gibbs and Claire Simpson. 2004. Conditions under which assessment supports students' learning Learning and Teaching in Higher Education. Learning and Teaching in Higher Education 1, 1: 3--31.

[9]

Sarah Gielen, Elien Peeters, Filip Dochy, Patrick Onghena, and Katrien Struyven. 2010. Improving the effectiveness of peer feedback for learning. Learning and Instruction 20, 4: 304--315.

[10]

Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip J Guo, and Robert C Miller. 2015. OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale. ACM Transactions on Computer-Human Interaction (TOCHI) 22, 2.

Digital Library

[11]

Elena L Glassman, Aaron Lin, Carrie J Cai, and Robert C Miller. 2016. Learnersourcing Personalized Hints. Proceedings of the 19th ACM Conference on ComputerSupported Cooperative Work & Social Computing.

Digital Library

[12]

Michael D Greenberg, Matthew W Easterday, and Elizabeth M Gerber. 2015. Critiki: A Scaffolded Approach to Gathering Design Feedback from Paid Crowdworkers. In Proceedings of the ACM SIGCHI Conference on Creativity and Cognition.

Digital Library

[13]

Björn Hartmann, Daniel Macdougall, Joel Brandt, and Scott R Klemmer. 2010. What Would Other Programmers Do? Suggesting Solutions to Error Messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.

Digital Library

[14]

John Hattie and Helen Timperley. 2007. The Power of Feedback. Review of Educational Research 77, 1: 81-- 112.

[15]

Andrew Head, Elena Glassman, Gustavo Soares, Ryo Suzuki, Lucas Figueredo, and Loris D 'Antoni. 2017. Writing Reusable Code Feedback at Scale with MixedInitiative Program Synthesis. In Proceedings of the Fourth ACM Conference on Learning@ Scale, 89--98.

Digital Library

[16]

Marti Hearst. 2009. Search User Interfaces. Cambridge University Press.

Digital Library

[17]

Catherine M Hicks, Vineet Pandey, C Ailie Fraser, and Scott Klemmer. Framing Feedback: Choosing Review Environment Features that Support High Quality Peer Assessment. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems.

Digital Library

[18]

David Kelley and Tom Kelley. 2013. Creative confidence?: unleashing the creative potential within us all. Crown Publishing Group.

[19]

Markus Krause, Tom Garncarz, Jiaojiao Song, Elizabeth M Gerber, Brian P Bailey, and Steven P Dow. 2017. Critique Style Guide?: Improving Crowdsourced Design Feedback with a Natural Language Model. In Proceedings of the CHI Conference on Human Factors in Computing Systems, 4627--4639.

Digital Library

[20]

Chinmay Kulkarni, Michael S Bernstein, and Scott Klemmer. 2015. PeerStudio?: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings of the Second ACM Conference on Learning@ Scale, 75--84.

Digital Library

[21]

Chinmay Kulkarni, Steven P Dow, and Scott R Klemmer. 2012. Early and Repeated Exposure to Examples Improves Creative Work. In Proceedings of the Cognitive Science Society.

[22]

Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R Klemmer. 2013. Peer and self assessment in massive online classes. ACM Transactions on ComputerHuman Interaction (TOCHI) 20, 6: 1--31.

Digital Library

[23]

Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R Klemmer. 2010. Designing with Interactive Example Galleries. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.

Digital Library

[24]

Lan Li, Xiongyi Liu, and Allen L. Steckelberg. 2010. Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology 41, 3: 525--536.

[25]

Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P Dow. 2015. Structuring, Aggregating, and Evaluating Crowdsourced Design Critique. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing.

Digital Library

[26]

Richard L. Marsh, Joshua D. Landau, and Jason L. Hicks. 1996. How examples may (and may not) constrain creativity. Memory & Cognition 24, 5: 669--680.

[27]

Yang Miao, Richard Badger, and Yu Zhen. 2006. A comparative study of peer and teacher feedback in a Chinese EFL writing class. Journal of Second Language Writing 15, 3: 179--200.

[28]

Susanne Narciss and Katja Huth. 2006. Fostering achievement and motivation with bug-related tutoring feedback in a computer-based training for written subtraction. Learning and Instruction 16, 4: 310--322.

[29]

Huy Nguyen, Wenting Xiong, and Diane Litman. 2016. Instant Feedback for Increasing the Presence of Solutions in Peer Reviews. In HLT-NAACL Demos, 6--10.

[30]

David J. Nicol and Debra Macfarlane-Dick. 2006. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education 31, 2: 199--218.

[31]

David J Nicol and Debra Macfarlane-dick. 2006. Formative assessment and self-regulated learning?: A model and seven principles of good feedback practice. Formative assessment and self-regulated learning?: A model and seven principles of good feedback practice. Studies in Higher Education, 31, 2: 199--218.

[32]

Arkalgud Ramaprasad. 1983. On the definition of feedback. Behavioral Science 28, 1: 4--13.

[33]

Brian J. Reiser. 2004. Scaffolding Complex Learning: The Mechanisms of Structuring and Problematizing Student Work. Journal of the Learning Sciences 13, 3: 273--304.

[34]

Rod D Roscoe, Laura K Allen, Jennifer L Weston, Scott A Crossley, and Danielle S Mcnamara. 2014. The Writing Pal Intelligent Tutoring System: Usability Testing and Development. Computers and Composition 34: 39--59.

[35]

D. Royce Sadler. 1989. Formative assessment and the design of instructional systems. Instrcutional Science 18: 119--144.

[36]

Donald A. Schon. 1984. The Reflective Practitioner: How Professionals Think In Action. Basic Books.

[37]

Amir Shareghi Najar, Antonija Mitrovic, and Bruce M Mclaren. 2014. Adaptive Support versus Alternating Worked Examples and Tutored Problems: Which Leads to Better Learning? LNCS 8538: 171--182.

[38]

Ben Shneiderman. 2007. Creativity Support Tools: Accelerating Discovery and Innovation. Communications of the ACM 50, 12.

Digital Library

[39]

Arjun Singh, Sergey Karayev, Kevin Gutowski, and Pieter Abbeel. 2017. Gradescope: a Fast, Flexible, and Fair System for Scalable Assessment of Handwritten Work. In Proceedings of the Fourth ACM Conference on Learning@ Scale, 81--88.

Digital Library

[40]

Nancy Sommers. 1982. Responding to Student Writing. College Composition and Communication 3311062, 2.

[41]

Maryam Tohidi, William Buxton, Ronald Baecker, and Abigail Sellen. 2006. Getting the Right Design and the Design Right?: Testing Many Is Better Than One. In Proceedings of the SIGCHI conference on Human Factors in computing systems, 1243--1252.

Digital Library

[42]

Keith Topping. 1998. Peer assessment between students in colleges and universities. Review of Educational Research 68, 3: 249--276.

[43]

Sheng-Chau Tseng and Chin-Chung Tsai. 2007. On-line peer assessment and the role of the peer feedback: A study of high school computer course. Computers & Education 49: 1161--1174.

Digital Library

[44]

Sara Värlander. 2008. The role of students' emotions in formal feedback situations. Teaching in Higher Education 13, 2: 145--156.

[45]

Wenting Xiong, Diane Litman, and Christian D. Schunn. 2012. Natural Language Processing techniques for researching and improving peer feedback. Journal of Writing Research 4, 2: 155--176.

[46]

Alvin Yuan, Kurt Luther, Markus Krause, Sophie Isabel Vennix, Steven P Dow, and Bjorn Hartmann. 2016. Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques. Proceedings of the 19th ACM Conference on ComputerSupported Cooperative Work & Social Computing: 1005--1017.

Digital Library

[47]

2017. Turnitin Feedback Studio. Retrieved from http://turnitin.com

Cited By

Peng ZChen QShen ZMa XOulasvirta A(2024)DesignQuizzer: A Community-Powered Conversational Agent for Learning Visual DesignProceedings of the ACM on Human-Computer Interaction10.1145/36373218:CSCW1(1-40)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637321
E JYen YPan ILin GLi MJin HChen MXia HDow S(2024)When to Give Feedback: Exploring Tradeoffs in the Timing of Design FeedbackProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656183(292-310)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656183
Duan PWarner JLi YHartmann B(2024)Generating Automatic Feedback on UI Mockups with Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642782(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642782
Show More Cited By

Index Terms

Interactive Guidance Techniques for Improving Creative Feedback
1. Human-centered computing

Recommendations

CritiqueKit: A Mixed-Initiative, Real-Time Interface For Improving Feedback
UIST '17 Adjunct: Adjunct Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

We present CritiqueKit, a mixed-initiative machine-learning system that helps students give better feedback to peers by reusing prior feedback, reducing it to be useful in a general context, and retraining the system about what is useful in real time. ...
Paragon: An Online Gallery for Enhancing Design Feedback with Visual Examples
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Examples provide a source of inspiration for creating designs, but can they help improve the feedback process? Supplementing design feedback with examples could help recipients see issues clearly, identify concrete steps for improvement, and integrate ...
Exploration-exploitation tradeoff in interactive relevance feedback
CIKM '10: Proceedings of the 19th ACM international conference on Information and knowledge management

We study an interesting optimization problem in interactive feedback that aims at optimizing the tradeoff between presenting search results with the highest immediate utility to a user (but not necessarily most useful for collecting feedback information)...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

April 2018

8489 pages

ISBN:9781450356206

DOI:10.1145/3173574

General Chairs:
Regan Mandryk
University of Saskatchewan, Canada
,
Mark Hancock
University of Waterloo, Canada
,
Program Chairs:
Mark Perry
Brunel University London, UK
,
Anna Cox
University College London, UK

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article

Funding Sources

Adobe Research Fellowship

Conference

CHI '18

Sponsor:

SIGCHI

CHI '18: CHI Conference on Human Factors in Computing Systems

April 21 - 26, 2018

Montreal QC, Canada

Acceptance Rates

CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

33
Total Citations
View Citations
1,497
Total Downloads

Downloads (Last 12 months)202
Downloads (Last 6 weeks)12

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Peng ZChen QShen ZMa XOulasvirta A(2024)DesignQuizzer: A Community-Powered Conversational Agent for Learning Visual DesignProceedings of the ACM on Human-Computer Interaction10.1145/36373218:CSCW1(1-40)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637321
E JYen YPan ILin GLi MJin HChen MXia HDow S(2024)When to Give Feedback: Exploring Tradeoffs in the Timing of Design FeedbackProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656183(292-310)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656183
Duan PWarner JLi YHartmann B(2024)Generating Automatic Feedback on UI Mockups with Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642782(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642782
Pang RSanty SJust RReinecke K(2024)BLIP: Facilitating the Exploration of Undesirable Consequences of Digital TechnologiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642054(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642054
Liao YJiang YChen ZSuleiman B(2024)FeedbackPulse: GPT-Enabled Feedback Assistant for Software Engineering Educators2024 36th International Conference on Software Engineering Education and Training (CSEE&T)10.1109/CSEET62301.2024.10662985(1-2)Online publication date: 29-Jul-2024
https://doi.org/10.1109/CSEET62301.2024.10662985
Jiang YBenatallah BBáez M(2024)Understanding how early-stage researchers leverage socio-technical affordances for distributed research supportInformation and Software Technology10.1016/j.infsof.2023.107340165:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.infsof.2023.107340
Beaudouin-Lafon ME JXia H(2023)Color Field: Developing Professional Vision by Visualizing the Effects of Color FiltersProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606828(1-16)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606828
Kretzer FMaedche A(2023)Making Usability Test Data Actionable! A Quantitative Test-Driven Prototyping ApproachExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585659(1-6)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585659
Cheng RFrens J(2022)Feedback Exchange and Online Affinity: A Case Study of Online Fanfiction WritersProceedings of the ACM on Human-Computer Interaction10.1145/35551276:CSCW2(1-29)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555127
Chung JHe SAdar E(2022)Artist Support Networks: Implications for Future Creativity Support ToolsProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533505(232-246)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533505
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents