demonstration

CritiqueKit: A Mixed-Initiative, Real-Time Interface For Improving Feedback

Authors:

Scott KlemmerAuthors Info & Claims

UIST '17 Adjunct: Adjunct Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

Pages 7 - 9

https://doi.org/10.1145/3131785.3131791

Published: 20 October 2017 Publication History

Get Access

Abstract

We present CritiqueKit, a mixed-initiative machine-learning system that helps students give better feedback to peers by reusing prior feedback, reducing it to be useful in a general context, and retraining the system about what is useful in real time. CritiqueKit exploits the fact that novices often make similar errors, leading reviewers to reuse the same feedback on many different submissions. It takes advantage of all prior feedback, and classifies feedback as the reviewer types it. CritiqueKit continually updates the corpus of feedback with new comments that are added, and it guides reviewers to improve their feedback, and thus the entire corpus, over time.

Supplementary Material

suppl.mov (uistde112.mp4)

Supplemental video

Download
13.75 MB

References

[1]

2017. Gradescope. (2017). https://gradescope.com

Google Scholar

[2]

Michael D. Greenberg, Matthew W. Easterday, and Elizabeth M. Gerber. 2015. Critiki: A Scaffolded Approach to Gathering Design Feedback from Paid Crowdworkers. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition - C&C '15. 235--244.

Digital Library

Google Scholar

[3]

J. Hattie and H. Timperley. 2007. The Power of Feedback. Review of Educational Research 77, 1 (mar 2007), 81--112.

Crossref

Google Scholar

[4]

David Kelley and Tom Kelley. 2013. Creative confidence: Unleashing the creative potential within us all. Crown Publishing Group, New York. 288 pages.

Google Scholar

[5]

Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R. Klemmer. 2013. Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction 20, 6 (Dec 2013), 1--31.

Digital Library

Google Scholar

[6]

Chinmay E. Kulkarni, Michael S. Bernstein, and Scott R. Klemmer. 2015. PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale - L@S '15. 75--84.

Digital Library

Google Scholar

[7]

Huy Nguyen, Wenting Xiong, and Diane Litman. 2016. Instant Feedback for Increasing the Presence of Solutions in Peer Reviews. HLT-NAACL Demos (2016), 6--10.

Google Scholar

[8]

D. Royce Sadler. 1989. Formative assessment and the design of instructional systems. Instructional Science 18, 2 (Jun 1989), 119--144.

Crossref

Google Scholar

[9]

Donald A. Schön. 1985. The design studio: An exploration of its traditions and potentials. RIBA Publications for RIBA Building Industry Trust. 99 pages.

Google Scholar

[10]

W Xiong, D. Litman, and C. Schunn. 2012. Natural Language Processing techniques for researching and improving peer feedback. Journal of Writing Research 4, 2 (Nov 2012), 155--176.

Crossref

Google Scholar

[11]

Alvin Yuan, Kurt Luther, Markus Krause, Sophie Isabel Vennix, Steven P Dow, and Björn Hartmann. 2016. Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing - CSCW '16. 1003--1015.

Digital Library

Google Scholar

Cited By

View all

Palani SRamos G(2024)Evolving Roles and Workflows of Creative Practitioners in the Age of Generative AIProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656190(170-184)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656190
Duan PWarner JLi YHartmann B(2024)Generating Automatic Feedback on UI Mockups with Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642782(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642782
Švábenský VVykopal JČeleda PDovjak J(2024)Automated feedback for participants of hands-on cybersecurity trainingEducation and Information Technologies10.1007/s10639-023-12265-829:9(11555-11584)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10639-023-12265-8
Show More Cited By

Index Terms

CritiqueKit: A Mixed-Initiative, Real-Time Interface For Improving Feedback
1. Human-centered computing

Recommendations

Interactive Guidance Techniques for Improving Creative Feedback
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Good feedback is critical to creativity and learning, yet rare. Many people do not know how to actually provide effective feedback. There is increasing demand for quality feedback -- and thus feedback givers -- in learning and professional settings. This ...
Voyant: generating structured feedback on visual designs using a crowd of non-experts
CSCW '14: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing

Feedback on designs is critical for helping users iterate toward effective solutions. This paper presents Voyant, a novel system giving users access to a non-expert crowd to receive perception-oriented feedback on their designs from a selected audience. ...
Paragon: An Online Gallery for Enhancing Design Feedback with Visual Examples
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Examples provide a source of inspiration for creating designs, but can they help improve the feedback process? Supplementing design feedback with examples could help recipients see issues clearly, identify concrete steps for improvement, and integrate ...

Comments

Information & Contributors

Information

Published In

UIST '17 Adjunct: Adjunct Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

October 2017

217 pages

ISBN:9781450354196

DOI:10.1145/3131785

General Chair:
Krzysztof Gajos
Harvard University
,
Program Chairs:
Jennifer Mankoff
Carnegie Mellon University
,
Chris Harrison
Carnegie Mellon University

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2017

Check for updates

Author Tags

Qualifiers

Demonstration

Funding Sources

Jacobs School of Engineering, University of California, San Diego

Conference

UIST '17

Sponsor:

UIST '17: The 30th Annual ACM Symposium on User Interface Software and Technology

October 22 - 25, 2017

QC, Québec City, Canada

Acceptance Rates

UIST '17 Adjunct Paper Acceptance Rate 73 of 324 submissions, 23%;

Overall Acceptance Rate 355 of 1,733 submissions, 20%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
340
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)4

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Palani SRamos G(2024)Evolving Roles and Workflows of Creative Practitioners in the Age of Generative AIProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656190(170-184)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656190
Duan PWarner JLi YHartmann B(2024)Generating Automatic Feedback on UI Mockups with Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642782(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642782
Švábenský VVykopal JČeleda PDovjak J(2024)Automated feedback for participants of hands-on cybersecurity trainingEducation and Information Technologies10.1007/s10639-023-12265-829:9(11555-11584)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10639-023-12265-8
Warner JPavel ANguyen TAgrawala MHartmann B(2023)SlideSpecs: Automatic and Interactive Presentation Feedback CollationProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584035(695-709)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584035
Chintalapati SBragg JWang L(2022)A Dataset of Alt Texts from HCI PublicationsProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544796(1-12)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3544796
Palani SLedo DFitzmaurice GAnderson F(2022)”I don’t want to feel like I’m working in a 1960s factory”: The Practitioner Perspective on Creativity Support Tool AdoptionProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501933(1-18)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501933
Lou YGao WChen PLiu XYang CSun L(2022)DesignEva: A Design-Supported Tool with Multi-faceted Perceptual EvaluationCross-Cultural Design. Interaction Design Across Cultures10.1007/978-3-031-06038-0_38(508-519)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-06038-0_38
Shi WJagannadharao ALee JBailey B(2021)Challenges and Opportunities for Data-Centric Peer Evaluation Tools for TeamworkProceedings of the ACM on Human-Computer Interaction10.1145/34795765:CSCW2(1-20)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479576
Britain GJain ALupfer NKerne APerrine ASeo JSungkajun ABernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Design is (A)live: An Environment Integrating Ideation and AssessmentExtended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3334480.3382947(1-8)Online publication date: 25-Apr-2020
https://dl.acm.org/doi/10.1145/3334480.3382947
Yen YKim JBailey BBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Decipher: An Interactive Visualization Tool for Interpreting Unstructured Design Feedback from Multiple ProvidersProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376380(1-13)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376380
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Interactive Guidance Techniques for Improving Creative Feedback

Voyant: generating structured feedback on visual designs using a crowd of non-experts

Paragon: An Online Gallery for Enhancing Design Feedback with Visual Examples