When to Give Feedback: Exploring Tradeoffs in the Timing of Design Feedback
Authors: 
Jane L. E, Yu-Chun Grace Yen, Isabelle Yan Pan, Grace Lin, Mingyi Li, Hyoungwook Jin, Mengyi Chen, 
Haijun Xia, 
Steven P. DowAuthors Info & Claims
Jane L. E, Yu-Chun Grace Yen, Isabelle Yan Pan, Grace Lin, Mingyi Li, Hyoungwook Jin, Mengyi Chen, Haijun Xia, Steven P. DowAuthors Info & ClaimsPages 292 - 310
Abstract
Advances in AI have opened up the potential for creativity tools to computationally generate design feedback. In a future when designers can request feedback anytime on demand, how would the timing of these requests impact novices’ creative learning processes? What are the tradeoffs of providing access to feedback throughout a design task (in-action) versus only providing feedback after (on-action)? We explored these questions through a Wizard-of-Oz study (N=20) using an interactive design probe, where participants could request feedback either throughout the design process or only after they complete a full draft. We found that in-action participants frequently request feedback, resulting in better improvements as indicated by a greater decrease in issues in their final design. However, we saw that in-action feedback can also risk users overly relying on feedback instead of engaging in more holistic self-evaluation. We discuss the implications of our insights on designing tools for creative feedback.
References
[1]
2022. CorelDRAW Graphics Suite. https://www.coreldraw.com/en/tips/graphic-design-principles/index.html
[2]
Susan A Ambrose, Michael W Bridges, Michele DiPietro, Marsha C Lovett, and Marie K Norman. 2010. How learning works: Seven research-based principles for smart teaching. John Wiley & Sons.
[3]
Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–13. https://doi.org/10.1145/3290605.3300233
[4]
Lorin W Anderson and David R Krathwohl. 2001. A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Longman.
[5]
Frederik Anseel, Filip Lievens, and Eveline Schollaert. 2009. Reflection as a strategy to enhance task performance after feedback. Organizational Behavior and Human Decision Processes 110, 1 (2009), 23–35. https://doi.org/10.1016/j.obhdp.2009.05.003
[6]
Hal R Arkes and Catherine Blumer. 1985. The psychology of sunk cost. Organizational behavior and human decision processes 35, 1 (1985), 124–140.
[7]
Terry Barrett. 1988. A comparison of the goals of studio professors conducting critiques and art education goals for teaching criticism. Studies in art education 30, 1 (1988), 22–27.
[8]
Leopold Bayerlein. 2014. Students’ feedback preferences: how do students react to timely and automatically generated assessment feedback?Assessment & Evaluation in Higher Education 39, 8 (2014), 916–931. https://doi.org/10.1080/02602938.2013.870531
[9]
Natalie C Benda, Laurie L Novak, Carrie Reale, and Jessica S Ancker. 2022. Trust in AI: why we should be designing for APPROPRIATE reliance. Journal of the American Medical Informatics Association 29, 1 (2022), 207–212.
[10]
Michaela Benk, Suzanne Tolmeijer, Florian von Wangenheim, and Andrea Ferrario. 2022. The Value of Measuring Trust in AI-A Socio-Technical System Perspective. arXiv preprint arXiv:2204.13480 (2022).
[11]
Aditya Bharadwaj, Pao Siangliulue, Adam Marcus, and Kurt Luther. 2019. Critter: Augmenting creative work with dynamic checklists, automated quality assurance, and contextual reviewer feedback. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12. https://doi.org/10.1145/3290605.3300769
[12]
Oloff C Biermann, Ning F Ma, and Dongwook Yoon. 2022. From tool to companion: Storywriters want AI writers to respect their personal values and writing strategies. In Designing Interactive Systems Conference. 1209–1227. https://doi.org/10.1145/3532106.3533506
[13]
Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–21. https://doi.org/10.1145/3449287
[14]
David Carless. 2006. Differing perceptions in the feedback process. Studies in higher education 31, 2 (2006), 219–233.
[15]
Alberto AP Cattaneo and Elisa Motta. 2021. “I reflect, therefore I am… a good professional”. On the relationship between reflection-on-action, reflection-in-action and professional performance in vocational education. Vocations and Learning 14, 2 (2021), 185–204. https://doi.org/10.1007/s12186-020-09259-9
[16]
Ruijia Cheng, Ziwen Zeng, Maysnow Liu, and Steven Dow. 2020. Critique Me: Exploring How Creators Publicly Request Feedback in an Online Critique Community. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1–24. https://doi.org/10.1145/3415232
[17]
Erin Cherry and Celine Latulipe. 2014. Quantifying the creativity support of digital tools through the creativity support index. ACM Transactions on Computer-Human Interaction (TOCHI) 21, 4 (2014), 1–25. https://doi.org/10.1145/2617588
[18]
Kwangsu Cho, Christian D Schunn, and Davida Charney. 2006. Commenting on writing: Typology and perceived helpfulness of comments from novice peer reviewers and subject matter experts. Written communication 23, 3 (2006), 260–294.
[19]
John Joon Young Chung and Eytan Adar. 2023. Artinter: AI-powered Boundary Objects for Commissioning Visual Arts(DIS ’23). Association for Computing Machinery, New York, NY, USA, 1997–2018. https://doi.org/10.1145/3563657.3595961
[20]
Maria Cutumisu and Daniel L. Schwartz. 2018. The impact of critical feedback choice on students’ revision, performance, learning, and memory. Computers in Human Behavior 78 (2018), 351–367. https://doi.org/10.1016/j.chb.2017.06.029
[21]
Maria Cutumisu and Daniel L Schwartz. 2018. The impact of critical feedback choice on students’ revision, performance, learning, and memory. Computers in Human Behavior 78 (2018), 351–367. https://doi.org/10.1016/j.chb.2017.06.029
[22]
Deanna Dannels, Amy Housley Gaffney, and Kelly Norris Martin. 2008. Beyond Content, Deeper than Delivery: What Critique Feedback Reveals about Communication Expectations in Design Education.International Journal for the Scholarship of teaching and Learning 2, 2 (2008), n2. https://doi.org/10.20429/ijsotl.2008.020212
[23]
Deanna P Dannels and Kelly Norris Martin. 2008. Critiquing critiques: A genre analysis of feedback across novice to expert design studios. Journal of Business and Technical Communication 22, 2 (2008), 135–159. https://doi.org/10.1177/1050651907311923
[24]
Niraj Ramesh Dayama, Simo Santala, Lukas Brückner, Kashyap Todi, Jingzhou Du, and Antti Oulasvirta. 2021. Interactive layout transfer. In 26th International Conference on Intelligent User Interfaces. 70–80. https://doi.org/10.1145/3397481.3450652
[25]
Elly De Bruijn and Yvonne Leeman. 2011. Authentic and self-directed learning in vocational education: Challenges to vocational educators. Teaching and Teacher Education 27, 4 (2011), 694–702. https://doi.org/10.1016/j.tate.2010.11.007
[26]
John Dewey. 2022. How we think. DigiCat.
[27]
Donis A Dondis. 1974. A primer of visual literacy. Mit Press.
[28]
Steven Dow, Julie Fortuna, Dan Schwartz, Beth Altringer, Daniel Schwartz, and Scott Klemmer. 2011. Prototyping dynamics: sharing multiple designs improves exploration, group rapport, and results. In Proceedings of the SIGCHI conference on human factors in computing systems. 2807–2816. https://doi.org/10.1145/1978942.1979359
[29]
Steven Dow, Elizabeth Gerber, and Audris Wong. 2013. A pilot study of using crowds in the classroom. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 227–236. https://doi.org/10.1145/2470654.2470686
[30]
Steven P Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L Schwartz, and Scott R Klemmer. 2010. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 4 (2010), 1–24. https://doi.org/10.1145/1879831.1879836
[31]
Steven P Dow, Kate Heddleston, and Scott R Klemmer. 2009. The efficacy of prototyping under time constraints. In Proceedings of the seventh ACM conference on Creativity and cognition. 165–174. https://doi.org/10.1145/1640233.1640260
[32]
Peitong Duan, Jeremy Warner, Yang Li, and Bjoern Hartmann. 2024. Generating Automatic Feedback on UI Mockups with Large Language Models. arXiv preprint arXiv:2403.13139 (2024). https://doi.org/arXiv.2403.13139
[33]
Jane L. E, Ohad Fried, and Maneesh Agrawala. 2019. Optimizing Portrait Lighting at Capture-Time Using a 360 Camera as a Light Probe. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. https://doi.org/10.1145/3332165.3347893
[34]
Jane L. E, Ohad Fried, Jingwan Lu, Jianming Zhang, Radomír Mech, Jose Echevarria, Pat Hanrahan, James A Landay, 2020. Adaptive photographic composition guidance. In CHI. https://doi.org/10.1145/3313831.3376635
[35]
Jane L E, Kevin Y. Zhai, Jose Echevarria, Ohad Fried, Pat Hanrahan, and James A. Landay. 2021. Dynamic Guidance for Decluttering Photographic Compositions. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 359–371. https://doi.org/10.1145/3472749.3474755
[36]
Sean B Eom, H Joseph Wen, and Nicholas Ashill. 2006. The determinants of students’ perceived learning outcomes and satisfaction in university online education: An empirical investigation. Decision Sciences Journal of Innovative Education 4, 2 (2006), 215–235.
[37]
K Anders Ericsson, Ralf T Krampe, and Clemens Tesch-Römer. 1993. The role of deliberate practice in the acquisition of expert performance.Psychological review 100, 3 (1993), 363.
[38]
Gerhard Fischer, Kumiyo Nakakoji, Jonathan Ostwald, Gerry Stahl, and Tamara Sumner. 1993. Embedding critics in design environments. The knowledge engineering review 8, 4 (1993), 285–307. https://doi.org/10.1017/S026988890000031X
[39]
Eureka Foong, Steven P Dow, Brian P Bailey, and Elizabeth M Gerber. 2017. Online feedback exchange: A framework for understanding the socio-psychological factors. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 4454–4467. https://doi.org/10.1145/3025453.3025791
[40]
Eureka Foong, Darren Gergle, and Elizabeth M Gerber. 2017. Novice and expert sensemaking of crowdsourced design feedback. Proceedings of the ACM on Human-Computer Interaction 1, CSCW (2017), 1–18.
[41]
Corey Ford and Nick Bryan-Kinns. 2023. Towards a Reflection in Creative Experience Questionnaire. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–16. https://doi.org/10.1145/3544548.3581077
[42]
Mary Forehand. 2010. Bloom’s taxonomy. Emerging perspectives on learning, teaching, and technology 41, 4 (2010), 47–56.
[43]
Krzysztof Z Gajos and Lena Mamykina. 2022. Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning. In 27th International Conference on Intelligent User Interfaces. 794–806. https://doi.org/10.1145/3490099.3511138
[44]
W Timothy Gallwey. 2014. The inner game of tennis: The classic guide to the mental side of peak performance. Macmillan.
[45]
William W Gaver. 1996. Situating action II: Affordances for interaction: The social is material for design. Ecological psychology 8, 2 (1996), 111–129.
[46]
Elizabeth Gerber and Maureen Carroll. 2012. The psychological experience of prototyping. Design studies 33, 1 (2012), 64–84.
[47]
Katy Ilonka Gero and Lydia B Chilton. 2019. Metaphoria: An algorithmic companion for metaphor creation. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–12. https://doi.org/10.1145/3290605.3300526
[48]
Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for science writing using language models. In Designing interactive systems conference. 1002–1019. https://doi.org/10.1145/3532106.3533533
[49]
Rex Hartson. 2003. Cognitive, physical, sensory, and functional affordances in interaction design. Behaviour & information technology 22, 5 (2003), 315–338.
[50]
James Hollan, Edwin Hutchins, and David Kirsh. 2000. Distributed cognition: toward a new foundation for human-computer interaction research. ACM Transactions on Computer-Human Interaction (TOCHI) 7, 2 (2000), 174–196. https://doi.org/10.1145/3313831.3376327
[51]
Jim Hollan and Scott Stornetta. 1992. Beyond being there. In Proceedings of the SIGCHI conference on Human factors in computing systems. 119–125. https://doi.org/10.1145/142750.142769
[52]
David G Jansson and Steven M Smith. 1991. Design fixation. Design studies 12, 1 (1991), 3–11. https://doi.org/10.1016/0142-694X(91)90003-F
[53]
Pratyusha Kalluri. 2020. Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583, 7815 (2020), 169–169. https://doi.org/10.1038/d41586-020-02003-
[54]
Joy Kim, Maneesh Agrawala, and Michael S Bernstein. 2017. Mosaic: designing online creative communities for sharing works-in-progress. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 246–258. https://doi.org/10.1145/2998181.2998195
[55]
David Kirsh and Paul Maglio. 1994. On distinguishing epistemic from pragmatic action. Cognitive science 18, 4 (1994), 513–549.
[56]
David A Kolb. 1984. Experience as the source of learning and development. Upper Sadle River: Prentice Hall (1984).
[57]
Yasmine Kotturi and McKayla Kingston. 2019. Why do Designers in the "Wild" Wait to Seek Feedback until Later in their Design Process? In Proceedings of the 2019 on Creativity and Cognition. 541–546. https://doi.org/10.1145/3325480.3326580
[58]
Yuki Koyama and Masataka Goto. 2022. BO as Assistant: Using Bayesian Optimization for Asynchronously Generating Design Suggestions. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–14. https://doi.org/10.1145/3526113.3545664
[59]
Markus Krause, Tom Garncarz, JiaoJiao Song, Elizabeth M Gerber, Brian P Bailey, and Steven P Dow. 2017. Critique style guide: Improving crowdsourced design feedback with a natural language model. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 4627–4639. https://doi.org/10.1145/3025453.3025883
[60]
Sneha R Krishna Kumaran, Wenxuan Wendy Shi, and Brian P Bailey. 2021. Am I Ready to Get Feedback? A Taxonomy of Factors Creators Consider Before Seeking Feedback on In-Progress Creative Work. In Creativity and Cognition. 1–10. https://doi.org/10.1145/3450741.3465255
[61]
Sneha R. Krishna Kumaran, Yue Yin, and Brian P. Bailey. 2021. Plan Early, Revise More: Effects of Goal Setting and Perceived Role of the Feedback Provider on Feedback Seeking Behavior. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 24 (apr 2021), 22 pages. https://doi.org/10.1145/3449098
[62]
Chinmay Kulkarni, Steven P Dow, and Scott R Klemmer. 2014. Early and repeated exposure to examples improves creative work. In Design thinking research. Springer, 49–62.
[63]
Chinmay E Kulkarni, Michael S Bernstein, and Scott R Klemmer. 2015. PeerStudio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the second (2015) ACM conference on learning@ scale. 75–84. https://doi.org/10.1145/2724660.2724670
[64]
Chunggi Lee, Sanghoon Kim, Dongyun Han, Hongjun Yang, Young-Woo Park, Bum Chul Kwon, and Sungahn Ko. 2020. GUIComp: A GUI design assistant with real-time, multi-faceted feedback. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–13. https://doi.org/10.1145/3313831.3376327
[65]
Fritz Lekschas, Spyridon Ampanavos, Pao Siangliulue, Hanspeter Pfister, and Krzysztof Z Gajos. 2021. Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–12. https://doi.org/10.1145/3411764.3445507
[66]
Jingyi Li, Eric Rawn, Jacob Ritchie, Jasper Tran O’Leary, and Sean Follmer. 2023. Beyond the Artifact: Power as a Lens for Creativity Support Tools. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–15. https://doi.org/10.1145/3586183.3606831
[67]
JiayiZhou Li, Junxiu Tang, Tan Tang, Haotian Li, Weiwei Cui, Yingcaui Wu, 2024. Understanding Nonlinear Collaboration between Human and AI Agents: A Co-design Framework for Creative Design. arXiv preprint arXiv:2401.07312 (2024). https://doi.org/10.48550/arXiv.2401.07312
[68]
William Lidwell, Kritina Holden, and Jill Butler. 2010. Universal principles of design, revised and updated: 125 ways to enhance usability, influence perception, increase appeal, make better design decisions, and teach through design. Rockport Pub.
[69]
J John Loughran. 2002. Developing reflective practice: Learning about teaching and learning through modelling. Routledge. https://doi.org/10.4324/9780203453995
[70]
Andrés Lucero. 2015. Using affinity diagrams to evaluate interactive prototypes. In Human-Computer Interaction–INTERACT 2015: 15th IFIP TC 13 International Conference, Bamberg, Germany, September 14-18, 2015, Proceedings, Part II 15. Springer, 231–248. https://doi.org/10.1007/978-3-319-22668-2_19
[71]
Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P Dow. 2015. Structuring, aggregating, and evaluating crowdsourced design critique. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing. 473–485. https://doi.org/10.1145/2675133.2675283
[72]
Stephen MacNeil, Zijian Ding, Kexin Quan, Thomas j Parashos, Yajie Sun, and Steven P Dow. 2021. Framing Creative Work: Helping Novices Frame Better Problems through Interactive Scaffolding. In Creativity and Cognition. 1–10. https://doi.org/10.1145/3450741.3465261
[73]
Yaoli Mao, Janet Rafner, Yi Wang, and Jacob Sherson. 2023. A hybrid intelligence approach to training generative design assistants: partnership between human experts and AI enhanced co-creative tools. In HHAI 2023: Augmenting Human Intellect. IOS Press, 108–123. https://doi.org/10.3233/FAIA230078
[74]
Jennifer Marlow and Laura Dabbish. 2014. From rookie to all-star: professional development in a graphic design social networking site. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 922–933. https://doi.org/10.1145/2531602.2531651
[75]
Tuuli Mattelmäki 2006. Design probes. Aalto University.
[76]
David Maulsby, Saul Greenberg, and Richard Mander. 1993. Prototyping an intelligent agent through Wizard of Oz. In Proceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 277–284. https://doi.org/10.1145/169059.169215
[77]
Thomas P Moran and John M Carroll. 2020. Design rationale: Concepts, techniques, and use. CRC Press.
[78]
Roxana Moreno. 2004. Decreasing cognitive load for novice students: Effects of explanatory versus corrective feedback in discovery-based multimedia. Instructional science 32, 1 (2004), 99–113. https://doi.org/10.1023/B:TRUC.0000021811.66966.1d
[79]
Tricia J Ngoon, C Ailie Fraser, Ariel S Weingarten, Mira Dontcheva, and Scott Klemmer. 2018. Interactive guidance techniques for improving creative feedback. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–11. https://doi.org/10.1145/3173574.3173629
[80]
Tricia J Ngoon, Joy O Kim, and Scott Klemmer. 2021. Shöwn: Adaptive Conceptual Guidance Aids Example Use in Creative Tasks. In Designing Interactive Systems Conference 2021. 1834–1845. https://doi.org/10.1145/3461778.3462072
[81]
Thi Thao Duyen T Nguyen, Thomas Garncarz, Felicia Ng, Laura A Dabbish, and Steven P Dow. 2017. Fruitful Feedback: Positive affective language and source anonymity improve critique reception and work outcomes. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing. 1024–1034. https://doi.org/10.1145/2998181.2998319
[82]
David J Nicol and Debra Macfarlane-Dick. 2006. Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in higher education 31, 2 (2006), 199–218.
[83]
Don Norman. 2004. Affordances and design. Unpublished article, available online at: http://www. jnd. org/dn. mss/affordances-and-design. html (2004).
[84]
Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. Designscape: Design with interactive layout suggestions. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 1221–1224. https://doi.org/10.1145/2702123.2702149
[85]
Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018. I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13. https://doi.org/10.1145/3173574.3174223
[86]
Jon L Pierce, Tatiana Kostova, and Kurt T Dirks. 2001. Toward a theory of psychological ownership in organizations. Academy of management review 26, 2 (2001), 298–310.
[87]
Garr Reynolds. 2011. Presentation Zen: Simple ideas on presentation design and delivery. New Riders.
[88]
Kathleen Dudden Rowlands. 2007. Check it out! Using checklists to support student learning. English Journal (2007), 61–66.
[89]
D Royce Sadler. 1989. Formative assessment and the design of instructional systems. Instructional science 18, 2 (1989), 119–144. https://doi.org/10.1007/BF00117714
[90]
Arvind Satyanarayan and Graham M. Jones. 2024. Intelligence as Agency: Evaluating the Capacity of Generative AI to Empower or Constrain Human Action. An MIT Exploration of Generative AI (mar 27 2024). https://mit-genai.pubpub.org/pub/94y6e0f8.
[91]
Harmen Schaap, Liesbeth Baartman, and Elly De Bruijn. 2012. Students’ learning processes during school-based learning and workplace learning in vocational education: A review. Vocations and learning 5 (2012), 99–117. https://doi.org/10.1007/s12186-011-9069-2
[92]
Roger C Schank, Tamara R Berman, and Kimberli A Macpherson. 1999. Learning by doing. Instructional-design theories and models: A new paradigm of instructional theory 2, 2 (1999), 161–181. https://doi.org/10.1016/0364-0213(94)90007-8
[93]
A Schön, Donald. 1984. The reflective practitioner: How professionals think in action. Basic Books.
[94]
Moushumi Sharmin and Brian P Bailey. 2013. ReflectionSpace: an interactive visualization tool for supporting reflection-on-action in design. In Proceedings of the 9th ACM Conference on Creativity & Cognition. 83–92. https://doi.org/10.1145/2466627.2466645
[95]
Ben Shneiderman. 2007. Creativity support tools: accelerating discovery and innovation. Commun. ACM 50, 12 (2007), 20–32. https://doi.org/10.1145/1323688.1323689
[96]
Masaki Suwa and Barbara Tversky. 2002. External representations contribute to the dynamic construction of ideas. In Diagrammatic Representation and Inference: Second International Conference, Diagrams 2002 Callaway Gardens, GA, USA, April 18–20, 2002 Proceedings 2. Springer, 341–343. https://doi.org/10.1007/3-540-46037-3_33
[97]
Michael Terry, Elizabeth D Mynatt, Kumiyo Nakakoji, and Yasuhiro Yamamoto. 2004. Variation in element and action: supporting simultaneous development of alternative solutions. In Proceedings of the SIGCHI conference on Human factors in computing systems. 711–718. https://doi.org/10.1145/985692.985782
[98]
Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and explore with a layout optimiser. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. 543–555. https://doi.org/10.1145/2901790.2901817
[99]
Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael Bernstein, and Ranjay Krishna. 2022. Explanations Can Reduce Overreliance on AI Systems During Decision-Making. arXiv preprint arXiv:2212.06823 (2022).
[100]
Jeremy Warner, Shuyao Zhou, and Björn Hartmann. 2023. Interactively Optimizing Layout Transfer for Vector Graphics. In Proceedings of the AI & HCI Workshop at the 40th International Conference on Machine Learning (ICML).
[101]
Robin Williams. 2015. The non-designer’s design book: Design and typographic principles for the visual novice. Pearson Education.
[102]
Naomi E. Winstone, Robert A. Nash, James Rowntree, and Michael Parker. 2017. ‘It’d be useful, but I wouldn’t use it’: barriers to university students’ feedback seeking and recipience. Studies in Higher Education 42, 11 (2017), 2026–2041. https://doi.org/10.1080/03075079.2015.1130032 arXiv:https://doi.org/10.1080/03075079.2015.1130032
[103]
Jun Xiao, John Stasko, and Richard Catrambone. 2004. An Empirical Study of the Effect of Agent Competence on User Performance and Perception. In Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems - Volume 1 (New York, New York) (AAMAS ’04). IEEE Computer Society, USA, 178–185.
[104]
Anbang Xu, Shih-Wen Huang, and Brian Bailey. 2014. Voyant: generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 1433–1444. https://doi.org/10.1145/2531602.2531604
[105]
Kenta Yamamoto, Yuki Koyama, and Yoichi Ochiai. 2022. Photographic Lighting Design with Photographer-in-the-Loop Bayesian Optimization. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–11. https://doi.org/10.1145/3526113.3545690
[106]
Yu-Chun Yen, Steven P Dow, Elizabeth Gerber, and Brian P Bailey. 2016. Social network, web forum, or task market? Comparing different crowd genres for design feedback exchange. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. 773–784. https://doi.org/10.1145/2901790.2901820
[107]
Yu-Chun Grace Yen, Steven P Dow, Elizabeth Gerber, and Brian P Bailey. 2017. Listen to others, listen to yourself: Combining feedback review and reflection to improve iterative design. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition. 158–170. https://doi.org/10.1145/3059454.3059468
[108]
Yu-Chun Grace Yen, Joy O Kim, and Brian P Bailey. 2020. Decipher: an interactive visualization tool for interpreting unstructured design feedback from multiple providers. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13. https://doi.org/10.1145/3313831.3376380
[109]
Alvin Yuan, Kurt Luther, Markus Krause, Sophie Isabel Vennix, Steven P Dow, and Bjorn Hartmann. 2016. Almost an expert: The effects of rubrics and expertise on perceived value of crowdsourced design critiques. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. 1005–1017. https://doi.org/10.1145/2818048.2819953
Index Terms
- When to Give Feedback: Exploring Tradeoffs in the Timing of Design Feedback
Recommendations
Feedback Conversations: Creating Feedback Dialogues with a New Textual Tool for Industrial Design Student Feedback
In this paper, the authors describe how a study of a large database of written university teacher feedback in the department of Industrial Design led to the development of a new conceptual framework for feedback and the design of a new feedback tool. ...
Exploration-exploitation tradeoff in interactive relevance feedback
CIKM '10: Proceedings of the 19th ACM international conference on Information and knowledge managementWe study an interesting optimization problem in interactive feedback that aims at optimizing the tradeoff between presenting search results with the highest immediate utility to a user (but not necessarily most useful for collecting feedback information)...
Effects of digital video-based feedback environments on pre-service teachers’ feedback competence
AbstractThe present study investigates the added value of blended digital video-based feedback environments in fostering pre-service teachers' feedback competence on teaching during a practicum. Pre-service teachers practised providing their ...
Highlights- Effects of face-to-face and digital video-based feedback environments were compared.
Comments
Information & Contributors
Information
Published In
June 2024
718 pages
ISBN:9798400704857
DOI:10.1145/3635636
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 23 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
C&C '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 108 of 371 submissions, 29%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 312Total Downloads
- Downloads (Last 12 months)312
- Downloads (Last 6 weeks)51
Reflects downloads up to 09 Jan 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in