poster

Generative Image AI Using Design Sketches as input: Opportunities and Challenges

Authors:

Chengzhi Zhang,

Nikolas Martelaro,

Daragh ByrneAuthors Info & Claims

C&C '23: Proceedings of the 15th Conference on Creativity and Cognition

Pages 254 - 261

https://doi.org/10.1145/3591196.3596820

Published: 19 June 2023 Publication History

Abstract

Generative image AI has sparked heated discussion among creative professionals. However, how it might be a tool for design practice is understudied, and it is not yet understood how designers may find image AI helpful across the stages of design practice. To address this, our preliminary study explores how designers use generative image AI accompanied by design sketches to inform early-stage 3D design. Further, we also examine the perceived limitations of text-to-image models. To do this, we recruited 11 Architecture graduate students with a median work experience of 2 years. Participants completed a design task using generative image AI packages and incorporated design sketches as inputs. The study findings provide insights into how image AI can or can not be a valuable resource for architectural design practitioners. Further, findings suggest possible directions for future image AI-assisted design tools and workflows.

References

[1]

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 (2022).

[2]

Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. 2022. GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images. https://doi.org/10.48550/arXiv.2209.11163 arXiv:2209.11163 [cs].

[3]

Shunan Guo, Zhuochen Jin, Fuling Sun, Jingwen Li, Zhaorui Li, Yang Shi, and Nan Cao. 2021. Vinci: An Intelligent Graphic Design System for Generating Advertising Posters. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–17. https://doi.org/10.1145/3411764.3445117

Digital Library

[4]

Melissa Heikkiläarchive. 2022. This artist is dominating ai-generated art. and he’s not happy about it.

[5]

Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems(CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–19. https://doi.org/10.1145/3491102.3502030

Digital Library

[6]

Vivian Liu and Lydia B. Chilton. 2021. Design Guidelines for Prompt Engineering Text-to-Image Generative Models. https://doi.org/10.48550/arXiv.2109.06977 arXiv:2109.06977 [cs].

[7]

Vivian Liu, Han Qiao, and Lydia Chilton. 2022. Opal: Multimodal Image Generation for News Illustration. https://doi.org/10.48550/arXiv.2204.09007 arXiv:2204.09007 [cs].

[8]

Michael Muller, Lydia B Chilton, Anna Kantosalo, Q Vera Liao, Mary Lou Maher, Charles Patrick Martin, and Greg Walsh. 2023. GenAICHI 2023: Generative AI and HCI at CHI 2023. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1–7.

Digital Library

[9]

Han Qiao, Vivian Liu, and Lydia Chilton. 2022. Initial Images: Using Image Prompts to Improve Subject Representation in Multimodal AI Generated Art. In Creativity and Cognition. 15–28.

[10]

Sarah Shaffi. 2023. ‘It’s the opposite of art’: why illustrators are furious about AI. The Guardian (Jan. 2023). https://www.theguardian.com/artanddesign/2023/jan/23/its-the-opposite-of-art-why-illustrators-are-furious-about-ai

[11]

Amy Smith, Hope Schroeder, Ziv Epstein, Michael Cook, Simon Colton, and Andrew Lippman. 2023. Trash to Treasure: Using text-to-image models to inform the design of physical artefacts. https://doi.org/10.48550/arXiv.2302.00561 arXiv:2302.00561 [cs].

[12]

Amy Smith, Hope Schroeder, Ziv Epstein, Michael Cook, Simon Colton, and Andrew Lippman. 2023. Trash to Treasure: Using text-to-image models to inform the design of physical artefacts. https://doi.org/10.48550/arXiv.2302.00561 arXiv:2302.00561 [cs].

[13]

Justin D. Weisz, Michael Muller, Jessica He, and Stephanie Houde. 2023. Toward General Design Principles for Generative AI Applications. https://doi.org/10.48550/arXiv.2301.05578 arXiv:2301.05578 [cs].

[14]

Paulina Yurman and Anuradha Venugopal Reddy. 2022. Drawing Conversations Mediated by AI. In Creativity and Cognition(C&C ’22). Association for Computing Machinery, New York, NY, USA, 56–70. https://doi.org/10.1145/3527927.3531448

Digital Library

[15]

Lvmin Zhang and Maneesh Agrawala. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. https://doi.org/10.48550/arXiv.2302.05543 arXiv:2302.05543 [cs].

[16]

Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. 2021. GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 748–760. https://doi.org/10.1109/ICSE43902.2021.00074 ISSN: 1558-1225.

Digital Library

Cited By

Yıldızoğlu N(2024)Sketching Versus Digital Design Tools in Architectural DesignJournal of Computational Design10.53710/jcode.15049475:2(301-316)Online publication date: 30-Sep-2024
https://doi.org/10.53710/jcode.1504947
Chen FMai MHuang XLi Y(2024)Enhancing the Sustainability of AI Technology in Architectural Design: Improving the Matching Accuracy of Chinese-Style BuildingsSustainability10.3390/su1619841416:19(8414)Online publication date: 27-Sep-2024
https://doi.org/10.3390/su16198414
Jin YYoon JSelf JLee K(2024)Understanding Fashion Designers’ Behavior Using Generative AI for Early-Stage Concept Ideation and RevisionArchives of Design Research10.15187/adr.2024.07.37.3.2537:3(25-45)Online publication date: 31-Jul-2024
https://doi.org/10.15187/adr.2024.07.37.3.25
Show More Cited By

Index Terms

Generative Image AI Using Design Sketches as input: Opportunities and Challenges
1. Applied computing
  1. Arts and humanities
    1. Architecture (buildings)
      1. Computer-aided design
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI

Recommendations

Architectural Drawing Using Pencil Sketches and AutoCAD
Design Challenges for Reconnecting in Later Life: A Qualitative Study
DIS '18 Companion: Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems

Friendships and social interactions are renown contributors to wellbeing. As such, keeping a healthy amount of relationships becomes very important as people age and the size of their social network tends to decrease. In this paper, we take a step back ...
Re-shaping Participation in the Field of Architecture through Service Design. Challenges in Knowledge Transfer Regarding End-user Needs in Interdisciplinary Design Processes Involving Architect and Service Design Expertise
PDC '24: Proceedings of the Participatory Design Conference 2024: Full Papers - Volume 1

This paper aims to understand the phenomenon of knowledge transfer between architects and service designers during participatory design processes in architectural design projects. While the relationship between planners and end-users in participatory ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

C&C '23: Proceedings of the 15th Conference on Creativity and Cognition

June 2023

564 pages

ISBN:9798400701801

DOI:10.1145/3591196

Copyright © 2023 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 June 2023

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

C&C '23

Sponsor:

SIGCHI

C&C '23: Creativity and Cognition

June 19 - 21, 2023

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 108 of 371 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
1,413
Total Downloads

Downloads (Last 12 months)891
Downloads (Last 6 weeks)68

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yıldızoğlu N(2024)Sketching Versus Digital Design Tools in Architectural DesignJournal of Computational Design10.53710/jcode.15049475:2(301-316)Online publication date: 30-Sep-2024
https://doi.org/10.53710/jcode.1504947
Chen FMai MHuang XLi Y(2024)Enhancing the Sustainability of AI Technology in Architectural Design: Improving the Matching Accuracy of Chinese-Style BuildingsSustainability10.3390/su1619841416:19(8414)Online publication date: 27-Sep-2024
https://doi.org/10.3390/su16198414
Jin YYoon JSelf JLee K(2024)Understanding Fashion Designers’ Behavior Using Generative AI for Early-Stage Concept Ideation and RevisionArchives of Design Research10.15187/adr.2024.07.37.3.2537:3(25-45)Online publication date: 31-Jul-2024
https://doi.org/10.15187/adr.2024.07.37.3.25
Park HEirich JLuckow ASedlmair M(2024)"We Are Visual Thinkers, Not Verbal Thinkers!": A Thematic Analysis of How Professional Designers Use Generative AI Image Generation ToolsProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685370(1-14)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3679318.3685370
Antony VHuang C(2024)ID.8: Co-Creating Visual Stories with Generative AIACM Transactions on Interactive Intelligent Systems10.1145/367227714:3(1-29)Online publication date: 15-Jun-2024
https://dl.acm.org/doi/10.1145/3672277
Ganguly AYan CChung JSun TKiheon YGingold YHong S(2024)ShadowMagic: Designing Human-AI Collaborative Support for Comic Professionals’ ShadowingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676332(1-15)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676332
Andrew SBishop CTigwell G(2024)Light and Dark ModeProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435398:1(1-23)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643539
Jahanlou AChilana P(2024)How Example-Based Authoring of Motion Graphics Impacts Creative Expression: Differences in Perceptions of Professional and Casual Motion DesignersProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656197(347-357)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656197
Palani SRamos G(2024)Evolving Roles and Workflows of Creative Practitioners in the Age of Generative AIProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656190(170-184)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656190
Han YQiu ZCheng JLC R(2024)When Teams Embrace AI: Human Collaboration Strategies in Generative Prompting in a Creative Design TaskProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642133(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642133
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents