research-article

Open access

Interactive Layout Transfer

Authors:

Niraj Ramesh Dayama,

Lukas Brückner,

Antti OulasvirtaAuthors Info & Claims

IUI '21: Proceedings of the 26th International Conference on Intelligent User Interfaces

Pages 70 - 80

https://doi.org/10.1145/3397481.3450652

Published: 14 April 2021 Publication History

All formats PDF

Abstract

During the design of graphical user interfaces (GUIs), one typical objective is to ensure compliance with pertinent style guides, ongoing design practices, and design systems. However, designing compliant layouts is challenging, time-consuming, and can distract creative thinking in design. This paper presents a method for interactive layout transfer, where the layout of a source design – typically an initial rough working draft – is transferred automatically using a selected reference/template layout while complying with relevant guidelines. Our integer programming (IP) method extends previous work in two ways: first, by showing how to transform a rough draft into the final target layout using a reference template and, second, by extending IP-based approaches to adhere to guidelines. We demonstrate how to integrate the method into a real-time interactive GUI sketching tool. Evaluation results are presented from a case study and from an online experiment where the perceived quality of layouts was assessed.

References

[1]

Apple. 2020. Human Interface Guidelines. https://developer.apple.com/design/human-interface-guidelines/[Retrieved 9 October, 2020].

[2]

Greg J. Badros, Alan Borning, and Peter J. Stuckey. 2001. The Cassowary Linear Arithmetic Constraint Solving Algorithm. ACM Trans. Comput.-Hum. Interact. 8, 4 (Dec. 2001), 267–306. https://doi.org/10.1145/504704.504705

Digital Library

[3]

Gilles Bailly, Antti Oulasvirta, Timo Kötzing, and Sabrina Hoppe. 2013. MenuOptimizer: Interactive Optimization of Menu Systems. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology(UIST ’13). ACM, New York, NY, USA, 331–342. https://doi.org/10.1145/2501988.2502024

Digital Library

[4]

Rafael Baptista. 2003. Fast Approximate Distance Functions. https://www.flipcode.com/archives/Fast_Approximate_Distance_Functions.shtml. Accessed: 9 October 2020.

[5]

Alan Borning, Richard Lin, and Kim Marriott. 1997. Constraints for the Web. In Proceedings of the Fifth ACM International Conference on Multimedia (Seattle, Washington, USA) (MULTIMEDIA ’97). Association for Computing Machinery, New York, NY, USA, 173–182. https://doi.org/10.1145/266180.266361

Digital Library

[6]

Niraj Ramesh Dayama, Kashyap Todi, Taru Saarelainen, and Antti Oulasvirta. 2020. GRIDS: Interactive Layout Design with Integer Programming. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376553

Digital Library

[7]

William Drenttel and Jessica Helfand. 1999. Method and system for computer screen layout based on a recombinant geometric modular structure. https://patents.google.com/patent/US7124360B1/en

[8]

Michael H Fischer, Richard R Yang, and Monica S Lam. 2020. ImagineNet: Restyling Apps Using Neural Style Transfer. arXiv preprint arXiv:2001.04932(2020).

[9]

Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: Automatically Generating User Interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces(IUI ’04). ACM, New York, NY, USA, 93–100. https://doi.org/10.1145/964442.964461

Digital Library

[10]

Krzysztof Gajos and Daniel S. Weld. 2005. Preference Elicitation for Interface Optimization. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology(UIST ’05). ACM, New York, NY, USA, 173–182. https://doi.org/10.1145/1095034.1095063

Digital Library

[11]

Leon A. Gatys, Alexander S. Ecker, and M. Bethge. 2015. A Neural Algorithm of Artistic Style. ArXiv abs/1508.06576(2015).

[12]

Michael Gleicher. 1992. Briar: A Constraint-based Drawing Program. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’92). ACM, New York, NY, USA, 661–662. https://doi.org/10.1145/142750.143074

Digital Library

[13]

Google. 2020. Material Design. https://material.io/design[Retrieved 9 October, 2020].

[14]

Scarlett R. Herring, Chia-Chen Chang, Jesse Krantzler, and Brian P. Bailey. 2009. Getting Inspired! Understanding How and Why Examples Are Used in Creative Design Practice. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for Computing Machinery, New York, NY, USA, 87–96. https://doi.org/10.1145/1518701.1518717

Digital Library

[15]

Yue Jiang, Wolfgang Stuerzlinger, Matthias Zwicker, and Christof Lutteroth. 2020. ORCSolver: An Efficient Solver for Adaptive GUI Layout with OR-Constraints. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376610

Digital Library

[16]

Janin Koch, Magda Laszlo, Andres Lucero Vera, Antti Oulasvirta, 2018. Surfing for Inspiration: digital inspirational material in design practice. In Design Research Society International Conference: Catalyst. Design Research Society.

[17]

Yuki Koyama, Daisuke Sakamoto, and Takeo Igarashi. 2014. Crowd-Powered Parameter Analysis for Visual Design Exploration. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 65–74. https://doi.org/10.1145/2642918.2647386

Digital Library

[18]

Ranjitha Kumar, Jerry O Talton, Salman Ahmad, and Scott R Klemmer. 2011. Bricolage: example-based retargeting for web design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2197–2206.

Digital Library

[19]

Markku Laine, Ai Nakajima, Niraj Dayama, and Antti Oulasvirta. 2020. Layout as a Service (LaaS): A Service Platform for Self-Optimizing Web Layouts. In Web Engineering, Maria Bielikova, Tommi Mikkonen, and Cesare Pautasso (Eds.). Springer International Publishing, Cham, 19–26. https://doi.org/10.1007/978-3-030-50578-3_2

Digital Library

[20]

Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R Klemmer. 2010. Designing with interactive example galleries. In Proceedings of the SIGCHI conference on human factors in computing systems. 2257–2266.

Digital Library

[21]

Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. 2019. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. https://openreview.net/forum?id=HJxB5sRcFQ

[22]

Xueting Li, Sifei Liu, J. Kautz, and M. Yang. 2018. Learning Linear Transformations for Fast Arbitrary Style Transfer. ArXiv abs/1808.04537(2018).

[23]

Minxuan Lin, Fan Tang, Weiming Dong, Xiao Li, Chongyang Ma, and C. Xu. 2020. Distribution Aligned Multimodal and Multi-Domain Image Stylization. ArXiv abs/2006.01431(2020).

[24]

Simon Lok and Steven Feiner. 2001. A Survey of Automated Layout Techniques for Information Presentations.

[25]

Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017. Deep Photo Style Transfer. arXiv preprint arXiv:1703.07511(2017).

[26]

Jonas Löwgren and Ulrika Laurén. 1993. Supporting the use of guidelines and style guides in professional user interface design. Interacting with Computers 5, 4 (1993), 385 – 396. https://doi.org/10.1016/0953-5438(93)90003-C

[27]

Brad Myers, Scott E. Hudson, and Randy Pausch. 2000. Past, Present, and Future of User Interface Software Tools. ACM Trans. Comput.-Hum. Interact. 7, 1 (March 2000), 3–28. https://doi.org/10.1145/344949.344959

Digital Library

[28]

Jeffrey Nichols, Duen Horng Chau, and Brad A Myers. 2007. Demonstrating the viability of automatically generated user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems. 1283–1292.

Digital Library

[29]

Jeffrey Nichols, Brad A. Myers, and Brandon Rothrock. 2006. UNIFORM: Automatically Generating Consistent Remote Control User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada) (CHI ’06). Association for Computing Machinery, New York, NY, USA, 611–620. https://doi.org/10.1145/1124772.1124865

Digital Library

[30]

Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. DesignScape: Design with Interactive Layout Suggestions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems(CHI ’15). ACM, New York, NY, USA, 1221–1224. https://doi.org/10.1145/2702123.2702149

Digital Library

[31]

Stephen Oney, Brad Myers, and Joel Brandt. 2012. ConstraintJS: Programming Interactive Behaviors for the Web by Integrating Constraints and States. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology(UIST ’12). ACM, New York, NY, USA, 229–238. https://doi.org/10.1145/2380116.2380146

Digital Library

[32]

Antti Oulasvirta, Niraj Ramesh Dayama, Morteza Shiripour, Maximilian John, and Andreas Karrenbauer. 2020. Combinatorial optimization of graphical user interface designs. Proc. IEEE 108, 3 (2020), 434–464.

[33]

E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley. 2001. Color Transfer between Images. IEEE Computer Graphics and Applications 21 (2001), 34–41.

Digital Library

[34]

Mindy Reyes (Ed.). 2019. Hack the design system. Idean Publishing. http://idean.com/learn

[35]

Karri Saarinen. [n.d.]. Building a Visual Language–Behind the scenes of our new design system. https://airbnb.design/building-a-visual-language/. Accessed: 23 April 2020.

[36]

Yi-Chang Shih, Sylvain Paris, Connelly Barnes, W. Freeman, and F. Durand. 2014. Style transfer for headshot portraits. ACM Trans. Graph. 33(2014), 148:1–148:14.

Digital Library

[37]

Kesler Tanner and James Landay. 2019. “I Know It When I See It”: How Experts and Novices Recognize Good Design. Springer International Publishing, Cham, 249–266. https://doi.org/10.1007/978-3-319-97082-0_13

[38]

Ondřej Texler, David Futschik, Michal Kučera, Ondřej Jamriška, Šárka Sochorová, Menglei Chai, Sergey Tulyakov, and Daniel Sýkora. 2020. Interactive Video Stylization Using Few-Shot Patch-Based Training. ACM Transactions on Graphics 39, 4 (2020), 73.

Digital Library

[39]

Kashyap Todi, Jussi Jokinen, Kris Luyten, and Antti Oulasvirta. 2018. Familiarisation: Restructuring Layouts with Visual Learning Models. In 23rd International Conference on Intelligent User Interfaces (Tokyo, Japan) (IUI ’18). Association for Computing Machinery, New York, NY, USA, 547–558. https://doi.org/10.1145/3172944.3172949

Digital Library

[40]

Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and Explore with a Layout Optimiser. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems(DIS ’16). ACM, New York, NY, USA, 543–555. https://doi.org/10.1145/2901790.2901817

Digital Library

[41]

Clemens Zeidler, Christof Lutteroth, and Gerald Weber. 2012. Constraint solving for beautiful user interfaces: how solving strategies support layout aesthetics. In Proceedings of the 13th International Conference of the NZ Chapter of the ACM’s Special Interest Group on Human-Computer Interaction. 72–79.

Digital Library

[42]

Shuyu Zheng, Ziniu Hu, and Yun Ma. 2019. FaceOff: Assisting the Manifestation Design of Web Graphical User Interface. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining(Melbourne VIC, Australia) (WSDM ’19). Association for Computing Machinery, New York, NY, USA, 774–777. https://doi.org/10.1145/3289600.3290610

Digital Library

[43]

Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau. 2019. Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics (TOG) 38, 4 (2019), 133.

Digital Library

Cited By

E JYen YPan ILin GLi MJin HChen MXia HDow S(2024)When to Give Feedback: Exploring Tradeoffs in the Timing of Design FeedbackProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656183(292-310)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656183
Liang XLin T(2024)Sketch2Wireframe: an automatic framework for transforming hand-drawn sketches to digital wireframes in UI designThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-023-03188-z40:9(6615-6625)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s00371-023-03188-z
Brie PBurny NSluÿters AVanderdonckt J(2023)Evaluating a Large Language Model on Searching for GUI LayoutsProceedings of the ACM on Human-Computer Interaction10.1145/35932307:EICS(1-37)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593230
Show More Cited By

Index Terms

Interactive Layout Transfer
1. Human-centered computing

Index terms have been assigned to the content through auto-classification.

Recommendations

Scenario-based interactive UI design
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Clearly picturing user behavior is one of the key requirements when designing successful interactive software. However, covering all possible user behaviors with one UI is a complex challenge. The Scenario-based Interactive UI Design tool is designed to ...
The University of Alberta user interface management system

In this paper the design and implementation of the University of Alberta user interface management system (UIMS) is discussed. This UIMS is based on the Seeheim model of user interfaces, which divides the user interface into three separate components. ...
The University of Alberta user interface management system
SIGGRAPH '85: Proceedings of the 12th annual conference on Computer graphics and interactive techniques

In this paper the design and implementation of the University of Alberta user interface management system (UIMS) is discussed. This UIMS is based on the Seeheim model of user interfaces, which divides the user interface into three separate components. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '21: Proceedings of the 26th International Conference on Intelligent User Interfaces

April 2021

618 pages

ISBN:9781450380171

DOI:10.1145/3397481

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 April 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Academy of Finland

Conference

IUI '21

Sponsor:

IUI '21: 26th International Conference on Intelligent User Interfaces

April 14 - 17, 2021

TX, College Station, USA

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
1,201
Total Downloads

Downloads (Last 12 months)260
Downloads (Last 6 weeks)36

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

E JYen YPan ILin GLi MJin HChen MXia HDow S(2024)When to Give Feedback: Exploring Tradeoffs in the Timing of Design FeedbackProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656183(292-310)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656183
Liang XLin T(2024)Sketch2Wireframe: an automatic framework for transforming hand-drawn sketches to digital wireframes in UI designThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-023-03188-z40:9(6615-6625)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s00371-023-03188-z
Brie PBurny NSluÿters AVanderdonckt J(2023)Evaluating a Large Language Model on Searching for GUI LayoutsProceedings of the ACM on Human-Computer Interaction10.1145/35932307:EICS(1-37)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593230
Warner JKim KHartmann B(2023)Interactive Flexible Style Transfer for Vector GraphicsProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606751(1-14)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606751
S. Kristiansen YGarrison LBruckner S(2022)Content-Driven Layout for Visualization DesignProceedings of the 15th International Symposium on Visual Information Communication and Interaction10.1145/3554944.3554950(1-8)Online publication date: 16-Aug-2022
https://dl.acm.org/doi/10.1145/3554944.3554950
Brückner LLeiva LOulasvirta A(2022)Learning GUI Completions with User-defined ConstraintsACM Transactions on Interactive Intelligent Systems10.1145/349003412:1(1-40)Online publication date: 4-Mar-2022
https://dl.acm.org/doi/10.1145/3490034

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents