research-article

Open access

ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints

Authors:

Wolfgang Stuerzlinger,

Christof LutterothAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 316, Pages 1 - 18

https://doi.org/10.1145/3411764.3445043

Published: 07 May 2021 Publication History

All formats PDF

Abstract

Reverse engineering (RE) of user interfaces (UIs) plays an important role in software evolution. However, the large diversity of UI technologies and the need for UIs to be resizable make this challenging. We propose ReverseORC, a novel RE approach able to discover diverse layout types and their dynamic resizing behaviours independently of their implementation, and to specify them by using OR constraints. Unlike previous RE approaches, ReverseORC infers flexible layout constraint specifications by sampling UIs at different sizes and analyzing the differences between them. It can create specifications that replicate even some non-standard layout managers with complex dynamic layout behaviours. We demonstrate that ReverseORC works across different platforms with very different layout approaches, e.g., for GUIs as well as for the Web. Furthermore, it can be used to detect and fix problems in legacy UIs, extend UIs with enhanced layout behaviours, and support the creation of flexible UI layouts.

Supplementary Material

VTT File (3411764.3445043_videofigurecaptions.vtt)

Download
6.44 KB

VTT File (3411764.3445043_videopreviewcaptions.vtt)

Download
.69 KB

MP4 File (3411764.3445043_videofigure.mp4)

Supplemental video

Download
16.65 MB

MP4 File (3411764.3445043_videopreview.mp4)

Preview video

Download
2.77 MB

References

[1]

Robert St Amant, Mark O Riedl, Frank E Ritter, and Andrew Reifers. 2005. Image Processing in Cognitive Models with SegMan. In Proceedings of the 11th International Conference on Human-Computer Interaction, HCII, Vol. 2005.

[2]

Pavol Bielik, Marc Fischer, and Martin Vechev. 2018. Robust Relational Layout Synthesis from Examples for Android. Proc. ACM Program. Lang. 2, OOPSLA, Article 156 (Oct. 2018), 29 pages. https://doi.org/10.1145/3276526

Digital Library

[3]

Michael Bolin, Matthew Webber, Philip Rha, Tom Wilson, and Robert C. Miller. 2005. Automation and Customization of Rendered Web Pages. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (Seattle, WA, USA) (UIST ’05). Association for Computing Machinery, New York, NY, USA, 163–172. https://doi.org/10.1145/1095034.1095062

Digital Library

[4]

Alan Borning, Richard Kuang-Hsu Lin, and Kim Marriott. 2000. Constraint-based document layout for the Web. Multimedia systems 8, 3 (2000), 177–189.

[5]

Deng Cai, Shipeng Yu, Ji-Rong Wen, and Wei-Ying Ma. 2003. VIPS: a Vision-based Page Segmentation Algorithm. Technical Report MSR-TR-2003-79. 28 pages. https://www.microsoft.com/en-us/research/publication/vips-a-vision-based-page-segmentation-algorithm/

[6]

Tsung-Hsiang Chang, Tom Yeh, and Robert C. Miller. 2010. GUI Testing Using Computer Vision. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 1535–1544. https://doi.org/10.1145/1753326.1753555

Digital Library

[7]

Yu Chen, Xing Xie, Wei-Ying Ma, and Hong-Jiang Zhang. 2005. Adapting web pages for small-screen devices. IEEE internet computing 9, 1 (2005), 50–56.

[8]

Morgan Dixon and James Fogarty. 2010. Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 1525–1534. https://doi.org/10.1145/1753326.1753554

Digital Library

[9]

Morgan Dixon, James Fogarty, and Jacob Wobbrock. 2012. A General-Purpose Target-Aware Pointing Enhancement Using Pixel-Level Analysis of Graphical Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York, NY, USA, 3167–3176. https://doi.org/10.1145/2207676.2208734

Digital Library

[10]

Morgan Dixon, Gierad Laput, and James Fogarty. 2014. Pixel-Based Methods for Widget State and Style in a Runtime Implementation of Sliding Widgets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 2231–2240. https://doi.org/10.1145/2556288.2556979

Digital Library

[11]

Morgan Dixon, Daniel Leventhal, and James Fogarty. 2011. Content and Hierarchy in Pixel-Based Methods for Reverse Engineering Interface Structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 969–978. https://doi.org/10.1145/1978942.1979086

Digital Library

[12]

Morgan Dixon, Alexander Nied, and James Fogarty. 2014. Prefab Layers and Prefab Annotations: Extensible Pixel-Based Interpretation of Graphical Interfaces. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 221–230. https://doi.org/10.1145/2642918.2647412

Digital Library

[13]

Carmel Domshlak, Samir Genaim, and Ronen Brafman. 2000. Preference-based configuration of web page content. In 14th European Conference on Artificial Intelligence (ECAI 2000), Configuration Workshop, Berlin, Germany. 19–22.

[14]

W. Keith Edwards, Scott E. Hudson, Joshua Marinacci, Roy Rodenstein, Thomas Rodriguez, and Ian Smith. 1997. Systematic Output Modification in a 2D User Interface Toolkit. In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology (Banff, Alberta, Canada) (UIST ’97). Association for Computing Machinery, New York, NY, USA, 151–158. https://doi.org/10.1145/263407.263537

Digital Library

[15]

Jan P Finis, Martin Raiber, Nikolaus Augsten, Robert Brunel, Alfons Kemper, and Franz Färber. 2013. Rws-diff: flexible and efficient change detection in hierarchical data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management. 339–348.

Digital Library

[16]

Jun Fujima, Aran Lunzer, Kasper Hornbæk, and Yuzuru Tanaka. 2004. Clip, Connect, Clone: Combining Application Elements to Build Custom Interfaces for Information Access. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (Santa Fe, NM, USA) (UIST ’04). Association for Computing Machinery, New York, NY, USA, 175–184. https://doi.org/10.1145/1029632.1029664

Digital Library

[17]

John Gerdes. 2009. User Interface Migration of Microsoft Windows Applications. Journal of Software Maintenance 21 (05 2009), 171–187. https://doi.org/10.1002/smr.400

[18]

Saul Greenberg and Bill Buxton. 2008. Usability Evaluation Considered Harmful (Some of the Time). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). Association for Computing Machinery, New York, NY, USA, 111–120. https://doi.org/10.1145/1357054.1357074

Digital Library

[19]

Björn Hartmann, Leslie Wu, Kevin Collins, and Scott R. Klemmer. 2007. Programming by a Sample: Rapidly Creating Web Applications with d.Mix. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology(Newport, Rhode Island, USA) (UIST ’07). Association for Computing Machinery, New York, NY, USA, 241–250. https://doi.org/10.1145/1294211.1294254

Digital Library

[20]

Masatomo Hashimoto and Akira Mori. 2008. Diff/TS: A tool for fine-grained structural change analysis. In 2008 15th working conference on reverse engineering. IEEE, 279–288.

[21]

Osamu Hashimoto and Brad A. Myers. 1992. Graphical Styles for Building Interfaces by Demonstration. In Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology (Monteray, California, USA) (UIST ’92). Association for Computing Machinery, New York, NY, USA, 117–124. https://doi.org/10.1145/142621.142635

Digital Library

[22]

Hiroshi Hosobe. 2005. Solving Linear and One-Way Constraints for Web Document Layout. In Proceedings of the 2005 ACM Symposium on Applied Computing (Santa Fe, New Mexico) (SAC ’05). Association for Computing Machinery, New York, NY, USA, 1252–1253. https://doi.org/10.1145/1066677.1066959

Digital Library

[23]

Scott E. Hudson, Jennifer Mankoff, and Ian Smith. 2005. Extensible Input Handling in the SubArctic Toolkit. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA) (CHI ’05). Association for Computing Machinery, New York, NY, USA, 381–390. https://doi.org/10.1145/1054972.1055025

Digital Library

[24]

Scott E. Hudson and Shamim P. Mohamed. 1990. Interactive Specification of Flexible User Interface Displays. ACM Trans. Inf. Syst. 8, 3 (July 1990), 269–288. https://doi.org/10.1145/98188.98201

Digital Library

[25]

Scott E. Hudson and Ian Smith. 1997. Supporting Dynamic Downloadable Appearances in an Extensible User Interface Toolkit. In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology(Banff, Alberta, Canada) (UIST ’97). Association for Computing Machinery, New York, NY, USA, 159–168. https://doi.org/10.1145/263407.263539

Digital Library

[26]

Scott E. Hudson and Kenichiro Tanaka. 2000. Providing Visually Rich Resizable Images for User Interface Components. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology (San Diego, California, USA) (UIST ’00). Association for Computing Machinery, New York, NY, USA, 227–235. https://doi.org/10.1145/354401.354783

Digital Library

[27]

Amy Hurst, Scott E. Hudson, and Jennifer Mankoff. 2010. Automatically Identifying Targets Users Interact with during Real World Tasks. In Proceedings of the 15th International Conference on Intelligent User Interfaces (Hong Kong, China) (IUI ’10). Association for Computing Machinery, New York, NY, USA, 11–20. https://doi.org/10.1145/1719970.1719973

Digital Library

[28]

Dugald Ralph Hutchings and John Stasko. 2005. Mudibo: Multiple Dialog Boxes for Multiple Monitors. In CHI ’05 Extended Abstracts on Human Factors in Computing Systems (Portland, OR, USA) (CHI EA ’05). Association for Computing Machinery, New York, NY, USA, 1471–1474. https://doi.org/10.1145/1056808.1056944

Digital Library

[29]

Jaekyu Ha, R. M. Haralick, and I. T. Phillips. 1995. Recursive X-Y cut using bounding boxes of connected components. In Proceedings of 3rd International Conference on Document Analysis and Recognition, Vol. 2. 952–955 vol.2. https://doi.org/10.1109/ICDAR.1995.602059

[30]

Yue Jiang, Ruofei Du, Christof Lutteroth, and Wolfgang Stuerzlinger. 2019. ORC Layout: Adaptive GUI Layout with OR-Constraints. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, Article 413, 12 pages. https://doi.org/10.1145/3290605.3300643

Digital Library

[31]

Yue Jiang, Wolfgang Stuerzlinger, Matthias Zwicker, and Christof Lutteroth. 2020. ORCSolver: An Efficient Solver for Adaptive GUI Layout with OR-Constraints. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376610

Digital Library

[32]

Solange Karsenty, James A. Landay, and Chris Weikart. 1993. Inferring Graphical Constraints With Rockit. In Proceedings of the Conference on People and Computers VII (York, United Kingdom) (HCI’92). Cambridge University Press, 137–153. https://doi.org/10.1007/3-540-58601-_91

Digital Library

[33]

R. Krosnick, S. W. Lee, W. S. Laseck, and S. Onev. 2018. Expresso: Building Responsive Interfaces with Keyframes. In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 39–47. https://doi.org/10.1109/VLHCC.2018.8506516

[34]

Gilly Leshed, Eben M. Haber, Tara Matthews, and Tessa Lau. 2008. CoScripter: Automating & Sharing How-to Knowledge in the Enterprise. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). Association for Computing Machinery, New York, NY, USA, 1719–1728. https://doi.org/10.1145/1357054.1357323

Digital Library

[35]

James Lin, Jeffrey Wong, Jeffrey Nichols, Allen Cypher, and Tessa A. Lau. 2009. End-User Programming of Mashups with Vegemite. In Proceedings of the 14th International Conference on Intelligent User Interfaces (Sanibel Island, Florida, USA) (IUI ’09). Association for Computing Machinery, New York, NY, USA, 97–106. https://doi.org/10.1145/1502650.1502667

Digital Library

[36]

Greg Little, Tessa A. Lau, Allen Cypher, James Lin, Eben M. Haber, and Eser Kandogan. 2007. Koala: Capture, Share, Automate, Personalize Business Processes on the Web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’07). Association for Computing Machinery, New York, NY, USA, 943–946. https://doi.org/10.1145/1240624.1240767

Digital Library

[37]

Christof Lutteroth. 2008. Automated Reverse Engineering of Hard-Coded GUI Layouts. In Proceedings of the Ninth Conference on Australasian User Interface - Volume 76 (Wollongong, Australia) (AUIC ’08). Australian Computer Society, Inc., AUS, 65–73. https://doi.org/10.5555/1378337.1378350

Digital Library

[38]

Christof Lutteroth, Robert Strandh, and Gerald Weber. 2008. Domain specific high-level constraints for user interface layout. Constraints 13, 3 (2008), 307–342.

Digital Library

[39]

Christof Lutteroth and Gerald Weber. 2006. User Interface Layout with Ordinal and Linear Constraints. In Proceedings of the 7th Australasian User Interface Conference - Volume 50 (Hobart, Australia) (AUIC ’06). Australian Computer Society, Inc., AUS, 53–60. https://doi.org/10.5555/1151758.1151764

Digital Library

[40]

Ethan Marcotte. 2011. Responsive Web Design. A book apart.

[41]

Melody Moore and Spencer Rugaber. 1997. Using Knowledge Representation to Understand Interactive Systems. In Proceedings of the 5th International Workshop on Program Comprehension (WPC ’97)(WPC ’97). IEEE Computer Society, USA, 60.

Digital Library

[42]

Melody M. Moore. 1996. Rule-Based Detection for Reverse Engineering User Interfaces. In Proceedings of the 3rd Working Conference on Reverse Engineering (WCRE ’96)(WCRE ’96). IEEE Computer Society, USA, 42.

Digital Library

[43]

Melody Marie Moore, James D. Foley, and Spencer Rugaber. 1998. User Interface Reengineering. Ph.D. Dissertation. USA. AAI9918460.

[44]

Melody M. Moore and Lilia Moshkina. 2000. Migrating Legacy User Interfaces to the Internet: Shifting Dialogue Initiative. In Proceedings of the Seventh Working Conference on Reverse Engineering (WCRE’00)(WCRE ’00). IEEE Computer Society, USA, 52.

[45]

Brad Myers, Scott E. Hudson, and Randy Pausch. 2000. Past, Present, and Future of User Interface Software Tools. ACM Trans. Comput.-Hum. Interact. 7, 1 (March 2000), 3–28. https://doi.org/10.1145/344949.344959

Digital Library

[46]

Brad Myers, Sun Young Park, Yoko Nakano, Greg Mueller, and Amy Ko. 2008. How designers design and program interactive behaviors. In 2008 IEEE Symposium on Visual Languages and Human-Centric Computing. IEEE, 177–184.

Digital Library

[47]

Brad A. Myers. 1995. User Interface Software Tools. ACM Trans. Comput.-Hum. Interact. 2, 1 (March 1995), 64–103. https://doi.org/10.1145/200968.200971

Digital Library

[48]

Brad A. Myers, Richard G. Mcdaniel, Robert C. Miller, Alan S. Ferrency, Andrew Faulring, Bruce D. Kyle, Ieee Computer Society, Ieee Computer Society, Andrew Mickish, Alex Klimovitski, and Patrick Doane. 1997. The Amulet Environment: New Models for Effective User Interface Software Development. IEEE Transactions on Software Engineering 23 (1997), 347–365.

Digital Library

[49]

George Nagy and Sharad C Seth. 1984. Hierarchical representation of optically scanned documents. (1984).

[50]

Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI. In Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering(Lincoln, Nebraska) (ASE ’15). IEEE Press, 248–259. https://doi.org/10.1109/ASE.2015.32

Digital Library

[51]

Jeffrey Nichols and Tessa Lau. 2008. Mobilization by Demonstration: Using Traces to Re-Author Existing Web Sites. In Proceedings of the 13th International Conference on Intelligent User Interfaces (Gran Canaria, Spain) (IUI ’08). Association for Computing Machinery, New York, NY, USA, 149–158. https://doi.org/10.1145/1378773.1378793

Digital Library

[52]

Dan R. Olsen. 2007. Evaluating User Interface Systems Research. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (Newport, Rhode Island, USA) (UIST ’07). Association for Computing Machinery, New York, NY, USA, 251–258. https://doi.org/10.1145/1294211.1294256

Digital Library

[53]

Dan R. Olsen, Scott E. Hudson, Thom Verratti, Jeremy M. Heiner, and Matt Phelps. 1999. Implementing Interface Attachments Based on Surface Representations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI ’99). Association for Computing Machinery, New York, NY, USA, 191–198. https://doi.org/10.1145/302979.303038

Digital Library

[54]

Dan R. Olsen, Trent Taufer, and Jerry Alan Fails. 2004. ScreenCrayons: Annotating Anything. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (Santa Fe, NM, USA) (UIST ’04). Association for Computing Machinery, New York, NY, USA, 165–174. https://doi.org/10.1145/1029632.1029663

Digital Library

[55]

Richard L Potter. 1992. Triggers: Guiding Automation with Pixels to Achieve Data Access. University of Maryland, Center for Automation Research, Human/Computer Interaction Laboratory.

[56]

Óscar Sánchez Ramón, Jesús Sánchez Cuadrado, and Jesús García Molina. 2014. Model-driven reverse engineering of legacy graphical user interfaces. Automated Software Engineering 21, 2 (2014), 147–186.

[57]

Óscar Sánchez Ramón, Jesús Sánchez Cuadrado, Jesús García Molina, and Jean Vanderdonckt. 2016. A layout inference algorithm for Graphical User Interfaces. Information and Software Technology 70 (2016), 155–175.

Digital Library

[58]

Alireza Sahami Shirazi, Niels Henze, Albrecht Schmidt, Robin Goldberg, Benjamin Schmidt, and Hansjörg Schmauder. 2013. Insights Into Layout Patterns of Mobile User Interfaces by an Automatic Analysis of Android Apps. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (London, United Kingdom) (EICS ’13). ACM, Gothenburg, Sweden, 275–284. https://doi.org/10.1145/3197231.3197249

Digital Library

[59]

Óscar Sánchez Ramón, Jesús Sánchez Cuadrado, and Jesús García Molina. 2010. Model-Driven Reverse Engineering of Legacy Graphical User Interfaces. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering(Antwerp, Belgium) (ASE ’10). Association for Computing Machinery, New York, NY, USA, 147–150. https://doi.org/10.1145/1858996.1859023

Digital Library

[60]

Adriano Scoditti and Wolfgang Stuerzlinger. 2009. A New Layout Method for Graphical User Interfaces. In Science and Technology for Humanity (TIC-STH), 2009 IEEE Toronto International Conference. IEEE, 642–647. https://doi.org/10.1016/j.infsof.2015.10.005

[61]

Robert St. Amant, Henry Lieberman, Richard Potter, and Luke Zettlemoyer. 2000. Programming by Example: Visual Generalization in Programming by Example. Commun. ACM 43, 3 (March 2000), 107–114. https://doi.org/10.1145/330534.330549

Digital Library

[62]

Stefan Staiger. 2007. Static Analysis of Programs with Graphical User Interface. In 11th European Conference on Software Maintenance and Reengineering (CSMR’07). 252–264. https://doi.org/10.1109/CSMR.2007.44

Digital Library

[63]

E. Stroulia, M. El-Ramly, P. Iglinski, and P. Sorenson. 2003. User Interface Reverse Engineering in Support of Interface Migration to the Web. Automated Software Engg. 10, 3 (July 2003), 271–301. https://doi.org/10.1023/A:1024460315173

Digital Library

[64]

Wolfgang Stuerzlinger, Olivier Chapuis, Dusty Phillips, and Nicolas Roussel. 2006. User Interface Façades: Towards Fully Adaptable User Interfaces. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (Montreux, Switzerland) (UIST ’06). Association for Computing Machinery, New York, NY, USA, 309–318. https://doi.org/10.1145/1166253.1166301

Digital Library

[65]

Amanda Swearngin, Amy J. Ko, and James Fogarty. 2017. Genie: Input Retargeting on the Web through Command Reverse Engineering. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 4703–4714. https://doi.org/10.1145/3025453.3025506

Digital Library

[66]

Desney S. Tan, Brian Meyers, and Mary Czerwinski. 2004. WinCuts: Manipulating Arbitrary Window Regions for More Effective Use of Screen Space. In CHI ’04 Extended Abstracts on Human Factors in Computing Systems (Vienna, Austria) (CHI EA ’04). Association for Computing Machinery, New York, NY, USA, 1525–1528. https://doi.org/10.1145/985921.986106

Digital Library

[67]

J. Vanderdonckt, L. Bouillon, and N. Souchon. 2001. Flexible Reverse Engineering of Web Pages with VAQUISTA. In Proceedings Eighth Working Conference on Reverse Engineering. 241–248. https://doi.org/10.1109/WCRE.2001.957828

[68]

Yuan Wang, David J DeWitt, and J-Y Cai. 2003. X-Diff: An effective change detection algorithm for XML documents. In Proceedings 19th international conference on data engineering (Cat. No. 03CH37405). IEEE, 519–530.

[69]

Gerald Weber. 2010. A Reduction of Grid-Bag Layout to Auckland Layout. In Proceedings of the 2010 21st Australian Software Engineering Conference(ASWEC ’10). IEEE Computer Society, 67–74. https://doi.org/10.1109/ASWEC.2010.38

Digital Library

[70]

Xing Xie, Chong Wang, Li-Qun Chen, and Wei-Ying Ma. 2005. An adaptive web page layout structure for small devices. Multimedia Systems 11, 1 (2005), 34–44.

Digital Library

[71]

Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. 2009. Sikuli: Using GUI Screenshots for Search and Automation. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (Victoria, BC, Canada) (UIST ’09). Association for Computing Machinery, New York, NY, USA, 183–192. https://doi.org/10.1145/1622176.1622213

Digital Library

[72]

Brad Vander Zanden and Brad A. Myers. 1990. Automatic, Look-and-Feel Independent Dialog Creation for Graphical User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, USA) (CHI ’90). ACM, Seattle, Washington, USA, 27–34. https://doi.org/10.1007/978-3-319-67744-_2

[73]

Clemens Zeidler, Christof Lutteroth, Gerald Weber, and Wolfgang Stürzlinger. 2012. The Auckland Layout Editor: An Improved GUI Layout Specification Process. In Proceedings of the 13th International Conference of the NZ Chapter of the ACM’s Special Interest Group on Human-Computer Interaction (Dunedin, New Zealand) (CHINZ ’12). Association for Computing Machinery, New York, NY, USA, 103. https://doi.org/10.1145/2379256.2379287

Digital Library

[74]

C. Zeidler, G. Weber, A. Gavryushkin, and Christof Lutteroth. 2017. Tiling algebra for constraint-based layout editing. J. Log. Algebraic Methods Program. 89 (2017), 67–94.

[75]

Luke S. Zettlemoyer and Robert St. Amant. 1999. A Visual Medium for Programmatic Control of Interactive Applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI ’99). Association for Computing Machinery, New York, NY, USA, 199–206. https://doi.org/10.1145/302979.303039

Digital Library

[76]

Luke S. Zettlemoyer, Robert St. Amant, and Martin S. Dulberg. 1998. IBOTS: Agent Control through the User Interface. In Proceedings of the 4th International Conference on Intelligent User Interfaces (Los Angeles, California, USA) (IUI ’99). Association for Computing Machinery, New York, NY, USA, 31–37. https://doi.org/10.1145/291080.291087

Digital Library

Cited By

Jiang YGuo ZRezazadegan Tavakoli HLeiva LOulasvirta A(2024)EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement LearningProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676436(1-15)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676436
Jiang Y(2024)Computational Representations for Graphical User InterfacesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3638191(1-6)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3638191
Jiang YLu YKnearem TKliman-Silver CLutteroth CLi TNichols JStuerzlinger W(2024)Computational Methodologies for Understanding, Automating, and Evaluating User InterfacesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636316(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3636316
Show More Cited By

Index Terms

ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints
1. Human-centered computing
2. Software and its engineering
  1. Software creation and management
    1. Software development techniques
    2. Software post-development issues
      1. Software reverse engineering
  2. Software notations and tools
    1. Development frameworks and environments

Index terms have been assigned to the content through auto-classification.

Recommendations

ORCSolver: An Efficient Solver for Adaptive GUI Layout with OR-Constraints
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

OR-constrained (ORC) graphical user interface layouts unify conventional constraint-based layouts with flow layouts, which enables the definition of flexible layouts that adapt to screens with different sizes, orientations, or aspect ratios with only a ...
ORC Layout: Adaptive GUI Layout with OR-Constraints
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

We propose a novel approach for constraint-based graphical user interface (GUI) layout based on OR-constraints (ORC) in standard soft/hard linear constraint systems. ORC layout unifies grid layout and flow layout, supporting both their features as well ...
Model-driven reverse engineering of legacy graphical user interfaces
ASE '10: Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineering

Businesses are more and more modernizing the legacy systems they developed with Rapid Application Development (RAD), so that they can benefit from the new platforms and technologies. In these systems, the Graphical User Interface (GUI) layout is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
1,575
Total Downloads

Downloads (Last 12 months)412
Downloads (Last 6 weeks)30

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Jiang YGuo ZRezazadegan Tavakoli HLeiva LOulasvirta A(2024)EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement LearningProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676436(1-15)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676436
Jiang Y(2024)Computational Representations for Graphical User InterfacesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3638191(1-6)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3638191
Jiang YLu YKnearem TKliman-Silver CLutteroth CLi TNichols JStuerzlinger W(2024)Computational Methodologies for Understanding, Automating, and Evaluating User InterfacesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636316(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3636316
Jiang YZhou CGarg VOulasvirta A(2024)Graph4GUI: Graph Neural Networks for Representing Graphical User InterfacesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642822(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642822
Yanagida TVanderdonckt JBurny N(2023)Adaptive GUI Layout by Satisfying Fuzzy ConstraintsCompanion Proceedings of the 2023 ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3596454.3597190(70-72)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3596454.3597190
Xu PLi YYang ZShi WFu HHuang H(2022)Hierarchical Layout Blending with Recursive Optimal CorrespondenceACM Transactions on Graphics10.1145/3550454.355544641:6(1-15)Online publication date: 30-Nov-2022
https://doi.org/10.1145/3550454.3555446
Chen YGrossman T(2021)Umitation: Retargeting UI Behavior Examples for Website DesignThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474796(922-935)Online publication date: 10-Oct-2021
https://dl.acm.org/doi/10.1145/3472749.3474796

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents