research-article

Individualising Graphical Layouts with Predictive Visual Search Models

Authors:

Antti OulasvirtaAuthors Info & Claims

ACM Transactions on Interactive Intelligent Systems (TiiS), Volume 10, Issue 1

Article No.: 9, Pages 1 - 24

https://doi.org/10.1145/3241381

Published: 30 August 2019 Publication History

Abstract

In domains where users are exposed to large variations in visuo-spatial features among designs, they often spend excess time searching for common elements (features) on an interface. This article contributes individualised predictive models of visual search, and a computational approach to restructure graphical layouts for an individual user such that features on a new, unvisited interface can be found quicker. It explores four technical principles inspired by the human visual system (HVS) to predict expected positions of features and create individualised layout templates: (I) the interface with highest frequency is chosen as the template; (II) the interface with highest predicted recall probability (serial position curve) is chosen as the template; (III) the most probable locations for features across interfaces are chosen (visual statistical learning) to generate the template; (IV) based on a generative cognitive model, the most likely visual search locations for features are chosen (visual sampling modelling) to generate the template. Given a history of previously seen interfaces, we restructure the spatial layout of a new (unseen) interface with the goal of making its features more easily findable. The four HVS principles are implemented in Familiariser, a web browser that automatically restructures webpage layouts based on the visual history of the user. Evaluation of Familiariser (using visual statistical learning) with users provides first evidence that our approach reduces visual search time by over 10%, and number of eye-gaze fixations by over 20%, during web browsing tasks.

References

[1]

John R. Anderson, D. Bothell, C. Lebiere, and M. Matessa. 1998. An integrated theory of list memory. Journal of Memory and Language 38, 4 (1998), 341--380.

[2]

John R. Anderson, Jon M. Fincham, and Scott Douglass. 1999. Practice and retention: A unifying analysis. Journal of Experimental Psychology-Learning Memory and Cognition 25, 5 (1999), 1120--1136.

[3]

Yigal Arens, Lawrence Miller, Stuart C. Shapiro, and Norman K. Sondheimer. 1988. Automatic construction of user-interface displays. In Proceedings of the 7th AAAI National Conference on Artificial Intelligence (AAAI’88). AAAI Press, 808--813. http://dl.acm.org/citation.cfm?id=2887965.2888108

Digital Library

[4]

Gilles Bailly, Antti Oulasvirta, Timo Kötzing, and Sabrina Hoppe. 2013. MenuOptimizer: Interactive optimization of menu systems. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST’13). ACM, New York, NY, 331--342.

Digital Library

[5]

C. M. Beshers and S. Feiner. 1989. Scope: Automated generation of graphical interfaces. In Proceedings of the 2nd Annual ACM SIGGRAPH Symposium on User Interface Software and Technology (UIST’89). ACM, New York, NY, 76--85.

Digital Library

[6]

David M. Blei. 2012. Probabilistic topic models. Communications of the ACM 55, 4 (2012), 77--84.

Digital Library

[7]

François Bodart, Anne-Marie Hennebert, Jean-Marie Leheureux, and Jean Vanderdonckt. 1994. Towards a dynamic strategy for computer-aided visual placement. In Proceedings of the Workshop on Advanced Visual Interfaces (AVI’94). ACM, New York, NY, 78--87.

Digital Library

[8]

Marvin M. Chun and Yuhong Jiang. 1998. Contextual cueing: Implicit learning and memory of visual context guides spatial attention. Cognitive Psychology 36, 1 (1998), 28--71.

[9]

Marvin M. Chun and Yuhong Jiang. 1999. Top-down attentional guidance based on implicit learning of visual covariation. Psychological Science 10, 4 (1999), 360--365.

[10]

Morgan Dixon and James Fogarty. 2010. Prefab: Implementing advanced behaviors using pixel-based reverse engineering of interface structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’10). ACM, New York, NY, 1525--1534.

Digital Library

[11]

Peter A. Frensch. 1994. Composition during serial learning: A serial position effect. Journal of Experimental Psychology Learning Memory and Cognition 20 (1994), 423--423.

[12]

Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: Automatically generating user interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI’04). ACM, New York, NY, 93--100.

Digital Library

[13]

Krzysztof Z. Gajos, Mary Czerwinski, Desney S. Tan, and Daniel S. Weld. 2006. Exploring the design space for adaptive graphical user interfaces (AVI’06). ACM, New York, NY, 201--208.

Digital Library

[14]

Krzysztof Z. Gajos, Jing Jing Long, and Daniel S. Weld. 2006. Automatically generating custom user interfaces for users with physical disabilities. In Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Assets’06). ACM, New York, NY, 243--244.

Digital Library

[15]

Krzysztof Z. Gajos, Jacob O. Wobbrock, and Daniel S. Weld. 2007. Automatically generating user interfaces adapted to users’ motor and vision capabilities. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST’07). ACM, New York, NY, 231--240.

Digital Library

[16]

Saul Greenberg and Ian H. Witten. 1985. Adaptive personalized interfaces—A question of viability. Behaviour and Information Technology 4, 1 (1985), 31--45.

[17]

Richard N. A. Henson. 1996. Unchained memory: Error patterns rule out chaining models of immediate serial recall. The Quarterly Journal of Experimental Psychology: Section A 49, 1 (1996), 80--115.

[18]

Helene Intraub and Michael Richardson. 1989. Wide-angle memories of close-up scenes. Journal of Experimental Psychology: Learning, Memory, and Cognition 15, 2 (1989), 179.

[19]

Laurent Itti and Christof Koch. 2000. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research 40, 10 (2000), 1489--1506.

[20]

Christian Janssen, Anette Weisbecker, and Jürgen Ziegler. 1993. Generating user interfaces from data models and dialogue net specifications. In Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems (CHI’93). ACM, New York, NY, 418--423.

Digital Library

[21]

Jussi P. P. Jokinen, Sayan Sarcar, Antti Oulasvirta, Chaklam Silpasuwanchai, Zhenxin Wang, and Xiangshi Ren. 2017. Modelling learning of new keyboard layouts. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI’17). ACM, New York, NY, 4203--4215.

Digital Library

[22]

Andruid Kerne, William A. Hamilton, and Zachary O. Toups. 2012. Culturally based design: Embodying trans-surface interaction in rummy. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW’12). ACM, New York, NY, 509--518.

Digital Library

[23]

Eileen Kowler. 2011. Eye movements: The past 25 years. Vision Research 51, 13 (2011), 1457--1483.

[24]

Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine Lim, Salman Ahmad, Scott R. Klemmer, and Jerry O. Talton. 2013. Webzeitgeist: Design mining the web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’13). ACM, New York, NY, 3083--3092.

Digital Library

[25]

Ranjitha Kumar, Jerry O. Talton, Salman Ahmad, and Scott R. Klemmer. 2011. Bricolage: Example-based retargeting for web design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). ACM, New York, NY, 2197--2206.

Digital Library

[26]

Talia Lavie and Joachim Meyer. 2010. Benefits and costs of adaptive user interfaces. International Journal of Human-Computer Studies 68, 8 (Aug. 2010), 508--524.

Digital Library

[27]

Jeffrey Nichols and Tessa Lau. 2008. Mobilization by demonstration: Using traces to re-author existing web sites. In Proceedings of the 13th International Conference on Intelligent User Interfaces (IUI’08). ACM, New York, NY, 149--158.

Digital Library

[28]

Jeffrey Nichols, Brad A. Myers, Michael Higgins, Joseph Hughes, Thomas K. Harris, Roni Rosenfeld, and Mathilde Pignol. 2002. Generating remote control interfaces for complex appliances. In Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology (UIST’02). ACM, New York, NY, 161--170.

Digital Library

[29]

Jeffrey Nichols, Brad A. Myers, and Brandon Rothrock. 2006. UNIFORM: Automatically generating consistent remote control user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’06). ACM, New York, NY, 611--620.

Digital Library

[30]

Jakob Nielsen. 1993. Usability Engineering. Morgan Kaufmann Publishers, Inc., San Francisco, CA.

Digital Library

[31]

Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann. 2014. Learning layouts for single-page graphic designs. IEEE Transactions on Visualization and Computer Graphics 20, 8 (Aug. 2014), 1200--1213.

Digital Library

[32]

Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. DesignScape: Design with interactive layout suggestions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI’15). ACM, New York, NY, 1221--1224.

Digital Library

[33]

D. R. Olsen, Jr.1989. A programming language basis for user interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’89). ACM, New York, NY, 171--176.

Digital Library

[34]

Angel R. Puerta, Henrik Eriksson, John H. Gennari, and Mark A. Musen. 1994. Model-based automated generation of user interfaces. In Proceedings of the 12h National Conference on Artificial Intelligence (Vol. 1) (AAAI’94). American Association for Artificial Intelligence, 471--477. http://dl.acm.org/citation.cfm?id=199288.184583

Digital Library

[35]

Keith Rayner. 2009. The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology 62, 8 (2009), 1457--1506.

[36]

John Rheinfrank and Shelley Evenson. 1996. Design languages. In Bringing Design to Software. ACM, 63--85.

Digital Library

[37]

Dario D. Salvucci. 2001. An integrated model of eye movements and visual encoding. Cognitive Systems Research 1, 4 (Feb. 2001), 201--220.

Digital Library

[38]

Sayan Sarcar, Jussi Jokinen, Antti Oulasvirta, Xiangshi Ren, Chaklam Silpasuwanchai, and Zhenxin Wang. 2018. Ability-based optimization: Designing smartphone text entry interface for older adults. IEEE Pervasive Computing 17 (2018), 15--26.

Digital Library

[39]

A. Sears. 1993. Layout appropriateness: A metric for evaluating user interface widget layout. IEEE Transactions on Software Engineering 19, 7 (July 1993), 707--719.

Digital Library

[40]

Andrew Sears, Julie A. Jacko, Josey Chu, and Francisco Moro. 2001. The role of visual search in the design of effective soft keyboards. Behaviour and Information Technology 20, 3 (2001), 159--166.

[41]

Kashyap Todi, Jussi Jokinen, Kris Luyten, and Antti Oulasvirta. 2018. Familiarisation: Restructuring layouts with visual learning models. In Proceedings of the 23rd International Conference on Intelligent User Interfaces (IUI’18). ACM, New York, NY, 547--558.

Digital Library

[42]

Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and explore with a layout optimiser. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS’16). ACM, New York, NY, 543--555.

Digital Library

[43]

Anne M. Treisman and Garry Gelade. 1980. A feature-integration theory of attention. Cognitive Psychology 12, 1 (1980), 97--136.

[44]

Martijn Van Welie and Gerrit C. Van der Veer. 2003. Pattern languages in interaction design: Structure and organization. In Proceedings of Interact, Vol. 3. 1--5.

[45]

Jacob O. Wobbrock, Shaun K. Kane, Krzysztof Z. Gajos, Susumu Harada, and Jon Froehlich. 2011. Ability-based design: Concept, principles and examples. ACM Transactions on Accessible Computing 3, 3 (April 2011), Article 9, 27 pages.

Digital Library

[46]

Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. 2009. Sikuli: Using GUI screenshots for search and automation. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST’09). ACM, New York, NY, 183--192.

Digital Library

Cited By

Yang CYuan BWu KYe J(2025)Aesthetic evaluation method for interactive interface layouts on the basis of visual cognitive multi-attribute fusion decision makingInternational Journal of Industrial Ergonomics10.1016/j.ergon.2024.103673105(103673)Online publication date: Jan-2025
https://doi.org/10.1016/j.ergon.2024.103673
Jiang YGuo ZRezazadegan Tavakoli HLeiva LOulasvirta A(2024)EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement LearningProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676436(1-15)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676436
Zuo YQi JFan ZWang ZXu HWang SZhang NHu J(2023)The influence of target layout and target graphic type on searching performance based on eye-tracking technologyFrontiers in Psychology10.3389/fpsyg.2023.105248814Online publication date: 9-Feb-2023
https://doi.org/10.3389/fpsyg.2023.1052488
Show More Cited By

Index Terms

Individualising Graphical Layouts with Predictive Visual Search Models
1. Human-centered computing
  1. Human computer interaction (HCI)
  2. Interaction design
    1. Interaction design process and methods
      1. User interface design

Recommendations

Familiarisation: Restructuring Layouts with Visual Learning Models
IUI '18: Proceedings of the 23rd International Conference on Intelligent User Interfaces

In domains where users are exposed to large variations in visuo-spatial features among designs, they often spend excess time searching for common elements (features) in familiar locations. This paper contributes computational approaches to restructuring ...
Towards accurate and practical predictive models of active-vision-based visual search
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Being able to predict the performance of interface designs using models of human cognition and performance is a long-standing goal of HCI research. This paper presents recent advances in cognitive modeling which permit increasingly realistic and ...
Personalizing graphical user interfaces on flexible widget layout
EICS '09: Proceedings of the 1st ACM SIGCHI symposium on Engineering interactive computing systems

The authors propose a method for personalizing the flexible widget layout (FWL) by adjusting the desirability of widgets with a pairwise comparison method, and show its implementation and that it actually works. Personalization of graphical user ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems

ACM Transactions on Interactive Intelligent Systems Volume 10, Issue 1

Special Issue on IUI 2018

March 2020

347 pages

ISSN:2160-6455

EISSN:2160-6463

DOI:10.1145/3352585

Editor:
Michelle X. Zhou
Juji, Inc., USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 August 2019

Accepted: 01 December 2018

Revised: 01 December 2018

Received: 01 May 2018

Published in TIIS Volume 10, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Academy of Finland project COMPUTED
European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
486
Total Downloads

Downloads (Last 12 months)52
Downloads (Last 6 weeks)3

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang CYuan BWu KYe J(2025)Aesthetic evaluation method for interactive interface layouts on the basis of visual cognitive multi-attribute fusion decision makingInternational Journal of Industrial Ergonomics10.1016/j.ergon.2024.103673105(103673)Online publication date: Jan-2025
https://doi.org/10.1016/j.ergon.2024.103673
Jiang YGuo ZRezazadegan Tavakoli HLeiva LOulasvirta A(2024)EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement LearningProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676436(1-15)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676436
Zuo YQi JFan ZWang ZXu HWang SZhang NHu J(2023)The influence of target layout and target graphic type on searching performance based on eye-tracking technologyFrontiers in Psychology10.3389/fpsyg.2023.105248814Online publication date: 9-Feb-2023
https://doi.org/10.3389/fpsyg.2023.1052488
Moon HOulasvirta ALee B(2023)Amortized Inference with User SimulationsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581439(1-20)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581439
Wu KYang C(2023)The Interaction Interface Design Method of Fresh Food E-Commerce Based on Kansei Engineering2023 4th International Conference on Intelligent Design（ICID）10.1109/ICID60307.2023.10396831(247-250)Online publication date: 20-Oct-2023
https://doi.org/10.1109/ICID60307.2023.10396831
Howes AJokinen JOulasvirta A(2023)Towards machines that understand peopleAI Magazine10.1002/aaai.1211644:3(312-327)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1002/aaai.12116
Wang WHong XDang SXu NQu J(2022)3D Space Layout Design of Holographic Command Cabin Information Display in Mixed Reality Environment Based on HoloLens 2Brain Sciences10.3390/brainsci1208097112:8(971)Online publication date: 23-Jul-2022
https://doi.org/10.3390/brainsci12080971
Oulasvirta AJokinen JHowes A(2022)Computational Rationality as a Theory of InteractionProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517739(1-14)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517739
Da Silva MLeite LGarcia Costa Dos Santos G(2022)Algorithms for the Development of Adaptive Web Interfaces: A Systematic Literature Review2022 XVII Latin American Conference on Learning Technologies (LACLO)10.1109/LACLO56648.2022.10013456(01-08)Online publication date: 17-Oct-2022
https://doi.org/10.1109/LACLO56648.2022.10013456
Cutting JDeterding S(2022)The task-attention theory of game learning: a theory and research agendaHuman–Computer Interaction10.1080/07370024.2022.204797139:5-6(257-287)Online publication date: 19-Apr-2022
https://doi.org/10.1080/07370024.2022.2047971
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents