research-article

The Emergence of Interactive Behavior: A Model of Rational Menu Search

Authors:

Duncan P. Brumby,

Antti Oulasvirta,

Andrew HowesAuthors Info & Claims

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Pages 4217 - 4226

https://doi.org/10.1145/2702123.2702483

Published: 18 April 2015 Publication History

Abstract

One reason that human interaction with technology is difficult to understand is because the way in which people perform interactive tasks is highly adaptive. One such interactive task is menu search. In the current article we test the hypothesis that menu search is rationally adapted to (1) the ecological structure of interaction, (2) cognitive and perceptual limits, and (3) the goal to maximise the trade-off between speed and accuracy. Unlike in previous models, no assumptions are made about the strategies available to or adopted by users, rather the menu search problem is specified as a reinforcement learning problem and behaviour emerges by finding the optimal Markov Decision Process (MDP). The model is tested against existing empirical findings concerning the effect of menu organisation and menu length. The model predicts the effect of these variables on task completion time and eye movements. The discussion considers the pros and cons of the modelling approach relative to other well-known mod- elling approaches.

Supplementary Material

MP4 File (p4217-chen.mp4)

Download
146.74 MB

References

[1]

Bailly, G., Malacria, S., et al. Menuinspector: Outil pour l'analyse des menus et cas d'étude. In 25'eme conférence francophone sur l'Interaction Homme-Machine, IHM'13 (2013).

Digital Library

[2]

Bailly, G., Oulasvirta, A., Brumby, D. P., and Howes, A. Model of visual search and selection time in linear menus. In ACM CHI'14 (2014), 3865--3874.

Digital Library

[3]

Bailly, G., Oulasvirta, A., Kötzing, T., and Hoppe, S. Menuoptimizer: Interactive optimization of menu systems. In ACM CHI'13 (2013), 331--342.

Digital Library

[4]

Baloh, R. W., Sills, A. W., Kumley, W. E., and Honrubia, V. Quantitative measurement of saccade amplitude, duration, and velocity. Neurology 25, 11 (1975), 1065--1065.

[5]

Baron, S., and Kleinman, D. L. The human as an optimal controller and information processor. Man-Machine Systems, IEEE Transactions on 10, 1 (1969), 9--17.

[6]

Brumby, D., and Howes, A. Strategies for Guiding Interactive Search: An Empirical Investigation Into the Consequences of Label Relevance for Assessment and Selection. Human-Computer Interaction 23, 1 (Jan. 2008), 1--46.

[7]

Brumby, D. P., Cox, A. L., Chung, J., and Fernandes, B. How does knowing what you are looking for change visual search behavior? In ACM CHI'14 (2014), 3895--3898.

Digital Library

[8]

Brumby, D. P., and Howes, A. Good enough but i'll just check: Web-page search as attentional refocusing. In ICCM (2004), 46--51.

[9]

Brumby, D. P., Salvucci, D. D., and Howes, A. Focus on driving: How cognitive constraints shape the adaptation of strategy when dialing while driving. In ACM CHI'09 (2009), 1629--1638.

Digital Library

[10]

Byrne, M. D. ACT-R/PM and menu selection: Applying a cognitive architecture to HCI. International Journal of Human-Computer Studies 55, 1 (2001), 41--84.

Digital Library

[11]

Charman, S. C., and Howes, A. The adaptive user: an investigation into the cognitive and task constraints on the generation of new methods. Journal of experimental psychology. Applied 9, 4 (2003), 236--48.

[12]

Chen, X, Howes, A., Lewis, R.L., Myers, C.W., Houpt, J. Discovering computationally rational eye movements in the distractor ratio task. In Reinforcement Learning and Decision Making (Princeton, 2013).

[13]

Cockburn, A., Gutwin, C., and Greenberg, S. A predictive model of menu performance. In ACM CHI'07 (2007), 627--636.

Digital Library

[14]

Fu, W.-T., and Gray, W. D. Resolving the paradox of the active user : stable suboptimal performance in interactive tasks. Cognitive Science 28 (2004), 901--935.

[15]

Fu, W.-T., and Pirolli, P. SNIF-ACT: A cognitive model of user navigation on the World Wide Web. Human-Computer Interaction 22, 4 (2007), 355--412.

Digital Library

[16]

Gray, W. D., Sims, C. R., Fu, W.-T., and Schoelles, M. J. The soft constraints hypothesis: a rational analysis approach to resource allocation for interactive behavior. Psychological review 113, 3 (2006), 461--82.

[17]

Halverson, T., and Hornof, A. J. A minimal model for predicting visual search in human-computer interaction. In ACM CHI'07, ACM (2007), 431--434.

Digital Library

[18]

Halverson, T., and Hornof, A. J. A computational model of active vision for visual search in human-computer interaction. Human-Computer Interaction 26, 4 (2011), 285--314.

Digital Library

[19]

Hayhoe, M., and Ballard, D. Modeling task control of eye movements. Current Biology 24, 13 (2014), R622--R628.

[20]

Hornof, A. J., and Halverson, T. Cognitive strategies and eye movements for searching hierarchical computer displays. In ACM CHI'03, ACM (2003), 249--256.

Digital Library

[21]

Howes, A., Lewis, R. L., and Vera, A. Rational adaptation under task and processing constraints: implications for testing theories of cognition and action. Psychological Review 116, 4 (2009), 717--751.

[22]

Howes, A., Vera, A., Lewis, R. L., and McCurdy, M. Cognitive constraint modeling: A formal approach to supporting reasoning about behavior. Proc. Cognitive Science Society (2004), 595--600.

[23]

Inhoff, A. W., and Rayner, K. Parafoveal word perception: A case against semantic preprocessing. Perception & Psychophysics 27, 5 (1980), 457--464.

[24]

Janssen, C. P., Brumby, D. P., Dowell, J., Chater, N., and Howes, A. Identifying optimum performance trade-offs using a cognitively bounded rational analysis model of discretionary task interleaving. Topics in Cognitive Science 3, 1 (2011), 123--139.

[25]

Kieras, D. E., and Hornof, A. J. Towards accurate and practical predictive models of active-vision-based visual search. In ACM CHI'14 (2014), 3875--3884.

Digital Library

[26]

Lelis, S., and Howes, A. Informing decisions: how people use online rating information to make choices. In Proceedings of the 2011 annual conference on Human factors in computing systems, ACM (2011), 2285--2294.

Digital Library

[27]

Lewis, R., Howes, A., and Singh, S. Computational rationality: linking mechanism and behavior through bounded utility maximization. Topics in Cognitive Science 6, 2 (2014), 279--311.

[28]

Miller, C. S., and Remington, R. W. Modeling information navigation: Implications for information architecture. Human-computer interaction 19, 3 (2004), 225--271.

Digital Library

[29]

Nunez-Varela, J., and Wyatt, J. L. Models of gaze control for manipulation tasks. ACM Transactions on Applied Perception (TAP) 10, 4 (2013), 20.

Digital Library

[30]

O'Hara, K. P., and Payne, S. J. The effects of operator implementation cost on planfulness of problem solving and learning. Cognitive psychology 35, 1 (1998), 34--70.

[31]

Payne, S. J., and Howes, A. Adaptive Interaction: A Utility Maximization Approach to Understanding Human Interaction with Technology. Synthesis Lectures on Human-Centered Informatics 6, 1 (2013), 1--111.

[32]

Payne, S. J., Howes, A., and Reader, W. R. Adaptively distributing cognition: a decision-making perspective on human-computer interaction. Behaviour & Information Technology 20, 5 (2001), 339--346.

[33]

Pirolli, P. Information foraging theory: Adaptive interaction with information, vol. 2. Oxford University Press, USA, 2007.

[34]

Pirolli, P., and Card, S. Information foraging. Psychological Review 106 (1999), 643--675.

[35]

Rayner, K. Eye movements in reading and information processing: 20 years of research. Psychological bulletin 124, 3 (1998), 372.

[36]

Russell, S., and Subramanian, D. Provably bounded-optimal agents. Journal of Artificial Intelligence Research 2 (1995), 575--609.

Digital Library

[37]

Sprague, N., and Ballard, D. Eye movements for reward maximization. Advances in neural information processing systems 16 (2004), 1419--1433.

[38]

Sutton, R. S., and Barto, A. G. Reinforcement learning: an introduction. MIT Press, 1998.

Digital Library

[39]

Trommershäuser, J., Maloney, L. T., and Landy, M. S. The Expected Utility of Movement. Brain (2009), 95--111.

[40]

Tseng, Y.-C., and Howes, A. The adaptation of visual search strategy to expected information gain. In ACM CHI '08 (2008), 1075--1084.

Digital Library

[41]

Vera, A., Howes, A., McCurdy, M., and Lewis, R. L. A constraint satisfaction approach to predicting skilled interactive cognition. In ACM CHI'04 (2004), 121--128.

Digital Library

[42]

Zhang, Y., and Hornof, A. J. Understanding multitasking through parallelized strategy exploration and individualized cognitive modeling. In ACM CHI'14 (2014), 3885--3894.

Digital Library

Cited By

Langerak TChristen SAlbaba MGebhardt CHolz CHilliges O(2024)MARLUI: Multi-Agent Reinforcement Learning for Adaptive Point-and-Click UIsProceedings of the ACM on Human-Computer Interaction10.1145/36611478:EICS(1-27)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3661147
Lorenz MAmorim TDey DSadeghi MEbel P(2024)Computational Models for In-Vehicle User Interface Design: A Systematic Literature ReviewProceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3640792.3675735(204-215)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3640792.3675735
Moon HLiao YLi CLee BOulasvirta A(2024)Real-time 3D Target Inference via Biomechanical SimulationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642131(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642131
Show More Cited By

Index Terms

The Emergence of Interactive Behavior: A Model of Rational Menu Search
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
      1. Cognitive science
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI theory, concepts and models
  2. Interaction design
    1. Interaction design theory, concepts and paradigms

Recommendations

Visual Grouping in Menu Interfaces
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Menu interfaces often arrange options into semantic groups. This semantic structure is then usually conveyed to the user by supplementary visual grouping cues. We investigate whether these visual grouping cues actually help users locate items in menus ...
Influence of Visual Salience on Webpage Product Searches

Visual salience can increase search efficiency in complex displays but does that influence persist when completing a specific search? In two experiments, participants were asked to search webpages for the prices of specific products. Those products were ...
Intelligent gaze-added interfaces
CHI '00: Proceedings of the SIGCHI conference on Human Factors in Computing Systems

We discuss a novel type of interface, the intelligent gaze-added interface, and describe the design and evaluation of a sample gaze-added operating-system interface. Gaze-added interfaces, like current gaze-based systems, allow users to execute commands ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

April 2015

4290 pages

ISBN:9781450331456

DOI:10.1145/2702123

General Chairs:
Bo Begole
Huawei, USA
,
Jinwoo Kim
Yonsei University, Korea
,
Program Chairs:
Kori Inkpen
Microsoft Research, USA
,
Woontack Woo
KAIST, Korea

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

EU-funded SPEEDD
Air Force Office of Scientific Research

Conference

CHI '15

Sponsor:

SIGCHI

CHI '15: CHI Conference on Human Factors in Computing Systems

April 18 - 23, 2015

Seoul, Republic of Korea

Acceptance Rates

CHI '15 Paper Acceptance Rate 486 of 2,120 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

41
Total Citations
View Citations
1,069
Total Downloads

Downloads (Last 12 months)171
Downloads (Last 6 weeks)14

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Langerak TChristen SAlbaba MGebhardt CHolz CHilliges O(2024)MARLUI: Multi-Agent Reinforcement Learning for Adaptive Point-and-Click UIsProceedings of the ACM on Human-Computer Interaction10.1145/36611478:EICS(1-27)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3661147
Lorenz MAmorim TDey DSadeghi MEbel P(2024)Computational Models for In-Vehicle User Interface Design: A Systematic Literature ReviewProceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3640792.3675735(204-215)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3640792.3675735
Moon HLiao YLi CLee BOulasvirta A(2024)Real-time 3D Target Inference via Biomechanical SimulationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642131(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642131
Lingler ATalypova DJokinen JOulasvirta AWintersberger P(2024)Supporting Task Switching with Reinforcement LearningProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642063(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642063
Wu CJokinen J(2024)Modeling the impact of mental models on interactive decision-making and behavior2024 16th International Conference on Human System Interaction (HSI)10.1109/HSI61632.2024.10613586(1-7)Online publication date: 8-Jul-2024
https://doi.org/10.1109/HSI61632.2024.10613586
Chandramouli SShi DPutkonen ADe Peuter SZhang SJokinen JHowes AOulasvirta A(2024)A Workflow for Building Computationally Rational Models of Human BehaviorComputational Brain & Behavior10.1007/s42113-024-00208-6Online publication date: 15-Aug-2024
https://doi.org/10.1007/s42113-024-00208-6
Aljasem DHowes A(2024)Understanding the Effects of Visual Impairment on Visual SearchUniversal Access in Human-Computer Interaction10.1007/978-3-031-60884-1_25(363-381)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-60884-1_25
Hämäläinen AÇelikok MKaski SEvans RShpitser I(2023)Differentiable user modelsProceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence10.5555/3625834.3625909(798-808)Online publication date: 31-Jul-2023
https://dl.acm.org/doi/10.5555/3625834.3625909
Keurulainen AWesterlund IKeurulainen OHowes A(2023)Amortised Experimental Design and Parameter Estimation for User Models of PointingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581483(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581483
Moon HOulasvirta ALee B(2023)Amortized Inference with User SimulationsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581439(1-20)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581439
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents