research-article

Predicting user error for ambient systems by integrating model-based UI development and cognitive modeling

Authors:

Marc Halbrügge,

Klaus-Peter Engelbrecht,

Sebastian Möller,

Sahin AlbayrakAuthors Info & Claims

UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

Pages 1028 - 1039

https://doi.org/10.1145/2971648.2971667

Published: 12 September 2016 Publication History

Abstract

With the move to ubiquitous computing, user interfaces (UI) are no longer bound to specific devices. While this problem can be tackled using the model-based UI development (MBUID) process, the usability of the device-specific interfaces is still an open question. We are presenting a combined system that integrates MBUID with a cognitive modeling framework in order to provide usability predictions at development time. Because of their potential impact, our focus within usability problems lies on user errors. These are captured in a cognitive model that capitalizes on meta-information provided by the MBUID system such as the abstract role of a UI element within a task sequence (e.g., input, output, command). The free parameters of the cognitive model were constrained using data from two previous studies. A validation experiment featuring a new application and UI yielded an unexpected error pattern that was nonetheless consistent with the model predictions.

References

[1]

Erik M Altmann and J Gregory Trafton. 2002. Memory for goals: An activation-based model. Cognitive science 26, 1 (2002), 39--83.

[2]

Maartje GA Ament, Anna L Cox, Ann Blandford, and Duncan P Brumby. 2013. Making a task difficult: Evidence that device-oriented steps are effortful and error-prone. Journal of experimental psychology: applied 19, 3 (2013), 195.

[3]

John R Anderson, Daniel Bothell, Michael D Byrne, Scott Douglass, Christian Lebiere, and Yulin Qin. 2004. An integrated theory of the mind. Psychological review 111, 4 (2004), 1036--1060.

[4]

Douglas Bates, Martin Maechler, Ben Bolker, and Steven Walker. 2013. lme4: Linear mixed-effects models using Eigen and S4. R package version 1.0-5.

[5]

Arthur N Beare and RE Dorris. 1983. A simulator-based study of human errors in nuclear power plant control room tasks. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 27. Sage Publications, 170--174.

[6]

Marco Blumendorf, Grzegorz Lehmann, and Sahin Albayrak. 2010. Bridging Models and Systems at Runtime to Build Adaptive User Interfaces. In Proceedings of the 2Nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '10). ACM, New York, NY, USA, 9--18.

Digital Library

[7]

Matthew L Bolton, Ellen J Bass, and Radu I Siminiceanu. 2012. Generating phenotypical erroneous human behavior to evaluate human--automation interaction using model checking. International journal of human-computer studies 70, 11 (2012), 888--906.

Digital Library

[8]

Michael D Byrne. 2001. ACT-R/PM and menu selection: Applying a cognitive architecture to HCI. International Journal of Human-Computer Studies 55, 1 (2001), 41--84.

Digital Library

[9]

Michael D Byrne and Elizabeth M Davis. 2006. Task structure and postcompletion error in the execution of a routine procedure. Human Factors: The Journal of the Human Factors and Ergonomics Society 48, 4 (2006), 627--638.

[10]

Gaëlle Calvary, Joëlle Coutaz, David Thevenin, Quentin Limbourg, Laurent Bouillon, and Jean Vanderdonckt. 2003. A Unifying Reference Framework for Multi-Target User Interfaces. Interacting with Computers 15, 3 (2003), 289--308.

[11]

Tim Clerckx, Kris Luyten, and Karin Coninx. 2004. DynaMo-AID: A Design Process and a Runtime Architecture for Dynamic Model-Based User Interface Development. In Engineering Human Computer Interaction and Interactive Systems. 77--95.

Digital Library

[12]

Joëlle Coutaz and Gaëlle Calvary. 2012. HCI and software engineering for user interface plasticity. In Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications (3rd ed.), Julie A Jacko (Ed.). CRC Press, Chapter 52, 1195--1220.

[13]

Klaus-Peter Engelbrecht, Michael Kruppa, Sebastian Möller, and Michael Quade. 2008. MeMo workbench for semi-automated usability testing. In INTERSPEECH. 1662--1665.

[14]

Racim Fahssi, Célia Martinie, and Philippe Palanque. 2015. Enhanced Task Modelling for Systematic Identification and Explicit Representation of Human Errors. In Human-Computer Interaction -- INTERACT 2015, Julio Abascal, Simone Barbosa, Mirko Fetter, Tom Gross, Philippe Palanque, and Marco Winckler (Eds.). Springer, 192--212.

[15]

Juan Manuel González-Calleros, Jan Patrick Osterloh, Rene Feil, and Andreas Lüdtke. 2014. Automated UI evaluation based on a cognitive architecture and UsiXML. Science of Computer Programming Journal 86 (2014), 43--57.

[16]

Christian Gram and Gilbert Cockton (Eds.). 1997. Design principles for interactive software. Chapman & Hall, Ltd., London, UK.

Digital Library

[17]

Wayne D. Gray. 2000. The Nature and Processing of Errors in Interactive Behavior. Cognitive Science 24, 2 (2000), 205--248.

[18]

Marc Halbrügge, Michael Quade, and Klaus-Peter Engelbrecht. 2015a. How can Cognitive Modeling Benefit from Ontologies? Evidence from the HCI Domain. In AGI 2015, Jordi Bieger, Ben Goertzel, and Alexey Potapov (Eds.). LNAI, Vol. 9205. Springer, Berlin, 261--271.

Digital Library

[19]

Marc Halbrügge, Michael Quade, and Klaus-Peter Engelbrecht. 2015b. A Predictive Model of Human Error based on User Interface Development Models and a Cognitive Architecture. In Proceedings of the 13th International Conference on Cognitive Modeling, Niels A Taatgen, Marieke K van Vugt, Jelmer P Borst, and Katja Mehlhorn (Eds.). University of Groningen, Groningen, the Netherlands, 238--243.

[20]

Marc Halbrügge, Michael Quade, and Klaus-Peter Engelbrecht. 2016. Cognitive Strategies in HCI and Their Implications on User Error. In Proceedings of the 38th Annual Meeting of the Cognitive Science Society. in press.

[21]

Laura M Hiatt and J Gregory Trafton. 2015. An Activation-Based Model of Routine Sequence Errors. In Proceedings of the 13th International Conference on Cognitive Modeling, Niels A Taatgen, Marieke K van Vugt, Jelmer P Borst, and Katja Mehlhorn (Eds.). University of Groningen, Groningen, the Netherlands, 244--249.

[22]

Kimberley Hiltz, Jonathan Back, and Ann Blandford. 2010. The roles of conceptual device models and user goals in avoiding device initialization errors. Interacting with Computers 22, 5 (2010), 363--374.

Digital Library

[23]

Erik Hollnagel. 1998. Cognitive reliability and error analysis method (CREAM). Elsevier, Oxford, UK.

[24]

Bonnie E John, Evan W Patton, Wayne D Gray, and Donald F Morrison. 2012. Tools for Predicting the Duration and Variability of Skilled Performance without Skilled Performers. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 56. SAGE Publications, 985--989.

[25]

David Kieras. 1997. A Guide to GOMS Model Usability Evaluation using NGOMSL. In Handbook of Human-Computer Interaction (2nd ed.), Marting G Helander, Thomas K Landauer, and Prasad V Prabhu (Eds.). North-Holland, Amsterdam, Chapter 31, 733 -- 766.

[26]

Barry Kirwan. 1997. Validation of human reliability assessment techniques: Part 2 -- Validation results. Safety Science 27, 1 (1997), 43--75.

[27]

Simon YW Li, Ann Blandford, Paul Cairns, and Richard M Young. 2008. The effect of interruptions on postcompletion and other procedural errors: an account based on the activation-based goal memory model. Journal of Experimental Psychology: Applied 14, 4 (2008), 314.

[28]

Quentin Limbourg, Jean Vanderdonckt, Benjamin Michotte, Laurent Bouillon, and Victor Lopez-Jaquero. 2005. USIXML: A Language Supporting Multi-path Development of User Interfaces. In Engineering Human Computer Interaction and Interactive Systems, Remi Bastide, Philippe Palanque, and Joerg Roth (Eds.). LNCS, Vol. 3425. Springer, Berlin, 200--220.

Digital Library

[29]

Joaquin Miller and Jishnu Mukerji. 2001. Model Driven Architecture (MDA). Technical Report ormsc/2001-07-01. Object Management Group, Architecture Board ORMSC. http://www.omg.org/cgi-bin/doc?ormsc/01-07-01.pdf

[30]

Giulio Mori, Fabio Paternò, and Carmen Santoro. 2004. Design and Development of Multidevice User Interfaces through Multiple Logical Descriptions. IEEE Trans. Softw. Eng. 30, 8 (2004), 507--520.

Digital Library

[31]

Donald A Norman. 2002. The design of everyday things. Basic books.

Digital Library

[32]

Philippe Palanque and Sandra Basnyat. 2004. Task patterns for taking into account in an efficient and systematic way both standard and erroneous user behaviours. In Human Error, Safety and Systems Development. Springer, 109--130.

[33]

Fabio Paternò. 2003. ConcurTaskTrees: An Engineered Notation for Task Models. In The Handbook of Task Analysis for Human-Computer Interaction, Dan Diaper and Neville Stanton (Eds.). Lawrence Erlbaum Associates, Mahwah, NJ, 483--501.

[34]

Fabio Paternò. 2005. Model-based Tools for Pervasive Usability. Interacting with Computers 17, 3 (2005), 291--315. http://dx.doi.org/10.1016/j.intcom.2004.06.017

[35]

Fabio Paternò and Carmen Santoro. 2002. Preventing user errors by systematic analysis of deviations from the system task model. International Journal of Human-Computer Studies 56, 2 (2002), 225--245.

Digital Library

[36]

Till Plumbaum, Sascha Narr, Elif Eryilmaz, Frank Hopfgartner, Funda Klein-Ellinghaus, Anna Reese, and Sahin Albayrak. 2014. Providing Multilingual Access to Health-Related Content. In eHealth -- For Continuity of Care: Proceedings of MIE2014. IOS Press, Amsterdam, NL, 393--397.

[37]

Michael Quade. 2015. Automation in Model-based Usability Evaluation of Adaptive User Interfaces by Simulating User Interaction. Ph.D. Dissertation. Fakultät IV, Technische Universität Berlin.

[38]

Michael Quade, Marc Halbrügge, Klaus-Peter Engelbrecht, Sahin Albayrak, and Sebastian Möller. 2014. Predicting Task Execution Times by Deriving Enhanced Cognitive Models from User Interface Development Models. In Proceedings of the 2014 ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '14). ACM, New York, NY, USA, 139--148.

Digital Library

[39]

Michael Quade, Grzegorz Lehmann, Klaus-Peter Engelbrecht, Dirk Roscher, and Sahin Albayrak. 2013. Automated Usability Evaluation of Model-Based Adaptive User Interfaces for Users with Special and Specific Needs by Simulating User Interaction. In User Modeling and Adaptation for Daily Routines, Estefania Martin, Pablo A Haya, and Rosa M Carro (Eds.). Springer, 219--247.

[40]

Jens Rasmussen. 1983. Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. Systems, Man and Cybernetics, IEEE Transactions on 13 (1983), 257--266. Issue 3.

[41]

James Reason. 1990. Human Error. Cambridge University Press, New York, NY.

[42]

Rimvydas Rukšėnas, Paul Curzon, Ann Blandford, and Jonathan Back. 2014. Combining human error verification and timing analysis: a case study on an infusion pump. Formal Aspects of Computing (2014).

[43]

Dario D Salvucci. 2010. On reconstruction of task context after interruption. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 89--92.

Digital Library

[44]

Mario Sanchez, Ivan Barrero, Jorge Villalobos, and Dirk Deridder. 2008. An Execution Platform for Extensible Runtime Models. In 3rd Int. Workshop on Models at Runtime at MoDELS'08.

[45]

Holger Schultheis, Thomas Barkowsky, and Sven Bertel. 2006. LTM C -- an improved long-term memory for cognitive architectures. In Proceedings of the Seventh International Conference on Cognitive Modeling. 274--279.

[46]

Jean-Sebastien Sottet, Gaelle Calvary, Joelle Coutaz, and Jean-Marie Favre. 2008. A Model-Driven Engineering Approach for the Usability of Plastic User Interfaces. In Engineering Interactive Systems, Jan Gulliksen, MortonBorup Harning, Philippe Palanque, GerritC. van der Veer, and Janet Wesson (Eds.). LNCS, Vol. 4940. Springer, Berlin, 140--157.

Digital Library

[47]

Neville A. Stanton. 2003. The Human-computer Interaction Handbook. L. Erlbaum Associates Inc., Hillsdale, NJ, USA, Chapter Human Error Identification in Human-computer Interaction, 371--383. http://dl.acm.org/citation.cfm?id=772072.772097

Digital Library

[48]

Terrence C Stewart and Robert L West. 2010. Testing for equivalence: a methodology for computational cognitive modelling. Journal of Artificial General Intelligence 2, 2 (2010), 69--87.

[49]

Leonghwee Teo and Bonnie E John. 2008. Towards a tool for predicting goal-directed exploratory behavior. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 52. SAGE Publications, 950--954.

[50]

J Gregory Trafton, Erik M Altmann, and Raj M Ratwani. 2011. A memory for goals model of sequence errors. Cognitive Systems Research 12 (2011), 134--143.

Digital Library

[51]

J Gregory Trafton and Raj M Ratwani. 2014. The law of unintended consequences: The case of external subgoal support. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 1767--1776.

Digital Library

[52]

Jean Vanderdonckt. 2008. Model-Driven Engineering of User Interfaces: Promises, Successes, Failures, and Challenges. In Proc. ROCHI 2008.

[53]

Scott D. Wood and David E. Kieras. 2002. Modeling Human Error For Experimentation, Training, And Error-Tolerant Design. In In Proceedings of the Interservice/Industry Training, Simulation and Education Conference.

Cited By

Ntoa S(2024)Usability and User Experience Evaluation in Intelligent Environments: A Review and ReappraisalInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2394724(1-30)Online publication date: 12-Sep-2024
https://doi.org/10.1080/10447318.2024.2394724
Stephanidis CAntona MNtoa S(2021)HUMAN FACTORS IN AMBIENT INTELLIGENCE ENVIRONMENTSHANDBOOK OF HUMAN FACTORS AND ERGONOMICS10.1002/9781119636113.ch41(1058-1084)Online publication date: 13-Aug-2021
https://doi.org/10.1002/9781119636113.ch41
Manca MPaternò FSantoro C(2018)A model-based framework for mobile apps customization through context-dependent rulesUniversal Access in the Information Society10.1007/s10209-018-0620-xOnline publication date: 6-Jun-2018
https://doi.org/10.1007/s10209-018-0620-x
Show More Cited By

Index Terms

Predicting user error for ambient systems by integrating model-based UI development and cognitive modeling
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Predicting task execution times by deriving enhanced cognitive models from user interface development models
EICS '14: Proceedings of the 2014 ACM SIGCHI symposium on Engineering interactive computing systems

Adaptive user interfaces (UI) offer the opportunity to adapt to changes in the context, but this also poses the challenge of evaluating the usability of many different versions of the resulting UI. Consequently, usability evaluations tend to become very ...
A design-and-play approach to accessible user interface development in Ambient Intelligence environments

User interface development in Ambient Intelligence (AmI) environments is anticipated to be a particularly complex and programming intensive endeavor. Additionally, AmI environments should ensure accessibility and usability of interactive technologies by ...
Automated Usability Evaluation during Model-Based Interactive System Development
HCSE-TAMODIA '08: Proceedings of the 2nd Conference on Human-Centered Software Engineering and 7th International Workshop on Task Models and Diagrams

In this paper we describe an approach to efficiently evaluate the usability of an interactive application that has been realized to support various platforms and modalities. Therefore we combine our Multi-Access Service Platform (MASP), a model-based ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 2016

1288 pages

ISBN:9781450344616

DOI:10.1145/2971648

General Chairs:
Paul Lukowicz
DFKI and University of Kaiserslautern
,
Antonio Krüger
DFKI and Saarland University
,
Program Chairs:
Andreas Bulling
Max Planck Institute for Informatics
,
Youn-Kyung Lim
KAIST
,
Shwetak N. Patel
University of Washington

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 September 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UbiComp '16

Sponsor:

UbiComp '16: The 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 12 - 16, 2016

Germany, Heidelberg

Acceptance Rates

UbiComp '16 Paper Acceptance Rate 101 of 389 submissions, 26%;

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
261
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ntoa S(2024)Usability and User Experience Evaluation in Intelligent Environments: A Review and ReappraisalInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2394724(1-30)Online publication date: 12-Sep-2024
https://doi.org/10.1080/10447318.2024.2394724
Stephanidis CAntona MNtoa S(2021)HUMAN FACTORS IN AMBIENT INTELLIGENCE ENVIRONMENTSHANDBOOK OF HUMAN FACTORS AND ERGONOMICS10.1002/9781119636113.ch41(1058-1084)Online publication date: 13-Aug-2021
https://doi.org/10.1002/9781119636113.ch41
Manca MPaternò FSantoro C(2018)A model-based framework for mobile apps customization through context-dependent rulesUniversal Access in the Information Society10.1007/s10209-018-0620-xOnline publication date: 6-Jun-2018
https://doi.org/10.1007/s10209-018-0620-x
Vigo MSantoro CPaterno F(2017)The usability of task modeling tools2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VLHCC.2017.8103455(95-99)Online publication date: Oct-2017
https://doi.org/10.1109/VLHCC.2017.8103455
Szabó G(2017)Usability of MachineryAdvances in Safety Management and Human Factors10.1007/978-3-319-60525-8_17(161-168)Online publication date: 23-Jun-2017
https://doi.org/10.1007/978-3-319-60525-8_17

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents