research-article

Pictorial usability inventory (PUI): a pilot study

Authors:

Jürgen Baumgartner,

Andreas SondereggerAuthors Info & Claims

MuC '20: Proceedings of Mensch und Computer 2020

Pages 43 - 52

https://doi.org/10.1145/3404983.3405522

Published: 06 September 2020 Publication History

Abstract

The Pictorial Usability Inventory (PUI) is an image-based instrument that was designed for the assessment of perceived usability of smartphone apps. The aim was to provide users with an easy and intuitive access to the evaluation of perceived usability. PUI was developed in several iterations using a user-centred approach. The pictorial items were based on selected items of various standardized usability questionnaires. A pilot study with 83 participants was conducted online to test the psychometric properties of the newly developed instrument. The factor structure, reliability, convergent, divergent and criterion-related validity, and sensitivity of the instrument were evaluated. Furthermore, respondent motivation, questionnaire preference, and questionnaire completion time were assessed. Participants interacted first with a prototype of the app in which the product-inherent usability was manipulated on two levels. The manipulation was implemented using a between-subjects design. Afterwards, participants completed the PUI and further verbal instruments to evaluate usability, as well as further questionnaires to determine the different forms of validity. First results indicate good psychometric qualities of the scale and an increased motivation among participants when completing the questionnaire. Nevertheless, 60% of the participants preferred the verbal scales. In addition, questionnaire completion time was longer for verbal scales. Some ideas for the further development of the PUI are discussed.

References

[1]

Ahlem Assila and Houcine Ezzedine. 2016. Standardized usability questionnaires: Features and quality focus. Electronic Journal of Computer Science and Information Technology: eJCIST 6, 1 (2016).

[2]

Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies 4, 3 (2009), 114--123.

Digital Library

[3]

Javier A. Bargas-Avila, Jonas Lötscher, Sébastien Orsini, and Klaus Opwis. 2009. Intranet satisfaction questionnaire: Development and validation of a questionnaire to measure user satisfaction with the Intranet. Computers in Human Behavior 25, 6 (November 2009), 1241--1250.

Digital Library

[4]

Juergen Baumgartner, Naomi Frei, Mascha Kleinke, Juergen Sauer, and Andreas Sonderegger. 2019. Pictorial System Usability Scale (P-SUS) Developing an Instrument for Measuring Perceived Usability. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1--11.

Digital Library

[5]

Juergen Baumgartner, Andreas Sonderegger, and Juergen Sauer. 2019. No need to read: Developing a pictorial single-item scale for measuring perceived usability. International Journal of Human-Computer Studies 122, (2019), 78--89.

[6]

Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1 (1994), 49--59.

[7]

John Brooke. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4--7.

[8]

John Brooke. 2013. SUS: A Retrospective. J. Usability Studies 8, 2 (February 2013), 29--40.

Digital Library

[9]

Fred D. Davis. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly (1989), 319--340.

Digital Library

[10]

Nicola Döring and Jürgen Bortz. 2016. Forschungsmethoden und Evaluation. Wiesbaden: Springerverlag (2016).

[11]

Kraig Finstad. 2010. The Usability Metric for User Experience. Interacting with Computers 22, 5 (September 2010), 323--327.

Digital Library

[12]

Meiyuzi Gao and Philip Kortum. 2015. The relationship between subjective and objective usability metrics for home healthcare devices. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, SAGE Publications Sage CA: Los Angeles, CA, 1001--1005.

[13]

Günther Gediga, Kai-Christoph Hamborg, and Ivo Düntsch. 1999. The IsoMetrics usability inventory: an operationalization of ISO 9241-10 supporting summative and formative evaluation of software systems. Behaviour & information technology 18, 3 (1999), 151--164.

[14]

Joseph F. Hair, William C. Black, Barry J. Babin, and Rolph E. Anderson. 2014. Multivariate Data Analysis. Pearson, Essex.

[15]

Marc Hassenzahl, Michael Burmester, and Franz Koller. 2003. AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In Mensch & computer 2003. Springer, 187--196.

[16]

Matt C. Howard. 2018. Scale pretesting. Practical Assessment, Research, and Evaluation 23, 1 (2018), 5.

[17]

International Organization for Standardization. 2016. Ergonomics of human-system interaction - Part 11: Usability: Definitions and concepts (Standard No. 9241-11.2). (2016). Retrieved from https://www.iso.org/standard/63500.html

[18]

J. Kirakowski, M. Corbett, and M. Sumi. 1993. The software usability measurement inventory. Br J Educ Technol 24, 3 (1993), 210--2.

[19]

Jurek Kirakowski and Bozena Cierlik. 1998. Measuring the Usability of Web Sites. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 42, 4 (October 1998), 424--428.

[20]

Clayton Lewis and Robert Mack. 1982. Learning to use a text processing system: Evidence from "thinking aloud" protocols. In Proceedings of the 1982 conference on Human factors in computing systems, 387--392.

Digital Library

[21]

James R. Lewis. 1995. IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction 7, 1 (January 1995), 57--78.

Digital Library

[22]

James R. Lewis. 2002. Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies. International Journal of Human-Computer Interaction 14, 3--4 (September 2002), 463--488.

[23]

James R. Lewis, Brian S. Utesch, and Deborah E. Maher. 2013. UMUX-LITE: When There's No Time for the SUS. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), ACM, New York, NY, USA, 2099--2102.

Digital Library

[24]

James R. Lewis, Brian S. Utesch, and Deborah E. Maher. 2015. Measuring Perceived Usability: The SUS, UMUX-LITE, and AltUsability. International Journal of Human-Computer Interaction 31, 8 (August 2015), 496--505.

[25]

Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In Proceedings of the 11 th Australasian Conference on Information Systems, 6--8.

[26]

Albert Mehrabian and James A. Russell. 1974. An approach to environmental psychology. The MIT Press, Cambridge, MA, US.

[27]

Michael Minge and Laura Riedel. 2013. meCUE-Ein modularer fragebogen zur erfassung des nutzungserlebens. Mensch & Computer 2013: Interaktive Vielfalt (2013).

[28]

Morten Moshagen and Meinald Thielsch. 2013. A short version of the visual aesthetics of websites inventory. Behaviour & Information Technology 32, 12 (2013), 1305--1311.

Digital Library

[29]

Jakob Nielsen and Rolf Molich. 1990. Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, 249--256.

Digital Library

[30]

Bernard Rummel. 2015. System Usability Scale - jetzt auch auf Deutsch. SAP User Experience Community. Retrieved December 20, 2019 from https://experience.sap.com/skillup/system-usability-scale-jetzt-auch-auf-deutsch/

[31]

Juergen Sauer, Juergen Baumgartner, Naomi Frei, and Andreas Sonderegger. in press. Pictorial scales in research and practice: a review. European Psychologist (in press).

[32]

Jeff Sauro. 2015. SUPR-Q: A comprehensive measure of the quality of the website user experience. Journal of usability studies 10, 2 (2015).

[33]

Jeff Sauro and James R. Lewis. 2009. Correlations among prototypical usability metrics: evidence for the construct of usability. In Proceedings of the SIGCHI conference on human factors in computing systems, ACM, 1609--1618.

[34]

Felix D. Schönbrodt and Marco Perugini. 2013. At what sample size do correlations stabilize? Journal of Research in Personality 47, 5 (October 2013), 609--612.

[35]

Martin Schrepp, Andreas Hinderks, and Jörg Thomaschewski. 2014. Applying the user experience questionnaire (UEQ) in different evaluation scenarios. In International Conference of Design, User Experience, and Usability, Springer, 383--392.

[36]

Andreas Sonderegger, Klaus Heyden, Alain Chavaillaz, and Juergen Sauer. 2016. AniSAM & AniAvatar: Animated visualizations of affective states. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, 4828--4837.

Digital Library

[37]

Andreas Sonderegger, Andreas Uebelbacher, Manuela Pugliese, and Juergen Sauer. 2014. The influence of aesthetics in usability testing: the case of dual-domain products. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 21--30.

Digital Library

[38]

Andreas Sonderegger, Andreas Uebelbacher, and Jürgen Sauer. 2019. The UX Construct-Does the Usage Context Influence the Outcome of User Experience Evaluations? In IFIP Conference on Human-Computer Interaction, Springer, 140--157.

Digital Library

[39]

Statista. Smartphone-Nutzung in der Schweiz. de.statista.com. Retrieved April 7, 2020 from https://de.statista.com/themen/3581/smartphone-nutzung-in-der-schweiz/

[40]

Statista. Österreich - Besitz von Smartphones bis 2019. de.statista.com. Retrieved April 7, 2020 from https://de.statista.com/statistik/daten/studie/322885/umfrage/umfrage-zum-besitz-von-smartphones-in-oesterreich/

[41]

Statista. Smartphones - Anteil der Nutzer in Deutschland 2018. de.statista.com. Retrieved April 7, 2020 from https://de.statista.com/statistik/daten/studie/585883/umfrage/anteil-der-smartphone-nutzer-in-deutschland/

[42]

Statista. Anzahl der Apps in den Top App-Stores 2019. de.statista.com. Retrieved April 7, 2020 from https://de.statista.com/statistik/daten/studie/208599/umfrage/anzahl-der-apps-in-den-top-app-stores/

[43]

Noam Tractinsky, Adi S. Katz, and Dror Ikar. 2000. What is beautiful is usable. Interacting with computers 13, 2 (2000), 127--145.

[44]

Matthias Wilde, Katrin Bätz, Anastassiya Kovaleva, and Detlef Urhahne. 2009. Überprüfung einer Kurzskala intrinsischer Motivation (KIM). Zeitschrift für Didaktik der Naturwissenschaften 15, (2009).

Cited By

Baumgartner JSonderegger ASauer J(2024)To Animate or Not to Animate Usability Scales: The Effect of Animation on Questionnaire Experience and Psychometric PropertiesInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2338333(1-12)Online publication date: 15-Apr-2024
https://doi.org/10.1080/10447318.2024.2338333

Index Terms

Pictorial usability inventory (PUI): a pilot study
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Pictorial System Usability Scale (P-SUS): Developing an Instrument for Measuring Perceived Usability
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

We have developed a pictorial multi-item scale, called P-SUS (Pictorial System Usability Scale), which aims to measure the perceived usability of mobile devices. The scale is based on the established verbal usability questionnaire SUS (System Usability ...
Questionnaire experience of the pictorial usability inventory (PUI) – a comparison of pictorial and hybrid usability scales
Highlights
- This article compares pictorial, hybrid and verbal usability questionnaires using classical psychometric parameters and respondent-centred measures.
- A recently developed questionnaire measuring ‘questionnaire experience’ was used to ...
Abstract
In recent years, alternative types of usability questionnaires using graphical elements (pictorial scales) or a combination of graphical and verbal elements (hybrid scales) have been introduced. Previous research indicates that these ...
Usability evaluation of a virtual reality smartphone app for a living museum
Abstract
This paper elaborates the empirical evidence of a usability evaluation of a VR and non-VR virtual tour application for a living museum. The System Usability Scale (SUS) was used in between participants experiments (Group 1: non-VR version and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MuC '20: Proceedings of Mensch und Computer 2020

September 2020

523 pages

ISBN:9781450375405

DOI:10.1145/3404983

General Chairs:
Bernhard Preim
Universität Magdeburg
,
Andreas Nürnberger
Universität Magdeburg
,
Christian Hansen
Universität Magdeburg

Copyright © 2020 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung

Conference

MuC'20

MuC'20: Mensch und Computer 2020

September 6 - 9, 2020

Magdeburg, Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
127
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)1

Reflects downloads up to 14 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Baumgartner JSonderegger ASauer J(2024)To Animate or Not to Animate Usability Scales: The Effect of Animation on Questionnaire Experience and Psychometric PropertiesInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2338333(1-12)Online publication date: 15-Apr-2024
https://doi.org/10.1080/10447318.2024.2338333

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents