research-article

Beyond Usability and Performance: A Review of User Experience-focused Evaluations in Visualization

Authors:

John StaskoAuthors Info & Claims

BELIV '16: Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization

Pages 133 - 142

https://doi.org/10.1145/2993901.2993903

Published: 24 October 2016 Publication History

Abstract

Traditionally, studies of data visualization techniques and systems have evaluated visualizations with respect to usability goals such as effectiveness and efficiency. These studies assess performance-related metrics such as time and correctness of participants completing analytic tasks. Alternatively, several studies in InfoVis recently have evaluated visualizations by investigating user experience goals such as memorability, engagement, enjoyment and fun. These studies employ somewhat different evaluation methodologies to assess these other goals. The growing number of these studies, their alternative methodologies, and disagreements concerning their importance have motivated us to more carefully examine them. In this article, we review this growing collection of visualization evaluations that examine user experience goals and we discuss multiple issues regarding the studies including questions about their motivation and utility. Our aim is to provide a resource for future work that plans to evaluate visualizations using these goals.

References

[1]

Baby Name Wizard - Name Voyager. http://www.babynamewizard.com/voyager.

[2]

S. Attfield, G. Kazai, M. Lalmas, and B. Piwowarski. Towards a science of user engagement (position paper). In WSDM workshop on user modelling for Web applications, Feb. 2011.

[3]

D. Baldauf, E. Burgard, and M. Wittmann. Time perception as a workload measure in simulated car driving. Applied ergonomics, 40(5):929--935, 2009.

[4]

C. M. Barnum and L. A. Palmer. More than a feeling: understanding the desirability factor in user experience. In CHI'10 Extended Abstracts on Human Factors in Computing Systems, pages 4703--4716. ACM, 2010.

Digital Library

[5]

S. Bateman, R. L. Mandryk, C. Gutwin, A. Genest, D. McDine, and C. Brooks. Useful junk?: The effects of visual embellishment on comprehension and memorability of charts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '10, pages 2573--2582, New York, NY, USA, 2010. ACM.

Digital Library

[6]

J. Benedek and T. Miner. Measuring desirability: New methods for evaluating desirability in a usability lab setting. In Proceedings of Usability Professionals Association, pages 8--12, July 2002.

[7]

R. Borgo, A. Abdul-Rahman, F. Mohamed, P. W. Grant, I. Reppa, L. Floridi, and M. Chen. An empirical study on using visual embellishments in visualization. IEEE Transactions on Visualization and Computer Graphics, 18(12):2759--2768, 2012.

Digital Library

[8]

M. Borkin, Z. Bylinskii, N. Kim, C. Bainbridge, C. Yeh, D. Borkin, H. Pfister, and A. Oliva. Beyond memorability: Visualization recognition and recall. IEEE Transactions on Visualization and Computer Graphics, 22(1):519--528, Jan 2016.

Digital Library

[9]

M. Borkin, A. Vo, Z. Bylinskii, P. Isola, S. Sunkavalli, A. Oliva, H. Pfister, et al. What makes a visualization memorable? IEEE Transactions on Visualization and Computer Graphics, 19(12):2306--2315, 2013.

Digital Library

[10]

J. Boy, F. Detienne, and J.-D. Fekete. Storytelling in information visualizations: Does it engage users to explore data? In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI '15, pages 1449--1458, New York, NY, USA, 2015. ACM.

Digital Library

[11]

P. B. Brandtzæg, A. Følstad, and J. Heim. Enjoyment: lessons from karasek. In Funology, pages 55--65. Springer, 2005.

[12]

M. Brehmer and T. Munzner. A multi-level typology of abstract visualization tasks. IEEE Transactions on Visualization and Computer Graphics, 19(12):2376--2385, 2013.

Digital Library

[13]

J. Brown, V. Lewis, and A. Monk. Memorability, word frequency and negative recognition. The Quarterly Journal of Experimental Psychology, 29(3):461--473, 1977.

[14]

S. Carpendale. Evaluating information visualizations. In A. Kerren, J. T. Stasko, J.-D. Fekete, and C. North, editors, Information Visualization, pages 19--45. Springer-Verlag, Berlin, Heidelberg, 2008.

Digital Library

[15]

M. Csikszentmihalyi. Flow: The Psychology of Optimal Experience. Harper Perennia, New York, 1990.

[16]

W. A. Davis. A causal theory of enjoyment. Mind, 91(362):240--256, 1982.

[17]

A. Dix. Human-computer interaction. Springer, 2009.

Digital Library

[18]

N. Elmqvist, A. Vande Moere, H.-C. Jetter, D. Cernea, H. Reiterer, and T. J. Jankun-Kelly. Fluid interaction for information visualization. Information Visualization, 10(4):327--340, Oct. 2011.

Digital Library

[19]

X. Feng, S. Chan, J. Brzezinski, and C. Nair. Measuring enjoyment of computer game play. In Proceedings of AMCIS 2008, 2008.

[20]

S. Few. Review of the Research Study "What Makes a Visualization Memorable?", 2013 (accessed February 3, 2016). https://www.perceptualedge.com/blog/?p=1770.

[21]

S. Few. Information Visualization Research as Pseudo-Science, 2015 (accessed February 3, 2016). https://www.perceptualedge.com/articles/visual_business_intelligence/infovis_research_as_pseudo-science.pdf.

[22]

B. Fredrickson. What good are positive emotions? Review of General Psychology, 3:300--3019, 1998.

[23]

F.-L. Fu, R.-C. Su, and S.-C. Yu. Egameflow: A scale to measure learners' enjoyment of e-learning games. Computers & Education, 52(1):101--112, Jan. 2009.

Digital Library

[24]

S. Haroz, R. Kosara, and S. L. Franconeri. Isotype visualization: Working memory, performance, and engagement with pictographs. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI '15, pages 1191--1200, New York, NY, USA, 2015. ACM.

Digital Library

[25]

L. Harrison, D. Skau, S. Franconeri, A. Lu, and R. Chang. Influencing visual judgment through affective priming. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '13, pages 2949--2958, New York, NY, USA, 2013. ACM.

Digital Library

[26]

J. Heer, F. van Ham, S. Carpendale, C. Weaver, and P. Isenberg. Creation and collaboration: Engaging new audiences for information visualization. In A. Kerren, J. T. Stasko, J.-D. Fekete, and C. North, editors, Information Visualization, pages 92--133. Springer-Verlag, Berlin, Heidelberg, 2008.

Digital Library

[27]

J. Hullman and N. Diakopoulos. Visualization rhetoric: Framing effects in narrative visualization. IEEE Transactions on Visualization and Computer Graphics, 17(12):2231--2240, Dec. 2011.

Digital Library

[28]

A. Isen. Positive affect facilitates creative problem solving. Journal of Personality and Social Psychology, 52(6):1122--1131, 1987.

[29]

R. Jianu, A. Rusu, Y. Hu, and D. Taggart. How to display group information on node-link diagrams: an evaluation. IEEE Transactions on Visualization and Computer Graphics, 20(11):1530--1541, 2014.

[30]

B. Jones. When Memorability Matters: Another Practitioner's View, 2015 (accessed February 3, 2016). http://dataremixed.com/2015/12/when-memorability-matters-another-practitioners-view/.

[31]

R. Kosara. Memorability, Science, and The Value of Thinking Outside the Box, 2015 (accessed February 3, 2016). https://eagereyes.org/blog/2015/memorability-science-and-the-value-of-thinking-outside-the-box.

[32]

R. Kosara and J. Mackinlay. Storytelling: The Next Step for Visualization. Computer, 46(5):44--50, May 2013.

Digital Library

[33]

H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale. Empirical studies in information visualization: Seven scenarios. IEEE Transactions on Visualization and Computer Graphics, 18(9):1520--1536, Sept. 2012.

Digital Library

[34]

H. Li and N. Moacdieh. Is "chart junk" useful? an extended examination of visual embellishment. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, volume 58, pages 1516--1520. SAGE Publications, 2014.

[35]

N. Mahyar, S.-H. Kim, and B. C. Kwon. Towards a taxonomy for evaluating user engagement in information visualization. Workshop on Personal Visualization: Exploring Everyday Life, 2015.

[36]

T. W. Malone. Toward a theory of intrinsically motivating instruction. Cognitive science, 5(4):333--369, 1981.

[37]

T. W. Malone. Heuristics for designing enjoyable user interfaces: Lessons from computer games. In Proceedings of the 1982 Conference on Human Factors in Computing Systems, CHI '82, pages 63--68, New York, NY, USA, 1982. ACM.

Digital Library

[38]

K. Marriott, H. Purchase, M. Wybrow, and C. Goncu. Memorability of visual features in network diagrams. IEEE Transactions on Visualization and Computer Graphics, 18(12):2477--2485, 2012.

Digital Library

[39]

T. Merčun. Evaluation of information visualization techniques: Analysing user experience with reaction cards. In Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization, BELIV '14, pages 103--109, New York, NY, USA, 2014. ACM.

Digital Library

[40]

T. Munzner. A nested model for visualization design and validation. IEEE Transactions on Visualization and Computer Graphics, 15(6):921--928, 2009.

Digital Library

[41]

D. A. Norman. Emotional design: Why we love (or hate) everyday things. Basic books, 2004.

[42]

D. A. Norman. The design of everyday things: Revised and expanded edition. Basic books, 2013.

[43]

H. L. O'Brien and E. G. Toms. What is user engagement? a conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59(6):938--955, 2008.

Digital Library

[44]

H. L. O'Brien and E. G. Toms. The development and evaluation of a survey to measure user engagement. Journal of the American Society for Information Science and Technology, 61(1):50--69, 2010.

Digital Library

[45]

F. Pachet and A. R. Addessi. When children reflect on their own playing style: Experiments with continuator and children. Comput. Entertain., 2(1):14--14, 2004.

Digital Library

[46]

R. Parncutt and G. E. McPherson. The Science & Psychology of Music Performance: Creative Strategies for Teaching and Learning Book. Oxford University Press, 2009.

[47]

W. A. Pike, J. Stasko, R. Chang, and T. A. O'Connell. The science of interaction. Information Visualization, 8(4):263--274, Dec. 2009.

Digital Library

[48]

K. Rathunde and M. Csikszentmihalyi. Middle school students's motivation and quality of experience: A comparison of montessori and traditional school environments. American Journal of Education, 111:341--371, 2005.

[49]

J. C. Read, S. MacFarlane, and C. Casey. Endurability, engagement and expectations: Measuring children's fun. In Interaction design and children, volume 2, pages 1--23. Shaker Publishing Eindhoven, 2002.

[50]

B. Saket, C. Scheidegger, and S. Kobourov. Towards understanding enjoyment and flow in information visualization (short paper). In EuroVis 2015 Conference, 2015.

[51]

B. Saket, C. Scheidegger, and S. G. Kobourov. Comparing node-link and node-link-group visualizations from an enjoyment perspective. Computer Graphics Forum, 35(3):41--50, 2016.

Digital Library

[52]

B. Saket, C. Scheidegger, S. G. Kobourov, and K. Börner. Map-based visualizations increase recall accuracy of data. Computer Graphics Forum, 34(3):441--450, 2015.

Digital Library

[53]

E. Segel and J. Heer. Narrative Visualization: Telling Stories with Data. IEEE Transactions on Visualization and Computer Graphics, 16(6):1139--1148, Dec. 2010.

Digital Library

[54]

H. Sharp, Y. Rogers, and J. Preece. Interaction Design: Beyond Human Computer Interaction. John Wiley & Sons, 2007.

Digital Library

[55]

B. Shneiderman. Designing for fun: how can we design user interfaces to be more fun? interactions, 11(5):48--50, 2004.

Digital Library

[56]

P. Sweetser and P. Wyeth. Gameflow: A model for evaluating player enjoyment in games. Computers in Entertainment, 3(3), July 2005.

Digital Library

[57]

M. Vass, J. M. Carroll, and C. A. Shaffer. Supporting creativity in problem solving environments. In Proceedings of the 4th Conference on Creativity & Cognition, pages 31--37, New York, NY, USA, 2002. ACM.

Digital Library

[58]

M. Wattenberg and J. Kriss. Designing for social data analysis. IEEE Transactions on Visualization and Computer Graphics, 12(4):549--557, July 2006.

Digital Library

[59]

L. Xie, A. N. Antle, and N. Motamedi. Are tangibles more fun?: Comparing children's enjoyment and engagement using physical, graphical and tangible user interfaces. In Proceedings of the 2nd international conference on Tangible and embedded interaction, pages 191--198. ACM, 2008.

Digital Library

[60]

J. S. Yi, Y. a. Kang, J. Stasko, and J. Jacko. Toward a deeper understanding of the role of interaction in information visualization. IEEE Transactions on Visualization and Computer Graphics, 13(6):1224--1231, Nov. 2007.

Digital Library

Cited By

Cabouat AHe TIsenberg PIsenberg T(2025)PREVis: Perceived Readability Evaluation for VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345631831:1(1083-1093)Online publication date: Jan-2025
https://doi.org/10.1109/TVCG.2024.3456318
Garreton MMoyano DGuerra JParra D(2024)Survey of data stories: Guidelines for data story authoringInformation Visualization10.1177/14738716241287116Online publication date: 23-Oct-2024
https://doi.org/10.1177/14738716241287116
Sinaei Hamed MKwan PKlich MAurisano JRajabiyazdi F(2024)The Elephant in the Room: Expert Experiences Designing, Developing and Evaluating Data Visualizations on Large DisplaysProceedings of the ACM on Human-Computer Interaction10.1145/36981398:ISS(301-329)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698139
Show More Cited By

Beyond Usability and Performance: A Review of User Experience-focused Evaluations in Visualization
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Evaluating Engagement in Reading: Comparing Children and Adult Assessors
IDC '16: Proceedings of the The 15th International Conference on Interaction Design and Children

This paper describes the findings of a study into how children engage with and enjoy reading digital stories. We considered stories created by children with an application called Fiabot! that we designed to support the creation of multimedia fairy tales ...
The experience of interactive storytelling: comparing “fahrenheit” with “façade”
ICEC'11: Proceedings of the 10th international conference on Entertainment Computing

At the intersection of multimedia, artificial intelligence, and gaming technology, new visions of future entertainment media arise that approximate the “Holodeck” ® idea of interactive storytelling. We report exploratory experiments on the user ...
Lab testing beyond usability: challenges and recommendations for assessing user experiences

In the "third wave" of human-computer interaction (HCI), the advent of the conceptual approach of UX broadens and changes the HCI landscape. Methods approved before, mainly within the conceptual approach of usability, are still widely used, and yet ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

BELIV '16: Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization

October 2016

177 pages

ISBN:9781450348188

DOI:10.1145/2993901

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

INRIA: INRIA Rhône-Alpes

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

BELIV '16

BELIV '16: Novel Evaluation Methods For Visualization

October 24, 2016

MD, Baltimore, USA

Acceptance Rates

Overall Acceptance Rate 45 of 64 submissions, 70%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

85
Total Citations
View Citations
1,806
Total Downloads

Downloads (Last 12 months)260
Downloads (Last 6 weeks)20

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cabouat AHe TIsenberg PIsenberg T(2025)PREVis: Perceived Readability Evaluation for VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345631831:1(1083-1093)Online publication date: Jan-2025
https://doi.org/10.1109/TVCG.2024.3456318
Garreton MMoyano DGuerra JParra D(2024)Survey of data stories: Guidelines for data story authoringInformation Visualization10.1177/14738716241287116Online publication date: 23-Oct-2024
https://doi.org/10.1177/14738716241287116
Sinaei Hamed MKwan PKlich MAurisano JRajabiyazdi F(2024)The Elephant in the Room: Expert Experiences Designing, Developing and Evaluating Data Visualizations on Large DisplaysProceedings of the ACM on Human-Computer Interaction10.1145/36981398:ISS(301-329)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698139
Gründling JWeyers B(2024)Immersive Analytics: The Influence of Flow, Sense of Agency, and Presence on Performance and SatisfactionProceedings of the ACM on Human-Computer Interaction10.1145/36611448:EICS(1-27)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3661144
Schenkluhn MKnierim MKiss FWeinhardt C(2024)Connecting Home: Human-Centric Setup Automation in the Augmented Smart HomeProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642862(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642862
Branco DMóteiro MBouça-Machado RMiranda RReis TDecoroso ÉCardoso RRamalho JRato FMalheiro JMiranda DCaniça VPona-Ferreira FGuerreiro DLeitão MBraz AJ Ferreira JGuerreiro T(2024)Co-designing Customizable Clinical Dashboards with Multidisciplinary Teams: Bridging the Gap in Chronic Disease CareProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642618(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642618
Herman BJackson CKeefe D(2024)Touching the Ground: Evaluating the Effectiveness of Data Physicalizations for Spatial Data Analysis TasksIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345637731:1(875-885)Online publication date: 10-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456377
Chen NZhang YXu JRen KYang Y(2024)VisEval: A Benchmark for Data Visualization in the Era of Large Language ModelsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345632031:1(1301-1311)Online publication date: 10-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456320
Ceneda DCollins CEl-Assady MMiksch STominski CArleo A(2024)A Heuristic Approach for Dual Expert/End-User Evaluation of Guidance in Visual AnalyticsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332715230:1(997-1007)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3327152
Fu YStasko J(2024)More Than Data Stories: Broadening the Role of Visualization in Contemporary JournalismIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328758530:8(5240-5259)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3287585
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents