abstract

Cross-study Reliability of the Open Card Sorting Method

Authors:

Christos Katsanos,

Nikolaos Tselios,

Nikolaos Avouris,

Stavros Demetriadis,

Ioannis Stamelos,

Lefteris AngelisAuthors Info & Claims

CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

Paper No.: LBW2718, Pages 1 - 6

https://doi.org/10.1145/3290607.3312999

Published: 02 May 2019 Publication History

Abstract

Information architecture forms the foundation of users' navigation experience. Open card sorting is a widely-used method to create information architectures based on users' groupings of the content. However, little is known about the method's cross-study reliability: Does it produce consistent content groupings for similar profile participants involved in different card sort studies? This paper presents an empirical evaluation of the method's cross-study reliability. Six card sorts involving 140 participants were conducted: three open sorts for a travel website, and three for an eshop. Results showed that participants provided highly similar card sorting data for the same content. A rather high agreement of the produced navigation schemes was also found. These findings provide support for the cross-study reliability of the open card sorting method.

References

[1]

Miranda G. Capra. 2005. Factor analysis of card sort data: an alternative to hierarchical cluster analysis. In Proc. of HFES, 691--695.

[2]

Jianming Dong, Shirley Martin, and Paul Waldo. 2001. A user input and analysis tool for information architecture. In Ext. Abstracts of CHI 2001, ACM Press, 23--24.

Digital Library

[3]

James Kalbach. 2007. Designing Web navigation: Optimizing the user experience. O'Reilly Media.

Digital Library

[4]

Christos Katsanos, Nikolaos Tselios, and Nikolaos Avouris. 2008. Automated semantic elaboration of web site information architecture. Interacting with Computers 20, 6, 535--544.

Digital Library

[5]

Christos Katsanos, Nikolaos Tselios, and Nikolaos Avouris. 2008. AutoCardSorter: designing the information architecture of a web site using latent semantic analysis. In Proc. of CHI 2008, ACM Press, 875--878.

Digital Library

[6]

N. Mantel. 1967. The detection of disease clustering and a generalized regression approach. Cancer Research 27, 2, 209-- 220.

[7]

Peter Morville. 2005. Ambient findability: What we find changes who we become. O'Reilly Media.

Digital Library

[8]

Peter Morville and Louis Rosenfeld. 2006. Information Architecture for the World Wide Web. O'Reilly Media.

Digital Library

[9]

Ather Nawaz. 2012. A comparison of card-sorting analysis methods. In Proc. of APCHI 2012, ACM Press, 583--592.

[10]

Jacob Nielsen. 2004. Card Sorting: How many users to test. Alertbox: http://www.useit.com/alertbox/20040719.html.

[11]

Sione Paea and Ross Baird. 2018. Information Architecture (IA): Using multidimensional scaling (MDS) and K-Means clustering algorithm for analysis of card sorting data. Journal of Usability Studies 13, 3, 138--157.

Digital Library

[12]

Celeste Lyn Paul. 2014. Analyzing card-sorting data using graph visualization. Journal of Usability Studies 9, 3, 87--104.

Digital Library

[13]

Helen Petrie, Christopher Power, Paul Cairns, and Cagla Seneler. 2011. Using card sorts for understanding website information architectures: technological, methodological and cultural issues. In Proc. of INTERACT 2011, 309--322.

Digital Library

[14]

Peter Pirolli. 2007. Information Foraging theory: Adaptive interaction with information. Oxford University Press, USA.

Digital Library

[15]

Carol Righi, Janice James, Michael Beasley, Donald L. Day, Jean E. Fox, Jennifer Gieber, Chris Howe and Laconya Ruby. 2013. Card sort analysis best practices. Journal of Usability Studies 8, 3, 69--89.

Digital Library

[16]

Per Rovegard, Lefteris Angelis, and Claes Wohlin. 2008. An empirical study on views of importance of change impact analysis issues. IEEE Transactions on Software Engineering 34, 4, 516--530.

Digital Library

[17]

Donna Spencer. 2009. Card sorting: designing usable categories. Rosenfeld Media, Brooklyn, N.Y.

[18]

Thomas Tullis and Larry Wood. 2005. How many users are enough for a card-sorting study? Presented at UPA 2004

[19]

Nikolaos Tselios, Christos Katsanos, and Nikolaos Avouris. 2009. Investigating the effect of hyperlink information scent on users' interaction with a web site. In Proc. of INTERACT 2009, 138--142.

Digital Library

[20]

Ian H. Witten and Elbe Frank. 2005. Data mining: practical machine learning tools and techniques. Morgan Kaufmann., Burlington, MA, USA.

Digital Library

[21]

Jed Wood and Larry Wood. 2008. Card sorting: current practices and beyond. Journal of Usability Studies 4, 1, 1--6.

Digital Library

Cited By

Gurcan F(2024)What are developers talking about information security? A large-scale study using semantic analysis of Q&A postsPeerJ Computer Science10.7717/peerj-cs.195410(e1954)Online publication date: 26-Mar-2024
https://doi.org/10.7717/peerj-cs.1954
Girjoaba ACapiluppi A(2024)Refactoring Legacy Code Using Cleaning Up Cycles: An Experience Report2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00079(753-764)Online publication date: 6-Oct-2024
https://doi.org/10.1109/ICSME58944.2024.00079
Paea SSharma BBulivou GKatsanos C(2024)Emergency Remote Teaching in Higher Education Institutes: A Taxonomy of Challenges Faced by First-Year Mathematics Students in the Pacific RegionIEEE Access10.1109/ACCESS.2024.335109812(6339-6355)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3351098
Show More Cited By

Index Terms

Cross-study Reliability of the Open Card Sorting Method
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI
    2. HCI design and evaluation methods

Recommendations

Test-Retest Reliability of the Open Card Sorting Method
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

Open card sorting is a popular method for designing information architectures. However, there is a lack of empirical evidence on the validity and reliability of the method. This paper explores the test-retest reliability of open card sorting. To this end,...
Validity of the Open Card Sorting Method for Producing Website Information Structures
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

When it comes to creating website information structures, open card sorting is the main approach used. However, scientific research for the method's validity is lacking. This paper explores the validity of open card sorting for website structural ...
Effect of Sense of Direction on Open Card Sorts for Websites
CHI Greece 2021: CHI Greece 2021: 1st International Conference of the ACM Greek SIGCHI Chapter

Card sorting is one widely used method to design website information architectures based on an understanding of their users’ mental models. This paper studies the effect of participants’ sense of direction on card sorting results. To this end, a two-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

May 2019

3673 pages

ISBN:9781450359719

DOI:10.1145/3290607

General Chairs:
Stephen Brewster
University of Glasgow, Scotland, UK
,
Geraldine Fitzpatrick
TU Wien, Austria
,
Program Chairs:
Anna Cox
University College London, UK
,
Vassilis Kostakos
University of Melbourne, Australia

Copyright © 2019 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Check for updates

Author Tags

Qualifiers

Abstract

Conference

CHI '19

Sponsor:

SIGCHI

CHI '19: CHI Conference on Human Factors in Computing Systems

May 4 - 9, 2019

Glasgow, Scotland Uk

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
315
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)5

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gurcan F(2024)What are developers talking about information security? A large-scale study using semantic analysis of Q&A postsPeerJ Computer Science10.7717/peerj-cs.195410(e1954)Online publication date: 26-Mar-2024
https://doi.org/10.7717/peerj-cs.1954
Girjoaba ACapiluppi A(2024)Refactoring Legacy Code Using Cleaning Up Cycles: An Experience Report2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00079(753-764)Online publication date: 6-Oct-2024
https://doi.org/10.1109/ICSME58944.2024.00079
Paea SSharma BBulivou GKatsanos C(2024)Emergency Remote Teaching in Higher Education Institutes: A Taxonomy of Challenges Faced by First-Year Mathematics Students in the Pacific RegionIEEE Access10.1109/ACCESS.2024.335109812(6339-6355)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3351098
Kloppenborg KPrice Ball MJonas SWolf GGreshake Tzovaras B(2024)Co-designing a wiki-based community knowledge management system for personal scienceRoyal Society Open Science10.1098/rsos.24027511:7Online publication date: 10-Jul-2024
https://doi.org/10.1098/rsos.240275
Nelson T(2024)Municipal Flag Design Preferences of United States ResidentsJournal of Nonprofit & Public Sector Marketing10.1080/10495142.2024.2361078(1-20)Online publication date: 6-Jun-2024
https://doi.org/10.1080/10495142.2024.2361078
Li JLin LYang MGuo ZZhang L(2024)A study on the impact of icon structural features in human–computer interface on visual search efficiencyErgonomics10.1080/00140139.2024.236634367:12(1925-1937)Online publication date: 2-Jul-2024
https://doi.org/10.1080/00140139.2024.2366343
Pechlevanoudis CZilidis GKatsanos C(2023)How Many Participants Do You Need for an Open Card Sort? A Case Study of E-commerce WebsitesHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42293-5_7(80-89)Online publication date: 26-Aug-2023
https://doi.org/10.1007/978-3-031-42293-5_7
Katsanos CChristoforidis VDemertzi C(2023)Task-Based Open Card Sorting: Towards a New Method to Produce Usable Information ArchitecturesHuman Interface and the Management of Information10.1007/978-3-031-35132-7_5(68-80)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-35132-7_5
O'Hagan JKhamis MMcGill MWilliamson J(2022)Exploring Attitudes Towards Increasing User Awareness of Reality From Within Virtual RealityProceedings of the 2022 ACM International Conference on Interactive Media Experiences10.1145/3505284.3529971(151-160)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1145/3505284.3529971
Ntouvaleti MKatsanos C(2022)Validity of the Open Card Sorting Method for Producing Website Information StructuresExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3519734(1-7)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519734
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents