abstract

Evaluating HCI Research beyond Usability

Authors:

Christian Remy,

Jennifer Mankoff,

Adrian FridayAuthors Info & Claims

CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems

Paper No.: SIG13, Pages 1 - 4

https://doi.org/10.1145/3170427.3185371

Published: 20 April 2018 Publication History

Abstract

Evaluating research artefacts is an important step to showcase the validity of a chosen approach. The CHI community has developed and agreed upon a large variety of evaluation methods for HCI research; however, sometimes those methods are not applicable or not sufficient. This is especially the case when the contribution lies within the context of the application area, such as for research in sustainable HCI, HCI for development, or design fiction and futures studies. In this SIG, we invite the CHI community to share their insights from projects that encountered problems in evaluating research and aim to discuss solutions for this difficult topic. We invite researchers from all areas of HCI research who are interested to engage in a debate of issues in the process of validating research artefacts.

References

[1]

Louise Barkhuus and Jennifer A. Rode. 2007. From Mice to Men - 24 Years of Evaluation in CHI. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM.

Digital Library

[2]

Alan J. Dix, Janet E. Finlay, Gregory D. Abowd, Russell Beale, and Janet E. Finley. 1998. HumanComputer Interaction. Prentice Hall, London, New York.

Digital Library

[3]

Geoffrey Ellis and Alan Dix. 2006. An Explorative Analysis of User Evaluation Studies in Information Visualisation. Proceedings of the 2006 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization, ACM, 1--7.

Digital Library

[4]

Saul Greenberg and Bill Buxton. 2008. Usability Evaluation Considered Harmful (Some of the Time). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 111--120.

Digital Library

[5]

Gillian R. Hayes. 2011. The Relationship of Action Research to Human-computer Interaction. ACM Trans. Comput.-Hum. Interact. 18, 3: 15:1--15:20.

Digital Library

[6]

International Organization for Standardization. 2010. ISO 9241--210:2010 - Ergonomics of humansystem interaction -- Part 210: Human-centred design for interactive systems.

[7]

Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research Methods in HumanComputer Interaction, Second Edition. Morgan Kaufmann, Cambridge, MA.

[8]

Joseph Lindley, Paul Coulton, and Emmett L. Brown. 2016. Peer Review and Design Fiction: "Honestly, they're not just made up." Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, ACM, 583-- 595.

Digital Library

[9]

Yong Liu, Jorge Goncalves, Denzil Ferreira, Bei Xiao, Simo Hosio, and Vassilis Kostakos. 2014. CHI 1994--2013: Mapping Two Decades of Intellectual Progress Through Co-word Analysis. Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems, ACM, 3553--3562.

Digital Library

[10]

Anton Lundström and Daniel Pargman. 2017. Developing a Framework for Evaluating the Sustainability of Computing Projects. Proceedings of the 2017 Workshop on Computing Within Limits, ACM, 111--117.

Digital Library

[11]

Jakob Nielsen. 1994. Usability engineering. Morgan Kaufmann Publishers, San Francisco, Calif.

Digital Library

[12]

Christian Remy, Oliver Bates, Alan Dix, et al. 2018. Evaluation beyond Usability: Validating Sustainable HCI Research. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.

Digital Library

[13]

Christian Remy, Oliver Bates, Thomas Vanessa, and Elaine May Huang. 2017. The Limits of Evaluating Sustainability. Proceedings of the Third Workshop on Computing within Limits, ACM Press.

Digital Library

[14]

Yvonne Rogers, Jenny Preece, and Helen Sharp. 2011. Interaction design. Wiley, Hoboken, N.J.; Chichester.

[15]

Antti Salovaara, Antti Oulasvirta, and Giulio Jacucci. 2017. Evaluation of Prototypes and the Problem of Possible Futures. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, ACM, 2064--2077.

Digital Library

[16]

Michael Sedlmair, Petra Isenberg, Miriah Meyer, and Tobias Isenberg. 2018. BELIV2018.

[17]

Ben Shneiderman, Catherine Plaisant, Maxine Cohen, and Steven Jacobs. 2009. Designing the User Interface: Strategies for Effective HumanComputer Interaction. Pearson, Boston.

Digital Library

[18]

M. Six Silberman, Eli Blevis, Elaine Huang, et al. 2014. What Have We Learned?: A SIGCHI HCI & Sustainability Community Workshop. CHI '14 Extended Abstracts on Human Factors in Computing Systems, ACM, 143--146.

Digital Library

[19]

M. Six Silberman and Bill Tomlinson. 2010. Toward an ecological sensibility: tools for evaluating sustainable HCI. CHI '10 Extended Abstracts on Human Factors in Computing Systems, ACM, 3469--3474.

Digital Library

[20]

Kentaro Toyama. 2015. Preliminary thoughts on a taxonomy of value for sustainable computing. First Monday 20, 8.

Cited By

Rawn ELi JPaulos EChasins S(2023)Understanding Version Control as Material Interaction with QuickposeProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581394(1-18)Online publication date: 19-Apr-2023
Marrella AWang LIocchi LNardi D(2022)A Methodology to Design and Evaluate HRI Teaming Tasks in Robotic CompetitionsACM Transactions on Human-Robot Interaction10.1145/352841511:3(1-22)Online publication date: 2-Sep-2022
Dix A(2022)Statistics for HCIundefinedOnline publication date: 8-Mar-2022
Show More Cited By

Index Terms

Evaluating HCI Research beyond Usability
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

This Changes Sustainable HCI
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

More than a decade into Sustainable HCI (SHCI) research, the community is still struggling to converge on a shared understanding of sustainability and HCI's role in addressing it. We think this is largely a positive sign, reflective of maturity; yet, ...
Evaluation Beyond Usability: Validating Sustainable HCI Research
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

The evaluation of research artefacts is an important step to validate research contributions. Sub-disciplines of HCI often pursue primary goals other than usability, such as Sustainable HCI (SHCI), HCI for development, or health and wellbeing. For such ...
Refugees & HCI SIG: Situating HCI Within Humanitarian Research
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

Currently, the United Nations High Commissioner for refugees estimates that there are around 65.8 million forcibly displaced people worldwide [16]. As digital technologies have become more available, humanitarian researchers and organizations have begun ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems

April 2018

3155 pages

ISBN:9781450356213

DOI:10.1145/3170427

General Chairs:
Regan Mandryk
University of Saskatchewan, Canada
,
Mark Hancock
University of Waterloo, Canada
,
Program Chairs:
Mark Perry
Brunel University London, UK
,
Anna Cox
University College London, UK

Copyright © 2018 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 April 2018

Check for updates

Author Tags

Qualifiers

Abstract

Conference

CHI '18

Sponsor:

SIGCHI

CHI '18: CHI Conference on Human Factors in Computing Systems

April 21 - 26, 2018

Montreal QC, Canada

Acceptance Rates

CHI EA '18 Paper Acceptance Rate 1,208 of 3,955 submissions, 31%;

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
429
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rawn ELi JPaulos EChasins S(2023)Understanding Version Control as Material Interaction with QuickposeProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581394(1-18)Online publication date: 19-Apr-2023
Marrella AWang LIocchi LNardi D(2022)A Methodology to Design and Evaluate HRI Teaming Tasks in Robotic CompetitionsACM Transactions on Human-Robot Interaction10.1145/352841511:3(1-22)Online publication date: 2-Sep-2022
Dix A(2022)Statistics for HCIundefinedOnline publication date: 8-Mar-2022
Remy CMacDonald Vermeulen LFrich JBiskjaer MDalsgaard PWakkary RAndersen KOdom WDesjardins APetersen M(2020)Evaluating Creativity Support Tools in HCI ResearchProceedings of the 2020 ACM Designing Interactive Systems Conference10.1145/3357236.3395474(457-476)Online publication date: 3-Jul-2020
Wacharamanotham CEisenring LHaroz SEchtler FBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Transparency of CHI Research Artifacts: Results of a Self-Reported SurveyProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376448(1-14)Online publication date: 21-Apr-2020
Frich JMacDonald Vermeulen LRemy CBiskjaer MDalsgaard PBrewster SFitzpatrick GCox AKostakos V(2019)Mapping the Landscape of Creativity Support Tools in HCIProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300619(1-18)Online publication date: 2-May-2019

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents