research-article

Galileo: Citizen-led Experimentation Using a Social Computing System

Authors:

Daniel McDonald,

Mad Price Ball,

Bastian Greshake Tzovaras,

Scott KlemmerAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 565, Pages 1 - 14

https://doi.org/10.1145/3411764.3445668

Published: 07 May 2021 Publication History

Abstract

People have scientific questions and folk theories; yet most lack the expertise to investigate them. How might people transform their questions into experiments that inform both science and their lives? This paper demonstrates how online volunteers can collaboratively design and run experiments using a novel social computing system. The Galileo system provides procedural support using three techniques: 1) experimental design workflow that provides just-in-time training; 2) review workflow with scaffolded questions; and 3) automated routines for data collection. We present two empirical investigations: a study and a field deployment with online volunteers across 16 and 8 countries respectively. People generated structurally-sound experiments on personally meaningful topics; three communities ran a week-long experiment each. We identify two key challenges for citizen-led experimentation—supporting different expertise levels and providing recruitment guidance—and provide specific suggestions from the social computing literature. Our results highlight the promise and challenges of citizen-led knowledge work like experimentation.

Supplementary Material

Supplementary Materials (3411764.3445668_supplementalmaterials.zip)

Download
312.14 KB

References

[1]

National Academies of Sciences, Engineering, and Medicine. 2018. Learning through citizen science: enhancing opportunities by design. National Academies Press.

[2]

Tim Althoff, Rok Sosič, Jennifer L. Hicks, Abby C. King, Scott L. Delp, and Jure Leskovec. 2017. Large-scale physical activity data reveal worldwide activity inequality. Nature 547, 7663: 336–339.

[3]

Michael S Bernstein, Joel Brandt, Robert C Miller, and David R Karger. 2011. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology, 33–42.

Digital Library

[4]

Rick Bonney, Caren B. Cooper, Janis Dickinson, Steve Kelling, Tina Phillips, Kenneth V. Rosenberg, and Jennifer Shirk. 2009. Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy. BioScience 59, 11: 977–984.

[5]

Carolin Cardamone, Kevin Schawinski, Marc Sarzi, Steven P Bamford, Nicola Bennert, C Megan Urry, Chris Lintott, William C Keel, John Parejko, Robert C Nichol, and others. 2009. Galaxy Zoo Green Peas: discovery of a class of compact extremely star-forming galaxies. Monthly Notices of the Royal Astronomical Society 399, 3: 1191–1205.

[6]

Dana Chandler and Adam Kapelner. 2013. Breaking monotony with meaning: Motivation in crowdsourcing markets. Journal of Economic Behavior and Organization 90: 123–133.

[7]

William G Chase and Herbert A Simon. 1973. Perception in chess. Cognitive psychology 4, 1: 55–81.

[8]

Quanze Chen, Jonathan Bragg, Lydia B Chilton, and Dan S Weld. 2019. Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 531.

Digital Library

[9]

Justin Cheng and Michael Bernstein. 2014. Catalyst: triggering collective action with thresholds. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, 1211–1221.

Digital Library

[10]

Stephanie N Chilton, Jeremy P Burton, and Gregor Reid. 2015. Inclusion of fermented foods in food guides around the world. Nutrients 7, 1: 390–404.

[11]

Caren B. Cooper, Jennifer Shirk, and Benjamin Zuckerberg. 2014. The invisible prevalence of citizen science in global research: Migratory birds and climate change. PLoS ONE 9, 9.

[12]

Seth Cooper, Firas Khatib, Adrien Treuille, and Et Al. 2010. Predicting protein structures with a multiplayer online game. Nature 466, 7307: 756–760.

[13]

Lorenzo Coviello, Yunkyu Sohn, Adam D.I. Kramer, Cameron Marlow, Massimo Franceschetti, Nicholas A. Christakis, and James H. Fowler. 2014. Detecting emotional contagion in massive social networks. PLoS ONE.

[14]

Nediyana Daskalova, Karthik Desingh, Alexandra Papoutsaki, Diane Schulze, Han Sha, and Jeff Huang. 2017. Lessons learned from two cohorts of personal informatics self-experiments. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 1, 3: 1–22.

Digital Library

[15]

Steven P. Dow, Anand Kulkarni, Scott R. Klemmer, and Bjorn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW ’12), 1013–1022.

Digital Library

[16]

Carol Dweck. 2016. What having a “growth mindset” actually means. Harvard Business Review 13: 213–226.

[17]

Irshaad O Ebrahim, Colin M Shapiro, Adrian J Williams, and Peter B Fenwick. 2013. Alcohol and sleep I: effects on normal sleep. Alcoholism: Clinical and Experimental Research 37, 4: 539–549.

[18]

Randall W Engle. 2002. Working memory capacity as executive attention. Current directions in psychological science 11, 1: 19–23.

[19]

E Ernst. 2003. Kombucha: a systematic review of the clinical evidence. Complementary Medicine Research 10, 2: 85–87.

[20]

f.lux. 2019. f.lux: sleep research. Retrieved from justgetflux.com/research.html

[21]

Björn Fjæstad. 2007. Why journalists report science as they do. Journalism, science and society: 123.

[22]

James Paul Gee. 2005. Semiotic social spaces and affinity spaces. Beyond communities of practice language power and social context 214232.

[23]

Fiona Godlee, Jane Smith, and Harvey Marcovitch. 2011. Wakefield's article linking MMR vaccine and autism was fraudulent. BMJ 342.

[24]

Jonathan Gray, Lucy Chambers, and Liliana Bounegru. 2012. The data journalism handbook: How journalists can use data to improve the news. O'Reilly Media, Inc.

[25]

John D Hansen and Justin Reich. 2015. Democratizing education? Examining access and usage patterns in massive open online courses. Science 350, 6265: 1245–1248.

[26]

National Institute of Health. 2018. Colloidal Silver | NCCIH. Retrieved from http://nccih.nih.gov/health/silver

[27]

Susanne Hecker, Muki Haklay, Anne Bowser, Zen Makuch, and Johannes Vogel. 2018. Citizen science: innovation in open science, society and policy. UCL Press.

[28]

Eric von Hippel. 2005. Democratizing innovation: The evolving phenomenon of user innovation. MIT.

[29]

Julie S Hui, Elizabeth M Gerber, and Darren Gergle. 2014. Understanding and leveraging social networks for crowdfunding: opportunities and challenges. In Proceedings of the 2014 conference on Designing interactive systems, 677–680.

Digital Library

[30]

Gerald C Kane. 2009. It's a Network, Not an Encyclopedia: A Social Network Perspective on Wikipedia Collaboration. In Academy of management proceedings, 1–6.

[31]

Ravi Karkar, Jessica Schroeder, Daniel A Epstein, Laura R Pina, Jeffrey Scofield, James Fogarty, Julie A Kientz, Sean A Munson, Roger Vilardaga, and Jasmine Zia. 2017. Tummytrials: a feasibility study of using self-experimentation to detect individualized food triggers. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 6850–6863.

Digital Library

[32]

Jennifer G Kim, Ha Kyung Kong, Karrie Karahalios, Wai-Tat Fu, and Hwajung Hong. 2016. The power of collective endorsements: credibility factors in medical crowdfunding campaigns. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 4538–4549.

Digital Library

[33]

Paul A Kirschner and Jeroen Van Merriënboer. 2008. Ten steps to complex learning a new approach to instruction and instructional design.

[34]

Rob Knight, J. Metcalf, and K. Amato. 2016. Gut Check: Exploring Your Microbiome. Coursera. Retrieved from https://www.coursera.org/learn/microbiome

[35]

Terry K Koo and Mae Y Li. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine 15, 2: 155–163.

[36]

Geza Kovacs, Zhengxuan Wu, and Michael S Bernstein. 2018. Rotating Online Behavior Change Interventions Increases Effectiveness But Also Increases Attrition. Proceedings of the ACM on Human-Computer Interaction 2, CSCW: 95.

Digital Library

[37]

Chinmay E Kulkarni, Michael S Bernstein, and Scott R Klemmer. 2015. PeerStudio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale, 75–84.

Digital Library

[38]

Walter Lasecki, Christopher Miller, Adam Sadilek, Andrew Abumoussa, Donato Borrello, Raja Kushalnagar, and Jeffrey Bigham. 2012. Real-time captioning by groups of non-experts. In Proceedings of the 25th annual ACM symposium on User interface software and technology, 23–34.

Digital Library

[39]

Samuel Lau, Tricia J Ngoon, Vineet Pandey, and Scott Klemmer. 2019. Experiment Reconstruction Reduces Fixation on Surface Details of Explanations. In Proceedings of the 2019 on Creativity and Cognition. 578–582.

Digital Library

[40]

Doris Lee, Joanne Lo, Moonhyok Kim, and Eric Paulos. 2016. Crowdclass: Designing classification-based citizen science learning modules. In Proceedings of the Fourth AAAI Conference on Human Computation and Crowdsourcing (HCOMP ’16).

[41]

Jayne Leonard. 2016. 15 Reasons You Need A Bottle Of Colloidal Silver In Your Home. Retrieved from naturallivingideas.com/colloidal-silver-benefits-and-uses/

[42]

Dana Lewis and Scott Leibrand. 2016. Real-World Use of Open Source Artificial Pancreas Systems. Journal of Diabetes Science and Technology 10, 6.

[43]

Kurt Luther, Scott Counts, Kristin B Stecher, Aaron Hoff, and Paul Johns. 2009. Pathfinder: an online collaboration environment for citizen scientists. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 239–248.

Digital Library

[44]

Wendy E Mackay, Caroline Appert, Michel Beaudouin-Lafon, Olivier Chapuis, Yangzhou Du, Jean-Daniel Fekete, and Yves Guiard. 2007. Touchstone: exploratory design of experiments. CHI ’07 Proceedings of the SIGCHI Conference on Human Factors in Computing System: 1425–1434.

Digital Library

[45]

D. W. Martin. 2007. Doing psychology experiments. Cengage Learning.

[46]

J Nathan Matias and Merry Mou. 2018. CivilServant: Community-led experiments in platform governance. In Proceedings of the 2018 CHI conference on human factors in computing systems, 9.

Digital Library

[47]

Daniel McDonald, Embriette Hyde, Justine W Debelius, James T Morton, Antonio Gonzalez, Gail Ackermann, Alexander A Aksenov, Bahar Behsaz, Caitriona Brennan, and Yingfeng Chen. 2018. American Gut: an Open Platform for Citizen Science Microbiome Research. mSystems 3, 3: e00031-18.

[48]

UK National Council for Voluntary Organisations. 2018. Why Volunteer? Retrieved from ncvo.org.uk/ncvo-volunteering/why-volunteer

[49]

NIH. 2015. NIH Clinical Trials Research and You. Retrieved from nih.gov/health-information/nih-clinical-research-trials-you/basics

[50]

Vineet Pandey, Amnon Amir, Justine Debelius, Embriette R Hyde, Rob Knight, and Scott Klemmer. 2017. Gut Instinct: Creating Scientific Theories with Online Learners. In 2017 CHI Conference on Human Factors in Computing Systems (pp. 6825-6836). ACM.

Digital Library

[51]

Vineet Pandey, Justine Debelius, Embriette R Hyde, Tomasz Kosciolek, Rob Knight, and Scott Klemmer. 2018. Docent: transforming personal intuitions to scientific hypotheses through content learning and process training. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale, 9.

Digital Library

[52]

Michael J Breus Ph.D. Alcohol and Sleep: What You Need to Know. Psychology Today. Retrieved from psychologytoday.com/us/blog/sleep-newzzz/201801/alcohol-and-sleep-what-you-need-know

[53]

Katharina Reinecke and Krzysztof Z Gajos. 2014. Quantifying visual preferences around the world. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 11–20.

Digital Library

[54]

Katharina Reinecke and Krzysztof Z Gajos. 2015. LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing.

Digital Library

[55]

Divit P Singh, Lee Lisle, T M Murali, and Kurt Luther. 2018. CrowdLayout. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18 (CHI ’18), 1–14.

[56]

Stefan Thomke. 2020. Building a culture of experimentation. Harvard Business Review 98, 2: 40–47.

[57]

Nai-Ching Wang, David Hicks, and Kurt Luther. 2018. Exploring Trade-Offs Between Learning and Productivity in Crowdsourced History. Proceedings of the ACM on Human-Computer Interaction 2, CSCW: 178.

Digital Library

[58]

Paul Wicks, Timothy E Vaughan, Michael P Massagli, and James Heywood. 2011. Accelerated clinical discovery using self-reported patient data collected online and a patient-matching algorithm. Nature Biotechnology 29, 5: 411–414.

[59]

Wikipedia. Community Insights/2018 Report. 2018. Retrieved from https://meta.wikimedia.org/wiki/Community_Insights/2018_Report#What_progress_has_been_made_in_the_diversity_of_Wikimedia_communities

[60]

Wikipedia. 2018. Bristol stool scale. Retrieved from en.wikipedia.org/wiki/Bristol_stool_scale

[61]

Wesley Willett, Jeffrey Heer, Joseph Hellerstein, and Maneesh Agrawala. 2011. CommentSpace: structured support for collaborative visual analysis. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 3131–3140.

Digital Library

[62]

Zooniverse. 2007. Galaxy Zoo. Retrieved December 31, 2016 from galaxyzoo.org

Cited By

Khadpe PLe LNowak KIqbal SSuh J(2024)DISCERN: Designing Decision Support Interfaces to Investigate the Complexities of Workplace Social Decision-Making With Line ManagersProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642685(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642685
Mutalova RSarrazin-Gendron RCai ERichard GGhasemloo Gheidari PCaisse SKnight RBlanchette MSzantner AWaldispühl J(2023)Playing the System: Can Puzzle Players Teach us How to Solve Hard Problems?Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581375(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581375
Daskalova NKyi EOuyang KBorem AChen SPark SNugent NHuang JKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Self-E: Smartphone-Supported Guidance for Customizable Self-ExperimentationProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445100(1-13)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445100

Index Terms

Galileo: Citizen-led Experimentation Using a Social Computing System
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Computer supported cooperative work
2. Social and professional topics
  1. Professional topics
    1. Computing and business
      1. Computer supported cooperative work

Index terms have been assigned to the content through auto-classification.

Recommendations

Volunteer computing: a model of the factors determining contribution to community-based scientific research
WWW '10: Proceedings of the 19th international conference on World wide web

Volunteer computing is a powerful way to harness distributed resources to perform large-scale tasks, similarly to other types of community-based initiatives. Volunteer computing is based on two pillars: the first is computational - allocating and ...
Ramble: A flexible, extensible, and composable experimentation framework
SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

Reproducibility and replicability are extremely important components of scientific computing, and any computational research. The ability to replicate a set of experiments aids many other computational use cases, such as systems acceptance where a ...
Designing online experiments: Citizen science approaches to research
CSCW '16 Companion: Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion

In this workshop we will host presentations from researchers performing online experiments and spend a half day helping attendees create experiments using the Volunteer Science platform. Current approaches to online science use new information ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
424
Total Downloads

Downloads (Last 12 months)65
Downloads (Last 6 weeks)13

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Khadpe PLe LNowak KIqbal SSuh J(2024)DISCERN: Designing Decision Support Interfaces to Investigate the Complexities of Workplace Social Decision-Making With Line ManagersProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642685(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642685
Mutalova RSarrazin-Gendron RCai ERichard GGhasemloo Gheidari PCaisse SKnight RBlanchette MSzantner AWaldispühl J(2023)Playing the System: Can Puzzle Players Teach us How to Solve Hard Problems?Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581375(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581375
Daskalova NKyi EOuyang KBorem AChen SPark SNugent NHuang JKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Self-E: Smartphone-Supported Guidance for Customizable Self-ExperimentationProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445100(1-13)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445100

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents