research-article

Open access

Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision

Authors:

Milagros Miceli,

Martin Schuessler,

Tianling YangAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 4, Issue CSCW2

Article No.: 115, Pages 1 - 25

https://doi.org/10.1145/3415186

Published: 15 October 2020 Publication History

Abstract

The interpretation of data is fundamental to machine learning. This paper investigates practices of image data annotation as performed in industrial contexts. We define data annotation as a sense-making practice, where annotators assign meaning to data through the use of labels. Previous human-centered investigations have largely focused on annotators? subjectivity as a major cause of biased labels. We propose a wider view on this issue: guided by constructivist grounded theory, we conducted several weeks of fieldwork at two annotation companies. We analyzed which structures, power relations, and naturalized impositions shape the interpretation of data. Our results show that the work of annotators is profoundly informed by the interests, values, and priorities of other actors above their station. Arbitrary classifications are vertically imposed on annotators, and through them, on data. This imposition is largely naturalized. Assigning meaning to data is often presented as a technical matter. This paper shows it is, in fact, an exercise of power with multiple implications for individuals and society.

References

[1]

Muhammad Ali, Piotr Sapiezynski, Miranda Bogen, Aleksandra Korolova, Alan Mislove, and Aaron Rieke. 2019. Discrimination Through Optimization : How Facebook 's Ad Delivery Can Lead to Biased Outcomes. Proc. ACM Hum.-Comput. Interact., Vol. 3, CSCW (Nov. 2019), 199:1--199:30. https://doi.org/10.1145/3359301

Digital Library

[2]

Ali Alkhatib and Michael Bernstein. 2019. Street-Level Algorithms : A Theory at the Gaps Between Policy and Decisions. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, 530:1--530:13. https://doi.org/10.1145/3290605.3300760

Digital Library

[3]

Luis Araujo and Martin Spring. 2006. Services, Products, and the Institutional Structure of Production. Industrial Marketing Management, Vol. 35, 7 (Oct. 2006), 797--805. https://doi.org/10.1016/j.indmarman.2006.05.013

[4]

Paul Baker and Amanda Potts. 2013. `Why Do White People Have Thin Lips?' Google and the Perpetuation of Stereotypes via Auto-Complete Search Forms. Critical Discourse Studies, Vol. 10, 2 (May 2013), 187--204. https://doi.org/10.1080/17405904.2012.744320

[5]

Chelsea Barabas, Colin Doyle, JB Rubinovitz, and Karthik Dinakar. 2020. Studying up: Reorienting the Study of Algorithmic Fairness around Issues of Power. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT * '20). Association for Computing Machinery, Barcelona, Spain, 167--176. https://doi.org/10.1145/3351095.3372859

[6]

Solon Barocas and Andrew D. Selbst. 2016. Big Data 's Disparate Impact. California Law Review, Vol. 104, 3 (2016), 671--732. https://doi.org/10.15779/Z38BG31

[7]

Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing : Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics, Vol. 6 (2018), 587--604. https://doi.org/10.1162/tacl_a_00041

[8]

Jeanette Blomberg and Chuck Darrah. 2015. Toward an Anthropology of Services. The Design Journal, Vol. 18, 2 (2015), 171--192. https://doi.org/10.2752/175630615X14212498964196

[9]

Pierre Bourdieu. 1977. Outline of a Theory of Practice .Cambridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511812507

[10]

Pierre Bourdieu. 1985. The Social Space and the Genesis of Groups. Theory and Society, Vol. 14, 6 (1985), 723--744. https://doi.org/10.1007/BF00174048

[11]

Pierre Bourdieu. 1989. Social Space and Symbolic Power. Sociological Theory, Vol. 7, 1 (1989), 14--25. https://doi.org/10.2307/202060

[12]

Pierre Bourdieu. 1990. The logic of practice reprinted ed.). Polity Press, Cambridge.

[13]

Pierre Bourdieu. 1992. Language and Symbolic Power new ed.). Blackwell Publishers, Cambridge.

[14]

Pierre Bourdieu. 2000. Pascalian Meditations .Stanford University Press, Stanford, Calif.

[15]

Geoffrey C. Bowker. 2000. Biodiversity Datadiversity. Social Studies of Science, Vol. 30, 5 (Oct. 2000), 643--683. https://doi.org/10.1177/030631200030005001

[16]

Geoffrey C. Bowker and Susan Leigh Star. 1999. Sorting Things out: Classification and Its Consequences .MIT Press, Cambridge, Mass. BD175 .B68 1999

Digital Library

[17]

danah boyd and Kate Crawford. 2012. Critical Questions for Big Data : Provocations for a Cultural, Technological, and Scholarly Phenomenon. Information, Communication & Society, Vol. 15, 5 (June 2012), 662--679. https://doi.org/10.1080/1369118X.2012.678878

[18]

C. E. Brodley and M. A. Friedl. 1999. Identifying Mislabeled Training Data. Journal of Artificial Intelligence Research, Vol. 11 (Aug. 1999), 131--167. https://doi.org/10.1613/jair.606

[19]

Joy Buolamwini and Timnit Gebru. 2018. Gender Shades : Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, Vol. 81. PMLR, 77--91.

[20]

Ryan Burns. 2019. New Frontiers of Philanthro -capitalism: Digital Technologies and Humanitarianism. Antipode, Vol. 51, 4 (April 2019), 1101--1122. https://doi.org/10.1111/anti.12534

[21]

Kathy Charmaz. 2006. Constructing Grounded Theory : A Practical Guide through Qualitative Analysis .Sage Publications, London ; Thousand Oaks, Calif. H61.24 .C45 2006

[22]

Justin Cheng and Dan Cosley. 2013. How Annotation Styles Influence Content and Preferences. In Proceedings of the 24th ACM Conference on Hypertext and Social Media - HT '13. Association for Computing Machinery, Paris, France, 214--218. https://doi.org/10.1145/2481492.2481519

Digital Library

[23]

Angèle Christin. 2016. From Daguerreotypes to Algorithms : Machines, Expertise, and Three Forms of Objectivity. SIGCAS Computers and Society, Vol. 46, 1 (2016), 27--32. https://doi.org/10.1145/2908216.2908220

Digital Library

[24]

Danielle Keats Citron and Frank Pasquale. 2014. The Scored Society : Due Process for Automated Predictions. Washington Law Review, Vol. 89, 1 (2014).

[25]

Juliet M. Corbin and Anselm L. Strauss. 2015. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory fourth edition ed.). SAGE, Los Angeles. HA29 .C7724 2015

[26]

Nick Couldry and Ulises A. Mejias. 2019. Data Colonialism : Rethinking Big Data 's Relation to the Contemporary Subject. Television & New Media, Vol. 20, 4 (May 2019), 336--349. https://doi.org/10.1177/1527476418796632

[27]

Kate Crawford and Trevor Paglen. 2019. Excavating AI. https://www.excavating.ai.

[28]

Ciaran Cronin. 1996. Bourdieu and Foucault on Power and Modernity. Philosophy & Social Criticism, Vol. 22, 6 (Nov. 1996), 55--85. https://doi.org/10.1177/019145379602200603

[29]

Hannah Davis. 2020. A Dataset Is a Worldview. https://towardsdatascience.com/a-dataset-is-a-worldview-5328216dd44d.

[30]

Catherine D'Ignazio and Lauren F. Klein. 2020. Data Feminism .The MIT Press, Cambridge, Massachusetts. HQ1190 .D574 2020

[31]

Ravit Dotan and Smitha Milli. 2020. Value-Laden Disciplinary Shifts in Machine Learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT * '20). Association for Computing Machinery, Barcelona, Spain, 294. https://doi.org/10.1145/3351095.3373157

Digital Library

[32]

Emile Durkheim and Marcel Mauss. 1963. Primitive Classification. University of Chicago Press.

[33]

M. C. Elish and danah boyd. 2018. Situating Methods in the Magic of Big Data and AI. Communication Monographs, Vol. 85, 1 (Jan. 2018), 57--80. https://doi.org/10.1080/03637751.2017.1375130

[34]

Virginia Eubanks. 2018. Automating Inequality : How High -Tech Tools Profile, Police, and Punish the Poor .St. Martin's Press, New York.

[35]

Melanie Feinberg. 2017. A Design Perspective on Data. In CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). Association for Computing Machinery, Denver, Colorado, USA, 2952--2963. https://doi.org/10.1145/3025453.3025837

Digital Library

[36]

Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating Named Entities in Twitter Data with Crowdsourcing. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon 's Mechanical Turk (CSLDAMT '10). Association for Computational Linguistics, Los Angeles, California, 80--88. https://doi.org/10.5555/1866696.1866709

Digital Library

[37]

Marion Fourcade and Kieran Healy. 2013. Classification Situations: Life -Chances in the Neoliberal Era. Accounting, Organizations and Society, Vol. 38, 8 (Nov. 2013), 559--572. https://doi.org/10.1016/j.aos.2013.11.002

[38]

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumeé III, and Kate Crawford. 2018. Datasheets for Datasets. arXiv:1803.09010 (March 2018). arxiv: 1803.09010

[39]

R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, and Jenny Huang. 2020. Garbage in, Garbage out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT * '20). Association for Computing Machinery, Barcelona, Spain, 325--336. https://doi.org/10.1145/3351095.3372862

Digital Library

[40]

Bhavya Ghai, Q. Vera Liao, Yunfeng Zhang, and Klaus Mueller. 2020. Measuring Social Biases of Crowd Workers Using Counterfactual Queries. In Workshop on Fair & Responsible AI at ACM CHI Conference on Human Factors in Computing Systems. Honolulu, HI, USA.

[41]

Tarleton Gillespie and Tarleton Gillespie. 2014. The Relevance of Algorithms. In Media Technologies : Essays on Communication, Materiality, and Society, Pablo J. Boczkowski and Kirsten A. Foot (Eds.). The MIT Press, 167--194. https://doi.org/10.7551/mitpress/9780262525374.003.0009

[42]

Lisa Gitelman (Ed.). 2013. "Raw Data" Is an Oxymoron .The MIT Press, Cambridge, Massachusetts ; London, England. Q355 .R385 2013

[43]

Barney G. Glaser and Anselm L. Strauss. 1998. Grounded theory: Strategien qualitativer Forschung. Huber, Bern.

[44]

Mary L. Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt, Boston.

[45]

Foad Hamidi, Morgan Klaus Scheuerman, and Stacy M. Branham. 2018. Gender Recognition or Gender Reductionism - The Social Implications of Embedded Gender Recognition Systems. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). Association for Computing Machinery, New York, 1--13. https://doi.org/10.1145/3173574.3173582

[46]

Alex Hanna, Emily Denton, Andrew Smart, and Jamila Smith-Loud. 2020. Towards a Critical Race Methodology in Algorithmic Fairness. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT * '20). Association for Computing Machinery, Barcelona, Spain, 501--512. https://doi.org/10.1145/3351095.3372826

Digital Library

[47]

Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018. The Dataset Nutrition Label : A Framework To Drive Higher Data Quality Standards. arXiv:1805.03677 (2018).

[48]

Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300637

Digital Library

[49]

Gunay Kazimzade and Milagros Miceli. 2020. Biased Priorities, Biased Outcomes : Three Recommendations for Ethics -Oriented Data Annotation Practices. In Proceedings of the AAAI /ACM Conference on Artificial Intelligence, Ethics, and Society. (AIES '20). Association for Computing Machinery, New York, NY, USA, 1--7. https://doi.org/10.1145/3375627.3375809

Digital Library

[50]

Lucy Kimbell and Jeanette Blomberg. 2017. The Object of Service Design. In Designing for Service : Key Issues and New Directions. Bloomsbury Publishing, 81--94.

[51]

Rob Kitchin. 2017. Thinking Critically about and Researching Algorithms. Information, Communication & Society, Vol. 20, 1 (Jan. 2017), 14--29. https://doi.org/10.1080/1369118X.2016.1154087

[52]

Gary Klein, Jennifer K. Phillips, Erica L. Rall, and Deborah A. Peluso. 2007. A Data-Frame Theory of Sensemaking. In Expertise out of Context: Proceedings of the Sixth International Conference on Naturalistic Decision Making. Lawrence Erlbaum Associates Publishers, Mahwah, NJ, US, 113--155.

[53]

Ulrike Klinger and Jakob Svensson. 2018. The End of Media Logics? On Algorithms and Agency. New Media & Society, Vol. 20, 12 (Dec. 2018), 4653--4670. https://doi.org/10.1177/1461444818779750

[54]

Natalia M Libakova and Ekaterina A Sertakova. 2015. The Method of Expert Interview as an Effective Research Procedure of Studying the Indigenous Peoples of the North. Journal of Siberian Federal University. Humanities & Social Sciences, Vol. 8, 1 (2015), 114--129. https://doi.org/10.17516/1997--1370--2015--8--1--114--129

[55]

Michael A. Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna Wallach. 2020. Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, Honolulu, HI, USA, 1--14. https://doi.org/10.1145/3313831.3376445

Digital Library

[56]

Astrid Mager. 2012. Algorithmic Ideology : How Capitalist Society Shapes Search Engines. Information, Communication & Society, Vol. 15, 5 (June 2012), 769--787. https://doi.org/10.1080/1369118X.2012.676056

[57]

Steffen Mau. 2019. The Metric Society: On the Quantification of the Social .Polity, Cambridge ; Medford, MA. MR 2800 M447 M5

[58]

Frauke Mörike. 2019. Ethnography for Human Factors Researchers. Collecting and Interweaving Threads of HCI.

[59]

Michael Muller. 2014. Curiosity, Creativity, and Surprise as Analytic Tools : Grounded Theory Method. In Ways of Knowing in HCI, Judith S. Olson and Wendy A. Kellogg (Eds.). Springer, New York, NY, 25--48. https://doi.org/10.1007/978--1--4939-0378--8_2

[60]

Michael Muller, Shion Guha, Eric P.S. Baumer, David Mimno, and N. Sadat Shami. 2016. Machine Learning and Grounded Theory Method : Convergence, Divergence, and Combination. In Proceedings of the 19th International Conference on Supporting Group Work (GROUP '16). Association for Computing Machinery, Sanibel Island, Florida, USA, 3--8. https://doi.org/10.1145/2957276.2957280

[61]

Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q. Vera Liao, Casey Dugan, and Thomas Erickson. 2019. How Data Science Workers Work with Data : Discovery, Capture, Curation, Design, Creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, Glasgow, Scotland Uk, 1--15. https://doi.org/10.1145/3290605.3300356

Digital Library

[62]

Safiya Umoja Noble. 2018. Algorithms of Oppression : How Search Engines Reinforce Racism .NYU Press, New York.

[63]

Cathy O'Neil. 2017. Weapons of Math Destruction : How Big Data Increases Inequality and Threatens Democracy .PENGUIN BOOKS, London.

[64]

Juho P"a"akkönen, Matti Nelimarkka, Jesse Haapoja, and Airi Lampinen. 2020. Bureaucracy as a Lens for Analyzing and Designing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, Honolulu, HI, USA., 1--14. https://doi.org/10.1145/3313831.3376780

[65]

Samir Passi and Solon Barocas. 2019. Problem Formulation and Fairness. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT * '19). Association for Computing Machinery, Atlanta, GA, USA, 39--48. https://doi.org/10.1145/3287560.3287567

Digital Library

[66]

Samir Passi and Steven Jackson. 2017. Data Vision : Learning to See Through Algorithmic Abstraction. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). Association for Computing Machinery, Portland, Oregon, USA, 2436--2447. https://doi.org/10.1145/2998181.2998331

Digital Library

[67]

Samir Passi and Steven J. Jackson. 2018. Trust in Data Science : Collaboration, Translation, and Accountability in Corporate Data Science Projects. Proc. ACM Hum.-Comput. Interact., Vol. 2, CSCW (Nov. 2018), 1--28. https://doi.org/10.1145/3274405

[68]

Kathleen H. Pine and Max Liboiron. 2015. The Politics of Measurement and Action. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). Association for Computing Machinery, New York, NY, USA, 3147--3156. https://doi.org/10.1145/2702123.2702298

Digital Library

[69]

Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché-Buc, Emily Fox, and Hugo Larochelle. 2020. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program ). arXiv:2003.12206 (April 2020). arxiv: 2003.12206

[70]

Alex Rosenblat, Tamara Kneese, and Danah Boyd. 2014. Networked Employment Discrimination. SSRN Electronic Journal (2014). https://doi.org/10.2139/ssrn.2543507

[71]

Morgan Klaus Scheuerman, Jacob M. Paul, and Jed R. Brubaker. 2019. How Computers See Gender : An Evaluation of Gender Classification in Commercial Facial Analysis Services. Proc. ACM Hum.-Comput. Interact., Vol. 3, CSCW (Nov. 2019). https://doi.org/10.1145/3359246

[72]

Morgan Klaus Scheuerman, Kandrea Wade, Caitlin Lustig, and Jed R Brubaker. 2020. How We 've Taught Algorithms to See Identity : Constructing Race and Gender in Image Databases for Facial Analysis. Proc. ACM Hum.-Comput. Interact., Vol. 4, CSCW1 (2020). https://doi.org/10.1145/3392866

[73]

Nick Seaver. 2019. Knowing Algorithms. In digitalSTS : A Field Guide for Science & Technology Studies. Princeton University Press, PRINCETON; OXFORD, 412--422.

[74]

Ismaïla Seck, Khouloud Dahmane, Pierre Duthon, and Gaëlle Loosli. 2018. Baselines and a Datasheet for the Cerema AWP Dataset. In Conférence d'Apprentissage CAp (Conférence d'Apprentissage Francophone 2018). Rouen, France. https://doi.org/10.13140/RG.2.2.36360.93448

[75]

Susan Leigh Star and Anselm Strauss. 1999. Layers of Silence, Arenas of Voice : The Ecology of Visible and Invisible Work. Computer Supported Cooperative Work, Vol. 8, 1--2 (March 1999), 9--30. https://doi.org/10.1023/A:1008651105359

Digital Library

[76]

Robert Thornberg. 2012. Informed Grounded Theory. Scandinavian Journal of Educational Research, Vol. 56, 3 (June 2012), 243--259. https://doi.org/10.1080/00313831.2011.581686

[77]

Fabian L. Wauthier and Michael I. Jordan. 2011. Bayesian Bias Mitigation for Crowdsourcing. In Proceedings of the 24th International Conference on Neural Information Processing Systems (NIPS '11). Curran Associates Inc., Granada, Spain, 1800--1808.

[78]

Jennifer Wortman Vaughan and Hanna Wallach. 2020. A Human -Centered Agenda for Intelligible Machine Learning. In Machines We Trust : Getting Along with Artificial Intelligence.

[79]

Eviatar Zerubavel. 1993. The Fine Line : Making Distinctions in Everyday Life. 2nd ed. ed.). University of Chicago Press.

[80]

Honglei Zhuang and Joel Young. 2015. Leveraging In -Batch Annotation Bias for Crowdsourced Active Learning. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining (WSDM '15). Association for Computing Machinery, Shanghai, China, 243--252. https://doi.org/10.1145/2684822.2685301

Digital Library

[81]

Shoshana Zuboff. 2019. The Age of Surveillance Capitalism: The Fight for the Future at the New Frontier of Power .Profile Books, London.

Cited By

Engdahl I(2024)Agreements ‘in the wild’: Standards and alignment in machine learning benchmark dataset constructionBig Data & Society10.1177/2053951724124245711:2Online publication date: 3-Apr-2024
https://doi.org/10.1177/20539517241242457
Posada J(2024)Deeply embedded wages: Navigating digital payments in data workBig Data & Society10.1177/2053951724124244611:2Online publication date: 30-Apr-2024
https://doi.org/10.1177/20539517241242446
Valentine MBohn RPratt AJain PSinger SBernstein M(2024)Constructing a Classification Scheme - and its Consequences: A Field Study of Learning to Label Data for Computer Vision in a Hospital Intensive Care UnitProceedings of the ACM on Human-Computer Interaction10.1145/36870298:CSCW2(1-29)Online publication date: 7-Nov-2024
https://dl.acm.org/doi/10.1145/3687029
Show More Cited By

Index Terms

Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision

Recommendations

Who is “I”?: Subjectivity and Ethnography in HCI
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

HCI research applies ethnographic methods to understand and represent practices that involve the use of interactive systems. A subdomain of this work is interpretivist ethnography, which positions the researcher's perspectival view [37] as central to ...
How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

With the rise of big data, there has been an increasing need for practitioners in this space and an increasing opportunity for researchers to understand their workflows and design new tools to improve it. Data science is often described as data-driven, ...
Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation
CSCW

Machine learning models are commonly used to detect toxicity in online conversations. These models are trained on datasets annotated by human raters. We explore how raters' self-described identities impact how they annotate toxicity in online comments. ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 4, Issue CSCW2

CSCW

October 2020

2310 pages

EISSN:2573-0142

DOI:10.1145/3430143

Editor:
Jeff Nichols
Apple Inc.

Issue’s Table of Contents

Copyright © 2020 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2020

Published in PACMHCI Volume 4, Issue CSCW2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Bundesministerium für Bildung und Forschung

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

75
Total Citations
View Citations
3,423
Total Downloads

Downloads (Last 12 months)1,074
Downloads (Last 6 weeks)154

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Engdahl I(2024)Agreements ‘in the wild’: Standards and alignment in machine learning benchmark dataset constructionBig Data & Society10.1177/2053951724124245711:2Online publication date: 3-Apr-2024
https://doi.org/10.1177/20539517241242457
Posada J(2024)Deeply embedded wages: Navigating digital payments in data workBig Data & Society10.1177/2053951724124244611:2Online publication date: 30-Apr-2024
https://doi.org/10.1177/20539517241242446
Valentine MBohn RPratt AJain PSinger SBernstein M(2024)Constructing a Classification Scheme - and its Consequences: A Field Study of Learning to Label Data for Computer Vision in a Hospital Intensive Care UnitProceedings of the ACM on Human-Computer Interaction10.1145/36870298:CSCW2(1-29)Online publication date: 7-Nov-2024
https://dl.acm.org/doi/10.1145/3687029
Lin HKarusala NOkolo CD'Ignazio CGajos K(2024)"Come to us first": Centering Community Organizations in Artificial Intelligence for Social Good PartnershipsProceedings of the ACM on Human-Computer Interaction10.1145/36870098:CSCW2(1-28)Online publication date: 7-Nov-2024
https://dl.acm.org/doi/10.1145/3687009
Yang TMiceli M(2024)"Guilds" as Worker Empowerment and Control in a Chinese Data Work PlatformProceedings of the ACM on Human-Computer Interaction10.1145/36869048:CSCW2(1-27)Online publication date: 7-Nov-2024
https://dl.acm.org/doi/10.1145/3686904
DiSalvo CRothschild ASchenck LShapiro BDiSalvo B(2024)When Workers Want to Say No: A View into Critical Consciousness and Workplace Democracy in Data WorkProceedings of the ACM on Human-Computer Interaction10.1145/36374338:CSCW1(1-24)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637433
Chandhiramowuli STaylor AHeitlinger SWang D(2024)Making Data Work CountProceedings of the ACM on Human-Computer Interaction10.1145/36373678:CSCW1(1-26)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637367
Pedersen ABossen C(2024)Data Work Between the Local and the Global: An Ethnography of a Healthcare Business Intelligence UnitProceedings of the ACM on Human-Computer Interaction10.1145/36373178:CSCW1(1-28)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637317
Ullstein CEngelmann SPapakyriakopoulos OIkkatai YArnez-Jordan NCaleno RMboya BHiguma SHartwig TYokoyama HGrossklags J(2024)Attitudes Toward Facial Analysis AI: A Cross-National Study Comparing Argentina, Kenya, Japan, and the USAProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3659038(2273-2301)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3659038
Rifat MSafir ASaha SJunaed JSaleki MAmin MAhmed S(2024)Data, Annotation, and Meaning-Making: The Politics of Categorization in Annotating a Dataset of Faith-based Communal ViolenceProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3659030(2148-2156)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3659030
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents