research-article

Open access

How Much Do Platform Workers Value Reviews? An Experimental Method

Authors:

Siddharth SuriAuthors Info & Claims

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Article No.: 71, Pages 1 - 11

https://doi.org/10.1145/3491102.3501900

Published: 28 April 2022 Publication History

All formats PDF

Abstract

Previous qualitative work has documented that platform workers place an immense importance on their reputation due to the use of algorithmic management by online labor platforms. We provide a general experimental method, which can be used across platforms and time, for numerically quantifying the intensity with which platform workers experience reputation system-based algorithmic management. Our method works via an experiment where workers choose between a monetary bonus or a positive review. We demonstrate this method by measuring the value that freelancers assigned to positive feedback on Upwork in June 2020. The median freelancer in our sample valued a single positive review at ∼ $49 USD. We also find that less experienced freelancers valued a positive review more highly than those with more experience. Qualitative data collected during the experiment indicates that many freelancers considered issues related to reputation system-based algorithmic management while choosing between the monetary reward and the positive review.

References

[1]

Hunt Allcott, Luca Braghieri, Sarah Eichmeyer, and Matthew Gentzkow. 2020. The welfare effects of social media. American Economic Review 110, 3 (2020), 629–76.

[2]

Christopher Avery, Paul Resnick, and Richard Zeckhauser. 1999. The market for evaluations. American economic review 89, 3 (1999), 564–584.

[3]

Gordon M Becker, Morris H DeGroot, and Jacob Marschak. 1964. Measuring utility by a single-response sequential method. Behavioral science 9, 3 (1964), 226–232.

[4]

Seth Benzell and Avinash Collis. 2020. How to Govern Facebook: A Structural Model for Taxing and Regulating Big Tech. Available at SSRN 3619535(2020).

[5]

Erik Brynjolfsson, Avinash Collis, and Felix Eggers. 2019. Using massive online choice experiments to measure changes in well-being. Proceedings of the National Academy of Sciences 116, 15(2019), 7250–7255.

[6]

Luis Cabral and Ali Hortacsu. 2010. The dynamics of seller reputation: Evidence from eBay. The Journal of Industrial Economics 58, 1 (2010), 54–78.

[7]

Juan Carlos Alvarez de la Vega, Marta E. Cecchinato, and John Rooksby. 2021. “Why lose control?” A Study of Freelancers’ Experiences with Gig Economy Platforms. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[8]

Richard T Carson, Theodore Groves, and John A List. 2014. Consequentiality: A theoretical and experimental exploration of a single binary choice. Journal of the Association of Environmental and Resource Economists 1, 1/2(2014), 171–207.

[9]

Jesse Chandler, Pam Mueller, and Gabriele Paolacci. 2014. Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods 46 (2014), 112–130. Issue 1.

[10]

Robert C Feenstra, Robert Inklaar, and Marcel P Timmer. 2015. The next generation of the Penn World Table. American economic review 105, 10 (2015), 3150–82.

[11]

Apostolos Filippas, John J. Horton, and Joseph M. Golden. 2020. Reputation Inflation. (April 2020). Under Review.

[12]

Terry N Flynn, Jordan J Louviere, Tim J Peters, and Joanna Coast. 2007. Best–worst scaling: what it can do for health care research and how to do it. Journal of health economics 26, 1 (2007), 171–189.

[13]

Andrey Fradkin, Elena Grewal, Dave Holtz, and Matthew Pearson. 2015. Bias and reciprocity in online reviews: Evidence from field experiments on airbnb. In Proceedings of the Sixteenth ACM Conference on Economics and Computation. 641–641.

Digital Library

[14]

Mary L Gray and Siddharth Suri. 2019. Ghost work: How to stop Silicon Valley from building a new global underclass. Eamon Dolan Books.

[15]

Kathleen Griesbach, Adam Reich, Luke Elliott-Negri, and Ruth Milkman. 2019. Algorithmic control in platform food delivery work. Socius 5(2019), 2378023119870041.

[16]

Chien-Ju Ho, Aleksandrs Slivkins, Siddharth Suri, and Jennifer Wortman Vaughan. 2015. Incentivizing high quality crowdwork. In Proceedings of the 24th International Conference on World Wide Web. 419–429.

Digital Library

[17]

Gerhard Hommel. 1988. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75, 2 (1988), 383–386.

[18]

John J Horton. 2021. The Ruble Collapse in an Online Marketplace: Some Lessons for Market Designers. Technical Report. National Bureau of Economic Research.

[19]

Mohammad Hossein Jarrahi, Will Sutherland, Sarah Nelson, and Steve Sawyer. 2020. Platformic Management, Boundary Resources for Gig Work, and Worker Autonomy. Computer Supported Cooperative Work (04 2020). https://doi.org/10.1007/s10606-019-09368-7

Digital Library

[20]

Eliscia Kinder, Mohammad Hossein Jarrahi, and Will Sutherland. 2019. Gig platforms, tensions, alliances and ecosystems: An actor-network perspective. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–26.

Digital Library

[21]

Min Kyung Lee, Daniel Kusbit, Evan Metsky, and Laura Dabbish. 2015. Working with Machines: The Impact of Algorithmic and Data-Driven Management on Human Workers. https://doi.org/10.1145/2702123.2702548

Digital Library

[22]

Lingfang Li. 2010. Reputation, trust, and rebates: How online auction markets can improve their feedback mechanisms. Journal of Economics & Management Strategy 19, 2 (2010), 303–331.

[23]

Lingfang Li, Steven Tadelis, and Xiaolan Zhou. 2020. Buying reputation as a signal of quality: Evidence from an online marketplace. The RAND Journal of Economics 51, 4 (2020), 965–988.

[24]

Lingfang Li and Erte Xiao. 2014. Money talks: Rebate mechanisms in reputation system design. Management Science 60, 8 (2014), 2054–2072.

Digital Library

[25]

David Martin, Benjamin V Hanrahan, Jacki O’Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 224–235.

Digital Library

[26]

Alexandre Mas and Amanda Pallais. 2017. Valuing alternative work arrangements. American Economic Review 107, 12 (2017), 3722–59.

[27]

Alexandre Mas and Amanda Pallais. 2019. Labor supply and the value of non-work time: Experimental estimates from the field. American Economic Review: Insights 1, 1 (2019), 111–26.

[28]

Mareike Möhlmann and Lior Zalmanson. 2017. Hands on the wheel: Navigating algorithmic management and Uber drivers’ autonomy.

[29]

Amanda Pallais. 2014. Inefficient hiring in entry-level labor markets. American Economic Review 104, 11 (2014), 3565–99.

[30]

Sungsik Park, Woochoel Shin, and Jinhong Xie. 2021. The fateful first consumer review. Marketing Science (2021).

[31]

Paul Resnick, Richard Zeckhauser, John Swanson, and Kate Lockwood. 2006. The value of reputation on eBay: A controlled experiment. Experimental economics 9, 2 (2006), 79–101.

[32]

Hilary C Robinson. 2017. Making a digital working class: Uber drivers in Boston, 2016-2017. Ph. D. Dissertation. Massachusetts Institute of Technology.

[33]

Alex Rosenblat and Luke Stark. 2016. Algorithmic Labor and Information Asymmetries: A Case Study of Uber’s Drivers. International Journal of Communication 10, 0 (2016). https://ijoc.org/index.php/ijoc/article/view/4892

[34]

Will Sutherland, Mohammad Hossein Jarrahi, Michael Dunn, and Sarah Beth Nelson. 2020. Work precarity and gig literacies in online freelancing. Work, Employment and Society 34, 3 (2020), 457–475.

[35]

Carlos Toxtli, Siddharth Suri, and Saiph Savage. 2021. Quantifying the Invisible Labor in Crowd Work. In Computer Supported Cooperative Work (CSCW 2021).

Digital Library

[36]

Prasad Vana and Anja Lambrecht. 2021. The effect of individual online reviews on purchase likelihood. Marketing Science (2021).

[37]

Alex Wood, Mark Graham, Vili Lehdonvirta, and Isis Hjorth. 2018. Good Gig, Bad Big: Autonomy and Algorithmic Control in the Global Gig Economy. Work, Employment and Society 33 (08 2018), 095001701878561. https://doi.org/10.1177/0950017018785616

[38]

Ming Yin, Siddharth Suri, and Mary L Gray. 2018. Running out of time: The impact and value of flexibility in on-demand crowdwork. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–11.

Digital Library

Cited By

Li ZLiang CPeng JYin M(2024)The Value, Benefits, and Concerns of Generative AI-Powered Assistance in WritingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642625(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642625
Ren JRaghupathi VRaghupathi W(2023)Exploring Influential Factors in Hiring Freelancers in Online Labor Platforms: An Empirical StudyEconomies10.3390/economies1103008011:3(80)Online publication date: 3-Mar-2023
https://doi.org/10.3390/economies11030080
Kim PCheon ESawyer S(2023)Online Freelancing on Digital Labor Platforms: A Scoping ReviewCompanion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3584931.3607011(259-266)Online publication date: 14-Oct-2023
https://dl.acm.org/doi/10.1145/3584931.3607011

Index Terms

How Much Do Platform Workers Value Reviews? An Experimental Method

Recommendations

Algorithmic Management Reimagined For Workers and By Workers: Centering Worker Well-Being in Gig Work
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Prior research has studied the detrimental impact of algorithmic management on gig workers and strategies that workers devise in response. However, little work has investigated alternative platform designs to promote worker well-being, particularly from ...
Platformic Management, Boundary Resources for Gig Work, and Worker Autonomy
Abstract
We advance the concept of platformic management, and the ways in which platforms help to structure project-based or “gig” work. We do so knowing that the popular press and a substantial number of the scholarly publications characterize the “rise ...
Platform-mediated Markets, Online Freelance Workers and Deconstructed Identities
CSCW

We advance the concept of deconstructed identity to explain how online workers' identities are being reshaped, diminished and controlled by digital labor platforms. We focus on online freelance workers and contribute to contemporary conceptualizations ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

10459 pages

ISBN:9781450391573

DOI:10.1145/3491102

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA
,
Steven Drucker
Microsoft Research, USA
,
Julie Williamson
University of Glasgow, UK
,
Koji Yatani
University of Tokyo, Japan

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
980
Total Downloads

Downloads (Last 12 months)285
Downloads (Last 6 weeks)34

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li ZLiang CPeng JYin M(2024)The Value, Benefits, and Concerns of Generative AI-Powered Assistance in WritingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642625(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642625
Ren JRaghupathi VRaghupathi W(2023)Exploring Influential Factors in Hiring Freelancers in Online Labor Platforms: An Empirical StudyEconomies10.3390/economies1103008011:3(80)Online publication date: 3-Mar-2023
https://doi.org/10.3390/economies11030080
Kim PCheon ESawyer S(2023)Online Freelancing on Digital Labor Platforms: A Scoping ReviewCompanion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3584931.3607011(259-266)Online publication date: 14-Oct-2023
https://dl.acm.org/doi/10.1145/3584931.3607011
Marčeková RŠebová LMalachovský A(2023)Reviews on the Internet Marketing Communication of the SPA Tourism Enterprises in SlovakiaOnline Reputation Management in Destination and Hospitality10.1108/978-1-80382-375-120231010(175-194)Online publication date: 9-Feb-2023
https://doi.org/10.1108/978-1-80382-375-120231010

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents