research-article

Soylent: a word processor with a crowd inside

Authors:

Michael S. Bernstein,

Robert C. Miller,

Björn Hartmann,

Mark S. Ackerman,

David R. Karger,

Katrina PanovichAuthors Info & Claims

UIST '10: Proceedings of the 23nd annual ACM symposium on User interface software and technology

Pages 313 - 322

https://doi.org/10.1145/1866029.1866078

Published: 03 October 2010 Publication History

Abstract

This paper introduces architectural and interaction patterns for integrating crowdsourced human contributions directly into user interfaces. We focus on writing and editing, complex endeavors that span many levels of conceptual and pragmatic activity. Authoring tools offer help with pragmatics, but for higher-level help, writers commonly turn to other people. We thus present Soylent, a word processing interface that enables writers to call on Mechanical Turk workers to shorten, proofread, and otherwise edit parts of their documents on demand. To improve worker quality, we introduce the Find-Fix-Verify crowd programming pattern, which splits tasks into a series of generation and review stages. Evaluation studies demonstrate the feasibility of crowdsourced editing and investigate questions of reliability, cost, wait time, and work time for edits.

Supplementary Material

JPG File (379s-bernstein.jpg)

Download
6.68 KB

MP4 File (379s-bernstein.mp4)

Download
27.29 MB

References

[1]

}}Bernstein, M., Marcus, A., Karger, D. R., and Miller, R. C. Enhancing Directed Content Sharing on the Web. CHI '10, ACM Press (2010).

Digital Library

[2]

}}Bernstein, M., Tan, D., Smith, G., Czerwinski, M., et al. Collabio: A Game for Annotating People within Social Networks. UIST '09, ACM Press (2009), 177--180.

Digital Library

[3]

}}Bigham, J. P., Jayant, C., Ji, H., Little, G., et al. VizWiz: Nearly Real-time Answers to Visual Questions. UIST '10, ACM Press (2010).

Digital Library

[4]

}}Clarke, J. and Lapata, M. Models for sentence compression: a comparison across domains, training requirements and evaluation measures. ACL '06, Association for Computational Linguistics (2006).

Digital Library

[5]

}}Cohn, T. and Lapata, M. Sentence compression beyond word deletion. COLING '08, (2008).

Digital Library

[6]

}}Cypher, A. Watch What I Do. MIT Press, Cambridge, MA, 1993.

[7]

}}Dourish, P. and Bellotti, V. Awareness and coordination in shared workspaces. CSCW '92, ACM Press (1992).

Digital Library

[8]

}}Evans, B. and Chi, E. Towards a model of understanding social search. CSCW '08, ACM Press (2008).

Digital Library

[9]

}}Hartmann, B., MacDougall, D., Brandt, J., and Klemmer, S. What Would Other Programmers Do? Suggesting Solutions to Error Messages. CHI '10, ACM Press (2010).

Digital Library

[10]

}}Heer, J. and Bostock, M. Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design. CHI '10, ACM Press (2010).

Digital Library

[11]

}}Kittur, A., Chi, E. H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. CHI '08, ACM Press (2008).

Digital Library

[12]

}}Knight, K. and Marcu, D. Summarization beyond sentence extraction: a probabilistic approach to sentence compression. Artificial Intelligence 139, 1 (2002).

Digital Library

[13]

}}Krieger, M., Stark, E. M., and Klemmer, S. R. Coordinating tasks on the commons: designing for personal goals, expertise and serendipity. CHI '09, ACM Press (2009).

Digital Library

[14]

}}Kukich, K. Techniques for automatically correcting words in text. ACM Computing Surveys (CSUR) 24, 4 (1992).

Digital Library

[15]

}}Lieberman, H. Your Wish is My Command. Morgan Kaufmann, San Francisco, 2001.

[16]

}}Little, G., Chilton, L., Goldman, M., and Miller, R. C. TurKit: Human Computation Algorithms on Mechanical Turk. UIST '10, ACM Press (2010).

Digital Library

[17]

}}Little, G., Lau, T. A., Cypher, A., Lin, J., et al. Koala: Capture, Share, Automate, Personalize Business Processes on the Web. CHI '07, (2007).

Digital Library

[18]

}}Marcu, D. The Theory and Practice of Discourse Parsing and Summarization. MIT Press, 2000.

Digital Library

[19]

}}Mason, W. and Watts, D. Financial Incentives and the Performance of Crowds. ACM SIGKDD Workshop on Human Computation, ACM Press (2009).

Digital Library

[20]

}}Miller, R. and Myers, B. Interactive simultaneous editing of multiple text regions. USENIX '01, (2001).

Digital Library

[21]

}}Quinn, A. J. and Bederson, B. B. A Taxonomy of Distributed Human Computation.

[22]

}}Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., et al. Who Are the Crowdworkers? Shifting Demographics in Amazon Mechanical Turk. alt.chi '10, ACM Press.

Digital Library

[23]

}}Sala, M., Partridge, K., Jacobson, L., and Begole, J. An Exploration into Activity-Informed Physical Advertising Using PEST. Pervasive '07, Springer Berlin Heidelberg (2007).

Digital Library

[24]

}}Simon, I., Morris, D., and Basu, S. MySong: automatic accompaniment generation for vocal melodies. Proc. CHI '08, ACM Press (2008).

Digital Library

[25]

}}Snow, R., O'Connor, B., Jurafsky, D., and Ng, A. Y. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. ACL '08, (2008).

Digital Library

[26]

}}Sorokin, A. and Forsyth, D. Utility data annotation with Amazon Mechanical Turk. CVPR '08, (2008).

[27]

}}von Ahn, L. and Dabbish, L. Labeling images with a computer game. CHI '04, ACM Press (2004).

Digital Library

Cited By

Deng ZXiang Y(2025)A partitioning Monte Carlo approach for consensus tasks in crowdsourcingExpert Systems with Applications10.1016/j.eswa.2024.125559262(125559)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125559
Ustalov DPavlichenko NTseitlin B(2024)Learning from Crowds with Crowd-KitJournal of Open Source Software10.21105/joss.062279:96(6227)Online publication date: Apr-2024
https://doi.org/10.21105/joss.06227
Cheng ZXu JJin H(2024)TreeQuestion: Assessing Conceptual Learning Outcomes with LLM-Generated Multiple-Choice QuestionsProceedings of the ACM on Human-Computer Interaction10.1145/36869708:CSCW2(1-29)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686970
Show More Cited By

Index Terms

Soylent: a word processor with a crowd inside
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Understanding the Microtask Crowdsourcing Experience for Workers with Disabilities: A Comparative View
CSCW

Microtask crowdsourcing holds great potential as an employment opportunity with the flexibility and anonymity that individuals with disability may require. Though prior research has explored the accessibility of crowd work, the lived crowd work ...
Who's the boss?: requester transparency and motivation in a microtask marketplace
CHI EA '14: CHI '14 Extended Abstracts on Human Factors in Computing Systems

Workers on crowdsourcing platforms such as Amazon's Mechanical Turk often receive little to no information about who is requesting the task they are asked to perform. This can lead to psychological distance and reduced work quality as a result. In this ...
Who are the crowdworkers?: shifting demographics in mechanical turk
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing Systems

Amazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is increasingly popular with researchers and developers. Here we extend previous ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '10: Proceedings of the 23nd annual ACM symposium on User interface software and technology

October 2010

476 pages

ISBN:9781450302715

DOI:10.1145/1866029

General Chair:
Ken Perlin
New York University
,
Program Chairs:
Mary Czerwinski
Microsoft Research
,
Rob Miller
MIT CSAIL

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 October 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UIST '10

Sponsor:

UIST '10: The 23nd Annual ACM Symposium on User Interface Software and Technology

October 3 - 6, 2010

New York, New York, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

478
Total Citations
View Citations
5,624
Total Downloads

Downloads (Last 12 months)401
Downloads (Last 6 weeks)25

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Deng ZXiang Y(2025)A partitioning Monte Carlo approach for consensus tasks in crowdsourcingExpert Systems with Applications10.1016/j.eswa.2024.125559262(125559)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125559
Ustalov DPavlichenko NTseitlin B(2024)Learning from Crowds with Crowd-KitJournal of Open Source Software10.21105/joss.062279:96(6227)Online publication date: Apr-2024
https://doi.org/10.21105/joss.06227
Cheng ZXu JJin H(2024)TreeQuestion: Assessing Conceptual Learning Outcomes with LLM-Generated Multiple-Choice QuestionsProceedings of the ACM on Human-Computer Interaction10.1145/36869708:CSCW2(1-29)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686970
Xiao YChen YYamashita NChen YLiu ZGao G(2024)(Dis)placed Contributions: Uncovering Hidden Hurdles to Collaborative Writing Involving Non-Native Speakers, Native Speakers, and AI-Powered Editing ToolsProceedings of the ACM on Human-Computer Interaction10.1145/36869428:CSCW2(1-31)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686942
Rajaram S(2024)Enabling Safer Augmented Reality Experiences: Usable Privacy Interventions for AR Creators and End-UsersAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686708(1-8)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686708
Laban PVig JHearst MXiong CWu C(2024)Beyond the Chat: Executable and Verifiable Text-Editing with LLMsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676419(1-23)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676419
Long TGero KChilton L(2024)Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI WorkflowProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661587(782-803)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661587
Mukhopadhyay AVenkatagiri SLuther K(2024)OSINT Research Studios: A Flexible Crowdsourcing Framework to Scale Up Open Source Intelligence InvestigationsProceedings of the ACM on Human-Computer Interaction10.1145/36373828:CSCW1(1-38)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637382
Fulker ZRiedl C(2024)Cooperation in the Gig Economy: Insights from Upwork FreelancersProceedings of the ACM on Human-Computer Interaction10.1145/36373148:CSCW1(1-20)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637314
Salim SHoque MMueller K(2024)Belief Miner: A Methodology for Discovering Causal Beliefs and Causal Illusions from General PopulationsProceedings of the ACM on Human-Computer Interaction10.1145/36372988:CSCW1(1-37)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637298
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents