research-article

Does Confidence Reporting from the Crowd Benefit Crowdsourcing Performance?

Authors:

Pramod K. VarshneyAuthors Info & Claims

SocialSens'17: Proceedings of the 2nd International Workshop on Social Sensing

Pages 49 - 54

https://doi.org/10.1145/3055601.3055607

Published: 18 April 2017 Publication History

Abstract

We explore the design of an effective crowdsourcing system for an M-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final classification decision. We consider the scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. Additionally, the workers report quantized confidence levels when they are able to submit definitive answers. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance. We obtain a couterintuitive result that the classification performance does not benefit from workers reporting quantized confidence. Therefore, the crowdsourcing system designer should employ the reject option without requiring confidence reporting.

References

[1]

Mohammad Allahbakhsh, Boualem Benatallah, Aleksandar Ignjatovic, Hamid Reza Motahari-Nezhad, Elisa Bertino, and Schahram Dustdar. 2013. Quality control in crowdsourcing systems: Issues and directions. IEEE Internet Comput. 2 (March 2013), 76--81.

Digital Library

[2]

Steven Burrows, Martin Potthast, and Benno Stein. 2013. Paraphrase acquisition via crowdsourcing and machine learning. ACM Trans. Intell. Syst. Technol. 4, 3 (July 2013), 43.

Digital Library

[3]

Zeineddin Chair and Pramod K. Varshney. 1986. Optimal data fusion in multiple sensor detection systems. IEEE Trans. Aerosp. Electron. Syst. AES-22, 1 (Jan. 1986), 98--101.

[4]

Wenda Chen, Mark Hasegawa-Johnson, and Nancy F. Chen. 2016. Mismatched Crowdsourcing based Language Perception for Under-resourced Languages. Procedia Computer Science 81 (2016), 23--29.

[5]

Ju Fan, Meihui Zhang, S. Kok, Meiyu Lu, and Beng Chin Ooi. 2015. CrowdOp: Query Optimization for Declarative Crowdsourcing Systems. IEEE Trans. Knowl. Data Eng. 27, 8 (Aug. 2015), 2078--2092. 2015.2407353

[6]

Mark Hasegawa-Johnson, Jennifer Cole, Preethi Jyothi, and Lav R. Varshney. 2015. Models of Dataset Size, Question Design, and Cross-Language Speech Perception for Speech Crowdsourcing Applications. Laboratory Phonology 6, 3-4 (Oct. 2015), 381--431.

[7]

Matthias Hirth, Tobias Hoßfeld, and Phuoc Tran-Gia. 2013. Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Math. Comput. Model. 57, 11 (July 2013), 2918--2932.

[8]

Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality Management on Amazon Mechanical Turk. In Proc. ACM SIGKDD Workshop Human Comput. (HCOMP'10). 64-67.

Digital Library

[9]

Preethi Jyothi and Mark Hasegawa-Johnson. 2015. Acquiring Speech Transcriptions Using Mismatched Crowdsourcing. In Proc. 29th AAAI Conf. Artificial Intelligence (AAAI'15).

[10]

David R. Karger, Sewoong Oh, and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. In Advances in Neural Information Processing Systems (NIPS) 24. MIT Press, Cambridge, MA, 1953--1961.

[11]

Gabriella Kazai. 2011. In search of quality in crowdsourcing for search engine evaluation. In European Conference on Information Retrieval. Springer, 165--176.

Digital Library

[12]

Xiang Kong, Preethi Jyothi, and Mark Hasegawa-Johnson. 2016. Performance Improvement of Probabilistic Transcriptions with Language-specific Constraints. Procedia Computer Science 81 (2016), 30--36. 2016.04.026

[13]

Q. Li, A. Vempaty, L. R. Varshney, and P. K. Varshney. 2017. Multi-Object Classification via Crowdsourcing With a Reject Option. IEEE Trans. Signal Process. 65, 4 (Feb 2017), 1068--1081.

Digital Library

[14]

Chunxi Liu, Preethi Jyothi, Hao Tang, Vimal Manohar, Rose Sloan, Tyler Kekona, Mark Hasegawa-Johnson, and Sanjeev Khudanpur. 2016. Adapting ASR for Under-Resourced Languages Using Mismatched Transcriptions.

[15]

George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review 63, 2 (1956),81.

[16]

Kaixiang Mo, Erheng Zhong, and Qiang Yang. 2013. Cross-task crowdsourcing. In Proc. ACM Int. Conf. Knowl Discovery Data Mining. 677-685.

Digital Library

[17]

Satoshi Oyama, Yukino Baba, Yuko Sakurai, and Hisashi Kashima. 2013. Accurate Integration of Crowdsourced Labels Using Workers' Self-reported Confidence Scores. In IJCAI.

[18]

Praveen Paritosh, Panos Ipeirotis, Matt Cooper, and Siddharth Suri. 2011. The computer is the new sewing machine: Benefits and perils of crowdsourcing. In Proc. 20th Int. Conf. World Wide Web (WWW'11). 325-326.

Digital Library

[19]

Alexander J. Quinn and Benjamin B. Bederson. 2011. Human computation: a survey and taxonomy of a growing field. In Proc. 2011 Annu. Conf. Hum. Factors Comput. Syst. (CHI 2011). 1403-1412. 1979148

Digital Library

[20]

Jana Rocker, Christopher M Yauch, Sumanth Yenduri, LA Perkins, and F Zand. 2007. Paper-based dichotomous key to computer based application for biological indentification. J. Comput. Sci. Coll. 22, 5 (May 2007), 30--38.

[21]

D. Sanchez-Charles, J. Nin, M. Sole, and V. Muntes-Mulero. 2014. Worker ranking determination in crowdsourcing platforms using aggregation functions. In Proc. IEEE Int. Conf. Fuzzy Syst. (FUZZ 2014). 1801-1808.

[22]

Nihar B Shah and Dengyong Zhou. 2014. Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing. ar Xiv preprint arXiv:1408.1387 (2014).

[23]

Lav R. Varshney, Preethi Jyothi, and Mark Hasegawa-Johnson. 2016. Language Coverage for Mismatched Crowdsourcing.

[24]

Lav R. Varshney, Aditya Vempaty, and Pramod K. Varshney. 2014. Assuring Privacy and Reliability in Crowdsourcing with Coding. In Proc. 2014 Inf. Theory Appl. Workshop.

[25]

Aditya Vempaty, Lav R. Varshney, and Pramod K. Varshney. 2014. Reliable Crowdsourcing for Multi-Class Labeling Using Coding Theory. IEEE J. Sel. Topics Signal Process. 8, 4 (Aug. 2014), 667--679.

[26]

Dejun Yue, Ge Yu, Derong Shen, and Xiaocong Yu. 2014. A weighted aggregation rule in crowdsourcing systems for high result accuracy. In Proc. IEEE 12th Int. Conf. Depend. Auton. Secure Comput. (DASC). 265-270.

Digital Library

[27]

Yu Zhang and Mihaela van der Schaar. 2012. Reputation-based incentive protocols in crowdsourcing applications. In Proc. 31st IEEE Conf. Computer Commun. (INFOCOM 2012). 2140-2148.

Cited By

Yasmin RHassan MGrassel JBhogaraju HEscobedo AFuentes O(2022)Improving Crowdsourcing-Based Image Classification Through Expanded Input Elicitation and Machine LearningFrontiers in Artificial Intelligence10.3389/frai.2022.8480565Online publication date: 29-Jun-2022
https://doi.org/10.3389/frai.2022.848056
Qu YRoitero KBarbera DSpina DMizzaro SDemartini G(2022)Combining Human and Machine Confidence in Truthfulness AssessmentJournal of Data and Information Quality10.1145/354691615:1(1-17)Online publication date: 28-Dec-2022
https://dl.acm.org/doi/10.1145/3546916
Geng BLi QVarshney P(2020)Prospect Theory Based Crowdsourcing for Classification in the Presence of SpammersIEEE Transactions on Signal Processing10.1109/TSP.2020.300675468(4083-4093)Online publication date: 2020
https://doi.org/10.1109/TSP.2020.3006754
Show More Cited By

Recommendations

Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Make Hay While the Crowd Shines: Towards Efficient Crowdsourcing on the Web
WWW '15 Companion: Proceedings of the 24th International Conference on World Wide Web

Within the scope of this PhD proposal, we set out to investigate two pivotal aspects that influence the effectiveness of crowdsourcing: (i) microtask design, and (ii) workers behavior. Leveraging the dynamics of tasks that are crowdsourced on the one ...
Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Crowdsourcing is increasingly being used as a means to tackle problems requiring human intelligence. With the ever-growing worker base that aims to complete microtasks on crowdsourcing platforms in exchange for financial gains, there is a need for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SocialSens'17: Proceedings of the 2nd International Workshop on Social Sensing

April 2017

97 pages

ISBN:9781450349772

DOI:10.1145/3055601

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

IEEE Signal Processing Society
SIGBED: ACM Special Interest Group on Embedded Systems
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CPS Week '17

Sponsor:

SIGBED

CPS Week '17: Cyber Physical Systems Week 2017

April 18 - 21, 2017

PA, Pittsburgh, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
88
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yasmin RHassan MGrassel JBhogaraju HEscobedo AFuentes O(2022)Improving Crowdsourcing-Based Image Classification Through Expanded Input Elicitation and Machine LearningFrontiers in Artificial Intelligence10.3389/frai.2022.8480565Online publication date: 29-Jun-2022
https://doi.org/10.3389/frai.2022.848056
Qu YRoitero KBarbera DSpina DMizzaro SDemartini G(2022)Combining Human and Machine Confidence in Truthfulness AssessmentJournal of Data and Information Quality10.1145/354691615:1(1-17)Online publication date: 28-Dec-2022
https://dl.acm.org/doi/10.1145/3546916
Geng BLi QVarshney P(2020)Prospect Theory Based Crowdsourcing for Classification in the Presence of SpammersIEEE Transactions on Signal Processing10.1109/TSP.2020.300675468(4083-4093)Online publication date: 2020
https://doi.org/10.1109/TSP.2020.3006754
Li QVarshney P(2018)Optimal Crowdsourced Classification with a Reject Option in the Presence of Spammers2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2018.8461615(4109-4113)Online publication date: Apr-2018
https://doi.org/10.1109/ICASSP.2018.8461615

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents