Cited By
View all- Lease MYilmaz E(2013)Crowdsourcing for information retrieval: introduction to the special issueInformation Retrieval10.1007/s10791-013-9222-716:2(91-100)Online publication date: 26-Mar-2013
CrowdTerrier: automatic crowdsourced relevance assessments with terrier
Page 1005
Abstract
In this demo, we present CrowdTerrier, an infrastructure extension to the open source Terrier IR platform that enables the semi-automatic generation of relevance assessments for a variety of document ranking tasks using crowdsourcing. The aim of CrowdTerrier is to reduce the time and expertise required to effectively Crowdsource relevance assessments by abstracting away from the complexities of the crowdsourcing process. It achieves this by automating the assessment process as much as possible, via a close integration of the IR system that ranks the documents (Terrier) and the crowdsourcing marketplace that is used to assess those documents (Amazon's Mechanical Turk).
References
[1]
R. McCreadie, C. Macdonald and I. Ounis. Identifying Top News using Crowdsourcing. Information Retrieval, 2012.
[2]
I. Ounis, G. Amati, V. Plachouras, B. He, C. Macdonald and C. Lioma. Terrier: A high performance and scalable information retrieval platform. In Proc. OSIR Workshop 2006.
Index Terms
- CrowdTerrier: automatic crowdsourced relevance assessments with terrier
Recommendations
Using crowdsourcing for TREC relevance assessment
Crowdsourcing has recently gained a lot of attention as a tool for conducting different kinds of relevance evaluations. At a very high level, crowdsourcing describes outsourcing of tasks to a large group of people instead of assigning such tasks to an ...
Identifying top news using crowdsourcing
AbstractThe influential Text REtrieval Conference (TREC) retrieval conference has always relied upon specialist assessors or occasionally participating groups to create relevance judgements for the tracks that it runs. Recently however, crowdsourcing has ...
Exploiting Document Content for Efficient Aggregation of Crowdsourcing Votes
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge ManagementThe use of crowdsourcing for document relevance assessment has been found to be a viable alternative to corpus annotation by highly trained experts. The question of quality control is a recurring challenge that is often addressed by aggregating multiple ...
Comments
Information & Contributors
Information
Published In
August 2012
1236 pages
ISBN:9781450314725
DOI:10.1145/2348283
- General Chair:
- William Hersh,
- Program Chairs:
- Jamie Callan,
- Yoelle Maarek,
- Mark Sanderson
Copyright © 2012 Authors.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 August 2012
Check for updates
Author Tags
Qualifiers
- Demonstration
Conference
SIGIR '12
Sponsor:
SIGIR '12: The 35th International ACM SIGIR conference on research and development in Information Retrieval
August 12 - 16, 2012
Oregon, Portland, USA
Acceptance Rates
Overall Acceptance Rate 792 of 3,983 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 190Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)0
Reflects downloads up to 14 Sep 2024
Other Metrics
Citations
Cited By
View all- Lease MYilmaz E(2013)Crowdsourcing for information retrieval: introduction to the special issueInformation Retrieval10.1007/s10791-013-9222-716:2(91-100)Online publication date: 26-Mar-2013
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in