research-article

News vertical search: when and what to display to users

Authors:

Richard McCreadie,

Craig Macdonald,

Iadh OunisAuthors Info & Claims

SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Pages 253 - 262

https://doi.org/10.1145/2484028.2484080

Published: 28 July 2013 Publication History

Abstract

News reporting has seen a shift toward fast-paced online reporting in new sources such as social media. Web Search engines that support a news vertical have historically relied upon articles published by major newswire providers when serving news-related queries. In this paper, we investigate to what extent real-time content from newswire, blogs, Twitter and Wikipedia sources are useful to return to the user in the current fast-paced news search setting. In particular, we perform a detailed user study using the emerging medium of crowdsourcing to determine when and where integrating news-related content from these various sources can better serve the user's news need. We sampled approximately 300 news-related search queries using Google Trends and Bitly data in real-time for two time periods. For these queries, we have crowdsourced workers compare Web search rankings for each, with similar rankings integrating real-time news content from sources such as Twitter or the blogosphere. Our results show that users exhibited a preference for rankings integrating newswire articles for only half of our queries, indicating that relying solely on newswire providers for news-related content is now insufficient. Moreover, our results show that users preferred rankings that integrate tweets more often than those that integrate newswire articles, showing the potential of using social media to better serve news queries.

References

[1]

O. Alonso. Crowdsourcing for relevance evaluation. In Tutorial at ECIR'10.

[2]

O. Alonso and R. Baeza-Yates. Design and implementation of relevance assessments using crowdsourcing. Information Retrieval Journal 2011.

[3]

O. Alonso, D. E. Rose, and B. Stewart. Crowdsourcing for relevance evaluation. SIGIR Forum, 2008.\enlargethispage1\baselineskip

Digital Library

[4]

J. Arguello, F. Diaz, J. Callan, and J.-F. Crespo. Sources of evidence for vertical selection. In Proc. of SIGIR'09.

Digital Library

[5]

J. Arguello, W.-C. Wu, D. Kelly and A. Edwards. Task Complexity, Vertical Display and User Interaction in Aggregated Search. In Proc. of SIGIR'12.

Digital Library

[6]

J. Atwood. Is Amazon's Mechanical Turk a failure? http://www.codinghorror.com/blog/2007/04/is-amazons-mechanical-turk-a-f%ailure.html, accessed on 28/01/2013.

[7]

R. Bandari, S. Asur, and B. Huberman. The pulse of news in social media: Forecasting popularity. In Proc. of ICWSM'12.

[8]

J. Bar-Ilan, Z. Zhu, and M. Levene. Topic-specific analysis of search queries. In Proc. of WSDM'09 workshop on Web Search Click Data.

Digital Library

[9]

B. Carterette. Low-Cost and Robust Evaluation of Information Retrieval Systems. PhD thesis, University of Massachusetts Amherst, 2009.

Digital Library

[10]

C. Callison-Burch. Fast, cheap, and creative: Evaluating translation quality using Amazon's Mechanical Turk. In Proc. of EMNLP'09.

Digital Library

[11]

C. W. Cleverdon. The significance of the Cranfield tests on index languages. In Proc. of SIGIR'91.

Digital Library

[12]

F. Diaz. Integration of news content into Web results. In Proc. of WSDM'09.

Digital Library

[13]

J. Downs, M. Holbrook, S. Sheng, and L. Cranor. Are your participants gaming the system? Screening Mechanical Turk workers. In Proc. of CHI'10.

Digital Library

[14]

S. Hansell. Google keeps tweaking its search engine. http://www.nytimes.com/2007/06/03/business/yourmoney/03google.html, accessed on 28/01/2013.

[15]

J. Howe. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Three Rivers Pr, 2009.

Digital Library

[16]

J. Howe. The rise of Crowdsourcing. http://www.wired.com/wired/archive/14.06/crowds.html, accessed on 28/01/2013.

[17]

P. G. Ipeirotis. Demographics of Mechanical Turk. Tech Report, New York University, 2010.

[18]

P. G. Ipeirotis. Crowdsourcing using Mechanical Turk: Quality Management and Scalability. In Proc. of the WSDM'11 workshop on Crowdsourcing for Search and Data Mining.

[19]

A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with Mechanical Turk. In Proc. of CHI'08.

Digital Library

[20]

A. C. König, M. Gamon, and Q. Wu. Click-through prediction for news queries. In Proc. of SIGIR'09.

Digital Library

[21]

H. Kwak, C. Lee, H. Park, and S. Moon. What is Twitter, a Social Network or a News Media. In Proc. of WWW'10.

Digital Library

[22]

L. Levon, P. Kaulgud, and S. Skiena. Newspapers vs. blogs: Who gets the scoop. In Proc. of the AAAI Spring Symposium on Computational Approaches to Analyzing Weblogs, 2006.

[23]

R. McCreadie, C. Macdonald, and I. Ounis. Crowdsourcing Blog Track Top News Judgments at TREC. In Proc. of the WSDM'11 workshop on Crowdsourcing for Search and Data Mining.

[24]

R. McCreadie, C. Macdonald, and I. Ounis. Identifying Top News using Crowdsourcing. Information Retrieval Journal, 2013.

Digital Library

[25]

J. Norman. Google processes 1,000,000,000 search queries per day. http://www.historyofinformation.com/index.php?id=3276, accessed on 28/01/2013.

[26]

T. O'Brien. Twitter breaks news of plane crash in the Hudson. http://www.switched.com/2009/01/15/twitter\\ -breaks-news-of-plane-crash-in-the-hudson/, accessed on 28/01/2013.

[27]

T. O'Reilly and S. Milstein. The Twitter Book. O'Reilly Media, Inc., May 2009.\enlargethispage1\baselineskip

Digital Library

[28]

P. Thomas and D. Hawking. Evaluation by comparing result sets in context. In Proc. of CIKM'06.

Digital Library

[29]

R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast--but is it good?: Evaluating non-expert annotations for natural language tasks. In Proc. of EMNLP'08.

Digital Library

[30]

K. Sparck-Jones and C. J. van Rijsbergen. Report on the need for and provision of an "ideal" judgements retrieval test collection. Technical Report, British Library Research and Development, 1975.

[31]

K. Zhou, R.Cummins, M. Lalmas and J. M. Jose. Evaluating Aggregated Search Pages. In Proc. of SIGIR'12.

Digital Library

[32]

D. Zhu and B. Carterette. An Analysis of Assessor Behavior in Crowdsourced Preference Judgments. In Proc. of SIGIR'10.

Cited By

Steichen BLowe R(2021)How do multilingual users search? An investigation of query and result list language choicesJournal of the Association for Information Science and Technology10.1002/asi.2444372:6(759-776)Online publication date: 14-May-2021
https://dl.acm.org/doi/10.1002/asi.24443
Verma RKarimi SLee DGnawali OShakery A(2019)Newswire versus Social Media for Disaster Response and Recovery2019 Resilience Week (RWS)10.1109/RWS47064.2019.8971973(132-141)Online publication date: Nov-2019
https://doi.org/10.1109/RWS47064.2019.8971973
Papakonstantinou MDelis A(2019)PolyRecs: Improving Page–View Rates Using Real-Time Data AnalysisReal-Time Business Intelligence and Analytics10.1007/978-3-030-24124-7_7(95-112)Online publication date: 11-Oct-2019
https://doi.org/10.1007/978-3-030-24124-7_7
Show More Cited By

Index Terms

News vertical search: when and what to display to users
1. Information systems
  1. Information retrieval

Recommendations

Improving Ranking Consistency for Web Search by Leveraging a Knowledge Base and Search Logs
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

In this paper, we propose a new idea called ranking consistency in web search. Relevance ranking is one of the biggest problems in creating an effective web search system. Given some queries with similar search intents, conventional approaches typically ...
Identifying popular search goals behind search queries to improve web search ranking
AIRS'11: Proceedings of the 7th Asia conference on Information Retrieval Technology

Web users usually have a certain search goal before they submit a search query. However, many laypersons can't transform their search goals into suitable queries. Thus, understanding original search goals behind a query is very important for search ...
Ranking Approaches for Microblog Search
WI-IAT '10: Proceedings of the 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology - Volume 01

Ranking microblogs, such as tweets, as search results for a query is challenging, among other things because of the sheer amount of microblogs that are being generated in real time, as well as the short length of each individual microblog. In this paper,...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

July 2013

1188 pages

ISBN:9781450320344

DOI:10.1145/2484028

General Chairs:
Gareth J.F. Jones
Dublin City University, Ireland
,
Páraic Sheridan
Dublin City University, Ireland
,
Program Chairs:
Diane Kelly
University of North Carolina, Chapel Hill, USA
,
Maarten de Rijke
University of Amsterdam, The Netherlands
,
Tetsuya Sakai
Microsoft Research Asia, China

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 July 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '13

Sponsor:

SIGIR

SIGIR '13: The 36th International ACM SIGIR conference on research and development in Information Retrieval

July 28 - August 1, 2013

Dublin, Ireland

Acceptance Rates

SIGIR '13 Paper Acceptance Rate 73 of 366 submissions, 20%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
527
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Steichen BLowe R(2021)How do multilingual users search? An investigation of query and result list language choicesJournal of the Association for Information Science and Technology10.1002/asi.2444372:6(759-776)Online publication date: 14-May-2021
https://dl.acm.org/doi/10.1002/asi.24443
Verma RKarimi SLee DGnawali OShakery A(2019)Newswire versus Social Media for Disaster Response and Recovery2019 Resilience Week (RWS)10.1109/RWS47064.2019.8971973(132-141)Online publication date: Nov-2019
https://doi.org/10.1109/RWS47064.2019.8971973
Papakonstantinou MDelis A(2019)PolyRecs: Improving Page–View Rates Using Real-Time Data AnalysisReal-Time Business Intelligence and Analytics10.1007/978-3-030-24124-7_7(95-112)Online publication date: 11-Oct-2019
https://doi.org/10.1007/978-3-030-24124-7_7
Zhang XHan SLu W(2018)Automatic prediction of news intent for search queriesThe Electronic Library10.1108/EL-06-2017-013436:5(938-958)Online publication date: Oct-2018
https://doi.org/10.1108/EL-06-2017-0134
Demirsoz OOzcan R(2017)Classification of news-related tweetsJournal of Information Science10.5555/3212218.321222443:4(509-524)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.5555/3212218.3212224
Lowe RSteichen BBielikova MHerder ECena FDesmarais M(2017)Multilingual Search User Behaviors -- Exploring Multilingual Querying and Result Selection Through CrowdsourcingProceedings of the 25th Conference on User Modeling, Adaptation and Personalization10.1145/3079628.3079702(303-307)Online publication date: 9-Jul-2017
https://dl.acm.org/doi/10.1145/3079628.3079702
Setty VAnand AMishra AAnand Ade Rijke MShokouhi MTomkins AZhang M(2017)Modeling Event Importance for Ranking Daily News EventsProceedings of the Tenth ACM International Conference on Web Search and Data Mining10.1145/3018661.3018728(231-240)Online publication date: 2-Feb-2017
https://dl.acm.org/doi/10.1145/3018661.3018728
Lehmann JCastillo CLalmas MBaeza-Yates R(2017)Story-focused reading in online news and its potential for user engagementJournal of the Association for Information Science and Technology10.1002/asi.2370768:4(869-883)Online publication date: 1-Apr-2017
https://dl.acm.org/doi/10.1002/asi.23707
Ren PChen ZMa JZhang ZSi LWang S(2017)Detecting temporal patterns of user queriesJournal of the Association for Information Science and Technology10.1002/asi.2357868:1(113-128)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1002/asi.23578
Demirsoz OOzcan R(2016)Classification of news-related tweetsJournal of Information Science10.1177/016555151665308243:4(509-524)Online publication date: Jun-2016
https://doi.org/10.1177/0165551516653082
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents