skip to main content
10.1145/1390334.1390417acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

Predicting information seeker satisfaction in community question answering

Published: 20 July 2008 Publication History

Abstract

Question answering communities such as Naver and Yahoo! Answers have emerged as popular, and often effective, means of information seeking on the web. By posting questions for other participants to answer, information seekers can obtain specific answers to their questions. Users of popular portals such as Yahoo! Answers already have submitted millions of questions and received hundreds of millions of answers from other participants. However, it may also take hours --and sometime days-- until a satisfactory answer is posted. In this paper we introduce the problem of predicting information seeker satisfaction in collaborative question answering communities, where we attempt to predict whether a question author will be satisfied with the answers submitted by the community participants. We present a general prediction model, and develop a variety of content, structure, and community-focused features for this task. Our experimental results, obtained from a largescale evaluation over thousands of real questions and user ratings, demonstrate the feasibility of modeling and predicting asker satisfaction. We complement our results with a thorough investigation of the interactions and information seeking patterns in question answering communities that correlate with information seeker satisfaction. Our models and predictions could be useful for a variety of applications such as user intent inference, answer ranking, interface design, and query suggestion and routing.

References

[1]
E. Agichtein, E. Brill, S. Dumais, and R. Ragno. Learning user interaction models for predicting web search result preferences. In Proc. of SIGIR, 2006.
[2]
E. Agichtein, C. Castillo, D. Donato, A. Gionis, and G. Mishne. Finding high-quality content in social media with an application to community-based question answering. In Proceedings of WSDM, 2008.
[3]
N. Belkin, R. N. Oddy, and H. M. Brooks. Information retrieval: Part ii. results of a design study. Journal of Documentation, 38(3):145--164, 1982.
[4]
N. J. Belkin. User modeling in information retrieval. Tutorial presented at the Sixth International Conference on User Modelling (UM97).
[5]
E. Brill, S. Dumais, and M. Banko. An analysis of the askmsr question-answering system. In Proceedings of EMNLP, 2002.
[6]
E. Cutrell and Z. Guan. Eye tracking in MSN Search: Investigating snippet length, target position and task types, MSR-TR-2007.
[7]
H. T. Dang, D. Kelly, and J. Lin. Overview of the TREC 2007 question answering track. In Proc.of TREC, 2007.
[8]
D. Demner-Fushman and J. Lin. Answering clinical questions with knowledge-based and statistical techniques. Computational Linguistics, 33(1):63--103, 2007.
[9]
D. Downey, S. T. Dumais, and E. Horvitz. Models of searching and browsing: Languages, studies, and applications. In Proc. of IJCAI, 2007.
[10]
Y. Freund and R. Schapire. Experiments with a new boosting algorithm. In Proc. of the 13th international conference on machine learning (ICML1996), 1996.
[11]
S. P. Harter and C. A. Hert. Evaluation of information retrieval systems: Approaches, issues, and methods.
[12]
J. Jeon, W. Croft, and J. Lee. Finding similar questions in large question and answer archives. In Proceedings of CIKM, 2005.
[13]
J. Jeon, W. Croft, J. Lee, and S. Park. A framework to predict the quality of answers with non-textual features. In Proceedings of SIGIR, 2006.
[14]
T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Trans. Inf. Syst., 25(2), 2007.
[15]
M. Kobayashi and K. Takeda. Information retrieval on the web. ACM Computing Surveys, 32(2), 2000.
[16]
J. Lin and D. Demner-Fushman. Methods for automatically evaluating answers to complex questions. Information Retrieval, 9(5):565--587, 2006.
[17]
J. Lin and P. Zhang. Deconstructing nuggets: the stability and reliability of complex question answering evaluation. In Proceedings of SIGIR, pages 327--334, 2007.
[18]
J. C. Platt. Fast training of support vector machines using sequential minimal optimization. Advances in Kernal Methods -- Support Vector Learning, pages 185--208, 1998.
[19]
J. Quinlan. Improved use of continuous attributes in c4.5. In Journal of Artificial Intelligence Research, 1996.
[20]
D. E. Rose and D. Levinson. Understanding user goals in web search. In Proceedings of WWW, 2004.
[21]
I. Ruthven, L. A. Glasgow, M. Baillie, R. Bierig, E. Nicol, S. Sweeney, and M. Yakici. Intra-assessor consistency in question answering. In Proceedings of SIGIR, pages 727--728, 2007.
[22]
R. Soricut and E. Brill. Automatic question answering: Beyond the factoid. In HLT-NAACL, 2004.
[23]
Q. Su, D. Pavlov, J. Chow, and W. Baker. Internet-scale collection of human-reviewed data. In Proc. of the 16th international conference on World Wide Web (WWW), 2007.
[24]
E. M. Voorhees. The philosophy of information retrieval evaluation. In Proceedings of (CLEF), 2001.
[25]
E. M. Voorhees. Overview of the TREC 2003 question answering track. In Text REtrieval Conference, 2003.
[26]
R. White, M. Bilenko, and S. Cucerzan. Studying the use of popular destinations to enhance web search interaction. In Proc. of SIGIR, 2007.
[27]
R. W. White and S. M. Drucker. Investigating behavioral variability in web search. In Proc. of WWW, 2007.
[28]
I. Witten and E. Frank. Data Mining: Practical machine learning tools and techniques. Morgan Kaufman, 2nd edition, 2005.
[29]
J. Zobel. How reliable are the results of large-scale information retrieval experiments? In Proceedings of SIGIR, pages 307--314, 1998.

Cited By

View all

Index Terms

  1. Predicting information seeker satisfaction in community question answering

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
        July 2008
        934 pages
        ISBN:9781605581644
        DOI:10.1145/1390334
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 20 July 2008

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. community question answering
        2. information seeker satisfaction

        Qualifiers

        • Research-article

        Conference

        SIGIR '08
        Sponsor:

        Acceptance Rates

        Overall Acceptance Rate 792 of 3,983 submissions, 20%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)38
        • Downloads (Last 6 weeks)9
        Reflects downloads up to 06 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media