research-article

Putting Question-Answering Systems into Practice: Transfer Learning for Efficient Domain Customization

Authors:

Bernhard Kratzwald,

Stefan FeuerriegelAuthors Info & Claims

ACM Transactions on Management Information Systems (TMIS), Volume 9, Issue 4

Article No.: 15, Pages 1 - 20

https://doi.org/10.1145/3309706

Published: 27 February 2019 Publication History

Abstract

Traditional information retrieval (such as that offered by web search engines) impedes users with information overload from extensive result pages and the need to manually locate the desired information therein. Conversely, question-answering systems change how humans interact with information systems: users can now ask specific questions and obtain a tailored answer—both conveniently in natural language. Despite obvious benefits, their use is often limited to an academic context, largely because of expensive domain customizations, which means that the performance in domain-specific applications often fails to meet expectations. This article proposes cost-efficient remedies: (i) we leverage metadata through a filtering mechanism, which increases the precision of document retrieval, and (ii) we develop a novel fuse-and-oversample approach for transfer learning to improve the performance of answer extraction. Here, knowledge is inductively transferred from related, yet different, tasks to the domain-specific application, while accounting for potential differences in the sample sizes across both tasks. The resulting performance is demonstrated with actual use cases from a finance company and the film industry, where fewer than 400 question-answer pairs had to be annotated to yield significant performance gains. As a direct implication to management, this presents a promising path to better leveraging of knowledge stored in information systems.

References

[1]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of theInternational Conference on Learning Representations.

[2]

Nicholas Belkin. 1993. Interaction with texts: Information retrieval as information-seeking behavior. Inform. Retriev. 93 (1993), 55--66.

[3]

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1533--1544.

[4]

Jiang Bian, Yandong Liu, Eugene Agichtein, and Hongyuan Zha. 2008. Finding the right facts in the crowd. In Proceedings of the International Conference on World Wide Web (WWW’08). 467--476.

Digital Library

[5]

Eric Brill, Susan Dumais, and Michele Banko. 2002. An analysis of the AskMSR question-answering system. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 257--264.

Digital Library

[6]

Chris Buckley and Mandar Mitra. 1997. SMART high precision: TREC 7. In Proceedings of the Text REtrieval Conference. 285--298.

[7]

Jinwei Cao and Jay F. Nunamaker. 2004. Question answering on lecture videos: A multifaceted approach. In Proceedings of the ACM/IEEE-CS Joint Conference on Digital Libraries. 214--215.

Digital Library

[8]

YongGang Cao, Feifan Liu, Pippa Simpson, Lamont Antieau, Andrew Bennett, James J. Cimino, John Ely, and Hong Yu. 2011. AskHERMES: An online question-answering system for complex clinical questions. J. Biomed. Inform. 44, 2 (2011), 277--288.

Digital Library

[9]

Michael Chau, Jialun Qin, Yilu Zhou, Chunju Tseng, and Hsinchun Chen. 2008. SpidersRUs: Creating specialized search engines in multiple languages. Dec. Supp. Syst. 45, 3 (2008), 621--640.

Digital Library

[10]

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. 1870--1879.

[11]

Abdessamad Echihabi, Ulf Hermjakob, Eduard Hovy, Daniel Marcu, Eric Melz, and Deepak Ravichandran. 2008. How to select an answer string? In Advances in Open Domain Question Answering, T. Strzalkowski and S. M. Harabagi (Eds.). Text, Speech and Language Technology, Vol 32. Springer, Dordrecht, 383--406.

[12]

Óscar Ferrández, Rubén Izquierdo, Sergio Ferrández, and José Luis Vicedo. 2009. Addressing ontology-based question answering with collections of user queries. Inform. Proc. Manag. 45, 2 (2009), 175--188.

Digital Library

[13]

D. A. Ferrucci. 2012. Introduction to “This is Watson.” IBM J. Res. Dev. 56, 3.4 (2012), 1--15.

Digital Library

[14]

Justin Scott Giboney, Susan A. Brown, Paul Benjamin Lowry, and Jay F. Nunamaker. 2015. User acceptance of knowledge-based system recommendations: Explanations, arguments, and fit. Dec. Supp. Syst. 72 (2015), 1--10.

Digital Library

[15]

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press, Cambridge, MA.

Digital Library

[16]

Nelson Granados, Alok Gupta, and Robert J. Kauffman. 2010. Research commentary-Information transparency in business-to-consumer markets: Concepts, framework, and research agenda. Inform. Syst. Res. 21, 2 (2010), 207--226.

Digital Library

[17]

Sanda Harabagiu, Dan Moldovan, Marius Pasca, Rada Mihalcea, Mihai Surdeanu, Razvan Bunescu, Roxana Girju, Vasile Rus, and Paul Morarescu. 2000. FALCON: Boosting knowledge for answer engines. In Proceedings of the 2000 Text REtrieval Conference. 479--488.

[18]

Nathalie Japkowicz and Shaju Stephen. 2002. The class imbalance problem: A systematic study. Intell. Data Anal. 6, 5 (2002), 429--449.

[19]

Daniel S. Jurafsky and James H. Martin. 2009. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition (2nd ed.). Pearson, Upper Saddle River, NJ.

Digital Library

[20]

Michael Kaisser and Tilman Becker. 2004. Question answering by searching large corpora with linguistic methods. In Proceedings of the Text REtrieval Conference.

[21]

Bernhard Kratzwald and Stefan Feuerriegel. 2018. Adaptive document retrieval for deep question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.

[22]

Bernhard Kratzwald, Suzana Ilić, Mathias Kraus, Stefan Feuerriegel, and Helmut Prendinger. 2018. Deep learning for affective computing: Text-based emotion recognition in decision support. Dec. Supp. Syst. 115 (2018), 24--35.

[23]

Mathias Kraus and Stefan Feuerriegel. 2017. Decision support from financial disclosures with deep neural networks and transfer learning. Dec. Supp. Syst. 104 (2017), 38--48.

Digital Library

[24]

M. Kraus, S. Feuerriegel, and A. Oztekin. 2018. Deep learning in business analytics and operations research: Models, applications, and managerial implications. arXiv1806.10897 (2018).

[25]

Ee-Peng Lim, Hsinchun Chen, and Guoqing Chen. 2013. Business intelligence and analytics. ACM Trans. Manag. Inform. Syst. 3, 4 (2013), 1--10.

Digital Library

[26]

Jimmy Lin. 2007. An exploration of the principles underlying redundancy-based factoid question answering. ACM Trans. Inform. Syst. 25, 2 (2007).

Digital Library

[27]

Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the Meeting of the Association for Computational Linguistics.

[28]

Xu-Ying Liu, Jianxin Wu, and Zhi-Hua Zhou. 2009. Exploratory undersampling for class-imbalance learning. IEEE Trans. Syst., Man, Cybern. 39, 2 (2009), 539--550.

Digital Library

[29]

Vanessa Lopez, Victoria Uren, Enrico Motta, and Michele Pasin. 2007. AquaLog: An ontology-driven question answering system for organizational semantic intranets. Web Sem.: Sci., Serv. Agents. World Wide Web 5, 2 (2007), 72--105.

Digital Library

[30]

A. Maedche and S. Staab. 2001. Ontology learning for the semantic web. IEEE Intell. Syst. 16, 2 (2001), 72--79.

Digital Library

[31]

Christopher D. Manning and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA.

Digital Library

[32]

Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1400--1409.

[33]

Dan Moldovan, Marius Paşca, Sanda Harabagiu, and Mihai Surdeanu. 2003. Performance issues and error analysis in an open-domain question answering system. ACM Trans. Inform. Syst. 21, 2 (2003), 133--154.

Digital Library

[34]

Diego Mollá and José Luis Vicedo. 2007. Question answering in restricted domains: An overview. Comput. Ling. 33, 1 (2007), 41--61.

Digital Library

[35]

Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2016. How transferable are neural networks in NLP applications? In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 479--489.

[36]

Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Trans. Know. Data Eng. 22, 10 (2010), 1345--1359.

Digital Library

[37]

Marius Pasca. 2005. Open-domain Question Answering from Large Text Collections. CSLI Studies in Computational Linguistics, Stanford, CA.

[38]

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1532--1543.

[39]

David Pinto, Michael Branstein, Ryan Coleman, W. Bruce Croft, Matthew King, Wei Li, and Xing Wei. 2002. QuASM: A system for question answering using semi-structured data. In Proceedings of the ACM/IEEE-CS Joint Conference on Digital Libraries. 46--55.

Digital Library

[40]

Dragomir Radev, Weiguo Fan, Hong Qi, Harris Wu, and Amardeep Grewal. 2005. Probabilistic question answering on the web. J. Assoc. Inform. Sci. Technol. 56, 6 (2005), 571--583.

Digital Library

[41]

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2383--2392.

[42]

Deepak Ravichandran and Eduard Hovy. 2002. Learning surface text patterns for a question-answering system. In Proceedings of the Meeting on Association for Computational Linguistics. 41--47.

Digital Library

[43]

Matthew Richardson. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 193--203.

[44]

Stephen Robertson. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inform. Retriev. 3, 4 (2009), 333--389.

Digital Library

[45]

Dmitri Roussinov and José A. Robles-Flores. 2007. Applying question-answering technology to locating malevolent online content. Dec. Supp. Syst. 43, 4 (2007), 1404--1418.

Digital Library

[46]

Gerard Salton. 1971. The SMART Retrieval System—Experiments in Automatic Document Processing. Prentice Hall, Upper Saddle River, NJ.

Digital Library

[47]

Jürgen Schmidhuber. 2015. Deep learning in neural networks: An overview. Neural Net. 61 (2015), 85--117.

Digital Library

[48]

Robert P. Schumaker and Hsinchun Chen. 2007. Leveraging question answer technology to address terrorism inquiry. Dec. Supp. Syst. 43, 4 (2007), 1419--1430.

Digital Library

[49]

Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proceedings of theInternational Conference on Learning Representations.

[50]

Dan Shen and Dietrich Klakow. 2006. Exploring correlation of dependency relation paths for answer extraction. In Proceedings of the Meeting of the Association for Computational Linguistics. 889--896.

Digital Library

[51]

Aditya Siddhant and Zachary C. Lipton. 2018. Deep Bayesian active learning for natural language processing: Results of a large-scale empirical study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2904--2909.

[52]

R. F. Simmons. 1965. Answering English questions by computer: A survey. Commun. ACM 8, 1 (1965), 53--70.

Digital Library

[53]

Amit Singhal, Chris Buckley, and Manclar Mitra. 1996. Pivoted document length normalization. In Proceedings of the ACM SIGIR Forum (1996), 21--29.

Digital Library

[54]

Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval: Document retrieval systems. J. Document. 28, 1 (1972), 11--21.

[55]

Christina Unger, Lorenz Bühmann, Jens Lehmann, Axel-Cyrille Ngonga Ngomo, Daniel Gerber, and Philipp Cimiano. 2012. Template-based question answering over RDF data. In Proceedings of the Conference on World Wide Web. 639.

Digital Library

[56]

David Vallet, Miriam Fernández, and Pablo Castells. 2005. An ontology-based information retrieval model. The Sem. Web: Res. App. 3532 (2005), 455--470.

Digital Library

[57]

Shahper Vodanovich, David Sundaram, and Michael Myers. 2010. Research commentary: Digital natives and ubiquitous information systems. Inform. Syst. Res. 21, 4 (2010), 711--723.

Digital Library

[58]

Ellen M. Voorhees. 2001. Overview of the TREC 9 question answering track. In Proceedings of the Text REtrieval Conference. 71--80.

[59]

Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, and Jing Jiang. 2018. R3: Reinforced ranker-reader for open-domain question answering. In Proceedings of the Conference on Artificial Intelligence.

[60]

Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2017a. Evidence aggregation for answer re-ranking in open-domain question answering. arXiv preprint arXiv:1711.05116 (2017).

[61]

Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017b. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the Meeting of the Association for Computational Linguistics. 189--198.

[62]

Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. 2009. Feature hashing for large scale multitask learning. In Proceedings of theInternational Conference on Machine Learning (ICML’09). 1--8.

Digital Library

[63]

Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016. Question answering on freebase via relation extraction and textual evidence. In Proceedings of the Meeting of the Association for Computational Linguistics. 2326--2336.

[64]

Weiwei Zong, Guang-Bin Huang, and Yiqiang Chen. 2013. Weighted extreme learning machine for imbalance learning. Neurocomputing 101 (2013), 229--242.

Digital Library

Cited By

Chib SKumar BAsha VSingh NLourens MBanerjee D(2024)Deep Learning Algorithms for Business Management: Ethical Considerations2024 International Conference on Communication, Computer Sciences and Engineering (IC3SE)10.1109/IC3SE62002.2024.10593494(1490-1495)Online publication date: 9-May-2024
https://doi.org/10.1109/IC3SE62002.2024.10593494
Chiesa MVerdi F(2023)Network Monitoring on Multi-Pipe SwitchesProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/35793217:1(1-31)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3579321
Guo KDiefenbach DGourru AGravier C(2023)Fine-tuning Strategies for Domain Specific Question Answering under Low Annotation Budget Constraints2023 IEEE 35th International Conference on Tools with Artificial Intelligence (ICTAI)10.1109/ICTAI59109.2023.00032(166-171)Online publication date: 6-Nov-2023
https://doi.org/10.1109/ICTAI59109.2023.00032
Show More Cited By

Index Terms

Putting Question-Answering Systems into Practice: Transfer Learning for Efficient Domain Customization
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Question answering
2. Social and professional topics
  1. Professional topics
    1. Computing and business

Recommendations

Quality-aware collaborative question answering: methods and evaluation
WSDM '09: Proceedings of the Second ACM International Conference on Web Search and Data Mining

Community Question Answering (QA) portals contain questions and answers contributed by hundreds of millions of users. These databases of questions and answers are of great value if they can be used directly to answer questions from any user. In this ...
Combining evidence with a probabilistic framework for answer ranking and answer merging in question answering

Question answering (QA) aims at finding exact answers to a user's question from a large collection of documents. Most QA systems combine information retrieval with extraction techniques to identify a set of likely candidates and then utilize some ...
Answer ranking based on named entity types for question answering
IMCOM '17: Proceedings of the 11th International Conference on Ubiquitous Information Management and Communication

Question answering (QA) using triples has been widely studied. One important aspect is answer ranking, that is, which answer candidates should be used to find correct answers. We are proposing a new method using type-matching information for ranking QA ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Management Information Systems

ACM Transactions on Management Information Systems Volume 9, Issue 4

December 2018

77 pages

ISSN:2158-656X

EISSN:2158-6578

DOI:10.1145/3316515

Editor:
Daniel Zeng
University of Arizona, USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 February 2019

Accepted: 01 January 2019

Revised: 01 November 2018

Received: 01 April 2018

Published in TMIS Volume 9, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Microsoft Azure for Research award

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
461
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)7

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chib SKumar BAsha VSingh NLourens MBanerjee D(2024)Deep Learning Algorithms for Business Management: Ethical Considerations2024 International Conference on Communication, Computer Sciences and Engineering (IC3SE)10.1109/IC3SE62002.2024.10593494(1490-1495)Online publication date: 9-May-2024
https://doi.org/10.1109/IC3SE62002.2024.10593494
Chiesa MVerdi F(2023)Network Monitoring on Multi-Pipe SwitchesProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/35793217:1(1-31)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3579321
Guo KDiefenbach DGourru AGravier C(2023)Fine-tuning Strategies for Domain Specific Question Answering under Low Annotation Budget Constraints2023 IEEE 35th International Conference on Tools with Artificial Intelligence (ICTAI)10.1109/ICTAI59109.2023.00032(166-171)Online publication date: 6-Nov-2023
https://doi.org/10.1109/ICTAI59109.2023.00032
Fang KXu K(2023)Automating Government Response to Citizens’ Questions: A Large Language Model-Based Question-Answering Guidance Generation System2023 3rd International Conference on Digital Society and Intelligent Systems (DSInS)10.1109/DSInS60115.2023.10455136(386-389)Online publication date: 10-Nov-2023
https://doi.org/10.1109/DSInS60115.2023.10455136
Escudero García DDeCastro-García NMuñoz Castañeda A(2023)An effectiveness analysis of transfer learning for the concept drift problem in malware detectionExpert Systems with Applications: An International Journal10.1016/j.eswa.2022.118724212:COnline publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1016/j.eswa.2022.118724
Schmitt AWambsganss TLeimeister J(2022)Conversational Agents for Information Retrieval in the Education DomainProceedings of the ACM on Human-Computer Interaction10.1145/35555876:CSCW2(1-22)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555587
Priya PMalik PMehbodniya AChaudhary VSharma ARay S(2022)The Relationship between Cloud Computing and Deep Learning towards Organizational Commitment2022 2nd International Conference on Innovative Practices in Technology and Management (ICIPTM)10.1109/ICIPTM54933.2022.9754046(21-26)Online publication date: 23-Feb-2022
https://doi.org/10.1109/ICIPTM54933.2022.9754046
Pisařovic IDařena FProcházka DJaniš V(2022)Preprocessing of normative documents for interactive question answeringExpert Systems with Applications: An International Journal10.1016/j.eswa.2021.116314191:COnline publication date: 6-May-2022
https://dl.acm.org/doi/10.1016/j.eswa.2021.116314
Liu ZSampaio PPishchulov GMehandjiev NCisneros-Cabrera SSchirrmann AJiru FBnouhanna N(2022)The architectural design and implementation of a digital platform for Industry 4.0 SME collaborationComputers in Industry10.1016/j.compind.2022.103623138:COnline publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1016/j.compind.2022.103623
Meng FYang T(2022)A Recognition Method of Basketball’s Shooting Trajectory Based On Transfer LearningMobile Networks and Applications10.1007/s11036-022-01949-z27:3(1271-1282)Online publication date: 21-Mar-2022
https://doi.org/10.1007/s11036-022-01949-z
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents