survey

A Review on Fact Extraction and Verification

Authors:

Giannis Bekoulis,

Christina Papagiannopoulou,

Nikos DeligiannisAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 55, Issue 1

Article No.: 12, Pages 1 - 35

https://doi.org/10.1145/3485127

Published: 23 November 2021 Publication History

Abstract

We study the fact-checking problem, which aims to identify the veracity of a given claim. Specifically, we focus on the task of Fact Extraction and VERification (FEVER) and its accompanied dataset. The task consists of the subtasks of retrieving the relevant documents (and sentences) from Wikipedia and validating whether the information in the documents supports or refutes a given claim. This task is essential and can be the building block of applications such as fake news detection and medical claim verification. In this article, we aim at a better understanding of the challenges of the task by presenting the literature in a structured and comprehensive way. We describe the proposed methods by analyzing the technical perspectives of the different approaches and discussing the performance results on the FEVER dataset, which is the most well-studied and formally structured dataset on the fact extraction and verification task. We also conduct the largest experimental study to date on identifying beneficial loss functions for the sentence retrieval component. Our analysis indicates that sampling negative sentences is important for improving the performance and decreasing the computational complexity. Finally, we describe open issues and future challenges, and we motivate future research in the task.

References

[1]

Lusine Abrahamyan, Yiming Chen, Giannis Bekoulis, and Nikos Deligiannis. 2021. Learned gradient compression for distributed deep learning. IEEE Trans. Neural Netw. Learn. Syst. (2021). https://doi.org/10.1109/TNNLS.2021.3084806

[2]

Naser Ahmadi, Joohyung Lee, Paolo Papotti, and Mohammed Saeed. 2019. Explainable fact checking with probabilistic answer set programming. In Proceedings of the Conference on Truth and Trust Online.

[3]

Tariq Alhindi, Savvas Petridis, and Smaranda Muresan. 2018. Where is your evidence: Improving fact-checking by justification modeling. In Proceedings of the 1st Workshop on Fact Extraction and VERification (FEVER).

[4]

Pepa Atanasova, Preslav Nakov, Georgi Karadzhov, Mitra Mohtarami, and Giovanni Da San Martino. 2019. Overview of the CLEF-2019 CheckThat! lab: Automatic identification and verification of claims. task 1: Check-worthiness. In Proceedings of CLEF (Working Notes). CEUR Workshop Proceedings.

[5]

Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020. Generating fact checking explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 7352–7364.

[6]

Pepa Atanasova, Dustin Wright, and Isabelle Augenstein. 2020. Generating label cohesive and well-formed adversarial claims. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.

[7]

Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 4685–4697.

[8]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference for Learning Representations.

[9]

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016).

[10]

Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Adversarial training for multi-context joint entity and relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2830–2836.

[11]

Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Joint entity recognition and relation extraction as a multi-head selection problem. Exp. Syst. Applic. 114 (2018), 34–45.

[12]

Alessandro Bondielli and Francesco Marcelloni. 2019. A survey on fake news and rumour detection techniques. Inf. Sci. 497 (2019), 38–55. DOI:DOI:

Digital Library

[13]

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 632–642.

[14]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei2020. Language models are few-shot learners. In Proceedings of the Conference on Advances in Neural Information Processing Systems, Vol. 33. 1877–1901.

[15]

Sylvie Cazalens, Julien Leblay, Philippe Lamarre, Ioana Manolescu, and Xavier Tannier. 2018. Computational fact checking: A content management perspective. Proc. VLDB Endow. 11, 12 (2018), 2110–2113.

Digital Library

[16]

Tuhin Chakrabarty, Tariq Alhindi, and Smaranda Muresan. 2018. Robust document retrieval and individual evidence modeling for fact extraction and verification. In Proceedings of the 1st Workshop on Fact Extraction and VERification (FEVER). Association for Computational Linguistics, 127–131.

[17]

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.

[18]

Jiangjie Chen, Qiaoben Bao, Jiaze Chen, Changzhi Sun, Hao Zhou, Yanghua Xiao, and Lei Li. 2020. LOREN: Logic enhanced neural reasoning for fact verification. arXiv preprint arXiv:2012.13577 (2020).

[19]

Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1657–1668.

[20]

Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. TabFact: A large-scale dataset for table-based fact verification. In Proceedings of the International Conference on Learning Representations.

[21]

Anton Chernyavskiy and Dmitry Ilvovsky. 2019. Extract and aggregate: A novel domain-independent approach to factual data verification. In Proceedings of the 2nd Workshop on Fact Extraction and VERification (FEVER).

[22]

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2924–2936.

[23]

Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 670–680.

[24]

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2475–2485.

[25]

Nadia K. Conroy, Victoria L. Rubin, and Yimin Chen. 2015. Automatic deception detection: Methods for finding fake news. Proc. Assoc. Inf. Sci. Technol. 52, 1 (2015), 1–4.

Digital Library

[26]

Leon Derczynski, Julie Binau, and Henri Schulte. 2020. Maintaining quality in FEVER annotation. In Proceedings of the 3rd Workshop on Fact Extraction and VERification (FEVER). 42–46.

[27]

Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval). 69–76.

[28]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

[29]

Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

[30]

Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER: A dataset for verification of real-world climate claims. In Proceedings of the Tackling Climate Change with Machine Learning Workshop at NeurIPS 2020.

[31]

Tien Huu Do, Xiao Luo, Duc Minh Nguyen, and Nikos Deligiannis. 2019. Rumour detection via news propagation dynamics and user representation learning. In Proceedings of the IEEE Data Science Workshop (DSW). IEEE, 196–200.

[32]

Joseph Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychol. Bull. 76, 5 (1971), 378–382.

[33]

Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of the Workshop for NLP Open Source Software (NLP-OSS).

[34]

Max Glockner, Ivan Habernal, and Iryna Gurevych. 2020. Why do you think that? Exploring faithful sentence-level rationales without supervision. In Findings of the Association for Computational Linguistics: EMNLP 2020. 1080–1095.

[35]

Peter Grabitz, Yuri Lazebnik, Josh Nicholson, and Sean Rife. 2017. Science with no fiction: Measuring the veracity of scientific reports by citation analysis. bioRxiv (2017). DOI:DOI:

[36]

Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. UKP-Athene: Multi-sentence textual entailment for claim verification. In Proceedings of the 1st Workshop on Fact Extraction and VERification (FEVER). Association for Computational Linguistics, 103–108.

[37]

Maram Hasanain, Reem Suwaileh, Tamer Elsayed, Alberto Barrón-Cedeno, and Preslav Nakov. 2019. Overview of the CLEF-2019 CheckThat! Lab: Automatic identification and verification of claims. Task 2: Evidence and factuality. In Proceedings of CLEF (Working Notes). CEUR Workshop Proceedings.

[38]

Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1923–1933.

[39]

Christopher Hidey, Tuhin Chakrabarty, Tariq Alhindi, Siddharth Varia, Kriste Krstovski, Mona Diab, and Smaranda Muresan. 2020. DeSePtion: Dual sequence prediction and adversarial examples for improved fact-checking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

[40]

Christopher Hidey and Mona Diab. 2018. Team SWEEPer: Joint sentence extraction and fact checking with pointer networks. In Proceedings of the 1st Workshop on Fact Extraction and VERification (FEVER).

[41]

Christopher Hidey and Kathy McKeown. 2019. Fixed that for you: Generating contrastive claims with semantic edits. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

[42]

Geoffrey E. Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural Comput. 14, 8 (2002), 1771–1800.

Digital Library

[43]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. 9, 8 (1997), 1735–1780.

Digital Library

[44]

Matthew Honnibal and Mark Johnson. 2015. An improved non-monotonic transition system for dependency parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.

[45]

Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A dataset for many-hop fact extraction and claim verification. In Findings of the Association for Computational Linguistics: EMNLP 2020.

[46]

Mayank Jobanputra. 2019. Unsupervised question answering for fact-checking. In Proceedings of the 2nd Workshop on Fact Extraction and VERification (FEVER).

[47]

Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2020. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

[48]

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). 6769–6781.

[49]

Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations.

[50]

Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).

[51]

Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Comput. Surv. 53, 1 (2020), 1–37.

Digital Library

[52]

Souvik Kundu and Hwee Tou Ng. 2018. A question-focused multi-factor attention network for question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.

Digital Library

[53]

Nayeon Lee, Belinda Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa. 2020. Language models as fact checkers? In Proceedings of the 3rd Workshop on Fact Extraction and VERification (FEVER).

[54]

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 7871–7880.

[55]

Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc.

[56]

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). IEEE, 2980–2988.

[57]

Hui Liu, Qingyu Yin, and William Yang Wang. 2019. Towards explainable NLP: A generative explanation framework for text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

[58]

Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, 3730–3740.

[59]

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv:1907.11692 (2019).

[60]

Zhenghao Liu, Chenyan Xiong, Zhuyun Dai, Si Sun, Maosong Sun, and Zhiyuan Liu. 2020. Adapting open domain fact extraction and verification to COVID-FACT through in-domain language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics.

[61]

Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

[62]

Jackson Luken, Nanjiang Jiang, and Marie-Catherine de Marneffe. 2018. QED: A fact verification system for the FEVER shared task. In Proceedings of the 1st Workshop on Fact Extraction and VERification (FEVER).

[63]

Jing Ma, Wei Gao, Shafiq Joty, and Kam-Fai Wong. 2019. Sentence-level evidence embedding for claim verification with hierarchical attention networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.

[64]

Christopher Malon. 2018. Team Papelo: Transformer networks at FEVER. In Proceedings of the 1st Workshop on Fact Extraction and VERification (FEVER). Association for Computational Linguistics, 109–113.

[65]

Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics.

[66]

Tanushree Mitra and Eric Gilbert. 2015. Credbank: A large-scale social media corpus with associated credibility annotations. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM).

[67]

Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

[68]

Moin Nadeem, Wei Fang, Brian Xu, Mitra Mohtarami, and James Glass. 2019. FAKTA: An automatic end-to-end fact checking system. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). Association for Computational Linguistics.

[69]

Duc Minh Nguyen, Tien Huu Do, Robert Calderbank, and Nikos Deligiannis. 2019. Fake news detection using deep Markov random fields. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

[70]

Yixin Nie, Lisa Bauer, and Mohit Bansal. 2020. Simple compounded-label training for fact extraction and verification. In Proceedings of the 3rd Workshop on Fact Extraction and VERification. Association for Computational Linguistics.

[71]

Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of the AAAI Conference on Artificial Intelligence.

Digital Library

[72]

Yixin Nie, Songhe Wang, and Mohit Bansal. 2019. Revealing the importance of semantic retrieval for machine reading at scale. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

[73]

Piotr Niewinski, Maria Pszona, and Maria Janicka. 2019. GEM: Generative enhanced model for adversarial attacks. In Proceedings of the 2nd Workshop on Fact Extraction and VERification (FEVER).

[74]

Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020. Zero-shot cross-lingual transfer with meta learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). 4547–4562.

[75]

Ray Oshikawa, Jing Qian, and William Yang Wang. 2020. A survey on natural language processing for fake news detection. In Proceedings of the 12th Language Resources and Evaluation Conference.

[76]

Wojciech Ostrowski, Arnav Arora, Pepa Atanasova, and Isabelle Augenstein. 2020. Multi-hop fact checking of political claims. arXiv preprint arXiv:2009.06401 (2020).

[77]

Wolfgang Otto. 2018. Team GESIS cologne: An all in all sentence-based approach for FEVER. In Proceedings of the 1st Workshop on Fact Extraction and VERification (FEVER). Association for Computational Linguistics.

[78]

Bhargavi Paranjape, Mandar Joshi, John Thickstun, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. An information bottleneck approach for controlling conciseness in rationale extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 1938–1952.

[79]

Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.

[80]

Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018. Automatic detection of fake news. In Proceedings of the 27th International Conference on Computational Linguistics.

[81]

Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).

[82]

Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktäschel, et al. 2020. KILT: A benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009.02252 (2020).

[83]

Przemysław Pobrotyn, Tomasz Bartczak, Mikołaj Synowiec, Radosław Białobrzeski, and Jarosław Bojar. 2020. Context-aware learning to rank with self-attention. In Proceedings of the SIGIR 2020 Workshop on eCommerce.

[84]

Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2016. Credibility assessment of textual claims on the web. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (CIKM’16). Association for Computing Machinery, New York, NY, 2173–2178.

Digital Library

[85]

Beatrice Portelli, Jason Zhao, Tal Schuster, Giuseppe Serra, and Enrico Santus. 2020. Distilling the evidence to augment fact verification models. In Proceedings of the 3rd Workshop on Fact Extraction and VERification (FEVER).

[86]

Adithya Pratapa, Sai Muralidhar Jayanthi, and Kavya Nerella. 2020. Constrained fact verification for FEVER. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).

[87]

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf.

[88]

Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2931–2937.

[89]

Estela Saquete, David Tomás, Paloma Moreda, Patricio Martínez-Barco, and Manuel Palomar. 2020. Fighting post-truth using natural language processing: A review and open challenges. Exp. Syst. Applic. 141 (2020), 112943.

[90]

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 815–823.

[91]

Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! Robust fact verification with contrastive evidence. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 624–643.

[92]

Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

[93]

Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv:1611.01603 (2016).

[94]

Darsh Shah, Tal Schuster, and Regina Barzilay. 2020. Automatic fact-guided sentence modification. Proc. AAAI Conf. Artif. Intell. 34, 05 (Apr. 2020), 8791–8798.

[95]

Gautam Kishore Shahi and Durgesh Nandini. 2020. FakeCovid-A multilingual cross-domain fact check news dataset for COVID-19. In Proceedings of the CySoc 2020 International Workshop on Cyber Social Threats. AAAI Press.

[96]

Karishma Sharma, Feng Qian, He Jiang, Natali Ruchansky, Ming Zhang, and Yan Liu. 2019. Combating fake news: A survey on identification and mitigation techniques. ACM Trans. Intell. Syst. Technol. 10, 3 (Apr 2019).

Digital Library

[97]

Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020. FakeNewsNet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big Data 8, 3 (2020), 171–188.

[98]

Amir Soleimani, Christof Monz, and Marcel Worring. 2020. BERT for evidence retrieval and claim verification. In Proceedings of the Conference on Advances in Information Retrieval. Springer International Publishing, 359–366.

Digital Library

[99]

Dominik Stammbach and Elliott Ash. 2020. e-FEVER: Explanations and summaries for automated fact checking. In Proceedings of the Conference on Truth and Trust Online.

[100]

Dominik Stammbach and Guenter Neumann. 2019. Team DOMLIN: Exploiting evidence enhancement for the FEVER shared task. In Proceedings of the 2nd Workshop on Fact Extraction and VERification (FEVER).

[101]

Shyam Subramanian and Kyumin Lee. 2020. Hierarchical evidence set modeling for automated fact extraction and verification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).

[102]

Sandeep Suntwal, Mithun Paul, Rebecca Sharp, and Mihai Surdeanu. 2019. On the importance of delexicalization for fact verification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

[103]

Eugenio Tacchini, Gabriele Ballarin, Marco L. Della Vedova, Stefano Moret, and Luca de Alfaro. 2017. Some like it Hoax: Automated fake news detection in social networks. In Proceedings of the 2nd Workshop on Data Science for Social Good. CEUR Workshop Proceedings, 1–15.

[104]

Motoki Taniguchi, Tomoki Taniguchi, Takumi Takahashi, Yasuhide Miura, and Tomoko Ohkuma. 2018. Integrating entity linking and evidence ranking for fact extraction and verification. In Proceedings of the 1st Workshop on Fact Extraction and VERification (FEVER). Association for Computational Linguistics, 124–126.

[105]

James Thorne, Max Glockner, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2021. Evidence-based verification for real world information needs. arXiv preprint arXiv:2104.00640 (2021).

[106]

James Thorne and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics.

[107]

James Thorne and Andreas Vlachos. 2019. Adversarial attacks against fact extraction and verification. arXiv:1903.05543.

[108]

James Thorne and Andreas Vlachos. 2020. Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation. arXiv preprint arXiv:2004.14366 (2020).

[109]

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A large-scale dataset for fact extraction and VERification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).

[110]

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Evaluating adversarial attacks against multiple fact verification systems. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

[111]

James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018. The fact extraction and VERification (FEVER) shared task. In Proceedings of the 1st Workshop on Fact Extraction and VERification (FEVER). Association for Computational Linguistics, 1–9.

[112]

James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019. The FEVER2.0 shared task. In Proceedings of the 2nd Workshop on Fact Extraction and VERification (FEVER).

[113]

Santosh Tokala, Vishal G., Avirup Saha, and Niloy Ganguly. 2019. AttentiveChecker: A bi-directional attention flow mechanism for fact verification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

[114]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Conference on Advances in Neural Information Processing Systems.

Digital Library

[115]

Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Proceedings of the Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc.

Digital Library

[116]

Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL Workshop on Language Technologies and Computational Social Science.

[117]

David Wadden, Kyle Lo, Lucy Lu Wang, Shanchuan Lin, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online, 17.

[118]

Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, and Sameer Singh. 2019. AllenNLP interpret: A framework for explaining predictions of NLP models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations. Association for Computational Linguistics.

[119]

Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. 2017. Deep metric learning with angular loss. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 2593–2601.

[120]

William Yang Wang. 2017. “Liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.

[121]

Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.

[122]

Chao-Yuan Wu, R. Manmatha, Alexander J. Smola, and Philipp Krahenbuhl. 2017. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 2840–2848.

[123]

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016).

[124]

Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). Association for Computing Machinery, New York, NY, 55–64.

Digital Library

[125]

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of the Conference on Advances in Neural Information Processing Systems.

Digital Library

[126]

Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential reasoning learning for language representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 7170–7186.

[127]

Wenpeng Yin and Dan Roth. 2018. TwoWingOS: A two-wing optimization strategy for evidential claim verification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.

[128]

Wenpeng Yin and Hinrich Schütze. 2018. Attentive convolution: Equipping CNNs with RNN-style attention mechanisms. Trans. Assoc. Comput. Ling. 6 (2018), 687–702.

[129]

Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. UCL machine reading group: Four factor framework for fact finding (HexaF). In Proceedings of the 1st Workshop on Fact Extraction and VERification (FEVER). Association for Computational Linguistics, 97–102.

[130]

Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, and Saurabh Tiwary. 2020. Transformer-XH: Multi-evidence reasoning with eXtra hop attention. In Proceedings of the International Conference on Learning Representations.

[131]

Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Reasoning over semantic-level graph for fact checking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 6170–6180.

[132]

Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and reasoning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 892–901.

[133]

Xinyi Zhou and Reza Zafarani. 2020. A survey of fake news: Fundamental theories, detection methods, and opportunities. ACM Comput. Surv. 53, 5 (Sept. 2020). DOI:DOI:https://doi.org/10.1145/3395046

[134]

Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. 2018. Detection and resolution of rumours in social media: A survey. ACM Comput. Surv. 51, 2 (2018), 1–36.

Digital Library

[135]

Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. 2016. Analysing how people orient to and spread rumours in social media by looking at conversational threads. PloS One 11, 3 (2016).

Cited By

Yang SYuan XGan TWu Y(2024)A Survey Of Automatic Fact Verification Research2024 7th World Conference on Computing and Communication Technologies (WCCCT)10.1109/WCCCT60665.2024.10541489(64-73)Online publication date: 12-Apr-2024
https://doi.org/10.1109/WCCCT60665.2024.10541489
Wu LYu DLiu PGao CWang Z(2024)Heuristic Heterogeneous Graph Reasoning Networks for Fact VerificationIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.328238035:10(14959-14973)Online publication date: Oct-2024
https://doi.org/10.1109/TNNLS.2023.3282380
Yang ZLin JGuo ZLi YLi XLi QLiu W(2024)Towards Rumor Detection With Multi-Granularity Evidences: A Dataset and BenchmarkIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.340170036:11(7188-7200)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TKDE.2024.3401700
Show More Cited By

Index Terms

A Review on Fact Extraction and Verification
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Information extraction

Recommendations

Evidence Distilling for Fact Extraction and Verification
Natural Language Processing and Chinese Computing
Abstract
There has been an increasing attention to the task of fact checking. Among others, FEVER is a recently popular fact verification task in which a system is supposed to extract information from given Wikipedia documents and verify the given claim. ...
Read it Twice: Towards Faithfully Interpretable Fact Verification by Revisiting Evidence
SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

Real-world fact verification task aims to verify the factuality of a claim by retrieving evidence from the source document. The quality of the retrieved evidence plays an important role in claim verification. Ideally, the retrieved evidence should be ...
WhatTheWikiFact: Fact-Checking Claims Against Wikipedia
CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

The rise of Internet has made it a major source of information. Unfortunately, not all information online is true, and thus a number of fact-checking initiatives have been launched, both manual and automatic, to deal with the problem. Here, we present ...

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 55, Issue 1

January 2023

860 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/3492451

Editor:
Albert Zomaya
University of Sydney, Australia

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 November 2021

Accepted: 01 August 2021

Revised: 01 July 2021

Received: 01 November 2020

Published in CSUR Volume 55, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Survey
Refereed

Funding Sources

MobiWave Innoviris Project

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
2,003
Total Downloads

Downloads (Last 12 months)514
Downloads (Last 6 weeks)35

Reflects downloads up to 30 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang SYuan XGan TWu Y(2024)A Survey Of Automatic Fact Verification Research2024 7th World Conference on Computing and Communication Technologies (WCCCT)10.1109/WCCCT60665.2024.10541489(64-73)Online publication date: 12-Apr-2024
https://doi.org/10.1109/WCCCT60665.2024.10541489
Wu LYu DLiu PGao CWang Z(2024)Heuristic Heterogeneous Graph Reasoning Networks for Fact VerificationIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.328238035:10(14959-14973)Online publication date: Oct-2024
https://doi.org/10.1109/TNNLS.2023.3282380
Yang ZLin JGuo ZLi YLi XLi QLiu W(2024)Towards Rumor Detection With Multi-Granularity Evidences: A Dataset and BenchmarkIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.340170036:11(7188-7200)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TKDE.2024.3401700
Stammbach DZhang BAsh E(2023)The Choice of Textual Knowledge Base in Automated Claim CheckingJournal of Data and Information Quality10.1145/356138915:1(1-22)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3561389
Aakur SSarkar S(2023)Leveraging Symbolic Knowledge Bases for Commonsense Natural Language Inference Using Pattern TheoryIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.328783745:11(13185-13202)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1109/TPAMI.2023.3287837
Lyu XHuang RYu X(2023)Ongraph Vector Computing for Solving Explicit Arithmetic Word Problems2023 International Conference on Intelligent Education and Intelligent Research (IEIR)10.1109/IEIR59294.2023.10391257(1-5)Online publication date: 5-Nov-2023
https://doi.org/10.1109/IEIR59294.2023.10391257
Spelda PStritecky VSymons J(2023)No-Regret Learning Supports Voters’ CompetenceSocial Epistemology10.1080/02691728.2023.225276338:5(543-559)Online publication date: 11-Sep-2023
https://doi.org/10.1080/02691728.2023.2252763
Correia PLopes Cardoso H(2023)Towards Explaining Shortcut Learning Through Attention Visualization and Adversarial AttacksEngineering Applications of Neural Networks10.1007/978-3-031-34204-2_45(558-569)Online publication date: 7-Jun-2023
https://doi.org/10.1007/978-3-031-34204-2_45
Naseer MWindiatmaja JAsvial MSari R(2022)RoBERTaEns: Deep Bidirectional Encoder Ensemble Model for Fact VerificationBig Data and Cognitive Computing10.3390/bdcc60200336:2(33)Online publication date: 22-Mar-2022
https://doi.org/10.3390/bdcc6020033
Zhang DVakili Tahami AAbualsaud MSmucker MAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Learning Trustworthy Web Sources to Derive Correct Answers and Reduce Health Misinformation in SearchProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531812(2099-2104)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531812
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents