review-article

A survey on legal question–answering systems

Author:

Jorge Martinez-GilAuthors Info & Claims

Volume 48, Issue C

https://doi.org/10.1016/j.cosrev.2023.100552

Published: 01 May 2023 Publication History

Abstract

Many legal professionals think the explosion of information about local, regional, national, and international legislation makes their practice more costly, time-consuming, and error-prone. The two main reasons are that most legislation is usually unstructured, and the tremendous amount and pace with which laws are released causes information overload in their daily tasks. In the case of the legal domain, the research community agrees that a system allowing to generate of automatic responses to legal questions could substantially impact many practical implications in daily activities. The degree of usefulness is such that even a semi-automatic solution could significantly help reduce the workload. This is mainly because a Question–Answering system could automatically process a massive amount of legal resources to answer a question or doubt in seconds. It could save resources in the form of effort, money, and time for many professionals in the legal sector. This work quantitatively and qualitatively survey the existing solutions to meet this challenge.

References

[1]

Bench-Capon T.J.M., Araszkiewicz M., Ashley K.D., Atkinson K., Bex F., Borges F., Bourcier D., Bourgine P., Conrad J.G., Francesconi E., Gordon T.F., Governatori G., Leidner J.L., Lewis D.D., Loui R.P., McCarty L.T., Prakken H., Schilder F., Schweighofer E., Thompson P., Tyrrell A., Verheij B., Walton D.N., Wyner A.Z., A history of AI and Law in 50 papers: 25 years of the international conference on AI and Law, Artif. Intell. Law 20 (3) (2012) 215–319,.

Digital Library

[2]

Kolomiyets O., Moens M., A survey on question answering technology from an information retrieval perspective, Inform. Sci. 181 (24) (2011) 5412–5434,.

Digital Library

[3]

Wang M., A survey of answer extraction techniques in factoid question answering, Comput. Linguist. 1 (1) (2006) 1–14.

[4]

Höffner K., Walter S., Marx E., Usbeck R., Lehmann J., Ngomo A.N., Survey on challenges of question answering in the semantic web, Semantic Web 8 (6) (2017) 895–920,.

Digital Library

[5]

Diefenbach D., López V., Singh K.D., Maret P., Core techniques of question answering systems over knowledge bases: a survey, Knowl. Inf. Syst. 55 (3) (2018) 529–569,.

Digital Library

[6]

Franco W., Avila C.V.S., Oliveira A., Maia G., Brayner A., Vidal V.M.P., Carvalho F., Pequeno V.M., Ontology-based question answering systems over knowledge bases: A survey, in: Filipe J., Smialek M., Brodsky A., Hammoudi S. (Eds.), Proceedings of the 22nd International Conference on Enterprise Information Systems, ICEIS 2020, Prague, Czech Republic, May 5-7, 2020, Volume 1, SCITEPRESS, 2020, pp. 532–539,.

[7]

Dimitrakis E., Sgontzos K., Tzitzikas Y., A survey on question answering systems over linked data and documents, J. Intell. Inf. Syst. 55 (2) (2020) 233–259,.

[8]

da Silva J.W.F., Venceslau A.D.P., Sales J.E., Maia J.G.R., Pinheiro V.C.M., Vidal V.M.P., A short survey on end-to-end simple question answering systems, Artif. Intell. Rev. 53 (7) (2020) 5429–5453,.

Digital Library

[9]

Rogers A., Gardner M., Augenstein I., QA dataset explosion: A taxonomy of NLP resources for question answering and reading comprehension, ACM Comput. Surv. 55 (10) (2023) 197:1–197:45,.

Digital Library

[10]

Kim M., Rabelo J., Goebel R., Statute law information retrieval and entailment, in: Proceedings of the Seventeenth International Conference on Artif. Intell. Law, ICAIL 2019, Montreal, QC, Canada, June 17–21, 2019, ACM, 2019, pp. 283–289,.

Digital Library

[11]

Yoshioka M., Aoki Y., Suzuki Y., BERT-based ensemble methods with data augmentation for legal textual entailment in COLIEE statute law task, in: Maranhão J., Wyner A.Z. (Eds.), ICAIL ’21: Eighteenth International Conference for Artificial Intelligence and Law, São Paulo Brazil, June 21 - 25, 2021, ACM, 2021, pp. 278–284,.

Digital Library

[12]

T. Mikolov, I. Sutskever, K. Chen, G.S. Corrado, J. Dean, Distributed Representations of Words and Phrases and their Compositionality, in: Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a Meeting Held December 5-8, 2013, Lake Tahoe, Nevada, United States, 2013, pp. 3111–3119.

[13]

Devlin J., Chang M., Lee K., Toutanova K., BERT: Pre-training of deep bidirectional transformers for language understanding, in: Burstein J., Doran C., Solorio T. (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Association for Computational Linguistics, 2019, pp. 4171–4186,.

[14]

Liu Y., Ott M., Goyal N., Du J., Joshi M., Chen D., Levy O., Lewis M., Zettlemoyer L., Stoyanov V., Roberta: A robustly optimized bert pretraining approach, 2019, arXiv preprint arXiv:1907.11692.

[15]

M.E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, Deep contextualized word representations, in: Proceedings of NAACL-HLT, 2018, pp. 2227–2237.

[16]

Cer D., Yang Y., Kong S., Hua N., Limtiaco N., John R.S., Constant N., Guajardo-Cespedes M., Yuan S., Tar C., Strope B., Kurzweil R., Universal sentence encoder for English, in: Blanco E., Lu W. (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 - November 4, 2018, Association for Computational Linguistics, 2018, pp. 169–174,.

[17]

Soares M.A.C., Parreiras F.S., A literature review on question answering techniques, paradigms and systems, J. King Saud Univ. Comput. Inf. Sci. 32 (6) (2020) 635–646,.

[18]

Kim M., Xu Y., Goebel R., Satoh K., Answering yes/no questions in legal bar exams, in: Nakano Y.I., Satoh K., Bekki D. (Eds.), New Frontiers in Artificial Intelligence - JSAI-IsAI 2013 Workshops, LENLS, JURISIN, MiMI, AAA, and DDS, Kanagawa, Japan, October 27-28, 2013, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 8417, Springer, 2013, pp. 199–213,.

Digital Library

[19]

Kim M., Xu Y., Goebel R., Legal question answering using ranking SVM and syntactic/semantic similarity, in: Murata T., Mineshima K., Bekki D. (Eds.), New Frontiers in Artificial Intelligence - JSAI-IsAI 2014 Workshops, LENLS, JURISIN, and GABA, Kanagawa, Japan, October 27-28, 2014, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 9067, Springer, 2014, pp. 244–258,.

Digital Library

[20]

Kim M., Xu Y., Lu Y., Goebel R., Question answering of bar exams by paraphrasing and legal text analysis, in: Kurahashi S., Ohta Y., Arai S., Satoh K., Bekki D. (Eds.), New Frontiers in Artificial Intelligence - JSAI-IsAI 2016 Workshops, LENLS, HAT-MASH, AI-Biz, JURISIN and SKL, Kanagawa, Japan, November 14-16, 2016, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 10247, Springer, 2016, pp. 299–313,.

Digital Library

[21]

Taniguchi R., Kano Y., Legal yes/no question answering system using case-role analysis, in: Kurahashi S., Ohta Y., Arai S., Satoh K., Bekki D. (Eds.), New Frontiers in Artificial Intelligence - JSAI-IsAI 2016 Workshops, LENLS, HAT-MASH, AI-Biz, JURISIN and SKL, Kanagawa, Japan, November 14-16, 2016, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 10247, Springer, 2016, pp. 284–298,.

Digital Library

[22]

Taniguchi R., Hoshino R., Kano Y., Legal question answering system using FrameNet, in: Kojima K., Sakamoto M., Mineshima K., Satoh K. (Eds.), New Frontiers in Artificial Intelligence - JSAI-IsAI 2018 Workshops, JURISIN, AI-Biz, SKL, LENLS, IDAA, Yokohama, Japan, November 12-14, 2018, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 11717, Springer, 2018, pp. 193–206,.

Digital Library

[23]

Kano Y., Hoshino R., Taniguchi R., Analyzable legal yes/no question answering system using linguistic structures, in: Satoh K., Kim M., Kano Y., Goebel R., Oliveira T. (Eds.), COLIEE 2017. 4th Competition on Legal Information Extraction and Entailment, Held in Conjunction with the 16th International Conference on Artif. Intell. Law, in: EPiC Series in Computing, vol. 47, ICAIL 2017 in King’s College London, UK, EasyChair, 2017, pp. 57–67.

[24]

Deerwester S.C., Dumais S.T., Landauer T.K., Furnas G.W., Harshman R.A., Indexing by latent semantic analysis, J. Am. Soc. Inf. Sci. 41 (6) (1990) 391–407.

[25]

Li Y., Bandar Z., McLean D., An approach for measuring semantic similarity between words using multiple information sources, IEEE Trans. Knowl. Data Eng. 15 (4) (2003) 871–882,.

Digital Library

[26]

Han L., Kashyap A.L., Finin T., Mayfield J., Weese J., UMBC_EBIQUITY-CORE: Semantic textual similarity systems, Proceedings of the Second Joint Conference on Lexical and Computational Semantics, Association for Computational Linguistics, 2013, pp. 44–52.

[27]

Aydin B.I., Yilmaz Y.S., Li Y., Li Q., Gao J., Demirbas M., Crowdsourcing for multiple-choice question answering, in: Brodley C.E., Stone P. (Eds.), Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, July 27–31, 2014, QuÉBec City, QuÉBec, Canada, AAAI Press, 2014, pp. 2946–2953.

[28]

Martinez-Gil J., Freudenthaler B., Tjoa A.M., Multiple choice question answering in the legal domain using reinforced co-occurrence, in: Hartmann S., Küng J., Chakravarthy S., Anderst-Kotsis G., Tjoa A.M., Khalil I. (Eds.), Database and Expert Systems Applications - 30th International Conference, DEXA 2019, Linz, Austria, August 26-29, 2019, Proceedings, Part I, in: Lecture Notes in Computer Science, vol. 11706, Springer, 2019, pp. 138–148,.

Digital Library

[29]

Chitta R., Hudek A.K., A reliable and accurate multiple choice question answering system for due diligence, in: Proceedings of the Seventeenth International Conference on Artif. Intell. Law, ICAIL 2019, Montreal, QC, Canada, June 17–21, 2019, ACM, 2019, pp. 184–188,.

Digital Library

[30]

Ferrucci D.A., Levas A., Bagchi S., Gondek D., Mueller E.T., Watson: Beyond jeopardy!, Artificial Intelligence 199–200 (2013) 93–105,.

Digital Library

[31]

Tesauro G., Gondek D., Lenchner J., Fan J., Prager J.M., Analysis of Watson’s strategies for playing Jeopardy!, J. Artificial Intelligence Res. 47 (2013) 205–251,.

[32]

Peñas A., Forner P., Sutcliffe R.F.E., Rodrigo Á., Forascu C., Alegria I., Giampiccolo D., Moreau N., Osenova P., Overview of ResPubliQA 2009: Question answering evaluation over European legislation, in: Peters C., Nunzio G.M.D., Kurimo M., Mandl T., Mostefa D., nas A.P., Roda G. (Eds.), Multilingual Information Access Evaluation I. Text Retrieval Experiments, 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 6241, Springer, 2009, pp. 174–196,.

[33]

Peñas A., Forner P., Rodrigo Á., Sutcliffe R.F.E., Forascu C., Mota C., Overview of ResPubliQA 2010: Question answering evaluation over European legislation, in: Braschler M., Harman D., Pianta E. (Eds.), CLEF 2010 LABs and Workshops, Notebook Papers, in: CEUR Workshop Proceedings, vol. 1176, 22-23 September 2010, Padua, Italy, CEUR-WS.org, 2010.

[34]

Brüninghaus S., Ashley K.D., Improving the representation of legal case texts with information extraction methods, in: Loui R.P. (Ed.), Proceedings of the Eigths International Conference on Artif. Intell. Law, ICAIL 2001, St. Louis, Missouri, USA, May 21-25, 2001, ACM, 2001, pp. 42–51,.

Digital Library

[35]

Quaresma P., Rodrigues I., A question-answering system for Portuguese juridical documents, in: Sartor G. (Ed.), The Tenth International Conference on Artificial Intelligence and Law, Proceedings of the Conference, June 6-11, 2005, Bologna, Italy, ACM, 2005, pp. 256–257,.

Digital Library

[36]

Maxwell K.T., Schafer B., Concept and context in legal information retrieval, in: Francesconi E., Sartor G., Tiscornia D. (Eds.), Legal Knowledge and Information Systems - JURIX 2008: The Twenty-First Annual Conference on Legal Knowledge and Information Systems, Florence, Italy, 10-13 December 2008, in: Frontiers in Artificial Intelligence and Applications, vol. 189, IOS Press, 2008, pp. 63–72,.

[37]

Monroy A., Calvo H., Gelbukh A.F., Using graphs for shallow question answering on legal documents, in: Gelbukh A.F., Morales E.F. (Eds.), MICAI 2008: Advances in Artificial Intelligence, 7th Mexican International Conference on Artificial Intelligence, Atizapán de Zaragoza, Mexico, October 27-31, 2008, Proceedings, in: Lecture Notes in Computer Science, vol. 5317, Springer, 2008, pp. 165–173,.

Digital Library

[38]

Monroy A., Calvo H., Gelbukh A.F., NLP for shallow question answering of legal documents using graphs, in: Gelbukh A.F. (Ed.), Computational Linguistics and Intelligent Text Processing, 10th International Conference, CICLing 2009, Mexico City, Mexico, March 1-7, 2009. Proceedings, in: Lecture Notes in Computer Science, vol. 5449, Springer, 2009, pp. 498–508,.

Digital Library

[39]

Tran O.T., Bach N.X., Nguyen M.L., Shimazu A., Answering legal questions by mining reference information, in: Nakano Y.I., Satoh K., Bekki D. (Eds.), New Frontiers in Artificial Intelligence - JSAI-IsAI 2013 Workshops, LENLS, JURISIN, MiMI, AAA, and DDS, Kanagawa, Japan, October 27-28, 2013, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 8417, Springer, 2013, pp. 214–229,.

Digital Library

[40]

Rodrigo Á., Pérez-Iglesias J., Peñas A., Garrido G., Araujo L., Answering questions about European legislation, Expert Syst. Appl. 40 (15) (2013) 5811–5816,.

Digital Library

[41]

D.S. Carvalho, V.D. Tran, K. Van Tran, V.D. Lai, M.-L. Nguyen, Lexical to discourse-level corpus modeling for legal question answering, in: Tenth International Workshop on Juris-Informatics, JURISIN, 2016.

[42]

Bach N.X., Cham L.T.N., Thien T.H.N., Phuong T.M., Question analysis for Vietnamese legal question answering, in: Nguyen T.T., Le A., Tojo S., Nguyen L., Phan X. (Eds.), 9th International Conference on Knowledge and Systems Engineering, KSE 2017, Hue, Vietnam, October 19-21, 2017, IEEE, 2017, pp. 154–159,.

[43]

van Kuppevelt D.E., van Dijck G., Answering legal research questions about dutch case law with network analysis and visualization, in: Wyner A.Z., Casini G. (Eds.), Legal Knowledge and Information Systems - JURIX 2017: The Thirtieth Annual Conference, Luxembourg, 13-15 December 2017, in: Frontiers in Artificial Intelligence and Applications, vol. 302, IOS Press, 2017, pp. 95–100,.

[44]

Delfino P., Cuconato B., Passos G.P., Zaverucha G., Rademaker A., Using OpenWordnet-PT for question answering on legal domain, in: Bond F., Vossen P., Fellbaum C. (Eds.), Proceedings of the 9th Global Wordnet Conference, GWC 2018, Nanyang Technological University (NTU), Singapore, January 8–12, 2018, Global Wordnet Association, 2018, pp. 105–112.

[45]

Hoshino R., Taniguchi R., Kiyota N., Kano Y., Question answering system for legal bar examination using predicate argument structure, in: Kojima K., Sakamoto M., Mineshima K., Satoh K. (Eds.), New Frontiers in Artificial Intelligence - JSAI-IsAI 2018 Workshops, JURISIN, AI-Biz, SKL, LENLS, IDAA, Yokohama, Japan, November 12-14, 2018, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 11717, Springer, 2018, pp. 207–220,.

Digital Library

[46]

McElvain G., Sanchez G., Matthews S., Teo D., Pompili F., Custis T., WestSearch plus: A non-factoid question-answering system for the legal domain, in: Piwowarski B., Chevalier M., Gaussier E., Maarek Y., Nie J., Scholer F. (Eds.), Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, ACM, 2019, pp. 1361–1364,.

Digital Library

[47]

McElvain G., Sanchez G., Teo D., Custis T., Non-factoid question answering in the legal domain, in: Piwowarski B., Chevalier M., Gaussier E., Maarek Y., Nie J., Scholer F. (Eds.), Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, ACM, 2019, pp. 1395–1396,.

Digital Library

[48]

Wehnert S., Hoque S.A., Fenske W., Saake G., Threshold-based retrieval and textual entailment detection on legal bar exam questions, 2019, arXiv preprint arXiv:1905.13350.

[49]

Verma A., Morato J., Jain A., Arora A., Relevant subsection retrieval for law domain question answer system, in: Data Visualization and Knowledge Engineering, Springer, 2020, pp. 299–319.

[50]

S. Khazaeli, J. Punuru, C. Morris, S. Sharma, B. Staub, M. Cole, S. Chiu-Webster, D. Sakalley, A Free Format Legal Question Answering System, in: Proceedings of the Natural Legal Language Processing Workshop 2021, 2021, pp. 107–113.

[51]

Hoppe C., Migenda N., Pelkmann D., Hötte D., Schenck W., Collaborative system for question answering in German case law documents, in: Collaborative Networks in Digitalization and Society 5.0: 23rd IFIP WG 5.5 Working Conference on Virtual Enterprises, PRO-VE 2022, Lisbon, Portugal, September 19–21, 2022, Proceedings, Springer, 2022, pp. 303–312.

[52]

Kim M., Goebel R., Two-step cascaded textual entailment for legal bar exam question answering, in: Keppens J., Governatori G. (Eds.), Proceedings of the 16th Edition of the International Conference on Artif. Intell. Law, ICAIL 2017, London, United Kingdom, June 12–16, 2017, ACM, 2017, pp. 283–290,.

Digital Library

[53]

Bennett Z., Russell-Rose T., Farmer K., A scalable approach to legal question answering, in: Keppens J., Governatori G. (Eds.), Proceedings of the 16th Edition of the International Conference on Artif. Intell. Law, ICAIL 2017, London, United Kingdom, June 12-16, 2017, ACM, 2017, pp. 269–270,.

Digital Library

[54]

Mimouni N., Nazarenko A., Salotti S., Answering complex queries on legal networks: A direct and a structured IR approaches, in: Pagallo U., Palmirani M., Casanovas P., Sartor G., Villata S. (Eds.), AI Approaches To the Complexity of Legal Systems - AICOL International Workshops 2015-2017: AICOL-VI JURIX 2015, AICOL-VII EKAW 2016, AICOL-VIII JURIX 2016, AICOL-IX ICAIL 2017, and AICOL-X JURIX 2017, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 10791, Springer, 2017, pp. 451–464,.

[55]

Martinez-Gil J., Freudenthaler B., Tjoa A.M., A general framework for multiple choice question answering based on mutual information and reinforced co-occurrence, Trans. Large Scale Data Knowl. Centered Syst. 42 (2019) 91–110,.

Digital Library

[56]

Lame G., Using NLP techniques to identify legal ontology components: Concepts and relations, Artif. Intell. Law 12 (4) (2004) 379–396,.

Digital Library

[57]

Xu K., Reddy S., Feng Y., Huang S., Zhao D., Question answering on freebase via relation extraction and textual evidence, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers, The Association for Computer Linguistics, 2016,.

[58]

Fawei B., Pan J.Z., Kollingbaum M.J., Wyner A.Z., A methodology for a criminal law and procedure ontology for legal question answering, in: Ichise R., Lécué F., Kawamura T., Zhao D., Muggleton S., Kozaki K. (Eds.), Semantic Technology - 8th Joint International Conference, JIST 2018, Awaji, Japan, November 26-28, 2018, Proceedings, in: Lecture Notes in Computer Science, vol. 11341, Springer, 2018, pp. 198–214,.

[59]

Fawei B., Pan J.Z., Kollingbaum M.J., Wyner A.Z., A semi-automated ontology construction for legal question answering, New Gener. Comput. 37 (4) (2019) 453–478,.

[60]

Veena G., Gupta D., Anil A., Akhil S., An ontology driven question answering system for legal documents, in: 2019 2nd International Conference on Intelligent Computing, Instrumentation and Control Technologies, Vol. 1, ICICICT, IEEE, 2019, pp. 947–951.

[61]

Kourtin I., Mbarki S., Mouloudi A., A legal question answering ontology-based system, in: Formalising Natural Languages: Applications to Natural Language Processing and Digital Humanities: 14th International Conference, Nooj 2020, Zagreb, Croatia, June 5–7, 2020, Revised Selected Papers 14, Springer International Publishing, 2021, pp. 218–229.

[62]

Kim M., Xu Y., Goebel R., Applying a convolutional neural network to legal question answering, in: Otake M., Kurahashi S., Ota Y., Satoh K., Bekki D. (Eds.), New Frontiers in Artificial Intelligence - JSAI-IsAI 2015 Workshops, LENLS, JURISIN, AAA, HAT-MASH, TSDAA, ASD-HR, and SKL, Kanagawa, Japan, November 16-18, 2015, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 10091, 2015, pp. 282–294,.

[63]

Do P., Nguyen H., Tran C., Nguyen M., Nguyen M., Legal question answering using ranking SVM and deep convolutional neural network, CoRR abs/1703.05320, 2017, arXiv:1703.05320.

[64]

Kim M., Lu Y., Goebel R., Textual entailment in legal bar exam question answering using deep Siamese networks, in: Arai S., Kojima K., Mineshima K., Bekki D., Satoh K., Ohta Y. (Eds.), New Frontiers in Artificial Intelligence - JSAI-IsAI Workshops, JURISIN, SKL, AI-Biz, LENLS, AAA, SCIDOCA, KNeXI, Tsukuba, Tokyo, Japan, November 13-15, 2017, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 10838, Springer, 2017, pp. 35–48.

[65]

A. John, L.D. Caro, G. Boella, C. Bartolini, An approach to information retrieval and question answering in the legal domain, in: Proceedings of the 10th International Workshop on Juris-Informatics, JURISIN 2016, 2016.

[66]

Morimoto A., Kubo D., Sato M., Shindo H., Matsumoto Y., Legal question answering system using neural attention, in: Satoh K., Kim M., Kano Y., Goebel R., Oliveira T. (Eds.), COLIEE 2017. 4th Competition on Legal Information Extraction and Entailment, Held in Conjunction with the 16th International Conference on Artif. Intell. Law, in: EPiC Series in Computing, vol. 47, ICAIL 2017 in King’s College London, UK, EasyChair, 2017, pp. 79–89.

[67]

Xiao G., Mo J., Chow E., Chen H., Guo J., Gong Z., Multi-task CNN for classification of Chinese legal questions, in: Hussain O., Jiang L., Fei X., Lan C., Chao K. (Eds.), 14th IEEE International Conference on E-Business Engineering, ICEBE 2017, Shanghai, China, November 4-6, 2017, IEEE Computer Society, 2017, pp. 84–90,.

[68]

D. Collarana, T. Heuss, J. Lehmann, I. Lytra, G. Maheshwari, R. Nedelchev, T. Schmidt, P. Trivedi, A Question Answering System on Regulatory Documents, in: JURIX, 2018, pp. 41–50.

[69]

Nicula B., Ruseti S., Rebedea T., Improving deep learning for multiple choice question answering with candidate contexts, in: Pasi G., Piwowarski B., Azzopardi L., Hanbury A. (Eds.), Advances in Information Retrieval - 40th European Conference on IR Research, ECIR 2018, Grenoble, France, March 26-29, 2018, Proceedings, in: Lecture Notes in Computer Science, vol. 10772, Springer, 2018, pp. 678–683,.

[70]

Liu L., Luo J., A question answering system based on deep learning, in: Huang D., Gromiha M.M., Han K., Hussain A. (Eds.), Intelligent Computing Methodologies - 14th International Conference, ICIC 2018, Wuhan, China, August 15-18, 2018, Proceedings, Part III, in: Lecture Notes in Computer Science, vol. 10956, Springer, 2018, pp. 173–181,.

Digital Library

[71]

Ravichander A., Black A.W., Wilson S., Norton T.B., Sadeh N.M., Question answering for privacy policies: Combining computational and legal perspectives, in: Inui K., Jiang J., Ng V., Wan X. (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Association for Computational Linguistics, 2019, pp. 4946–4957,.

[72]

Zhong H., Wang Y., Tu C., Zhang T., Liu Z., Sun M., Iteratively questioning and answering for interpretable legal judgment prediction, in: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, the Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, the Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, AAAI Press, 2020, pp. 1250–1257.

[73]

Zhong H., Xiao C., Tu C., Zhang T., Liu Z., Sun M., JEC-QA: A legal-domain question answering dataset, in: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, the Thirty-Second Innovative Applications of Artificial Intelligence Conference, iaai 2020, the Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, AAAI Press, 2020, pp. 9701–9708.

[74]

Kien P.M., Nguyen H., Bach N.X., Tran V., Nguyen M.L., Phuong T.M., Answering legal questions by learning neural attentive text representation, in: Scott D., Bel N., Zong C. (Eds.), Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8–13, 2020, International Committee on Computational Linguistics, 2020, pp. 988–998,.

[75]

J. Wu, J. Liu, X. Luo, Few-Shot Legal Knowledge Question Answering System for COVID-19 Epidemic, in: 2020 3rd International Conference on Algorithms, Computing and Artificial Intelligence, 2020, pp. 1–6.

[76]

Huang W., Liao X., Xie Z., Qian J., Zhuang B., Wang S., Xiao J., Generating reasonable legal text through the combination of language modeling and question answering, in: Bessiere C. (Ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, ijcai.org, 2020, pp. 3687–3693,.

[77]

Van H.N., Nguyen D., Nguyen P.M., Nguyen M.L., Miko team: Deep learning approach for legal question answering in ALQAC 2022, in: Hung N.B., Vinh L.S., Minh N.L., Hoang H.M., Lam B.T., Hieu P.X., Lap N.T., Hieu V.D., Nhat V.H.Q. (Eds.), 14th International Conference on Knowledge and Systems Engineering, KSE 2022, Nha Trang, Vietnam, October 19-21, 2022, IEEE, 2022, pp. 1–5,.

[78]

Zhong B., He W., Huang Z., Love P.E.D., Tang J., Luo H., A building regulation question answering system: A deep learning methodology, Adv. Eng. Inform. 46 (2020),.

[79]

Linh L.H., Long N.H., Yen N.H., Vuong T., Vietnamese legal question answering with combined features and deep learning, in: Theeramunkong T., Thuy N.T., Minh N.L., Supnith T., Bui L.T., Miyake Y., Racharak T., Hashimoto K., Yingthawornsuk T. (Eds.), 13th International Conference on Knowledge and Systems Engineering, KSE 2021, Bangkok, Thailand, November 10–12, 2021, IEEE, 2021, pp. 1–4,.

[80]

Zhang N.-N., Xing Y., Questions and answers on legal texts based on BERT-BiGRU, in: Journal of Physics: Conference Series, 1828, IOP Publishing, 2021.

[81]

Kia M.A., Garifullina A., Kern M., Chamberlain J., Jameel S., Adaptable closed-domain question answering using contextualized CNN-attention models and question expansion, IEEE Access 10 (2022) 45080–45092.

[82]

Holzenberger N., Blair-Stanek A., Van Durme B., A dataset for statutory reasoning in tax law entailment and question answering, 2020, arXiv preprint arXiv:2005.05257.

[83]

Hoppe C., Pelkmann D., Migenda N., Hötte D., Schenck W., Towardss intelligent legal advisors for document retrieval and question-answering in German legal documents, in: Fourth IEEE International Conference on Artificial Intelligence and Knowledge Engineering, AIKE 2021, Laguna Hills, CA, USA, December 1–3, 2021, IEEE, 2021, pp. 29–32,.

[84]

Huy N.Q., Tuan N.M.D., Duong N.A., Minh P.Q.N., AimeLaw at ALQAC 2021: Enriching neural network models with legal-domain knowledge, in: Theeramunkong T., Thuy N.T., Minh N.L., Supnith T., Bui L.T., Miyake Y., Racharak T., Hashimoto K., Yingthawornsuk T. (Eds.), 13th International Conference on Knowledge and Systems Engineering, KSE 2021, Bangkok, Thailand, November 10-12, 2021, IEEE, 2021, pp. 1–6,.

[85]

Tieu T., Chau C., Bui N., Nguyen T., Nguyen L., Apply bert-based models and domain knowledge for automated legal question answering tasks at ALQAC 2021, in: Theeramunkong T., Thuy N.T., Minh N.L., Supnith T., Bui L.T., Miyake Y., Racharak T., Hashimoto K., Yingthawornsuk T. (Eds.), 13th International Conference on Knowledge and Systems Engineering, KSE 2021, Bangkok, Thailand, November 10-12, 2021, IEEE, 2021, pp. 1–6,.

[86]

Vold A., Conrad J.G., Using transformers to improve answer retrieval for legal questions, in: Maranhão J., Wyner A.Z. (Eds.), ICAIL ’21: Eighteenth International Conference for Artificial Intelligence and Law, São Paulo Brazil, June 21 - 25, 2021, ACM, 2021, pp. 245–249,.

Digital Library

[87]

Wu J., Luo X., Alignment-based graph network for judicial examination task, in: Qiu H., Zhang C., Fei Z., Qiu M., Kung S. (Eds.), Knowledge Science, Engineering and Management - 14th International Conference, KSEM 2021, Tokyo, Japan, August 14-16, 2021, Proceedings, Part III, in: Lecture Notes in Computer Science, vol. 12817, Springer, 2021, pp. 386–400,.

Digital Library

[88]

Zhang H., Zhang T., Cao F., Wang Z., Zhang Y., Sun Y., Vicente M.A., BCA: Bilinear convolutional neural networks and attention networks for legal question answering, AI Open 3 (2022) 172–181,.

[89]

Nguyen H.-L., Nguyen T.-B., Nguyen T.-M., Nguyen H.-T., Vuong H.-Y.T., Vlh team at alqac 2022: Retrieving legal document and extracting answer with bert-based model, in: 2022 14th International Conference on Knowledge and Systems Engineering, KSE, IEEE, 2022, pp. 1–6.

[90]

S. Abualhaija, C. Arora, L.C. Briand, COREQQA: a COmpliance REQuirements understanding using question answering tool, in: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 1682–1686.

[91]

E.Q. Caballero, M.S. Rahman, T. Cerny, P. Rivas, G. Bejarano, Study of Question Answering on Legal Software Document using BERT based models, in: LatinX in Natural Language Processing Research Workshop, 2022.

[92]

Chalkidis I., Fergadiotis M., Malakasiotis P., Aletras N., Androutsopoulos I., LEGAL-BERT: “Preparing the muppets for court”, in: Cohn T., He Y., Liu Y. (Eds.), Findings of the Association for Computational Linguistics, in: Findings of ACL, vol. EMNLP 2020, EMNLP 2020, Online Event, 16-20 November 2020, Association for Computational Linguistics, 2020, pp. 2898–2904,.

[93]

Górski L., Ramakrishna S., Explainable artificial intelligence, lawyer’s perspective, in: Maranhão J., Wyner A.Z. (Eds.), ICAIL ’21: Eighteenth International Conference for Artificial Intelligence and Law, São Paulo Brazil, June 21 - 25, 2021, ACM, 2021, pp. 60–68,.

Digital Library

[94]

Selvaraju R.R., Cogswell M., Das A., Vedantam R., Parikh D., Batra D., Grad-CAM: Visual explanations from deep networks via gradient-based localization, in: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, IEEE Computer Society, 2017, pp. 618–626,.

[95]

Ribeiro M.T., Singh S., Guestrin C., ”Why should I trust you?”: Explaining the predictions of any classifier, in: Krishnapuram B., Shah M., Smola A.J., Aggarwal C.C., Shen D., Rastogi R. (Eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, ACM, 2016, pp. 1135–1144,.

Digital Library

[96]

S.M. Lundberg, S. Lee, A Unified Approach to Interpreting Model Predictions, in: I. Guyon, U. von Luxburg, S. Bengio, H.M. Wallach, R. Fergus, S.V.N. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017, pp. 4765–4774.

[97]

He S., Liu S., Chen Y., Zhou G., Liu K., Zhao J., CASIA QALD-3: A question answering system over linked data, in: Forner P., Navigli R., Tufis D., Ferro N. (Eds.), Working Notes for CLEF 2013 Conference, in: CEUR Workshop Proceedings, vol. 1179, Valencia, Spain, September 23-26, 2013, CEUR-WS.org, 2013.

[98]

Filtz E., Building and processing a knowledge-graph for legal data, in: Blomqvist E., Maynard D., Gangemi A., Hoekstra R., Hitzler P., Hartig O. (Eds.), The Semantic Web - 14th International Conference, ESWC 2017, Portorož, Slovenia, May 28 - June 1, 2017, Proceedings, Part II, in: Lecture Notes in Computer Science, vol. 10250, 2017, pp. 184–194,.

Digital Library

[99]

Tong P., Zhang Q., Yao J., Leveraging domain context for question answering over knowledge graph, Data Sci. Eng. 4 (4) (2019) 323–335.

[100]

Sovrano F., Palmirani M., Vitali F., Legal knowledge extraction for knowledge graph based question-answering, in: Serena V., Harasta J., Kremen P. (Eds.), Legal Knowledge and Information Systems - JURIX 2020: The Thirty-Third Annual Conference, Brno, Czech Republic, December 9-11, 2020, in: Frontiers in Artificial Intelligence and Applications, vol. 334, IOS Press, 2020, pp. 143–153,.

[101]

Sovrano F., Palmirani M., Vitali F., Legal knowledge extraction for knowledge graph based question-answering, in: Legal Knowledge and Information Systems, IOS Press, 2020, pp. 143–153.

[102]

Huang W., Jiang J., Qu Q., Yang M., AILA: A question answering system in the legal domain, in: Bessiere C. (Ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, ijcai.org, 2020, pp. 5258–5260,.

[103]

Filtz E., Kirrane S., Polleres A., The linked legal data landscape: linking legal data across different countries, Artif. Intell. Law (2021) 1–55.

[104]

Dai F., Zhao Z., Sun C., Li B., Intelligent audit question answering system based on knowledge graph and semantic similarity, in: 2022 11th International Conference of Information and Communication Technology, ICTech, IEEE, 2022, pp. 125–132.

[105]

Thomas A., Sangeetha S., Knowledge graph based question-answering system for effective case law analysis, in: Evolution in Computational Intelligence: Proceedings of the 9th International Conference on Frontiers in Intelligent Computing: Theory and Applications, FICTA 2021, Springer, 2022, pp. 291–300.

[106]

Martinez-Gil J., Yin S., Küng J., Morvan F., Knowledge graph augmentation for increased question answering accuracy, Trans. Large Scale Data Knowl. Centered Syst. 52 (2022) 70–85,.

[107]

Sovrano F., Palmirani M., Distefano B., Sapienza S., Vitali F., A dataset for evaluating legal question answering on private international law, in: Maranhão J., Wyner A.Z. (Eds.), ICAIL ’21: Eighteenth International Conference for Artificial Intelligence and Law, São Paulo Brazil, June 21 - 25, 2021, ACM, 2021, pp. 230–234,.

Digital Library

[108]

Yih W., Chang M., Meek C., Pastusiak A., Question answering using enhanced lexical semantic models, in: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, The Association for Computer Linguistics, 2013, pp. 1744–1753.

[109]

P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang, SQuAD: 100,000+ Questions for Machine Comprehension of Text, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016, pp. 2383–2392.

[110]

Chalkidis I., Kampas D., Deep learning in law: early adaptation and legal word embeddings trained on large corpora, Artif. Intell. Law 27 (2) (2019) 171–198,.

Digital Library

[111]

Nguyen H., Bui M., Nguyen C., Le T., Nguyen M.P., Dang T.B., Yen V.T.H., Racharak T., Minh N.L., Tran D., Anh P.V., Son N.T., Nguyen H.T., Butr-indr B., Vateekul P., Boonkwan P., A summary of the ALQAC 2021 competition, in: Theeramunkong T., Thuy N.T., Minh N.L., Supnith T., Bui L.T., Miyake Y., Racharak T., Hashimoto K., Yingthawornsuk T. (Eds.), 13th International Conference on Knowledge and Systems Engineering, KSE 2021, Bangkok, Thailand, November 10–12, 2021, IEEE, 2021, pp. 1–5,.

[112]

Bench-Capon T.J.M., Coenen F., The maintenance of legal knowledge based systems, Artif. Intell. Rev. 6 (2) (1992) 129–143,.

[113]

Martinez-Gil J., Automated knowledge base management: A survey, Comput. Sci. Rev. 18 (2015) 1–9,.

Digital Library

[114]

Cimiano P., López V., Unger C., Cabrio E., Ngomo A.N., Walter S., Multilingual question answering over linked data (QALD-3): lab overview, in: Forner P., Müller H., Paredes R., Rosso P., Stein B. (Eds.), Information Access Evaluation. Multilinguality, Multimodality, and Visualization - 4th International Conference of the CLEF Initiative, CLEF 2013, Valencia, Spain, September 23-26, 2013. Proceedings, in: Lecture Notes in Computer Science, vol. 8138, Springer, 2013, pp. 321–332,.

Digital Library

[115]

Zou L., Huang R., Wang H., Yu J.X., He W., Zhao D., Natural language question answering over RDF: a graph data driven approach, in: Dyreson C.E., Li F., Özsu M.T. (Eds.), International Conference on Management of Data, SIGMOD 2014, Snowbird, UT, USA, June 22-27, 2014, ACM, 2014, pp. 313–324,.

Digital Library

[116]

Zhong Q., Fan X., Luo X., Toni F., An explainable multi-attribute decision model based on argumentation, Expert Syst. Appl. 117 (2019) 42–61,.

[117]

Atkinson K., Bench-Capon T.J.M., Bollegala D., Explanation in AI and law: Past, present and future, Artificial Intelligence 289 (2020),.

[118]

Doshi-Velez F., Kim B., Towards a rigorous science of interpretable machine learning, 2017, arXiv preprint arXiv:1702.08608.

[119]

Martinez-Gil J., Chaves-Gonzalez J.M., A novel method based on symbolic regression for interpretable semantic similarity measurement, Expert Syst. Appl. 160 (2020),.

[120]

Rodrigo Á., Peñas A., A study about the future evaluation of Question-Answering systems, Knowl.-Based Syst. 137 (2017) 83–93,.

Digital Library

[121]

Peters C., Clough P.D., Gonzalo J., Jones G.J.F., Kluck M., Magnini B. (Eds.), Multilingual Information Access for Text, Speech and Images, 5th Workshop of the Cross-Language Evaluation Forum, clef 2004, Bath, UK, September 15-17, 2004, Revised Selected Papers, in: Lecture Notes in Computer Science, vol. 3491, Springer, 2005,.

[122]

Peñas A., Rodrigo Á., A simple measure to assess non-response, in: Lin D., Matsumoto Y., Mihalcea R. (Eds.), The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA, The Association for Computer Linguistics, 2011, pp. 1415–1424.

[123]

Chen A., Stanovsky G., Singh S., Gardner M., Evaluating question answering evaluation, in: Fisch A., Talmor A., Jia R., Seo M., Choi E., Chen D. (Eds.), Proceedings of the 2nd Workshop on Machine Reading for Question Answering, MRQA-EMNLP 2019, Hong Kong, China, November 4, 2019, Association for Computational Linguistics, 2019, pp. 119–124,.

[124]

Yan M., Zhang H., Jin D., Zhou J.T., Multi-source meta transfer for low resource multiple-choice question answering, in: Jurafsky D., Chai J., Schluter N., Tetreault J.R. (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5–10, 2020, Association for Computational Linguistics, 2020, pp. 7331–7341,.

[125]

Ashley K.D., Rissland E.L., Law, learning and representation, Artificial Intelligence 150 (1–2) (2003) 17–58,.

Digital Library

[126]

S. Riedel, L. Yao, A. McCallum, B.M. Marlin, Relation extraction with matrix factorization and universal schemas, in: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2013, pp. 74–84.

[127]

Grave E., Bojanowski P., Gupta P., Joulin A., Mikolov T., Learning word vectors for 157 languages, 2018, arXiv preprint arXiv:1802.06893.

[128]

Goldberg Y., Neural Network Methods for Natural Language Processing, in: Synthesis Lectures on Human Language Technologies, Morgan & Claypool Publishers, 2017,.

[129]

Satoh K., Takahashi K., Kawasaki T., Interactive system for arranging issues based on PROLEG in civil litigation, in: Maranhão J., Wyner A.Z. (Eds.), ICAIL ’21: Eighteenth International Conference for Artificial Intelligence and Law, São Paulo Brazil, June 21 - 25, 2021, ACM, 2021, pp. 273–274,.

Digital Library

Cited By

Zhang HSun YXu BLin H(2024)LegalATLE: an active transfer learning framework for legal triple extractionApplied Intelligence10.1007/s10489-024-05842-y54:24(12835-12850)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s10489-024-05842-y
Askari AYang ZRen ZVerberne S(2024)Answer Retrieval in Legal Community Question AnsweringAdvances in Information Retrieval10.1007/978-3-031-56063-7_40(477-485)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56063-7_40
Chen GLuo XZhu J(2023)A Legal Multi-Choice Question Answering Model Based on BERT and AttentionKnowledge Science, Engineering and Management10.1007/978-3-031-40292-0_21(250-266)Online publication date: 16-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-40292-0_21

Index Terms

A survey on legal question–answering systems

Index terms have been assigned to the content through auto-classification.

Recommendations

Multiple Choice Question Answering in the Legal Domain Using Reinforced Co-occurrence
Database and Expert Systems Applications
Abstract
Nowadays, the volume of legal information available is continuously growing. As a result, browsing and querying this huge legal corpus in search of specific information is currently a tedious task exacerbated by the fact that data presentation ...
Improving legal quality: an application report
ICAIL '03: Proceedings of the 9th international conference on Artificial intelligence and law

Problems with legal quality will not only increase effort and costs of the law enforcement organisations, but also undermines the regulating power of the legislator. Unintended use or even abuse of the law may be the result. Governments therefore should ...
BERT-CNN based evidence retrieval and aggregation for Chinese legal multi-choice question answering
Abstract
Legal question answering is an important natural language processing application in the legal domain. The Judicial Examination of Chinese Question Answering dataset is the most prominent and more challenging legal question answering dataset, which ...

Comments

Information & Contributors

Information

Published In

cover image Computer Science Review

Computer Science Review Volume 48, Issue C

May 2023

426 pages

ISSN:1574-0137

Issue’s Table of Contents

Elsevier Inc.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 May 2023

Author Tags

Qualifiers

Review-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 31 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang HSun YXu BLin H(2024)LegalATLE: an active transfer learning framework for legal triple extractionApplied Intelligence10.1007/s10489-024-05842-y54:24(12835-12850)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s10489-024-05842-y
Askari AYang ZRen ZVerberne S(2024)Answer Retrieval in Legal Community Question AnsweringAdvances in Information Retrieval10.1007/978-3-031-56063-7_40(477-485)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56063-7_40
Chen GLuo XZhu J(2023)A Legal Multi-Choice Question Answering Model Based on BERT and AttentionKnowledge Science, Engineering and Management10.1007/978-3-031-40292-0_21(250-266)Online publication date: 16-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-40292-0_21

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents