TAT-LLM: A Specialized Language Model for Discrete Reasoning over Financial Tabular and Textual Data

In this work, we develop a specialized language model with strong discrete reasoning capabilities to tackle question answering (QA) over hybrid tabular and textual data in finance. Compared with adopting online LLMs, specializing smaller LLMs is more advantageous in response to users’ concerns about cost, network latency, and data security risks. To this end, we first abstract a Step-wise Pipeline for tabular and textual QA to help LLMs better execute multi-step inference, which includes three key steps, i.e. Extractor, Reasoner and Executor. This pipeline is proved to bring great performance grains compared with applying other prompting strategies like Chain-of-Thought (CoT), and meanwhile provides better interpretability to the derivation of the answer, with fixed inference steps and intermediate outcomes as references. We then develop a TAT-LLM model by fine-tuning LLaMA 2 with the training data generated automatically from existing datasets following the Step-wise Pipeline. The experimental results have verified that our TAT-LLM model can outperform all compared models, including prior best fine-tuned models and very large-scale LLMs like GPT-4 on FinQA, TAT-QA and TAT-DQA benchmarks. It is hoped that this work will shed light on practical solutions to the intelligent understanding of financial documents in the future. The generated datasets and trained models will be made publicly available to facilitate future research on the development of financial LLMs.