TableGPT2: A Large Multimodal Model
with Tabular Data Integration

In the initial version of TableGPT [18], we introduced approaches such as structured Domain-Specific Languages (DSLs) and specialized table encoder to manage complex table-based queries. These techniques showed promise in structured data handling but also highlighted areas for further refinement. Our latest iteration, TableGPT2, advances these foundations with substantial improvements. We have scaled up data and training protocols, carefully redesigned each component, and introduced additional technical enhancements to improve robustness, broaden applicability, and optimize performance in diverse BI tasks.

To provide context, TableGPT2 is a large-scale multimodal model available in two configurations—a 7-billion-parameter model and a 72-billion-parameter model—both based on the Qwen2.5 model family. Training encompassed over 86 billion tokens for continual pretraining (CPT), more than 437.5K table-language interleaved samples for encoder training and over 2.36M high quality query-table-output tuples being utilized for supervised fine-tuning. This scale of data is unprecedented in related research, ensuring that TableGPT2 meets the rigorous demands of modern applications involving structured or tabular data.

TableGPT2’s language framework is built on the Qwen2.5 [1] architecture. It underwent continual pretraining (CPT), supervised fine-tuning (SFT), and an agent framework for production-level ability. These steps differ from traditional LLMs as our pretraining and fine-tuning place a stronger emphasis on coding, multi-turn reasoning, and tool usage. These characteristics ensure the model is not only adept at natural language processing but also highly capable of handling the intricate requirements of table-related tasks.

In addition, we preliminarily explored the possibilities of multimodal alignment for tabular data. Specifically, TableGPT2 innovatively incorporates a separate modality module specifically for reading and interpreting tabular data. Similar to vision-language models (VLMs), TableGPT2 involve a tabular data reading module that generates specialized embeddings concatenated with the token embeddings from textual input. This additional module allows TableGPT2 to better capture the structure and semantics of tabular data, enabling more accurate tabular comprehension in complex BI scenarios.

An overall model framework is depicted in Figure 1.

Given the immense importance of coding abilities for BI applications, our data curation process is divided into two main categories: coding data and general data.

For coding data, we sourced primarily from StackOverflow (stack-v2) and GitHub, two of the most comprehensive repositories of code examples and developer discussions. After gathering the data, we performed a deduplication process to eliminate overlapping code and ensure that the dataset remained diverse and non-redundant. For general data, we selected textbooks from fields such as finance, mathematics, and biology, ensuring a well-rounded representation of domain-specific knowledge. In addition to these, we incorporated data analysis materials (from the source including Kaggle [26]) to further refine the model’s understanding of data-driven decision-making processes.

Special emphasis was placed on selecting data that included numbers, code snippets, and tables, as these elements are critical in the BI space. It ensures that TableGPT2 can efficiently handle tasks that require reasoning across both textual and structured data formats.

We adopted a two-stage filtering flow, including document-level and token-level filtering to ensure the data quality of the CPT procedure.

We selected Qwen-2.5-Base [1] as the base model and trained with context lengths of 8192 tokens. To create batches of data, we shuffled and merged documents, then truncated them to the specified context lengths. A cosine learning rate schedule was employed, with a maximum learning rate of 3e-5 and 3% warm-up steps. We used BFloat16 mixed precision to ensure training stability. To further reduce instances of truncation during data batching, we grouped the data based on token lengths, ensuring that each batch contained as many complete data samples as possible, minimizing the need for truncation and preserving the integrity of the data.

The standard approach for utilizing LLMs in table-related tasks involves serializing tables into formats like Markdown, HTML, or XML to make them compatible with text-based inputs [31, 32]. While larger context windows enable LLMs to handle more table content, a fundamental gap remains between language-based models and structured table understanding. This gap is evident in three key aspects: 1) Structural incompatibility: the bi-dimensional structure of tabular data is inherently misaligned with the sequential, autoregressive nature of language models, making LLMs less sensitive to structure-related patterns within the context. Previous studies have also highlighted this as a critical limitation of LLMs in table understanding [33, 34]. 2) Expressive inefficiency: feeding LLMs with serialized tables is highly token-inefficient, particularly when dealing with large tables or databases. This leads to excessive computational costs and potential performance degradation due to the long contexts required. 3) Lack of generalizability: real-world tables are often noisy and ambiguous [35], and when directly serialized into prompts, they can negatively impact the LLMs’ reasoning capabilities. Furthermore, due to context length limitations, including only a partial view of the table in the prompt may introduce context biases, misleading the model’s interpretation. To address these challenges, we propose a dedicated Semantic Table Encoder that provides the LLM with a structure-aware, token-efficient representation of table semantics.

Our design for the Semantic Table Encoder consists of three main components: 1) the semantic Table Encoder, which generates structure-enriched semantic embeddings from the input tables; 2) the table-language Adapter, which aggregates the encoder’s output embeddings and aligns them with the textual features; and 3) the backbone LLM, which utilizes these high-level table representations to perform reasoning within the joint table-text space.

Recall that our aim is to handle flexible and dynamic data analysis tasks. We believe that supervision of processes involving the selection, analysis, computation, and manipulation of table contents is crucial. However, existing table analysis datasets fall short of providing such process-level supervision. A significant portion of TableQA datasets only contain table-question-answer pairs, which limits the activation of large models’ capabilities for quantitative table operations. Meanwhile, available NL2SQL datasets are either too simplistic in question complexity or lack sufficient data volume. To address this gap, in addition to gathering existing tabular datasets, we have collected an additional batch of data containing 115K tables along with 479K samples. Particularly, our approach involves several key types of data: general corpora, coding corpora, and table-related query-answer pairs.

In addition to these data types, our SFT process also incorporates both single-turn and multi-turn interactions, particularly for table-related tasks. This reflects real-world scenarios where multi-turn dialogue is often necessary to retrieve or analyze tabular data over multiple exchanges.

It is important to note that recent research [46] has suggested that many Tabular LLMs fail at general language tasks as a trade-off for optimized table-related performance. However, in TableGPT2, we maintain a balanced ratio of general data during fine-tuning. By carefully tuning the SFT process, we address this trade-off and ensure strong performance across both general and table-specific tasks, shown in Section 7. In the sequel, we illustrate our data collection process in detail.

The data cleaning phase is crucial and continues throughout the training process, ensuring the quality and relevance of the data used. The goal is to remove irrelevant, inconsistent, or erroneous data while preserving the core information that enhances the performance of the model for the wide application scenarios. Here’s a breakdown of the most critical aspects.

Data augmentation plays a critical role in improving the robustness of our training datasets. We employ several strategies to diversify the input and enhance the model’s ability to handle varied data scenarios. The key augmentation methods are as follows:

• •

  diverse table serialization input formats: We utilize multiple formats for serializing table input data, including html, xml, df.markdown(), df.to_string(), json, and various custom table serialization methods. These are randomly combined with table field descriptions, schema information, and value enumeration data, resulting in more than 20 different ways of constructing the input table-info.

• •

  query augmentation: In BI-related queries, a significant number of queries tend to be ambiguous. To ensure that our model handles query ambiguity effectively, we aim to avoid direct mapping between fields in the query and table names or field names. Specifically, we ensure that quoted field or table names are removed from queries. Experimental results show that this method significantly reduces instances of key-errors in TableGPT2, especially in realistic BI cases.

• •

  table data augmentation: We anonymize table fields and categorical data values, enhancing the TableGPT2’s robustness when handling dirty or noisy table data.

• •

  single-turn and multi-turn QA pair combination augmentation: We combine single-turn and multi-turn QA data using different prompt formats and output structures. This reduces TableGPT2’s sensitivity to specific system or user instructions, ensuring better generalization.

• •

  post-processing data augmentation: During the data generation process, we randomly add details such as chart plotting instructions or summary explanations to create diverse outputs and further augment the dataset.

Our supervised fine-tuning (SFT) process is generally divided into two primary scenarios. The first scenario involves standard SFT with input-output data, which covers most general-purpose tasks that do not involve structured tables. These tasks typically involve common textual inputs and outputs, where the focus is on aligning user queries with generated responses. The details of these general types of datasets can be found in Table 3.

The second scenario deals with table-related data, where we combine the decoder and encoder using a query-based transformer mechanism inspired by Q-Former [24]. This mechanism is specifically trained on table-query-answer tuples, which allows the model to process structured data efficiently. In this setup, the encoder processes table information, generating embeddings that are then combined with the query, aiding the decoder in producing accurate results.

Noticed that the encoder are first pretrained in a self-supervised manner prior to the SFT process, focusing on tasks such as schema inference, which helps the model understand general table semantics more effectively. While these two SFT scenarios (general and table-related) are distinct in their approach, they are combined during the training process. This combined approach allows the model to handle both free-form text queries and structured table data, making it flexible and adaptable across various BI tasks while keeping the general abilities in TableGPT2. By dynamically switching between these two modes, the model can effectively process both unstructured and structured information depending on the task at hand.

Finally, we experimented with several training configurations to optimize the SFT process. In some cases, we froze the decoder and only trained the encoder and projection layers. In other setups, we trained the entire model end-to-end, which allowed us to learn simultaneously from both general and table-related data. These different training configurations provided valuable insights into how to best balance the model’s training between its encoder and decoder components.

The prompt engineering primarily involves the design of effective system instructions and the integration of RAG infrastructure.

Enterprise-level data warehouses often contain petabytes (PB) of data and complex schemas with thousands of tables. Even though traditional large language models (LLMs) possess large context-handling capabilities, they are still unable to process such vast amounts of data at once. Inputting excessive context into the model can lead to distracted attention, significantly affecting both the accuracy and the response speed of the model. Therefore, RAG technology [25] plays a crucial role in such scenarios.

When dealing with large datasets and complex database schemas, our RAG engine first retrieves enterprise-specific data (such as database schemas and business documents) and selects highly relevant contextual information for the current analysis task through two stages: embedding and reranking. This avoids unnecessary information overload. The agent then generates code or interprets the data based on this refined context, focusing on the most important data and analysis goals within the limited context window.

In data analysis tasks, RAG is applied in two primary scenarios:

• •

  data schema retrieval and context aggregation: when facing multiple tables or complex database structures, the agent uses RAG to retrieve table structures, field information, and relationship dependencies that are relevant to the current analysis task, and then generates the appropriate code. This approach effectively reduces unnecessary context loading, allowing the agent to quickly pinpoint data sources and generate more accurate code.

• •

  domain-specific knowledge retrieval and analysis: in scenarios where the analysis involves specific business contexts (e.g., interpreting enterprise documents or historical analysis reports), RAG retrieves relevant document fragments and merges them with the current task context. This enables the agent to leverage domain-specific knowledge, improving the professionalism and credibility of its responses.

By integrating RAG technology, the agent can intelligently understand context when faced with complex data warehouses or multi-dimensional business needs. This allows it to generate analysis code or interpret data in a way that aligns with business requirements, significantly improving its usefulness and reliability in enterprise-level data environments.

In the agent workflow, the code generated by the agent needs to be executed in a safe, isolated, and controlled environment. To ensure that the execution of the code does not pose a security threat to the main system and can be quickly isolated and recovered in case of an error, we designed and implemented a specialized code sandbox environment. The code sandbox has the following core design objectives:

• •

  security: prevent unauthorized operations on the host system or other critical services by the generated code.

• •

  isolation: ensure that each code execution environment is independent and does not share any system resources or files.

• •

  easy recovery: in case of errors or improper behavior, the environment can be quickly destroyed and recreated without affecting other running tasks.

We adopt a containerized approach to package the iPython executor as a code sandbox, using Kubernetes as the management platform for these sandboxes. Each piece of code generated by the agent is executed within an independent sandbox, providing the following safeguards:

• •

  isolated execution environment: each container has its own isolated file system, process space, and network space, ensuring no direct impact on the host system.

• •

  resource quota control: Kubernetes’ resource quota functionality (such as CPU and memory limits) is used to prevent any single task from consuming excessive resources.

• •

  timeout execution strategy: all code execution tasks are set with a maximum execution time (timeout), after which the environment is automatically terminated and destroyed.

The data analysis tasks typically consist of multiple steps, with potential dependencies between each step. Therefore, the lifecycle of a code sandbox must cover the entire duration of a complete data analysis task. However, enterprise-level data analysis tools often handle large numbers of independent analysis tasks, each executed in its own sandbox, which can consume significant cluster resources. To ensure efficient utilization of resources, we designed an automatic destruction mechanism for the code sandbox. When a sandbox has been idle for an extended period, the container is automatically destroyed to free up resources.

We have made the entire agent workflow open-source on our Github repository. See Section 2.

In Table 5, we present the primary comparisons of TableGPT2 (in both 7B and 72B versions) against the most advanced LLM competitors discussed above.

Notably, without extensive training on any specific benchmark training set, TableGPT2 achieves significantly better results than nearly all other LLM-based methods. Empirically, on certain benchmarks, TableGPT2 even reaches results superior or comparable to those of GPT-4o.

Moreover, on complex data benchmarks such as HiTab, which involve hierarchically structured tables, most current LLMs show poor performance. TableGPT2, however, demonstrates a substantial improvement, achieving over a 60% absolute increase in execution accuracy over Qwen2.5 family.

In addition, we present our results on the RealTabBench dataset, in Table 6.

For qualitative illustration, we pick a few comparative cases in Figure 4, covering scenarios involving irregular tables, general tables, and ambiguous tables.

As mentioned, tuning an LLM for table-related tasks should not compromise its overall capabilities. To verify this, we conducted evaluations on popular benchmarks, including MBPP [67], HumanEval [66], CMMLU [68], and MMLU [69], which measure general coding and language comprehension skills (see Table 7). Unlike some other approaches highlighted in  [46], TableGPT2 maintains strong performance on these benchmarks, demonstrating no degradation in general competency.

While we utilized Python and SQL data for fine-tuning TableGPT2, specific domains often require specialized coding practices for security and efficiency reasons. For example, some industries employ pseudo-SQL or domain-specific languages (DSLs) designed to limit access or adhere to strict compliance standards. In such cases, it becomes essential to build interpreters dedicated to these specific domains, ensuring that the generated code integrates seamlessly into existing data infrastructures.

A key challenge here is enabling LLMs to quickly adapt to enterprise-specific DSLs or pseudo-code. Although LLMs like TableGPT2 can generate code, the question remains: how can we efficiently bridge the gap between LLM-generated code and the specific requirements of enterprise data infrastructures? One possible solution lies in the development of encapsulated programming languages, such as Logic-LM [70], which provide a structured framework that can be tailored to specific use cases.

In previous version of TableGPT2 [18], we adopted a mixed output approach that combines both structured DSL output and standard programming code. The generation of this hybrid output was guided by prompt templates, and further reinforced during the supervised fine-tuning (SFT) process. This allowed the model to fluidly produce both structured and unstructured code, offering flexibility while maintaining the structure needed for domain-specific applications.

These domain-specific languages offer several benefits, such as better interpretability, and allowing users to interact more directly with the LLM’s output through a user-friendly interface. Additionally, these languages can lead to safer and more robust solutions by minimizing potential security risks and errors. However, this encapsulation can limit the flexibility of the code, as it is constrained within predefined structures.

In conclusion, coding in production environments goes beyond simple code generation. It requires careful consideration of domain-specific needs, infrastructure compatibility, and the ability to strike a balance between flexibility and safety, especially when using mixed approaches of DSL and general-purpose code.

Although TableGPT2 achieves state-of-the-art performance in table-related tasks, we still cannot yet expect a single end-to-end LLM to fully solve complex, real-world problems independently. Recently, we have been closely following a new line of research focused on automated agency design [71], which builds on the principles to automate the orchestration of LLM workflows. In this approach, several LLMs are organized into a directed acyclic graph (DAG) structure, such that the input queries are automatically routed through a sequence of LLMs by the topological order of the graph. Each LLM performs a specialized functionality. The flow through the DAG is determined by the system itself, making decisions about which LLMs to involve based on the task at hand. These models may vary in their system prompt templates, retrieval-augmented generation (RAG) configurations, in-context learning (ICL) examples, and others. This automated flow engineering creates a flexible, modular pipeline that adjusts dynamically based on the problem’s requirements, much like how AutoML systems automatically configure machine learning models for optimal performance.

For example, in our small app designed for equity and fund recommendations via natural language, we needed to connect the LLM with real-time market data. In this multi-agent architecture, we generally assign distinct roles to different LLMs, each fine-tuned (SFT-ed) on data specifically tailored to its function. One LLM is dedicated to precise intent recognition, trained with datasets focused on understanding user queries and intents. Another LLM is specialized in code generation, data interaction, and tool invocation. A third LLM handles domain-specific and in-depth analysis, fine-tuned on industry-specific datasets to ensure expertise in the relevant field. Each LLM is further configured with distinct prompt templates and retrieval-augmented generation (RAG) setups for the input, while varied coding or conversation logic is applied to the output stage. This tailored fine-tuning at each stage ensures that the overall pipeline delivers precise, accurate, and context-aware responses, addressing the complex nature of real-world applications.

Indeed, while there is ongoing speculation that a single foundation model with sufficiently advanced capabilities might eventually replace the need for chaining multiple models, this remains largely theoretical. Such a model would possess enough general-purpose intelligence to handle various tasks within a unified framework. However, based on our experience with specialized projects, we typically require more than two LLMs to address the full complexity of real-world applications. Just as MetaGPT [72] suggests, unified models are promising in their design, but it is evident that for now, one LLM is not sufficient to manage the intricacies of complex workflows. Much like the Qwen and LLaMA model families, where specialized models are developed for tasks such as mathematics or coding, it remains uncertain when a single model can seamlessly solve problems across multiple domains at a high level of proficiency, especially towards production.

While TableGPT2 primarily focuses on business intelligence (BI) applications, where databases or data warehouses serve as the upstream sources of structured data, another common and unneglectable form of tabular data originates from apps like Apple Pages or Microsoft Excel. These types of tables differ significantly from those in data infrastructures because they often exhibit irregularities. For example, tables in Excel or Pages frequently have merged cells, inconsistent row or column structures, and non-standard data formatting, making them more complex to process. These tables can vary widely in organization, where cells might contain free-form text, be partially filled, or use multi-level headers, making them far less uniform than typical database tables.

In the agent workflow where TableGPT2 model resides, we fine-tuned a separate LLM specifically for normalizing irregular tables and integrated it into a holistic system. However, handling these irregular tables still leaves considerable room for improvement, especially given the substantial commercial potential for production use.

Thus, we may hypothesize that addressing such irregularities should begin during the pretraining stage to ensure the model becomes adept at handling the wide array of formats that tables can take. Many current LLMs, along with retrieval-augmented generation (RAG) processes, do not adequately process or handle these non-standard table structures. Moreover, many existing pretraining corpora tend to overlook this type of data [73]. This gap presents a valuable opportunity for future research to enhance models capable of effectively processing versatile structured data formats, particularly those widely used, such as Excel and Pages.