-
Background
This paper addresses the capability of LLMs in interpreting and reasoning over tabular data, an area that has not been fully explored yet. While LLMs have shown a strong ability to understand rich textual content, their capability to interpret and reason about tabular structures remains a challenge. -
Existing Work Current LLMs show limitations in interpreting and reasoning over tabular data, mainly due to the structural nature of tables. LLMs are inherently designed for parsing and processing large amounts of unstructured text, and they struggle with tables that possess significant structural and relational information. Tasks like precise localization and complex statistical analyses, as well as the added complexity of numerical reasoning and aggregation, pose extra challenges.
- Introduced a method for table structure normalization
-
Challenge 1: Robustness against structural variations of tables There is a notable performance decline in LLMs when dealing with tables with structural variations, especially in symbolic reasoning tasks. The proposed method for table structure normalization aims to enhance the robustness of LLMs when encountering structural changes in tables.
-
Challenge 2: Comparison between textual and symbolic reasoning Textual reasoning slightly surpasses symbolic reasoning, and detailed error analysis shows different strengths of each strategy depending on the task at hand. By aggregating both textual and symbolic reasoning pathways, and with the aid of a mix self-consistency mechanism, the method achieved SOTA performance with 73.6% accuracy on WIKITABLEQUESTIONS, representing a significant improvement over previous table processing paradigms in LLMs.
-
Experiments were conducted on state-of-the-art LLMs such as GPT-3.5, revealing that LLMs are good at semantically interpreting tables but their capability in resisting structural variance and understanding table structures is suboptimal. Furthermore, experiments showed that textual reasoning outperforms symbolic reasoning in contexts with limited table content, challenging the conventional belief in the dominance of symbolic reasoning in other domains. The paper demonstrates that the application of a mix self-consistency mechanism, aggregating the unique advantages of textual and symbolic reasoning, can lead to SOTA performance in Table QA tasks.
The paper delves into the understanding and reasoning capabilities of LLMs over tabular data, contributing insights into the robustness of table structure, the comparison of textual versus symbolic reasoning, and the impact of aggregating multiple reasoning pathways on model performance. The proposed table structure normalization method and the mix self-consistency mechanism are instrumental in enhancing LLMs' performance in tabular data reasoning.