-
Background Existing Multimodal Large Language Models (MLLMs) for document understanding mainly focus on a global understanding of document structure. However, they face challenges in zero-shot document understanding as they require fine-tuning with annotated data for downstream tasks.
-
Existing Work Many pre-training tasks designed by previous research enhanced the model's understanding of document layouts. Still, they were limited by focusing only on global document understanding and didn't make full use of local features in the document, such as tables, headlines, and images. Additionally, these approaches lacked explicit learning of layout information during supervised fine-tuning, essential for document understanding.
- Introduced LayoutLLM with a layout instruction tuning strategy
-
Challenge 1: Enhanced understanding of document layouts The researchers propose to learn document layouts from a global to a local perspective through different levels of pre-training tasks. All these tasks are implemented via instruction tuning, allowing LayoutLLM to understand and utilize the document structure comprehensively.
-
Challenge 2: Integration of layout information into fine-tuning A novel LayoutCoT (Chain-of-Thought) strategy is introduced that explicitly incorporates layout information into supervised fine-tuning. This enables LayoutLLM to focus on relevant areas in the document and leverage their features to generate accurate answers, offering a degree of interpretability in the answer generation process.
-
The newly developed LayoutLLM model and layout instruction tuning strategy are specifically designed to enhance the comprehension and utilization of document layouts. In implementation, the researchers use three groups of pre-training tasks that learn document layouts from global to local levels and employ the LayoutCoT strategy for supervised fine-tuning. In experiments on zero-shot document understanding, LayoutLLM demonstrated superior performance over other open-source LLMs and MLLMs on benchmarks such as document visual question answering and visual information extraction.
The paper successfully proposes the LayoutLLM model and its layout instruction tuning strategy, significantly improving the model's understanding and utilization of document layouts, especially demonstrating outstanding performance in zero-shot document understanding tasks.