-
Background This paper discusses the recent advancements in Large Language Models (LLMs), focusing on how increasing the model size enhances the model's language understanding capabilities. Simultaneously, researchers are trying to further augment text-based LLMs by incorporating additional modalities such as images and videos, leading to the development of multi-modal LLMs (MM-LLMs).
-
Existing Work The main progress in the research field is In-Context Learning (ICL), which refers to the ability to learn and predict based only on a few examples in the context without updating any parameters of LLMs. Although ICL can bring significant performance improvements to MM-LLMs, its improvements still lag behind fine-tuning for downstream tasks.
- Introduced a new paradigm called Multi-Modal In-Context Tuning (MMICT)
-
Challenge 1: Combining ICL and fine-tuning to enhance MM-LLM performance Existing methods show that while ICL does increase model performance, there is still a significant gap compared to fine-tuning on downstream tasks. Seeing this, researchers proposed the possibility of combining these two learning paradigms to achieve further enhancements in fine-tuning performance on downstream multi-modal tasks.
-
Challenge 2: Designing a single module that captures various multi-modal features according to different inputs and objectives The paper presents a unified module named "Multi-Modal Hub" (M-Hub), an innovation that captures multiple multi-modal features according to different inputs and objectives. This module allows MM-LLMs to learn from in-context visually-guided textual features and subsequently generate outputs based on text-guided visual features. The flexibility of M-Hub also allowed researchers to design various context demonstrations.
-
Extensive experiments show that MMICT significantly outperforms traditional fine-tuning strategies and the vanilla ICT method, which directly inputs a mixture of different modalities' information. MMICT improves fine-tuning by learning from the visually-guided textual features of the examples, further predicting textual labels under visual inputs and textual instructions. Through the exploration of various demonstration formats, the research discovered several intriguing key findings, providing direction for future research in this field.
The work introduces a new paradigm with MMICT to showcase the use of in-context learning capabilities to enhance fine-tuning performance on large multi-modal language models. By designing the versatile M-Hub module and conducting various context demonstration experiments, the study reveals the potential of in-context learning to improve performance on multi-modal tasks.