-
Background The paper discusses the critical design decisions in constructing Vision-Language Models (VLMs) and highlights how unsupported decisions impede progress in the field.
-
Existing Work The existing diverse design choices in VLMs are often not experimentally justified or only minimally so, which makes it difficult to identify which decisions genuinely enhance model performance.
- Created an efficient visionary language model, Idefics2, based on extensive experimental research
-
Challenge 1: Impact of model architecture choice on inference efficiency The study examined how different connector modules that fuse vision and text modalities affect the model's inference efficiency. By comparing different design choices in a controlled environment, it was found that the fully autoregressive architecture performs better than the cross-attention architecture while ensuring stable training.
-
Challenge 2: Impact of multimodal training procedures on training stability By modifying image processing methods to balance inference cost and model performance and adapting the pre-trained vision backbone and the connecting modules to efficiently handle images in their original ratio and size without compromising downstream performance, the study addresses this issue.
-
Idefics2, as a foundational VLM with 8 billion parameters, achieves state-of-the-art performance across various benchmarks and is more efficient in terms of inference. Its performance on some vision-language benchmarks is comparable to state-of-the-art models four times its size, and it matches the performance of Gemini 1.5 Pro on some challenging benchmarks. Moreover, the base, instructed, and chat versions of Idefics2 were released alongside the datasets created for training the model.
This paper thoroughly investigates the critical design choices impacting VLMs' performance through extensive experiments, introduces the efficient foundational VLM Idefics2, and proves its superior performance in multiple standard tests.