-
Background The paper discusses that text embeddings, as vector representations of natural language, play a significant role in various NLP tasks like information retrieval, question answering, etc. Existing BERT-based methods that fine-tune word embeddings on NLI datasets, although efficient, fail to capture the richness of natural language contexts. State-of-the-art methods like E5 and BGE use a more complex multistage training paradigm involving pre-training on billions of weakly-supervised text pairs and subsequent fine-tuning on several labeled datasets.
-
Existing Work Current multistage training methods maintain several drawbacks, including complex training pipelines that require extensive engineering effort to manage large relevance pairs, and the use of manually-collected datasets, which suffer from limited diversity in tasks and language coverage.
- Introduced a new text embedding method using LLM-generated synthetic data
-
Challenge 1: Simplifying the training process and reducing dependence on manually-annotated datasets The proposed method does not rely on complex training pipelines or manually curated datasets but instead employs proprietary LLMs to generate synthetic data for millions of text embedding tasks. This approach allows for competitive performance on leading text embedding benchmarks with less than 1,000 training steps and no labeled data.
-
Challenge 2: Enhancing diversity across languages and task types The research utilizes a two-step prompting strategy: first brainstorming a candidate task pool with LLMs, then generating data conditioned on a given task, thereby covering a wide range of application scenarios and increasing data diversity through multiple prompt templates.
-
The paper illustrates that the Mistral-7B model, fine-tuned only with synthetic data, achieves competitive performance on the BEIR and MTEB benchmarks. When further fine-tuned with a mix of synthetic and labeled data, the model sets a new benchmark record by surpassing previous methods by 2%. Moreover, it effectively performs personalized passkey retrieval for inputs up to 32k tokens by beyond the conventional 512 token context limit, showing outstanding multilingual performance, especially with high-resource languages.
The paper introduces an innovative text embedding approach utilizing the latest LLMs and synthetic data to match performance on competitive benchmarks with fewer than 1,000 training steps and no label data, offering strong evidence for further advancements in text embedding technology.