-
Background Large language models (LLMs) are effective across various tasks, such as response generation, logical reasoning, and program synthesis. Their use as planning agents is an active research area. Good planning involves selecting the right tools to complete sub-tasks, and while LLMs offer excellent generation capabilities, they have limitations, including a lack of up-to-date information and a tendency to hallucinate tools.
-
Existing Work Current research in using retrieval to integrate tools into LLM planning is fresh, with a focus on refining retrieval methods for better handling of under-specified or ambiguous queries. However, these works primarily handle queries that are already well-defined, and CoT mechanisms to augment queries with context may lead to overly lengthy inputs, which are problematic due to the limited context window size of LLMs.
- Introduced a new approach
-
Challenge 1: Improving the retrieval step The challenge lies in the failure of semantic search methods when queries are incomplete or lack context. Context Tuning for RAG is introduced to address this by using a smart context retrieval system that retrieves information to improve tool retrieval and plan generation. The lightweight context retrieval model uses numerical, categorical, and habitual usage signals to retrieve and rank context items.
-
Challenge 2: Reducing hallucination errors After the correct tool retrieval, incorporation of relevant context into plan generation helps reduce hallucination errors and increase accuracy. The paper uses a lightweight LambdaMART-based RRF method to tackle this, which outperforms GPT-4 based retrieval and offers improvement without needing to augment the query.
-
The planner was fine-tuned using OpenLLaMA-v2-7B, and performance was assessed using AST matching strategy to measure plan accuracy. Empirical results showed that context tuning significantly enhanced the performance of context retrieval with a 3.5-fold improvement in Recall@K, and a 1.5-fold improvement in tool retrieval tasks, resulting in an 11.6% increase in LLM-based planner accuracy.
The paper presents context tuning as a novel component that enhances RAG-based planning, enabling it to effectively handle incomplete or under-specified queries and reduce hallucinations. It systematically compares various retrieval methods in lightweight models and LLMs, showcasing the effectiveness of context tuning in improving contextual understanding.