Skip to content

Latest commit

 

History

History
20 lines (15 loc) · 2.53 KB

2403.05217.md

File metadata and controls

20 lines (15 loc) · 2.53 KB
Raw

Background

  • Background The paper addresses the spotlight research area of Open-domain question answering (ODQA). Existing methods primarily collect evidence through two paradigms: (1) The retrieve-then-read uses an external corpus to obtain relevant documents and (2) The generate-then-read approach utilizes large language models (LLMs) to generate relevant documents. However, neither can fully address the multi-faceted requirements for evidence.

  • Existing Work Current paradigms cannot fully satisfy the varied and comprehensive requirements for evidence; therefore, an integrative solution is needed that offers factual reliability and expands the diversity of evidence.

Core Contributions

  • Introduced LLMQA, a generalized framework

    • Challenge 1: Multi-faceted evidence requirements Current methods do not entirely meet the broad evidence demands of ODQA. LLMQA addresses this by combining the retrieval and generative paradigms, classifying the ODQA process into three basic steps: query expansion, document selection, and answer generation, thus taking advantage of both paradigms to improve evidence gathering.

    • Challenge 2: Guiding LLMs to fulfill multiple roles within the framework LLMs exhibit their capabilities in various tasks. The paper enables LLMs to act as generators, rerankers, and evaluators within the proposed framework. They work together in the ODQA process to enhance overall performance. Additionally, a novel prompt optimization algorithm is introduced to further guide LLMs in producing high-quality evidence and answers.

Implementation and Deployment

Experiments on well-known ODQA benchmarks such as NQ, WebQ, and TriviaQA show that LLMQA achieves the best performance in terms of answer accuracy and evidence quality, underscoring its potential to advance ODQA research and applications. LLMQA outpaces the baselines with significant improvements in EM scores (4.0@TriviaQA, 2.7@WebQ, 3.1@NQ), proving the efficacy of multi-role LLMs for ODQA. The query expansion generator role reaches recall scores of 73%-87% for the answers in generated expansions, while the reranker role increases answer coverage by about 8.1%.

Summary

LLMQA is a novel generalized framework that combines strengths of retrieval- and generation-based evidence collection. By enabling LLMs to take on multiple roles within the framework, the paper significantly improves the overall performance of ODQA systems, with experimental results demonstrating its effectiveness over existing methods.