Skip to content

Latest commit

 

History

History
20 lines (15 loc) · 2.59 KB

2312.07476.md

File metadata and controls

20 lines (15 loc) · 2.59 KB
Raw

Background

  • Background The paper discusses how Large Language Models (LLMs) use a few examples for In-Context Learning (ICL). However, the restricted number of examples leads to demonstration bias, meaning that the input-label mapping deduced by LLMs misinterprets the task's essence. Inspired by human experience, the authors seek to alleviate this bias from the perspective of the inter-demonstration relationship.

  • Existing Work There is currently insufficient understanding of how ICL operates. Although prior research has attempted to explore the ICL mechanism from different angles (such as the importance of input-label format, the role of label space, the impact of example distribution), the potential impact of the inter-demonstration relationship in ICL has not been widely discussed.

Core Contributions

  • Introduced Comparable Demonstrations (CDs)

    • Challenge 1: Demonstration Bias Refers to the bias in ICL where due to limited examples, LLMs may induce input-label mappings that do not conform to the essence of the task. The paper reduces bias by constructing CDs—minimally editing texts to flip the corresponding labels—strengthening inter-demonstration comparisons, emphasizing the task's essence, and minimizing potential spurious correlations.

    • Challenge 2: The Cost of Manual Annotation Although ICL requires relatively few examples, manual annotation is still costly, and automatic generation of CDs is currently not feasible. Moreover, CDs currently only consider one-to-one relationships between demonstrations, not fully exploiting the potential of many-to-many relationships.

Implementation and Deployment

A series of experiments demonstrated the effectiveness of CDs, especially in the Out-Of-Distribution (OOD) scenario, where CDs showed performance improvements. The results show that in tasks of numerical reasoning, CDs are more robust compared to other strategies based on semantic similarity. The conclusion mentions that future work will explore ICL in more complex scenarios (e.g., mathematical reasoning, multi-hop inference) and rigorously analyze the ICL mechanisms from the perspective of the inter-demonstration relationship.

Summary

The study explores ICL from the perspective of the inter-demonstration relationship, proposing the minimally edited text construction of Comparable Demonstrations (CDs) to alleviate potential demonstration bias. The experiments confirm the performance gains of CDs in OOD scenarios, emphasizing their particular necessity in simpler tasks and demonstrating their robustness with respect to the number of examples.