-
Background
The article introduces Adapters, an open-source library that unifies parameter-efficient and modular transfer learning methods for large pre-trained language models (LLMs). Adapters simplifies the process of using and configuring these transfer learning methods for researchers and practitioners by providing a unified interface. The literature points out that the growing size of LLMs has made the traditional paradigm of fine-tuning all model parameters on a downstream task extremely expensive, and further challenges include negative transfer, a lack of positive transfer, catastrophic forgetting, and poor generalization. -
Existing Work Past studies tried addressing these problems by designing smaller components called adapters for fine-tuning language models on labeled task data. These adapters are known for their parameter efficiency and modularity, manifested in the initial release of AdapterHub. However, as interest in adapter methods grew, existing efforts focused on parameter efficiency overlooked the modularity aspect. Thus, the paper proposes a new library, Adapters, which not only explores parameter efficiency but also strengthens the feature of modularity.
- Introduced a unified Adapter library
-
Challenge 1: Parameter efficiency Existing works or libraries like OpenDelta and HuggingFace's PEFT have achieved certain successes in addressing the parameter efficiency issue, but the Adapters library builds on this to enhance the flexibility of defining complex adapter setups and compound structures.
-
Challenge 2: Modularity In existing works, particularly in the first generation of AdapterHub, modular design choices in transfer learning have not been fully developed. Hence, Adapters expands both "horizontally" and "vertically," not only supporting more pre-trained neural structures but also adding new composition and processing capabilities, enabling rapid deployment and adapter capabilities of modularity.
-
The Adapters library offers tight integration with the widely used HuggingFace Transformers library, enabling adaptation of pre-existing fine-tuning scripts with minimal code changes. It supports a variety of model architectures and provides a unified adapter interface that covers the entire lifecycle associated with working with adapters, including adding, attaching, activating, saving, loading, aggregating, and deleting adapter modules. The authors validate the effectiveness of Adapters through experimental comparisons with traditional full fine-tuning on various NLP tasks, proving its efficacy in parameter-efficient and modular transfer learning.
The paper proposes a unified library—Adapters—that integrates and extends parameter-efficient and modular transfer learning methods. It achieves close integration with the Transformers library and demonstrates its effectiveness through comparative experiments on several NLP tasks.