-
Background The article introduces the importance of examining the information encoded in the hidden representations of large language models (LLMs) to explain the behavior of models and to verify their alignment with human values. Since LLMs are capable of generating text that humans can understand, the researchers propose to leverage the model itself to explain its internal representations in natural language.
-
Existing Work Previous studies have proposed a variety of interpretability tools, which mainly relied on three approaches: training linear classifiers called probes on top of hidden representations, projecting representations to the model's vocabulary space, and intervening in the computation to identify dependencies for certain predictions. Although these methods have been widely successful, they have practical shortcomings such as inability to inspect early layers and lack of expressiveness.
- Introduced a framework called Patchscopes
-
Challenge 1: Enhancing the expressiveness and scalability of interpretability methods Using the advanced capabilities of LLMs to produce human-like text, the Patchscopes framework can be configured to query various information from LLM representations. It achieves the decoding of specific information by "patching" a representation into a separate inference pass that extracts that information independently of the original context.
-
Challenge 2: Unifying existing interpretation techniques and extending new applications Patchscopes can act as a superset of existing interpretation methods, offering more effective solutions to the same problems while overcoming previous methods’ limitations. Additionally, it introduces new possibilities such as using a more capable model to explain the representations of a smaller model and new applications like self-correction in multi-hop reasoning.
-
The researchers conducted a series of experiments focusing on auto-regressive LLMs with Patchscopes. Initially, they considered the problem of estimating the model’s next-token prediction from its intermediate representations, demonstrating that using a few-shot token identity prompt results in significant improvements over vocabulary projection methods. Next, they evaluated how well Patchscopes can decode specific attributes of an entity when detached from the original context. Experiments show that without any training data, Patchscopes outperforms probes in six out of twelve commonsense and factual reasoning tasks and is comparable in another five. Patchscopes also tackle questions that are difficult for existing methods, such as assessing how LLMs contextualize input entity names in early layers and how a more expressive model can be used to inspect the hidden representations of a smaller model. Finally, the utility of Patchscopes for practical applications was showcased, such as correcting latent multi-hop reasoning errors when the model can conduct each reasoning step correctly but fails to process their connection in-context. Using Patchscope improved accuracy from a baseline of 19.57% to 50%.
The paper presents a framework named Patchscopes, offering a novel approach to interpret the information encoded in the hidden representations of large language models (LLMs) and to correct multi-hop reasoning errors. Patchscopes serves as a general modular framework, unifying existing interpretative tools and addressing their deficiencies, while also paving the way for new research and application opportunities.