-
Background The paper investigates the usage of large language models (LLMs) to answer agriculture exam questions, particularly focusing on enhancing the performance of models on subject-specific agricultural questions. Researchers note that despite llms' strong capabilities in many domains, they still face challenges in understanding and accurately responding to subject-specific queries, especially in agriculture. This is due to the models' lack of sufficient domain-specific knowledge and the difficulty in applying their pre-trained knowledge to particular questions which necessitates a method for improving LLCs in domain-specific performance.
-
Existing Work Existing methods like chain-of-thought and self-consistency have tried to improve LLM performance through enhanced understanding and reasoning. However, these techniques often fall short in handling complex agriculture-specific problems as they overlook the importance of effectively integrating domain knowledge into LLMs. Moreover, they do not consider how to optimize the model's performance in the absence of specific textual material as background.
- Introduced a new strategy called Ensemble Refinement (ER)
-
Challenge 1: Enhancing the model's performance on agriculture-domain questions ER overcomes the shortcomings of previous methods by incorporating techniques like chain-of-thought and self-consistency over multiple stages. In the first stage, the model generates various possible explanations and answers. In the second stage, the model synthesizes these generations to produce a more refined answer. By integrating the strengths and weaknesses of the different explanations, ER enables LLMs to generate more precise responses.
-
Challenge 2: Reducing the resource cost associated with repeated sampling ER is only implemented for multiple-choice evaluations, reducing the high resource cost that comes with repeated model sampling. This strategy effectively refines the LLMs' generations to extract the most accurate answer, culminating in a plurality vote for the final response.
-
The research evaluated the GPT-3.5 and GPT-4 models using Azure Open AI deployments and employed Llama2 models (13B and 70B parameters). Evaluations included video-based questions relying on LLMs' pre-trained knowledge and text-based questions utilizing RAG, ER, and preambles. GPT-4 consistently outperformed other models in various scenarios; especially with the introduction of preambles, its performance increased from 79% without ER to 83% with ER. For text-based questions with given context, GPT-4's performance using ER went up to 84%, and further up to 93% when employing RAG technique.
This study presents a new approach in employing LLMs for answering questions in the field of agriculture, significantly enhancing LLMs' performance on multiple-choice questions through the Ensemble Refinement strategy, showing the broad potential in handling domain-specific problems.