- Background
The paper describes one-on-one tutoring as an effective instructional method, particularly when it comes to improving student learning outcomes. However, the high demand for qualified tutors is a challenge that often requires the training of novice tutors to ensure effective tutoring. It is noted that timely explanatory feedback can facilitate the trainee's training process but is challenged by the time-consuming nature of human experts assessing trainees' performance. - Existing Work The quality of automated explanatory feedback is currently lacking, with many systems unable to provide learners with accurate feedback. The paper highlights the fact that providing professional-level personalized explanatory feedback requires significant time and effort investment. The need to scale such training to meet the increasing demand in educational settings poses a considerable challenge. However, breakthroughs in large language models (LLMs) present a possible way to simplify this process. Models like the Generative Pre-trained Transformer (GPT) could potentially automate the generation of personalized, real-time feedback for tutors, which could mitigate resource burdens and improve feedback specificity and precision by accurately identifying tutors' personalized needs.
- Introduced an explanatory feedback system using GPT-4 for trainees
- Challenge 1: Accurately identifying incorrect trainee feedback The paper poses a question regarding whether a large language model can accurately identify trainees' incorrect responses when they fail to effectively guide students in specific training scenarios. With zero-shot and few-shot learning approaches, a binary classifier was developed to judge tutors' correct and incorrect responses. The results showed that the five-shot learning approach achieved acceptable performance in identifying incorrect responses.
- Challenge 2: Enhancing the effectiveness of trainees' responses in specific training scenarios The second question addresses whether GPT-4 can be utilized to enhance the effectiveness of trainees' responses in specific training scenarios. After identifying incorrect responses, the paper explored the idea of rephrasing these incorrect responses using GPT-4 to determine if it is possible to ensure the rephrased feedback is accurate with minimal changes from the original incorrect feedback. Combining the results from the first and second questions, a feedback system was built to provide explanatory feedback for the incorrect trainee responses.
The study found that using a few-shot approach, the GPT-4 model was able to effectively identify correct/incorrect trainees' responses from three training lessons with an average F1 score of 0.84 and an AUC score of 0.85. Additionally, using the few-shot approach, the GPT-4 model adeptly rephrased incorrect trainees' responses into desired responses, achieving performance comparable to human experts. The work involved providing suitable prompts to GPT-4 to optimize feedback and ultimately build a feedback system to offer explanatory feedback to trainees with incorrect responses.
The paper investigates the construction of an automated feedback system using GPT-4 to assist in the training of tutors in one-on-one classes, aiming to alleviate the resource burden of traditional personalized instructional feedback and provide high-quality and specific feedback. It falls under the category of knowledge retrieval and evaluation research.