Improved Program Repair Methods using Refactoring with GPT Models

Teachers often utilize automatic program repair methods to provide feedback on submitted student code using model answer code. A state-of-the-art tool is Refactory, which achieves a high repair success rate and small patch size (less code repair) by refactoring code to expand the variety of correct code samples that can be referenced. However, Refactory has two major limitations. First, it cannot fix code with syntax errors. Second, it has difficulty fixing code when there are few correct submissions. Herein we propose a new method that combines Refactory and OpenAI's GPT models to address these issues and conduct a performance measurement experiment. The experiment uses a dataset consisting of 5 programming assignment problems and almost 1,800 real-life incorrect Python program submissions from 361 students for an introductory programming course at a large public university. The proposed method improves the repair success rate by 1-21% when the set of correct code samples is sufficient and the patch size is smaller than Refactory alone in 16-45% of the cases. When there was no set of correct code samples at all (only the model answer code was used as a reference for repair), method improves the repair success rate by 1-43% and the patch size is smaller than Refactory alone in 42-68% of the cases.