A small pilot of an AI tutor for introductory programming found that its large language model (LLM) grader was better at judging whether students' explanations were correct than at deciding how complete they were. Against an expert reference, the grader's F1 score, a combined measure of precision and recall, was 0.91 and its accuracy was 83.8%. Against the expert's completeness judgment, the score had an AUC of 0.67, a measure of ranking ability, and a mean absolute error of 0.45, indicating that it ranked responses better than it calibrated the raw score. The work is an arXiv preprint, version 1, dated 25 August 2026.
A tutor built around revision
The system, called ESSE, asks students to explain each highlighted code line in their own words. Students can revise an answer repeatedly, and advancement is unlocked after a correct explanation. A GPT-4o mini prompt returns three outputs: a correctness judgment, a completeness score from 0 to 1, and the concepts covered. ESSE treats a score of at least 0.5 as sufficiently complete. Its feedback is immediate and layered, covering correctness, completeness and concepts that are present or missing.
The classroom pilot involved eight consenting students in a single introductory Java CS1 section. They explained 30 code lines across four worked examples, producing 409 attempts, of which 407 received feedback. The researchers describe the deployment as a pilot, and its small, narrow sample limits what can be inferred from the observations.
To assess the grader, the researchers compared it with two human reference standards. One was an expert's judgment on 358 attempts. The other came from an independent Amazon Mechanical Turk re-rating in which 124 workers produced 1,696 ratings across 216 explanations, with uneven coverage. The analysis used precision, recall and F1 for correctness, plus AUC and mean absolute error for completeness. Mixed-effects models included a random intercept for student to account for repeated observations.
Correctness was the easier test
Against the expert, the model's precision was 0.95, recall 0.87, F1 0.91 and accuracy 83.8%. It recorded 42 false negatives and 16 false positives. McNemar's test gave p = 0.001, while kappa, a chance-adjusted agreement measure, was 0.32. The error pattern was uneven, but the main correctness measures were strong.
The crowd comparison produced a similar correctness result: precision 0.90, recall 0.95 and F1 0.93. False positives outnumbered false negatives, 19 to 9. Completeness was less settled. Against the expert, the completeness score had a point-biserial correlation of 0.39, p < 0.001, and AUC 0.67, with mean absolute error 0.45. Against the crowd, AUC was 0.76, but mean absolute error was still 0.42. In other words, the score could rank explanations to a degree, but its numerical value was not a dependable measure of absolute completeness.
Students revised, but learning remained uncertain
The student behavior data showed that revision was tied to the tutor's verdict. Students averaged 1.66 attempts per explanation, and 38% of explanations received a second attempt. Revision occurred 63% of the time after an incorrect verdict, compared with 35% after a correct verdict. Across successive attempts, completeness rose substantially, with a Wilcoxon effect size of 0.84 and p < 0.001. Changes in completeness were negatively correlated with changes in word count, with a Spearman correlation of -0.49 and p < 0.001.
A mixed-effects model also gave conceptual content a larger completeness coefficient than word-count volume: 0.50 for concept content versus 0.33 for volume, with both results marked p < 0.001. Put simply, the model associated completeness more strongly with what an explanation contained than with how many words it used. That is an association within this pilot, not evidence that the tutor caused students to produce more conceptually complete answers.
The learning test, however, was too thin for a firm conclusion. Only three of the eight students completed the posttest. Their normalized gains were 0.67, -0.25 and 0.78, a spread the authors describe as suggestive because the pilot was too small. The eight-student survey was also preliminary: six students disagreed that the feedback was irrelevant or incorrect or misleading, and six disagreed that they would use the system without feedback.
A small test with a narrow reach
Those limits shape how far the findings can travel. This was a single-section pilot with eight students; only three had valid posttest data, the expert standard came from one expert, and the evaluation used one LLM grader with one prompt configuration. The authors interpret the correctness judgments as reliable enough to power tutor feedback, while completeness remains the harder task because ranking was better than calibration. The paper therefore offers cautious evidence that students revised their explanations and that concept content was more strongly associated with completeness in the scoring model, but it does not establish general learning benefits or show that the approach will work across courses, models or prompts.
Paper data and sources
Original title: Self-Explanation Tutor for Active Study of CS1 Worked Examples
Authors: Arun-Balajiee Lekshmi-Narayanan, Mohammad Hassany, Kamil Akhuseyinoglu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text