In a preprint, combining execution logs with keystroke-level editing data produced the strongest early signal for separating Breakthrough (BT) student-exercise instances from Fully Stuck (FS) instances. At the first submission segment, the Combined model reached an AUROC of 0.674, compared with 0.575 for execution features alone, a reported gain of 0.098. AUROC, the study's main metric, indicates how well the model separates the two groups across its scores. A paired bootstrap gave the gain a 95% confidence interval of +0.078 to +0.119.
The headline result needs a practical qualifier. At k=1, the Combined model's precision was 0.214 at FS recall of 0.70. Put plainly, a setting that captured 70% of FS cases produced a flagged group in which just over one in five cases were actually FS. The model supplied a useful ranking signal in this analysis, but its alert list still included many cases outside that target group.
The study asked whether keystroke-level editing features added information beyond execution-log features alone, and which exercise stage predicted BT and FS most accurately. It compared three feature configurations—ExecOnly, CMOnly and Combined—across submission-based segments.
How the comparison was built
The comparison used the 2019-1 semester, which included 507 analyzed students. BT was defined as a student-exercise pair with at least one earlier 0% submission followed by a final 100% submission. FS meant that every submission received 0% and no correct solution appeared. The analysis identified 11,999 BT instances and 2,101 FS instances, while excluding pairs with partial credit of 33%, 50% or 67%.
Those numbers are counts of student-exercise instances, not unique students in each outcome group. The results therefore describe patterns across exercise cases rather than a simple tally of people in each group.
What the model measured
To capture when a signal appeared, the authors divided activity into segments: the intervals between consecutive submissions. The analysis ran through k=10. For BT, the final segment ending in the 100% submission was removed; all FS segments were retained, and any segment without editing events was excluded. The three input sets were ExecOnly, with four features; CMOnly, with eight editing features; and Combined, with all 12.
Random Forest and logistic regression models were trained separately for each segment and feature set. Validation used five-fold GroupKFold at the student level, keeping every record from a student in one fold so that records from the same student were not split between training and testing. The analysis pooled out-of-fold predictions to calculate AUROC and used 2,000 student resamples for paired 95% bootstrap intervals.
The signal was strongest at the start
The signal was front-loaded. The first segment had the highest AUROC for every configuration. Combined declined from 0.674 at k=1 to 0.549 later in the sequence, while CMOnly fell from 0.654 to 0.529. ExecOnly took a less steady path, moving from 0.484 at k=4 to 0.543 at k=10.
At the opening segment, editing features alone beat execution features: CMOnly recorded AUROC 0.654 versus 0.575 for ExecOnly. The bootstrap 95% interval for that difference was +0.056 to +0.104. Adding the two feature types produced the largest reported improvement, +0.098 over ExecOnly, and the analysis found reliable gains through k=4.
What the alert numbers mean
Because FS was the positive class and relatively uncommon in the primary analysis, the study also reported PR-AUC, a measure that emphasizes the balance between finding FS cases and keeping flagged cases precise. At k=1, PR-AUC was 0.281 for Combined, 0.256 for CMOnly and 0.186 for ExecOnly, against an FS-prevalence baseline of 0.151. All three were above that baseline, but the precision result shows why the ranking should not be read as a final yes-or-no judgment about individual cases.
A supplementary analysis used a broader FS-versus-rest test, rather than the primary BT-versus-FS contrast. There, Combined reached AUROC 0.839 and PR-AUC 0.224. FS prevalence was 0.060, making 0.224 equal to 3.74 times the random-classifier baseline. The stronger AUROC belongs to that different comparison and should not be mistaken for the 0.674 primary-task result.
A label with an important ambiguity
FS may include early disengagement as well as persistent struggle: 25.4% of FS instances in 2019-1 had only one submission. When those one-submission instances were excluded, the Combined AUROC at the first segment fell from 0.674 to 0.650. The change suggests that part of the measured separation depends on which kind of stuck behavior the label captures.
The model's feature rankings offer clues, but not a causal explanation. In the k=1 CMOnly analysis, the reported Gini importances were 0.149 for duration_sec, 0.139 for code_change and 0.131 for max_pause_sec. These numbers describe how the model ranked its inputs in that analysis; they do not show that duration, code changes or pauses cause students to struggle.
The preprint's status and access
The document is an arXiv version 1 item dated 26 Aug 2026. The work was supported by JSPS KAKENHI No. JP26K21197 and the Support Center for Advanced Telecommunications Technology Research.
Preprocessing and analysis scripts, library-version information and reproduction instructions are available through the stated Zenodo DOI, while access to the CodeBench dataset is subject to the provider's terms of use.
Paper data and sources
Original title: Predicting Struggling Students in CS1 Programming Using Keystroke-Level Editing Features
Authors: Yasuyo Kofune, Kazumasa Shimari, Kenichi Matsumoto
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text