A preprint describes a code-training pipeline that reported the strongest listed results across the benchmarks it tested. Called RobustTests, it combines test cases generated from deliberately flawed programs with a step-by-step reward based on pass rates. Its reported LiveCodeBench Score was 68.39. On CodeForces, it recorded a Score of 38.50, a Rating of 85.99 and a Percentile of 94.67. The paper also reports an approximately absolute 3% advantage over baselines on both benchmarks, although it does not specify the exact comparator or calculation behind that figure.
Those numbers are benchmark comparisons, not proof that the recipe itself caused the difference. The same Qwen3-32B model was used across the compared methods, and each method was capped at approximately 40 test cases per problem. Baseline reinforcement-learning runs received binary sparse rewards of 0 or 1, while RobustTests used the denser pass-rate signal.
How the test sets are built
RobustTests begins by generating near-correct programs that fail on some inputs. Fully correct and completely incorrect candidates are excluded, and candidates with identical execution vectors are deduplicated. In plain terms, the process keeps different failure patterns rather than a pile of programs that behave the same way. The proposal then combines that faulty-code step with test filtering, diversity-driven selection and reinforcement learning from verifiable rewards, or RLVR, where test outcomes provide the training signal.
Candidate tests pass through three checks: input verification, validation that the LLM followed the instructions for the test, and diversity-driven selection. The diversity step uses execution behavior from the faulty programs to choose representatives. In the paper's case study, two refinement stages produced a final suite of 248 tests, after which selection retained 40 representative cases.
For training, the researchers filtered CodeContests+ with Qwen3-32B over ten trials per problem. They retained problems whose pass@10, a pass-rate measure calculated over ten trials per problem, fell between 0.2 and 0.9. That left approximately 3.3k of the dataset's 11,636 programming problems, and the refined RobustTests set is described as containing approximately 200 diverse test cases.
What the benchmark comparison found
In the main comparison, RobustTests had the highest listed values. It scored 68.39 on LiveCodeBench. On CodeForces, it scored 38.50, with a Rating of 85.99 and a Percentile of 94.67. The paper reports an approximately absolute 3% advantage over the baseline methods on both benchmarks. Since the exact comparator and calculation are not reported, that percentage carries more uncertainty than the displayed scores.
The component comparisons pointed in the same direction, but they are still comparisons between configurations. The integrated RobustTests* setup scored 67.84 on LiveCodeBench, versus 66.91 without test-case synthesis, 66.82 without diversity-driven selection and 65.41 without both modules. The integrated configuration had the highest listed score in that set.
The reward comparison also favored the denser signal in the reported results. On CodeContests+ training, the sparse-reward score was 65.41, compared with 66.30 for dense reward. For RobustTests*, the scores were 67.84 with sparse reward and 68.39 with dense reward. Without the validator, the corresponding scores were 66.02 and 67.21. Dense reward was higher in all three pairs.
The paper also reports a positive association between test diversity and downstream performance using Test case Space Polarization, or TSP. RobustTests had the highest TSP and the best LiveCodeBench performance among the compared methods, while methods with lower TSP had weaker downstream performance. The paper describes this as a positive correlation; the comparison does not establish that TSP itself caused the score differences.
Validation leaves a warning
The audit of the generated tests adds a significant qualification. Across approximately 3.3k training problems, each with around 200 raw generated tests, approximately 30% were rejected as invalid. About 10% of the accepted tests still remained invalid, and false-negative judgments were reported for roughly 10% of all tests. No reference-execution failures were observed, but the validator did not eliminate every problematic case.
That audit matters because RobustTests uses test outcomes as its reward signal. The reported advantage therefore comes with uncertainty about how residual invalid tests affected the comparison. The supplied results also give no variance estimates for the benchmark figures, so readers cannot tell from the report how much run-to-run variation lies behind the displayed differences.
Taken together, the findings support a narrower conclusion: in these Qwen3-32B reinforcement-learning experiments on competitive-programming data, the RobustTests configuration had the strongest listed benchmark results and the highest TSP among the compared methods. They do not establish that the pipeline caused the gains or that validation removed all invalid synthetic tests. The document is an arXiv version 1 preprint dated 25 Aug 2026 and reports support from the Ant Group Research Intern Program.
Paper data and sources
Original title: Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping
Authors: Yiwen Zhang, Xiaodong Yan, Zhenyu Huang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text