A version-1 arXiv preprint reports that a gradient-based filter for AI training was associated with faster convergence and stronger scores across several benchmark settings, especially when preference data were deliberately corrupted. The method, called Dynamic Trajectory Valuation, or DTV, and its leave-one-out version, DTV-Loo, was tested in reinforcement learning, mathematical reasoning and preference optimization. The gains came with a clear cost: on AIME, training with valuation took longer than Vanilla GRPO.
A filter for the training stream
The basic idea is to judge training units while a mini-batch is being processed, using the gradients already produced by that batch. DTV and DTV-Loo use a fixed zero threshold for filtering, rather than requiring a separately chosen filtering rate or selection budget. The paper says the approach needs no fixed dataset, auxiliary valuation signal or extra rollouts. DTV-Loo goes further by removing the self-term and relying only on cross-unit alignment; the authors present that as a way to stop one harmful unit from dominating the batch evaluation.
Gains across three tests
In the PPO experiments, the filter was tested on two partially observable MiniGrid environments. The comparison used five seeds and 1,000 evaluation episodes per checkpoint, and DTV-based filtering was associated with faster convergence. DTV-Loo reached the target earlier on Empty-8x8, while both DTV variants later stayed ahead of IIF and Vanilla PPO on DoorKey-8x8.
On GSM8K, DTV had the best exact and partial accuracy in both the Clean and Mismatch-20% conditions. Under the shared training budget and five matched seeds, its exact-accuracy improvement over the shared pre-trained model was 0.0760 in Clean and 0.0690 in Mismatch-20%. DTV-Loo also improved over Vanilla GRPO.
The AIME comparison was limited to one training run per method because a single seed required more than 140 hours. DTV-Loo had the best performance across all reported measures. Its scores were 0.1333 versus 0.1063 for Pass@1, 0.4333 versus 0.3333 for Pass@16, and 0.3667 versus 0.2333 for Maj@16, compared with the shared pre-trained model.
The clearest edge came with corrupted data
A separate DPO experiment used UltraFeedback and a two-stage SFT-then-DPO pipeline. The study compared Clean, Mismatch-20% and Mismatch-40% conditions. In the two mismatch conditions, 20% or 40% of DPO training pairs were corrupted, while held-out and test data remained clean.
The main DPO comparison used five matched seeds, with run, data-shuffle and curation seeds matched across methods. DTV-Loo had the best AUC and final preference accuracy in all three conditions. Against Reward 10%, its AUC margin widened from 0.0071 in Clean, where the scores were 0.5723 versus 0.5652, to 0.0274 under Mismatch-40%, where they were 0.5406 versus 0.5132. Its final-accuracy margin likewise grew from 0.0033 in Clean to 0.0339 under Mismatch-40%.
Paired tests comparing DTV-Loo only with Reward 10% found its AUC advantage statistically significant in all three conditions, with reported p-values of 0.0006, below 0.0001, and below 0.0001, respectively. The reported final-accuracy improvement was significant under Mismatch-40%, with p = 0.0176.
Better scores came with a time penalty
Under corrupted preference settings, DTV-Loo reached the target accuracy T95 in approximately 30 minutes. On AIME, Vanilla GRPO required approximately 120 to 144 hours, compared with approximately 168 to 192 hours for DTV and DTV-Loo; the paper attributes the difference to per-completion valuation.
External instruction-following results were mixed: DTV remained broadly competitive, while DTV-Loo's stronger in-domain gains did not uniformly transfer across external metrics. The document is an arXiv version-1 preprint dated 28 September 2026. The authors acknowledge support and computational access from the Google TPU Builders Program and Google Tunix library.
Paper data and sources
Original title: Not All Rollouts Are Worth Learning: On Trajectory Valuation for Post-Training Reinforcement Learning
Authors: Xuesong Jia, Ziao Yang, Zhanhe Huang, Hongfu Liu
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-28
DOI: Not available
Original paper · Full text