An arXiv preprint dated 20 August 2026 reports that direct preference tuning in continuous-time flow models can increase a model’s score on a preference signal while pushing final samples away from the pretrained data manifold. In a synthetic benchmark, vanilla FlowDPO at β = 1 reached 92% on the paper’s Win measure, while OnManifold fell to 0%; StrictWin and WinQuality also collapsed.
The authors interpret this pattern as a structural source of reward hacking: a preference score can improve even as manifold validity degrades. The study asks whether anchoring the preferred “winner” to the manifold—the set of outputs represented by the pretrained model—can preserve that structure while improving preference alignment.
A mathematical warning
Under exact optimization and stated regularity assumptions, the authors’ theorem says that optimal Flow Matching recovers the data distribution at the terminal, or final, time.
A separate result gives a sufficient condition for drift: an endpoint update with a nonzero component normal to the manifold is enough for off-manifold movement. It does not say that every FlowDPO update drifts.
The proposed anchor
ThermoDPO-weighted adds a temperature-controlled winner-side manifold anchor to pairwise preference optimization. At positive temperature, the theory decomposes the objective into FlowDPO plus that anchor; its guarantee is pointwise, bounding a reconstructed-winner distance rather than preserving the full generated distribution.
On the main toy benchmark, ThermoDPO-weighted recorded a StrictScore of 0.899, compared with 0.629 for FlowDPO and 0.857 for FlowDPO+RFT, the highest reported value among the highlighted comparisons. The comparison used a common three-layer flow-matching MLP, common winner–loser pairs, a 10,000-step fine-tuning budget and 10,000 generated samples.
From toy manifold to images
The empirical study then tested the method on synthetic and real-world image benchmarks against RFT and FlowDPO variants, using automated and human metrics. The offline real-image dataset contained 1,000 prompts, 64 candidate images per prompt and one winner–loser pair per prompt.
On SD3.5-M, with CFG set to 4.5, the paper reports a 47.5% improvement in OCR and a 16.0% average gain across the reported metrics over the baseline.
In pairwise human judgments covering 30 prompts, ThermoDPO-weighted remained competitive on text accuracy and was generally preferred on visual quality. But each comparison was labeled once by a single paper author, with no external crowd workers, paid participants or compensation.
The evidence has limits
The study is offline and uses a fixed winner–loser dataset. It does not test online RLHF or transfer to other continuous-time or diffusion-based alignment algorithms, and its thermodynamic motivation is not developed into a fully principled physical derivation.
An appendix adds a further model check, but not a model-independent result: its prose reports that at CFG = 3.5, ThermoDPO-weighted improved GenEval, UniReward, HPSv3.0 and OCR over the pretrained model, while FlowDPO had higher OCR but lower values on the other listed metrics. The authors say this does not establish model-independent behavior.
The preprint therefore offers conditional mathematical evidence and benchmark comparisons in the tested settings, rather than a universal guarantee. It does not establish that every FlowDPO update drifts, that the proposed bound keeps the full output distribution on the manifold, or that the reported trade-off transfers to online RLHF or other alignment objectives.
Paper data and sources
Original title: Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hacking
Authors: Yansen Han, Shengyi Liao, Yuanxing Zhang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text