A model-based reinforcement-learning policy, called MB IQL in the paper, recorded 7.9640% higher per-user net profit than TD3+BC in one online test, while per-user revenue was 6.8004% higher. The reported cost change was 1.4978%, with a p-value of 0.9631; the reported p-values for revenue and net profit were below 0.0001. The result appears in an arXiv preprint, version 1, dated 28 Aug 2026.
The decision problem
The paper frames incentive allocation as a cost-sensitive sequential decision problem. Incentives create monetizable ad opportunities before downstream revenue is observed, so a policy must make spending decisions before the full value of an attempt is known.
Each incentive attempt is treated as one transition, and its reward is defined as realized RTB revenue minus a fixed cost multiplier applied to the completion-and-payout indicator and the offered incentive amount.
The framework learns a structured user-response world model from offline logs, performs conservative model-based policy optimization and then screens candidate policies with an independent offline tool called CES.
From logs to live comparisons
The industrial analysis drew on three large logged environments: about 200 million records in DY-1, 400 million in DY-2 and 50 million in DH. The study used user-disjoint training and test splits, keeping users in one split out of the other.
Candidate policies were trained for 1 million optimization steps with a 15-minute real-time discount half-life. Each configuration used three independent seeds, and offline means and standard deviations were calculated across three seed-specific CES evaluation pipelines.
Online evidence came from four sequential, non-overlapping A/B tests. Each group had more than 10 million unique DIDs, and measurement traffic was held at 10% after ramping. The analysis used CUPED-adjusted lifts, two-sided Welch t-tests on CUPED-adjusted DID-level outcomes and multiplicity adjustment for the reported p-values.
What the comparisons found
In the offline comparisons, the structured-world-model variants scored higher than matched model-free backbones. In DY-2, MB IQL scored a 20.8% per-user net-profit gain, compared with 15.9% for IQL, a difference of 4.9 percentage points and a reported relative gain of 30.8%.
The reported matched-cost online ordering placed DESCN below TD3+BC, with TD3+BC approximately tied with IQL, and IQL below MB IQL. The paper states that this ordering was consistent with CES.
In Experiment C, MB IQL showed 6.8004% higher per-user revenue and 7.9640% higher net profit than TD3+BC. Its cost changed by 1.4978%; the reported p-value was 0.9631, compared with values below 0.0001 for revenue and net profit.
In the reversal Experiment D, plain IQL had 21.81% more incentive attempts and 23.68% more exposures than MB IQL, but recorded 4.77% lower revenue, 6.56% lower net profit and 5.32% lower ROI. Cost was statistically unchanged.
In a factual DY-2 CES evaluation, adding request context corresponded to exposure AUC changing from 0.85 to 0.91, completion AUC from 0.86 to 0.93 and conditional-revenue wMAPE from 0.83 to 0.32. Under a DY-2in-to-DY-2out support shift, the corresponding values were 0.78, 0.79 and 0.99.
Where the evaluation weakens
Cross-support diagnostics found near agreement between CESin and CESall for BC and IQL at a cost multiplier of 1.5, but divergence for IQL at 8.0 when it moved outside the inner support. In the latter case, CESin overestimated cost and revenue and consequently underestimated net profit. The diagnostic used bucket-average actions rather than checking every state-action pair.
Autoregressive world-model rollouts accumulated error, with substantial deterioration after one to two steps. The paper therefore emphasizes local one-step rollouts.
The paper's stated limits are substantial. The formulation targets short-horizon windows and net profit, while longer-term outcomes are monitored rather than optimized. CES is a support-aware ranking tool, not a full long-horizon counterfactual estimator. The world model and CES also share substantial inputs and prediction targets, which may create correlated errors and overestimate policy value.
Paper data and sources
Original title: Learning to Allocate Incentives for Incentivized Advertising via Offline Model-Based Reinforcement Learning
Authors: Zilin Zhao, Han Yang, Tianpei Yang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text