In a simulated robot benchmark, JEPA-x recorded much lower forecast drift and higher task success than a visual-only baseline. Relative rollout drift was 0.104 for JEPA-x, compared with 0.361 for VISUAL, while mean control success was 78.2% versus 53.6%—a difference of 24.6 percentage points under the same planner, action prior and candidate budget.
Learning from two views
The study asked whether privileged physical trajectories could make visual dynamics more forecastable and improve control. JEPA-x treats visual observations and physical states as paired views of the same action-conditioned trajectory. During training, a shared predictor advances a history from either modality and is trained to match future representations in both. At deployment, the physical branch is discarded.
To test forecastability independently of the model’s own predictor, the visual encoder was frozen, its co-trained predictor was discarded, and the same lightweight action-conditioned predictor was fitted from scratch. Lower relative rollout drift in this fresh-predictor probe was treated as better forecastability.
The gain held across tasks, with a catch
The multi-task training corpus contained 29,260 episodes and approximately 1.44 million transitions across 22 object–task configurations. The suite covered 13 distinct assets, with 450 clean episodes per configuration before noise augmentation. Evaluation used 50 held-out scenarios for each interaction family and seed, or 300 scenarios per model drawn from 266 source episodes; the scenarios used fixed seeds and shared initial states across methods.
In matched single-task tests, JEPA-x also had lower relative drift than the visual baseline in Two-Room, OGBench-Block, Push-T and Reacher. Drift fell from 0.503 to 0.221 in Two-Room, 0.507 to 0.269 in OGBench-Block, 0.399 to 0.258 in Push-T and 0.313 to 0.233 in Reacher.
Control was less uniform. JEPA-x improved reported success on Two-Room and OGBench-Block, but its mean success on Push-T and Reacher was within one percentage point of VISUAL. Each single-task controller was tested on the same 150 paired scenarios, and the paper describes the control gains as task dependent.
Prediction was not the whole story
A separate state-regression baseline showed that stronger physical-state decodability did not by itself reproduce the forecastability or control gains. REGRESS had a manipulated-object position R2 of 0.991, compared with 0.978 for both JEPA-x and VISUAL. Its control success was 53.7% and its relative drift was 0.373.
The ablations pointed to different roles for the training ingredients. CROSS-ONLY reached 73.0% control and ALIGN-ONLY 72.6%, compared with 56.7% for SHARE-ONLY; ALIGN-ONLY also had higher drift than JEPA-x, 0.158 versus 0.104. A shuffled visual–physical pairing had only 16.6% control but drift of 0.167, lower than the visual baseline. The result is a caution that low drift alone did not deliver usable control.
For suite-level comparisons, the analysis used episode-clustered paired sign-flip tests with 105 Monte Carlo assignments. Single-task comparisons used exact McNemar tests on the same 150 paired scenarios.
Still a simulation result
The evidence has a narrow boundary. The evaluation was limited to simulation, and JEPA-x required paired privileged physical trajectories during training while relying on a fixed rigid-body state schema. Real-world observations and broader physical interactions were not evaluated, so whether the approach transfers beyond this setup remains open.
Paper data and sources
Original title: JEPA-x: Cross-Predictive Physics Grounding for Forecastable Latent Dynamics
Authors: Kehan Wen, Ziming Li, Siyuan Luo, Fan Shi
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text