An arXiv version 1 preprint dated 20 Aug. 2026 compares 41 design choices for latent actions in robot learning. LAPO had the highest overall mean score across the tested benchmarks, at 0.7327, followed by ΔDINO at 0.7280.
Latent actions are internal representations of movement used between visual input and physical commands to a robot. The comparison covered latent-action models, learning objectives and regularization, and ways to integrate latent actions into the system that predicts robot actions.
Inside the comparison
The study used three stages: pre-training for representation learning, mid-training for vision-language model backbone fine-tuning, and post-training for downstream policy learning. Stages I and II used raw video without robot-action annotations, drawing on a heterogeneous corpus of approximately 59 million frames.
The evaluations covered 7-DoF single-arm end-effector delta control and 14-DoF dual-arm joint-position control. Ablation results were averaged over three random seeds, and the RoboTwin2.0 experiments used 12 selected tasks.
The settings that stood out
Regularization — a penalty that shapes what a model learns — had a relatively limited effect by type in the reported comparison, while its strength was more consequential. Overly strong settings were associated with poorer reconstruction and weaker physical-action prediction.
The paper recommends regularization strengths of 10−7 for its VAE setup, 10−5 for Sparsity, 10−3 for SIGReg and 1 for VQ-VAE. These are reported recommendations, not formally validated optima.
The preferred action-head design varied with the available training information. DAP was the weakest of the compared designs, while LAP performed better than DAP; the authors recommend LAP when tuning uses video alone and JAP-LAP when robot-action annotations are available.
The authors reported that a 32-dimensional latent action offered the best overall trade-off in both the single-arm and dual-arm settings. The result was not identical across every benchmark, so the dimension should not be read as a universal optimum.
Video scale and normalization in the tested comparisons
In the comparison of Stage-II video scales, the setting using 100% rather than 14.5% of the full corpus showed higher downstream performance across all the tested benchmarks. The largest reported difference was 9.0% on LIBERO-Plus.
Normalization was not consistently associated with higher scores. Unnormalized latent actions scored higher in 28 of 33 method-benchmark combinations, with an average gain of 0.0115 and a largest reported gain of 0.0160 on LIBERO-Plus.
The physical-robot comparison
In the real-world comparison, the latent-action-tuned model completed 317 of 400 rollouts successfully, compared with 259 of 400 for the baseline. That corresponded to success rates of 79.25% and 64.75%, respectively — a difference of 14.5 percentage points, or 22.4% in relative terms.
The test used four Franka Panda manipulation tasks, with 50 demonstrations per task and 100 independent evaluations per task for each method. The robot viewed the scene through a single front-facing RGB camera.
Evidence with clear boundaries
The study found that reconstruction-based proxy metrics were more useful for coarse screening than for fine-grained ranking. They showed stronger correlations with downstream performance than linear or MLP probes, but they do not replace direct task evaluation.
The comparisons were conducted on existing open-source robotic datasets, selected manipulation benchmarks and robotic-arm experiments. The authors identify broader in-the-wild video and platforms beyond robotic arms as areas for future work.
Because the experiments were nonrandomized comparisons, the results indicate associations within the tested configurations; they do not establish that any one design caused the improvement. The preprint did not report formal significance tests or uncertainty intervals for the comparisons.
The evidence is limited to the tested robotic datasets, manipulation benchmarks and robotic-arm experiments. The document is an arXiv version 1 preprint dated 20 Aug. 2026.
Paper data and sources
Original title: What Matters for Latent Actions in Robot Learning
Authors: Xizhou Bu, Qingda Hu, Lei Zhou et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text