An arXiv preprint reports that DreamHand outperformed listed baselines on 45 of 48 primary metric comparisons across five challenging egocentric benchmarks. The offline system reconstructs 3D trajectories for both hands from first-person video, including scenes with severe occlusion or hand absence.
A video model used as a motion encoder
DreamHand repurposes a pretrained video diffusion model—not to generate new footage, but as a deterministic encoder of a clean latent, the model’s internal representation before noise is added. A bidirectional spatiotemporal decoder uses context from across a clip, while a ray-based camera solver estimates the geometry needed to place the hands in 3D, without an external detector.
This was a dataset-based methods study using public annotated egocentric video and hand-image datasets rather than newly collected human-subject data. The test manifests list 34 ARCTIC clips, 72 HOT3D, 46 H2O, 77 OakInk2 and 166 held-out HOI4D clips. All methods were evaluated on identical video segments in sequence-disjoint test splits, with HOI4D held out from training and tested zero-shot.
The biggest gains came when hands left view
In the main comparison with ViDiHand, DreamHand’s MPJPE-p—a wrist-aligned pose error that also penalizes missed on-screen detections—was 15.26 mm versus 21.67 mm on ARCTIC, 12.89 versus 21.51 mm on HOT3D, and 23.03 versus 30.09 mm on zero-shot HOI4D. The paper reports those as reductions of 30%, 40% and 23%, respectively, and says DreamHand led on 27 of 29 reported metrics.
When out-of-sight frames were included, DreamHand’s MPJPE+OOS was 16.8 mm on ARCTIC and 17.3 mm on HOT3D, compared with 31.05 and 44.44 mm for ViDiHand. The reported reductions were 45.9% and 61.1%; in the out-of-sight-only measure, DreamHand recorded 35.2 and 38.6 mm, versus 149.1 and 151.7 mm for ViDiHand.
Those figures capture continuity through the gap, not certainty about where the hidden hand was in absolute space. The metric is wrist-aligned, and the paper reports no benchmark table for absolute out-of-sight placement.
The pattern extended to the additional H2O and OakInk2 evaluations: DreamHand’s MPJPE-p was 9.223 mm on H2O versus 16.219 mm for the best listed baseline, and 8.988 mm on OakInk2 versus 26.520 mm. The reported reductions were 43% and 66%, respectively. The analysis classifies these comparisons as in-domain for DreamHand and out-of-domain for every baseline; H2O’s label distribution was also seen during DreamHand training.
Calibration and speed came with caveats
A separately trained K-free configuration removes test-time camera calibration. It matched the standard model’s ARCTIC MPJPE-p at 15.26 mm, but on wide-angle HOT3D its detection coverage, measured by FAcc, dropped from 0.986 to 0.752 and its MPJPE-p rose from 12.89 to 18.04 mm. The K-free and standard figures come from separately trained models rather than a test-time switch.
On one A100, standard DreamHand ran at 63.1 frames per second and K-free at 63.3 fps, against 1.91 fps for ViDiHand—an approximately 33-fold reported throughput gap. The timing used repeated passes after warm-up, but the paper documents protocol departures for some baselines, making this a measurement under the stated setup rather than a universal hardware comparison.
What the numbers do not settle
The headline results are descriptive benchmark comparisons. The paper reports no confidence intervals or across-run uncertainty estimates, so the supplied results do not show how much the scores might vary with retraining.
Dataset boundaries also matter: only ARCTIC and H2O are subject-disjoint; HOT3D and OakInk2 splits may allow subject overlap, while HOI4D and H2O use partially pseudo-ground-truth annotations.
DreamHand is an offline, clip-level system that uses bidirectional context. The study therefore does not establish suitability for causal streaming or closed-loop control, where future frames would not be available, nor does it test whether the recovered trajectories improve downstream robot-policy learning.
The work is an arXiv version-1 preprint dated 20 August 2026; no journal publication is reported in the supplied metadata.
Paper data and sources
Original title: DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery
Authors: Yufei Liu, Xixi Wang, Hao Li et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text