A preliminary surgical AI model performed better at forecasting future surgical frames and two-dimensional instrument trajectories when it predicted three steps at a time and fed each forecast into the next stage, rather than making a single one-shot prediction. The advantage was reported across all five future segments and every evaluated metric.
The findings appear in an arXiv v1 preprint dated 20 Aug 2026.
How the forecast worked
The model encoded observed surgical frames and historical tool trajectories into internal representations, then used separate prediction heads for future visual states and two-dimensional instrument trajectories.
Under the benchmark’s history-only protocol, the first five frames were used as history and the next 15 visual states and trajectory points were predicted. In the chunked setup, the model forecast three future steps, recursively fed its predicted visual and motion representations back into the model, and produced the 15-step rollout through five chunks. The comparison was direct one-shot prediction under the same protocol.
The researchers used PSNR, SSIM and LPIPS as visual measures, and ADE and FDE as trajectory measures. The data came from SurgWMBench: 1,637 valid motion segments and 32,740 annotated frames from real clinical robot-assisted radical prostatectomy videos, with a 20-point two-dimensional instrument trajectory for each segment.
The early advantage was substantial
For the first future segment, chunked rollout had a PSNR of 23.105 decibels, compared with 18.859 for one-shot prediction. Its SSIM was 0.7971 versus 0.4713, while its LPIPS was 0.1783 versus 0.6045; the paper reports a 70.5% reduction in LPIPS.
The instrument-trajectory measures also favored chunking in that first segment. ADE was 22.22 pixels versus 45.77 pixels, and FDE was 31.64 pixels versus 49.47 pixels for one-shot prediction.
At the final reported segment, t+13:15, chunked rollout still had better scores, although performance had declined. PSNR was 17.982 decibels versus 16.872, SSIM was 0.6343 versus 0.4264, and LPIPS was 0.3047 versus 0.6244. ADE was 158.52 pixels versus 195.27, and FDE was 172.12 versus 208.04; the reported relative reductions were about 18.8% and 17.3%.
Longer forecasts still drift
Chunked rollout reduced but did not eliminate the decline over longer horizons. Both visual quality and trajectory accuracy deteriorated as the forecast moved farther into the future, and qualitative examples showed reduced fidelity and endpoint errors for both settings at t + 12 and t + 15.
The study therefore presents a preliminary benchmark result rather than a complete solution to long-horizon surgical forecasting. Whether action conditioning, uncertainty modeling or stronger visual-motion consistency can improve the later stages remains an open question identified by the authors.
What the benchmark cannot answer
The evaluation was based on SurgWMBench and the reported visual and trajectory metrics. It does not establish how the approach would perform on other procedures, instruments or surgical datasets, and it does not show clinical benefit, patient safety or effective low-level robotic control.
The comparison also does not isolate the separate contributions of chunking, scheduled sampling and other architectural components. No inferential tests, confidence intervals, repeated-seed analysis or external validation were reported.
The acknowledgments report support from the University of Macau, the Dr. Stanley Ho Medical Development Foundation and the Science and Technology Development Fund of Macao S.A.R. The authors reported no competing interests.
Paper data and sources
Original title: Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning
Authors: Weiliang Huang, Huanrong Liu, Bob Zhang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text