A new arXiv preprint describes a prototype that combines coding-agent reasoning and executable code for world evolution with video-model generation for visual realization. In reported examples, generated videos followed proxy-specified entity positions and motion, scene layout and camera trajectories. The paper presents the work as a prototype and does not demonstrate autonomous construction of a complete open-world game or simulator.
A bridge from world state to video
The proposed pipeline starts with an interaction intent, translates it into code and compiles the updated world state into a proxy—a deliberately coarse, programmable description of what matters visually. A video model then receives a proxy video and structured text to generate the observation. The proxy can encode the camera, entity positions, poses, trajectories and spatial relationships, and is rendered deterministically into the proxy video.
The implemented proxy used one-quarter of the target video resolution along each spatial dimension and added 1/16 as many visual tokens. The authors described the additional inference burden as negligible.
Built on a small gameplay set
The training material came from 157 gameplay takes containing approximately 5.6 hours of source video. The researchers sampled 9,420 clips, each five seconds long, at two-second intervals.
They fine-tuned the official MiniMax-H3 Ref2VA backbone for proxy-conditioned video generation. Target clips contained 124 frames at 1,344 by 768 pixels and 24 frames per second, while proxy sequences contained 124 frames at 336 by 192 pixels. The encoder received 11 proxy frames.
Fine-tuning used a rank-128 LoRA adaptation across 50 transformer blocks and approximately 596 million trainable parameters. Training ran for three epochs, or 3,534 optimizer steps.
Control was visible, but not measured
At inference, GPT-5.6 Sol was supplied with existing game-engine code, basic controls, collision handling, runtime updates and the proxy primitives used during training. The resulting coding-agent-built worlds were reported to be executable and player-controllable, with coarse proxy geometry for visually relevant state. The setup still fell short of complete AAA-scale construction from scratch.
After the video model was adapted on the paired gameplay data, qualitative examples were reported to closely follow the proxy’s conditions for entity positions and motion, scene layout and camera trajectories. In evaluations using proxies from simple coding-agent-built worlds, the model was also reported to preserve visual detail and dynamics while following those conditions. A visual-quality caption reported generalization across diverse characters, environments, motions and camera trajectories after five hours of GTA V fine-tuning.
The authors also reported more precise and responsive control of character motion, actions and camera movement than action- or camera-conditioned video world models in qualitative comparisons. That comparison excluded inference latency and supplied no quantitative baseline metric, so the paper does not provide a measured advantage. The proxy’s small footprint was described as imposing negligible additional inference burden.
What the prototype does not yet show
The paper includes a separate test with KITTI-360, but it was an offline proof of concept for constructing paired RGB observations and proxy conditions. Calibrated camera poses, 3D reconstruction and object annotations were used while building those pairs, not provided to the video model. It was not reported as a real-world model-training or deployment evaluation.
The authors acknowledge that compute-limited training constrained the work and that autoregressive real-time generation was not implemented. They also say the prototype did not demonstrate autonomous construction of a complete open-world game or simulator. With its reported results limited to qualitative examples and an offline data-construction proof of concept, the system is best read as a feasibility signal for connecting editable world state to detailed video, not as a validated general-purpose world simulator.
Paper data and sources
Original title: Code World Model: Coding Agent as World Brain
Authors: Yiwen Chen, Guosheng Lin, Chi Zhang
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text