Multimodal AI agents made most of their progress on a digital drawing task within the first 10% of the available action budget, then often plateaued or degraded the canvas, according to an arXiv preprint dated 26 August 2026. The result exposes a weakness in sustained visual tool use: agents can make an initially useful response to what they see, but often fail to turn later feedback into steady correction.
A test of controlled visual action
The study asks whether an agent can convert visual evidence into precise, parameterized actions, meaning actions whose settings directly govern the visual result, and use the updated state to decide what to do next.
To test that question, EASEL uses a controlled 2D digital canvas. It includes 110 reference-guided reconstruction samples and five instruction-conditioned semantic tasks, with up to 11,392 interactions in a full evaluation.
Researchers evaluated 25 multimodal agents: six closed-source APIs and 19 open-source models spanning 4 billion to 90 billion parameters.
For reconstruction, its Final Similarity score combines structural similarity, known as SSIM, normalized RGB L1 distance and Edge IoU, weighted 0.5, 0.3 and 0.2 respectively.
A wide gap at the top
Gemini 3.1 Pro led standard reconstruction with a Final Similarity score of 0.535. Claude Opus 4.7 followed at 0.472, while open-source models clustered between 0.40 and 0.45.
Even the leader remained well below the benchmark's data-policy reference. That anchor scored 0.665 when matched to the agents' action budget and 0.715 in its native setting, exceeding every evaluated agent. The spread indicates substantial headroom in the benchmark itself, rather than a task that all systems were close to solving.
These are point estimates. No confidence intervals or other formal uncertainty estimates were reported, and the supplied analysis does not give run-to-run variability for the rankings.
A modest signal from training
The authors also ran a separate training probe. They fine-tuned Qwen3.5-9B with LoRA in two stages, first on C1 and then on C2. The training material, EASEL-Data, used 11,000 reference images, generated a 250-step vector trajectory per reference and yielded 440,000 next-action samples, including 44,000 CoT samples.
That EASEL-9B variant was reported as third overall and 6.3% ahead of the base model. But the comparison was a nonrandomized model ablation, so it is an observed difference, not a causal estimate that the training data produced the gain.
In the curriculum breakdown, the base model scored 0.432, the C1-only version scored 0.456 and the full C1+C2 version scored 0.459. The reported differences were +0.024 and +0.027 versus the base, with C2 adding +0.003 beyond C1. Most of the numerical change appeared after C1, with a smaller increment after C2.
Early progress, weak correction
The benchmark also examined the path to the final image, not just the final score. Most reconstruction gains came in the first 10% of the step budget; later trajectories plateaued or degraded. GPT-5.5 showed the strongest reported degradation, with an F-B Gap of 0.073, an AUC of 0.440 and a Final Similarity of 0.426. The F-B Gap measures the difference between an agent's best intermediate result and its final one.
The semantic tasks exposed a different split. Across five representative models, Gemini 3.1 Pro and GPT-5.5 each scored 1.000 on Hello, and GPT-5.5 scored 1.000 on Simple Maze. H-Maze was much weaker, with scores ranging from 0.000 to 0.047.
Action traces showed different failure modes. Claude Opus 4.7 made 124 non-maze undos versus three maze undos, consistent with unstable draw-undo loops. Claude Sonnet 4.6 rarely used undo but had near-maximal maze invalid rates.
What the benchmark leaves open
On listed secondary perception checks, reported changes after training were small: +0.7% on MMVP, +2.0% on POPE, +0.6% on HallusionBench, 0% on BLINK spatial reasoning and +1.7% on BLINK counting. These comparisons showed small positive or neutral changes, but no uncertainty intervals were reported.
The paper frames EASEL as a controlled 2D proxy that trades breadth for control. By isolating visual goals, parameterized actions and closed-loop feedback, it reduces confounding variables, but it does not directly test physical tools or open-ended embodied work.
The semantic analysis covered five representative models and omitted models with near-zero scores on all semantic tasks. The broader evaluation also reports no inferential tests or repeated-run variability, leaving open whether the rankings would remain stable under repeated evaluation.
The document is arXiv:2608.25417v1, dated 26 August 2026. Its front matter lists a GitHub repository and a Hugging Face page, but the supplied material does not establish what those artifacts contain or whether the release is complete.
Paper data and sources
Original title: Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents
Authors: Shudong Liu, Dongyang Chen, Enci Zhang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text