A synthetic-data system for training vision-and-language models recorded higher scores on tests of multimodal instruction following, while also posting a higher average across a broader set of image-and-text benchmarks. The results come from an arXiv preprint describing VISA, a system that continually checks and revises the examples it generates.
The document is labeled arXiv:2608.26013v2 and dated 27 Aug 2026. The supplied analysis reports no confidence intervals, p-values, significance tests, or formal inferential model for the benchmark comparisons.
A training pipeline that learns from its own checks
VISA treats instruction synthesis as a closed loop with four stages: Perception, Planning and Execution, Reflection, and Memory Update. In practical terms, the system examines an image and a requested task, plans an example, reflects on failed samples, and updates a running store of constraints for later rounds.
After a sample fails, the loop uses diagnostic-guided recovery. It also probes accepted samples with the target model to estimate their difficulty, then reuses verifier contracts, or checks for whether an instruction has been met, as reinforcement-learning rewards. The reported setup does this without a separately trained reward model.
The experiments used Qwen3.5-27B as the synthesis agent and Qwen3.5-4B as the model trained with supervised fine-tuning or reinforcement learning. Qwen3.5-9B served as a same-family reference model. The training runs used 128 L40s GPUs; the visual encoder layers were frozen, while the language-model backbone and alignment module were fine-tuned.
Higher scores on the main instruction test
The clearest reported differences appeared on MM-IFEval, a multimodal instruction-following evaluation. The base Qwen3.5-4B condition had an average score of 60.8. With the approximately 15,000-example VISA supervised-fine-tuning set, the reported average was 63.9, while the VISA reinforcement-learning condition reached 64.9. Its P-Level score was 59.0.
The component comparisons followed the same direction. Under supervised fine-tuning, the reported average was 60.2 for a static pipeline, 61.2 after adding Reflection, 63.1 after adding Reflection and Memory, and 63.9 for the full VISA setup. The sequence describes differences among the tested configurations, but does not establish randomized attribution for the individual components.
The reported comparison also covered seven general multimodal benchmarks: MMBench, MMStar, MM-Vet, HallusionBench, MathVista, OCRBench, and AI2D. The average score was 70.5 for the base condition, 71.0 with VISA supervised fine-tuning, and 72.9 with VISA reinforcement learning. The reinforcement-learning condition scored above base on five of the seven benchmarks.
What the generated corpus looked like
The experimental setup describes an approximately 15,000-sample VISA training dataset based on the MM-IFInstruct image pool. A separate diversity and difficulty analysis covered 15,668 samples produced by the system. The synthesized instructions included every one of the 90 off-diagonal category pairs tracked in the analysis, and instruction embeddings had a mean pairwise cosine distance of 0.977, a numerical measure of how differently the examples were represented.
The system's constraint library expanded during 126 self-evolution rounds. VISA added 234 constraint types, reaching 310 types in all, a reported 4.1-fold increase without additional human annotation. The generated samples contained a mean of 4.87 constraints, with a median of five and a standard deviation of 2.12.
Most examples were labeled easy by the target-model probing protocol: 82.6% were easy, 15.9% medium, and 1.5% hard. Verifier failures affected 5.6% of constraint instances overall. Counting, lexical, and format checks had the highest reported failure rates, at 11.6%, 10.2%, and 8.1%; style failures were 0.16%, while conditional checks recorded 0%.
The evidence remains a benchmark result
A manual audit of a sampled subset found approximately 95% agreement between the system's LLM-judge decisions and human annotations. The supplied analysis does not report the audit sample size or explain whether that subset represents the full corpus, so the figure cannot establish how the judge would perform across all generated examples.
The supplied evidence is benchmark-based and does not include a human-user or deployment evaluation. It is centered on Qwen3.5 target models and the listed benchmarks, so broader transfer is not demonstrated. The paper also notes that P-Level MM-IFEval cases depend on visual perception and grounding, making them an imperfect measure of instruction-following supervision alone.
The system adds state updates, iterative recovery, and target-aware probing compared with a static one-shot pipeline. Those design choices are part of the reported method and help explain why the benchmark comparison also involves a different training process, rather than a single isolated change.
The front matter identifies vivo AI Lab, but the supplied text reports no funding statement.
Paper data and sources
Original title: VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following
Authors: Min Zeng, Guanxin Tan, Libin Cen et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text