AI systems can answer a video-related request and still fail the request itself, according to a new benchmark of multimodal large language models. Video-IFBench found that models often met some constraints in a prompt without meeting all of its requirements. In plain terms, a response could be broadly on task but still miss a condition, detail, or required presentation.
The strongest overall result came from Gemini-3-Pro, which scored 76.5 on the task-gated constraint satisfaction rate, or TCSR, and 54.5 on the task-gated instruction satisfaction rate, or TISR. Qwen3.5-397B-A17B-Think was the best open-source model, with scores of 69.6 and 46.1. It was 6.9 points behind on TCSR and 8.4 points behind on TISR.
A benchmark built around instructions
The benchmark covered over 700 videos from public sources, ranging from 10 seconds to 10 minutes each, with approximately 49 hours of footage in total.
Its semi-automatic construction pipeline combined MLLM extraction, programmatic rules and human verification. The process produced about 1.5K high-quality samples.
The instruction taxonomy used four templates: Single, Multi, Selection and Nested. Across them, the benchmark covered 32 manually curated video-understanding task types and 39 semantic or format constraints. That design tested not only what a model said about a video, but also whether it followed the requested form and conditions.
More than 20 recent models were evaluated across instruction types and complexity levels.
The harder the prompt, the wider the gap
Structure mattered. Many models scored higher on Multi instructions than on Single instructions, while Selection and Nested instructions were harder. Larger models generally performed better within the same model family. In the benchmark's shorthand, the four categories moved from one or several requests to choices among branches and instructions nested inside other conditions.
Nested prompts were particularly difficult. Gemini-3-Pro recorded 53.7 TCSR and 46.0 TISR on this category. The best open-source model recorded 33.0 and 28.2, respectively. Those figures were below the models' overall scores, showing how much harder layered instructions were in this evaluation.
Content rules were harder than presentation rules. The benchmark found that models performed substantially worse on semantic constraints, which govern what an answer must capture, than on format constraints, which govern how the answer is delivered. A polished format could therefore coexist with a response that missed the required meaning.
Adding more requirements also exposed a weakness that a simple task-pass score could hide. Task pass rates stayed relatively stable across different numbers of tasks, but TISR fell as the number of constraints rose. For Gemini-3-Pro, it dropped from 62.7% with one to four constraints to 46.9% with nine or more. Qwen3.5-397B-A17B-Think fell from 58.2% to 42.5% across the same range.
Depth had a similar effect on nested trees. Gemini-3-Pro's task pass rate fell from 64.6% at a depth of two to four to 55.6% at eight to 14. Qwen3.5-397B-A17B-Think fell from 40.5% to 22.2%. The benchmark found a steeper decline for the open-source model in this comparison, although it did not report formal uncertainty estimates.
Selection tasks became less reliable as the model had more branches to consider or as the correct branch appeared later. In reported examples, Gemini-3-Flash fell from a 66.7% pass rate with two branches to 56.5% with four, while Qwen3.5-397B-A17B-Instruct fell from 58% to 39.1%. When the correct branch moved from position one to position three, pass rates fell from 76.5% to 52.9% for the Thinking model and from 70.6% to 29.4% for the Instruct model.
What the scores can and cannot say
The scoring method adds an important qualification. Responses were checked with a hybrid protocol that combined an LLM acting as a judge with programmatic checklist verification. When the researchers used a smaller judge, absolute scores generally rose, but the relative pattern across models remained largely consistent.
Thinking mode was not a universal advantage. It produced clear gains for Qwen3 and Qwen3.5 models, but little improvement or performance drops for InternVL3.5. The gains were generally larger on semantic constraints than on format constraints.
These results are best read as descriptive comparisons within a curated benchmark. The benchmark uses public videos and a semi-automatic construction process that combines model extraction, rules and human verification. The paper reports no confidence intervals or inferential uncertainty estimates, so the size and generality of the differences remain unresolved.
The document is an arXiv preprint, version one, dated 26 August 2026. Its central message is straightforward: strong performance on video understanding does not automatically amount to faithful instruction following, especially when a prompt combines many constraints, branches or nested conditions.
Paper data and sources
Original title: Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios
Authors: Hongbo Liu, Peixian Chen, Sihan Liu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text