AI models can get the scientific answer right and still fail to follow the instructions wrapped around it, according to a new benchmark of scientific questions. Across the evaluated models, fewer than 30% of cases were both scientifically correct and constraint-compliant. About 20% were correct but violated at least one constraint, while incorrect answers that followed the constraints exceeded 30% for several models. The strongest discipline-level instruction scores appeared in biology and materials science; chemistry was markedly weaker, and geography was also described as difficult.
A test built around scientific instructions
The benchmark, called SciMIF, contains 2,527 samples across chemistry, geography, biology, materials science and physics. Images appeared alongside 27.50% of the samples, making multimodal input a minority of the test rather than its default format. The dataset draws on 13 existing scientific datasets and covers 22 task types across the five disciplines.
To test instruction following, the researchers augmented existing scientific questions with two kinds of rules: constraints tied to the subject matter and general constraints. Their taxonomy spans 22 tasks, 10 functional groups and 42 discipline-adapted constraints; the augmentation procedure used N = 3 and k = 3 to balance diversity and consistency.
Performance was split into separate measures. Constraint Satisfaction Rate and Instruction Satisfaction Rate captured whether instructions were met, while Decomposed Requirements Following Ratio, or DRFR, counted how many individual requirements in a prompt were followed. The checks combined exact-match and precision-based measures with language-model judging. Scientific answer correctness was scored separately with a binary decision from CompassVerifier-32B, while deterministic constraints were checked programmatically and semantic requirements were assessed with GPT-4.1.
The verification process included human checks: two annotators reached 100% observed agreement on 20 calibration instances, and 884 samples were manually revised. Those checks helped prepare the benchmark, but the later semantic assessments still depended on an LLM judge.
Chemistry exposed the hardest instructions
On ISR, GPT-5.2 scored 46.33% in chemistry, compared with 72.82% in biology and 78.07% in materials science. Those are instruction-level scores: the reported measure concerns whether the instructions were satisfied, rather than scientific correctness alone.
Chemistry also stood out when the benchmark was broken down by the type of requirement. Its average DRFR was 68.2%, the lowest domain-level figure identified in the analysis. For GPT-5.2, following scientific constraints reached 88.74%, versus 74.65% for general constraints. Among individual constraint groups, letter requirements were hardest at 51.30%, followed by number requirements at 68.62% and terminology at 73.59%.
Correctness and compliance came apart
The split between the two abilities was not a minor edge case. Across the evaluated models, the rate of answers that were both correct and compliant stayed below 30%. Roughly one in five answers was scientifically correct but broke a constraint, and for several models more than 30% of answers followed the instructions while getting the science wrong. A system can therefore satisfy the requested format or rule set without delivering a correct scientific answer, and the reverse can also happen.
Statistically, correctness and adherence moved together only weakly. Five of six model-specific Pearson tests remained significant after a Bonferroni correction, using a threshold of 0.00833, but the phi coefficients for those models ranged only from 0.0672 to 0.2069. Grok-4-fast showed no meaningful association in this analysis, with p = 0.68 and phi = 0.0081. The Jaccard overlap between correct cases and constraint-following cases ranged from 27.04% to 36.40%.
Bigger did not mean reliably better
Model family mattered more clearly than model size in the displayed comparisons. Closed-source models held an advantage across disciplines: GPT-5.2 recorded an overall score of 65.67%, against 57.39% for the leading open-source model. But larger open-source models did not consistently score higher. InternVL3.5 moved from 51.37% at 8B to 51.64% at 38B, while Qwen3.5 fell from 51.37% at 27B to 50.41% at 122B. These comparisons show a pattern in the benchmark, not that parameter count causes the result.
Image inputs were associated with lower instruction adherence in the one model-specific modality comparison. For Gemini-3.1-Pro-Preview, DRFR was 64.67% with multimodal inputs and 87.92% with text-only inputs in geography, a 23.25-point gap. The differences were smaller in materials science, 85.50% versus 90.16%, and physics, 74.49% versus 83.76%, corresponding to decreases of 4.66 and 9.27 points. The analysis covered only those three disciplines and one representative model, so it does not establish that images themselves caused the lower scores.
Results also varied within a discipline. For GPT-5.2, materials-science DRFR ranged from 63.92% on LLM4Mat-Bench to 96.28% on MatSciBench. That spread suggests that a single discipline average can hide differences between the underlying source datasets.
A benchmark signal, not a real-world guarantee
The findings are about model behavior on a constructed benchmark, not a direct measure of human scientific work or reliability in a real laboratory or research workflow. That distinction matters because the benchmark combines heterogeneous source datasets and task types, and because semantic constraints and procedures were judged by language models while scientific correctness was scored separately. The preprint is arXiv:2608.25973v1, dated 26 August 2026; its abstract says the data and code will be released. For now, the result is a warning that scientific answer quality and obedience to detailed instructions need to be tested as separate capabilities.
Paper data and sources
Original title: SciMIF: Understanding Multimodal Instruction Following in Scientific Domains
Authors: Ye Shen, Yuting Zheng, Dun Pei et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text