Text-to-image systems can lose track of how many objects they are asked to draw, and the problem becomes severe as the requested crowd gets larger, according to a new arXiv preprint. In tests covering nine systems, the requested count showed the strongest measured sensitivity, ahead of spatial layout, two-category composition and appearance condition.
The benchmark’s main measure, confidence-weighted numeric precision or cw-NPS, is a detector-based score for how accurately an image contains the requested number of objects. On single-category prompts, the specialized CountGen method scored .692 when the request was for 1 to 25 objects, but .129 at 76 to 100. Bounded Attention fell from .623 to .110 over the same ranges, while both open models scored below .09 in the final range.
A controlled test of counting
The study set out to test whether a large, controlled benchmark could show how requested count, composition, layout and appearance relate to counting failures. NumBench contains 1,600 categories and single-category requests ranging from 1 to 100 objects. Its core set has 160,000 prompts, split evenly between single-category and double-category requests, then expanded across two layout and two appearance conditions to reach 640,000 prompts.
The evaluation covered five commercial systems, two open models and two specialized counting methods. The open and specialized systems were run on all 640,000 prompts, with four images generated for each prompt. Commercial systems were tested on a balanced 3,200-prompt subset per system, also with four images per prompt. The commercial outputs were collected between January 2025 and July 2026.
To calculate cw-NPS, the study combined object proposals from OW-DETR, Grounding DINO and OWL-ViT after category-specific suppression of overlapping detections. The detector scores were calibrated on a separate set of 5,000 images before being combined.
Count mattered most, but format changed the result
Across the nine systems, the study measured each factor’s relative contrast by comparing the best and worst scores within otherwise defined cells. The resulting sensitivity values were .81 for count range, .57 for layout, .51 for composition and .45 for appearance condition. The authors describe these figures as associations within the benchmark, not causal coefficients.
Among the three guided settings reported, grid layouts had the highest average cw-NPS at .45. Scene and random layouts were close, at .31 and .32. The ordering is observational, and the study did not have the matched free-form aggregate needed for a complete four-layout comparison.
The strongest method also depended on what the prompt asked for. CountGen led across the single-category ranges, while Bounded Attention led across the double-category ranges. In the 1 to 25 range for double-category prompts, Bounded Attention scored .641 compared with .610 for CountGen.
A proposed explanation for crowded scenes
The paper proposes a mechanism in which requested instances compete for a limited number of regions that can be resolved separately. Under that model, visible undercounting should rise roughly quadratically at low occupancy, while regular or coordinated layouts should degrade more slowly. The benchmark’s layout ordering is presented as consistent with that prediction, but the model is only a proposed mechanism and the results do not establish spatial collision as the sole cause of failure.
The layout comparison cannot isolate spatial capacity from coordination, object size, overlap, background complexity or detector confidence. The proposed collision model also does not account for overcounting, category substitution, changes in object scale or learned preferences for scene size. Those factors remain possible contributors within the study’s measurements.
The score was most dependable below 50 objects
The researchers checked the detector measure against human counts. Three annotators independently counted 1,600 images per system, for 14,400 images in all, with 400 images from each count range per system across layouts, appearance conditions and both composition settings.
Agreement was strongest at lower counts. Fleiss’ kappa, a measure of agreement among multiple raters, was .82 for 1 to 25 objects and .58 for 26 to 50, falling to .33 for 76 to 100. Detector-human correlation was above .80 in the first two strata, about .40 at 76 to 100 and above .75 overall. The study did not report a separate estimate for 51 to 75.
The measurement itself was also sensitive to the detector choice. Replacing the ensemble with any single detector shifted aggregate cw-NPS by at least .10, a result the study says underscores the role of the scoring pipeline in the reported totals.
Natural language showed the same broad pattern
The researchers then tested 243 more natural-language prompts. Method rankings and performance gaps remained broadly consistent with NumBench, but no system exceeded a cw-NPS of .67. Every prompt in this transfer set requested fewer than ten objects, so the test did not show how high-count performance would carry over to natural language.
The controlled templates leave out paraphrases, number words, multilingual instructions and long natural-language prompts. The appearance condition also combines image-quality degradation with changes in the visual evidence used to separate individual instances. Commercial systems were unversioned snapshots, so later service updates could change the results.
What the benchmark can and cannot settle
Taken together, the results support using count range as the first warning sign when assessing whether an image generator can follow numeric instructions. They also point to layout and composition as meaningful reporting dimensions, while suggesting that scores above 50 objects should be treated as stress tests rather than equally precise estimates across the full range.
The complete 640,000-prompt dataset is included with the supplementary submission. The planned full release after acceptance includes prompts, factor labels, source code and experiment metadata. The supplied document identifies the work as an arXiv version 1 preprint dated 28 Aug 2026.
Paper data and sources
Original title: NumBench: Diagnosing Counting Failures in Text-to-Image Models
Authors: Sandeep Wadhwa, Mayank Vatsa, Richa Singh et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text