AI systems that generate stories across text, pictures and speech still have trouble keeping the story world visually consistent. In a benchmark of 32 system configurations, orchestration led the overall comparison, while image performance was the clearest observed weakness. The benchmark therefore tests more than whether each output is plausible on its own: it tests whether the outputs stay aligned as the story moves forward.
The study asked whether a system could preserve the current story context and realize a specified next step consistently across an image, a narration and speech. That makes consistency across modalities central to the evaluation, rather than treating each output as a separate task.
A test of complete story transitions
The benchmark contains 900 transitions drawn from openly licensed children's books. Human annotators reviewed and revised the examples after automated scoring selected an initial 1,000 pairs; 16 reviewers took part in that filtering process. The comparison covered orchestration, semi-orchestration and native any-to-any systems.
The evaluation combined separate checks for the three output types with an integrated assessment. It used BERTScore for text similarity, CLIP Similarity for image-text alignment, Speech Metadata Accuracy for speech properties, and three-judge language-model assessments covering text, image, speech and the combined result. The main quality scores used valid evaluator outputs when the required material was present; missing outputs were also examined separately with zero-filled scores.
Reliability mattered as much as quality
Orchestration came out ahead in the overall comparison. Semi-orchestrated and native any-to-any systems had more difficulty producing all requested modalities reliably and in a controllable way. The finding puts output completeness alongside quality in judging the systems.
Some failures were stark. MMaDA produced 187 failed plans per configuration, while Emu3-9B produced 50. In those failed examples, all three requested outputs were absent. These are counts within this benchmark, not estimates of how often the systems would fail in wider use, but they show why output completeness can change the practical ranking of a multimodal system.
The hardest part was carrying the scene forward
Image was the weakest modality by several descriptive measures. It was the weakest component for 14 configurations, had the largest average gap from the other modalities at 6.85 percentage points, and showed the widest spread in scores, with an interquartile range of 1.79. The bottom third of configurations reached no more than 6.60 on the Total Average.
The analysis suggests that the problem was not simply making a generally plausible picture. It was preserving concrete visual state across a transition. The diagnostic is descriptive and does not establish a causal bottleneck.
Text and speech had the strongest association of the three modality pairings. The comparison did not isolate the effect of backbone architecture, so the association does not establish a causal direction.
Longer narration did not mean closer wording
The study also found a mismatch between what some automated measures rewarded and what the reference stories contained. Reference narrations averaged 107 characters, while large vision-language models produced narrations averaging more than 200 characters. Those longer outputs received high language-model judge scores but only modest gains on BERTScore, a measure of overlap with the reference wording. The finding does not explain why judges favored longer answers, but it warns against treating one score as a complete measure of quality.
The top of the ranking was close
An independent human study involving 24 annotators broadly supported the automated comparisons across all 32 configurations and four evaluation tracks. Human and automated rankings were positively associated for text, image, speech and integrated performance. The Spearman correlations were 0.863 for text, 0.882 for image, 0.724 for speech, 0.884 for integrated evaluation and 0.945 for the four-track composite. Speech showed the weakest agreement, which makes fine distinctions in that track less secure.
The leading configuration also should not be treated as a clear winner over its closest rival. Its point estimate was 7.7491, but the paired difference was only 0.0196, with a 95% bootstrap interval from −0.0110 to 0.0504. It ranked first in 88.14% of bootstrap draws, yet the interval crossed zero, so the evidence did not support a firm ordering of the close pair.
What the benchmark leaves open
The benchmark contains 900 transitions from openly licensed children's books, and the comparison covers 32 configurations. That makes it a focused test of this task rather than a general verdict on every language, genre or real-world interaction. The comparison is descriptive, and the shared architecture issue means it cannot establish a causal advantage for orchestration.
The study's central question is whether a system can preserve context and realize a specified next narrative state across modalities. The findings leave open whether native any-to-any systems can achieve the same completeness and controllability, and whether visual continuity improves with better cross-modal planning.
The document is a preprint labeled arXiv:2609.37317v1 and dated 29 September 2026. Its findings are best read as an evaluation of the reported benchmark and methods, rather than a settled verdict on story-generating AI.
Paper data and sources
Original title: What Comes Next? Omni-StoryBench for Evaluating Story-Grounded Omnimodal Generation
Authors: Sieun Hyeon, Yejoon Lee, Mintaek Lim et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-29
DOI: Not available
Original paper · Full text