A benchmark of language-model agents found that strong performance on static enterprise questions did not reliably translate into the strongest results when tasks became interactive. Across Consulting cases, the Beer Game and Enterprise Digital Twin settings, the ranking of methods changed with both the task and the underlying language model.
A benchmark built around the jump from answers to decisions
EnterpriseBench combines a foundational enterprise and financial question-answering suite with three interactive settings: Consulting, Beer Game and Enterprise Digital Twin. Together, they assess both static capabilities and interactive decision-making, allowing the evaluation to test whether performance on questions carries over to tasks that require action.
The reported corpus contains 21,789 entries in all: 15,631 in training, 1,452 in validation and 4,706 in testing. The Consulting component contains 423 curated cases. The evaluation used nine agent methods with DeepSeek-V3 and GPT-4.1, and seven representative methods with DeepSeek-V4-Pro and GLM-5.2.
The winner depended on the model
The two simulation settings use different measures. Beer Game requires 25 sequential replenishment decisions and is scored by accumulated cost, with a lower cost better. Enterprise Digital Twin begins with scenario configuration and then a rollout, with accumulated earnings as its measure, so higher earnings are better.
That distinction showed up in Consulting. Under DeepSeek-V3, GEPA had the highest overall score of 8.28, with DC second at 8.00. Under GPT-4.1, DC led at 8.08 and GEPA was second at 7.89. The top two therefore switched places when the underlying model changed.
The pattern was not limited to Consulting. With DeepSeek-V3, Discussion had the lowest Beer Game cost, while AMEM had the highest Enterprise Digital Twin earnings at 6.76 million. With GPT-4.1, GEPA had the lowest Beer Game cost and the highest Enterprise Digital Twin result, at 8.16 million.
Harder questions exposed uneven skills
On the foundational questions, difficulty mattered differently by category. Information Extraction varied relatively little as tasks moved from easy to hard, while Numerical Calculation and Domain Knowledge showed sharp declines under both tested language models. The pattern points to uneven performance across capabilities.
The scoring checks were encouraging, within limits
The study also tested whether a language-model judge for Consulting tracked human assessments. In an initial audit of 20 cases, the reported Pearson correlation, a measure of how closely two sets of scores move together, was 0.71; the p-value was below 0.001, and the average gap from human scores was 8.5%. An expanded validation covering 100 cases and 500 transcripts reported a correlation of 0.887, with 85.0% of ratings within one point and a human inter-rater reliability statistic of 0.892.
Other checks focused on whether the benchmark's scoring and difficulty labels were stable. Across Consulting interviewer and judge prompt variants, the maximum change in the overall score was 0.12 points, or 1.45% relative to baseline. A stratified audit of 50 tasks found close agreement between language-model difficulty annotations and human ratings: the reported Pearson correlation was 0.96, the mean absolute difference was 0.47 points, and 94% of samples were within one point of the human score.
A small proof-of-concept test of adaptive routing produced a modest gap. AOA scored 0.729 on overall QA, compared with 0.725 for ACE; ExpertLLaMA scored 0.684 and Multi-expert Prompting 0.613. That result suggests task-specific selection may help in this setup, but it does not establish general superiority.
A benchmark, not a workplace
The benchmark's main caveat is that its interactive environments are controlled. The authors say they do not fully capture the social, organizational and institutional factors of real enterprise decision-making. The text-based benchmark omits multimodal evidence such as slides, spreadsheets, emails, dashboards and meeting transcripts, and its three interactive tasks do not cover all industries, organizational scales or decision types.
Paper data and sources
Original title: EnterpriseBench: Benchmarking LLM Agents on Enterprise-Level Strategic Reasoning and Decision-Making
Authors: Min Yang, Yichen Pan, Jinghua Piao et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-29
DOI: Not available
Original paper · Full text