The main result from a computational benchmark is straightforward: the study reports that CoT+Ours achieved the best results in its main comparison. The benchmark was built to evaluate whether language models can induce explicit task-level strategies. But that result does not show that strategy-only solving was consistently better. The study reports that Ours alone did not always outperform CoT.
The benchmark behind the comparison
After filtering, the benchmark retained 44 tasks, 356 subtasks and 123,845 examples. The study evaluates whether models can induce explicit task-level strategies, and prompting-based inference used three few-shot examples.
The evaluation covered two sides of the task: the quality of the strategy itself and its downstream utility. Reported measures included correctness, sampling consistency, robustness consistency, input-length efficiency, output-length efficiency, conciseness and format compliance. Model answers were compared with gold answers using exact match for correctness.
For the strategy analysis, the study used two reference sets. The effective set, Seff, contained 22 tasks, 241 subtasks and 11,993 strategies. The statistically filtered set, Ssig, contained 21 tasks, 221 subtasks and 7,801 strategies.
The result changed across the tested setups
In the main comparison, the authors reported CoT+Ours as achieving the best results. The qualification is important: Ours alone did not always outperform CoT. The main-comparison result therefore does not amount to a consistent advantage for strategy-only solving.
The study also examined generator–executor pairings. The paper reports that scaling the executor to Qwen3-8B did not consistently improve correctness. In the reported comparison, the tested executor size did not produce a uniform correctness gain.
At 30 shots, the reported scores were 89.83 for conciseness, 14.58 for format compliance, 25.49 for correctness and 51.29 for sampling consistency. The paper presents this result as a quality–utility tradeoff, rather than an improvement on every measure.
Prompt format produced another split in the reported results. The Structured Prompt scores were 91.60, 22.79, 20.86 and 48.55, while the Free-form Prompt scores were 90.91, 57.38, 16.96 and 45.02. The two variants had different quality and utility profiles.
Fine-tuning improved stability, not correctness
Supervised fine-tuning was tested through the Ours+SFT variant. On the ID, OOD and BBH test sets, Ours+SFT improved conciseness and format compliance, and it also improved sampling consistency and robustness consistency across all three sets.
Correctness did not follow the same pattern. Ours+SFT did not improve Corr. and even led to a slight decrease. In the reported tests, higher strategy quality and stability measures were not accompanied by better single-run correctness.
A narrow test bed
The benchmark’s reach is narrower than the headline may suggest. It is mainly built from BIG-Bench and BBH, while the experiments use models from the Qwen3 family. The reported evidence therefore describes those task sources and that model family; it does not establish that the same pattern holds across other model families or broader task settings.
This is an arXiv preprint based on computational comparisons. Its results vary across the tested configurations: CoT+Ours led the main comparison, Ours alone did not always outperform CoT, scaling the executor to Qwen3-8B did not consistently improve correctness, and Ours+SFT did not improve Corr. The evidence is best read as a conditional report from the benchmark and models that were tested.
Paper data and sources
Original title: StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models
Authors: Jinghan Tan, Yuanzheng Wang, Lu Chen et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-24
DOI: Not available
Original paper · Full text