An arXiv preprint testing Qwen2.5-Instruct models from 0.5B to 7B on the 1,034-example Spider development set found that spending more inference compute on decoding usually improved smaller models, but generally did not make them match larger ones.
The search strategy mattered
Researchers compared grammar-constrained decoding, which restricts generated SQL to permitted forms, using two ways to spend the inference budget: beam search, which keeps several candidate queries in play, and sample-and-vote, which draws several outputs and uses their agreement. The budgets were 1, 2, 4 and 8, sample-and-vote used a temperature of 0.7, and the models ran at 4-bit precision.
Every configuration used the same full Spider development set of 1,034 examples. Execution accuracy counted a generated query as correct when it returned the same result set as the reference query. The analysis compared paired predictions with an exact McNemar test, a test for paired results, and used 95% Wilson score intervals for the accuracy estimates.
Beam search had the clearest edge. In 16 matched-budget comparisons, it was significantly ahead of sample-and-vote 11 times and significantly behind zero times, using exact McNemar tests on paired predictions.
Compute narrowed the gap, but did not erase it
Extra search helped most at smaller sizes. At 1.5B, widening the beam added about 15 percentage points of execution accuracy; at 7B, the gain was about 5 points. The shrinking returns suggest that more decoding effort became less valuable as the model grew, but the comparison was confined to this benchmark.
Model size remained difficult to replace. Giving a smaller model eight times more inference compute did not compensate for downgrades from 7B to 3B or from 1.5B to 0.5B. For the tested 3B-to-1.5B downgrade, a twofold increase appeared sufficient.
At the eight-beam budget, the extra computation closed between 50% and 76% of the accuracy gap to the next larger model. Only one of the three model-size steps was fully bridged: the move from 1.5B to 3B.
The grammar was not neutral
The unconstrained baseline was higher in both tested size comparisons. At 3B, unconstrained greedy decoding reached 47.7% execution accuracy versus 44.5% for grammar-constrained decoding at budget 1; at 7B, the figures were 64.3% and 60.0%.
The grammar also rejected outputs that could otherwise be valid. At 3B, all 400 testable differing predictions in the detailed comparison were rejected by the grammar, including predictions that an independent SQLite parser judged correct. That finding indicates incomplete coverage of valid SQL forms.
A result with a narrow reach
The evidence has a narrow reach. It covers one model family, one benchmark and one decoding run per configuration; sample-and-vote used a single random seed, so repeated-run variability was not analyzed.
These results do not show that beam search will beat sample-and-vote in unconstrained reasoning or in other constrained tasks, or that extra compute can never compensate for a smaller model. One of the three tested downgrades appeared compensable.
Paper data and sources
Original title: Beam Search, Self-Consistency, and the Limits of Inference-Time Scaling for Grammar-Constrained Text-to-SQL in Small Language Models
Authors: Ty Chermsirivatana, John MacCormick
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text