A controlled benchmark of an AI coding agent found that a bare user story was associated with a substantially higher bill than a full task specification. Retaining only the user story was linked to 29.7% higher list-price cost and 16.4% more agent turns. The estimated solve-rate difference was negative 1.9 percentage points, with a reported uncertainty range from negative 7.4 to positive 3.6 points. The cost gap was therefore clearer than any difference in whether the task was solved.
The test covered 2,700 controlled runs. It paired five coding tasks with 12 prompt specifications and three thinking-effort settings, repeating every combination 15 times. Kimi K3 ran through Modal at temperature 1.0, using mini-swe-agent in the standard SWE-bench Docker image. The setup had no network access and included screening for solution leakage.
Researchers recorded each run's list-price cost in US dollars, input, cached and output token counts, number of agent turns, and whether the patch was resolved under the SWE-bench harness. To summarize the pattern across tasks, they fit one Bayesian hierarchical model for each prompt section and outcome. That approach estimated a typical effect and how much tasks disagreed, with posterior medians and 90% credible intervals.
The cost gap depended on the task
The result changed depending on which part of the specification was removed. Removing the acceptance-scenarios section was associated with 7.0% more turns than the full specification. Its solve-rate estimate was negative 0.2 points, with a 90% credible interval from negative 5.6 to positive 5.1 points, leaving the effect on solve rate uncertain.
The size of the cost difference also depended sharply on the task. For the bare-user-story cut, the five task-level differences were 13%, 16%, 21%, 24% and 115%. By contrast, the cost effect associated with removing acceptance scenarios was positive on four of the five tasks and ranged from negative 8% to positive 53%. The 115% gap shows how an average across tasks can hide a large workload-specific swing.
Thinking effort added another layer. From low to high to maximum effort, the ratio between the most and least expensive specifications fell from 2.13 times to 1.67 times and then 1.61 times. At the same time, geometric-mean cost rose from $0.117 to $0.561 per run. The acceptance-scenario turn difference fell from 20.1% at low effort to 4.5% at high effort and 2.1% at maximum effort. Specification had no credible effect on solve rate in this comparison. Four of the five tasks had solve rates of at least 98%, while django-15503 stood at 86.3%.
Repeated runs were variable, but the prompt cuts did not measurably change that variability. The median geometric standard deviation, a measure of multiplicative run-to-run spread, was 1.34 times overall and ranged from 1.29 to 1.40 across effort settings. Absolute spread tracked mean cost closely, with a spread-to-cost slope of 1.08 and a correlation of 0.95. In this test, more expensive settings showed larger absolute swings, while prompt variation itself was not a measurable source of extra spread.
In this test, the bill was not determined simply by how many tokens appeared in a run. Under the tested Kimi K3 prices and cache conditions, output tokens accounted for 2.7% of processed tokens but 51.1% of the dollars. Fresh input represented 13.4% of spending, alongside a 96.3% cache hit rate. Those shares are specific to the price schedule and caching behavior used in the test.
Testing a forecast for an unseen task
The researchers also tested a way to estimate the cost distribution for a task held out from calibration. In a leave-one-task-out design, they learned a shared prompt-by-effort cost shape from four tasks, held out the fifth, and evaluated 160 settings across five folds. The five-fold exercise used 400 independent probe draws.
Without a probe, the median prediction error was 161%. One probe, costing $0.11, was associated with a median error of 36% and put 67% of settings within a plus-or-minus 50% error band. Ten probes cost $1.13, and the median error was 25%, with 76% of settings within the same band. The largest change was between zero and one probe.
For individual runs within a single thinking level, the correlation between predicted and actual cost was only 0.08 without a probe, with a reported interval from 0.04 to 0.12. After one probe, it was 0.72, with an interval from 0.67 to 0.78. The authors identify the use of a single model, the sparse task sample and the predictor's omission of task-specific prompt sensitivity as key limits on generalization. The results describe a narrow tested configuration, not a universal price rule.
Paper data and sources
Original title: Can your AI agent be cheaper? Investigating the effects of task specifications on token spend in agentic coding tasks
Authors: Jakub Smékal
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text