A preprint reports a consistent pattern in benchmark tests: task-specific harnesses generated by a learned meta-agent were linked to higher AI-agent scores. Here, a harness is the software and control setup that shapes how a model tackles a job. Across all 18 directly matched backbone-benchmark pairs, the generated harness outperformed the default scaffold in the reported comparisons. The nine-benchmark average for GLM-5.2 rose from 74.1 to 81.8, a 7.7-point difference. For DeepSeek-V4-Flash, it rose from 66.7 to 75.5, an 8.8-point difference.
That result is a comparison, not a causal proof that the generated harness produced the gains. The analysis reports no confidence intervals, standard errors, p-values or other inferential uncertainty measures. It also does not report run-level variability, so the published averages do not show how stable the gaps would be under repeated runs.
A harness built around the task
The evaluation covered nine benchmarks grouped into four task types. The study asks whether a trained meta-agent can generate a task-specific harness on demand for execution by an agentic language model. Its four-module protocol separates history compression, local-intent formation, tool and skill orchestration, and control advancement. In plain terms, it determines what past information to retain, what the next local goal is, which tools or skills to coordinate, and when to move the task forward.
Training follows three stages. First, it teaches task-conditioned customization. It then turns failed generations into bounded repair trajectories. Finally, Evo-GDPO proposes harnesses that improve the archive frontier, while reward, latency and cost are normalized separately. JIT-Agent was trained based on Qwen3.6-27B and primarily instantiated with GLM-5.2 and DeepSeek-V4-Flash-Preview.
The gains held under tighter comparisons
In backbone-controlled comparisons, JIT-Agent achieved the highest performance in four of six settings. Against the strongest fixed harness in named examples, it gained 4.7 points on DeepSeek-V4-Flash's DeepSearchQA test and 4.0 points on its xBench-DS test. For Qwen3.6-Flash, the gains were 7.0 points on xBench-DS and 2.9 points on AgentIF. The preprint characterizes the generated harnesses as performance-competitive with OpenCode and Claude Code.
Token and cost numbers pointed in the same direction. JIT-Agent had the lowest token consumption and API cost in all six controlled settings. Relative to the cheapest fixed harness, the reported cost reduction ranged from 14.9% to 54.1%, averaging 36.0%. In the highlighted DeepSeek-V4-Flash xBench-DS example, token use fell from 527K to 212K and cost per case from $0.075 to $0.039, while the score rose from 78.0 to 82.0.
The preprint also reports favorable margins against GPT-5.6 in selected examples. With JIT-Agent, DeepSeek-V4-Flash exceeded GPT-5.6 by 9.1 points on DeepSearchQA and 4.3 points on OdysseyBench. GLM-5.2 gained up to 20.2 points. The reported comparisons are benchmark results within the tested configurations.
The pattern extended across model families
A broader comparison covered 24 matched backbone-benchmark comparisons against ReAct. JIT-generated harnesses were ahead by an average of 7.6 points. Average gains were 10.2 points for DeepSeek V4, 4.0 points for Qwen 3.6, and 8.6 points for Mimo 2.5.
The authors read this spread as evidence that harness intelligence can transfer across model families. The supplied analysis limits that conclusion to the models, benchmarks, harnesses and environments evaluated; transfer beyond them remains untested, and no independent replication estimates are reported.
A system that can change its route
In a separate test-time evolution comparison, streaming JIT finished with higher cumulative accuracy than static JIT on DeepPlanning-Shopping, DeepPlanning-Travel and OfficeBench. Cost and tool-call trajectories remained task-dependent, and the extracted results do not provide numeric endpoint values for those measures.
The visual examples show what task specialization looks like. For one task, the generator produced graph execution with an artifact store. For another, it produced recursive orchestration with a fact store. These examples are illustrative qualitative evidence, not a quantitative comparison of which execution design is best.
What remains untested
Important details needed to judge repeatability are not supplied. Exact task-case totals, per-benchmark run counts and run-level variability are not reported. The paper also says that production harnesses expose substantially richer mechanisms than the compact four-module instantiation tested here. That leaves open how the approach would behave on longer task streams or richer production runtimes.
The document is identified as arXiv version 1, dated 26 Aug 2026. It lists GitHub and Hugging Face links, but does not describe what artifacts are available or the conditions for access. Funding information is not reported in the supplied contributor material.
Paper data and sources
Original title: JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution
Authors: Guibin Zhang, Leo Lu, Fangzhou Xie et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text