The benchmark gap
A local language-model system for building collider-simulation workflows had its highest descriptive benchmark pass rate when the full guarded agent was used. In a matched comparison of 61 cases, direct native generation passed 2 cases, or 3.3%. A structured approach that first converted the request into a formal description and then built the workflow passed 8, or 13.1%. The full guarded agent passed 52, or 85.2%, after fixed representation normalization.
The evaluation centered on a practical design question: is it useful to separate a language model’s interpretation of a request from the deterministic construction and execution of the workflow? In the system described, deterministic builders construct the execution artifacts, which are shown for inspection before approval. Approval does not trigger another planning call. The setup is therefore an inspectable, approval-gated constructor, with the comparison focused on what that pipeline produces.
The primary paired comparison planned 60 model-request cases. It obtained usable responses for 47, while 13 were classed as infrastructure-unavailable. In those 47 matched request units, the first structured proposal passed 7 cases, or 14.9%, without modification. The full pipeline passed 19, or 40.4%. After fixed representation normalization, the counts were 11, or 23.4%, for the first proposal and 43, or 91.5%, for the full pipeline.
The jump between the two scoring views needs care. The paper describes the gap as a bookkeeping mismatch involving the launch form, two fixed control lines and the name of the output directory. It says the mismatch was not a difference in physics content. Normalization changed how the benchmark counted the workflows, not the physics content described in the comparison.
The boundary problem
An extended evaluation planned 160 cases and had 158 usable model responses. It comprised 62 valid-supported cases and 96 cases expected not to proceed to an executable workflow. Among the 62 valid-supported cases, the requested physics was preserved in 57, or 91.9%, and there were no false rejections. Five failures, however, reached approval without preserving the requested physics.
The sharper warning came from the 96 problematic requests. The system stopped 39 before approval, but 57 reached approval in modified form. The evaluation conservatively counted those 57 as false acceptances, equal to 59.4% of the set. That is a stress-test result, not an estimate of ordinary-use failure: the challenge set was designed to press on difficult boundary cases.
The review identified several mechanisms behind those modified approvals. A sanitized remainder was retained in 16 cases, and 15 scans were reduced to a single run. Contradictions went undetected in 11 cases. Eight ambiguous requests received defaults, four malformed-token cases were reinterpreted, and three involved resource limits handled at warning level.
The primary paired study did not use model-guided repair in any unit. Deterministic corrections were applied in 36 of the 47 matched units. The comparison therefore reflects the pipeline as configured for the study, while the effect of adding model-guided repair was not measured.
What the runtime checks showed
Runtime checks offered a small confirmation that some constructed workflows could complete. Four of seven tested workflows finished normally. In repeated pairs, Drell-Yan cross sections were 844.3 and 841.7 pb, while top-pair results were 505.7 and 504.1 pb; each pair agreed to 0.3%. The remaining tests exposed practical failures: one larger top-pair sample timed out, and two Higgs-plus-jet examples failed during MadGraph generation.
These runtime checks do not establish scientific validation. They show completion and repeatability in a small subset, while technical failures remained. Within the tested scope, the system’s design combines deterministic construction with an approval gate and puts the resulting artifacts up for inspection. The boundary tests show why boundary handling still needs close attention.
Taken together, the descriptive results support the usefulness of separating language-model interpretation from deterministic construction for the bounded workflows tested here. But a system can score strongly on normalized construction tasks and still let altered, problematic requests reach approval. The runtime agreement adds evidence of repeatability in the tested examples, not scientific validation.
Paper data and sources
Original title: HEPLocalAgent 1.0: Running Collider Simulations from Plain-Language Requests on Your Own Computer
Authors: Aadarsh Singh, Sudhir K. Vempati
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text