An external workflow guide was associated with higher policy-compliance scores for customer-service AI agents in a new benchmark comparison. Across airline, retail and telecom tasks, PolicyGuide had a mean Pass4 score of 0.62, compared with 0.42 for unguided execution.
The document is an arXiv v1 preprint dated 20 Aug 2026. It examines whether guiding an agent through a policy-required workflow can produce better benchmark outcomes than relying on an unguided actor or a safeguard that checks only the next action.
Keeping track of the whole request
The benchmark uses English customer-service tasks in three domains: airlines, retail and telecom. Each task asks an agent either to refuse a policy violation or to carry out a permitted change correctly. Success requires both the final database state and the required natural-language assertions to be correct.
PolicyGuide separates the preparation of policy rules from their use during a conversation. Policy requirements are compiled offline into a reusable workflow bundle. Online, the verifier reads the conversation, tool results and persisted request state, moves to the first unmet requirement, and returns remediation together with updated state.
In practical terms, the guide is designed to remember where a request stands in a sequence instead of treating every tool call as an isolated decision. The main comparison used ReAct, PolicyGuard and PolicyGuide on the same task IDs with a GPT 5.4 substrate. ToolGuard was included on Airline, and the main cells used four trials each.
Telecom showed the widest gap
Telecom showed the largest reported difference. Its Pass4 score changed from 0.19 for the comparator to 0.61 with PolicyGuide. Pass4 is a strict task-level measure: it equals one only when all four trials for a task succeed.
Pooling the three domains, the paper reports higher Pass4 for PolicyGuide than for ReAct and PolicyGuard. The reported significance levels were p < 10−8 against ReAct and p < 10−12 against PolicyGuard. For common-task comparisons, the analysis used 95% confidence intervals from paired bootstrap resampling with 10,000 task-level resamples.
A diagnostic comparison also examined the value of the workflow graph against a raw-policy guide. The reported overall Pass4 difference was 0.100 on Airline, 0.150 on Retail and 0.325 on Telecom. The paper describes these results as diagnostic rather than as a complete factorial analysis isolating every runtime component.
Tests beyond the final database state
The same workflows were evaluated with Claude Sonnet 4.6 and Gemini 2.5 Pro agents. That reported transfer across agent families suggests the workflow was not evaluated only with the agent family used for the main comparison, although it does not settle how other systems would perform.
The study also tested resistance to CRAFT attacks, which are adversarial attempts to make the agent violate policy. In the reported Airline evaluation, PolicyGuide had the lowest attack-success rate, or ASR@k, at every tested k. Its per-trial ASR was 0.087, compared with 0.125 for PolicyGuard and 0.200 for ReAct.
A separate Telecom audit looked at ordered traces, or whether the agent’s steps followed the required process. In that author-designed audit, PolicyGuide had a process-valid rate of 56.2%, compared with 17.5% for ReAct and 13.1% for PolicyGuard. It also led the audit’s two conditional diagnostics.
What the benchmark cannot answer
These are comparative benchmark findings, not a causal guarantee that the verifier will improve every customer-service system. The evaluation used a frozen user simulator across three English domains, so it does not establish performance with live customers, other languages or different policy regimes.
The Pass4 score checks the final database state and natural-language assertions. It does not by itself establish that every required step occurred in the correct order and at the correct time. The ordered-trace result therefore adds a diagnostic view of process following rather than replacing a comprehensive test of temporal compliance.
The CRAFT finding is limited to the reported Airline attack evaluation and does not establish robustness across domains or against adaptive attacks. Likewise, the transfer result shows that the same workflows were tested with Claude Sonnet 4.6 and Gemini 2.5 Pro, but it does not establish how workflows authored by other models or under other seeds would perform.
The trace audit was author-designed and exploratory, and its conditional diagnostics were reported only for traces that already passed the outcome check. That makes the result useful as supporting evidence about procedure-following, but not as a definitive independent measure of all policy-compliant behavior.
A promising benchmark result, with work still ahead
For teams building policy-compliant customer-service agents, the study points to a specific design choice: maintain state for the whole policy workflow rather than focusing only on the next proposed action. The reported results are consistent with that approach being especially useful when a task requires a longer ordered chain of steps, but the benchmark does not establish a universal advantage.
The study’s coverage remains deliberately narrow. It uses a frozen simulator, three English benchmark domains and four trials per multi-trial cell. The authors state that prompts and workflow schemas will be released to support reproducibility, creating a basis for testing the approach in additional settings.
The open questions are whether the findings hold with live users, other languages and policy regimes; whether workflows from other models or seeds transfer equally well; and whether broader intervention coverage can strengthen policy adherence without turning a benchmark improvement into an unsupported real-world guarantee.
Paper data and sources
Original title: PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents
Authors: Seongjae Kang, Taehyung Yu, Sung Ju Hwang
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text