MACGen, a four-agent code-generation system, reported higher average F&S@1 scores than Direct prompting by 19.61 percentage points on CWEval and 10.57 percentage points on BaxBench. On CWEval, however, its average Func@1 score fell by 0.56 percentage points while its average Sec@1 score rose by 23.82 percentage points.
The paper tracks three primary measures: Func@1 for functionality, Sec@1 for security, and F&S@1 for their combined result. Func@1 and F&S@1 use all generated outputs, whereas Sec@1 uses compilable outputs only.
A closer look at the combined score
CWEval's main evaluation comprised 119 tasks across five languages. MACGen was the best overall reported method across all six evaluated LLMs, reaching 70.59% F&S@1.
Averaged over GPT-4o and GPT-4o-mini, MACGen achieved the best F&S@1 across all five CWEval languages.
Results across the benchmarks
BaxBench covered 392 tasks across six languages. MACGen had the highest reported F&S@1 on both GPT-4o and GPT-4o-mini. For GPT-4o, it reached 31.12%, compared with 21.17% for Direct prompting and 30.36% for INDICT.
The authors also evaluated HumanEval and HumanEval+, where MACGen maintained performance comparable to Direct across all evaluated LLMs. Some models showed improvements in those comparisons, according to the paper.
On LLMSecEval, MACGen had the highest reported Sec@1 across all evaluated models. The benchmark contains 150 natural-language programming tasks in Python and C, but functional test oracles are absent, so this result is security-only static-analysis evidence. It does not establish that the outputs were both functional and secure.
The design was tested in pieces
MACGen divides secure code generation into planning, security analysis, code synthesis and refinement, handled by four specialized agents. They coordinate through structured, artifact-only interfaces. The coordination comparison used a version called MACGen-Shared.
MACGen consistently outperformed MACGen-Shared across the reported benchmarks and models. The reported F&S@1 differences were +20.17% for GPT-4o-mini on CWEval and +8.67% for GPT-4o on BaxBench.
In the GPT-4o CWEval ablation, the full configuration reached 70.59% F&S@1, compared with 50.42% when the Planner, Security Advisor and Reviewer were all absent. Removing standards-driven generation, code-based generation or validation degraded F&S@1 in both evaluated model ablations. Leaving out either generation step favored security at the expense of functionality.
A narrow cost comparison
Across 25 Python CWEval tasks, MACGen used 220,916 total tokens, compared with 1,411,754 for INDICT. Estimated API cost was $0.91 versus $4.68, and average end-to-end latency was 28.34 seconds versus 213.44 seconds. The paper reports that MACGen's cost was $3.77 lower and its token use was 16% of INDICT's.
Because the comparison covered 25 Python CWEval tasks, these figures describe that test rather than every possible use of the systems.
Preprint status and planned materials
The document is an arXiv preprint, version 1, dated 26 August 2026. No journal venue is reported.
The paper says it plans to release code, evaluation scripts, prompt templates and MACGen guideline-generation procedures for reproducibility and transparency.
Funding and conflicts of interest are not reported in the supplied paper text.
Paper data and sources
Original title: MACGen: Toward Functionally Correct and Secure Code Generation via Multi-Agent Collaboration
Authors: Miseon Yu, Jaehoon Choi, Younghan Lee, Yunheung Paek
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text