BreakGuard, a method that uses a large language model to generate migration tests, detected 27 of 89 dependency-breaking updates in its best reported setup—a rate of 30.3%. The configuration used GPT-4o with Class context and was the best configuration reported for detection.
The result came with a major validity problem. Compilation was the main reported failure among invalid tests, averaging 79.7% across configurations and ranging from 66% to 98%; even the best configuration had 66.5% noncompiling tests.
Inside the test
BreakGuard statically scans a client project for target-library call sites, groups them by their enclosing “focal method”—the method containing the call—and generates one test file for each method.
Criterion-based filtering left 89 BUMP breaking-update instances with feasible Docker environments, supported test runners and direct library call sites in production code, spanning 31 client projects and 25 open-source libraries.
Across those instances, static analysis identified 5,790 distinct focal methods. The experiment compared three LLMs—GPT-4o, Qwen3-coder-480B and GPT-OSS-120B—at three context levels: Minimal, Method and Class. The labels refer to how much surrounding client code the model received.
Generated tests were compiled first; only those that compiled were then executed against the library version before the update and the breaking version. A change counted as detected when at least one focal method had a valid test on the pre-breaking version but failed on the breaking version.
In this study, a valid test had to compile, run successfully and pass its assertions on the pre-breaking version. Qwen3-coder with Class context produced valid tests for 22.4% of focal methods in a single shot and covered 34 of the 89 BUMP instances.
The catches were mostly crashes
With GPT-4o and Class context, detections covered five library categories—JSON libraries, Logging, parser generators, Maven plugins and HTTP clients—but were concentrated in the Logging API and JSON libraries.
Among 3,566 tests that detected a break, 3,390, or 95.1%, ended as Maven Surefire ERROR outcomes and 176, or 4.9%, ended in FAILURE. The reported root causes were dominated by missing classes or methods: NoClassDefFoundError appeared in 1,441 cases, NoSuchMethodError in 984 and ClassNotFoundException in 965.
Only 25 tests checked returned values, while 136 mainly checked whether execution threw an exception. Value-based tests accounted for 0.7% of all detecting tests. The authors therefore describe BreakGuard as most useful for crash-type breaking changes; the findings do not show that behavioral changes are absent.
Why some breaks went unnoticed
Among 2,111 tests from BUMP instances where detection failed entirely, 1,656, or 78.4%, never loaded the broken API class. The remaining 455 tests, or 21.6%, loaded it but passed.
A manual inspection of 80 of those 455 tests found three patterns: dependency context allowed correct-input tests to pass in 37 cases, catch-block suppression appeared in 32, and inputs failed to exercise the change in 11.
The price of more context
Across configurations, mean cost per BUMP instance ranged from $0.005 to $0.90, while median cost ranged from $0.001 to $0.088.
Class context required roughly three to four times as many tokens as Minimal context. For GPT-4o, total use increased from 1,119 tokens per focal method with Minimal context to 4,321 with Class context.
The authors interpret GPT-4o with Class context as offering the best detection-cost trade-off among the configurations tested.
A result with narrow edges
The sample was criterion-filtered to cases with feasible Docker environments, supported test runners and direct production-code calls to the library. It therefore excluded some forms of use, including reflection, dynamic loading and access only through a transitive dependency.
The model and context comparisons were descriptive rather than randomized or inferential, so the study does not establish that Class context itself caused the higher detection rate. The paper reported no inferential tests or confidence intervals for its main performance estimates.
The evaluation used one single-shot response per focal method, without feedback or repair. The authors regard the reported detection rates as lower bounds, while results outside the filtered Java/Maven setting remain uncertain.
Paper data and sources
Original title: BreakGuard: Towards Detecting Dependency Breaking Changes with LLM-Generated Tests
Authors: Rachna Raj, Benoit Baudry, Diego Elias Costa
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text