An identifier-renaming attack was associated with a sharp change in code-search rankings in a new preprint, which reports an absolute drop in Mean Reciprocal Rank (MRR) of up to 77% across all tested models on the CosQA benchmark. The study evaluates a method intended to align a snippet with a target query without changing its functionality. It changes user-defined names while keeping the code’s parsed structure unchanged. MRR is the study’s retrieval-ranking measure, so the reported drop represents a worsening in where matching code appeared in that benchmark.
How the names were changed
The attack replaced user-defined identifiers and used gradient information from a white-box surrogate model to choose replacements. It also preserved identifier style and used similarity between query tokens and candidate names as part of that selection. In ordinary language, the method steered a snippet’s visible vocabulary toward a search request while keeping its structure fixed.
The threat model gives the attacker full white-box access to the surrogate embedding model and assumes that modified code is injected during corpus indexing to affect later retrieval. The method constrains the changes to preserve execution logic and keep the abstract syntax tree, or AST—the parsed map of a program’s structure—unchanged before and after the attack.
Similarity moved between models
Main effectiveness and transfer tests used 20,000 sampled query–code pairs from CosQA and CLARC. The attacks used CodeT5+ or OASIS as surrogates, and the default pairwise protocol ran for five iterations. A surrogate is the model used to guide the attack; transfer was assessed by applying the resulting changes across surrogate/evaluation model pairs.
On the surrogate models, query–code similarity rose for more than 97% of examples, with generally larger gains for CodeT5+ than for OASIS. Across surrogate/evaluation pairs, similarity rose in more than 95% of cases, and gains on the surrogate generally tracked gains after transfer to other models. The paper reports means and standard deviations for these changes, but no confidence intervals or formal significance tests.
Against the listed baselines, the method showed larger ∆Sim values, where ∆Sim means the change in query–code similarity. In the white-box comparison with DAMP, ∆Sim was 18.66 ± 5.52 versus 10.45 ± 5.48, with 47 reported GPU minutes versus 81. In the black-box comparison with CodeAttack, ∆Sim was 28.03 ± 8.29 versus 5.18 ± 4.18, with 1,000 API calls versus 10.8 million.
From similarity to retrieval
In a separate shared-corpus setting, 20 attacked snippets represented 4% of the corpus. With those 20 snippets, reported MRR drops were 38.2 for CodeT5+, 11.9 for OASIS and 14.4 for Nomic. With five attacked snippets, the corresponding drops were 4.4, 4.3 and 5.8.
Transfer reached other models and languages
The transfer also appeared in tests of large language models. On RepoQA, Full Attack accuracy was lower than Original for both evaluated LLMs in every listed language. For Python and C++, GPT-5.4-mini recorded 81% and 70% under Full Attack, compared with 96% and 92% on Original; Gemini-3.1-Pro recorded 79% and 72%, compared with 95% and 92%.
Cross-language analysis used HumanEval-X to form 26,896 query–code pairs from 164 queries with solutions in Python, C++, Java, JavaScript and Go. The paper reported effective, transferable attacks across all five languages.
A defense trade-off
The defense question focused on fine-tuning (FT), an additional training step used here to produce robust-only and mixed checkpoints. On three retrieval measures—MRR, NDCG and Recall@5—Robust-Only FT had values of 34.1, 37.6 and 42.6, compared with 72.4, 77.3 and 88.0 for the pretrain baseline. Mixed FT scored 72.3, 77.6 and 89.4 on the same measures. The numbers show the robust-only checkpoint performing far below the baseline, while the mixed checkpoint remained close to it.
An adaptive test complicated that picture. Under direct white-box attacks on the fine-tuned checkpoints, CodeT5+ ∆Sim was 7.14 for Robust-Only FT and 34.70 for Mixed FT. Transferred attacks on Nomic gave 6.57 and 29.93, respectively. The contrast is the trade-off reported in the paper: the checkpoint with lower retrieval values also showed the smaller direct attack similarity change, while the mixed checkpoint stayed near baseline on retrieval and showed a larger change.
Ablation tests—comparisons made after removing one component—gave lower scores in some of the reduced settings. Without gradient guidance, ∆Sim was 30.91 for CodeT5+ and 35.35 for Nomic, versus 38.86 and 39.25 for the full method. Without query-token similarity, Nomic’s ∆Sim was 22.68. Removing style preservation produced values close to the full method: 39.48 for CodeT5+ and 39.95 for Nomic. The paper reports means and standard deviations for these comparisons but no significance tests.
What remains unclear
Why the attack transfers remains unresolved. The authors say the causes of cross-model transfer are unclear, and that limited public information about pretraining prevents a definitive explanation.
Reproduction is also limited because the attack code is not publicly released; the authors invite researchers to contact the first author for implementation details.
The document identifies itself as arXiv:2608.26031v1, dated 26 August 2026, and remains a preprint.
Paper data and sources
Original title: Vulnerable Code Search: Transferable Attack for Code Language Models
Authors: Kaicheng Wang, Liyan Huang, Jesse Thomason, Weihang Wang
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text