Preprint competition paper flags contamination risks in LLM evaluation

A new arXiv paper says benchmark contamination can materially distort large language model evaluation, especially for software engineering tasks, and argues that current auditing methods are not yet reliable enough to certify benchmark integrity.

The report, titled The Poisoned Chalice of LLM Evaluation Report, was posted on July 8 and describes what it calls the first Poisoned Chalice of LLM Evaluation Competition. The competition was co-located with the FSE-AIWare 2026 Competition Track in Montreal, Canada, from July 5 to July 9.

What the paper claims

The authors frame contamination detection as a white-box membership inference problem on source code. In that setup, participants were given curated member and non-member datasets, target models, and baseline attacks, then evaluated on a held-out model and dataset.

According to the paper, the goal was to reduce dependence on superficial dataset artifacts and test whether methods generalize beyond a single training setting.

The report says benchmark contamination can inflate apparent model performance and lead to misleading conclusions about capability. It also argues that many modern models cannot be reliably decontaminated because training corpora are only partly disclosed.

Why it matters

The stakes are highest for software engineering benchmarks, where teams may use evaluation results to judge coding ability, compare models, or make deployment decisions. If a model has already seen benchmark material, the resulting score may overstate real-world performance.

The paper’s broader point is that statistical contamination checks can help, but they do not yet replace transparent data provenance. That aligns with a separate June 2026 paper that reported reliability gaps in benchmark auditing and warned that detection methods can fail under realistic conditions.

Background from the same research group

The same research group previously released The Heap, a contamination-free multilingual code dataset covering 57 programming languages. That earlier work was aimed at making evaluation cleaner by reducing the risk that training data and test data overlap.

The new report sits in that same line of research, but shifts from dataset construction to attack-and-audit evaluation. Instead of assuming contamination can be measured cleanly, it tests how far current methods can go in a competition setting.

What to watch next

The paper is a research report, not a policy change or product launch. The most immediate follow-up questions are whether the competition releases code or datasets, whether a companion proceedings entry adds formal publication context, and whether other researchers find the methods generalize beyond this setup.

For now, the paper’s practical message is straightforward: contamination remains a real risk in LLM evaluation, and the field still lacks a robust way to prove a benchmark is clean.