A preprint reports that a model fine-tuned for Russian 1C:Enterprise and BSL code retrieval ranked first on a 3,413-query benchmark, with a balanced macro nDCG@10 of 0.5992. The paper’s main ranking measure, nDCG@10, looks at how well relevant code is placed among the first 10 results.
The system, called PruhaNLP/USER2-1C-code, is a bi-encoder: it turns the question and the code document into separate representations and compares them. It was built by fine-tuning deepvk/USER2-base with asymmetric query and document prompts. Training used 784,057 synthetic triplets, Matryoshka Representation Learning and a privacy-aware tokenizer; a separate retrieval step selected hard negatives from the top 20 candidates.
A test built from real questions
The benchmark was assembled from real 1C questions and solutions rather than synthetic questions. Its source contained 19,041 records; the final set held 2,883 forum queries and 530 fastcode queries. The PII-scrubbed benchmark uses a closed set with one binary gold document marked for each query.
The model’s nDCG@10 was 0.4617 on forum and 0.7366 on fastcode. Giving the two subsets equal weight produced the 0.5992 macro score, while weighting queries by their actual counts produced a query-weighted micro score of 0.5044.
Where the score held up
In paired-bootstrap comparisons, the proposed model’s balanced macro score was 0.106 higher than USER2-base, with a 95% confidence interval of 0.087 to 0.125. It was 0.0588 higher than EmbeddingGemma, with a confidence interval of 0.042 to 0.075. The reported one-sided P value was below 0.001 for both comparisons.
Adding a lexical search method did not improve the dense rankings in the tested setup. BM25Okapi lagged the stronger dense systems, while untuned reciprocal-rank fusion, or RRF, lowered the proposed model’s macro score from 0.5992 to 0.4300. The same combination lowered USER2-base from 0.4932 to 0.3908; only that untuned RRF configuration was evaluated.
The authors also audited exact matches and 13-gram overlap between the training and benchmark data. On the audit-clean subsets, the model scored 0.6011 balanced macro and 0.5010 micro nDCG@10. Clean fastcode scored 0.7369, compared with 0.7366 before exclusions. The audit does not address semantic paraphrase overlap.
Smaller vectors, larger caveats
A second result concerns storage. Truncating the model’s embeddings from 768 to 256 dimensions retained 99.9% of the full-dimensional nDCG@10 quality and reduced the index to one-third its size. The paper describes a theoretical 3× speedup for exact scoring, but it did not measure end-to-end retrieval latency.
The benchmark’s single-gold design leaves room for false negatives: code fragments that may be relevant but are not marked as the one correct answer. In a manual check of 50 queries—25 from each subset, using seed 42—five cases, or 10%, contained multiple relevant fragments. The study did not estimate the overall false-negative rate.
The paper’s practical offering is a public package comprising the benchmark, a CLI/Python evaluation harness, PII-scrubbed training triplets and an MRL reference bi-encoder. Its results remain an offline ranking evaluation on these corpora; they do not measure human search success, developer productivity or deployed-system performance.
The manuscript is an arXiv version-1 preprint dated 20 Aug 2026. It does not report a funding source, and the authors declare no relevant competing interests.
Paper data and sources
Original title: Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder
Authors: Konstantin Chesnokov, Chingiz Mingazov
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text