Jev scored higher than Qwen3.8-27B on 27 of 37 public benchmark datasets and was above Gemma-4-E4B on all 37, according to an arXiv preprint. The lead was uneven, with performance changing sharply across tasks and languages.
The evaluation tested Jev zero-shot, meaning it received no examples in the prompt, using one frozen template for each dataset and full evaluation splits totaling 346,009 requests. Qwen3.8-27B and Gemma-4-E4B were scored on exactly the same requests, using next-token probabilities over answer-option codes. Thinking was disabled for the reference models, and they generated no text.
The comparison was not equally clear everywhere. Eleven of Jev's leads over Qwen had non-overlapping 95% bootstrap confidence intervals, while none of Qwen's leads did. The intervals were calculated over examples with 500 resamples.
Strong on familiar tasks, less steady at the edges
Jev was strongest on several short-text and commonsense tasks: accuracy was 96.5% on IMDB, 96.4% on SST-2, 95.5% on HellaSwag and 99.3% on ARC-Easy. Results were weaker on noisy, fine-grained or low-resource benchmarks, including Emotion at 58.5%, SST-5 at 57.9% and AfriXNLI at 64.0%.
One of the more useful findings concerned confidence. Across 22 Choice datasets and 279,925 answers, pooled expected calibration error, or ECE, was 0.028. ECE measures how closely a model's stated confidence matches its actual accuracy. The unweighted mean ECE across datasets was 0.061, while miscalibration was concentrated on lower-accuracy datasets: 0.279 for Emotion and 0.165 for AfriXNLI. Dataset accuracy and ECE had a Spearman correlation of negative 0.83.
The study also tested selective prediction, in which only the most confident answers are kept. At 50% coverage, Jev's accuracy was 96.3% on Banking77, compared with 79.7% across the full set; 86.9% on ANLI versus 73.9%; and 73.3% on Emotion versus 58.5%. These are benchmark results for a selected subset, not a measure of production performance.
Thresholds changed some of the scores
Thresholds also mattered. For binary decisions, the default cutoff was 0.5, while tuned thresholds used 1,000 training or validation examples before being applied unchanged to evaluation data. On Jev, that changed UNFAIR-ToS micro-F1 from 0.499 to 0.748, GoEmotions macro-F1 from 0.243 to 0.353, and ToxicChat jailbreak F1 from 0.717 to 0.833.
On rubric-scoring tasks, probability-weighted scores correlated strongly with human ratings for STS-B, with a Spearman correlation of 0.890, and for SST-5 and ToxiGen offensiveness, at 0.851 and 0.841. The relationship was weaker for HelpSteer2, at 0.399 for helpfulness and 0.369 for correctness, showing substantial variation across rubric dimensions.
The multilingual gap was stark
The largest gaps appeared across languages. Belebele accuracy was 97.2% for English but 37.4% for Nigerian Fulfulde. SIB-200 accuracy reached 19.6% for Santali in Ol Chiki, while AfriXNLI ranged from 33.8% to 70.3% across 16 African languages. The weakest results were concentrated in several low-resource settings.
A calculation result needs a closer look
A separate MMLU probe raised a different concern. Jev scored 94.3% on calculation-heavy subjects, compared with 91.3% on other subjects. Rotating the answer options left the first figure unchanged at 94.3%, but withholding the question reduced accuracy to 31.5%, close to chance. The probe cannot distinguish genuine solving from memorized question-answer pairs, so it does not settle why the calculation score was so high.
A bounded, reproducible comparison
The study reports a low-cost evaluation setup: 217.9 million input tokens for the main run, US$9.15 in cost, and an average 0.36 seconds per request. Including the pilot and additional analyses, the full study cost US$9.40. The reported open-model compute costs depend on hardware and implementation and are not directly comparable with API prices.
The comparison remains tightly bounded. Jev used one prompt template per dataset, with no prompt tuning, and each request was run once, so run-to-run variation was not measured. The reference models used prompts written for Jev, with thinking disabled and no generated reasoning. Binary decisions generally used a fixed 0.5 threshold, while the tuned thresholds used only 1,000 training or validation examples. The results therefore describe this benchmark setup rather than a general production advantage.
The authors published the benchmark definition, evaluation harness, analysis scripts and all model responses through a repository and Zenodo DOI 10.5281/zenodo.23039006. The document is an arXiv version 1 preprint dated 29 Sep 2026. The research was partially funded by Germany's Federal Ministry of Education and Research and the state of North-Rhine Westphalia through the Lamarr-Institute for Machine Learning and Artificial Intelligence.
Paper data and sources
Original title: Evaluating and Benchmarking the System One Model Jev
Authors: Tobias Deußer, Lorenz Sparrenberg, Rafet Sifa
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-29
DOI: Not available
Original paper · Full text