The preprint reports that BERT-LER, an AI model for coded electronic health records, produced higher reported scores than CLMBR on five listed laboratory tasks in the EHRShot benchmark. Across the study’s asthma tasks, it recorded the strongest reported results.
The authors caution that laboratory ordering can reflect the care setting, clinical concern or acuity, and that correlated tokens can share attribution. The explanations were assessed qualitatively, so the findings should not be read as evidence of causal risk factors or clinical benefit.
A model that records more than the test name
BERT-LER is a BERT-style masked-language-model encoder that reads sequences of coded health-record events, including laboratory-code tokens and percentile-based value information. Each laboratory result is placed into one of 10 percentile categories, using the entire TriNetX Dataworks dataset as the reference population.
The study applies Integrated Gradients after prediction to assign attribution scores to individual tokens. This post-hoc method leaves the model’s parameters and predictions unchanged, making the scores an interpretation of the model rather than an additional prediction step.
Pretraining and the custom benchmark used de-identified TriNetX Dataworks records for approximately 75 million patients. EHRShot is a public Stanford Medicine dataset with 6,739 patients and 14 prediction tasks.
Higher scores on the reported tests
On the EHRShot laboratory tasks, BERT-LER’s ROC-AUC point estimates were 0.910 for anemia, 0.79 for hyperkalemia, 0.77 for hypoglycemia, 0.867 for hyponatremia and 0.808 for thrombocytopenia. The corresponding CLMBR estimates were 0.891, 0.77, 0.75, 0.733 and 0.783.
In the custom asthma evaluation, BERT-LER recorded PR-AUC/ROC-AUC values of 0.599/0.682 for loss of control, 0.627/0.667 for multiclass disease-state prediction and 0.232/0.724 for exacerbation. These were the highest reported values across the asthma tasks.
Removing the percentile-value embeddings lowered the point estimates on every asthma task and metric. For loss of control, the full model scored 0.599 PR-AUC and 0.682 ROC-AUC, compared with 0.586 and 0.676 without those embeddings; for the multiclass task, the figures were 0.627 and 0.667 versus 0.609 and 0.649.
For exacerbation, the full model scored 0.232 PR-AUC and 0.724 ROC-AUC, compared with 0.213 and 0.706 for the version without percentile-value embeddings.
How the asthma evaluation was set up
The custom asthma cohort was defined by mild-to-moderate asthma, a primary-position asthma diagnosis and GINA Step 2 through Step 4 therapy during a 365-day baseline, followed by a fixed 365-day follow-up.
Each asthma task used a 70/15/15 patient-level train, validation and test split. Loss-of-control and multiclass prediction each had 19,575 training, 4,143 validation and 4,108 test patients; exacerbation had 50,438 training, 10,748 validation and 10,859 test patients.
Training used stratified sampling and early stopping. EHRShot was evaluated with ROC-AUC, while the asthma tasks were evaluated with both ROC-AUC and PR-AUC; 95% confidence intervals were estimated from 1,000 bootstrap resamples.
Explanations remain an open question
Integrated Gradients attributions for anemia-related results and asthma loss of control were reported as largely consistent with expected risk-factor patterns and comparator models. That alignment describes model behavior; it does not establish that those factors cause an outcome.
The study does not formally test whether the explanations are faithful to the model’s reasoning or stable across similar records. The authors note that attribution can be distributed across correlated tokens, while laboratory ordering may itself reflect care processes and clinical acuity.
Still an early model evaluation
The work is identified as an arXiv version 1 preprint dated 20 August 2026.
The paper reports that its training package and explainability software are available through GitHub repositories.
Paper data and sources
Original title: Explainable Transformer Models for Clinical Prediction Tasks on Structured Electronic Health Records
Authors: Jun Ni Du, Lukas Adamek, Maxim Kryukov et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text