A preprint reports that protein-language-model embeddings appear to encode biochemical patterns linked to fitness prediction. When selected biochemical features were projected out of those embeddings, a downstream classifier lost predictive performance, while a projection based on randomized feature subspaces left average Matthews correlation coefficient, or MCC, essentially unchanged. In the 650M-parameter ESM-1v model, for example, the MCC score fell from 0.51 to 0.37 on the Expression task after linear projection.
How the test was built
The analysis used a filtered subset of ProteinGym deep-mutational-scanning assays for proteins from the human proteome. It covered 113,000 mutated sequences for Organismal Fitness, 69,000 for Activity, 51,000 for Expression and 18,000 for Binding. The groups came from 24, 15, 13 and 6 reference wild-types, respectively, and had positive-to-negative class ratios of 1.28, 1.54, 1.63 and 1.23. The study used stratified five-fold cross-validation. Stability was excluded because only 796 samples remained after filtering.
Feature selection used L1-regularized logistic regression and the 1-SE rule, which kept the sparsest model within one standard error of the best one. The selection step was nested inside the outer cross-validation. On average, it retained 204, 249, 260 and 193 single-amino-acid features for Organismal Fitness, Activity, Expression and Binding, or 53%, 64%, 67% and 50%. Grouping features by chemical properties retained 53, 43, 45 and 30 features, or 80%, 64%, 67% and 44%.
The researchers then used orthogonal projection, a mathematical way to remove the part of an embedding aligned with selected inputs. They projected embeddings against the selected tabular features to test linear effects, then against a learned representation generated by a multilayer perceptron, or MLP, to test higher-order and interaction effects. The downstream model was deliberately simple: L2-regularized logistic regression. MCC measured predictive performance and AUC selected the regularization strength.
What disappeared after projection
The baseline comparisons pointed to a different conclusion from the ablation. The largest embeddings, ESM-1v 650M and ESM-2 650M, performed best, while adding tabular inputs produced little or no additional performance. Those inputs described biochemical substitutions and predicted wild-type structure. ESM-1v and ESM-2 embeddings were mean-pooled to the sequence level, with ESM-2 versions at 35 million, 150 million and 650 million parameters; the downstream input was each variant's embedding difference from its wild-type sequence. For ESM-1v 650M, average MCC was 0.39 for Organismal Fitness, 0.47 for Activity, 0.51 for Expression and 0.41 for Binding.
A conditional-independence test asked whether one input type contained information beyond the other. Embeddings added information beyond tabular features in 15 of 16 cases, with ESM-2 35M for Activity the exception. Tabular features added information beyond embeddings in four cases: Activity for three models and Expression for ESM-2 35M. The analysis reported p-values rather than effect sizes, and the document gives inconsistent Bonferroni cutoff values, leaving the exact significance threshold unclear.
Direct projection left ESM-1v 650M at MCC values of 0.30, 0.34, 0.37 and 0.31 for Organismal Fitness, Activity, Expression and Binding, respectively. The corresponding losses from the unprojected scores were 0.09, 0.13, 0.14 and 0.10. The authors report that these declines exceeded fold-level standard deviations, although confidence intervals were not reported.
Projection against the MLP representation produced larger losses than direct linear projection. For ESM-1v 650M, the decreases across the four function types were 0.13, 0.18, 0.20 and 0.18. With ESM-2 35M, average MCC fell from 0.35 before projection to 0.15 after it. These comparisons used fold averages and estimated standard deviations, and no formal effect test was reported.
An evaluation model supplied a second measure of overlap between the embeddings and the tabular inputs. For unprojected ESM-1v 650M embeddings, adjusted R² ranged from 0.153 to 0.262 for the tabular features and from 0.206 to 0.289 for their representation. After regular projection, those ranges dropped to 0.002 to 0.018 and 0.006 to 0.011. Randomized projection remained close to unprojected values. Adjusted R² here is a model-based association measure: it describes variation in predicted scores, not a causal share of biology.
Patterns in the model's scores
In the ESM-1v 650M evaluation model, one structural measure, RSA, had positive and significant coefficients across all four functions, at 0.316, 0.425, 0.467 and 0.339. pLDDT coefficients were negative, at -0.139, -0.138, -0.117 and -0.097, and significant in Organismal Fitness, Activity and Expression but not Binding. Substitution patterns varied by function: nearly every reference amino acid changed to Proline was associated with a lower predicted score for Organismal Fitness, Activity and Expression; glutamine substitutions were mostly positive for Organismal Fitness, while aromatic-to-non-aromatic substitutions were negative in those three functions.
What the result does not establish
The findings concern model representations within the tested computational setting. They do not establish that the models causally use the identified biochemical features or that the coefficient patterns identify a unique biochemical mechanism. The analysis covered filtered ProteinGym groups from proteins in the human proteome and four function types, and used ESM-family embeddings with a deliberately simple L2-regularized logistic regression. Its structural inputs were based on predicted wild-type structures. The result supports predictive association within this setup, not a causal explanation.
The paper is an arXiv version 1 preprint dated 26 August 2026. It acknowledges funding from the German Research Foundation through grants KL 3037/7-1 and RE 3474/8-1, project P5 in Research Unit KI-FOR 5363, and grant 459422098.
Paper data and sources
Original title: Interpreting Protein Language Model Embeddings via Orthogonal Projection for Protein Fitness Prediction
Authors: Paulo Yanez Sarmiento, Pia Francesca Rissom, Manuel Pfeuffer et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text