A new machine-learning method is designed to tell whether changing data contains different inputs or reflects a different input-output relationship. The study calls the method the Conditional Jensen–Shannon Discrepancy, or CJSD, and targets decisions about expert reuse, drift diagnosis and category discovery.
Under pure covariate shift—when the input distribution changes but the input-output relationship does not—the method’s functional discrepancy is exactly zero, even when the two input supports are disjoint. A zero result cannot identify differences in parts of the input space where the task mixtures provide no overlap.
Two views of the same task
CJSD compares two tasks with a pair of task-indicator discriminators, or models trained to distinguish the tasks. One sees only the input, X; the other sees the input and its outcome, (X,Y). The difference between their held-out cross-entropy losses forms the functional part of the discrepancy, while an information-theoretic chain rule supplies a separate covariate axis.
The estimator clips predicted probabilities, uses cross-fitting so each held-out loss is evaluated by a model that did not train on that observation, and balances the comparison by subsampling the two tasks to an equal prior weight.
In a special case with identical input distributions and deterministic labels, the normalized functional discrepancy, divided by ln 2, equals the probability mass of the region where the two labeling rules disagree. The paper also distinguishes a fixed-reference version, described as a true metric on conditional mechanisms, from a pair-dependent version that is not guaranteed to obey the triangle inequality.
The benchmark picture
In a battery of 202 constructed task pairs covering null, covariate-shift, concept-shift and combined-shift cases, CJSD achieved perfect concept-versus-covariate discrimination, with an AUC of 1.00. The k-nearest-neighbor conditional mutual-information plug-in also reached 1.00, while MMD and SW2 scored 0.0, a CCIT-style classifier scored 0.90, and the listed exchange measures ranged from 0.78 to 0.82.
The contrast was clearest in a support-shift test. The estimated functional discrepancy stayed between −0.002 and 0.002, while an exchange score rose from 0.051 when the supports were separated by s=2 to 0.346 at s=4. The result illustrates the distinction CJSD is intended to make: the input-related score changed while the functional discrepancy remained near zero.
The methods did not behave equally as the number of dimensions grew. The k-nearest-neighbor estimator had an AUC of 0.72 at 64 dimensions and was at chance from 256 dimensions. Random-forest discriminators held an AUC of 1.00 through 128 dimensions, and gradient boosting restored 1.00 at 256 dimensions. In a broader sample-size grid, the random-forest frontier moved from 64 dimensions with 1,000 samples per side to 256 dimensions with 12,000; its AUC was 0.92 at 512 dimensions, while histogram-based gradient boosting reached 1.00 across the grid at 12,000 samples per side.
Tests on drift and synthetic data
On the INSECTS benchmark, the method separated documented drifts into covariate and functional components: the covariate axis ranged from 0.26 to 0.31, and the functional discrepancy from 0.13 to 0.24. Recurring segment pairs—(0, 2), (0, 5) and (2, 5)—had functional discrepancies no higher than 0.03 and clustered with within-segment null comparisons.
A synthetic-data audit also found a conditional signal that marginal quality checks missed. Shuffled-label generators produced values from 0.041 to 0.074, while subtler conditional flips produced 0.008 to 0.038; in the hardest case, the reported z-score was about 20 even though marginal metrics passed.
For sequential monitoring, the paper reports that betting-based e-processes built from discriminator-loss differences produced zero false alarms in the described experiment. They added a 20% delay compared with repeated confidence intervals, and the reuse rule required an indifference zone extending from τ to 3τ. Results for bounded-memory lifetime recency were attributed to a companion paper.
A diagnostic with clear limits
The authors frame CJSD as a decision aid for expert reuse, drift diagnosis and category discovery. But the study reports no fine-tuning-transfer signal for CJSD; by comparison, LEEP showed a reported correlation of 0.86 with fine-tuned accuracy in the authors’ analysis.
Misspecified discriminators can make the loss gap overshoot or undershoot the target, with each direction bounded by the excess risk of one discriminator, but the bias direction is not generally controlled. The discrepancy also attenuates differences in low-overlap regions and cannot identify a difference outside the overlap.
The document is an arXiv version-1 preprint dated 20 August 2026. Its supplementary package is reported to include all experiments, 46 scripts, checkpointed JSON results and figures.
Paper data and sources
Original title: Separating Covariate Shift from Mechanism Change with Two Discriminators: CJSD, a Conditional Discrepancy with an Exact Covariate-Concept Decomposition
Authors: Kentaro Oda
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text