Contents
The invisible contamination problem
Benchmark contamination — a test item, its paraphrase, or its source document sitting in the training corpus — inflates scores by making them partly a report of what the model memorised. The problem is old, documented at scale in the GPT-3 report, and has grown worse as corpora grow and stop being disclosed ¹. The three standard detection families each hit a different wall. n-gram and substring overlap search needs the training corpus, which for most released models does not exist publicly. Likelihood-based membership inference fixes a scalar test statistic in advance and performs close to chance on genuinely exchangeable pretraining-scale data. Canary strings must be planted before the benchmark is published ¹. A fourth option recently appeared: Liu et al. 2024 train a linear probe on internal activations and report state-of-the-art AUC on WikiMIA. But a probe separates whatever distinguishes two item sets, and Das et al. 2024 show that a classifier with no access to the target model reaches high AUC because the member and non-member sets were split by publication date, differing in topic and style ¹. On the deliberately contaminated checkpoints of Oren et al. 2024, where the split is not temporal, Liu’s probe reaches only 54.0 and 51.9 AUC at duplication counts of 1 and 2 against a 50.0 chance baseline ¹. Independently, an analysis found roughly 29% of MMLU questions contaminated in Common Crawl data, and controlled work shows paraphrased items, which sail past n-gram decontamination filters, still inflate scores beyond genuine capability ²¹.
The method: Excess Separability
Excess Separability (arXiv:2608.12652) frames the probe as the right instrument and the reported quantity as the wrong one ¹. Its protocol, Residual-Stream Contamination Probing (RSCP), reads activations at the item prefix, because a state computed over the answer would let the detector read the model’s own competence — the very effect contamination is supposed to explain ¹. The method’s core readout compares the two item sets as a zero-sum contrast across depth: probe accuracy per layer, re-centred against a placebo baseline that matches difficulty level, and tested against a null built by permuting labels, with the reference sample kept twice as large as the suspect sample ¹. Each of those four choices replaces a simpler alternative that was measured and rejected; the level-matched placebo baseline is the paper’s main technical contribution ¹. The nuisance-control idea is a direct import from causal inference: controlling for confounders before measuring an effect is standard in epidemiology and econometrics, and applying it to activation-space geometry to distinguish contamination from domain confounds is new ². The method needs only inference-time access to intermediate activations — no training corpus is consulted and no reference model is retrained ¹².
Validation, and the four choices it forced
Validation forced four choices by measuring failures. Reporting the level of separability is unusable: the false positive rate tracks the size of the analyst’s own control set, from 0.03 to 0.99 under a true null ¹. Contrasting against a flat depth profile fails in both directions: it rejects a true null 0.72 of the time when surface decodability climbs with depth, and it throws away its power entirely when decodability drops — the very shape produced by a residual stream that accumulates noise ¹. The null must refit the probe: an item bootstrap that holds the fitted probe fixed rejects up to 0.09 of the time, where a permutation null that refits it holds 0.02 ¹. And a half-size baseline triples the error rate ¹. The corrected protocol holds the nominal rate across all of these under two deliberately dissimilar simulators, and its power reaches 0.8 once the item sets drift apart by one or two accuracy points between the embedding layer and the depth where the profile peaks ¹.
Real models, and honest limits
On real Pythia transformers, the audits found that baseline depth profiles refuse to stay flat: the spread on a temporal split reaches 29.1 accuracy points, and how uneven each profile is follows the surface difference between the two item sets (r=0.87 over the six audits). That makes the correction bite hardest exactly where the drift is largest ¹. The four well-matched Pile arms all return null — consistent with what prior literature predicts for a corpus each model has seen only about once — and on the temporal split the protocol declines to call a verdict rather than reporting one ¹. The paper is frank about its own boundary: the single positive result sits on the split where exchangeability breaks down, so it cannot settle whether transformers carry a familiarity direction at all ¹. Post-training is also flagged as a threat to the clean story (L6) ¹. Independently, Excess Separability is reported to survive RL post-training that scrubs the behavioral signals earlier detectors rely on; that account cites a UIUC-and-University-of-Washington GRPO study and a Peking University analysis at ICLR 2026 showing RL post-training opens a gap where existing detectors largely fail ². A related concurrent method, LaRA (arXiv:2605.29888), also uses internal representations but requires semantic perturbations across multiple forward passes per item; Excess Separability measures static separability in the unperturbed residual stream without variants ².
What it means
The paper’s principal practical constraint is access: it needs intermediate activations, which most commercial APIs do not expose, so the realistic use cases are audits of open-weight models or self-audits inside a lab ². It also repurposes interpretability infrastructure — the same residual-stream layer where SAE tooling such as GemmaScope, LlamaScope, and Qwen-Scope already operates — into audit infrastructure, giving a regulator or independent evaluator with inference access a way to verify decontamination claims rather than taking the lab’s word ². Published on arXiv on 12 August 2026 and accepted at MAPR 2026 in Hue City, Vietnam, the work releases code and artifacts at github.com/mabushi-lab/residual-stream-contamination-probing, archived at doi:10.5281/zenodo.21855510; the build regenerates every number in about ten minutes on a laptop, and a pre-registered follow-on programme sets a compute budget on the order of 100 GPU-hours ¹². The takeaway is a corrective more than a silver bullet: the protocol is as much a catalogue of what does not work as it is a detector, and whether transformers genuinely carry a contamination-readable familiarity direction remains an open question ¹.