AI Self-Reports Are Unreliable: Open-Weight Study Maps Introspection Failure

Ask a language model whether something changed inside it and you will almost certainly get a confident answer. That answer carries no reliable information. A new study from the University of Southern California has measured that gap precisely, and the result has direct consequences for how AI safety audits are designed and what counts as valid oversight of a deployed model.

The paper, published today on arXiv by Emilio Ferrara of USC's Information Sciences Institute, introduces the Open-Weight Masked Introspection framework — OWMI — and uses it to run more than 78,000 controlled measurements across eight open-weight language models from seven families. The finding is clear and statistically bounded: no model's verbal report discriminates a real internal intervention from a fake one at above-chance accuracy. The pooled AUROC across the study is approximately 0.5007 — a score that is, within a margin of 0.15 percentage points, statistically indistinguishable from random guessing.

The more important result is what the study found when it looked inside the same models. A simple linear probe trained on the models' internal activations recovers the intervention signal at up to 95.8% held-out accuracy at the final layer before the model speaks. The information is present in the model's representation. It is not reaching the model's words.

How OWMI Measures What Models Can Report About Themselves

The core challenge in studying LLM introspection is establishing ground truth. When a model says it "noticed" a change in its processing, there is normally nothing to check that claim against. Prior methods have typically relied on behavioral measures — does the model's output change when its activations change? — but this conflates change detection with verbal reporting. A model whose output shifts after an intervention might be responding to the intervention without being able to report that response; a model that generates the same verbal disclaimer regardless of whether anything changed cannot be said to be accurately introspecting. OWMI sidesteps this by imposing changes rather than inferring them, and by separating the question of whether the change affected output from the question of whether the model can report it.

The framework selects an open-weight model and modifies one of three classes of internal computational object: a position in the residual stream, the output of a specific attention head, or a direction identified by a sparse autoencoder as an interpretable feature. All three sit deep inside the transformer architecture, invisible to a user watching the text output. After imposing the change, OWMI asks the model a standardized question about whether its computation was altered and records both the yes-or-no answer and the confidence attached to it.

To make the result mean anything, the framework requires three null conditions. The first is a sham run — the identical procedure with nothing actually modified — which ensures a model that always says "yes" still scores at chance. The second is an impact-matched random direction: a perturbation of the same magnitude as the real intervention but in a random direction rather than a meaningful one, testing whether any apparent detection reflects generic sensitivity to disruption rather than awareness of the specific object. The third is a text-only observer that sees only the model's visible output, ruling out effects an external reader could infer from surface behavior without access to activations.

The OWMI library releases all of these components openly under a MIT license on Hugging Face: the intervention operators, three control conditions, probe bank, paired scoring estimators, and equivalence-testing procedure. The intent is to make this class of measurement reproducible as the field's models and capabilities develop.

A Residual Stream Primer: What Is Being Altered

A transformer-based language model passes information through layers via what researchers call the residual stream. Each layer reads from this stream, processes it, and adds its output back rather than replacing the stream outright. The stream accumulates an increasingly abstract representation of the input as it moves through the network. A token's final representation at the last layer is what gets projected into vocabulary space to produce the next word.

This architecture has a property that matters for OWMI's design: because each layer adds to the stream rather than replacing it, an intervention at layer 10 will still be present — as an added signal — when the model generates its answer at layer 30. The intervention does not disappear; it propagates forward with the residual. This means a model that failed to verbally report the intervention did not fail because the altered signal was washed out downstream. The altered signal was still there.

OWMI intervenes directly on sites in this stream — adding a vector that represents a specific concept or activation — then asking the model whether it can report that something changed. Attention heads are another natural target because their output projections are interpretable components that contribute to the residual stream at each layer. The third intervention class uses sparse autoencoder features, which are directions in activation space that have been identified through interpretability research as corresponding to human-understandable concepts. These are among the most semantically meaningful objects in the model's representation system. When OWMI targets one of these directions, it is modifying something that researchers already know carries interpretable signal.

The sparse autoencoder angle deserves a brief explanation because it is doing real interpretability work here. Transformers store vastly more information in their activations than they have neurons by representing concepts as combinations of neuron directions — a phenomenon researchers call superposition. A sparse autoencoder is trained to decompose these overcomplete representations into a larger dictionary of sparse, more human-legible directions, each corresponding to something like a topic, a syntactic role, or a semantic concept. When OWMI uses an SAE feature as an intervention target, it is modifying a direction that research has already associated with a specific interpretable meaning — making the question more tractable and the null result more striking. The model is not just failing to detect a random noise injection; it is failing to detect a change to something its own activations were already organized around.

The result is a precise measurement setup. The experimenter knows exactly what changed, exactly when, and exactly what signal would need to reach language output for the verbal report to be accurate.

The Dissociation: Information Present, Verbal Report Absent

Across the dose battery — 11,216 paired trials across eight models — no model's verbal report outperforms a coin flip. The equivalence test bounds the discrimination advantage to less than 0.15 percentage points of AUROC with a p-value below 0.0001. This is not a failure-to-reject-null result that might reflect low statistical power; it is a bounded null that rules out meaningful effect sizes.

A separate breadth battery adds 19,520 measurements on three models, crossing three classes of computational object, six depths from layer 8 to layer 31, four benchmark domains, and four probe families. At no level of any axis does the verbal report rise above chance. The conclusion holds whether the intervention targets the residual stream, an attention head, or a sparse autoencoder feature; whether it happens early or late in the network; and whether the domain involves language, code, or reasoning tasks.

What the study also finds is equally important. When Ferrara trains a linear probe on the same activations the model uses to produce its verbal answer — reading the model's internal representation rather than its output — the probe recovers the intervention at 75.0% and 95.8% held-out accuracy in the two dose-calibrated models. At the last layer before the model writes its answer, both dose-calibrated models and Mistral-7B-Instruct-v0.3 separate intervention from sham with no held-out error, and none of 200 label-shuffled permutations reaches the measured margin. The signal sharpens rather than fades as the probe captures activations closer to the output layer.

This is the paper's key finding: the failure is not informational. The model has access to the relevant computation. Something in the path from internal activation to verbal output drops that information before it becomes language.

A fine-tuning result confirms the task is solvable. A model explicitly trained to report this class of intervention achieves a d′ of 5.15 and an AUROC of approximately 1.0 on the identical pipeline — near-perfect recovery on held-out directions. The barrier is not architectural impossibility. It is that untrained verbal output does not route through the channel carrying the signal.

One model produced an additional finding worth noting. Its yes-or-no verbal answer never varies across trials — a constant response regardless of whether an intervention occurred. But the confidence attached to that constant answer separates intervention from sham at AUROC 0.647. This is a dissociation between two channels of the same self-report: the graded probability carries some signal; the discrete word does not. In the paper's corrected description, this is not evidence of meta-level awareness without first-order access, but rather evidence that even the marginal signal present in the system leaks through confidence rather than through the model's stated conclusion.

What OWMI Finds That Prior Introspection Research Did Not

The closest prior work is a January 2026 paper by Jack Lindsey at Anthropic, "Emergent Introspective Awareness in Large Language Models," which injected concept representations into model activations and found that Claude Opus 4 and Claude Opus 4.1, the most capable models tested, could detect and name injected concepts with above-chance accuracy in certain conditions. That finding suggested at least a weak form of introspective awareness was present in frontier closed-weight models, and triggered a wave of follow-on research to understand whether it generalized, what mechanisms produced it, and how it varied across model families.

OWMI differs from that work on several dimensions that matter for interpreting the gap. Lindsey studied frontier closed-weight models, primarily Claude variants; OWMI covers open-weight models across seven families. Lindsey asked models to detect and name an injected concept; OWMI asks models to detect whether their computation was altered at all — a related but distinct task. Lindsey's controls were sham-like baselines; OWMI uses a three-arm design with an impact-matched random direction that specifically tests whether models track the semantic content of the intervention or merely respond to generic disruption.

These methodological differences mean the two studies are not straightforwardly contradictory. It is consistent with both results that frontier closed-weight models have developed a weak introspective channel that open-weight models have not, or that named-concept detection (Lindsey) is a different and easier task than arbitrary computation-alteration detection (OWMI). The former may require only that the model encode a concept strongly enough to notice it verbally; the latter requires the model to independently notice that some aspect of its computation differed from baseline — a more abstract and challenging detection task. What the two studies together establish is that the field does not yet have a single clean answer to whether language models can introspect — the answer depends on which model, which task, and which controls are applied. OWMI's specific contribution is the most rigorous controlled multi-model null result on the computation-detection version of the question.

A parallel 2026 study began with results suggesting Llama 3.1 8B Instruct could detect interventions with 97.3% accuracy in one early-layer condition, before the authors discovered that the effect was almost perfectly correlated with a general yes-bias introduced by the intervention itself. Across 40 combinations of layer and injection strength, apparent detection and this general yes-bias were correlated at 0.999. After controlling for that bias, the remaining signal collapsed to near-zero — a cautionary case for how apparent introspective ability can dissolve under rigorous controls. OWMI's equivalence-test framework is designed specifically to prevent that category of error.

The broader research picture shows how contested and rapidly evolving this area is. Plunkett et al. (2025) found that GPT-4o and GPT-4o-mini, after targeted fine-tuning, could accurately report the internal decision weights guiding their choices across complex preference tasks — but this is behavioral self-knowledge acquired through explicit training, not spontaneous introspective access. Binder et al. (2024) found that frontier models could be fine-tuned to predict their own behavior in hypothetical situations, providing evidence of privileged self-access. Song et al. (2025) argued that meaningful AI introspection requires privileged access — information available to the model about itself that is not derivable by an external observer — and found that existing metrics often fail to establish this threshold rigorously. The OWMI result sits within this contested landscape as the strongest large-scale evidence that untrained verbal self-reporting, specifically about computation-level changes, currently carries no reliable signal across open-weight model families.

Why This Matters for AI Oversight Architecture

The safety implication is direct. AI oversight frameworks under development in regulatory contexts — the EU AI Act, the International AI Safety Report 2026, enterprise AI governance programs — increasingly discuss model transparency and self-reporting as potential oversight mechanisms. If a model can accurately report on anomalous internal states, that capability could substitute for some external monitoring; a model flagging that something unusual occurred in its processing would be a useful early-warning signal.

The OWMI result makes the case against that approach as a primary safety mechanism. Verbal self-reports are not just noisy — they are structurally disconnected from the mechanistic channel that carries the relevant information. Asking a model "did your computation change?" is statistically equivalent to asking it to flip a coin and report the result as introspection. The model's confident answer tells you nothing about what actually occurred in its weights.

This is not merely a theoretical concern for future AI systems. Current enterprise AI deployments often include monitoring steps where models are asked to reflect on their outputs, flag uncertainties, or explain their reasoning. These mechanisms are useful for many purposes — user trust, error detection, audit trails — but the OWMI result suggests they should not be understood as reading genuine internal states. A model that says "I am confident in this answer" and a model that says "I am uncertain" are both reporting through the verbal output channel that OWMI shows carries no reliable signal about actual internal states. The confident-sounding answer and the uncertain-sounding answer may be equally disconnected from what the model's activations actually contain.

The alternative — reading activations directly using probes, sparse autoencoders, or mechanistic interpretability tools — works. The same activations that produce uninformative verbal reports can be decoded at up to 95.8% accuracy by an external linear classifier. This suggests that investment in activation-level monitoring infrastructure, rather than in systems that query models about their own states, is the more reliable path toward meaningful oversight. Anthropic, Google DeepMind, and several academic groups have built prototype activation-reading systems; the OWMI result gives those approaches a stronger empirical foundation and a clearer rationale.

This has a secondary implication for AI governance. The EU AI Act and related frameworks require certain transparency and documentation for high-risk AI deployments. If self-reports are used to satisfy transparency obligations — asking a model to explain its reasoning or flag its uncertainties — OWMI provides empirical grounds for questioning whether those self-reports are grounded in actual internal states. External interpretability tools that read activations are more expensive to build but more informative. The policy question is whether transparency requirements should specify the mechanism by which transparency is achieved, not merely that a model is capable of producing explanatory text.

What Limits the Study and What Comes Next

The OWMI study is not without its own caveats, which the paper and the Hugging Face model card document carefully. Most notably, only one of the eight tested models carries a genuinely impact-matched random control in the sense the framework specifies: Qwen2.5-7B-Instruct, where calibration landed within 1.1% of target. Mistral-7B-Instruct-v0.3 overshot its impact target by 41.2%. The remaining six models carry a unit-norm random direction, which equalizes perturbation size but not its functional effect on the model's computation. The paper's per-model contrasts should be read with this in mind.

The model card also flags specific implementation gaps. The reconstruction probe family is implemented in the OWMI library but was not scored in the reported batteries. The spontaneous track, which tests whether models flag internal changes without being asked, executed but produced no complete intervention-sham pair. Models that emit a reasoning chain before answering need a probe budget well above the 128-token default, or their answers are truncated before reaching a scorable report — a practical constraint that affects which architectures can be cleanly evaluated on longer-form tasks.

Qwen3-14B was tested but excluded from all reported results: only 5 of 384 trials produced scorable output, and the model completed no intervention-sham pair. The eight-model roster is what the numbers are computed over, and that roster spans seven laboratory families — but the composition skews toward smaller 7-to-14 billion parameter models. Whether the null result holds at larger open-weight scales (70B+, 100B+) is an empirical question the current study cannot resolve.

The OWMI paper is a preprint under review. Its 45-page manuscript has not yet been through formal peer review, and its results — while consistent across two independently published sources (the arXiv abstract and the detailed Hugging Face model card) — await independent replication. The equivalence-test framework is a strength of the methodology, but the specific numerical bounds (95.8%, 0.5007) are author-reported from a single research group.

The framework also covers open-weight models only. Whether the null result holds for frontier closed-weight models — the class Lindsey found to show at least limited introspection — is an open empirical question OWMI cannot answer with the current dataset. This is not a minor gap. The most safety-relevant models are precisely the frontier closed-weight systems deployed in high-stakes applications. OWMI's finding applies most directly to the open-weight ecosystem, which is large and growing but distinct from the proprietary frontier where the most consequential safety questions currently reside.

Ferrara releases the OWMI library precisely because the picture should be tracked over time. Introspective capability, if it develops in future model generations, would show up as AUROC rising above chance on the same pipeline used here. The framework is designed to be a longitudinal measurement instrument, not just a one-time study. What the current generation of open-weight models cannot do, a future one might — and the community will need controlled infrastructure to recognize that transition when it occurs. The fine-tuning result already points to the direction: a model that can be explicitly trained to report computation-level interventions at near-perfect accuracy has the necessary architecture. The open question is whether the corresponding capability emerges through general training at scale without being explicitly elicited.

Governance frameworks tracking model capability over time would benefit from including OWMI-style introspection measurement alongside standard capability benchmarks, because the moment a model's verbal report starts carrying reliable internal signal is also the moment oversight strategies based on self-testimony become meaningfully more defensible. Until that moment arrives, the field has a clear empirical mandate: build oversight on what works. For now, that means reading activations. The information is in the model. The words do not carry it.