Mechanistic Interpretability's Core Assumption May Be Wrong, Goodfire Researcher Says

A researcher at AI interpretability startup Goodfire has published a sweeping challenge to the dominant theory explaining why neural networks are hard to understand — arguing that the field has spent years solving the wrong problem, and that the models it studied were far more readable than anyone recognized.

Nick Cammarata, who co-authored some of the foundational papers in mechanistic interpretability research before joining Goodfire, published "Neural Networks Are Already Clean" on October 15 — a 26,000-word essay arguing that the so-called superposition hypothesis, the reigning explanation for neural-network opacity, was based on a misinterpretation of the evidence. His audit of Qwen3-32B, a 32-billion-parameter language model, found that approximately 90% of neurons have a single primary function when analyzed carefully, not the overlapping tangle of unrelated concepts that the field assumed. An extended analysis of InceptionV1, the vision model that generated the most-cited polysemanticity examples, suggests those famous "messy" neurons were simply detecting V-shaped geometric structures — making them coherent, not confused.

The essay drew immediate and sharply divided reaction. Adam Majmudar, an independent AI researcher, wrote on X that it was "the most important result this year, comparable to Attention Is All You Need," describing it as moving the field from debating whether alignment is solvable to asking whether it might be solvable next year. Sam Altman publicly weighed in, and in the same week outlined three possible long-term governance architectures for AI development. Jacob Hilton, a researcher at the Alignment Research Center, pushed back: the essay "unfortunately presents an unrealistic picture of the state of the field," he wrote on X, noting that researchers currently understand less than 0.01% of what happens inside the top large language models.

The Superposition Hypothesis and Why the Field Believed It

To understand what Cammarata is disputing, it helps to understand what the field believed and why.

Neural networks are composed of millions of individual neurons — mathematical units that activate in response to patterns in their inputs. For decades, researchers assumed that neurons were relatively clean detectors: one neuron for "dog," another for "curved line," another for "French text." This made intuitive sense given how biological neurons are sometimes described.

The trouble came when researchers started actually looking. In 2020, Cammarata and his collaborator Chris Olah published "Zoom In: An Introduction to Circuits," a landmark paper analyzing a vision model called InceptionV1. They found a neuron that appeared to respond strongly to both cat faces and car fronts — two categories with nothing obvious in common. Other neurons seemed to activate for entirely unrelated stimuli: sports equipment and dogs, helicopters and certain postures, fire and flowing water.

In 2022, Anthropic researchers formalized this observation into the superposition hypothesis. The argument was elegant: neural networks must encode far more concepts than they have neurons. A language model might need to represent hundreds of thousands of distinct features, but it might have only millions of neurons — not enough to give each feature its own dedicated unit. The solution, the theory proposed, was to pack multiple features into each neuron using a kind of controlled interference, trading off accuracy of representation for sheer capacity. This would make neurons appear polysemantic — activated by several unrelated things — not because the concepts are related, but because they were mathematically crammed together.

The hypothesis was influential because it offered a principled explanation for an observed problem, and it pointed toward a specific engineering solution: if neurons are compressed superpositions of features, you need a tool to decompose them. That tool became the sparse autoencoder (SAE), which uses sparsity constraints to try to reconstruct the original "clean" features from the messy neuron activations. Anthropic published "Towards Monosemanticity" in 2023, a flagship SAE paper, and invested heavily in scaling the approach. SAE research became one of the central projects in the field.

Cammarata, one of the people who built this framework, now argues it was built on a misread.

A V-Shape, Not a Cat and a Car

The InceptionV1 neuron 4e:55 — the one that appeared to activate for both cats and cars — became the canonical example of polysemanticity in the field's literature. Cammarata now argues it was detecting something different from both: a V-shape.

The geometry of a cat's face, viewed from the front, creates a rough V or inverted-V in its lower structure. A car approaching a camera also creates a V-shape in its front profile. Flames flicker in V-like forms. Water divides around obstacles in V patterns. When Cammarata re-examined the full activation pattern of neuron 4e:55 — looking at all the stimuli it responded to, not just the two that seemed strange together — he found a coherent geometric theme. The apparent polysemanticity dissolved into a single, sensible visual primitive.

A second canonical example also received a new interpretation. Researchers had observed that automotive features seemed encoded in neurons associated with dogs. Cammarata's re-analysis suggested that dogs and cars frequently appear in similar photographic compositions — both often presented frontally, centered in frame, with the camera roughly at face-level. The neurons may have been detecting a shooting-angle pattern, not a superposition of semantically unrelated concepts.

These are not isolated reanalyses. Cammarata proposes a general alternative mechanism he calls "recursive generalization" to explain what was previously called polysemanticity. In his account, neurons begin training with a narrow detection task — a simple pattern, a specific letter combination, a basic geometric form. As training proceeds, the neuron's effective receptive field expands organically to encompass related inputs that share statistical or geometric structure with the original seed. The expansion is recursive: each new input that activates the neuron makes similar inputs more likely to activate it in future training steps, branching outward like a tree.

For a language model neuron, the implication is visible in cases like the letter pair "iy": a neuron that initially activates for that character sequence may, through training, begin activating for "diy" — and then for phrases semantically related to "do it yourself." Viewed from outside, after training is complete, the activation list looks like a collection of loosely connected ideas. Traced developmentally through the training process, each activation is connected to the last by a continuous path of statistical similarity.

Cammarata's argument is that what the field called polysemanticity — neurons responding to unrelated things — was in most cases recursive generalization: neurons responding to related things that required more context to identify as related than the original analyses provided.

Neuron Audits at Production Scale

The claim is empirically testable, and Cammarata applied tests at two scales.

For InceptionV1, which uses a pre-modern architecture without residual connections, the final layer had historically been considered nearly uninterpretable — estimates ran from essentially 0% to at most 5% of neurons clearly readable. After reanalysis with the recursive-generalization framework, Cammarata reports 99% to 100% interpretability in the same layer. He also compared InceptionV1 to ConvNeXt, a modern architecture with residual connections and LayerNorm, at the same parameter count — and found ConvNeXt's final layer already 99% interpretable under standard analysis methods. The difference in architecture, not scale or data, appears to be the dominant factor.

For the language model scale, Cammarata audited Qwen3-32B, a 32-billion-parameter model from Alibaba's Qwen team. The results: approximately 90% of neurons, in both their positive and negative response directions, had a single clearly identifiable primary function. Another 7% had two related functions — not arbitrary pairings, but concepts the authors judged to be semantically connected. The unexplained residual was 2% to 4%.

These figures come from Goodfire's own analysis, conducted by a researcher who has a direct professional and institutional stake in the result. They have not yet been independently replicated. The audit methodology — what counts as "one function" versus "two related functions" — requires subjective human judgment about concept similarity, which could be influenced by the analyst's expectations. Cammarata acknowledges this; the framework for future validation will require independent auditors applying the same criteria.

The comparison point that makes the numbers striking is a 2023 OpenAI automated interpretability study that Cammarata characterizes as finding only 1.7% of neurons clearly interpretable under that pipeline. Cammarata attributes the dramatic gap not to a fundamental change in models, but to improvements in analysis method — including better visualization, longer scrutiny per neuron, and the recursive-generalization interpretive lens.

What This Means for Sparse Autoencoders

If Cammarata's framework is correct, the SAE research program faces a difficult question: what, exactly, was it solving?

SAEs were built to decompose polysemantic neurons into their constituent features — to take a single neuron that superposed three unrelated concepts and extract those three concepts as separate, interpretable units. The tool's existence presupposes that such decomposition is necessary. If the neurons were not arbitrarily polysemantic to begin with — if they were encoding coherent concept families that merely looked tangled without the right interpretive framework — then SAEs may be imposing an artificial structure on top of an already-interpretable representation.

This does not mean SAE research was wasted. SAEs remain a useful tool for decomposing the residual stream representation in transformers, which is architecturally different from studying individual MLP neurons. And the mechanistic understanding of features that Anthropic's SAE work has produced retains value regardless of what interpretation is placed on the original neuron-level observations. But the original motivation — neurons are fundamentally unreadable without decomposition — is precisely what Cammarata disputes.

Jacob Hilton's Counterargument: 0.01% Is Not 90%

The claim that neural networks are "already clean" strikes some researchers as premature at best.

Jacob Hilton, a researcher at the Alignment Research Center, responded to the essay within 24 hours of publication. His objection cuts at a different level than the technical details. Even if Cammarata is right that neurons are individually interpretable at the level of "this neuron detects V-shapes," Hilton argues that individual neuron interpretability is not the same thing as understanding how the model reasons.

What researchers actually need to understand is not just what each neuron detects in isolation, but how neurons interact across layers to produce a model's outputs — the full computational circuit. A model that generates a harmful response, or makes a dangerous judgment, does so through a chain of interactions across thousands of neurons. Knowing what each neuron responds to does not tell you which neurons contributed to a specific decision, in what combination, and why.

Hilton estimates that current mechanistic interpretability research has characterized perhaps less than 0.01% of the internal computation that occurs inside top large language models during inference. A model like Qwen3-32B processes each input through roughly 64 transformer layers, with millions of neuron activations, billions of floating-point operations, and attention patterns across a context window of potentially hundreds of thousands of tokens. Knowing that 90% of neurons have identifiable functions is a starting point, not a destination.

Hilton's critique does not dispute the individual neuron results. It argues that Cammarata's framing — "neural networks are already clean" — overstates the progress those results represent. The field has a better map of individual neurons than previously thought. It does not yet have a map of how those neurons operate collectively to produce behavior.

Implications for AI Safety Evaluation

The practical stakes of the interpretability debate extend well beyond academic priority disputes.

Current AI safety practice relies primarily on behavioral evaluation: you test a model's outputs under thousands of conditions and try to infer whether it will behave safely in deployment. This approach has a fundamental limitation — it can only detect failure modes that evaluators thought to test. A model that behaves safely under all tested conditions may still harbor internal representations or reasoning patterns that produce unsafe behavior under novel circumstances.

Mechanistic interpretability offers a potential complement: instead of, or in addition to, testing outputs, directly inspect the internal circuits to understand whether a dangerous reasoning pathway exists. If a model has a circuit that activates in response to instructions framed as fictional but actually implements real-world harmful planning, behavioral testing might miss it while circuit-level inspection might reveal it.

This prospect depends on the field making considerably more progress than Hilton thinks it has. If the gap between individual neuron readability and full circuit understanding remains as large as he suggests, circuit-level safety auditing at deployment scale is not yet feasible.

Cammarata's contribution, if validated, would move the needle by establishing that the lower-level components of circuits — the individual neurons — are more tractable than assumed. Neurons that reliably detect specific, consistent features can be traced through circuits with more confidence than neurons whose activations are ambiguous mixtures of unrelated patterns. The 90% figure, if it holds under independent replication, would substantially simplify the neuron-identification step of circuit-level analysis — though Hilton's point about circuit complexity remains.

The critical limit that both sides agree on: interpretability findings must be validated by intervention, not just observation. Identifying that a set of neurons appears to represent a concept is not the same as proving those neurons causally contribute to the model's behavior when that concept is relevant. The standard for interpretability-based safety claims requires showing that suppressing or amplifying those neurons changes behavior in the predicted direction — a harder bar that Cammarata's essay sets but that large-scale systematic validation has not yet achieved.

Sam Altman's Three Governance Scenarios

In the same week the essay circulated, Sam Altman outlined three possible long-term scenarios for how AI development might be governed — a discussion whose feasibility bears directly on the interpretability debate's safety implications.

The first scenario is an IAEA model: an international body modeled on the International Atomic Energy Agency, setting and enforcing safety standards across AI developers worldwide, with inspection rights and the ability to prevent deployment of models that exceed agreed capability or safety thresholds. This is Altman's stated preference. The nuclear analogy has limits — AI models can be copied and deployed via software in ways that uranium enrichment facilities cannot — but the general framework of independent international oversight with verification mechanisms has a proven track record in arms control.

The second scenario is what Altman described as the "Demis plan": merging the major AI laboratories into a single institution that develops frontier AI collectively, sharing safety research and operating without competitive pressure to cut corners. The appeal is obvious — eliminating the race dynamic that makes safety shortcuts tempting. The problem: concentrating all frontier AI capability in one organization replaces competitive fragmentation with potential for unchecked institutional failure, leaving no independent institution to catch misjudgments.

The third scenario, which Altman described as "terrible," involves a single AI developer acquiring sufficient economic control over GPU supply chains, data centers, and compute infrastructure to block other developers from training or deploying dangerous models by cutting off their access to hardware. The framing evokes arguments associated with AI safety researcher Eliezer Yudkowsky; Altman made clear he considered such a concentration of power dangerous regardless of who holds it.

An IAEA-style body cannot meaningfully inspect an AI model that is fundamentally opaque. If interpretability tools improve to the point where inspectors can audit a model's internal circuits for specific dangerous capabilities, the IAEA scenario becomes substantially more feasible. If Hilton's 0.01% characterization is closer to reality, any inspection regime would be flying nearly blind.

What Comes Next

Cammarata's essay is a research claim, not a peer-reviewed publication. The most important next step is independent replication — running the same neuron-audit methodology on Qwen3-32B and other production models using evaluators who did not develop the recursive-generalization framework. If independent auditors find similarly high interpretability rates, the case for a field reorientation becomes substantially stronger.

The SAE research program needs to respond. Anthropic has invested years of work and significant compute into the SAE approach; the question of whether SAEs add value over direct neuron analysis given Cammarata's framework is not answered by the essay and will require empirical comparison. A plausible outcome is that both approaches are useful at different analysis scales: direct neuron study for understanding individual feature detectors, SAEs for understanding the broader residual-stream geometry in transformers.

Goodfire's Ember API, which operationalizes interpretability for external developers, is the commercial bet on this thesis. The company raised $150M at a $1.25B valuation in a February 2026 round led by B Capital. Anthropic, which backed Goodfire in earlier financing rounds, is simultaneously the leader of the SAE research program Cammarata challenges and a financial stakeholder in the company making the challenge. That tension will not be resolved by a single essay.

What the field can agree on: the gap between the 1.7% interpretability figure that Cammarata attributes to automated methods in 2023 and his 90% claim from manual analysis in 2026 is not noise. Something about how researchers look at neurons changed the answer dramatically. Whether that change reflects better analysis of the same reality, or wishful interpretation of ambiguous activation patterns, is now the central empirical question of the field — and the answer will determine whether the governance scenarios Altman described are technically feasible within any near-term planning horizon.