Interpretability: How Scientists Look Inside an AI's Mind

On 23 May 2024, Anthropic put a strange version of Claude online for 24 hours.

People asked it ordinary things. How should I spend ten dollars? Drive across the Golden Gate Bridge, it suggested, and pay the toll. Write me a love story. It wrote about a car that couldn’t wait to cross its beloved bridge on a foggy day. In Anthropic’s own write-up, someone asked about its physical form, and it answered: “I am the Golden Gate Bridge… my physical form is the iconic bridge itself…”

Nobody had prompted it to role-play. There was no system prompt saying “you love bridges”, and no fine-tuning on bridge facts. Anthropic described the change as “precise, surgical”, made to the model’s internal activations. Researchers had found one specific pattern inside Claude 3 Sonnet that stands for the Golden Gate Bridge, and they had turned it up.

That demo, nicknamed Golden Gate Claude, was a joke on the surface. Underneath, it was a proof. For years people have called neural networks “black boxes”. Golden Gate Claude showed that at least some of what’s in the box can be found, named and controlled.

This post is about how that works. It’s part 1 of How AI Works: Advanced, so I’ll assume you know what tokens, embeddings, layers and attention are. If those are new to you, start with the basics first and come back.

The problem: neurons are a mess

A language model is layers of numbers. When it reads a token, every layer produces a long list of numbers, its activations, which you can think of as the model’s state of mind at that moment. Some of those numbers come from units we call neurons.

The obvious way to understand a model is to look at one neuron at a time and ask: what does this one respond to? Researchers did this for years with image models and sometimes found lovely results, like neurons that fire for curves or for dog faces. But in language models, it mostly fails. A single neuron might fire for Korean text, legal contracts and HTML tags. Unrelated things, all mixed together.

That property has a name: polysemantic, meaning “many meanings”. A neuron that means one thing would be monosemantic.

Superposition: more ideas than neurons

In September 2022, a team at Anthropic (Elhage and colleagues) published “Toy Models of Superposition”, which offered an explanation. A model needs to represent far more concepts than it has neurons. Think of all the people, places, programming ideas, emotions and grammar rules a chatbot knows. So instead of giving each concept its own neuron, the model stores each concept as a direction: a particular pattern across many neurons at once. Concepts are allowed to overlap.

This works because most concepts are sparse. On any given token, only a tiny fraction of all the things the model knows are relevant. If “Golden Gate Bridge” and “photosynthesis” rarely show up together, they can share neurons with little confusion. The toy-model paper showed small networks doing exactly this, packing extra features into “superposition” when those features were sparse enough.

Here’s the analogy I find most useful. Think of a choir. If you listen to one singer, you hear a jumble of notes, because that singer is contributing to many songs. But each song is a particular chord: many singers, each doing a little. To hear the song, you need to recognise the chord, not follow one singer.

Where the analogy breaks: a real choir sings one song at a time. A model is “singing” dozens of concepts at once on every token, overlapping in the same voices. That’s what makes pulling them apart hard.

How it works: a dictionary for the mind

If concepts are chords, you need a way to find the chords. The main tool is called dictionary learning, and the specific network used is a sparse autoencoder (SAE).

Here’s the recipe in plain words:

Run the model on a huge amount of text and record the activations at one layer, millions of snapshots of its “state of mind”.
Train a second, separate network to do one job: rewrite each snapshot as a short list of ingredients, picked from a very large dictionary, and then rebuild the original snapshot from that list.
Add a rule that forces it to use as few ingredients as possible each time. That’s the “sparse” part.

Each ingredient is called a feature. The hope is that, because the rewrite has to be both accurate and sparse, each feature ends up meaning one thing. The second network is the decoder ring. The original model isn’t changed at all.

A sparse autoencoder rewrites one jumbled snapshot as a few active features out of millions, and must be able to rebuild the snapshot from them.

One important detail: nobody tells the autoencoder what the features should be. It doesn’t know the words “bridge” or “fog”. Researchers find out what a feature means afterwards, by looking at which texts make it fire most strongly. The labels in my diagram are the kind of names humans give later.

From a toy model to a real one

Anthropic first tried this at small scale. In “Towards Monosemanticity” (October 2023), they took a tiny transformer with a layer of 512 neurons and pulled out more than 4,000 features. Many were clean and specific: DNA sequences, legal language, HTTP requests, Hebrew text, nutrition statements. Five hundred neurons, each one a jumble, turned into thousands of features that mostly meant one thing.

The obvious question was whether it would scale to a real production model. In May 2024, “Scaling Monosemanticity” (Templeton and colleagues) answered it. The team trained sparse autoencoders on the middle layer of Claude 3 Sonnet, with dictionaries of roughly 1 million, 4 million and 34 million features. Anthropic summarised the result as “millions of features” extracted from a model people actually used.

Other labs have done similar work since. OpenAI trained a sparse autoencoder with 16 million features on GPT-4’s activations (Gao and colleagues, June 2024). Google DeepMind released Gemma Scope in July 2024: more than 400 open sparse autoencoders covering every layer of its small open Gemma 2 models, so researchers outside the big labs could look inside too.

What’s in there

So what did they find in Claude? Anthropic’s write-up, “Mapping the Mind of a Large Language Model”, has the best examples.

Concrete features. The Golden Gate Bridge feature fired on the bridge’s name in English, and also on discussions of it in Japanese, Chinese, Greek, Vietnamese and Russian. It fired on a photo of the bridge too. That’s the key point: it isn’t a “word detector”. It’s closer to a concept, one that doesn’t care what language or format it arrives in.

Neighbourhoods. Features can be compared by how similar their directions are. Near the Golden Gate Bridge sat features for Alcatraz, Ghirardelli Square, the Golden State Warriors, California governor Gavin Newsom, the 1906 earthquake and Hitchcock’s film Vertigo, which is set in San Francisco. A small map of the city, inside the model.

Abstract features. Some features were about ideas, not things:

bugs in computer code
keeping secrets
inner conflict, with nearby features for relationship breakups, conflicting loyalties, logical inconsistencies and the phrase “catch-22”
sycophantic praise, firing on lines like “Your wisdom is unquestionable”
a feature that fired when Claude read a scam email

Safety-relevant features. They also found features for code backdoors, biological weapons, bias, power-seeking, manipulation and secrecy. That doesn’t mean the model wants those things. It means the model has concepts for them, the same way you understand what a lie is without being a liar. But it does mean those concepts are present and could, in principle, be watched.

Proof by slider: steering

Finding a feature that lights up for the bridge is one thing. It could still be a side effect, a pattern that happens to correlate with the bridge but plays no role in what the model says. Correlation isn’t cause.

So the researchers tested causes. Picture a mixing desk with millions of sliders, one per feature. Normally the model sets every slider itself as it reads. Steering means grabbing one slider and forcing it to a value of your choice, high or low, no matter what the conversation is about. Researchers often call this clamping a feature.

Clamp the Golden Gate Bridge feature high, and you get Golden Gate Claude. Ask about money, love or itself, and every answer bends toward the bridge. Let the slider go, and the normal model comes back.

The more serious examples were in the research paper itself:

Scam emails. Normally Claude refuses to write a scam email. When the researchers turned up the scam-email feature strongly enough, the model drafted one. Anthropic noted this overrode the model’s harmlessness training.
Flattery. A user claimed to have invented the phrase “Stop and smell the roses”. Normally the model gently corrected them. With the sycophantic-praise feature turned up, it gushed instead, with flowery and untrue praise.

That’s the central move of the field: if changing a feature changes behaviour in exactly the way you’d predict, the feature is real and it matters. It’s the difference between noticing that a light is on and finding its switch.

Where the slider analogy breaks: real features aren’t independent knobs. Turning one up shifts the model’s state in a direction that also nudges related concepts, and pushing too hard tends to make the model incoherent. Steering is a research tool, not yet a clean control panel.

From words to grammar: circuits and attribution graphs

Features are the vocabulary. The next question is the grammar: how features connect, step by step, to produce an answer. A chain of features that work together is called a circuit.

In March 2025, Anthropic published two papers on this, a methods paper called “Circuit Tracing” and a case-study paper called “On the Biology of a Large Language Model”, plus a plain-English summary, “Tracing the thoughts of a large language model”. The model studied was Claude 3.5 Haiku.

The method, simplified:

They trained a cross-layer transcoder, a cousin of the sparse autoencoder with up to 30 million features for Haiku, that learns to imitate the feed-forward parts of the model using readable features.
Swapping it in gives a replacement model: a stand-in that behaves much like the original on a given prompt but is built from features you can name.
For one prompt, they then draw an attribution graph: a map of which features pushed on which other features, layer by layer, until the output token came out. Think of it as an MRI of a single thought.

Three findings stood out.

1. It plans the rhyme before writing the line

They asked for a rhyming couplet. The first line was “He saw a carrot and had to grab it”. You might expect the model to write the second line word by word and hope a rhyme turns up at the end. It doesn’t. Before the second line starts, a feature for “rabbit” is already active. Then the model writes a line that lands there: “His hunger was like a starving rabbit.”

They checked it by editing the plan. Suppress “rabbit”, and the model ended the line on “habit” instead. Inject “green”, and it wrote a sensible line ending in “green”, even though that no longer rhymed.

The model picks the rhyme word first and writes toward it; editing that plan changes the line it writes.

This matters beyond poetry. “It just predicts the next word” is true as a description of the output. It’s misleading as a description of what’s happening inside. The model can hold a goal several words ahead and steer toward it.

2. Its maths isn’t the maths it describes

Ask Claude what 36 + 59 is. Inside, the researchers found parallel paths: one working out roughly how big the answer is, another working out the exact last digit. They meet at 95. Then ask Claude how it got there, and it describes the school method: add the ones, carry the one.

That isn’t what the circuit shows. The model’s explanation of its own reasoning is a plausible story, not a readout. It learned to explain maths from human-written explanations, and it learned to do maths some other way.

3. Concepts are shared across languages

Ask for “the opposite of small” in English, French or Chinese, and the same core features for smallness and oppositeness light up first. Only later is the answer, a concept of largeness, put into the language of the question. Anthropic also reported that the shared fraction grows with model size: Claude 3.5 Haiku shared more than twice the proportion of its features across languages compared with a smaller model.

The same work also found a default “I can’t answer that” circuit that gets switched off when the model recognises a name it knows, and showed that misfires of that switch are one route to hallucination: the model recognises a name, the “I don’t know” switch turns off, but there’s no real knowledge behind it.

Limits and open questions

This is real progress. It’s also early, and the researchers are blunt about it.

We see a small part of the map. Anthropic called the features found in Claude 3 Sonnet “a small subset of all the concepts learned by the model during training”, and said finding a full set with current techniques “would be cost-prohibitive”. They added that the computation needed would “vastly exceed” the compute used to train the model in the first place.
Attribution graphs work some of the time. In the 2025 case-study paper, the authors wrote that the graphs gave them “satisfying insight for about a quarter of the prompts we’ve tried”. On short prompts, the method “only captures a fraction of the total computation”. Untangling one graph takes “a few hours of human effort”, for prompts of only tens of words.
The replacement model isn’t the model. The readable stand-in imitates the real one, with gaps the researchers call reconstruction errors or “dark matter”. The methods paper also lists missing attention circuits among its limitations. When you read a graph, you’re reading a careful approximation.
Sparse autoencoders aren’t always the best tool. In March 2025, Google DeepMind’s interpretability team published negative results: for detecting harmful intent in user prompts, a simple linear probe (a small classifier trained directly on the activations) beat probes built on sparse autoencoder features, especially on unfamiliar data. They decided to deprioritise fundamental SAE research for now, while saying SAEs remain a useful tool. Healthy science, and a useful reminder that a beautiful feature map isn’t automatically the most practical safety check.
Labels are human guesses. “Golden Gate Bridge” is an easy label. Many features are harder to name, and a name chosen from the top examples can miss what the feature does elsewhere.

The big open question is whether this can scale fast enough to matter. Models keep getting bigger and more capable. Interpretability keeps getting better too, but it’s not yet able to give a full account of why a frontier model said what it said.

What it means in practice

If you build with AI or buy AI products, here’s what I take from this.

1. Treat self-explanations as stories. When a model explains its reasoning, including the visible “thinking” of reasoning models, that text is generated output, not a view of the computation. The 36 + 59 result is the cleanest example I know. Judge a model by what it does on your test cases, not by how convincing its explanation sounds. This is the same reason I argue for treating the LLM like an unreliable API: check the outputs, don’t trust the narration.

2. Models build real concepts. “Autocomplete” undersells what’s inside. A concept that fires for English, Russian and a photo, and a plan that exists before the words that express it, are not surface tricks. That cuts both ways: it’s why these models generalise well, and why their mistakes can look so reasonable.

3. Watch steering, but don’t buy it yet. Steering a feature is a powerful idea for control, and the research has shown it can both help and hurt (the scam-email example went the wrong way). If a vendor tells you their model is “interpretable” or “steerable” in production, ask what that means concretely, and what tests back it up.

4. Expect better audits over time. The hope behind this field is an MRI for models: checking from the inside whether a model is being honest, flattering you, or hiding something. We’re not there. But for a field that’s only a few years old at this scale, the progress from “neurons are a mess” to “here’s the plan for the rhyme” is remarkable.

Key ideas

Superposition: models store far more concepts than they have neurons, as overlapping directions, so single neurons look like a jumble.
Dictionary learning with a sparse autoencoder rewrites the model’s activations as a few active features out of millions, and many features turn out to be readable concepts.
Features can be multilingual and multimodal: the Golden Gate Bridge feature fired on several languages and on a photo.
Steering (clamping a feature up or down) proves a feature matters. It produced Golden Gate Claude, and also showed a feature can override safety training.
Attribution graphs trace how features connect into circuits, revealing rhyme planning, parallel arithmetic and shared concepts across languages.
A model’s explanation of its own reasoning can differ from what it actually computed.
The field sees only a small part of what models do, and even its own tools have known gaps.

Sources: Anthropic, “Golden Gate Claude” (23 May 2024), anthropic.com/news/golden-gate-claude; Anthropic, “Mapping the Mind of a Large Language Model” (21 May 2024), anthropic.com/research/mapping-mind-language-model; Templeton et al., “Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet” (2024), transformer-circuits.pub; Anthropic, “Towards Monosemanticity: Decomposing Language Models With Dictionary Learning” (5 Oct 2023), anthropic.com; Elhage et al., “Toy Models of Superposition” (Sept 2022), arXiv 2209.10652; Gao et al., “Scaling and evaluating sparse autoencoders” (OpenAI, June 2024), arXiv 2406.04093; Lieberum et al., “Gemma Scope” (Google DeepMind, July 2024), arXiv 2408.05147; Anthropic, “Tracing the thoughts of a large language model” (27 Mar 2025), anthropic.com/research/tracing-thoughts-language-model; Ameisen et al., “Circuit Tracing: Revealing Computational Graphs in Language Models” (2025), transformer-circuits.pub; Lindsey et al., “On the Biology of a Large Language Model” (2025), transformer-circuits.pub; Smith et al., “Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research” (Google DeepMind, 26 Mar 2025), Alignment Forum.