How interpretability tools look inside a model

Why not just ask what each neuron does?

The idea

The obvious first plan for understanding a network is to go neuron by neuron: find what makes each one fire and write down a label. Attempts kept hitting neurons that fired for things that seemed unrelated, first in image models and later in language models. Anthropic's 2023 “Towards Monosemanticity” described a neuron in a one-layer transformer that responds to academic citations, English dialogue, HTTP requests and Korean text. These are called polysemantic neurons. (The Goodfire essay further down re-reads that neuron as most likely a “don't say a dot” neuron.)

The most influential explanation is superposition. A model may need to track far more concepts than it has neurons. If most concepts are rare, it can pack them in as directions that sit at angles to each other, accepting a little interference on the rare occasions two are active together. Anthropic's 2022 paper “Toy Models of Superposition” showed small networks doing exactly this: five features squeezed into two dimensions arrange themselves into a pentagon, which is the layout in the animation.

What the animation shows

1. Two dimensions give a flat plane, drawn here as two neurons. Each arrow is one feature the model wants to represent. (In the paper's pentagon the two dimensions aren't neurons; a separate experiment in the paper shows the same packing in neurons.)
2. When one feature switches on, the activation moves out along its arrow. Its shadow on each axis is what each neuron reports.
3. Almost every feature moves both neurons, so each neuron ends up responding to several unrelated features.
4. When two features fire together, their sum can point straight at a third feature. That's interference, the price of packing.

Why it shaped the field

If superposition is common, single neurons are the wrong unit of study, and you need tools that recover the hidden directions. That reasoning is a main motivation for sparse autoencoders, and for probing directions rather than individual neurons.

Now contested. In 2026 a Goodfire researcher argued that most neurons in real models are already interpretable, and that much apparent polysemanticity comes from weak tools and from neurons that grow related jobs, not from packing. See Latest research.

Is concept X present at this layer, in a form a straight line can read?

The idea

A probe is a small detector you attach to a model to test whether it “knows” something. Say you want to know whether a model tracks if a statement is true. You can't ask its activations directly. But you can show it many statements whose answers you already know, record its activations for each, and see whether a very simple classifier can sort them. If it can, the information must be in there.

“Linear” is the important word. The classifier may only draw a straight line (in high dimensions, a flat cut). That stops the probe from doing the thinking itself. If a straight cut works, the concept is linearly decodable at that layer: it lies along a direction, the format later layers could read most easily.

Step by step

1. Gather a labelled dataset, such as a few thousand true and false sentences.
2. Run each sentence through the model and save the activation vector at one layer, usually at the last token.
3. Train logistic regression on those vectors. It learns a weight vector w, the probe direction, and a bias.
4. Test on sentences the probe has never seen. Repeat at every layer to see where the concept appears and where it fades.

What probes have found

Probes have found linearly readable directions for truth, sentiment and a model's own uncertainty. In a model trained only on lists of Othello moves, probes recovered the board state: the first study (Li et al., 2022) needed nonlinear probes, and the board became linearly readable once it was encoded as “mine versus the opponent's” (Nanda et al., 2023). Probes are cheap, fast and often match fancier methods at simply detecting a concept, which keeps them a workhorse in safety research, for example as monitors for deceptive or harmful intent.

What it can't tell you. A probe shows the information is there, not that the model uses it. Information can sit unused, like a note in a margin nobody reads. A powerful probe can also learn the task itself, so researchers keep probes simple and compare against control tasks with random labels. To show the direction matters, follow up with patching or steering.

What features is the model representing, without choosing labels first?

The idea

A probe needs you to know what to look for. A sparse autoencoder (SAE) tries to list everything, without labels. Think of an activation as a chord played on a piano with thousands of keys. You hear one blended sound; the SAE tries to work out which few keys were pressed.

An SAE is a small extra network trained on one layer's activations. It expands each vector into a much longer list of possible features, then rebuilds the original vector from that list. A sparsity penalty allows only a handful of features to be on at once. To rebuild well under that rule, each feature tends to settle on one recognisable thing.

Step by step

1. Collect activations from one layer over millions to billions of tokens of ordinary text.
2. Encoder: multiply each vector by a wide matrix, add a bias and apply ReLU, giving one score per feature. Most scores end up at exactly zero.
3. Decoder: add up each active feature's direction, scaled by its score, to rebuild the activation.
4. Train so reconstruction error stays low while the number of active features (called L0) stays small.
5. Label each feature by reading the texts that activate it most, often with another model's help, then check the label by steering with it.

What SAEs have found

Anthropic's 2024 work on Claude 3 Sonnet found millions of features, from concrete ones like the Golden Gate Bridge to abstract ones like bugs in code, sycophancy and inner conflict. Turning the bridge feature up produced a model that brought the bridge into most of its replies. Open SAE suites such as Gemma Scope now let anyone explore features in public models.

What it can't tell you. Features are found, not guaranteed. As SAEs grow, features split into finer, more specific ones (feature splitting). A broad feature can fail to fire where a narrower one has absorbed it (feature absorption). And whatever the SAE fails to reconstruct can hide what matters. In 2025 Google DeepMind's interpretability team reported SAEs underperforming plain linear probes at detecting harmful intent in user prompts, tested on jailbreaks the detectors hadn't seen. The 2026 Goodfire essay below goes further: an SAE trained on a deliberately squeezed layer did not recover the clean features of the full model, and splicing it in cost a lot of accuracy.

What would the model predict if it stopped at this layer?

The idea

At the very end of a model, the final residual vector is turned into a prediction by the unembedding: a matrix that scores how well the vector matches every word in the vocabulary. The logit lens, first described by the blogger nostalgebraist in 2020, asks a cheeky question. What if you applied that same final step to a vector from the middle of the model?

It works surprisingly often, because every layer writes into the same shared stream and later layers mostly refine rather than replace. A middle-layer vector is a rough draft of the final answer, and the lens lets you read the draft.

Step by step

1. Run the prompt and save the residual stream after every layer.
2. For each layer, apply the model's final layer norm, then the unembedding matrix.
3. Softmax the result to get a probability for every token in the vocabulary.
4. Track when the correct answer first appears and how its probability grows.

What it shows

Predictions usually form gradually, the answer's probability rising over the later layers. In Llama-2 working in other languages, a lens study found that the middle layers often decode the right next word in English first, before the model switches to the input language (Wendler et al., 2024). The same trick works on single directions: pushing a probe or SAE direction through the unembedding shows which words it promotes.

What it can't tell you. Early layers often aren't in a format the unembedding understands, so they decode to noise even when the information is there. The tuned lens (2023) fixes much of this by training a small translator for each layer. Neither shows why the guess changes; for that you need patching.

Which activations actually cause the model's answer?

The idea

Reading tools find correlations. Patching tests cause, using a transplant. Take two prompts that are identical except for one key detail, so the model gives different answers. Then move a single activation from one run into the other. If that one transplant brings the other answer back, that activation was carrying the information that mattered.

Step by step

1. Pick a clean prompt (“The Eiffel Tower is in” → Paris) and a corrupted one that changes the key fact (“The Roman Colosseum is in” → Rome).
2. Run the clean prompt and save every activation at every layer and position.
3. Run the corrupted prompt, but at one chosen layer and position, overwrite the activation with the saved clean one.
4. Measure how much of the clean answer comes back, usually as the change in the gap between the “Paris” and “Rome” scores.
5. Repeat for every layer and position to build the heatmap. The same works for single heads or single MLPs.

What it has found

The heatmap's shape resembles causal tracing of how models recall facts (Meng et al., 2022), which corrupted the subject by adding noise rather than swapping prompts. It found that MLPs in the middle layers, at the subject's last token, look the fact up, and attention then carries it to the final position. Patching is the backbone of most circuit discoveries, including the attention circuit later on this page.

What it can't tell you. Results depend on how you corrupt the input: swapping in a different city, adding noise, and averaging over many prompts can each give different maps. Patching shows where the important information sits, not how it's computed. Path patching and attribution graphs narrow it to specific connections.

Does this direction control the model's behaviour?

The idea

If a concept lives along a direction, you should be able to turn it up or down like a dial. Steering does exactly that: while the model runs, add a multiple of the direction to the residual stream and see what it writes. It's the most direct test that a direction means what you think, and it doubles as a way to control a model without retraining it.

Step by step

1. Get a direction v: a probe's weight vector, an SAE feature, or the average activation on cheerful prompts minus the average on gloomy ones.
2. Pick a layer, usually somewhere in the middle.
3. On every forward pass, replace the residual vector h with h + αv.
4. Generate text across a range of α, and check the change matches your hypothesis and nothing else.

What it has found

Simple difference-of-means directions can steer sentiment, topic, honesty and refusal. In 2024 researchers found that removing a single “refusal direction” from 13 open chat models largely disabled their refusals, a striking sign of how linear some behaviours are. Steering is also how many interpretations get checked: when Goodfire steered a “Colorado” neuron in a Qwen model, the model wanted to go hiking, which supported the label.

What it can't tell you. Large α pushes activations somewhere the model never saw in training, and the text degrades, as at the ends of the slider. A direction can also carry side effects you didn't intend, so steering results need checking on unrelated tasks.

Where is this head looking, and what is it moving?

The idea

Imagine every word in a sentence holding up two cards. The key says “here's what I am”. The value says “here's what I'll tell you if you look at me”. The word currently being processed holds up a query: “here's what I'm looking for”. Attention matches the query against every key, then collects values from the best matches.

Each layer has many heads (12 per layer in GPT-2 small), and each has its own query, key and value maps, so each can look for something different: the previous word, the subject of the sentence, a matching bracket.

Step by step

1. Each token's residual vector is multiplied by three learned matrices to make its query, key and value.
2. The current token's query is dot-producted with every earlier token's key, giving one score per token. Later tokens are masked out, so the model can't peek ahead.
3. Softmax turns the scores into weights between 0 and 1 that add to 1. This row of weights is the attention pattern.
4. The values are summed using those weights, passed through an output matrix, and added back into the current token's residual stream.

Two circuits in one head

Researchers split each head into two independent pieces. The QK circuit (steps 1 to 3) decides where to look. The OV circuit (step 4) decides what gets moved. A head can look in the right place and copy something useless, so you study both. Pattern heatmaps show QK. OV is studied by pushing a head's output through the unembedding, like a logit lens for one head.

What it can't tell you. A pattern shows where a head looks, not what it does with what it finds. Many heads dump spare weight on the first token or on punctuation as a resting spot, called an attention sink, so a bright cell can mean “nothing to do here”. Read QK and OV together, and test with patching.

How does a model copy a pattern it has seen only once, earlier in the prompt?

The idea

Read “Mr and Mrs Dursley … Mr D” and you'll guess “ursley”. You aren't recalling a fact; you're noticing the pattern happened before in the same text. Induction heads give models the same trick: if A was followed by B earlier, and A appears again, predict B.

No single head can do this alone, because a query can only match on what a token holds. The solution is two heads in sequence. An earlier previous-token head writes “the token before me was A” into each position. The later induction head, sitting on the second A, searches for a position tagged “preceded by A”, finds B, and copies it forward. The animation shows the simplest place this appears: a two-layer, attention-only toy model.

How researchers find them

1. Generate a run of random tokens and repeat it. Random tokens guarantee the model can't lean on real-world knowledge.
2. Record every head's attention pattern on the repeated half.
3. Score each head by how much weight it puts on “the token right after my earlier copy”. That's the offset stripe in the right-hand heatmap.
4. Confirm by ablation: knock out the high-scoring heads and loss on the repeated half shoots back up.

Why they matter

Anthropic's 2022 study showed induction heads form suddenly in a short window early in training, visible as a bump in the loss curve, and that in-context learning improves at that same moment. They were one of the first mechanisms found in tiny models that also show up in large ones.

What it can't tell you. The stripe identifies one behaviour on one test. Real induction heads also do fuzzier matching, such as completing patterns across a translation, and take on other jobs. The name is a summary, not a full description.

Which heads produce this answer, and what happens without them?

The idea

Before the final layer norm, the residual stream is literally a sum: the token and position embeddings, plus every head's output, plus every MLP's output. Treating the layer norm's scaling as fixed, you can split the final answer into one contribution per component. That's direct logit attribution (DLA). Pick the right answer (“Mary”) and a wrong one (“John”), project each head's output onto the direction that separates them, and you get one number per head saying how hard it pushed.

Attribution only reads. Ablation tests: remove a head's output (set it to zero, or better, replace it with its average) and measure how much the answer suffers.

Step by step

1. Choose a task with a clear right and wrong token, like this indirect-object sentence.
2. Run the model and save every head's output at the final position.
3. Dot each output with the difference between the “Mary” and “John” unembeddings. Plot the 144 numbers as a grid.
4. Ablate the biggest contributors and re-measure the score.
5. Use patching to find heads that matter indirectly, by influencing other heads.

What it found

Wang et al. (2022) mapped this task in GPT-2 small to 26 heads in seven main classes. Name mover heads attend to the correct name and copy it. S-inhibition heads stop the name movers from copying the repeated name (“John”). Negative name movers push against the answer (the authors speculate this helps the model hedge), and backup name movers stay quiet until the main ones are removed, then take over. It's still one of the most complete circuits mapped in a real language model.

What it can't tell you. DLA only sees a head's direct effect on the output, so the S-inhibition heads score near zero despite being essential. Self-repair means a single ablation can understate a head's importance: with the name movers removed, the paper measured only a 5% drop in the logit difference. Head numbers follow the published study; the values shown are illustrative.

Is superposition really the reason neurons look messy?

In a long essay, “Neural Networks Are Already Clean”, Goodfire's Nick Cammarata (the language-model section with Curt Tigges) argues the field may have taken a wrong turn. His claim is that most neurons in real networks are interpretable as they are, so researchers can study the network directly instead of first untangling it with tools like SAEs. Cammarata co-authored the 2020 circuits work whose car-and-dog example is widely cited as evidence for superposition, which gives the revision extra weight.

The core idea: recursive generalisation

Superposition says a neuron holds unrelated features packed together to save space. The essay proposes a different reason a neuron can look mixed. A neuron learns one job, then generalises it a small step, then generalises the new version another step, like a plant sending out offshoots. After a few steps the far ends look unrelated, but each step was small and sensible. The animation shows the essay's “iy” example of this, then two neurons the essay explains in other ways: one stores a running summary at commas, and one does a single job in several encodings.

The two explanations predict different things. Generalisation predicts a neuron's jobs are locally connected by a chain of small steps. Superposition predicts non-local groupings with no chain at all. The essay notes you can tell them apart by watching a neuron form across training checkpoints.

Five reasons a neuron can look messy

1. Bad tools. Feature visualisations that were reward-hacked, contexts too short to show long-range neurons, or methods that only looked backward. The “unrelated jobs” of one “Titanic” neuron were often detecting that “boat” comes next.
2. Bad explanations. Built from those weak tools: for example, listing the nouns a neuron fires on (cats, cars) when what they share is a V shape.
3. Bad learning. The network learned poorly because of its architecture, training data or setup; InceptionV1, for one, had no residual stream or layer norm.
4. Recursive generalisation. One job that grew outward.
5. Superposition. Unrelated features packed together.

Superposition's only symptom is “this neuron doesn't make sense”, which the other four produce too. So the essay argues it should be the last explanation checked, not the first one assumed.

The evidence offered

In InceptionV1, a classic vision model, the share of interpretable neurons rose as tools improved (the first figure is the author's recollection; others put it near zero), rose again when the same design was retrained with modern regularisation, and was near-total in a modern ConvNeXt of the same size. The network didn't change in the first jump; only the microscope did.

In language models, preliminary audits of Qwen3-32B found about 90% of neuron poles with one main job, often with small related offshoots.

A pressure test pointed the same way. InceptionV1 retrained from scratch with one layer cut from 480 neurons to 10 still worked, at 58.35% top-1 accuracy on ImageNet. If superposition were the network's main strategy, those 10 neurons should be packed with features. Instead they became coarser, broader detectors. An SAE trained on the squeezed layer did not recover the clean curve detectors of the full model, and splicing it in dropped accuracy from 58.8% to 43.0% (measured on 10,000 validation images).

What it means for the tools on this page

• Neurons first. Study the network's own neurons and circuits before decomposing them, and keep a neuron's grouped meanings together rather than splitting them apart.
• SAEs as a fallback. Still useful where a network really is knotted, but the essay argues the field reached for them too early.
• Attribution graphs explain a neuron through its neighbours, which helps less when those neighbours are themselves unexplained.
• Steering stays central as the check on any explanation.
• Attention and punctuation. Attention often gathers on commas and full stops, so neurons there can carry a running summary of the text (the essay's “summary motif”). So a bright attention cell on punctuation isn't always a resting spot (a sink); it can be where a summary is stored.
• Use the leftovers. The unexplained remainder is the best place to hunt for bugs in your tools.

How settled is this? Not very. The author calls it a working hypothesis. Several numbers rest on small samples and on judgement calls about what counts as “one job”. Detailed language-model results are promised for a forthcoming companion paper; the essay reports similar results in models with trillions of parameters, but says full audits at that scale haven't begun. The essay also sets aside the weaker sense of superposition, simply having more features than dimensions. Treat it as a strong claim to test, not a consensus.