Part II, Sparse autoencoders
From features to circuits
An SAE describes what a layer represents. Transcoders, crosscoders and cross-layer transcoders try to describe what it computes, and attribution graphs chain them together to trace a single prompt.
Assumes pieces 6 and 8; how attention and MLP layers write into the residual stream.
An SAE reads an activation and writes back the same activation. That makes it a description of a representation. To explain a computation, we need units that read what goes into a layer and write what comes out of it.
Transcoders replace an MLP layer with a sparse approximation of its input-to-output map: encode the MLP’s input, decode to its output.Dunefsky, Chlenski and Nanda, Transcoders Find Interpretable LLM Feature Circuits, 2024. Because the replacement is sparse and its decoder is linear, the effect of one transcoder feature on a later one is a single number per pair, and circuits can be read off directly. Skip transcoders add a linear bypass for the part of the map that is not sparse. On the same model and data, transcoders turned out more interpretable than SAEs.Paulo, Shabalin and Belrose, Transcoders Beat Sparse Autoencoders for Interpretability, 2025.
Crosscoders read from and write to several layers at once, or to the same layer in two models.Lindsey et al., Transformer Circuits Thread, 2024. The two-model form is a tool for model diffing: which latents belong only to a chat model, and which it shares with its base model? The choice of objective turned out to matter. L1 crosscoders falsely labelled shared concepts as chat-only. BatchTopK crosscoders largely fixed this and recovered genuinely chat-specific latents in Gemma 2 2B, for false information, personal questions and several kinds of refusal.Minder et al., 2025.
Cross-layer transcoders let each feature read from one layer and write to the MLP outputs of every later layer. A feature that a model computes once and then uses repeatedly appears once, instead of being duplicated in every layer’s dictionary.
Attribution graphs
Replace every MLP in a model with a cross-layer transcoder. For one prompt, freeze the attention patterns at the values the real model computed. Add an error node at each position for whatever the transcoders fail to reconstruct. In this replacement model, the path from input tokens through features to an output logit is linear, so the direct effect of every feature on every other can be computed and then pruned to a readable graph.Ameisen et al., Circuit Tracing: Revealing Computational Graphs in Language Models, Transformer Circuits Thread, March 2025.
The numbers are worth stating plainly. Anthropic’s cross-layer transcoder on Claude 3.5 Haiku had 30 million features and left 21.7% of normalised reconstruction error. In an 18-layer test model, the replacement matched the real model’s next token about half the time. Anthropic reported satisfying insight on about a quarter of the prompts it tried.
The method spread quickly once it was released. The open-source circuit-tracer, built with Anthropic Fellows and Decode Research and running on Gemma-2-2B and Llama-3.2-1B through Neuronpedia, produced almost 5,000 user-made graphs in its first two weeks.Related work: Sparse Feature Circuits (Marks et al., 2024) used SAE features to find and edit circuits, including removing gender features from a profession classifier (SHIFT). Lorsa (2025) builds a sparse replacement for attention heads. CLT-Forge (2026) trains cross-layer transcoders across many machines.
Why must attention patterns be frozen for the replacement model to be linear from features to logits? What kind of mechanism does this hide?
Solution
An attention pattern is a softmax of products of activations, so it depends on the features themselves. Holding it fixed makes each attention layer a fixed linear map for this prompt. What it hides is any mechanism that works by changing where the model attends, for example a feature that matters because it moves attention to a different token.