A wide, sparse dictionary

A sparse autoencoder takes an activation $x \in \mathbb{R}^n$, say 512 numbers from one layer, maps it to a much wider vector of $m$ latents, say 4,096, and maps that back to $n$ numbers. A ReLU in the middle sets most latents to exactly zero.

$$\begin{aligned} f(x) &= \mathrm{ReLU}\big(W_{\mathrm{enc}}(x - b_{\mathrm{dec}}) + b_{\mathrm{enc}}\big),\\ \hat x &= W_{\mathrm{dec}}\, f(x) + b_{\mathrm{dec}} \;=\; b_{\mathrm{dec}} + \textstyle\sum_i f_i(x)\, d_i . \end{aligned}$$

The columns $d_i$ of $W_{\mathrm{dec}}$ are the dictionary. Each is a candidate feature direction, and $f_i(x)$ says how strongly it is present in $x$.Subtracting $b_{\mathrm{dec}}$ before encoding centres the input. Decoder columns are usually constrained or normalised to unit length, so that the sparsity penalty cannot be dodged by shrinking $f$ and growing $d$.

Training

The original objective is reconstruction error plus an L1 penalty on the latents:

$$\mathcal{L}(x) = \lVert x - \hat x\rVert_2^2 + \lambda\, \lVert f(x)\rVert_1 .$$

What we actually want to penalise is the number of active latents, $\lVert f\rVert_0$. The L1 norm is its convex stand-in, and the substitution has a cost. L1 charges for size as well as activity, so the optimum pulls every active latent toward zero. This bias is called shrinkage, and most of the architecture work in later pieces sets out to remove it. The coefficient $\lambda$ trades sparsity against reconstruction, so every trained SAE is one point on a sparsity–fidelity frontier.

What gets reported

Measure	Definition	Reading
$L_0$	$\mathbb{E}_x\,\lVert f(x)\rVert_0$	average active latents per token
FVU	$\dfrac{\mathbb{E}\lVert x-\hat x\rVert^2}{\mathbb{E}\lVert x-\bar x\rVert^2}$	fraction of variance unexplained
loss recovered	$\dfrac{\mathcal{L}_{\text{abl}} - \mathcal{L}_{\text{SAE}}}{\mathcal{L}_{\text{abl}} - \mathcal{L}_{\text{clean}}}$	splice $\hat x$ into the forward pass; share of the gap to zero- or mean-ablation that is closed

The third measure is the one that tests whether the dictionary keeps what the model uses. A reconstruction can explain most of the variance and still damage next-token prediction, if the variance it misses is the variance that matters. Interpretability is then scored separately, usually by a language model that writes an explanation of a latent from its top-activating examples and is then tested on predicting new activations from that explanation.

The first results, 2023

Two papers in the autumn of 2023 established the method.

Cunningham and colleagues trained ReLU+L1 SAEs on the residual stream and MLP outputs of Pythia-70M and Pythia-410M.Cunningham, Ewart, Riggs, Huben and Sharkey, Sparse Autoencoders Find Highly Interpretable Features in Language Models, ICLR 2024. The proceedings list the authors as Huben, Cunningham, Smith, Ewart and Sharkey. Residual dictionaries were mostly twice the activation width, with $\lambda = 8.6 \times 10^{-4}$. By automated interpretability scores their features beat the neuron basis, PCA, ICA and random directions in early layers. By layer 4 of Pythia-70M, ICA was comparable. Fidelity was the weak point: splicing the reconstruction into Pythia-70M at layer 2 raised Pile perplexity from 25 to 40. On the indirect-object identification task in Pythia-410M, patching SAE features made smaller and more targeted edits than patching PCA directions. That advantage disappeared when the sparsity penalty was removed.

Bricken and colleagues at Anthropic worked with a one-layer transformer with a 512-neuron MLP, about 8 billion activation samples, and dictionaries from 512 to 131,072 features, studying a 4,096-feature run in depth.Bricken et al., Towards Monosemanticity, Transformer Circuits Thread, October 2023. Their most consequential observation was feature splitting. As the dictionary grows, a feature such as one for base64 text divides into several narrower, related features. Dictionaries of different sizes give overlapping, partly hierarchical vocabularies, and nothing singles out one size as correct.