Measuring a dictionary

Every architecture in piece 8 was justified by a better sparsity–fidelity frontier. The obvious next question is whether a better frontier gives a better tool.

Benchmarks

SAEBench evaluated more than 200 open SAEs on Gemma-2-2B and Pythia-160M, covering seven architectures, three widths and sparsities from 20 to 1,000 active latents, on eight measures.Karvonen et al., SAEBench, 2025. Its headline finding is that “gains on proxy metrics do not reliably translate to better practical performance.” Absorption got worse with width for every architecture except Matryoshka. As a working compromise it suggests $L_0$ between 50 and 150.

Explanations

Automated interpretability scores say SAE latents are much easier to explain than neurons, even neurons sparsified to their top $k$.Paulo et al., Automatically Interpreting Millions of Features in Large Language Models, 2024. How much those scores can be trusted is less settled. An earlier study of neuron explanations found that “even the most confident explanations have high error rates and little to no causal efficacy.”Huang et al., Rigorously Assessing Natural Language Explanations of Neurons, 2023.

A 2026 preprint re-examined Neuronpedia’s fuzzing runs for 500 GPT-2 Small features. In 91.3% of the activating examples, the text had been cropped so that the peak activation sat on the centre token. A rule that answers “yes if the centre token is marked” reached a balanced accuracy of 0.979, against 0.838 for the stored LM scorers. On OpenAI’s simulation benchmark, a rule using the first five tokens reportedly beat the published GPT-4 simulator.Chen, arXiv:2610.37676, October 2026: a single-author preprint, not yet reviewed. Reported in the bitfilter review. If this holds, part of what these scores measure is the format of the examples rather than the quality of the explanations.

Against supervised methods

When the target concept is known, an SAE can be compared with a method trained to find that concept. On indirect-object identification, SAEs gave less control than a supervised dictionary, because of feature occlusion and over-splitting.Makelov, Lange and Nanda, 2024. On RAVEL, open GPT-2 SAEs “struggle to reach the neuron baseline, and none come close” to the supervised method.Chaudhary and Geiger, 2024.

AxBench made the comparison at scale on Gemma 2, for both detecting a concept and steering generation toward it.Wu et al., AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders, 2025.

Difference-of-means directions and linear probes detect concepts at about 0.94 AUROC; SAE latents reach 0.695. For steering, SAE latents scored lowest of the methods tested. Sparse probing tells a similar story: across 113 binary datasets, Gemma Scope SAE probes on Gemma-2-9B beat baselines on 2.2% of them.Kantamneni et al., Are Sparse Autoencoders Useful? A Case Study in Sparse Probing, 2025.

One result points the other way. Choosing latents by their output effect, rather than by what makes them fire, gives two to three times better steering.Arad, Mueller and Belinkov, 2025. Part of the gap may come from how latents are selected rather than from the dictionary itself.Jørgensen and Hansen (2026) report Gemma Scope steering close to LoRA when latents are chosen with supervision. Reported in the bitfilter review; not checked against the paper.