Thirty thousand times larger

The first dictionaries had about a thousand features. By May 2024 there was one with 34 million.

Work	Model	Arch.	Scale and what was reported
Scaling Monosemanticity, Anthropic, May 2024	Claude 3 Sonnet, middle residual stream	L1	1M, 4M and 34M features; fewer than 300 active per token; reconstructions explain at least 65% of variance
Gao et al., OpenAI, June 2024	GPT-4	TopK	16M latents, 40B training tokens; spliced loss equals a model trained with about 10% of GPT-4’s compute
Gemma Scope, DeepMind, Aug 2024	Gemma 2 2B and 9B, selected 27B layers	JumpReLU	every layer and sublayer; more than 400 SAEs; about 15% of Gemma 2 9B’s training compute; about 20 PiB of stored activations
Llama Scope, Fudan, 2024	Llama-3.1-8B		256 SAEs, 32K and 128K features each
Gemma Scope 2, DeepMind, Dec 2025	Gemma 3, 270M to 27B, base and instruction-tuned	mixed	SAEs, transcoders, skip and cross-layer transcoders, some Matryoshka; about 110 PB stored

Compute and storage figures are the developers’ own.Templeton et al., Transformer Circuits Thread, 2024; Gao et al., Scaling and evaluating sparse autoencoders, 2024; Lieberum et al., Gemma Scope, 2024; He et al., Llama Scope, 2024.

Scaling laws

Gao and colleagues found that reconstruction loss falls as a smooth power law in the number of latents $m$, in the number of active latents $k$, and in training tokens.Fitting these laws was made easier by TopK, which fixes $k$ directly instead of leaving it to be set indirectly through $\lambda$. See piece 8. The laws make dictionary size a budgeting question. They do not say whether a larger dictionary is more useful, which is a different question taken up in piece 11.

The GPT-4 number deserves a second reading. With 16 million latents, splicing the reconstruction back in still degraded GPT-4 to the level of a model trained with about a tenth of its compute. Even very large dictionaries leave a gap in fidelity.

Dead latents

Large dictionaries waste capacity. A dead latent never fires on the training distribution. On Claude 3 Sonnet the dead fraction rose from about 2% at 1M features to 35% at 4M and 65% at 34M. OpenAI’s TopK runs kept it to 7% at 16M by initialising the encoder to the transpose of the decoder and adding an auxiliary loss that asks the dead latents to model the residual error.

The open suites

Frontier runs showed that the method works on production models. The open suites are what made academic research possible: Gemma Scope, Llama Scope and EleutherAI’s sparsify (TopK SAEs and transcoders on Llama 3 8B) give anyone a trained dictionary for every layer of a capable model. They are also expensive. Gemma Scope’s 15% of a 9B model’s training compute is not a rounding error, and it is one reason the field has looked for cheaper training.A November 2026 scaling-laws paper reports that a small replayed activation buffer matches quality while cutting storage by 8 to 64 times. This is a recent result and has not yet been peer reviewed.