Wider than the input

Make the code wider than the input, $m > n$, and the bottleneck is gone. Many settings of the weights reconstruct perfectly, and nothing in the loss prefers one of them. An overcomplete autoencoder needs a different constraint, and the field has tried several:

Regulariser	Idea	Introduced
sparsity penalty	most latents should be zero on any input	Ranzato et al., 2006; Ng’s notes, 2011
denoising	reconstruct the clean input from a corrupted copy	Vincent et al., 2008
contractive	penalise the encoder’s Jacobian, so the code ignores small changes	Rifai et al., 2011
k-sparse	keep only the $k$ largest latents	Makhzani and Frey, 2013

The first and last are the ancestors of the sparse autoencoders used in interpretability today.Ranzato, Poultney, Chopra and LeCun, NIPS 2006; Vincent, Larochelle, Bengio and Manzagol, ICML 2008; Rifai et al., ICML 2011; Makhzani and Frey, k-Sparse Autoencoders, ICLR 2014. Ng’s CS294A notes penalised the KL divergence between each unit’s mean activation and a small target rate ρ. The TopK SAEs that OpenAI trained on GPT-4 in 2024 build directly on the k-sparse autoencoder of 2013.

Without a penalty

The figure trains a small autoencoder live, in your browser. The data are 5 sparse features in the plane, the superposition setting of piece 5. The autoencoder has 8 ReLU latents with unit-length decoder directions, and starts with a sparsity penalty of 0.00.

With no penalty the reconstruction becomes essentially perfect and the code is useless for reading: several latents fire on each input, and their directions bear little relation to the dashed lines. Any two non-parallel directions span the plane, so the autoencoder has no reason to find the features.

Now raise $\lambda$ to about 0.3. The error rises a little, the number of active latents falls toward the data’s own, and the decoder directions swing onto the true features. Push $\lambda$ further and latents die or merge, and the error climbs. This trade-off is the subject of pieces 5 to 8.

On images

The same experiment on the digit images. The autoencoder now has the shape of the SAEs in pieces 5 to 8: one linear layer into 32 ReLU latents, one linear layer back out (with a sigmoid to keep pixels between 0 and 1), and an L1 penalty on the latents. Each tile in the top row is one decoder column, the image a latent adds when it fires. The bars under them show how often each latent fires.

With the penalty off, the code is dense. About 28 of the 32 latents fire on a typical image and the reconstruction is good, around 17% unexplained, but each column is a speckled, signed pattern: every image is a large sum of mostly cancelling parts. With $\lambda =$ 0.10, about 5 latents fire per image. Reconstruction is worse, about 38% unexplained, and many latents die. The columns that survive are smooth and legible: most look like a template for one digit or a few similar ones, drawn in blue, with red where a neighbouring digit has ink this one lacks. An image is now a template or two plus small corrections, and its decomposition can be read off term by term. Add input noise to make it a denoising autoencoder as well.

The encoder as one step of sparse coding

Classical sparse coding fixes a dictionary $D$ and finds the code by optimisation:

$$f^\star(x) = \arg\min_{f \ge 0}\; \tfrac12\lVert x - D f\rVert^2 + \lambda\lVert f\rVert_1 .$$

A standard solver is ISTA, iterative shrinkage-thresholding.Daubechies, Defrise and De Mol, 2004. Each step takes a gradient step on the reconstruction term, then applies the soft threshold. For non-negative codes:

$$f \leftarrow \mathrm{ReLU}\big(f + \eta D^\top (x - D f) - \eta\lambda\big).$$

Start from $f = 0$ and take a single step:

$$f = \mathrm{ReLU}\big(\eta D^\top x - \eta\lambda\big).$$

This is exactly a sparse autoencoder’s encoder, with $W_{\mathrm{enc}} = \eta D^\top$ and $b_{\mathrm{enc}} = -\eta\lambda$. An SAE learns $W_{\mathrm{enc}}$ and $b_{\mathrm{enc}}$ freely instead of tying them to the dictionary, so it is a single learned step of ISTA. Unrolling several steps and learning them is LISTA.Gregor and LeCun, Learning Fast Approximations of Sparse Coding, ICML 2010.

One step has a cost. Full sparse coding lets latents compete: if two dictionary atoms overlap, later iterations let the better one explain away the other. A one-step encoder cannot do this, so on the same dictionary its codes are less sparse or less accurate than the optimum. This difference is the amortisation gap, and it grows as dictionary atoms become more similar, which is precisely the superposition regime.Later work quantifies this gap for SAEs and finds that spending more computation at inference time recovers better codes than a single linear-plus-ReLU step: O’Neill et al., 2024. SAEs accept the gap in exchange for speed. One matrix multiply per input is what makes it possible to encode billions of activations.

Where the first four pieces leave us

• An autoencoder learns what its constraint forces it to keep.
• Reconstruction alone fixes a subspace, not a basis.
• Sparsity picks a basis, and when the data really are sparse along some directions, it picks those.
• A sparsity penalty is a sparse prior: L1 is MAP inference under a Laplace prior, while the Gaussian prior of a VAE cannot pick axes.
• A wide ReLU autoencoder with an L1 penalty is amortised sparse coding.

Piece 5 applies this machine to the activations of a language model.