Out of Sample
A course in applied neural networks.
The figures here move: push them. Every setting you reach is kept in the address, so a link carries it.
Select any passage to write beside it, in the left margin.
Turn with the arrow keys, the space bar, a swipe, or the arrows at the top right.Part I, Optimisation, generalisation, adaptationWinter 2027
Part I
- Downhill
Training a network is walking down a surface you cannot see, one gradient at a time. How big a step you take matters more than almost anything else.
- The part you didn’t fit
A model judged on the data it learned from will always flatter itself. Hide some of the data, and the flattery stops.
- A small change to a large matrix
Fine-tuning a pretrained model rarely needs to move every weight. LoRA trains a low-rank correction instead, and loses surprisingly little.
- What survives the bottleneck
An autoencoder learns by copying its input through a code. Make the code narrow, make it a distribution, or make it sparse, and you get three different models that answer three different questions.