The part you didn’t fit

Here is a rule, and some points scattered around it by noise. You do not get to see the rule, only the points. Your job is to draw a curve through them that will also pass close to the next points, the ones that have not arrived yet.

The curve in the figure is a polynomial of degree 3. Degree zero is a flat line, degree one a slope, and every degree above that adds one more bend the curve is allowed to make. More bends, more flexibility.

Two kinds of error

Raise the degree and watch the faint line in the lower panel, the error on the points the curve has seen. It only ever goes down. Give a curve enough bends and it can pass through every point it is shown, noise included, and report an error of nothing at all.With k points, a polynomial of degree k − 1 can always pass through all of them exactly.

The blue line tells the truth. It is the error on the hollow points, the ones held back. It falls at first, while the extra bends are capturing the real shape of the rule. Then, somewhere, it turns and climbs: the bends are now spent on the noise in the points it saw, and the noise in the points it didn’t see is different.

That turning point is the central problem of this course in miniature. A polynomial stands in for a network here because its flexibility is a single number; with networks, width, depth, training time and regularisation all move the same dial. Below it, the model is too stiff to follow the rule; above it, the model has started memorising accidents.The dotted line marks the noise itself. No curve, however good, should beat it on points it has not seen. A held-out error below that line is luck.

Things worth breaking

• Raise the noise to 0.25 and beyond. The best degree drops: when the points are less trustworthy, a stiffer curve is the safer bet.
• Increase the number of points. Overfitting needs room to hide; with plenty of data the blue line stays low for longer.
• Press draw again a few times. The best degree moves around from one draw to the next. Choosing the degree is itself a fit, and it can overfit too.

That last point is why careful work keeps a third set of data, untouched until the very end, and looks at it once.In the language of this course’s title: the data you learned from is in sample. Everything the model will actually be used on is out of sample.

What changes with networks

Two things complicate the picture for neural networks. First, training time behaves like flexibility. Early in training a network has effectively used only part of its capacity, so stopping when the held-out loss turns up, early stopping, is a form of regularisation.

Second, very large networks can pass the point where they fit the training data exactly, and their held-out error can fall again.This is “double descent” (Belkin et al., 2019). Raise the flexibility here past the number of seen points and you will see only the first half of the story: a polynomial has no way of choosing among the many curves that fit exactly. Which interpolating solution training finds then matters, and that depends on the optimiser and the initialisation. It is one reason the first piece in this part spent so long on how descent moves.