Downhill

Every network you will train in this course is fitted the same way. You have parameters $\theta$, a loss $L(\theta)$ that measures how wrong the network is on the training data, and the gradient $\nabla L(\theta)$, which points uphill. You step the other way:

$$\theta_{t+1} = \theta_t - \eta \, \nabla L(\theta_t).$$

The step size $\eta$ is called the learning rate.“Rate” is a misnomer we are stuck with. It is a length scale, not a speed: the distance moved is $\eta$ times the length of the gradient. Real networks have millions of parameters, but almost everything that goes right or wrong with gradient descent can be seen with two. The figure below shows two parameters, their loss as contour lines, and the path the parameters take.

The step that is too large

Start on the bowl at a step size of 0.60 and drag it upward. At first the path just gets there sooner. Past 1 it starts to zig-zag across the steep direction of the bowl, and past 2 it stops converging and flies off.

That threshold is no accident. Near a minimum every smooth loss looks like a quadratic, $L(\theta) \approx \tfrac12 \theta^\top H \theta$, where $H$ is the Hessian. Write $\theta$ in the eigenbasis of $H$, with eigenvalues $\lambda_i$. Each coordinate then evolves on its own:

$$\theta^{(i)}_{t+1} = (1 - \eta \lambda_i)\, \theta^{(i)}_t .$$

A coordinate shrinks only if $|1 - \eta\lambda_i| \lt 1$, that is, if $\eta \lt 2/\lambda_i$. The largest curvature sets the limit for the whole problem:

$$\eta \lt \frac{2}{\lambda_{\max}}.$$

In the bowl here, $\lambda_{\max} = 1$, so the edge is at $\eta = 2$.The readout beside the figure reports $\eta\lambda_{\max}$ for the bowl. Watch it cross 2.

The direction that is too flat

The other coordinate shrinks by $1 - \eta\lambda_{\min}$ each step. If the step is held under the limit set by the steep direction, the flat direction crawls. The ratio $\kappa = \lambda_{\max}/\lambda_{\min}$, the condition number, measures how bad the mismatch is. Push the bowl’s elongation κ up and watch the number of steps needed grow roughly in proportion.

Badly conditioned losses are the normal case in deep learning, not the exception. Much of what follows in this course, including normalisation layers, careful initialisation and adaptive optimisers, is about making $\kappa$ smaller, or acting as if it were.

Momentum

Heavy-ball momentum keeps a running velocity:

$$v_{t+1} = \beta v_t - \eta \nabla L(\theta_t), \qquad \theta_{t+1} = \theta_t + v_{t+1}.$$

Along the steep direction, successive gradients alternate in sign and largely cancel. Along the flat direction they agree and accumulate. Set momentum to 0.9 on the elongated bowl and compare the step counts. Then try the valley surface, Rosenbrock’s banana, where the floor curves and momentum has to turn a corner.

Adam goes further and rescales each coordinate by a running estimate of its gradient’s size.Kingma and Ba, 2015. In effect it is a cheap, diagonal guess at the curvature. On the bowl it is nearly indifferent to κ. On the saddle surface, start near the ridge and compare how long plain descent and Adam take to escape it.