Of Loaves Baked by the Trayful, Not the Crumb

Welcome back

Last time, backstage at the theatre, we learned about regularisation: dropout, which randomly benches neurons during training, and weight decay, which keeps weights modest. Both stop a network leaning too hard on its training quirks.

Today, a practical question that turns out to matter a great deal: how many examples should the network look at before it adjusts its weights? For the answer, let’s go into a bakery before dawn.

Three bakers

Picture three bakers, each trying to perfect a new bread recipe. Each one adjusts the recipe, a little more salt or a little less time in the oven, based on how the bread turns out.

The first baker bakes the entire day’s order, a thousand loaves, and only then tastes them all and tweaks the recipe. Her adjustments are well judged, because she’s seen so much. But she can only make one adjustment a day. Progress is painfully slow.

The second baker bakes one single bread roll, tastes it, and adjusts. Then another roll, and adjusts again. He makes hundreds of tweaks a day. But one roll can be misleading. Maybe it sat in a hot corner of the oven. His recipe lurches about, chasing every odd roll.

The third baker bakes one tray at a time, say thirty-two rolls. She tastes the tray, adjusts, and bakes the next. A tray is big enough that one odd roll doesn’t fool her, and small enough that she can adjust many times a day. She improves quickly and steadily.

The third baker is the one we copy.

Batches

In neural networks, a tray of examples is called a batch (or often a mini-batch).

Instead of running the training loop on one example at a time, or on all of them at once, we group the examples into batches, commonly 32, 64 or 256 at a time. For each batch, the network:

1. Makes guesses for every example in the batch.
2. Measures the loss on each and averages them.
3. Uses backpropagation to work out the blame, averaged over the batch.
4. Takes one step downhill.

Then on to the next batch. One epoch, a lap through all the data, is now made of many batches. With ten thousand examples and batches of 100, that’s a hundred steps per epoch.

Stochastic gradient descent

You may remember gradient descent, our wanderer in the fog, feeling for the downhill direction. When we estimate “downhill” from a small, randomly chosen batch instead of the whole dataset, the method is called stochastic gradient descent, or SGD. Stochastic is just a formal word for “involving chance.”

Because each batch is a random handful, each step’s sense of “downhill” is a little rough. It’s not quite the true slope of the whole hillside. But it’s close, and it’s cheap to compute, so we can take many steps quickly. On average, those quick, slightly wobbly steps head the right way.

What to carry with you