Welcome back
Last time, we played the whisper game. A message passed down a long line of people can fade to nothing or swell into nonsense. Deep networks face the same danger: the learning signal can vanish or explode as it travels back through many layers.
Today we look at one of the simplest, gentlest remedies. It happens before training even begins, in the quiet moment when the seeds go into the soil.
Before the first season
Picture a gardener on a spring morning, holding a packet of seeds. Where should they go?
If she pours every seed into one hole, they crowd each other and grow up identical and weak. If she buries them too deep, they never reach the light. If she scatters them on the surface, the wind takes them. A good gardener spaces them out a little, at just the right depth, and then lets the season do its work.
A neural network has its own spring morning. Before it has seen a single example, every weight needs a starting value. Choosing those values is called weight initialisation. It sounds like a small housekeeping detail, but it shapes everything that follows.
Why not start everything at zero?
The tidiest idea would be to set every weight to zero, or to the same number. That’s like pouring all the seeds into one hole.
Think back to the neurons in a layer. If they all start with identical weights, they all compute the same thing. During backpropagation, they all receive the same blame, so they all change in the same way. They stay twins forever. A layer of a hundred neurons ends up doing the work of one.
So we give each weight a small random value instead. That little bit of randomness breaks the symmetry. Each neuron starts out slightly different, so each one can grow into its own role. One might come to notice edges, another curves, another colour.
Not too deep, not too shallow
Random alone isn’t enough. The size of those random values matters just as much as a seed’s depth.
- Too large, and signals grow bigger at every layer as they flow forward. The learning signal grows on its way back too. This is the exploding problem from the whisper game.
- Too small, and signals shrink layer after layer until almost nothing reaches the end. The learning signal fades on its way back as well. This is the vanishing problem.
The aim is to keep the signal at roughly the same strength as it passes through each layer, neither swelling nor shrinking.
The gardener’s rule of thumb
Researchers found a lovely simple guide. The right size for a neuron’s starting weights depends on how many inputs that neuron has.
A neuron that listens to many others should start with smaller weights, because it adds up many signals and the total could easily grow too large. A neuron that listens to only a few can start with slightly larger ones. Put simply, the more neighbours a seedling has, the more space each one needs.
Two well-known recipes follow this idea. They are named after the people who described them: Xavier (or Glorot) initialisation, and He initialisation, which suits the popular ReLU activation function. You don’t need to remember the names. Just remember the spirit: random, small, and scaled to fit the number of inputs.
What to carry with you
Start the weights small, random, and scaled to how many inputs each neuron has. That way every neuron grows into its own role, and the signal stays steady from the first layer to the last.