Of Many Desks Reading All Pages at Once

Welcome back

Last time, at the dinner party, we met attention. It lets a network look across every word at once and decide how much each one matters right now. It’s like a host who hears “her” and instantly knows to think of Clara’s sister.

Today we take that one idea and build an entire building out of it.

The night desk

Picture a great newspaper’s newsroom on the night before a big story breaks. A stack of reports has just come in.

In an old-fashioned office, one editor might read the pages in order, one at a time, keeping notes as she goes. That’s our recurrent network: careful, but slow, and prone to forgetting page one by the time she reaches page forty.

This newsroom works differently. Every desk receives every page at once. And each desk has its own speciality:

• The politics desk looks for who said what.
• The legal desk looks for what’s been promised.
• The local desk looks for which streets and names are involved.

Each desk reads everything and pays attention to what matters for its question. Then they share notes, the copy gets revised, and it goes around the room again for a deeper reading.

The Transformer

A Transformer is that newsroom, built as a neural network. It has three main ideas.

1. Everything is read at once. There’s no reading word by word. All the words in a passage are taken in together, and attention connects each word to every other directly. This also makes Transformers fast to train, because many calculations can happen side by side.

2. Many desks, not one. Instead of a single attention, a Transformer runs several in parallel. This is called multi-head attention. Each “head” learns to look for a different kind of relationship. One might follow grammar, another meaning, and another which noun a pronoun points to. Nobody assigns these specialities. They emerge during training.

3. Round after round. After the attention step, each word passes through a small ordinary network of neurons, the same kind you’ve known since the termites. Together, those two steps form one layer. Then the whole thing repeats, layer upon layer, often dozens of times, just like the aviation lesson. Early layers notice simple things. Later layers grasp richer meaning.

A puzzle about order

If every desk sees every page at once, how does anyone know which page came first? “Dog bites man” and “man bites dog” use the same words.

The Transformer solves this by stamping each word with a little marker of its position before it enters the room, like page numbers in the corner. This is called positional encoding. Now the network can know both what each word is and where it sits.

Why it changed everything

Before Transformers, language networks read slowly and forgot easily. Transformers read in parallel, connect distant words directly, and grow well as you give them more data and more layers. Since around 2017, they have become the foundation of nearly every modern language model, and they’re now used for images, sound and much more.

The famous paper that introduced them had a fitting title: Attention Is All You Need.

What to carry with you