The trick that lets you stack many layers.

Video transcript

A transformer stacks many layers. Each layer takes numbers in, and passes new numbers on.

So why not stack a hundred? Would it just get smarter?

Not by itself. Each layer rewrites the signal, so after many layers the original information fades, and learning signals fade too.

The trick is a shortcut. We add the layer's input back to its output. Output equals x plus layer of x.

So if a layer has nothing useful to add, the original still gets through. What do you think the shortcut keeps?

Yes. Layers only add small changes. Still, numbers can drift to wildly different sizes, as in these bars.

Layer norm fixes that. It rescales each word's numbers to a steady size, so deep stacks stay stable and trainable.