The tiny √d fix that saves everything.

Video transcript

A score is a dot product of a query and a key. That's a sum of d products. What goes wrong when d is big?

More terms means a bigger spread. If each term is random with variance one, the sum has variance d, so scores grow like the square root of d.

So the scores are huge. Why is that bad for softmax?

Softmax exponentiates. With big gaps, one word gets almost all the weight and the rest get almost zero. The slopes vanish, so learning stalls. If scores are this large, what should we do?

Divide by square root of d. The spread was square root of d, so dividing returns it to about one, whatever d is.

So softmax stays soft, and the gradients survive. Such a tiny fix.

Exactly. One small division keeps attention trainable at any size.