All of attention in one line of math.

Video transcript

This one line is the whole attention idea? Where do I even start reading it?

Start inside. Q K transpose multiplies the query matrix by the key matrix. One multiply gives every query's score against every key.

So for a sentence of four words, that's a four by four grid of scores, all at once?

Yes. Then divide by root d to keep scores calm. Next, softmax turns each row into percentages. Before we continue, what does each row of the grid describe?

Right, one word's percentages over every word, adding to one hundred percent.

Last step: multiply by V. Each word's output is a weighted mix of the value vectors, again for all words in one matrix multiply.

So scores, soften, percentages, mix. Every word at once, in one line.