Attention mixes. This part thinks.

Video transcript

Here are three words, each one a list of numbers. Attention is the step where words talk to each other.

Right, each word looks at the others and pulls in information. So is that where the thinking happens?

Attention mostly mixes. It gathers information from other words. After the mixing, each word needs to process what it gathered.

That is the feed-forward layer. A small network takes one word's numbers and transforms them. Every word goes through the same network, alone.

So attention moves information between words, and feed-forward works on each word separately, with shared weights.

Exactly. It first expands the numbers to a larger size, applies a nonlinearity, then shrinks back. That is where the model computes on what it gathered.