The flaw: it can’t tell “dog bites man” from “man bites dog.”

Video transcript

These two sentences use the same words, but they mean very different things. Can attention tell them apart?

Think about what attention does. Each word looks at every other word and scores them. Nothing says who is first or last. It's like a bag of words.

So dog looks at man, and man looks at dog, in both sentences. Would the scores come out the same?

Yes. Same words, same scores, same result. Attention alone can't tell who bit whom. That's the flaw.

So we must tell each word where it sits? Like stamping a number on it?

Exactly. We add a position signal to every word before attention. Next, we'll build that signal using sine waves.