No peeking: the mask.

Video transcript

In training, we already know the full French answer. It's like an answer sheet: Le chat dort ici.

So the decoder could just read the next word and copy it? That sounds like cheating.

Exactly, so we cover the answer sheet. When predicting a word, the model sees only earlier words. Each word's row of attention scores is a grid.

So the squares for future words must be blocked. Should those blocked scores become zero percent after softmax?

Yes, zero. We set those scores to minus infinity before softmax, so they become zero percent.

So each word looks only at itself and the past. That makes training match real writing.