One paper. Then ChatGPT.

Video transcript

This one paper, Attention Is All You Need, introduced the Transformer. Almost every modern language model grows from it.

So BERT and GPT are different from it? The paper had both an encoder and a decoder.

They split it. BERT keeps the encoder, which reads a whole sentence. GPT keeps the decoder, which writes one word at a time. Which part do you think ChatGPT's text writing comes from?

The decoder, yes. Both keep the same core: attention, many heads, stacked layers, trained on huge amounts of text.

So the title was right. Attention really was all we needed, just scaled up.