Where do Q, K, V come from?

Video transcript

Last time each word had a query, a key and a value. But where do those come from? Does the word just have them?

Good question. A word starts with just one vector, its embedding. Q, K and V are not stored in it. We make them.

We multiply the same vector by three different matrices. One for queries, one for keys, one for values.

So one word becomes three vectors, because the matrices differ? Then do all words share the same three matrices?

Yes, shared. And nobody writes their numbers by hand. They start random, and training slowly adjusts them until the matches become useful.

So the model learns what to ask, what to advertise, and what to hand over. Neat.