Computers don’t read words. They read pieces.

Video transcript

So the computer reads words as numbers. Does it use one number for each whole word?

Not quite. First it chops words into small pieces called tokens. Look at the word unhappiness.

It can split into un, happi, and ness. Each piece gets its own numbers.

But why bother? Wouldn't a list of whole words be simpler?

Think of a new word, like unkindness. With whole words, it's unknown. With pieces, it's un, kind, and ness, which the computer already knows.

Oh, so a small set of pieces can build almost any word, even new ones.

Exactly. Fewer things to remember, and nothing is truly unknown. Next, we'll see how each piece becomes a point in space.