The scoreboard: did it actually work?

Video transcript

How do you grade a translation? Compare the machine's sentence to one a human wrote.

But the two sentences differ a bit, and both seem fine. Is the machine wrong?

Not exactly wrong. BLEU counts how many words and short phrases overlap with the human version. More overlap gives a higher score, from zero to one hundred.

So a higher BLEU means closer to a human translation. If the Transformer scored 28.4 on English to German, was that good compared to older systems?

Yes. 28.4 beat the best systems of the time, including combinations of several models.

And it got there with a fraction of the training cost. Better score, much cheaper. So yes, it worked.