Start slow, speed up, slow down.

Video transcript

You said the step size comes next. Should it just stay the same the whole time?

The paper changes it. The learning rate climbs linearly for four thousand steps, then falls slowly, like one over the square root of the step.

Why start slow? At the start the model knows nothing, so I'd want big steps.

Right. Early on, the gradients are noisy and unreliable. Warmup avoids a bad first leap. Then we speed up, and later slow down to settle gently near the lowest point.

So warmup is caution, and decay is precision. Got it. Anything else in training?

Three quick ones. Adam is the optimizer that adapts each step. Dropout randomly switches off pieces to prevent memorizing. Label smoothing softens the right answer a little to avoid overconfidence.