Loss Functions

A loss function turns 'the model was wrong' into a single number to minimize. It defines what the model is trying to do — choose it carefully.

A forward pass produces an output. To improve it, you first need to know how wrong it is — as a number, not a feeling. That number is the loss, and the function that computes it is the loss function. It is the single most consequential design choice in training, because it defines what “good” means. The model will chase whatever the loss rewards, so if the loss measures the wrong thing, a perfectly trained model will do the wrong thing perfectly.

Turning “wrong” into one number

Training needs a target to minimize. The loss function compares the model’s prediction to the known correct answer and returns a single non-negative number: zero means perfect, larger means worse.

Why a single number? Because gradient descent needs one quantity to push downhill. If wrongness were reported as five separate numbers, there would be no unambiguous “down.” The loss function collapses all the error in a prediction into one scalar that the rest of training relentlessly minimizes.

The right way to measure wrongness depends on the task, so different tasks use different losses.

Regression: mean squared error

When the model predicts a number — a price, a temperature — the natural error is the difference between prediction and truth. Predict 180 when the answer is 200, and you are off by 20.

Mean squared error (MSE) squares each difference and averages over the examples:

loss = average of (prediction - truth)^2

Squaring does two useful things. It makes every error positive, so overshooting and undershooting both count as wrong. And it punishes large errors disproportionately: being off by 20 costs 400, being off by 40 costs 1,600 — four times as much for double the error. That makes MSE intolerant of big misses, which is usually what you want. Its cousin, mean absolute error, punishes errors in proportion instead and is more forgiving of the occasional outlier. The choice encodes how much you fear large mistakes.

Classification: cross-entropy

When the model predicts a category, squared error is the wrong tool. The model outputs a probability — “80% cat” — and you want a loss that rewards confident correct answers and punishes confident wrong ones.

Cross-entropy does this. It looks at the probability the model assigned to the correct class and penalizes it for being low. If the true answer is “cat” and the model said 0.8, the loss is small. If it said 0.05, the loss is large. Crucially, the penalty grows sharply as confidence in the right answer approaches zero: being confidently wrong is punished brutally, which pushes the model toward honest probabilities rather than reckless guesses.

This is the loss behind essentially every LLM. Predicting the next token is classification over the whole vocabulary — tens of thousands of classes — and the training signal is cross-entropy on the probability the model gave to the token that actually came next. The language modeling objective is exactly this loss applied across trillions of tokens.

The loss defines the goal

This is the idea to carry away: the model does not pursue your intentions, it pursues low loss. Whatever the loss rewards is what you get.

Ask for the wrong thing and you get the wrong thing. If a medical classifier’s loss treats a missed disease and a false alarm as equally bad, it will happily trade dangerous misses for fewer false alarms — because the loss told it those were equal. Fixing that means changing the loss to weight the errors differently, not lecturing the model. When you read that aligning a model to human preferences required a new training objective, this is why: to change behavior, you change what the loss rewards.

Loss is not your evaluation metric

One subtlety worth internalizing. The loss must be smooth enough for gradient descent to work — small weight changes must produce small, measurable loss changes. But the thing you actually care about, like accuracy, is often not smooth (it jumps in steps as predictions flip). So you train against a smooth surrogate loss (cross-entropy) and evaluate against the metric you care about (accuracy). The two are related but not identical, which is why loss going down does not always mean your real metric improved — a tension explored across the evaluation layer.

What to remember

• The loss function reduces “how wrong was the prediction” to one number that training minimizes; a single scalar is what gradient descent needs.
• Mean squared error suits regression and punishes large errors disproportionately; cross-entropy suits classification and punishes confident wrong answers.
• LLMs train on cross-entropy over next-token prediction — classification across the whole vocabulary.
• The model optimizes the loss, not your intent, so the loss is the goal — choose it to reward what you actually want.
• Train on a smooth loss but evaluate on the metric you truly care about; they are not the same thing.

Next: Gradient Descent — how the model uses the loss to decide which way to adjust its weights.