Backpropagation

Backpropagation is how a network computes the gradient for every weight efficiently: apply the chain rule backward through the layers, reusing work as you go.

Gradient descent needs the gradient — the slope of the loss for every weight. In a network with billions of weights spread across dozens of layers, computing that efficiently is not obvious. Backpropagation is the algorithm that does it. Without it, training deep networks would be computationally hopeless; with it, the cost of the gradient is about the same as one forward pass.

The problem: blame assignment across layers

The loss is measured at the very end, after the input has passed through every layer. But a weight in the first layer also influenced that loss — its effect rippled forward through all the layers in between.

The question backpropagation answers: how much did each weight, however deep, contribute to the final error? A first-layer weight’s influence is indirect, filtered through everything downstream. You cannot read its effect off the loss directly. You have to trace responsibility backward through the chain of computations.

The chain rule, made practical

The tool is the chain rule from calculus. Informally: if A affects B and B affects C, then A’s effect on C is A’s effect on B multiplied by B’s effect on C. Effects along a chain multiply.

A network is exactly such a chain. Weight → weighted sum → activation → next layer → … → loss. To find how a deep weight affects the loss, multiply together the local effects along the path from that weight to the loss.

Each local effect is simple to compute — each step is a basic operation whose slope is known. The chain rule stitches these simple local slopes into the full slope of a weight buried deep in the network. Backpropagation is, at heart, the chain rule applied systematically to every weight at once.

Why “backward” and why it is fast

Here is the insight that makes it efficient. Computing each weight’s gradient from scratch, tracing its own path forward, would repeat an enormous amount of shared work — nearby weights share most of their path to the loss.

Backpropagation instead works from the loss backward, one layer at a time, carrying an intermediate quantity: how much the loss changes with respect to each layer’s output. Call it the error signal.

• Start at the output. Compute how the loss changes with the final layer’s output — directly, from the loss function.
• Move back one layer. Using the chain rule, convert that into how the loss changes with this layer’s output, and with this layer’s weights. The weight gradients for this layer are now known.
• Pass the error signal back another layer. Repeat.

Each layer’s computation reuses the error signal already computed for the layer after it. Nothing is recomputed. This reuse is why the whole backward pass costs roughly one forward pass, no matter how many weights there are. That efficiency is the entire reason large networks are trainable at all.

Forward stores, backward uses

The two passes are partners. During the forward pass, the network computes and remembers the intermediate values at each layer — the inputs and outputs of every neuron. The backward pass needs those exact values to compute the local slopes.

This is why training a model takes far more memory than running it. Inference can discard each layer’s output once the next layer has consumed it. Training cannot — it must hold every intermediate value from the forward pass until the backward pass reaches back to use it. When you read that training a model needs many times the memory of running it, this storage is a large part of why.

Vanishing and exploding gradients

Because backpropagation multiplies many local slopes along the chain, a pathology follows directly. If those slopes are consistently smaller than 1, their product shrinks toward zero as it travels back through many layers — the vanishing gradient. Early layers then receive almost no signal and stop learning. If the slopes are consistently larger than 1, the product explodes and training destabilizes.

This is not an abstract worry; it is why several core design choices exist. ReLU was adopted partly because it does not shrink the signal for active neurons. Normalization layers and the residual connections in Transformers exist largely to give the gradient a clean path backward through very deep networks. Once you see backprop as a long chain of multiplications, the reason deep networks need these tricks becomes obvious.

What you do not have to do by hand

Modern frameworks compute all of this automatically through automatic differentiation. You define the forward computation; the framework records the operations and can then run the backward pass for you, applying the chain rule step by step. You will almost never derive a gradient manually.

But understanding what happens under that abstraction pays off constantly. It explains why training is memory-hungry, why network depth causes gradient problems, why activation and normalization choices matter, and why a bug that makes the loss stop decreasing often traces back to gradients that vanished or blew up. Backpropagation is the quiet engine under every pretraining and fine-tuning run.

What to remember

• Backpropagation computes the gradient for every weight by applying the chain rule backward through the layers.
• Working backward and reusing a per-layer error signal makes the full gradient cost about one forward pass, which is what makes deep networks trainable.
• The forward pass must store its intermediate values for the backward pass to use, which is why training needs far more memory than inference.
• Multiplying many slopes causes vanishing or exploding gradients, motivating ReLU, normalization, and residual connections.
• Frameworks do this automatically via automatic differentiation, but knowing the mechanism explains many practical training behaviors.

Next: Overfitting and Generalization — why a model that learns its training data too well can still fail.