Weights and Biases

Weights and biases are the numbers a neural network learns. Weights set how much each input matters; biases shift the result. Everything a model knows lives here.

A neural network is a fixed pattern of arithmetic. What makes one network recognize cats and another translate French is not the pattern — it is the specific numbers plugged into it. Those numbers are the weights and biases, and they are the only thing training changes. Everything a model has learned is stored here and nowhere else.

Weights: how much each input matters

Recall a neuron’s core operation: multiply each input by a weight, add them up, add a bias.

sum = w1*x1 + w2*x2 + w3*x3 + b

Each weight w controls the influence of one input. A large positive weight means “when this input goes up, push the output up hard.” A negative weight flips that. A weight near zero means “ignore this input.”

Picture a network scoring loan applications. If the weight on income is large and positive, income strongly raises the score. If the weight on an irrelevant field is near zero, the network has effectively learned to ignore it. Nobody set those weights by hand — they emerged from training. The weights are the learned rule, expressed as a pile of numbers rather than as English.

Bias: shifting the baseline

The bias b is added regardless of the inputs. It sets the neuron’s default lean before any input speaks.

Without a bias, every neuron is forced to output its activation’s midpoint when all inputs are zero. That is an arbitrary constraint. The bias frees the neuron to fire easily (high bias) or reluctantly (low bias). Concretely: if you want a neuron that activates only when the evidence is strong, a negative bias raises the bar the weighted sum must clear. It is the intercept in y = mx + b — the weights are the slopes, the bias is where the line sits.

Small in count, but they matter. Remove biases and networks lose a surprising amount of flexibility.

Why these are “parameters”

Weights and biases together are the model’s parameters — the adjustable numbers set by learning. When a model is described as “7B parameters”, that seven billion is the total count of weights and biases across all its layers.

The number is large because layers are dense. A layer taking 1,000 inputs into 1,000 neurons needs 1,000 x 1,000 = one million weights for that layer alone, plus 1,000 biases. Stack dozens of wide layers and the count climbs into the billions fast. Weights vastly outnumber biases — one bias per neuron, but a weight for every input-to-neuron connection.

This is also why a trained model is a large file. Saving a model means saving every one of those numbers. There is no separate database of knowledge; as What Is an LLM? puts it, the parameters are the knowledge.

Where they start and where they go

At the very beginning, weights are set to small random numbers and the network’s output is meaningless noise. Randomness is deliberate — if every weight started identical, every neuron in a layer would compute the same thing and stay forever identical, learning nothing. Random starts break that symmetry.

Then training takes over. Each step measures how wrong the output is with a loss function, works out which direction to nudge every weight and bias to reduce that wrongness, and takes a small step. That direction comes from gradient descent; the method for computing it across all layers is backpropagation. Repeat billions of times and the random noise resolves into numbers that produce coherent behavior.

A useful mental model

Think of the weights and biases as billions of small dials. Training is the slow process of turning every dial a hair at a time until the whole machine behaves. There is no moment where the network “understands” — there is only a configuration of dials that happens to produce the right outputs across the training data, and, if training went well, on new data too.

That framing demystifies a lot. Fine-tuning is nudging an already-good set of dials for a narrower task. Quantization is storing each dial with less precision to save memory. LoRA is learning a small correction to the dials instead of moving all of them. Every one of these techniques is, at bottom, a different way of managing weights and biases.

What to remember

• Weights scale how much each input influences a neuron; biases shift its baseline output. Together they are the model’s parameters.
• These numbers are the only thing training changes, and they hold everything the model has learned.
• Parameter counts reach billions because dense layers need a weight per connection; weights far outnumber biases.
• Weights start as small random values (random to break symmetry) and are nudged by gradient descent until the network behaves.

Next: The Forward Pass — how a network uses these numbers to turn an input into an output.