Self-Attention, Explained Visually

How each word decides which other words matter to it. Queries, keys, and values — the mechanism at the center of every language model.

Consider the sentence:

The animal didn’t cross the street because it was too tired.

What does it refer to? Obviously the animal. Change one word:

The animal didn’t cross the street because it was too wide.

Now it is the street. The word it did not change. The word that resolved it sits six positions away.

Self-attention is the mechanism that lets it reach across the sentence and pull meaning from the right place. It is the single most important idea in modern language models, and it is more approachable than its reputation suggests.

The core move

Every token gets to ask a question about what it needs, check that question against what every other token offers, and take a weighted blend of the content it finds most relevant.

Three roles, derived from each token’s embedding by three separate learned projections:

• Query — what am I looking for?
• Key — what do I have to offer?
• Value — the actual content I contribute if selected.

The library analogy holds up well. Your query is the topic you want. Each book’s key is its spine label. The value is what is inside. You compare your query against every spine, then read proportionally — mostly the best matches, a little of the near-misses.

The four steps

1 · Score. Take one token’s query and compute a dot product against every token’s key. High dot product means the vectors point in similar directions, meaning this key is relevant to this query. For a 10-token sequence, each token produces 10 scores.

2 · Scale. Divide by the square root of the key dimension. Without this, large dimensions produce large dot products, softmax saturates, and gradients vanish. A small detail with real consequences.

3 · Softmax. Convert scores into weights that sum to 1. Now it might weight animal at 0.7, street at 0.1, and spread 0.2 across everything else.

4 · Blend. Multiply every token’s value vector by its weight and sum. The output is a new vector for it that now contains substantial information from animal.

Every token does this simultaneously. The whole operation is a handful of matrix multiplications, which is exactly why it runs efficiently on GPUs — the parallelism that made transformers practical.

Why it beats the alternatives

Direct access at any distance. Position 500 attending to position 3 is one operation, identical in cost to attending to position 499. Earlier architectures had to relay information step by step, degrading it along the way.

The weights are computed, not fixed. it resolves to animal or street depending on whether the sentence ends in tired or wide. Attention weights are recalculated for every input, which is what makes them context-sensitive rather than a static lookup.

Interpretable, up to a point. You can inspect the weights and see which tokens attended to which. Attention maps often show recognizable structure: pronouns finding referents, verbs finding subjects, brackets finding partners. Worth a caveat — attention weights show where information flowed, not why, and reading them as explanations of the model’s reasoning overstates what they demonstrate.

Causal masking

In a language model, token 5 must not see token 6 — it is trying to predict token 6.

So before softmax, all scores for future positions are set to negative infinity. Softmax sends them to zero. Each position attends only to itself and what precedes it.

This is causal masking, and it is why the architecture is called decoder-only. It also enables a large optimization: since past tokens can never attend to future ones, their computed keys and values never change as generation proceeds, so they can be stored and reused. That is the KV cache.

Self vs cross

Self-attention means queries, keys, and values all come from the same sequence — the tokens are examining each other.

Cross-attention draws queries from one sequence and keys and values from another. Used when a model conditions on separate input, as in translation or image captioning. Decoder-only language models use self-attention throughout.

Where the cost lives

Every token scores against every token: n² comparisons for n tokens. Double the context and quadruple the attention work.

This is the fundamental constraint on context length, and the reason long-context requests are priced as they are. Why Long Contexts Cost So Much covers it, along with the approximations people use to soften it.

What to remember

• Each token forms a query, matches it against every token’s key, and blends their values by relevance.
• Four steps: score by dot product, scale, softmax into weights, blend values.
• Weights are recomputed per input, which is what makes attention context-sensitive.
• Causal masking blocks future positions and is what makes the KV cache possible.
• Cost grows with the square of sequence length.

Next: Why Multiple Attention Heads? — one attention pass is not enough.