In-Context Learning: Why Few-Shot Works
Show a model a few examples in the prompt and it improves on the spot, with no weight change. Here is the theory of why that happens and where it breaks.
On this page
Put three labeled examples in the prompt and the model gets better at the fourth. No training, no weight update, no gradient. The improvement lives entirely inside a single forward pass and vanishes the moment the prompt ends.
That is in-context learning, and it is strange enough to deserve an explanation. A system whose parameters are frozen still “learns” from examples it sees at inference time. How?
Learning without weight change
Start by separating two things that share the word “learning”.
Training adjusts weights with gradient descent. It is permanent and it is what produced the model.
In-context learning adjusts nothing. The examples in your prompt become part of the input, the model reads them alongside your actual question, and its output shifts accordingly. Close the session and the effect is gone.
The mechanism is not memory in any lasting sense. It is that the model conditions its next-token prediction on everything currently in the context window, and your examples are now part of “everything”. The behavior you see is the model’s frozen parameters reacting to a richer input.
The pattern-completion view
The most reliable way to reason about few-shot prompting is this: the model is not “understanding your task” so much as completing the pattern you started.
Give it:
Review: "Loved it, would buy again." Sentiment: positive
Review: "Broke after one day." Sentiment: negative
Review: "Arrived on time, works fine." Sentiment:The model has seen, during pretraining, enormous quantities of structured text. Your three lines establish a format and a mapping. Predicting the next token that best continues this specific pattern happens to be the correct label. The task got solved as a side effect of pattern completion.
This framing explains a lot of otherwise confusing behavior, which the next sections cover.
What the examples actually teach
A counterintuitive finding: the format of your examples matters more than whether their labels are correct.
Studies that scrambled the labels in few-shot prompts — pairing inputs with wrong answers — found accuracy dropped far less than you would expect. What the examples mainly convey is the shape of the task: the label space (these are the allowed answers), the input distribution (this is what inputs look like), and the format (this is how output should be structured).
The practical reading is not “labels do not matter”, it is that examples do several jobs at once, and demonstrating the format and space is a bigger share of the value than any single correct answer. It is why one badly formatted example poisons a prompt more than one mislabeled one.
Why it emerges at all
In-context learning was not designed in. Nobody added a “learn from examples” module. It appeared as models scaled, the same way other emergent capabilities did.
The plausible explanation: to predict next tokens well across all of human text, a model must handle documents that contain their own patterns — glossaries, translation tables, formatted lists, worked examples followed by more of the same. Getting good at that general skill means getting good at “infer the rule from the visible instances and apply it”. In-context learning is that skill, pointed at your prompt.
This is why it is stronger in larger models. The internal machinery that tracks and extends patterns is exactly what scale builds.
The limits that follow
Because it is pattern completion inside a fixed input, in-context learning inherits hard boundaries.
It fits in the context window. Your examples cost tokens, on every single call. Twenty examples is a different economic proposition than three, and there is a ceiling. When you need hundreds of examples, that signals fine-tuning, where examples move into the weights and stop costing per-request tokens.
It does not persist. Nothing carries to the next conversation. The model has no memory of having seen your examples before.
It cannot add missing capability. If the base model cannot do arithmetic, examples of arithmetic will not teach it — they can only surface and direct ability that is already latent. Few-shot prompting redistributes existing competence. It does not create new competence, which is the same limit fine-tuning hits.
It is sensitive to order and selection. Which examples, and in what order, measurably changes output. This is a feature of pattern completion, not a bug to be prompted away.
Using the theory
Concrete consequences for how you write prompts:
- Make format identical to what you want back. The examples teach shape first. Inconsistent formatting across examples teaches inconsistency.
- Cover the label space. If a class never appears in your examples, the model has weak evidence it is allowed.
- Pick examples near the hard cases, not the obvious ones — the same selection principle that governs dataset curation.
- Stop adding examples when accuracy plateaus. More tokens past that point is pure cost.
What to remember
- In-context learning improves output from examples in the prompt with zero weight change — the effect lives in one forward pass and disappears after.
- Treat it as pattern completion: the model extends the structure your examples establish, and solving your task is a side effect.
- Examples teach format and label space as much as correct answers, so formatting consistency matters enormously.
- It emerged from scale as a consequence of next-token prediction over patterned text, not from explicit design.
- Limits are real: it costs tokens per call, does not persist, and cannot add capability the base model lacks.