What Happens During Pretraining?

Hide the next token, guess it, adjust on error, repeat a trillion times. How a simple objective at enormous scale produces broad capability.

Pretraining is where nearly all of a model’s capability comes from. Everything after it — instruction tuning, preference training — shapes behavior rather than building ability.

The objective is almost absurdly simple.

The objective

Take text. Show the model a prefix. Have it predict the next token. Compare with the truth. Adjust.

The cat sat on the ___ → model predicts a distribution → actual token was ␣mat → if the model gave ␣mat low probability, adjust parameters to raise it.

No labels. No human annotation. The text supplies its own answers, which is what allows training on a corpus far larger than anyone could label. This is why it is called self-supervised.

Crucially, the model predicts at every position simultaneously, not just the last one. A 1,000-token document provides 1,000 training signals in one pass.

Where capability comes from

The reason a next-token objective produces a broadly capable system is worth sitting with: being good at this task requires machinery you did not ask for.

Consider what accurate prediction demands:

• The keys to the cabinet ___ → subject-verb agreement across an intervening phrase
• Marie told Ann that she ___ → tracking which entity is which
• 247 + 158 = ___ → actual arithmetic
• def is_prime(n): if n < 2: return ___ → the semantics of the function
• The killer was revealed to be the ___ → holding an entire plot in working memory

No lookup table does these. The only route to low prediction error across all of human text is to develop internal representations of grammar, entities, causality, arithmetic, and code semantics.

Capability is not the objective. It is what the objective forces.

Data

The corpus is a large fraction of accessible written text: web pages, books, code repositories, reference works, forums.

Composition matters more than raw size. Two things labs invest heavily in:

Filtering. Removing boilerplate, spam, and machine-generated junk. Quality of remaining text affects final capability substantially.

Deduplication. Repeated passages cause memorization rather than generalization, and inflate apparent dataset size without adding information.

Mixture is deliberate. More code improves reasoning even on non-code tasks — a repeatedly observed effect, plausibly because code demands precise multi-step structure. Language proportions determine multilingual strength. And whatever dominated the corpus determined the tokenizer’s merge decisions, permanently.

Scale and its surprises

Three quantities scale together: parameters, data, and compute. Getting the ratio wrong wastes the run — too many parameters for too little data, and the model underlearns relative to its size.

The result that reoriented the field: certain capabilities do not improve gradually with scale. They are absent, then present. Multi-step arithmetic, following instructions in unfamiliar formats, translating language pairs never explicitly paired in training — these appeared in larger models without appearing weakly in smaller ones.

More of the same simple thing produced qualitative change. That observation is why the industry pursued scale so aggressively.

What pretraining does not produce

An assistant. A pretrained model continues text. Ask it a question and you may get more questions. Base Models vs Chat Models covers the gap.

Current knowledge. The cutoff is absolute, and the model cannot tell you where it falls.

Reliable factuality. The objective rewards plausible continuation, and plausible is not the same as true. This is the deep root of hallucination — it comes from pretraining itself, not from a downstream mistake.

Any notion of source. Whatever the model retains is diffused across parameters as statistical structure. There is no stored document to cite.

What to remember

• Pretraining predicts the next token across enormous text, using the text as its own labels.
• Broad capability emerges because accurate prediction requires grammar, entity tracking, arithmetic, and code understanding.
• Data composition and deduplication matter as much as volume; more code improves general reasoning.
• Some capabilities appear abruptly with scale rather than improving smoothly.
• Pretraining alone yields no assistant behavior, no current knowledge, and no factual guarantee.

Next: What Does “7B Parameters” Mean?