How Embedding Models Are Trained

A general model's internal vectors are not the same as a good embedding. What the contrastive objective optimizes, and why hard negatives make or break it.

You might assume an embedding is just a vector pulled from inside a language model. You can do that, and the result is mediocre. A model trained to predict the next token organizes its internal space for prediction, not for the thing embeddings are actually used for: measuring whether two pieces of text mean the same. Getting a good embedding model means training specifically for that — and the objective that does it is contrastive, with one detail that dominates everything else.

What an embedding needs that prediction does not

The whole job of a retrieval embedding is a geometric property: texts with similar meaning should land close together, and different meanings should land far apart, so that similarity search by distance actually retrieves relevant results.

A next-token model was never optimized for that. Its internal representations are shaped to predict continuations, so two sentences that mean the same thing but continue differently can sit far apart, and two that continue similarly but mean different things can sit close. Usable in a pinch, wrong for retrieval. The fix is not a better extraction trick; it is a different training objective aimed straight at the distance property.

The contrastive objective, applied to text

The training idea is the same contrastive learning that aligns images and captions, pointed at pairs of text. You need examples of what should be close:

• a question and a passage that answers it,
• two paraphrases of one sentence,
• a query and a document a user actually clicked.

These are positive pairs — texts that ought to embed near each other. Training pulls each positive pair together while pushing it away from other texts in the batch. Over millions of pairs the space reorganizes so that “means the same” becomes “is nearby”, which is exactly the property retrieval needs.

The positives are the easy half to reason about. The negatives — what you push away from — are where a good embedding model is won or lost.

Why hard negatives are the whole game

The simplest way to get negatives is to treat every other text in the batch as one: for a given query, its own passage is the positive, and everyone else’s passages are negatives. This works, but the negatives are too easy. A random passage is usually so obviously unrelated to the query that the model learns to tell them apart with almost no effort — and learns almost nothing useful in the process.

The distinctions that matter in real retrieval are subtle. Two passages can both mention the query’s keywords while only one truly answers it. If the model only ever trained against random, obviously-wrong negatives, it never had to learn that distinction, and it fails exactly there in production.

Hard negatives are the fix: texts that are similar to the query but wrong — same topic, same vocabulary, but not the actual answer. Training against these forces the model to encode the fine difference between “on topic” and “correct”, which is precisely the difference retrieval quality depends on. Where the CLIP recipe could lean on huge batches for its negatives, text retrieval lives or dies on mining hard ones deliberately.

Mining and its hazard

Hard negatives do not come for free — you have to find them. The standard approach is to use an existing retriever to fetch the top results for a query and take the ones that are not the known answer as hard negatives. They are, by construction, the confusing near-misses.

This carries a real hazard: false negatives. A “hard negative” mined this way might actually be a correct answer that simply was not labeled as the positive. Training pushes it away, teaching the model that a right answer is wrong — which actively degrades the space. Managing this — filtering, thresholds, careful labeling — is much of the practical craft of building a good embedding model, and it is why quality embedding datasets are valuable.

The balance to strike: negatives hard enough to teach the real distinction, but not so hard they are secretly positives.

Choices that shape the final model

A few decisions determine what the trained embedding is good at:

What counts as a positive encodes your definition of similarity. Train on question-answer pairs and you get a model tuned for retrieval; train on paraphrases and you get one tuned for semantic equivalence. The pairs are the task definition.

Symmetric versus asymmetric. Some tasks match like-to-like (two sentences), others match short-to-long (a query to a document). Models are often trained for one shape, and a query-document model may underperform on sentence-to-sentence comparison, because the two jobs have different geometry.

Pooling. A model produces a vector per token; an embedding needs one vector for the whole text. How you combine them — average, or a designated summary position — is a design choice trained end to end with the objective.

The through-line: an embedding model is defined less by its architecture than by what it was told is similar. Match that definition to your use case, and mind the negatives, and the rest follows.

What to remember

• General model internals make poor embeddings because prediction geometry is not similarity geometry — embeddings must be trained for distance directly.
• The objective is contrastive: pull positive pairs (question-answer, paraphrases, clicks) together, push everything else apart.
• Hard negatives — similar but wrong texts — are what force the model to learn the fine “on topic vs correct” distinction retrieval depends on.
• Mining hard negatives risks false negatives (unlabeled correct answers), and managing them is much of the practical craft.
• Positives define the notion of similarity; symmetry and pooling shape what the model is good at.

Next: Similarity Search