Multimodal Embeddings
Train images and text into one shared vector space and you can search pictures with words. The idea behind CLIP and most cross-modal retrieval.
On this page
An embedding space for text places similar sentences near each other. An embedding space for images places similar pictures near each other.
Now put both in the same space, so that a photograph of a dog and the text “a dog” land in nearly the same place.
That is a multimodal embedding, and it unlocks a capability neither modality has alone: searching images with words, with no labels, tags, or captions anywhere in the system.
Contrastive training
The training method is the whole idea, and it is elegant.
Collect a large number of image-caption pairs from the web. Take a batch of them — say 1,000 pairs. Encode all 1,000 images and all 1,000 captions.
Now you have a 1,000×1,000 grid of similarity scores. Exactly 1,000 of those cells are correct pairings; the rest are mismatches. Train so that matching pairs score high and every other combination scores low.
That is contrastive learning. No labels are needed beyond the pairing itself, which is why it scales to hundreds of millions of examples scraped from the open web.
The consequence is a shared space. The image encoder and text encoder were trained together toward the same objective, so their outputs are directly comparable — you can compute cosine similarity between a picture and a sentence.
Note what the negatives do here. Learning is driven largely by pushing apart the 999,000 wrong pairings, not just pulling together the 1,000 right ones. Larger batches give more negatives per step, which is why batch size matters unusually much for this kind of training.
What it enables
Text-to-image search. Embed a query, find nearest image vectors. Works on any image collection with no tagging.
Image-to-image search. Find visually or semantically similar pictures.
Zero-shot classification. To classify an image among n categories, embed the strings “a photo of a {category}” and pick the nearest. No training on those categories required — the classifier is constructed at query time from words. This was the result that made the approach famous.
Conditioning generators. Diffusion models need text and image representations that relate to each other; a shared space provides exactly that.
Deduplication and clustering across mixed media.
Where it falls short
The limitations are consistent and worth knowing before building on it.
Compositional binding. “A red cube on a blue sphere” embeds close to “a blue cube on a red sphere.” The representation captures which concepts are present far better than how they relate — the same failure that shows up in image generation prompts.
Counting. Poorly represented, as everywhere else in vision.
Fine-grained distinctions. Bird species, product variants, near-identical documents. The space is coarse.
Text rendered inside images. These models often match on the presence of text-like pixels rather than reading it — one reason a picture containing the word “apple” can match the query “apple” for the wrong reason.
Web-scraped bias. The space inherits the associations in its caption data, exactly as text embedding spaces do. It is a compressed record of how people captioned images on the internet.
Practical use
Retrieval quality drops on specialized domains. Medical imaging, industrial inspection, and technical diagrams were thinly represented in web captions. Test on your own data before assuming it transfers.
Prompt phrasing matters. “A photo of a {x}” reliably outperforms bare “{x}” for zero-shot classification, because it matches the caption style in training. Small template changes shift accuracy measurably.
Combine with keyword filtering. The same argument as hybrid search for text: embeddings handle semantics, exact metadata handles the rest.
Rerank when precision matters. A cross-encoder that examines query and candidate together beats independently-computed similarity, in this modality as in text.
What to remember
- Multimodal embeddings put images and text in one shared space, so they can be compared directly.
- Trained contrastively on web image-caption pairs: matching pairs pulled together, all other combinations pushed apart.
- Enables text-to-image search, zero-shot classification, and conditioning for generators — with no labels.
- Weak at compositional relationships, counting, fine-grained distinctions, and reading text in images.
- Test on your domain; web-caption training does not transfer evenly.