word2vec: Where Word Vectors Began

The idea that changed NLP: represent each word as a dense vector learned from its neighbours, so that similar words land near each other and meaning becomes arithmetic.

Bag-of-words had one flaw that no amount of clever weighting could fix: every word was an isolated symbol. cat and kitten were as unrelated as cat and parliament. In 2013, word2vec broke that wall, and the idea it introduced — the dense word vector — sits underneath every model built since.

The insight: a word is known by its company

The linguist J.R. Firth put it in 1957: “You shall know a word by the company it keeps.” Words that appear in similar contexts tend to mean similar things. cat and dog both show up near pet, vet, feed, fur. Monday and Tuesday share their neighbours almost entirely.

word2vec turned this observation into a training procedure. Instead of assigning each word a one-hot slot in a giant sparse vector, it assigns each word a short dense vector — say 300 numbers — and tunes those numbers so that words appearing in similar contexts get similar vectors.

The result is a learned representation. Nobody hand-labels that cat is animal-like; the model discovers it purely from which words surround which, across billions of sentences.

How it learns: predict the neighbours

word2vec is a small neural network trained on a fake task whose real purpose is the vectors it produces as a side effect. There are two variants.

Skip-gram. Given a center word, predict the words around it. Feed in fox and train the model to raise the probability of the words that actually appeared nearby in the corpus — quick, brown, jumps.

CBOW (continuous bag of words). The reverse: given the surrounding words, predict the center one. Given quick brown ___ jumps, predict fox.

Either way, the network has one hidden layer, and the weights of that layer are the word vectors. Training on a next-word-style objective forces words with similar contexts to develop similar weights, because they need to make similar predictions. The prediction task is throwaway; the weights are the prize.

One practical trick made it fast enough to train on billions of words: negative sampling. Rather than computing probabilities over the entire vocabulary every step, the model just learns to separate real neighbour pairs from a handful of random fake ones. That approximation is what made word2vec cheap enough to matter.

Meaning becomes geometry

Because similar words end up with similar vectors, the space they live in has structure you can measure. Distance means something.

• The nearest neighbours of Paris are London, Berlin, Madrid — other capitals.
• The nearest neighbours of running are ran, runs, jogging.

More strikingly, directions in the space carry meaning. The famous example:

vector(king) − vector(man) + vector(woman) ≈ vector(queen)

The direction from man to woman is roughly the same as the direction from king to queen. The model was never told about gender or royalty; the geometry of these relationships fell out of the training data. This is the property explored in depth in embedding space geometry — word2vec is where it was first observed.

The straight line to modern embeddings

A word2vec vector is exactly what we now call an embedding: a dense vector where position encodes meaning. Everything downstream in this course — vector databases, similarity search, RAG retrieval — rests on the representation word2vec proved could be learned.

The lineage is direct. word2vec learns one vector per word. Modern transformer embeddings learn one vector per token in context, so that bank gets a different vector by the river than in a vault. But the core bet is identical: meaning can be packed into a few hundred numbers, and those numbers can be learned from raw text.

The limitation that motivated what came next

word2vec assigns each word exactly one vector, fixed forever after training. That is its ceiling. bank gets a single vector that has to average the financial sense and the river sense together, and it can never tell them apart, because it has no access to the sentence the word appears in.

This is the static embedding problem. A word’s meaning shifts with context, and a lookup table cannot represent that. Closing this gap required models that read the whole sequence and produced context-dependent representations — first RNNs, and eventually the self-attention mechanism that computes a fresh vector for every word based on its actual neighbours.

What to remember

• word2vec learns a dense vector for each word by training a network to predict a word’s neighbours.
• Words in similar contexts get similar vectors, so distance in the space encodes similarity.
• Relationships appear as consistent directions: king − man + woman ≈ queen.
• These vectors are the direct ancestor of modern embeddings, which power search and retrieval.
• The ceiling: one fixed vector per word, so it cannot separate bank the riverside from bank the institution. Context-aware models fixed that.

Next: Recurrent Neural Networks — the first architecture built to read a sequence in order and carry meaning forward.