Bag of Words and N-grams

The first way anyone turned text into numbers: count the words, ignore the order. Simple, surprisingly effective, and the baseline everything else is measured against.

Suppose you want a computer to tell spam from real email. The computer works with numbers, and email is text. You need a way to turn one into the other. The oldest answer, and still a strong baseline, is disarmingly simple: count the words and ignore the order.

That is the bag-of-words model. It sounds too crude to work. It works.

Counting words into a vector

Take two documents, already preprocessed into clean tokens:

• Doc A: "the cat sat on the mat"
• Doc B: "the dog sat on the log"

Build a vocabulary of every unique word across both: [cat, dog, log, mat, on, sat, the]. Now represent each document as a vector of counts, one slot per vocabulary word:

Each document is now a point in a 7-dimensional space. A classifier can measure distances between these vectors, and documents that share words land near each other. This is the “bag”: you tossed all the words into a bag and counted them, throwing away the sequence entirely.

Why throwing away order still works

"dog bites man" and "man bites dog" produce identical bag-of-words vectors, which is obviously wrong. And yet for many tasks it barely matters.

To decide if an email is spam, the mere presence of viagra, winner, and free is enormously predictive regardless of their arrangement. To sort news articles into sports versus finance, the words quarterback and dividend do the job on their own. For topic-level tasks, word presence carries most of the signal and word order carries little. Bag-of-words exploits exactly that.

The failure cases are where order and structure carry the meaning: sentiment (“not good” vs “good”), translation, question answering. Those had to wait for models that read sequences, which is where RNNs come in later.

TF-IDF: not all words are equally useful

Raw counts have a bias. Common words get high counts in every document, so they dominate the vector without distinguishing anything. TF-IDF fixes this by weighting each word by how informative it is.

Two factors multiplied together:

• Term Frequency (TF) — how often the word appears in this document. More mentions, more weight.
• Inverse Document Frequency (IDF) — how rare the word is across all documents. A word in every document (like the) gets a weight near zero; a word in only a few documents gets a high weight.

The effect: the is downweighted to near-nothing automatically, without a hand-built stop list, while a distinctive word like photosynthesis gets amplified. TF-IDF was the workhorse of search engines and text classifiers for two decades, and it remains a genuinely good baseline today.

N-grams: putting a little order back

Bag-of-words loses all sequence, which hurts. N-grams buy some of it back by treating short runs of consecutive words as single units.

From "the cat sat":

• Unigrams (n=1): the, cat, sat
• Bigrams (n=2): the cat, cat sat
• Trigrams (n=3): the cat sat

Now not good becomes a single bigram token, distinct from good, and your sentiment classifier can finally tell them apart. N-grams also powered the earliest language models: predict the next word from the previous n−1 words, using nothing but counts.

The problem is combinatorial explosion. A 10,000-word vocabulary has 100 million possible bigrams and a trillion possible trigrams. Most never occur, so the vectors become enormous and almost entirely zeros. This sparsity is the wall n-grams hit: you cannot extend the window far enough to capture long-range meaning without the representation blowing up.

What these methods could never do

Bag-of-words and n-grams share one fatal limitation: every word is an isolated symbol. In the count vector, cat and kitten are as unrelated as cat and democracy. The model has no notion that some words mean similar things. It sees only exact matches.

A search for “car” misses documents about “automobiles.” A classifier trained on “excellent” learns nothing about “superb.” Each word is a separate dimension with no relationship to any other. This is the specific gap that word2vec was invented to close, by giving words dense vectors where similar meanings sit close together.

What to remember

• Bag-of-words represents a document as word counts, discarding order. Crude, but strong for topic-level tasks where word presence carries the signal.
• TF-IDF weights each word by frequency times rarity, automatically suppressing common words and amplifying distinctive ones.
• N-grams capture short-range order by treating word sequences as units, but explode combinatorially and grow sparse.
• The shared flaw: every word is an isolated symbol, so cat and kitten are treated as completely unrelated.
• These remain useful baselines, and losing to them is a good sanity check for any fancier method.

Next: word2vec — giving words dense vectors so that similar meanings finally sit close together.