How AI Works
From machine learning basics to how a transformer works inside.
1 Entry
- The AI Jargon Decoder Thirty AI terms, each in one sentence, each linked to a fuller explanation. Use it as an index when a word blocks you mid-article.
- In What Order Should You Learn AI? A dependency-ordered path through AI concepts, from tokens to agents — plus three routes depending on how deep you actually need to go.
- What AI Can and Cannot Do (Yet) A capability map: what current AI is genuinely good at, what it fails at, and the dangerous middle category where it looks competent but isn't.
- What Is AI, and How Do People Actually Use It? Forget the definitions. Here are the five things people actually do with AI every day, and what connects them.
2 ML Foundations
- Probability for AI Models do not output answers, they output probability distributions. Understanding distributions, softmax, and confidence explains how AI systems actually behave.
- What Is Machine Learning? Machine learning is programming by example instead of by rule. Here are the three ways a machine can learn from data, and when each one applies.
- How Data Becomes Numbers A model does arithmetic, so every input — text, images, categories — must first become a vector of numbers. Here is how that conversion works and why it matters.
- What Is a Neural Network? A neural network is layers of simple units that each compute a weighted sum. Stack enough of them and they approximate almost any function. Here is how.
- Weights and Biases Weights and biases are the numbers a neural network learns. Weights set how much each input matters; biases shift the result. Everything a model knows lives here.
- The Forward Pass The forward pass is how a network turns an input into an output: layer by layer, each computing weighted sums and activations. This is inference in one word.
- Activation Functions Activation functions add the nonlinearity that lets deep networks learn curved patterns. Here is why they are non-negotiable and how ReLU and sigmoid differ.
- Loss Functions A loss function turns 'the model was wrong' into a single number to minimize. It defines what the model is trying to do — choose it carefully.
- Gradient Descent Gradient descent is how models learn: measure the slope of the loss, step downhill, repeat. The single algorithm behind training almost every neural network.
- Backpropagation Backpropagation is how a network computes the gradient for every weight efficiently: apply the chain rule backward through the layers, reusing work as you go.
- Overfitting and Generalization A model that memorizes its training data fails on new data. Overfitting versus generalization is the central tension in all of machine learning.
- Train, Validation, Test Split your data into three parts with three different jobs. Confuse them and your model looks great in development and fails in the real world.
3 Classic NLP
- Recurrent Neural Networks The first neural network built to read sequences one step at a time, carrying a hidden state forward. How RNNs work, what they made possible, and the flaw that limited them.
- LSTM and GRU: Memory That Lasts Recurrent networks that forgot too fast got a fix: gated memory cells. How LSTMs and GRUs decide what to keep, what to discard, and why they ruled NLP for a decade.
- Sequence to Sequence and Encoder-Decoder How two recurrent networks were chained to turn one sequence into another — the architecture behind neural machine translation, and the bottleneck that led to attention.
- The Origin of Attention Before self-attention, there was Bahdanau attention: a 2014 fix for translation that let the decoder look back at the whole input. Where the idea came from and how it worked.
- Classic Text Preprocessing Before models learned to read raw text, engineers cleaned it by hand: tokenizing, lowercasing, removing stop words, and stemming words to their roots.
- Bag of Words and N-grams The first way anyone turned text into numbers: count the words, ignore the order. Simple, surprisingly effective, and the baseline everything else is measured against.
- Why Transformers Won RNNs read one step at a time and forgot too much. Transformers process everything at once and reach any distance in one hop. The three reasons the field switched, in full.
- word2vec: Where Word Vectors Began The idea that changed NLP: represent each word as a dense vector learned from its neighbours, so that similar words land near each other and meaning becomes arithmetic.
4 Foundations
- Logits and Softmax How a model's raw output scores become a probability distribution over the next token. The softmax function, what temperature does to it, and why the numbers are relative, not absolute.
- What Is a Token in AI? A token is the unit an AI model actually reads — not a word, not a letter. It explains your bill, your context limit, and why models miscount letters.
- What Is an Embedding? Embeddings turn meaning into coordinates, which is what lets a machine compute with language. The foundation under search, RAG, and attention.
- Where Does the Vocabulary Come From? Byte Pair Encoding in four steps, and why the resulting vocabulary is a permanent fossil of the text it was built from.
- What Is an LLM? A large language model is a machine trained to predict the next token. Here is why that simple objective produces something that looks like thinking.
- How Does an LLM Actually Write? One token at a time, each conditioned on everything before it. The generation loop explains streaming, cost, and why models cannot revise.
- Base Models vs Chat Models A pretrained model continues text; it does not answer questions. What instruction tuning adds, and why the assistant persona is a trained layer.
- The Language Modeling Objective Predicting the next token is not a metaphor for what an LLM does — it is the exact training target. Here is how that objective is defined and why it needs no human labels.
- Perplexity: A Language Model's Intrinsic Measure Perplexity scores how surprised a model is by real text. What the number means, how it relates to loss, and why a lower score is not the same as a better assistant.
- Temperature and Top-p, Explained Why the same prompt gives different answers, and how the two main sampling knobs actually reshape the model's choices.
- Training vs Inference One is an enormous one-time expense that changes the model. The other is what happens every time you send a prompt, and it changes nothing.
- What Happens During Pretraining? Hide the next token, guess it, adjust on error, repeat a trillion times. How a simple objective at enormous scale produces broad capability.
- What Does "7B Parameters" Mean? Parameters are the adjustable numbers that hold everything a model learned. Why the count matters, and why it stopped being the whole story.
5 Internals
- Decoding Strategies: Greedy Search and Beam Search Once a model produces a probability distribution, something has to turn it into a sequence. The deterministic strategies — greedy and beam search — and why the locally best token is not the globally best sentence.
- The Geometry of Meaning Embedding spaces have structure: directions carry relationships, clusters carry categories, and the geometry behaves in ways that are useful and occasionally misleading.
- How Does a Transformer Work? The architecture behind every current language model, traced from input tokens to output probabilities — one layer at a time.
- Encoder, Decoder, and Encoder-Decoder Architectures Three ways to arrange a transformer stack — encoder-only, decoder-only, and encoder-decoder — differ in one thing: which positions can see which. That choice determines what each is good for.
- The Feed-Forward Network in a Transformer Half of every transformer layer is not attention — it is a small per-token network holding most of the model's parameters. What it computes and why factual knowledge appears to live there.
- Layers, Residuals, and Depth Why models stack dozens of identical blocks, and the two small tricks that make deep stacks trainable at all.
- Normalization Layers: LayerNorm and RMSNorm Why every transformer block rescales its activations, how LayerNorm and RMSNorm do it, and why this unglamorous step is what makes deep models trainable at all.
- The Output Projection: From Hidden State to Vocabulary Logits The final step of a transformer turns one hidden vector into a score for every token in the vocabulary. How that projection works, why it is the largest single matrix, and why models tie it to the embeddings.
- Self-Attention, Explained Visually How each word decides which other words matter to it. Queries, keys, and values — the mechanism at the center of every language model.
- What Is a KV Cache? Generating token 500 should not require recomputing tokens 1 through 499. The cache that prevents it, and the memory it costs instead.
- Why Multiple Attention Heads? One attention pass produces one blend. Running many in parallel lets a model track grammar, reference, and position at the same time.
- How Does a Model Know Word Order? Attention is order-blind by construction. Position has to be injected separately, and how it is injected determines how far context can stretch.
- Why Long Contexts Cost So Much Attention compares every token with every token, so doubling the context quadruples the work. The constraint behind every context limit and price tier.
6 Prompting
- What Is a Context Window? One token budget covering your prompt, the conversation history, retrieved documents, and the answer being generated. Everything competes for the same space.
- Why Do AI Models Hallucinate? Fluent, confident, wrong. Hallucination is not a bug in the system — it follows directly from what the system was built to do.