Training & Architecture
Fine-tuning, adaptation, and the architectures behind modern models.
12 Training and Adaptation
- How Fine-Tuning Actually Works Continue training an existing model on your own data. What it changes, what it cannot change, and why the dataset is the whole job.
- Model Distillation Train a small model to imitate a large one. Often better than training the small model from scratch, and the reason cheap models got good.
- Hyperparameter Tuning Some settings are learned during training; others you must choose before it starts. How to search the ones you choose without fooling yourself or wasting compute.
- LoRA and Parameter-Efficient Fine-Tuning Train a small number of extra weights instead of all of them. Nearly all the benefit at a fraction of the cost, plus swappable adapters.
- Reading Model Benchmarks Benchmark tables are marketing surfaces with real numbers in them. How to extract signal, and why your own twenty examples beat any leaderboard.
- Optimizers and Learning Rate Gradient descent says which way is downhill. The optimizer and learning rate decide how far to step and how to remember past steps. Why SGD gave way to Adam.
- What Is Quantization? Store weights with fewer bits. Large memory and speed savings for small quality loss, and the main reason big models run on small hardware.
- Regularization: Dropout and Weight Decay Techniques that make a model deliberately worse at fitting training data so it does better on data it has never seen. Why handicapping a model helps.
- How Models Are Aligned Turning a text predictor into an assistant takes preference training. What that adds, what it costs, and why refusals feel inconsistent.
- How Reasoning Models Work Models trained to think at length before answering. The same token-equals-computation principle, moved from your prompt into the training.
- Synthetic Data Generating training data with models. It works when you can verify or filter the output, and fails quietly when you cannot.
- Transfer Learning: Why Pretraining Reuses A model trained on one huge task can be repurposed for a different, smaller one with a fraction of the data. Why representations transfer, and when they do not.
13 Advanced Architecture
- Attention Variants: Sparse, Linear, and Sliding Window Full attention compares every token to every other, and the cost grows with the square of length. The variants that break that quadratic, and what each gives up.
- How Embedding Models Are Trained A general model's internal vectors are not the same as a good embedding. What the contrastive objective optimizes, and why hard negatives make or break it.
- How Inference Serving Works Batching, scheduling, and memory management. Why throughput and latency pull against each other, and what a serving system is actually doing.
- How Long Context Became Possible Context windows grew from thousands to millions of tokens. Four changes made that happen, and none of them repealed the quadratic.
- Mixture of Experts Hold many parameters, use a few per token. How models grew far larger without getting proportionally slower.
- Merging and Combining Models Average the weights of two fine-tuned models and you often get one that does both jobs. Surprisingly effective, and nobody fully knows why.
- Speculative Decoding Guess several tokens with a small model, verify them all at once with the large one. Free speed, identical output.
- State Space Models An alternative to attention with linear cost and constant memory. Where it wins, where it loses, and why hybrids are winning.