Multimodal
Images, audio, and documents beyond plain text.
11 Multimodal
- How Do Models See Images? An image becomes a sequence of patch tokens, and from there the machinery is the same one that reads text.
- How Image Generation Works Start with noise and repeatedly remove a little of it. Diffusion is a different mechanism from text generation, and the difference explains the controls.
- Multimodal Embeddings Train images and text into one shared vector space and you can search pictures with words. The idea behind CLIP and most cross-modal retrieval.
- Contrastive Learning and CLIP How images and text end up in one shared space where a photo and its caption land near each other. The training idea that made multimodal models possible.
- How Speech Recognition Works Audio becomes a spectrogram, the spectrogram becomes tokens, and from there it is sequence-to-sequence translation.
- How Text-to-Speech Works Two stages: text becomes a spectrogram, the spectrogram becomes audio. Plus why prosody is the hard part and latency is the practical one.
- Vision-Language Models Put image patches and text tokens in one sequence and attention does the rest. How a language model gains eyes, and what it still cannot do.
- Document and OCR Understanding Extracting text is solved. Extracting structure is not, and structure is usually what you actually need.