How Speech Recognition Works

Audio becomes a spectrogram, the spectrogram becomes tokens, and from there it is sequence-to-sequence translation.

Modern speech recognition is a transformer doing translation — from audio to text. The interesting part is how audio becomes something a transformer can read.

Audio to spectrogram

Raw audio is a waveform: tens of thousands of amplitude samples per second. Feeding that directly to a transformer would produce absurdly long sequences.

So audio is converted to a spectrogram first. Slice the waveform into short overlapping windows — typically 25 milliseconds — and for each window compute which frequencies are present and how strongly.

The result is a 2D array: time along one axis, frequency along the other, intensity as the value. Essentially a picture of the sound.

Two refinements are standard. Frequency bins are spaced according to human hearing sensitivity — finer at low frequencies, coarser at high — producing a mel spectrogram. And intensities are log-scaled, matching how loudness is perceived.

Now the representation is compact and structured, and the same patch-token approach used for images applies. Spectrogram frames become a sequence of vectors.

Encoder-decoder

Speech recognition is one of the places where the full encoder-decoder transformer is still the right architecture, rather than decoder-only.

The encoder reads the entire spectrogram bidirectionally. Every frame can attend to every other frame, which matters because later audio disambiguates earlier audio — you often cannot resolve a word until you have heard the end of the sentence.

The decoder generates text tokens autoregressively, attending to the encoder output through cross-attention. Standard next-token prediction, conditioned on audio.

This is why speech models handle homophones reasonably: “their” versus “there” is decided by the language model half, using surrounding context, not by the acoustics.

What multitask training buys

The influential design choice in recent systems is training one model on several tasks at once, distinguished by special tokens in the decoder prompt: transcribe in the same language, translate to English, detect the language, predict timestamps, and detect whether speech is present at all.

One model, several capabilities, and they reinforce each other — language identification improves transcription, and timestamp prediction improves segmentation.

It also means the decoder is a genuine language model. Which produces a characteristic failure: on silence or noise, it can hallucinate fluent text, generating plausible sentences from nothing. This is the same dynamic as text hallucination — a model trained to produce plausible continuations does so whether or not there is signal. Voice-activity detection before transcription is the standard defense.

Streaming versus batch

A bidirectional encoder needs the whole clip, which is fine for recorded audio and unacceptable for live captioning.

Streaming systems make a tradeoff: process in chunks with limited lookahead, accepting lower accuracy for low latency. Some emit provisional text and revise it as more audio arrives — the visible flicker in live captions.

If you do not need real-time output, batch processing is both more accurate and simpler. Choose deliberately.

What affects accuracy

Audio quality dominates everything else. Sample rate, background noise, microphone distance, compression artifacts.

Overlapping speakers are handled poorly. Separating who said what — diarization — is a distinct problem usually solved by a separate model.

Domain vocabulary. Product names, medical terms, and proper nouns transcribe badly if they were rare in training. Many systems accept a hint list or initial prompt to bias toward expected terms — cheap and effective.

Accents and dialects vary with training representation, and the disparity is real rather than incidental.

Language. Well-resourced languages transcribe far better, mirroring the tokenizer fossil problem in text.

Practical notes

Fix the audio before tuning the model. Better microphone placement and noise reduction beat any parameter change.

Run voice-activity detection first, both to save cost and to prevent hallucinated transcripts on silence.

Post-process with a language model for punctuation, formatting, and domain corrections. Transcription and cleanup are separable problems, and treating them separately works better.

Never trust timestamps precisely. They are predicted, not measured, and drift on long audio.

What to remember

• Audio becomes a mel spectrogram — a picture of frequencies over time — then a token sequence.
• Encoder-decoder architecture: bidirectional audio encoding, autoregressive text generation.
• Multitask training gives transcription, translation, and language ID in one model — and makes the decoder capable of hallucinating on silence.
• Streaming trades accuracy for latency; use batch when you can.
• Audio quality, overlapping speakers, and domain vocabulary drive accuracy more than model choice.

Next: How Text-to-Speech Works