Speech & Audio AI — Overview

A completely separate modality from text and images, with its own representation problem before any model can even start: raw audio is a continuous pressure wave sampled thousands of times a second — turning that into something a neural network can learn from is most of the work in this section.

What's in this section

Audio Fundamentals — how sound is digitized, the Fourier transform, spectrograms, and MFCCs — the representations every downstream audio model is actually built on.
Speech & Audio Tasks — ASR (speech-to-text), TTS (text-to-speech), speaker recognition, diarization, audio classification, speech enhancement, and audio-language models.

Why Audio Needs Its Own Representation Layer

Text arrives already discrete (characters, then subword tokens). Images arrive as a fixed spatial grid a CNN can convolve over directly. Raw audio is neither — a few seconds of CD-quality audio is tens of thousands of raw amplitude samples, far too long and far too low-level a sequence for a Transformer to attend over directly, and with no obvious "token" boundary the way text has words. Every technique in Audio Fundamentals exists to solve exactly this: turning a raw waveform into a shorter, richer sequence of features a model can actually work with.

Audio duration3s

Bar length is log-scaled -- the raw sample count is genuinely 161x longer, not just visually longer.

3s of audio at 16kHz is 48,000 raw amplitude samples -- far too long and low-level a sequence to feed a Transformer directly. Framing it into 25ms windows every 10ms (the same STFT framing from Audio Fundamentals) collapses that down to 298 frames -- a 161x reduction, and the actual reason a spectrogram (not the raw waveform) is what gets modeled.

The Shape of the Task Landscape

Before Speech & Audio Tasks goes deep on each one, it's worth seeing the whole landscape at a glance — every task below is defined entirely by what goes in and what comes out, and that input/output shape is what determines which architecture pattern applies:

Input

Audio waveform

→

ASR

→

Output

Text

Spoken audio in, a transcript out -- a sequence-to-sequence problem.

ASR and TTS are exact mirror images of each other (audio↔text, in opposite directions); diarization and classification both take audio in, but produce a structural answer (a segmentation, a label) rather than another modality entirely.

Latency Is a First-Class Design Constraint

Unlike a lot of ML systems where "make it more accurate" is close to the whole story, audio systems routinely have a hard wall-clock budget baked into the product itself — a voice assistant that takes 3 seconds to respond has failed at its job regardless of transcript accuracy. The same task gets built completely differently depending on which latency zone it has to live in:

Latency budget300ms

Real-time, turn-taking

Voice assistant response, live translation

The same task (ASR, TTS) gets architected completely differently depending on which of these zones it has to live in.

Requires streaming inference (process audio as it arrives, not after the fact) and small, efficient models -- accuracy is traded for speed.

This is why "streaming ASR" and "offline ASR" are practically different engineering problems built from the same underlying ideas, not just the same model run with a stopwatch.

Evaluation Is Task-Specific

There is no single "audio accuracy" metric — what "correct" means changes with the task:

Metric	Word Error Rate (WER)
Measures	Edit distance (insertions + deletions + substitutions) between predicted and reference transcript, divided by reference word count.
Direction	Lower is better -- 0% is a perfect transcript.

Every task here needs its own metric because 'correct' means something different each time -- exact text match, perceived naturalness, correct speaker attribution, and correct label are four genuinely different notions of success.

See the roadmap for the full ordered path.