From Text to Sound: A Deep Dive into Text-to-Speech Generation and its Emerging Applications

Written September 2023. The original claimed my team and I were “pioneering models that can grasp and reproduce a spectrum of emotions.” That was not true. I was reading about this field, not working in it, and inventing a lab to write from was indefensible. Rewritten in 2026, with the invented credentials gone, the architecture corrected, and, since it is a post about sound, audio.

Text-to-speech is the problem of going from a string to a waveform. It is unusually good for watching a field’s architecture get simpler as it gets better, because TTS ran that arc in about eight years and every stage is still visible in something deployed.

The classical pipeline

For most of its history TTS was four components, each with its own failure mode.

"Dr. Smith read 1984."
      |
      v  text normalisation   -> "Doctor Smith read nineteen eighty-four"
      v  grapheme-to-phoneme   -> /d ɑ k t ər  s m ɪ θ  r ɛ d .../
      v  acoustic model        -> mel spectrogram (pitch, timing, energy)
      v  vocoder               -> waveform

The first stage is the one nobody expects to be hard. Dr. is Doctor or Drive depending on position. 1984 is a title, a year or a quantity depending on what it modifies. read is /riːd/ or /rɛd/ depending on tense, which you cannot determine without parsing the sentence.

None of that is acoustics. It is linguistics, and the original version of this post asserted that a “startling share” of remaining production errors still come from there.

I never checked. So this time I did.

Testing the front end

Seven sentences through gpt-4o-mini-tts, five takes each where a number was being measured. Two kinds of test, because they need different instruments.

Normalisation changes which word comes out, so speech recognition can check it. Homographs keep the word and change only the sound, so ASR is useless and the waveform has to be measured directly. Nothing here asks a model to grade a model.

Abbreviations: correct

“Dr. Sarkar lives on Elm Dr.”: the same three characters, twice, expanded two different ways.

Transcribing it back gives “Dr. Sarkar lives on Elm Drive”: the first read as Doctor, the second as Drive. Had it read both the same way the transcript would say so. This is the textbook hard case and it is handled.

Homographs: also correct

“I read the paper yesterday” then “I will read the paper tomorrow.” Same four letters, different vowel.

“The exit was too close to risk it” then “The exit was too stiff to close it.” Listen to the last consonant.

These I had to measure. For read, the contrast is vowel height: /iː/ is a high vowel and high vowels have a low first formant, so I ran LPC over the vowel nucleus with the flanking consonants trimmed off. For close, the contrast is voicing in the final fricative, so I found the sibilant by its high-frequency energy and measured periodicity there: /z/ keeps vocal-fold vibration, /s/ does not.

Strip plots of five takes each. Vowel height for read separates cleanly at 427 hertz past tense against 270 infinitive. Final-consonant voicing for close separates cleanly at 0.41 unvoiced against 0.77 voiced. The number does not separate: 1984 bytes takes 740 milliseconds, the short year reading, where spelling the count out takes 1107.

Both separate completely, with no overlap between the two sets of takes. F1 averages 427 Hz for the past tense against 270 Hz for the infinitive; the lowest past-tense take still sits above the highest infinitive one. Final-consonant periodicity averages 0.41 for the adjective against 0.77 for the verb, and again the ranges do not touch.

So the answer to the question the original post never asked is that the classic front-end failures are solved. Not improved. Solved, on these cases, by a model that reads the whole sentence before it opens its mouth. That is the correction. I was repeating a 2015 complaint in 2023 without testing whether it still held.

One case it gets wrong

“The file is 1984 bytes long” then the same count spelled out in words. The first one is shorter, because it is not saying the same thing.

1984 after “in” is a year and gets the four-syllable reading. 1984 before “bytes” is a count and should get “one thousand nine hundred eighty-four.” It does not.

Whisper renormalises spoken numbers back to digits, so ASR cannot see this at all: the transcript reads “1984 bytes” either way. Duration can see it. The token takes 740 ms in the sentence, and spelling the count out in words takes 1,107 ms across three control takes, with no overlap between the two sets. It is giving the short reading.

The failure is not where the textbooks put it. Deciding that “read” is past tense needs syntax, and the model has syntax. Deciding that a number before “bytes” is a quantity needs to know what bytes are.

That is a much narrower and more useful claim than the one I made in 2023, and it predicts where to look: units, identifiers, version strings, part numbers, anything where the reading depends on the domain rather than the grammar. Those are also the cases where a wrong reading is least likely to be caught, because the output stays completely fluent. Nobody hears a stumble. They hear a confident, well-prosodied wrong number.

What collapsed, and in what order

The vocoder went first. WaveNet, in 2016, generated raw audio sample by sample and was the first system most people could not distinguish from a recording. It was also far too slow to use, at 24,000 sequential forward passes per second of audio, and the following years went into making that practical, ending with GAN vocoders like HiFi-GAN that run far faster than real time.

Then the acoustic model absorbed the front end. Tacotron 2 took characters or phonemes straight to spectrograms with attention, deleting most of the hand-built intermediate representations. Prosody stopped being a set of rules and became something learned.

Then the whole thing became a language model. The current generation treats audio as a sequence of discrete tokens from a learned neural codec and generates those tokens the way a text model generates words. VALL-E is the clearest statement of it. Once audio is tokens, everything the language-modelling world knows applies directly, including in-context learning, which is why current systems copy a voice from a few seconds of reference audio with no fine-tuning at all.

It also explains the result above. The front end stopped being a rule-based stage that could be wrong about tense, and became a model that reads the sentence. What it does not have is a reason to know that bytes are counted.

What is still hard

Domain-dependent normalisation, per the above. The measurable one, and the one you can regression-test in your own vocabulary.
Long-form prosody. A sentence sounds excellent. Twenty minutes of audiobook drifts: emphasis lands oddly, energy stops tracking the narrative. The model has no representation of where it is in the story.
Low-resource languages. Real, unglamorous, mostly a data problem rather than an architecture one. A language with a hundred hours of clean speech is in a completely different position from one with none.
Evaluation. Mean opinion score needs human listeners and saturates once output is close to natural. Telling which of two very good systems is better is now genuinely difficult, which slows everything down.

The part that needs saying out loud

Voice cloning from a few seconds of audio is deployed, cheap, and available to anyone. The original version of this post treated that as an exciting frontier with a passing note about “malevolent hands.”

That framing has not aged well. Voice is used as an authentication factor by banks, and as an informal one by everybody; recognising a relative on the phone is the oldest identity check there is. A technology that defeats both, from a public clip of someone speaking, is not a footnote to a post about architecture. The mitigations that exist (watermarking, provenance standards, detection models) are all partial, and detection in particular is a structurally losing position. It deserves its own post; a paragraph is better than the sentence it replaces.

Method and caveats

One system, one voice, English. gpt-4o-mini-tts, voice “alloy”. A different system would give different answers and the point of publishing the script is that you can run it against yours.
Five takes per measured sentence, three for the spelled-out control. Small, but the separations are complete rather than marginal, which is why I am willing to state them.
The first version of this measurement used the spectral centroid of the whole word as a stand-in for F2 and produced an answer in the wrong direction. That was the instrument, not the model: the centroid of “read” is dominated by the /r/, which is exactly the sound that suppresses F2. Both attempts are in the repository.
Whisper is used only to locate word boundaries, never to judge pronunciation, because it cannot: it normalises away the distinctions under test.

Code and audio: experiments/tts-frontend.

References

WaveNet, van den Oord et al., 2016.
Tacotron 2. Shen et al., 2018.
HiFi-GAN. Kong et al., 2020. The vocoder that made this deployable.
VALL-E. Wang et al., 2023. Audio as discrete tokens, and voice cloning as in-context learning.