Chapter 09: Quality Benchmarks & Honest Naturalness Matrix

No single score captures all of these. The metrics below each capture a slice, which is why you need several of them — and why a top leaderboard rank is necessary but never sufficient.

The core metrics: MOS, CMOS, MUSHRA, WER

MOS — Mean Opinion Score. The classic subjective metric, standardized in ITU-T P.800. Human listeners rate audio on a 1–5 scale (1 = bad, 5 = excellent) and you average across raters and samples. A modern "very good" neural TTS lands roughly in the 4.0–4.5 range; natural human recordings typically score around 4.5–4.8 (not a perfect 5, because raters are harsh and recordings have their own flaws).

MOS limitations are severe and worth internalizing:

• It is subjective and pool-dependent — the same model scored by two different rater groups can differ by several tenths of a point, so a "4.3" in one paper is not comparable to a "4.3" in another.
• It is absolute with no anchor — listeners drift; some never give 5s, some never give 1s.
• It is influenced by playback conditions (headphones vs laptop speakers, loudness).
• It is expensive and slow to collect properly.

The practical consequence: never compare MOS numbers across different papers or vendors as if they were on the same ruler. A MOS is only meaningful relative to other systems scored in the same study by the same listeners.

CMOS — Comparative MOS. Listeners hear two samples (A vs B) and rate the difference, typically on a −3 to +3 scale (−3 = A much worse, +3 = A much better, 0 = tie). This is far more sensitive than absolute MOS for detecting subtle differences, because you remove the "where is 4 on my personal scale" problem. A CMOS near 0 means two systems are perceptually a tie; a CMOS of −0.3 to −0.5 is a small but real preference. CMOS is the right tool when you are deciding between two genuinely good options — which is exactly your situation.

MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor, ITU-R BS.1534). Listeners rate several systems at once on a 0–100 scale, with a hidden high-quality reference and a deliberately degraded "anchor" mixed in to calibrate the scale. More efficient than MOS when comparing many systems, and the hidden reference/anchor make scores somewhat more stable. You will see it in audio-codec and academic TTS papers more than in product marketing.

WER — Word Error Rate. This is the only objective, fully automatic metric of the four. You feed the synthesized audio into a strong ASR (speech-to-text) system, transcribe it, and compare to the text you fed in:

WER = (Substitutions + Insertions + Deletions) / Total reference words

Lower is better; a clean modern English TTS run through a good ASR often lands in the low single-digit percentages. WER measures intelligibility, not naturalness — a flat robotic voice can have a near-zero WER. Its limitations:

• It inherits the ASR's own errors and biases. If your ASR is weak in Czech, a perfectly intelligible Czech TTS will score a bad WER for reasons that have nothing to do with the TTS.
• It says nothing about prosody, warmth, or expressiveness.
• It can be gamed by hyper-articulated, unnatural speech.

WER is most useful as a guardrail: it catches catastrophic failures (skipped sentences, hallucinated words, garbled numbers) that a quick listen might miss. Pair it with a human/CMOS judgment of naturalness — never use it alone.

Public leaderboard #1: TTS Arena (Hugging Face)

The most-cited community benchmark is TTS Arena by the TTS-AGI group on Hugging Face, now in its V2 generation. Its design borrows directly from the LMSYS Chatbot Arena idea: instead of asking people for an absolute MOS, it shows a listener two anonymized audio clips of the same text generated by two different (randomly chosen) models, the listener picks which sounds better, and those pairwise votes are aggregated into an Elo-style ranking. This is essentially crowd-sourced CMOS at scale.

Why this design is good:

• Blind and randomized — listeners don't know which model is which, removing brand bias.
• Pairwise — sidesteps the "where is 4 on my scale" drift of absolute MOS.
• In the wild — many listeners, many texts, large vote counts, continuously updated.
• It mixes commercial APIs and open-source models in the same pool, which is rare and valuable.

V2 expanded the model roster and refined the methodology over the original arena, pulling in a wide range of contemporary systems — commercial engines (e.g., ElevenLabs, MiniMax, Cartesia, PlayHT, Inworld, Hume) alongside open models (e.g., Kokoro, Sesame CSM, and other open releases). The original Hugging Face blog post "TTS Arena: Benchmarking Text-to-Speech Models in the Wild" documents the methodology; the V2 leaderboard is a fresh Elo restart (V1 votes do not carry over).

As a moment-in-time illustration (and only that — rankings move): in 2026 snapshots, the very top of the board is held by frontier commercial systems — recent entries from Inworld, MiniMax (Speech-02 / 2.x HD lines), ElevenLabs (which routinely places several models in the top 10 across its Turbo/Flash/Multilingual tiers), Hume's Octave, and Google's Gemini-based TTS, with Elo spreads of only a few dozen points separating the leaders (i.e., perceptually close). The instructive open-source data point is Kokoro-82M, which sits well down the board (roughly mid-table) yet is remarkable for an 82M-parameter open model with one of the largest vote samples on the leaderboard — proof that you can get respectable English naturalness from a tiny free model, but also a reminder that the open models are not (yet) beating the commercial frontier on raw English naturalness.

Critical caveats — read these before you trust a rank:

1. English-only by design. This is not a soft bias — the TTS Arena maintainers state explicitly that all test cases are in English and models are graded on US/UK English across four use-case categories (assistants, customer service, entertainment, knowledge-sharing). A model's Arena rank tells you about its English naturalness and nothing about Czech. XTTS supports ~17 languages and Kokoro ~9, but the arena captures none of that multilingual ability; some top-ranked commercial models are English-only. Do not let a high Arena rank convince you a model is good in Czech — that has to be checked separately. (Maintainers have discussed adding multilingual evaluation via Common Voice sentences, but as of this writing the public board is English.)
2. Voter composition is uncontrolled. Anyone can vote; there is no trained-listener panel, no audio-condition control, and self-selection bias is real.
3. Snapshot, not gospel. Rankings shift as models are added/removed and as vote volume changes; treat any specific ordering as a moment-in-time snapshot, not a permanent truth. (For that reason this chapter deliberately does not transcribe a frozen "model X is #1" list — go read the live board.)
4. Same-text constraint flattens some strengths. Voice cloning, long-form stability, and expressive range may not show up in short same-text clips.

Public leaderboard #2: Artificial Analysis TTS

Artificial Analysis runs a parallel "Text to Speech" leaderboard and arena (artificialanalysis.ai/text-to-speech). It is more commercial-provider focused than the Hugging Face arena and is widely cited in industry write-ups. Like TTS Arena it uses head-to-head preference voting to produce a quality ranking, and it pairs that with provider-level data on price and speed/latency, which is genuinely useful for an engineering decision. The same English-bias and snapshot caveats apply: it is primarily an English-quality and commercial-coverage view. Use it to compare commercial options on the quality/price/latency triangle, and cross-check its quality ordering against the Hugging Face arena.

Public benchmark #3: TTSDS2 (objective, distribution-based)

A useful complement to the preference arenas is TTSDS2 (Text-to-Speech Distribution Score 2), a 2025 academic benchmark (arXiv:2506.19441). Instead of crowd votes, it computes how closely a system's synthetic speech matches real human speech distributions across several factors — prosody, speaker identity, intelligibility, and general naturalness — and the authors report it correlates strongly with human ratings (validated against ~11,800 collected listening-test ratings). Why it matters for you: it is objective and reproducible (no rater pool to drift), and the framework is designed to be re-run on new languages and datasets. It is still research-grade and not a product leaderboard, but it is the most credible objective naturalness metric to cite in 2026, and it is the kind of thing you could point at Czech data yourself.

Other evidence sources (and how much to trust them)

• Academic papers (arXiv) & Papers With Code — report MOS/CMOS/WER on standard datasets (LibriTTS, VCTK, LJSpeech, SEED-TTS-eval, etc.). Trust the relative comparisons within a single paper; distrust absolute numbers across papers. Almost all of these datasets are English or Chinese — Czech is essentially absent.
• Vendor benchmark pages (e.g., a "new SOTA" launch post) — treat as marketing until independently reproduced. Vendors choose the test set, the competitors, the metric, and the cherry-picked samples. A "lowest WER on our multilingual benchmark" claim is the vendor grading its own homework on a private dataset.
• Community testing threads (r/LocalLLaMA "I tested most of the open-source TTS models", and similar) — surprisingly useful as a reality check on stability and ease-of-use, because the testers run the models themselves and report failure modes vendors hide. But they are anecdotal and English-heavy.
• Vendor demo pages — the most dangerous of all (see the warning below).

The honest naturalness matrix: EN vs CZ

Here is the part the public leaderboards cannot give you: a per-engine grade that separates English from Czech and attaches a confidence level to each grade. The grades below are qualitative bands synthesized from the arenas (for English), from documented language support, and from community/community-listening evidence (for Czech). They are not transcribed MOS numbers — see the warnings above about why that would be dishonest. Confidence reflects how much independent, Czech-specific evidence exists.

Grade legend: A = clearly natural / among the best · B = good, usually natural · C = usable but noticeably synthetic or inconsistent · D = weak / robotic · n/a = not supported. Confidence: High = strong independent evidence · Med = some evidence · Low = sparse, mostly inferred.

How to read this matrix for your decision (natural EN and CZ, free/near-free):

• The engines that pass the hard Czech filter with reasonable confidence are ElevenLabs, Google Cloud, Azure (commercial, free tiers then paid) and Coqui XTTS v2 (free, self-hosted, explicitly Czech).
• The fashionable English-arena darlings (Kokoro, Cartesia, and to a lesser degree MiniMax/OpenAI for Czech specifically) either don't support Czech or have unverified Czech — a great English rank is a trap here.
• For the "free or near-free + natural + Czech" sweet spot, the realistic shortlist is Coqui XTTS v2 self-hosted (free, Czech confirmed, clone-capable) versus a commercial free tier (ElevenLabs / Google / Azure) whose Czech is a step more natural but metered. The deciding factor is your own ears on your own Czech text — which brings us to the only benchmark that actually settles this.

Run your own blind A/B test (the only benchmark that matters)

Public leaderboards are English. Vendor demos are cherry-picked. Your texts are yours. So build a tiny, honest, blind test harness. It takes an afternoon and will out-decide every leaderboard for your specific use case.

Step 1 — Pick representative texts. Choose 8–15 short passages (2–5 sentences each) drawn from your real content, split roughly half English, half Czech. Deliberately include the hard cases that break TTS: numbers and dates, an acronym, a person's name, a foreign loanword, a question, and one long compound sentence. Save them as texts_en.txt and texts_cs.txt.

Step 2 — Generate the same texts from each candidate engine. Render every passage with every candidate into samples/<engine>/<id>.wav. For self-hosted XTTS, that's a local call; for cloud engines, their SDK. Keep loudness consistent (normalize all clips) so loudness doesn't bias votes.

Step 3 — Blind-randomize and listen. Strip engine names from filenames, shuffle, and rate each clip 1–5 (MOS-style) and do pairwise A/B picks (CMOS-style) per text. Keep a hidden key mapping shuffled IDs back to engines. Critically: rate Czech and English separately and keep separate scoreboards — an engine can be A in English and C in Czech.

A minimal harness to randomize, collect votes, and reveal the key:

import os, random, json, glob

# samples/<engine>/<id>.wav  -> blind A/B over the SAME text id
engines = [d for d in os.listdir("samples") if os.path.isdir(f"samples/{d}")]
ids = sorted({os.path.splitext(os.path.basename(p))[0]
              for p in glob.glob("samples/*/*.wav")})

votes, key = [], {}
for tid in ids:
    a, b = random.sample(engines, 2)              # two random engines, same text
    label_a, label_b = f"{tid}-A", f"{tid}-B"
    key[label_a], key[label_b] = a, b             # hidden mapping
    print(f"\nText {tid}")
    print(f"  Play A: samples/{a}/{tid}.wav")
    print(f"  Play B: samples/{b}/{tid}.wav")
    pick = input("  Which sounds more natural? [a/b/tie]: ").strip().lower()
    votes.append({"text": tid, "winner": pick, "A": a, "B": b})

# tally
from collections import Counter
score = Counter()
for v in votes:
    if v["winner"] == "a": score[v["A"]] += 1
    elif v["winner"] == "b": score[v["B"]] += 1
    else: score[v["A"]] += 0.5; score[v["B"]] += 0.5
print("\nBlind A/B results:", score.most_common())
json.dump({"votes": votes, "key": key}, open("results.json", "w"), indent=2)

Step 4 — Add an objective WER guardrail. Transcribe every generated clip with a strong multilingual ASR (e.g., a Whisper-class model, which handles Czech reasonably) and compute WER against the source text. This catches the silent killers — a skipped clause, a mangled number, a hallucinated word — that a quick listen misses. Use a Czech-capable ASR for the Czech clips, otherwise the WER reflects the ASR's Czech weakness, not the TTS.

# pip install faster-whisper jiwer
from faster_whisper import WhisperModel
from jiwer import wer
import glob, os

asr = WhisperModel("large-v3")   # multilingual; decent Czech
refs = dict(l.split("\t", 1) for l in open("refs.tsv"))   # id<TAB>text

for wav in glob.glob("samples/*/*.wav"):
    tid = os.path.splitext(os.path.basename(wav))[0]
    lang = "cs" if tid.startswith("cs") else "en"
    segs, _ = asr.transcribe(wav, language=lang)
    hyp = " ".join(s.text for s in segs).strip()
    print(f"{wav}\tWER={wer(refs[tid].lower(), hyp.lower()):.3f}")

Step 5 — Decide on the combined picture. Rank by your blind A/B/MOS for each language separately, then disqualify anything with alarming Czech WER. The engine that wins your Czech scoreboard while staying free/near-free is your answer — regardless of where it sits on any public English leaderboard.

Warning: do not trust vendor demos

Every vendor's landing page plays its best clip: a sentence the model was tuned on, in the language it's strongest in (almost always English), at the optimal length, with the best voice, possibly hand-edited. This is the single most misleading "evidence" in the whole field. Specific traps:

• Language bait-and-switch. The slick demo is English; the Czech you'll actually generate is a fraction as polished — and they never demo Czech.
• Cherry-picked length. Demos are short. Your articles are long. Long-form is where stability, speaker drift, and pronunciation consistency fall apart.
• Hand-picked voice. The demo uses the one flagship voice; the other 40 voices (and most non-English locales) are weaker.
• Self-graded benchmarks. "Lowest WER / highest similarity" on a vendor-chosen private dataset against vendor-chosen competitors is an ad, not a measurement.

The antidote is the harness above: your texts, your languages, blind, with an objective WER guardrail. Two hours of that beats two weeks of reading leaderboards.

Key Takeaways

• No single metric captures TTS quality. MOS = absolute naturalness (pool-dependent, not cross-study comparable); CMOS = sensitive A-vs-B preference; MUSHRA = many systems at once; WER = objective intelligibility but inherits ASR bias and ignores prosody. Use several together.
• Never compare MOS across different papers/vendors — they're on different rulers. Within one study, relative ordering is meaningful; absolute numbers are not.
• TTS Arena (Hugging Face, V2) and Artificial Analysis are the two leaderboards worth reading — both use blind, randomized, pairwise (Elo) voting that mixes commercial and open models. But both are English-biased snapshots: a high rank means good English, says little about Czech, and shifts over time.
• Czech is essentially unbenchmarked publicly. Standard datasets and arenas barely touch it, so Czech quality must be graded separately and treated with lower confidence than English.
• Honest EN-vs-CZ shortlist for natural + free/near-free: Czech is reliably supported by ElevenLabs, Google Cloud, Azure (commercial, free tier) and Coqui XTTS v2 (free, self-hosted, Czech confirmed). English arena favorites like Kokoro (no Czech) and Cartesia / MiniMax / OpenAI (Czech unverified or weak) can be traps if you forget the Czech filter.
• Run your own blind A/B test on your own EN + CZ texts, score the two languages on separate boards, and add a Czech-capable-ASR WER guardrail. This single afternoon of work decides the question better than any public benchmark.
• Distrust vendor demos and self-graded benchmarks — they are short, cherry-picked, flagship-voice, English-first, and choose their own test set and competitors.

Key References

• TTS-AGI. TTS Arena V2: A Community-Driven Benchmark for TTS (Hugging Face blog). 2025. https://huggingface.co/blog/arena-tts — Methodology of the blind, randomized, Elo-style pairwise TTS leaderboard and its V2 model roster.
• TTS-AGI. TTS Arena V2 (Hugging Face Space). 2025–2026. https://huggingface.co/spaces/TTS-AGI/TTS-Arena-V2 — Live community leaderboard; vote interface and current rankings.
• TTS-AGI. About — TTS Arena V2. 2025–2026. https://tts-agi-tts-arena-v2.hf.space/about — States explicitly that the arena is English-only (US/UK accents, four use-case categories) and a fresh Elo restart from V1.
• Artificial Analysis. Text to Speech AI Model & Provider Leaderboard. 2025–2026. https://artificialanalysis.ai/text-to-speech/leaderboard — Commercial-focused TTS quality (Elo) / price / speed leaderboard with head-to-head Speech Arena; ~76–83 models tracked in 2026.
• Minixhofer, C. et al. TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems. arXiv:2506.19441. 2025. https://arxiv.org/abs/2506.19441 — Objective, distribution-based naturalness benchmark correlating strongly with human ratings; complements preference arenas.
• AssemblyAI. How to Evaluate Speech Synthesis & TTS Quality. 2024–2025. https://www.assemblyai.com/blog/how-to-evaluate-speech-synthesis-and-tts-quality — Clear explanation of MOS, CMOS, MUSHRA and WER with scales and limitations.
• Deepgram. What is Mean Opinion Score (MOS) and How is it Calculated? 2024. https://www.deepgram.com/ai-glossary/mean-opinion-score-mos — MOS scale, ITU-T P.800 background, and rater-pool caveats.
• Papers With Code. Text-to-Speech Synthesis & Naturalness Evaluation of Speech. 2025. https://paperswithcode.com/task/text-to-speech-synthesis — Standard datasets (LibriTTS, VCTK, LJSpeech) and metrics used in academic TTS evaluation.
• Coqui. XTTS-v2 model card (Hugging Face). 2024. https://huggingface.co/coqui/XTTS-v2 — Confirms the 17 supported languages including Czech (cs); zero-shot voice cloning.
• ElevenLabs. Text-to-Speech capabilities / supported languages. 2025. https://elevenlabs.io/docs/capabilities/text-to-speech — Documents 70+ supported languages including Czech for the multilingual models.
• Google Cloud. Chirp 3: HD voices. 2025–2026. https://docs.cloud.google.com/text-to-speech/docs/chirp3-hd — Confirms cs-CZ is among the Chirp 3 HD supported languages (8 voices, 4M/4F); higher naturalness than older Czech WaveNet/Neural2 voices.
• TTS-AGI / Pendrokar. Support evaluations of other languages? (TTS Arena discussion). 2025. https://huggingface.co/spaces/Pendrokar/TTS-Spaces-Arena/discussions/10 — Community thread documenting the English-only limitation and plans for multilingual (Common Voice) evaluation.
• MiniMax. MiniMax Speech-02: A New Benchmark in TTS. 2025. https://www.minimax.io/news/minimax-speech-02 — Vendor-claimed multilingual WER/similarity results; example of self-graded benchmark to treat cautiously (Czech inclusion unconfirmed).
• rhasspy. Piper — a fast, local neural text to speech system (GitHub). 2024–2025. https://github.com/rhasspy/piper — Open, lightweight engine with Czech voice models; intelligible but clearly synthetic.