Which offline speech-to-text engine for Python makes the fewest mistakes?
Parakeet, faster-whisper, whisper.cpp, Whisper, Moonshine, Vosk: each one tested on the same 130 English clips with reference transcripts: audiobooks, read sentences, meetings (headset and table microphone) and earnings calls, 14.9 minutes. New PyPI releases are tested again by themselves. Updated 06 November 2026.
130 files from Speech test set (LibriSpeech, FLEURS, AMI meetings, Earnings-22 calls);
130 of them give text to at least one library.
| Library | 🔴 Fails where another reads | 🟡 Empty where another reads |
⏱ Hangs or crashes | ✅ Word error rate | Median ms/file | Total seconds |
|---|---|---|---|---|---|---|
| 1 Parakeet 0.12.0 · model TDT 0.6B v2 Note: With huggingface-hub 1.31.0: with 1.32+ (new shared cache layout) and current onnxruntime, the model fails to load ("External data path escapes model directory"), as many other ONNX models do. why | 0 | 0 | 0 | 11.1% | 770 | 124.2 |
| 2 faster-whisper 1.2.1 · model base.en int8 Note: With PyAV 18.1.0: faster-whisper 1.2.1 (the latest release) fails on every file with PyAV 19 (fixed on its main branch, not released yet). why | 0 | 0 | 0 | 16.8% | 466 | 79.8 |
| 3 whisper.cpp 1.5.1 · model base.en | 0 | 0 | 0 | 16.8% | 1226 | 162.9 |
| 4 Whisper 20250625 · model base.en | 0 | 0 | 0 | 17.6% | 1142 | 164.9 |
| 5 Moonshine 20251121 · model base | 0 | 0 | 0 | 19.8% | 1228 | 167.9 |
| 6 Vosk 0.3.45 · model small en-us | 0 | 1 | 0 | 37.1% | 918 | 138.2 |
By kind of clip: word error rate
| Clips | Parakeet | faster-whisper | whisper.cpp | Whisper | Moonshine | Vosk |
|---|---|---|---|---|---|---|
| ami-close-mic 20 clips | 12.4% | 22.7% | 23.4% | 22.7% | 18.3% | 44.3% |
| ami-far-field 15 clips | 29.3% | 47.6% | 46.6% | 49.0% | 50.5% | 79.3% |
| earnings-calls 25 clips | 19.1% | 24.5% | 23.9% | 26.6% | 37.5% | 60.4% |
| fleurs 20 clips | 9.3% | 12.4% | 12.9% | 12.9% | 14.8% | 32.5% |
| librispeech-clean 30 clips | 1.3% | 3.6% | 3.3% | 3.6% | 3.6% | 10.3% |
| librispeech-other 20 clips | 6.4% | 10.9% | 11.8% | 11.5% | 10.6% | 23.8% |
Don't want to run it yourself? I also run Audio & Video Transcriber as a pay-per-use API on Apify (it uses faster-whisper). Disclosure: that tool is mine; the rankings above come only from the measurements.
How this is measured
- Each library runs a short recipe from its own docs (board/targets.py):
open the file, get all text. Installed with the extras the docs recommend for reading
(
pypdf[full]: AES decryption, Brotli), nothing else. One worker process per CPU; per file 60 s / 120 s / 180 s / 300 s at most (the any-document table allows longer: some tools load models), killed above the memory limit. - Fails where another reads: the library raised an error, hung or crashed on a file from which another library in the same table got text. Files that every library fails on (encrypted, truncated, fuzzed) are not counted against anyone.
- Empty where another reads: no error, but no text. This is often the recipe, not the library: for example
python-docx's paragraphs+tables recipe skips headers, text boxes and nested tables. Check the file list. - Any document → text: a file type the tool's docs don't list, or that the tool itself rejects as unsupported, counts as "not supported", not as a failure (lists in board/libs.json). Returning a file's raw RTF source counts as empty, for every tool. Each tool runs with default settings: no OCR switched on by hand.
- OCR: each image comes with the human-written ground-truth text published with its dataset. Words found is the share of its words that appear anywhere in the engine's output (reading order doesn't matter); an image that fails or comes back empty counts as 0. The ground truth of receipts and document pages covers only their annotated text, so a character error rate would punish engines for reading more than was annotated; it is in the raw results but not ranked. English models, CPU, default settings.
- Speech: each clip comes with its dataset's reference transcript. Both texts go through Whisper's English text normalizer (case, punctuation, "twenty" = "20"); word error rate is all substituted, missing and extra words over all reference words, the usual corpus-level WER. A clip that fails or comes back empty counts all its words as errors. The three Whisper implementations use the same base.en model; each other engine its documented default model. English, CPU, default decoding settings.
- Language detection: each text's language is known from Tatoeba. An answer counts as correct if the library's code matches after mapping variants to ISO 639-1 (zh-cn → zh, nb → no, iw → he). Close languages count as different: Croatian detected as Bosnian or Serbian, or Norwegian as Danish, is wrong. One call per text with default settings and all languages enabled.
- Text quality (order, tables, layout) is not ranked in the other tables; they are about not breaking. The raw results have every output's size and hash.
- Same files for every version: the test collections are pinned to one commit. GitHub-hosted runner, so times are comparable within one table, roughly.