Which Python tool turns PDFs into the most faithful Markdown?
Docling, pymupdf4llm, Marker, Kreuzberg, markitdown: each one tested on the same 190 PDF pages with checks written by AI2's olmOCR-bench: multi-column articles, tiny print, old scans, tables, pages with headers and footers. New PyPI releases are tested again by themselves. Updated 06 November 2026.
190 files from olmOCR-bench (AI2), 190 pages;
190 of them give text to at least one library.
| Library | 🔴 Fails where another reads | 🟡 Empty where another reads |
⏱ Hangs or crashes | ✅ Checks passed | Median ms/file | Total seconds |
|---|---|---|---|---|---|---|
| 1 Docling 2.148.0 | 0 | 0 | 0 | 64.1% | 5697 | 1,564.0 |
| 2 pymupdf4llm 2.1 | 0 | 55 | 0 | 38.6% | 407 | 116.5 |
| 3 Marker 2.1.0 Note: Marker 2 runs its OCR model (Surya 2.5, 1.2 GB) through llama.cpp's llama-server, installed separately as its docs say: official CPU build b12177. | 93 | 0 | 0 | 36.6% | 636 | 228.9 |
| 4 Kreuzberg 4.10.4 | 0 | 54 | 0 | 28.8% | 20 | 9.1 |
| 5 markitdown 0.1.9 | 1 | 54 | 0 | 27.0% | 381 | 99.0 |
By kind of page: checks passed
| Pages | Docling | pymupdf4llm | Marker | Kreuzberg | markitdown |
|---|---|---|---|---|---|
| headers & footers 40 pages | 89.0% | 40.4% | 72.5% | 61.5% | 38.5% |
| multi-column 40 pages | 73.7% | 61.2% | 47.4% | 56.6% | 47.4% |
| old scans 40 pages | 20.9% | 0.0% | 0.0% | 0.0% | 0.0% |
| tables 40 pages | 68.8% | 64.1% | 32.8% | 0.5% | 25.0% |
| tiny text 30 pages | 79.2% | 34.5% | 46.9% | 43.4% | 32.7% |
How this is measured
- Each library runs a short recipe from its own docs (board/targets.py):
open the file, get all text. Installed with the extras the docs recommend for reading
(
pypdf[full]: AES decryption, Brotli), nothing else. One worker process per CPU; per file 60 s / 120 s / 180 s / 300 s / 600 s at most (the any-document table allows longer: some tools load models), killed above the memory limit. - Fails where another reads: the library raised an error, hung or crashed on a file from which another library in the same table got text. Files that every library fails on (encrypted, truncated, fuzzed) are not counted against anyone.
- Empty where another reads: no error, but no text. This is often the recipe, not the library: for example
python-docx's paragraphs+tables recipe skips headers, text boxes and nested tables. Check the file list. - Any document → text: a file type the tool's docs don't list, or that the tool itself rejects as unsupported, counts as "not supported", not as a failure (lists in board/libs.json). Returning a file's raw RTF source counts as empty, for every tool. Each tool runs with default settings: no OCR switched on by hand.
- OCR: each image comes with the human-written ground-truth text published with its dataset. Words found is the share of its words that appear anywhere in the engine's output (reading order doesn't matter); an image that fails or comes back empty counts as 0. The ground truth of receipts and document pages covers only their annotated text, so a character error rate would punish engines for reading more than was annotated; it is in the raw results but not ranked. English models, CPU, default settings.
- Speech: each clip comes with its dataset's reference transcript. Both texts go through Whisper's English text normalizer (case, punctuation, "twenty" = "20"); word error rate is all substituted, missing and extra words over all reference words, the usual corpus-level WER. A clip that fails or comes back empty counts all its words as errors. The three Whisper implementations use the same base.en model; each other engine its documented default model. English, CPU, default decoding settings.
- Language detection: each text's language is known from Tatoeba. An answer counts as correct if the library's code matches after mapping variants to ISO 639-1 (zh-cn → zh, nb → no, iw → he). Close languages count as different: Croatian detected as Bosnian or Serbian, or Norwegian as Danish, is wrong. One call per text with default settings and all languages enabled.
- PDF → Markdown: pages from AI2's olmOCR-bench (ODC-BY), scored with its own checks (vendored, Apache-2.0): a sentence must be present, a page header or footer absent, two passages in the right order, a table cell next to the right neighbour. Checks passed is all passed checks over all checks; a page that fails or comes back empty passes none. Math pages are left out (their checks need a browser to render formulas). Default settings: whatever OCR a tool does on its own, it does.
- Text quality (order, tables, layout) is not ranked in the other tables; they are about not breaking. The raw results have every output's size and hash.
- Same files for every version: the test collections are pinned to one commit. GitHub-hosted runner, so times are comparable within one table, roughly.