Multilingual OCR: routing documents to different engines by difficulty

"What is the best OCR for multilingual documents?" has an unsatisfying answer: it depends on the page. A clean, typed English invoice and a skewed phone photo of a handwritten Arabic customs form are different problems. An engine that is excellent and cheap on the first is useless on the second, and an engine capable of the second is slow, expensive, and unnecessarily risky on the first.

So the better question is how to build a pipeline that picks the engine per page. The pattern we use:

1. Detect language and script per page, and per region where pages are mixed.
2. Analyze layout: text blocks, tables, handwriting, stamps.
3. Score the page's difficulty from cheap signals.
4. Route to the cheapest tier likely to succeed.
5. Check the result, and escalate on low confidence.
6. Verify anything a generative model produced, and queue what remains for a person.

This article covers each step, the script-specific traps, and how to evaluate the whole thing.

Why one engine is not enough

Three families of engines are in play, with genuinely different profiles:

Two things follow. If most of your volume is easy pages — and it usually is — sending everything to the top tier multiplies cost and latency for no accuracy gain. And if you send everything to the bottom tier, the hard minority of pages fails silently. Routing gets you the accuracy of the top tier where it matters at close to the cost of the bottom tier.

Verify language coverage per vendor and per feature before committing. A service may read printed text in a hundred languages but support handwriting or form extraction in only a handful.

Step 1: detect language and script — per page, then per region

Document-level language metadata is unreliable. A shipping file is an English bill of lading, a Chinese commercial invoice, and an Arabic certificate of origin in one PDF. Detect per page at minimum.

Script before language. Script (Latin, Arabic, Cyrillic, Han, Devanagari) is what determines which recognition model is needed, and it is detectable from the image. Language within a script (Spanish or Portuguese) mostly matters downstream, and is easy to determine from recognized text.

Practical methods:

• Image-based script detection. Tesseract's orientation and script detection mode returns the dominant script and page rotation in one cheap pass:

tesseract page.png stdout --psm 0

• Text-based identification after a first pass. Run a fast multilingual recognizer, then classify the text. Counting characters by Unicode script block is trivial and robust. For language within a script, fastText's lid.176 language identification model is small and fast.
• Born-digital shortcut. If the PDF has a real text layer, check it first — many pages need no OCR at all. Validate the layer, though: some PDFs carry a garbage or visually ordered text layer from a previous bad OCR run.

Then go per region. Mixed pages are common: bilingual forms with parallel columns, Latin part numbers inside Chinese descriptions, an English letterhead over Arabic body text. Page-level detection picks the majority script and the recognizer mangles the minority. Detect script per text block after layout analysis, and recognize each block with the right model. With Tesseract you can also load several languages together (-l ara+eng), which works but tends to lower accuracy compared with a correctly chosen single model, so prefer per-region selection when layout analysis is available.

Step 2: layout analysis

Recognition accuracy is wasted if reading order is wrong. A two-column page read straight across produces perfectly spelled nonsense.

A layout stage segments the page into typed regions — paragraphs, headings, tables, figures, handwriting, stamps, signatures — and establishes reading order. Open options include PaddleOCR's PP-Structure, Surya, and the DocLayout-YOLO family of detectors; the cloud document services return layout as part of their output.

Layout output feeds routing directly:

• Tables go to a table-aware extractor. Generic line-by-line OCR destroys cell structure.
• Handwritten regions are escalated on their own, without dragging the printed majority of the page up a tier.
• Stamps and seals overlapping text are a classic cause of localized failure; flag the overlapped block for escalation.
• Region-level cropping lets you send a single difficult block to an expensive engine rather than the whole page.

Step 3: score difficulty from cheap signals

Everything here costs milliseconds per page, which is what makes routing economical.

The blur check, for instance, is two lines:

import cv2

gray = cv2.imread("page.png", cv2.IMREAD_GRAYSCALE)
sharpness = cv2.Laplacian(gray, cv2.CV_64F).var()  # calibrate the cutoff on your own scans

Do not copy thresholds from a blog post, including this one. Calibrate them on your documents: plot each signal against measured error rate on a labeled sample and set cutoffs where error starts climbing.

Fix what you can before escalating. Deskewing, rotation correction, denoising, contrast normalization, and upscaling small text are cheap and frequently turn a tier-2 page back into a tier-1 page. Preprocess, re-score, then route.

Step 4: route, check, escalate

The router is deliberately simple: rules over the signals above, with escalation on low confidence. Rules are easy to audit and easy to adjust when you add a language.

def route_page(page):
    if page.has_valid_text_layer:
        return extract_text_layer(page)

    page = preprocess(page)            # deskew, denoise, normalize, upscale
    regions = analyze_layout(page)     # typed blocks in reading order
    results = []

    for region in regions:
        script = detect_script(region)
        tier = initial_tier(region, script)
        result = None

        while tier <= 3:
            result = ENGINES[tier].recognize(region, script)
            if tier == 3:
                result = verify_generative(result, region)   # see next section
            if accept(result, region, tier):
                break
            tier += 1

        if result is None or not accept(result, region, tier=3):
            result = send_to_review_queue(region, result)

        results.append(result)

    return assemble_in_reading_order(results)


def initial_tier(region, script):
    if region.kind == "handwriting":
        return 2 if script in HANDWRITING_SUPPORTED_TIER2 else 3
    if script not in TIER1_STRONG_SCRIPTS:
        return 2
    if region.sharpness < SHARPNESS_MIN or region.text_height_px < TEXT_HEIGHT_MIN:
        return 2
    return 1


def accept(result, region, tier):
    if result.mean_confidence < CONF_MEAN_MIN[tier]:
        return False
    if result.p10_confidence < CONF_P10_MIN[tier]:      # catches a few badly wrong words
        return False
    if not passes_sanity_checks(result, region):        # script matches, not empty, plausible length
        return False
    return True

Notes on the acceptance check:

• Use a low percentile, not just the mean. A page with 95% confident words and a few garbage ones has a fine average and a wrong invoice number.
• Sanity checks are cheap and catch a lot: output script matches detected script; character count is plausible for the region's area; a dictionary hit rate for languages where that is meaningful; field-level validators such as date formats, checksums, and totals that add up.
• Weight by importance. Low confidence in a footer matters less than in an amount field. If you know which regions feed downstream extraction, hold them to a tighter threshold.
• A data-residency rule may override everything. If certain documents cannot leave your environment, the cloud tier is not eligible for them, and the escalation path goes from local OCR to a self-hosted VLM.

Step 5: verify the generative tier

This is the step most pipelines skip, and the one that matters most.

The model is doing what it was trained to do: produce likely text. For reading damaged documents, "likely" and "what is on the page" diverge exactly where it matters. So the top tier's output is treated as a claim to be checked, not a result:

• Verbatim grounding. Require the model to transcribe, not summarize or normalize, and to mark unreadable spans explicitly with a token such as [illegible] rather than guess. Then check that transcribed spans can be located on the page — aligning against even a poor tier-1 pass with word coordinates catches text that appears nowhere in the image. This is the principle behind our own verbatim extraction work.
• Cross-engine agreement. Compare the VLM transcription with the tier-1 or tier-2 output at the character level. Where they agree, confidence is high. Where they disagree, that span — not the whole page — goes to review. Two engines with different failure modes are much less likely to agree on the same wrong reading.
• Confidence proxies. If you serve the model yourself, token log-probabilities over the transcription are a usable signal; low-probability spans cluster on the uncertain words. Running the model twice and diffing the outputs is a cruder version of the same idea.
• Field validators. Anything with structure — dates, totals, identifiers with check digits, codes from a known list — gets validated against that structure.
• Human review queue. What survives none of the above goes to a person, shown as the cropped region beside the candidate text. Record corrections; they become evaluation data.

If the routing is working, the review queue is a small fraction of volume, and it is the right fraction.

Script-specific traps

Right-to-left and mixed-direction text. Arabic and Hebrew lines contain left-to-right runs: numbers, Latin product codes, email addresses. Output should be in logical order (the order characters are read and typed), leaving display to the Unicode Bidirectional Algorithm. Engines and PDF text layers that emit visual order produce reversed numbers or reversed words that look fine until someone searches for them. Test specifically with lines that mix directions. Also decide how to handle Arabic-Indic digits versus Western digits, and normalize consistently.

Contextual shaping and presentation forms. Arabic letters change shape by position. Some engines emit legacy presentation-form code points instead of the base letters, which breaks search and matching. Unicode NFKC normalization folds these back; apply it deliberately, knowing it also folds other compatibility characters such as full-width Latin letters.

CJK. There are thousands of character classes, many differing by a stroke, so resolution matters more than for alphabetic scripts. Vertical text and mixed horizontal and vertical layouts need layout support. Distinguish Simplified from Traditional Chinese and Japanese from both — they share many characters but need different models. There are no spaces between words, so measure character error rate, not word error rate.

Diacritics. Vietnamese stacked tone marks, Turkish dotted and dotless i, Polish and Czech accents, and Arabic vowel marks are small features that vanish first at low resolution. A dropped diacritic is a different word or a different name. Normalize to NFC before comparing, and do not strip accents "for matching" unless you keep the original.

Indic and other complex scripts. Conjuncts and vowel signs reorder visually relative to logical order; conventional-engine quality varies a great deal by language. Expect these to start at tier 2 until your own measurements say otherwise.

Evaluation: per language, per tier

One aggregate accuracy number hides every problem this architecture exists to solve.

• Build a labeled set stratified by language, document type, and quality band. Include the ugly pages in proportion to how often they occur, plus extra, because they are where the decisions get made.
• Measure character error rate (CER) everywhere, and word error rate where words are well defined. Both are edit distance divided by reference length; the jiwer library computes them. Normalize Unicode and whitespace identically on both sides first.
• Measure field-level accuracy on what downstream systems consume. A 1% CER concentrated in amounts is worse than 3% spread across boilerplate.
• Report a matrix, not a number: CER by language by tier. This tells you where tier 1 is good enough (route more there) and where it is not (start those at tier 2).
• Evaluate the router itself. Of the pages accepted at tier 1, how many were actually wrong — the false accepts? Of those escalated, how many would have been fine? These two rates are what you tune thresholds against.
• Track the invented-text rate for the generative tier separately: spans in the output with no support on the page. It should be driven toward zero by the verification step, and it is the number an auditor will care about.

Thinking about cost per page

We will not quote prices; they change, and yours depend on volume and hosting. The structure is what matters:

cost per page = sum over tiers of (share of pages reaching tier x tier cost per page)
              + share reaching human review x review cost per page

Three observations. Human review is by far the most expensive term per page, so verification that shrinks the review queue pays for itself quickly. Tier-1 cost is effectively compute you already own, so every page that preprocessing rescues back to tier 1 is nearly free. And escalated pages pay for every tier they pass through, which is why the initial tier choice should skip tier 1 when the signals clearly say it will fail.

Measure the share of pages reaching each tier on a real sample before estimating anything. The distribution is usually far more favorable than teams fear — and the hard tail is usually harder.

Summary

Treat multilingual OCR as a routing problem rather than an engine selection problem. Detect script per region, score difficulty with cheap signals, start each page at the lowest tier likely to succeed, escalate on measured confidence, and never accept generative output without checking it against the page. Evaluate per language and per tier so you know which routes are earning their cost. More on how we approach this kind of pipeline is on our document intelligence page.