Correct to Select: Choosing OCR Without Ground Truth

TL;DR for operators

A team has a new collection of forms, receipts, exam papers, or clinical notes and must choose an OCR engine before paying to transcribe the collection. A fixed benchmark winner is not enough: in the paper’s evaluation, no single OCR engine consistently leads across datasets and languages.

That creates a practical sequencing problem. The best engine depends on the documents, but directly measuring which engine is best normally requires the ground-truth labels the team is trying to avoid creating before selection.

DocOCR-Eval¹ proposes an unlabeled selection signal. It runs candidate OCR engines on the collection, lets multimodal language models diagnose and correct their outputs, and treats the amount of change required as evidence about how wrong the original OCR may have been. Engines requiring less correction are ranked more highly, and the paper checks whether that ordering agrees with rankings obtained from ground-truth evaluation.

For operators, the value is collection-specific OCR selection before large-scale annotation. The decision can affect extraction quality, annotation cost, downstream workflow reliability, latency, and infrastructure or API spending. The method is still a selection aid rather than an accuracy certificate: its reliability depends on the documents, OCR engines, and correcting models being evaluated, while the correction process itself adds compute, latency, and cost.

The best OCR engine changes with the documents

Suppose a team receives a new stream of forms, receipts, exam papers, or clinical notes. The simplest procurement decision is to choose the OCR engine with the strongest benchmark reputation and standardize on it.

The paper’s ground-truth results give a reason not to do that.

Across the evaluated collections, leadership changes with domain and language. Cloud Vision has the highest reported ANLS on EPHOIE at 0.44 among the conventional OCR engines, while PaddleOCR leads FUNSD at 0.42 and RXPAD at 0.66. On XFUND language subsets, EasyOCR leads Spanish, German, and Portuguese in the reported table, while Cloud Vision leads Japanese and Italian.

These results are main comparative evidence, not an ablation. Their purpose is to establish the selection problem itself: OCR quality is collection-dependent.

That changes the operational decision. A team running one homogeneous document stream may care less about average performance across public benchmarks than about which engine works best on its documents. Yet measuring that directly usually means producing reference transcriptions first, which can make tool selection expensive before the actual pipeline has even been chosen.

Correction becomes a measurement signal

DocOCR-Eval addresses that sequencing problem by using correction as measurement rather than only as post-processing.

For each OCR text block, the framework checks three kinds of possible failure: character noise, tokenization errors, and semantic inconsistency with the source document. The diagnostic state is represented as

Character or tokenization problems can be routed to a text-based MLLM corrector. Semantic inconsistency can trigger vision-based re-reading of the source document.

The key measurement idea comes after correction. Instead of assuming the corrected output is ground truth, the framework compares the original OCR text with its corrected version. Greater similarity means the correcting model found less to change; larger discrepancy implies that the original OCR output required more intervention. The paper uses ANLS for this original-versus-corrected comparison and aggregates it into an engine-level score.

This is an important boundary. Post-correction ANLS is not OCR accuracy. It is a proxy constructed from correction behavior.

The method therefore rests on a testable assumption: better OCR should generally require less correction. The paper’s contribution is not simply proposing that assumption, but checking whether rankings generated from it resemble rankings obtained when actual annotations are available.

The proxy tracks ground-truth rankings closely, but not perfectly

To validate the proxy, the authors compare the OCR ordering produced by correction scores with the ordering produced from annotated ground truth.

NDCG is used here as a ranking-agreement measure: a value near 1 means that the correction-based ordering places engines in nearly the same order as the annotation-based evaluation.

The reported stepwise results are strong within the tested settings:

The FUNSD result is particularly clear: under the full cascade, the correction-based ranking exactly matches the ground-truth ordering of the four candidate OCR engines.

But the table also prevents a stronger claim. Exact recovery is configuration-dependent. RXPAD, for example, reaches NDCG 1.0000 with tokenization correction alone and with character-plus-tokenization correction, while adding the full semantic cascade reduces agreement to 0.9548.

This table is best read as a sensitivity test of the measurement mechanism. It shows that the proxy can remain highly informative while the optimal way to construct that proxy changes with the document collection.

More correction is not automatically better evaluation

The stage-by-stage results reveal a second operational issue: there is no universal correction recipe.

On EPHOIE, semantic consistency checking by itself reproduces the ground-truth ordering exactly. On FUNSD, the full cascade performs best. On RXPAD, the strongest reported ordering comes from text-focused configurations rather than the full cascade.

That matters because every correction stage consumes resources and can introduce its own model behavior. A production team should therefore distinguish two decisions that are easy to conflate:

1. Which OCR engine should process this collection?
2. How much MLLM intervention is justified to make that selection?

The paper does not establish that every production document should pass through all correction stages. Its latency and cost analysis instead supports treating correction as configurable. A high-value extraction workflow may justify richer evaluation and re-reading; a large indexing pipeline with looser accuracy requirements may not.

The business implication is therefore not “attach an MLLM to every OCR request.” It is to spend additional inference selectively when the expected cost of choosing the wrong OCR engine is large enough.

Multiple correcting models can reduce dependence on one judge

A correction-based evaluator inherits another risk: the correcting MLLM has its own biases and variance. If one model systematically prefers particular wording or handles a document type poorly, its correction discrepancy can distort the OCR ranking.

DocOCR-Eval tests equal-weight aggregation across multiple correcting MLLMs:

The paper reports that averaging these scores improves top-tool selection relative to the single-MLLM setting and narrows model-specific score differences.

This is better interpreted as a robustness mechanism than as evidence that ensembles eliminate measurement error. SROIE still remains a disagreement in the reported multi-model results. Aggregation reduces dependence on one evaluator; it does not transform the proxy into ground truth.

For higher-stakes document pipelines, that distinction has governance value. A procurement or model-selection decision can be less exposed to the behavior of one correcting model without pretending that several automated judges constitute independent human validation.

Where this fits in an OCR deployment process

The paper directly shows that correction-derived rankings can approximate annotation-derived rankings across the evaluated public datasets and four candidate OCR engines. It also shows that engine leadership and correction strategy vary with the data.

Cognaptus inference: a practical deployment process could use DocOCR-Eval before large-scale annotation. A team could take an unlabeled sample from a new collection, run several OCR engines, apply a chosen correction configuration, and rank the candidates. The paper itself uses 50 sampled test images per test split; that supports the feasibility of sample-based evaluation, but it does not establish 50 documents as a universal sample-size rule.

The likely economic value comes from moving manual labeling later in the decision sequence. Instead of fully transcribing a collection merely to decide which OCR engine deserves further evaluation, the team can use the correction proxy to narrow the field first.

The remaining uncertainty is consequential. Validation covers a bounded set of public datasets, OCR engines, correcting MLLMs, and sampled images. Future engines, specialized document types, or distributions substantially different from FUNSD, SROIE, EPHOIE, RXPAD, and XFUND may produce different relationships between correction discrepancy and true accuracy.

A selection layer, not an accuracy certificate

“Annotation-free” needs a precise reading here. The study itself uses annotated ground truth extensively: first to benchmark OCR performance, then to verify whether the proposed unlabeled ranking agrees with the ranking those annotations produce. What operates without labels is the intended selection procedure, not the experimental validation of that procedure.

That is also the right boundary for deployment.

DocOCR-Eval provides evidence that MLLM correction can serve as a useful proxy when a team needs to choose among OCR engines before ground-truth transcription exists. It does not show that correction scores can certify absolute OCR accuracy, remove the need for validation in consequential workflows, or guarantee the same ranking on unseen document collections.

Its more defensible contribution is narrower and operationally meaningful: when OCR quality depends on the collection, teams can measure local fit before committing to the expense of full labeling—and they can make the strength of that measurement process proportional to the cost of getting the selection wrong.

Cognaptus: Automate the Present, Incubate the Future.