Introducing retab-consensus
The reconciliation layer for N-extractor pipelines.
Today, we're releasing retab-consensus in the API and in the Retab dashboard. It's our most capable reconciliation engine for high-stakes document extraction. We're also releasing retab-consensus-large in the API, for teams who want maximum alignment quality on the hardest nested schemas.
retab-consensus brings together the best of our recent advances in extraction accuracy, model agreement, and review tooling into a single product. It builds on the list-alignment engine we shipped earlier this year, while substantially improving how the system handles disagreement across long, deeply structured documents.
The result is an extraction pipeline that returns a single trusted answer for every document, with confidence on every field - delivering a result your team can act on without manual reconciliation.
In the Retab dashboard, retab-consensus surfaces the documents that need attention, highlights the handful of fields where the model is uncertain, and lets reviewers compare every candidate answer side by side.
In the API, retab-consensus turns multiple independent extractions of the same document into a single reconciled answer. It accepts up to 16 extractions per document and returns a merged result with a confidence score on every field.
AI-native workflows
Before the mechanics, a vocabulary clarification - because "agent" and "workflow" get used interchangeably, and the rest of this post depends on the distinction.
An agent is an LLM wrapped in a harness: tools it can call, memory it can write to, a sandboxed execution environment, and a control loop that lets it decide what to do next. Agents are general-purpose by construction - you hand them an objective and they navigate. That generality is also the cost: their runtime is open-ended, their failure modes are hard to enumerate, and their outputs drift between runs.
An AI-native workflow is a different shape: a graph with three kinds of nodes. Deterministic code is the glue: routing, persistence, validation, retries; same input, same output, every run. Narrow AI-enabled operations are single, well-scoped LLM calls - extract these fields, classify this document, normalise this address; not deterministic, but small enough that you can measure, score, and bound the variance.
Narrow here refers to scope, not model capability: Retab calls frontier generalist models for these operations - the same class of model an agent would call - because unseen-domain document workloads transfer from a frontier model more reliably than from a task-specific narrower one.
Retab lives in the middle row. Its extraction, classification, and parsing primitives are narrow non-deterministic operations stitched together by deterministic orchestration, with agents reserved for the parts of the graph that actually need them. That positioning is what makes consensus not just useful but structurally necessary: when the non-deterministic step is narrow, you can afford to run it K times in parallel, and you can reconcile field-by-field because the output shape is fixed.
Why consensus
Document-AI in production isn't a single model call. Teams build pipelines: an extractor that returns typed JSON, and an agent layer that reviews, enriches, and routes the result. Retab is the extractor. The agent layer can only act on a value once it knows the extractor's confidence in it. Consensus is how Retab returns that confidence per field.
the hybrid workflow
LLMs are strong on open-ended reasoning and weak on mechanical extraction. The best production pipelines split the work accordingly: Retab handles the structured layer - typed fields, schema-conformant output, repeatable across millions of documents - and agent loops handle the interpretive layer: domain judgement, cross-referencing, edge-case reasoning, and narrative summaries.
A growing number of document-AI teams now treat Retab as the backbone and wrap it in agent workflows for review, enrichment, and downstream routing. The extractor never hallucinates a new field shape; the agent never has to guess at a schema. Each layer does what it's good at.
where hybrid breaks
The wall is scale. A typical loan file runs ~100 pages; a production schema for that packet has ~40 nested fields; running it through a 4-extractor ensemble yields on the order of 16,000 values per document. At production throughput, exhaustive human audit stops being possible somewhere between the first document and the hundredth.
Two failure modes follow. Teams that pipe every extracted field through an agent for review blow up their token budget and latency: the audit costs more than the extraction it's auditing. Teams that skip review, or random-sample it, accept silent errors and miss exactly the fields where the extractor was uncertain. Without a per-field signal of uncertainty, both paths converge on the same outcome: errors ship.
agent in the loop is the new human in the loop
Document-AI pipelines have long wrapped extractors in a review-based: reviewers read the output, correct errors, and approve before downstream systems consume the JSON. That pattern was tenable when a document produced dozens of fields. At the scale described above, the loop does not close - the reviewer caps out long before the pipeline does.
The shape that does scale is an agent-in-the-loop: a second LLM process that reads the extractor's output, flags or corrects errors, and escalates to a human only on the fields it cannot resolve itself. Agents do not tire, do not skim, and can be run in parallel across every document in the queue.
The missing piece is a per-field signal of uncertainty. With it, the agent reviews the small fraction of fields where extractors disagreed and ignores the rest; without it, every value looks equally suspect and the audit cost runs away. Consensus is that signal.
Experiments
Traditional document-AI evaluation compares extracted values to a manually annotated ground truth. The dataset is expensive to build, the annotations drift between reviewers, and the worst failure mode is silent: the annotator is wrong, the model is right, and the eval counts it as a regression. At the schema sizes described above, building and maintaining ground truth is its own full-time pipeline.
Consensus produces a different signal for free. Run the same extraction K times under the same configuration - Retab defaults to K=5, with 3 and 7 available - and the distribution of answers is already there. For every leaf field, plurality voting picks the dominant answer and the likelihood is the fraction of runs that voted for it: five out of five is 1.0, three out of five is 0.6. Tight agreement means the field is settled. Wide disagreement means the field is ambiguous, and the ambiguity is a property of the document, not of any one run. No annotation required.
why experiments beat evals
Ground-truth evals have three well-known failure modes. They are slow to stand up: a schema of 40 nested fields across 500 documents is a quarter of a quality engineer's year before the first number comes back. They are noisy: two annotators on the same document disagree on the fields a metric most needs to be stable on. And they are adversarial to the model: when the annotator and the extractor disagree, the convention is to call the extractor wrong, even in the cases where the document supports the extractor's reading.
Experiments invert the setup. Instead of comparing N extractions against one fixed label, we compare N extractions against each other and report, per field, how concentrated their agreement is. An experiment in Retab points at a single block - Extract, Split, or Classify - picks a set of test documents from run history or uploads, and replays that block K times per document. The snapshot it returns gives you five views: an overall score, a per-run breakdown that surfaces the weakest documents, a per-field breakdown that surfaces the weakest paths in the schema, a field-choices view that shows the actual voter outputs behind any single cell, and a heatmap across runs and fields. The signal is denser than pass/fail labels because every field carries a continuous likelihood instead of a binary verdict.
A single document's consensus spread is noisy on its own: maybe one run misread a smudged date, maybe another tripped on an unusual layout. Aggregated across a test set of dozens or hundreds of documents, those peculiarities cancel out. What survives the averaging is structural: a field the schema cannot disambiguate, a prompt that leaves room for interpretation, an extractor that systematically underperforms on a specific region.
what this unlocks
Once likelihood is the objective, the agent-in-the-loop stops being just a reviewer and starts being an optimizer. An agent can propose schema refinements, prompt edits, or extractor mixture changes, re-run the experiment, and read off the delta against the previous snapshot. Each Retab experiment carries its signed delta versus the last run so the agent sees movement, not just absolute scores.
A staleness check invalidates the comparison when the block's config or test set has changed, so the number the agent reads is always comparing the change it just made against a fixed baseline. The learning loop that used to require a labelled corpus now runs on unlabelled documents.
Consensus is the runtime signal for reviewers, the evaluation signal for engineers, and the optimization signal for agents. One mechanism, three uses - and none of them require a hand-labelled dataset to get started.