Ten documents and ten thousand are not the same problem. With ten, the cost of a mistake is the document: somebody notices and fixes it. With ten thousand, the cost of a mistake is the repetition — the natural response to a load that stopped halfway is to run it again, and that is where second copies of the same contract and second calls to the destination system come from. Capturing documents at volume is not about reading faster: it is about making repetition harmless, and about still proving months later what was delivered.
What changes between ten and ten thousand
With ten documents the process is one person's attention: excellent, and incapable of scaling. Suppose a load of ten thousand files in which forty fail: the difference between an afternoon and a week is knowing which forty they were, repeating only those, and repeating the others by accident being harmless. A network scanner digitises well and email is unbeatable at receiving; what neither answers is “has this file been here before?”, the question volume asks ten thousand times.
Recognising is not enough: the document has to become structure
Extracting text from an image is the first step and the least interesting. What decides the outcome is classifying the file against the organisation's processes, and that decision has two thresholds, not one. It is not enough for the best candidate's score to be high: there has to be enough distance to the second. Two high, tied scores are not a decision: they are a tie, and a tie goes to a person. Past both thresholds, the dossier is created from the process with its expected documents listed and the file in the right place; otherwise the row goes to the review queue with the reason recorded, as the document intake chapter shows; the intake channels covers where files arrive.
Idempotency starts here: each file is identified by its content — a SHA-256 fingerprint — plus the channel and the source, up to 100 MB per file; submitting it again returns the document that already exists. A multi-page scan is cut into several documents before filing — by separator sheet, by a barcode on the first page of each case, or by a fixed number of pages — with the cuts previewed and corrected by hand first.
Delivery to the external system
A governed process does not end in a folder: it ends by calling the system owning the operation — the ERP that creates the supplier, the portal that registers the policy. Today it delivers JSON: a payload template defined by the customer, in which every value carries its evidence — where it came from: a dossier field, the source email's metadata, the extracted text, the AI's reading. Paths marked sensitive are redacted before it leaves.
The delivery has three guarantees:
| Guarantee | What it does |
|---|---|
| Idempotency key | In the header of every execution; a destination that honours it recognises the repeat |
| Retries with backoff | Three attempts by default, thirty seconds of backoff, thirty seconds request timeout |
| Callback window | Twenty-four hours by default for the destination to answer asynchronously |
The dossier shows the wait: there is a state of its own for “the external operation is under way — sending, waiting for the outcome, or retrying”, leaving for completed or failed, with a code. The destination is validated before every attempt and after every redirect, and private-network addresses are refused by default — see security. The integrations page frames the design, and the process design shows where it fits.
Carrying the file is an option, not a rule
Not every destination wants the same thing: an ERP creating a supplier wants the tax number and the account, not fifty pages; a digital archive or a signature system needs the original. So it is a per-operation decision.
Today it delivers the structure: values, metadata and up to 8000 characters of extracted text per document — a deliberate ceiling, because a whole fifty-page contract blows up the model's context for nothing. Delivering the document itself alongside the structure, in the body of the call or through a temporary link the destination downloads it from, is on the roadmap.
Exceptions at volume: a work queue, not an error report
An error report says what went wrong; a work queue says what to do next and lets you do it there: each row carries the reason, the detected type, the score and the actions — resolve, discard, retry, or create the dossier by hand.
Underneath is the rule that makes repeating safe: nothing is discarded. Files that come in are never deleted, and a document that failed is repeated from the start. The pace today is that of background processing, with per-pass ceilings; explicit throughput and back-pressure control — how many documents run in parallel, queues with a maximum depth — is on the roadmap.
The proof, months later
The question that turns up six months after a migration is never “did the load work”, but “was this invoice delivered, when, and what is it called on the other side?”.
Every execution keeps the answer: the payload as it was sent, already redacted, the evidence for each value, the URL and the method, the response code, the attempt count and the reference the destination returned. That last one closes the loop: it is the name the case has on the other side, and without it “we delivered it” is half a conversation. The dossier keeps, next to it, the documents that produced the values.
FAQ
Does repeating a load create duplicate documents?
No. Each file is identified by its content — a SHA-256 fingerprint — plus the channel and the source, and submitting it again returns the document that already exists.
How is the destination system kept from receiving the same case twice?
Every execution carries an idempotency key in the header, which a destination that honours it uses to recognise the repeat. On this side, the three retries with backoff and the twenty-four-hour window stop a slow answer being read as a failure.
What happens to documents that fail in a load?
They go to the review queue with the reason, the detected type and the score. Nothing is deleted, so they are repeated from the start once the cause is fixed: a tuned threshold, a missing process.
Can I send ten thousand documents in a single request?
Not today: each submission carries one file and each dossier is created one at a time; volume arrives through the continuous channels — email, the scanner folder, the API. Batch loading with tracking and a count of exceptions is on the roadmap.