IDP Software Evaluation Checklist: How to Test Vendors Beyond the Demo

Every intelligent document processing (IDP) demo goes well. The vendor uploads a clean invoice, the fields light up, the totals match, and the room nods. The trouble starts in month two, when your own scanned forms, odd supplier layouts and 40-page contracts hit the system and a queue of exceptions lands on someone's desk.

That gap is why buyers keep switching. In a 2025 survey of 600 enterprises by Deep Analysis and AIIM, 66% of organizations planning a new IDP project said it would replace an IDP system they already had.

Choosing IDP software well comes down to testing what the demo hides. Below are 12 checks, grouped into four questions, and a scoring table you can use to compare vendors side by side.

Build your own test set first

A vendor demo is built to succeed. Before you watch another one, put together the set of documents every vendor will be judged on.

Pull a sample from every document class you process and every channel it arrives through: email attachments, portal uploads, scanned paper and faxes. Paper still matters here. The same Deep Analysis and AIIM survey found that 61% of IDP processes still include paper documents, and 48% of respondents expect paper volumes to grow. Include the ugly ones: skewed scans, handwriting in the margins, multi-page tables.

Then label the ground truth, the correct value for every field you actually use downstream. Split the set in two. Vendors may tune on the first part. The second part is your blind holdout set, and they never see it until scoring day. Give every vendor the same set, so their numbers are comparable.

Treat the pilot as a loop rather than a single pass: the vendor tunes, you score on the blind set, they adjust, you score again. Buyers in the survey gave high ratings to proof-of-concept testing with their own documents when choosing a product. If you have already run a similar exercise for web data vendors, our data extraction evaluation checklist follows the same logic.

Score each of the 12 items below 0 (fails), 1 (partial) or 2 (passes).

How accurate is it on your documents?

Accuracy is where every vendor claims to win, and where the numbers are least comparable. These four checks turn a marketing figure into something you can measure.

1. Accuracy on your documents, not the vendor's

Attribute	Detail
Ask the vendor	What accuracy do you expect on our document classes, and at what level is it measured?
Test it	Run the blind set. Measure accuracy per field and per document class.
A pass looks like	Results reported by class and by field, on your blind set, close to what the vendor predicted.
Demo shortcut it catches	A headline accuracy figure measured on clean, standard documents.

Vendor accuracy numbers usually come from the vendor's own benchmark, often dominated by clean invoices. Your document mix decides the number that matters.

2. Whole-document accuracy, not per field alone

Attribute	Detail
Ask the vendor	What share of our documents will pass with every required field correct and no human touch?
Test it	Count blind-set documents where all required fields are right. Compare with field-level accuracy.
A pass looks like	The vendor quotes straight-through processing on your documents, and your count confirms it.
Demo shortcut it catches	"99% accurate" meaning 99% of fields, not 99% of documents.

Field accuracy compounds. If a document has 15 fields and each is extracted correctly 99% of the time, the chance that all 15 are right is about 86%. At 30 fields it drops to about 74%. (This is illustrative arithmetic and assumes errors are independent.) The number your operations team feels is straight-through processing (STP), the share of documents that reach the target system with no human touch.

A short script is enough to score this yourself on the blind set:

Python: score field accuracy, whole-document accuracy and review rate on your blind set

import json

def score(ground_truth, predictions, threshold=0.9):
    """Field accuracy, whole-document accuracy and review rate for one vendor."""
    fields_total = fields_correct = docs_all_correct = docs_to_review = 0
    for doc_id, truth in ground_truth.items():
        pred = predictions.get(doc_id, {})
        doc_ok, needs_review = True, False
        for field, value in truth.items():
            got = pred.get(field, {})
            fields_total += 1
            if got.get("value") == value:
                fields_correct += 1
            else:
                doc_ok = False
            if got.get("confidence", 0) < threshold:
                needs_review = True
        docs_all_correct += doc_ok
        docs_to_review += needs_review
    n = len(ground_truth)
    return {
        "field_accuracy": fields_correct / fields_total,
        "document_accuracy": docs_all_correct / n,
        "review_rate": docs_to_review / n,
    }

truth = json.load(open("blind_set_truth.json"))
vendor = json.load(open("vendor_a_output.json"))
print(score(truth, vendor))

3. Confidence scores you can set a threshold on

Attribute	Detail
Ask the vendor	What does a confidence score of 0.9 mean on our documents, and can we set the review threshold per field?
Test it	At the vendor's default threshold, count how many documents go to review and how many wrong values slip through marked high-confidence.
A pass looks like	High-confidence values are almost always right, and moving the threshold shows a clear trade-off between review volume and errors.
Demo shortcut it catches	Confidence scores shown on screen that do not track real errors.

The confidence score decides what a person checks and what flows straight into your systems. A wrong value marked as certain is the most expensive error an IDP system can make. Nobody looks at it, so its blast radius is every report and payment that uses it.

4. The hard documents: tables, handwriting, scans and long files

Attribute	Detail
Ask the vendor	How do you handle multi-page tables, handwriting, low-resolution scans and files over 100 pages?
Test it	Put your worst examples of each in the blind set and score them separately.
A pass looks like	Table rows stay aligned across pages, handwritten fields are read or flagged, and values come back with page references.
Demo shortcut it catches	Single-page, typed, well-lit samples.

Tables are where extraction quietly breaks: a merged cell shifts every value in a row. Our guide to table extraction in IDP covers why. As a reference point for what to ask about, Forage AI's in-house model detects tables with 95% accuracy across table types and processes documents over 2,000 pages long.

How the system runs after go-live

Accuracy tells you how the system performs on the day you test it. These three checks tell you what it costs to run every day after that.

5. The exception queue

Attribute	Detail
Ask the vendor	Who reviews low-confidence documents, what does the reviewer see, and do corrections improve the model?
Test it	Have your own staff work the queue for one day of pilot volume, and time them.
A pass looks like	The reviewer sees the source image beside each value, and you can estimate review hours per 1,000 documents.
Demo shortcut it catches	The vendor's own team clearing exceptions during the pilot.

In our experience, the review queue, not the model, decides whether an IDP rollout holds. If your team cannot clear it at production volume, automation stalls. See how review fits into document workflow automation and why human-in-the-loop checks are what keep accuracy high.

6. Model updates and generated values

Attribute	Detail
Ask the vendor	How are model updates released? Can we pin a version? Will you re-run our blind set before we switch?
Test it	Ask the system for a field that is not on the page. It should return empty, not a plausible value.
A pass looks like	Version pinning, a regression report on your blind set for every update, and empty fields when data is missing.
Demo shortcut it catches	Output that looks well-formed but was invented.

As of 2026, most IDP products use large language models somewhere in the pipeline. That brings a specific risk, which NIST's Generative AI Profile (July 2024) calls confabulation: "the production of confidently stated but erroneous or false content." In document extraction, that looks like a correctly formatted invoice number that does not exist on the invoice.

7. Upkeep when layouts change

Attribute	Detail
Ask the vendor	What happens when a new supplier or form layout appears? Who retrains the model, how long does it take and what does it cost?
Test it	Add two unseen layouts late in the pilot and time how long they take to handle.
A pass looks like	New layouts are handled by configuration or a short tuning cycle that is included in the contract.
Demo shortcut it catches	Templates built for the demo documents.

In practice, this is the check buyers skip and later regret. When the survey asked why organizations were replacing their IDP system, the top answer was that it was too limited for current needs, followed closely by needing too much external help to maintain and operate.

Will it fit your systems and data rules?

A system that reads documents well still fails if the data never reaches the right place, or if security blocks the rollout.

8. Integration with your downstream systems

Attribute	Detail
Ask the vendor	Can we define the output schema? Do you support webhooks, retries and connectors for our ERP or workflow tools?
Test it	Push pilot output into a staging copy of your target system, end to end.
A pass looks like	No re-keying, and you own the field mapping.
Demo shortcut it catches	Results that only ever appear in the vendor's interface.

Integration ranked as the second-biggest concern about implementing IDP in the Deep Analysis and AIIM survey, and difficulty integrating was the third most common reason for replacing a system. Our guide to integrating IDP with enterprise systems covers the patterns to ask about.

9. Security and the use of your documents

Attribute	Detail
Ask the vendor	Where are documents and outputs stored? How long are they kept? Are our documents used to train shared models? Can you run in a private cloud or on-premises?
Test it	Request current third-party audit reports and the data-processing terms, and have your security team review them.
A pass looks like	Clear answers written into the contract, including retention, deletion and training use.
Demo shortcut it catches	A security slide with no contract terms behind it.

Data security and privacy was the number one concern about implementing IDP in the Deep Analysis and AIIM survey of 600 enterprises (2025). The survey authors tied the rise of that concern partly to generative AI makers training models on proprietary data. The training-use question belongs in the contract, not the sales call.

What does it cost to run?

Price sheets for IDP software are hard to compare, because vendors charge by different units. These three checks put every quote on the same footing.

10. Pricing unit and cost per clean document

Attribute	Detail
Ask the vendor	Do you charge per page, per document, per field or per seat? What counts as a page? What happens above our committed volume?
Test it	Work out the cost per clean document: software fees plus review labour plus integration upkeep, divided by the documents that reach your system correct.
A pass looks like	A cost per clean document you can defend to finance, based on your pilot numbers.
Demo shortcut it catches	A low per-page price paired with a high review rate.

For a sense of scale, cloud document APIs list plain optical character recognition (OCR) at about $1.50 per 1,000 pages and structured extraction at roughly $30 to $70 per 1,000 pages, as of July 2026. Software fees at that level are rarely the biggest line. The review labour behind a low STP rate usually is.

11. Contract terms and your exit

Attribute	Detail
Ask the vendor	What are the minimum commitments? Is there a cap on renewal increases? Who owns the trained models, labels and field mappings?
Test it	Ask for a sample export of your data, labels and mappings in a named format.
A pass looks like	A renewal cap and an export format written into the contract.
Demo shortcut it catches	First-year pricing with nothing said about year two.

With two in three new IDP projects replacing an existing system, plan how you would leave before you sign. Mappings locked inside a vendor become technical debt the day you decide to move. Switching cost tends to sit in schema and integration rework, not in the extraction itself, so owning your mappings is what keeps a future move affordable.

12. Proof from production, not pilots

Attribute	Detail
Ask the vendor	Can we speak to two customers running our document type in production, at similar volume, for at least a year?
Test it	Ask those references for their STP rate and weekly review hours.
A pass looks like	References confirm numbers close to what you measured in your pilot.
Demo shortcut it catches	A slide of customer logos.

A pilot that works is not the same as a system that runs. Gartner predicted in July 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs or unclear business value. That figure covers generative AI broadly, not IDP alone, but the causes read like this checklist.

IDP vendor scoring table

With the 12 checks run, the scoring table turns them into one comparison. Copy this table into your evaluation document. Score each vendor 0, 1 or 2 per item, multiply by the weight, and add up the totals. The weights are a starting point: prioritize the items that carry the most risk for your documents.

#	Checklist item	Weight	A pass looks like	Vendor A	Vendor B	Vendor C
1	Accuracy on your documents	3	Per-class, per-field results on your blind set
2	Whole-document accuracy	3	Quoted STP confirmed by your count
3	Confidence scores	2	High-confidence values almost always right
4	Hard documents	2	Tables aligned, handwriting read or flagged
5	Exception queue	3	Review hours per 1,000 documents estimated
6	Model updates and generated values	2	Version pinning, empty when data is missing
7	Upkeep when layouts change	2	New layouts handled within the contract
8	Integration	2	End-to-end push, you own the mapping
9	Security and document use	3	Retention and training use in the contract
10	Cost per clean document	3	Defensible figure from pilot numbers
11	Contract terms and exit	2	Renewal cap and export format in writing
12	Production proof	1	References confirm your pilot numbers
	Weighted total (max 56)

Two rules keep the total honest. First, a 0 on item 1, 2, 5 or 9 rules a vendor out, whatever its total. Second, if two vendors finish within 5% of each other, break the tie on item 10, cost per clean document, and then on item 5, because those are the two numbers you will live with every month.

When a managed IDP service fits better

A strong score still leaves one question open: whether your team should run the system at all. An IDP platform fits well when your documents are steady and transactional and you have a team ready to own the exception queue. A managed service fits better when your document mix is varied, volumes spike, or nobody has the hours to clear exceptions.

Forage AI works on the managed side. We run enhanced OCR that reads faded and handwritten scans, human experts validate the output, and the data arrives in your ERP, CRM or database through webhooks, typically one to two weeks from brief to a live pipeline. The same 12 checks apply to us, and we would rather be scored on your hardest documents than on ours. For a view of the platforms on the market, see our roundup of IDP solutions.

The demo shows a vendor's best day. The checklist shows you month two, before you have signed for it. A checklist is only useful if it gets used more than once. Run it again at every renewal with a fresh blind set, share the scoring table with the people who will work the queue, and send us the document that broke your last system.

Frequently asked questions

What accuracy should IDP software reach?

It depends on your document classes, so measure it per class on your own blind set. Look at field-level accuracy and at whole-document accuracy, the share of documents with every required field right. The second number is the one that decides how much review work you keep.

What is straight-through processing in IDP?

Straight-through processing is the share of documents that go from intake to your target system with no human touch. It is lower than field-level accuracy, because one wrong field in a document sends the whole document to review.

How many documents do you need for an IDP pilot?

There is no universal number. Cover every document class, layout and channel you receive, include your hardest examples, and keep a separate blind set large enough that one or two errors do not swing the result.

How long should an IDP evaluation take?

Long enough to run at least two scoring rounds on the blind set, work the exception queue with your own staff, and push output into a staging copy of your target system. Plan the timeline around those three steps rather than a fixed number of weeks.

Should you choose an IDP platform or a managed service?

Choose a platform if your documents are consistent and you have people to run the review queue. Choose a managed service if your documents vary widely or nobody can own exceptions. Score both types against the same checklist.

Related articles

• Top 10 Intelligent Document Processing Solutions: a roundup of IDP platforms to build your shortlist from.
• Rossum Alternatives: 15 IDP Platforms Compared: how IDP platforms differ on pricing shape and document mix.
• Data Extraction Company: Enterprise Evaluation Checklist: the same evaluation approach for web data extraction vendors.
• Document Workflow Automation: how extraction, validation and exception handling fit together.