How to Evaluate an AI Vendor's Proposal Before You Sign

An AI development proposal is harder to evaluate than an ordinary software proposal for one reason: the deliverable is probabilistic. A payroll system either computes the right number or it has a bug. An AI system is right some percentage of the time, and the proposal in front of you probably does not say what percentage, measured how, on whose data. Everything else in this checklist matters, but that gap is where most bad engagements start.

The short version: before you sign, the proposal should tell you what will be delivered, how both sides will agree it works, who owns every artifact, what it will cost to build and to run, and who is responsible once it is live. If any of those five is vague, the vagueness will be resolved later, at your expense.

We should say plainly that we are a vendor. We write proposals like the ones this post teaches you to pick apart, and we have an interest in buyers choosing firms like ours. We have tried to write the checklist we would want a client to use on us, and on our competitors. Apply it to both.

1. Scope and deliverables

Read the scope section and ask whether a stranger could tell, six months from now, whether each item was delivered.

What good looks like:

• Deliverables are nouns, not activities. "A deployed extraction service with an API, a review interface, and an evaluation harness" is a deliverable. "AI strategy support and model optimization" is an activity.
• Inputs are specified: which document types, which languages, what volume, what quality of scans, which systems it integrates with.
• Exclusions are written down. Every real proposal has them. If you cannot find any, they exist only in the vendor's head.
• The evaluation harness and test data are listed as deliverables in their own right. They are what lets you change models or vendors later without starting again.
• Assumptions about your side are explicit: who provides data, who labels examples, how many hours of subject-matter-expert time are needed, and by when.

Ask verbatim: "What is explicitly out of scope, and what would make you raise a change order?"

2. Acceptance criteria tied to an evaluation set

This is the section most proposals are missing, and the one to insist on.

A demo is a vendor-selected sample. Acceptance has to rest on a buyer-influenced sample: an evaluation set of real or realistic inputs with known-correct outputs, agreed before the build starts.

What good looks like:

• A named evaluation set, with a size (hundreds of examples, not a dozen), drawn from your actual data, including the ugly cases.
• A held-out portion the vendor does not see during development, so the final score means something.
• Metrics defined per task. For extraction: field-level precision and recall. For classification: per-class accuracy, not only the overall number. For generation: a grading rubric and who applies it.
• Thresholds. "At least 95% field-level accuracy on the held-out set, and no critical-field error rate above 1%" is a criterion. "High accuracy" is not.
• Non-functional criteria alongside quality: latency at a stated percentile under a stated load, throughput, and cost per unit of work.
• A defined process for what happens if the threshold is missed: remediation period, price adjustment, or exit.

A vendor may fairly push back that they cannot commit to a number before seeing the data. The honest answer to that is a short paid discovery phase that produces the evaluation set and a baseline, followed by a build phase with thresholds. A vendor who will not commit to a number after discovery is telling you something.

Ask verbatim: "Who builds the evaluation set, who holds the held-out portion, and what happens commercially if the system misses the threshold?"

Our post on the three measurements every production LLM system needs explains why this harness keeps paying for itself long after the engagement ends.

3. Ownership: models, prompts, data, code

Traditional software contracts cover code. AI engagements produce several other artifacts, and proposals are often silent on them.

Ask verbatim: "If we part ways the day after go-live, list everything we would need from you to keep running and keep improving the system. Which of those items do we own outright?"

4. Data handling

You are about to give an outside firm your documents, your customers' data, or both.

• Where does data go? Name every third-party service it touches during development and in production: model APIs, labeling tools, logging services, the vendor's own laptops.
• Training use. Confirm in writing that neither the vendor nor any subprocessor trains on your data. For model APIs, this depends on the specific service tier and its terms, so ask which tier.
• Environment. Can development happen inside your environment, or on de-identified data? For regulated data this is often the only acceptable answer. See our notes on data residency.
• Retention and deletion. What is deleted at the end of the engagement, by when, and how is that confirmed?
• Access. Which named individuals will see production data? Are subcontractors involved, and where are they located?
• Incidents. Notification timeline and responsibilities if something goes wrong.

Ask verbatim: "Draw the data flow for us, development and production, and mark every point where our data leaves our control."

5. Cost exposure

The proposal's price is the build cost. Your exposure is the build cost plus everything after it.

Build cost. Fixed-price and time-and-materials are both legitimate. Fixed price suits a well-defined scope after discovery; the vendor carries overrun risk and will price that in. Time-and-materials suits exploratory work; you carry the risk, so you need a cap, a burn report, and checkpoints at which you can stop. Be wary of fixed price on vague scope: the vendor will protect their margin with change orders.

Run cost. This is where AI proposals most often go quiet. Ask for a run-cost model:

• Cost per unit of work (per document, per conversation, per thousand requests) at your expected volume and at three times that volume.
• Who pays the model API or GPU bill, and whose account it is in. If it is the vendor's account with a markup, know the markup.
• What drives the cost: tokens per request, context length, retries, the share of traffic sent to the expensive model.
• For self-hosted systems: hardware, power, hosting, and the engineering time to keep it patched. We lay out that math in self-hosted vs API total cost of ownership.

Ongoing fees. Licenses for vendor components, support retainers, per-seat charges for review tools. Get the three-year total, not the first-year number.

Ask verbatim: "What will this cost us per month to run in year two at our projected volume, and which line items in that number do you control?"

6. Operational responsibility after go-live

AI systems degrade in ways ordinary software does not. Input data drifts. Hosted models get updated or retired by their providers. A prompt that worked in March behaves differently in September. Someone has to be watching.

The proposal should say who:

• Monitors quality, not only uptime, and with what tooling.
• Re-runs the evaluation set when the model, prompt, or provider changes.
• Responds when accuracy drops, and within what time.
• Handles model deprecations. Hosted models are retired on the provider's schedule, not yours.
• Carries the pager, if the system is business-critical.

There are two honest answers. Either the vendor operates it under a defined service agreement, or your team operates it and the engagement includes real handover: runbooks, training, and a period of shadowing. The dishonest answer is silence, which means you operate it by default, without preparation.

Ask verbatim: "Six months after launch, accuracy drops five points. Walk us through who notices, how, and what happens next."

7. Exit and handover

Assume the relationship ends at some point, amicably or otherwise. The proposal should make that survivable.

• Everything runs in accounts you own, or can be transferred to them, with a documented procedure.
• Documentation sufficient for a competent third party to take over: architecture, deployment, evaluation procedure, known limitations.
• A handover period is priced, even if you never use it.
• No component you cannot replace. If the system depends on the vendor's proprietary platform, know what migration away from it would involve.

Red flags

None of these is automatically disqualifying. Each should slow you down.

• Accuracy claims with no dataset attached. "99% accurate" means nothing without knowing on what, measured how.
• No evaluation harness in the deliverables. The vendor either does not build them or does not want you to have one.
• The demo used their data, and they are reluctant to run it on yours before signing, even under NDA on a small sample.
• Timeline with no discovery phase on a problem involving your messy data. Nobody can estimate that accurately sight unseen.
• Silence on run costs, or a run cost quoted only at demo volume.
• The senior people in the sales meetings are not named in the delivery team.
• Everything is an agent. Architecture chosen for fashion rather than need costs more to run and is harder to test. Ask why a simpler design would not do.
• Ownership language that grants you "use of" the system rather than ownership of the artifacts.
• No mention of what happens when it is wrong. Every AI system is wrong sometimes. A serious design has a review path, confidence thresholds, or a fallback. A proposal that never mentions errors has not designed for them.
• References who can speak to a demo but not to a system in production for more than a few months.

A one-page scoring sheet

Score each line 0 (absent), 1 (vague), or 2 (specific and written down). Compare vendors on the same sheet.

A proposal scoring under 12 out of 18 is not ready to sign, however good the demo was. The useful move is usually not to reject it but to send the gaps back as questions. How a vendor responds to being asked for acceptance criteria and ownership terms tells you most of what you need to know about working with them.

If you are still deciding whether to hire a firm at all, our buy vs build vs hire decision matrix covers that earlier fork. And if you would like us to be one of the vendors you run through this sheet, we are happy to be.