The vendor evaluation
Two different failure modes, one gate. The first is a vendor whose capability is real but hasn’t survived contact with your data. The second is a vendor whose contract is priced on a metric only they can see. Watch for both before you sign anything.
Sitting through the demo
A demo is designed to show what works. It is not designed to answer these, and nobody else in the room is positioned to raise them for you. Ask them, in order.
Was the output on screen produced live, or selected in advance from many attempts? Ask them to run it again, on something they haven’t seen.
Everything so far ran on their curated dataset. Ask what happens on data as messy as yours — the duplicates, the missing fields, the format nobody cleaned up.
Ask them to name where it breaks. “It doesn’t” is the worst answer in the room, not the best one.
Ask how many people are quietly correcting the system, off screen, for it to look this good.
Ask what you’d keep if you stopped paying in month six. If the honest answer is nothing, the dependency is the product.
Look around the room, including your own side of it. Who is paid more, promoted sooner or proven right if you believe this today — the vendor’s team, your sponsor, the integrator, the advisor who scored the shortlist? How each of them is paid is part of the answer.
“Show me the last deployment that didn’t go well — and what you changed because of it.” An answer in specifics can be checked. An answer in reassurance can’t.
Part two
Before you initial the pricing clause
“Paid only on the savings we prove” sounds safe — until you ask who counts the savings.
Find the sentence that defines the metric. If it lives in the vendor’s methodology document rather than your contract, it can be revised without your agreement. Find the sentence that sets the baseline. A baseline drawn from your worst quarter guarantees the bonus, and counts improvements your own team would have made as the vendor’s. Then ask who can actually audit the number once it’s published — “transparent reporting” is not an audit right, and if the platform doing the work is also the one reporting the result, the scorekeeper and the player are the same party.
None of this needs bad faith. A vendor paid on a number it also measures is doing what the arrangement rewards. What’s missing is someone on the buy side who owns the measurement.
Where the solution is built in-house, the same questions apply twice: to each vendor layer inside it (the model, the framework, the evaluation harness), and, with “the vendor” read as “your own team”, to the internal layer. Who picked the benchmark, who ran the demo, who keeps the score.
If the vendor proves the savings, the vendor owns the deal. Verification belongs to the buyer, or it isn’t verification.→ These six and three are the fast version. Bring a real proposal and we’ll run it against the full ten-question Measurement Clause Review.
“Because whoever picks the benchmark, runs the demo or keeps the score has a stake in how it comes out — and the buyer's own side of the table is not exempt.”