Every initiative, its own ruler
When AI is adopted function by function, each initiative arrives with its own measure. Finance has one, operations another, customer service a third. Each was written when its initiative began, by whoever was closest to it. Nobody chose to have one ruler per initiative. The institution now has as many rulers as initiatives, and no way to lay them side by side.
How a ruler was written depends on where the initiative came from.
A vendor’s defined solution. The ruler comes with the product, so the first check happens before the contract. Trace the problem statement to its first sentence, then run the contract questions on the metric, the baseline and the audit right.
Built in-house. There is no seller, and also no contract. That removes the moment that forces a check. The team best placed to write the ruler is the team with the most riding on the result, and it usually reports the result too. Nobody has to be careless for that to happen. What stands in for the contract is three separations, written down before the build starts: the success measure, fixed before delivery; a named checker who was not part of the build or of the original definition; and a date on which the check recurs whether or not anyone asks. A measure written down in advance is also what protects a team when its number is questioned a year later.
Both (hybrid). An in-house build rarely stands alone. The model, the framework or the evaluation harness usually comes from a vendor. Treat each layer with a different owner as its own artifact: the vendor’s layer has the vendor’s ruler, the internal layer has the team’s. Run the checks once per layer and read the findings separately. A single overall verdict is how one layer’s metric ends up vouching for another.
The register
One row per live initiative, whatever its origin:
- ├──Metric: Who defined it?
- ├──Baseline: Who set it, and has it moved since?
- ├──Results: Who reports them?
- ├──Checking: Who verifies, and were they part of the build?
- └──Source: Vendor, in-house, or both — and if both, which layer owns which number?
Then count the distinct authors. Look for rows where one name appears in three columns. Look for any row with nothing in the Checking column. Rows with three different names are the initiatives that can survive a hard question.
Where a layer is vendor-supplied, the Measurement Clause Checklist applies. Where it isn’t, the register is the check.
Take the initiative your institution is proudest of. For its headline number, can you name who defined it, who reports it, and who would be free to say it had stopped holding? If all three are the same team, the number is a claim that team is making about itself. That is not a criticism of the team. It is a role nobody was assigned.
“Because a measure written and reported by the party that built the thing is a claim, not a check — whether that party is a vendor or your own team.”