The golden set: the one evaluation every AI purchase needs

In short: a golden set is a few hundred real cases from your own work, each with an answer your experts agree is correct, plus a set of requests the system must refuse. Run every candidate on it before buying and every version after launch. It measures what public benchmarks cannot: accuracy on your documents, your edge cases and your rules.^1,2

A golden set is a collection of real cases from your own work, each paired with an answer your experts agree is correct. Run every candidate system on it before you buy, and again every time the system changes. It is the most useful evaluation tool a manager can own, and most organizations do not have one.