Every second brief we receive now includes an AI requirement. Roughly half of those requirements describe something a well-designed form and a database index would do faster, cheaper and more reliably. The other half are genuinely transformative. Telling them apart early is the single most valuable thing we do in a discovery sprint.
The three-question test
Before we prototype anything, a proposed AI feature has to survive three questions. They are deliberately blunt.
- Is the input unstructured? Language, documents, images and messy human intent are where models earn their cost. Structured fields in a table are not.
- Is the task high-volume? A model that saves four minutes on a task performed twice a month is a science project, not a feature.
- Is a good-enough answer useful? If only a provably correct answer is acceptable and no human will review it, you want deterministic code.
A feature that answers yes three times is worth a prototype. A feature that answers no to any of them usually becomes a better form, a saved view, or a scheduled report, and the client is measurably happier six months later.
Prototype against real data, in two weeks
Demos on synthetic data are theatre. Real corpora are inconsistent, badly scanned, full of internal abbreviations and contradicted by a memo from 2019. That mess is the actual engineering problem, and it should surface in week one rather than week twelve.
We scope the first prototype to a single narrow question type against genuine documents. It takes about two weeks and it produces evidence: a number, a demo and a list of the failure modes we found. Sometimes that evidence kills the project. That is a good outcome delivered cheaply.
Build the evaluation set before you scale
The moment a prototype looks promising, the temptation is to add features. Resist it and build the golden dataset instead: a few hundred real questions with reviewed answers, scored automatically on every change.
Without an evaluation set, 'it seems better' is the only metric available, and it is wrong about as often as it is right.
With one in place, prompt changes, model upgrades and retrieval tuning become measurable engineering decisions. Without one, every deployment is a vibe check, and quality erodes in ways nobody notices until a customer does.
Ship with the controls already on
Rate limits, fallbacks, refusal behaviour, human review paths and complete request logging are not hardening tasks for later. They ship with the feature, because the first incident is exactly when you will wish you had the logs.
None of this is glamorous. It is also the difference between an AI feature that quietly compounds value for years and one that gets switched off after a bad week.
- AI
- Product
- Evaluation