Start with shipped work, not slides
Ask for AI systems that are live with real users today. Not a demo video, not a pilot that quietly ended. Live. A portfolio of prototypes proves a team can build something impressive once, under ideal conditions, on data they chose themselves. Production proves they can keep it accurate, secure and affordable while the inputs get messy and the volume climbs.
Then ask what broke. Every system that has run for six months has a failure story. A team that cannot tell you theirs has either never been in production or is not being straight with you.
Ask how they measure quality
A serious partner describes an evaluation process before you think to ask for one. A test set built from your own cases. Scores for accuracy and for safety. A rule about what has to hold before a release goes out, and a way to put the previous version back when it does not.
If the answer is that they test it manually, expect surprises after launch. Manual testing catches the cases someone thought of that morning. The cases that hurt are the ones nobody thought of.
Check how they protect your data
- Will your data be used to train anyone's model? Get the answer in writing, for every provider in the stack.
- Can they run the system inside your own cloud account if your policy requires it?
- Who on their team can read production conversations, and where is that access logged?
- What happens to your data when the contract ends?
These are contract questions as much as engineering ones. If an agency treats them as an afterthought, you have learned where security sits in the way they build.
Look for engineering beyond the model
The model is a small part of the work. Most of the effort goes into the integration with the systems you already run, the interface your team touches every day, the infrastructure underneath, and the monitoring that tells you when quality drifts.
Choose a team that covers web, mobile, cloud and design, so nothing falls between vendors. When the model provider, the app builder and the hosting team are three different companies, you have become the integrator.
Agree on a small first milestone
The best engagements open with two to four weeks of work on real inputs and a decision point at the end. It proves value early, surfaces the risks while they are still cheap, and gives both sides an honest exit if the fit is wrong.
Be wary of anyone who wants a six figure commitment before they have seen your data.
The ten questions, in one place
- Which of your AI systems are live with real users right now?
- What has failed in production, and what did you change afterwards?
- How would you build a test set from our cases?
- What score has to hold before you ship a release?
- How do you roll back a release that regresses?
- Will our data train any model, and can we see that in writing?
- Can you deploy inside our own cloud account?
- Who on your side can access production data, and how is it logged?
- Who owns the code, the prompts and the evaluation set when we part ways?
- What does the first four weeks produce, and what does it cost?
Send the same ten to every agency on your shortlist. The gaps between the answers will be sharper than anything in the proposals.