Choosing your first AI agent — a five-question buyer framework

You've watched the demo. You're impressed. Now what?

The honest answer is: now you need to find out whether what you saw on a Zoom call will survive contact with production. The demo was almost certainly engineered to work. The agent ran in a sandbox. The inputs were curated. The failure modes were hidden by careful slide order.

None of that is dishonest. It's normal sales. But it means the second call — not the demo call, the one after — is where you find out if you're buying a product or buying a press release. Below are the five questions to ask on that call. They take about thirty minutes total. They'll save you six months.

Question 1: "Walk me through the audit trail for a real customer task."

Not a sample. Not a demo. A real task, run by a real customer, two weeks ago. Anonymized if needed. You want to see:

• The original input the agent received.
• The agent's reasoning — what it considered, what it ruled out.
• The tools it called, with arguments.
• The responses those tools returned.
• The decisions made between steps.
• The final output.

If the vendor can show you this in a clean, readable format, you're looking at a real production system. If they hand-wave ("our system is observable"), or show you raw LLM logs that nobody on your team could ever audit, or claim "for security reasons we can't share customer data" without offering you a sanitized real example — you're looking at a demo wrapped in marketing.

Question 2: "What does the agent do when it's uncertain?"

Bad agents either guess (and tell you about it after the fact) or fail silently. Good agents have an explicit escalation policy. Ask:

• What signals trigger an escalation?
• Where does the work go — to whom, in what format, with what context?
• What is the operator expected to do, and how does their response flow back into the agent?
• Is there a confidence threshold I can tune?

If the answer is "the agent never escalates because it's always right," walk away. No agent is always right. A vendor who claims otherwise either doesn't know their system well enough to ship it, or is lying. Neither is good.

A production AI agent without an escalation policy is a liability with confidence.

Question 3: "Show me a failure and how you recovered from it."

Every production system has had failures. The question is what happened next. You're looking for evidence of:

• Detection time. How fast did the vendor know something was wrong? Hours? Days? Customer complaint?
• Containment. Did the failure spread to other customers, other tasks, other workflows? Or was it isolated?
• Reversibility. Could the affected actions be rolled back? Were customers made whole?
• Postmortem. What was the root cause? What changed afterward so it can't happen again?

A vendor who answers "we haven't had any failures" is either too new to have learned anything, or not honest enough to tell you. Either way, that's information.

Question 4: "What is the agent allowed to do, and who decided?"

Every agent in production has a permission model. You need to understand it before the contract, not after the first surprise. Specifically:

• Can the agent send emails? To which addresses? Internal only, or external?
• Can the agent spend money? Is there a per-action cap? Per-day cap? Total cap?
• Can the agent modify or delete records in your systems? What's the rollback policy?
• Can the agent call other tools or APIs not on the originally agreed list?
• How are these permissions changed — by whom, with what review, with what logging?

Good vendors hand you a written permissions document on day one. Mediocre vendors have a verbal policy and ask you to trust them. Bad vendors haven't thought about it.

Question 5: "What does the contract look like if I want to leave in 12 months?"

This is the most revealing question of the five. The answer tells you whether the vendor is confident enough in their product to make leaving easy, or whether their business model depends on lock-in.

Look for:

• Data portability. Can you export all your data — including the agent's training history, the audit logs, your knowledge graph — in standard formats?
• Notice period. 30 days? 90 days? A year? Annual contracts with auto-renewal locked in mid-term are a yellow flag.
• Off-boarding support. Will the vendor help your team migrate, or do you get a dead inbox and a "good luck"?
• Cost ramp. Does the price go up 3× in year 2? Is the pricing tied to outcomes you can verify, or to vanity metrics like "agent tasks executed"?

What scoring looks like.

You don't need a formal scorecard. After the call, ask yourself for each question: did the answer feel rehearsed, or did it feel like the vendor was telling you something true about their actual system?

Three out of five "felt true" is enough to proceed with a paid pilot. Less than three, walk away. All five, you've found a real partner — which is rare enough that you should move fast.

The honest meta-question.

The hardest thing about buying AI agents in 2026 is that the technology is moving faster than the procurement processes designed to evaluate it. Your CISO has compliance checklists built for SaaS. Your finance team has procurement frameworks built for software with linear pricing. Neither of those was designed for systems that take autonomous actions on live customer data with non-deterministic outputs.

The five-question framework above is a bridge. It's not a replacement for proper procurement. But used early, it filters out roughly 80% of vendors before you waste your legal team's time. That alone is worth the thirty minutes.

One more thing. If a vendor refuses to answer any of these five questions on a normal sales call, that's the answer. They are not a fit for production work. The good vendors love these questions because they get to show off — they've thought about all of this and they want you to know. That's the signal.