Eight Questions to Ask Before You Buy an AI Agent

The demo was flawless. The vendor typed "summarize my pipeline and draft follow-ups for stale deals," and the agent pulled from a suspiciously tidy HubSpot instance, wrote four crisp emails, and the room nodded. Three weeks into the pilot, the same agent quietly skipped half the deals because a HubSpot rate limit hit mid-run, nobody got an error, and the one email it did send went out under the wrong rep's name. Nobody in the buying committee had asked what happens when it fails, because nobody had an ai agent checklist. They had a demo scorecard, and demos are designed to be passed.

This article is the checklist. Eight questions, in the order that kills bad deals fastest. Every one of them is a question a vendor should be able to answer in plain language, live, without scheduling a follow-up with their solutions engineer. If they cannot, that is itself the answer.

Why you need an AI agent checklist, not a demo scorecard

Most AI agent evaluations are run backwards. Teams start with "what can it do," collect a feature matrix, and pick the longest column. Then they spend month two discovering the questions that actually mattered: where the data went, what broke silently, who approved the action that shouldn't have happened, and what it costs to leave.

The failure modes of AI agents are not feature gaps. They are operational gaps. An agent that does 40 things adequately but fails loudly, logs everything, and asks before acting will beat an agent that does 200 things and fails silently. That is the core inversion an ai agent checklist forces: you evaluate the product on its worst day, not its best demo.

A related trap: capability claims in this market are aspirational by default. Vendors describe what the architecture could do, not what the shipped product reliably does today. Before you evaluate any specific product, it helps to calibrate on what AI agents genuinely cannot do yet, so you can hear the difference between a roadmap and a feature.

The eight questions below split into four themes: data path (questions 1 and 6), failure behavior (question 2), the permission model (questions 3 and 4), and total cost of ownership including the cost of leaving (questions 5, 7, and 8).

Question 1: Where does my data actually go?

Not "is it secure." That question gets you a paragraph of adjectives. Ask for the path, hop by hop: when a user types a prompt that touches our Salesforce data, what services does that request transit, which subprocessors see it, where is it stored, for how long, and is any of it ever used to train models?

The answers you want are specific and boring. Encryption at rest with a named standard, AES-256 is the common floor. TLS 1.3 in transit. Per-tenant isolation at the row level, so one customer's query can never surface another customer's records. A written statement that customer data never trains models, in the contract, not just the marketing site. Named subprocessors with regions.

The red flags are just as recognizable. "We anonymize data before using it for model improvement" means your data trains models. "We use industry-standard security" without naming a standard means nobody has written it down. A pause when you ask about tenant isolation means the answer is a shared database with an application-layer filter, which is exactly the architecture behind most cross-tenant leak incidents.

One more question inside this question: does the agent's retrieval layer respect the source system's permissions? If your Jira instance restricts a project to six people, and the agent indexed it, can person seven now ask the agent about it? This is the single most common data-path failure in retrieval-augmented products, and you should make the vendor demonstrate the denial, not describe it. We keep a fuller security checklist for connecting AI to your tools if you want the long version with the OAuth scope questions included.

Question 2: What happens when it fails?

Every agent fails. APIs rate-limit, tokens expire, a webhook arrives malformed, a model returns something that doesn't parse. The question is never whether, it is what the product does next, and whether you find out.

Make the vendor show you a failed run inside the product. Not a slide about reliability. The actual screen. You are looking for four things:

• Visibility. Is there a run history where a human can see every execution, what succeeded, what failed, and why? If failures only exist in the vendor's internal logs, you will learn about them from your customers.
• Retry behavior. Does a transient failure retry automatically, with backoff? Or does the whole run die because HubSpot returned a 429 at step three of seven?
• Partial completion. If the agent processed 60 of 120 records before failing, what happens on the next run? Does it resume, restart, or worse, reprocess and send 60 duplicate emails? Idempotency is a word the vendor should use before you do.
• Blast radius. When a run fails, does it fail closed? A failed read is annoying. A half-executed write, an email sent to the wrong segment, a Stripe refund issued twice, is an incident.

This is where architecture shows. In Skopx, workflows run with retries built in, every version of a workflow is kept, and every run leaves a full history you can open and inspect. That is not a premium tier feature, it is the minimum for trusting scheduled automation, and it is the standard you should hold any vendor to.

The follow-up question that separates mature products: "show me how I would debug a run that produced a wrong answer, not a crashed one." Silent wrongness is harder than loud failure, and the honest vendors will admit it and show you their citation or trace tooling. The evasive ones will change the subject.

Question 3: What can it do without asking me?

This is the permission model question, and it has two halves: what the agent is technically able to do, and what it will do unprompted.

The first half is about scopes. When the agent connects to Gmail, did it request read-only access or full send-as-you rights? When it connects to Stripe, can it read charges or issue refunds? Least privilege is easy to promise and rare to implement, because broad scopes make demos smoother. Ask to see the exact OAuth consent screens for the three tools you care most about, and check what an admin can revoke without breaking everything else. The mechanics of how agents authenticate to tools matter more than most buyers expect; the MCP versus OAuth question for business tools is worth twenty minutes of your evaluation.

The second half is about autonomy, and here you want precision, not vibes. A well-designed agent draws a hard line: actions inside your tools happen on a human's instruction with a human's approval, while autonomous behavior is confined to surfaces that cannot cause harm, things like generating a morning briefing, monitoring for changes, or publishing content a human already scheduled and approved. Skopx draws exactly that line: chat actions execute on your instruction with your approval, and the autonomous surfaces are briefings, monitoring with approval-gated follow-ups, and scheduled publishing. Whatever product you buy, get its version of that line in writing, because "the agent decided to" is not a phrase you want in a postmortem.

Question 4: Where is the audit trail, and can I trust the answers?

Assume the agent will one day do something a human questions. The controller of that moment is the audit trail. Who asked what, what did the agent access, what did it do, and who approved it? If the answer involves exporting vendor logs through a support ticket, you do not have an audit trail, you have a favor.

The other half of trust is citations. When an agent answers "Q3 churn was concentrated in the starter tier," can you click through to the source: the actual query, the actual document, the actual CRM records? An answer without a source is a guess wearing a suit. This matters most for the questions that drive decisions, and those are precisely the questions executives will ask the agent. Insist on seeing citations on a hard question against your own messy data, not the vendor's demo workspace. Any product where answers cite their sources by default, the way every Skopx answer does, has made a structural commitment to being checkable. Products that add citations "on request" have made a structural commitment to being impressive.

Question 5: What does it cost at real usage?

AI agent pricing hides in three places: the seat price, the usage markup, and the overage cliff.

The seat price is the number on the pricing page. Fine. The usage markup is the quiet one: many vendors resell model tokens at a multiple of what the model provider charges, which means your bill scales with adoption in a way the pricing page never showed. Ask directly: "do you mark up AI usage, and by how much?" Then ask what happens in month four when your team's usage doubles because the product is working. Is that a budget conversation or an automatic overage charge?

Run the math on your own numbers, not the vendor's example customer. Take your realistic team size, assume the heavy users run twenty to fifty agent interactions a day, and ask the vendor to project the monthly bill in writing. If they cannot or will not, the bill is designed to surprise you.

For calibration on structure rather than a price war: Skopx charges $16 per seat per month with 2.3 million AI tokens included per seat, no API key required, and a $5 Solo plan where you bring your own key and pay the provider directly. Either way there is zero markup on AI usage. You do not have to buy that plan, but you should use it as leverage: any vendor quoting materially more should be able to say exactly what the difference buys you. The pricing page shows the full breakdown if you want the comparison numbers in front of you during vendor calls.

Question 6: How deep are the integrations, really?

"Integrates with Salesforce" is a claim with a hundred possible meanings, from "full read-write across objects with your permission model respected" down to "we can create a contact." Integration count is a marketing number. Integration depth is an engineering fact, and it is the fact that determines whether the agent handles your actual workflow or just the demo version of it.

For each of your five most important tools, ask three things. First, which specific actions are supported: in HubSpot, contacts only, or deals, tickets, and workflows too? Second, is the integration a native API connection or browser puppeteering that breaks when the UI changes? Third, what happens when the tool's API errors mid-action, which loops back to question 2.

Then ask the uncomfortable one: "of your integrations, how many have you personally seen a customer use in production?" A platform with nearly 1,000 connected tools, which is the scale Skopx operates at, should still be able to answer depth questions per tool without flinching. A vendor with 30 integrations who knows all 30 cold may serve you better than one with 500 shallow ones. The number is not the point. The match with your stack is.

Question 7: Who runs this after the pilot team moves on?

Most AI agent purchases die between month three and month six, and the autopsy usually reads the same: the champion who configured everything changed roles, and nobody else could maintain it. This is the pattern behind most stalled deployments, and why AI pilots stall is a topic worth reading before you sign anything, not after.

The questions that predict survival:

• Can a non-technical operations person build and modify an automation, or does every change route through engineering? If the vendor's answer involves "our professional services team," multiply the sticker price accordingly.
• Are automations versioned? When someone edits a workflow and Tuesday's run breaks, can you see what changed and roll back?
• Is there a single place a manager can see everything the agent did this week, or does oversight require assembling screenshots?

The build-versus-buy calculus lives here too. If the product requires so much configuration and maintenance that you are effectively building on top of it, you should price the honest alternative; our breakdown of build versus buy for AI agents covers where that line sits for teams under a few hundred people.

Question 8: What is my exit strategy?

Ask this one while everyone is still friendly, because you are negotiating the terms of a breakup during the honeymoon, which is the only time you have leverage.

Concretely: if you cancel, what do you get out? Your documents and data should be exportable in usable formats. Your workflow logic, the actual automation definitions your team spent months refining, should be inspectable and portable enough to rebuild elsewhere, not trapped in a proprietary visual format with no export. Your run history and audit logs should be downloadable, because compliance obligations outlive vendor contracts. And deletion should be verifiable: a written commitment on timeline for purging your data, including from backups and any vector indexes built on your documents.

Also ask what degrades before you leave. If the vendor gets acquired and the product you bought gets folded into a suite, what does your contract say? If they deprecate the integration your core workflow depends on, what notice do you get? These feel like paranoid questions until the year you need them, and in this market, consolidation years come frequently.

The AI agent checklist in one table

Print this, take it into the vendor call, and write the answers down in the vendor's own words. The middle and right columns are the point: most vendors pass the left column and fail the right one.

If a vendor scores strong on six or more of the eight, you are in rare territory and price becomes the conversation. Four or five, pilot narrowly with read-only scopes first. Three or fewer, the demo was the product.

FAQ: using the AI agent checklist in practice

Does this checklist change for a five-person team versus a 500-person company?

The questions stay identical, the weighting shifts. Small teams should weight questions 5 and 7 heaviest, because they have no budget slack for usage surprises and no bench to absorb a maintenance burden. Larger companies should weight 1, 4, and 8, because data path, auditability, and exit terms are where enterprise risk concentrates. No size of company gets to skip question 2. Failure behavior is universal.

How long should an evaluation actually take?

Two to four weeks of calendar time, most of it a narrow pilot, is enough if you run it against real work. One structured vendor call gets written answers to all eight questions. Then pilot one real workflow, with real data, read-only scopes where possible, and deliberately induce a failure: revoke a token mid-pilot, or feed it a malformed input, and watch what the product does. A 90-day evaluation usually means nobody defined what passing looks like. For a broader evaluation sequence, our AI agent buyer's guide covers scoping the pilot itself.

What is the single biggest red flag?

A vendor who cannot show you a failure. Every production system fails, so a product with no visible failure states has either hidden them from you or hidden them from itself. Both are disqualifying. The healthiest signal in this entire process is a vendor who volunteers "here is what we are bad at."

Should I pilot with real data or sandbox data?

Real data, restricted scope. Sandbox pilots test the vendor's data, which is always clean, always well-labeled, and nothing like your CRM with its six years of duplicate contacts and half-filled custom fields. Restrict the blast radius with read-only access and a small user group, but let the agent see the mess. The mess is what you are buying help with.

The vendor answered everything well. What now?

Get the answers in writing, attached to the contract, especially the training-data commitment, the markup answer, and the exit terms. Verbal answers from a sales call have a shelf life of exactly one account manager transition. Then negotiate the pilot success criteria before the pilot: three named workflows, a defined accuracy bar, and a specific date on which you decide.

Run the checklist before the contract, not after

Every question on this list costs the asker nothing and reveals something the demo was built to hide. The pattern across all eight is the same: evaluate the worst day, not the best one. Where does data go when nobody is watching, what happens when the API says no, what can the agent do without a human, and what does leaving cost.

Vendors who build for that worst day tend to answer these questions quickly, because the answers are just descriptions of how the product already works. We built Skopx to sit on the right side of this table, and we would rather you show up to the call with this list than without it, whoever you end up buying from. The teams that get burned are almost never the ones who asked too many questions.