AI Agent Products: How to Spot One That Does Real Work

The demo runs forty seconds. An agent product notices that a customer's usage dropped, pulls the account owner from the CRM, drafts a check-in email in the rep's voice, posts a summary to Slack, and logs a task. Nobody touches a keyboard. The room goes quiet in the good way.

Six weeks after the contract is signed, the thing has produced four Slack messages and one email that got sent to a person who had already churned. What went wrong is not that the vendor lied. It is that the demo tested one path through one seeded dataset, and nobody in the room asked the four questions that would have exposed how narrow that path was.

This guide is those four questions. They are deliberately unglamorous: what can it read, what is it allowed to change, what happens when a step fails, and does it show you where the answer came from. Every serious evaluation of an ai agent product comes down to these, and almost no vendor deck answers them directly.

What the phrase "agent product" is actually hiding

"Agent" has become a category label the way "cloud" was in 2011. It is applied to a chat box with a system prompt, to a scheduled script with a language model in the middle, to a retrieval system with citations, and to a genuinely stateful planner that decomposes goals and recovers from errors. Those are not variations on a theme. They have different failure modes, different security surfaces, and different prices.

Buyers get hurt when they price and staff for one and receive another. If you want the taxonomy properly, Types of AI Agents: A Practical Map for Work Software walks through the distinctions in detail. For evaluation purposes, the short version is enough:

Most products sold as agents today sit in rows two and three, with a chat interface on top. That is not a scandal. Retrieval with good coverage is genuinely useful, and scripted automation that a non-engineer can author is genuinely useful. The problem is only that they are priced and pitched as row five.

So the first move in evaluating ai agent products is to place the thing on this table before you discuss anything else. Ask the vendor directly: does your product plan its own steps, or does it execute steps that a person or a template defined? A good vendor answers plainly. A vendor who cannot answer plainly is telling you something.

Question one: what can the agent product read?

An agent is bounded by its inputs before it is bounded by its intelligence. The most capable model in the world cannot tell you why a renewal stalled if the reason lives in a Slack thread, a support ticket, and a payment failure, and the agent can only see the CRM.

This is where most agent products for business quietly fail. The demo dataset contains the answer. Production does not.

What to actually test:

Ask a question whose answer spans three systems. Not "summarize this document." Something like: "Which accounts renewing in the next sixty days have both an open support escalation and a payment issue this quarter?" That question requires the CRM, the helpdesk, and the billing system to be readable at once, joined by a customer identity that is spelled differently in each. If the product cannot do it, you have learned the real boundary in one question.

Check whether reach means live or copied. Some products read your tools in real time through an API. Others require you to load data into their store first, on a schedule, which means every answer is as stale as the last sync and every new source is a project. Neither is wrong, but the second one has a hidden cost you should price: someone maintains those pipelines forever.

Check whether permissions are respected per person. If the agent indexes everything under a service account and then answers whoever asks, you have built a permissions bypass with a friendly interface. Ask what happens when a user asks about a document they cannot open in the source system. The correct answer is that they get nothing. Private AI for Business: Keeping Company Data Yours covers the surrounding questions on residency, retention, and model training that belong in the same conversation.

Ask what happens with unstructured text. A great deal of the useful context inside a company lives in prose: meeting notes, threads, comments on a deal. A product that only reads structured fields will miss the reason for everything. This is why narrow, well-executed capture tools matter alongside broader ones. If meetings are where your decisions actually get made, the comparison in Fireflies vs Otter AI: Which Note Taker Fits Your Team is worth reading, because an agent with no access to what was said in the room is guessing.

Reach is not a feature you can add later by asking nicely. It is architecture. Establish it first.

Question two: what is the agent product allowed to change?

Reading is low stakes. Writing is where an ai agent product becomes either valuable or dangerous, and usually both.

There is a genuine spectrum here, and you want to know exactly where a product sits:

• Read only. It answers and drafts. A human copies, edits, sends. Lowest risk, lowest leverage.
• Write to its own surface. It creates tasks, notes, or documents inside the vendor's product. Contained, reversible, mildly useful.
• Scoped writes with approval. It proposes a specific change in a specific external system, shows you the exact payload, and waits. This is where most real value sits for most companies.
• Unattended writes. It changes records in your CRM, sends email to customers, moves money, without a person in the loop. Occasionally correct. Frequently sold before it is earned.

The questions that separate a serious vendor from a confident one:

What credentials does it use, and whose are they? If the agent acts as a shared admin service account, every action it takes is attributed to a robot with keys to everything. If it acts as the requesting user with that user's scopes, blast radius is naturally limited and your existing access reviews still mean something.

Can I set the approval boundary myself? The right threshold differs by team. Sending an internal Slack message is not the same as emailing a customer or issuing a refund. A product with one global "autonomy" toggle has not thought about this. A product where approval rules are per action type, and can be tightened per workspace, has.

Is there an audit trail I can read without the vendor's help? Every action, with timestamp, actor, target record, input, and outcome. If the audit view only shows successful runs, it is a marketing surface, not an audit trail.

What is the undo story? Most write actions in business systems are not cleanly reversible. An email is sent. A ticket is closed and the customer got the notification. Ask the vendor what happens after a wrong write, and listen for whether they have thought about it at all.

A useful heuristic: the more autonomy a product claims, the more specific its answers to these four questions should be. Vagueness here correlates almost perfectly with a product that has not been run at scale against messy data.

Question three: what happens when a step fails?

This is the question that most reliably ends a sales conversation early, which is exactly why it is worth asking on the first call.

Demos never fail. Production fails constantly, and not dramatically. An OAuth token expires. An API rate limits at the worst moment. A required field is empty because someone in sales skipped it. A record was deleted between the read and the write. A model returns a plausible-looking value in the wrong format.

An agent product that does real work has opinions about all of that. Here is what a mature failure story looks like, and what a thin one looks like:

Two specifics are worth insisting on. The first is idempotency: if a run is retried, does it create a second ticket and send a second email? Ask for the mechanism, not the reassurance. The second is failure visibility by default: does someone find out, without going looking? Automations that fail quietly are worse than no automation, because the team has already stopped doing the manual version.

Ask to see the run history view for a real customer workload with the names removed. If the vendor cannot show a screen full of both successes and failures, the product has probably not run long enough anywhere to have a screen like that.

Question four: does it show its sources?

An answer without provenance is a rumor with good grammar.

For any agent product that answers questions, the citation behavior is the single most informative thing you can inspect, because it reveals whether the system retrieved something real or generated something plausible. Look for four properties:

1. The citation points to a record, not a system. "From HubSpot" is not a citation. A link to the specific deal, with the field and the value, is.
2. It is clickable and it resolves. Open three at random during the evaluation. Broken or approximate links mean the citation layer was added late.
3. It carries a timestamp. Data recency changes the meaning of an answer. "Twelve open escalations" is a different sentence if the read was four hours ago.
4. The product declines when it cannot support an answer. Ask something the connected tools genuinely cannot answer. The correct response is a clear "I do not have that." A confident fabrication in an evaluation is a confident fabrication in production, in front of a customer.

Provenance also does something structural for adoption: it converts an unverifiable claim into a checkable one, which is what allows a skeptical finance or ops lead to start trusting the system. Teams that get this right tend to have thought hard about the surrounding habits too, which is the subject of AI Native Organization: Best Practices That Hold Up.

A scoring framework for evaluating AI agent products

Run every candidate through the same grid. Score each row 0, 1, or 2. Anything scoring below 8 out of 16 is an assistant being sold as an agent.

The last row deserves a note. An ai agent vendor checklist that ignores who builds the automations misses the largest hidden cost in this category. If every new workflow requires a services engagement or an internal engineer, your realistic throughput is a handful of automations per year, and the tool's ceiling is set by your backlog rather than its capabilities.

The demo tricks worth watching for

None of these are fraud. All of them make a narrow product look general.

The seeded account. Every entity in the demo tenant is clean, consistent, and cross-referenced. Ask to run the same flow in a sandbox connected to your own data, even one system, even read-only.

The single golden prompt. The presenter types one carefully chosen question. Take the keyboard. Type your own question, badly phrased, the way your team will actually type it.

The connector that "we support." There is a large gap between an integration that authenticates and an integration that reads the specific object you need with the specific field you need. Name the object. Ask them to show it.

Autonomy framed as a slider. Products that describe autonomy as a dial from "assist" to "full auto" are usually describing a prompt change, not an architectural difference. Ask what technically differs between the settings.

The absent unhappy path. If you never see an error state during a demo, you have not seen the product. Ask them to disconnect something on purpose.

Pricing that hides the model bill. Some products bundle inference into a per-seat price with an invisible markup and undisclosed limits. Others let you supply your own model key so you pay the provider directly. Neither is automatically better, but you should know which you are buying. Affordable AI Orchestration: What You Should Pay For breaks down where the money actually goes in this category.

Where Skopx fits as an agent product, and where it does not

Skopx is an AI workspace that connects to nearly 1,000 tools a company already uses, including Gmail, Slack, Stripe, HubSpot, QuickBooks, and Google Analytics. Against the four questions above, here is the accurate answer, including the parts that are not flattering.

Reach. This is the strongest part. Chat answers questions using live data pulled from your connected tools, and answers arrive with citations pointing back to the records they came from. A question that spans billing, CRM, and email is the normal case, not the hard case.

Writes and approval. Skopx writes to connected tools, and a human still approves anything consequential. Automations are described in chat and then reviewed before they run: you can read workflows for how that authoring path works. The honest framing is supervised, not autonomous. Skopx is not going to run your renewals unattended, and any vendor promising that deserves harder questions than this article contains.

Failure and visibility. Automations report failures with reasons, retry where retrying is safe, and stop rather than looping. The insights engine runs continuously against connected data and surfaces anomalies and risks, so the common case of "something changed and nobody noticed" gets caught by the system rather than by a person remembering to check. A morning brief collects what changed overnight.

Provenance. Answers cite the source record. If the connected tools cannot support an answer, the correct behavior is to say so.

Cost structure. Bring your own AI key for any major model, with zero markup on model usage. Plans are Solo at $5 per month and Team at $16 per seat per month, listed on pricing. The model bill is yours and it is visible to you, which is the point.

Now the limits, stated plainly.

Skopx is not a BI or dashboard-building tool. If your requirement is a governed semantic layer with certified metrics and a library of visualizations for a hundred people, that is a different category, and Data Visualization Tools: How to Choose the Right One is a better starting point than this page.

Skopx is not a data warehouse or an ETL tool. It reads your systems where they live. It does not consolidate them into a modeled store, and it will not replace the pipeline layer if you have real warehouse requirements.

Skopx is not a CRM. It reads and updates the one you have. If you are shopping for the system of record itself, Open Source CRM: Self-Hosted Options and Real Tradeoffs is the relevant guide.

And Skopx is not an unattended autonomous agent. A person approves anything that matters. For some buyers that is a limitation. For most, after they have watched an unattended system email the wrong customer once, it is the feature.

Here is the shape of a typical supervised automation, so the approval boundary is concrete rather than abstract:

Failed payment triage with human approval

Payment fails in Stripe

Webhook from the billing system starts the run

Pull account context

Owner, plan, renewal date, open tickets

Assess account risk

Flags accounts with an open escalation or a recent usage drop

Draft the outreach

Written in the account owner's usual tone, with the invoice link

Owner approves

Nothing sends until a person reads the draft

Send and log

Email goes out, the CRM activity is written

Post to the account channel

Team sees what happened and why

A ninety minute evaluation you can run this week

You do not need a three month bake-off. You need one afternoon and a willingness to be rude.

Minutes 0 to 15: place it on the spectrum. Ask whether it plans steps or executes authored ones. Ask what it writes to. Write down the answers verbatim so you can compare vendors later on the same words.

Minutes 15 to 40: the cross-system question. Connect two or three of your real systems, read-only if the vendor allows it, and ask a question whose answer requires all of them. Then ask a follow-up that depends on the first answer. Then ask something unanswerable and see whether it declines.

Minutes 40 to 60: audit the citations. Open five at random. Check that they resolve to specific records and that the numbers match the source system when you look manually.

Minutes 60 to 75: break it. Revoke a token, or ask for an action against a system where your account lacks permission. Watch what the product does and, more importantly, whether anyone would find out.

Minutes 75 to 90: price the second year. Who authors new automations. What the model usage costs and who pays the provider. What happens when the person who set it up leaves. What the export path looks like if you stop paying.

Nine times out of ten, the answers from minute 40 onward decide the purchase, and the answers from minutes 0 to 15 decide whether you should have been in the meeting at all.

Frequently asked questions

Is an agent product different from workflow automation with AI in it?

Sometimes, and less often than the pricing suggests. If a human authored every branch and the model only fills in text, you have workflow automation with a language model inside it, which is a perfectly good thing to buy. It becomes an agent when the system chooses steps at runtime based on what it finds. Ask which one you are being sold, then check that the price matches the answer.

How much autonomy should we actually allow at the start?

Start with read plus draft, and approval on every external write. Run it for a month and look at the approval log: the actions you approved unchanged, every single time, are the candidates for loosening. That gives you an evidence-based path to autonomy rather than a leap of faith, and it surfaces the categories where the system is genuinely reliable in your environment rather than in the vendor's.

What is the single fastest way to disqualify an agent product?

Ask it a question that requires data it cannot see, and watch whether it says so. Fabrication under evaluation conditions, when the vendor is trying hardest to impress you, is as good as a guarantee of fabrication in production. The second fastest way is to ask to see a failed run.

Do we need engineers to run agent products for business?

For the authoring layer, ideally not: if building a new automation requires an engineer or a services contract, your throughput is limited by your backlog rather than by the product. You do still need someone accountable for connections, permissions, and reviewing what the automations did, which is usually an ops person rather than a developer.

Does bringing our own model key actually matter?

It matters for two reasons beyond price. You see real usage instead of an abstraction, which makes cost forecasting possible. And you keep the ability to change models when a better or cheaper one appears, rather than waiting for the vendor to migrate. The saving is real, but the leverage is the better argument.

How do we compare vendors that describe themselves in completely different language?

Force them onto the same grid. The scoring table above works because every row is a factual question with a checkable answer, not a positioning claim. Fill it in yourself during the demo rather than from the vendor's materials, and the marketing differences collapse into four or five real ones.