Data Quality Tools: Catching Bad Data Before It Ships

The number was wrong for nine days before anyone noticed. A nightly sync from the order system timed out, retried, and loaded the same day of rows twice. Nothing errored. The pipeline reported success, the dashboard refreshed on schedule, and one acquisition channel quietly looked about twice as good as it was. The problem surfaced when a marketing lead asked why she could not find the extra orders in the source system.

That failure mode is the entire reason data quality tools exist, and it is also the reason so many teams buy the wrong one. The category contains two genuinely different products marketed with the same vocabulary. One lets you assert what must be true and stops the pipeline when it is not. The other watches your tables and tells you when something looks statistically unusual compared to last week. Teams routinely buy the second, expensive one before writing a single instance of the first, cheap one. A duplicate load like the one above is caught by a five line uniqueness assertion that costs nothing and runs in seconds. It is also eventually caught by an anomaly detector, usually with a weaker signal and a longer delay.

This is a guide to telling the two apart, choosing between them honestly, and writing the handful of checks that catch most real incidents.

Two things get sold as data quality tools

Here is the split, stated plainly.

Testing frameworks are code. You declare an expectation ("order_id is unique in this table", "amount is never null", "status is one of these four values"), the framework compiles it to a query, and the query runs as a step in your pipeline. If it fails, you decide what happens: warn, or halt the run so downstream models and dashboards never see the bad data. dbt's built in tests, Great Expectations, Soda Core, Pandera, and Deequ all live here. The defining property is that a human wrote down what correct means, in advance, and the check runs before the data ships.

Observability platforms are monitoring. You connect them to your warehouse, they read metadata, query logs, and samples, and they learn baselines for freshness, row volume, schema shape, null rates, and column distributions. When today deviates from the learned pattern, they alert. Monte Carlo, Bigeye, Metaplane, Anomalo, and Elementary sit in this space, along with diffing tools like Datafold that compare a table before and after a change. The defining property is that nobody had to predict the failure. The platform notices that something moved.

Both are legitimate. They answer different questions, and the difference matters more than any feature comparison between two vendors in the same tier.

Read the "when it fires" row twice. A test is preventive. Monitoring is detective. The nine day duplicate above needed prevention, and no amount of detection spend substitutes for it.

What a testing framework actually does

A test is a query that returns rows only when something is wrong, plus a threshold for how many failing rows are tolerable. That is the whole mechanism. In dbt you write a few lines of YAML next to a model and get uniqueness, not null, accepted values, and relationship checks for free. In Great Expectations or Soda you write a suite in Python or YAML that runs against a batch. The output is the same: a pass, a warning, or a failure with the offending rows available for inspection.

The reason this is the cheap layer is that the expensive part is not the software, it is knowing your data. The person who can say "this table is one row per order line, not one row per order" is the person the whole discipline depends on. Once that is written down as an assertion, every future contributor gets caught the moment they break it.

Two details separate teams that get value from teams that abandon this after a month:

Tests belong in CI and in production runs, not just one of them. Running them only in CI means a bad upstream load passes untested. Running them only in production means a broken pull request gets merged and then wakes someone up.

A failing test needs an owner and a severity. Split checks into blocking and warning. Blocking means the run stops and downstream tables stay stale rather than wrong, which is almost always the better trade. Warning means log it and move on. A repository where every check is blocking gets its checks disabled within a quarter.

What data observability tools actually do

Observability platforms earn their price in a specific situation: you have more tables than you have people, and a meaningful share of those tables were built by someone who has left or by a pipeline nobody reviews. Under those conditions you cannot write assertions for everything, and broad automated coverage is genuinely worth paying for.

The core capabilities are consistent across vendors. Freshness monitoring flags a table that has not updated within its usual cadence. Volume monitoring flags a row count outside its normal range. Schema monitoring flags a column added, dropped, or retyped upstream, one of the most common causes of silent breakage. Distribution monitoring flags shifts in null rate, cardinality, or numeric spread. Lineage ties a failing table to the dashboards and models that consume it, so the alert says who is affected and not just what broke.

The honest limitation is baked into the method. Anomaly detection compares against history, so it cannot tell you a number is wrong, only that it changed. A field miscalculated since the day it was built looks perfectly healthy. A genuine business change, like a pricing launch or a large enterprise deal, looks like an incident. Tuning that ratio is ongoing work, and a platform nobody tunes becomes a channel nobody reads.

Data observability tools also need somewhere to observe. If your reporting lives in spreadsheets and SaaS exports rather than a warehouse, most of this category has nothing to connect to. The prerequisites and cost of getting there are covered in Enterprise Data Warehouse: Concept, Examples, and Cost, and the sequencing is worth stating plainly: warehouse first, then tests, then monitoring.

The five checks worth writing on day one

If you write nothing else, write these. They are cheap, they run in seconds, and between them they catch the majority of incidents that reach a dashboard.

1. Uniqueness on the grain. For every table, name the column or combination of columns that identifies one row, and assert that it is unique. This is the single check that catches duplicate loads, double firing webhooks, and retried syncs. It is also the check that forces you to state the grain, which is valuable on its own.

2. Not null on join keys and money columns. Nulls in a join key silently drop rows. Nulls in an amount column silently understate totals. Both look like a smaller number rather than an error, which is why they survive so long. Assert not null on every column that participates in a join and every column that gets summed.

3. Freshness. Assert that the maximum timestamp in the table is within the window it should be. A stale table is more dangerous than a missing one, because a missing table produces an obvious error while a stale one produces a confident, outdated answer. Set the window to the real cadence plus a small buffer, not to a hopeful number.

4. Row count within an accepted range. Not anomaly detection, just a floor and a ceiling you are willing to defend. Zero rows loaded is the classic case, and a hard floor of one catches it immediately. A ceiling catches the duplicate load from the opening scenario even when uniqueness is not enforceable.

5. Accepted values and referential integrity. Assert that a status column contains only the values your logic handles, and that every foreign key resolves to a row in the parent table. New enum values appearing upstream is one of the most common ways business logic starts quietly mishandling a slice of records, because the unhandled value usually falls into an else branch and disappears.

There is a sixth check that is not a test in the framework sense but catches the most expensive errors: reconciliation against the source of truth. Compare your warehouse revenue total to what the payment processor reports, and your recorded fees to what actually cleared. The mechanics and the specific traps in that comparison are covered in Stripe QuickBooks Integration: Reconciling Fees and Payouts. If you sell through a storefront as well, the same reconciliation discipline applies to order and refund flows, as described in Shopify Integrations: Orders, Books, and Marketing in Sync. No amount of statistical data quality monitoring replaces tying a number back to the system that owns it.

Where each category fails

Testing frameworks fail through coverage gaps. Tests only exist where someone wrote them, and the tables nobody tests are usually the ones nobody understands, which is exactly where the risk sits. They also go stale: a check written against last year's schema can pass while asserting something that no longer matters. And they cover only modeled data, so raw landing tables and one off exports stay untested by construction.

Observability platforms fail through noise and cost. The first month is genuinely impressive because every finding is new. By month four, if nobody owns triage, the alert channel is muted and the subscription is renewing anyway. Cost also scales with tables watched, and profiling queries consume warehouse compute on top of the license, a line item teams frequently forget to model.

There is also a shared failure no tool addresses: nobody agreed on the definition. If sales and finance count a customer differently, every check passes and both numbers are still wrong for somebody. That is a governance problem, and it needs a written definition with an owner before any data quality platform can help.

How to choose between data quality tools

Choose by trigger, not by maturity model. The following sequence reflects what actually forces each purchase.

Two notes on that table. The third row is where most companies actually sit, and it is where the wrong purchase is most tempting, because vendors sell into it hard. The last row is about control and evidence more than data engineering, and the split between what to buy and what to wire up there is covered in AI Compliance Software: What to Buy and What to Wire Up.

What data quality tools cost, and what you are paying for

Open source testing frameworks carry no license cost, and that number misleads in both directions. There is real cost in engineering time to write and maintain checks, in CI and warehouse compute to run them, and in someone being on the hook when a blocking test fails at 3am. There is also real saving, because the compute cost of a uniqueness check is trivial next to the cost of a wrong board deck.

Observability platforms are typically priced by tables or monitors covered, sometimes by warehouse consumption. The list price is the visible part. Model the rest before signing: warehouse compute for profiling, engineering days to connect sources and set ownership, and ongoing triage time. A platform with no named triage owner produces zero value at full price.

Then there is a third family that shares the vocabulary and solves a different problem. Enterprise data quality management tools such as Informatica, Talend, Ataccama, and Collibra sell matching, deduplication, standardization, address and name parsing, and stewardship workflows, mostly aimed at master data in large organizations. If your pain is "we have four records for the same company in the CRM and nobody knows which is real", that is what this family of data quality management software is for, and warehouse era testing frameworks will not fix it. If your pain is "the pipeline loaded twice", the reverse holds. Buying across that boundary is a common and expensive mistake.

Data quality without a warehouse

Plenty of companies run on spreadsheets, SaaS reports, and a shared drive, and most of this category has nothing to sell them. The underlying discipline still transfers.

Write definitions down before you write formulas. Keep control totals on every sheet that matters, meaning a cell that recomputes the total a second way and a cell that flags when the two disagree. Reconcile to the system of record on a fixed cadence rather than when something looks off. Timestamp every manual export, because an undated CSV is the spreadsheet equivalent of a stale table. If operational tracking lives in a tool like Smartsheet, the same freshness and ownership questions apply to the views built on top of it, the subject of Smartsheet Dashboards: Build One and Keep It Current. The same goes for process docs that drift out of sync with the systems they describe, a pattern covered in Notion Integrations: Connecting Notion to the Rest of Work.

Where Skopx fits, and where it does not

Skopx is an AI workspace that connects to nearly 1,000 tools a company already uses, including Gmail, Slack, Stripe, HubSpot, QuickBooks, and Google Analytics. You ask questions in chat and get answers with citations back to the source records, you get a morning brief, its insights engine surfaces risks and anomalies in the business metrics it can see, and you build workflows by describing them in chat. You bring your own AI key for any major model with zero markup. Pricing is $5 per month for Solo and $16 per seat per month for Team, listed on the pricing page.

Now the part that matters here. Skopx is not a data quality platform and it is not one of the data observability tools described above. It does not sit in your pipeline, it does not assert expectations against warehouse tables, it cannot fail a dbt run, and it will not tell you that a foreign key stopped resolving in a staging model. It is not a warehouse, not an ETL tool, and not a BI dashboard builder. If you need pipeline level testing, you need a testing framework. If you need table level monitoring across hundreds of models, you need an observability vendor.

What Skopx does is narrower and different: its insights engine looks at the business metrics visible through connected tools and flags things that look off, a revenue pattern that broke, a pipeline stage that stopped moving, an invoice cohort aging past its usual window. That is anomaly surfacing at the business metric layer, on data the connected tools expose, not integrity checking at the row and column layer. The two get confused because both are called anomaly detection. They operate on different objects and catch different problems, and a team that has one still needs the other.

Where Skopx genuinely helps a data team is the routing around an incident, not the detection of it. When a test fails, what follows is coordination: telling the right people, noting affected dashboards, logging what happened. That is a workflow you can describe in chat.

Route failed data quality checks to the right owner

Test failure webhook

Your pipeline posts the failing check name, table, and severity

Blocking only

Warnings are logged, not paged

Notify the owning channel

Post the table, the check, and the downstream consumers

Append to incident log

Date, table, cause, resolution, for the monthly review

The same honesty applies elsewhere on the stack. Tools that promise automatic understanding of messy inputs are worth the same scrutiny, a theme covered in Automated Market Analysis Software: What Actually Works, and the operational side of chasing bad or stale records shows up in Accounts Receivable Automation Software: Get Paid Faster.

Frequently asked questions

Do I need both a testing framework and a data observability platform?

Eventually, most teams past a few hundred models do. In order, though, tests come first. Tests are cheap, preventive, and encode knowledge that survives staff turnover. Observability supplements them by covering the tables nobody wrote tests for. Buying monitoring while your models are untested means paying a subscription to be told, after the fact, about failures a free assertion would have blocked.

What is the difference between data quality software and data observability tools?

In current usage, data quality software usually means the assertion layer, checks you author against known rules, while data observability refers to automated monitoring of freshness, volume, schema, and distribution with lineage attached. The older enterprise meaning of data quality management tools is different again: matching, deduplication, and standardization of master records. Ask any vendor which of the three they are, because all three use the same words on their homepage.

How many tests are enough?

Coverage on every model that a human or a dashboard consumes, with the five day one checks as the baseline, is a defensible target. Chasing a coverage percentage across every raw table is not, because tests on tables nobody reads generate maintenance without reducing risk. A better metric than count is this: for each incident that reaches a business user, ask whether a check could have caught it, and write that check.

Who should own data quality monitoring?

One named person per domain, not a committee and not "the data team" in the abstract. The owner decides severity, triages alerts, and has authority to mute a noisy monitor. Unowned alerting reliably decays into ignored alerting, which is worse than no alerting because it produces a false sense of coverage.

Can AI find data quality issues automatically?

It can help with two narrow things: suggesting checks by profiling a table, and summarizing an incident once it fires. It cannot know your grain, your definitions, or which discrepancy actually matters to the business. Treat generated checks as a draft a person reviews, because a confidently wrong assertion is harder to spot than a missing one.