Data Catalog Tools: Do You Need One at Your Data Size?

An analyst posts in Slack: "which table has the real revenue number?" Three people answer, and they name three different tables. One is the raw Stripe sync, one is a model somebody built for a board deck in March, one is deprecated but nobody removed it. This is the exact moment a data catalog is supposed to prevent, and also the exact moment where buying one is often the wrong call. At twelve people the question gets a correct answer in four minutes because everyone knows who built what. At two hundred people it never gets a correct answer at all, and the cost of guessing compounds quietly across every report in the company.

The useful question is not whether data catalog tools are good. They are fine. It is where your organization sits relative to the threshold at which discovery stops being solvable by humans talking to each other. Below that line a catalog is a maintenance obligation dressed as a solution. Above it, the alternative is a slow tax on every analyst you hire. This article gives the threshold first, then compares options on the three things that genuinely differ: lineage depth, what the crawlers can and cannot see, and how much unpaid curation each vendor quietly assumes you will supply.

What a data catalog actually is, and what it is not

A data catalog is an inventory of your data assets with search on top. A crawler connects to your warehouse, BI tool, object storage, and orchestrator, reads the technical metadata (table and column names, types, row counts, last modified), and builds a searchable index. On top of that sits human-supplied meaning: descriptions, owners, business glossary terms, certification badges, deprecation flags. Better tools add lineage, mapping how data flows from source to table to dashboard, and usage signals mined from query logs.

Three things it is not, and each confusion costs money:

It is not a data quality product. A catalog tells you the table exists and who owns it, not whether the numbers in it are correct. Freshness and null-rate checks belong in a testing layer, a separate purchase and a separate discipline covered in Data Quality Tools: Catching Bad Data Before It Ships. Several vendors now bundle a quality module, and those modules are typically thinner than a dedicated tool.

It is not a governance program. Metadata management gives a governance program somewhere to record its decisions. It does not make the decisions, staff the stewards, or enforce the policy. A catalog with no policy behind it is an expensive wiki.

It is not a map of your company's data. It maps the systems its crawlers reach, which in practice means your warehouse and BI layer. Revenue in Stripe, contact history in HubSpot, and ledger detail in QuickBooks appear only if a pipeline has already landed them in the warehouse. If most of your operational truth still lives in the SaaS tools themselves, a catalog describes a small and possibly unrepresentative slice of your business.

The threshold: when a data catalog starts paying for itself

Discovery cost scales with the product of two numbers, not either alone: how many distinct assets exist, and how many people must find them without asking whoever built them. Five hundred tables and two analysts is not a catalog problem. Eighty tables and forty self-serve business users is a training problem. Five hundred tables and forty people is where it breaks.

Four signals, and a rule for combining them:

Asset count. Count physical tables and views in production schemas, not staging. Under roughly two hundred, discovery is a naming problem, and naming discipline is free. Between two hundred and a thousand, lightweight options win. Past a thousand, human memory is out of the picture.

Query writers and report builders. Anyone who chooses which table to use. Under ten, your catalog is a Slack channel and it works well. Between ten and twenty five it works badly but survives. Past twenty five, the same question is asked and answered repeatedly by people who never see each other's threads.

Independent producer teams. How many teams can create a table in a shared schema without coordinating. One means ambiguity is accidental and fixable. Three or more means it is structural, because two groups will independently build "monthly recurring revenue" with different rules and both will be right in their own context.

External lineage pressure. An auditor, an acquirer's diligence team, or a regulator asking in writing where a specific number came from. This is a step change rather than a slope. Once you have had that question, you will have it again, and lineage tooling stops being optional.

The rule that holds up in practice: you need a catalog when at least two of those are true. One signal alone is usually solvable more cheaply. Two together means the informal system has already failed and people are simply not telling you.

Counter-check with arithmetic on your own numbers rather than a vendor's. Take the people who lose time hunting for the right table, multiply by hours per week and a loaded hourly rate, and compare that to license cost plus the fraction of a role curation will consume. Run the same sum for the middle option, better naming plus a maintained ownership document. At two hundred tables the middle option usually wins by a wide margin. At two thousand it loses badly. Most teams never run the calculation and buy on a demo built against a clean schema.

What actually differs between the platforms

Every data catalog platform lists search, lineage, glossary, and collaboration. Those are table stakes, nearly indistinguishable in a demo. Four differences decide whether your rollout survives eighteen months.

Lineage depth. Table-level lineage says model B reads from table A. Column-level says the revenue_usd field on the executive dashboard traces back to invoices.amount_cents through a specific transformation with a specific currency rule. Column-level requires parsing SQL in your dialect, so support is uneven: strong for warehouse SQL and dbt projects, patchy for stored procedures, mostly absent for transformations written in application code or inside a BI tool's own calculated fields. That last gap matters more than it sounds, because the number an executive disputes is very often a BI-layer calculation no crawler ever saw. If your reporting looks like the layouts in Financial Dashboard Examples Finance Teams Rely On, ask whether the vendor parses your BI semantic layer or merely records that the dashboard exists.

Crawler coverage. Ask for the connector list, then cross out everything you do not run. What remains is the real coverage. Warehouses and BI tools are well served everywhere. Orchestrators, streaming systems, notebooks, and object storage vary a lot. Operational SaaS is the consistent weak spot, because catalogs crawl systems with schemas and a CRM's custom-field sprawl is a hard target, as the field-level mess in CRM Database: What It Stores and Where It Falls Short illustrates. If your pipelines run on an integration platform, check whether the catalog reads that platform's metadata or whether lineage stops at the landing table, a distinction covered from the pipeline side in Cloud Integration Platforms (iPaaS) Compared for 2026.

Assumed curation load. The number no vendor prints. Crawled technical metadata is free and arrives on day one. Business meaning is not. Someone writes descriptions, assigns owners, defines glossary terms, certifies the good tables, and deprecates the bad ones, then keeps doing it as the warehouse changes. Governance-heavy platforms assume a staffed stewardship function with review workflows and approval chains. Lighter tools assume one motivated data lead. Generated descriptions help with the first pass and are risky on the second, because a model reading a column named net_rev writes a fluent sentence about net revenue without knowing your net excludes refunds processed after ninety days. That business rule was the only part anyone needed.

Usage signal. The most valuable feature for a mid-size team is query-log mining, which ranks assets by who queries them and how often. It answers "which of these six revenue tables do people actually use" with zero human curation, the closest thing in the category to free documentation. The Alation data catalog built much of its early reputation on this behavioral approach and most serious competitors now offer a version. Weight it heavily: it is the one capability that keeps working when curation lapses.

A fifth dimension is genuinely enterprise-only: write-back, where the catalog pushes classification tags into the warehouse so masking policies key off them, and documentation becomes infrastructure.

Data catalog tools compared by category

Commercial pricing is almost universally sales-led, so treat secondhand figures as unreliable and get a quote against your actual asset count.

Category boundaries matter more than brand names. Data discovery tools bought to satisfy an auditor disappoint, and a governance suite bought to help analysts find tables gets abandoned by those analysts, because its workflow overhead is aimed at a different job.

Open source options and what they really cost

An open source data catalog removes the license line and replaces it with an infrastructure line and an engineering line. DataHub gives you a well-designed metadata graph and an extensible entity model, exactly what you want if your metadata needs are unusual and exactly what you do not want if nobody can extend it. OpenMetadata ships a more complete out-of-the-box experience with a large connector set. Amundsen remains a clean search-first implementation. Apache Atlas is bound tightly to the Hadoop lineage it was built for.

The honest accounting for self-hosting: a search index, a metadata store, an ingestion scheduler, upgrade cycles, single sign-on integration, and someone who owns all of it. That is a modest but permanent claim on a platform engineer, the same trade made with any self-hosted infrastructure, including the kind weighed up in Is PostgreSQL a Vector Database? Pinecone, Weaviate, Chroma. Open source wins when you have that engineer and want control of the metadata model. It loses when the real motivation was avoiding procurement, because the cost reappears as an under-maintained internal service nobody is accountable for.

One point in favor of the open tier is easy to miss: OpenLineage is a standard rather than a product, and emitting lineage events from your orchestrator in a portable format is worth doing whichever catalog you eventually buy. It turns a future migration into a configuration change instead of a rebuild.

An unmaintained data catalog is worse than no catalog

This is the failure mode that sinks most rollouts, and it deserves stating plainly. With no catalog, an analyst who does not know which table to use experiences uncertainty, and uncertainty makes them ask a human. Slow, but it terminates in a correct answer.

With a stale catalog, the same analyst reads a confident description written eighteen months ago by someone who has since left, sees a certification badge applied during the rollout push, and proceeds without asking anyone. The catalog has converted "I do not know" into "I know, incorrectly." Stale confidence propagates into reports and is far harder to detect than an open question, because nothing looks wrong until a number is disputed in a meeting.

Three mitigations that work:

Expire certifications. A badge with no expiry date is a claim about the past presented as a claim about the present. Ninety or one hundred eighty day expiries force a decision instead of silent decay.

Show freshness on the entry. Not just the data's last refresh, but the metadata's last human review, displayed beside the description. A description last touched in 2024 should look old.

Scope the initial rollout brutally. Catalog the fifty tables carrying most real query volume and leave the rest as bare crawled metadata with no human claims attached. A small curated core plus a large honestly uncurated periphery beats a warehouse-wide sweep of generated descriptions nobody has read.

If you cannot name the person whose job includes catalog maintenance and estimate the hours, you are not ready to buy. That one question predicts outcomes better than any feature comparison.

The cheaper first step most teams skip

Below the threshold, or in the ambiguous middle band, a version of this costs nothing but discipline and captures a surprising share of the value.

Enforce naming conventions that encode layer and grain, so fct_orders_daily is self-describing and revenue_final_v2_USE_THIS cannot exist. Maintain one page listing the twenty to fifty assets that matter, each with an owner, a one-line definition, and a last-reviewed date. Deprecate tables by dropping or renaming them, not by adding a comment. Use your warehouse's metadata views to find tables nobody has queried in ninety days, and delete them. Most catalog value at small scale comes from reducing the asset count, not documenting all of it.

Then automate the nagging, because ownership documents rot when nobody is reminded. A scheduled check against your own ownership list, not a crawl of your warehouse, closes the loop:

Weekly data ownership review

Every Monday 08:00

Weekly cadence, before planning

Read ownership sheet

The list your team maintains by hand

Flag stale entries

No owner, or reviewed more than 90 days ago

Message each owner

One direct message per person, listing only their assets

Post unowned assets

Anything with no owner goes to the data channel

That pattern is deliberately dumb. It reads a human-maintained list and applies a date rule. Build it in workflows by describing it in chat, or in any automation tool using the patterns in n8n Automation: 12 Workflow Examples Worth Copying. The maintenance habit, not the tooling, decides whether documentation survives.

Where Skopx fits, and where it does not

Skopx is not a data catalog. It does not crawl your warehouse, build a lineage graph, or maintain a business glossary, and it will not tell you which of your two thousand tables is the certified revenue source. It is not a warehouse, not an ETL tool, not a BI tool, and not a CRM. If you have crossed the threshold above, you need a real catalog, and no chat interface substitutes for one.

What Skopx does is adjacent, and it addresses the smaller version of the same problem. Catalogs exist because people cannot trust an answer whose origin they cannot verify. Skopx connects to nearly 1,000 tools a company already uses, including Gmail, Slack, Stripe, HubSpot, QuickBooks, and Google Analytics, and answers questions in chat with citations back to the exact source records. Ask what last month's collected revenue was and the answer arrives with the underlying records identified, so the next person can check it rather than take it on faith. That is provenance at the answer level, not the schema level, and it is honest to call it the narrower scope.

For companies whose operational truth still lives inside their SaaS tools rather than a modeled warehouse, the narrower version is frequently the whole problem. The revenue question there is not "which table" but "which system, and does the accounting tool match the payment processor," closer to the reconciliation work in QuickBooks Integrations: Picking the Ones Worth Wiring Up. Alongside cited chat, Skopx sends a morning brief, runs an insights engine that surfaces anomalies and risks, and builds automations from a description in chat, using your own AI key with zero markup at Solo $5 per month and Team $16 per seat per month on pricing.

The dividing line is clean. Schema-level trust across a large modeled warehouse with many producers is a catalog job. Answer-level trust across the operational tools a business runs on is what Skopx does. Some need both, most small companies need only the second, and a reliable way to waste a year is buying the first when the second was the gap. If your evaluation involves pointing a general AI assistant at company data, set the boundaries in ChatGPT Data Privacy at Work: What Leaves Your Company first.

How to run the evaluation

Skip the standard demo and do these four things instead.

Bring your worst schema, not the clean marts: the legacy area with inconsistent naming and three overlapping revenue tables. A tool that produces a clean lineage graph on that will produce one on anything. While it is loaded, ask one lineage question, tracing a specific dashboard number back to its source column. Wherever the trace stops is the product's real limit.

Mark every connector against your stack as full, partial, or absent, and ask what partial means in each case. It usually means table-level metadata with no lineage.

Estimate curation hours per month for the first year, then name the person supplying them. If nobody can be named, stop the evaluation.

Agree the rollout scope before signing: the fifty highest-traffic assets, expiring certifications, nothing else claimed as reviewed. Vendors push a wider initial sweep because breadth looks like progress. Breadth is how catalogs die.

Frequently asked questions

Is a data catalog the same as a data dictionary?

No. A dictionary is a static definition list describing fields and their meanings. A catalog is a live index built by crawlers, kept in sync with the systems it describes, with search, lineage, and usage signals on top. A dictionary is a fine substitute below the threshold, and it stops working as soon as the asset count exceeds what a human can keep current by hand.

Can we just use dbt docs instead of buying a catalog?

For many teams in the two hundred to one thousand table range, yes. If your transformations already live in dbt you get column-level lineage across the DAG, descriptions maintained in the same pull requests as the code, and no separate license or deployment. The limits are real: nothing outside dbt appears, source systems show up only as declared sources, and BI-layer logic is invisible. Look at a dedicated platform when lineage must cross those boundaries, or when non-technical people need to search without reading a docs site.

How long does implementing a data catalog actually take?

Crawling is fast and is not the project. Connecting sources and producing a searchable index of technical metadata typically takes days to a few weeks. The project is curation and adoption, which runs for months and never fully ends. Plan the rollout as a scoped curation exercise on a small set of high-traffic assets with a named owner and a review cadence, not an integration exercise measured by connector count.

Is an open source data catalog genuinely cheaper?

Cheaper in license, not necessarily in total. You take on hosting, upgrades, single sign-on, connector maintenance, and the support burden of an internal service. Worth it if you have platform engineering capacity and want control of the metadata model. A poor trade if the driver was avoiding procurement, because the cost moves somewhere nobody tracks it.

Do AI assistants and semantic search remove the need for a catalog?

They move the effort without removing it. Natural language search over metadata makes a well-curated catalog much easier to use, and generated descriptions speed up the first pass. Neither invents the business rules that make a definition correct, and a model summarizing column names produces confident text that is wrong in exactly the way that matters. AI reduces the cost of using a catalog, not of maintaining one.

What should a fifty-person company do right now?

Delete unused tables, enforce a naming convention that encodes layer and grain, list your twenty most important assets with an owner, a definition, and a review date, then automate the reminder. Revisit the buy decision when two threshold signals are true.