Automatic Data Collection System: What It Actually Means and How to Build One

An automatic data collection system is any setup that gathers data on a schedule or in response to an event, without a person manually exporting, copying or retyping it. In practice it has four parts: a source you pull from (an API, a database, a sensor, a form, a web page, a file drop), a trigger that decides when to pull (a cron schedule, a webhook, a change in the source, a file arriving), a transport that moves the data, and a destination that stores it with enough structure that someone can query it later. If a system has all four and runs unattended after setup, it qualifies. If a person clicks "export CSV" once a week, it does not, no matter how good the dashboard on the other end is.

Most teams building one for the first time should not build one at all. Before you write a collector, check whether the data is already collected somewhere and you simply cannot see it. This is the single most common wasted project in operations: a company builds a nightly job to pull Stripe charges into a warehouse when the finance question they actually had was "which customers downgraded last month and why", and the "why" was never in Stripe at all. Collection is cheap. Knowing which questions the collected data can answer is the hard part, and it is worth doing first.

The Four Components, Concretely

The component people underinvest in is the trigger. A collection system that runs on a cron and writes to a table looks healthy right up until the source API starts returning 429s at 3am and the job exits zero because nobody checked the response code. Six weeks later somebody notices the chart is flat. The fix is not more sophistication in the collector, it is a freshness check on the destination: a query that asserts the newest row is younger than the expected interval, and something that shouts when it is not.

Push Versus Pull: Pick Before You Build

Pull means you ask the source on a schedule. Push means the source tells you when something happens. The choice determines almost everything downstream, and the wrong one is expensive to undo.

Pull is right when the source has no webhooks, when you need a complete picture rather than a stream of changes, when the source is a database you control, or when you are reconciling. Pull is forgiving: if a run fails you rerun it. Pull is also lossy in a specific way. If you pull orders every hour and an order is created and deleted within the same hour, you never see it. For financial or compliance data this matters.

Push is right when you need low latency, when volume is high enough that repeated full pulls are wasteful, and when the source emits reliable events. Push is unforgiving: if your endpoint is down for ten minutes, whatever the provider sent during those ten minutes may be gone, depending on their retry policy. Stripe retries webhooks for up to three days with exponential backoff. Many smaller providers retry three times over fifteen minutes and then drop the event permanently.

The mature answer for anything that matters is both. Push for freshness, pull on a slower schedule for correctness. A nightly reconciliation pull that compares row counts and checksums against what the webhook stream delivered will catch drift within a day. Teams that skip the reconciliation pull generally discover their gap during an audit.

Worked Example: Collecting Support Ticket Data

Say you want a system that automatically collects support tickets so you can measure resolution time by customer segment.

The naive build: a cron job every hour that calls the Zendesk API for tickets updated in the last hour, upserts them into a Postgres table keyed by ticket ID, and joins to a customer table on email.

Here is what will actually go wrong, in the order it will happen:

Week one, pagination. The first backfill pulls 100 tickets, which is the default page size, and stops. The table looks populated so nobody checks. Fix: always assert that you consumed every page, and log the total count against the source's own count endpoint if it has one.

Week three, clock skew. "Updated in the last hour" uses your server's clock and the API filters on its own. A few seconds of drift plus a job that starts late means tickets fall through the gap. Fix: overlap your windows. Pull the last 90 minutes every 60 minutes and rely on the upsert to deduplicate. Overlap is nearly free and closes an entire class of bug.

Week six, schema drift. Zendesk adds a custom field, your insert has a fixed column list, and the new field is silently dropped. This is not a failure, it is a slow loss. Fix: store the raw payload as JSON alongside your parsed columns. Storage is cheaper than a re-backfill, and when someone asks a question your schema did not anticipate, the answer is already sitting in the raw column.

Month three, the join breaks. Tickets arrive with the email of whoever wrote in, which is often not the billing contact you keyed your customer table on. Roughly ten percent of tickets fail to join and quietly vanish from every segment chart. Fix: never inner join in the reporting layer without also counting what fell out. An unmatched-rows count that someone actually looks at is worth more than most monitoring.

Month six, the real problem. You have clean resolution times by segment, and the numbers do not explain anything. The reason enterprise tickets take four days is that three of those days are spent waiting on an engineering escalation that lives in Linear, and the decision to prioritise it was made in a Slack thread. The data you automatically collected was real, complete and insufficient, because the causal information was never structured data in the first place.

That last failure is the important one, and it is not a bug in the pipeline.

When the Simple Answer Breaks

Automatic collection works cleanly when the data is already structured, already has a stable identifier, and already means the same thing in every row. Three situations break it.

Unstructured evidence. Decisions, context and reasons live in messages, email threads, call notes and documents. You can collect these files automatically, but collecting them does not make them queryable. A pipeline that dumps Slack exports into object storage has automated the archiving, not the answering.

Semantic drift across sources. Your CRM says a customer is "active". Your billing system says "active". They do not mean the same thing: one means an open opportunity, the other means a non-cancelled subscription. Automatic collection dutifully brings both in, and now you have two columns named the same thing with different truth. No amount of pipeline engineering fixes this; it is a definitions problem that requires a human to make a ruling and write it down.

Data that is expensive to collect and rarely used. Every collector is a permanent maintenance obligation. Source APIs change, credentials expire, rate limits tighten. A rough test: if a dataset will not be queried at least weekly, or is not required for compliance, the ongoing cost of keeping its collector alive probably exceeds the value of having it pre-collected. Pull it on demand instead.

A Practical Build Order

1. Write down the three questions the data must answer. If you cannot, stop.
2. Check where each answer already lives. Often at least one is a query away, not a pipeline away.
3. Pick push or pull per source, and add the reconciliation pull if the data is financial or regulated.
4. Store raw payloads alongside parsed columns from day one.
5. Add a freshness assertion per destination table before you add a second source.
6. Add an unmatched-rows count on every join used in reporting.
7. Only then add sources two through twenty.

Steps 4, 5 and 6 are what separate a system that survives a year from one that quietly rots. They are also the steps that get cut when the project is running late.

Collection Is Not the Same as Access

The reason step one matters is that the finished pipeline still leaves you with a table and a person who has to write SQL against it. For a data team that is fine. For the operations manager who wanted to know why refunds spiked, it is another queue.

This is where working directly across connected tools changes the shape of the problem. Instead of collecting everything into a warehouse first and querying second, you can ask the question against the live sources: the database, the CRM, the billing system, and also the Slack thread and the email that explain what happened. The tools that connect only to databases and modelled sources cannot see that last category, because a sentence in a message was never a row anywhere. Skopx connects to nearly 1,000 SaaS tools plus direct database connections and answers with citations back to the source, and it can turn a recurring question into a small internal app that reads live and shows the same view to everyone who needs it.

Build the pipeline when you need history, aggregation over time, or a system of record. Skip it when what you actually needed was an answer.