Building a Static Integration Catalog for Reliability

When your integrations page is empty after every deploy, you have a reliability problem. This is the story of how we learned that lesson at Skopx, why we now ship 1,028 integrations as a committed JSON file instead of fetching from a third-party API at runtime, and how this pattern applies to any application that depends on external catalogs.

The Problem

Skopx connects to over a thousand third-party tools through an integration partner. Our integrations page displayed the full catalog of available connectors, letting users browse and discover what they could connect. The catalog was fetched at runtime from the partner's API.

This worked perfectly in development. It worked perfectly in staging. And it worked perfectly in production, until it did not.

The failure modes were varied and unpredictable:

• The partner API returned 500 errors during our deployment window
• Rate limiting kicked in during traffic spikes
• API response times spiked from 200ms to 8 seconds during partner maintenance
• Schema changes in the API response broke our rendering logic
• DNS resolution failures caused silent empty responses

Each failure had the same visible result: users visited the integrations page and saw nothing. No error message, no fallback, just an empty page. For a platform that positions itself on the breadth of its integrations, an empty integrations page is a credibility disaster.

Why Runtime Fetching Fails for Catalogs

The core insight is that integration catalogs change slowly. A new connector might be added once a week. A description might be updated once a month. The data is effectively static on any reasonable request-by-request timescale.

Yet by fetching at runtime, we were treating this slowly changing data as if it were real-time. We were paying the reliability cost of a network dependency for data that was identical across 99.99% of requests.

Caching (the second row) seems like the obvious fix, but it introduces its own problems. Cache invalidation timing, cold-start scenarios after deployments, and the complexity of cache warming all create new failure modes. We tried this approach first and still experienced intermittent empty pages during cache refreshes.

The Static File Approach

The solution was simple: fetch the catalog once, commit it as a JSON file, and ship it with the application code.

Implementation

The process works as follows:

1. A script fetches the full integration catalog from the partner API
2. The response is normalized, validated, and written to a JSON file in the repository
3. The JSON file is committed alongside application code
4. The application reads the local file at startup (or imports it directly in the build)
5. A scheduled job runs weekly to check for catalog updates and opens a PR if changes are detected

The JSON file contains the essential metadata for each integration: name, slug, category, icon URL, description, and supported actions. At 1,028 integrations, the file is approximately 850KB uncompressed, well within any reasonable bundle size budget.

The Update Workflow

Keeping the catalog current requires a deliberate update process. We use a scheduled CI job that:

1. Fetches the current catalog from the partner API
2. Compares it against the committed JSON file
3. If differences exist, opens a pull request with the changes
4. A team member reviews and merges the PR

This process adds a human review step, which turned out to be a benefit. Twice, the partner API returned malformed data that would have broken our rendering. The PR review caught both issues before they reached production.

Build-Time Validation

The JSON file passes through validation during the build process. The build fails if:

• The JSON is malformed
• Any required field (name, slug, category) is missing
• Duplicate slugs exist
• The total count drops below a threshold (protection against accidental truncation)

This validation ensures that the catalog is always complete and well-formed before deployment.

Tradeoffs

Freshness vs. Reliability

The primary tradeoff is freshness. A new integration added by our partner will not appear on our page until the next scheduled update is merged. In practice, this delay is one to seven days, which is acceptable for a catalog that users browse, not a real-time feed.

For our use case, 100% availability with slightly stale data is categorically better than 99.5% availability with real-time data. The 0.5% downtime costs more in user trust than a three-day delay in showing a new integration.

Repository Size

Adding a 850KB JSON file to the repository is a negligible cost. Over time, Git history accumulates diffs of the file, but since the catalog changes slowly, these diffs are small.

Deployment Coupling

The catalog is now coupled to deployment. Updating the catalog requires a deploy. For applications with continuous deployment pipelines, this is not a meaningful constraint. For applications with infrequent, ceremony-heavy releases, it could be a friction point.

When to Use This Pattern

Static catalogs work well when the data meets these criteria:

• Changes infrequently (less than daily)
• Is the same for all users (not personalized)
• Is critical to the user experience (empty state is unacceptable)
• Comes from a source you do not fully control (third-party API)
• Does not require real-time accuracy

Common examples beyond integration catalogs include: feature comparison tables, pricing tier definitions, supported language/region lists, connector documentation, and template libraries.

When Not to Use This Pattern

This approach is wrong for data that:

• Changes frequently (inventory levels, pricing, availability)
• Is personalized per user or tenant
• Requires real-time accuracy for correctness
• Is too large to commit to a repository (gigabytes of data)

Lessons Learned

The most important lesson was that reliability is a feature. Users do not care whether your integrations page shows 1,028 or 1,031 connectors. They care enormously whether it shows 1,028 or zero.

The second lesson was that simplicity wins. Our first attempted fix (multi-layer caching with TTL and fallback) was more complex and less reliable than the final solution (a committed JSON file). The simplest approach that meets your requirements is usually the most reliable.

The third lesson was that scheduled update PRs with human review are a feature, not a limitation. They provide a natural review point for changes from external dependencies and have caught real issues. At Skopx, the static catalog pattern has been in production for months with zero downtime incidents on the integrations page.

Applying This to Your Stack

If your application has a page or feature that depends on slowly changing external data and that feature has experienced reliability issues, consider the static file approach. The implementation cost is low (a fetch script, a JSON file, a scheduled job), and the reliability improvement is dramatic. The cognitive shift is small but important: treat slowly changing external data as deployment artifacts, not runtime dependencies.