A multi-source data pipeline, built to be cloned

A public-data pipeline with connectors to 33 state and municipal registries — connector per source, validation against ground truth, and a readiness gate that decides which sources are trustworthy enough to use. Then cloned to a second market in weeks.

PythonETLPlaywright

The challenge

Public data lives in dozens of incompatible places — Socrata APIs, flat CSV dumps, XML, and ASP.NET forms that actively fight automation — and every source drifts over time. Assembling it into something trustworthy and current is mostly grunt work, which is exactly where automation earns its keep. The harder problem isn't collecting the data. It's knowing when to trust it.

The build

A connector per source type across 33 state and municipal registries, with a single source of truth for what's actually usable from each. A validation layer re-checks output against live ground truth so nothing ships unverified, and a change-detection engine flags what's new since the last run. Where sites block automation outright, a real-browser egress path handles it. Sources that throttle get a circuit breaker that rests them and escalates the backoff, because losing an endpoint costs a whole jurisdiction rather than a single request — and the health report names every resting source, so one being quietly avoided is never invisible.

Knowing which data you can trust

The part I'd build again first: a readiness gate that decides, per source, whether the data is good enough to act on. A source qualifies only after enough clean runs to arm its own volume guard, a passing data-quality scan, and a freshness check calibrated to how slowly that particular source actually publishes. The first run of that gate cleared four of eighteen sources — and that was the correct answer, not a disappointing one. Most pipelines will happily serve you stale or half-broken data with total confidence. This one refuses, and tells you which test it failed.

First real run: 4 of 18 cleared — and that was the correct answer. Most pipelines serve stale or half-broken data with total confidence; this one refuses, and tells you which test each held source failed.

Reusing the engine

The real return showed up on the second build. A different market, a different public data source, a different buyer — and the collection, normalization, change-detection, validation, and delivery layers all transferred as-is. What had taken months the first time stood up in weeks, because the engine had already been paid for. That's the argument for building this kind of thing properly once: the first one is a project, and every one after it is a configuration.

The result

Jurisdictions assembled and kept current with no manual touch, a test suite in the hundreds guarding the parts most likely to drift silently, and a reusable pattern for any 'pull data from everywhere, keep it clean, act on what changed' problem. The honest shape of the coverage, measured 2026-09-09: 33 registry connectors, 32 states holding data, 30 producing that week. Those middle numbers move week to week; one moved the day before this was written, when a state's robots policy changed and its connector correctly refused to run. How many of those are commercially productised is a separate and smaller number — the gap there is data readiness and licensing law, not engineering. Reliability is the whole point: a pipeline that fails silently is worse than no pipeline, so this one checks its own work and says so when it can't.