Public Data & Data Pipelines
Collecting public data is close to a solved problem now — tools generate scrapers on demand. What still isn't solved is joining it. One public record tells you a license was issued; joined to the business-filing registry it tells you the company is four months old, who signed for it, and whether it's in good standing — which is the difference between a name on a list and a reason to call. Those joins are where the value is, and they're where this work quietly goes wrong: match too loosely and you merge two businesses into one confident, wrong row that nobody catches.
What you get
Joins & enrichment
The part that creates the value: licence records matched to business filings, addresses normalized to a postal standard so they actually join, and entity resolution done on a dual key — because matching on name alone produces collisions and matching on address alone returns the registered agent. Ambiguous matches are marked as candidates, never merged.
Source discovery & assessment
What public data exists about your market, how current it really is, what the terms allow, and — stated honestly — when the answer is that there isn't anything worth having.
The pipeline, and proof it still works
A connector per source, normalization into one shape, validation against live ground truth, and change detection. Built to run unattended and to tell you when it can't — a pipeline that fails silently is worse than no pipeline.
How I work
This isn't a side skill. I've built this engine across thirty-plus state jurisdictions — with cross-registry joins, a readiness gate that decides per source whether the data is trustworthy enough to act on, and a firm line about what never gets appended — then reused the whole thing for a second market in weeks. The reuse is the point: the first build is a project, every one after it is a configuration.
FAQ
Does my industry even have public data worth using?
If it's licensed, permitted, inspected, or registered by a state or county, almost certainly yes. Whether it's worth collecting is a different question — volume, freshness, and how well it identifies a real buying moment all matter. That's the first thing an audit answers, including when the answer is no.
Isn't this just scraping?
Scraping is the commodity half — there are tools that generate a scraper for you now. The work that still takes judgment is joining records across registries that were never designed to be joined, and knowing when the result is trustworthy. A loose match merges two companies into one row that looks perfectly fine and is wrong, and nothing downstream will flag it. That's the part I build carefully.
Do you append phone numbers and email addresses?
No — and that's deliberate. Appending personal contact data is a different business with a different risk profile, and it's the thing that gets a data product into trouble. I join public records to other public records: a licence to a business filing, a filing to a normalized address. Where a state publishes business contact details in the record itself, that's already public and comes along; nothing is bought from a broker or scraped from a personal profile.
Is it legal to use?
Public records are generally usable, but not universally — some states restrict commercial solicitation sourced from their rosters, and individual datasets carry their own terms. Those terms get checked per source before anything is built on them, and any that don't clear are excluded and named.
Why would I need you again after it's built?
Because sources rot. Government portals migrate, drop fields, and change URLs — sometimes within days — and an unmaintained pipeline dies quietly while appearing to work. Most clients keep a maintenance retainer for exactly that reason.
Can you build the dashboard and internal tools too?
Yes — the pipeline and the interface that surfaces it, built together so the data is actually usable rather than just collected. Internal tools and admin dashboards are part of the same work.