A distributed system designed to fail safely
Two hosts, one writer, and an irreplaceable database: an ownership lease with fencing, a standby that refuses to push backups, a ~20-failure-mode threat model, and a failover drilled live in both directions.
The challenge
One of my pipelines accumulates something that cannot be re-bought: longitudinal change history. Current state can be re-pulled from public sources any day; the record of what changed and when can only be collected as it happens, and it is the entire competitive moat of the product it feeds. The system itself is a weekly batch job — if the runner is down mid-week, nothing downstream notices. So this is explicitly a data-durability problem, not an availability problem, and the design keeps saying so: the expensive machinery guards the history, and almost nothing guards uptime, because uptime is not what's at stake.
Start from the threat model, not the tooling
Before any architecture: a written threat model of roughly twenty failure modes — hardware, software, human, malicious, environmental, third-party — each with likelihood, impact, the current control, and the honest gap. The two that actually get people are the meta-failures: backups that silently stop working, and backups that were never restorable in the first place. Both are boring, both are common, and both got named as threats before either could happen. The plan that fell out has RPO and RTO stated per phase, so at every stage there's a written answer to 'how much can we lose, and how long can recovery take?' rather than a feeling.
The dangerous failure is the one that looks like normal writes
Physical corruption is the easy case — checksums catch it, a backup fixes it. The dangerous case is logical corruption: a bad deploy or a fat-fingered fix writing valid-but-wrong data. It passes every storage-level check, and a rotating backup faithfully copies it into the backups — wait long enough and every surviving copy is poisoned. The answer has three legs: detect fast (integrity and anomaly checks on every run, so the window shrinks), retain deep in a form that can't be poisoned or deleted (versioned, immutable object storage on a grandfather-father-son ladder), and recover to a point in time (continuous WAL archiving), so a known-good moment before the bad write always exists to land on.
A second box is a second writer
Adding a standby host sounds like pure safety. It isn't: a second machine is a second potential writer, and an ungated standby coming up on the code it went down with would quietly fork the history — the exact asset all of this protects. So the failover is built around a single-writer ownership lease with fencing: the primary heartbeats a lease and stops writing if it can't renew within thirty minutes; a standby may only seize a lease older than sixty. Fence-before-takeover is the safety property — by the time a standby is allowed to take over, the old primary has provably been silent for at least the difference. The standby must restore from the off-site backup before claiming the lease, and refuses the takeover entirely if it can't. Drilled live in both directions; takeover completes in about thirty seconds. And said precisely: this was not built for uptime — there is still no user-facing service to keep up. It is data durability wearing an availability shape.
The standby that refuses to help
The design decision I'd defend hardest looks like a bug report waiting to happen: the standby deliberately declines to push off-site backups. The off-site copy is the exact object a failover restores from. A helpful standby pushing its own database there would make a stale replica the newest restore point — so a real failover would 'restore' weeks-old data and report success. Single writer for the database means single writer for its backups. Each box still keeps unconditional local snapshots as its own safety net; the shared restore path has exactly one author.
Verify the artifact, not the signal
The sharpest lesson came from monitoring, not storage. A deploy health check watched HTTP status codes — and the hosting platform returns 200 for any path, including pages that don't exist. The check structurally could not fail: it reported nineteen of nineteen pages working while the site had been undeployed for four days. It was rebuilt to verify the artifact itself — the page must carry its own expected state, the CSV must begin with its own header — and re-proven both ways: real content passes, a bogus deploy fails. The restore drill follows the same principle: a backup is fiction until a restore of it has actually run, so proving restores is a scheduled job, not a hope.
The result
An irreplaceable dataset with immutable versioned history off-site, point-in-time recovery, integrity gates that stop corruption aging out the good copies, and a two-host failover that cannot fork the data it exists to protect — with the deliberate gaps written down next to the protections, because an engineer who documents what isn't protected is the one you want examining what you've got. This is the reliability thinking that ships with every build I deliver: not the maximum amount of infrastructure, but a written answer to what can be lost, and proof the recovery path actually works.