Nothing has been lost. Nothing has failed. No restore has been attempted and come back empty. I want to say that plainly up front, because the rest of this post is uncomfortable in a specific way, and it would be dishonest in the other direction to dress it up as more dramatic than it is. This is a post about a gap that exists right now, today, that I know about, and that I have not closed. It’s a currently-tracked, admitted risk: the kind of thing that’s easy to keep meaning to get to, and worth writing down precisely because writing it down is a form of not letting that keep happening quietly.
The gap: production data exists without a verified, tested, off-node restore path. There is a copy job. There is not a demonstrated way to take that copy and rebuild a working system from it, on hardware that isn’t the machine currently holding the original.
Why “I have data” and “I have a backup” are different claims
It’s tempting to treat “the data is copied somewhere” as equivalent to “the data is backed up,” and it isn’t. A backup is a claim about the future: specifically, a claim that when something goes wrong with the original, there’s a path from the copy back to a working state. “The data is copied somewhere” is a claim about the present: right now, bytes exist in two places. Those are genuinely different properties, and the gap between them is exactly where backup strategies quietly fail without anyone noticing until the moment they’re needed.
The gap has a few different ways to hide. A copy job can be running and silently failing partway through: writing a partial file, or stopping on an error that gets logged somewhere nobody’s watching. A copy can complete perfectly and still be useless, because what’s on the other end isn’t sufficient to reconstruct a running system: maybe it’s the data files but not the configuration that makes sense of them, or a database dump taken mid-write that’s internally inconsistent in a way that only shows up when you try to load it. A copy can be both complete and consistent and still fail at restore time for reasons that have nothing to do with the data itself: permissions that don’t translate to different hardware, an assumed directory layout that isn’t there, a dependency on something specific to the original machine that nobody thought to write down because it was never tested elsewhere.
None of those failure modes are exotic. They’re the ordinary, boring ways backups fail, and the only thing that rules all of them out at once is doing the restore: on different hardware, from nothing, and checking that what comes out the other end actually works. Until that’s happened, “I have a copy” and “I have a backup” are not the same sentence, no matter how confident the copy job’s last log line looked.
Why untested is functionally the same as absent
The uncomfortable conclusion this leads to is that an untested backup and no backup at all are closer to each other, in terms of what you can rely on, than either one is to a tested backup. That sounds like an overstatement until you sit with what “untested” means: it means the restore path has an unknown probability of working, and that probability has never been measured. It could be high. It could also be low, for reasons that are invisible until the moment you need it, and the moment you need it is, by definition, the worst possible moment to discover which one it was. A known gap you can plan around. An assumed safety net that isn’t actually there is a much worse position, because it changes what other risks feel acceptable to take on, based on a guarantee that hasn’t been earned yet.
That’s the honest reason this gap is worth naming publicly rather than sitting on quietly until it’s fixed: right now, decisions about what else is acceptable to risk on this infrastructure are implicitly leaning on a restore capability that hasn’t been demonstrated to exist. Naming that gap explicitly is the only way to make sure it’s weighed as a gap, rather than silently treated as covered ground.
What closing this requires
Closing this isn’t a bigger version of the copy job that already exists. Three things have to be true, and none of them are optional:
The copy has to live somewhere genuinely independent of the original: off-node in a real sense, not just a different disk in the same machine, and not just a different machine that shares a failure domain with the original (same power, same network segment, same physical location), in a way that means whatever takes out the original has a decent chance of taking out the copy too.
The restore has to actually be exercised, not assumed. That means taking the off-node copy and rebuilding a working system from it somewhere that isn’t the original: on a genuinely separate machine, ideally one that doesn’t already have the answer sitting on it from some other route. A restore test that quietly falls back on state still present on the original machine hasn’t tested anything.
And it has to be exercised more than once, on a cadence, not as a one-time proof that then goes stale. Configuration drifts, data grows, dependencies change: a restore that worked six months ago is evidence about six months ago, not about today. This is the part that’s easiest to let slip, because a restore test that passed once feels like it settled the question permanently, when it only settled it for that moment.
None of the three is exotic engineering. All three take real, deliberate time, which is exactly why this is the kind of gap that survives quietly for longer than it should: there’s always something more urgent-feeling than proving a thing you hope you’ll never need.
Why a copy on different hardware isn’t automatically “off-node”
One trap worth naming specifically, because it’s the one most likely to produce false confidence: having a second copy on a second machine is not the same thing as having genuinely independent off-node storage if that second machine shares enough of the first one’s failure domain. Same power circuit, same network switch, same physical room, same administrative account with access to both: any of those is a path by which whatever takes out the original could plausibly take out the copy at the same time, or shortly after. A copy that dies alongside the original was never really a second copy in the sense that matters; it was one failure domain with two disks in it.
Independence, in the sense a real backup needs, means asking specifically what has to go wrong for both the original and the copy to be unavailable at once, and being satisfied that the answer is a meaningfully different, less likely event than what would take out the original alone. That’s a higher bar than “it’s a different piece of hardware,” and it’s the bar this setup doesn’t yet clear: the honest answer right now to “where does the off-node copy live, and what does it not share with the original” isn’t settled enough to call the gap closed.
The trap of treating this as always next on the list
Part of why this kind of gap survives is a specific, familiar shape of self-deception: it never fails the “is this urgent” test on any given day, because on any given day nothing has gone wrong yet, and the actual cost of not having a tested restore path is zero right up until the one day it isn’t. That asymmetry (every day of delay looks free, until suddenly one day it very much isn’t) is exactly what lets genuinely important, non-urgent work get quietly outcompeted by whatever feels more pressing this week, indefinitely. There’s no natural deadline pressure pushing back, because the risk doesn’t announce itself with a ticking clock. It just sits there, correctly modeled as “someday this could matter enormously” and incorrectly treated, in practice, as “not today.”
Writing this down is a small, specific countermeasure to that pattern. A gap that only exists as a private, hazy sense of “I should get to that” is easy to keep deprioritizing indefinitely, because there’s no fixed artifact forcing a comparison between today’s date and whatever this was supposed to have been fixed by. A gap that’s dated, published, and specific about the three conditions that close it is a different kind of thing: it’s a claim with a shape that can be checked against, later, and found either honored or not. That doesn’t fix the underlying asymmetry between how urgent this feels day-to-day and how much it would matter on the day it’s tested. It does make it harder to keep quietly deferring without at least noticing that’s what’s happening.
Why publish this instead of waiting until it’s fixed
The honest version of a status update on this kind of risk is not “everything’s covered,” said before it’s true. It’s this — a clear statement of what’s actually in place, what isn’t, and what specifically needs to happen to close the distance between them. That’s a legitimate thing to publish before the fix lands, not a confession that should wait for a tidier ending. A gap that’s written down, dated, and specific about what closing it requires is a gap that’s much harder to quietly forget about than one that only exists as a vague background worry. This post is that: the risk model, named, with the three concrete conditions that close it — off-node storage that’s genuinely independent, a restore that’s actually been run, and a repeat cadence so that proof doesn’t go stale. Whichever of those lands first, it’ll be a shorter, better follow-up than this one.
In the meantime, Disaster Recovery Before Real Backups covers the compensating control that makes this gap survivable while it’s still open — rehearsed triage and documentation-as-source-of-truth, which reduce the damage from a failure without pretending to be a substitute for the restore path this post is about.