I’ve written elsewhere in this log about a backup gap on this platform that isn’t fully closed yet, a real, currently open risk, not a solved problem I’m retroactively narrating as handled. This post is the other half of that honesty: what exists because the backup story isn’t finished. Rehearsed triage flow and documentation treated as the source of truth aren’t a replacement for backups (nothing replaces backups), but they’re the compensating control that makes the gap survivable while it’s being closed, instead of just an unaddressed risk sitting quietly until it isn’t quiet anymore.
To be clear about what this post is not: it’s not a retelling of a specific outage. Nothing described here narrates an incident that already happened on this platform. It’s the general discipline, the flow that gets used if and when something breaks, written down now, deliberately, rather than improvised for the first time under pressure.
Runbooks and guides are different documents
It’s easy to lump all operational documentation into one undifferentiated pile labeled “docs,” but two of the categories in that pile do genuinely different jobs, and conflating them produces documentation that’s bad at both.
A runbook answers “what do I do when X breaks,” and it earns that name by being repeatable: the same steps, in the same order, producing the same outcome, regardless of who’s running it or how calm they are at 3am. A good runbook reads almost like a script: check this, if it’s true do that, if it’s false do this other thing. It doesn’t need to explain why each step exists, because the person following it during an incident doesn’t have spare attention for the reasoning, they need the procedure to just work.
A guide answers a different question: “how do I reason about doing X for the first time.” It’s the document you read before there’s a runbook for something, or when the runbook doesn’t cover the specific way this particular failure is showing up. A guide explains the mental model: here’s how this subsystem works, here’s what normal looks like, here’s how to tell the difference between three failure modes that look similar on the surface. It’s slower to read and slower to act on, and that’s fine, because a guide’s job is building the understanding a runbook can later compress into a fast procedure.
The mistake is writing guides when you need runbooks (nobody wants to reason from first principles while a service is down) or writing runbooks when you need guides (a rigid procedure for a novel failure mode just fails silently the moment reality doesn’t match the script). Knowing which one a given situation calls for is itself part of the discipline, and often the honest answer, the first time a particular kind of failure shows up, is that only a guide exists yet, and the runbook gets written afterward, informed by what the guide-level reasoning turned up.
A general triage flow
Underneath the specific runbooks for specific systems, there’s a shape that most failure triage on this platform follows, roughly in this order:
1. Identify the actual scope of what’s broken. Not what’s reported broken: what’s actually broken. A user-facing symptom (“the site is down”) is frequently a downstream effect of something else entirely, and jumping straight to fixing the visible symptom without scoping the real failure wastes time and sometimes makes the real problem harder to see. Scoping means figuring out which layer the failure is in before touching anything.
2. Check cluster and network state. Before assuming an application-level failure, rule out (or confirm) that the infrastructure underneath it is healthy: is the network reachable end to end, is the underlying cluster or host layer reporting healthy, is this an application problem or an infrastructure problem wearing an application symptom. Skipping this step is how people spend twenty minutes debugging application code for a problem that was really a network partition one layer down.
3. Rebuild by treating the documentation as the source of truth. This is the step that matters most for this post’s argument. When something needs to be rebuilt — a service redeployed, a piece of infrastructure recreated — the rebuild follows what’s written down, not what someone remembers doing last time. Tribal memory and muscle memory are exactly what fails under the two conditions that make disaster recovery hard: it’s been a while since anyone did this procedure, or the person who usually does it isn’t available. A rebuild driven by documentation is reproducible regardless of who’s doing it or how long it’s been. A rebuild driven by memory is a bet on a specific person’s recall being accurate under stress, and that’s a bad bet to structurally depend on.
4. Validate that the rebuild worked. Rebuilding something and confirming it looks right are not the same step, and skipping the second one is how a recovery effort quietly leaves a system in a state that appears fine and isn’t. Validation means checking the specific things that matter for that system — not just “does it respond,” but “does it respond correctly, with the data and configuration it’s supposed to have.”
5. Record what was learned back into the docs. The loop closes here or it doesn’t close at all. If the rebuild revealed that a runbook step was wrong, missing, or based on an assumption that turned out not to hold, that goes back into the documentation immediately — not as a someday task, but as part of finishing the recovery. A disaster recovery effort that doesn’t update its own instructions afterward guarantees the next person hits the exact same gap, at the exact worst possible time to discover it for the first time.
Why this is a compensating control, not a substitute
None of this makes backups optional. A rehearsed triage flow and trustworthy documentation reduce the damage from a class of failures (the ones where infrastructure needs to be recreated, reconfigured, or debugged from a degraded state), but they do nothing for the failure mode backups exist to cover: data that’s gone. If the underlying data behind a system is lost and there’s no backup to restore it from, the best triage flow in the world rebuilds an empty shell of the system that used to hold that data. Documentation tells you how to recreate infrastructure faithfully. It cannot recreate data that no longer exists anywhere.
That’s the honest boundary of what this discipline buys, and it’s worth being explicit about it rather than letting “we have good runbooks” quietly stand in for “we have backups,” because those are different sentences and only one of them is currently fully true here. What the triage flow and documentation-as-source-of-truth practice do, while the backup gap is being closed, is shrink the set of failures that turn into prolonged or repeated outages (a config gets corrupted, a host needs to be rebuilt from scratch, a service needs redeploying after a bad change) down to something that’s boring and procedural instead of something that depends on someone remembering the right sequence of commands under pressure. That’s real value. It’s just not the same value a working backup restores, and pretending otherwise would be exactly the kind of comfortable overstatement this log has tried to avoid.
Documentation earns “source of truth” by being exercised
There’s a trap specific to step three of the triage flow that’s worth calling out on its own: documentation only functions as a source of truth if it’s accurate, and the only way to know it’s accurate is to have followed it recently enough to notice where it’s drifted from reality. A runbook that was correct when written and hasn’t been touched since is a runbook that’s been silently accumulating small inaccuracies every time the underlying system changed without the doc being updated to match: a renamed config key, a step that used to be manual and is now automated, a dependency that got swapped out. None of those individually feel worth a doc update in the moment they happen. Collectively, they’re what turns “follow the documented steps” from a reliable procedure into a procedure that gets you 90% of the way there and then strands you at the one step that quietly stopped being true.
The only real defense against that drift is treating a rebuild (any rebuild, not just one forced by a real failure) as an implicit test of the documentation itself, and that’s exactly what step five closes the loop on. A routine, planned rebuild (standing up a replacement host, redeploying a service as part of ordinary maintenance rather than incident response) is a low-stakes opportunity to discover the same drift a real disaster would expose, without the time pressure of a real outage attached to the discovery. Deliberately using ordinary maintenance work this way, following the documented procedure even when muscle memory would be faster in the moment, is what keeps the documentation trustworthy enough to lean on later, when there isn’t the luxury of muscle memory being an option at all because the person who usually does this isn’t the one dealing with it.
The failure mode this whole discipline exists to prevent
It’s worth being precise about what goes wrong when this discipline is missing, because “no backups” and “no rehearsed recovery process” fail in different, specific ways rather than blurring into one generic “bad at ops” description. Missing backups fails as data loss: something that existed is now gone, permanently, and no procedure recovers it. A missing or untrustworthy recovery process fails as duration and inconsistency: the outage lasts longer than it needed to, or gets resolved differently depending on who happened to be on call, because there’s nothing written down reliable enough to follow, and the person handling it is reconstructing the procedure from memory and judgment calls instead of executing something already verified to work. Both failure modes are real risks. They’re just not the same risk, and closing one doesn’t touch the other, which is precisely why this platform is working on both in parallel rather than treating either as a stand-in for the other.