Get in touch →
← All posts
Operations2026.05.27

Back to Deploying Everything, Every Run

The optimization looked obviously correct going in: don’t redeploy things that didn’t change. A pipeline with several independently deployable pieces was spending time and doing work on every run redeploying pieces whose files hadn’t been touched since the last run. Path-based detection is a well-known, widely used pattern for exactly this: diff the current run against some prior reference point, look at which paths changed, and only trigger the deploy steps whose paths show up in that diff. It’s a standard enough idea that CI platforms build native support for it, and third-party actions exist specifically to make it more robust than a naive diff.

We built a version of it: on every run, compute what changed relative to the last known state, and only run the deploy step for a component if its paths appeared in that diff. Everything else that hadn’t changed got skipped. On a normal day, with a normal incremental commit history, this worked exactly as advertised: a change to one component redeployed that component and left the others untouched, and the pipeline got measurably lighter for it.

Where the diff has nothing to compare against

The failure mode showed up on the runs that didn’t have a normal incremental history to diff against. A fresh checkout, or a re-run in a context with no prior recorded state, has nothing to compare the current tree to. There’s no “last known state,” there’s just the current state, full stop. And a diff against nothing doesn’t produce “everything changed,” it produces nothing to report as changed, because there’s no baseline to compare against in the first place. The path-detection logic asked “which paths differ from last time,” got back an empty or meaningless answer, and correctly-by-its-own-logic concluded that nothing needed to be deployed.

That’s the trap in path-based triggers generally, not just in whatever specific diffing logic we’d built: the mechanism assumes a meaningful prior reference always exists, and quietly does the wrong thing the moment that assumption breaks. This isn’t a hypothetical edge case unique to our setup: it’s a documented characteristic of how git-diff-based path filtering works. GitHub’s own documentation on path filters describes how the diff a workflow trigger evaluates against is computed differently for pushes to new branches versus existing ones, and carves out explicit behavior for large diffs and diff timeouts specifically because the underlying comparison can silently fail to represent “everything actually changed” in exactly the situations you’d least expect it to. Third-party tools built around the same pattern, like the paths-filter action, document their own explicit handling for the case where there’s no common ancestor or no previous commit to diff against — precisely because leaving that case unhandled means the filter can’t tell “nothing changed” apart from “there was nothing to compare.”

Our version didn’t have that explicit handling. It treated an empty diff result as equivalent to “no changes,” full stop, whether that empty result came from a genuinely unchanged path or from having no baseline at all. On a fresh checkout or a state-losing re-run, every component looked unchanged, and the pipeline correctly, by its own internal logic, deployed nothing.

Path-detection branches on a diff that assumes a baseline; always-deploy has no such assumption to break.

The revert

The fix we actually shipped wasn’t a smarter diff, a special-cased “no baseline” branch, or a fallback path that deploys everything only when the diff computation looks suspicious. Those are all real options, and there are mature tools that implement exactly that kind of guard. We went a different direction: rip out path detection entirely, and unconditionally deploy every component on every run. (This is the containerized side of the two deploy mechanisms this infrastructure runs; a static site never had this failure mode to begin with, for reasons covered in Two Deploy Paths, Two Artifacts.)

That’s a real trade, not a free win. Every run now does strictly more work than the path-detection version did on its best day — components that haven’t changed in weeks still go through a full deploy cycle every time the pipeline runs. On a small setup that redundant work costs a few extra minutes and some extra load on the deploy target, not a real capacity problem, so the trade was worth making here. On a larger fleet with expensive or slow deploy steps, “always deploy everything” would be a much heavier hit, and the smarter-diff route, with an explicit, tested fallback for the no-baseline case, would earn its complexity back.

What made this an easy call for us specifically was what the failure mode actually was: not a slow pipeline, but a silently correct-looking one. A run that skips deploying something because it “detected no changes” produces no error, no failed step, nothing that looks worth investigating: it just quietly doesn’t ship a component that should have shipped. That’s a much worse category of problem than wasted compute time. A pipeline that’s slow but honest about what it did is something you can live with and optimize later. A pipeline that’s fast but wrong in a way that looks like success is something that erodes trust in every other green checkmark it’s ever produced, because now every “deploy succeeded, nothing to do” result carries a question mark that wasn’t there before.

Why this was hard to notice at first

Part of what made this worth writing up is how unremarkable the failure looked while it was happening. There was no crash, no failed step, no red X anywhere in the pipeline’s output. Every run reported success — because by the pipeline’s own internal accounting, it had succeeded: it correctly computed a diff, correctly found no matching paths in that diff, and correctly skipped deploy steps that its logic said weren’t needed. Every individual decision in the chain was locally correct given its input. The only thing wrong was the premise feeding the first decision — that a meaningful diff existed to compute at all — and nothing downstream was positioned to question that premise, because by the time the “which paths changed” question got asked, the pipeline had no way to distinguish “compared against a real baseline and found zero changes” from “had no real baseline to compare against.”

That’s a useful shape to recognize in general, past this specific pipeline: a chain of individually correct steps built on an unexamined assumption produces a result that looks exactly like success. There’s no step where the logic visibly breaks, because nothing in the chain breaks: the whole thing behaves exactly as designed, on an input the design never accounted for. Catching that kind of gap usually isn’t about better error handling in any one step. It’s about someone asking, before trusting the mechanism, what its unstated assumptions are and what happens when reality doesn’t hand it one.

What actually surfaced this

The way this got noticed wasn’t through the pipeline flagging anything — it noticed nothing, by design, as described above. It got noticed the ordinary way this class of problem usually does: someone expected a deployed change to be live and it wasn’t, went looking for why, and traced it back through the pipeline’s logs to a run that had reported success while doing nothing. That’s a slower, more manual detection path than a good pipeline should rely on, and it’s worth naming as its own gap: a healthy deploy pipeline would ideally make “deployed nothing” distinguishable from “deployed, no changes needed” in its own output, rather than requiring an operator to notice a mismatch between expected and actual state from the outside. The always-deploy revert fixes the specific false-negative failure mode. It doesn’t, on its own, fix the larger habit of trusting a green pipeline run without an independent way to confirm what shipped, and that’s a separate improvement still worth making regardless of which deploy strategy is running underneath it.

What we’d reconsider

If path detection comes back, it isn’t going back in as a bare diff-and-skip. The actual gap wasn’t the idea of skipping unchanged work: that’s a legitimate optimization plenty of serious CI setups run safely. The gap was treating “no diff to compare” and “diff shows no changes” as the same signal, when they aren’t. A version worth trusting would need an explicit, separate check for whether a valid baseline exists at all, with a hard rule that the absence of a baseline forces a full deploy rather than an empty one, the same shape of guard that dorny/paths-filter documents for the no-common-ancestor case. Until that guard exists and is actually exercised by a test, deploying everything, every time, is the version of this pipeline we trust.

Sources checked while writing this: GitHub’s workflow syntax documentation describes how path-filter diffs are computed and where that computation intentionally gives up (large diffs, diff timeouts); the dorny/paths-filter README documents its explicit handling for missing-baseline comparisons. Our own pipeline is not GitHub Actions and doesn’t use that action — the underlying failure mode (a path diff with no baseline to compare against) is generic to the pattern, not specific to either tool.

The broader lesson we took from this isn’t “path-based triggers are bad”: they’re a real, widely used optimization, and the tools that implement them well clearly put real thought into the edge cases that make them safe. It’s that adopting an optimization built around a diff means adopting responsibility for every case where that diff can fail to represent reality, and that responsibility doesn’t go away just because the common case works fine in testing. If we bring path detection back, the fallback-on-missing-baseline behavior needs to be there from day one, not discovered the same way this gap was — after something quietly didn’t ship.

← All posts