Keeping it running is its own piece of work.

Any service deployed this way is expected to already have a Prometheus target, an Uptime Kuma monitor, and ideally a Grafana dashboard before it ships — that expectation is what the next section is actually about.

This site — the one serving the page you're reading — is a good concrete example, because one pipeline deploys two genuinely different kinds of artifact from the same monorepo.

git push → GitHub webhook → Jenkins
  → docs: mkdocs build --strict → rsync to the internal proxy
  → webpage: Docker build → Harbor push → Compose deploy on the public-facing host
  → health check both projects
  → notify Telegram per project, or once on pipeline failure
  → agent workspace cleaned either way

A few design choices behind it, worth knowing rather than just assuming:

• Docs deploy by rsync, not by container. The docs site is static HTML; packaging it as an image just to redeploy through Harbor would add a registry round-trip for nothing. The proxy already serves files directly, so Jenkins only needs to build and copy.
• The webpage deploys as an image. It builds from apps/webpage, publishes to Harbor tagged by short commit and build number, and redeploys via Compose on the public-facing host.
• Both projects redeploy on every push, path detection or not. Path-detection logic kept breaking in first-build and replay scenarios in Jenkins, in ways that were easy to miss. Both builds are small enough that rebuilding everything every time is simpler and safer than being clever about it.
• Telegram is the entire notification path. One success message per deployed project, one failure message with the failed stage, the commit, and a console link if the pipeline breaks. No email, no dashboard I have to remember to check.
• The build workspace is always cleaned, success or failure. cleanWs() runs unconditionally in post { cleanup {} }, so a failed run never leaves half-built artifacts for the next one to trip over.

The stack behind that table splits into five pieces, each doing exactly one job:

Prometheus scrapes on a 15-second interval, and targets that vary by host or segment come from file_sd_configs JSON files rather than the main config — adding a machine is usually editing one target file, not rewriting Prometheus itself.

Grafana Alloy ships logs from every host that runs it to Loki, kept for 30 days on local filesystem storage. Loki's ruler already points at the same Alertmanager metrics alerting uses, so log-based alerting can reuse that exact path whenever those rules actually get written — right now, none exist yet.

Alertmanager and Uptime Kuma both notify through Telegram, but they're answering different questions. Alertmanager fires on what the metrics say is wrong — high CPU, a target gone silent, a database with no free connections — with critical alerts repeating faster than warnings, and a warning for a host suppressed while a critical alert for that same host is already firing. Uptime Kuma doesn't look at metrics at all; it's an outside-in check, does this endpoint actually respond. The overlap is deliberate: one catches internal degradation the other can't see, the other catches "simply unreachable" even when every internal metric still looks fine.

A handful of rules keep the whole stack from sprawling as it grows:

• Every new host gets node_exporter and Grafana Alloy. No exceptions — that's the baseline before anything else gets installed on it.
• cAdvisor only where Docker actually runs. No point scraping container metrics on a host that has none.
• Uptime Kuma only where user-facing reachability matters. Not every internal service needs an outside-in check — just the ones people, or clients, actually hit.
• Alert rules exist only for conditions someone should act on. No alert without an action behind it, or it's just noise that gets muted a week later.
• Firewall rules follow the scrape paths, not the other way around. Metrics don't get a bypass for being metrics — Prometheus reaching a database exporter is still an explicit, narrow rule.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no

That last part — node_exporter and Grafana Alloy on every single host — is what keeps the observability section above honest. A machine that isn't reporting metrics within one scrape interval of coming online is treated as a setup bug, not a later cleanup task.

$ ./bootstrap-host.sh vm-207
==> creating admin user, disabling root SSH login
==> disabling password authentication
==> installing node_exporter, grafana-alloy
==> registering Prometheus target: node-internal
[ok] vm-207 reporting metrics — 1 scrape interval elapsed

Rebooting a Proxmox node for a kernel update or hardware work follows the same discipline in the other direction: drain it first — live-migrate what can move, stop deliberately what can't — confirm nothing's left running locally, then reboot and watch it rejoin the cluster. Nodes go one at a time, never more, because the cluster needs at least two of the three healthy to keep quorum; the next one doesn't start until pvecm status shows the previous one fully back.

$ qm list --node pve02        # live-migrate anything running
$ pct list --node pve02       # stop what can't move
$ qm list --node pve02 && pct list --node pve02
(empty — pve02 is clear)
$ reboot
...
$ pvecm status | grep Quorate
Quorate: Yes

Service defined → VLAN chosen → address and DNS assigned
  → proxy/firewall path reviewed → observability added
  → persistence risk noted → documentation updated

The two decisions that actually matter are which segment it lands in and which of the three proxies fronts it — both follow directly from what the service is, not from convenience:

Everything after that placement follows mechanically: the narrowest firewall rule that matches the actual path, never a broad segment-to-segment allow; the standard observability agents; an Uptime Kuma monitor if it's user-facing; and, if it stores anything, an explicit note that the data isn't backed up yet — pretending otherwise would undercut the point of writing any of this down honestly.

$ pvecm status
Quorum information
------------------
Nodes:            3
Quorate:          Yes

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 pve01 (local)
0x00000002          1 pve02
0x00000003          1 pve03

The runbooks that exist today, grouped by what they cover:

The second is automation, or the current lack of it. Base machine setup — the bootstrap sequence covered earlier, under keeping hosts consistent — is still typed by hand on every new host, not run from Ansible or OpenTofu. It's consistent because I follow the same runbook every time, not because a machine enforces it, which is a meaningfully weaker guarantee.

One more small one: one of the three nodes doesn't have Wake-on-LAN properly configured yet. The reason is about as trivial as it gets — nobody's connected a monitor to flip the relevant BIOS setting — and since that node already has out-of-band power management, it hasn't been urgent enough to fix.

Both of the bigger ones are tracked, not ignored — the full, current backlog (backup targets, restore testing, identity and secrets rollout, infrastructure-as-code) lives on /roadmap, filed as an honest punch list rather than a wishlist.

• Ansible and OpenTofu should have started on day one, not now. Back when this was two machines, adopting infrastructure-as-code would have cost almost nothing. With the current sprawl of hosts and services, retrofitting it is real, ongoing friction — enough that I keep not getting to it, which is exactly the automation gap described above. Doing it early would have kept every host's configuration synced without touching each one by hand.
• Segmenting the network was worth it, complexity and all. Eight segments and the firewall rules that go with them make day-to-day changes genuinely more work than one flat network would. I'd make the same call again — the isolation it buys is worth the friction of maintaining it.

From here: /system covers what actually exists — the hardware, the network, the services this pipeline deploys and this stack watches. /security covers how access into all of it is controlled. /roadmap is the current backlog, in order.