Keeping it running is a job of its own.

Anything deployed this way should already have a Prometheus target, an Uptime Kuma monitor and ideally a Grafana dashboard before it ships. That's what the next section is about.

This website ships the same way, through one pipeline that deploys two different kinds of artifact from the same monorepo:

git push → GitHub webhook → Jenkins
  → docs: mkdocs build --strict → rsync to the internal proxy
  → webpage: Docker build → Harbor push → Compose deploy on the public host
  → health check both projects
  → Telegram message per project, or one on failure
  → agent workspace cleaned either way

The decisions behind it:

• The docs deploy with rsync, not as a container. The docs site is static HTML built with mkdocs --strict, so a broken link fails the build. Packaging it as an image would add a trip through Harbor for nothing: Caddy already serves files directly, so Jenkins builds them and copies them over as a deploy user that can only write to that one directory.
• The website deploys as an image. It builds from apps/webpage, is pushed to Harbor tagged with the short commit and build number, and redeploys with Docker Compose on the public host.
• Both projects redeploy on every push. I used to detect which paths had changed. It kept breaking on Jenkins' first builds and replays, in ways that were easy to miss. Both builds are small, so rebuilding everything every time is simpler and safer.
• Telegram is the only notification channel. One message per deployed project on success, and one message with the failed stage, the commit and a console link on failure. No email, and no dashboard I have to remember to check.
• The workspace is always cleaned. cleanWs() runs in post { cleanup {} } whatever the result, so a failed run never leaves half-built files for the next one to trip over.

Five components, each with one job:

Prometheus scrapes every 15 seconds. Targets that change by host or VLAN come from file_sd_configs JSON files, one per zone, so adding a machine means editing a target file instead of the main config. Besides node_exporter, it scrapes cAdvisor on the Docker hosts (with some high-cardinality series dropped before storage so the series count doesn't explode), pve_exporter for the cluster, postgres_exporter for both databases, and the native metrics of Authentik, Zitadel and Grafana.

Grafana Alloy ships logs from every host to Loki, which keeps 30 days on local storage with the TSDB schema. Loki's ruler already points at the same Alertmanager, so log-based alerts will use the same path once I write them. Right now there are none.

Alertmanager and Uptime Kuma both notify through Telegram, but they answer different questions. Alertmanager fires on what the metrics say: high CPU, a target that stopped answering, a database out of connections. Critical alerts wait less and repeat sooner than warnings, and a host's warnings are muted while a critical alert for that host is firing, so one failure doesn't send me five messages. Uptime Kuma ignores metrics completely and only asks whether an endpoint responds. I want both: one catches slow internal degradation, the other catches "it's simply down" even when every internal metric looks fine.

A few rules keep the stack from sprawling as it grows:

• Every new host gets node_exporter and Grafana Alloy. No exceptions. That's the baseline before anything else is installed.
• cAdvisor only where Docker runs. No point scraping container metrics on a host without containers.
• Uptime Kuma only where reachability matters to someone. Not every internal service needs an outside check, only the ones people or clients actually use.
• An alert exists only if someone should act on it. An alert with no action behind it is noise, and noise gets muted a week later.
• Firewall rules follow the scrape paths. Metrics don't get a shortcut. Prometheus reaching a database exporter is still an explicit, narrow rule.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no

The last step, node_exporter and Grafana Alloy on every host, is what keeps the monitoring above trustworthy. If a machine isn't reporting metrics one scrape interval after it comes online, I treat it as a setup bug, not something to clean up later. One detail from the runbook worth repeating: test the key login from a second terminal before closing the first, or you can lock yourself out.

$ adduser alberto && usermod -aG sudo alberto
$ ssh-copy-id -i ~/.ssh/id_ed25519.pub alberto@<new-host>
$ ssh alberto@<new-host>        # second terminal, before closing the first
$ sudoedit /etc/ssh/sshd_config && sudo systemctl restart sshd
$ sudo systemctl enable --now node_exporter alloy
$ curl -s localhost:9100/metrics | head -n 3

Rebooting a Proxmox node for a kernel update or hardware work follows the same discipline in reverse. First I drain it: live-migrate the VMs that can move, shut down deliberately whatever can't, and check that nothing is left running. Then I reboot and watch it rejoin. One node at a time, always, because the cluster needs two of three nodes healthy to keep quorum, and the next one doesn't start until pvecm status shows the previous one fully back. pve01 has one extra catch: its kernel is pinned, because a new kernel would boot without the NVIDIA driver the GPU needs.

# on pve02
$ qm list                          # VMs still running here
$ qm migrate 105 pve03 --online
$ pct list                         # containers
$ pct shutdown 210                 # local storage, stop it cleanly
$ qm list && pct list              # nothing running: pve02 is clear
$ reboot
...
$ pvecm status | grep Quorate
Quorate:          Yes

Service defined → VLAN chosen → address and DNS assigned
  → proxy/firewall path reviewed → observability added
  → persistence risk noted → documentation updated

The two decisions that matter are which VLAN it lands in and which of the three proxies fronts it. Both follow from what the service is:

Everything after that is mechanical: the narrowest firewall rule that matches the real path (never a broad VLAN-to-VLAN allow), the host firewall, the observability agents, an Uptime Kuma monitor if people use it, and, if it stores anything, a written note that its data isn't backed up yet.

$ pvecm status
Quorum information
------------------
Nodes:            3
Quorate:          Yes

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 pve01 (local)
0x00000002          1 pve02
0x00000003          1 pve03

The runbooks I have today, by area:

The second is automation, or the lack of it. The base machine setup from earlier is still typed by hand on every new host, not run from Ansible or OpenTofu. It's consistent because I follow the same runbook every time, not because a tool enforces it, and that's a much weaker guarantee.

A smaller one: pve03 doesn't have Wake-on-LAN configured yet. The reason is about as mundane as it gets: I haven't connected a monitor to change the BIOS setting. Since it has an iDRAC for remote power anyway, it hasn't been urgent.

Both of the big ones are on the list. The full backlog (backup target, restore tests, SSO and secrets rollout, infrastructure as code) is on /roadmap.

• Ansible and OpenTofu should have come on day one. When this was two machines, adopting infrastructure as code would have cost almost nothing. With today's number of hosts and services, adding it afterwards is real, constant friction, enough that I keep putting it off. That's exactly the automation gap above. Starting early would have kept every host's configuration in sync without touching each one by hand.
• Splitting the network was worth it. Eight VLANs and their firewall rules make every change more work than a flat network would. I'd still make the same call: the isolation is worth the friction of maintaining it.

Next: /system covers what exists (the hardware, the network, and the services this pipeline deploys and this stack watches), /security covers how access is controlled, and /roadmap is the backlog, in order.