Monitoring & Observability
How I know things are working — and how I find out when they’re not.
Uptime Kuma (service health)
- Where: Docker container on the Docker VM (Desktop Host).
- Exposed as:
status.detellem.comanduptime.detellem.com(LAN-only via nginxallow/deny). - What it does: HTTP/HTTPS uptime checks against all public services (Vaultwarden, Mealie, Homepage, OpenWebUI, Plex, IT-Tools, ConvertX, Donetick, Stoat Chat, Paperless, and more). Status dashboard and alerting on downtime.
This is the “first glance” layer — if a service is reachable or not.
Centralized logging and metrics (Loki + Prometheus + Grafana)
A full observability stack runs on the Docker VM:
| Component | Role | Port |
|---|---|---|
| Loki | Log aggregation — receives logs from all hosts | Internal |
| Prometheus | Metrics storage — scrapes and stores time-series data | Internal |
| Grafana | Dashboards and alerting — visualizes logs and metrics | logs.detellem.com (LAN-only) |
All three run as Docker containers in a shared Compose stack. Grafana is the single pane of glass: I use it for historical trending, alert rules, and ad-hoc log queries.
Grafana · Loki · Prometheus
Grafana Alloy (agents on every host)
Grafana Alloy is deployed on every host in the lab:
| Host | Deployment | Ships |
|---|---|---|
| Desktop Host (Windows) | Windows service | Windows Event Logs (System, Application, Security), Plex logs, Ollama logs, host metrics (CPU, memory, disk, network) |
| Docker VM | systemd service | systemd journal, Docker container logs, host metrics |
| NAS Host | systemd service | systemd journal, Docker container logs (Pi-hole), host metrics, ZFS metrics |
| Proxy Host | systemd service | systemd journal, Docker container logs (Pi-hole), host metrics |
| nginx VM | systemd service | systemd journal, host metrics |
| Minecraft VM | systemd service | systemd journal, host metrics |
| Stoat VM | systemd service | systemd journal, Docker container logs, host metrics |
Each Alloy agent pushes logs to Loki and host metrics to Prometheus on the Docker VM. This means I can query logs from any host in one place (Grafana) without SSH-ing around.
ZFS health monitoring
The NAS Host runs zfs_exporter which exports ZFS pool health, capacity, fragmentation, and dataset-level metrics to Prometheus via Alloy. Grafana alerting rules fire on:
- Pool status DEGRADED or FAULTED
- Capacity exceeding 80% (warning) or 90% (critical)
- Fragmentation exceeding 50%
SMART disk health is monitored independently via smartd on the NAS Host.
Alerting
Grafana alerting rules cover:
- ZFS pool degradation
- Disk capacity thresholds
- Certificate expiry approaching (Let’s Encrypt)
- Service-level alerts from Uptime Kuma (via its own notification channels)
Audit system (config drift detection)
Beyond real-time monitoring, I run periodic audit scripts that SSH into all hosts and verify state:
make audit(quick): Service reachability, port availability, container status, systemd unit health. Run before sessions or after changes.make audit-thorough(full): Everything above plus config drift comparison (live nginx/keepalived/fail2ban vs repo-generated configs), firewall rule validation, backup recency checks, SSH hardening verification, router/AP state comparison, and ZFS health. Takes a few minutes; produces a full report with diffs for any drift.
This bridges the gap between real-time monitoring (Uptime Kuma, Grafana alerts) and structural correctness (is what’s deployed what the repo says should be deployed?).
Summary
| Layer | Tool | What it catches |
|---|---|---|
| Service availability | Uptime Kuma | “Is this reachable right now?” |
| Logs (all hosts) | Loki + Alloy | Application errors, auth failures, container crashes |
| Metrics (all hosts) | Prometheus + Alloy | CPU/RAM/disk trends, ZFS health, capacity planning |
| Dashboards + alerts | Grafana | Visual overview + proactive alerting on degradation |
| Config correctness | Audit scripts | “Does live state match the repo?” |