Observability: Prometheus, Grafana and the nightly load checks

SupaCloud exposes a single Prometheus /metrics endpoint, ships a ready-made Grafana dashboard and alert-rule file, and runs a nightly real-backend load check that gates p95 latency against a committed baseline. This page wires all three so you can see request rate/errors/latency, DB-pool pressure and dispatch health, and catch a performance regression before it reaches users.

Scrape /metrics with Prometheus

The server exposes GET /metrics on its root router (not under /api), so it is outside the API auth stack and the governor rate-limit layer. It renders Prometheus text exposition (text/plain; version=0.0.4) and does DB + Docker work per request, so do not scrape it more often than you need.

Lock it down on a public origin

/metrics carries instance-wide AI spend plus task and container gauges — business telemetry an anonymous caller must not read. Set SUPACLOUD_METRICS_TOKEN (env or the OpenBao supacloud/app map) and the endpoint requires Authorization: Bearer <token> with a constant-time compare; any missing or wrong token is 401. While the token is unset the endpoint stays open so an existing unauthenticated scrape config keeps working — you opt into the lockdown by setting the token and mirroring it in the scrape config below.

1. (Recommended) Set a metrics token so the endpoint is not world-readable:

   SUPACLOUD_METRICS_TOKEN=<a long random string>

   See the environment-variable reference for where your deployment reads env, and the OpenBao path under Use the OpenBao secret backend.

2. Point Prometheus at the server. The server listens on SERVER_PORT (default 8080). Add a scrape_config — include the bearer token only if you set one:

   scrape_configs:
     - job_name: supacloud
       metrics_path: /metrics
       scheme: http            # https if you terminate TLS in front of the server
       # Drop this block entirely if SUPACLOUD_METRICS_TOKEN is unset:
       authorization:
         type: Bearer
         credentials: <the same value as SUPACLOUD_METRICS_TOKEN>
       static_configs:
         - targets: ["supacloud.example.com:8080"]

3. Reload Prometheus and confirm the supacloud target is UP on the Targets page, then check a series renders, e.g. supacloud_http_requests_total.

What the endpoint exposes

RED — rate, errors, duration (per route)

A middleware records every HTTP response under the matched Axum route template (e.g. /api/runs/{id}/events), never the raw path — so the label space is bounded by the route table plus a single {unmatched} fallback and never grows with tenant ids or query strings. Three series carry the RED signal, labelled by method, route and status:

Metric	Type	Signal
supacloud_http_route_requests_total	counter	Rate — requests served
supacloud_http_route_errors_total	counter	Errors — responses with status ≥ 500
supacloud_http_route_duration_seconds	histogram	Duration — latency buckets (_bucket/_sum/_count)

The duration histogram uses fixed le buckets from 5 ms to 60 s, so histogram_quantile() gives you a real p95 per route. Process-wide counters (supacloud_http_requests_total, supacloud_management_requests_total, supacloud_webhook_requests_total, supacloud_task_launch_requests_total, supacloud_scheduler_ticks_total) and business gauges (supacloud_tasks_total{status}, supacloud_ai_cost_month_usd, supacloud_agent_containers_active) round out the snapshot.

DB pool saturation

The PostgreSQL connection pool is exported so you can see pool pressure before it turns into request latency:

Metric	Type	Signal
supacloud_db_pool_connections{state="active|idle|open"}	gauge	Connections by state
supacloud_db_pool_max_connections	gauge	Configured pool maximum
supacloud_db_pool_saturation	gauge	active / max ratio (0–1)

A supacloud_db_pool_saturation riding near 1.0 means requests are queueing on the pool — scale the pool or shed load.

Slow-query signal (logs, not a metric)

Slow statements surface in the server logs, not on /metrics. The pool is configured to log any statement slower than a threshold at WARN level via sqlx’s slow-statement logging. The threshold is SUPACLOUD_SLOW_QUERY_LOG_MS (default 250 ms; a zero or unparseable value falls back to 250). Grep the server log for these WARN lines, or ship the logs to Loki, to find the queries behind a route p95 spike.

Dispatch operability + alert state

The delivery engine’s health is exported so the same Prometheus can alert on a stalled scheduler or a growing backlog (the human-facing view of this lives in Operate the Delivery Engine):

Metric	Type	Signal
supacloud_scheduler_seconds_since_tick	gauge	Seconds since the last scheduler tick (-1 before the first tick)
supacloud_backlog_queued_items	gauge	Backlog items queued awaiting dispatch
supacloud_backlog_oldest_queued_age_seconds	gauge	Age of the oldest queued item
supacloud_dispatch_alert{kind="…"}	gauge	Evaluated alert state per kind (1 = breaching, 0 = ok)

The supacloud_dispatch_alert kinds (tick_sla, queue_age, error_spike, budget_80, weekly_window_low) are evaluated server-side against named-const thresholds, so the gauge already tells you whether a rule is breaching — you do not have to re-derive the thresholds in PromQL.

Wire the Grafana dashboard and alert rules

Both artifacts are committed in the repo under observability/ — import them as-is.

1. Import the dashboard. In Grafana, Dashboards → New → Import, and upload observability/grafana/supacloud-s4-dashboard.json. Pick your Prometheus datasource when prompted (the dashboard exposes a datasource template variable). It ships five panels: HTTP Route p95, HTTP Route Throughput, HTTP Route 5xx, DB Pool Saturation, and Dispatch SLA + Queue.

2. Load the alert rules. observability/prometheus/supacloud-s4-rules.yml is a Prometheus rule group. Reference it from your prometheus.yml (rule_files:) or load it into Grafana-managed alerting. It defines four rules:

   Alert	Fires when	Severity
   SupaCloudHttpRouteP95High	a route’s p95 > 2 s for 15 m	warning
   SupaCloudHttpRouteErrors	sustained 5xx rate > 0.05/s for 10 m	warning
   SupaCloudDbPoolSaturation	supacloud_db_pool_saturation > 0.85 for 10 m	warning
   SupaCloudDispatchTickSla	supacloud_dispatch_alert{kind="tick_sla"} == 1 for 5 m	critical

3. Attach a contact point. Route the severity / stream: s4 labels to your on-call channel so the critical tick-SLA alert pages and the warnings notify.

These are starting thresholds

The 2 s p95 and 0.85 saturation thresholds in the rule file are sane defaults, not a per-deployment SLO. Tune them in your copy of the rules to match the latency your hardware and traffic actually warrant.

The nightly E2E + k6 load checks

A scheduled job (.forgejo/workflows/e2e-nightly.yml, cron 0 6 * * *, also runnable on demand via workflow_dispatch) boots a real backend stack — server, Postgres, web — seeds it, and runs two checks against it:

• Playwright real-backend E2E (npm run test:e2e:nightly) drives the UI against the live server, not mocks.
• k6 hot-endpoint budgets (scripts/perf/run-k6-nightly.sh → k6/s4-hot-endpoints.js) load the six hottest read paths and gate their p95.

The k6 scenarios cover the boards and the two hot non-board paths: tasks_board, projects_board, runs_board, run_events, dispatch_tick (/api/operator/v1/dispatch/metrics) and mcp_gateway (a tools/list call to /api/mcp). Each runs at K6_VUS virtual users for K6_DURATION (the nightly uses 4 VUs for 2 m).

The p95 budget gate

Each scenario writes a k6 summary that scripts/perf/check-k6-budget.mjs compares against a committed baseline in k6/baselines/:

• With K6_REQUIRE_BASELINE=1 (the nightly default) a missing or non-positive baseline p95 fails the job — the gate is real from day one.
• A scenario whose p95 regresses by more than PERF_P95_REGRESSION_PCT (nightly: 20) percent over its baseline fails the job.
• Each k6 scenario also carries an absolute in-script p95 ceiling (e.g. tasks_board at 1200 ms, runs_board at 1500 ms) and an error-rate threshold of < 1%; breaching either fails k6 directly.

When pg_stat_statements is available the runner also records the DB-call delta per scenario and writes a db_queries_per_request figure into the budget report, so a query-count regression (an N+1 creeping in) is visible alongside latency. Artifacts (k6 summaries, server/Postgres/web logs, the Playwright report) are uploaded on every run, and a failure posts to the S4_FAILURE_WEBHOOK if configured.

Refresh the baseline deliberately

The committed baselines under k6/baselines/ are bootstrap values. Once your environment is stable, replace a scenario’s *.summary.json with a reviewed last-known-good nightly result to tighten the gate — that is the one place you intentionally edit the baseline.

Troubleshooting

Symptom	Likely cause	What to check
Prometheus target is DOWN with 401	SUPACLOUD_METRICS_TOKEN set but the scrape config has the wrong/no bearer token	The authorization.credentials matches the env value exactly
/metrics is publicly readable	No token configured (open by design)	Set SUPACLOUD_METRICS_TOKEN and mirror it in the scrape config
Route p95 panel is flat / empty	No traffic yet, or you queried a raw path	Series are labelled by the route template; check supacloud_http_route_duration_seconds_bucket exists
A route is slow but the cause is opaque	The slow query is in the logs, not on /metrics	Grep the server log for WARN slow-statement lines; lower SUPACLOUD_SLOW_QUERY_LOG_MS to widen the net
SupaCloudDispatchTickSla is firing	Scheduler/dispatch tick has not advanced inside the SLA window	supacloud_scheduler_seconds_since_tick; see Operate the Delivery Engine
Nightly fails on a missing baseline	K6_REQUIRE_BASELINE=1 and a k6/baselines/<scenario>.summary.json is absent or has a non-positive p95	Commit a valid baseline summary for that scenario