Skip to content
SupaCloud Docs
Select themeSelect language

Observability: Prometheus, Grafana and the nightly load checks

SupaCloud exposes a single Prometheus /metrics endpoint, ships a ready-made Grafana dashboard and alert-rule file, and runs a nightly real-backend load check that gates p95 latency against a committed baseline. This page wires all three so you can see request rate/errors/latency, DB-pool pressure and dispatch health, and catch a performance regression before it reaches users.

Scrape /metrics with Prometheus

Section titled “Scrape /metrics with Prometheus”

The server exposes GET /metrics on its root router (not under /api), so it is outside the API auth stack and the governor rate-limit layer. It renders Prometheus text exposition (text/plain; version=0.0.4) and does DB + Docker work per request, so do not scrape it more often than you need.

  1. (Recommended) Set a metrics token so the endpoint is not world-readable:

    SUPACLOUD_METRICS_TOKEN=<a long random string>

    See the environment-variable reference for where your deployment reads env, and the OpenBao path under Use the OpenBao secret backend.

  2. Point Prometheus at the server. The server listens on SERVER_PORT (default 8080). Add a scrape_config — include the bearer token only if you set one:

    scrape_configs:
      - job_name: supacloud
        metrics_path: /metrics
        scheme: http # https if you terminate TLS in front of the server
        # Drop this block entirely if SUPACLOUD_METRICS_TOKEN is unset:
        authorization:
          type: Bearer
          credentials: <the same value as SUPACLOUD_METRICS_TOKEN>
        static_configs:
          - targets: ["supacloud.example.com:8080"]

  3. Reload Prometheus and confirm the supacloud target is UP on the Targets page, then check a series renders, e.g. supacloud_http_requests_total.

What the endpoint exposes

Section titled “What the endpoint exposes”

RED — rate, errors, duration (per route)

Section titled “RED — rate, errors, duration (per route)”

A middleware records every HTTP response under the matched Axum route template (e.g. /api/runs/{id}/events), never the raw path — so the label space is bounded by the route table plus a single {unmatched} fallback and never grows with tenant ids or query strings. Three series carry the RED signal, labelled by method, route and status:

Metric Type Signal
supacloud_http_route_requests_total counter Rate — requests served
supacloud_http_route_errors_total counter Errors — responses with status ≥ 500
supacloud_http_route_duration_seconds histogram Duration — latency buckets (_bucket/_sum/_count)

The duration histogram uses fixed le buckets from 5 ms to 60 s, so histogram_quantile() gives you a real p95 per route. Process-wide counters (supacloud_http_requests_total, supacloud_management_requests_total, supacloud_webhook_requests_total, supacloud_task_launch_requests_total, supacloud_scheduler_ticks_total) and business gauges (supacloud_tasks_total{status}, supacloud_ai_cost_month_usd, supacloud_agent_containers_active) round out the snapshot.

DB pool saturation

Section titled “DB pool saturation”

The PostgreSQL connection pool is exported so you can see pool pressure before it turns into request latency:

Metric Type Signal
supacloud_db_pool_connections{state="active|idle|open"} gauge Connections by state
supacloud_db_pool_max_connections gauge Configured pool maximum
supacloud_db_pool_saturation gauge active / max ratio (0–1)

A supacloud_db_pool_saturation riding near 1.0 means requests are queueing on the pool — scale the pool or shed load.

Slow-query signal (logs, not a metric)

Section titled “Slow-query signal (logs, not a metric)”

Slow statements surface in the server logs, not on /metrics. The pool is configured to log any statement slower than a threshold at WARN level via sqlx’s slow-statement logging. The threshold is SUPACLOUD_SLOW_QUERY_LOG_MS (default 250 ms; a zero or unparseable value falls back to 250). Grep the server log for these WARN lines, or ship the logs to Loki, to find the queries behind a route p95 spike.

Dispatch operability + alert state

Section titled “Dispatch operability + alert state”

The delivery engine’s health is exported so the same Prometheus can alert on a stalled scheduler or a growing backlog (the human-facing view of this lives in Operate the Delivery Engine):

Metric Type Signal
supacloud_scheduler_seconds_since_tick gauge Seconds since the last scheduler tick (-1 before the first tick)
supacloud_backlog_queued_items gauge Backlog items queued awaiting dispatch
supacloud_backlog_oldest_queued_age_seconds gauge Age of the oldest queued item
supacloud_dispatch_alert{kind="…"} gauge Evaluated alert state per kind (1 = breaching, 0 = ok)

The supacloud_dispatch_alert kinds (tick_sla, queue_age, error_spike, budget_80, weekly_window_low) are evaluated server-side against named-const thresholds, so the gauge already tells you whether a rule is breaching — you do not have to re-derive the thresholds in PromQL.

Wire the Grafana dashboard and alert rules

Section titled “Wire the Grafana dashboard and alert rules”

Both artifacts are committed in the repo under observability/ — import them as-is.

  1. Import the dashboard. In Grafana, Dashboards → New → Import, and upload observability/grafana/supacloud-s4-dashboard.json. Pick your Prometheus datasource when prompted (the dashboard exposes a datasource template variable). It ships five panels: HTTP Route p95, HTTP Route Throughput, HTTP Route 5xx, DB Pool Saturation, and Dispatch SLA + Queue.

  2. Load the alert rules. observability/prometheus/supacloud-s4-rules.yml is a Prometheus rule group. Reference it from your prometheus.yml (rule_files:) or load it into Grafana-managed alerting. It defines four rules:

    Alert Fires when Severity
    SupaCloudHttpRouteP95High a route’s p95 > 2 s for 15 m warning
    SupaCloudHttpRouteErrors sustained 5xx rate > 0.05/s for 10 m warning
    SupaCloudDbPoolSaturation supacloud_db_pool_saturation > 0.85 for 10 m warning
    SupaCloudDispatchTickSla supacloud_dispatch_alert{kind="tick_sla"} == 1 for 5 m critical

  3. Attach a contact point. Route the severity / stream: s4 labels to your on-call channel so the critical tick-SLA alert pages and the warnings notify.

The E2E + k6 load checks

Section titled “The E2E + k6 load checks”

The manual qualification workflow (.forgejo/workflows/e2e-nightly.yml, via workflow_dispatch) boots a real backend stack — server, Postgres, web —, seeds it, and runs three checks against it. Its predominantly red recurring schedule is disabled until the lane is repaired and has a reviewed green run:

  • Playwright real-backend E2E (npm run test:e2e:nightly) drives the UI against the live server, not mocks.
  • k6 hot-endpoint budgets (scripts/perf/run-k6-nightly.sh → k6/s4-hot-endpoints.js) load the six hottest read paths and gate their p95.
  • k6 2.0 WebSocket streaming (scripts/perf/run-k6-streaming.sh → k6/s4-ws-streaming.ts) authenticates against the real server, receives live task events through k6/websockets, and records sequence gaps, server resync frames and exact-after_sequence recovery as custom metrics.

The k6 scenarios cover the boards and the two hot non-board paths: tasks_board, projects_board, runs_board, run_events, dispatch_tick (/api/operator/v1/dispatch/metrics) and mcp_gateway (a tools/list call to /api/mcp). Each runs at K6_VUS virtual users for K6_DURATION (the nightly uses 4 VUs for 2 m).

The p95 budget gate

Section titled “The p95 budget gate”

Each scenario writes a k6 summary that scripts/perf/check-k6-budget.mjs compares against the committed governed baseline in k6/baselines/:

  • With K6_REQUIRE_BASELINE=1 (the nightly default) a missing or non-positive baseline p95 fails the job — the gate is real from day one.
  • The H4 manifest is validated on every run. While it is pending, the existing reviewed single bootstrap file is used. Once active, only the three checksum-pinned samples are accepted and their per-metric median is used; an invalid set never falls back to the single file.
  • A scenario whose p95 regresses by more than 20% over its governed baseline fails the job. PERF_P95_REGRESSION_PCT is retained as an explicit workflow declaration, but any value other than 20 now fails closed.
  • Each k6 scenario also carries an absolute in-script p95 ceiling (e.g. tasks_board at 1200 ms, runs_board at 1500 ms) and an error-rate threshold of < 1%; breaching either fails k6 directly.

When pg_stat_statements is available the runner also records the DB-call delta per scenario and writes a db_queries_per_request figure into the budget report, so a query-count regression (an N+1 creeping in) is visible alongside latency. Artifacts (k6 summaries, server/Postgres/web logs, the Playwright report) are uploaded on every run, and a failure posts to the S4_FAILURE_WEBHOOK if configured.

Two independent, manually dispatched lanes exercise longer-lived failure modes without joining the deploy graph. soak-weekly.yml runs 90 minutes and gates the linear slope (after a ten-minute burn-in) of server RSS, open file descriptors, PostgreSQL connections and the durable scheduler_tick_durations signal. It deliberately does not infer a leak from one absolute snapshot. chaos-weekly.yml places Toxiproxy in front of PostgreSQL and WebSocket traffic, then runs five bounded experiments: proxy latency, Pumba pause, Pumba netem, Pumba kill/restart and a PostgreSQL restart. Health, a DB query, an authenticated API read and a real WS event stream must all recover after every experiment.

These lanes require the self-hosted e2e-host-net runner’s Docker socket and network-emulation privileges. Tool versions are fail-closed (k6 2.0.0, Toxiproxy 2.12.0, Pumba 1.1.7; the nettools helper is digest-pinned), and main dispatches require the exact server SHA image. Recurring soak/chaos activation is gated on each lane’s first reviewed green run. The confirmed H4 governance records all three successful raw sample sets side by side with immutable provenance, checksums and review metadata, then computes the deterministic per-metric median at gate time. All three runs must have distinct identities and content but the same exact SHA and declared environment. Nightly captures evidence only; it never curates or updates a committed baseline. The existing reviewed single baseline and unchanged 20% p95 threshold remain authoritative while the committed H4 manifest is pending.

Playwright flake trend and quarantine governance

Section titled “Playwright flake trend and quarantine governance”

The same nightly workflow writes a Playwright JSON report for the real and mock suites. JSON is intentional: it retains every results[] retry attempt, so a failed first attempt followed by a pass remains visible. JUnit collapses that case to a pass and is kept only for the human-readable job summary.

Set the repository secret TREND_DB_DSN to a PostgreSQL DSN for the dedicated CI trend database. Provision it separately from both the SupaCloud product database and Forgejo’s database. The in-run reporter creates and fills ci_playwright_trend idempotently; it never writes an application migration or uses the SupaCloud product database. The DSN is optional: when it is absent the test lane stays green and prints a skip notice. When it is configured, a broken connection or insert is a real instrumentation failure.

  1. Create a Grafana PostgreSQL datasource for the same CI trend database. Give the datasource account read-only access to ci_playwright_trend.

  2. Import the flake dashboard manually. In Dashboards → New → Import, upload observability/grafana/flake-trend-panel.json and select the PostgreSQL datasource. Select repository and lane explicitly. A run counts as a flake only when an earlier attempt failed, timed out or was interrupted and its final attempt passed; a final failure stays a failure and does not inflate the flake rate. Rolling windows and series are isolated by repository, lane and test ID. The panels also show p95 duration drift and hashed first-line error clusters. Error text is not stored in the trend table.

  3. Keep quarantine exceptional. web/tests/e2e/quarantine.json is normally empty. A temporary entry must match an explicit @quarantine test-title tag and carry owner, reason and an expiring UTC review_by. The validate job rejects stale IDs, expired entries and tag/list drift. PR/main lanes exclude only that explicit tag; the nightly quarantine lanes run tagged mock and real tests five times with retries disabled.

  4. Read the changed-spec burn-in separately. Nightly changed mock and real specs run ten times on chromium-desktop with retries disabled. The last ci-green/web-lint marker is the preferred diff anchor, with the commit from 24 hours ago as fallback. If the diff cannot be computed, the lane runs the full applicable suite instead of silently skipping it.

Troubleshooting

Section titled “Troubleshooting”
Symptom Likely cause What to check
Prometheus target is DOWN with 401 SUPACLOUD_METRICS_TOKEN set but the scrape config has the wrong/no bearer token The authorization.credentials matches the env value exactly
/metrics is publicly readable No token configured (open by design) Set SUPACLOUD_METRICS_TOKEN and mirror it in the scrape config
Route p95 panel is flat / empty No traffic yet, or you queried a raw path Series are labelled by the route template; check supacloud_http_route_duration_seconds_bucket exists
A route is slow but the cause is opaque The slow query is in the logs, not on /metrics Grep the server log for WARN slow-statement lines; lower SUPACLOUD_SLOW_QUERY_LOG_MS to widen the net
SupaCloudDispatchTickSla is firing Scheduler/dispatch tick has not advanced inside the SLA window supacloud_scheduler_seconds_since_tick; see Operate the Delivery Engine
Nightly fails on baseline governance While H4 is pending, a k6/baselines/<scenario>.summary.json is absent/non-positive; when active, the manifest, provenance, review, checksum or three-sample contract is invalid Pending: restore the reviewed single file. Active: run k6-baseline-governance.mjs validate; fix and review the evidence set rather than falling back
The Playwright flake dashboard is empty TREND_DB_DSN is unset/wrong, or Grafana points at product/Forgejo storage Check the workflow’s ingest notice and the read-only datasource against the dedicated CI trend DB’s ci_playwright_trend table
Validate rejects quarantine.json Entry expired, test ID disappeared, inventory changed, or tag/list no longer agree Regenerate web/tests/e2e/test-inventory.json, then remove the quarantine or update its bounded owner review