Runner-fleet defense-in-depth and elasticity

This page explains the defense-in-depth and elasticity hardening (ADR 0046) of the runner fleet. It is the companion to Runner-fleet security model and is deliberately explicit about what is GA, what is experimental, and what is still an infrastructure rollout. The earlier Phase-5 design shipped only thin code hooks here (an infra-terminated mTLS hook, a microVM grant with no executor, an autoscaling signal); the B+C wave (issue #521) turns those hooks into real, feature-gated, fail-closed code — a mutual-TLS hub listener, a microVM execution backend, and a built-in autoscaler — each off/disabled by default so a single host stays byte-identical.

Note

None of this changes a single-host deploy. Hub mode is off by default, so no assignment rows are ever created, the fencing epoch stays 0, leases are never renewed, the pickup sweep finds nothing, and the autoscale signal reports a desired count of zero. A single host is byte-identical to before Phase 5.

Lease-fencing epoch (the centerpiece)

A claimed assignment carries a monotonic fencing epoch. Every time the hub hands the assignment to a runner — including a self-reclaim when a lease expired — the epoch is bumped by one and the new value is returned to the claiming runner. The hub then requires that exact epoch on every report and renewal.

Why this matters: a runner can outlive its lease (a long, silent agent step, a GC pause, a network blip). When that happens the hub treats the holder as dead and the work returns to the queue; the same runner (or another) re-leases the row. Without fencing, the original — still-running — process could later report against the row and clobber the reclaimed work: drop the live per-run secret, flip the run terminal. The epoch closes this by construction. The current holder is the only party carrying the row’s current epoch; a reclaimed zombie carries a strictly-lower one, so its late report matches nothing and is rejected. This makes the existing “only the claiming runner, only an open row” guard provably total.

The same fence protects the connector queue, including the abandoned-assignment late-retry path: a current-epoch retry can still complete a hub-abandoned row (idempotency), but an epoch-stale zombie cannot complete over it.

Lease renewal and the pickup timeout

A live runner renews its lease on a fixed time cadence (≈ lease ÷ 3) for the duration of a run — on a timer, never gated on event traffic, so a silent long step still keeps the lease alive. Renewal does not bump the epoch (it is the same run continuing) and is fenced the same way, so a reclaimed zombie cannot renew.

Renewal guarantees no corruption (via fencing); it does not guarantee exactly-once execution. A lost renewal still lets the lease lapse and the task re-run — fencing only stops the stale report from corrupting the new run’s state. This was already true before Phase 5 (the lease-expiry reclaim re-ran work); Phase 5 makes the reporting safe.

The pickup timeout catches the other gap: a runner that claimed work and then died before starting it. Such a row would otherwise sit until its full lease expired. The existing liveness sweep now also fails any assignment that is claimed but never started past a short pickup window (SUPACLOUD_RUNNER_PICKUP_SECS, default 120s, floored at 60s and always below the claim lease), drops its at-rest secret, and surfaces the run as failed so it can re-dispatch.

Static-token expiry (optional, fail-closed)

The enrolled-runner JWT path has a short TTL (≤ 1 h) and auto-refreshes: the runner daemon re-exchanges its scrnj_… JWT before it expires — by default at roughly half the TTL (derived from runners.token_expires_at / the JWT exp), or at an explicit lead time set with SUPACLOUD_RUNNER_TOKEN_REFRESH_SECS. Each refresh hits the hub’s grant-rechecking refresh endpoint, so a revoked workspace takes effect within one token lifetime. The static scrn_ token, by contrast, has no expiry and is never refreshed — rotation is its only renewal/revocation path. Phase 5 adds an optional per-row expiry: register or rotate a runner with a token_ttl_secs and its token fails authentication closed after that instant. The default is unchanged — NULL expiry, never expires — so an existing runner is unaffected until its token is rotated with a TTL. The expiry is enforced per-row in the auth check; the JWT path is governed by its own exp and is untouched. Token rotation is the renewal path for a static token. Third-party edge runners should still prefer the JWT enrollment path; the static-token expiry is defence-in-depth for first-party runners.

Isolation tiers (container vs microVM)

A runner enrolls with an isolation grant — container or microvm — and runs a matching execution backend. The two tiers trade isolation strength against operational cost:

• container (default, GA). Each task runs in a hardened container on the runner’s Docker host: cap_drop=ALL, no-new-privileges, a read-only rootfs, tmpfs mounts, and memory/CPU/pids caps. It shares the host kernel, so the boundary is a kernel namespace + cgroup. This is the right tier for a runner you trust.
• microvm (experimental, feature-gated). Each task runs in its own short-lived hardware-virtualised guest (a Firecracker/Kata-class microVM), so a kernel-level escape is contained behind a VM boundary rather than only a namespace. This is the strongest tier — reserve it for untrusted or edge runners. The backend is selected with SUPACLOUD_RUNNER_EXECUTION_BACKEND=microvm and is compiled only into a binary built with the microvm-backend cargo feature (default off, like ssh-backends); selecting it without the feature is a hard boot error (fail-closed — never a silent downgrade). SupaCloud ships no launcher: the operator provisions one and names it via SUPACLOUD_RUNNER_MICROVM_CMD.

The grant and the backend are enforced independently and both fail-closed: a microVM-required task is dispatched only to a runner with a microvm grant (the dispatch filter), and that runner only delivers true VM isolation if it actually runs the microVM backend. A container-grant runner is unaffected.

Per-workspace fairness (shared-pool admission)

On a fleet shared by several workspaces, one busy workspace can otherwise drain the whole pool. The per-workspace fairness switch admits work fairly per workspace at dispatch — admission fairness, not a hard quota — so no single workspace starves the others. It is a persisted global switch (workspace_fairness_enabled, default off, fail-closed), mirroring the least-loaded switch exactly; an optional global integer workspace_fair_share_cap bounds the in-flight share any one workspace may hold. While the switch is off, no per-workspace accounting runs and dispatch is byte-identical to a fleet without it.

Autoscaling: built-in actuator or external signal

The fleet can grow and shrink on demand. SupaCloud computes the decision and can optionally actuate it itself.

The decision is computed from the fleet pull-queue depth — the count of unclaimed assigned rows across both fleet queues, not the autonomous-delivery backlog_items queue — plus the online runner count, the in-flight work, and how many runners are draining/cordoned. It yields a desired_runners count and a human-readable reason, exposed on the operator dispatch-metrics snapshot and on GET /api/operator/v1/fleet/autoscale. A desired count of 0 with no work is a scale-to-zero signal.

The built-in autoscaler (experimental). With SUPACLOUD_RUNNER_AUTOSCALE=on the hub runs the decision on a loop and actuates it itself by spawning and removing local-Docker runner containers, clamped to SUPACLOUD_RUNNER_AUTOSCALE_MIN/_MAX, throttled by SUPACLOUD_RUNNER_AUTOSCALE_COOLDOWN_SECS, launching SUPACLOUD_RUNNER_AUTOSCALE_RUNNER_IMAGE on SUPACLOUD_RUNNER_AUTOSCALE_NETWORK (and handing each new container a runner token). It is off by default — no loop runs, no behaviour change. v1 is local-Docker only: it does not yet drive a cloud ASG or Kubernetes, and there is no per-cloud cost ceiling, so treat it as an experimental edge and set _MAX conservatively.

Or consume the signal externally. Leave the autoscaler off and let an external controller (KEDA, a Komodo cron, a cloud ASG) read the decision and actuate it. In either path scale-in reuses the existing drain primitive — in-flight work is never force-killed:

1. desired_runners < online ⇒ remove some runners.
2. Cordon each victim via POST /api/runners/{id}/drain {drain_state:"cordoned"} (the dispatch ranker already excludes a cordoned runner from new work).
3. Poll GET /api/runners/dispatch-state for each cordoned runner’s in_flight count and wait until it reaches zero (the safe_to_remove gate).
4. Only then terminate the runner.

A runner holding in-flight work is never reported safe. And even a premature kill is non-corrupting: the work is reclaimed and the zombie is fenced.

What is code vs infrastructure (no false green)

Capability	Status
Lease-fencing epoch (both queues)	Code, tested
Lease renewal + pickup timeout	Code, tested
Static-token expiry / TTL + JWT auto-refresh	Code, tested
Isolation dispatch filter (container vs microvm grant)	Code, tested
Per-workspace fairness switch + share cap	Code, tested (default off)
Autoscaling decision + fleet-queue signal + drain-gated safe_to_remove	Code, tested
Mutual TLS (hub listener + runner client identity)	Code (the hub serves TLS and optionally requires + verifies a client cert; the runner presents a client identity and can pin the hub CA). All env-gated, fail-closed, off by default; the live mTLS handshake is not CI-verified end-to-end
microVM execution backend (Firecracker/Kata)	Code, experimental + feature-gated (microvm-backend, default off). The dispatch filter is fail-closed; the microVM launcher itself is operator-provisioned (SUPACLOUD_RUNNER_MICROVM_CMD) — SupaCloud ships none
Built-in autoscaler (spawn/scale-in local-Docker runners)	Code, experimental, default off; local-Docker only in v1
Cloud-ASG / Kubernetes autoscaling, snapshot / hibernate	Infra rollout. Consume the GET …/fleet/autoscale signal + the drain-gated safe_to_remove from an external controller

Mutual TLS (hub ↔ runner)

By default the runner trusts the hub over one-way TLS (server cert only). To require a mutual handshake, turn TLS on at the hub with SUPACLOUD_TLS_CERT_PATH plus SUPACLOUD_TLS_KEY_PATH and add SUPACLOUD_TLS_CLIENT_CA_PATH to require and verify a client certificate against that CA; on each runner present SUPACLOUD_RUNNER_TLS_CLIENT_CERT_PATH plus SUPACLOUD_RUNNER_TLS_CLIENT_KEY_PATH (and optionally pin the hub with SUPACLOUD_RUNNER_TLS_CA_PATH). Each pair is all-or-nothing — a half-configured side is a hard boot error (fail-closed), and no TLS environment leaves plain HTTP exactly as before (byte-identical default). The operator-facing steps are in Set up a runner securely → secure the runner transport with mTLS. Terminating mTLS at a fronting proxy instead remains a valid bw-infra option; the in-process listener is for deployments without one.

microVM and scale-to-zero

The microVM tier is a real, fail-closed execution backend (see Isolation tiers above), gated behind the microvm-backend cargo feature and an operator-provided launcher — SupaCloud ships no Firecracker/Kata launcher. Scale-to-zero / hibernate is the desired_runners = 0 end of the autoscaling decision: the built-in autoscaler removes idle local-Docker runners down to SUPACLOUD_RUNNER_AUTOSCALE_MIN, and an external controller can drive a true scale-to-zero (or a cloud-native snapshot/hibernate) off the same signal. No hibernation controller ships in SupaCloud.