Steer & monitor runners

Once you have more than one runner (see Set up a runner securely), you steer which runner gets which work. Every lever here is off by default — a fleet with none of them set dispatches “first online runner that can run the image”, so you opt into exactly the complexity you need.

The ladder

Dispatch is one decision function with stacked, default-off filters and ordering: capability/label match → drain state → concurrency cap → (optional) least-loaded ordering → priority within a runner’s queue → optional pin. Removing a setting drops you back to the rung below.

Labels & capabilities (routing)

A runner advertises capabilities at registration (and you can re-register or rotate to change them). Beyond images, advertise free-form labels:

{ "images": ["unified"], "labels": ["gpu", "eu-west"], "max_parallel": 4 }

A task that requires labels is sent only to a runner advertising all of them (subset match, like CI runs-on). A task with no required labels ignores labels entirely — so adding labels never changes routing for existing work.

Drain & cordon (graceful quiesce)

To take a runner out of rotation without killing its running tasks, set its drain state. A cordoned / draining runner keeps heartbeating and finishes its in-flight work but receives nothing new; active restores it.

curl -X POST https://supacloud.example.com/api/runners/<id>/drain \
  -H "Authorization: Bearer <admin-token>" \
  -H "Content-Type: application/json" \
  -d '{ "drain_state": "draining" }'   # active | cordoned | draining

This is the graceful alternative to disabling a runner (POST /api/runners/{id}/status {"enabled": false}), which parks it abruptly and rejects its heartbeats.

Concurrency cap

capabilities.max_parallel caps how many tasks a runner runs at once. The server counts a runner’s in-flight assignments and stops dispatching to it at the cap — the work simply waits for a free slot or another eligible runner.

Least-loaded ordering

By default eligible runners keep registry order. Turn on weighted least-loaded to send each new task to the least-busy runner instead. It is a persisted global switch (dispatch_least_loaded, default off), toggled via the operator switches:

curl -X PUT https://supacloud.example.com/api/operator/v1/switches/dispatch_least_loaded \
  -H "Authorization: Bearer <operator-token>" \
  -H "Content-Type: application/json" \
  -d '{ "enabled": true, "reason": "balance load across the fleet" }'

A runner’s capabilities.weight (default 1) scales its share: a weight: 2 runner absorbs roughly twice the load before it is deprioritised — useful when machines differ in size.

Per-workspace fairness (shared-pool admission)

Least-loaded balances across runners; per-workspace fairness balances across workspaces competing for one shared pool. With it on, the dispatcher admits work fairly per workspace so one busy workspace cannot monopolise the fleet and starve the others — admission fairness on a shared pool, not a hard quota. It is a persisted global switch (workspace_fairness_enabled, default off), toggled exactly like the least-loaded switch via the operator switches:

curl -X PUT https://supacloud.example.com/api/operator/v1/switches/workspace_fairness_enabled \
  -H "Authorization: Bearer <operator-token>" \
  -H "Content-Type: application/json" \
  -d '{ "enabled": true, "reason": "stop one workspace monopolising the shared fleet" }'

An optional global integer workspace_fair_share_cap bounds how much of the pool any single workspace may hold in flight at once. While the switch is off, dispatch is byte-identical to today — no workspace accounting runs.

Priority & pinning (per task)

Two per-task controls live in the task’s config:

• priority (integer, default 0) — within a runner’s queue a higher-priority task is claimed first.
• pinned_runner_id — force the task onto one specific runner. If that runner is offline or at capacity the task waits for it (queue-pending) rather than running elsewhere. Set "pin_fallback": true to instead fall back to normal ranking when the pinned runner is unavailable.

Precedence

Today pin/priority are read from the task’s config. The designed precedence is agent-profile → project → task (first value wins, so an operator policy on the profile or project beats an ad-hoc per-task value); the profile/project sources land with the worker-groups workstream.

Monitor: dispatch state

GET /api/runners/dispatch-state returns, per runner, its live in-flight count, advertised cap, drain state and whether its heartbeat is stale — the real signal for “is the fleet keeping up”, replacing any guesswork from container ids.

curl https://supacloud.example.com/api/runners/dispatch-state \
  -H "Authorization: Bearer <admin-token>"

Every steering mutation (register, enable/disable, drain, token rotation, hub-mode) writes an audit_events row, so fleet changes are attributable.

Autoscaler (experimental)

Rather than running a fixed fleet, the hub can grow and shrink it on demand. The built-in autoscaler reads the fleet pull-queue signal and spawns new local-Docker runner containers when work is queued, then scales them back in when the pool is idle. It is off by default (SUPACLOUD_RUNNER_AUTOSCALE=off) — no loop is spawned and dispatch is unchanged.

SUPACLOUD_RUNNER_AUTOSCALE=on
SUPACLOUD_RUNNER_AUTOSCALE_MIN=1
SUPACLOUD_RUNNER_AUTOSCALE_MAX=8
SUPACLOUD_RUNNER_AUTOSCALE_RUNNER_IMAGE=<your-supacloud-image>
SUPACLOUD_RUNNER_AUTOSCALE_COOLDOWN_SECS=120
SUPACLOUD_RUNNER_AUTOSCALE_NETWORK=supacloud-agents

Each spawned runner needs a token, so the autoscaler must be configured with a runner token to hand the container it launches — see the environment-variable reference. Scale-in is ARC-shaped and drain-gated: it cordons a victim (so the ranker stops sending it new work), waits for its in_flight count to reach zero, and only then removes it — running work is never force-killed.

Experimental, local-Docker only in v1

The autoscaler is experimental. v1 spawns runners as local Docker containers on the hub’s own Docker host only — it does not yet drive a remote fleet, a cloud ASG or Kubernetes. The honest edge: it grows the pool from a simple queue signal and there is no per-cloud cost ceiling, so set SUPACLOUD_RUNNER_AUTOSCALE_MAX conservatively and watch it. For an external actuator (KEDA, a cloud ASG) consuming the signal instead, see Runner-fleet defense-in-depth → autoscaling.

Troubleshooting

Symptom	Likely cause	What to check
Task runs locally despite a healthy runner	Hub mode off, or no runner advertises the task’s image	GET /api/runners/config; the runner’s capabilities.images
Runner shows online but gets no work	Cordoned/draining, at its max_parallel cap, or a required label it lacks	dispatch-state (drain_state, in_flight vs cap); the task’s required labels
Runner flips to offline on its own	Heartbeat stale past the window (default 90 s)	Worker clock/network; SUPACLOUD_RUNNER_STALE_SECS on the hub
A pinned task never starts	Pinned runner offline/at-capacity and pin_fallback not set (queue-pending is intended)	The runner’s state; set pin_fallback if you want it to run elsewhere
One workspace’s tasks starve the others on a shared pool	Per-workspace fairness off	Turn on workspace_fairness_enabled; optionally set workspace_fair_share_cap
A microVM-required task waits while container runners sit idle	No runner holds a microvm grant (the filter is fail-closed)	Enroll a runner with isolation: microvm running the microVM backend
“Bridge health check timed out” on a remote run	Agent container not on the runner’s Docker network	AGENT_NETWORK exists on the worker and matches