Operate the Delivery Engine

The Autonomous Delivery Engine (ADR 0045) has two operability surfaces, scoped differently:

• The operator console at /operator is the cross-tenant cockpit for an instance admin: dispatcher health across every workspace, tripped circuit breakers, an explainable routing log, the cross-tenant audit query, and the global system switches.
• The operability console at /operations is a tenant-scoped, read-only view of a single workspace for that workspace’s owners and admins: the workspace’s own dispatch metrics, its routing decision-log, and its tripped circuit breakers. It does not expose any other tenant and carries no switches.

This guide covers reading both surfaces and the two safety controls — the global dispatch kill-switch (instance owner) and the per-workspace pause (workspace owner).

Who can open which

The cross-tenant operator console is for instance admins — the Operator item in the sidebar appears only for an admin, and the console opens with your normal login session. The tenant operability console is for a workspace’s owners and admins and shows only that workspace. Every cross-tenant read and switch change is audited as actor_type=operator.

Read your workspace’s operability console

A workspace owner or admin opens /operations to see how the delivery engine is treating their workspace, without instance-admin access. The page uses the familiar tabbed Settings layout.

It is backed by three tenant-scoped, read-only endpoints — the workspace is resolved from your session, never passed as a parameter:

• GET /api/operator/v1/tenant/dispatch/metrics — this workspace’s queue depth, age and throughput, plus its own per-credential provider-quota saturation.
• GET /api/operator/v1/tenant/routing — this workspace’s routing decision-log.
• GET /api/operator/v1/tenant/circuit-breakers — this workspace’s tripped project-scope breakers.

A member or viewer who deep-links to /operations is refused — the console requires the workspace owner or admin role.

Open the console

1. Sign in as an instance admin and open /operator (the Operator item in the sidebar appears only for admins).

2. The console loads the cross-tenant overview, the dispatch-metrics panel, the routing decision-log, the audit query, and the system switches. Select Refresh to re-read at any time.

Read the dispatch-metrics panel

The Dispatch metrics section aggregates the dispatcher’s health across all workspaces. It is a read-only snapshot, refreshed each time you select Refresh.

Queue and tick gauges

The four top gauges report the backlog queue and the scheduler’s health:

Gauge	Meaning
Queue depth	Backlog items currently queued awaiting dispatch.
Oldest queued	Age of the oldest queued item (queue staleness). Shown as Ns / Nm / Nh, or “never” when the queue is empty.
Dispatched (recent)	Backlog runs dispatched in the recent window — throughput.
Tick health	Time since the last scheduler tick. A growing value here means the dispatch ticker has stalled.

Below them, the Priority distribution bars show how the queued items break down by priority bucket (highest-priority first); a long tail in the lowest bucket is normal, a backlog piling up in the highest bucket is not.

Provider quota-saturation gauges

The Provider saturation card shows, per model provider, how much of the provider’s metered quota is consumed (used / limit, summed across the provider’s credentials):

• A gauge fill turns amber past 60% saturation and red past 85% — the dispatcher’s throttle floor. A red gauge means the dispatcher is about to back off that provider to avoid hitting the ceiling.
• A provider whose limit is unknown in every metered window renders as unknown (no gauge) — SupaCloud cannot compute saturation without a known ceiling.
• The “near exhaustion” line under a gauge counts windows already past the 85% floor.

Error-recovery rates

The Error recovery card shows, per agent-error class (transient_server, usage_limit, context_degraded, auth_config, fatal), the fraction of recent errors of that class the recovery layer marked auto-retryable. A high recovery rate for transient_server is healthy (blips are being retried); a low rate for a class that should be recoverable is worth investigating.

The routing decision-log

The Routing log card is the dispatcher’s explainability surface. Each entry is one backlog routing decision, newest first, showing:

• the item title (or id) the decision applied to;
• the routing source — classify, escalate, fallback, or manual — the path that produced the decision;
• the verbatim decision factors JSON the classifier or escalation wrote (why the item dispatched, which model was chosen, why it was paused). Hover an entry to read the full JSON in the tooltip;
• the decision’s timestamp.

Use this log to answer “why did this item run on that model?” or “why is this item still waiting?” without reading server logs.

Note

This console reads cross-tenant. The same routing log, scoped to a single workspace, is what a workspace owner or admin sees in their own operability console at GET /api/operator/v1/tenant/routing.

Read circuit-breaker state

A circuit breaker stops the dispatcher from hammering a provider or project that is in a sustained outage. Each breaker is keyed by a scope (provider or project) and an error class, and is one of three states:

• closed — requests flow normally.
• open — the breaker has tripped; requests fail fast or fail over. It trips after three consecutive failures of the same scope and class, and stays open for a 60-second cooldown.
• half-open — after the cooldown, one probe request is admitted. A success closes the breaker; a failure re-opens it and restarts the cooldown.

A tripped provider stays tripped across a server restart (the state is durable). Currently-tripped breakers are listed at GET /api/operator/v1/circuit-breakers — the endpoint returns only the non-closed breakers, with their scope, error class, state, and trip count.

Tip

When a provider has any open provider-outage breaker (transient_server or usage_limit), the dispatcher fails that provider over to an alternative credential or provider rather than launching a run that will only error again. A breaker self-heals on the next success after its cooldown — you do not normally clear it by hand.

Use the global kill-switch

The global dispatch switch is the engine’s master kill-switch. While it is disabled, the backlog ticker dispatches nothing across every workspace; the queue still fills but no new autonomous runs are launched.

1. In the operator console, scroll to System switches.

2. Enter a reason in the reason field — a switch change is rejected without one. The reason is written to the audit trail.

3. Select the action button on the global_dispatch row to toggle it. The button label is the action it performs (Enable / Disable). The toggle calls PUT /api/operator/v1/switches/{key} and is restricted to the instance owner.

The other switches on this panel — task_launch, scheduler, auto_developer, webhook_automation, outbound_notifications, auto_model_selection — are finer-grained global gates over the same audited mechanism. Toggling any of them also requires a reason.

Pause a single workspace

The kill-switch above is global. To pause one workspace without affecting the rest, set its workspace-scoped switch. A workspace-scoped switch overrides the global value for that workspace only; an unset workspace switch inherits the global value (which defaults to allowed). This is applied through the workspace’s own settings or the management API rather than the operator console’s global switch panel.

Caution

Every switch change — global or per-workspace — writes an audit_events row with the actor, the target switch, and the required reason. Treat the reason as the operational record of why dispatch was paused.

See also

• The self-learning Governor — how the engine learns model selection and what the L3 governor proposes (nothing auto-applies).
• The full design rationale is in ADR 0045.