E2E spec truth table

The Blockworx engineering handbook made its real-flow E2E requirement binding on 2026-09-04 (docs/70-standards/testing/e2e.md). Rule 8 of that standard asks every repository to keep a spec truth table: one row per spec, its surface, and the class of shortcut it still takes — with line references and counts — so the remaining shortcuts are visible rather than assumed. This page is that table for SupaCloud. It is maintained by hand and re-counted whenever a spec changes; the counts below were taken on 2026-09-04 from web/tests/e2e and web/tests/e2e-real.

Classes

The handbook’s classes, plus one this repository needs:

Class	Meaning
A	Real flow through the real surface: a booted backend, the built web bundle, UI clicks or public API calls, no direct database write inside the flow.
B	A database step inside the flow: something the flow should produce is written directly (seed, stamp, UPDATE).
C	The result is read from the database instead of the artefact.
D	Skipped cells (test.skip / fixme).
E	Timestamp manipulation in rows instead of an E2E-gated clock.
F	A route no user can reach.
M	Mocked API: the browser drives the built web bundle, but every /api/** call is answered by page.route fixtures (web/tests/e2e/vite-mock-api.ts, fixtures.ts). By the handbook’s definition (“hits the real running app”) this is not an E2E test; it is the contract-conformance harness over the ADR 0047 fixtures (fixtures.contract.test.ts gates the fixtures against the OpenAPI schema). The mechanics rules (no sleep, no skip, retries, video) apply to it all the same; the real-flow rules 1–4 cannot be met by it and are not claimed.

Lanes

Lane	Config	Spec directory	Cells	Runs in	Browser
Mock (smoke)	web/playwright.config.ts	tests/e2e	208 (113 chromium-desktop, 95 mobile-chrome)	pr-checks.yml (core-loop net, @pr-core-loop), e2e-nightly.yml mock-smoke + mock-full	Chromium
Real (nightly)	web/playwright.nightly.config.ts (local: playwright.real.config.ts)	tests/e2e-real	41 (1 auth-setup, 20 chromium-desktop, 20 mobile-chrome)	e2e-nightly.yml real-backend job, 06:00 UTC	Chromium

Layout gating is a project property since 2026-09-04: a test titled @desktop-only never enters the mobile-chrome project and @mobile-only never enters chromium-desktop (grepInvert on the project). The 19 test.skip cells that used to express this are gone — a gated test is not listed, a skipped one was listed and silent.

The table

Selector columns count call sites in the spec file: testid = getByTestId / data-testid; role = getByRole; text = getByText / getByLabel / getByPlaceholder; locator = .locator(; css = a .locator( anchored on a class or id. Rule 6 of the standard admits data-testid only, so every non-zero role, text and css figure is open debt, listed here so it cannot be assumed away.

Mock lane (web/tests/e2e)

Mock lane, migrated 2026-09-05. Every mock spec now selects by data-testid; the counts below are the state after that migration. Two .locator( calls remain by design and are library-owned DOM below an element we DO tag: .cm-content inside memory-editor-body (CodeMirror) and .xterm-screen inside log-terminal (xterm.js). visual.spec.ts keeps two document-level probes (html[data-theme] and the screenshot mask over time elements) - the document is not a product element. Attribute selectors that START from a data-testid ([data-testid="backlog-column"] [data-state=queued], [data-testid][data-variant=nav]) count as locator, not as debt. marketplace-subscription.spec.ts was the last spec still mixing roles and is on ids since the same day.

Spec	Cells (desktop / mobile)	Class	Shortcut, with reference	testid / role / text / locator / css	Routes driven
accessibility.spec.ts	8 / 8	M	mocked API	1 / 0 / 0 / 0 / 0	one axe pass per listed path
app-flows.spec.ts	8 / 14	M	mocked API; 6 tests @mobile-only	64 / 0 / 0 / 0 / 0	/, /inbox, /inbox?lang=de, /inbox?modal=…, /login?lang=de, …
badge-coordinator.spec.ts	2 / 2	M	mocked API; asserts the leader-tab fetch count over page.route	1 / 0 / 0 / 0 / 0	/ in three tabs
core-loop.spec.ts	5 / 0	M	mocked API; describe @desktop-only	17 / 0 / 0 / 0 / 0	/backlog, /cli, /runs/{id}, /tasks/{id}
crud-matrix.spec.ts	9 / 0	M	mocked API; describe @desktop-only	82 / 0 / 0 / 1 / 1	/projects, /projects/new, /resources, /reports/schedules, /intelligence/memory, …
delivery-debt-surfaces.spec.ts	14 / 5	M	mocked API; 9 tests @desktop-only	72 / 0 / 0 / 1 / 1	/inbox?source=governor, /reports/usage, /settings?tab=…, /admin?tab=payouts…, /cli
marketplace-subscription.spec.ts	7 / 7	M	mocked API	25 / 0 / 0 / 3 / 3	/marketplace/{item}, /admin/marketplace/payouts?tab=fees
memory.spec.ts	1 / 1	M	mocked API	15 / 0 / 0 / 0 / 0	/intelligence/memory
mobile-capture.spec.ts	19 / 19	M	mocked API; screenshot harness, no DOM assertion beyond the capture	0 / 0 / 0 / 0 / 0	the 19 capture routes
mobile-feed-scroll.spec.ts	0 / 1	M	mocked API; @mobile-only	3 / 0 / 0 / 0 / 0	/runs/{id}
mock-capture-desktop.spec.ts	9 / 9	M	mocked API; screenshot harness	0 / 0 / 0 / 0 / 0	the 9 desktop capture routes
mock-conformance.spec.ts	9 / 9	M	mocked API; conformance of screens to fixtures	4 / 0 / 0 / 2 / 0	the 9 conformance routes
optimistic-conflicts.spec.ts	2 / 0	M	mocked API; describe @desktop-only	11 / 0 / 0 / 2 / 0	/runs, /projects/{id}/backlog
visual.spec.ts	20 / 20	M	mocked API; toHaveScreenshot	6 / 0 / 0 / 2 / 2	the 20 visual routes

Real lane (web/tests/e2e-real)

Spec	Cells (desktop / mobile)	Class	Shortcut, with reference	testid / role / text / locator / css	Routes driven
auth.setup.ts	1 (project auth-setup)	A	registers and logs in through the public API (/api/auth/register line 13, /api/auth/login line 19) and stores the session — master data, allowed by rule 1	–	–
app-real-flows.spec.ts	7 / 7	D in CI (parked), A ×6 + B ×1 when run	selectors are test ids since the evening pass (what remains: five locator("body") content reads and one a[href] enumeration, not control selectors); the run-detail test (line 373, S4_NIGHTLY_SEEDED) inspects a run that scripts/ui-capture/seed.sql wrote with psql in the workflow’s “Seed nightly data” step (.forgejo/workflows/e2e-nightly.yml lines 115–137). The run is the artefact under test and is seeded, not produced by a flow — class B. The other six drive login, workspace and project creation, every reachable route surface, the legacy-route redirects and the mobile bottom bar through the UI.	0 / 18 / 6 / 13 / 2	/build, /projects, /settings?tab=workspace, /runs/{id}, every user-reachable route, the legacy stubs
app-negative-flows.spec.ts	13 / 13	D in CI (parked), A ×13 when run	selectors are test ids since the second evening pass (what remains: four locator("body") content reads and one status-text read); API reads via page.request.get (lines 84–110, 488, 580) are additional consistency checks through the public API (rule 3 allows them beside a DOM assertion); the admin request context (line 135) and request.delete (line 455) are calls a signed-in admin can make (rule 1). No database access.	12 / 26 / 9 / 38 / 29	/login, /, /admin, /build/apps/new, project / run / deploy / marketplace / invite flows

Helper drivers (scripts/e2e, not Playwright)

The #1161 measurement drivers run against a booted stack from the command line. They are listed because rule 1 applies to them as well.

Driver	Class	Shortcut, with reference	Callers
raise-gate.mjs	A	mints the task session through the product (POST /api/workspaces/{id}/tasks/{task_id}/e2e-session, owner/admin, only with SUPACLOUD_E2E_TASK_SESSION_MINT on the bench server; MODE=prod refuses the flag at boot). Until 2026-09-04 this was the one class-B stamp in the driver set (UPDATE tasks SET session_id through psql); everything after it (the MCP handler, the task_tool_approvals insert, the hub event, the Telegram push) was already the real path, and now the entry is too.	telegram-surface-check.mjs, form-surface-check.mjs
all other drivers	A	public API and the chat surfaces only; telegram-surface-check.mjs reads a count through psql (line 468) as an additional check beside the surface assertion	–

The former B row was replaced by a product capability, not a driver change: the E2E-gated session mint (the same shape as the handbook’s E2E clock, a capability only the E2E mode exposes). The bench server sets the flag in boot-nightly-stack.sh.

Correction of 2026-09-04, evening

The first version of this page counted the mock lane and wrote “A” over the real lane. It was wrong: both real-lane spec files began with test.skip(!process.env.E2E_REAL_QUALIFICATION, …), and nothing sets that variable — not the nightly workflow, not package.json, not a config. The last green nightly (2026-08-29, task 206965) ran “41 tests: 40 skipped, 1 passed” — the one passing cell is auth.setup. The lane has been an empty green since 2026-08-06 (commit 34d95957a: the #791 C2/C3/E1 qualification classes rode red in every nightly, #840, so they were parked; #791 records reactivation as per-lane and owner-decided). The parking stands until the owner rules; the MECHANISM changed on feat/e2e-real-lane-testids: every parked cell carries @qualification, the configs exclude the tag at project level unless E2E_REAL_QUALIFICATION=1, and the listing shows 1 test (parked) or 39 (qualification on) instead of 40 silent skips. The same pass turned the seed precondition into @nightly-seeded, the two layout skips into project tags, and removed the real lane’s own sleeps (three 100 ms pacing sleeps and two retry backoffs), which the first version had not counted either.

Counts

Class	Cells / sites	Where
A	40 real-lane cells + auth.setup — only when E2E_REAL_QUALIFICATION=1; in the nightly as it runs today they are D	tests/e2e-real
B	1	app-real-flows.spec.ts:373 (seeded run)
C	0	–
D	40 in CI (the parked real lane, see the correction above); 0 test.skip calls remain in either lane	19 mock-lane skips became project tags on 2026-09-04 (app-flows 6 @mobile-only, delivery-debt-surfaces 9 @desktop-only, core-loop / crud-matrix / optimistic-conflicts describe-level @desktop-only, mobile-feed-scroll @mobile-only)
E	0	no created_at / updated_at write in scripts/e2e or the specs
F	0	every driven route is a signed-in user’s or admin’s route; the legacy stubs are driven on purpose to assert the redirect
M	208 cells	tests/e2e
Rule 6 debt	real lane: cleared on 2026-09-04 (content reads excepted); mock lane: role 178, text 70, .locator( 46 (27 class- or id-anchored) still open — the mock lane is not E2E by the standard, the migration there is a hygiene pass	per spec above

Mechanics — where each rule stands

Handbook rule	State on 2026-09-04
Playwright, Chromium primary	Yes. Firefox and WebKit smoke projects are absent: the CI image ships its browsers pre-installed (PLAYWRIGHT_BROWSERS_PATH, no npx playwright install in the workflows), so the two extra browsers arrive with the image (bw-infra scripts/ci-images) and the configs then add the two projects.
trace: on-first-retry, video: retain-on-failure	Yes, all three configs.
Auto-retry at most once	Yes: retries: process.env.CI ? 1 : 0 in all three configs (was 2).
No waitForTimeout / sleep	Yes: 0 calls in tests/e2e and tests/e2e-real. Seven sites were rewritten on 2026-09-04: badge-coordinator.spec.ts (three sleeps → expect.poll on the per-tab fetch counter), core-loop.spec.ts (one sleep → page.waitForResponse on the snapshot GET that the test’s own route handler primes), mobile-capture / mock-capture-desktop / mock-conformance (one sleep each → settleForCapture in route-helpers.ts: network idle, fonts ready, no aria-busy, two painted frames).
No test.skip / fixme; pinned defects use test.fail() + ticket	Yes: 0 skips; no defect is pinned today, so 0 test.fail().
data-testid only	No — see the rule-6 debt row; the migration is additive commits on the product (a test id is added where the test looks), spec by spec.
Falsification proof for every rewritten spec	Recorded below for the 2026-09-04 rewrite.
Spec truth table	This page.
≤ 10 minutes wall-clock	Mock lane: 3.5 min locally (208 cells, 8 workers); 9.8 min in the nightly mock-e2e-full job on the 4-CPU runner, build included — at the budget’s edge, so the build belongs in its own step before Playwright’s clock starts. Real lane: the nightly job’s Playwright step; its duration is read from the workflow run once the lane starts again (PR #1265).
Local only, never staging	Yes: the real lane boots its own stack (scripts/e2e/boot-nightly-stack.sh), the mock lane needs no backend. No config points at a stage or production origin.
Every production incident names the spec that would have caught it	2026-09-01: the nightly real-backend lane stopped starting (act validated the workflow’s --memory=12g against the 6 GiB runner lane cap and refused the job before its first step). No spec can catch a lane that does not start; the guard is the workflow declaration (--memory=6g, PR #1265) plus the consumer-side memory guard in bw-infra.

Falsification record — 2026-09-04 mechanics rewrite

Rule 7 asks for three runs per rewritten spec: (a) the old spec stays green with the product path broken, (b) the new spec goes red on the same break, (c) the new spec is green on the intact path. The rewrite of 2026-09-04 changed waits, not assertions — the old specs already asserted the fetch count and the follow snapshot, they merely slept a fixed time before doing so. Run (a) therefore cannot show a gap the old spec had; what the runs prove is that the new waits are real conditions (red when the path is broken) and that the intact path is green.

Run	What	Result
(c) intact	full mock lane, both projects, prebuilt web (PLAYWRIGHT_PREBUILT_WEB=1), --ignore-snapshots as the CI job runs it	see the lane history below: the run before the fixture fix reproduced the CI failure exactly (204 passed, 4 failed, 2.1 min); the final run after the fix is recorded there
(b) broken, badge	the product’s badge fetch (inbox-actionable.ts, both fetch sites) never happens; rebuilt; badge-coordinator.spec.ts	red on both tests: expect.poll on the per-tab fetch counter times out after 10 s (“no tab fetched … hits=[0,0,0]”)
(a) old, badge	the pre-rewrite spec on the same broken build	red as well: after its fixed 2.5 s sleep the same counter assertion fails — the rewrite changed the wait, not the assertion, so there is no gap for (a) to expose
(b) broken, follow	the cold snapshot GET is never answered (the mock backend hangs it); core-loop.spec.ts follow test	red: page.waitForResponse on the snapshot never resolves (test timeout 30 s) — the wait is a real condition, not a no-op
(a) old, follow	the pre-rewrite spec on the same hang	red too, later and less precisely: its expect.poll on the request counter passes (the request fired), the 400 ms sleep elapses, and the resync’s backfill assertion then fails

Lane history, read from the Forgejo actions database on 2026-09-04

Lane	State	Since	Cause	Fix
Mock full (mock-e2e-full)	red	2026-09-02 (green on 08-31 and 09-01)	marketplace-subscription.spec.ts carried a fixed paid-through date (2026-09-01); on 09-02 the card started saying “access has ended” — 4 cells red on both projects, 204 green, 9.8 min on the 4-CPU runner including the build	fixture dates and the two date expectations follow the run clock (daysFromNow, calendarDay) — 14/14 locally
Mock full, local	red on any dev box with a backend on :8080	—	the preview’s SSR auth gate resolved the cookie against a stray local backend and redirected to /login before the in-page mocks were reached (≈60 cells)	webServer.env.BACKEND_URL pinned to a dead port in playwright.config.ts
Real nightly (e2e-nightly)	red	2026-08-30	08-30/31: a comment inside the continued docker run chain of boot-nightly-stack.sh ended the command before the image name (already fixed on main, the script documents it); 09-01 onwards: act refused the job before its first step (--memory=12g against the 6 GiB lane cap)	PR #1265 (--memory=6g) plus the consumer-side guard in bw-infra
Smoke core-loop (smoke-e2e, PR net)	green	—	—	—
Mock full, local final	green	2026-09-04	the final intact run after the fixes, both projects, --ignore-snapshots as in CI	208 of 208 cells passed in 3.5 min (the pre-fix run: 204 / 4 in 2.1 min)