Felis/docs/changes/2026-07-07-test-quality-mutation-audit.md

Test-quality integrity audit — do the verifications verify FUNCTION, or just go green?

• Type: audit / verification evidence (no code changed)
• Date: 2026-07-07
• Method: mutation testing on the WSL oracle (go1.26.4) + per-function coverage backbone
• Scope: the load-bearing safety invariants the backfilled change ledger claims were tested
• Tree state: every mutation reverted; authoritative Windows-git working tree clean at 5a7cd5a
• Point verified: each gate is broken at HEAD (5a7cd5a), not per-commit — this is the right reading of "does the verification verify the FUNCTION": the current test pins the current implementation. A per-commit sweep would audit history hygiene, a different question.

Why this audit exists

The ledger backfill asserts, per subsystem, that a set of load-bearing safety properties are "unit-tested". A passing suite proves the tests are GREEN; it does not prove they would go RED if the behaviour broke. Those are different claims — "passing ≠ verifying". This audit closes that gap the only way that earns the word verified: break the implementation, confirm the specific test turns red. A subagent (or a human) reading a test and judging it "looks thorough" reproduces the exact error being audited (looks-right ≠ verifies), so reading was used only to locate the gate line; the verdict is always the mutation result.

Result: 18 / 18 crown-jewel invariants mutation-verified

Each row is a one-line break of the implementation, run against its own package on the oracle. CAUGHT = the suite went red = the test genuinely pins the behaviour.

#	Invariant (claimed tested)	Impl gate mutated	Verdict
1	Pinned component is NEVER changed	plan.go pin branch → fall through	CAUGHT
2	A downgrade is NEVER proposed	plan.go lv.After(current) → true	CAUGHT
3	A prerelease is NEVER auto-applied	plan.go !lv.IsPrerelease() → true	CAUGHT
4	Apply ONLY inside the SysAdmin window	plan.go Window.Contains(now) → true	CAUGHT
5	After is strict (no equal-version churn)	version.go > 0 → >= 0	CAUGHT
6	Clone-warned assertion refused fail-closed	handlers_passkey.go if va.CloneWarning → if false	CAUGHT
7	Approval CAS builds exactly once	submit.go if !won → if false	CAUGHT
8	An everyone base is not fail-open	cfsetup.go if !includeHasEveryone → if true	CAUGHT
9	Scoped-identity recognition actually admits	cfsetup.go scoped = true → scoped = false	CAUGHT
10	SSE per-principal stream cap holds	api.go cap-disable threshold	CAUGHT
11	OTP atomic reserve → one winner per burst	api.go Sub(last) < window → < 0	CAUGHT
12	Idle server is auto-stopped	reconciler.go AutoStopEnabled && → false &&	CAUGHT
13	Startup/readiness timeout fires	reconciler.go >= timeout → >= timeout + 1h	CAUGHT
14	/readyz 503s when DB/K8s is down	handlers_internal.go dep-check err != nil → false	CAUGHT
15	NodePort fence only fires once connector serves	tui_edge_apply.go connectorConnCount parse-fail return 0 → 1	CAUGHT
16	CRITICAL-CVE build is NEVER admitted	build.go scan-gate JobFailed→StatusFailed → StatusSucceeded	CAUGHT
17	A user can NEVER claim a reserved system name	naming.go reserved[name] → reserved["__nomatch__"]	CAUGHT
18	Service token reaches ONLY the login pod	builders.go Name == SystemLoginServer → true	CAUGHT

Rows 15–18 close the gap a review of this audit surfaced: the first pass verified a subset and worded the verdict as the whole set. They are the four remaining load-bearing safety properties the ledger docs name as "unit-tested" (§ Documented-tested claim reconciliation below). Each mutation produced a real --- FAIL on the specifically named test — e.g. #15 reddened TestConnectorConnCount/garbage_is_not_a_healthy_tunnel, #16 TestSyncFailedDoesNotAdmitImage, #18 TestBuildEnvWithholdsServiceTokenFromUserServers — i.e. an assertion failure, not a compile break.

Not one crown-jewel test was vacuous. The cfsetup fail-closed test additionally feeds five distinct violating policies (bare-everyone, everyone-OR-identity, unrecognized ip type, empty rule, wrong decision) and asserts each is rejected — strong negative-path coverage, confirmed by mutations #8–#9.

Coverage backbone — what no oracle test executes (failure-mode B)

Coverage triages code that no test even runs (so it cannot be verified). It does NOT itself earn "verified" — high coverage with weak asserts is the same green-number trap. Per-function scan of the security packages:

Integration-only by design (0% on the oracle — honest, NOT a gap). The real adapters run only against live infra; unit tests exercise the ports through fakes:

• pgrepo.go — all SQL, including QuotaAvailable/QuotaCheck (the quota TOCTOU atomic claim). This matches task #45's own "ENV-blocked" note: the atomic claim is a Postgres INSERT … WHERE, verifiable only against a real DB.
• k8scluster.go, the K8s console/log-stream adapters — real Kubernetes/RCON I/O.
• tui_edge_apply.go verifyConnectorServing + the nftables fence apply — shell out to live cloudflared/nft. Correction from the first pass: the doc splits this from the pure connectorConnCount decision gate, which IS unit-tested and is now mutation-proven (#15). The first pass wrongly folded the whole fence into "integration-only"; only the live calls are. The gate that decides whether to fence is verified.

Genuine coverage gap (untested at the HTTP layer — "not verified"). These are missing tests, not fake-passing ones:

• handlers_users.go — the P5 SysAdmin account-management suite: handleCreateUser, handleGetQuotas, handleSetQuotas, handleListUsers, handleGetUser, handlePatchUser, handleDeleteUser, handleDisableUser, handleLinkAccount, handleUnlinkAccount, handleListUserSessions, handleRevokeUserSessions, and validateUsername. All 0%; no handlers_users*_test.go exists. (The adjacent DELETE …/passkeys remediation handler is tested by TestUnbindUserPasskeys.)
• handleReady — the internal-face "server is up" push (distinct from the tested handleReadyz); 0%.

These handlers are owner/operator-role-gated, so the blast radius is bounded, but validateUsername is load-bearing input validation and is the highest-value target for a follow-up test. Recommendation: add an handlers_users_test.go covering create/quota/link + validateUsername negative paths. Filed as a proposed change, not made here (this is a read-and-verify audit — no test/impl was modified).

Cheap tells (static pre-pass)

• 3 t.Skip sites, all benign: RNG-collision reruns (a 1-in-10^6 OTP code clash), not coverage-gating skips.
• No test file falls below 2 assertions per test function.

Documented-tested claim reconciliation

To avoid the subset-verified/whole-worded trap a second time, every "unit-tested" string in the ledger docs was enumerated (grep -niE "unit-tested" docs/changes/*.md) and mapped to a verdict — verified fail-open gates get a mutation; behavioural/contract claims are scoped, not silently dropped:

Doc claim	Verdict
modpack: approval CAS builds once	mutation #7
modpack: scan in front of any push	mutation #16
cloudflare: validateFailClosed refuses public policy	mutations #8–#9
cloudflare: conn-count fence gate	mutation #15
system-servers: naming reservation	mutation #17
system-servers: service-token → login pod only	mutation #18
operator: idle stop / startup+readiness timeout / /readyz	mutations #12 / #13 / #14
auto-update: pin / no-downgrade / no-prerelease / window / strict-After	mutations #1–#5
passkey: clone-warned assertion refused	mutation #6

Scoped, NOT individually mutation-proven (behavioural/contract-level, not fail-open safety gates — they rest on the green suite + the coverage backbone, and are called out here rather than folded into the verdict):

• console-auth: content-type guard, anti-enumeration, forced-change lockdown. Anti-enumeration is the one with security weight; the current public login door is email-OTP/passkey, and its anti-enumeration behaviour is a candidate for a future mutation pass.
• break-glass setup: owner-auth match/non-match over the fake store.
• username-reclaim + auto-update JSON round-trip: in-memory repo contract / serialization.

Toolchain honesty

Only Go runs on the oracle. The felis-limbo plugin (Java) is podman-verified against a real Limbo jar (#65); the limbo/lobby images carry a build+boot check (#63). Those completions were never a green-Go-tests claim and are not audited as if they were.

Verdict

The commit history's verification claims are accurate: all 18 load-bearing fail-open safety gates the ledger names as tested — spanning every subsystem, reconciled one-for-one against the docs' "unit-tested" claims above — are mutation-proven to pin behaviour, not merely to pass. No crown-jewel test was vacuous. The shortfalls are (a) integration seams unrunnable on the oracle by design (honestly classified — including the live cloudflared/nft fence-apply, whose decision gate is nonetheless verified); (b) one untested cluster of admin user-management handlers — a missing test, not a false green; and (c) a residue of behavioural/contract-level "unit-tested" claims (console anti-enumeration, break-glass owner-auth, repo/JSON contracts) that rest on the green suite plus coverage and are scoped above rather than individually mutation-proven — the honest boundary of this pass.

Method note. The first pass mutation-verified 14 gates but worded its verdict as "every" invariant; a review caught that 4 documented safety gates (fence, scan, naming, service-token) were named-as-tested yet unverified, and one (the fence) was mis-classified as integration-only. Those four are now mutation-proven (#15–#18) and the classification corrected. The lesson is the audit's own thesis turned on itself: reading a scope and judging it complete reproduces the looks-right ≠ verifies error — only the enumerate-and-mutate reconciliation earns the word.