Real-machine closure for everything batched since batch 45: #75/#76 verified
end to end on auditfix77 (cooldowns 429, pending cap 403, 2GiB budget 403 via
the real accounting path, failure-does-not-burn-window, withdraw/admin-delete
row+blob double clean, panel two-step confirms in CDP), tracker #8 and #1
landed and verified (#77 panel redirect, #78 felis converge), plus #79 docs.
Also: pgint's new assertions first run on real PG (17/17) and the release
pipeline's first-ever run — tag v0.1.0-rc1, multiarch assets published and
executed on the target arch, /releases/latest deliberately untouched.
The legend said "§12 lists the one field this still applies to", but the last
inert field (spec.storage.retainOnDelete) was removed rather than implemented
(§13), and §12 has said "every field below is read by a controller" since.
Reworded so a reader whose change looks ignored follows the condition question
instead of hunting for a dead field.
Provisioning is create-if-absent, so a field the desired spec gained after an
install (spec.rcon, spec.startup.healthHTTPPort, a derived env key) never
reaches the existing login/lobby CR while every re-run of setup reports
success — the reported 'configuration updates never reach an installed
deployment' symptom. converge is the explicit pass: it fills exactly the
zero-valued whitelist fields and the derived env (including a missing key,
which refreshDerivedEnv deliberately never adds), and never overwrites a
non-zero value. The timing stays with the operator because enabling RCON or
the HTTP readiness gate on a pre-listener image would wedge that server in
Starting until it was marked Failed.
Tests: fills predated fields while operator edits survive / non-zero values
left alone / absent + foreign + unset-image guards. usage table updated so the
router-parity test passes; troubleshooting gains §12b.
A session that still owes forced onboarding gets 403 setup_required from every
protected route, but the panel rendered it as the generic forbidden line — the
one step that unlocks the app read as missing authorization. api.ts now
announces the code on a window event (client module has no router) and the App
shell, inside the Router, navigates to /setup; the wizard resumes from the
surviving session with or without a token. Other 403s are untouched.
Panel tests: +2 (fires on setup_required, silent on any other 403).
Nothing ever removed a submission: users could not retract a pending row, no
route deleted blobs or rows, and the reaper never touches uploads — so every
upload accumulated on the 5 GiB PVC forever and the only cleanup was SQL or
kubectl against the store.
- Blobs.Delete on both transports (local: RemoveAll of the id-namespaced dir,
id re-validated at the boundary; S3: idempotent object DELETE).
- Store: DeleteSubmission (admin, any status) and DeletePendingSubmission
(owner+pending CAS — a reviewed row can never be withdrawn out from under
its build).
- Manager.Delete / Manager.Withdraw delete the ROW first (under the CAS for
withdraw) and the blob after, so a live row can never point at a reaped
blob; a cleanup failure names the orphan explicitly instead of failing mute.
- API: DELETE /me/submissions/{id} (withdraw, app tier) and
DELETE /api/v1/submissions/{id} (admin) both return the row as it was;
audit events submission.withdraw / submission.delete; openapi documents both
paths; admin route pinned in the admin-only table.
- Panel: two-step withdraw on a pending row (frees the pending slot and the
storage budget); two-step delete on every admin row; zh/en copy; wire tests.
Unit: submit (withdraw happy path / wrong owner / reviewed row / no transport /
blob-cleanup failure), local+S3 delete idempotence, api handlers (200/404/409/
503 + route tier); pgint: withdraw CAS + admin delete exactly-once.
go vet/go test/gofmt clean; panel vitest 120 + typecheck green.
A logged-in user could file submissions without bound and stream a 1 GiB
context per submission. The only limits were the single-blob size cap and the
5 GiB uploads PVC (platform/workloads.go); nothing counted a user's rows or
bytes, so one account could fill the volume and every other user's upload
would start failing.
- Create: per-user pending_review cap (default 5) — the review queue cannot
be parked full of one account's rows. Check-then-insert, documented soft.
- UploadContext: per-user stored-context budget (default 2 GiB) charged
against the blob store's REAL sizes (new Blobs.Size on local/S3 stores), so
the sum cannot drift from the volume; the write is capped at the remaining
budget, so the excess is refused before it is persisted, and a re-upload is
charged only for its new bytes.
- API: per-user create/upload throttles (30s/15s, cmd/felis-wired) on a
dedicated cooldown keyspace, reserve→release so a failed attempt never
burns the window and a burst collapses to one winner; ErrQuotaExceeded →
403 submission_quota_exceeded (distinct from the 400 an oversize blob
gets), 429 submission_cooldown for the throttles.
- Panel: zh/en copy for both codes; openapi documents 403/429 on the two
user routes; pgint covers the pending-queue count.
Unit tests: submit package (cap, budget boundary/exact-fit/replacement,
oversize-vs-quota split) and api handlers (quota 403 both paths, throttle
429 + recovery + failure-release). go vet/go test/gofmt clean; panel
vitest 118 + typecheck green.
Six CR-level run/stop cycles with three chaos injections (operator pod
kill, postgres restart, api pod kill) all converged (Running 23-29s,
Stopped 3s, pods gone per cycle); PG outage keeps the 503-not-401
session semantics; 30-request burst all 200; ownerless wake correctly
403s. The build lane's missing Job TTL (#67, fixed in 2755e41) was
verified live on auditfix62: a real build Job carries ttl=604800 and an
old Job patched to ttl=30s was reaped, pod and all, within 40s.
Every other Job family carries a TTLSecondsAfterFinished (fileedit 2m,
backup/restore 10m) but the build lane never set one: each build left a
completed Job + Pod in felis-build indefinitely (5 already on the drill
cluster, oldest 26h), growing etcd and — since completed pods count
against the node's pod budget (110 on stock k3s) — eventually blocking
new builds. The code even anticipated GC it never got ("JobUnknown
means the Job was not found (e.g. GC'd)").
Set a deliberately long TTL (7 days): the kaniko log is the admin
failure-triage surface (GET /images/build/{id}/logs), so the window
keeps a week of logs while bounding steady-state pods; terminal builds
are idempotent under Sync, so a late log-404 is the only cost.
The release workflow has never executed (no tags/releases exist yet).
Its three load-bearing contracts were verified against their other
halves: the asset names vs bootstrap's download_release_binary (and its
'felis ${REF}' convergence check), the local-exporter path vs the
Dockerfile's final COPY, and the version assertion vs cmdVersion's first
line. The one un-replicable check (file(1) machine type) was probed
with a cross-compiled linux/arm64 binary: 'ARM aarch64' matches. No
pre-tag fixes needed.
Module-by-module coverage matrix against the ledger: repo hygiene clean,
no TODO/FIXME debt, CLI dispatch test-guarded, CI syntax-checks every
tracked shell script, all 22 internal packages accounted for (CRD types
and game-image dirs covered indirectly), panel's 17 page routes drilled
in earlier batches. Fixes this batch: #66 README_EN was missing the
private-repo install workaround and the rerun/upgrade notes (an English
reader 404s on step one); the §28 claim/link sequence diagrams were
realigned with the audited implementations. A local-only scripts/
sync.sh conflict (plugins/ exclusion vs go:embed) was reproduced and
fixed on the workstation — private per .git/info/exclude, not committed.
The claim-transaction note still described the pre-audit-#4 shape (SELECT
EXISTS plus a quota pre-check outside the transaction); the implemented
ClaimServer serializes on pg_advisory_xact_lock(user_id), takes the row
FOR UPDATE and re-runs the four-dimension gate inside the same
transaction — the diagram's QuotaAvailable step is only a fast path.
The link flow now selects the code FOR UPDATE, treats a same-user
re-verify as idempotent, and lets a live caller take over a retired
(soft-deleted) owner's link — the 409 is only for a different, live user.
Both re-read from internal/api/pgrepo.go and the handler mappings.
README.md grew a required workaround — the repository is private, so the
plain raw.githubusercontent one-liner returns 404. The credentialed form
(token handed to curl via --config - so it never touches argv, sudo -E so
the installer inherits it) and the rerun/upgrade notes (the rerun upgrades
felis-api; the channel is not inherited, so main-followers need
FELIS_VERSION_BOOTSTRAP=dev) never made it into README_EN.md, which is
linked as the English entry point. An English reader following that page
could not install at all. Bring it back in sync with the Chinese one.
The last module family nobody ever built (#64 gradlew exec bit, #65
license metadata), closed with three real dedicated-server boots
(fabric/forge/neoforge: mod loads, /link registers, console refused)
and a permanent JDK-17 gate in CI + release. Reachability table now
#1–#65; conclusions item 20; remaining-queue note updated. CI run
35892544563 = 5/5 with the new mods job green on its first run.
Nothing ever compiled these modules — no CI job, no install path — which is
why the gradlew exec-bit bug (previous commit) shipped unnoticed. Add
plugins/test-mods.sh: runs each module's vendored wrapper under JDK 17
(they target the Java-17 Minecraft lines; paper/limbo stay on the JDK 21
gate). ci.yml gains a `mods` job (temurin 17 + setup-gradle, wrapper
pinned per module), release.yml gates both plugin gates before shipping.
README: status now records the server-boot verification, the Building
section pins limbo's `-PlimboVersion=<release>` (the `+` default is
unresolvable from the LOOHP repo) and documents both gates.
The three loader mods declared `license = "MIT"` (and Forge/NeoForge a
placeholder `issueTrackerURL = https://example.invalid/felis`) while the
repository is AGPL-3.0-only (README.md:73, LICENSE). The mods were added
2026-06-26, the LICENSE landed 2026-07-12 — stale leftovers that nothing
ever read back: no CI job, no install path. Fabric loader prints the
license from fabric.mod.json at boot and both mods.toml files are parsed
by their loaders, so the wrong claim is user-visible. Align both with
reality — boot-verified on real fabric/forge/neoforge dedicated servers
(AUDIT-2026-09-22.md, batch 41).
All three vendored wrappers were tracked as 100644, so the README's documented
'plugins/<loader>/gradlew -p plugins/<loader> build' failed on every fresh
clone with 'Permission denied' (exit 126) — and since no install path or CI job
ever ran them, nothing caught it.
git update-index --chmod=+x for the three files; the VM then built all three
modules for the first time (results in the gate commit and the audit).
#63 fixed in f5a76cf (demo-up delegates to bootstrap; T1/T2/T3 real-machine);
plugins/test.sh + the new CI plugins job verified on the VM and in CI
(c59b387, run 35888009965, 4/4 jobs); #62 re-checked against all three
install paths and screened out as unreachable/nonexistent; plugin-layer
real-machine E2E via the MC status/login probes recorded.
The three framework-free test mains under plugins/*/test were never run by
anything — not CI, not the plugin builds — and the velocity/paper/limbo jars
were only ever compiled by deploy/bootstrap.sh on a live host. CI gains a
'plugins' job (JDK 21 plus the Gradle 8.14 the plugin Dockerfiles pin) running
plugins/test.sh: the three mains (InviteCardTest's jars fetched from Maven
Central, pinned and digest-checked) and the three production builds.
The first real run surfaced and fixed two untested assumptions: InviteCardTest's
documented javac line omitted examination-api (adventure-api's Component
signatures reference Examinable, so javac needs it too), and limbo's '+' version
default cannot resolve — LOOHP's repository serves no maven-metadata — so the
script resolves the current release off the Limbo CI artifact name (the same
source bootstrap reads) and plugins/README.md stops advertising a bare
'gradle -p plugins/limbo build' that can never work.
Verified in gradle:8.14-jdk21 on the VM: mains OK (32/36/48 checks);
velocity/paper/limbo BUILD SUCCESSFUL.
demo-up.sh grew its own image lane in July, before bootstrap could build the
game stack. The copy has since drifted from the installer it duplicates: it
pinned Paper 1.21.8 while bootstrap derives one version from the Limbo login
gate (both hops of a login must speak one protocol), it never built
felis-velocity.jar (a proxy without it silently routes nothing), it imported
images under local tags the [velocity] wiring no longer points at, and it left
the docker daemon running.
It now runs deploy/bootstrap.sh (unless SKIP_BOOTSTRAP=1), checks that the
earlier run really left the full stack behind, and hands over to 'felis setup'
— so a half-installed base is a clear error instead of a proxy that accepts
logins and routes nowhere.
Verified on the VM: healthy base passes to the handoff (exit 0); a hidden
felis-velocity.jar dies naming the jar and the remedy (exit 1, jar restored).
Live with real LuckPerms 5.5.85: every lp command returns an empty RCON body
(list/plugins answer normally; a standalone RCON client sees the same, and
creategroup/permission-set still persist), so the read projection can never
populate and the page asserted "no parent groups / no explicit nodes" for a
state it could not actually read. The raw reply now rides the same disclosure
the rosters carry, a silent entry-less reply shows an explicit notice instead
of the false empty claims, and the write history's placeholder no longer
dresses up a fabricated "[RCON] ..." line as output.
Live: the files page against a server whose world claim does not exist (never
started, or reaped) created a Job whose Pod stayed Pending on
FailedScheduling (persistentvolumeclaim not found) until the executor's 90s
wait expired — a 90s spinner answered by a misleading 504 files_timeout, for
a request that is knowably impossible. Backup and restore have refused this
shape with 409 no_world_volume since the #42 round; the file routes now run
the same gate before any Job is created, and the panel maps the code to a
localized message (it previously fell back to the English server text).
sendCommand returns the RCON reply and the component's own contract says it
"displays its plain-text reply", but the reply was discarded and the pod log
does not echo command output, so a sent command produced no visible result at
all (verified live with 'list'). The last command echo + reply now render
above the prompt, terminal-style.
The whitelist/ban/kick mutations reported a canned success message and threw
away the server's reply, so a refused command still read as done: live, the
vanilla server answers "That player does not exist" for a name it has never
seen (any player who has not joined yet), while the panel said the player had
been whitelisted/banned. api.ts documents these replies as "surfaced verbatim
as confirmation"; now they are. The localized string stays as the fallback
for a silent server.
A configured Yggdrasil root answering 200 with a name outside the Minecraft charset (or an identity UUID that does not parse) was rejected one layer up in the handler: a silent 204 with no log line, and because the rejection returned instead of continuing, every source behind the broken one was unreachable for that login. The resolver already treats the same class (200 without a usable profile, non-200, unreachable) as skip + log + failed; the name/UUID screens lived above it and silently stopped the ladder instead.
Live on the audit box, a single sloppy root produced 204s with no trace anywhere, and [bad root, valid root] answered 204 where the valid root would have admitted the login; nothing else in the nano matrix (60 checks across input validation, canonical rewrite, premium rename, failure modes, failover, log discipline, properties relay) was red.
Screen both shapes inside resolveHasJoined, before a 200 can win: identity ids must parse, third-party names must match the charset. A bad answer is logged ('unusable profile name' / 'unparseable profile id'), skipped, and counted as failed — 503 when nothing else validates, and later sources get their turn. The handler's guards stay as the last line before anything leaves (comments updated).
Gates: gofmt, go vet, go test ./..., deploy/bootstrap_test.sh all clean. Green live (v0.0.0+fix55): the five bad-name cases and the two failover cases all pass; matrix rerun 60/60.
Records: (1) the /updates maintenance-window API sweep (unset null, 400s/415 for the invalid set, write->read-back->survives API pod restart, 401/403 auth) and the CDP panel sweep (status transitions, validation copy, clear, audit x3, zero console errors); (2) #53 red/green with the live fix53 binary ('MliroLirrorsIngenuity/Felis' -> 'FelisMC/Felis' in the 404 line) plus the token'd check (old path 301 / new path 200) and the fact the new home has no stable release yet; (3) #54 red/green with the live fix54 binary (installer one-liner + single trailer replace 'run: sudo felis setup'), the setup-vs-installer evidence, and the doc/test synchronization. Reachability table gains #53 (2) and #54 (2, docs); stats 54 total; repro-entry notes updated to the new remote, host binary v0.0.0+fix54 and the batch's artifacts.
On a completed install 'felis setup' never re-runs the installer: its host-bootstrap phase only runs while an install marker is missing, so it opens the config console and moves no component. Live on the audit box, a clean 'felis setup' run left /opt/felis/velocity/velocity.jar's mtime and hash untouched while an installer re-run logged 'resolving the newest Velocity 3.5.1 build'. The 'felis update' guidance was wrong three ways accordingly: 'run: sudo felis setup' for panel/velocity/plugins, the 'felis setup is idempotent and re-runs the installer' trailer, and the felis-api-only exception block, whose scoping taught the same false model for velocity.
Point every planner-backed selector at the tested path -- re-running the installer (the README's install one-liner) -- and replace the scoped caveat with one trailer: the channel is not persisted (pass FELIS_VERSION_BOOTSTRAP=dev on a host that tracks main), the private repo's one-liner needs the README's token'd form, and 'felis setup is not this path'. troubleshooting.md SS15 drops the same false alternative and gains the channel caveat.
Gates: gofmt, go vet, go test ./..., deploy/bootstrap_test.sh all clean. Green live (v0.0.0+fix54 installed to /usr/local/bin over the fix52 backup, sha 0bd49467...): --panel and --velocity print the installer one-liner plus the single trailer, --mc stays command-free, --all prints the trailer once.
The repository moved to FelisMC/Felis, but the felis-api release coordinate, the installer's default FELIS_REPO_URL, the PaperMC user-agent strings and both READMEs still named MliroLirrorsIngenuity/Felis. Live on the audit box, 'felis update' reported 'github: MliroLirrorsIngenuity/Felis releases/latest returned HTTP 404 -- ...', pointing operators at a coordinate that no longer exists; the old path keeps answering today only because GitHub still 301s the transfer (verified with a read token against api.github.com: old path 301, new path 200), and if that redirect is ever retired every install and every update check breaks with it.
Replace the coordinate in the six tracked files: the updater topology and both test fixtures, the bootstrap default URL and user-agent strings, and README.md/README_EN.md. Green live: the same command now reports 'github: FelisMC/Felis releases/latest returned HTTP 404 -- ...' (still 404 because the new home has published no stable release yet -- a release-process fact, not a code bug).
Gates: gofmt, go vet, go test ./..., deploy/bootstrap_test.sh all clean.
Lemon-miaow