feat(cli): 新增 felis status、doctor 和 support-bundle，一屏看全平台、按区域列出问题、打出脱敏诊断包

README.md

@@ -19,6 +19,7 @@ A Kubernetes-driven Minecraft server hosting platform — one command to deploy,
 - **Web 控制面板**：浏览器中查看服务器状态、在线玩家与资源用量，管理备份与恢复。
 - **备份与恢复**：一键把整服数据（世界、配置、插件/模组，即整个 /data 卷）打包进集群内的归档库，支持从任意备份点回滚；默认安装就已启用（归档 PVC 与路径由安装器一并生成）。
 - **控制面数据库备份**：账号、服务器归属、配额与存档索引所在的数据库每天自动备份，每次升级迁移前先快照，出错可用 `felis db restore` 整库原子回滚；面板「维护与备份」页显示备份是否新鲜（见 [故障排查 §16](docs/troubleshooting.md)）。
+- **运维自检**：`sudo felis status` 一屏列出节点、控制面、游戏代理、每台服务器、备份与未解决的告警；`sudo felis doctor` 把健康检查全跑一遍，按区域给出问题和下一步去哪看，不发邮件；`sudo felis support-bundle` 打出一个脱敏的诊断包，求助时直接附上（见 [故障排查 §0](docs/troubleshooting.md)）。
 - **智慧回收（可选开启）**：超过 15 天无人游玩的世界自动备份后删除，释放磁盘空间；安装时设置 `FELIS_WORLDS_HOST_PATH`（k3s 默认 `/var/lib/rancher/k3s/storage`）即启用每日回收，不设置则不删任何世界。过期备份无论是否开启都会每天清理。
 - **多核心支持**：兼容 Paper、Fabric、Forge、NeoForge，经由 Velocity 代理统一入口。
 - **模组自助提交**：玩家自行上传模组包，服主审批通过后自动构建；构建产物进入镜像白名单，可直接选用为服务器镜像完成部署。

README_EN.md

@@ -19,6 +19,7 @@ Table of Contents
 - **Web Dashboard**: Monitor server status, online players, and resource usage from your browser, with backup and restore management.
 - **Backup & Restore**: One-click snapshots of a server's whole data volume (worlds, config, plugins/mods — the entire /data volume) into the cluster's archive store, with rollback from any backup point — enabled by default (the installer renders the archive PVC and its path).
 - **Control-plane database backups**: The database holding accounts, server ownership, quotas and the archive index is backed up daily and snapshotted before every upgrade migrates it; `felis db restore` rolls it back atomically, and the panel's Maintenance & Backups page shows whether the newest backup is fresh (see [troubleshooting §16](docs/troubleshooting.md)).
+- **Self-check for operators**: `sudo felis status` shows the node, the control plane, the game proxy, every server, the backups and the open alerts on one screen; `sudo felis doctor` runs every health check once and lists each problem by area with where to look next, mailing nothing; `sudo felis support-bundle` writes one redacted diagnostics archive to attach when asking for help (see [troubleshooting §0](docs/troubleshooting.md)).
 - **World Reaper** (opt in): Worlds idle for more than 15 days are automatically backed up and removed to free disk space. Enable it by setting `FELIS_WORLDS_HOST_PATH` at install time (on k3s: `/var/lib/rancher/k3s/storage`); without it, no world is ever deleted.
 - **Multi-core Support**: Compatible with Paper, Fabric, Forge, and NeoForge, federated behind a Velocity proxy.
 - **Modpack Submission**: Players submit custom modpacks; admin approval triggers an automatic build, and the result is whitelisted as a server image you can select to deploy.

cmd/felis/doctor.go

@@ -0,0 +1,387 @@
+package main
+
+import (
+	"context"
+	"errors"
+	"flag"
+	"fmt"
+	"io"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"slices"
+	"strings"
+	"time"
+
+	"felis.lolicon.best/internal/config"
+	"felis.lolicon.best/internal/watchdog"
+)
+
+// systemdUnitDir is where the installer writes its units.
+const systemdUnitDir = "/etc/systemd/system"
+
+// hostCommand runs a host tool (systemctl, journalctl, k3s) and returns its
+// stdout. A tool that exits non-zero still returns what it printed:
+// `systemctl is-active` prints "inactive" and exits 3.
+func hostCommand(ctx context.Context, name string, args ...string) ([]byte, error) {
+	return exec.CommandContext(ctx, name, args...).Output()
+}
+
+// hostServices are the long-running units a full install depends on, checked
+// when their unit file is present: k3s runs the cluster, felis-velocity is the
+// game proxy, felis-nano the single-binary host that runs without k3s.
+var hostServices = []string{"k3s.service", "felis-velocity.service", "felis-nano.service"}
+
+// cmdDoctor runs every check felis watchdog runs, with the settings
+// felis-watchdog.service gives it, plus what only the host shows (systemd
+// units that failed or stopped, timers that no longer fire, alerts that reach
+// no one), and prints them grouped by area with where to look next. It mails
+// nothing, pings no heartbeat and leaves the watchdog's state alone: it is
+// safe to run at any time, as often as wanted.
+func cmdDoctor(args []string, stdout, stderr io.Writer) int {
+	fs := flag.NewFlagSet("doctor", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	unitDir := fs.String("systemd-dir", systemdUnitDir, "where the installer's systemd units are")
+	if err := fs.Parse(args); err != nil {
+		if errors.Is(err, flag.ErrHelp) {
+			return 0
+		}
+		return 2
+	}
+	if os.Geteuid() != 0 {
+		fmt.Fprintln(stderr, "felis doctor: run as root (sudo felis doctor): the checks read root-only state under /etc/felis and /var/lib/felis")
+		return 1
+	}
+	host, _ := os.Hostname()
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+	defer cancel()
+	return runDoctor(ctx, doctorEnv{unitDir: *unitDir, run: hostCommand, now: time.Now(), host: host}, stdout)
+}
+
+// doctorEnv is what one doctor run reads the host through.
+type doctorEnv struct {
+	unitDir string
+	run     func(ctx context.Context, name string, args ...string) ([]byte, error)
+	now     time.Time
+	host    string
+}
+
+// doctorAreas are the report's headings in order, by the area findingArea
+// puts a finding under.
+var doctorAreas = []struct{ key, title string }{
+	{key: "config", title: "configuration"},
+	{key: "cluster", title: "Kubernetes cluster"},
+	{key: "postgres", title: "PostgreSQL"},
+	{key: "proxy", title: "game proxy"},
+	{key: "db-backup", title: "database backups"},
+	{key: "offsite", title: "off-site copy"},
+	{key: "scan-db", title: "build scan database"},
+	{key: "disk", title: "disk space"},
+	{key: "memory", title: "memory"},
+	{key: "k3s-certs", title: "k3s certificates"},
+	{key: "host-address", title: "node address"},
+	{key: "clock", title: "clock"},
+	{key: "systemd", title: "systemd units and timers"},
+	{key: "alerts", title: "alerting"},
+}
+
+func runDoctor(ctx context.Context, env doctorEnv, stdout io.Writer) int {
+	unit := filepath.Join(env.unitDir, "felis-watchdog.service")
+	w, found, unitErr := watchdogUnitFlags(unit)
+	var report watchdog.Report
+	var notes []string
+	fmt.Fprintf(stdout, "felis doctor on %s at %s\n", env.host, env.now.UTC().Format("2006-01-02 15:04 UTC"))
+	switch {
+	case unitErr != nil:
+		report.Findings = append(report.Findings, watchdog.Finding{
+			Key: "watchdog/unit", Severity: watchdog.Critical,
+			SummaryEN: fmt.Sprintf("cannot read the watchdog's settings: %v", unitErr),
+			Hint:      "rerun the installer (deploy/bootstrap.sh) to rewrite felis-watchdog.service",
+		})
+		fmt.Fprintf(stdout, "checks run with the watchdog's defaults (config %s)\n", w.cfgPath)
+	case !found:
+		report.Findings = append(report.Findings, watchdog.Finding{
+			Key: "watchdog/unit", Severity: watchdog.Critical,
+			SummaryEN: fmt.Sprintf("%s is not installed: nothing checks this host or mails anyone when it breaks", unit),
+			Hint:      "rerun the installer (deploy/bootstrap.sh), which installs felis-watchdog.timer",
+		})
+		fmt.Fprintf(stdout, "checks run with the watchdog's defaults (config %s)\n", w.cfgPath)
+	default:
+		fmt.Fprintf(stdout, "checks run as %s runs them (config %s)\n", unit, w.cfgPath)
+	}
+	fmt.Fprintln(stdout)
+
+	skip := map[string]string{}
+	cfg, cfgErr := config.Load(w.cfgPath)
+	if cfgErr != nil {
+		report.Findings = append(report.Findings, watchdog.Finding{
+			Key: "config", Severity: watchdog.Critical,
+			SummaryEN: cfgErr.Error(),
+			Hint:      "the installer writes it (deploy/bootstrap.sh); felis watchdog fails on every run until it loads",
+		})
+		// The host's units are read without it; whether alerts reach anyone
+		// is not known without its relay.
+		for _, a := range doctorAreas {
+			if a.key != "config" && a.key != "systemd" {
+				skip[a.key] = "the configuration did not load"
+			}
+		}
+	} else {
+		_, owners, ownersErr := watchdogProbes(ctx, w, cfg, env.now, &report)
+		report.Findings = append(report.Findings, alertReachFindings(cfg, owners, ownersErr)...)
+		if w.proxyAddr == "" {
+			skip["proxy"] = "no -proxy-addr"
+		}
+		if w.backupDir == "" {
+			skip["db-backup"] = "no -backup-dir"
+		}
+		if !cfg.Offsite.Enabled() {
+			skip["offsite"] = "not configured"
+		}
+		if !usesMirroredScanDB(cfg) {
+			skip["scan-db"] = "builds do not scan against the registry's copy"
+		}
+		if len(splitList(w.certDirs)) == 0 {
+			skip["k3s-certs"] = "no -k3s-cert-dirs"
+		}
+		if w.nodeIP == "" {
+			skip["host-address"] = "no -node-ip"
+		}
+	}
+	report.Findings = append(report.Findings, unitFindings(ctx, env)...)
+
+	switch url, err := readHeartbeatURL(w.heartbeatFile); {
+	case err != nil:
+		report.Findings = append(report.Findings, watchdog.Finding{
+			Key: "watchdog/heartbeat", Severity: watchdog.Warning,
+			SummaryEN: fmt.Sprintf("the heartbeat URL is unusable, so no run pings it: %v", err),
+			Hint:      "rerun the installer with FELIS_WATCHDOG_HEARTBEAT_URL set (docs/troubleshooting.md §14)",
+		})
+	case url == "":
+		notes = append(notes, "no heartbeat URL is set: a host that goes down entirely, or a watchdog that stops running, alerts no one. "+
+			"Rerun the installer with FELIS_WATCHDOG_HEARTBEAT_URL (docs/troubleshooting.md §14)")
+	}
+	if until := watchdog.QuietUntil(w.quietPath); env.now.Before(until) {
+		notes = append(notes, fmt.Sprintf("the watchdog mails nothing until %s (%s): the installer holds it while it restarts things on purpose, "+
+			"and a marker an installer killed mid-run left behind holds it until then",
+			until.UTC().Format("2006-01-02 15:04 UTC"), w.quietPath))
+	}
+	return printDoctorReport(stdout, report.Findings, skip, notes)
+}
+
+// printDoctorReport prints each area's findings under its heading, an area
+// with none as fine or, when skip says why, as not checked, then the notes
+// and the count. It returns the exit status: 1 when anything was found.
+func printDoctorReport(stdout io.Writer, findings []watchdog.Finding, skip map[string]string, notes []string) int {
+	byArea := map[string][]watchdog.Finding{}
+	for _, f := range findings {
+		a := findingArea(f.Key)
+		byArea[a] = append(byArea[a], f)
+	}
+	var critical, warning int
+	for _, a := range doctorAreas {
+		fs := byArea[a.key]
+		switch {
+		case len(fs) > 0:
+		case skip[a.key] != "":
+			fmt.Fprintf(stdout, "-  %s: not checked, %s\n", a.title, skip[a.key])
+			continue
+		default:
+			fmt.Fprintf(stdout, "✓  %s\n", a.title)
+			continue
+		}
+		mark := "!"
+		if slices.ContainsFunc(fs, func(f watchdog.Finding) bool { return f.Severity == watchdog.Critical }) {
+			mark = "✗"
+		}
+		fmt.Fprintf(stdout, "%s  %s\n", mark, a.title)
+		for _, f := range fs {
+			if f.Severity == watchdog.Critical {
+				critical++
+			} else {
+				warning++
+			}
+			fmt.Fprintf(stdout, "     %-8s %s: %s\n", f.Severity, f.Key, f.SummaryEN)
+			if f.Hint != "" {
+				fmt.Fprintf(stdout, "              → %s\n", f.Hint)
+			}
+		}
+	}
+	for _, n := range notes {
+		fmt.Fprintf(stdout, "\nnote: %s\n", n)
+	}
+	fmt.Fprintln(stdout)
+	if critical+warning == 0 {
+		fmt.Fprintln(stdout, "no problems found")
+		return 0
+	}
+	fmt.Fprintf(stdout, "%d problem(s): %d critical, %d warning(s)\n", critical+warning, critical, warning)
+	return 1
+}
+
+// findingArea is the report heading a finding key goes under.
+func findingArea(key string) string {
+	head, _, _ := strings.Cut(key, "/")
+	switch head {
+	case "kube-api", "deployment", "system-server", "server-failed", "job-failed", "reaper-stale", "node":
+		return "cluster"
+	case "db-backup", "db-backup-servers":
+		return "db-backup"
+	case "unit", "timer":
+		return "systemd"
+	case "watchdog":
+		return "alerts"
+	}
+	return head
+}
+
+// alertReachFindings is why the watchdog's alerts would reach no one, which
+// its own runs only log: no relay, or no owner with a verified address.
+func alertReachFindings(cfg *config.Config, owners []string, ownersErr error) []watchdog.Finding {
+	var out []watchdog.Finding
+	if cfg.SMTP.Host == "" {
+		out = append(out, watchdog.Finding{
+			Key: "alerts/relay", Severity: watchdog.Warning,
+			SummaryEN: "no [smtp] relay is configured: the watchdog logs its alerts to the journal and mails no one",
+			Hint:      "sudo felis setup, step SMTP",
+		})
+	}
+	if ownersErr == nil && len(owners) == 0 {
+		out = append(out, watchdog.Finding{
+			Key: "alerts/recipients", Severity: watchdog.Warning,
+			SummaryEN: "no owner account has a verified email: the watchdog's alerts reach no one",
+			Hint:      "an owner verifies an address in the panel's account settings",
+		})
+	}
+	return out
+}
+
+// unitFindings reports the installer's systemd units that failed, the
+// long-running ones that are not running, and timers that no longer fire.
+func unitFindings(ctx context.Context, env doctorEnv) []watchdog.Finding {
+	var out []watchdog.Finding
+	seen := map[string]bool{}
+	failed, err := env.run(ctx, "systemctl", "list-units", "--all", "--plain", "--no-legend", "--no-pager", "--state=failed", "felis-*", "k3s.service")
+	if err != nil && len(failed) == 0 {
+		return []watchdog.Finding{{
+			Key: "unit/systemctl", Severity: watchdog.Warning,
+			SummaryEN: fmt.Sprintf("systemctl list-units failed, so no unit was checked: %v", err),
+		}}
+	}
+	for _, line := range strings.Split(string(failed), "\n") {
+		fields := strings.Fields(line)
+		if len(fields) == 0 {
+			continue
+		}
+		name := fields[0]
+		seen[name] = true
+		out = append(out, watchdog.Finding{
+			Key: "unit/" + name, Severity: watchdog.Critical,
+			SummaryEN: name + " failed",
+			Hint:      fmt.Sprintf("journalctl -u %s -n 100 --no-pager; once fixed, sudo systemctl reset-failed %s (a timer's job clears on its next good run)", name, name),
+		})
+	}
+	for _, name := range hostServices {
+		if seen[name] {
+			continue
+		}
+		if _, err := os.Stat(filepath.Join(env.unitDir, name)); err != nil {
+			continue
+		}
+		if state := unitActiveState(ctx, env, name); state != "active" {
+			out = append(out, watchdog.Finding{
+				Key: "unit/" + name, Severity: watchdog.Critical,
+				SummaryEN: fmt.Sprintf("%s is %s", name, state),
+				Hint:      fmt.Sprintf("sudo systemctl start %s; journalctl -u %s -n 100 --no-pager", name, name),
+			})
+		}
+	}
+	timers, _ := filepath.Glob(filepath.Join(env.unitDir, "felis-*.timer"))
+	for _, path := range timers {
+		name := filepath.Base(path)
+		if state := unitActiveState(ctx, env, name); state != "active" {
+			out = append(out, watchdog.Finding{
+				Key: "timer/" + name, Severity: watchdog.Warning,
+				SummaryEN: fmt.Sprintf("%s is %s: the job it starts no longer runs", name, state),
+				Hint:      fmt.Sprintf("sudo systemctl enable --now %s", name),
+			})
+		}
+	}
+	return out
+}
+
+// unitActiveState is what `systemctl is-active` says of unit.
+func unitActiveState(ctx context.Context, env doctorEnv, unit string) string {
+	out, err := env.run(ctx, "systemctl", "is-active", unit)
+	if state := strings.TrimSpace(string(out)); state != "" {
+		return state
+	}
+	return fmt.Sprintf("in an unknown state (systemctl is-active: %v)", err)
+}
+
+// watchdogUnitFlags reads the flags felis-watchdog.service runs felis
+// watchdog with. found is false when there is no such unit; w is then the
+// watchdog's defaults.
+func watchdogUnitFlags(path string) (w watchdogFlags, found bool, err error) {
+	fs := flag.NewFlagSet("watchdog", flag.ContinueOnError)
+	fs.SetOutput(io.Discard)
+	w.register(fs)
+	raw, err := os.ReadFile(path)
+	if errors.Is(err, os.ErrNotExist) {
+		return w, false, nil
+	}
+	if err != nil {
+		return w, false, err
+	}
+	for _, line := range strings.Split(string(raw), "\n") {
+		cmd, ok := strings.CutPrefix(strings.TrimSpace(line), "ExecStart=")
+		if !ok {
+			continue
+		}
+		fields := execArgs(cmd)
+		i := slices.Index(fields, "watchdog")
+		if i < 0 {
+			continue
+		}
+		if err := fs.Parse(fields[i+1:]); err != nil {
+			return w, true, fmt.Errorf("%s: %w", path, err)
+		}
+		return w, true, nil
+	}
+	return w, true, fmt.Errorf("%s runs no `felis watchdog`", path)
+}
+
+// execArgs splits an ExecStart= command line into its words. A word may be
+// quoted with " or ', as systemd allows, which is how an empty value is
+// written.
+func execArgs(s string) []string {
+	var out []string
+	var cur strings.Builder
+	inWord := false
+	var quote rune
+	for _, r := range s {
+		switch {
+		case quote != 0:
+			if r == quote {
+				quote = 0
+			} else {
+				cur.WriteRune(r)
+			}
+		case r == '"' || r == '\'':
+			quote, inWord = r, true
+		case r == ' ' || r == '\t':
+			if inWord {
+				out = append(out, cur.String())
+				cur.Reset()
+				inWord = false
+			}
+		default:
+			cur.WriteRune(r)
+			inWord = true
+		}
+	}
+	if inWord {
+		out = append(out, cur.String())
+	}
+	return out
+}

cmd/felis/doctor_test.go

@@ -0,0 +1,423 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	"felis.lolicon.best/internal/config"
+	"felis.lolicon.best/internal/watchdog"
+)
+
+// bootstrapWatchdogExecStart is the ExecStart= line deploy/bootstrap.sh writes
+// into felis-watchdog.service, with its variables filled in as an install
+// fills them.
+func bootstrapWatchdogExecStart(t *testing.T, vars map[string]string) string {
+	t.Helper()
+	raw, err := os.ReadFile("../../deploy/bootstrap.sh")
+	if err != nil {
+		t.Fatal(err)
+	}
+	var line string
+	for _, l := range strings.Split(string(raw), "\n") {
+		if strings.HasPrefix(l, "ExecStart=${HOST_BIN} watchdog -config") {
+			line = l
+		}
+	}
+	if line == "" {
+		t.Fatal("deploy/bootstrap.sh writes no `ExecStart=${HOST_BIN} watchdog -config` line")
+	}
+	line = strings.ReplaceAll(line, "${NODE_IP:+ -node-ip ${NODE_IP}}", " -node-ip "+vars["NODE_IP"])
+	line = regexp.MustCompile(`\$\{([A-Za-z_]+)\}`).ReplaceAllStringFunc(line, func(m string) string {
+		v, ok := vars[m[2:len(m)-1]]
+		if !ok {
+			t.Fatalf("bootstrap's watchdog ExecStart= uses %s, which this test does not fill in", m)
+		}
+		return v
+	})
+	return line
+}
+
+// felis doctor reads the watchdog's settings from the unit the installer
+// writes, so it checks the paths the timer's runs check.
+func TestWatchdogUnitFlagsReadsTheInstallersUnit(t *testing.T) {
+	dir := t.TempDir()
+	exec := bootstrapWatchdogExecStart(t, map[string]string{
+		"HOST_BIN": "/usr/local/bin/felis", "STATE_DIR": "/srv/felis-etc", "WATCHDOG_STATE": "/srv/watchdog/state.json",
+		"WATCHDOG_QUIET_FILE": "/srv/quiet-until", "FELIS_DB_BACKUP_DIR": "/srv/db-backups", "FELIS_GAME_PORT": "25577",
+		"disks": "/,/srv/data", "NODE_IP": "10.0.0.5",
+	})
+	unit := filepath.Join(dir, "felis-watchdog.service")
+	writeTestFile(t, unit, "[Unit]\nDescription=Felis watchdog\n\n[Service]\nType=oneshot\n"+exec+"\nTimeoutStartSec=3min\n", 0o644)
+
+	w, found, err := watchdogUnitFlags(unit)
+	if err != nil || !found {
+		t.Fatalf("found %v, err %v", found, err)
+	}
+	got := []string{w.cfgPath, w.statePath, w.quietPath, w.backupDir, w.proxyAddr, w.diskPaths, w.nodeIP, w.heartbeatFile, w.controlNS}
+	want := []string{"/srv/felis-etc/felis.host.toml", "/srv/watchdog/state.json", "/srv/quiet-until", "/srv/db-backups", "127.0.0.1:25577", "/,/srv/data", "10.0.0.5", defaultHeartbeatFile, "felis"}
+	if strings.Join(got, "|") != strings.Join(want, "|") {
+		t.Errorf("read\n %q\nwant\n %q", got, want)
+	}
+
+	w, found, err = watchdogUnitFlags(filepath.Join(dir, "missing.service"))
+	if err != nil || found || w.cfgPath != "/etc/felis/felis.toml" || w.backupDir != "/var/lib/felis/db-backups" {
+		t.Errorf("no unit: found %v, err %v, config %q, backups %q; want the watchdog's defaults", found, err, w.cfgPath, w.backupDir)
+	}
+
+	writeTestFile(t, unit, "[Service]\nExecStart=/usr/local/bin/felis version\n", 0o644)
+	if _, found, err := watchdogUnitFlags(unit); !found || err == nil || !strings.Contains(err.Error(), "runs no `felis watchdog`") {
+		t.Errorf("a unit that runs something else: found %v, err %v", found, err)
+	}
+	writeTestFile(t, unit, "[Service]\nExecStart=/usr/local/bin/felis watchdog -no-such-flag x\n", 0o644)
+	if _, _, err := watchdogUnitFlags(unit); err == nil {
+		t.Error("a flag this binary does not know was accepted")
+	}
+	writeTestFile(t, unit, "[Service]\nExecStart=/usr/local/bin/felis watchdog -backup-dir \"\"\t-proxy-addr '127.0.0.1:1' -disk-paths \"/a b\"\n", 0o644)
+	if w, _, err := watchdogUnitFlags(unit); err != nil || w.backupDir != "" || w.proxyAddr != "127.0.0.1:1" || w.diskPaths != "/a b" {
+		t.Errorf("quoted words: backups %q, proxy %q, disks %q, err %v", w.backupDir, w.proxyAddr, w.diskPaths, err)
+	}
+}
+
+func TestFindingArea(t *testing.T) {
+	for key, want := range map[string]string{
+		"kube-api":                           "cluster",
+		"deployment/felis-api":               "cluster",
+		"system-server/lobby":                "cluster",
+		"server-failed/survival":             "cluster",
+		"job-failed/reaper-123":              "cluster",
+		"reaper-stale":                       "cluster",
+		"node/felis-1/NotReady":              "cluster",
+		"postgres":                           "postgres",
+		"proxy":                              "proxy",
+		"db-backup":                          "db-backup",
+		"db-backup-servers":                  "db-backup",
+		"offsite":                            "offsite",
+		"scan-db":                            "scan-db",
+		"disk//var/lib/felis":                "disk",
+		"memory":                             "memory",
+		"k3s-certs":                          "k3s-certs",
+		"host-address":                       "host-address",
+		"clock":                              "clock",
+		"config":                             "config",
+		"unit/felis-offsite.service":         "systemd",
+		"timer/felis-db-backup.timer":        "systemd",
+		"watchdog/unit":                      "alerts",
+		"watchdog/heartbeat":                 "alerts",
+		"alerts/relay":                       "alerts",
+		"alerts/recipients":                  "alerts",
+		"something-a-later-release-reported": "something-a-later-release-reported",
+	} {
+		if got := findingArea(key); got != want {
+			t.Errorf("findingArea(%q) = %q, want %q", key, got, want)
+		}
+	}
+}
+
+func TestAlertReachFindings(t *testing.T) {
+	relay := &config.Config{SMTP: config.SMTPConfig{Host: "smtp.example.com"}}
+	keys := func(fs []watchdog.Finding) string {
+		var k []string
+		for _, f := range fs {
+			k = append(k, f.Key)
+		}
+		return strings.Join(k, ",")
+	}
+	for _, tc := range []struct {
+		what      string
+		cfg       *config.Config
+		owners    []string
+		ownersErr error
+		want      string
+	}{
+		{"a relay and an owner", relay, []string{"[email protected]"}, nil, ""},
+		{"no relay", &config.Config{}, []string{"[email protected]"}, nil, "alerts/relay"},
+		{"no owner with an address", relay, nil, nil, "alerts/recipients"},
+		{"PostgreSQL down: its own finding says so", relay, nil, errors.New("refused"), ""},
+		{"neither", &config.Config{}, nil, nil, "alerts/relay,alerts/recipients"},
+	} {
+		if got := keys(alertReachFindings(tc.cfg, tc.owners, tc.ownersErr)); got != tc.want {
+			t.Errorf("%s: %q, want %q", tc.what, got, tc.want)
+		}
+	}
+}
+
+// fakeSystemctl answers list-units with failed and is-active from states;
+// a unit missing from states is "inactive", as systemctl says, and one whose
+// state is "" gets no answer.
+func fakeSystemctl(failed string, states map[string]string) func(ctx context.Context, name string, args ...string) ([]byte, error) {
+	return func(_ context.Context, name string, args ...string) ([]byte, error) {
+		if name != "systemctl" || len(args) == 0 {
+			return nil, errors.New("unexpected command " + name)
+		}
+		switch args[0] {
+		case "list-units":
+			return []byte(failed), nil
+		case "is-active":
+			if s, ok := states[args[1]]; ok && s == "" {
+				return nil, errors.New("signal: killed")
+			} else if ok {
+				return []byte(s + "\n"), nil
+			}
+			return []byte("inactive\n"), errors.New("exit status 3")
+		}
+		return nil, errors.New("unexpected systemctl " + args[0])
+	}
+}
+
+func TestUnitFindings(t *testing.T) {
+	dir := t.TempDir()
+	for _, f := range []string{"k3s.service", "felis-velocity.service", "felis-db-backup.timer", "felis-offsite.timer", "felis-offsite.service"} {
+		writeTestFile(t, filepath.Join(dir, f), "[Unit]\n", 0o644)
+	}
+	env := doctorEnv{unitDir: dir, run: fakeSystemctl(
+		"felis-offsite.service loaded failed failed Felis off-site copy\nfelis-velocity.service loaded failed failed Velocity\n",
+		map[string]string{"k3s.service": "active", "felis-db-backup.timer": "active", "felis-offsite.timer": "inactive"},
+	)}
+	var got []string
+	for _, f := range unitFindings(context.Background(), env) {
+		got = append(got, string(f.Severity)+" "+f.Key+": "+f.SummaryEN)
+	}
+	want := []string{
+		"critical unit/felis-offsite.service: felis-offsite.service failed",
+		"critical unit/felis-velocity.service: felis-velocity.service failed",
+		"warning timer/felis-offsite.timer: felis-offsite.timer is inactive: the job it starts no longer runs",
+	}
+	if strings.Join(got, "\n") != strings.Join(want, "\n") {
+		t.Errorf("findings:\n%s\nwant (felis-nano.service has no unit file here, and a failed unit is reported once):\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
+	}
+
+	env.run = fakeSystemctl("", map[string]string{"k3s.service": "activating", "felis-velocity.service": "active", "felis-db-backup.timer": "active", "felis-offsite.timer": "active"})
+	got = nil
+	for _, f := range unitFindings(context.Background(), env) {
+		got = append(got, f.Key+": "+f.SummaryEN)
+	}
+	if strings.Join(got, "\n") != "unit/k3s.service: k3s.service is activating" {
+		t.Errorf("findings %q, want only k3s.service, which is not active", got)
+	}
+
+	env.run = fakeSystemctl("", map[string]string{"k3s.service": "active", "felis-velocity.service": "active", "felis-db-backup.timer": "", "felis-offsite.timer": "active"})
+	got = nil
+	for _, f := range unitFindings(context.Background(), env) {
+		got = append(got, f.Key+": "+f.SummaryEN)
+	}
+	if want := "timer/felis-db-backup.timer: felis-db-backup.timer is in an unknown state (systemctl is-active: signal: killed): the job it starts no longer runs"; strings.Join(got, "\n") != want {
+		t.Errorf("findings %q, want %q", got, want)
+	}
+
+	env.run = func(context.Context, string, ...string) ([]byte, error) { return nil, errors.New("no systemctl") }
+	if fs := unitFindings(context.Background(), env); len(fs) != 1 || fs[0].Key != "unit/systemctl" || fs[0].Severity != watchdog.Warning {
+		t.Errorf("without systemctl: %+v, want the one unit/systemctl warning", fs)
+	}
+}
+
+// quoteArgs writes args as an ExecStart= line does, each word in quotes so an
+// empty one survives.
+func quoteArgs(args []string) string {
+	q := make([]string, len(args))
+	for i, a := range args {
+		q[i] = `"` + a + `"`
+	}
+	return strings.Join(q, " ")
+}
+
+// doctorHost is a host with the installer's watchdog unit, whose API server
+// and PostgreSQL are down.
+func doctorHost(t *testing.T, cfg string, extraFlags string) (env doctorEnv, h *watchdogHost) {
+	t.Helper()
+	h = newWatchdogHost(t, cfg, nil)
+	unitDir := t.TempDir()
+	writeTestFile(t, filepath.Join(unitDir, "felis-watchdog.service"),
+		"[Service]\nType=oneshot\nExecStart=/usr/local/bin/felis watchdog "+quoteArgs(h.args)+
+			// Off this machine's disk, whose free space is not the test's.
+			` -disk-paths "/nonexistent-felis-doctor-test"`+extraFlags+"\n", 0o644)
+	writeTestFile(t, filepath.Join(unitDir, "felis-velocity.service"), "[Unit]\n", 0o644)
+	writeTestFile(t, filepath.Join(unitDir, "felis-offsite.timer"), "[Unit]\n", 0o644)
+	return doctorEnv{
+		unitDir: unitDir, now: time.Now(), host: "felis-test",
+		run: fakeSystemctl("felis-db-backup.service loaded failed failed Felis database backup\n",
+			map[string]string{"felis-velocity.service": "active", "felis-offsite.timer": "active"}),
+	}, h
+}
+
+func TestDoctorReportsByArea(t *testing.T) {
+	env, _ := doctorHost(t, testWatchdogConfig, "")
+	var out bytes.Buffer
+	code := runDoctor(context.Background(), env, &out)
+	got := out.String()
+	if code != 1 {
+		t.Errorf("exit %d, want 1 with problems found", code)
+	}
+	for _, want := range []string{
+		"felis doctor on felis-test at ",
+		"checks run as " + filepath.Join(env.unitDir, "felis-watchdog.service") + " runs them (config ",
+		"✓  configuration\n",
+		"✗  Kubernetes cluster\n     critical kube-api: ",
+		"✗  PostgreSQL\n     critical postgres: ",
+		"-  game proxy: not checked, no -proxy-addr\n",
+		"-  database backups: not checked, no -backup-dir\n",
+		"-  off-site copy: not checked, not configured\n",
+		"-  build scan database: not checked, builds do not scan against the registry's copy\n",
+		"-  k3s certificates: not checked, no -k3s-cert-dirs\n",
+		"-  node address: not checked, no -node-ip\n",
+		"✗  systemd units and timers\n     critical unit/felis-db-backup.service: felis-db-backup.service failed\n" +
+			"              → journalctl -u felis-db-backup.service -n 100 --no-pager; once fixed, sudo systemctl reset-failed felis-db-backup.service",
+		"!  alerting\n     warning  alerts/relay: no [smtp] relay is configured",
+		"\n4 problem(s): 3 critical, 1 warning(s)\n",
+	} {
+		if !strings.Contains(got, want) {
+			t.Errorf("report lacks %q:\n%s", want, got)
+		}
+	}
+	if strings.Contains(got, "alerts/recipients") {
+		t.Errorf("reported no recipients although PostgreSQL, which names them, is down:\n%s", got)
+	}
+	if strings.Contains(got, "note: no heartbeat URL") {
+		t.Errorf("the host has a heartbeat URL:\n%s", got)
+	}
+}
+
+// A doctor run is only a look: whatever is due to be mailed stays due, no
+// heartbeat is pinged, and the watchdog's state is left as it was.
+func TestDoctorMailsPingsAndSavesNothing(t *testing.T) {
+	cfg := testWatchdogConfig + "[smtp]\nhost = \"smtp.example.com\"\nport = 587\nfrom = \"[email protected]\"\n"
+	env, h := doctorHost(t, cfg, "")
+	if err := watchdog.SaveState(h.statePath, duePostgres("cached-pw")); err != nil {
+		t.Fatal(err)
+	}
+	before, err := os.ReadFile(h.statePath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	watchdogSender = h.rec.sender
+	defer func() { watchdogSender = smtpSender }()
+	var out bytes.Buffer
+	runDoctor(context.Background(), env, &out)
+	if !strings.Contains(out.String(), "critical postgres: ") {
+		t.Fatalf("the due PostgreSQL alert was not seen:\n%s", out.String())
+	}
+	if len(h.rec.sent) != 0 || len(h.rec.relays) != 0 {
+		t.Errorf("mailed %v", h.rec.sent)
+	}
+	if n := len(h.pings.pings); n != 0 {
+		t.Errorf("pinged the heartbeat %d times", n)
+	}
+	after, err := os.ReadFile(h.statePath)
+	if err != nil || !bytes.Equal(before, after) {
+		t.Errorf("the watchdog state changed (err %v)", err)
+	}
+	if _, err := os.Stat(h.fallbackPath); !errors.Is(err, os.ErrNotExist) {
+		t.Errorf("a fallback state was written: %v", err)
+	}
+}
+
+func TestDoctorNotes(t *testing.T) {
+	env, h := doctorHost(t, testWatchdogConfig, "")
+	if err := os.Remove(filepath.Join(h.dir, "watchdog-heartbeat-url")); err != nil {
+		t.Fatal(err)
+	}
+	until := env.now.Add(10 * time.Minute).Unix()
+	writeTestFile(t, filepath.Join(h.dir, "quiet"), strconv.FormatInt(until, 10)+"\n", 0o644)
+	var out bytes.Buffer
+	runDoctor(context.Background(), env, &out)
+	for _, want := range []string{
+		"\nnote: no heartbeat URL is set: ",
+		"\nnote: the watchdog mails nothing until " + time.Unix(until, 0).UTC().Format("2006-01-02 15:04 UTC") + " (" + filepath.Join(h.dir, "quiet") + ")",
+	} {
+		if !strings.Contains(out.String(), want) {
+			t.Errorf("report lacks %q:\n%s", want, out.String())
+		}
+	}
+
+	writeTestFile(t, filepath.Join(h.dir, "watchdog-heartbeat-url"), "not a url\n", 0o600)
+	out.Reset()
+	runDoctor(context.Background(), env, &out)
+	if !strings.Contains(out.String(), "warning  watchdog/heartbeat: the heartbeat URL is unusable") {
+		t.Errorf("an unusable heartbeat URL is not reported:\n%s", out.String())
+	}
+}
+
+// Without the watchdog's unit the doctor still checks, with the watchdog's
+// defaults, and says the host has no watchdog; a configuration that does not
+// load leaves the checks that need it unchecked and the host's own checks on.
+func TestDoctorWithoutUnitOrConfig(t *testing.T) {
+	env, _ := doctorHost(t, testWatchdogConfig, "")
+	if err := os.Remove(filepath.Join(env.unitDir, "felis-watchdog.service")); err != nil {
+		t.Fatal(err)
+	}
+	writeTestFile(t, filepath.Join(env.unitDir, "felis-watchdog.service"), "[Service]\nExecStart=/usr/local/bin/felis watchdog -config "+filepath.Join(t.TempDir(), "absent.toml")+"\n", 0o644)
+	env.run = fakeSystemctl("", map[string]string{"felis-velocity.service": "active", "felis-offsite.timer": "active"})
+	var out bytes.Buffer
+	if code := runDoctor(context.Background(), env, &out); code != 1 {
+		t.Errorf("exit %d, want 1", code)
+	}
+	for _, want := range []string{
+		"✗  configuration\n     critical config: ",
+		"-  Kubernetes cluster: not checked, the configuration did not load\n",
+		"-  PostgreSQL: not checked, the configuration did not load\n",
+		"-  disk space: not checked, the configuration did not load\n",
+		"✓  systemd units and timers\n",
+		"-  alerting: not checked, the configuration did not load\n",
+	} {
+		if !strings.Contains(out.String(), want) {
+			t.Errorf("report lacks %q:\n%s", want, out.String())
+		}
+	}
+
+	if err := os.Remove(filepath.Join(env.unitDir, "felis-watchdog.service")); err != nil {
+		t.Fatal(err)
+	}
+	out.Reset()
+	runDoctor(context.Background(), env, &out)
+	if !strings.Contains(out.String(), "critical watchdog/unit: "+filepath.Join(env.unitDir, "felis-watchdog.service")+" is not installed") ||
+		!strings.Contains(out.String(), "checks run with the watchdog's defaults (config /etc/felis/felis.toml)") {
+		t.Errorf("a host without the watchdog's unit:\n%s", out.String())
+	}
+
+	unit := filepath.Join(env.unitDir, "felis-watchdog.service")
+	writeTestFile(t, unit, "[Service]\nExecStart=/usr/local/bin/felis watchdog -no-such-flag x\n", 0o644)
+	out.Reset()
+	runDoctor(context.Background(), env, &out)
+	if !strings.Contains(out.String(), "critical watchdog/unit: cannot read the watchdog's settings: "+unit+": flag provided but not defined: -no-such-flag\n") ||
+		!strings.Contains(out.String(), "checks run with the watchdog's defaults (config /etc/felis/felis.toml)") {
+		t.Errorf("a watchdog unit this binary cannot read:\n%s", out.String())
+	}
+}
+
+func TestPrintDoctorReport(t *testing.T) {
+	var out bytes.Buffer
+	if code := printDoctorReport(&out, nil, map[string]string{"proxy": "no -proxy-addr"}, nil); code != 0 {
+		t.Errorf("exit %d with nothing found, want 0", code)
+	}
+	want := "✓  configuration\n✓  Kubernetes cluster\n✓  PostgreSQL\n-  game proxy: not checked, no -proxy-addr\n✓  database backups\n✓  off-site copy\n" +
+		"✓  build scan database\n✓  disk space\n✓  memory\n✓  k3s certificates\n✓  node address\n✓  clock\n✓  systemd units and timers\n✓  alerting\n" +
+		"\nno problems found\n"
+	if out.String() != want {
+		t.Errorf("report:\n%s\nwant:\n%s", out.String(), want)
+	}
+
+	out.Reset()
+	code := printDoctorReport(&out, []watchdog.Finding{
+		{Key: "unit/systemctl", Severity: watchdog.Warning, SummaryEN: "systemctl list-units failed"},
+		{Key: "postgres", Severity: watchdog.Critical, SummaryEN: "PostgreSQL is down", Hint: "kubectl -n felis get pods"},
+	}, map[string]string{"postgres": "a skip loses to what was found"}, []string{"a note"})
+	if code != 1 {
+		t.Errorf("exit %d with problems, want 1", code)
+	}
+	for _, want := range []string{
+		"✗  PostgreSQL\n     critical postgres: PostgreSQL is down\n              → kubectl -n felis get pods\n✓  game proxy\n",
+		"!  systemd units and timers\n     warning  unit/systemctl: systemctl list-units failed\n✓  alerting\n",
+		"✓  alerting\n\nnote: a note\n\n2 problem(s): 1 critical, 1 warning(s)\n",
+	} {
+		if !strings.Contains(out.String(), want) {
+			t.Errorf("report lacks %q:\n%s", want, out.String())
+		}
+	}
+}

cmd/felis/run.go

@@ -34,6 +34,9 @@ Commands:
   converge          Fill in fields a newer desired spec added to already-installed system servers
   rotate-token      Replace a generated credential and restart what reads it (velocity|limbo|build|ops|registry|forwarding|db; prints the plan, -yes applies; requires root/sudo)
   domain            Move the install to a new root domain on every surface that carries it, or check each one (set|check; requires root/sudo)
+  status            Print the platform at a glance: node, control plane, proxy, servers, backups, host, open alerts (requires root/sudo)
+  doctor            Run every health check once, grouped by area, with where to look next; mails nothing (requires root/sudo)
+  support-bundle    Collect status, doctor, logs and cluster state into one redacted tar.gz to share when asking for help (requires root/sudo)
   watchdog          Check the platform once and mail the owners what has gone wrong (run by felis-watchdog.timer)
   version           Print the build stamp of this binary
   update            Report which platform components have updates available
@@ -84,6 +87,9 @@ var commands = map[string]func(args []string, stdout, stderr io.Writer) int{
 	"version":            cmdVersion,
 	"update":             cmdUpdate,
 	"watchdog":           cmdWatchdog,
+	"status":             cmdStatus,
+	"doctor":             cmdDoctor,
+	"support-bundle":     cmdSupportBundle,
 }
 
 // run dispatches a subcommand. It is separate from main so the router is

cmd/felis/status.go

@@ -0,0 +1,438 @@
+package main
+
+import (
+	"bufio"
+	"context"
+	"errors"
+	"flag"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+	"syscall"
+	"text/tabwriter"
+	"time"
+
+	"felis.lolicon.best/internal/apis/felis/v1alpha1"
+	"felis.lolicon.best/internal/config"
+	"felis.lolicon.best/internal/dbbackup"
+	"felis.lolicon.best/internal/offsite"
+	"felis.lolicon.best/internal/platform"
+	"felis.lolicon.best/internal/store"
+	"felis.lolicon.best/internal/watchdog"
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+)
+
+// cmdStatus prints the platform at a glance: the release and the node, the
+// control plane's workloads, the game proxy, every server with its players
+// and newest world backup, the database backups and the off-site copy, the
+// host's disks and memory, and what the watchdog has open. It changes
+// nothing, and a part that is down (the cluster, PostgreSQL) reads as such
+// while the rest still prints. felis doctor says what is wrong and where to
+// look.
+func cmdStatus(args []string, stdout, stderr io.Writer) int {
+	fs := flag.NewFlagSet("status", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	unitDir := fs.String("systemd-dir", systemdUnitDir, "where the installer's systemd units are")
+	if err := fs.Parse(args); err != nil {
+		if errors.Is(err, flag.ErrHelp) {
+			return 0
+		}
+		return 2
+	}
+	if os.Geteuid() != 0 {
+		fmt.Fprintln(stderr, "felis status: run as root (sudo felis status): it reads root-only state under /etc/felis and /var/lib/felis")
+		return 1
+	}
+	env, err := hostStatusEnv(*unitDir)
+	if err != nil {
+		fmt.Fprintf(stderr, "felis status: %v\n", err)
+		return 1
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
+	defer cancel()
+	printStatus(ctx, env, stdout)
+	return 0
+}
+
+// statusEnv is what one status report reads the host through.
+type statusEnv struct {
+	cfg *config.Config
+	// w is how felis-watchdog.service runs the watchdog: where the backups,
+	// the off-site record and the proxy are.
+	w       watchdogFlags
+	cl      client.Client // nil while the API server is unreachable
+	clErr   error
+	backups func(ctx context.Context) (map[string]time.Time, error)
+	run     func(ctx context.Context, name string, args ...string) ([]byte, error)
+	unitDir string
+	meminfo string
+	host    string
+	now     time.Time
+}
+
+// hostStatusEnv reads this host: the watchdog's settings, the configuration
+// they name, and the cluster.
+func hostStatusEnv(unitDir string) (statusEnv, error) {
+	w, _, err := watchdogUnitFlags(filepath.Join(unitDir, "felis-watchdog.service"))
+	if err != nil {
+		return statusEnv{}, err
+	}
+	cfg, err := config.Load(w.cfgPath)
+	if err != nil {
+		return statusEnv{}, err
+	}
+	cl, clErr := buildSystemServerClient()
+	if clErr != nil {
+		cl = nil
+	}
+	host, _ := os.Hostname()
+	return statusEnv{
+		cfg: cfg, w: w, cl: cl, clErr: clErr,
+		backups: func(ctx context.Context) (map[string]time.Time, error) {
+			return newestWorldBackups(ctx, cfg.Database.URL)
+		},
+		run: hostCommand, unitDir: unitDir, meminfo: "/proc/meminfo", host: host, now: time.Now(),
+	}, nil
+}
+
+func printStatus(ctx context.Context, env statusEnv, out io.Writer) {
+	fmt.Fprintf(out, "felis %s on %s at %s\n", resolvedVersion(), env.host, env.now.UTC().Format("2006-01-02 15:04 UTC"))
+	if env.cl == nil {
+		fmt.Fprintf(out, "cluster:  unreachable (%v)\n", env.clErr)
+	} else {
+		statusCluster(ctx, env, out)
+	}
+	statusProxy(ctx, env, out)
+	statusServers(ctx, env, out)
+	statusBackups(env, out)
+	statusHost(env, out)
+	statusWatchdog(ctx, env, out)
+}
+
+func statusCluster(ctx context.Context, env statusEnv, out io.Writer) {
+	var nodes corev1.NodeList
+	if err := env.cl.List(ctx, &nodes); err != nil {
+		fmt.Fprintf(out, "cluster:  unreachable (%v)\n", err)
+		return
+	}
+	for _, n := range nodes.Items {
+		ready := "NotReady"
+		for _, c := range n.Status.Conditions {
+			if c.Type == corev1.NodeReady && c.Status == corev1.ConditionTrue {
+				ready = "Ready"
+			}
+		}
+		info := n.Status.NodeInfo
+		fmt.Fprintf(out, "node:     %s %s, k3s %s, %s, kernel %s\n", n.Name, ready, info.KubeletVersion, info.OSImage, info.KernelVersion)
+	}
+
+	ns := env.w.controlNS
+	var deps appsv1.DeploymentList
+	var pods corev1.PodList
+	err := env.cl.List(ctx, &deps, client.InNamespace(ns))
+	if err == nil {
+		err = env.cl.List(ctx, &pods, client.InNamespace(ns))
+	}
+	fmt.Fprintf(out, "\ncontrol plane (namespace %s):\n", ns)
+	if err != nil {
+		fmt.Fprintf(out, "  cannot list it: %v\n", err)
+		return
+	}
+	sort.Slice(deps.Items, func(i, j int) bool { return deps.Items[i].Name < deps.Items[j].Name })
+	tw := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
+	for _, d := range deps.Items {
+		want := int32(1)
+		if d.Spec.Replicas != nil {
+			want = *d.Spec.Replicas
+		}
+		image := "-"
+		if cs := d.Spec.Template.Spec.Containers; len(cs) > 0 {
+			image = shortImage(cs[0].Image)
+		}
+		fmt.Fprintf(tw, "  %s\t%d/%d ready\t%s\trestarts %d\n", d.Name, d.Status.ReadyReplicas, want, image, podRestarts(d.Spec.Selector, pods.Items))
+	}
+	tw.Flush()
+}
+
+// podRestarts adds up the container restarts of the pods selector picks (the
+// API server refuses a Deployment whose selector is empty).
+func podRestarts(selector *metav1.LabelSelector, pods []corev1.Pod) int32 {
+	sel, err := metav1.LabelSelectorAsSelector(selector)
+	if err != nil {
+		return 0
+	}
+	var n int32
+	for _, p := range pods {
+		if !sel.Matches(labels.Set(p.Labels)) {
+			continue
+		}
+		for _, cs := range p.Status.ContainerStatuses {
+			n += cs.RestartCount
+		}
+	}
+	return n
+}
+
+// shortImage is an image reference without its registry and repository path,
+// and with its digest cut to 12 characters.
+func shortImage(ref string) string {
+	if i := strings.LastIndex(ref, "/"); i >= 0 {
+		ref = ref[i+1:]
+	}
+	if name, digest, ok := strings.Cut(ref, "@sha256:"); ok && len(digest) > 12 {
+		ref = name + "@" + digest[:12]
+	}
+	return ref
+}
+
+func statusProxy(ctx context.Context, env statusEnv, out io.Writer) {
+	var parts []string
+	if _, err := os.Stat(filepath.Join(env.unitDir, "felis-velocity.service")); err == nil {
+		parts = append(parts, "felis-velocity "+unitActiveState(ctx, doctorEnv{run: env.run}, "felis-velocity.service"))
+	}
+	if env.w.proxyAddr != "" {
+		d := net.Dialer{Timeout: 3 * time.Second}
+		if conn, err := d.DialContext(ctx, "tcp", env.w.proxyAddr); err != nil {
+			parts = append(parts, fmt.Sprintf("%s refuses connections (%v)", env.w.proxyAddr, err))
+		} else {
+			conn.Close()
+			parts = append(parts, env.w.proxyAddr+" accepts connections")
+		}
+	}
+	if len(parts) == 0 {
+		parts = append(parts, "not on this host")
+	}
+	fmt.Fprintf(out, "\nproxy:    %s\n", strings.Join(parts, ", "))
+}
+
+func statusServers(ctx context.Context, env statusEnv, out io.Writer) {
+	ns := env.cfg.K8s.Namespace
+	if ns == "" {
+		ns = platform.DefaultMinecraftNamespace
+	}
+	if env.cl == nil {
+		fmt.Fprintf(out, "\nservers (namespace %s): unknown while the cluster is unreachable\n", ns)
+		return
+	}
+	var list v1alpha1.MinecraftServerList
+	if err := env.cl.List(ctx, &list, client.InNamespace(ns)); err != nil {
+		fmt.Fprintf(out, "\nservers (namespace %s): cannot list them: %v\n", ns, err)
+		return
+	}
+	newest, backupErr := env.backups(ctx)
+	running, online := 0, int32(0)
+	for _, ms := range list.Items {
+		if ms.Status.Phase == v1alpha1.PhaseRunning {
+			running++
+			online += ms.Status.Players.Online
+		}
+	}
+	fmt.Fprintf(out, "\nservers (namespace %s): %d, %d running, %d players online\n", ns, len(list.Items), running, online)
+	if len(list.Items) == 0 {
+		return
+	}
+	sort.Slice(list.Items, func(i, j int) bool { return list.Items[i].Name < list.Items[j].Name })
+	tw := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
+	fmt.Fprintln(tw, "  NAME\tROLE\tDESIRED\tPHASE\tPLAYERS\tNEWEST WORLD BACKUP")
+	for _, ms := range list.Items {
+		role := ms.Labels[v1alpha1.LabelSystemRole]
+		if role == "" {
+			role = "-"
+		}
+		phase := string(ms.Status.Phase)
+		if phase == "" {
+			phase = "-"
+		}
+		players := "-"
+		if ms.Status.Phase == v1alpha1.PhaseRunning {
+			players = fmt.Sprintf("%d/%d", ms.Status.Players.Online, ms.Status.Players.Max)
+		}
+		backup := "none"
+		switch at, ok := newest[ms.Name]; {
+		case backupErr != nil:
+			backup = "?"
+		case ok:
+			backup = dbbackup.Age(env.now.Sub(at)) + " ago"
+		}
+		fmt.Fprintf(tw, "  %s\t%s\t%s\t%s\t%s\t%s\n", ms.Name, role, orDash(string(ms.Spec.DesiredState)), phase, players, backup)
+	}
+	tw.Flush()
+	if backupErr != nil {
+		fmt.Fprintf(out, "  world backups unknown: %v\n", backupErr)
+	}
+}
+
+func orDash(s string) string {
+	if s == "" {
+		return "-"
+	}
+	return s
+}
+
+// newestWorldBackups is when each server's newest world backup that a restore
+// can use was taken.
+func newestWorldBackups(ctx context.Context, url string) (map[string]time.Time, error) {
+	ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
+	defer cancel()
+	drv, err := store.Open(ctx, url)
+	if err != nil {
+		return nil, err
+	}
+	defer drv.Close()
+	rows, err := drv.DB().QueryContext(ctx, `SELECT server_name, max(created_at) FROM world_backups
+		WHERE status = 'present' AND corrupt_at IS NULL GROUP BY server_name`)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	out := map[string]time.Time{}
+	for rows.Next() {
+		var name string
+		var at time.Time
+		if err := rows.Scan(&name, &at); err != nil {
+			return nil, err
+		}
+		out[name] = at
+	}
+	return out, rows.Err()
+}
+
+func statusBackups(env statusEnv, out io.Writer) {
+	fmt.Fprintln(out, "\nbackups:")
+	switch bundles, err := dbbackup.List(env.w.backupDir); {
+	case env.w.backupDir == "":
+		fmt.Fprintln(out, "  database: not checked (felis-watchdog.service names no -backup-dir)")
+	case err != nil:
+		fmt.Fprintf(out, "  database: cannot read %s: %v\n", env.w.backupDir, err)
+	case len(bundles) == 0:
+		fmt.Fprintf(out, "  database: none in %s\n", env.w.backupDir)
+	default:
+		fmt.Fprintf(out, "  database: newest %s, %s ago; %d bundles in %s\n",
+			bundles[0].Name, dbbackup.Age(env.now.Sub(bundles[0].Created)), len(bundles), env.w.backupDir)
+	}
+	if !env.cfg.Offsite.Enabled() {
+		fmt.Fprintln(out, "  off-site: not configured, every backup is on this machine only")
+		return
+	}
+	switch st, err := offsite.ReadStatus(env.w.offsiteStatus); {
+	case err != nil:
+		fmt.Fprintf(out, "  off-site: %v\n", err)
+	case st == nil:
+		fmt.Fprintln(out, "  off-site: never synced")
+	case st.LastSuccess.IsZero():
+		fmt.Fprintf(out, "  off-site: never succeeded; last attempt %s ago: %s\n", dbbackup.Age(env.now.Sub(st.LastAttempt)), st.LastError)
+	default:
+		line := fmt.Sprintf("  off-site: last good sync %s ago to %s", dbbackup.Age(env.now.Sub(st.LastSuccess)), st.Bucket)
+		if st.LastAttempt.After(st.LastSuccess) { // a failed run records no success
+			line += fmt.Sprintf("; the last attempt, %s ago, failed: %s", dbbackup.Age(env.now.Sub(st.LastAttempt)), st.LastError)
+		}
+		fmt.Fprintln(out, line)
+	}
+}
+
+func statusHost(env statusEnv, out io.Writer) {
+	fmt.Fprintln(out, "\nhost:")
+	seen := map[uint64]bool{}
+	for _, p := range splitList(env.w.diskPaths) {
+		var st syscall.Stat_t
+		if err := syscall.Stat(p, &st); err != nil {
+			continue
+		}
+		dev := uint64(st.Dev) // int32 on darwin
+		if seen[dev] {
+			continue
+		}
+		seen[dev] = true
+		var fs syscall.Statfs_t
+		if err := syscall.Statfs(p, &fs); err != nil || fs.Blocks == 0 {
+			continue
+		}
+		bsize := uint64(fs.Bsize) // uint32 on darwin
+		total, avail := uint64(fs.Blocks)*bsize, uint64(fs.Bavail)*bsize
+		fmt.Fprintf(out, "  disk %s: %s free of %s (%.0f%% free)\n", p, offsite.HumanBytes(int64(avail)), offsite.HumanBytes(int64(total)), float64(fs.Bavail)/float64(fs.Blocks)*100)
+	}
+	if total, avail, ok := readMeminfo(env.meminfo); ok {
+		fmt.Fprintf(out, "  memory: %s available of %s\n", offsite.HumanBytes(int64(avail)), offsite.HumanBytes(int64(total)))
+	}
+}
+
+// readMeminfo reads MemTotal and MemAvailable, in bytes, from a /proc/meminfo
+// style file.
+func readMeminfo(path string) (total, avail uint64, ok bool) {
+	f, err := os.Open(path)
+	if err != nil {
+		return 0, 0, false
+	}
+	defer f.Close()
+	sc := bufio.NewScanner(f)
+	for sc.Scan() {
+		fields := strings.Fields(sc.Text())
+		if len(fields) < 2 {
+			continue
+		}
+		v, _ := strconv.ParseUint(fields[1], 10, 64) // the kernel writes numbers
+		switch fields[0] {
+		case "MemTotal:":
+			total = v * 1024
+		case "MemAvailable:":
+			avail = v * 1024
+		}
+	}
+	return total, avail, total > 0
+}
+
+func statusWatchdog(ctx context.Context, env statusEnv, out io.Writer) {
+	fmt.Fprintln(out, "\nwatchdog:")
+	if _, err := os.Stat(filepath.Join(env.unitDir, "felis-watchdog.timer")); err != nil {
+		fmt.Fprintln(out, "  not installed: nothing checks this host")
+		return
+	}
+	line := "  timer " + unitActiveState(ctx, doctorEnv{run: env.run}, "felis-watchdog.timer")
+	show, _ := env.run(ctx, "systemctl", "show", "--timestamp=unix", "-p", "Result", "-p", "ExecMainExitTimestamp", "felis-watchdog.service")
+	props := map[string]string{}
+	for _, ln := range strings.Split(string(show), "\n") {
+		if k, v, ok := strings.Cut(strings.TrimSpace(ln), "="); ok {
+			props[k] = v
+		}
+	}
+	// ExecMainExitTimestamp is empty until the service has run.
+	if sec, _ := strconv.ParseInt(strings.TrimPrefix(props["ExecMainExitTimestamp"], "@"), 10, 64); sec > 0 {
+		line += fmt.Sprintf(", last run %s ago (%s)", dbbackup.Age(env.now.Sub(time.Unix(sec, 0))), orDash(props["Result"]))
+	}
+	fmt.Fprintln(out, line)
+
+	state, err := watchdog.LoadState(watchdog.NewestState(env.w.statePath, env.w.fallbackState))
+	if err != nil {
+		fmt.Fprintf(out, "  alerts: unknown (%v)\n", err)
+		return
+	}
+	var open []string
+	for key, a := range state.Alerts {
+		if !a.ClearedAt.IsZero() {
+			continue
+		}
+		if a.Notified.IsZero() {
+			open = append(open, fmt.Sprintf("%s (%s, seen %s ago, not mailed yet)", key, a.Severity, dbbackup.Age(env.now.Sub(a.FirstSeen))))
+		} else {
+			open = append(open, fmt.Sprintf("%s (%s, mailed %s ago)", key, a.Severity, dbbackup.Age(env.now.Sub(a.Notified))))
+		}
+	}
+	if len(open) == 0 {
+		fmt.Fprintln(out, "  alerts: none open")
+		return
+	}
+	sort.Strings(open)
+	fmt.Fprintf(out, "  alerts: %d open (sudo felis doctor says where to look)\n", len(open))
+	for _, o := range open {
+		fmt.Fprintf(out, "    %s\n", o)
+	}
+}

cmd/felis/status_test.go

@@ -0,0 +1,490 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"path/filepath"
+	"slices"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	"felis.lolicon.best/internal/apis/felis/v1alpha1"
+	"felis.lolicon.best/internal/config"
+	"felis.lolicon.best/internal/dbbackup"
+	"felis.lolicon.best/internal/offsite"
+	"felis.lolicon.best/internal/watchdog"
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/meta"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
+)
+
+// statusCluster is a one-node install: felis-api with a restarted pod, three
+// servers (one of them the lobby), and a pod of another app whose restarts
+// are not felis-api's.
+func statusClusterObjects() []client.Object {
+	replicas := int32(1)
+	apiLabels := map[string]string{"app": "felis-api"}
+	return []client.Object{
+		&corev1.Node{
+			ObjectMeta: metav1.ObjectMeta{Name: "felis-1"},
+			Status: corev1.NodeStatus{
+				Conditions: []corev1.NodeCondition{{Type: corev1.NodeReady, Status: corev1.ConditionTrue}},
+				NodeInfo:   corev1.NodeSystemInfo{KubeletVersion: "v1.36.4+k3s1", OSImage: "CentOS Stream 9", KernelVersion: "5.14.0-630.el9.aarch64"},
+			},
+		},
+		&appsv1.Deployment{
+			ObjectMeta: metav1.ObjectMeta{Name: "felis-api", Namespace: "felis"},
+			Spec: appsv1.DeploymentSpec{
+				Replicas: &replicas,
+				Selector: &metav1.LabelSelector{MatchLabels: apiLabels},
+				Template: corev1.PodTemplateSpec{Spec: corev1.PodSpec{Containers: []corev1.Container{{
+					Name: "api", Image: "registry.felis.svc:5000/felis/felis-api:v1.4.0@sha256:0123456789abcdef0123456789abcdef",
+				}}}},
+			},
+			Status: appsv1.DeploymentStatus{ReadyReplicas: 1},
+		},
+		&corev1.Pod{
+			ObjectMeta: metav1.ObjectMeta{Name: "felis-api-7d9", Namespace: "felis", Labels: apiLabels},
+			Status:     corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{{Name: "api", RestartCount: 2}}},
+		},
+		&corev1.Pod{
+			ObjectMeta: metav1.ObjectMeta{Name: "other-1", Namespace: "felis", Labels: map[string]string{"app": "other"}},
+			Status:     corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{{Name: "x", RestartCount: 5}}},
+		},
+		&v1alpha1.MinecraftServer{
+			ObjectMeta: metav1.ObjectMeta{Name: "survival", Namespace: "minecraft"},
+			Spec:       v1alpha1.MinecraftServerSpec{DesiredState: v1alpha1.DesiredRunning},
+			Status:     v1alpha1.MinecraftServerStatus{Phase: v1alpha1.PhaseRunning, Players: v1alpha1.PlayersStatus{Online: 2, Max: 20}},
+		},
+		&v1alpha1.MinecraftServer{
+			ObjectMeta: metav1.ObjectMeta{Name: "creative", Namespace: "minecraft"},
+			Spec:       v1alpha1.MinecraftServerSpec{DesiredState: v1alpha1.DesiredStopped},
+			Status:     v1alpha1.MinecraftServerStatus{Phase: v1alpha1.PhaseStopped},
+		},
+		&v1alpha1.MinecraftServer{
+			ObjectMeta: metav1.ObjectMeta{Name: "lobby", Namespace: "minecraft", Labels: map[string]string{v1alpha1.LabelSystemRole: "lobby"}},
+			Spec:       v1alpha1.MinecraftServerSpec{DesiredState: v1alpha1.DesiredRunning},
+			Status:     v1alpha1.MinecraftServerStatus{Phase: v1alpha1.PhaseStarting},
+		},
+		// Another namespace's server is not this install's.
+		&v1alpha1.MinecraftServer{ObjectMeta: metav1.ObjectMeta{Name: "elsewhere", Namespace: "other"}},
+	}
+}
+
+// statusTestEnv is a host with the cluster above, a database backup nine hours
+// old, an off-site copy last good two hours ago, and one alert open.
+func statusTestEnv(t *testing.T) statusEnv {
+	t.Helper()
+	now := time.Date(2026, 9, 27, 12, 0, 0, 0, time.UTC)
+	dir := t.TempDir()
+	var w watchdogFlags
+	w.controlNS = "felis"
+	w.backupDir = filepath.Join(dir, "db-backups")
+	w.offsiteStatus = filepath.Join(dir, "offsite-status.json")
+	w.statePath = filepath.Join(dir, "state.json")
+	w.fallbackState = filepath.Join(dir, "fallback.json")
+	w.diskPaths = "/nonexistent-felis-status-test"
+	unitDir := filepath.Join(dir, "systemd")
+	for _, d := range []string{w.backupDir, unitDir} {
+		if err := os.MkdirAll(d, 0o700); err != nil {
+			t.Fatal(err)
+		}
+	}
+	writeTestFile(t, filepath.Join(w.backupDir, dbbackup.BundleName(now.Add(-9*time.Hour), dbbackup.LabelDaily)), "x", 0o600)
+	writeTestFile(t, filepath.Join(w.backupDir, dbbackup.BundleName(now.Add(-33*time.Hour), dbbackup.LabelDaily)), "x", 0o600)
+	if err := offsite.WriteStatus(w.offsiteStatus, offsite.Status{
+		LastAttempt: now.Add(-2 * time.Hour), LastSuccess: now.Add(-2 * time.Hour), Bucket: "felis-dr", Endpoint: "https://s3.example.com",
+	}); err != nil {
+		t.Fatal(err)
+	}
+	state := &watchdog.State{Alerts: map[string]*watchdog.Alert{
+		"db-backup-servers": {Finding: watchdog.Finding{Key: "db-backup-servers", Severity: watchdog.Warning}, FirstSeen: now.Add(-3 * time.Hour), Notified: now.Add(-90 * time.Minute)},
+		"proxy":             {Finding: watchdog.Finding{Key: "proxy", Severity: watchdog.Critical}, FirstSeen: now.Add(-time.Hour), Notified: now.Add(-time.Hour), ClearedAt: now.Add(-10 * time.Minute)},
+		"disk//var":         {Finding: watchdog.Finding{Key: "disk//var", Severity: watchdog.Critical}, FirstSeen: now.Add(-4 * time.Minute)},
+	}}
+	if err := watchdog.SaveState(w.statePath, state); err != nil {
+		t.Fatal(err)
+	}
+	meminfo := filepath.Join(dir, "meminfo")
+	writeTestFile(t, meminfo, "MemTotal:        8000000 kB\nMemFree:          100000 kB\nMemAvailable:    2000000 kB\n", 0o644)
+	writeTestFile(t, filepath.Join(unitDir, "felis-watchdog.timer"), "[Unit]\n", 0o644)
+	cfg := &config.Config{Offsite: config.OffsiteConfig{Endpoint: "https://s3.example.com", Bucket: "felis-dr"}}
+	return statusEnv{
+		cfg: cfg, w: w, cl: fake.NewClientBuilder().WithScheme(newSystemServerScheme(t)).WithObjects(statusClusterObjects()...).Build(),
+		backups: func(context.Context) (map[string]time.Time, error) {
+			return map[string]time.Time{"survival": now.Add(-3 * time.Hour)}, nil
+		},
+		run: func(_ context.Context, name string, args ...string) ([]byte, error) {
+			switch strings.Join(append([]string{name}, args...), " ") {
+			case "systemctl is-active felis-watchdog.timer":
+				return []byte("active\n"), nil
+			case "systemctl show --timestamp=unix -p Result -p ExecMainExitTimestamp felis-watchdog.service":
+				return []byte("Result=success\nExecMainExitTimestamp=@" + strconv.FormatInt(now.Add(-70*time.Second).Unix(), 10) + "\n"), nil
+			}
+			return nil, errors.New("unexpected")
+		},
+		unitDir: unitDir, meminfo: meminfo, host: "felis-test", now: now,
+	}
+}
+
+// lineFields is the words of the first line of out that starts with prefix
+// once trimmed.
+func lineFields(out, prefix string) []string {
+	for _, l := range strings.Split(out, "\n") {
+		if strings.HasPrefix(strings.TrimSpace(l), prefix) {
+			return strings.Fields(l)
+		}
+	}
+	return nil
+}
+
+func TestStatusReport(t *testing.T) {
+	env := statusTestEnv(t)
+	var out bytes.Buffer
+	printStatus(context.Background(), env, &out)
+	got := out.String()
+	for _, want := range []string{
+		"on felis-test at 2026-09-27 12:00 UTC\n",
+		"node:     felis-1 Ready, k3s v1.36.4+k3s1, CentOS Stream 9, kernel 5.14.0-630.el9.aarch64\n",
+		"\ncontrol plane (namespace felis):\n",
+		"\nproxy:    not on this host\n",
+		"\nservers (namespace minecraft): 3, 1 running, 2 players online\n",
+		"  database: newest " + dbbackup.BundleName(env.now.Add(-9*time.Hour), dbbackup.LabelDaily) + ", 9h0m ago; 2 bundles in " + env.w.backupDir + "\n",
+		"  off-site: last good sync 2h0m ago to felis-dr\n",
+		"  memory: 1.9 GiB available of 7.6 GiB\n",
+		"  timer active, last run 1m ago (success)\n",
+		"  alerts: 2 open (sudo felis doctor says where to look)\n" +
+			"    db-backup-servers (warning, mailed 1h30m ago)\n" +
+			"    disk//var (critical, seen 4m ago, not mailed yet)\n",
+	} {
+		if !strings.Contains(got, want) {
+			t.Errorf("status lacks %q:\n%s", want, got)
+		}
+	}
+	for prefix, want := range map[string]string{
+		"felis-api": "felis-api 1/1 ready felis-api:v1.4.0@0123456789ab restarts 2",
+		"NAME":      "NAME ROLE DESIRED PHASE PLAYERS NEWEST WORLD BACKUP",
+		"creative":  "creative - Stopped Stopped - none",
+		"lobby":     "lobby lobby Running Starting - none",
+		"survival":  "survival - Running Running 2/20 3h0m ago",
+	} {
+		if f := strings.Join(lineFields(got, prefix), " "); f != want {
+			t.Errorf("row %q = %q, want %q\n%s", prefix, f, want, got)
+		}
+	}
+	if strings.Contains(got, "elsewhere") || strings.Contains(got, "other-1") {
+		t.Errorf("status shows what is not this install's:\n%s", got)
+	}
+}
+
+// Whatever is down reads as down, and the rest of the report still prints.
+func TestStatusWithPartsDown(t *testing.T) {
+	env := statusTestEnv(t)
+	env.cl, env.clErr = nil, errors.New("connection refused")
+	env.cfg = &config.Config{}
+	var out bytes.Buffer
+	printStatus(context.Background(), env, &out)
+	for _, want := range []string{
+		"cluster:  unreachable (connection refused)\n",
+		"\nservers (namespace minecraft): unknown while the cluster is unreachable\n",
+		"  off-site: not configured, every backup is on this machine only\n",
+		"  timer active, last run",
+	} {
+		if !strings.Contains(out.String(), want) {
+			t.Errorf("status lacks %q:\n%s", want, out.String())
+		}
+	}
+
+	env = statusTestEnv(t)
+	env.backups = func(context.Context) (map[string]time.Time, error) { return nil, errors.New("postgres is down") }
+	out.Reset()
+	printStatus(context.Background(), env, &out)
+	if f := strings.Join(lineFields(out.String(), "survival"), " "); f != "survival - Running Running 2/20 ?" {
+		t.Errorf("survival with PostgreSQL down = %q", f)
+	}
+	if !strings.Contains(out.String(), "  world backups unknown: postgres is down\n") {
+		t.Errorf("status does not say why the backups are unknown:\n%s", out.String())
+	}
+}
+
+func TestShortImage(t *testing.T) {
+	for in, want := range map[string]string{
+		"registry.felis.svc:5000/felis/felis-api:v1.4.0":                                       "felis-api:v1.4.0",
+		"docker.io/library/postgres:17@sha256:0123456789abcdef0123":                            "postgres:17@0123456789ab",
+		"felis-operator@sha256:fedcba9876543210fedcba9876543210fedcba9876543210fedcba98765432": "felis-operator@fedcba987654",
+		"busybox": "busybox",
+	} {
+		if got := shortImage(in); got != want {
+			t.Errorf("shortImage(%q) = %q, want %q", in, got, want)
+		}
+	}
+}
+
+// A node that is not ready, a Deployment without replicas or containers, and
+// lists the API server refuses each read as such.
+func TestStatusClusterEdges(t *testing.T) {
+	env := statusTestEnv(t)
+	env.cfg = &config.Config{K8s: config.K8sConfig{Namespace: "games"}}
+	env.cl = fake.NewClientBuilder().WithScheme(newSystemServerScheme(t)).WithObjects(
+		&corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "felis-1"}, Status: corev1.NodeStatus{
+			Conditions: []corev1.NodeCondition{{Type: corev1.NodeReady, Status: corev1.ConditionFalse}, {Type: corev1.NodeMemoryPressure, Status: corev1.ConditionTrue}}}},
+		&appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: "felis-bare", Namespace: "felis"}},
+		&corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "p", Namespace: "felis"}, Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{{RestartCount: 4}}}},
+	).Build()
+	var out bytes.Buffer
+	printStatus(context.Background(), env, &out)
+	got := out.String()
+	for _, want := range []string{
+		"node:     felis-1 NotReady, k3s , , kernel \n",
+		"\nservers (namespace games): 0, 0 running, 0 players online\n\nbackups:\n",
+	} {
+		if !strings.Contains(got, want) {
+			t.Errorf("status lacks %q:\n%s", want, got)
+		}
+	}
+	if f := strings.Join(lineFields(got, "felis-bare"), " "); f != "felis-bare 0/1 ready - restarts 0" {
+		t.Errorf("row felis-bare = %q, want its one replica wanted, no image, and no pod of its own:\n%s", f, got)
+	}
+
+	failing := func(what client.ObjectList) {
+		env.cl = fake.NewClientBuilder().WithScheme(newSystemServerScheme(t)).WithObjects(statusClusterObjects()...).
+			WithInterceptorFuncs(interceptor.Funcs{List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error {
+				if fmt.Sprintf("%T", list) == fmt.Sprintf("%T", what) {
+					return errors.New("forbidden")
+				}
+				return c.List(ctx, list, opts...)
+			}}).Build()
+		out.Reset()
+		printStatus(context.Background(), env, &out)
+	}
+	for _, tc := range []struct {
+		list client.ObjectList
+		want string
+	}{
+		{&corev1.NodeList{}, "cluster:  unreachable (forbidden)\n\nproxy:"},
+		{&appsv1.DeploymentList{}, "\ncontrol plane (namespace felis):\n  cannot list it: forbidden\n\nproxy:"},
+		{&corev1.PodList{}, "\ncontrol plane (namespace felis):\n  cannot list it: forbidden\n\nproxy:"},
+		{&v1alpha1.MinecraftServerList{}, "\nservers (namespace games): cannot list them: forbidden\n\nbackups:"},
+	} {
+		failing(tc.list)
+		if !strings.Contains(out.String(), tc.want) {
+			t.Errorf("listing %T refused: status lacks %q:\n%s", tc.list, tc.want, out.String())
+		}
+	}
+}
+
+func TestStatusProxy(t *testing.T) {
+	env := statusTestEnv(t)
+	writeTestFile(t, filepath.Join(env.unitDir, "felis-velocity.service"), "[Unit]\n", 0o644)
+	env.run = fakeSystemctl("", map[string]string{"felis-velocity.service": "active"})
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	env.w.proxyAddr = ln.Addr().String()
+	var out bytes.Buffer
+	statusProxy(context.Background(), env, &out)
+	if want := "\nproxy:    felis-velocity active, " + env.w.proxyAddr + " accepts connections\n"; out.String() != want {
+		t.Errorf("proxy listening: %q, want %q", out.String(), want)
+	}
+	ln.Close()
+	out.Reset()
+	statusProxy(context.Background(), env, &out)
+	if want := "\nproxy:    felis-velocity active, " + env.w.proxyAddr + " refuses connections (dial tcp " + env.w.proxyAddr + ": "; !strings.HasPrefix(out.String(), want) {
+		t.Errorf("proxy gone: %q, want it to start %q", out.String(), want)
+	}
+}
+
+func TestStatusBackups(t *testing.T) {
+	env := statusTestEnv(t)
+	status := func() string {
+		var out bytes.Buffer
+		statusBackups(env, &out)
+		return out.String()
+	}
+	for _, tc := range []struct {
+		what, dir, want string
+	}{
+		{"no -backup-dir", "", "  database: not checked (felis-watchdog.service names no -backup-dir)\n"},
+		{"an empty directory", t.TempDir(), "  database: none in %s\n"},
+		{"a file where the directory should be", filepath.Join(env.w.backupDir, dbbackup.BundleName(env.now.Add(-9*time.Hour), dbbackup.LabelDaily)), "  database: cannot read %s: "},
+	} {
+		env.w.backupDir = tc.dir
+		want := tc.want
+		if strings.Contains(want, "%s") {
+			want = fmt.Sprintf(want, tc.dir)
+		}
+		if got := status(); !strings.Contains(got, want) {
+			t.Errorf("%s: %q, want %q", tc.what, got, want)
+		}
+	}
+
+	for _, tc := range []struct {
+		what   string
+		status *offsite.Status
+		raw    string
+		want   string
+	}{
+		{"never synced", nil, "", "  off-site: never synced\n"},
+		{"never succeeded", &offsite.Status{LastAttempt: env.now.Add(-time.Hour), LastError: "403 Forbidden"}, "", "  off-site: never succeeded; last attempt 1h0m ago: 403 Forbidden\n"},
+		{"the last attempt failed", &offsite.Status{LastAttempt: env.now.Add(-30 * time.Minute), LastSuccess: env.now.Add(-26 * time.Hour), LastError: "timeout", Bucket: "felis-dr"}, "",
+			"  off-site: last good sync 26h0m ago to felis-dr; the last attempt, 30m ago, failed: timeout\n"},
+		{"an error an earlier attempt left", &offsite.Status{LastAttempt: env.now.Add(-2 * time.Hour), LastSuccess: env.now.Add(-time.Hour), LastError: "timeout", Bucket: "felis-dr"}, "",
+			"  off-site: last good sync 1h0m ago to felis-dr\n"},
+		{"an unreadable record", nil, "{", "  off-site: unexpected end of JSON input\n"},
+	} {
+		os.Remove(env.w.offsiteStatus)
+		switch {
+		case tc.status != nil:
+			if err := offsite.WriteStatus(env.w.offsiteStatus, *tc.status); err != nil {
+				t.Fatal(err)
+			}
+		case tc.raw != "":
+			writeTestFile(t, env.w.offsiteStatus, tc.raw, 0o600)
+		}
+		if got := status(); !strings.HasSuffix(got, tc.want) {
+			t.Errorf("%s: %q, want it to end %q", tc.what, got, tc.want)
+		}
+	}
+}
+
+func TestStatusHost(t *testing.T) {
+	env := statusTestEnv(t)
+	dir := t.TempDir()
+	env.w.diskPaths = "/nonexistent-felis-status-test," + dir + "," + dir
+	env.meminfo = filepath.Join(dir, "meminfo")
+	var out bytes.Buffer
+	statusHost(env, &out)
+	lines := strings.Split(strings.TrimSuffix(out.String(), "\n"), "\n")
+	if len(lines) != 3 || lines[0] != "" || lines[1] != "host:" || !strings.HasPrefix(lines[2], "  disk "+dir+": ") || !strings.HasSuffix(lines[2], "% free)") {
+		t.Errorf("host: %q, want one line for %s (a path on a disk already shown, and one that is not there, print none) and no memory line", lines, dir)
+	}
+
+	writeTestFile(t, env.meminfo, "MemTotal: 4096 kB\ngarbage\nMemAvailable: 1024 kB\n", 0o644)
+	if total, avail, ok := readMeminfo(env.meminfo); !ok || total != 4096*1024 || avail != 1024*1024 {
+		t.Errorf("readMeminfo = %d, %d, %v", total, avail, ok)
+	}
+	writeTestFile(t, env.meminfo, "MemAvailable: 1024 kB\n", 0o644)
+	if _, _, ok := readMeminfo(env.meminfo); ok {
+		t.Error("readMeminfo without MemTotal: ok")
+	}
+}
+
+func TestStatusWatchdog(t *testing.T) {
+	env := statusTestEnv(t)
+	status := func() string {
+		var out bytes.Buffer
+		statusWatchdog(context.Background(), env, &out)
+		return out.String()
+	}
+	run := env.run
+	env.run = func(ctx context.Context, name string, args ...string) ([]byte, error) {
+		if len(args) > 0 && args[0] == "show" {
+			return []byte("Result=success\nExecMainExitTimestamp=\n"), nil
+		}
+		return run(ctx, name, args...)
+	}
+	if err := watchdog.SaveState(env.w.statePath, &watchdog.State{Alerts: map[string]*watchdog.Alert{
+		"proxy": {Finding: watchdog.Finding{Key: "proxy", Severity: watchdog.Critical}, FirstSeen: env.now.Add(-time.Hour), ClearedAt: env.now.Add(-time.Minute)},
+	}}); err != nil {
+		t.Fatal(err)
+	}
+	if got, want := status(), "\nwatchdog:\n  timer active\n  alerts: none open\n"; got != want {
+		t.Errorf("a watchdog that has not run and has nothing open: %q, want %q", got, want)
+	}
+
+	writeTestFile(t, env.w.statePath, "{", 0o600)
+	if got := status(); !strings.Contains(got, "\n  alerts: unknown (") {
+		t.Errorf("an unreadable state: %q", got)
+	}
+
+	if err := os.Remove(filepath.Join(env.unitDir, "felis-watchdog.timer")); err != nil {
+		t.Fatal(err)
+	}
+	if got, want := status(), "\nwatchdog:\n  not installed: nothing checks this host\n"; got != want {
+		t.Errorf("no watchdog timer: %q, want %q", got, want)
+	}
+}
+
+// Rows print by name in whatever order the API server lists them, a server
+// the operator has not reconciled yet reads as dashes, and open alerts print
+// by key in whatever order the state's map yields them.
+func TestStatusOrder(t *testing.T) {
+	env := statusTestEnv(t)
+	objs := append(statusClusterObjects(),
+		&appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: "felis-operator", Namespace: "felis"}},
+		&v1alpha1.MinecraftServer{ObjectMeta: metav1.ObjectMeta{Name: "fresh", Namespace: "minecraft"}})
+	env.cl = fake.NewClientBuilder().WithScheme(newSystemServerScheme(t)).WithObjects(objs...).
+		WithInterceptorFuncs(interceptor.Funcs{List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error {
+			// The fake lists by name; the other way round, then.
+			if err := c.List(ctx, list, opts...); err != nil {
+				return err
+			}
+			items, err := meta.ExtractList(list)
+			if err != nil {
+				return err
+			}
+			slices.Reverse(items)
+			return meta.SetList(list, items)
+		}}).Build()
+	var out bytes.Buffer
+	printStatus(context.Background(), env, &out)
+	var rows []string
+	for _, l := range strings.Split(out.String(), "\n") {
+		if f := strings.Fields(l); len(f) > 0 && (strings.HasPrefix(f[0], "felis-") || slices.Contains([]string{"creative", "fresh", "lobby", "survival"}, f[0])) {
+			rows = append(rows, strings.Join(f, " "))
+		}
+	}
+	want := []string{
+		"felis-api 1/1 ready felis-api:v1.4.0@0123456789ab restarts 2",
+		"felis-operator 0/1 ready - restarts 0",
+		"creative - Stopped Stopped - none",
+		"fresh - - - - none",
+		"lobby lobby Running Starting - none",
+		"survival - Running Running 2/20 3h0m ago",
+	}
+	if !slices.Equal(rows, want) {
+		t.Errorf("rows %q, want %q:\n%s", rows, want, out.String())
+	}
+
+	alerts := map[string]*watchdog.Alert{}
+	for i, key := range []string{"a", "b", "c", "d"} {
+		alerts[key] = &watchdog.Alert{Finding: watchdog.Finding{Key: key, Severity: watchdog.Warning}, FirstSeen: env.now.Add(-time.Duration(i+1) * time.Minute)}
+	}
+	if err := watchdog.SaveState(env.w.statePath, &watchdog.State{Alerts: alerts}); err != nil {
+		t.Fatal(err)
+	}
+	wantAlerts := "  alerts: 4 open (sudo felis doctor says where to look)\n" +
+		"    a (warning, seen 1m ago, not mailed yet)\n    b (warning, seen 2m ago, not mailed yet)\n" +
+		"    c (warning, seen 3m ago, not mailed yet)\n    d (warning, seen 4m ago, not mailed yet)\n"
+	// A map starts its walk at random: fifty walks all in order by chance
+	// is out of the question.
+	for range 50 {
+		out.Reset()
+		statusWatchdog(context.Background(), env, &out)
+		if !strings.HasSuffix(out.String(), wantAlerts) {
+			t.Fatalf("alerts %q, want them to end %q", out.String(), wantAlerts)
+		}
+	}
+}
+
+// A selector the API server would have refused picks no pods.
+func TestPodRestartsBadSelector(t *testing.T) {
+	pods := []corev1.Pod{{Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{{RestartCount: 3}}}}}
+	if n := podRestarts(&metav1.LabelSelector{MatchExpressions: []metav1.LabelSelectorRequirement{{Key: "app", Operator: "Near"}}}, pods); n != 0 {
+		t.Errorf("podRestarts with a bad selector = %d, want 0", n)
+	}
+	if n := podRestarts(&metav1.LabelSelector{}, pods); n != 3 {
+		t.Errorf("podRestarts with a selector that picks all = %d, want 3", n)
+	}
+}

cmd/felis/supportbundle.go

@@ -0,0 +1,862 @@
+package main
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"errors"
+	"flag"
+	"fmt"
+	"io"
+	"io/fs"
+	"net/url"
+	"os"
+	"path"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+	"text/tabwriter"
+	"time"
+
+	"felis.lolicon.best/internal/apis/felis/v1alpha1"
+	"felis.lolicon.best/internal/config"
+	"felis.lolicon.best/internal/platform"
+	"felis.lolicon.best/internal/watchdog"
+	appsv1 "k8s.io/api/apps/v1"
+	batchv1 "k8s.io/api/batch/v1"
+	corev1 "k8s.io/api/core/v1"
+	networkingv1 "k8s.io/api/networking/v1"
+	"k8s.io/apimachinery/pkg/api/meta"
+	"k8s.io/client-go/kubernetes"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/yaml"
+)
+
+// supportBundleDir is where felis support-bundle writes unless told otherwise.
+const supportBundleDir = "/var/lib/felis/support"
+
+// redacted stands in for every value the bundle leaves out.
+const redacted = "<redacted>"
+
+// minScrubLen is the shortest known secret value the bundle searches for: a
+// shorter one would blank ordinary words, and every secret the installer
+// generates is far longer.
+const minScrubLen = 8
+
+// hostSecretSources are the files on a Felis host that hold secrets; the
+// bundle reads them only to take each value out of what it collects.
+var hostSecretSources = secretSources{
+	envFiles:   []string{"/etc/felis/secrets.env", defaultOffsiteEnvFile},
+	valueFiles: []string{hostSMTPPasswordPath, hostUploadsS3AccessKeyPath, hostUploadsS3SecretKeyPath, defaultHeartbeatFile, "/opt/felis/velocity/forwarding.secret"},
+	propsFiles: []string{"/opt/felis/velocity/plugins/felis-link/felis-link.properties"},
+	tokenFiles: []string{"/var/lib/rancher/k3s/server/token", "/var/lib/rancher/k3s/server/agent-token"},
+}
+
+// cmdSupportBundle collects what someone helping with this host needs into
+// one tar.gz: felis status and felis doctor, the logs of the control plane,
+// the builds and the systemd units, the cluster's workloads and events, and a
+// summary of the configuration. It never collects a Secret, a ConfigMap, the
+// contents of a configuration file, the database or a world, and takes every
+// value of the host's secret files out of what it does collect. Game server
+// logs, which carry player names, addresses and chat, only with -server-logs.
+func cmdSupportBundle(args []string, stdout, stderr io.Writer) int {
+	fs := flag.NewFlagSet("support-bundle", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	outDir := fs.String("o", supportBundleDir, "directory to write the bundle to (created mode 0700 when missing)")
+	logLines := fs.Int64("log-lines", 2000, "lines kept from the end of each pod log and each unit's journal")
+	since := fs.Duration("since", 48*time.Hour, "how far back each unit's journal is read")
+	serverLogs := fs.Bool("server-logs", false, "also collect the game servers' own logs, which carry player names, IP addresses and chat")
+	unitDir := fs.String("systemd-dir", systemdUnitDir, "where the installer's systemd units are")
+	if err := fs.Parse(args); err != nil {
+		if errors.Is(err, flag.ErrHelp) {
+			return 0
+		}
+		return 2
+	}
+	if os.Geteuid() != 0 {
+		fmt.Fprintln(stderr, "felis support-bundle: run as root (sudo felis support-bundle): it reads root-only logs and state")
+		return 1
+	}
+	b := hostSupportBundle(*unitDir, *logLines, *since, *serverLogs)
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
+	defer cancel()
+	path, err := b.write(ctx, *outDir)
+	if err != nil {
+		fmt.Fprintf(stderr, "felis support-bundle: %v\n", err)
+		return 1
+	}
+	size := int64(0)
+	if st, err := os.Stat(path); err == nil {
+		size = st.Size()
+	}
+	fmt.Fprintf(stdout, "wrote %s (%s, mode 0600)\n", path, humanSize(size))
+	fmt.Fprintln(stdout, "MANIFEST.txt inside says what it holds and what was taken out. Read it through before you send it anywhere:")
+	fmt.Fprintln(stdout, "redaction finds this host's known secrets and the common ways a secret is logged, and a secret logged another way stays in.")
+	if !*serverLogs {
+		fmt.Fprintln(stdout, "Game server logs are left out; -server-logs adds them (player names, IP addresses, chat).")
+	}
+	return 0
+}
+
+func humanSize(n int64) string {
+	switch {
+	case n >= 1<<20:
+		return fmt.Sprintf("%.1f MiB", float64(n)/(1<<20))
+	case n >= 1<<10:
+		return fmt.Sprintf("%.1f KiB", float64(n)/(1<<10))
+	}
+	return fmt.Sprintf("%d B", n)
+}
+
+// shortDuration is d as Duration.String writes it, less the zero minutes and
+// seconds after whole hours or minutes: 48h, and 90m as 1h30m.
+func shortDuration(d time.Duration) string {
+	s := d.String()
+	if strings.HasSuffix(s, "m0s") {
+		s = strings.TrimSuffix(s, "0s")
+	}
+	if strings.HasSuffix(s, "h0m") {
+		s = strings.TrimSuffix(s, "0m")
+	}
+	return s
+}
+
+// supportBundle is one collection and what it reads the host through.
+type supportBundle struct {
+	host    string
+	now     time.Time
+	unitDir string
+	// w is how felis-watchdog.service runs the watchdog, wErr why it could
+	// not be read (w then holds the defaults).
+	w          watchdogFlags
+	wErr       error
+	cfg        *config.Config // nil when cfgErr
+	cfgErr     error
+	cl         client.Client // nil when clErr
+	clErr      error
+	logs       func(ctx context.Context, ns, pod, container string, previous bool) ([]byte, error)
+	run        func(ctx context.Context, name string, args ...string) ([]byte, error)
+	backups    func(ctx context.Context) (map[string]time.Time, error)
+	doctor     func(ctx context.Context, out io.Writer)
+	secrets    secretSources
+	logLines   int64
+	since      time.Duration
+	serverLogs bool
+	// Host files read whole, and the directory whose listing is kept.
+	meminfo, osRelease, procVersion, stateDir string
+}
+
+func hostSupportBundle(unitDir string, logLines int64, since time.Duration, serverLogs bool) *supportBundle {
+	host, _ := os.Hostname()
+	b := &supportBundle{
+		host: host, now: time.Now(), unitDir: unitDir, run: hostCommand, secrets: hostSecretSources,
+		logLines: logLines, since: since, serverLogs: serverLogs,
+		meminfo: "/proc/meminfo", osRelease: "/etc/os-release", procVersion: "/proc/version", stateDir: "/etc/felis",
+	}
+	var found bool
+	b.w, found, b.wErr = watchdogUnitFlags(filepath.Join(unitDir, "felis-watchdog.service"))
+	if b.wErr == nil && !found {
+		b.wErr = fmt.Errorf("%s is not installed; read the watchdog's defaults", filepath.Join(unitDir, "felis-watchdog.service"))
+	}
+	if b.cfg, b.cfgErr = config.Load(b.w.cfgPath); b.cfgErr == nil {
+		cfg := b.cfg
+		b.backups = func(ctx context.Context) (map[string]time.Time, error) {
+			return newestWorldBackups(ctx, cfg.Database.URL)
+		}
+	} else {
+		b.cfg = nil
+	}
+	if b.cl, b.clErr = buildSystemServerClient(); b.clErr != nil {
+		b.cl = nil
+	} else if rc, err := hostRESTConfig(); err != nil {
+		b.clErr = err
+		b.cl = nil
+	} else if cs, err := kubernetes.NewForConfig(rc); err != nil {
+		b.clErr = err
+		b.cl = nil
+	} else {
+		limit := int64(8 << 20)
+		b.logs = func(ctx context.Context, ns, pod, container string, previous bool) ([]byte, error) {
+			return cs.CoreV1().Pods(ns).GetLogs(pod, &corev1.PodLogOptions{
+				Container: container, Previous: previous, Timestamps: true, TailLines: &logLines, LimitBytes: &limit,
+			}).DoRaw(ctx)
+		}
+	}
+	b.doctor = func(ctx context.Context, out io.Writer) {
+		runDoctor(ctx, doctorEnv{unitDir: unitDir, run: hostCommand, now: b.now, host: host}, out)
+	}
+	return b
+}
+
+// bundleWriter streams scrubbed files into the archive and keeps what went
+// wrong while collecting, for MANIFEST.txt.
+type bundleWriter struct {
+	tw     *tar.Writer
+	prefix string
+	now    time.Time
+	scrub  *scrubber
+	errs   []string
+}
+
+// logText adds a log, or a report that quotes errors: known secrets and
+// anything logged as one come out.
+func (bw *bundleWriter) logText(name string, data []byte) error {
+	return bw.add(name, bw.scrub.text(data))
+}
+
+// plain adds a file whose secrets were already taken out by structure (the
+// cluster's objects, the configuration summary) or that names none (the
+// release, disk use, addresses): known secret values still come out, and
+// nothing else is rewritten.
+func (bw *bundleWriter) plain(name string, data []byte) error {
+	return bw.add(name, bw.scrub.values(data))
+}
+
+func (bw *bundleWriter) add(name string, data []byte) error {
+	hdr := &tar.Header{Name: path.Join(bw.prefix, name), Mode: 0o600, Size: int64(len(data)), ModTime: bw.now, Typeflag: tar.TypeReg}
+	if err := bw.tw.WriteHeader(hdr); err != nil {
+		return err
+	}
+	_, err := bw.tw.Write(data)
+	return err
+}
+
+func (bw *bundleWriter) failed(what string, err error) {
+	bw.errs = append(bw.errs, string(bw.scrub.text([]byte(fmt.Sprintf("%s: %v", what, err)))))
+}
+
+// write collects the bundle into dir and returns its path.
+func (b *supportBundle) write(ctx context.Context, dir string) (string, error) {
+	if _, err := os.Stat(dir); errors.Is(err, fs.ErrNotExist) {
+		if err := os.MkdirAll(dir, 0o700); err != nil {
+			return "", err
+		}
+	}
+	stamp := b.now.UTC().Format("20060102T150405Z")
+	base := fmt.Sprintf("felis-support-%s-%s", safeName(b.host), stamp)
+	final := filepath.Join(dir, base+".tar.gz")
+	f, err := os.CreateTemp(dir, "."+base+".*.partial") // mode 0600
+	if err != nil {
+		return "", err
+	}
+	keep := false
+	defer func() {
+		if !keep {
+			f.Close()
+			os.Remove(f.Name())
+		}
+	}()
+	gz := gzip.NewWriter(f)
+	bw := &bundleWriter{tw: tar.NewWriter(gz), prefix: base, now: b.now, scrub: b.scrubber()}
+	if err := b.collect(ctx, bw); err != nil {
+		return "", err
+	}
+	if err := bw.tw.Close(); err != nil {
+		return "", err
+	}
+	if err := gz.Close(); err != nil {
+		return "", err
+	}
+	if err := f.Sync(); err != nil {
+		return "", err
+	}
+	if err := f.Close(); err != nil {
+		return "", err
+	}
+	if err := os.Rename(f.Name(), final); err != nil {
+		return "", err
+	}
+	keep = true
+	return final, nil
+}
+
+// safeName keeps a host name usable in a file name.
+func safeName(s string) string {
+	s = strings.Map(func(r rune) rune {
+		if r == '-' || r == '.' || r == '_' || (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') {
+			return r
+		}
+		return '_'
+	}, s)
+	if s == "" {
+		return "host"
+	}
+	return s
+}
+
+func (b *supportBundle) collect(ctx context.Context, bw *bundleWriter) error {
+	var buf bytes.Buffer
+	cmdVersion(nil, &buf, io.Discard)
+	for _, p := range []string{b.osRelease, b.procVersion} {
+		if raw, err := os.ReadFile(p); err == nil {
+			fmt.Fprintf(&buf, "\n# %s\n%s", p, raw)
+		}
+	}
+	if err := bw.plain("version.txt", buf.Bytes()); err != nil {
+		return err
+	}
+
+	buf.Reset()
+	switch {
+	case b.cfgErr != nil:
+		fmt.Fprintf(&buf, "felis status: the configuration did not load: %v\n", b.cfgErr)
+	default:
+		printStatus(ctx, statusEnv{
+			cfg: b.cfg, w: b.w, cl: b.cl, clErr: b.clErr, backups: b.backups, run: b.run,
+			unitDir: b.unitDir, meminfo: b.meminfo, host: b.host, now: b.now,
+		}, &buf)
+	}
+	if err := bw.logText("status.txt", buf.Bytes()); err != nil {
+		return err
+	}
+	buf.Reset()
+	b.doctor(ctx, &buf)
+	if err := bw.logText("doctor.txt", buf.Bytes()); err != nil {
+		return err
+	}
+	if b.cfg != nil {
+		if err := bw.plain("config.txt", configSummary(b.cfg, b.w.cfgPath)); err != nil {
+			return err
+		}
+	}
+	if err := b.collectHost(ctx, bw); err != nil {
+		return err
+	}
+	if err := b.collectJournal(ctx, bw); err != nil {
+		return err
+	}
+	if b.cl == nil {
+		bw.failed("cluster", b.clErr)
+	} else if err := b.collectCluster(ctx, bw); err != nil {
+		return err
+	}
+	return bw.add("MANIFEST.txt", b.manifest(bw))
+}
+
+func (b *supportBundle) collectHost(ctx context.Context, bw *bundleWriter) error {
+	commands := []struct {
+		name string
+		argv []string
+	}{
+		{"host/systemd-units.txt", []string{"systemctl", "list-units", "--all", "--no-pager", "--plain", "felis-*", "k3s.service"}},
+		{"host/systemd-timers.txt", []string{"systemctl", "list-timers", "--all", "--no-pager", "felis-*"}},
+		{"host/df.txt", []string{"df", "-h"}},
+		{"host/addresses.txt", []string{"ip", "-brief", "address"}},
+	}
+	for _, c := range commands {
+		out, err := b.run(ctx, c.argv[0], c.argv[1:]...)
+		if err != nil && len(out) == 0 {
+			bw.failed(strings.Join(c.argv, " "), err)
+			continue
+		}
+		if err := bw.plain(c.name, out); err != nil {
+			return err
+		}
+	}
+	if raw, err := os.ReadFile(b.meminfo); err == nil {
+		if err := bw.plain("host/meminfo.txt", raw); err != nil {
+			return err
+		}
+	}
+	listing, err := dirListing(b.stateDir)
+	if err != nil {
+		bw.failed("list "+b.stateDir, err)
+		return nil
+	}
+	return bw.plain("host/etc-felis.txt", listing)
+}
+
+// dirListing names every file under dir with its mode, size and time, and
+// holds nothing of what is in them.
+func dirListing(dir string) ([]byte, error) {
+	var buf bytes.Buffer
+	tw := tabwriter.NewWriter(&buf, 0, 0, 2, ' ', 0)
+	fmt.Fprintf(tw, "# %s: names, modes, sizes and times only; no contents\n", dir)
+	err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
+		if err != nil {
+			return err
+		}
+		info, err := d.Info()
+		if err != nil {
+			return err
+		}
+		fmt.Fprintf(tw, "%s\t%d\t%s\t%s\n", info.Mode(), info.Size(), info.ModTime().UTC().Format(time.RFC3339), p)
+		return nil
+	})
+	tw.Flush()
+	return buf.Bytes(), err
+}
+
+func (b *supportBundle) collectJournal(ctx context.Context, bw *bundleWriter) error {
+	units, _ := filepath.Glob(filepath.Join(b.unitDir, "felis-*.service"))
+	if _, err := os.Stat(filepath.Join(b.unitDir, "k3s.service")); err == nil {
+		units = append(units, filepath.Join(b.unitDir, "k3s.service"))
+	}
+	since := "@" + strconv.FormatInt(b.now.Add(-b.since).Unix(), 10)
+	for _, u := range units {
+		unit := filepath.Base(u)
+		out, err := b.run(ctx, "journalctl", "-u", unit, "--since", since, "-n", strconv.FormatInt(b.logLines, 10), "--no-pager", "-o", "short-iso")
+		if err != nil && len(out) == 0 {
+			bw.failed("journalctl -u "+unit, err)
+			continue
+		}
+		if err := bw.logText("journal/"+strings.TrimSuffix(unit, ".service")+".log", out); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// bundleNamespaces are the namespaces whose objects and logs the bundle
+// collects: the control plane, the builds and the game servers.
+func (b *supportBundle) bundleNamespaces() (control, build, minecraft string) {
+	control, build, minecraft = b.w.controlNS, platform.DefaultBuildNamespace, platform.DefaultMinecraftNamespace
+	if b.cfg != nil {
+		if b.cfg.Registry.BuildNamespace != "" {
+			build = b.cfg.Registry.BuildNamespace
+		}
+		if b.cfg.K8s.Namespace != "" {
+			minecraft = b.cfg.K8s.Namespace
+		}
+	}
+	return control, build, minecraft
+}
+
+func (b *supportBundle) collectCluster(ctx context.Context, bw *bundleWriter) error {
+	control, build, minecraft := b.bundleNamespaces()
+	dump := func(name string, list client.ObjectList, opts ...client.ListOption) error {
+		if err := b.cl.List(ctx, list, opts...); err != nil {
+			bw.failed("list "+name, err)
+			return nil
+		}
+		redactList(list)
+		out, err := yaml.Marshal(list)
+		if err != nil {
+			bw.failed("encode "+name, err)
+			return nil
+		}
+		return bw.plain(name, out)
+	}
+	if err := dump("cluster/nodes.yaml", &corev1.NodeList{}); err != nil {
+		return err
+	}
+	if err := dump("cluster/persistentvolumes.yaml", &corev1.PersistentVolumeList{}); err != nil {
+		return err
+	}
+	if err := dump("cluster/minecraftservers.yaml", &v1alpha1.MinecraftServerList{}, client.InNamespace(minecraft)); err != nil {
+		return err
+	}
+	for _, ns := range []string{control, build, minecraft} {
+		for _, k := range []struct {
+			name string
+			list client.ObjectList
+		}{
+			{"pods", &corev1.PodList{}},
+			{"deployments", &appsv1.DeploymentList{}},
+			{"statefulsets", &appsv1.StatefulSetList{}},
+			{"jobs", &batchv1.JobList{}},
+			{"cronjobs", &batchv1.CronJobList{}},
+			{"services", &corev1.ServiceList{}},
+			{"persistentvolumeclaims", &corev1.PersistentVolumeClaimList{}},
+			{"networkpolicies", &networkingv1.NetworkPolicyList{}},
+			{"events", &corev1.EventList{}},
+		} {
+			if err := dump("cluster/"+ns+"/"+k.name+".yaml", k.list, client.InNamespace(ns)); err != nil {
+				return err
+			}
+		}
+	}
+
+	var all corev1.PodList
+	if err := b.cl.List(ctx, &all); err != nil {
+		bw.failed("list every pod", err)
+	} else if err := bw.plain("cluster/pods-all-namespaces.txt", podTable(all.Items, b.now)); err != nil {
+		return err
+	}
+
+	for _, ns := range []string{control, build, minecraft} {
+		var pods corev1.PodList
+		if err := b.cl.List(ctx, &pods, client.InNamespace(ns)); err != nil {
+			continue // already recorded by the dump above
+		}
+		for _, p := range pods.Items {
+			if err := b.collectPodLogs(ctx, bw, p, ns == minecraft && !b.serverLogs); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+// collectPodLogs keeps the tail of each container's log, and of its previous
+// run when it restarted. initOnly keeps only the init containers: a game
+// server's own log carries player names, addresses and chat.
+func (b *supportBundle) collectPodLogs(ctx context.Context, bw *bundleWriter, p corev1.Pod, initOnly bool) error {
+	type ctr struct {
+		name     string
+		restarts int32
+	}
+	var ctrs []ctr
+	restarts := map[string]int32{}
+	for _, cs := range append(append([]corev1.ContainerStatus(nil), p.Status.InitContainerStatuses...), p.Status.ContainerStatuses...) {
+		restarts[cs.Name] = cs.RestartCount
+	}
+	for _, c := range p.Spec.InitContainers {
+		ctrs = append(ctrs, ctr{c.Name, restarts[c.Name]})
+	}
+	if !initOnly {
+		for _, c := range p.Spec.Containers {
+			ctrs = append(ctrs, ctr{c.Name, restarts[c.Name]})
+		}
+	}
+	for _, c := range ctrs {
+		for _, previous := range []bool{false, true} {
+			if previous && c.restarts == 0 {
+				continue
+			}
+			name := fmt.Sprintf("logs/%s/%s/%s.log", p.Namespace, p.Name, c.name)
+			if previous {
+				name = fmt.Sprintf("logs/%s/%s/%s.previous.log", p.Namespace, p.Name, c.name)
+			}
+			out, err := b.logs(ctx, p.Namespace, p.Name, c.name, previous)
+			if err != nil {
+				bw.failed(name, err)
+				continue
+			}
+			if err := bw.logText(name, out); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+// podTable is every pod on the node, one line each.
+func podTable(pods []corev1.Pod, now time.Time) []byte {
+	sort.Slice(pods, func(i, j int) bool {
+		if pods[i].Namespace != pods[j].Namespace {
+			return pods[i].Namespace < pods[j].Namespace
+		}
+		return pods[i].Name < pods[j].Name
+	})
+	var buf bytes.Buffer
+	tw := tabwriter.NewWriter(&buf, 0, 0, 2, ' ', 0)
+	fmt.Fprintln(tw, "NAMESPACE\tNAME\tPHASE\tREADY\tRESTARTS\tAGE")
+	for _, p := range pods {
+		var ready, restarts int32
+		for _, cs := range p.Status.ContainerStatuses {
+			if cs.Ready {
+				ready++
+			}
+			restarts += cs.RestartCount
+		}
+		age := "-"
+		if !p.CreationTimestamp.IsZero() {
+			age = now.Sub(p.CreationTimestamp.Time).Round(time.Minute).String()
+		}
+		fmt.Fprintf(tw, "%s\t%s\t%s\t%d/%d\t%d\t%s\n", p.Namespace, p.Name, p.Status.Phase, ready, len(p.Spec.Containers), restarts, age)
+	}
+	tw.Flush()
+	return buf.Bytes()
+}
+
+// redactList takes out of every object what the bundle must not carry: the
+// literal env values of pod specs and of MinecraftServers (valueFrom
+// references stay, naming the Secret without its contents), the
+// last-applied-configuration annotation that repeats them, and managedFields.
+func redactList(list client.ObjectList) {
+	items, err := meta.ExtractList(list)
+	if err != nil {
+		return
+	}
+	for _, it := range items {
+		if acc, err := meta.Accessor(it); err == nil {
+			acc.SetManagedFields(nil)
+			if ann := acc.GetAnnotations(); ann != nil {
+				delete(ann, corev1.LastAppliedConfigAnnotation)
+				acc.SetAnnotations(ann)
+			}
+		}
+		switch o := it.(type) {
+		case *corev1.Pod:
+			redactPodSpec(&o.Spec)
+		case *appsv1.Deployment:
+			redactPodSpec(&o.Spec.Template.Spec)
+		case *appsv1.StatefulSet:
+			redactPodSpec(&o.Spec.Template.Spec)
+		case *batchv1.Job:
+			redactPodSpec(&o.Spec.Template.Spec)
+		case *batchv1.CronJob:
+			redactPodSpec(&o.Spec.JobTemplate.Spec.Template.Spec)
+		case *v1alpha1.MinecraftServer:
+			for i := range o.Spec.Env {
+				if o.Spec.Env[i].Value != "" {
+					o.Spec.Env[i].Value = redacted
+				}
+			}
+		}
+	}
+}
+
+func redactPodSpec(s *corev1.PodSpec) {
+	blank := func(cs []corev1.Container) {
+		for i := range cs {
+			for j := range cs[i].Env {
+				if cs[i].Env[j].Value != "" {
+					cs[i].Env[j].Value = redacted
+				}
+			}
+		}
+	}
+	blank(s.InitContainers)
+	blank(s.Containers)
+	for i := range s.EphemeralContainers {
+		for j := range s.EphemeralContainers[i].Env {
+			if s.EphemeralContainers[i].Env[j].Value != "" {
+				s.EphemeralContainers[i].Env[j].Value = redacted
+			}
+		}
+	}
+}
+
+// configSummary is felis.toml without a single credential: the hostnames,
+// namespaces and which features are on. Fields are picked one by one, so a
+// field added later stays out until someone decides it is safe.
+func configSummary(c *config.Config, path string) []byte {
+	var buf bytes.Buffer
+	line := func(k string, v any) { fmt.Fprintf(&buf, "%-34s %v\n", k, v) }
+	fmt.Fprintf(&buf, "# a summary of %s; no password, key or token is in it\n", path)
+	line("server.root_domain", c.Server.RootDomain)
+	line("server.listen", c.Server.Listen)
+	db := "(unparsable)"
+	if u, err := url.Parse(c.Database.URL); err == nil {
+		db = u.Scheme + "://" + u.User.Username() + "@" + u.Host + u.Path
+	}
+	line("database.url (no password)", db)
+	line("database.deployment", c.Database.Deployment)
+	line("velocity.public_ip", c.Velocity.PublicIP)
+	line("velocity.game_port", c.Velocity.GamePort)
+	line("velocity.login_image", c.Velocity.LoginImage)
+	line("velocity.lobby_image", c.Velocity.LobbyImage)
+	line("auth.panel_hostname", c.Auth.PanelHostname)
+	line("auth.admin_hostname", c.Auth.AdminHostname)
+	line("auth.client_ip_header", c.Auth.ClientIPHeader)
+	line("k8s.namespace", c.K8s.Namespace)
+	line("k8s.egress_mode", c.K8s.EgressMode)
+	line("k8s.metallb_pool", c.K8s.MetalLBPool)
+	line("registry.url", c.Registry.URL)
+	line("registry.build_namespace", c.Registry.BuildNamespace)
+	line("registry.trivy_db_repository", c.Registry.TrivyDBRepository)
+	line("registry.build_user_namespaces", c.Registry.BuildUserNamespaces)
+	line("registry.build_runtime_class", c.Registry.BuildRuntimeClass)
+	line("registry.max_concurrent_builds", c.Registry.MaxConcurrentBuilds)
+	line("registry.user_uploads_context", c.Registry.UserUploadsContext)
+	line("archive.store", c.Archive.Store)
+	line("archive.local_path", c.Archive.LocalPath)
+	line("archive.retention", c.Archive.Retention)
+	line("archive.scheduled_every", c.Archive.ScheduledEvery)
+	line("archive.scheduled_keep", c.Archive.ScheduledKeep)
+	line("offsite (configured)", c.Offsite.Enabled())
+	if c.Offsite.Enabled() {
+		line("offsite.endpoint", c.Offsite.Endpoint)
+		line("offsite.bucket", c.Offsite.Bucket)
+		line("offsite.prefix", c.Offsite.Prefix)
+	}
+	line("smtp.host", c.SMTP.Host)
+	if c.SMTP.Host != "" {
+		line("smtp.port", c.SMTP.Port)
+		line("smtp.require_tls", c.SMTP.TLSRequired())
+		line("smtp.max_per_hour", c.SMTP.MaxPerHour)
+	}
+	for i, s := range c.AuthSources {
+		line(fmt.Sprintf("auth_source[%d]", i), s.Tag+" "+s.Prefix+" "+s.URL)
+	}
+	return buf.Bytes()
+}
+
+func (b *supportBundle) manifest(bw *bundleWriter) []byte {
+	control, build, minecraft := b.bundleNamespaces()
+	var buf bytes.Buffer
+	fmt.Fprintf(&buf, "Felis support bundle\nhost %s, collected %s, felis %s\n\n", b.host, b.now.UTC().Format(time.RFC3339), resolvedVersion())
+	fmt.Fprintf(&buf, `What it holds:
+  status.txt, doctor.txt   felis status and felis doctor at collection time
+  version.txt              the release, the OS and the kernel
+  config.txt               a summary of felis.toml: hostnames, namespaces, which features are on
+  host/                    systemd units and timers, disk use, memory, addresses, and the names,
+                           modes, sizes and times of the files under %s (not what is in them)
+  journal/                 up to %d lines per Felis unit and k3s, from the last %s
+  logs/                    up to %d lines of each container of the pods in %s and %s, and of the
+                           init containers of the game server pods in %s; the previous run too
+                           where a container restarted
+  cluster/                 nodes, volumes, the MinecraftServers, and the pods, workloads, services,
+                           volume claims, network policies and events of %s, %s and %s
+`, b.stateDir, b.logLines, shortDuration(b.since), b.logLines, control, build, minecraft, control, build, minecraft)
+	if b.serverLogs {
+		fmt.Fprintln(&buf, "\nThe game servers' own logs are in logs/ (-server-logs): they carry player names, IP addresses and chat.")
+	} else {
+		fmt.Fprintln(&buf, "\nThe game servers' own logs are left out (they carry player names, IP addresses and chat; -server-logs adds them).")
+	}
+	fmt.Fprint(&buf, `
+Never collected: Kubernetes Secrets and ConfigMaps, what is in /etc/felis or any configuration
+file, the database, worlds, uploads.
+
+Taken out:
+`)
+	if len(bw.scrub.sources) > 0 {
+		fmt.Fprintf(&buf, "  - %d secret values, wherever they appear, read from:\n", len(bw.scrub.vals))
+		for _, s := range bw.scrub.sources {
+			fmt.Fprintf(&buf, "      %s\n", s)
+		}
+	} else {
+		fmt.Fprintln(&buf, "  - no secret file was found on this host to take values from")
+	}
+	fmt.Fprint(&buf, `  - passwords in URLs, private keys, Bearer and Basic credentials
+  - in logs and command output, whatever follows password=, secret=, token=, api_key=,
+    access_key=, private_key= or credentials= (and the same with a colon)
+  - every literal env value in pod specs and MinecraftServers (valueFrom references stay)
+
+Read it through before you send it anywhere: redaction finds this host's known secrets and the
+common ways a secret is logged, and a secret logged another way stays in.
+`)
+	if b.wErr != nil {
+		fmt.Fprintf(&buf, "\nThe watchdog's settings: %v\n", b.wErr)
+	}
+	if len(bw.errs) > 0 {
+		fmt.Fprintln(&buf, "\nNot collected:")
+		for _, e := range bw.errs {
+			fmt.Fprintf(&buf, "  - %s\n", e)
+		}
+	}
+	// Its own words name what is redacted, and would be redacted themselves;
+	// what it quotes was scrubbed as it was recorded.
+	return buf.Bytes()
+}
+
+// secretSources are files that hold secrets, by how each is laid out.
+type secretSources struct {
+	envFiles   []string // KEY=VALUE lines, every value a secret (a commented-out one too)
+	valueFiles []string // one secret, the whole file
+	propsFiles []string // key=value lines; the keys naming a token, secret, password or key hold one
+	tokenFiles []string // k3s join tokens: the whole token and its secret part after the last ':'
+}
+
+// scrubber takes secrets out of what the bundle collects.
+type scrubber struct {
+	vals    []string // longest first, so a secret that contains another goes whole
+	sources []string // the files vals came from
+}
+
+var (
+	pemPrivateKey = regexp.MustCompile(`(?s)-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----.*?-----END [A-Z0-9 ]*PRIVATE KEY-----`)
+	urlUserinfo   = regexp.MustCompile(`([A-Za-z][A-Za-z0-9+.-]*://[^/\s:@]*:)[^/\s@]+@`)
+	authScheme    = regexp.MustCompile(`(?i)\b(bearer|basic)\s+[A-Za-z0-9._~+/=-]{8,}`)
+	secretAssign  = regexp.MustCompile(`(?i)((?:password|passwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|credentials?)[A-Za-z0-9_.-]*"?[ \t]*[=:][ \t]*"?)([^\s"',;&]{4,})`)
+	secretPropKey = regexp.MustCompile(`(?i)token|secret|password|key`)
+)
+
+func (b *supportBundle) scrubber() *scrubber {
+	s := &scrubber{}
+	s.readSources(b.secrets)
+	if b.cfg != nil {
+		if u, err := url.Parse(b.cfg.Database.URL); err == nil {
+			if pw, ok := u.User.Password(); ok {
+				s.add(pw)
+			}
+		}
+		for _, ref := range []string{
+			b.cfg.Velocity.ServiceTokenRef, b.cfg.SMTP.PasswordRef,
+			b.cfg.Offsite.AccessKeyRef, b.cfg.Offsite.SecretKeyRef, b.cfg.Offsite.KeyRef,
+			b.cfg.Registry.S3.AccessKeyRef, b.cfg.Registry.S3.SecretKeyRef,
+			b.cfg.Archive.S3.AccessKeyRef, b.cfg.Archive.S3.SecretKeyRef,
+		} {
+			if ref != "" {
+				s.add(os.Getenv(ref))
+			}
+		}
+	}
+	if st, err := watchdog.LoadState(watchdog.NewestState(b.w.statePath, b.w.fallbackState)); err == nil {
+		s.add(st.SMTPPassword)
+	}
+	return s
+}
+
+func (s *scrubber) add(v string) bool {
+	v = strings.TrimSpace(v)
+	if len(v) < minScrubLen {
+		return false
+	}
+	for _, have := range s.vals {
+		if have == v {
+			return true
+		}
+	}
+	s.vals = append(s.vals, v)
+	sort.SliceStable(s.vals, func(i, j int) bool { return len(s.vals[i]) > len(s.vals[j]) })
+	return true
+}
+
+func (s *scrubber) readSources(src secretSources) {
+	read := func(p string, take func(content string) bool) {
+		raw, err := os.ReadFile(p)
+		if err != nil {
+			return
+		}
+		if take(string(raw)) {
+			s.sources = append(s.sources, p)
+		}
+	}
+	keyValues := func(content string, keep func(key string) bool) bool {
+		found := false
+		for _, line := range strings.Split(content, "\n") {
+			k, v, ok := strings.Cut(line, "=")
+			if !ok || !keep(strings.TrimSpace(k)) {
+				continue
+			}
+			v = strings.TrimSpace(v)
+			if len(v) >= 2 && (v[0] == '\'' || v[0] == '"') && v[len(v)-1] == v[0] {
+				v = v[1 : len(v)-1]
+			}
+			found = s.add(v) || found
+		}
+		return found
+	}
+	for _, p := range src.envFiles {
+		read(p, func(c string) bool { return keyValues(c, func(string) bool { return true }) })
+	}
+	for _, p := range src.propsFiles {
+		read(p, func(c string) bool { return keyValues(c, secretPropKey.MatchString) })
+	}
+	for _, p := range src.valueFiles {
+		read(p, func(c string) bool { return s.add(c) })
+	}
+	for _, p := range src.tokenFiles {
+		read(p, func(c string) bool {
+			c = strings.TrimSpace(c)
+			whole := s.add(c)
+			part := false
+			if i := strings.LastIndex(c, ":"); i >= 0 {
+				part = s.add(c[i+1:])
+			}
+			return whole || part
+		})
+	}
+}
+
+// values takes every known secret value out of data.
+func (s *scrubber) values(data []byte) []byte {
+	t := pemPrivateKey.ReplaceAllString(string(data), "<redacted private key>")
+	for _, v := range s.vals {
+		t = strings.ReplaceAll(t, v, redacted)
+	}
+	t = urlUserinfo.ReplaceAllString(t, "${1}"+redacted+"@")
+	t = authScheme.ReplaceAllString(t, "${1} "+redacted)
+	return []byte(t)
+}
+
+// text is values, and whatever a log names as a secret as well.
+func (s *scrubber) text(data []byte) []byte {
+	return secretAssign.ReplaceAll(s.values(data), []byte("${1}"+redacted))
+}

cmd/felis/supportbundle_test.go

@@ -0,0 +1,433 @@
+package main
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"testing"
+	"time"
+
+	"felis.lolicon.best/internal/apis/felis/v1alpha1"
+	"felis.lolicon.best/internal/config"
+	"felis.lolicon.best/internal/watchdog"
+	appsv1 "k8s.io/api/apps/v1"
+	batchv1 "k8s.io/api/batch/v1"
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+)
+
+// The secrets a bundle host holds, each of which must be nowhere in the bundle.
+var plantedSecrets = map[string]string{
+	"secrets.env SERVICE_TOKEN":     "svc-token-planted-0a1b2c3d",
+	"secrets.env DB_PASSWORD":       "db-pass-planted-4e5f6a7b",
+	"offsite.env FELIS_OFFSITE_KEY": "offsite-key-planted+8c9d/0e1f=",
+	"smtp-password":                 "smtp-pass-planted-2a3b",
+	"felis-link.properties token":   "props-token-planted-4c5d",
+	"k3s token secret part":         "k3s-secret-part-planted-6e7f",
+	"watchdog state relay password": "cached-relay-pw-planted-8a9b",
+	"pod env literal":               "env-literal-planted-0c1d",
+	"deployment env literal":        "deploy-env-planted-2e3f",
+	"MinecraftServer env literal":   "cr-env-planted-4a5b",
+	"last-applied annotation":       "annotation-planted-6c7d",
+	"logged password":               "hunter2-planted-8e9f",
+	"logged bearer":                 "bearerplanted0a1b2c3d",
+	"URL password":                  "urlpass-planted-4e5f",
+	"token in an error":             "errtoken-planted-0f1e",
+	"felis.host.toml DB password":   "cfg-db-pass-planted-1c2d",
+	"service token by its env ref":  "env-ref-token-planted-3e4f",
+	"init container env literal":    "init-env-planted-5a6b",
+	"ephemeral container env":       "ephemeral-env-planted-7c8d",
+	"statefulset env literal":       "sts-env-planted-9e0f",
+	"job env literal":               "job-env-planted-1a2b",
+	"cronjob env literal":           "cronjob-env-planted-3c4d",
+	"commented-out offsite key":     "old-offsite-key-planted-5e6f",
+	"doctor quoting a password":     "doctor-pw-planted-7a8b",
+	"status quoting a password":     "status-pw-planted-9c0d",
+	"journal quoting a password":    "journal-pw-planted-1e2f",
+}
+
+// bundleHost is a Felis host for felis support-bundle: its secret files, its
+// watchdog unit and state, a cluster with a control-plane pod that restarted
+// and a game server, and logs and a journal that name secrets.
+func bundleHost(t *testing.T) *supportBundle {
+	t.Helper()
+	p := plantedSecrets
+	dir := t.TempDir()
+	etc := filepath.Join(dir, "etc-felis")
+	for _, d := range []string{etc, filepath.Join(dir, "systemd"), filepath.Join(dir, "k3s")} {
+		if err := os.MkdirAll(d, 0o700); err != nil {
+			t.Fatal(err)
+		}
+	}
+	writeTestFile(t, filepath.Join(etc, "secrets.env"), "SERVICE_TOKEN="+p["secrets.env SERVICE_TOKEN"]+"\nDB_PASSWORD="+p["secrets.env DB_PASSWORD"]+"\nSHORT=abc\n", 0o600)
+	writeTestFile(t, filepath.Join(etc, "offsite.env"), "# before the rotation\n# FELIS_OFFSITE_KEY="+p["commented-out offsite key"]+"\nFELIS_OFFSITE_KEY='"+p["offsite.env FELIS_OFFSITE_KEY"]+"'\n", 0o600)
+	writeTestFile(t, filepath.Join(etc, "smtp-password"), p["smtp-password"]+"\n", 0o600)
+	writeTestFile(t, filepath.Join(etc, "felis-link.properties"), "api-base-url=http://10.43.0.10:8081\nservice-token="+p["felis-link.properties token"]+"\nroot-domain=games.example.org\n", 0o640)
+	writeTestFile(t, filepath.Join(dir, "k3s", "token"), "K10deadbeefcafe::server:"+p["k3s token secret part"]+"\n", 0o600)
+	cfgPath := filepath.Join(etc, "felis.host.toml")
+	writeTestFile(t, cfgPath, "[database]\nurl = \"postgres://felis:"+p["felis.host.toml DB password"]+"@127.0.0.1:1/felis?sslmode=disable&connect_timeout=1\"\n"+
+		"[server]\nroot_domain = \"games.example.org\"\n[archive]\nstore = \"tarLocal\"\n[k8s]\negress_mode = \"nodeport\"\n"+
+		"[velocity]\nservice_token_ref = \"FELIS_BUNDLE_TEST_SERVICE_TOKEN\"\n", 0o600)
+	t.Setenv("FELIS_BUNDLE_TEST_SERVICE_TOKEN", p["service token by its env ref"])
+	statePath := filepath.Join(dir, "state.json")
+	if err := watchdog.SaveState(statePath, &watchdog.State{SMTPPassword: p["watchdog state relay password"]}); err != nil {
+		t.Fatal(err)
+	}
+	unitDir := filepath.Join(dir, "systemd")
+	writeTestFile(t, filepath.Join(unitDir, "felis-watchdog.service"), "[Service]\nExecStart=/usr/local/bin/felis watchdog -config "+cfgPath+" -state "+statePath+"\n", 0o644)
+	writeTestFile(t, filepath.Join(unitDir, "felis-offsite.service"), "[Unit]\n", 0o644)
+	writeTestFile(t, filepath.Join(unitDir, "k3s.service"), "[Unit]\n", 0o644)
+	meminfo := filepath.Join(dir, "meminfo")
+	writeTestFile(t, meminfo, "MemTotal: 8000000 kB\nMemAvailable: 2000000 kB\n", 0o644)
+
+	cfg, err := config.Load(cfgPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var w watchdogFlags
+	w, _, err = watchdogUnitFlags(filepath.Join(unitDir, "felis-watchdog.service"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	w.diskPaths = "/nonexistent-felis-bundle-test"
+
+	secretEnv := corev1.EnvVar{Name: "FELIS_SMTP_PASSWORD", ValueFrom: &corev1.EnvVarSource{SecretKeyRef: &corev1.SecretKeySelector{
+		LocalObjectReference: corev1.LocalObjectReference{Name: "felis-smtp"}, Key: "password"}}}
+	replicas := int32(1)
+	cl := fake.NewClientBuilder().WithScheme(newSystemServerScheme(t)).WithObjects(
+		&corev1.Pod{
+			ObjectMeta: metav1.ObjectMeta{
+				Name: "felis-api-7d9", Namespace: "felis",
+				Annotations:   map[string]string{corev1.LastAppliedConfigAnnotation: `{"env":"` + p["last-applied annotation"] + `"}`, "felis.lolicon.best/kept": "yes"},
+				ManagedFields: []metav1.ManagedFieldsEntry{{Manager: "kubectl-client-side-apply"}},
+			},
+			Spec: corev1.PodSpec{
+				InitContainers: []corev1.Container{{Name: "wait-db", Image: "busybox", Env: []corev1.EnvVar{{Name: "FELIS_INIT_SETTING", Value: p["init container env literal"]}}}},
+				Containers: []corev1.Container{{Name: "api", Image: "felis-api:v1", Env: []corev1.EnvVar{
+					{Name: "FELIS_PLAIN_SETTING", Value: p["pod env literal"]}, secretEnv,
+				}}},
+				EphemeralContainers: []corev1.EphemeralContainer{{EphemeralContainerCommon: corev1.EphemeralContainerCommon{
+					Name: "debug", Env: []corev1.EnvVar{{Name: "FELIS_DEBUG_SETTING", Value: p["ephemeral container env"]}}}}},
+			},
+			Status: corev1.PodStatus{Phase: corev1.PodRunning, ContainerStatuses: []corev1.ContainerStatus{{Name: "api", RestartCount: 1, Ready: true}}},
+		},
+		&appsv1.Deployment{
+			ObjectMeta: metav1.ObjectMeta{Name: "felis-api", Namespace: "felis"},
+			Spec: appsv1.DeploymentSpec{Replicas: &replicas, Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"app": "felis-api"}},
+				Template: corev1.PodTemplateSpec{Spec: corev1.PodSpec{Containers: []corev1.Container{{Name: "api", Env: []corev1.EnvVar{
+					{Name: "FELIS_DEPLOY_SETTING", Value: p["deployment env literal"]},
+				}}}}}},
+		},
+		&corev1.Pod{
+			ObjectMeta: metav1.ObjectMeta{Name: "survival-0", Namespace: "minecraft"},
+			Spec: corev1.PodSpec{
+				InitContainers: []corev1.Container{{Name: "prepare-data"}, {Name: "egress-gate"}},
+				Containers:     []corev1.Container{{Name: "server"}},
+			},
+			Status: corev1.PodStatus{Phase: corev1.PodRunning},
+		},
+		&v1alpha1.MinecraftServer{
+			ObjectMeta: metav1.ObjectMeta{Name: "survival", Namespace: "minecraft"},
+			Spec:       v1alpha1.MinecraftServerSpec{Env: []v1alpha1.EnvVar{{Name: "DISCORD_WEBHOOK", Value: p["MinecraftServer env literal"]}}},
+		},
+		&appsv1.StatefulSet{
+			ObjectMeta: metav1.ObjectMeta{Name: "felis-postgres", Namespace: "felis"},
+			Spec:       appsv1.StatefulSetSpec{Template: envTemplate("FELIS_STS_SETTING", p["statefulset env literal"])},
+		},
+		&batchv1.Job{
+			ObjectMeta: metav1.ObjectMeta{Name: "build-1", Namespace: "felis-build"},
+			Spec:       batchv1.JobSpec{Template: envTemplate("FELIS_JOB_SETTING", p["job env literal"])},
+		},
+		&batchv1.CronJob{
+			ObjectMeta: metav1.ObjectMeta{Name: "felis-reaper", Namespace: "felis"},
+			Spec:       batchv1.CronJobSpec{JobTemplate: batchv1.JobTemplateSpec{Spec: batchv1.JobSpec{Template: envTemplate("FELIS_CRON_SETTING", p["cronjob env literal"])}}},
+		},
+		&corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "felis-1"}},
+	).Build()
+
+	secretLog := fmt.Sprintf("started with SERVICE_TOKEN=%s\nlogin password=%s ok\nAuthorization: Bearer %s\ndial postgres://felis:%s@db:5432/felis\n",
+		p["secrets.env SERVICE_TOKEN"], p["logged password"], p["logged bearer"], p["URL password"])
+	return &supportBundle{
+		host: "felis-test", now: time.Date(2026, 9, 27, 12, 0, 0, 0, time.UTC), unitDir: unitDir,
+		w: w, cfg: cfg, cl: cl,
+		logs: func(_ context.Context, ns, pod, container string, previous bool) ([]byte, error) {
+			if container == "wait-db" {
+				return nil, errors.New("container \"wait-db\" is waiting to start; token=" + p["token in an error"])
+			}
+			return []byte(fmt.Sprintf("log of %s/%s/%s previous=%v\n%s", ns, pod, container, previous, secretLog)), nil
+		},
+		run: func(_ context.Context, name string, args ...string) ([]byte, error) {
+			switch name {
+			case "journalctl":
+				return []byte("journal of " + args[1] + "\nFELIS_OFFSITE_KEY=" + p["offsite.env FELIS_OFFSITE_KEY"] + "\nrelay " + p["smtp-password"] + " refused\n" +
+					"relay login with the cached " + p["watchdog state relay password"] + "\ndatabase auth failed for " + p["felis.host.toml DB password"] +
+					"\nservice token " + p["service token by its env ref"] + " rejected\nnode joined with " + p["k3s token secret part"] + "\n" +
+					"old copies sealed with " + p["commented-out offsite key"] + "\nretrying with password=" + p["journal quoting a password"] + "\n"), nil
+			case "systemctl", "df", "ip":
+				return []byte(name + " output\n"), nil
+			}
+			return nil, errors.New("unexpected " + name)
+		},
+		backups: func(context.Context) (map[string]time.Time, error) {
+			return nil, errors.New("connect: password=" + p["status quoting a password"])
+		},
+		doctor: func(_ context.Context, out io.Writer) {
+			fmt.Fprintf(out, "doctor report; the k3s token K10deadbeefcafe::server:%s leaked here\nprobe said password=%s\n", p["k3s token secret part"], p["doctor quoting a password"])
+		},
+		secrets: secretSources{
+			envFiles:   []string{filepath.Join(etc, "secrets.env"), filepath.Join(etc, "offsite.env"), filepath.Join(etc, "absent.env")},
+			valueFiles: []string{filepath.Join(etc, "smtp-password"), filepath.Join(etc, "uploads-s3-secret-key")},
+			propsFiles: []string{filepath.Join(etc, "felis-link.properties")},
+			tokenFiles: []string{filepath.Join(dir, "k3s", "token")},
+		},
+		logLines: 500, since: 48 * time.Hour,
+		meminfo: meminfo, osRelease: filepath.Join(dir, "os-release"), procVersion: filepath.Join(dir, "version"), stateDir: etc,
+	}
+}
+
+// envTemplate is a pod template whose one container sets name to a literal.
+func envTemplate(name, value string) corev1.PodTemplateSpec {
+	return corev1.PodTemplateSpec{Spec: corev1.PodSpec{Containers: []corev1.Container{{Name: "main", Env: []corev1.EnvVar{{Name: name, Value: value}}}}}}
+}
+
+// readBundle is every file in the bundle at path, by its name inside the
+// bundle's directory, and each one's mode.
+func readBundle(t *testing.T, path string) (map[string]string, map[string]int64) {
+	t.Helper()
+	f, err := os.Open(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer f.Close()
+	gz, err := gzip.NewReader(f)
+	if err != nil {
+		t.Fatal(err)
+	}
+	tr := tar.NewReader(gz)
+	files, modes := map[string]string{}, map[string]int64{}
+	prefix := strings.TrimSuffix(filepath.Base(path), ".tar.gz") + "/"
+	for {
+		hdr, err := tr.Next()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			t.Fatal(err)
+		}
+		name, ok := strings.CutPrefix(hdr.Name, prefix)
+		if !ok {
+			t.Errorf("%s is outside the bundle's directory %s", hdr.Name, prefix)
+		}
+		raw, err := io.ReadAll(tr)
+		if err != nil {
+			t.Fatal(err)
+		}
+		files[name], modes[name] = string(raw), hdr.Mode
+	}
+	return files, modes
+}
+
+func TestSupportBundle(t *testing.T) {
+	b := bundleHost(t)
+	out := filepath.Join(t.TempDir(), "support")
+	path, err := b.write(context.Background(), out)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if want := filepath.Join(out, "felis-support-felis-test-20260927T120000Z.tar.gz"); path != want {
+		t.Errorf("path %s, want %s", path, want)
+	}
+	for p, want := range map[string]os.FileMode{out: 0o700 | os.ModeDir, path: 0o600} {
+		if st, err := os.Stat(p); err != nil || st.Mode() != want {
+			t.Errorf("%s: mode %v (err %v), want %v", p, st.Mode(), err, want)
+		}
+	}
+	if left, _ := filepath.Glob(filepath.Join(out, ".*partial")); len(left) != 0 {
+		t.Errorf("left behind %v", left)
+	}
+
+	files, modes := readBundle(t, path)
+	var names []string
+	for n := range files {
+		names = append(names, n)
+		if modes[n] != 0o600 {
+			t.Errorf("%s: mode %o in the archive, want 600", n, modes[n])
+		}
+	}
+	sort.Strings(names)
+	want := []string{
+		"MANIFEST.txt",
+		"cluster/felis-build/cronjobs.yaml", "cluster/felis-build/deployments.yaml", "cluster/felis-build/events.yaml", "cluster/felis-build/jobs.yaml",
+		"cluster/felis-build/networkpolicies.yaml", "cluster/felis-build/persistentvolumeclaims.yaml", "cluster/felis-build/pods.yaml",
+		"cluster/felis-build/services.yaml", "cluster/felis-build/statefulsets.yaml",
+		"cluster/felis/cronjobs.yaml", "cluster/felis/deployments.yaml", "cluster/felis/events.yaml", "cluster/felis/jobs.yaml",
+		"cluster/felis/networkpolicies.yaml", "cluster/felis/persistentvolumeclaims.yaml", "cluster/felis/pods.yaml",
+		"cluster/felis/services.yaml", "cluster/felis/statefulsets.yaml",
+		"cluster/minecraft/cronjobs.yaml", "cluster/minecraft/deployments.yaml", "cluster/minecraft/events.yaml", "cluster/minecraft/jobs.yaml",
+		"cluster/minecraft/networkpolicies.yaml", "cluster/minecraft/persistentvolumeclaims.yaml", "cluster/minecraft/pods.yaml",
+		"cluster/minecraft/services.yaml", "cluster/minecraft/statefulsets.yaml",
+		"cluster/minecraftservers.yaml", "cluster/nodes.yaml", "cluster/persistentvolumes.yaml", "cluster/pods-all-namespaces.txt",
+		"config.txt", "doctor.txt",
+		"host/addresses.txt", "host/df.txt", "host/etc-felis.txt", "host/meminfo.txt", "host/systemd-timers.txt", "host/systemd-units.txt",
+		"journal/felis-offsite.log", "journal/felis-watchdog.log", "journal/k3s.log",
+		"logs/felis/felis-api-7d9/api.log", "logs/felis/felis-api-7d9/api.previous.log",
+		"logs/minecraft/survival-0/egress-gate.log", "logs/minecraft/survival-0/prepare-data.log",
+		"status.txt", "version.txt",
+	}
+	if strings.Join(names, "\n") != strings.Join(want, "\n") {
+		t.Errorf("bundle holds\n  %s\nwant\n  %s", strings.Join(names, "\n  "), strings.Join(want, "\n  "))
+	}
+
+	for what, secret := range plantedSecrets {
+		for n, body := range files {
+			if strings.Contains(body, secret) {
+				t.Errorf("%s (%s) is in %s:\n%s", what, secret, n, body)
+			}
+		}
+	}
+
+	// What was taken out leaves what a reader needs around it.
+	for name, wants := range map[string][]string{
+		"logs/felis/felis-api-7d9/api.previous.log": {"log of felis/felis-api-7d9/api previous=true\n", "SERVICE_TOKEN=<redacted>\n", "login password=<redacted> ok\n", "Authorization: Bearer <redacted>\n", "postgres://felis:<redacted>@db:5432/felis\n"},
+		"journal/k3s.log": {"journal of k3s.service\n", "FELIS_OFFSITE_KEY=<redacted>\n", "relay <redacted> refused\n",
+			"relay login with the cached <redacted>\n", "database auth failed for <redacted>\n", "service token <redacted> rejected\n", "node joined with <redacted>\n"},
+		"cluster/felis/statefulsets.yaml": {"name: FELIS_STS_SETTING\n"},
+		"cluster/felis/cronjobs.yaml":     {"name: FELIS_CRON_SETTING\n"},
+		"cluster/felis-build/jobs.yaml":   {"name: FELIS_JOB_SETTING\n"},
+		"cluster/felis/pods.yaml":         {"name: FELIS_PLAIN_SETTING\n        value: <redacted>\n", "name: FELIS_SMTP_PASSWORD\n        valueFrom:\n          secretKeyRef:\n            key: password\n            name: felis-smtp\n", "felis.lolicon.best/kept: \"yes\"", "name: FELIS_INIT_SETTING\n", "name: FELIS_DEBUG_SETTING\n"},
+		"cluster/felis/deployments.yaml":  {"name: FELIS_DEPLOY_SETTING\n            value: <redacted>\n"},
+		"cluster/minecraftservers.yaml":   {"name: DISCORD_WEBHOOK\n      value: <redacted>\n"},
+		"config.txt":                      {fmt.Sprintf("%-34s %s\n", "server.root_domain", "games.example.org"), fmt.Sprintf("%-34s %s\n", "database.url (no password)", "postgres://[email protected]:1/felis")},
+		"doctor.txt":                      {"the k3s token <redacted> leaked here\nprobe said password=<redacted>\n"},
+		"status.txt":                      {"  world backups unknown: connect: password=<redacted>\n"},
+		"host/etc-felis.txt":              {"secrets.env\n", "smtp-password\n", "felis-link.properties\n"},
+		"MANIFEST.txt": {
+			"The game servers' own logs are left out",
+			"  journal/                 up to 500 lines per Felis unit and k3s, from the last 48h\n",
+			"  - 11 secret values, wherever they appear, read from:\n",
+			"secrets.env\n", "offsite.env\n", "smtp-password\n", "felis-link.properties\n", "token\n",
+			"logs/felis/felis-api-7d9/wait-db.log: container \"wait-db\" is waiting to start; token=<redacted>\n",
+			// Its own words about what is redacted come through whole.
+			"  - passwords in URLs, private keys, Bearer and Basic credentials\n",
+			"access_key=, private_key= or credentials= (and the same with a colon)\n",
+			"Read it through before you send it anywhere",
+		},
+	} {
+		for _, w := range wants {
+			if !strings.Contains(files[name], w) {
+				t.Errorf("%s lacks %q:\n%s", name, w, files[name])
+			}
+		}
+	}
+	for _, gone := range []string{"managedFields", "last-applied-configuration"} {
+		if strings.Contains(files["cluster/felis/pods.yaml"], gone) {
+			t.Errorf("pods.yaml keeps %s:\n%s", gone, files["cluster/felis/pods.yaml"])
+		}
+	}
+	if strings.Contains(files["MANIFEST.txt"], "absent.env") || strings.Contains(files["MANIFEST.txt"], "uploads-s3-secret-key") {
+		t.Errorf("MANIFEST names files this host does not have:\n%s", files["MANIFEST.txt"])
+	}
+}
+
+// -server-logs adds the game servers' own logs, and says so.
+func TestSupportBundleServerLogs(t *testing.T) {
+	b := bundleHost(t)
+	b.serverLogs = true
+	path, err := b.write(context.Background(), t.TempDir())
+	if err != nil {
+		t.Fatal(err)
+	}
+	files, _ := readBundle(t, path)
+	if _, ok := files["logs/minecraft/survival-0/server.log"]; !ok {
+		t.Error("no server.log with -server-logs")
+	}
+	if !strings.Contains(files["MANIFEST.txt"], "The game servers' own logs are in logs/ (-server-logs)") {
+		t.Errorf("MANIFEST does not say server logs are in:\n%s", files["MANIFEST.txt"])
+	}
+}
+
+// With the cluster and the configuration gone the bundle still holds what the
+// host shows, and MANIFEST says what it could not collect.
+func TestSupportBundleWithTheClusterDown(t *testing.T) {
+	b := bundleHost(t)
+	b.cl, b.clErr = nil, errors.New("connection refused")
+	b.cfg, b.cfgErr = nil, errors.New("felis.host.toml: no such file")
+	b.wErr = errors.New("felis-watchdog.service is not installed; read the watchdog's defaults")
+	path, err := b.write(context.Background(), t.TempDir())
+	if err != nil {
+		t.Fatal(err)
+	}
+	files, _ := readBundle(t, path)
+	for _, n := range []string{"status.txt", "doctor.txt", "host/etc-felis.txt", "journal/k3s.log"} {
+		if _, ok := files[n]; !ok {
+			t.Errorf("no %s", n)
+		}
+	}
+	for n := range files {
+		if strings.HasPrefix(n, "cluster/") || strings.HasPrefix(n, "logs/") || n == "config.txt" {
+			t.Errorf("%s with no cluster and no configuration", n)
+		}
+	}
+	if !strings.Contains(files["status.txt"], "felis status: the configuration did not load: felis.host.toml: no such file") {
+		t.Errorf("status.txt:\n%s", files["status.txt"])
+	}
+	if !strings.Contains(files["MANIFEST.txt"], "\nThe watchdog's settings: felis-watchdog.service is not installed; read the watchdog's defaults\n") ||
+		!strings.Contains(files["MANIFEST.txt"], "  - cluster: connection refused\n") {
+		t.Errorf("MANIFEST.txt:\n%s", files["MANIFEST.txt"])
+	}
+}
+
+func TestShortDuration(t *testing.T) {
+	for d, want := range map[time.Duration]string{
+		48 * time.Hour:             "48h",
+		90 * time.Minute:           "1h30m",
+		45 * time.Minute:           "45m",
+		30 * time.Second:           "30s",
+		time.Hour + 30*time.Second: "1h0m30s",
+	} {
+		if got := shortDuration(d); got != want {
+			t.Errorf("shortDuration(%v) = %q, want %q", d, got, want)
+		}
+	}
+}
+
+func TestScrubber(t *testing.T) {
+	s := &scrubber{}
+	for _, v := range []string{"known-secret-value", "known-secret-value-longer", "short", "known-secret-value"} {
+		s.add(v)
+	}
+	if got := strings.Join(s.vals, ","); got != "known-secret-value-longer,known-secret-value" {
+		t.Errorf("kept %q; want each value once, longest first, none under %d characters", s.vals, minScrubLen)
+	}
+	for in, want := range map[string]string{
+		"a known-secret-value-longer b":                                     "a <redacted> b",
+		"a known-secret-value b":                                            "a <redacted> b",
+		"password=abcd1234 next":                                            "password=<redacted> next",
+		`{"token": "abcd1234"}`:                                             `{"token": "<redacted>"}`,
+		"SMTP_PASSWORD: s3cr3t!x":                                           "SMTP_PASSWORD: <redacted>",
+		"x-api-key=zzzz9999&q=1":                                            "x-api-key=<redacted>&q=1",
+		"Authorization: Basic dXNlcjpwYXNzd29yZA==":                         "Authorization: Basic <redacted>",
+		"s3://AKIA:secretpart123@bucket/key":                                "s3://AKIA:<redacted>@bucket/key",
+		"https://user@host/path":                                            "https://user@host/path",
+		"-----BEGIN EC PRIVATE KEY-----\nMHc\n-----END EC PRIVATE KEY-----": "<redacted private key>",
+		"tokens: 3": "tokens: 3",
+		"read the relay password (keeping the cached one)": "read the relay password (keeping the cached one)",
+		`secrets "felis-smtp" not found`:                   `secrets "felis-smtp" not found`,
+	} {
+		if got := string(s.text([]byte(in))); got != want {
+			t.Errorf("text(%q) = %q, want %q", in, got, want)
+		}
+	}
+	// Structured files keep what only looks like a secret.
+	if got := string(s.values([]byte("secretName: felis-forwarding-secret and known-secret-value"))); got != "secretName: felis-forwarding-secret and <redacted>" {
+		t.Errorf("values() = %q", got)
+	}
+}

cmd/felis/systemservers.go

@@ -15,6 +15,7 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 	clientgoscheme "k8s.io/client-go/kubernetes/scheme"
+	"k8s.io/client-go/rest"
 	"k8s.io/client-go/tools/clientcmd"
 	"k8s.io/client-go/util/retry"
 	ctrl "sigs.k8s.io/controller-runtime"
@@ -243,6 +244,15 @@ func buildSystemServerClient() (client.Client, error) {
 	if err := v1alpha1.AddToScheme(scheme); err != nil {
 		return nil, err
 	}
+	cfg, err := hostRESTConfig()
+	if err != nil {
+		return nil, err
+	}
+	return client.New(cfg, client.Options{Scheme: scheme})
+}
+
+// hostRESTConfig is the cluster connection buildSystemServerClient describes.
+func hostRESTConfig() (*rest.Config, error) {
 	cfg, err := ctrl.GetConfig()
 	if err != nil {
 		cfg, err = clientcmd.BuildConfigFromFlags("", hostBootstrapKubeconfigPath)
@@ -250,7 +260,7 @@ func buildSystemServerClient() (client.Client, error) {
 			return nil, fmt.Errorf("no reachable kubeconfig (tried in-cluster/$KUBECONFIG/~/.kube and %s): %w", hostBootstrapKubeconfigPath, err)
 		}
 	}
-	return client.New(cfg, client.Options{Scheme: scheme})
+	return cfg, nil
 }
 
 // systemServerOutcome records what ensureSystemServers did with one service so

cmd/felis/watchdog.go

@@ -35,22 +35,8 @@ const proxyFor = 3 * time.Minute
 func cmdWatchdog(args []string, stdout, stderr io.Writer) int {
 	fs := flag.NewFlagSet("watchdog", flag.ContinueOnError)
 	fs.SetOutput(stderr)
-	cfgPath := fs.String("config", "/etc/felis/felis.toml", "path to felis.toml (the host copy, which reaches PostgreSQL on 127.0.0.1)")
+	var w watchdogFlags
-	statePath := fs.String("state", "/var/lib/felis/watchdog/state.json", "state kept between runs (root only: it caches the relay password)")
+	w.register(fs)
-	fallbackState := fs.String("fallback-state", watchdog.FallbackStatePath, "where a run keeps its state while -state cannot be written, so what it mailed is not mailed again (tmpfs: until the host restarts; \"\" keeps none)")
-	smtpPasswordFile := fs.String("smtp-password-file", hostSMTPPasswordPath, "the relay password `felis setup` keeps on the host; the felis-smtp Secret stands in while it is missing")
-	quietPath := fs.String("quiet-file", "/run/felis/watchdog-quiet-until", "Unix time before which nothing is mailed; the installer writes it while it restarts things on purpose")
-	backupDir := fs.String("backup-dir", "/var/lib/felis/db-backups", `control-plane database backups to check for freshness ("" skips the check)`)
-	diskPaths := fs.String("disk-paths", "/,/var/lib/rancher/k3s,/var/lib/felis", "comma-separated paths whose filesystems must keep free space")
-	certDirs := fs.String("k3s-cert-dirs", strings.Join(watchdog.K3sCertDirs, ","), `k3s certificate directories whose *.crt files must not be near expiry ("" skips the check)`)
-	proxyAddr := fs.String("proxy-addr", "", `game proxy address to dial, e.g. 127.0.0.1:25565 ("" skips the check)`)
-	nodeIP := fs.String("node-ip", "", `the node address the install was made on, which must stay on this host ("" skips the check)`)
-	controlNS := fs.String("control-namespace", platform.DefaultControlNamespace, "namespace of the control plane")
-	offsiteStatus := fs.String("offsite-status", offsite.DefaultStatusFile, "the record `felis offsite sync` leaves, checked when [offsite] is configured")
-	toolsStatus := fs.String("build-tools-status", defaultBuildToolsStatus, "the record `felis mirror-build-tools` leaves, checked when builds scan against the registry's DB copy")
-	dryRun := fs.Bool("dry-run", false, "print every finding and the mail that is due; send nothing and keep the state as it was")
-	heartbeatFile := fs.String("heartbeat-file", defaultHeartbeatFile, "file holding the heartbeat URL each run pings, a dead man's switch at a monitoring service that alerts when the pings stop (no file pings nothing)")
-	unitFailed := fs.Bool("unit-failed", false, "report a failed run of felis-watchdog.service instead of checking; felis-watchdog-failed.service runs this through OnFailure=")
 	if err := fs.Parse(args); err != nil {
 		if errors.Is(err, flag.ErrHelp) {
 			return 0
@@ -58,20 +44,20 @@ func cmdWatchdog(args []string, stdout, stderr io.Writer) int {
 		return 2
 	}
 	now := time.Now()
-	if *unitFailed {
+	if w.unitFailed {
 		return watchdogUnitFailed(unitFailedRun{
-			cfgPath: *cfgPath, statePath: *statePath, fallbackPath: *fallbackState, quietPath: *quietPath,
+			cfgPath: w.cfgPath, statePath: w.statePath, fallbackPath: w.fallbackState, quietPath: w.quietPath,
-			offsiteStatus: *offsiteStatus, heartbeatFile: *heartbeatFile,
+			offsiteStatus: w.offsiteStatus, heartbeatFile: w.heartbeatFile,
 			result: os.Getenv("MONITOR_SERVICE_RESULT"), exitStatus: os.Getenv("MONITOR_EXIT_STATUS"),
 			send: watchdogSender, client: http.DefaultClient, now: now,
 		}, stdout, stderr)
 	}
-	cfg, err := config.Load(*cfgPath)
+	cfg, err := config.Load(w.cfgPath)
 	if err != nil {
 		fmt.Fprintf(stderr, "felis watchdog: %v\n", err)
 		return 1
 	}
-	loadPath := watchdog.NewestState(*statePath, *fallbackState)
+	loadPath := watchdog.NewestState(w.statePath, w.fallbackState)
 	state, aside, err := watchdog.RecoverState(loadPath, now)
 	if err != nil {
 		fmt.Fprintf(stderr, "felis watchdog: %v\n", err)
@@ -81,72 +67,19 @@ func cmdWatchdog(args []string, stdout, stderr io.Writer) int {
 	defer cancel()
 
 	var report watchdog.Report
-	add := func(f *watchdog.Finding) {
-		if f != nil {
-			report.Findings = append(report.Findings, *f)
-		}
-	}
 	if aside != "" {
 		fmt.Fprintf(stderr, "felis watchdog: %s was unreadable; moved it to %s and started over\n", loadPath, aside)
-		f := watchdog.StateSetAside(aside)
+		report.Findings = append(report.Findings, watchdog.StateSetAside(aside))
-		add(&f)
-	}
-
-	// The cluster: one unreachable API server stands in for every check behind it.
-	minecraftNS := cfg.K8s.Namespace
-	if minecraftNS == "" {
-		minecraftNS = platform.DefaultMinecraftNamespace
-	}
-	cl, err := buildSystemServerClient()
-	var found []watchdog.Finding
-	if err == nil {
-		found, err = watchdog.Cluster{Client: cl, ControlNamespace: *controlNS, MinecraftNamespace: minecraftNS}.Check(ctx, now)
-	}
-	if err != nil {
-		f := watchdog.KubeAPIDown(err)
-		add(&f)
-		report.Unknown = append(report.Unknown, watchdog.ClusterPrefixes...)
-	} else {
-		report.Findings = append(report.Findings, found...)
 	}
+	cl, owners, ownersErr := watchdogProbes(ctx, w, cfg, now, &report)
 	if cfg.SMTP.Host != "" {
-		var secrets client.Client
+		refreshSMTPPassword(ctx, w.smtpPasswordFile, cl, w.controlNS, state, stderr)
-		if err == nil {
-			secrets = cl
-		}
-		refreshSMTPPassword(ctx, *smtpPasswordFile, secrets, *controlNS, state, stderr)
 	}
 	state.Relay = cachedRelay(cfg.SMTP)
-
+	if ownersErr == nil {
-	if recipients, err := ownerEmails(ctx, cfg.Database.URL); err != nil {
+		state.Recipients = owners
-		f := watchdog.PostgresDown(err)
-		add(&f)
-	} else {
-		state.Recipients = recipients
 	}
 
-	if *proxyAddr != "" {
-		add(proxyFinding(ctx, *proxyAddr))
-	}
-	if *backupDir != "" {
-		add(watchdog.BackupFinding(*backupDir, now))
-	}
-	if cfg.Offsite.Enabled() {
-		add(watchdog.OffsiteFinding(*offsiteStatus, now))
-	}
-	if usesMirroredScanDB(cfg) {
-		add(watchdog.ScanDBFinding(*toolsStatus, now))
-	}
-	report.Findings = append(report.Findings, watchdog.DiskFindings(splitList(*diskPaths))...)
-	add(watchdog.MemoryFinding("/proc/meminfo"))
-	add(watchdog.CertFinding(splitList(*certDirs), now))
-	if *nodeIP != "" {
-		if held, err := watchdog.HostAddresses(); err == nil {
-			add(watchdog.AddressFinding(*nodeIP, held))
-		}
-	}
-	add(watchdog.ClockFinding(watchdog.ClockStatus()))
-
 	if len(report.Findings) == 0 {
 		fmt.Fprintln(stdout, "felis watchdog: every check passed")
 	}
@@ -158,13 +91,13 @@ func cmdWatchdog(args []string, stdout, stderr io.Writer) int {
 	host, _ := os.Hostname()
 	subject, body := plan.Message(host, now)
 	beat := heartbeat{
-		standby: standsBy(cfg.Offsite.Enabled(), *offsiteStatus),
+		standby: standsBy(cfg.Offsite.Enabled(), w.offsiteStatus),
-		quiet:   now.Before(watchdog.QuietUntil(*quietPath)),
+		quiet:   now.Before(watchdog.QuietUntil(w.quietPath)),
 	}
-	if beat.url, err = readHeartbeatURL(*heartbeatFile); err != nil {
+	if beat.url, err = readHeartbeatURL(w.heartbeatFile); err != nil {
 		fmt.Fprintf(stderr, "felis watchdog: %v; pinging no heartbeat\n", err)
 	}
-	if *dryRun {
+	if w.dryRun {
 		if plan.Empty() {
 			fmt.Fprintln(stdout, "felis watchdog: nothing is due to be mailed")
 		} else {
@@ -177,8 +110,8 @@ func cmdWatchdog(args []string, stdout, stderr io.Writer) int {
 	}
 
 	m := configMailer(cfg.SMTP, state.SMTPPassword, watchdogSender)
-	unheard, mailFailed := m.deliver(ctx, state, plan, subject, body, mailHold(*quietPath, cfg.Offsite.Enabled(), *offsiteStatus, now), now, stdout, stderr)
+	unheard, mailFailed := m.deliver(ctx, state, plan, subject, body, mailHold(w.quietPath, cfg.Offsite.Enabled(), w.offsiteStatus, now), now, stdout, stderr)
-	saveErr := watchdog.SaveStateOr(*statePath, *fallbackState, state)
+	saveErr := watchdog.SaveStateOr(w.statePath, w.fallbackState, state)
 	if saveErr != nil {
 		fmt.Fprintf(stderr, "felis watchdog: save state: %v\n", saveErr)
 	}
@@ -191,6 +124,93 @@ func cmdWatchdog(args []string, stdout, stderr io.Writer) int {
 	return 0
 }
 
+// watchdogFlags are felis watchdog's flags. felis doctor reads them back from
+// the ExecStart= line of felis-watchdog.service, so it checks what the timer's
+// runs check, with the same paths.
+type watchdogFlags struct {
+	cfgPath, statePath, fallbackState, smtpPasswordFile, quietPath string
+	backupDir, diskPaths, certDirs, proxyAddr, nodeIP, controlNS   string
+	offsiteStatus, toolsStatus, heartbeatFile                      string
+	dryRun, unitFailed                                             bool
+}
+
+func (w *watchdogFlags) register(fs *flag.FlagSet) {
+	fs.StringVar(&w.cfgPath, "config", "/etc/felis/felis.toml", "path to felis.toml (the host copy, which reaches PostgreSQL on 127.0.0.1)")
+	fs.StringVar(&w.statePath, "state", "/var/lib/felis/watchdog/state.json", "state kept between runs (root only: it caches the relay password)")
+	fs.StringVar(&w.fallbackState, "fallback-state", watchdog.FallbackStatePath, "where a run keeps its state while -state cannot be written, so what it mailed is not mailed again (tmpfs: until the host restarts; \"\" keeps none)")
+	fs.StringVar(&w.smtpPasswordFile, "smtp-password-file", hostSMTPPasswordPath, "the relay password `felis setup` keeps on the host; the felis-smtp Secret stands in while it is missing")
+	fs.StringVar(&w.quietPath, "quiet-file", "/run/felis/watchdog-quiet-until", "Unix time before which nothing is mailed; the installer writes it while it restarts things on purpose")
+	fs.StringVar(&w.backupDir, "backup-dir", "/var/lib/felis/db-backups", `control-plane database backups to check for freshness ("" skips the check)`)
+	fs.StringVar(&w.diskPaths, "disk-paths", "/,/var/lib/rancher/k3s,/var/lib/felis", "comma-separated paths whose filesystems must keep free space")
+	fs.StringVar(&w.certDirs, "k3s-cert-dirs", strings.Join(watchdog.K3sCertDirs, ","), `k3s certificate directories whose *.crt files must not be near expiry ("" skips the check)`)
+	fs.StringVar(&w.proxyAddr, "proxy-addr", "", `game proxy address to dial, e.g. 127.0.0.1:25565 ("" skips the check)`)
+	fs.StringVar(&w.nodeIP, "node-ip", "", `the node address the install was made on, which must stay on this host ("" skips the check)`)
+	fs.StringVar(&w.controlNS, "control-namespace", platform.DefaultControlNamespace, "namespace of the control plane")
+	fs.StringVar(&w.offsiteStatus, "offsite-status", offsite.DefaultStatusFile, "the record `felis offsite sync` leaves, checked when [offsite] is configured")
+	fs.StringVar(&w.toolsStatus, "build-tools-status", defaultBuildToolsStatus, "the record `felis mirror-build-tools` leaves, checked when builds scan against the registry's DB copy")
+	fs.BoolVar(&w.dryRun, "dry-run", false, "print every finding and the mail that is due; send nothing and keep the state as it was")
+	fs.StringVar(&w.heartbeatFile, "heartbeat-file", defaultHeartbeatFile, "file holding the heartbeat URL each run pings, a dead man's switch at a monitoring service that alerts when the pings stop (no file pings nothing)")
+	fs.BoolVar(&w.unitFailed, "unit-failed", false, "report a failed run of felis-watchdog.service instead of checking; felis-watchdog-failed.service runs this through OnFailure=")
+}
+
+// watchdogProbes is one pass of every check, appended to report. cl is the
+// cluster client, nil while the API server is unreachable; owners is who the
+// alerts go to, and ownersErr why PostgreSQL did not say.
+func watchdogProbes(ctx context.Context, w watchdogFlags, cfg *config.Config, now time.Time, report *watchdog.Report) (cl client.Client, owners []string, ownersErr error) {
+	add := func(f *watchdog.Finding) {
+		if f != nil {
+			report.Findings = append(report.Findings, *f)
+		}
+	}
+
+	// The cluster: one unreachable API server stands in for every check behind it.
+	minecraftNS := cfg.K8s.Namespace
+	if minecraftNS == "" {
+		minecraftNS = platform.DefaultMinecraftNamespace
+	}
+	cl, err := buildSystemServerClient()
+	var found []watchdog.Finding
+	if err == nil {
+		found, err = watchdog.Cluster{Client: cl, ControlNamespace: w.controlNS, MinecraftNamespace: minecraftNS}.Check(ctx, now)
+	}
+	if err != nil {
+		cl = nil
+		f := watchdog.KubeAPIDown(err)
+		add(&f)
+		report.Unknown = append(report.Unknown, watchdog.ClusterPrefixes...)
+	} else {
+		report.Findings = append(report.Findings, found...)
+	}
+
+	if owners, ownersErr = ownerEmails(ctx, cfg.Database.URL); ownersErr != nil {
+		f := watchdog.PostgresDown(ownersErr)
+		add(&f)
+	}
+
+	if w.proxyAddr != "" {
+		add(proxyFinding(ctx, w.proxyAddr))
+	}
+	if w.backupDir != "" {
+		add(watchdog.BackupFinding(w.backupDir, now))
+	}
+	if cfg.Offsite.Enabled() {
+		add(watchdog.OffsiteFinding(w.offsiteStatus, now))
+	}
+	if usesMirroredScanDB(cfg) {
+		add(watchdog.ScanDBFinding(w.toolsStatus, now))
+	}
+	report.Findings = append(report.Findings, watchdog.DiskFindings(splitList(w.diskPaths))...)
+	add(watchdog.MemoryFinding("/proc/meminfo"))
+	add(watchdog.CertFinding(splitList(w.certDirs), now))
+	if w.nodeIP != "" {
+		if held, err := watchdog.HostAddresses(); err == nil {
+			add(watchdog.AddressFinding(w.nodeIP, held))
+		}
+	}
+	add(watchdog.ClockFinding(watchdog.ClockStatus()))
+	return cl, owners, ownersErr
+}
+
 // failureReport is what the heartbeat's failure ping carries, "" when the run
 // pings success: the alerts this run knows of reach no one (a mail that
 // failed, or no relay or recipient while something is open), or the state did

docs/troubleshooting.md

@@ -37,6 +37,85 @@ concrete host.
 
 ---
 
+## 0. First look: `felis status`, `felis doctor`, `felis support-bundle`
+
+Three read-only commands, all run as root on the node, give the state of the
+whole host before any of the sections below. They change nothing and mail
+nothing, so they are safe to run at any time.
+
+**`sudo felis status`** prints the platform at a glance: the release, the node
+and its kubelet, each control-plane Deployment (ready replicas, image, pod
+restarts), the game proxy (the `felis-velocity` unit and whether the game port
+accepts connections), every server with its desired state, phase, players and
+newest world backup, the newest database bundle and the off-site copy, the
+watched disks and memory, and the alerts the watchdog has open. A part that is
+down reads as such and the rest still prints: with k3s stopped the cluster
+line says `unreachable (...)` and the servers `unknown while the cluster is
+unreachable`; with PostgreSQL down the backup column reads `?` and a line under
+the table says why. [GO-TESTED: `TestStatusReport`, `TestStatusWithPartsDown`,
+`TestStatusClusterEdges`, `TestStatusBackups`, `TestStatusWatchdog`]
+
+**`sudo felis doctor`** runs every check `felis watchdog` runs, with the
+settings `felis-watchdog.service` gives it, plus what only the host shows: a
+Felis unit that failed or a long-running one (`k3s`, `felis-velocity`) that
+stopped, a timer that no longer fires, alerts that reach no one (no `[smtp]`
+relay, no owner with a verified address), an unusable heartbeat URL. It prints
+one line per area, `✓` fine, `!` warnings, `✗` something critical, `-` not
+checked and why (the off-site copy on an install without one), each finding
+with where to look next:
+
+```text
+felis doctor on felis-1 at 2026-09-27 12:00 UTC
+checks run as /etc/systemd/system/felis-watchdog.service runs them (config /etc/felis/felis.host.toml)
+
+✓  configuration
+✓  Kubernetes cluster
+✗  PostgreSQL
+     critical postgres: PostgreSQL is unreachable: sign-in, the panel and server management fail
+              → k3s kubectl -n felis get pods -l app.kubernetes.io/component=postgres; k3s kubectl -n felis logs deploy/felis-postgres --tail=100 (...)
+-  off-site copy: not checked, not configured
+...
+1 problem(s): 1 critical, 0 warning(s)
+```
+
+It exits 1 when it found anything and 0 otherwise, so a script can run it. It
+never mails, pings the heartbeat or touches the watchdog's state: an alert it
+shows is mailed by the watchdog's own next run, on the watchdog's delays (§14).
+A missing or unreadable `felis-watchdog.service` is itself a critical finding,
+and the checks then run with the watchdog's defaults. [GO-TESTED:
+`TestDoctorReportsByArea`, `TestDoctorMailsPingsAndSavesNothing`,
+`TestDoctorWithoutUnitOrConfig`, `TestUnitFindings`, `TestPrintDoctorReport`]
+
+**`sudo felis support-bundle`** collects what someone helping needs into one
+file, `/var/lib/felis/support/felis-support-<host>-<time>.tar.gz` (mode 0600;
+`-o DIR` writes elsewhere): `status.txt` and `doctor.txt`, the release, a
+summary of the configuration (names and settings, no credentials), the Felis
+units and timers, `df`, addresses and memory, the names of the files under
+`/etc/felis` (no contents), each Felis unit's and k3s's journal (`-since 48h`,
+`-log-lines 2000`), the nodes, volumes and servers, and the pods, Deployments,
+StatefulSets, Jobs, CronJobs, Services, claims, network policies and events of
+the control-plane, build and server namespaces, and the tail of every
+control-plane and build container's log (and of its previous run after a
+restart). Game servers contribute their init containers' logs only; their own
+logs carry player names, IP addresses and chat, and `-server-logs` adds them.
+
+What it leaves out: every Secret and ConfigMap, the contents of every
+configuration file, the database and every world. What it takes out of what it
+keeps: every value in the host's secret files (`secrets.env`, `offsite.env`,
+the relay and upload keys, the heartbeat URL, the forwarding secret, the
+proxy's token, the k3s tokens), the database password, every environment value
+in a pod spec or a server, passwords in URLs, private keys, `Bearer`/`Basic`
+credentials, and anything a log or report writes as `password=`, `token=`,
+`secret=` and the like. `MANIFEST.txt` inside lists what the bundle holds, what
+was taken out and what could not be collected (k3s down, a log that is gone).
+**Read it through before sending the bundle anywhere**: a secret logged in a
+form none of these rules knows stays in. [GO-TESTED: `TestSupportBundle`
+plants 25 secrets across every source and finds none in the bundle,
+`TestSupportBundleWithTheClusterDown`, `TestSupportBundleServerLogs`,
+`TestScrubber`]
+
+---
+
 ## 1. Server is stuck in `Starting` and never becomes `Running`
 
 `MinecraftServer.status.phase` stays `Starting`. A start that never succeeds is
@@ -3088,6 +3167,7 @@ end, has not been run on a cluster.]
 
 | Symptom | Section |
 |---|---|
+| Where to start: what is up, what is wrong, what to send when asking for help | §0 |
 | Stuck `Starting`, never `Running` | §1 |
 | `Starting` with `PodNotReady` (image? PVC? boot?) | §1a |
 | RCON secret/auth/port errors | §1b, §1c |