diff --git a/README.md b/README.md index bacd6a3..1bb4723 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ A Kubernetes-driven Minecraft server hosting platform — one command to deploy, - **Web 控制面板**：浏览器中查看服务器状态、在线玩家与资源用量，管理备份与恢复。 - **备份与恢复**：一键把整服数据（世界、配置、插件/模组，即整个 /data 卷）打包进集群内的归档库，支持从任意备份点回滚；默认安装就已启用（归档 PVC 与路径由安装器一并生成）。 - **控制面数据库备份**：账号、服务器归属、配额与存档索引所在的数据库每天自动备份，每次升级迁移前先快照，出错可用 `felis db restore` 整库原子回滚；面板「维护与备份」页显示备份是否新鲜（见 [故障排查 §16](docs/troubleshooting.md)）。 +- **运维自检**：`sudo felis status` 一屏列出节点、控制面、游戏代理、每台服务器、备份与未解决的告警；`sudo felis doctor` 把健康检查全跑一遍，按区域给出问题和下一步去哪看，不发邮件；`sudo felis support-bundle` 打出一个脱敏的诊断包，求助时直接附上（见 [故障排查 §0](docs/troubleshooting.md)）。 - **智慧回收（可选开启）**：超过 15 天无人游玩的世界自动备份后删除，释放磁盘空间；安装时设置 `FELIS_WORLDS_HOST_PATH`（k3s 默认 `/var/lib/rancher/k3s/storage`）即启用每日回收，不设置则不删任何世界。过期备份无论是否开启都会每天清理。 - **多核心支持**：兼容 Paper、Fabric、Forge、NeoForge，经由 Velocity 代理统一入口。 - **模组自助提交**：玩家自行上传模组包，服主审批通过后自动构建；构建产物进入镜像白名单，可直接选用为服务器镜像完成部署。 diff --git a/README_EN.md b/README_EN.md index a51df1c..7bd9752 100644 --- a/README_EN.md +++ b/README_EN.md @@ -19,6 +19,7 @@ Table of Contents - **Web Dashboard**: Monitor server status, online players, and resource usage from your browser, with backup and restore management. - **Backup & Restore**: One-click snapshots of a server's whole data volume (worlds, config, plugins/mods — the entire /data volume) into the cluster's archive store, with rollback from any backup point — enabled by default (the installer renders the archive PVC and its path). - **Control-plane database backups**: The database holding accounts, server ownership, quotas and the archive index is backed up daily and snapshotted before every upgrade migrates it; `felis db restore` rolls it back atomically, and the panel's Maintenance & Backups page shows whether the newest backup is fresh (see [troubleshooting §16](docs/troubleshooting.md)). +- **Self-check for operators**: `sudo felis status` shows the node, the control plane, the game proxy, every server, the backups and the open alerts on one screen; `sudo felis doctor` runs every health check once and lists each problem by area with where to look next, mailing nothing; `sudo felis support-bundle` writes one redacted diagnostics archive to attach when asking for help (see [troubleshooting §0](docs/troubleshooting.md)). - **World Reaper** (opt in): Worlds idle for more than 15 days are automatically backed up and removed to free disk space. Enable it by setting `FELIS_WORLDS_HOST_PATH` at install time (on k3s: `/var/lib/rancher/k3s/storage`); without it, no world is ever deleted. - **Multi-core Support**: Compatible with Paper, Fabric, Forge, and NeoForge, federated behind a Velocity proxy. - **Modpack Submission**: Players submit custom modpacks; admin approval triggers an automatic build, and the result is whitelisted as a server image you can select to deploy. diff --git a/cmd/felis/doctor.go b/cmd/felis/doctor.go new file mode 100644 index 0000000..f0e0329 --- /dev/null +++ b/cmd/felis/doctor.go @@ -0,0 +1,387 @@ +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "slices" + "strings" + "time" + + "felis.lolicon.best/internal/config" + "felis.lolicon.best/internal/watchdog" +) + +// systemdUnitDir is where the installer writes its units. +const systemdUnitDir = "/etc/systemd/system" + +// hostCommand runs a host tool (systemctl, journalctl, k3s) and returns its +// stdout. A tool that exits non-zero still returns what it printed: +// `systemctl is-active` prints "inactive" and exits 3. +func hostCommand(ctx context.Context, name string, args ...string) ([]byte, error) { + return exec.CommandContext(ctx, name, args...).Output() +} + +// hostServices are the long-running units a full install depends on, checked +// when their unit file is present: k3s runs the cluster, felis-velocity is the +// game proxy, felis-nano the single-binary host that runs without k3s. +var hostServices = []string{"k3s.service", "felis-velocity.service", "felis-nano.service"} + +// cmdDoctor runs every check felis watchdog runs, with the settings +// felis-watchdog.service gives it, plus what only the host shows (systemd +// units that failed or stopped, timers that no longer fire, alerts that reach +// no one), and prints them grouped by area with where to look next. It mails +// nothing, pings no heartbeat and leaves the watchdog's state alone: it is +// safe to run at any time, as often as wanted. +func cmdDoctor(args []string, stdout, stderr io.Writer) int { + fs := flag.NewFlagSet("doctor", flag.ContinueOnError) + fs.SetOutput(stderr) + unitDir := fs.String("systemd-dir", systemdUnitDir, "where the installer's systemd units are") + if err := fs.Parse(args); err != nil { + if errors.Is(err, flag.ErrHelp) { + return 0 + } + return 2 + } + if os.Geteuid() != 0 { + fmt.Fprintln(stderr, "felis doctor: run as root (sudo felis doctor): the checks read root-only state under /etc/felis and /var/lib/felis") + return 1 + } + host, _ := os.Hostname() + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + return runDoctor(ctx, doctorEnv{unitDir: *unitDir, run: hostCommand, now: time.Now(), host: host}, stdout) +} + +// doctorEnv is what one doctor run reads the host through. +type doctorEnv struct { + unitDir string + run func(ctx context.Context, name string, args ...string) ([]byte, error) + now time.Time + host string +} + +// doctorAreas are the report's headings in order, by the area findingArea +// puts a finding under. +var doctorAreas = []struct{ key, title string }{ + {key: "config", title: "configuration"}, + {key: "cluster", title: "Kubernetes cluster"}, + {key: "postgres", title: "PostgreSQL"}, + {key: "proxy", title: "game proxy"}, + {key: "db-backup", title: "database backups"}, + {key: "offsite", title: "off-site copy"}, + {key: "scan-db", title: "build scan database"}, + {key: "disk", title: "disk space"}, + {key: "memory", title: "memory"}, + {key: "k3s-certs", title: "k3s certificates"}, + {key: "host-address", title: "node address"}, + {key: "clock", title: "clock"}, + {key: "systemd", title: "systemd units and timers"}, + {key: "alerts", title: "alerting"}, +} + +func runDoctor(ctx context.Context, env doctorEnv, stdout io.Writer) int { + unit := filepath.Join(env.unitDir, "felis-watchdog.service") + w, found, unitErr := watchdogUnitFlags(unit) + var report watchdog.Report + var notes []string + fmt.Fprintf(stdout, "felis doctor on %s at %s\n", env.host, env.now.UTC().Format("2006-01-02 15:04 UTC")) + switch { + case unitErr != nil: + report.Findings = append(report.Findings, watchdog.Finding{ + Key: "watchdog/unit", Severity: watchdog.Critical, + SummaryEN: fmt.Sprintf("cannot read the watchdog's settings: %v", unitErr), + Hint: "rerun the installer (deploy/bootstrap.sh) to rewrite felis-watchdog.service", + }) + fmt.Fprintf(stdout, "checks run with the watchdog's defaults (config %s)\n", w.cfgPath) + case !found: + report.Findings = append(report.Findings, watchdog.Finding{ + Key: "watchdog/unit", Severity: watchdog.Critical, + SummaryEN: fmt.Sprintf("%s is not installed: nothing checks this host or mails anyone when it breaks", unit), + Hint: "rerun the installer (deploy/bootstrap.sh), which installs felis-watchdog.timer", + }) + fmt.Fprintf(stdout, "checks run with the watchdog's defaults (config %s)\n", w.cfgPath) + default: + fmt.Fprintf(stdout, "checks run as %s runs them (config %s)\n", unit, w.cfgPath) + } + fmt.Fprintln(stdout) + + skip := map[string]string{} + cfg, cfgErr := config.Load(w.cfgPath) + if cfgErr != nil { + report.Findings = append(report.Findings, watchdog.Finding{ + Key: "config", Severity: watchdog.Critical, + SummaryEN: cfgErr.Error(), + Hint: "the installer writes it (deploy/bootstrap.sh); felis watchdog fails on every run until it loads", + }) + // The host's units are read without it; whether alerts reach anyone + // is not known without its relay. + for _, a := range doctorAreas { + if a.key != "config" && a.key != "systemd" { + skip[a.key] = "the configuration did not load" + } + } + } else { + _, owners, ownersErr := watchdogProbes(ctx, w, cfg, env.now, &report) + report.Findings = append(report.Findings, alertReachFindings(cfg, owners, ownersErr)...) + if w.proxyAddr == "" { + skip["proxy"] = "no -proxy-addr" + } + if w.backupDir == "" { + skip["db-backup"] = "no -backup-dir" + } + if !cfg.Offsite.Enabled() { + skip["offsite"] = "not configured" + } + if !usesMirroredScanDB(cfg) { + skip["scan-db"] = "builds do not scan against the registry's copy" + } + if len(splitList(w.certDirs)) == 0 { + skip["k3s-certs"] = "no -k3s-cert-dirs" + } + if w.nodeIP == "" { + skip["host-address"] = "no -node-ip" + } + } + report.Findings = append(report.Findings, unitFindings(ctx, env)...) + + switch url, err := readHeartbeatURL(w.heartbeatFile); { + case err != nil: + report.Findings = append(report.Findings, watchdog.Finding{ + Key: "watchdog/heartbeat", Severity: watchdog.Warning, + SummaryEN: fmt.Sprintf("the heartbeat URL is unusable, so no run pings it: %v", err), + Hint: "rerun the installer with FELIS_WATCHDOG_HEARTBEAT_URL set (docs/troubleshooting.md §14)", + }) + case url == "": + notes = append(notes, "no heartbeat URL is set: a host that goes down entirely, or a watchdog that stops running, alerts no one. "+ + "Rerun the installer with FELIS_WATCHDOG_HEARTBEAT_URL (docs/troubleshooting.md §14)") + } + if until := watchdog.QuietUntil(w.quietPath); env.now.Before(until) { + notes = append(notes, fmt.Sprintf("the watchdog mails nothing until %s (%s): the installer holds it while it restarts things on purpose, "+ + "and a marker an installer killed mid-run left behind holds it until then", + until.UTC().Format("2006-01-02 15:04 UTC"), w.quietPath)) + } + return printDoctorReport(stdout, report.Findings, skip, notes) +} + +// printDoctorReport prints each area's findings under its heading, an area +// with none as fine or, when skip says why, as not checked, then the notes +// and the count. It returns the exit status: 1 when anything was found. +func printDoctorReport(stdout io.Writer, findings []watchdog.Finding, skip map[string]string, notes []string) int { + byArea := map[string][]watchdog.Finding{} + for _, f := range findings { + a := findingArea(f.Key) + byArea[a] = append(byArea[a], f) + } + var critical, warning int + for _, a := range doctorAreas { + fs := byArea[a.key] + switch { + case len(fs) > 0: + case skip[a.key] != "": + fmt.Fprintf(stdout, "- %s: not checked, %s\n", a.title, skip[a.key]) + continue + default: + fmt.Fprintf(stdout, "✓ %s\n", a.title) + continue + } + mark := "!" + if slices.ContainsFunc(fs, func(f watchdog.Finding) bool { return f.Severity == watchdog.Critical }) { + mark = "✗" + } + fmt.Fprintf(stdout, "%s %s\n", mark, a.title) + for _, f := range fs { + if f.Severity == watchdog.Critical { + critical++ + } else { + warning++ + } + fmt.Fprintf(stdout, " %-8s %s: %s\n", f.Severity, f.Key, f.SummaryEN) + if f.Hint != "" { + fmt.Fprintf(stdout, " → %s\n", f.Hint) + } + } + } + for _, n := range notes { + fmt.Fprintf(stdout, "\nnote: %s\n", n) + } + fmt.Fprintln(stdout) + if critical+warning == 0 { + fmt.Fprintln(stdout, "no problems found") + return 0 + } + fmt.Fprintf(stdout, "%d problem(s): %d critical, %d warning(s)\n", critical+warning, critical, warning) + return 1 +} + +// findingArea is the report heading a finding key goes under. +func findingArea(key string) string { + head, _, _ := strings.Cut(key, "/") + switch head { + case "kube-api", "deployment", "system-server", "server-failed", "job-failed", "reaper-stale", "node": + return "cluster" + case "db-backup", "db-backup-servers": + return "db-backup" + case "unit", "timer": + return "systemd" + case "watchdog": + return "alerts" + } + return head +} + +// alertReachFindings is why the watchdog's alerts would reach no one, which +// its own runs only log: no relay, or no owner with a verified address. +func alertReachFindings(cfg *config.Config, owners []string, ownersErr error) []watchdog.Finding { + var out []watchdog.Finding + if cfg.SMTP.Host == "" { + out = append(out, watchdog.Finding{ + Key: "alerts/relay", Severity: watchdog.Warning, + SummaryEN: "no [smtp] relay is configured: the watchdog logs its alerts to the journal and mails no one", + Hint: "sudo felis setup, step SMTP", + }) + } + if ownersErr == nil && len(owners) == 0 { + out = append(out, watchdog.Finding{ + Key: "alerts/recipients", Severity: watchdog.Warning, + SummaryEN: "no owner account has a verified email: the watchdog's alerts reach no one", + Hint: "an owner verifies an address in the panel's account settings", + }) + } + return out +} + +// unitFindings reports the installer's systemd units that failed, the +// long-running ones that are not running, and timers that no longer fire. +func unitFindings(ctx context.Context, env doctorEnv) []watchdog.Finding { + var out []watchdog.Finding + seen := map[string]bool{} + failed, err := env.run(ctx, "systemctl", "list-units", "--all", "--plain", "--no-legend", "--no-pager", "--state=failed", "felis-*", "k3s.service") + if err != nil && len(failed) == 0 { + return []watchdog.Finding{{ + Key: "unit/systemctl", Severity: watchdog.Warning, + SummaryEN: fmt.Sprintf("systemctl list-units failed, so no unit was checked: %v", err), + }} + } + for _, line := range strings.Split(string(failed), "\n") { + fields := strings.Fields(line) + if len(fields) == 0 { + continue + } + name := fields[0] + seen[name] = true + out = append(out, watchdog.Finding{ + Key: "unit/" + name, Severity: watchdog.Critical, + SummaryEN: name + " failed", + Hint: fmt.Sprintf("journalctl -u %s -n 100 --no-pager; once fixed, sudo systemctl reset-failed %s (a timer's job clears on its next good run)", name, name), + }) + } + for _, name := range hostServices { + if seen[name] { + continue + } + if _, err := os.Stat(filepath.Join(env.unitDir, name)); err != nil { + continue + } + if state := unitActiveState(ctx, env, name); state != "active" { + out = append(out, watchdog.Finding{ + Key: "unit/" + name, Severity: watchdog.Critical, + SummaryEN: fmt.Sprintf("%s is %s", name, state), + Hint: fmt.Sprintf("sudo systemctl start %s; journalctl -u %s -n 100 --no-pager", name, name), + }) + } + } + timers, _ := filepath.Glob(filepath.Join(env.unitDir, "felis-*.timer")) + for _, path := range timers { + name := filepath.Base(path) + if state := unitActiveState(ctx, env, name); state != "active" { + out = append(out, watchdog.Finding{ + Key: "timer/" + name, Severity: watchdog.Warning, + SummaryEN: fmt.Sprintf("%s is %s: the job it starts no longer runs", name, state), + Hint: fmt.Sprintf("sudo systemctl enable --now %s", name), + }) + } + } + return out +} + +// unitActiveState is what `systemctl is-active` says of unit. +func unitActiveState(ctx context.Context, env doctorEnv, unit string) string { + out, err := env.run(ctx, "systemctl", "is-active", unit) + if state := strings.TrimSpace(string(out)); state != "" { + return state + } + return fmt.Sprintf("in an unknown state (systemctl is-active: %v)", err) +} + +// watchdogUnitFlags reads the flags felis-watchdog.service runs felis +// watchdog with. found is false when there is no such unit; w is then the +// watchdog's defaults. +func watchdogUnitFlags(path string) (w watchdogFlags, found bool, err error) { + fs := flag.NewFlagSet("watchdog", flag.ContinueOnError) + fs.SetOutput(io.Discard) + w.register(fs) + raw, err := os.ReadFile(path) + if errors.Is(err, os.ErrNotExist) { + return w, false, nil + } + if err != nil { + return w, false, err + } + for _, line := range strings.Split(string(raw), "\n") { + cmd, ok := strings.CutPrefix(strings.TrimSpace(line), "ExecStart=") + if !ok { + continue + } + fields := execArgs(cmd) + i := slices.Index(fields, "watchdog") + if i < 0 { + continue + } + if err := fs.Parse(fields[i+1:]); err != nil { + return w, true, fmt.Errorf("%s: %w", path, err) + } + return w, true, nil + } + return w, true, fmt.Errorf("%s runs no `felis watchdog`", path) +} + +// execArgs splits an ExecStart= command line into its words. A word may be +// quoted with " or ', as systemd allows, which is how an empty value is +// written. +func execArgs(s string) []string { + var out []string + var cur strings.Builder + inWord := false + var quote rune + for _, r := range s { + switch { + case quote != 0: + if r == quote { + quote = 0 + } else { + cur.WriteRune(r) + } + case r == '"' || r == '\'': + quote, inWord = r, true + case r == ' ' || r == '\t': + if inWord { + out = append(out, cur.String()) + cur.Reset() + inWord = false + } + default: + cur.WriteRune(r) + inWord = true + } + } + if inWord { + out = append(out, cur.String()) + } + return out +} diff --git a/cmd/felis/doctor_test.go b/cmd/felis/doctor_test.go new file mode 100644 index 0000000..239ced9 --- /dev/null +++ b/cmd/felis/doctor_test.go @@ -0,0 +1,423 @@ +package main + +import ( + "bytes" + "context" + "errors" + "os" + "path/filepath" + "regexp" + "strconv" + "strings" + "testing" + "time" + + "felis.lolicon.best/internal/config" + "felis.lolicon.best/internal/watchdog" +) + +// bootstrapWatchdogExecStart is the ExecStart= line deploy/bootstrap.sh writes +// into felis-watchdog.service, with its variables filled in as an install +// fills them. +func bootstrapWatchdogExecStart(t *testing.T, vars map[string]string) string { + t.Helper() + raw, err := os.ReadFile("../../deploy/bootstrap.sh") + if err != nil { + t.Fatal(err) + } + var line string + for _, l := range strings.Split(string(raw), "\n") { + if strings.HasPrefix(l, "ExecStart=${HOST_BIN} watchdog -config") { + line = l + } + } + if line == "" { + t.Fatal("deploy/bootstrap.sh writes no `ExecStart=${HOST_BIN} watchdog -config` line") + } + line = strings.ReplaceAll(line, "${NODE_IP:+ -node-ip ${NODE_IP}}", " -node-ip "+vars["NODE_IP"]) + line = regexp.MustCompile(`\$\{([A-Za-z_]+)\}`).ReplaceAllStringFunc(line, func(m string) string { + v, ok := vars[m[2:len(m)-1]] + if !ok { + t.Fatalf("bootstrap's watchdog ExecStart= uses %s, which this test does not fill in", m) + } + return v + }) + return line +} + +// felis doctor reads the watchdog's settings from the unit the installer +// writes, so it checks the paths the timer's runs check. +func TestWatchdogUnitFlagsReadsTheInstallersUnit(t *testing.T) { + dir := t.TempDir() + exec := bootstrapWatchdogExecStart(t, map[string]string{ + "HOST_BIN": "/usr/local/bin/felis", "STATE_DIR": "/srv/felis-etc", "WATCHDOG_STATE": "/srv/watchdog/state.json", + "WATCHDOG_QUIET_FILE": "/srv/quiet-until", "FELIS_DB_BACKUP_DIR": "/srv/db-backups", "FELIS_GAME_PORT": "25577", + "disks": "/,/srv/data", "NODE_IP": "10.0.0.5", + }) + unit := filepath.Join(dir, "felis-watchdog.service") + writeTestFile(t, unit, "[Unit]\nDescription=Felis watchdog\n\n[Service]\nType=oneshot\n"+exec+"\nTimeoutStartSec=3min\n", 0o644) + + w, found, err := watchdogUnitFlags(unit) + if err != nil || !found { + t.Fatalf("found %v, err %v", found, err) + } + got := []string{w.cfgPath, w.statePath, w.quietPath, w.backupDir, w.proxyAddr, w.diskPaths, w.nodeIP, w.heartbeatFile, w.controlNS} + want := []string{"/srv/felis-etc/felis.host.toml", "/srv/watchdog/state.json", "/srv/quiet-until", "/srv/db-backups", "127.0.0.1:25577", "/,/srv/data", "10.0.0.5", defaultHeartbeatFile, "felis"} + if strings.Join(got, "|") != strings.Join(want, "|") { + t.Errorf("read\n %q\nwant\n %q", got, want) + } + + w, found, err = watchdogUnitFlags(filepath.Join(dir, "missing.service")) + if err != nil || found || w.cfgPath != "/etc/felis/felis.toml" || w.backupDir != "/var/lib/felis/db-backups" { + t.Errorf("no unit: found %v, err %v, config %q, backups %q; want the watchdog's defaults", found, err, w.cfgPath, w.backupDir) + } + + writeTestFile(t, unit, "[Service]\nExecStart=/usr/local/bin/felis version\n", 0o644) + if _, found, err := watchdogUnitFlags(unit); !found || err == nil || !strings.Contains(err.Error(), "runs no `felis watchdog`") { + t.Errorf("a unit that runs something else: found %v, err %v", found, err) + } + writeTestFile(t, unit, "[Service]\nExecStart=/usr/local/bin/felis watchdog -no-such-flag x\n", 0o644) + if _, _, err := watchdogUnitFlags(unit); err == nil { + t.Error("a flag this binary does not know was accepted") + } + writeTestFile(t, unit, "[Service]\nExecStart=/usr/local/bin/felis watchdog -backup-dir \"\"\t-proxy-addr '127.0.0.1:1' -disk-paths \"/a b\"\n", 0o644) + if w, _, err := watchdogUnitFlags(unit); err != nil || w.backupDir != "" || w.proxyAddr != "127.0.0.1:1" || w.diskPaths != "/a b" { + t.Errorf("quoted words: backups %q, proxy %q, disks %q, err %v", w.backupDir, w.proxyAddr, w.diskPaths, err) + } +} + +func TestFindingArea(t *testing.T) { + for key, want := range map[string]string{ + "kube-api": "cluster", + "deployment/felis-api": "cluster", + "system-server/lobby": "cluster", + "server-failed/survival": "cluster", + "job-failed/reaper-123": "cluster", + "reaper-stale": "cluster", + "node/felis-1/NotReady": "cluster", + "postgres": "postgres", + "proxy": "proxy", + "db-backup": "db-backup", + "db-backup-servers": "db-backup", + "offsite": "offsite", + "scan-db": "scan-db", + "disk//var/lib/felis": "disk", + "memory": "memory", + "k3s-certs": "k3s-certs", + "host-address": "host-address", + "clock": "clock", + "config": "config", + "unit/felis-offsite.service": "systemd", + "timer/felis-db-backup.timer": "systemd", + "watchdog/unit": "alerts", + "watchdog/heartbeat": "alerts", + "alerts/relay": "alerts", + "alerts/recipients": "alerts", + "something-a-later-release-reported": "something-a-later-release-reported", + } { + if got := findingArea(key); got != want { + t.Errorf("findingArea(%q) = %q, want %q", key, got, want) + } + } +} + +func TestAlertReachFindings(t *testing.T) { + relay := &config.Config{SMTP: config.SMTPConfig{Host: "smtp.example.com"}} + keys := func(fs []watchdog.Finding) string { + var k []string + for _, f := range fs { + k = append(k, f.Key) + } + return strings.Join(k, ",") + } + for _, tc := range []struct { + what string + cfg *config.Config + owners []string + ownersErr error + want string + }{ + {"a relay and an owner", relay, []string{"owner@example.com"}, nil, ""}, + {"no relay", &config.Config{}, []string{"owner@example.com"}, nil, "alerts/relay"}, + {"no owner with an address", relay, nil, nil, "alerts/recipients"}, + {"PostgreSQL down: its own finding says so", relay, nil, errors.New("refused"), ""}, + {"neither", &config.Config{}, nil, nil, "alerts/relay,alerts/recipients"}, + } { + if got := keys(alertReachFindings(tc.cfg, tc.owners, tc.ownersErr)); got != tc.want { + t.Errorf("%s: %q, want %q", tc.what, got, tc.want) + } + } +} + +// fakeSystemctl answers list-units with failed and is-active from states; +// a unit missing from states is "inactive", as systemctl says, and one whose +// state is "" gets no answer. +func fakeSystemctl(failed string, states map[string]string) func(ctx context.Context, name string, args ...string) ([]byte, error) { + return func(_ context.Context, name string, args ...string) ([]byte, error) { + if name != "systemctl" || len(args) == 0 { + return nil, errors.New("unexpected command " + name) + } + switch args[0] { + case "list-units": + return []byte(failed), nil + case "is-active": + if s, ok := states[args[1]]; ok && s == "" { + return nil, errors.New("signal: killed") + } else if ok { + return []byte(s + "\n"), nil + } + return []byte("inactive\n"), errors.New("exit status 3") + } + return nil, errors.New("unexpected systemctl " + args[0]) + } +} + +func TestUnitFindings(t *testing.T) { + dir := t.TempDir() + for _, f := range []string{"k3s.service", "felis-velocity.service", "felis-db-backup.timer", "felis-offsite.timer", "felis-offsite.service"} { + writeTestFile(t, filepath.Join(dir, f), "[Unit]\n", 0o644) + } + env := doctorEnv{unitDir: dir, run: fakeSystemctl( + "felis-offsite.service loaded failed failed Felis off-site copy\nfelis-velocity.service loaded failed failed Velocity\n", + map[string]string{"k3s.service": "active", "felis-db-backup.timer": "active", "felis-offsite.timer": "inactive"}, + )} + var got []string + for _, f := range unitFindings(context.Background(), env) { + got = append(got, string(f.Severity)+" "+f.Key+": "+f.SummaryEN) + } + want := []string{ + "critical unit/felis-offsite.service: felis-offsite.service failed", + "critical unit/felis-velocity.service: felis-velocity.service failed", + "warning timer/felis-offsite.timer: felis-offsite.timer is inactive: the job it starts no longer runs", + } + if strings.Join(got, "\n") != strings.Join(want, "\n") { + t.Errorf("findings:\n%s\nwant (felis-nano.service has no unit file here, and a failed unit is reported once):\n%s", strings.Join(got, "\n"), strings.Join(want, "\n")) + } + + env.run = fakeSystemctl("", map[string]string{"k3s.service": "activating", "felis-velocity.service": "active", "felis-db-backup.timer": "active", "felis-offsite.timer": "active"}) + got = nil + for _, f := range unitFindings(context.Background(), env) { + got = append(got, f.Key+": "+f.SummaryEN) + } + if strings.Join(got, "\n") != "unit/k3s.service: k3s.service is activating" { + t.Errorf("findings %q, want only k3s.service, which is not active", got) + } + + env.run = fakeSystemctl("", map[string]string{"k3s.service": "active", "felis-velocity.service": "active", "felis-db-backup.timer": "", "felis-offsite.timer": "active"}) + got = nil + for _, f := range unitFindings(context.Background(), env) { + got = append(got, f.Key+": "+f.SummaryEN) + } + if want := "timer/felis-db-backup.timer: felis-db-backup.timer is in an unknown state (systemctl is-active: signal: killed): the job it starts no longer runs"; strings.Join(got, "\n") != want { + t.Errorf("findings %q, want %q", got, want) + } + + env.run = func(context.Context, string, ...string) ([]byte, error) { return nil, errors.New("no systemctl") } + if fs := unitFindings(context.Background(), env); len(fs) != 1 || fs[0].Key != "unit/systemctl" || fs[0].Severity != watchdog.Warning { + t.Errorf("without systemctl: %+v, want the one unit/systemctl warning", fs) + } +} + +// quoteArgs writes args as an ExecStart= line does, each word in quotes so an +// empty one survives. +func quoteArgs(args []string) string { + q := make([]string, len(args)) + for i, a := range args { + q[i] = `"` + a + `"` + } + return strings.Join(q, " ") +} + +// doctorHost is a host with the installer's watchdog unit, whose API server +// and PostgreSQL are down. +func doctorHost(t *testing.T, cfg string, extraFlags string) (env doctorEnv, h *watchdogHost) { + t.Helper() + h = newWatchdogHost(t, cfg, nil) + unitDir := t.TempDir() + writeTestFile(t, filepath.Join(unitDir, "felis-watchdog.service"), + "[Service]\nType=oneshot\nExecStart=/usr/local/bin/felis watchdog "+quoteArgs(h.args)+ + // Off this machine's disk, whose free space is not the test's. + ` -disk-paths "/nonexistent-felis-doctor-test"`+extraFlags+"\n", 0o644) + writeTestFile(t, filepath.Join(unitDir, "felis-velocity.service"), "[Unit]\n", 0o644) + writeTestFile(t, filepath.Join(unitDir, "felis-offsite.timer"), "[Unit]\n", 0o644) + return doctorEnv{ + unitDir: unitDir, now: time.Now(), host: "felis-test", + run: fakeSystemctl("felis-db-backup.service loaded failed failed Felis database backup\n", + map[string]string{"felis-velocity.service": "active", "felis-offsite.timer": "active"}), + }, h +} + +func TestDoctorReportsByArea(t *testing.T) { + env, _ := doctorHost(t, testWatchdogConfig, "") + var out bytes.Buffer + code := runDoctor(context.Background(), env, &out) + got := out.String() + if code != 1 { + t.Errorf("exit %d, want 1 with problems found", code) + } + for _, want := range []string{ + "felis doctor on felis-test at ", + "checks run as " + filepath.Join(env.unitDir, "felis-watchdog.service") + " runs them (config ", + "✓ configuration\n", + "✗ Kubernetes cluster\n critical kube-api: ", + "✗ PostgreSQL\n critical postgres: ", + "- game proxy: not checked, no -proxy-addr\n", + "- database backups: not checked, no -backup-dir\n", + "- off-site copy: not checked, not configured\n", + "- build scan database: not checked, builds do not scan against the registry's copy\n", + "- k3s certificates: not checked, no -k3s-cert-dirs\n", + "- node address: not checked, no -node-ip\n", + "✗ systemd units and timers\n critical unit/felis-db-backup.service: felis-db-backup.service failed\n" + + " → journalctl -u felis-db-backup.service -n 100 --no-pager; once fixed, sudo systemctl reset-failed felis-db-backup.service", + "! alerting\n warning alerts/relay: no [smtp] relay is configured", + "\n4 problem(s): 3 critical, 1 warning(s)\n", + } { + if !strings.Contains(got, want) { + t.Errorf("report lacks %q:\n%s", want, got) + } + } + if strings.Contains(got, "alerts/recipients") { + t.Errorf("reported no recipients although PostgreSQL, which names them, is down:\n%s", got) + } + if strings.Contains(got, "note: no heartbeat URL") { + t.Errorf("the host has a heartbeat URL:\n%s", got) + } +} + +// A doctor run is only a look: whatever is due to be mailed stays due, no +// heartbeat is pinged, and the watchdog's state is left as it was. +func TestDoctorMailsPingsAndSavesNothing(t *testing.T) { + cfg := testWatchdogConfig + "[smtp]\nhost = \"smtp.example.com\"\nport = 587\nfrom = \"felis@example.com\"\n" + env, h := doctorHost(t, cfg, "") + if err := watchdog.SaveState(h.statePath, duePostgres("cached-pw")); err != nil { + t.Fatal(err) + } + before, err := os.ReadFile(h.statePath) + if err != nil { + t.Fatal(err) + } + watchdogSender = h.rec.sender + defer func() { watchdogSender = smtpSender }() + var out bytes.Buffer + runDoctor(context.Background(), env, &out) + if !strings.Contains(out.String(), "critical postgres: ") { + t.Fatalf("the due PostgreSQL alert was not seen:\n%s", out.String()) + } + if len(h.rec.sent) != 0 || len(h.rec.relays) != 0 { + t.Errorf("mailed %v", h.rec.sent) + } + if n := len(h.pings.pings); n != 0 { + t.Errorf("pinged the heartbeat %d times", n) + } + after, err := os.ReadFile(h.statePath) + if err != nil || !bytes.Equal(before, after) { + t.Errorf("the watchdog state changed (err %v)", err) + } + if _, err := os.Stat(h.fallbackPath); !errors.Is(err, os.ErrNotExist) { + t.Errorf("a fallback state was written: %v", err) + } +} + +func TestDoctorNotes(t *testing.T) { + env, h := doctorHost(t, testWatchdogConfig, "") + if err := os.Remove(filepath.Join(h.dir, "watchdog-heartbeat-url")); err != nil { + t.Fatal(err) + } + until := env.now.Add(10 * time.Minute).Unix() + writeTestFile(t, filepath.Join(h.dir, "quiet"), strconv.FormatInt(until, 10)+"\n", 0o644) + var out bytes.Buffer + runDoctor(context.Background(), env, &out) + for _, want := range []string{ + "\nnote: no heartbeat URL is set: ", + "\nnote: the watchdog mails nothing until " + time.Unix(until, 0).UTC().Format("2006-01-02 15:04 UTC") + " (" + filepath.Join(h.dir, "quiet") + ")", + } { + if !strings.Contains(out.String(), want) { + t.Errorf("report lacks %q:\n%s", want, out.String()) + } + } + + writeTestFile(t, filepath.Join(h.dir, "watchdog-heartbeat-url"), "not a url\n", 0o600) + out.Reset() + runDoctor(context.Background(), env, &out) + if !strings.Contains(out.String(), "warning watchdog/heartbeat: the heartbeat URL is unusable") { + t.Errorf("an unusable heartbeat URL is not reported:\n%s", out.String()) + } +} + +// Without the watchdog's unit the doctor still checks, with the watchdog's +// defaults, and says the host has no watchdog; a configuration that does not +// load leaves the checks that need it unchecked and the host's own checks on. +func TestDoctorWithoutUnitOrConfig(t *testing.T) { + env, _ := doctorHost(t, testWatchdogConfig, "") + if err := os.Remove(filepath.Join(env.unitDir, "felis-watchdog.service")); err != nil { + t.Fatal(err) + } + writeTestFile(t, filepath.Join(env.unitDir, "felis-watchdog.service"), "[Service]\nExecStart=/usr/local/bin/felis watchdog -config "+filepath.Join(t.TempDir(), "absent.toml")+"\n", 0o644) + env.run = fakeSystemctl("", map[string]string{"felis-velocity.service": "active", "felis-offsite.timer": "active"}) + var out bytes.Buffer + if code := runDoctor(context.Background(), env, &out); code != 1 { + t.Errorf("exit %d, want 1", code) + } + for _, want := range []string{ + "✗ configuration\n critical config: ", + "- Kubernetes cluster: not checked, the configuration did not load\n", + "- PostgreSQL: not checked, the configuration did not load\n", + "- disk space: not checked, the configuration did not load\n", + "✓ systemd units and timers\n", + "- alerting: not checked, the configuration did not load\n", + } { + if !strings.Contains(out.String(), want) { + t.Errorf("report lacks %q:\n%s", want, out.String()) + } + } + + if err := os.Remove(filepath.Join(env.unitDir, "felis-watchdog.service")); err != nil { + t.Fatal(err) + } + out.Reset() + runDoctor(context.Background(), env, &out) + if !strings.Contains(out.String(), "critical watchdog/unit: "+filepath.Join(env.unitDir, "felis-watchdog.service")+" is not installed") || + !strings.Contains(out.String(), "checks run with the watchdog's defaults (config /etc/felis/felis.toml)") { + t.Errorf("a host without the watchdog's unit:\n%s", out.String()) + } + + unit := filepath.Join(env.unitDir, "felis-watchdog.service") + writeTestFile(t, unit, "[Service]\nExecStart=/usr/local/bin/felis watchdog -no-such-flag x\n", 0o644) + out.Reset() + runDoctor(context.Background(), env, &out) + if !strings.Contains(out.String(), "critical watchdog/unit: cannot read the watchdog's settings: "+unit+": flag provided but not defined: -no-such-flag\n") || + !strings.Contains(out.String(), "checks run with the watchdog's defaults (config /etc/felis/felis.toml)") { + t.Errorf("a watchdog unit this binary cannot read:\n%s", out.String()) + } +} + +func TestPrintDoctorReport(t *testing.T) { + var out bytes.Buffer + if code := printDoctorReport(&out, nil, map[string]string{"proxy": "no -proxy-addr"}, nil); code != 0 { + t.Errorf("exit %d with nothing found, want 0", code) + } + want := "✓ configuration\n✓ Kubernetes cluster\n✓ PostgreSQL\n- game proxy: not checked, no -proxy-addr\n✓ database backups\n✓ off-site copy\n" + + "✓ build scan database\n✓ disk space\n✓ memory\n✓ k3s certificates\n✓ node address\n✓ clock\n✓ systemd units and timers\n✓ alerting\n" + + "\nno problems found\n" + if out.String() != want { + t.Errorf("report:\n%s\nwant:\n%s", out.String(), want) + } + + out.Reset() + code := printDoctorReport(&out, []watchdog.Finding{ + {Key: "unit/systemctl", Severity: watchdog.Warning, SummaryEN: "systemctl list-units failed"}, + {Key: "postgres", Severity: watchdog.Critical, SummaryEN: "PostgreSQL is down", Hint: "kubectl -n felis get pods"}, + }, map[string]string{"postgres": "a skip loses to what was found"}, []string{"a note"}) + if code != 1 { + t.Errorf("exit %d with problems, want 1", code) + } + for _, want := range []string{ + "✗ PostgreSQL\n critical postgres: PostgreSQL is down\n → kubectl -n felis get pods\n✓ game proxy\n", + "! systemd units and timers\n warning unit/systemctl: systemctl list-units failed\n✓ alerting\n", + "✓ alerting\n\nnote: a note\n\n2 problem(s): 1 critical, 1 warning(s)\n", + } { + if !strings.Contains(out.String(), want) { + t.Errorf("report lacks %q:\n%s", want, out.String()) + } + } +} diff --git a/cmd/felis/run.go b/cmd/felis/run.go index 8a019de..f3310a2 100644 --- a/cmd/felis/run.go +++ b/cmd/felis/run.go @@ -34,6 +34,9 @@ Commands: converge Fill in fields a newer desired spec added to already-installed system servers rotate-token Replace a generated credential and restart what reads it (velocity|limbo|build|ops|registry|forwarding|db; prints the plan, -yes applies; requires root/sudo) domain Move the install to a new root domain on every surface that carries it, or check each one (set|check; requires root/sudo) + status Print the platform at a glance: node, control plane, proxy, servers, backups, host, open alerts (requires root/sudo) + doctor Run every health check once, grouped by area, with where to look next; mails nothing (requires root/sudo) + support-bundle Collect status, doctor, logs and cluster state into one redacted tar.gz to share when asking for help (requires root/sudo) watchdog Check the platform once and mail the owners what has gone wrong (run by felis-watchdog.timer) version Print the build stamp of this binary update Report which platform components have updates available @@ -84,6 +87,9 @@ var commands = map[string]func(args []string, stdout, stderr io.Writer) int{ "version": cmdVersion, "update": cmdUpdate, "watchdog": cmdWatchdog, + "status": cmdStatus, + "doctor": cmdDoctor, + "support-bundle": cmdSupportBundle, } // run dispatches a subcommand. It is separate from main so the router is diff --git a/cmd/felis/status.go b/cmd/felis/status.go new file mode 100644 index 0000000..fee0a95 --- /dev/null +++ b/cmd/felis/status.go @@ -0,0 +1,438 @@ +package main + +import ( + "bufio" + "context" + "errors" + "flag" + "fmt" + "io" + "net" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "syscall" + "text/tabwriter" + "time" + + "felis.lolicon.best/internal/apis/felis/v1alpha1" + "felis.lolicon.best/internal/config" + "felis.lolicon.best/internal/dbbackup" + "felis.lolicon.best/internal/offsite" + "felis.lolicon.best/internal/platform" + "felis.lolicon.best/internal/store" + "felis.lolicon.best/internal/watchdog" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/labels" + "sigs.k8s.io/controller-runtime/pkg/client" +) + +// cmdStatus prints the platform at a glance: the release and the node, the +// control plane's workloads, the game proxy, every server with its players +// and newest world backup, the database backups and the off-site copy, the +// host's disks and memory, and what the watchdog has open. It changes +// nothing, and a part that is down (the cluster, PostgreSQL) reads as such +// while the rest still prints. felis doctor says what is wrong and where to +// look. +func cmdStatus(args []string, stdout, stderr io.Writer) int { + fs := flag.NewFlagSet("status", flag.ContinueOnError) + fs.SetOutput(stderr) + unitDir := fs.String("systemd-dir", systemdUnitDir, "where the installer's systemd units are") + if err := fs.Parse(args); err != nil { + if errors.Is(err, flag.ErrHelp) { + return 0 + } + return 2 + } + if os.Geteuid() != 0 { + fmt.Fprintln(stderr, "felis status: run as root (sudo felis status): it reads root-only state under /etc/felis and /var/lib/felis") + return 1 + } + env, err := hostStatusEnv(*unitDir) + if err != nil { + fmt.Fprintf(stderr, "felis status: %v\n", err) + return 1 + } + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + defer cancel() + printStatus(ctx, env, stdout) + return 0 +} + +// statusEnv is what one status report reads the host through. +type statusEnv struct { + cfg *config.Config + // w is how felis-watchdog.service runs the watchdog: where the backups, + // the off-site record and the proxy are. + w watchdogFlags + cl client.Client // nil while the API server is unreachable + clErr error + backups func(ctx context.Context) (map[string]time.Time, error) + run func(ctx context.Context, name string, args ...string) ([]byte, error) + unitDir string + meminfo string + host string + now time.Time +} + +// hostStatusEnv reads this host: the watchdog's settings, the configuration +// they name, and the cluster. +func hostStatusEnv(unitDir string) (statusEnv, error) { + w, _, err := watchdogUnitFlags(filepath.Join(unitDir, "felis-watchdog.service")) + if err != nil { + return statusEnv{}, err + } + cfg, err := config.Load(w.cfgPath) + if err != nil { + return statusEnv{}, err + } + cl, clErr := buildSystemServerClient() + if clErr != nil { + cl = nil + } + host, _ := os.Hostname() + return statusEnv{ + cfg: cfg, w: w, cl: cl, clErr: clErr, + backups: func(ctx context.Context) (map[string]time.Time, error) { + return newestWorldBackups(ctx, cfg.Database.URL) + }, + run: hostCommand, unitDir: unitDir, meminfo: "/proc/meminfo", host: host, now: time.Now(), + }, nil +} + +func printStatus(ctx context.Context, env statusEnv, out io.Writer) { + fmt.Fprintf(out, "felis %s on %s at %s\n", resolvedVersion(), env.host, env.now.UTC().Format("2006-01-02 15:04 UTC")) + if env.cl == nil { + fmt.Fprintf(out, "cluster: unreachable (%v)\n", env.clErr) + } else { + statusCluster(ctx, env, out) + } + statusProxy(ctx, env, out) + statusServers(ctx, env, out) + statusBackups(env, out) + statusHost(env, out) + statusWatchdog(ctx, env, out) +} + +func statusCluster(ctx context.Context, env statusEnv, out io.Writer) { + var nodes corev1.NodeList + if err := env.cl.List(ctx, &nodes); err != nil { + fmt.Fprintf(out, "cluster: unreachable (%v)\n", err) + return + } + for _, n := range nodes.Items { + ready := "NotReady" + for _, c := range n.Status.Conditions { + if c.Type == corev1.NodeReady && c.Status == corev1.ConditionTrue { + ready = "Ready" + } + } + info := n.Status.NodeInfo + fmt.Fprintf(out, "node: %s %s, k3s %s, %s, kernel %s\n", n.Name, ready, info.KubeletVersion, info.OSImage, info.KernelVersion) + } + + ns := env.w.controlNS + var deps appsv1.DeploymentList + var pods corev1.PodList + err := env.cl.List(ctx, &deps, client.InNamespace(ns)) + if err == nil { + err = env.cl.List(ctx, &pods, client.InNamespace(ns)) + } + fmt.Fprintf(out, "\ncontrol plane (namespace %s):\n", ns) + if err != nil { + fmt.Fprintf(out, " cannot list it: %v\n", err) + return + } + sort.Slice(deps.Items, func(i, j int) bool { return deps.Items[i].Name < deps.Items[j].Name }) + tw := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0) + for _, d := range deps.Items { + want := int32(1) + if d.Spec.Replicas != nil { + want = *d.Spec.Replicas + } + image := "-" + if cs := d.Spec.Template.Spec.Containers; len(cs) > 0 { + image = shortImage(cs[0].Image) + } + fmt.Fprintf(tw, " %s\t%d/%d ready\t%s\trestarts %d\n", d.Name, d.Status.ReadyReplicas, want, image, podRestarts(d.Spec.Selector, pods.Items)) + } + tw.Flush() +} + +// podRestarts adds up the container restarts of the pods selector picks (the +// API server refuses a Deployment whose selector is empty). +func podRestarts(selector *metav1.LabelSelector, pods []corev1.Pod) int32 { + sel, err := metav1.LabelSelectorAsSelector(selector) + if err != nil { + return 0 + } + var n int32 + for _, p := range pods { + if !sel.Matches(labels.Set(p.Labels)) { + continue + } + for _, cs := range p.Status.ContainerStatuses { + n += cs.RestartCount + } + } + return n +} + +// shortImage is an image reference without its registry and repository path, +// and with its digest cut to 12 characters. +func shortImage(ref string) string { + if i := strings.LastIndex(ref, "/"); i >= 0 { + ref = ref[i+1:] + } + if name, digest, ok := strings.Cut(ref, "@sha256:"); ok && len(digest) > 12 { + ref = name + "@" + digest[:12] + } + return ref +} + +func statusProxy(ctx context.Context, env statusEnv, out io.Writer) { + var parts []string + if _, err := os.Stat(filepath.Join(env.unitDir, "felis-velocity.service")); err == nil { + parts = append(parts, "felis-velocity "+unitActiveState(ctx, doctorEnv{run: env.run}, "felis-velocity.service")) + } + if env.w.proxyAddr != "" { + d := net.Dialer{Timeout: 3 * time.Second} + if conn, err := d.DialContext(ctx, "tcp", env.w.proxyAddr); err != nil { + parts = append(parts, fmt.Sprintf("%s refuses connections (%v)", env.w.proxyAddr, err)) + } else { + conn.Close() + parts = append(parts, env.w.proxyAddr+" accepts connections") + } + } + if len(parts) == 0 { + parts = append(parts, "not on this host") + } + fmt.Fprintf(out, "\nproxy: %s\n", strings.Join(parts, ", ")) +} + +func statusServers(ctx context.Context, env statusEnv, out io.Writer) { + ns := env.cfg.K8s.Namespace + if ns == "" { + ns = platform.DefaultMinecraftNamespace + } + if env.cl == nil { + fmt.Fprintf(out, "\nservers (namespace %s): unknown while the cluster is unreachable\n", ns) + return + } + var list v1alpha1.MinecraftServerList + if err := env.cl.List(ctx, &list, client.InNamespace(ns)); err != nil { + fmt.Fprintf(out, "\nservers (namespace %s): cannot list them: %v\n", ns, err) + return + } + newest, backupErr := env.backups(ctx) + running, online := 0, int32(0) + for _, ms := range list.Items { + if ms.Status.Phase == v1alpha1.PhaseRunning { + running++ + online += ms.Status.Players.Online + } + } + fmt.Fprintf(out, "\nservers (namespace %s): %d, %d running, %d players online\n", ns, len(list.Items), running, online) + if len(list.Items) == 0 { + return + } + sort.Slice(list.Items, func(i, j int) bool { return list.Items[i].Name < list.Items[j].Name }) + tw := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0) + fmt.Fprintln(tw, " NAME\tROLE\tDESIRED\tPHASE\tPLAYERS\tNEWEST WORLD BACKUP") + for _, ms := range list.Items { + role := ms.Labels[v1alpha1.LabelSystemRole] + if role == "" { + role = "-" + } + phase := string(ms.Status.Phase) + if phase == "" { + phase = "-" + } + players := "-" + if ms.Status.Phase == v1alpha1.PhaseRunning { + players = fmt.Sprintf("%d/%d", ms.Status.Players.Online, ms.Status.Players.Max) + } + backup := "none" + switch at, ok := newest[ms.Name]; { + case backupErr != nil: + backup = "?" + case ok: + backup = dbbackup.Age(env.now.Sub(at)) + " ago" + } + fmt.Fprintf(tw, " %s\t%s\t%s\t%s\t%s\t%s\n", ms.Name, role, orDash(string(ms.Spec.DesiredState)), phase, players, backup) + } + tw.Flush() + if backupErr != nil { + fmt.Fprintf(out, " world backups unknown: %v\n", backupErr) + } +} + +func orDash(s string) string { + if s == "" { + return "-" + } + return s +} + +// newestWorldBackups is when each server's newest world backup that a restore +// can use was taken. +func newestWorldBackups(ctx context.Context, url string) (map[string]time.Time, error) { + ctx, cancel := context.WithTimeout(ctx, 15*time.Second) + defer cancel() + drv, err := store.Open(ctx, url) + if err != nil { + return nil, err + } + defer drv.Close() + rows, err := drv.DB().QueryContext(ctx, `SELECT server_name, max(created_at) FROM world_backups + WHERE status = 'present' AND corrupt_at IS NULL GROUP BY server_name`) + if err != nil { + return nil, err + } + defer rows.Close() + out := map[string]time.Time{} + for rows.Next() { + var name string + var at time.Time + if err := rows.Scan(&name, &at); err != nil { + return nil, err + } + out[name] = at + } + return out, rows.Err() +} + +func statusBackups(env statusEnv, out io.Writer) { + fmt.Fprintln(out, "\nbackups:") + switch bundles, err := dbbackup.List(env.w.backupDir); { + case env.w.backupDir == "": + fmt.Fprintln(out, " database: not checked (felis-watchdog.service names no -backup-dir)") + case err != nil: + fmt.Fprintf(out, " database: cannot read %s: %v\n", env.w.backupDir, err) + case len(bundles) == 0: + fmt.Fprintf(out, " database: none in %s\n", env.w.backupDir) + default: + fmt.Fprintf(out, " database: newest %s, %s ago; %d bundles in %s\n", + bundles[0].Name, dbbackup.Age(env.now.Sub(bundles[0].Created)), len(bundles), env.w.backupDir) + } + if !env.cfg.Offsite.Enabled() { + fmt.Fprintln(out, " off-site: not configured, every backup is on this machine only") + return + } + switch st, err := offsite.ReadStatus(env.w.offsiteStatus); { + case err != nil: + fmt.Fprintf(out, " off-site: %v\n", err) + case st == nil: + fmt.Fprintln(out, " off-site: never synced") + case st.LastSuccess.IsZero(): + fmt.Fprintf(out, " off-site: never succeeded; last attempt %s ago: %s\n", dbbackup.Age(env.now.Sub(st.LastAttempt)), st.LastError) + default: + line := fmt.Sprintf(" off-site: last good sync %s ago to %s", dbbackup.Age(env.now.Sub(st.LastSuccess)), st.Bucket) + if st.LastAttempt.After(st.LastSuccess) { // a failed run records no success + line += fmt.Sprintf("; the last attempt, %s ago, failed: %s", dbbackup.Age(env.now.Sub(st.LastAttempt)), st.LastError) + } + fmt.Fprintln(out, line) + } +} + +func statusHost(env statusEnv, out io.Writer) { + fmt.Fprintln(out, "\nhost:") + seen := map[uint64]bool{} + for _, p := range splitList(env.w.diskPaths) { + var st syscall.Stat_t + if err := syscall.Stat(p, &st); err != nil { + continue + } + dev := uint64(st.Dev) // int32 on darwin + if seen[dev] { + continue + } + seen[dev] = true + var fs syscall.Statfs_t + if err := syscall.Statfs(p, &fs); err != nil || fs.Blocks == 0 { + continue + } + bsize := uint64(fs.Bsize) // uint32 on darwin + total, avail := uint64(fs.Blocks)*bsize, uint64(fs.Bavail)*bsize + fmt.Fprintf(out, " disk %s: %s free of %s (%.0f%% free)\n", p, offsite.HumanBytes(int64(avail)), offsite.HumanBytes(int64(total)), float64(fs.Bavail)/float64(fs.Blocks)*100) + } + if total, avail, ok := readMeminfo(env.meminfo); ok { + fmt.Fprintf(out, " memory: %s available of %s\n", offsite.HumanBytes(int64(avail)), offsite.HumanBytes(int64(total))) + } +} + +// readMeminfo reads MemTotal and MemAvailable, in bytes, from a /proc/meminfo +// style file. +func readMeminfo(path string) (total, avail uint64, ok bool) { + f, err := os.Open(path) + if err != nil { + return 0, 0, false + } + defer f.Close() + sc := bufio.NewScanner(f) + for sc.Scan() { + fields := strings.Fields(sc.Text()) + if len(fields) < 2 { + continue + } + v, _ := strconv.ParseUint(fields[1], 10, 64) // the kernel writes numbers + switch fields[0] { + case "MemTotal:": + total = v * 1024 + case "MemAvailable:": + avail = v * 1024 + } + } + return total, avail, total > 0 +} + +func statusWatchdog(ctx context.Context, env statusEnv, out io.Writer) { + fmt.Fprintln(out, "\nwatchdog:") + if _, err := os.Stat(filepath.Join(env.unitDir, "felis-watchdog.timer")); err != nil { + fmt.Fprintln(out, " not installed: nothing checks this host") + return + } + line := " timer " + unitActiveState(ctx, doctorEnv{run: env.run}, "felis-watchdog.timer") + show, _ := env.run(ctx, "systemctl", "show", "--timestamp=unix", "-p", "Result", "-p", "ExecMainExitTimestamp", "felis-watchdog.service") + props := map[string]string{} + for _, ln := range strings.Split(string(show), "\n") { + if k, v, ok := strings.Cut(strings.TrimSpace(ln), "="); ok { + props[k] = v + } + } + // ExecMainExitTimestamp is empty until the service has run. + if sec, _ := strconv.ParseInt(strings.TrimPrefix(props["ExecMainExitTimestamp"], "@"), 10, 64); sec > 0 { + line += fmt.Sprintf(", last run %s ago (%s)", dbbackup.Age(env.now.Sub(time.Unix(sec, 0))), orDash(props["Result"])) + } + fmt.Fprintln(out, line) + + state, err := watchdog.LoadState(watchdog.NewestState(env.w.statePath, env.w.fallbackState)) + if err != nil { + fmt.Fprintf(out, " alerts: unknown (%v)\n", err) + return + } + var open []string + for key, a := range state.Alerts { + if !a.ClearedAt.IsZero() { + continue + } + if a.Notified.IsZero() { + open = append(open, fmt.Sprintf("%s (%s, seen %s ago, not mailed yet)", key, a.Severity, dbbackup.Age(env.now.Sub(a.FirstSeen)))) + } else { + open = append(open, fmt.Sprintf("%s (%s, mailed %s ago)", key, a.Severity, dbbackup.Age(env.now.Sub(a.Notified)))) + } + } + if len(open) == 0 { + fmt.Fprintln(out, " alerts: none open") + return + } + sort.Strings(open) + fmt.Fprintf(out, " alerts: %d open (sudo felis doctor says where to look)\n", len(open)) + for _, o := range open { + fmt.Fprintf(out, " %s\n", o) + } +} diff --git a/cmd/felis/status_test.go b/cmd/felis/status_test.go new file mode 100644 index 0000000..28612b5 --- /dev/null +++ b/cmd/felis/status_test.go @@ -0,0 +1,490 @@ +package main + +import ( + "bytes" + "context" + "errors" + "fmt" + "net" + "os" + "path/filepath" + "slices" + "strconv" + "strings" + "testing" + "time" + + "felis.lolicon.best/internal/apis/felis/v1alpha1" + "felis.lolicon.best/internal/config" + "felis.lolicon.best/internal/dbbackup" + "felis.lolicon.best/internal/offsite" + "felis.lolicon.best/internal/watchdog" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" +) + +// statusCluster is a one-node install: felis-api with a restarted pod, three +// servers (one of them the lobby), and a pod of another app whose restarts +// are not felis-api's. +func statusClusterObjects() []client.Object { + replicas := int32(1) + apiLabels := map[string]string{"app": "felis-api"} + return []client.Object{ + &corev1.Node{ + ObjectMeta: metav1.ObjectMeta{Name: "felis-1"}, + Status: corev1.NodeStatus{ + Conditions: []corev1.NodeCondition{{Type: corev1.NodeReady, Status: corev1.ConditionTrue}}, + NodeInfo: corev1.NodeSystemInfo{KubeletVersion: "v1.36.4+k3s1", OSImage: "CentOS Stream 9", KernelVersion: "5.14.0-630.el9.aarch64"}, + }, + }, + &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{Name: "felis-api", Namespace: "felis"}, + Spec: appsv1.DeploymentSpec{ + Replicas: &replicas, + Selector: &metav1.LabelSelector{MatchLabels: apiLabels}, + Template: corev1.PodTemplateSpec{Spec: corev1.PodSpec{Containers: []corev1.Container{{ + Name: "api", Image: "registry.felis.svc:5000/felis/felis-api:v1.4.0@sha256:0123456789abcdef0123456789abcdef", + }}}}, + }, + Status: appsv1.DeploymentStatus{ReadyReplicas: 1}, + }, + &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Name: "felis-api-7d9", Namespace: "felis", Labels: apiLabels}, + Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{{Name: "api", RestartCount: 2}}}, + }, + &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Name: "other-1", Namespace: "felis", Labels: map[string]string{"app": "other"}}, + Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{{Name: "x", RestartCount: 5}}}, + }, + &v1alpha1.MinecraftServer{ + ObjectMeta: metav1.ObjectMeta{Name: "survival", Namespace: "minecraft"}, + Spec: v1alpha1.MinecraftServerSpec{DesiredState: v1alpha1.DesiredRunning}, + Status: v1alpha1.MinecraftServerStatus{Phase: v1alpha1.PhaseRunning, Players: v1alpha1.PlayersStatus{Online: 2, Max: 20}}, + }, + &v1alpha1.MinecraftServer{ + ObjectMeta: metav1.ObjectMeta{Name: "creative", Namespace: "minecraft"}, + Spec: v1alpha1.MinecraftServerSpec{DesiredState: v1alpha1.DesiredStopped}, + Status: v1alpha1.MinecraftServerStatus{Phase: v1alpha1.PhaseStopped}, + }, + &v1alpha1.MinecraftServer{ + ObjectMeta: metav1.ObjectMeta{Name: "lobby", Namespace: "minecraft", Labels: map[string]string{v1alpha1.LabelSystemRole: "lobby"}}, + Spec: v1alpha1.MinecraftServerSpec{DesiredState: v1alpha1.DesiredRunning}, + Status: v1alpha1.MinecraftServerStatus{Phase: v1alpha1.PhaseStarting}, + }, + // Another namespace's server is not this install's. + &v1alpha1.MinecraftServer{ObjectMeta: metav1.ObjectMeta{Name: "elsewhere", Namespace: "other"}}, + } +} + +// statusTestEnv is a host with the cluster above, a database backup nine hours +// old, an off-site copy last good two hours ago, and one alert open. +func statusTestEnv(t *testing.T) statusEnv { + t.Helper() + now := time.Date(2026, 9, 27, 12, 0, 0, 0, time.UTC) + dir := t.TempDir() + var w watchdogFlags + w.controlNS = "felis" + w.backupDir = filepath.Join(dir, "db-backups") + w.offsiteStatus = filepath.Join(dir, "offsite-status.json") + w.statePath = filepath.Join(dir, "state.json") + w.fallbackState = filepath.Join(dir, "fallback.json") + w.diskPaths = "/nonexistent-felis-status-test" + unitDir := filepath.Join(dir, "systemd") + for _, d := range []string{w.backupDir, unitDir} { + if err := os.MkdirAll(d, 0o700); err != nil { + t.Fatal(err) + } + } + writeTestFile(t, filepath.Join(w.backupDir, dbbackup.BundleName(now.Add(-9*time.Hour), dbbackup.LabelDaily)), "x", 0o600) + writeTestFile(t, filepath.Join(w.backupDir, dbbackup.BundleName(now.Add(-33*time.Hour), dbbackup.LabelDaily)), "x", 0o600) + if err := offsite.WriteStatus(w.offsiteStatus, offsite.Status{ + LastAttempt: now.Add(-2 * time.Hour), LastSuccess: now.Add(-2 * time.Hour), Bucket: "felis-dr", Endpoint: "https://s3.example.com", + }); err != nil { + t.Fatal(err) + } + state := &watchdog.State{Alerts: map[string]*watchdog.Alert{ + "db-backup-servers": {Finding: watchdog.Finding{Key: "db-backup-servers", Severity: watchdog.Warning}, FirstSeen: now.Add(-3 * time.Hour), Notified: now.Add(-90 * time.Minute)}, + "proxy": {Finding: watchdog.Finding{Key: "proxy", Severity: watchdog.Critical}, FirstSeen: now.Add(-time.Hour), Notified: now.Add(-time.Hour), ClearedAt: now.Add(-10 * time.Minute)}, + "disk//var": {Finding: watchdog.Finding{Key: "disk//var", Severity: watchdog.Critical}, FirstSeen: now.Add(-4 * time.Minute)}, + }} + if err := watchdog.SaveState(w.statePath, state); err != nil { + t.Fatal(err) + } + meminfo := filepath.Join(dir, "meminfo") + writeTestFile(t, meminfo, "MemTotal: 8000000 kB\nMemFree: 100000 kB\nMemAvailable: 2000000 kB\n", 0o644) + writeTestFile(t, filepath.Join(unitDir, "felis-watchdog.timer"), "[Unit]\n", 0o644) + cfg := &config.Config{Offsite: config.OffsiteConfig{Endpoint: "https://s3.example.com", Bucket: "felis-dr"}} + return statusEnv{ + cfg: cfg, w: w, cl: fake.NewClientBuilder().WithScheme(newSystemServerScheme(t)).WithObjects(statusClusterObjects()...).Build(), + backups: func(context.Context) (map[string]time.Time, error) { + return map[string]time.Time{"survival": now.Add(-3 * time.Hour)}, nil + }, + run: func(_ context.Context, name string, args ...string) ([]byte, error) { + switch strings.Join(append([]string{name}, args...), " ") { + case "systemctl is-active felis-watchdog.timer": + return []byte("active\n"), nil + case "systemctl show --timestamp=unix -p Result -p ExecMainExitTimestamp felis-watchdog.service": + return []byte("Result=success\nExecMainExitTimestamp=@" + strconv.FormatInt(now.Add(-70*time.Second).Unix(), 10) + "\n"), nil + } + return nil, errors.New("unexpected") + }, + unitDir: unitDir, meminfo: meminfo, host: "felis-test", now: now, + } +} + +// lineFields is the words of the first line of out that starts with prefix +// once trimmed. +func lineFields(out, prefix string) []string { + for _, l := range strings.Split(out, "\n") { + if strings.HasPrefix(strings.TrimSpace(l), prefix) { + return strings.Fields(l) + } + } + return nil +} + +func TestStatusReport(t *testing.T) { + env := statusTestEnv(t) + var out bytes.Buffer + printStatus(context.Background(), env, &out) + got := out.String() + for _, want := range []string{ + "on felis-test at 2026-09-27 12:00 UTC\n", + "node: felis-1 Ready, k3s v1.36.4+k3s1, CentOS Stream 9, kernel 5.14.0-630.el9.aarch64\n", + "\ncontrol plane (namespace felis):\n", + "\nproxy: not on this host\n", + "\nservers (namespace minecraft): 3, 1 running, 2 players online\n", + " database: newest " + dbbackup.BundleName(env.now.Add(-9*time.Hour), dbbackup.LabelDaily) + ", 9h0m ago; 2 bundles in " + env.w.backupDir + "\n", + " off-site: last good sync 2h0m ago to felis-dr\n", + " memory: 1.9 GiB available of 7.6 GiB\n", + " timer active, last run 1m ago (success)\n", + " alerts: 2 open (sudo felis doctor says where to look)\n" + + " db-backup-servers (warning, mailed 1h30m ago)\n" + + " disk//var (critical, seen 4m ago, not mailed yet)\n", + } { + if !strings.Contains(got, want) { + t.Errorf("status lacks %q:\n%s", want, got) + } + } + for prefix, want := range map[string]string{ + "felis-api": "felis-api 1/1 ready felis-api:v1.4.0@0123456789ab restarts 2", + "NAME": "NAME ROLE DESIRED PHASE PLAYERS NEWEST WORLD BACKUP", + "creative": "creative - Stopped Stopped - none", + "lobby": "lobby lobby Running Starting - none", + "survival": "survival - Running Running 2/20 3h0m ago", + } { + if f := strings.Join(lineFields(got, prefix), " "); f != want { + t.Errorf("row %q = %q, want %q\n%s", prefix, f, want, got) + } + } + if strings.Contains(got, "elsewhere") || strings.Contains(got, "other-1") { + t.Errorf("status shows what is not this install's:\n%s", got) + } +} + +// Whatever is down reads as down, and the rest of the report still prints. +func TestStatusWithPartsDown(t *testing.T) { + env := statusTestEnv(t) + env.cl, env.clErr = nil, errors.New("connection refused") + env.cfg = &config.Config{} + var out bytes.Buffer + printStatus(context.Background(), env, &out) + for _, want := range []string{ + "cluster: unreachable (connection refused)\n", + "\nservers (namespace minecraft): unknown while the cluster is unreachable\n", + " off-site: not configured, every backup is on this machine only\n", + " timer active, last run", + } { + if !strings.Contains(out.String(), want) { + t.Errorf("status lacks %q:\n%s", want, out.String()) + } + } + + env = statusTestEnv(t) + env.backups = func(context.Context) (map[string]time.Time, error) { return nil, errors.New("postgres is down") } + out.Reset() + printStatus(context.Background(), env, &out) + if f := strings.Join(lineFields(out.String(), "survival"), " "); f != "survival - Running Running 2/20 ?" { + t.Errorf("survival with PostgreSQL down = %q", f) + } + if !strings.Contains(out.String(), " world backups unknown: postgres is down\n") { + t.Errorf("status does not say why the backups are unknown:\n%s", out.String()) + } +} + +func TestShortImage(t *testing.T) { + for in, want := range map[string]string{ + "registry.felis.svc:5000/felis/felis-api:v1.4.0": "felis-api:v1.4.0", + "docker.io/library/postgres:17@sha256:0123456789abcdef0123": "postgres:17@0123456789ab", + "felis-operator@sha256:fedcba9876543210fedcba9876543210fedcba9876543210fedcba98765432": "felis-operator@fedcba987654", + "busybox": "busybox", + } { + if got := shortImage(in); got != want { + t.Errorf("shortImage(%q) = %q, want %q", in, got, want) + } + } +} + +// A node that is not ready, a Deployment without replicas or containers, and +// lists the API server refuses each read as such. +func TestStatusClusterEdges(t *testing.T) { + env := statusTestEnv(t) + env.cfg = &config.Config{K8s: config.K8sConfig{Namespace: "games"}} + env.cl = fake.NewClientBuilder().WithScheme(newSystemServerScheme(t)).WithObjects( + &corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "felis-1"}, Status: corev1.NodeStatus{ + Conditions: []corev1.NodeCondition{{Type: corev1.NodeReady, Status: corev1.ConditionFalse}, {Type: corev1.NodeMemoryPressure, Status: corev1.ConditionTrue}}}}, + &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: "felis-bare", Namespace: "felis"}}, + &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "p", Namespace: "felis"}, Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{{RestartCount: 4}}}}, + ).Build() + var out bytes.Buffer + printStatus(context.Background(), env, &out) + got := out.String() + for _, want := range []string{ + "node: felis-1 NotReady, k3s , , kernel \n", + "\nservers (namespace games): 0, 0 running, 0 players online\n\nbackups:\n", + } { + if !strings.Contains(got, want) { + t.Errorf("status lacks %q:\n%s", want, got) + } + } + if f := strings.Join(lineFields(got, "felis-bare"), " "); f != "felis-bare 0/1 ready - restarts 0" { + t.Errorf("row felis-bare = %q, want its one replica wanted, no image, and no pod of its own:\n%s", f, got) + } + + failing := func(what client.ObjectList) { + env.cl = fake.NewClientBuilder().WithScheme(newSystemServerScheme(t)).WithObjects(statusClusterObjects()...). + WithInterceptorFuncs(interceptor.Funcs{List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if fmt.Sprintf("%T", list) == fmt.Sprintf("%T", what) { + return errors.New("forbidden") + } + return c.List(ctx, list, opts...) + }}).Build() + out.Reset() + printStatus(context.Background(), env, &out) + } + for _, tc := range []struct { + list client.ObjectList + want string + }{ + {&corev1.NodeList{}, "cluster: unreachable (forbidden)\n\nproxy:"}, + {&appsv1.DeploymentList{}, "\ncontrol plane (namespace felis):\n cannot list it: forbidden\n\nproxy:"}, + {&corev1.PodList{}, "\ncontrol plane (namespace felis):\n cannot list it: forbidden\n\nproxy:"}, + {&v1alpha1.MinecraftServerList{}, "\nservers (namespace games): cannot list them: forbidden\n\nbackups:"}, + } { + failing(tc.list) + if !strings.Contains(out.String(), tc.want) { + t.Errorf("listing %T refused: status lacks %q:\n%s", tc.list, tc.want, out.String()) + } + } +} + +func TestStatusProxy(t *testing.T) { + env := statusTestEnv(t) + writeTestFile(t, filepath.Join(env.unitDir, "felis-velocity.service"), "[Unit]\n", 0o644) + env.run = fakeSystemctl("", map[string]string{"felis-velocity.service": "active"}) + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + env.w.proxyAddr = ln.Addr().String() + var out bytes.Buffer + statusProxy(context.Background(), env, &out) + if want := "\nproxy: felis-velocity active, " + env.w.proxyAddr + " accepts connections\n"; out.String() != want { + t.Errorf("proxy listening: %q, want %q", out.String(), want) + } + ln.Close() + out.Reset() + statusProxy(context.Background(), env, &out) + if want := "\nproxy: felis-velocity active, " + env.w.proxyAddr + " refuses connections (dial tcp " + env.w.proxyAddr + ": "; !strings.HasPrefix(out.String(), want) { + t.Errorf("proxy gone: %q, want it to start %q", out.String(), want) + } +} + +func TestStatusBackups(t *testing.T) { + env := statusTestEnv(t) + status := func() string { + var out bytes.Buffer + statusBackups(env, &out) + return out.String() + } + for _, tc := range []struct { + what, dir, want string + }{ + {"no -backup-dir", "", " database: not checked (felis-watchdog.service names no -backup-dir)\n"}, + {"an empty directory", t.TempDir(), " database: none in %s\n"}, + {"a file where the directory should be", filepath.Join(env.w.backupDir, dbbackup.BundleName(env.now.Add(-9*time.Hour), dbbackup.LabelDaily)), " database: cannot read %s: "}, + } { + env.w.backupDir = tc.dir + want := tc.want + if strings.Contains(want, "%s") { + want = fmt.Sprintf(want, tc.dir) + } + if got := status(); !strings.Contains(got, want) { + t.Errorf("%s: %q, want %q", tc.what, got, want) + } + } + + for _, tc := range []struct { + what string + status *offsite.Status + raw string + want string + }{ + {"never synced", nil, "", " off-site: never synced\n"}, + {"never succeeded", &offsite.Status{LastAttempt: env.now.Add(-time.Hour), LastError: "403 Forbidden"}, "", " off-site: never succeeded; last attempt 1h0m ago: 403 Forbidden\n"}, + {"the last attempt failed", &offsite.Status{LastAttempt: env.now.Add(-30 * time.Minute), LastSuccess: env.now.Add(-26 * time.Hour), LastError: "timeout", Bucket: "felis-dr"}, "", + " off-site: last good sync 26h0m ago to felis-dr; the last attempt, 30m ago, failed: timeout\n"}, + {"an error an earlier attempt left", &offsite.Status{LastAttempt: env.now.Add(-2 * time.Hour), LastSuccess: env.now.Add(-time.Hour), LastError: "timeout", Bucket: "felis-dr"}, "", + " off-site: last good sync 1h0m ago to felis-dr\n"}, + {"an unreadable record", nil, "{", " off-site: unexpected end of JSON input\n"}, + } { + os.Remove(env.w.offsiteStatus) + switch { + case tc.status != nil: + if err := offsite.WriteStatus(env.w.offsiteStatus, *tc.status); err != nil { + t.Fatal(err) + } + case tc.raw != "": + writeTestFile(t, env.w.offsiteStatus, tc.raw, 0o600) + } + if got := status(); !strings.HasSuffix(got, tc.want) { + t.Errorf("%s: %q, want it to end %q", tc.what, got, tc.want) + } + } +} + +func TestStatusHost(t *testing.T) { + env := statusTestEnv(t) + dir := t.TempDir() + env.w.diskPaths = "/nonexistent-felis-status-test," + dir + "," + dir + env.meminfo = filepath.Join(dir, "meminfo") + var out bytes.Buffer + statusHost(env, &out) + lines := strings.Split(strings.TrimSuffix(out.String(), "\n"), "\n") + if len(lines) != 3 || lines[0] != "" || lines[1] != "host:" || !strings.HasPrefix(lines[2], " disk "+dir+": ") || !strings.HasSuffix(lines[2], "% free)") { + t.Errorf("host: %q, want one line for %s (a path on a disk already shown, and one that is not there, print none) and no memory line", lines, dir) + } + + writeTestFile(t, env.meminfo, "MemTotal: 4096 kB\ngarbage\nMemAvailable: 1024 kB\n", 0o644) + if total, avail, ok := readMeminfo(env.meminfo); !ok || total != 4096*1024 || avail != 1024*1024 { + t.Errorf("readMeminfo = %d, %d, %v", total, avail, ok) + } + writeTestFile(t, env.meminfo, "MemAvailable: 1024 kB\n", 0o644) + if _, _, ok := readMeminfo(env.meminfo); ok { + t.Error("readMeminfo without MemTotal: ok") + } +} + +func TestStatusWatchdog(t *testing.T) { + env := statusTestEnv(t) + status := func() string { + var out bytes.Buffer + statusWatchdog(context.Background(), env, &out) + return out.String() + } + run := env.run + env.run = func(ctx context.Context, name string, args ...string) ([]byte, error) { + if len(args) > 0 && args[0] == "show" { + return []byte("Result=success\nExecMainExitTimestamp=\n"), nil + } + return run(ctx, name, args...) + } + if err := watchdog.SaveState(env.w.statePath, &watchdog.State{Alerts: map[string]*watchdog.Alert{ + "proxy": {Finding: watchdog.Finding{Key: "proxy", Severity: watchdog.Critical}, FirstSeen: env.now.Add(-time.Hour), ClearedAt: env.now.Add(-time.Minute)}, + }}); err != nil { + t.Fatal(err) + } + if got, want := status(), "\nwatchdog:\n timer active\n alerts: none open\n"; got != want { + t.Errorf("a watchdog that has not run and has nothing open: %q, want %q", got, want) + } + + writeTestFile(t, env.w.statePath, "{", 0o600) + if got := status(); !strings.Contains(got, "\n alerts: unknown (") { + t.Errorf("an unreadable state: %q", got) + } + + if err := os.Remove(filepath.Join(env.unitDir, "felis-watchdog.timer")); err != nil { + t.Fatal(err) + } + if got, want := status(), "\nwatchdog:\n not installed: nothing checks this host\n"; got != want { + t.Errorf("no watchdog timer: %q, want %q", got, want) + } +} + +// Rows print by name in whatever order the API server lists them, a server +// the operator has not reconciled yet reads as dashes, and open alerts print +// by key in whatever order the state's map yields them. +func TestStatusOrder(t *testing.T) { + env := statusTestEnv(t) + objs := append(statusClusterObjects(), + &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: "felis-operator", Namespace: "felis"}}, + &v1alpha1.MinecraftServer{ObjectMeta: metav1.ObjectMeta{Name: "fresh", Namespace: "minecraft"}}) + env.cl = fake.NewClientBuilder().WithScheme(newSystemServerScheme(t)).WithObjects(objs...). + WithInterceptorFuncs(interceptor.Funcs{List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + // The fake lists by name; the other way round, then. + if err := c.List(ctx, list, opts...); err != nil { + return err + } + items, err := meta.ExtractList(list) + if err != nil { + return err + } + slices.Reverse(items) + return meta.SetList(list, items) + }}).Build() + var out bytes.Buffer + printStatus(context.Background(), env, &out) + var rows []string + for _, l := range strings.Split(out.String(), "\n") { + if f := strings.Fields(l); len(f) > 0 && (strings.HasPrefix(f[0], "felis-") || slices.Contains([]string{"creative", "fresh", "lobby", "survival"}, f[0])) { + rows = append(rows, strings.Join(f, " ")) + } + } + want := []string{ + "felis-api 1/1 ready felis-api:v1.4.0@0123456789ab restarts 2", + "felis-operator 0/1 ready - restarts 0", + "creative - Stopped Stopped - none", + "fresh - - - - none", + "lobby lobby Running Starting - none", + "survival - Running Running 2/20 3h0m ago", + } + if !slices.Equal(rows, want) { + t.Errorf("rows %q, want %q:\n%s", rows, want, out.String()) + } + + alerts := map[string]*watchdog.Alert{} + for i, key := range []string{"a", "b", "c", "d"} { + alerts[key] = &watchdog.Alert{Finding: watchdog.Finding{Key: key, Severity: watchdog.Warning}, FirstSeen: env.now.Add(-time.Duration(i+1) * time.Minute)} + } + if err := watchdog.SaveState(env.w.statePath, &watchdog.State{Alerts: alerts}); err != nil { + t.Fatal(err) + } + wantAlerts := " alerts: 4 open (sudo felis doctor says where to look)\n" + + " a (warning, seen 1m ago, not mailed yet)\n b (warning, seen 2m ago, not mailed yet)\n" + + " c (warning, seen 3m ago, not mailed yet)\n d (warning, seen 4m ago, not mailed yet)\n" + // A map starts its walk at random: fifty walks all in order by chance + // is out of the question. + for range 50 { + out.Reset() + statusWatchdog(context.Background(), env, &out) + if !strings.HasSuffix(out.String(), wantAlerts) { + t.Fatalf("alerts %q, want them to end %q", out.String(), wantAlerts) + } + } +} + +// A selector the API server would have refused picks no pods. +func TestPodRestartsBadSelector(t *testing.T) { + pods := []corev1.Pod{{Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{{RestartCount: 3}}}}} + if n := podRestarts(&metav1.LabelSelector{MatchExpressions: []metav1.LabelSelectorRequirement{{Key: "app", Operator: "Near"}}}, pods); n != 0 { + t.Errorf("podRestarts with a bad selector = %d, want 0", n) + } + if n := podRestarts(&metav1.LabelSelector{}, pods); n != 3 { + t.Errorf("podRestarts with a selector that picks all = %d, want 3", n) + } +} diff --git a/cmd/felis/supportbundle.go b/cmd/felis/supportbundle.go new file mode 100644 index 0000000..232f4c1 --- /dev/null +++ b/cmd/felis/supportbundle.go @@ -0,0 +1,862 @@ +package main + +import ( + "archive/tar" + "bytes" + "compress/gzip" + "context" + "errors" + "flag" + "fmt" + "io" + "io/fs" + "net/url" + "os" + "path" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" + "text/tabwriter" + "time" + + "felis.lolicon.best/internal/apis/felis/v1alpha1" + "felis.lolicon.best/internal/config" + "felis.lolicon.best/internal/platform" + "felis.lolicon.best/internal/watchdog" + appsv1 "k8s.io/api/apps/v1" + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + networkingv1 "k8s.io/api/networking/v1" + "k8s.io/apimachinery/pkg/api/meta" + "k8s.io/client-go/kubernetes" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/yaml" +) + +// supportBundleDir is where felis support-bundle writes unless told otherwise. +const supportBundleDir = "/var/lib/felis/support" + +// redacted stands in for every value the bundle leaves out. +const redacted = "" + +// minScrubLen is the shortest known secret value the bundle searches for: a +// shorter one would blank ordinary words, and every secret the installer +// generates is far longer. +const minScrubLen = 8 + +// hostSecretSources are the files on a Felis host that hold secrets; the +// bundle reads them only to take each value out of what it collects. +var hostSecretSources = secretSources{ + envFiles: []string{"/etc/felis/secrets.env", defaultOffsiteEnvFile}, + valueFiles: []string{hostSMTPPasswordPath, hostUploadsS3AccessKeyPath, hostUploadsS3SecretKeyPath, defaultHeartbeatFile, "/opt/felis/velocity/forwarding.secret"}, + propsFiles: []string{"/opt/felis/velocity/plugins/felis-link/felis-link.properties"}, + tokenFiles: []string{"/var/lib/rancher/k3s/server/token", "/var/lib/rancher/k3s/server/agent-token"}, +} + +// cmdSupportBundle collects what someone helping with this host needs into +// one tar.gz: felis status and felis doctor, the logs of the control plane, +// the builds and the systemd units, the cluster's workloads and events, and a +// summary of the configuration. It never collects a Secret, a ConfigMap, the +// contents of a configuration file, the database or a world, and takes every +// value of the host's secret files out of what it does collect. Game server +// logs, which carry player names, addresses and chat, only with -server-logs. +func cmdSupportBundle(args []string, stdout, stderr io.Writer) int { + fs := flag.NewFlagSet("support-bundle", flag.ContinueOnError) + fs.SetOutput(stderr) + outDir := fs.String("o", supportBundleDir, "directory to write the bundle to (created mode 0700 when missing)") + logLines := fs.Int64("log-lines", 2000, "lines kept from the end of each pod log and each unit's journal") + since := fs.Duration("since", 48*time.Hour, "how far back each unit's journal is read") + serverLogs := fs.Bool("server-logs", false, "also collect the game servers' own logs, which carry player names, IP addresses and chat") + unitDir := fs.String("systemd-dir", systemdUnitDir, "where the installer's systemd units are") + if err := fs.Parse(args); err != nil { + if errors.Is(err, flag.ErrHelp) { + return 0 + } + return 2 + } + if os.Geteuid() != 0 { + fmt.Fprintln(stderr, "felis support-bundle: run as root (sudo felis support-bundle): it reads root-only logs and state") + return 1 + } + b := hostSupportBundle(*unitDir, *logLines, *since, *serverLogs) + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + path, err := b.write(ctx, *outDir) + if err != nil { + fmt.Fprintf(stderr, "felis support-bundle: %v\n", err) + return 1 + } + size := int64(0) + if st, err := os.Stat(path); err == nil { + size = st.Size() + } + fmt.Fprintf(stdout, "wrote %s (%s, mode 0600)\n", path, humanSize(size)) + fmt.Fprintln(stdout, "MANIFEST.txt inside says what it holds and what was taken out. Read it through before you send it anywhere:") + fmt.Fprintln(stdout, "redaction finds this host's known secrets and the common ways a secret is logged, and a secret logged another way stays in.") + if !*serverLogs { + fmt.Fprintln(stdout, "Game server logs are left out; -server-logs adds them (player names, IP addresses, chat).") + } + return 0 +} + +func humanSize(n int64) string { + switch { + case n >= 1<<20: + return fmt.Sprintf("%.1f MiB", float64(n)/(1<<20)) + case n >= 1<<10: + return fmt.Sprintf("%.1f KiB", float64(n)/(1<<10)) + } + return fmt.Sprintf("%d B", n) +} + +// shortDuration is d as Duration.String writes it, less the zero minutes and +// seconds after whole hours or minutes: 48h, and 90m as 1h30m. +func shortDuration(d time.Duration) string { + s := d.String() + if strings.HasSuffix(s, "m0s") { + s = strings.TrimSuffix(s, "0s") + } + if strings.HasSuffix(s, "h0m") { + s = strings.TrimSuffix(s, "0m") + } + return s +} + +// supportBundle is one collection and what it reads the host through. +type supportBundle struct { + host string + now time.Time + unitDir string + // w is how felis-watchdog.service runs the watchdog, wErr why it could + // not be read (w then holds the defaults). + w watchdogFlags + wErr error + cfg *config.Config // nil when cfgErr + cfgErr error + cl client.Client // nil when clErr + clErr error + logs func(ctx context.Context, ns, pod, container string, previous bool) ([]byte, error) + run func(ctx context.Context, name string, args ...string) ([]byte, error) + backups func(ctx context.Context) (map[string]time.Time, error) + doctor func(ctx context.Context, out io.Writer) + secrets secretSources + logLines int64 + since time.Duration + serverLogs bool + // Host files read whole, and the directory whose listing is kept. + meminfo, osRelease, procVersion, stateDir string +} + +func hostSupportBundle(unitDir string, logLines int64, since time.Duration, serverLogs bool) *supportBundle { + host, _ := os.Hostname() + b := &supportBundle{ + host: host, now: time.Now(), unitDir: unitDir, run: hostCommand, secrets: hostSecretSources, + logLines: logLines, since: since, serverLogs: serverLogs, + meminfo: "/proc/meminfo", osRelease: "/etc/os-release", procVersion: "/proc/version", stateDir: "/etc/felis", + } + var found bool + b.w, found, b.wErr = watchdogUnitFlags(filepath.Join(unitDir, "felis-watchdog.service")) + if b.wErr == nil && !found { + b.wErr = fmt.Errorf("%s is not installed; read the watchdog's defaults", filepath.Join(unitDir, "felis-watchdog.service")) + } + if b.cfg, b.cfgErr = config.Load(b.w.cfgPath); b.cfgErr == nil { + cfg := b.cfg + b.backups = func(ctx context.Context) (map[string]time.Time, error) { + return newestWorldBackups(ctx, cfg.Database.URL) + } + } else { + b.cfg = nil + } + if b.cl, b.clErr = buildSystemServerClient(); b.clErr != nil { + b.cl = nil + } else if rc, err := hostRESTConfig(); err != nil { + b.clErr = err + b.cl = nil + } else if cs, err := kubernetes.NewForConfig(rc); err != nil { + b.clErr = err + b.cl = nil + } else { + limit := int64(8 << 20) + b.logs = func(ctx context.Context, ns, pod, container string, previous bool) ([]byte, error) { + return cs.CoreV1().Pods(ns).GetLogs(pod, &corev1.PodLogOptions{ + Container: container, Previous: previous, Timestamps: true, TailLines: &logLines, LimitBytes: &limit, + }).DoRaw(ctx) + } + } + b.doctor = func(ctx context.Context, out io.Writer) { + runDoctor(ctx, doctorEnv{unitDir: unitDir, run: hostCommand, now: b.now, host: host}, out) + } + return b +} + +// bundleWriter streams scrubbed files into the archive and keeps what went +// wrong while collecting, for MANIFEST.txt. +type bundleWriter struct { + tw *tar.Writer + prefix string + now time.Time + scrub *scrubber + errs []string +} + +// logText adds a log, or a report that quotes errors: known secrets and +// anything logged as one come out. +func (bw *bundleWriter) logText(name string, data []byte) error { + return bw.add(name, bw.scrub.text(data)) +} + +// plain adds a file whose secrets were already taken out by structure (the +// cluster's objects, the configuration summary) or that names none (the +// release, disk use, addresses): known secret values still come out, and +// nothing else is rewritten. +func (bw *bundleWriter) plain(name string, data []byte) error { + return bw.add(name, bw.scrub.values(data)) +} + +func (bw *bundleWriter) add(name string, data []byte) error { + hdr := &tar.Header{Name: path.Join(bw.prefix, name), Mode: 0o600, Size: int64(len(data)), ModTime: bw.now, Typeflag: tar.TypeReg} + if err := bw.tw.WriteHeader(hdr); err != nil { + return err + } + _, err := bw.tw.Write(data) + return err +} + +func (bw *bundleWriter) failed(what string, err error) { + bw.errs = append(bw.errs, string(bw.scrub.text([]byte(fmt.Sprintf("%s: %v", what, err))))) +} + +// write collects the bundle into dir and returns its path. +func (b *supportBundle) write(ctx context.Context, dir string) (string, error) { + if _, err := os.Stat(dir); errors.Is(err, fs.ErrNotExist) { + if err := os.MkdirAll(dir, 0o700); err != nil { + return "", err + } + } + stamp := b.now.UTC().Format("20060102T150405Z") + base := fmt.Sprintf("felis-support-%s-%s", safeName(b.host), stamp) + final := filepath.Join(dir, base+".tar.gz") + f, err := os.CreateTemp(dir, "."+base+".*.partial") // mode 0600 + if err != nil { + return "", err + } + keep := false + defer func() { + if !keep { + f.Close() + os.Remove(f.Name()) + } + }() + gz := gzip.NewWriter(f) + bw := &bundleWriter{tw: tar.NewWriter(gz), prefix: base, now: b.now, scrub: b.scrubber()} + if err := b.collect(ctx, bw); err != nil { + return "", err + } + if err := bw.tw.Close(); err != nil { + return "", err + } + if err := gz.Close(); err != nil { + return "", err + } + if err := f.Sync(); err != nil { + return "", err + } + if err := f.Close(); err != nil { + return "", err + } + if err := os.Rename(f.Name(), final); err != nil { + return "", err + } + keep = true + return final, nil +} + +// safeName keeps a host name usable in a file name. +func safeName(s string) string { + s = strings.Map(func(r rune) rune { + if r == '-' || r == '.' || r == '_' || (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') { + return r + } + return '_' + }, s) + if s == "" { + return "host" + } + return s +} + +func (b *supportBundle) collect(ctx context.Context, bw *bundleWriter) error { + var buf bytes.Buffer + cmdVersion(nil, &buf, io.Discard) + for _, p := range []string{b.osRelease, b.procVersion} { + if raw, err := os.ReadFile(p); err == nil { + fmt.Fprintf(&buf, "\n# %s\n%s", p, raw) + } + } + if err := bw.plain("version.txt", buf.Bytes()); err != nil { + return err + } + + buf.Reset() + switch { + case b.cfgErr != nil: + fmt.Fprintf(&buf, "felis status: the configuration did not load: %v\n", b.cfgErr) + default: + printStatus(ctx, statusEnv{ + cfg: b.cfg, w: b.w, cl: b.cl, clErr: b.clErr, backups: b.backups, run: b.run, + unitDir: b.unitDir, meminfo: b.meminfo, host: b.host, now: b.now, + }, &buf) + } + if err := bw.logText("status.txt", buf.Bytes()); err != nil { + return err + } + buf.Reset() + b.doctor(ctx, &buf) + if err := bw.logText("doctor.txt", buf.Bytes()); err != nil { + return err + } + if b.cfg != nil { + if err := bw.plain("config.txt", configSummary(b.cfg, b.w.cfgPath)); err != nil { + return err + } + } + if err := b.collectHost(ctx, bw); err != nil { + return err + } + if err := b.collectJournal(ctx, bw); err != nil { + return err + } + if b.cl == nil { + bw.failed("cluster", b.clErr) + } else if err := b.collectCluster(ctx, bw); err != nil { + return err + } + return bw.add("MANIFEST.txt", b.manifest(bw)) +} + +func (b *supportBundle) collectHost(ctx context.Context, bw *bundleWriter) error { + commands := []struct { + name string + argv []string + }{ + {"host/systemd-units.txt", []string{"systemctl", "list-units", "--all", "--no-pager", "--plain", "felis-*", "k3s.service"}}, + {"host/systemd-timers.txt", []string{"systemctl", "list-timers", "--all", "--no-pager", "felis-*"}}, + {"host/df.txt", []string{"df", "-h"}}, + {"host/addresses.txt", []string{"ip", "-brief", "address"}}, + } + for _, c := range commands { + out, err := b.run(ctx, c.argv[0], c.argv[1:]...) + if err != nil && len(out) == 0 { + bw.failed(strings.Join(c.argv, " "), err) + continue + } + if err := bw.plain(c.name, out); err != nil { + return err + } + } + if raw, err := os.ReadFile(b.meminfo); err == nil { + if err := bw.plain("host/meminfo.txt", raw); err != nil { + return err + } + } + listing, err := dirListing(b.stateDir) + if err != nil { + bw.failed("list "+b.stateDir, err) + return nil + } + return bw.plain("host/etc-felis.txt", listing) +} + +// dirListing names every file under dir with its mode, size and time, and +// holds nothing of what is in them. +func dirListing(dir string) ([]byte, error) { + var buf bytes.Buffer + tw := tabwriter.NewWriter(&buf, 0, 0, 2, ' ', 0) + fmt.Fprintf(tw, "# %s: names, modes, sizes and times only; no contents\n", dir) + err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + info, err := d.Info() + if err != nil { + return err + } + fmt.Fprintf(tw, "%s\t%d\t%s\t%s\n", info.Mode(), info.Size(), info.ModTime().UTC().Format(time.RFC3339), p) + return nil + }) + tw.Flush() + return buf.Bytes(), err +} + +func (b *supportBundle) collectJournal(ctx context.Context, bw *bundleWriter) error { + units, _ := filepath.Glob(filepath.Join(b.unitDir, "felis-*.service")) + if _, err := os.Stat(filepath.Join(b.unitDir, "k3s.service")); err == nil { + units = append(units, filepath.Join(b.unitDir, "k3s.service")) + } + since := "@" + strconv.FormatInt(b.now.Add(-b.since).Unix(), 10) + for _, u := range units { + unit := filepath.Base(u) + out, err := b.run(ctx, "journalctl", "-u", unit, "--since", since, "-n", strconv.FormatInt(b.logLines, 10), "--no-pager", "-o", "short-iso") + if err != nil && len(out) == 0 { + bw.failed("journalctl -u "+unit, err) + continue + } + if err := bw.logText("journal/"+strings.TrimSuffix(unit, ".service")+".log", out); err != nil { + return err + } + } + return nil +} + +// bundleNamespaces are the namespaces whose objects and logs the bundle +// collects: the control plane, the builds and the game servers. +func (b *supportBundle) bundleNamespaces() (control, build, minecraft string) { + control, build, minecraft = b.w.controlNS, platform.DefaultBuildNamespace, platform.DefaultMinecraftNamespace + if b.cfg != nil { + if b.cfg.Registry.BuildNamespace != "" { + build = b.cfg.Registry.BuildNamespace + } + if b.cfg.K8s.Namespace != "" { + minecraft = b.cfg.K8s.Namespace + } + } + return control, build, minecraft +} + +func (b *supportBundle) collectCluster(ctx context.Context, bw *bundleWriter) error { + control, build, minecraft := b.bundleNamespaces() + dump := func(name string, list client.ObjectList, opts ...client.ListOption) error { + if err := b.cl.List(ctx, list, opts...); err != nil { + bw.failed("list "+name, err) + return nil + } + redactList(list) + out, err := yaml.Marshal(list) + if err != nil { + bw.failed("encode "+name, err) + return nil + } + return bw.plain(name, out) + } + if err := dump("cluster/nodes.yaml", &corev1.NodeList{}); err != nil { + return err + } + if err := dump("cluster/persistentvolumes.yaml", &corev1.PersistentVolumeList{}); err != nil { + return err + } + if err := dump("cluster/minecraftservers.yaml", &v1alpha1.MinecraftServerList{}, client.InNamespace(minecraft)); err != nil { + return err + } + for _, ns := range []string{control, build, minecraft} { + for _, k := range []struct { + name string + list client.ObjectList + }{ + {"pods", &corev1.PodList{}}, + {"deployments", &appsv1.DeploymentList{}}, + {"statefulsets", &appsv1.StatefulSetList{}}, + {"jobs", &batchv1.JobList{}}, + {"cronjobs", &batchv1.CronJobList{}}, + {"services", &corev1.ServiceList{}}, + {"persistentvolumeclaims", &corev1.PersistentVolumeClaimList{}}, + {"networkpolicies", &networkingv1.NetworkPolicyList{}}, + {"events", &corev1.EventList{}}, + } { + if err := dump("cluster/"+ns+"/"+k.name+".yaml", k.list, client.InNamespace(ns)); err != nil { + return err + } + } + } + + var all corev1.PodList + if err := b.cl.List(ctx, &all); err != nil { + bw.failed("list every pod", err) + } else if err := bw.plain("cluster/pods-all-namespaces.txt", podTable(all.Items, b.now)); err != nil { + return err + } + + for _, ns := range []string{control, build, minecraft} { + var pods corev1.PodList + if err := b.cl.List(ctx, &pods, client.InNamespace(ns)); err != nil { + continue // already recorded by the dump above + } + for _, p := range pods.Items { + if err := b.collectPodLogs(ctx, bw, p, ns == minecraft && !b.serverLogs); err != nil { + return err + } + } + } + return nil +} + +// collectPodLogs keeps the tail of each container's log, and of its previous +// run when it restarted. initOnly keeps only the init containers: a game +// server's own log carries player names, addresses and chat. +func (b *supportBundle) collectPodLogs(ctx context.Context, bw *bundleWriter, p corev1.Pod, initOnly bool) error { + type ctr struct { + name string + restarts int32 + } + var ctrs []ctr + restarts := map[string]int32{} + for _, cs := range append(append([]corev1.ContainerStatus(nil), p.Status.InitContainerStatuses...), p.Status.ContainerStatuses...) { + restarts[cs.Name] = cs.RestartCount + } + for _, c := range p.Spec.InitContainers { + ctrs = append(ctrs, ctr{c.Name, restarts[c.Name]}) + } + if !initOnly { + for _, c := range p.Spec.Containers { + ctrs = append(ctrs, ctr{c.Name, restarts[c.Name]}) + } + } + for _, c := range ctrs { + for _, previous := range []bool{false, true} { + if previous && c.restarts == 0 { + continue + } + name := fmt.Sprintf("logs/%s/%s/%s.log", p.Namespace, p.Name, c.name) + if previous { + name = fmt.Sprintf("logs/%s/%s/%s.previous.log", p.Namespace, p.Name, c.name) + } + out, err := b.logs(ctx, p.Namespace, p.Name, c.name, previous) + if err != nil { + bw.failed(name, err) + continue + } + if err := bw.logText(name, out); err != nil { + return err + } + } + } + return nil +} + +// podTable is every pod on the node, one line each. +func podTable(pods []corev1.Pod, now time.Time) []byte { + sort.Slice(pods, func(i, j int) bool { + if pods[i].Namespace != pods[j].Namespace { + return pods[i].Namespace < pods[j].Namespace + } + return pods[i].Name < pods[j].Name + }) + var buf bytes.Buffer + tw := tabwriter.NewWriter(&buf, 0, 0, 2, ' ', 0) + fmt.Fprintln(tw, "NAMESPACE\tNAME\tPHASE\tREADY\tRESTARTS\tAGE") + for _, p := range pods { + var ready, restarts int32 + for _, cs := range p.Status.ContainerStatuses { + if cs.Ready { + ready++ + } + restarts += cs.RestartCount + } + age := "-" + if !p.CreationTimestamp.IsZero() { + age = now.Sub(p.CreationTimestamp.Time).Round(time.Minute).String() + } + fmt.Fprintf(tw, "%s\t%s\t%s\t%d/%d\t%d\t%s\n", p.Namespace, p.Name, p.Status.Phase, ready, len(p.Spec.Containers), restarts, age) + } + tw.Flush() + return buf.Bytes() +} + +// redactList takes out of every object what the bundle must not carry: the +// literal env values of pod specs and of MinecraftServers (valueFrom +// references stay, naming the Secret without its contents), the +// last-applied-configuration annotation that repeats them, and managedFields. +func redactList(list client.ObjectList) { + items, err := meta.ExtractList(list) + if err != nil { + return + } + for _, it := range items { + if acc, err := meta.Accessor(it); err == nil { + acc.SetManagedFields(nil) + if ann := acc.GetAnnotations(); ann != nil { + delete(ann, corev1.LastAppliedConfigAnnotation) + acc.SetAnnotations(ann) + } + } + switch o := it.(type) { + case *corev1.Pod: + redactPodSpec(&o.Spec) + case *appsv1.Deployment: + redactPodSpec(&o.Spec.Template.Spec) + case *appsv1.StatefulSet: + redactPodSpec(&o.Spec.Template.Spec) + case *batchv1.Job: + redactPodSpec(&o.Spec.Template.Spec) + case *batchv1.CronJob: + redactPodSpec(&o.Spec.JobTemplate.Spec.Template.Spec) + case *v1alpha1.MinecraftServer: + for i := range o.Spec.Env { + if o.Spec.Env[i].Value != "" { + o.Spec.Env[i].Value = redacted + } + } + } + } +} + +func redactPodSpec(s *corev1.PodSpec) { + blank := func(cs []corev1.Container) { + for i := range cs { + for j := range cs[i].Env { + if cs[i].Env[j].Value != "" { + cs[i].Env[j].Value = redacted + } + } + } + } + blank(s.InitContainers) + blank(s.Containers) + for i := range s.EphemeralContainers { + for j := range s.EphemeralContainers[i].Env { + if s.EphemeralContainers[i].Env[j].Value != "" { + s.EphemeralContainers[i].Env[j].Value = redacted + } + } + } +} + +// configSummary is felis.toml without a single credential: the hostnames, +// namespaces and which features are on. Fields are picked one by one, so a +// field added later stays out until someone decides it is safe. +func configSummary(c *config.Config, path string) []byte { + var buf bytes.Buffer + line := func(k string, v any) { fmt.Fprintf(&buf, "%-34s %v\n", k, v) } + fmt.Fprintf(&buf, "# a summary of %s; no password, key or token is in it\n", path) + line("server.root_domain", c.Server.RootDomain) + line("server.listen", c.Server.Listen) + db := "(unparsable)" + if u, err := url.Parse(c.Database.URL); err == nil { + db = u.Scheme + "://" + u.User.Username() + "@" + u.Host + u.Path + } + line("database.url (no password)", db) + line("database.deployment", c.Database.Deployment) + line("velocity.public_ip", c.Velocity.PublicIP) + line("velocity.game_port", c.Velocity.GamePort) + line("velocity.login_image", c.Velocity.LoginImage) + line("velocity.lobby_image", c.Velocity.LobbyImage) + line("auth.panel_hostname", c.Auth.PanelHostname) + line("auth.admin_hostname", c.Auth.AdminHostname) + line("auth.client_ip_header", c.Auth.ClientIPHeader) + line("k8s.namespace", c.K8s.Namespace) + line("k8s.egress_mode", c.K8s.EgressMode) + line("k8s.metallb_pool", c.K8s.MetalLBPool) + line("registry.url", c.Registry.URL) + line("registry.build_namespace", c.Registry.BuildNamespace) + line("registry.trivy_db_repository", c.Registry.TrivyDBRepository) + line("registry.build_user_namespaces", c.Registry.BuildUserNamespaces) + line("registry.build_runtime_class", c.Registry.BuildRuntimeClass) + line("registry.max_concurrent_builds", c.Registry.MaxConcurrentBuilds) + line("registry.user_uploads_context", c.Registry.UserUploadsContext) + line("archive.store", c.Archive.Store) + line("archive.local_path", c.Archive.LocalPath) + line("archive.retention", c.Archive.Retention) + line("archive.scheduled_every", c.Archive.ScheduledEvery) + line("archive.scheduled_keep", c.Archive.ScheduledKeep) + line("offsite (configured)", c.Offsite.Enabled()) + if c.Offsite.Enabled() { + line("offsite.endpoint", c.Offsite.Endpoint) + line("offsite.bucket", c.Offsite.Bucket) + line("offsite.prefix", c.Offsite.Prefix) + } + line("smtp.host", c.SMTP.Host) + if c.SMTP.Host != "" { + line("smtp.port", c.SMTP.Port) + line("smtp.require_tls", c.SMTP.TLSRequired()) + line("smtp.max_per_hour", c.SMTP.MaxPerHour) + } + for i, s := range c.AuthSources { + line(fmt.Sprintf("auth_source[%d]", i), s.Tag+" "+s.Prefix+" "+s.URL) + } + return buf.Bytes() +} + +func (b *supportBundle) manifest(bw *bundleWriter) []byte { + control, build, minecraft := b.bundleNamespaces() + var buf bytes.Buffer + fmt.Fprintf(&buf, "Felis support bundle\nhost %s, collected %s, felis %s\n\n", b.host, b.now.UTC().Format(time.RFC3339), resolvedVersion()) + fmt.Fprintf(&buf, `What it holds: + status.txt, doctor.txt felis status and felis doctor at collection time + version.txt the release, the OS and the kernel + config.txt a summary of felis.toml: hostnames, namespaces, which features are on + host/ systemd units and timers, disk use, memory, addresses, and the names, + modes, sizes and times of the files under %s (not what is in them) + journal/ up to %d lines per Felis unit and k3s, from the last %s + logs/ up to %d lines of each container of the pods in %s and %s, and of the + init containers of the game server pods in %s; the previous run too + where a container restarted + cluster/ nodes, volumes, the MinecraftServers, and the pods, workloads, services, + volume claims, network policies and events of %s, %s and %s +`, b.stateDir, b.logLines, shortDuration(b.since), b.logLines, control, build, minecraft, control, build, minecraft) + if b.serverLogs { + fmt.Fprintln(&buf, "\nThe game servers' own logs are in logs/ (-server-logs): they carry player names, IP addresses and chat.") + } else { + fmt.Fprintln(&buf, "\nThe game servers' own logs are left out (they carry player names, IP addresses and chat; -server-logs adds them).") + } + fmt.Fprint(&buf, ` +Never collected: Kubernetes Secrets and ConfigMaps, what is in /etc/felis or any configuration +file, the database, worlds, uploads. + +Taken out: +`) + if len(bw.scrub.sources) > 0 { + fmt.Fprintf(&buf, " - %d secret values, wherever they appear, read from:\n", len(bw.scrub.vals)) + for _, s := range bw.scrub.sources { + fmt.Fprintf(&buf, " %s\n", s) + } + } else { + fmt.Fprintln(&buf, " - no secret file was found on this host to take values from") + } + fmt.Fprint(&buf, ` - passwords in URLs, private keys, Bearer and Basic credentials + - in logs and command output, whatever follows password=, secret=, token=, api_key=, + access_key=, private_key= or credentials= (and the same with a colon) + - every literal env value in pod specs and MinecraftServers (valueFrom references stay) + +Read it through before you send it anywhere: redaction finds this host's known secrets and the +common ways a secret is logged, and a secret logged another way stays in. +`) + if b.wErr != nil { + fmt.Fprintf(&buf, "\nThe watchdog's settings: %v\n", b.wErr) + } + if len(bw.errs) > 0 { + fmt.Fprintln(&buf, "\nNot collected:") + for _, e := range bw.errs { + fmt.Fprintf(&buf, " - %s\n", e) + } + } + // Its own words name what is redacted, and would be redacted themselves; + // what it quotes was scrubbed as it was recorded. + return buf.Bytes() +} + +// secretSources are files that hold secrets, by how each is laid out. +type secretSources struct { + envFiles []string // KEY=VALUE lines, every value a secret (a commented-out one too) + valueFiles []string // one secret, the whole file + propsFiles []string // key=value lines; the keys naming a token, secret, password or key hold one + tokenFiles []string // k3s join tokens: the whole token and its secret part after the last ':' +} + +// scrubber takes secrets out of what the bundle collects. +type scrubber struct { + vals []string // longest first, so a secret that contains another goes whole + sources []string // the files vals came from +} + +var ( + pemPrivateKey = regexp.MustCompile(`(?s)-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----.*?-----END [A-Z0-9 ]*PRIVATE KEY-----`) + urlUserinfo = regexp.MustCompile(`([A-Za-z][A-Za-z0-9+.-]*://[^/\s:@]*:)[^/\s@]+@`) + authScheme = regexp.MustCompile(`(?i)\b(bearer|basic)\s+[A-Za-z0-9._~+/=-]{8,}`) + secretAssign = regexp.MustCompile(`(?i)((?:password|passwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|credentials?)[A-Za-z0-9_.-]*"?[ \t]*[=:][ \t]*"?)([^\s"',;&]{4,})`) + secretPropKey = regexp.MustCompile(`(?i)token|secret|password|key`) +) + +func (b *supportBundle) scrubber() *scrubber { + s := &scrubber{} + s.readSources(b.secrets) + if b.cfg != nil { + if u, err := url.Parse(b.cfg.Database.URL); err == nil { + if pw, ok := u.User.Password(); ok { + s.add(pw) + } + } + for _, ref := range []string{ + b.cfg.Velocity.ServiceTokenRef, b.cfg.SMTP.PasswordRef, + b.cfg.Offsite.AccessKeyRef, b.cfg.Offsite.SecretKeyRef, b.cfg.Offsite.KeyRef, + b.cfg.Registry.S3.AccessKeyRef, b.cfg.Registry.S3.SecretKeyRef, + b.cfg.Archive.S3.AccessKeyRef, b.cfg.Archive.S3.SecretKeyRef, + } { + if ref != "" { + s.add(os.Getenv(ref)) + } + } + } + if st, err := watchdog.LoadState(watchdog.NewestState(b.w.statePath, b.w.fallbackState)); err == nil { + s.add(st.SMTPPassword) + } + return s +} + +func (s *scrubber) add(v string) bool { + v = strings.TrimSpace(v) + if len(v) < minScrubLen { + return false + } + for _, have := range s.vals { + if have == v { + return true + } + } + s.vals = append(s.vals, v) + sort.SliceStable(s.vals, func(i, j int) bool { return len(s.vals[i]) > len(s.vals[j]) }) + return true +} + +func (s *scrubber) readSources(src secretSources) { + read := func(p string, take func(content string) bool) { + raw, err := os.ReadFile(p) + if err != nil { + return + } + if take(string(raw)) { + s.sources = append(s.sources, p) + } + } + keyValues := func(content string, keep func(key string) bool) bool { + found := false + for _, line := range strings.Split(content, "\n") { + k, v, ok := strings.Cut(line, "=") + if !ok || !keep(strings.TrimSpace(k)) { + continue + } + v = strings.TrimSpace(v) + if len(v) >= 2 && (v[0] == '\'' || v[0] == '"') && v[len(v)-1] == v[0] { + v = v[1 : len(v)-1] + } + found = s.add(v) || found + } + return found + } + for _, p := range src.envFiles { + read(p, func(c string) bool { return keyValues(c, func(string) bool { return true }) }) + } + for _, p := range src.propsFiles { + read(p, func(c string) bool { return keyValues(c, secretPropKey.MatchString) }) + } + for _, p := range src.valueFiles { + read(p, func(c string) bool { return s.add(c) }) + } + for _, p := range src.tokenFiles { + read(p, func(c string) bool { + c = strings.TrimSpace(c) + whole := s.add(c) + part := false + if i := strings.LastIndex(c, ":"); i >= 0 { + part = s.add(c[i+1:]) + } + return whole || part + }) + } +} + +// values takes every known secret value out of data. +func (s *scrubber) values(data []byte) []byte { + t := pemPrivateKey.ReplaceAllString(string(data), "") + for _, v := range s.vals { + t = strings.ReplaceAll(t, v, redacted) + } + t = urlUserinfo.ReplaceAllString(t, "${1}"+redacted+"@") + t = authScheme.ReplaceAllString(t, "${1} "+redacted) + return []byte(t) +} + +// text is values, and whatever a log names as a secret as well. +func (s *scrubber) text(data []byte) []byte { + return secretAssign.ReplaceAll(s.values(data), []byte("${1}"+redacted)) +} diff --git a/cmd/felis/supportbundle_test.go b/cmd/felis/supportbundle_test.go new file mode 100644 index 0000000..cdb080b --- /dev/null +++ b/cmd/felis/supportbundle_test.go @@ -0,0 +1,433 @@ +package main + +import ( + "archive/tar" + "compress/gzip" + "context" + "errors" + "fmt" + "io" + "os" + "path/filepath" + "sort" + "strings" + "testing" + "time" + + "felis.lolicon.best/internal/apis/felis/v1alpha1" + "felis.lolicon.best/internal/config" + "felis.lolicon.best/internal/watchdog" + appsv1 "k8s.io/api/apps/v1" + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client/fake" +) + +// The secrets a bundle host holds, each of which must be nowhere in the bundle. +var plantedSecrets = map[string]string{ + "secrets.env SERVICE_TOKEN": "svc-token-planted-0a1b2c3d", + "secrets.env DB_PASSWORD": "db-pass-planted-4e5f6a7b", + "offsite.env FELIS_OFFSITE_KEY": "offsite-key-planted+8c9d/0e1f=", + "smtp-password": "smtp-pass-planted-2a3b", + "felis-link.properties token": "props-token-planted-4c5d", + "k3s token secret part": "k3s-secret-part-planted-6e7f", + "watchdog state relay password": "cached-relay-pw-planted-8a9b", + "pod env literal": "env-literal-planted-0c1d", + "deployment env literal": "deploy-env-planted-2e3f", + "MinecraftServer env literal": "cr-env-planted-4a5b", + "last-applied annotation": "annotation-planted-6c7d", + "logged password": "hunter2-planted-8e9f", + "logged bearer": "bearerplanted0a1b2c3d", + "URL password": "urlpass-planted-4e5f", + "token in an error": "errtoken-planted-0f1e", + "felis.host.toml DB password": "cfg-db-pass-planted-1c2d", + "service token by its env ref": "env-ref-token-planted-3e4f", + "init container env literal": "init-env-planted-5a6b", + "ephemeral container env": "ephemeral-env-planted-7c8d", + "statefulset env literal": "sts-env-planted-9e0f", + "job env literal": "job-env-planted-1a2b", + "cronjob env literal": "cronjob-env-planted-3c4d", + "commented-out offsite key": "old-offsite-key-planted-5e6f", + "doctor quoting a password": "doctor-pw-planted-7a8b", + "status quoting a password": "status-pw-planted-9c0d", + "journal quoting a password": "journal-pw-planted-1e2f", +} + +// bundleHost is a Felis host for felis support-bundle: its secret files, its +// watchdog unit and state, a cluster with a control-plane pod that restarted +// and a game server, and logs and a journal that name secrets. +func bundleHost(t *testing.T) *supportBundle { + t.Helper() + p := plantedSecrets + dir := t.TempDir() + etc := filepath.Join(dir, "etc-felis") + for _, d := range []string{etc, filepath.Join(dir, "systemd"), filepath.Join(dir, "k3s")} { + if err := os.MkdirAll(d, 0o700); err != nil { + t.Fatal(err) + } + } + writeTestFile(t, filepath.Join(etc, "secrets.env"), "SERVICE_TOKEN="+p["secrets.env SERVICE_TOKEN"]+"\nDB_PASSWORD="+p["secrets.env DB_PASSWORD"]+"\nSHORT=abc\n", 0o600) + writeTestFile(t, filepath.Join(etc, "offsite.env"), "# before the rotation\n# FELIS_OFFSITE_KEY="+p["commented-out offsite key"]+"\nFELIS_OFFSITE_KEY='"+p["offsite.env FELIS_OFFSITE_KEY"]+"'\n", 0o600) + writeTestFile(t, filepath.Join(etc, "smtp-password"), p["smtp-password"]+"\n", 0o600) + writeTestFile(t, filepath.Join(etc, "felis-link.properties"), "api-base-url=http://10.43.0.10:8081\nservice-token="+p["felis-link.properties token"]+"\nroot-domain=games.example.org\n", 0o640) + writeTestFile(t, filepath.Join(dir, "k3s", "token"), "K10deadbeefcafe::server:"+p["k3s token secret part"]+"\n", 0o600) + cfgPath := filepath.Join(etc, "felis.host.toml") + writeTestFile(t, cfgPath, "[database]\nurl = \"postgres://felis:"+p["felis.host.toml DB password"]+"@127.0.0.1:1/felis?sslmode=disable&connect_timeout=1\"\n"+ + "[server]\nroot_domain = \"games.example.org\"\n[archive]\nstore = \"tarLocal\"\n[k8s]\negress_mode = \"nodeport\"\n"+ + "[velocity]\nservice_token_ref = \"FELIS_BUNDLE_TEST_SERVICE_TOKEN\"\n", 0o600) + t.Setenv("FELIS_BUNDLE_TEST_SERVICE_TOKEN", p["service token by its env ref"]) + statePath := filepath.Join(dir, "state.json") + if err := watchdog.SaveState(statePath, &watchdog.State{SMTPPassword: p["watchdog state relay password"]}); err != nil { + t.Fatal(err) + } + unitDir := filepath.Join(dir, "systemd") + writeTestFile(t, filepath.Join(unitDir, "felis-watchdog.service"), "[Service]\nExecStart=/usr/local/bin/felis watchdog -config "+cfgPath+" -state "+statePath+"\n", 0o644) + writeTestFile(t, filepath.Join(unitDir, "felis-offsite.service"), "[Unit]\n", 0o644) + writeTestFile(t, filepath.Join(unitDir, "k3s.service"), "[Unit]\n", 0o644) + meminfo := filepath.Join(dir, "meminfo") + writeTestFile(t, meminfo, "MemTotal: 8000000 kB\nMemAvailable: 2000000 kB\n", 0o644) + + cfg, err := config.Load(cfgPath) + if err != nil { + t.Fatal(err) + } + var w watchdogFlags + w, _, err = watchdogUnitFlags(filepath.Join(unitDir, "felis-watchdog.service")) + if err != nil { + t.Fatal(err) + } + w.diskPaths = "/nonexistent-felis-bundle-test" + + secretEnv := corev1.EnvVar{Name: "FELIS_SMTP_PASSWORD", ValueFrom: &corev1.EnvVarSource{SecretKeyRef: &corev1.SecretKeySelector{ + LocalObjectReference: corev1.LocalObjectReference{Name: "felis-smtp"}, Key: "password"}}} + replicas := int32(1) + cl := fake.NewClientBuilder().WithScheme(newSystemServerScheme(t)).WithObjects( + &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{ + Name: "felis-api-7d9", Namespace: "felis", + Annotations: map[string]string{corev1.LastAppliedConfigAnnotation: `{"env":"` + p["last-applied annotation"] + `"}`, "felis.lolicon.best/kept": "yes"}, + ManagedFields: []metav1.ManagedFieldsEntry{{Manager: "kubectl-client-side-apply"}}, + }, + Spec: corev1.PodSpec{ + InitContainers: []corev1.Container{{Name: "wait-db", Image: "busybox", Env: []corev1.EnvVar{{Name: "FELIS_INIT_SETTING", Value: p["init container env literal"]}}}}, + Containers: []corev1.Container{{Name: "api", Image: "felis-api:v1", Env: []corev1.EnvVar{ + {Name: "FELIS_PLAIN_SETTING", Value: p["pod env literal"]}, secretEnv, + }}}, + EphemeralContainers: []corev1.EphemeralContainer{{EphemeralContainerCommon: corev1.EphemeralContainerCommon{ + Name: "debug", Env: []corev1.EnvVar{{Name: "FELIS_DEBUG_SETTING", Value: p["ephemeral container env"]}}}}}, + }, + Status: corev1.PodStatus{Phase: corev1.PodRunning, ContainerStatuses: []corev1.ContainerStatus{{Name: "api", RestartCount: 1, Ready: true}}}, + }, + &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{Name: "felis-api", Namespace: "felis"}, + Spec: appsv1.DeploymentSpec{Replicas: &replicas, Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"app": "felis-api"}}, + Template: corev1.PodTemplateSpec{Spec: corev1.PodSpec{Containers: []corev1.Container{{Name: "api", Env: []corev1.EnvVar{ + {Name: "FELIS_DEPLOY_SETTING", Value: p["deployment env literal"]}, + }}}}}}, + }, + &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Name: "survival-0", Namespace: "minecraft"}, + Spec: corev1.PodSpec{ + InitContainers: []corev1.Container{{Name: "prepare-data"}, {Name: "egress-gate"}}, + Containers: []corev1.Container{{Name: "server"}}, + }, + Status: corev1.PodStatus{Phase: corev1.PodRunning}, + }, + &v1alpha1.MinecraftServer{ + ObjectMeta: metav1.ObjectMeta{Name: "survival", Namespace: "minecraft"}, + Spec: v1alpha1.MinecraftServerSpec{Env: []v1alpha1.EnvVar{{Name: "DISCORD_WEBHOOK", Value: p["MinecraftServer env literal"]}}}, + }, + &appsv1.StatefulSet{ + ObjectMeta: metav1.ObjectMeta{Name: "felis-postgres", Namespace: "felis"}, + Spec: appsv1.StatefulSetSpec{Template: envTemplate("FELIS_STS_SETTING", p["statefulset env literal"])}, + }, + &batchv1.Job{ + ObjectMeta: metav1.ObjectMeta{Name: "build-1", Namespace: "felis-build"}, + Spec: batchv1.JobSpec{Template: envTemplate("FELIS_JOB_SETTING", p["job env literal"])}, + }, + &batchv1.CronJob{ + ObjectMeta: metav1.ObjectMeta{Name: "felis-reaper", Namespace: "felis"}, + Spec: batchv1.CronJobSpec{JobTemplate: batchv1.JobTemplateSpec{Spec: batchv1.JobSpec{Template: envTemplate("FELIS_CRON_SETTING", p["cronjob env literal"])}}}, + }, + &corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "felis-1"}}, + ).Build() + + secretLog := fmt.Sprintf("started with SERVICE_TOKEN=%s\nlogin password=%s ok\nAuthorization: Bearer %s\ndial postgres://felis:%s@db:5432/felis\n", + p["secrets.env SERVICE_TOKEN"], p["logged password"], p["logged bearer"], p["URL password"]) + return &supportBundle{ + host: "felis-test", now: time.Date(2026, 9, 27, 12, 0, 0, 0, time.UTC), unitDir: unitDir, + w: w, cfg: cfg, cl: cl, + logs: func(_ context.Context, ns, pod, container string, previous bool) ([]byte, error) { + if container == "wait-db" { + return nil, errors.New("container \"wait-db\" is waiting to start; token=" + p["token in an error"]) + } + return []byte(fmt.Sprintf("log of %s/%s/%s previous=%v\n%s", ns, pod, container, previous, secretLog)), nil + }, + run: func(_ context.Context, name string, args ...string) ([]byte, error) { + switch name { + case "journalctl": + return []byte("journal of " + args[1] + "\nFELIS_OFFSITE_KEY=" + p["offsite.env FELIS_OFFSITE_KEY"] + "\nrelay " + p["smtp-password"] + " refused\n" + + "relay login with the cached " + p["watchdog state relay password"] + "\ndatabase auth failed for " + p["felis.host.toml DB password"] + + "\nservice token " + p["service token by its env ref"] + " rejected\nnode joined with " + p["k3s token secret part"] + "\n" + + "old copies sealed with " + p["commented-out offsite key"] + "\nretrying with password=" + p["journal quoting a password"] + "\n"), nil + case "systemctl", "df", "ip": + return []byte(name + " output\n"), nil + } + return nil, errors.New("unexpected " + name) + }, + backups: func(context.Context) (map[string]time.Time, error) { + return nil, errors.New("connect: password=" + p["status quoting a password"]) + }, + doctor: func(_ context.Context, out io.Writer) { + fmt.Fprintf(out, "doctor report; the k3s token K10deadbeefcafe::server:%s leaked here\nprobe said password=%s\n", p["k3s token secret part"], p["doctor quoting a password"]) + }, + secrets: secretSources{ + envFiles: []string{filepath.Join(etc, "secrets.env"), filepath.Join(etc, "offsite.env"), filepath.Join(etc, "absent.env")}, + valueFiles: []string{filepath.Join(etc, "smtp-password"), filepath.Join(etc, "uploads-s3-secret-key")}, + propsFiles: []string{filepath.Join(etc, "felis-link.properties")}, + tokenFiles: []string{filepath.Join(dir, "k3s", "token")}, + }, + logLines: 500, since: 48 * time.Hour, + meminfo: meminfo, osRelease: filepath.Join(dir, "os-release"), procVersion: filepath.Join(dir, "version"), stateDir: etc, + } +} + +// envTemplate is a pod template whose one container sets name to a literal. +func envTemplate(name, value string) corev1.PodTemplateSpec { + return corev1.PodTemplateSpec{Spec: corev1.PodSpec{Containers: []corev1.Container{{Name: "main", Env: []corev1.EnvVar{{Name: name, Value: value}}}}}} +} + +// readBundle is every file in the bundle at path, by its name inside the +// bundle's directory, and each one's mode. +func readBundle(t *testing.T, path string) (map[string]string, map[string]int64) { + t.Helper() + f, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer f.Close() + gz, err := gzip.NewReader(f) + if err != nil { + t.Fatal(err) + } + tr := tar.NewReader(gz) + files, modes := map[string]string{}, map[string]int64{} + prefix := strings.TrimSuffix(filepath.Base(path), ".tar.gz") + "/" + for { + hdr, err := tr.Next() + if err == io.EOF { + break + } + if err != nil { + t.Fatal(err) + } + name, ok := strings.CutPrefix(hdr.Name, prefix) + if !ok { + t.Errorf("%s is outside the bundle's directory %s", hdr.Name, prefix) + } + raw, err := io.ReadAll(tr) + if err != nil { + t.Fatal(err) + } + files[name], modes[name] = string(raw), hdr.Mode + } + return files, modes +} + +func TestSupportBundle(t *testing.T) { + b := bundleHost(t) + out := filepath.Join(t.TempDir(), "support") + path, err := b.write(context.Background(), out) + if err != nil { + t.Fatal(err) + } + if want := filepath.Join(out, "felis-support-felis-test-20260927T120000Z.tar.gz"); path != want { + t.Errorf("path %s, want %s", path, want) + } + for p, want := range map[string]os.FileMode{out: 0o700 | os.ModeDir, path: 0o600} { + if st, err := os.Stat(p); err != nil || st.Mode() != want { + t.Errorf("%s: mode %v (err %v), want %v", p, st.Mode(), err, want) + } + } + if left, _ := filepath.Glob(filepath.Join(out, ".*partial")); len(left) != 0 { + t.Errorf("left behind %v", left) + } + + files, modes := readBundle(t, path) + var names []string + for n := range files { + names = append(names, n) + if modes[n] != 0o600 { + t.Errorf("%s: mode %o in the archive, want 600", n, modes[n]) + } + } + sort.Strings(names) + want := []string{ + "MANIFEST.txt", + "cluster/felis-build/cronjobs.yaml", "cluster/felis-build/deployments.yaml", "cluster/felis-build/events.yaml", "cluster/felis-build/jobs.yaml", + "cluster/felis-build/networkpolicies.yaml", "cluster/felis-build/persistentvolumeclaims.yaml", "cluster/felis-build/pods.yaml", + "cluster/felis-build/services.yaml", "cluster/felis-build/statefulsets.yaml", + "cluster/felis/cronjobs.yaml", "cluster/felis/deployments.yaml", "cluster/felis/events.yaml", "cluster/felis/jobs.yaml", + "cluster/felis/networkpolicies.yaml", "cluster/felis/persistentvolumeclaims.yaml", "cluster/felis/pods.yaml", + "cluster/felis/services.yaml", "cluster/felis/statefulsets.yaml", + "cluster/minecraft/cronjobs.yaml", "cluster/minecraft/deployments.yaml", "cluster/minecraft/events.yaml", "cluster/minecraft/jobs.yaml", + "cluster/minecraft/networkpolicies.yaml", "cluster/minecraft/persistentvolumeclaims.yaml", "cluster/minecraft/pods.yaml", + "cluster/minecraft/services.yaml", "cluster/minecraft/statefulsets.yaml", + "cluster/minecraftservers.yaml", "cluster/nodes.yaml", "cluster/persistentvolumes.yaml", "cluster/pods-all-namespaces.txt", + "config.txt", "doctor.txt", + "host/addresses.txt", "host/df.txt", "host/etc-felis.txt", "host/meminfo.txt", "host/systemd-timers.txt", "host/systemd-units.txt", + "journal/felis-offsite.log", "journal/felis-watchdog.log", "journal/k3s.log", + "logs/felis/felis-api-7d9/api.log", "logs/felis/felis-api-7d9/api.previous.log", + "logs/minecraft/survival-0/egress-gate.log", "logs/minecraft/survival-0/prepare-data.log", + "status.txt", "version.txt", + } + if strings.Join(names, "\n") != strings.Join(want, "\n") { + t.Errorf("bundle holds\n %s\nwant\n %s", strings.Join(names, "\n "), strings.Join(want, "\n ")) + } + + for what, secret := range plantedSecrets { + for n, body := range files { + if strings.Contains(body, secret) { + t.Errorf("%s (%s) is in %s:\n%s", what, secret, n, body) + } + } + } + + // What was taken out leaves what a reader needs around it. + for name, wants := range map[string][]string{ + "logs/felis/felis-api-7d9/api.previous.log": {"log of felis/felis-api-7d9/api previous=true\n", "SERVICE_TOKEN=\n", "login password= ok\n", "Authorization: Bearer \n", "postgres://felis:@db:5432/felis\n"}, + "journal/k3s.log": {"journal of k3s.service\n", "FELIS_OFFSITE_KEY=\n", "relay refused\n", + "relay login with the cached \n", "database auth failed for \n", "service token rejected\n", "node joined with \n"}, + "cluster/felis/statefulsets.yaml": {"name: FELIS_STS_SETTING\n"}, + "cluster/felis/cronjobs.yaml": {"name: FELIS_CRON_SETTING\n"}, + "cluster/felis-build/jobs.yaml": {"name: FELIS_JOB_SETTING\n"}, + "cluster/felis/pods.yaml": {"name: FELIS_PLAIN_SETTING\n value: \n", "name: FELIS_SMTP_PASSWORD\n valueFrom:\n secretKeyRef:\n key: password\n name: felis-smtp\n", "felis.lolicon.best/kept: \"yes\"", "name: FELIS_INIT_SETTING\n", "name: FELIS_DEBUG_SETTING\n"}, + "cluster/felis/deployments.yaml": {"name: FELIS_DEPLOY_SETTING\n value: \n"}, + "cluster/minecraftservers.yaml": {"name: DISCORD_WEBHOOK\n value: \n"}, + "config.txt": {fmt.Sprintf("%-34s %s\n", "server.root_domain", "games.example.org"), fmt.Sprintf("%-34s %s\n", "database.url (no password)", "postgres://felis@127.0.0.1:1/felis")}, + "doctor.txt": {"the k3s token leaked here\nprobe said password=\n"}, + "status.txt": {" world backups unknown: connect: password=\n"}, + "host/etc-felis.txt": {"secrets.env\n", "smtp-password\n", "felis-link.properties\n"}, + "MANIFEST.txt": { + "The game servers' own logs are left out", + " journal/ up to 500 lines per Felis unit and k3s, from the last 48h\n", + " - 11 secret values, wherever they appear, read from:\n", + "secrets.env\n", "offsite.env\n", "smtp-password\n", "felis-link.properties\n", "token\n", + "logs/felis/felis-api-7d9/wait-db.log: container \"wait-db\" is waiting to start; token=\n", + // Its own words about what is redacted come through whole. + " - passwords in URLs, private keys, Bearer and Basic credentials\n", + "access_key=, private_key= or credentials= (and the same with a colon)\n", + "Read it through before you send it anywhere", + }, + } { + for _, w := range wants { + if !strings.Contains(files[name], w) { + t.Errorf("%s lacks %q:\n%s", name, w, files[name]) + } + } + } + for _, gone := range []string{"managedFields", "last-applied-configuration"} { + if strings.Contains(files["cluster/felis/pods.yaml"], gone) { + t.Errorf("pods.yaml keeps %s:\n%s", gone, files["cluster/felis/pods.yaml"]) + } + } + if strings.Contains(files["MANIFEST.txt"], "absent.env") || strings.Contains(files["MANIFEST.txt"], "uploads-s3-secret-key") { + t.Errorf("MANIFEST names files this host does not have:\n%s", files["MANIFEST.txt"]) + } +} + +// -server-logs adds the game servers' own logs, and says so. +func TestSupportBundleServerLogs(t *testing.T) { + b := bundleHost(t) + b.serverLogs = true + path, err := b.write(context.Background(), t.TempDir()) + if err != nil { + t.Fatal(err) + } + files, _ := readBundle(t, path) + if _, ok := files["logs/minecraft/survival-0/server.log"]; !ok { + t.Error("no server.log with -server-logs") + } + if !strings.Contains(files["MANIFEST.txt"], "The game servers' own logs are in logs/ (-server-logs)") { + t.Errorf("MANIFEST does not say server logs are in:\n%s", files["MANIFEST.txt"]) + } +} + +// With the cluster and the configuration gone the bundle still holds what the +// host shows, and MANIFEST says what it could not collect. +func TestSupportBundleWithTheClusterDown(t *testing.T) { + b := bundleHost(t) + b.cl, b.clErr = nil, errors.New("connection refused") + b.cfg, b.cfgErr = nil, errors.New("felis.host.toml: no such file") + b.wErr = errors.New("felis-watchdog.service is not installed; read the watchdog's defaults") + path, err := b.write(context.Background(), t.TempDir()) + if err != nil { + t.Fatal(err) + } + files, _ := readBundle(t, path) + for _, n := range []string{"status.txt", "doctor.txt", "host/etc-felis.txt", "journal/k3s.log"} { + if _, ok := files[n]; !ok { + t.Errorf("no %s", n) + } + } + for n := range files { + if strings.HasPrefix(n, "cluster/") || strings.HasPrefix(n, "logs/") || n == "config.txt" { + t.Errorf("%s with no cluster and no configuration", n) + } + } + if !strings.Contains(files["status.txt"], "felis status: the configuration did not load: felis.host.toml: no such file") { + t.Errorf("status.txt:\n%s", files["status.txt"]) + } + if !strings.Contains(files["MANIFEST.txt"], "\nThe watchdog's settings: felis-watchdog.service is not installed; read the watchdog's defaults\n") || + !strings.Contains(files["MANIFEST.txt"], " - cluster: connection refused\n") { + t.Errorf("MANIFEST.txt:\n%s", files["MANIFEST.txt"]) + } +} + +func TestShortDuration(t *testing.T) { + for d, want := range map[time.Duration]string{ + 48 * time.Hour: "48h", + 90 * time.Minute: "1h30m", + 45 * time.Minute: "45m", + 30 * time.Second: "30s", + time.Hour + 30*time.Second: "1h0m30s", + } { + if got := shortDuration(d); got != want { + t.Errorf("shortDuration(%v) = %q, want %q", d, got, want) + } + } +} + +func TestScrubber(t *testing.T) { + s := &scrubber{} + for _, v := range []string{"known-secret-value", "known-secret-value-longer", "short", "known-secret-value"} { + s.add(v) + } + if got := strings.Join(s.vals, ","); got != "known-secret-value-longer,known-secret-value" { + t.Errorf("kept %q; want each value once, longest first, none under %d characters", s.vals, minScrubLen) + } + for in, want := range map[string]string{ + "a known-secret-value-longer b": "a b", + "a known-secret-value b": "a b", + "password=abcd1234 next": "password= next", + `{"token": "abcd1234"}`: `{"token": ""}`, + "SMTP_PASSWORD: s3cr3t!x": "SMTP_PASSWORD: ", + "x-api-key=zzzz9999&q=1": "x-api-key=&q=1", + "Authorization: Basic dXNlcjpwYXNzd29yZA==": "Authorization: Basic ", + "s3://AKIA:secretpart123@bucket/key": "s3://AKIA:@bucket/key", + "https://user@host/path": "https://user@host/path", + "-----BEGIN EC PRIVATE KEY-----\nMHc\n-----END EC PRIVATE KEY-----": "", + "tokens: 3": "tokens: 3", + "read the relay password (keeping the cached one)": "read the relay password (keeping the cached one)", + `secrets "felis-smtp" not found`: `secrets "felis-smtp" not found`, + } { + if got := string(s.text([]byte(in))); got != want { + t.Errorf("text(%q) = %q, want %q", in, got, want) + } + } + // Structured files keep what only looks like a secret. + if got := string(s.values([]byte("secretName: felis-forwarding-secret and known-secret-value"))); got != "secretName: felis-forwarding-secret and " { + t.Errorf("values() = %q", got) + } +} diff --git a/cmd/felis/systemservers.go b/cmd/felis/systemservers.go index 61c1e95..7acfabd 100644 --- a/cmd/felis/systemservers.go +++ b/cmd/felis/systemservers.go @@ -15,6 +15,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" clientgoscheme "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" "k8s.io/client-go/tools/clientcmd" "k8s.io/client-go/util/retry" ctrl "sigs.k8s.io/controller-runtime" @@ -243,6 +244,15 @@ func buildSystemServerClient() (client.Client, error) { if err := v1alpha1.AddToScheme(scheme); err != nil { return nil, err } + cfg, err := hostRESTConfig() + if err != nil { + return nil, err + } + return client.New(cfg, client.Options{Scheme: scheme}) +} + +// hostRESTConfig is the cluster connection buildSystemServerClient describes. +func hostRESTConfig() (*rest.Config, error) { cfg, err := ctrl.GetConfig() if err != nil { cfg, err = clientcmd.BuildConfigFromFlags("", hostBootstrapKubeconfigPath) @@ -250,7 +260,7 @@ func buildSystemServerClient() (client.Client, error) { return nil, fmt.Errorf("no reachable kubeconfig (tried in-cluster/$KUBECONFIG/~/.kube and %s): %w", hostBootstrapKubeconfigPath, err) } } - return client.New(cfg, client.Options{Scheme: scheme}) + return cfg, nil } // systemServerOutcome records what ensureSystemServers did with one service so diff --git a/cmd/felis/watchdog.go b/cmd/felis/watchdog.go index 7971505..d63c1f6 100644 --- a/cmd/felis/watchdog.go +++ b/cmd/felis/watchdog.go @@ -35,22 +35,8 @@ const proxyFor = 3 * time.Minute func cmdWatchdog(args []string, stdout, stderr io.Writer) int { fs := flag.NewFlagSet("watchdog", flag.ContinueOnError) fs.SetOutput(stderr) - cfgPath := fs.String("config", "/etc/felis/felis.toml", "path to felis.toml (the host copy, which reaches PostgreSQL on 127.0.0.1)") - statePath := fs.String("state", "/var/lib/felis/watchdog/state.json", "state kept between runs (root only: it caches the relay password)") - fallbackState := fs.String("fallback-state", watchdog.FallbackStatePath, "where a run keeps its state while -state cannot be written, so what it mailed is not mailed again (tmpfs: until the host restarts; \"\" keeps none)") - smtpPasswordFile := fs.String("smtp-password-file", hostSMTPPasswordPath, "the relay password `felis setup` keeps on the host; the felis-smtp Secret stands in while it is missing") - quietPath := fs.String("quiet-file", "/run/felis/watchdog-quiet-until", "Unix time before which nothing is mailed; the installer writes it while it restarts things on purpose") - backupDir := fs.String("backup-dir", "/var/lib/felis/db-backups", `control-plane database backups to check for freshness ("" skips the check)`) - diskPaths := fs.String("disk-paths", "/,/var/lib/rancher/k3s,/var/lib/felis", "comma-separated paths whose filesystems must keep free space") - certDirs := fs.String("k3s-cert-dirs", strings.Join(watchdog.K3sCertDirs, ","), `k3s certificate directories whose *.crt files must not be near expiry ("" skips the check)`) - proxyAddr := fs.String("proxy-addr", "", `game proxy address to dial, e.g. 127.0.0.1:25565 ("" skips the check)`) - nodeIP := fs.String("node-ip", "", `the node address the install was made on, which must stay on this host ("" skips the check)`) - controlNS := fs.String("control-namespace", platform.DefaultControlNamespace, "namespace of the control plane") - offsiteStatus := fs.String("offsite-status", offsite.DefaultStatusFile, "the record `felis offsite sync` leaves, checked when [offsite] is configured") - toolsStatus := fs.String("build-tools-status", defaultBuildToolsStatus, "the record `felis mirror-build-tools` leaves, checked when builds scan against the registry's DB copy") - dryRun := fs.Bool("dry-run", false, "print every finding and the mail that is due; send nothing and keep the state as it was") - heartbeatFile := fs.String("heartbeat-file", defaultHeartbeatFile, "file holding the heartbeat URL each run pings, a dead man's switch at a monitoring service that alerts when the pings stop (no file pings nothing)") - unitFailed := fs.Bool("unit-failed", false, "report a failed run of felis-watchdog.service instead of checking; felis-watchdog-failed.service runs this through OnFailure=") + var w watchdogFlags + w.register(fs) if err := fs.Parse(args); err != nil { if errors.Is(err, flag.ErrHelp) { return 0 @@ -58,20 +44,20 @@ func cmdWatchdog(args []string, stdout, stderr io.Writer) int { return 2 } now := time.Now() - if *unitFailed { + if w.unitFailed { return watchdogUnitFailed(unitFailedRun{ - cfgPath: *cfgPath, statePath: *statePath, fallbackPath: *fallbackState, quietPath: *quietPath, - offsiteStatus: *offsiteStatus, heartbeatFile: *heartbeatFile, + cfgPath: w.cfgPath, statePath: w.statePath, fallbackPath: w.fallbackState, quietPath: w.quietPath, + offsiteStatus: w.offsiteStatus, heartbeatFile: w.heartbeatFile, result: os.Getenv("MONITOR_SERVICE_RESULT"), exitStatus: os.Getenv("MONITOR_EXIT_STATUS"), send: watchdogSender, client: http.DefaultClient, now: now, }, stdout, stderr) } - cfg, err := config.Load(*cfgPath) + cfg, err := config.Load(w.cfgPath) if err != nil { fmt.Fprintf(stderr, "felis watchdog: %v\n", err) return 1 } - loadPath := watchdog.NewestState(*statePath, *fallbackState) + loadPath := watchdog.NewestState(w.statePath, w.fallbackState) state, aside, err := watchdog.RecoverState(loadPath, now) if err != nil { fmt.Fprintf(stderr, "felis watchdog: %v\n", err) @@ -81,72 +67,19 @@ func cmdWatchdog(args []string, stdout, stderr io.Writer) int { defer cancel() var report watchdog.Report - add := func(f *watchdog.Finding) { - if f != nil { - report.Findings = append(report.Findings, *f) - } - } if aside != "" { fmt.Fprintf(stderr, "felis watchdog: %s was unreadable; moved it to %s and started over\n", loadPath, aside) - f := watchdog.StateSetAside(aside) - add(&f) - } - - // The cluster: one unreachable API server stands in for every check behind it. - minecraftNS := cfg.K8s.Namespace - if minecraftNS == "" { - minecraftNS = platform.DefaultMinecraftNamespace - } - cl, err := buildSystemServerClient() - var found []watchdog.Finding - if err == nil { - found, err = watchdog.Cluster{Client: cl, ControlNamespace: *controlNS, MinecraftNamespace: minecraftNS}.Check(ctx, now) - } - if err != nil { - f := watchdog.KubeAPIDown(err) - add(&f) - report.Unknown = append(report.Unknown, watchdog.ClusterPrefixes...) - } else { - report.Findings = append(report.Findings, found...) + report.Findings = append(report.Findings, watchdog.StateSetAside(aside)) } + cl, owners, ownersErr := watchdogProbes(ctx, w, cfg, now, &report) if cfg.SMTP.Host != "" { - var secrets client.Client - if err == nil { - secrets = cl - } - refreshSMTPPassword(ctx, *smtpPasswordFile, secrets, *controlNS, state, stderr) + refreshSMTPPassword(ctx, w.smtpPasswordFile, cl, w.controlNS, state, stderr) } state.Relay = cachedRelay(cfg.SMTP) - - if recipients, err := ownerEmails(ctx, cfg.Database.URL); err != nil { - f := watchdog.PostgresDown(err) - add(&f) - } else { - state.Recipients = recipients + if ownersErr == nil { + state.Recipients = owners } - if *proxyAddr != "" { - add(proxyFinding(ctx, *proxyAddr)) - } - if *backupDir != "" { - add(watchdog.BackupFinding(*backupDir, now)) - } - if cfg.Offsite.Enabled() { - add(watchdog.OffsiteFinding(*offsiteStatus, now)) - } - if usesMirroredScanDB(cfg) { - add(watchdog.ScanDBFinding(*toolsStatus, now)) - } - report.Findings = append(report.Findings, watchdog.DiskFindings(splitList(*diskPaths))...) - add(watchdog.MemoryFinding("/proc/meminfo")) - add(watchdog.CertFinding(splitList(*certDirs), now)) - if *nodeIP != "" { - if held, err := watchdog.HostAddresses(); err == nil { - add(watchdog.AddressFinding(*nodeIP, held)) - } - } - add(watchdog.ClockFinding(watchdog.ClockStatus())) - if len(report.Findings) == 0 { fmt.Fprintln(stdout, "felis watchdog: every check passed") } @@ -158,13 +91,13 @@ func cmdWatchdog(args []string, stdout, stderr io.Writer) int { host, _ := os.Hostname() subject, body := plan.Message(host, now) beat := heartbeat{ - standby: standsBy(cfg.Offsite.Enabled(), *offsiteStatus), - quiet: now.Before(watchdog.QuietUntil(*quietPath)), + standby: standsBy(cfg.Offsite.Enabled(), w.offsiteStatus), + quiet: now.Before(watchdog.QuietUntil(w.quietPath)), } - if beat.url, err = readHeartbeatURL(*heartbeatFile); err != nil { + if beat.url, err = readHeartbeatURL(w.heartbeatFile); err != nil { fmt.Fprintf(stderr, "felis watchdog: %v; pinging no heartbeat\n", err) } - if *dryRun { + if w.dryRun { if plan.Empty() { fmt.Fprintln(stdout, "felis watchdog: nothing is due to be mailed") } else { @@ -177,8 +110,8 @@ func cmdWatchdog(args []string, stdout, stderr io.Writer) int { } m := configMailer(cfg.SMTP, state.SMTPPassword, watchdogSender) - unheard, mailFailed := m.deliver(ctx, state, plan, subject, body, mailHold(*quietPath, cfg.Offsite.Enabled(), *offsiteStatus, now), now, stdout, stderr) - saveErr := watchdog.SaveStateOr(*statePath, *fallbackState, state) + unheard, mailFailed := m.deliver(ctx, state, plan, subject, body, mailHold(w.quietPath, cfg.Offsite.Enabled(), w.offsiteStatus, now), now, stdout, stderr) + saveErr := watchdog.SaveStateOr(w.statePath, w.fallbackState, state) if saveErr != nil { fmt.Fprintf(stderr, "felis watchdog: save state: %v\n", saveErr) } @@ -191,6 +124,93 @@ func cmdWatchdog(args []string, stdout, stderr io.Writer) int { return 0 } +// watchdogFlags are felis watchdog's flags. felis doctor reads them back from +// the ExecStart= line of felis-watchdog.service, so it checks what the timer's +// runs check, with the same paths. +type watchdogFlags struct { + cfgPath, statePath, fallbackState, smtpPasswordFile, quietPath string + backupDir, diskPaths, certDirs, proxyAddr, nodeIP, controlNS string + offsiteStatus, toolsStatus, heartbeatFile string + dryRun, unitFailed bool +} + +func (w *watchdogFlags) register(fs *flag.FlagSet) { + fs.StringVar(&w.cfgPath, "config", "/etc/felis/felis.toml", "path to felis.toml (the host copy, which reaches PostgreSQL on 127.0.0.1)") + fs.StringVar(&w.statePath, "state", "/var/lib/felis/watchdog/state.json", "state kept between runs (root only: it caches the relay password)") + fs.StringVar(&w.fallbackState, "fallback-state", watchdog.FallbackStatePath, "where a run keeps its state while -state cannot be written, so what it mailed is not mailed again (tmpfs: until the host restarts; \"\" keeps none)") + fs.StringVar(&w.smtpPasswordFile, "smtp-password-file", hostSMTPPasswordPath, "the relay password `felis setup` keeps on the host; the felis-smtp Secret stands in while it is missing") + fs.StringVar(&w.quietPath, "quiet-file", "/run/felis/watchdog-quiet-until", "Unix time before which nothing is mailed; the installer writes it while it restarts things on purpose") + fs.StringVar(&w.backupDir, "backup-dir", "/var/lib/felis/db-backups", `control-plane database backups to check for freshness ("" skips the check)`) + fs.StringVar(&w.diskPaths, "disk-paths", "/,/var/lib/rancher/k3s,/var/lib/felis", "comma-separated paths whose filesystems must keep free space") + fs.StringVar(&w.certDirs, "k3s-cert-dirs", strings.Join(watchdog.K3sCertDirs, ","), `k3s certificate directories whose *.crt files must not be near expiry ("" skips the check)`) + fs.StringVar(&w.proxyAddr, "proxy-addr", "", `game proxy address to dial, e.g. 127.0.0.1:25565 ("" skips the check)`) + fs.StringVar(&w.nodeIP, "node-ip", "", `the node address the install was made on, which must stay on this host ("" skips the check)`) + fs.StringVar(&w.controlNS, "control-namespace", platform.DefaultControlNamespace, "namespace of the control plane") + fs.StringVar(&w.offsiteStatus, "offsite-status", offsite.DefaultStatusFile, "the record `felis offsite sync` leaves, checked when [offsite] is configured") + fs.StringVar(&w.toolsStatus, "build-tools-status", defaultBuildToolsStatus, "the record `felis mirror-build-tools` leaves, checked when builds scan against the registry's DB copy") + fs.BoolVar(&w.dryRun, "dry-run", false, "print every finding and the mail that is due; send nothing and keep the state as it was") + fs.StringVar(&w.heartbeatFile, "heartbeat-file", defaultHeartbeatFile, "file holding the heartbeat URL each run pings, a dead man's switch at a monitoring service that alerts when the pings stop (no file pings nothing)") + fs.BoolVar(&w.unitFailed, "unit-failed", false, "report a failed run of felis-watchdog.service instead of checking; felis-watchdog-failed.service runs this through OnFailure=") +} + +// watchdogProbes is one pass of every check, appended to report. cl is the +// cluster client, nil while the API server is unreachable; owners is who the +// alerts go to, and ownersErr why PostgreSQL did not say. +func watchdogProbes(ctx context.Context, w watchdogFlags, cfg *config.Config, now time.Time, report *watchdog.Report) (cl client.Client, owners []string, ownersErr error) { + add := func(f *watchdog.Finding) { + if f != nil { + report.Findings = append(report.Findings, *f) + } + } + + // The cluster: one unreachable API server stands in for every check behind it. + minecraftNS := cfg.K8s.Namespace + if minecraftNS == "" { + minecraftNS = platform.DefaultMinecraftNamespace + } + cl, err := buildSystemServerClient() + var found []watchdog.Finding + if err == nil { + found, err = watchdog.Cluster{Client: cl, ControlNamespace: w.controlNS, MinecraftNamespace: minecraftNS}.Check(ctx, now) + } + if err != nil { + cl = nil + f := watchdog.KubeAPIDown(err) + add(&f) + report.Unknown = append(report.Unknown, watchdog.ClusterPrefixes...) + } else { + report.Findings = append(report.Findings, found...) + } + + if owners, ownersErr = ownerEmails(ctx, cfg.Database.URL); ownersErr != nil { + f := watchdog.PostgresDown(ownersErr) + add(&f) + } + + if w.proxyAddr != "" { + add(proxyFinding(ctx, w.proxyAddr)) + } + if w.backupDir != "" { + add(watchdog.BackupFinding(w.backupDir, now)) + } + if cfg.Offsite.Enabled() { + add(watchdog.OffsiteFinding(w.offsiteStatus, now)) + } + if usesMirroredScanDB(cfg) { + add(watchdog.ScanDBFinding(w.toolsStatus, now)) + } + report.Findings = append(report.Findings, watchdog.DiskFindings(splitList(w.diskPaths))...) + add(watchdog.MemoryFinding("/proc/meminfo")) + add(watchdog.CertFinding(splitList(w.certDirs), now)) + if w.nodeIP != "" { + if held, err := watchdog.HostAddresses(); err == nil { + add(watchdog.AddressFinding(w.nodeIP, held)) + } + } + add(watchdog.ClockFinding(watchdog.ClockStatus())) + return cl, owners, ownersErr +} + // failureReport is what the heartbeat's failure ping carries, "" when the run // pings success: the alerts this run knows of reach no one (a mail that // failed, or no relay or recipient while something is open), or the state did diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index c2921ff..ba0f426 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -37,6 +37,85 @@ concrete host. --- +## 0. First look: `felis status`, `felis doctor`, `felis support-bundle` + +Three read-only commands, all run as root on the node, give the state of the +whole host before any of the sections below. They change nothing and mail +nothing, so they are safe to run at any time. + +**`sudo felis status`** prints the platform at a glance: the release, the node +and its kubelet, each control-plane Deployment (ready replicas, image, pod +restarts), the game proxy (the `felis-velocity` unit and whether the game port +accepts connections), every server with its desired state, phase, players and +newest world backup, the newest database bundle and the off-site copy, the +watched disks and memory, and the alerts the watchdog has open. A part that is +down reads as such and the rest still prints: with k3s stopped the cluster +line says `unreachable (...)` and the servers `unknown while the cluster is +unreachable`; with PostgreSQL down the backup column reads `?` and a line under +the table says why. [GO-TESTED: `TestStatusReport`, `TestStatusWithPartsDown`, +`TestStatusClusterEdges`, `TestStatusBackups`, `TestStatusWatchdog`] + +**`sudo felis doctor`** runs every check `felis watchdog` runs, with the +settings `felis-watchdog.service` gives it, plus what only the host shows: a +Felis unit that failed or a long-running one (`k3s`, `felis-velocity`) that +stopped, a timer that no longer fires, alerts that reach no one (no `[smtp]` +relay, no owner with a verified address), an unusable heartbeat URL. It prints +one line per area, `✓` fine, `!` warnings, `✗` something critical, `-` not +checked and why (the off-site copy on an install without one), each finding +with where to look next: + +```text +felis doctor on felis-1 at 2026-09-27 12:00 UTC +checks run as /etc/systemd/system/felis-watchdog.service runs them (config /etc/felis/felis.host.toml) + +✓ configuration +✓ Kubernetes cluster +✗ PostgreSQL + critical postgres: PostgreSQL is unreachable: sign-in, the panel and server management fail + → k3s kubectl -n felis get pods -l app.kubernetes.io/component=postgres; k3s kubectl -n felis logs deploy/felis-postgres --tail=100 (...) +- off-site copy: not checked, not configured +... +1 problem(s): 1 critical, 0 warning(s) +``` + +It exits 1 when it found anything and 0 otherwise, so a script can run it. It +never mails, pings the heartbeat or touches the watchdog's state: an alert it +shows is mailed by the watchdog's own next run, on the watchdog's delays (§14). +A missing or unreadable `felis-watchdog.service` is itself a critical finding, +and the checks then run with the watchdog's defaults. [GO-TESTED: +`TestDoctorReportsByArea`, `TestDoctorMailsPingsAndSavesNothing`, +`TestDoctorWithoutUnitOrConfig`, `TestUnitFindings`, `TestPrintDoctorReport`] + +**`sudo felis support-bundle`** collects what someone helping needs into one +file, `/var/lib/felis/support/felis-support--