- [registry] gains kaniko_image / trivy_image / build_cpu_limit / build_mem_limit overrides; empty keeps the compiled-in defaults. An air-gapped or mirrored install has no route to gcr.io/aquasec (the build egress policy allows only DNS + registry + package mirrors), so builds previously could not even start their executors. - deferred-seams: the uploads-context entry now records WHY a mount is impossible (PVCs cannot cross namespaces) and that the s3 lane also lacks credentials in the build Pod — options captured for the real fix. - troubleshooting 8e (executor ImagePullBackOff + the overrides), 13b rewritten (verified eviction refusal, 5m pressure-transition, image-GC recovery), 15 (upgrade/rollback runbook for Recreate). - Backup semantics decided and documented: a backup is the whole /data volume (worlds + config + plugins + cache) and a restore rolls all of it back — OpenAPI/README wording updated to match (same-tag images are still watched for regressions by the openapi parity gate).
This commit is contained in:
10 files changed
+165
-37
No files matched your search
+2
-2
@@ -19,14 +19,14 @@ IPv6-only 接入(`ssh -6 -i ~/.ssh/id_ed25519 root@fdb2:2c26:f4e4:0:21c:42ff:f
|
||||
| 19 | **reaper CronJob 渲染位置错误,永远无法调度**:`felis manifests` 把 CronJob 渲染到控制 ns,却引用 minecraft ns 的备份 PVC(Pod 不能跨 ns 挂 PVC:真机 `FailedScheduling: persistentvolumeclaim "felis-backups" not found`);修正 ns 后又发现 ServerAccount 也不能跨 ns 使用(`serviceaccount "felis-reaper" not found`)。而 reaper 的 Role/RoleBinding 本就在 minecraft ns | 真机三层取证(PVC/SA/调度)| CronJob 与 SA、RoleBinding subject 全部移到 `MinecraftNamespace`(提交 `e4f2cff`+`c839454`)。**修复后完整演练通过**(见下) |
|
||||
| 6 | **默认安装无备份能力 + 回收链路不可达/不可用**:(a) 没有任何环节渲染归档 PVC,`FELIS_BACKUP_PVC` 永远为空 → backup/restore 恒 503;(b) bootstrap 从不下传 reaper 三旗标 → 官方安装路径根本无法启用回收;(c) 即便启用,stock k3s 的 `<root>/<pvc>` 布局不存在,且 k3s storage root 是 `0700 root:root`、reaper pod 以 uid 1000 运行 → 真机 `lstat /worlds/…: permission denied`(fail-closed 跳过,但纯空转);(d) README 口径(“定时备份”)不实 | 真机全链路:默认渲染 PVC+env → 建服→marker→备份 202→jobs 端点 running→succeeded→改 marker→restore→读回原值 ✅;再建 resolvecheck 世界(20d idle,marker)→ CronJob 手动 Job:`evaluated=2 reaped=1`,归档含 marker、PVC+宿主目录回收、`world_backups` 得 `inactive_15d` 行、servers 行/CR 保留 ✅ | `platform/workloads.go` 渲染归档 PVC(minecraft ns、RWO 10Gi、默认 SC),`felis manifests --backup-pvc` 默认 `felis-backups`(`=` 空为显式关闭),bootstrap 统一下传 env/旗标并写 [archive] local_path;`resolveWorldDir` 新增精确 local-path 目录解析(读 PVC `spec.volumeName`,非 glob,杜绝陈旧 PV 目录顶替);ReaperRole 增 `pvc:get`;bootstrap 设 `FELIS_WORLDS_HOST_PATH` 时给 uid 1000 授 traverse(setfacl/o+x);README/故障手册改写。提交 `fd33fd0`+`2b87a5a` |
|
||||
| 8 | **磁盘打满灾难链**:DiskPressure → kubelet 驱逐控制面(无 PriorityClass 保护)→ 镜像被 GC(无外网、registry 空)→ 全部 ImagePullBackOff;释放后数分钟才恢复调度。恢复依赖人工 `docker save | k3s ctr images import -` | 三轮真机 drill(原缺陷复现 + 两轮修复验证):①自定义 1e6 类:游戏 pod 先走、控制面随后仍被驱逐(kubelet 日志逐条列出 ranked/evicted),随后镜像被 GC → ErrImagePull;②内建 `system-cluster-critical`(2e9):填盘至 1.7G free,kubelet 对 felis-api/operator/registry 全部报 **“cannot evict a critical pod”**,三者在整个 DiskPressure 期间保持 Running;游戏 pod(0)被驱逐;③释放磁盘后:`DiskPressure` 经 ~5 分钟(`eviction-pressure-transition-period`)转 False——即“恢复慢”的主因;被 GC 的游戏镜像按 runbook 重导入后 25s 恢复 ✅ | 控制面四类 pod 挂内建 `system-cluster-critical`(用户自定义类值上限 1e9,达不到 2e9 临界阈值;preemption 保留=管理面可调度,已记录权衡);troubleshooting §13b 固化事件链、5 分钟条件过渡与镜像恢复路径。提交见下 |
|
||||
| 9 | **升级策略 Recreate**:单副本 + Recreate:任何控制面升级=停机;坏升级(错 tag)中断约 95s 且需人工 `rollout undo`(无自动回滚) | 复核部署模板(Recreate + 单副本 + 无 leader election 的注释理由成立) | 不改策略(Recreate 是对无 leader election 的正确取舍),改为固化 runbook:troubleshooting §15 = 升级即重跑安装器;停机窗口=rollout 时长;坏镜像在安装器 180s 等待内以 `kubectl describe` 诊断呈现;回滚 `kubectl rollout undo`(镜像被 GC 时先按 §13b 重导入)。提交见下 |
|
||||
| 10 | **备份语义**:归档包含整个 /data(jar、libraries、cache),167MB;是否符合 world backup 定位待评估 | 真机 tar 清单(cache/、config/、eula.txt、server.properties、jar)+ 恢复语义(overlay+prune) | 评估结论:**整卷备份是正确语义**(回滚=整服状态回滚,含配置/插件),保留行为;文档化:troubleshooting §10「backup contains」+ OpenAPI/README 口径改为“整服数据卷”。提交见下 |
|
||||
|
||||
## 待决策台账(未修)
|
||||
|
||||
| # | 主题 | 说明 |
|
||||
|---|------|------|
|
||||
| 7 | 异步失败不可感知 | backup/restore 失败后无状态出口:restore 行不变、backup 无行;只有集群侧 Job/日志可查。建议状态字段或 `?failed` 查询 |
|
||||
| 9 | 升级策略 Recreate | 单副本 + Recreate:任何控制面升级=停机;坏升级(实测错 tag)服务中断约 95s 且需人工 `rollout undo`(无自动回滚)。建议 runbook/文档化 |
|
||||
| 10 | 备份语义 | 归档包含整个 /data(jar、libraries、cache),167MB;是否符合"world backup"定位待评估 |
|
||||
| 11 | PG 断连表现 | 会话查询失败报 401 而非 503(fail-closed 但误导;用户以为没登录) |
|
||||
| 12 | ready 门滞后 | 容器 Ready 后 6~10s 内 API 仍 409 not_running |
|
||||
| 13 | setup token 截断 | 43 字符 token + 长域名,80 列终端下 TUI 截断显示(复现:tmux 80 列) |
|
||||
|
||||
Reference in new issue
Block a user