Loading cmd/felis/db.go +5 −4 Changes for cmd/felis/db.go: 5 added lines, 4 removed lines. Original line number Diff line number Diff line Loading @@ -415,10 +415,11 @@ func humanBytes(n int64) string { return fmt.Sprintf("%.1f %ciB", float64(n)/float64(div), "KMGTPE"[exp]) } // dbCheck is the freshness probe: exit 1 when the newest bundle is missing or // older than -max-age, for a monitor or the break-glass console to act on. // dbCheck is the freshness probe: exit 1 when the newest daily bundle is // missing or older than -max-age, for a monitor or the break-glass console to // act on. func dbCheck(fs *flag.FlagSet, dir *string, args []string, stdout, stderr io.Writer) int { maxAge := fs.Duration("max-age", dbbackup.StaleAfter, "oldest acceptable newest bundle") maxAge := fs.Duration("max-age", dbbackup.StaleAfter, "oldest acceptable newest daily bundle") if err := fs.Parse(args); err != nil { return 2 } Loading @@ -427,7 +428,7 @@ func dbCheck(fs *flag.FlagSet, dir *string, args []string, stdout, stderr io.Wri fmt.Fprintf(stderr, "felis db check: %v\n", err) return 1 } fmt.Fprintf(stdout, "felis db check: ok, newest backup %s (%s ago)\n", b.Name, dbbackup.Age(time.Since(b.Created))) fmt.Fprintf(stdout, "felis db check: ok, newest daily backup %s (%s ago)\n", b.Name, dbbackup.Age(time.Since(b.Created))) return 0 } Loading docs/openapi.yaml +13 −3 Changes for docs/openapi.yaml: 13 added lines, 3 removed lines. Original line number Diff line number Diff line Loading @@ -382,9 +382,19 @@ components: Why the bundle lacks the MinecraftServer objects (the cluster did not answer the export), when it does. A restore from it brings back the database but no servers. daily_at: type: string format: date-time description: > When the newest daily bundle (felis-db-backup.timer) in dir was written, as of this record; absent when dir held none. Equal to at when this record is a daily one. stale: type: boolean description: True when there is no record or it is older than max_age_seconds. description: > True when there is no record, no daily bundle, or the newest daily bundle is older than max_age_seconds. A newer manual, pre-migrate or off-site bundle leaves it as it is: the daily timer has still stopped. max_age_seconds: type: integer format: int64 Loading Loading @@ -3684,8 +3694,8 @@ paths: description: >- What the host's felis-db-backup.timer (or a manual `felis db backup`) last recorded in platform_settings. last is null before the first backup; stale is true then, and whenever the newest backup is older than max_age_seconds. Read-only: backups run on the host, never through the API. backup; stale is true then, and whenever the newest daily backup (last.daily_at) is missing or older than max_age_seconds. Read-only: backups run on the host, never through the API. x-felis-face: [external] x-felis-tier: admin security: [{ sessionCookie: [] }] Loading docs/operations.md +1 −1 Changes for docs/operations.md: 1 added line, 1 removed line. Original line number Diff line number Diff line Loading @@ -690,7 +690,7 @@ production install: - **Rehearse the rebuild** once on a spare VM: troubleshooting.md §16 "Rebuild on a new host", every step but 8 (take-over) and 11 (the tunnel), then its checks: sign in with an email code, restore one world and join it. `felis offsite status` and `felis db check` exit non-zero when the copy or the newest bundle is stale; wire them into your monitoring, non-zero when the copy or the newest daily bundle is stale; wire them into your monitoring, or rely on the watchdog's mail. ### Moving to another host (planned) Loading docs/troubleshooting.md +10 −5 Changes for docs/troubleshooting.md: 10 added lines, 5 removed lines. Original line number Diff line number Diff line Loading @@ -1665,7 +1665,7 @@ Every two minutes the host checks: | Node `NotReady`, or kubelet reports Disk/Memory/PID pressure (§13b) | 2–5 min | critical | | PostgreSQL unreachable | 3 min | critical | | The game proxy (`felis-velocity`) refuses connections on the game port | 3 min | critical | | Newest control-plane database backup over 26h old, or none (§16) | 10 min | critical | | Newest daily control-plane database backup over 26h old, or none (§16); a newer manual or off-site bundle leaves it standing | 10 min | critical | | A watched filesystem below 15% free (below 5%: critical) | 15 min (5 min) | warning | | Host memory available below 10% | 15 min | warning | | The host no longer holds the address the install was made on (§13c) | 5 min | critical | Loading Loading @@ -2187,13 +2187,18 @@ Installer knobs: `FELIS_DB_BACKUP_DIR`, `FELIS_DB_BACKUP_KEEP`, ### Is the newest backup fresh? Three places answer, all with the same 26 h limit: Four places answer, all with the same 26 h limit on the newest **daily** bundle, the one `felis-db-backup.timer` writes. A manual, `pre-migrate` or `offsite` bundle taken since counts for a restore and leaves the alarm standing: the timer has still stopped, and that bundle only ages from here. - The panel: **管理 → 维护与备份** shows the newest backup, its kind and size, and turns red with the fix commands when it is missing or overdue (read from the `db_backup_last` platform setting each backup writes). and turns red with the fix commands when the daily one is missing or overdue (read from the `db_backup_last` platform setting each backup writes; its `daily_at` is the newest daily bundle on disk when it was written). - `sudo felis db check` exits 1 with the reason; `sudo felis db list` shows every bundle with its age. - The watchdog mails the owners (§14). - Prometheus: `FelisDBBackupStale` (critical) and `FelisDBBackupMetricMissing` (warning) in `deploy/alerts/`. They read `felis_db_backup_last_success_timestamp_seconds`, which each daily run writes Loading @@ -2207,7 +2212,7 @@ When a backup is overdue: ``` sudo systemctl status felis-db-backup.timer # enabled? next run? sudo journalctl -u felis-db-backup -n 50 --no-pager # why the last run failed sudo felis db backup # take one now (label manual) sudo systemctl start felis-db-backup.service # run the daily backup now; clears the alarm ``` **A bundle without the MinecraftServer objects.** When the cluster does not Loading internal/api/handlers_dbbackup.go +10 −3 Changes for internal/api/handlers_dbbackup.go: 10 added lines, 3 removed lines. Original line number Diff line number Diff line Loading @@ -16,8 +16,8 @@ import ( // something on this install right now". // dbBackupView is the wire shape. Last is null until the first backup has been // recorded; Stale is true for a missing record too, so the panel has a single // flag for "nobody could restore today's state". // recorded; Stale is true for a missing record or daily backup too, so the // panel has a single flag for "the daily backups have stopped". type dbBackupView struct { Last *dbbackup.Status `json:"last"` Stale bool `json:"stale"` Loading @@ -43,7 +43,14 @@ func (a *API) handleGetDBBackup(w http.ResponseWriter, r *http.Request) { writeError(w, r, err) return } // Staleness goes by the daily timer's newest bundle: a manual or // pre-migrate one recorded since would hide a timer that has stopped. A // daily record is its own daily bundle, also when written before daily_at // existed. No daily bundle leaves the zero time, centuries past the limit. if st.Label == dbbackup.LabelDaily { st.DailyAt = st.At } view.Last = &st view.Stale = st.At.IsZero() || a.now().Sub(st.At) > dbbackup.StaleAfter view.Stale = a.now().Sub(st.DailyAt) > dbbackup.StaleAfter writeJSON(w, http.StatusOK, view) } Loading
cmd/felis/db.go +5 −4 Changes for cmd/felis/db.go: 5 added lines, 4 removed lines. Original line number Diff line number Diff line Loading @@ -415,10 +415,11 @@ func humanBytes(n int64) string { return fmt.Sprintf("%.1f %ciB", float64(n)/float64(div), "KMGTPE"[exp]) } // dbCheck is the freshness probe: exit 1 when the newest bundle is missing or // older than -max-age, for a monitor or the break-glass console to act on. // dbCheck is the freshness probe: exit 1 when the newest daily bundle is // missing or older than -max-age, for a monitor or the break-glass console to // act on. func dbCheck(fs *flag.FlagSet, dir *string, args []string, stdout, stderr io.Writer) int { maxAge := fs.Duration("max-age", dbbackup.StaleAfter, "oldest acceptable newest bundle") maxAge := fs.Duration("max-age", dbbackup.StaleAfter, "oldest acceptable newest daily bundle") if err := fs.Parse(args); err != nil { return 2 } Loading @@ -427,7 +428,7 @@ func dbCheck(fs *flag.FlagSet, dir *string, args []string, stdout, stderr io.Wri fmt.Fprintf(stderr, "felis db check: %v\n", err) return 1 } fmt.Fprintf(stdout, "felis db check: ok, newest backup %s (%s ago)\n", b.Name, dbbackup.Age(time.Since(b.Created))) fmt.Fprintf(stdout, "felis db check: ok, newest daily backup %s (%s ago)\n", b.Name, dbbackup.Age(time.Since(b.Created))) return 0 } Loading
docs/openapi.yaml +13 −3 Changes for docs/openapi.yaml: 13 added lines, 3 removed lines. Original line number Diff line number Diff line Loading @@ -382,9 +382,19 @@ components: Why the bundle lacks the MinecraftServer objects (the cluster did not answer the export), when it does. A restore from it brings back the database but no servers. daily_at: type: string format: date-time description: > When the newest daily bundle (felis-db-backup.timer) in dir was written, as of this record; absent when dir held none. Equal to at when this record is a daily one. stale: type: boolean description: True when there is no record or it is older than max_age_seconds. description: > True when there is no record, no daily bundle, or the newest daily bundle is older than max_age_seconds. A newer manual, pre-migrate or off-site bundle leaves it as it is: the daily timer has still stopped. max_age_seconds: type: integer format: int64 Loading Loading @@ -3684,8 +3694,8 @@ paths: description: >- What the host's felis-db-backup.timer (or a manual `felis db backup`) last recorded in platform_settings. last is null before the first backup; stale is true then, and whenever the newest backup is older than max_age_seconds. Read-only: backups run on the host, never through the API. backup; stale is true then, and whenever the newest daily backup (last.daily_at) is missing or older than max_age_seconds. Read-only: backups run on the host, never through the API. x-felis-face: [external] x-felis-tier: admin security: [{ sessionCookie: [] }] Loading
docs/operations.md +1 −1 Changes for docs/operations.md: 1 added line, 1 removed line. Original line number Diff line number Diff line Loading @@ -690,7 +690,7 @@ production install: - **Rehearse the rebuild** once on a spare VM: troubleshooting.md §16 "Rebuild on a new host", every step but 8 (take-over) and 11 (the tunnel), then its checks: sign in with an email code, restore one world and join it. `felis offsite status` and `felis db check` exit non-zero when the copy or the newest bundle is stale; wire them into your monitoring, non-zero when the copy or the newest daily bundle is stale; wire them into your monitoring, or rely on the watchdog's mail. ### Moving to another host (planned) Loading
docs/troubleshooting.md +10 −5 Changes for docs/troubleshooting.md: 10 added lines, 5 removed lines. Original line number Diff line number Diff line Loading @@ -1665,7 +1665,7 @@ Every two minutes the host checks: | Node `NotReady`, or kubelet reports Disk/Memory/PID pressure (§13b) | 2–5 min | critical | | PostgreSQL unreachable | 3 min | critical | | The game proxy (`felis-velocity`) refuses connections on the game port | 3 min | critical | | Newest control-plane database backup over 26h old, or none (§16) | 10 min | critical | | Newest daily control-plane database backup over 26h old, or none (§16); a newer manual or off-site bundle leaves it standing | 10 min | critical | | A watched filesystem below 15% free (below 5%: critical) | 15 min (5 min) | warning | | Host memory available below 10% | 15 min | warning | | The host no longer holds the address the install was made on (§13c) | 5 min | critical | Loading Loading @@ -2187,13 +2187,18 @@ Installer knobs: `FELIS_DB_BACKUP_DIR`, `FELIS_DB_BACKUP_KEEP`, ### Is the newest backup fresh? Three places answer, all with the same 26 h limit: Four places answer, all with the same 26 h limit on the newest **daily** bundle, the one `felis-db-backup.timer` writes. A manual, `pre-migrate` or `offsite` bundle taken since counts for a restore and leaves the alarm standing: the timer has still stopped, and that bundle only ages from here. - The panel: **管理 → 维护与备份** shows the newest backup, its kind and size, and turns red with the fix commands when it is missing or overdue (read from the `db_backup_last` platform setting each backup writes). and turns red with the fix commands when the daily one is missing or overdue (read from the `db_backup_last` platform setting each backup writes; its `daily_at` is the newest daily bundle on disk when it was written). - `sudo felis db check` exits 1 with the reason; `sudo felis db list` shows every bundle with its age. - The watchdog mails the owners (§14). - Prometheus: `FelisDBBackupStale` (critical) and `FelisDBBackupMetricMissing` (warning) in `deploy/alerts/`. They read `felis_db_backup_last_success_timestamp_seconds`, which each daily run writes Loading @@ -2207,7 +2212,7 @@ When a backup is overdue: ``` sudo systemctl status felis-db-backup.timer # enabled? next run? sudo journalctl -u felis-db-backup -n 50 --no-pager # why the last run failed sudo felis db backup # take one now (label manual) sudo systemctl start felis-db-backup.service # run the daily backup now; clears the alarm ``` **A bundle without the MinecraftServer objects.** When the cluster does not Loading
internal/api/handlers_dbbackup.go +10 −3 Changes for internal/api/handlers_dbbackup.go: 10 added lines, 3 removed lines. Original line number Diff line number Diff line Loading @@ -16,8 +16,8 @@ import ( // something on this install right now". // dbBackupView is the wire shape. Last is null until the first backup has been // recorded; Stale is true for a missing record too, so the panel has a single // flag for "nobody could restore today's state". // recorded; Stale is true for a missing record or daily backup too, so the // panel has a single flag for "the daily backups have stopped". type dbBackupView struct { Last *dbbackup.Status `json:"last"` Stale bool `json:"stale"` Loading @@ -43,7 +43,14 @@ func (a *API) handleGetDBBackup(w http.ResponseWriter, r *http.Request) { writeError(w, r, err) return } // Staleness goes by the daily timer's newest bundle: a manual or // pre-migrate one recorded since would hide a timer that has stopped. A // daily record is its own daily bundle, also when written before daily_at // existed. No daily bundle leaves the zero time, centuries past the limit. if st.Label == dbbackup.LabelDaily { st.DailyAt = st.At } view.Last = &st view.Stale = st.At.IsZero() || a.now().Sub(st.At) > dbbackup.StaleAfter view.Stale = a.now().Sub(st.DailyAt) > dbbackup.StaleAfter writeJSON(w, http.StatusOK, view) }