diff --git a/deploy/crd/felis.lolicon.best_minecraftservers.yaml b/deploy/crd/felis.lolicon.best_minecraftservers.yaml index 4f4f926..4d34df9 100644 --- a/deploy/crd/felis.lolicon.best_minecraftservers.yaml +++ b/deploy/crd/felis.lolicon.best_minecraftservers.yaml @@ -101,6 +101,8 @@ spec: EmptySecondsBeforeStop is how long the server may sit empty before the operator scales it down. format: int32 + maximum: 604800 + minimum: 0 type: integer type: object image: @@ -127,6 +129,8 @@ spec: description: TerminationGracePeriodSeconds is the pod grace period (default 300). format: int64 + maximum: 3600 + minimum: 0 type: integer type: object motd: @@ -159,6 +163,8 @@ spec: port: description: Port is the RCON TCP port (default 25575). format: int32 + maximum: 65535 + minimum: 0 type: integer secretRef: description: SecretRef points at the Secret holding the RCON password. @@ -172,6 +178,10 @@ spec: - name type: object type: object + x-kubernetes-validations: + - message: the allow-rcon NetworkPolicy admits only port 25575; leave + port unset + rule: '!has(self.port) || self.port == 0 || self.port == 25575' reaperExempt: description: ReaperExempt opts this server out of the world reaper entirely (spec §18). @@ -250,16 +260,22 @@ spec: (notably LOOHP/Limbo) where the felis-limbo plugin reports true readiness only after the first server tick. format: int32 + maximum: 65535 + minimum: 0 type: integer readinessTimeoutSeconds: description: ReadinessTimeoutSeconds is the budget for the first successful RCON probe. format: int32 + maximum: 86400 + minimum: 0 type: integer timeoutSeconds: description: TimeoutSeconds is the overall budget before the server is marked Failed. format: int32 + maximum: 86400 + minimum: 0 type: integer type: object storage: diff --git a/docs/operations.md b/docs/operations.md index 1aab255..4a5af99 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -44,7 +44,9 @@ node's local-path storage, so a game server's pod is pinned to the node that fir scheduled it and cannot move when that node fails; the operator and felis-api each run as a single replica without leader election, so an upgrade or a node restart pauses wakes and stops until their pod is back. Joining k3s agents to the cluster is untested -and gains no failover. +and gains no failover. A multi-node shape would need, at least, storage that can follow a +pod to another node and leader election in felis-operator (controller-runtime's +`LeaderElection`) so a second replica can stand by. ## 2. Sizing @@ -261,6 +263,37 @@ sudo systemctl start postgresql sudo k3s kubectl -n felis scale deploy/felis-api deploy/felis-operator --replicas=1 ``` +### The MinecraftServer CRD [VM-VERIFIED] + +Every rerun applies the CRD embedded in the `felis` binary (`felis bootstrap-assets crd`). +It serves and stores the single version `v1alpha1`, and the apiserver refuses values the +operator cannot act on: + +| Field | Accepted | +|---|---| +| `spec.rcon.port` | unset, `0` or `25575`: the allow-rcon NetworkPolicy opens only 25575, so any other port leaves the server unprobeable | +| `spec.startup.timeoutSeconds`, `readinessTimeoutSeconds` | 0 – 86400 | +| `spec.startup.healthHTTPPort` | 0 – 65535 | +| `spec.lifecycle.terminationGracePeriodSeconds` | 0 – 3600 | +| `spec.idle.emptySecondsBeforeStop` | 0 – 604800 (the panel caps it at 86400) | + +`0` means the operator's default throughout. An object stored before these rules keeps an +out-of-range value until someone edits that field (CRD validation ratcheting). The operator +reads a negative value as its default and an oversized one as written, so fix such a +value by hand: `kubectl -n minecraft edit minecraftserver `. + +**Moving to `v1beta1` (planned, not built).** The first breaking change to the spec ships as a new +version, in this order, each step one release: + +1. The CRD serves `v1alpha1` and `v1beta1`, storage stays `v1alpha1`. While the two + schemas carry the same fields, `conversion.strategy: None` suffices; a renamed or + reshaped field needs a conversion webhook, which felis-operator would serve. +2. Storage moves to `v1beta1`. The installer rewrites every object so etcd holds the new + version (`kubectl get minecraftservers -A -o json | kubectl replace -f -`), then sets + `status.storedVersions` of the CRD to `["v1beta1"]`. +3. A later release stops serving `v1alpha1`. Felis itself reads through one Go type at a + time, so the operator and felis-api switch in the release that moves storage. + ## 5. Disaster recovery The procedures are in §16: what a database bundle holds, restoring one on the same host, diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 502924c..2c0de7f 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -140,8 +140,9 @@ message is the verbatim dial error: backend's `rcon.password`. Reconcile the two. [GO-TESTED that this maps to `RconNotReachable`.] - `connection refused` / `i/o timeout` → the backend has not opened the RCON - port yet, RCON is disabled in `server.properties`, or `spec.rcon.port` - (default 25575) is wrong. [INTEGRATION-ONLY for the live handshake.] + port yet, RCON is disabled in `server.properties`, or the image listens on a + port other than 25575 (the CRD accepts only that one for `spec.rcon.port`, + operations.md §4). [INTEGRATION-ONLY for the live handshake.] The per-probe timeout is a fixed 5s in code (`prober.go:45`, shortened further if the reconcile context has a nearer deadline). It is **not** derived from diff --git a/go.mod b/go.mod index 80ce152..3630971 100644 --- a/go.mod +++ b/go.mod @@ -19,11 +19,13 @@ require ( k8s.io/api v0.31.3 k8s.io/apimachinery v0.31.3 k8s.io/client-go v0.31.0 + k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 sigs.k8s.io/controller-runtime v0.19.3 sigs.k8s.io/yaml v1.4.0 ) require ( + github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a // indirect github.com/atotto/clipboard v0.1.4 // indirect github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect github.com/beorn7/perks v1.0.1 // indirect @@ -108,7 +110,6 @@ require ( gopkg.in/yaml.v3 v3.0.1 // indirect k8s.io/apiextensions-apiserver v0.31.0 // indirect k8s.io/klog/v2 v2.130.1 // indirect - k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 // indirect sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect diff --git a/go.sum b/go.sum index bf06b98..02b68a8 100644 --- a/go.sum +++ b/go.sum @@ -2,6 +2,8 @@ github.com/BurntSushi/toml v1.6.0 h1:dRaEfpa2VI55EwlIW72hMRHdWouJeRF7TPYhI+AUQjk github.com/BurntSushi/toml v1.6.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho= github.com/MakeNowJust/heredoc v1.0.0 h1:cXCdzVdstXyiTqTvfqk9SDHpKNjxuom+DOlyEeQ4pzQ= github.com/MakeNowJust/heredoc v1.0.0/go.mod h1:mG5amYoWBHf8vpLOuehzbGGw0EHxpZZ6lCpQ4fNJ8LE= +github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a h1:idn718Q4B6AGu/h5Sxe66HYVdqdGu2l9Iebqhi/AEoA= +github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4= github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI= github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k= diff --git a/internal/apis/felis/v1alpha1/crd_test.go b/internal/apis/felis/v1alpha1/crd_test.go new file mode 100644 index 0000000..2a1a94e --- /dev/null +++ b/internal/apis/felis/v1alpha1/crd_test.go @@ -0,0 +1,105 @@ +package v1alpha1_test + +import ( + "encoding/json" + "os" + "testing" + + "k8s.io/kube-openapi/pkg/validation/spec" + "k8s.io/kube-openapi/pkg/validation/strfmt" + "k8s.io/kube-openapi/pkg/validation/validate" + "sigs.k8s.io/yaml" +) + +// crdSchema loads the openAPIV3Schema the cluster enforces, from the same YAML +// `felis bootstrap-assets crd` applies. The x-kubernetes-validations (CEL) rules +// ride along as extensions and are checked live with a server-side dry run. +func crdSchema(t *testing.T) *spec.Schema { + t.Helper() + raw, err := os.ReadFile("../../../../deploy/crd/felis.lolicon.best_minecraftservers.yaml") + if err != nil { + t.Fatal(err) + } + var crd struct { + Spec struct { + Versions []struct { + Schema struct { + OpenAPIV3Schema json.RawMessage `json:"openAPIV3Schema"` + } `json:"schema"` + } `json:"versions"` + } `json:"spec"` + } + if err := yaml.Unmarshal(raw, &crd); err != nil { + t.Fatal(err) + } + if len(crd.Spec.Versions) != 1 { + t.Fatalf("versions = %d, want the single v1alpha1", len(crd.Spec.Versions)) + } + var s spec.Schema + if err := json.Unmarshal(crd.Spec.Versions[0].Schema.OpenAPIV3Schema, &s); err != nil { + t.Fatal(err) + } + return &s +} + +// server is a minimal valid MinecraftServer with one extra spec section. +func server(section string, body map[string]any) map[string]any { + sp := map[string]any{"image": "paper", "subdomain": "survival"} + if section != "" { + sp[section] = body + } + return map[string]any{ + "apiVersion": "felis.lolicon.best/v1alpha1", + "kind": "MinecraftServer", + "metadata": map[string]any{"name": "survival"}, + "spec": sp, + } +} + +func TestCRDBoundsNumericFields(t *testing.T) { + schema := crdSchema(t) + cases := []struct { + name string + section string + body map[string]any + ok bool + }{ + {"defaults", "", nil, true}, + {"rcon port 0 means default", "rcon", map[string]any{"port": 0}, true}, + {"rcon port 25575", "rcon", map[string]any{"port": 25575}, true}, + {"rcon port negative", "rcon", map[string]any{"port": -1}, false}, + {"rcon port past 65535", "rcon", map[string]any{"port": 70000}, false}, + {"grace period 3600", "lifecycle", map[string]any{"terminationGracePeriodSeconds": 3600}, true}, + {"grace period negative", "lifecycle", map[string]any{"terminationGracePeriodSeconds": -1}, false}, + {"grace period past an hour", "lifecycle", map[string]any{"terminationGracePeriodSeconds": 3601}, false}, + {"startup timeout a day", "startup", map[string]any{"timeoutSeconds": 86400}, true}, + {"startup timeout negative", "startup", map[string]any{"timeoutSeconds": -5}, false}, + {"startup timeout past a day", "startup", map[string]any{"timeoutSeconds": 100000}, false}, + {"readiness timeout negative", "startup", map[string]any{"readinessTimeoutSeconds": -1}, false}, + {"readiness timeout past a day", "startup", map[string]any{"readinessTimeoutSeconds": 86401}, false}, + {"health port 8080", "startup", map[string]any{"healthHTTPPort": 8080}, true}, + {"health port past 65535", "startup", map[string]any{"healthHTTPPort": 70000}, false}, + {"health port negative", "startup", map[string]any{"healthHTTPPort": -1}, false}, + {"idle a week", "idle", map[string]any{"emptySecondsBeforeStop": 604800}, true}, + {"idle negative", "idle", map[string]any{"emptySecondsBeforeStop": -1}, false}, + {"idle past a week", "idle", map[string]any{"emptySecondsBeforeStop": 604801}, false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + // Round-trip through JSON so numbers arrive as the float64 the + // apiserver's decoder hands the validator. + raw, _ := json.Marshal(server(tc.section, tc.body)) + var obj any + if err := json.Unmarshal(raw, &obj); err != nil { + t.Fatal(err) + } + err := validate.AgainstSchema(schema, obj, strfmt.Default) + if tc.ok && err != nil { + t.Fatalf("rejected a valid spec: %v", err) + } + if !tc.ok && err == nil { + t.Fatal("accepted an out-of-range value") + } + }) + } +} diff --git a/internal/apis/felis/v1alpha1/minecraftserver_types.go b/internal/apis/felis/v1alpha1/minecraftserver_types.go index 31b85cb..d907def 100644 --- a/internal/apis/felis/v1alpha1/minecraftserver_types.go +++ b/internal/apis/felis/v1alpha1/minecraftserver_types.go @@ -174,10 +174,17 @@ type MotdSpec struct { } // RconSpec configures RCON (spec §4 spec.rcon). +// +// The port stays the default: the allow-rcon NetworkPolicy (internal/platform +// netpol.go) admits the operator and felis-api on 25575 only, so any other port +// would leave the server unprobeable and stuck in Starting. +// +kubebuilder:validation:XValidation:rule="!has(self.port) || self.port == 0 || self.port == 25575",message="the allow-rcon NetworkPolicy admits only port 25575; leave port unset" type RconSpec struct { // Enabled must be true for readiness probing and graceful shutdown. Enabled bool `json:"enabled,omitempty"` // Port is the RCON TCP port (default 25575). + // +kubebuilder:validation:Minimum=0 + // +kubebuilder:validation:Maximum=65535 Port int32 `json:"port,omitempty"` // SecretRef points at the Secret holding the RCON password. SecretRef SecretKeyRef `json:"secretRef,omitempty"` @@ -202,14 +209,20 @@ type StorageSpec struct { // then the time the server has to finish its own shutdown save after SIGTERM. type LifecycleSpec struct { // TerminationGracePeriodSeconds is the pod grace period (default 300). + // +kubebuilder:validation:Minimum=0 + // +kubebuilder:validation:Maximum=3600 TerminationGracePeriodSeconds int64 `json:"terminationGracePeriodSeconds,omitempty"` } // StartupSpec bounds the Starting phase (spec §5). type StartupSpec struct { // TimeoutSeconds is the overall budget before the server is marked Failed. + // +kubebuilder:validation:Minimum=0 + // +kubebuilder:validation:Maximum=86400 TimeoutSeconds int32 `json:"timeoutSeconds,omitempty"` // ReadinessTimeoutSeconds is the budget for the first successful RCON probe. + // +kubebuilder:validation:Minimum=0 + // +kubebuilder:validation:Maximum=86400 ReadinessTimeoutSeconds int32 `json:"readinessTimeoutSeconds,omitempty"` // HealthHTTPPort, when > 0, switches the pod readiness probe from the default // plain-TCP check on the game port to an HTTP GET on this container port. It @@ -219,6 +232,8 @@ type StartupSpec struct { // only after the first server tick. The operator's readiness path is otherwise // unchanged — with rcon disabled, passing this probe (readyReplicas >= 1) is // what marks the server Ready. + // +kubebuilder:validation:Minimum=0 + // +kubebuilder:validation:Maximum=65535 HealthHTTPPort int32 `json:"healthHTTPPort,omitempty"` // HealthHTTPPath is the path for the HTTP readiness probe (default "/healthz" // when HealthHTTPPort is set). @@ -246,7 +261,10 @@ type IdleSpec struct { // AutoStopEnabled turns on idle auto-stop. AutoStopEnabled bool `json:"autoStopEnabled,omitempty"` // EmptySecondsBeforeStop is how long the server may sit empty before the - // operator scales it down. + // operator scales it down. The API caps what the panel sets well below the + // schema's week. + // +kubebuilder:validation:Minimum=0 + // +kubebuilder:validation:Maximum=604800 EmptySecondsBeforeStop int32 `json:"emptySecondsBeforeStop,omitempty"` } diff --git a/internal/platform/netpol.go b/internal/platform/netpol.go index 1e1550b..48b7846 100644 --- a/internal/platform/netpol.go +++ b/internal/platform/netpol.go @@ -21,14 +21,11 @@ const gamePort = operator.GamePort // so the policy and the server container's default RCON port are one source of // truth. // -// LIMITATION (honestly labeled, not verifiable without a cluster): RCON is -// per-server overridable via spec.rcon.port (internal/operator.rconPort), but this -// is one namespace-wide policy that can open only a single port. It opens the -// default. A server that overrides spec.rcon.port to a non-default value would have -// its RCON port denied by this fence, so the operator's readiness prober could not -// reach it. The supported deployment keeps the default RCON port; a per-server-port -// deployment would need per-server NetworkPolicies, deferred until a concrete need -// exists. +// This is one namespace-wide policy, so it opens a single port: the default. The +// CRD holds spec.rcon.port to unset, 0 or 25575 with a CEL rule (RconSpec in +// internal/apis/felis/v1alpha1), so no server can move RCON behind this fence and +// leave the operator's readiness prober outside it. Per-server ports would need +// per-server NetworkPolicies and a relaxed rule, together. const rconPort = operator.DefaultRconPort // serverPodSelector matches every operator-managed Minecraft server pod by the