fix(platform): front the felis-api internal face on its own ClusterIP Service

The login limbo pod dials FELIS_API_BASE_URL = felis-api.<ns>.svc:8081 (the
internal face, service-token auth) to mint bind codes and poll link status, but
the only Service named felis-api is the external NodePort face and declares only
port 443. A Service answers only on its declared ports, so felis-api:8081 had no
backend and every login-pod internal call silently failed to connect.

Render a separate ClusterIP Service felis-api-internal for port 8081 and repoint
InternalAPIBaseURL at it. A second port on the NodePort Service is not an option:
Type=NodePort allocates a node port for every declared port with no per-port
opt-out, so it would publish the no-Zero-Trust internal face on every node's
external IP. A distinct ClusterIP Service keeps 8081 in-cluster only, reachable
by the login pod via DNS and by the on-node break-glass console via the
ClusterIP (exported as APIInternalServiceName / APIInternalPort).

Manifest-level fix; the live packet path is pending real-cluster verification.

deploy/limbo/README.md

@@ -131,11 +131,16 @@ set them by hand:
   via a `secretKeyRef`, keyed off the reserved `login` name. Until the token is
   present the plugin fail-safes to readiness-only, so the gate is never broken — it
   simply does not authenticate yet.
+- **Service:** the login pod dials `FELIS_API_BASE_URL`, which resolves to the
+  ClusterIP Service `felis-api-internal` (control namespace) that fronts the api
+  pod's internal port 8081. That Service is deliberately separate from the external
+  NodePort `felis-api` (443) so the no-Zero-Trust internal face is never published on
+  a node's external IP.
 - **NetworkPolicy:** none is required today — neither the minecraft-namespace egress
   nor the control-namespace ingress is policy-locked, so the login pod's call to the
-  API internal port is already reachable. If a future deployment adds a minecraft
-  egress lock or a control-namespace ingress fence, it must also open the
-  login-pod → felis-api internal-port (8081) path.
+  API internal port is reachable. If a future deployment adds a minecraft egress lock
+  or a control-namespace ingress fence, it must also open the login-pod →
+  felis-api-internal (8081) path.
 
 The Velocity default-landing and waiting-park wiring is printed by `felis setup`
 and enforces the invariant: fresh connections hit `login` first; nothing falls

docs/troubleshooting.md

@@ -280,6 +280,17 @@ The internal face (`--internal-addr :8081`, routes under
 `/api/v1/internal/...`) is **never** Zero-Trust; it authenticates a single
 service token via `Authorization: Bearer <token>`, compared in constant time.
 
+In-cluster it is reached through the ClusterIP Service `felis-api-internal` (port
+8081), which is separate from the external NodePort `felis-api` (443) precisely so
+the no-Zero-Trust face is never exposed on a node. On the control-plane node the
+break-glass console reaches it by resolving that Service's ClusterIP and dialing
+`:8081`.
+
+- **Internal calls fail to *connect* (not 401)** → the `felis-api-internal` Service
+  is missing or its selector no longer matches the api pods. `kubectl -n felis get
+  svc felis-api-internal` must show a ClusterIP with 8081; a bare `felis-api` name
+  serves only 443 and every internal call would hang/refuse.
+
 - **All internal calls 401** → the token is unset or wrong. The API reads env
   `FELIS_SERVICE_TOKEN`. If unset, startup logs:
 

internal/platform/workloads.go

@@ -128,17 +128,28 @@ const (
 	nonRootUID int64 = 1000
 )
 
+// APIInternalServiceName is the ClusterIP Service that fronts the felis-api
+// internal face (8081). It is SEPARATE from the external NodePort Service (SAAPI)
+// on purpose — see apiInternalService. The login pod resolves it by cross-namespace
+// DNS; the on-node break-glass console resolves its ClusterIP and dials it directly.
+const APIInternalServiceName = SAAPI + "-internal"
+
+// APIInternalPort is the felis-api internal-face port, exported for the on-node
+// console which builds http://<clusterIP>:APIInternalPort after a Service lookup.
+const APIInternalPort = apiInternalPort
+
 // InternalAPIBaseURL returns the in-cluster base URL of the felis-api INTERNAL
 // face for a caller in another namespace — specifically the login system server,
 // which dials it with the service token to mint bind codes and poll link status.
-// It single-sources the Service name (SAAPI, in the control namespace) and the
-// internal port with the Deployment/Service above, so a rename or port change here
-// can never drift from what the login pod is told to call. Cross-namespace DNS is
-// always resolvable; reachability additionally depends on there being no fence in
-// the way (today neither the minecraft-ns egress nor the control-ns ingress is
-// policy-locked, so the path is open — see internal/platform/netpol.go).
+// It single-sources the internal Service name (APIInternalServiceName, in the
+// control namespace) and the internal port with the Deployment/Service above, so a
+// rename or port change here can never drift from what the login pod is told to
+// call. Cross-namespace DNS is always resolvable, and apiInternalService actually
+// programs 8081 on that ClusterIP; reachability additionally depends on there being
+// no fence in the way (today neither the minecraft-ns egress nor the control-ns
+// ingress is policy-locked, so the path is open — see internal/platform/netpol.go).
 func InternalAPIBaseURL(controlNamespace string) string {
-	return fmt.Sprintf("http://%s.%s.svc.cluster.local:%d", SAAPI, controlNamespace, apiInternalPort)
+	return fmt.Sprintf("http://%s.%s.svc.cluster.local:%d", APIInternalServiceName, controlNamespace, apiInternalPort)
 }
 
 // Workloads renders the running control-plane: the felis-api Deployment, the
@@ -151,6 +162,7 @@ func Workloads(p Params) []Object {
 	objs := []Object{
 		APIDeployment(p),
 		apiService(p),
+		apiInternalService(p),
 		OperatorDeployment(p),
 		registryDeployment(p),
 		registryService(p),
@@ -300,6 +312,35 @@ func apiService(p Params) *corev1.Service {
 	}
 }
 
+// apiInternalService fronts the felis-api INTERNAL face (service-token, no Zero
+// Trust) on a ClusterIP-only Service, kept SEPARATE from the external NodePort
+// apiService on purpose: a NodePort Service allocates a node port for EVERY declared
+// port with no per-port opt-out, so folding 8081 into apiService would publish the
+// no-Zero-Trust internal face on every node's external IP — a hard red line for a
+// face whose only guard is the bearer service token. A distinct ClusterIP Service
+// exposes 8081 in-cluster only: reachable by the login pod (cross-namespace DNS to
+// APIInternalServiceName) and, on the k3s node, by the break-glass console dialing
+// this Service's ClusterIP. Without it the felis-api DNS name has no 8081 port and
+// every internal-face call silently fails to connect.
+func apiInternalService(p Params) *corev1.Service {
+	p = p.withDefaults()
+	labels := controlPlanePodLabels(ComponentAPI)
+	return &corev1.Service{
+		TypeMeta:   metav1.TypeMeta{APIVersion: "v1", Kind: "Service"},
+		ObjectMeta: metav1.ObjectMeta{Name: APIInternalServiceName, Namespace: p.ControlNamespace, Labels: labels},
+		Spec: corev1.ServiceSpec{
+			Type:     corev1.ServiceTypeClusterIP,
+			Selector: labels,
+			Ports: []corev1.ServicePort{{
+				Name:       "internal",
+				Port:       apiInternalPort,
+				TargetPort: intstr.FromString("internal"),
+				Protocol:   corev1.ProtocolTCP,
+			}},
+		},
+	}
+}
+
 // OperatorDeployment renders the felis-operator Deployment (spec §5). It runs as
 // the felis-operator SA and carries controlPlanePodLabels(operator), the second
 // pod the allow-rcon peer admits (the readiness prober dials RCON). It takes NO

internal/platform/workloads_test.go

@@ -277,6 +277,38 @@ func TestAPIService_NodePort(t *testing.T) {
 	}
 }
 
+// TestAPIInternalService_ClusterIP pins the separate internal-face Service: it must
+// be ClusterIP (never NodePort — the internal face is service-token-only and must not
+// be published on a node's external IP), expose 8081 -> the api pod's "internal"
+// port, carry NO nodePort, and select the same api pods as the external Service. It
+// is what makes the felis-api DNS name actually answer on 8081 (the login pod path)
+// and gives the on-node console a ClusterIP to dial.
+func TestAPIInternalService_ClusterIP(t *testing.T) {
+	p := testParams()
+	svc := apiInternalService(p)
+	dep := APIDeployment(p)
+
+	if svc.Name != APIInternalServiceName || svc.Namespace != p.ControlNamespace {
+		t.Errorf("internal Service = %s/%s, want %s/%s", svc.Namespace, svc.Name, p.ControlNamespace, APIInternalServiceName)
+	}
+	if svc.Name == SAAPI {
+		t.Errorf("internal Service must not collide with the external Service name %q", SAAPI)
+	}
+	if svc.Spec.Type != corev1.ServiceTypeClusterIP {
+		t.Errorf("internal Service type = %s, want ClusterIP (never expose the no-Zero-Trust face on a node)", svc.Spec.Type)
+	}
+	if !mapSelectorMatches(svc.Spec.Selector, dep.Spec.Template.Labels) {
+		t.Errorf("internal Service selector %v does not select api pod labels %v", svc.Spec.Selector, dep.Spec.Template.Labels)
+	}
+	if len(svc.Spec.Ports) != 1 {
+		t.Fatalf("internal Service ports = %v, want one", svc.Spec.Ports)
+	}
+	port := svc.Spec.Ports[0]
+	if port.Port != apiInternalPort || port.TargetPort.StrVal != "internal" || port.NodePort != 0 {
+		t.Errorf("internal Service port = %#v, want %d -> internal with no nodePort", port, apiInternalPort)
+	}
+}
+
 // TestAPIDeployment_BackupPVC proves the FELIS_BACKUP_PVC env appears only when a
 // backup PVC is named.
 func TestAPIDeployment_BackupPVC(t *testing.T) {
@@ -375,18 +407,26 @@ func TestRegistry_DeploymentServicePVC(t *testing.T) {
 }
 
 // TestWorkloads_BundleContents sanity-checks the slice Workloads returns: the two
-// control-plane Deployments, the api Service, and the registry Deployment/Service/PVC,
-// every one with TypeMeta (so its YAML header renders).
+// control-plane Deployments, the api external+internal Services, and the registry
+// Deployment/Service/PVC, every one with TypeMeta (so its YAML header renders). The
+// internal Service must be present or the login pod's felis-api:8081 path is dead.
 func TestWorkloads_BundleContents(t *testing.T) {
 	objs := Workloads(testParams())
-	if len(objs) != 7 {
-		t.Fatalf("Workloads returned %d objects, want 7", len(objs))
+	if len(objs) != 8 {
+		t.Fatalf("Workloads returned %d objects, want 8", len(objs))
 	}
+	var haveInternalSvc bool
 	for _, o := range objs {
 		gvk := o.GetObjectKind().GroupVersionKind()
 		if gvk.Kind == "" || gvk.Version == "" {
 			t.Errorf("%T missing TypeMeta (kind=%q version=%q)", o, gvk.Kind, gvk.Version)
 		}
+		if svc, ok := o.(*corev1.Service); ok && svc.Name == APIInternalServiceName {
+			haveInternalSvc = true
+		}
+	}
+	if !haveInternalSvc {
+		t.Errorf("Workloads bundle is missing the internal-face Service %q", APIInternalServiceName)
 	}
 }