Unverified Commit 02fd2de5 authored by Lemon-miaow's avatar Lemon-miaow
Browse files

fix(build): three drill-driven fixes so the lane actually completes on a starter node 

The first live build (Kaniko v1.24, 4 vCPU / 5.5 GiB node) walked the new
transport end to end and hit three real defects, each invisible to unit tests:

- The Job requested its FULL limits (2 CPU / 4Gi per container), so the build
  Pod never scheduled on the platform's own starter node: FailedScheduling /
  Insufficient memory, Pending forever. Requests are now a small floor
  (250m / 512Mi, never above a configured cap) while the limits stay the
  safety caps.
- Kaniko re-copies the Dockerfile out of the context and chowns/chmods it to
  the source owner; a 65532-owned context (the distroless felis image uid)
  fails that under the pod's dropped capabilities ('copying dockerfile:
  chown /kaniko/Dockerfile: operation not permitted'). The fetch container
  now extracts as root — the uid Kaniko already runs as — so the copy
  succeeds; the pod was root by necessity regardless.
- Trivy's DB fetch is exactly what the build egress lock denies: the scan
  step failed closed on mirror.gcr.io. New [registry] trivy_db_repository
  renders --db-repository, and docs/troubleshooting.md §8e now carries the
  verified mirror recipe (docker pull/tag/push of aquasec/trivy-db:2 into the
  internal registry; --insecure already covers its plain HTTP).

Verified live after this batch: fetch initContainer streamed the blob through
the API + netpol + token, Kaniko built and pushed registry.felis.svc:5000/
user-uploads/sub-<id>:latest, and Trivy scanned against the mirrored DB.
parent f79e5ebb

cmd/felis/api.go

+3 −0
Changes for cmd/felis/api.go: 3 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -421,6 +421,9 @@ func buildConfig(cfg *config.Config) build.Config { 
		TrivyImage:  cfg.Registry.TrivyImage,

		CPULimit:    cfg.Registry.BuildCPULimit,

		MemLimit:    cfg.Registry.BuildMemLimit,

		// Empty keeps Trivy's own default; an install with builds points this at

		// the internal DB mirror (see config.RegistryConfig.TrivyDBRepository).

		TrivyDBRepository: cfg.Registry.TrivyDBRepository,

	}

}

docs/deferred-seams.md

+6 −3
Changes for docs/deferred-seams.md: 6 added lines, 3 removed lines.
Original line number Diff line number Diff line
@@ -55,9 +55,12 @@ A grep across `*.md` and `*.go` returns both sets; only the Go ones are seams. 
  filesystem view or object-store credentials. Kaniko/Trivy images are

  external-only by default; `[registry] kaniko_image / trivy_image /

  build_cpu_limit / build_mem_limit` override them for mirrored or air-gapped

  installs, and Trivy's vulnerability DB download needs the same treatment (a

  `package_source_cidrs` allowance or an internal `TRIVY_DB_REPOSITORY` mirror) or

  the scan step fails closed on an egress-locked install.

  installs. Trivy's vulnerability DB is the same story, and now has its own knob:

  `[registry] trivy_db_repository` points `--db-repository` at an internal mirror

  (recipe in docs/troubleshooting.md §8e). Left unset on an egress-locked box the

  scan step fails closed — Kaniko pushes, Trivy exits on the DB download — which

  is the correct fail direction but leaves the build unfinished, so the mirror is

  part of a production build install.



## Built; only its I/O is unverifiable from this repo

docs/troubleshooting.md

+22 −3
Changes for docs/troubleshooting.md: 22 added lines, 3 removed lines.
Original line number Diff line number Diff line
@@ -408,14 +408,33 @@ url = "registry.felis.svc:5000" 
build_namespace = "felis-build"

kaniko_image = "registry.felis.svc:5000/mirror/kaniko:v1.23.2"

trivy_image  = "registry.felis.svc:5000/mirror/trivy:0.58.1"

trivy_db_repository = "registry.felis.svc:5000/mirror/trivy-db:2"

build_cpu_limit = "2"

build_mem_limit = "4Gi"

```



then restart `felis-api` (it renders the Job from this config). Unset fields keep

the defaults. Note the user-modpack context topologies are a separate,

still-open seam (see `docs/deferred-seams.md`); this section only makes the

executors reachable.

the defaults.



`trivy_db_repository` is not optional on an egress-locked box. Trivy fetches its

vulnerability DB from `mirror.gcr.io`/`ghcr.io` unless told otherwise, and the

build egress policy denies those hosts — so the scan step fails closed

(`failed to download vulnerability DB`) and NO build ever completes, even though

Kaniko pushed the image. Mirror the DB into the internal registry once:



```

# On a host with internet + docker access to the cluster's registry

# (add its address to the daemon's insecure-registries first; the registry

# serves plain HTTP):

#   docker pull mirror.gcr.io/aquasec/trivy-db:2

#   docker tag  mirror.gcr.io/aquasec/trivy-db:2 <registry-addr>:5000/mirror/trivy-db:2

#   docker push <registry-addr>:5000/mirror/trivy-db:2

```



The Job's Trivy container already runs with `--insecure`, so the internal

registry's plain HTTP works for the DB pull exactly as it does for the scanned

image. Re-mirror the tag periodically (Trivy refreshes the DB several times a

day upstream; a stale mirror only means stale CVE data, never a failed gate).



---

internal/build/build.go

+6 −0
Changes for internal/build/build.go: 6 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -220,6 +220,11 @@ type Config struct { 
	// only when a build's ContextRef is an http(s) URL (the submit lane's derived

	// shape); an install that never builds user submissions can leave it empty.

	FelisImage string

	// TrivyDBRepository overrides where Trivy fetches its vulnerability DB

	// (--db-repository). Empty keeps Trivy's upstream default, which the build

	// egress lock denies — an install with builds must point this at an internal

	// mirror (see config.RegistryConfig.TrivyDBRepository).

	TrivyDBRepository string

	// KanikoImage / TrivyImage are the executor images.

	KanikoImage string

	TrivyImage  string
@@ -361,6 +366,7 @@ func (b *Builder) jobParams(bld *Build, cfg Config) JobParams { 
		ServiceAccount:    cfg.ServiceAccount,

		RegistryURL:       cfg.RegistryURL,

		FelisImage:        cfg.FelisImage,

		TrivyDBRepository: cfg.TrivyDBRepository,

		KanikoImage:       cfg.KanikoImage,

		TrivyImage:        cfg.TrivyImage,

		Deadline:          cfg.Deadline,

internal/build/jobspec.go

+58 −10
Changes for internal/build/jobspec.go: 58 added lines, 10 removed lines.
Original line number Diff line number Diff line
@@ -64,6 +64,10 @@ type JobParams struct { 
	// FelisImage runs the context-fetch initContainer (the felis binary's

	// fetch-context entrypoint). Required when ContextRef is an http(s) URL.

	FelisImage string

	// TrivyDBRepository overrides Trivy's vulnerability-DB source (the

	// --db-repository flag). Empty keeps Trivy's own default; see

	// build.Config.TrivyDBRepository for why an in-cluster install sets it.

	TrivyDBRepository string

	KanikoImage       string

	TrivyImage        string

	Deadline          time.Duration
@@ -134,6 +138,19 @@ func BuildJob(p JobParams) (*batchv1.Job, error) { 
			return nil, fmt.Errorf("build: context ref %q needs FelisImage for the fetch initContainer", p.ContextRef)

		}

		contextPath = contextMountPath

		// The fetch container runs as root while Kaniko keeps the image default

		// (also root): Kaniko re-copies the Dockerfile out of the context and

		// chowns/chmods it to the SOURCE file's owner, which fails for any other

		// owner without CAP_CHOWN/CAP_FOWNER — capabilities this pod deliberately

		// drops (the live drill hit exactly this: "copying dockerfile: chown

		// /kaniko/Dockerfile: operation not permitted" with the distroless uid

		// 65532). Extracting as root, the uid Kaniko itself runs as, keeps the

		// context owned by the only user that can satisfy that copy. The pod is

		// root by necessity regardless: Kaniko unpacks base-image layers into its

		// own filesystem.

		fetchSec := sec.DeepCopy()

		fetchSec.RunAsUser = int64Ptr(0)

		fetchSec.RunAsGroup = int64Ptr(0)

		fetch := corev1.Container{

			Name:  ContainerFetch,

			Image: p.FelisImage,
@@ -155,8 +172,8 @@ func BuildJob(p JobParams) (*batchv1.Job, error) { 
				}},

			}},

			VolumeMounts:    []corev1.VolumeMount{{Name: contextVolume, MountPath: contextMountPath}},

			Resources:       corev1.ResourceRequirements{Limits: limits, Requests: limits},

			SecurityContext: sec,

			Resources:       corev1.ResourceRequirements{Limits: limits, Requests: buildRequests(limits)},

			SecurityContext: fetchSec,

		}

		initContainers = append(initContainers, fetch)

		kanikoMounts = []corev1.VolumeMount{{Name: contextVolume, MountPath: contextMountPath, ReadOnly: true}}
@@ -185,23 +202,32 @@ func BuildJob(p JobParams) (*batchv1.Job, error) { 
			"--skip-tls-verify",

		},

		VolumeMounts:    kanikoMounts,

		Resources:       corev1.ResourceRequirements{Limits: limits, Requests: limits},

		Resources:       corev1.ResourceRequirements{Limits: limits, Requests: buildRequests(limits)},

		SecurityContext: sec,

	}

	initContainers = append(initContainers, kaniko)



	trivy := corev1.Container{

		Name:  ContainerTrivy,

		Image: p.TrivyImage,

		Args: []string{

	trivyArgs := []string{

		"image",

		"--exit-code", "1",

		"--severity", "CRITICAL",

		"--no-progress",

		"--insecure",

			p.ImageRef,

		},

		Resources:       corev1.ResourceRequirements{Limits: limits, Requests: limits},

	}

	// The DB source is configurable because the default (mirror.gcr.io/ghcr.io)

	// is exactly what the build egress lock denies: an install that never mirrors

	// the DB cannot complete a scan, and the gate fails closed on purpose. The

	// supported shape is the internal registry (`--insecure` above already covers

	// its plain HTTP).

	if p.TrivyDBRepository != "" {

		trivyArgs = append(trivyArgs, "--db-repository", p.TrivyDBRepository)

	}

	trivyArgs = append(trivyArgs, p.ImageRef)

	trivy := corev1.Container{

		Name:            ContainerTrivy,

		Image:           p.TrivyImage,

		Args:            trivyArgs,

		Resources:       corev1.ResourceRequirements{Limits: limits, Requests: buildRequests(limits)},

		SecurityContext: sec,

	}
@@ -391,6 +417,28 @@ func resourceLimits(cpu, mem string) (corev1.ResourceList, error) { 
	}, nil

}



// buildRequests is the scheduler floor a build container asks for while its

// configured limit stays the safety cap. Reserving the full cap as a request is

// what once made a default install on the platform's starter node (4 vCPU /

// 5.5 GiB) unable to schedule ANY build — caught by the live end-to-end drill, not

// by any unit test. A build is best-effort batch work: it may be throttled or

// evicted under contention, which fails the Job loudly, and the caps still stop a

// runaway build from exhausting the node.

func buildRequests(limits corev1.ResourceList) corev1.ResourceList {

	req := corev1.ResourceList{}

	for res, floor := range map[corev1.ResourceName]resource.Quantity{

		corev1.ResourceCPU:    resource.MustParse("250m"),

		corev1.ResourceMemory: resource.MustParse("512Mi"),

	} {

		limit, ok := limits[res]

		if ok && limit.Cmp(floor) < 0 {

			floor = limit // never ask for more than the cap

		}

		req[res] = floor

	}

	return req

}



func boolPtr(b bool) *bool    { return &b }

func int32Ptr(i int32) *int32 { return &i }