Merge pull request #375 from OpenMS/claude/parallel-webapp-memory-optimization-RoNnJ

Refactor K8s deployment to use memory-tier components

Author: t0mdavid-m

.claude/skills/configure-k8s-deployment.md

@@ -20,13 +20,9 @@ Run `configure-app-settings` first. This skill assumes `settings.json`, the Dock
 
    Push your changes to `main` or create a tag. The workflow `.github/workflows/build-and-test.yml` builds both the full and lightweight (`-simple`) variants and publishes them to `ghcr.io/<your-org>/<your-repo>:<tag>-full` and `:<tag>-simple` respectively. The unsuffixed `latest` tag tracks the full variant on the `main` branch.
 
-2. **Copy the template overlay.**
+2. **Edit `k8s/overlays/prod/kustomization.yaml` in place.**
 
-    ```bash
-    cp -r k8s/overlays/template-app k8s/overlays/<your-app-name>
-    ```
-
-3. **Edit `k8s/overlays/<your-app-name>/kustomization.yaml`**:
+   Each fork ships a single production overlay at `k8s/overlays/prod/`. Edit the existing file rather than copying — the repo itself identifies the app, so there is no per-app overlay subdirectory.
 
    - Change `namePrefix` from `template-app-` to `<your-app-name>-`
    - Change `commonLabels.app` from `template-app` to `<your-app-name>`
@@ -38,18 +34,31 @@ Run `configure-app-settings` first. This skill assumes `settings.json`, the Dock
 
    The overlay leaves the nginx `Ingress` unpatched because production deployments use Traefik. If you are deploying to an nginx-only cluster, add an overlay patch for both `rules[].host` entries in the base `Ingress` (same `.de` / `.org` pattern) instead of the IngressRoute.
 
+3. **Select a memory tier.**
+
+   The overlay pulls in one of two Kustomize components under `components:`. The default `memory-tier-low` pins the app to the low-mem node (~64 GB) with modest resource requests/limits — correct for most template-based apps. Switch to `memory-tier-high` only if the workload genuinely needs the large-memory node (DIA spectral-library construction, OpenSwath peak picking, DIA-LFQ):
+
+   ```yaml
+   components:
+     - ../../components/memory-tier-low    # default: light app
+     # OR
+     - ../../components/memory-tier-high   # memory-intensive app
+   ```
+
+   The cluster node labels `openms.de/memory-tier=low` and `openms.de/memory-tier=high` must already exist; the tier component adds the matching `nodeSelector`.
+
 4. **Validate the overlay builds.**
 
     ```bash
-    kubectl kustomize k8s/overlays/<your-app-name>/
+    kubectl kustomize k8s/overlays/prod/
     ```
 
     Should print the rendered manifests with no errors.
 
 5. **Deploy.**
 
     ```bash
-    kubectl apply -k k8s/overlays/<your-app-name>/
+    kubectl apply -k k8s/overlays/prod/
     ```
 
 6. **Verify.**
@@ -63,18 +72,19 @@ Run `configure-app-settings` first. This skill assumes `settings.json`, the Dock
 
 ## Reference Files
 
-- Overlay template: `k8s/overlays/template-app/kustomization.yaml`
+- Overlay: `k8s/overlays/prod/kustomization.yaml`
+- Memory-tier components: `k8s/components/memory-tier-{low,high}/`
 - Base manifests: `k8s/base/*.yaml`
 - CI: `.github/workflows/build-and-test.yml` (unified build + lint + kind integration), `.github/workflows/ghcr-cleanup.yml` (scheduled tag retention)
 - Full reference: see the "Developers Guide: Kubernetes Deployment" Documentation page in the running Streamlit app.
 
 ## Checklist
 
 - [ ] Image built and pushed to GHCR (via CI or manual push to `main`/tag)
-- [ ] Overlay copied to `k8s/overlays/<your-app-name>/`
-- [ ] `namePrefix`, `commonLabels.app`, `images[0].newName` updated
+- [ ] `namePrefix`, `commonLabels.app`, `images[0].newName` updated in `k8s/overlays/prod/kustomization.yaml`
 - [ ] IngressRoute patch updated (both hostnames + service reference)
 - [ ] Redis URL updated in both Deployment patches
+- [ ] Memory tier selected (`memory-tier-low` or `memory-tier-high`)
 - [ ] `kubectl kustomize` succeeds
 - [ ] `kubectl apply -k` succeeds
 - [ ] All pods Running, `rollout status` succeeds

.github/kind-config.yaml

@@ -0,0 +1,27 @@
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+# Two-node cluster that mirrors the production memory-tier topology, so
+# the Build-and-Test job passes regardless of which tier a fork selects
+# in its overlay (memory-tier-low or memory-tier-high). Without both
+# labels present, flipping the overlay would leave pods Pending on the
+# single kind node.
+nodes:
+  - role: control-plane
+    # Multi-node kind clusters taint the control-plane with
+    # node-role.kubernetes.io/control-plane:NoSchedule. Clear it so
+    # app pods with nodeSelector memory-tier=low can actually land
+    # here (single-node kind had no such taint, hence the original
+    # workflow passed without this patch).
+    kubeadmConfigPatches:
+      - |
+        kind: InitConfiguration
+        nodeRegistration:
+          taints: []
+    labels:
+      openms.de/memory-tier: low
+      # ingress-ready is required by the kind variant of the
+      # ingress-nginx deploy manifest applied in CI.
+      ingress-ready: "true"
+  - role: worker
+    labels:
+      openms.de/memory-tier: high

.github/workflows/build-and-test.yml

@@ -35,7 +35,7 @@ jobs:
 
       - name: Validate kustomized overlay output
         run: |
-          kubectl kustomize k8s/overlays/template-app/ | \
+          kubectl kustomize k8s/overlays/prod/ | \
             kubeconform -summary -strict -kubernetes-version 1.28.0 -skip IngressRoute
 
   build:
@@ -105,6 +105,7 @@ jobs:
         uses: helm/kind-action@v1
         with:
           cluster_name: test-cluster
+          config: .github/kind-config.yaml
 
       - name: Load image into kind cluster
         run: kind load docker-image openms-streamlit:test --name test-cluster
@@ -117,7 +118,7 @@ jobs:
       - name: Deploy with Kustomize
         run: |
           # Filter out Traefik IngressRoute (kind cluster uses nginx) and force imagePullPolicy=Never
-          kubectl kustomize k8s/overlays/template-app/ | \
+          kubectl kustomize k8s/overlays/prod/ | \
             yq 'select(.kind != "IngressRoute")' | \
             sed 's|imagePullPolicy: IfNotPresent|imagePullPolicy: Never|g' | \
             sed 's|storageClassName: cinder-csi|storageClassName: standard|g' > /tmp/manifests.yaml
@@ -176,6 +177,7 @@ jobs:
         uses: helm/kind-action@v1
         with:
           cluster_name: traefik-test
+          config: .github/kind-config.yaml
 
       - name: Load image into kind cluster
         run: kind load docker-image openms-streamlit:test --name traefik-test
@@ -194,7 +196,7 @@ jobs:
 
       - name: Deploy with Kustomize (full manifests, no filter)
         run: |
-          kubectl kustomize k8s/overlays/template-app/ | \
+          kubectl kustomize k8s/overlays/prod/ | \
             sed 's|imagePullPolicy: IfNotPresent|imagePullPolicy: Never|g' | \
             sed 's|storageClassName: cinder-csi|storageClassName: standard|g' > /tmp/manifests.yaml
           for i in 1 2 3 4 5; do

docs/kubernetes-deployment.md

@@ -162,15 +162,13 @@ Update `settings.json`, choose a Dockerfile, and update `README.md`. If you are
 
 Push your changes to `main` or create a tag. The workflow `.github/workflows/build-and-test.yml` builds both the full (`Dockerfile`) and lightweight (`Dockerfile_simple`) variants and pushes each to `ghcr.io/<your-org>/<your-repo>` with variant-suffixed tags: `<branch>-full` / `<branch>-simple`, `v<version>-full` / `v<version>-simple`, and `<sha>-full` / `<sha>-simple`. The unsuffixed `latest` tag tracks the full variant on `main`.
 
-### Step 3 — Create your overlay
+### Step 3 — Edit the production overlay
 
-```bash
-cp -r k8s/overlays/template-app k8s/overlays/<your-app-name>
-```
+Each fork ships a single production overlay at `k8s/overlays/prod/`. Edit this file in place — the forked repository itself identifies the app, so no per-app overlay subdirectory is created.
 
 ### Step 4 — Edit `kustomization.yaml`
 
-Open `k8s/overlays/<your-app-name>/kustomization.yaml` and change the following fields:
+Open `k8s/overlays/prod/kustomization.yaml` and change the following fields:
 
 | Field | Set to |
 |-------|--------|
@@ -184,6 +182,19 @@ Open `k8s/overlays/<your-app-name>/kustomization.yaml` and change the following
 
 The overlay leaves the nginx `Ingress` unpatched because Traefik is the production ingress. If you are deploying to an nginx-only cluster, add an overlay patch for both `rules[].host` entries in the base `Ingress` (same `.de` / `.org` pattern) instead of the IngressRoute patch.
 
+### Step 4b — Select a memory tier
+
+The overlay pulls in one of two Kustomize components under `components:`:
+
+```yaml
+components:
+  - ../../components/memory-tier-low    # default: light app on low-mem node
+  # OR
+  - ../../components/memory-tier-high   # memory-intensive app on high-mem node
+```
+
+`memory-tier-low` is the right choice for most apps. Switch to `memory-tier-high` only if the workload genuinely needs tens of GB of RAM (DIA spectral-library + OpenSwath peak picking, DIA-LFQ). The tier component adds the matching `nodeSelector: openms.de/memory-tier=<tier>` plus `requests`/`limits` sized for that node, so cluster nodes must already be labelled `openms.de/memory-tier=low` / `...=high`.
+
 ### Step 5 — Configure the admin password (optional)
 
 Skip this step if you don't need the "Save as Demo" feature. The Streamlit Deployment mounts the `streamlit-secrets` Secret with `optional: true`, so the pod starts either way — the admin UI simply reports "Admin not configured" when the Secret is absent.
@@ -213,18 +224,18 @@ Because the Secret is created outside Kustomize, the overlay's `namePrefix` does
 Copy the template and fill it in:
 
 ```bash
-cp k8s/base/streamlit-secrets.yaml.example k8s/overlays/<your-app-name>/streamlit-secrets.yaml
+cp k8s/base/streamlit-secrets.yaml.example k8s/overlays/prod/streamlit-secrets.yaml
 # edit the password
 ```
 
-Add `- streamlit-secrets.yaml` to the `resources:` list in `k8s/overlays/<your-app-name>/kustomization.yaml`. The filename `streamlit-secrets.yaml` is gitignored (`.gitignore`: `k8s/**/streamlit-secrets.yaml`); always confirm with `git status` before committing — never commit a filled-in copy.
+Add `- streamlit-secrets.yaml` to the `resources:` list in `k8s/overlays/prod/kustomization.yaml`. The filename `streamlit-secrets.yaml` is gitignored (`.gitignore`: `k8s/**/streamlit-secrets.yaml`); always confirm with `git status` before committing — never commit a filled-in copy.
 
 When the Secret is managed through Kustomize, the overlay's `namePrefix` rewrites both the Secret name and the Deployment reference, so no manual renaming is needed.
 
 ### Step 6 — Deploy
 
 ```bash
-kubectl apply -k k8s/overlays/<your-app-name>/
+kubectl apply -k k8s/overlays/prod/
 ```
 
 ### Step 7 — Verify
@@ -252,13 +263,13 @@ One unified workflow owns manifest lint, Docker build, push, and kind integratio
 - **Trigger:** pull request to `main`, push to `main`, push of a `v*` tag, or manual workflow dispatch.
 - **Job 1 — `lint-manifests`:**
   - `kubeconform` runs against `k8s/base/*.yaml` with strict mode and Kubernetes 1.28 schemas (excluding `kustomization.yaml` and the Traefik CRD `traefik-ingressroute.yaml`).
-  - `kubectl kustomize k8s/overlays/template-app/` must succeed; the kustomized output is re-validated through `kubeconform` (with `IngressRoute` skipped).
+  - `kubectl kustomize k8s/overlays/prod/` must succeed; the kustomized output is re-validated through `kubeconform` (with `IngressRoute` skipped).
   - Takes ~30s. Fails fast so manifest typos never trigger the hours-long full Docker build.
 - **Job 2 — `build`** (`needs: lint-manifests`, matrix over `[full, simple]`):
   - Builds `Dockerfile` (full, includes TOPP tools) or `Dockerfile_simple` (pyOpenMS only) depending on the matrix leg.
   - **Buildx registry cache** (`type=registry,…,mode=max`) stored at `ghcr.io/<repo>/cache:full` and `:simple`. A `cache-from` read is attempted on every event; `cache-to` write only on push/tag/workflow_dispatch (fork PRs can't write). Repeat builds with an unchanged Dockerfile finish in minutes.
   - **Push** on push/tag/workflow_dispatch events (not on PRs). Tags: `<branch>-full` / `<branch>-simple`, `v<version>-full` / `v<version>-simple`, `<sha>-full` / `<sha>-simple`. `latest` is emitted only for the full variant on push to `main`.
-  - **Kind integration** runs per variant: creates a kind cluster, loads the just-built image, installs the nginx ingress controller, applies the kustomized `template-app` overlay (filtering Traefik `IngressRoute`, forcing `imagePullPolicy: Never` and `storageClassName: standard`), asserts Redis + deployments become ready, and curls both `.de` and `.org` hostnames through the nginx ingress to verify dual-host routing.
+  - **Kind integration** runs per variant: creates a kind cluster, loads the just-built image, installs the nginx ingress controller, applies the kustomized `prod` overlay (filtering Traefik `IngressRoute`, forcing `imagePullPolicy: Never` and `storageClassName: standard`), asserts Redis + deployments become ready, and curls both `.de` and `.org` hostnames through the nginx ingress to verify dual-host routing.
 - **Job 3 — `traefik-integration`** (`needs: lint-manifests`, runs once on `Dockerfile_simple`): builds the simple image, brings up a second kind cluster, installs Traefik via Helm (`service.type=ClusterIP`), applies the full kustomized overlay without filtering the `IngressRoute` (still patching `imagePullPolicy: Never` and `storageClassName: standard` for kind compatibility), and curls both hostnames through Traefik. Catches IngressRoute-syntax regressions that the nginx-side test cannot.
 - **Auth:** uses the workflow's `GITHUB_TOKEN` for GHCR login and as a build argument for in-image private-resource access. Fork PRs skip login (their `GITHUB_TOKEN` is read-only) but can still read the public cache.
 - **PR behavior:** all three jobs run on pull requests. No tags are pushed and no cache is written. The kind integration still runs, exercising manifests end-to-end. If branch protection requires these checks, a failure blocks merge.

k8s/base/kustomization.yaml

@@ -14,3 +14,4 @@ resources:
   - ingress.yaml
   - traefik-ingressroute.yaml
   - cleanup-cronjob.yaml
+  - limitrange.yaml

k8s/base/limitrange.yaml

@@ -0,0 +1,16 @@
+apiVersion: v1
+kind: LimitRange
+metadata:
+  name: default-container-limits
+spec:
+  limits:
+    - type: Container
+      defaultRequest:
+        memory: "512Mi"
+        cpu: "250m"
+      default:
+        memory: "2Gi"
+        cpu: "2"
+      max:
+        memory: "200Gi"
+        cpu: "20"

k8s/base/redis.yaml

@@ -21,11 +21,11 @@ spec:
             - containerPort: 6379
           resources:
             requests:
-              memory: "64Mi"
-              cpu: "50m"
+              memory: "256Mi"
+              cpu: "100m"
             limits:
               memory: "256Mi"
-              cpu: "250m"
+              cpu: "100m"
           readinessProbe:
             exec:
               command: ["redis-cli", "ping"]

k8s/base/rq-worker-deployment.yaml

@@ -48,13 +48,6 @@ spec:
               mountPath: /app/settings-overrides.json
               subPath: settings-overrides.json
               readOnly: true
-          resources:
-            requests:
-              memory: "1Gi"
-              cpu: "500m"
-            limits:
-              memory: "8Gi"
-              cpu: "4"
       volumes:
         - name: workspaces
           persistentVolumeClaim:

k8s/base/streamlit-deployment.yaml

@@ -65,13 +65,6 @@ spec:
               port: 8501
             initialDelaySeconds: 30
             periodSeconds: 30
-          resources:
-            requests:
-              memory: "1Gi"
-              cpu: "500m"
-            limits:
-              memory: "8Gi"
-              cpu: "4"
       volumes:
         - name: workspaces
           persistentVolumeClaim:

k8s/components/memory-tier-high/kustomization.yaml

@@ -0,0 +1,15 @@
+apiVersion: kustomize.config.k8s.io/v1alpha1
+kind: Component
+
+patches:
+  - path: nodeselector.yaml
+    target:
+      kind: Deployment
+  - path: streamlit-resources.yaml
+    target:
+      kind: Deployment
+      name: streamlit
+  - path: worker-resources.yaml
+    target:
+      kind: Deployment
+      name: rq-worker