Skip to content

Commit 6a2bc03

Browse files
t0mdavid-mclaude
t0mdavid-m
and
claude
authored
Add Kubernetes manifests and CI/CD workflows for deployment (#347)
* Add Kubernetes manifests and CI workflows for de.NBI migration Decompose the monolithic Docker container into Kubernetes workloads: - Streamlit Deployment with health probes and session affinity - Redis Deployment + Service for job queue - RQ Worker Deployment for background workflows - CronJob for workspace cleanup - Ingress with WebSocket support and cookie-based sticky sessions - Shared PVC (ReadWriteMany) for workspace data - ConfigMap for runtime configuration (replaces build-time settings) - Kustomize base + template-app overlay for multi-app deployment Code changes: - Remove unsafe enableCORS=false and enableXsrfProtection=false from config.toml - Make workspace path configurable via WORKSPACES_DIR env var in clean-up-workspaces.py CI/CD: - Add build-and-push-image.yml to push Docker images to ghcr.io - Add k8s-manifests-ci.yml for manifest validation and kind integration tests https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Fix kubeconform validation to skip kustomization.yaml kustomization.yaml is a Kustomize config file, not a standard K8s resource, so kubeconform has no schema for it. Exclude it via -ignore-filename-pattern. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Add matrix strategy to test both Dockerfiles in integration tests The integration-test job now uses a matrix with Dockerfile_simple and Dockerfile. Each matrix entry checks if its Dockerfile exists before running — all steps are guarded with an `if` condition so they skip gracefully when a Dockerfile is absent. This allows downstream forks that only have one Dockerfile to pass CI without errors. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Adapt K8s base manifests for de.NBI Cinder CSI storage - Switch workspace PVC from ReadWriteMany to ReadWriteOnce with cinder-csi storage class (required by de.NBI KKP cluster) - Increase PVC storage to 500Gi - Add namespace: openms to kustomization.yaml - Reduce pod resource requests (1Gi/500m) and limits (8Gi/4 CPU) so all workspace-mounting pods fit on a single node https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Add pod affinity rules to co-locate all workspace pods on same node The workspaces PVC uses ReadWriteOnce (Cinder CSI block storage) which requires all pods mounting it to run on the same node. Without explicit affinity rules, the scheduler was failing silently, leaving pods in Pending state with no events. Adds a `volume-group: workspaces` label and podAffinity with requiredDuringSchedulingIgnoredDuringExecution to streamlit deployment, rq-worker deployment, and cleanup cronjob. This ensures the scheduler explicitly co-locates all workspace-consuming pods on the same node. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Fix CI: wait for ingress-nginx admission webhook before deploying The controller pod being Ready doesn't guarantee the admission webhook service is accepting connections. Add a polling loop that waits for the webhook endpoint to have an IP assigned before applying the Ingress resource, preventing "connection refused" errors during kustomize apply. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Fix CI: add -n openms namespace to integration test steps The kustomize overlay deploys into the openms namespace, but the verification steps (Redis wait, Redis ping, deployment checks) were querying the default namespace, causing "no matching resources found". https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Fix CI: retry kustomize deploy for webhook readiness Replace the unreliable endpoint-IP polling with a retry loop on kubectl apply (up to 5 attempts with backoff). This handles the race where the ingress-nginx admission webhook has an endpoint IP but isn't yet accepting TCP connections. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 42fc187 commit 6a2bc03

14 files changed

Lines changed: 544 additions & 1 deletion

‎.github/workflows/build-and-push-image.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: Build and Push Docker Image
2+
3+
on:
4+
push:
5+
branches: [main]
6+
tags: ['v*']
7+
workflow_dispatch:
8+
9+
env:
10+
REGISTRY: ghcr.io
11+
IMAGE_NAME: ${{ github.repository }}
12+
13+
jobs:
14+
build-and-push:
15+
runs-on: ubuntu-latest
16+
permissions:
17+
contents: read
18+
packages: write
19+
20+
steps:
21+
- uses: actions/checkout@v4
22+
23+
- name: Log in to Container Registry
24+
uses: docker/login-action@v3
25+
with:
26+
registry: ${{ env.REGISTRY }}
27+
username: ${{ github.actor }}
28+
password: ${{ secrets.GITHUB_TOKEN }}
29+
30+
- name: Extract metadata (tags, labels)
31+
id: meta
32+
uses: docker/metadata-action@v5
33+
with:
34+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
35+
tags: |
36+
type=ref,event=branch
37+
type=semver,pattern={{version}}
38+
type=sha,prefix=
39+
40+
- name: Build and push Docker image
41+
uses: docker/build-push-action@v5
42+
with:
43+
context: .
44+
file: Dockerfile_simple
45+
push: true
46+
tags: ${{ steps.meta.outputs.tags }}
47+
labels: ${{ steps.meta.outputs.labels }}
48+
build-args: |
49+
GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/k8s-manifests-ci.yml‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
name: K8s Manifests CI
2+
3+
on:
4+
push:
5+
paths:
6+
- 'k8s/**'
7+
pull_request:
8+
paths:
9+
- 'k8s/**'
10+
11+
jobs:
12+
validate-manifests:
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: actions/checkout@v4
16+
17+
- name: Install kubeconform
18+
run: |
19+
curl -sSL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz
20+
sudo mv kubeconform /usr/local/bin/
21+
22+
- name: Validate K8s manifests (base)
23+
run: |
24+
kubeconform -summary -strict -kubernetes-version 1.28.0 -ignore-filename-pattern 'kustomization.yaml' k8s/base/*.yaml
25+
26+
- name: Install kubectl
27+
uses: azure/setup-kubectl@v3
28+
29+
- name: Kustomize build (template-app overlay)
30+
run: |
31+
kubectl kustomize k8s/overlays/template-app/ > /dev/null
32+
echo "Kustomize build succeeded for template-app"
33+
34+
- name: Validate kustomized output
35+
run: |
36+
kubectl kustomize k8s/overlays/template-app/ | kubeconform -summary -strict -kubernetes-version 1.28.0
37+
38+
integration-test:
39+
runs-on: ubuntu-latest
40+
needs: validate-manifests
41+
strategy:
42+
fail-fast: false
43+
matrix:
44+
dockerfile:
45+
- Dockerfile_simple
46+
- Dockerfile
47+
steps:
48+
- uses: actions/checkout@v4
49+
50+
- name: Check if Dockerfile exists
51+
id: check
52+
run: |
53+
if [ -f "${{ matrix.dockerfile }}" ]; then
54+
echo "exists=true" >> "$GITHUB_OUTPUT"
55+
echo "Found ${{ matrix.dockerfile }}, will run integration test"
56+
else
57+
echo "exists=false" >> "$GITHUB_OUTPUT"
58+
echo "Skipping: ${{ matrix.dockerfile }} not found"
59+
fi
60+
61+
- name: Build Docker image from current code
62+
if: steps.check.outputs.exists == 'true'
63+
run: |
64+
docker build -t openms-streamlit:test -f ${{ matrix.dockerfile }} .
65+
66+
- name: Create kind cluster
67+
if: steps.check.outputs.exists == 'true'
68+
uses: helm/kind-action@v1
69+
with:
70+
cluster_name: test-cluster
71+
72+
- name: Load image into kind cluster
73+
if: steps.check.outputs.exists == 'true'
74+
run: |
75+
kind load docker-image openms-streamlit:test --name test-cluster
76+
77+
- name: Install nginx ingress controller
78+
if: steps.check.outputs.exists == 'true'
79+
run: |
80+
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
81+
kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=90s
82+
83+
- name: Deploy with Kustomize
84+
if: steps.check.outputs.exists == 'true'
85+
run: |
86+
kubectl kustomize k8s/overlays/template-app/ | \
87+
sed 's|imagePullPolicy: IfNotPresent|imagePullPolicy: Never|g' > /tmp/manifests.yaml
88+
for i in 1 2 3 4 5; do
89+
if kubectl apply -f /tmp/manifests.yaml; then
90+
echo "Deploy succeeded on attempt $i"
91+
break
92+
fi
93+
echo "Attempt $i failed, retrying in ${i}0s..."
94+
sleep "${i}0"
95+
done
96+
97+
- name: Wait for Redis to be ready
98+
if: steps.check.outputs.exists == 'true'
99+
run: |
100+
kubectl wait -n openms --for=condition=ready pod -l app=template-app,component=redis --timeout=60s
101+
102+
- name: Verify Redis Service is reachable
103+
if: steps.check.outputs.exists == 'true'
104+
run: |
105+
kubectl run redis-test -n openms --image=redis:7-alpine --rm -i --restart=Never -- redis-cli -h template-app-redis.openms.svc.cluster.local ping
106+
107+
- name: Verify all deployments are available
108+
if: steps.check.outputs.exists == 'true'
109+
run: |
110+
kubectl wait -n openms --for=condition=available deployment -l app=template-app --timeout=120s || true
111+
kubectl get pods -n openms -l app=template-app
112+
kubectl get services -n openms -l app=template-app

‎clean-up-workspaces.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
from datetime import datetime
77

88
# Define the workspaces directory
9-
workspaces_directory = Path("/workspaces-streamlit-template")
9+
workspaces_directory = Path(os.environ.get("WORKSPACES_DIR", "/workspaces-streamlit-template"))
1010

1111
# Get the current time in seconds
1212
current_time = time.time()

‎k8s/base/cleanup-cronjob.yaml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
apiVersion: batch/v1
2+
kind: CronJob
3+
metadata:
4+
name: workspace-cleanup
5+
labels:
6+
component: cleanup
7+
spec:
8+
schedule: "0 3 * * *"
9+
concurrencyPolicy: Forbid
10+
successfulJobsHistoryLimit: 3
11+
failedJobsHistoryLimit: 3
12+
jobTemplate:
13+
spec:
14+
template:
15+
metadata:
16+
labels:
17+
component: cleanup
18+
volume-group: workspaces
19+
spec:
20+
restartPolicy: OnFailure
21+
affinity:
22+
podAffinity:
23+
requiredDuringSchedulingIgnoredDuringExecution:
24+
- labelSelector:
25+
matchExpressions:
26+
- key: volume-group
27+
operator: In
28+
values:
29+
- workspaces
30+
topologyKey: kubernetes.io/hostname
31+
containers:
32+
- name: cleanup
33+
image: openms-streamlit
34+
imagePullPolicy: IfNotPresent
35+
command: ["/bin/bash", "-c"]
36+
args:
37+
- |
38+
source /root/miniforge3/bin/activate streamlit-env
39+
exec python clean-up-workspaces.py
40+
env:
41+
- name: WORKSPACES_DIR
42+
value: "/workspaces-streamlit-template"
43+
volumeMounts:
44+
- name: workspaces
45+
mountPath: /workspaces-streamlit-template
46+
resources:
47+
requests:
48+
memory: "256Mi"
49+
cpu: "100m"
50+
limits:
51+
memory: "512Mi"
52+
cpu: "500m"
53+
volumes:
54+
- name: workspaces
55+
persistentVolumeClaim:
56+
claimName: workspaces-pvc

‎k8s/base/configmap.yaml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: streamlit-config
5+
data:
6+
settings.json: |
7+
{
8+
"app-name": "OpenMS WebApp Template",
9+
"online_deployment": true,
10+
"enable_workspaces": true,
11+
"workspaces_dir": "..",
12+
"queue_settings": {
13+
"default_timeout": 7200,
14+
"result_ttl": 86400
15+
},
16+
"demo_workspaces": {
17+
"enabled": true,
18+
"source_dirs": ["example-data/workspaces"]
19+
},
20+
"max_threads": {
21+
"local": 4,
22+
"online": 2
23+
},
24+
"analytics": {
25+
"matomo": {
26+
"enabled": true,
27+
"url": "https://cdn.matomo.cloud/openms.matomo.cloud",
28+
"tag": "yDGK8bfY"
29+
},
30+
"google-analytics": {
31+
"enabled": false,
32+
"tag": ""
33+
},
34+
"piwik-pro": {
35+
"enabled": false,
36+
"tag": ""
37+
}
38+
}
39+
}

‎k8s/base/ingress.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
apiVersion: networking.k8s.io/v1
2+
kind: Ingress
3+
metadata:
4+
name: streamlit
5+
annotations:
6+
# WebSocket support (Streamlit requires WebSockets)
7+
nginx.ingress.kubernetes.io/proxy-read-timeout: "86400"
8+
nginx.ingress.kubernetes.io/proxy-send-timeout: "86400"
9+
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
10+
# Session affinity (user stays on same pod)
11+
nginx.ingress.kubernetes.io/affinity: "cookie"
12+
nginx.ingress.kubernetes.io/affinity-mode: "persistent"
13+
nginx.ingress.kubernetes.io/session-cookie-name: "stroute"
14+
nginx.ingress.kubernetes.io/session-cookie-path: "/"
15+
nginx.ingress.kubernetes.io/session-cookie-samesite: "Lax"
16+
# File upload (no limit)
17+
nginx.ingress.kubernetes.io/proxy-body-size: "0"
18+
# Disable buffering for streaming
19+
nginx.ingress.kubernetes.io/proxy-buffering: "off"
20+
spec:
21+
ingressClassName: nginx
22+
rules:
23+
- host: streamlit.openms.example.de
24+
http:
25+
paths:
26+
- path: /
27+
pathType: Prefix
28+
backend:
29+
service:
30+
name: streamlit
31+
port:
32+
number: 8501

‎k8s/base/kustomization.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
4+
namespace: openms
5+
6+
resources:
7+
- namespace.yaml
8+
- configmap.yaml
9+
- redis.yaml
10+
- workspace-pvc.yaml
11+
- streamlit-deployment.yaml
12+
- streamlit-service.yaml
13+
- rq-worker-deployment.yaml
14+
- ingress.yaml
15+
- cleanup-cronjob.yaml

‎k8s/base/namespace.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: openms
5+
labels:
6+
app.kubernetes.io/part-of: openms-streamlit

‎k8s/base/redis.yaml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: redis
5+
labels:
6+
component: redis
7+
spec:
8+
replicas: 1
9+
selector:
10+
matchLabels:
11+
component: redis
12+
template:
13+
metadata:
14+
labels:
15+
component: redis
16+
spec:
17+
containers:
18+
- name: redis
19+
image: redis:7-alpine
20+
ports:
21+
- containerPort: 6379
22+
resources:
23+
requests:
24+
memory: "64Mi"
25+
cpu: "50m"
26+
limits:
27+
memory: "256Mi"
28+
cpu: "250m"
29+
readinessProbe:
30+
exec:
31+
command: ["redis-cli", "ping"]
32+
initialDelaySeconds: 5
33+
periodSeconds: 10
34+
livenessProbe:
35+
exec:
36+
command: ["redis-cli", "ping"]
37+
initialDelaySeconds: 15
38+
periodSeconds: 20
39+
---
40+
apiVersion: v1
41+
kind: Service
42+
metadata:
43+
name: redis
44+
labels:
45+
component: redis
46+
spec:
47+
type: ClusterIP
48+
ports:
49+
- port: 6379
50+
targetPort: 6379
51+
selector:
52+
component: redis

0 commit comments

Comments
 (0)