@@ -5,7 +5,60 @@ metadata:
55 labels :
66 component : rq-worker
77spec :
8- replicas : 1
8+ # Two workers, not one. RQ runs one job per worker process, so `replicas: 1`
9+ # serialised every workflow in a deployment however much hardware sat
10+ # underneath it - and on the de.NBI pair the second node could not have been
11+ # used anyway, because the memory-tier component pinned every Deployment to
12+ # one labelled node and the workspace claim was ReadWriteOnce on a Cinder
13+ # volume that attaches to exactly one node. Both of those are gone. This is
14+ # the line that actually spends the second node.
15+ #
16+ # Fixed rather than autoscaled: each worker is Guaranteed QoS at whichever
17+ # size k8s/components/memory-tier-*/worker-resources.yaml gives it, so N
18+ # replicas reserve N times that size for as long as they run. The count is a
19+ # capacity decision against real nodes - see the worker-size and
20+ # worker-replica questions in .claude/skills/configure-k8s-deployment.md.
21+ #
22+ # 2 is one per node on the production pair, which is the point at which the
23+ # spread constraint below has anything to do. A higher count is legal
24+ # (maxSkew 1 permits 2/1 across two nodes) but only pays off where a node can
25+ # hold two workers of this size at once. It is also the first configuration
26+ # in which two workflow runs are genuinely concurrent: everything they share
27+ # goes through the workspace volume, so a fork that adds shared mutable state
28+ # outside it has no serialisation left to hide behind.
29+ replicas : 2
30+ # Explicit, because the default is wrong here in a way that only shows up
31+ # once replicas > 1. RollingUpdate defaults to 25%/25%, which against 2
32+ # replicas rounds to maxSurge 1 and maxUnavailable **0** (maxUnavailable
33+ # rounds down, and the 0/0 fallback does not apply because surge is 1). A
34+ # Deployment's Available condition needs `replicas - maxUnavailable` ready,
35+ # so with 0 the whole Deployment goes non-Available the moment either worker
36+ # is Pending or NotReady - and both of those are routine here rather than
37+ # exceptional:
38+ # - whenUnsatisfiable: DoNotSchedule below parks a replica in Pending
39+ # during any node drain, since a cordoned node still counts as a domain
40+ # under the default nodeTaintsPolicy;
41+ # - the readinessProbe below fails every worker whose mount is blocked
42+ # during a routine Ganesha restart.
43+ # Either would then hang `kubectl rollout status deployment/<slug>-rq-worker`
44+ # and fail CI's `kubectl wait --for=condition=available`, turning a
45+ # degradation that the design calls survivable into a blocked rollout.
46+ #
47+ # maxUnavailable 1: one worker down is a queue running at half rate, which is
48+ # what "visible and recoverable" has to mean if it is to mean anything. That
49+ # the OTHER replica is up is not weakened by this - it is asserted directly,
50+ # against .spec.replicas, by assert_workers_spread_across_nodes.
51+ #
52+ # maxSurge 0: a surge replica needs a whole extra worker's worth of memory
53+ # free somewhere, and these are Guaranteed pods sized to most of a node
54+ # (180Gi in the high tier). Surging first would leave the new pod Pending on
55+ # a cluster that has exactly enough room for the steady-state count, which is
56+ # the normal case. Terminate-then-replace instead.
57+ strategy :
58+ type : RollingUpdate
59+ rollingUpdate :
60+ maxSurge : 0
61+ maxUnavailable : 1
962 selector :
1063 matchLabels :
1164 component : rq-worker
1467 labels :
1568 component : rq-worker
1669 spec :
70+ # Spread the workers over nodes: one each until every node has one.
71+ #
72+ # maxSkew is measured over ELIGIBLE domains, which is why this had to
73+ # wait for the nodeSelector deletion rather than shipping alongside it.
74+ # With a selector matching exactly one node there is one eligible domain,
75+ # every distribution has skew 0, and a constraint written here would have
76+ # passed while changing nothing at all.
77+ #
78+ # DoNotSchedule, not ScheduleAnyway: soft spreading is a scoring
79+ # preference the scheduler can trade away against other priorities, and a
80+ # worker that usually spreads is a second node that is used right up
81+ # until the day it matters.
82+ #
83+ # The cost is a Pending replica during node maintenance: with the default
84+ # nodeTaintsPolicy a cordoned node still counts as a domain, so a drain
85+ # parks the evicted worker in Pending rather than doubling it up on the
86+ # survivor. That is visible in `kubectl get pods` and it clears when the
87+ # node comes back - but only because `strategy.maxUnavailable` above is
88+ # 1. At the default of 0 for this replica count the same Pending replica
89+ # would take the Deployment out of Available and hang every subsequent
90+ # `kubectl rollout status`, which is a blocked rollout, not a
91+ # degradation. The two settings are a pair; changing one without the
92+ # other is what makes this bite. Add `nodeTaintsPolicy: Honor` here
93+ # (Kubernetes >= 1.27) if node maintenance becomes routine enough that
94+ # even the Pending replica is unwelcome.
95+ #
96+ # The selector names `component` alone. The overlay's
97+ # `commonLabels: {app: <slug>}` is copied into it by kustomize, whose
98+ # commonLabels field specs reach
99+ # topologySpreadConstraints/labelSelector/matchLabels with create: false
100+ # (verified against the kustomize 5.7.1 built into kubectl). That scoping
101+ # is load-bearing: every fork of this template deploys into the same
102+ # `openms` namespace and topology spread counts pods per namespace, so
103+ # without the app label one fork's workers would be counted against
104+ # another's skew. A fork that drops commonLabels from its overlay loses
105+ # that scoping, and the constraint starts balancing every rq-worker in
106+ # the namespace as though they were all its own.
107+ topologySpreadConstraints :
108+ - maxSkew : 1
109+ topologyKey : kubernetes.io/hostname
110+ whenUnsatisfiable : DoNotSchedule
111+ labelSelector :
112+ matchLabels :
113+ component : rq-worker
17114 containers :
18115 - name : rq-worker
19116 image : openms-streamlit
@@ -30,6 +127,93 @@ spec:
30127 value : " redis://redis:6379/0"
31128 - name : WORKSPACES_DIR
32129 value : " /workspaces-streamlit-template"
130+ # Downward API. The storage heartbeat is keyed per node
131+ # (storage:ok:<node>) and the node registration alongside it
132+ # (storage:node:<node>) is what makes a *missing* heartbeat
133+ # readable, so a healthy node cannot mask a node whose mount has
134+ # wedged - with one shared key the indicator would stay green
135+ # straight through a real outage. Without this env var
136+ # health.current_node_name() falls back to the pod hostname, which
137+ # changes on every restart and keys the heartbeat per pod instead
138+ # of per node.
139+ - name : NODE_NAME
140+ valueFrom :
141+ fieldRef :
142+ fieldPath : spec.nodeName
143+ # Readiness only, and deliberately no liveness counterpart: a
144+ # liveness failure would SIGKILL the container mid-TOPP-job, and a
145+ # restart cannot fix a wedged NFS mount - it would destroy hours of
146+ # work while the fault persists. rq-worker sits behind no Service, so
147+ # readiness has no traffic effect either way; it is purely an
148+ # alertable signal, and it doubles as the writer of the sidebar's
149+ # storage heartbeat (src/workflow/health.py).
150+ #
151+ # One process, not a shell pipeline, because the ordering inside it
152+ # is the contract and belongs somewhere a test can drive it
153+ # (tests/test_storage_health.py). `probe_storage()` registers the
154+ # node, *then* writes to the volume, and publishes the heartbeat only
155+ # if that write succeeded. Nothing here is wrapped in `|| true`: a
156+ # `|| true` on the storage check would report Ready on a wedged mount
157+ # and publish a green heartbeat with it, which is the whole failure
158+ # this probe exists to catch.
159+ #
160+ # A write, not a `stat`: with default `actimeo` a stat can be answered
161+ # from the client's attribute cache while the server is gone, which
162+ # is a false green. A write always reaches the server, and it creates
163+ # the sentinel on a fresh volume instead of failing forever on a file
164+ # nothing else makes. The file is dot-named and not a directory, so
165+ # clean-up-workspaces.py skips it twice over.
166+ #
167+ # The interpreter is named explicitly rather than activated: `conda
168+ # activate` costs a few hundred ms of shell hooks on every probe, and
169+ # this is the same env the container command activates one screen up.
170+ #
171+ # Timing, against the ~90s NFSv4 grace period a Ganesha restart
172+ # imposes - all I/O blocks until it ends, so the numbers have to
173+ # outlast it or every routine restart flaps the worker:
174+ # inner timeout 15s - a cold interpreter plus `import redis` is
175+ # ~200ms idle, but this pod is by design
176+ # CPU-saturated running TOPP tools under a CFS
177+ # quota, and the two Redis calls are bounded at
178+ # health.REDIS_SOCKET_TIMEOUT (2s) each. The
179+ # budget has to cover all of that *and* leave
180+ # room for a slow-but-alive mount, or the probe
181+ # reports a storage fault that is really a busy
182+ # node.
183+ # timeoutSeconds 20 - kubelet's own bound, above the inner one so
184+ # that a reachable-but-slow mount produces a
185+ # clean exit 1 from `timeout`. On a genuinely
186+ # wedged `hard` mount it is kubelet that ends
187+ # the attempt, not `timeout`: the blocked write
188+ # sits in uninterruptible sleep, TERM and KILL
189+ # are both ignored, and `timeout` then blocks in
190+ # wait() for a child that cannot die. The probe
191+ # still fails, and the process stays until the
192+ # mount recovers.
193+ # periodSeconds 30 - one attempt per 30s. Those unkillable
194+ # processes accumulate one per period during an
195+ # outage, so the cadence bounds the pile.
196+ # failureThreshold 5 - 4 x 30s + up to 20s = ~140s of continuous
197+ # failure before NotReady, ~50s clear of the
198+ # grace period.
199+ # health.STORAGE_HEARTBEAT_TTL (120s) is 4 x periodSeconds, so the
200+ # sidebar indicator goes red after three missed refreshes - still
201+ # before the worker is marked NotReady. Degradation should be visible,
202+ # not fatal.
203+ readinessProbe :
204+ exec :
205+ command :
206+ - /bin/bash
207+ - -c
208+ - |
209+ set -u
210+ py=/root/miniforge3/envs/streamlit-env/bin/python
211+ exec timeout -k 3 15 "$py" -m src.workflow.health --probe
212+ initialDelaySeconds : 15
213+ periodSeconds : 30
214+ timeoutSeconds : 20
215+ failureThreshold : 5
216+ successThreshold : 1
33217 volumeMounts :
34218 - name : workspaces
35219 mountPath : /workspaces-streamlit-template
40224 volumes :
41225 - name : workspaces
42226 persistentVolumeClaim :
43- claimName : workspaces-pvc
227+ claimName : workspaces-nfs- pvc
44228 - name : config
45229 configMap :
46230 name : streamlit-config
0 commit comments