Some K8s Questions

[andrewheberle]

I have only just found that the CSI driver now exists for Kubernetes, which is amazing news, but I had some queries:

1. Cache Volume: Is it safe/possible/advisable to use an emptyDir volume for cache storage?
2. Gateway: The current example uses two single replica StatefulSet's however given the nature of this service would this make more sense as a single StatefulSet with two replicas?
3. Compactor: Would having the compactor as a separate Deployment be best practice?

For item 1 and 2 above, I am currently running things this way myself, so I am happy to submit a PR to change the examples if this is a good idea.

For the cache volume question I haven't actually tested this myself and wasn't sure if this could lead to data loss, with the main reason for asking if the gateway can be treated as "ephemeral" (but with a stable identity for each pod).

[Barre]

Hi,

Cache (emptyDir):

• It is safe, and there is no data-loss risk. The on-disk cache is a read-through cache of decrypted SST blocks; all durable state is in S3 and unflushed writes live in memory, not in the cache dir. Losing it just means a cold cache.

Gateway topology:

• Two single-replica StatefulSets are used because the two nodes need asymmetric config (node_id, peers, replication_listen, separate Secrets). A single STS works too but needs an entrypoint/init container to template that config from the pod ordinal, plus anti-affinity. I believe the static two-STS form is just easier to read as a reference.

Compactor:

• Yes, a separate Deployment is supported and is a good pattern for write-heavy setups: run gateways with zerofs run --no-compactor and a Deployment of stateless zerofs compactor workers (no identity/volume needed, safe to run many).
• Autoscaling: workers are stateless, so an HPA on CPU/network could work.

[andrewheberle]

Thanks for the quick response.

My concern with the two seperate StatefulSet's was mainly that there is no coordination between them for changes that might affect both when used with a GitOps solution like Fleet or Flux (eg updating the image version). We use Fleet in our case, so I have the gateway running as a two replica StatefulSet but with two different configurations named based on the seperate pod names and those are referenced via an environment variable in the pod args as follows:

spec:
  template:
    spec:
      containers:
        - name: zerofs
          args: ["run", "-c", "/etc/zerofs/$(POD_NAME).toml", "--no-compactor"]
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

Having said that as an example of deploying the Gateway the current manifest is probably easier to understand as you said, and simple enough to adapt for someone that wants to do this.

Thanks again.

[andrewheberle]

Another question I wasn’t sure about was relating to the WAL feature and if both gateways in the HA pair need access to the same WAL storage?

The documentation says the WAL and HA are compatible but does this need to be the same storage or could it be a local PVC?

If it needs to be the “same” storage could it be a ReadWriteMany PVC or is S3 the only option here?

Finally if shared object storage is required for the WAL is there any advantage to configuring a WAL to the same S3 storage provider/tier as the standard storage used for ZeroFS or does it only help if WAL storage is faster/lower latency?

Also if there is a better way to ask these sort of questions without cluttering up your GitHub Issues please let me know.

[Barre]

Another question I wasn’t sure about was relating to the WAL feature and if both gateways in the HA pair need access to the same WAL storage?

The documentation says the WAL and HA are compatible but does this need to be the same storage or could it be a local PVC?

Yes, they do need to access the same storage. The WAL holds what hasn't yet been materialised into L0 SSTs. Essentially, it only exists to absorb fsync latency rather than creating an L0 per fsync.

If it needs to be the “same” storage could it be a ReadWriteMany PVC or is S3 the only option here?

This would work!

Finally if shared object storage is required for the WAL is there any advantage to configuring a WAL to the same S3 storage provider/tier as the standard storage used for ZeroFS or does it only help if WAL storage is faster/lower latency?

If your goal is more performance, local NVMe or something like S3 Express would work great. Otherwise, using the same bucket or a different bucket won't make any difference (assuming latency to both buckets is the same).

Also if there is a better way to ask these sort of questions without cluttering up your GitHub Issues please let me know.

There's a github discussions tab on this repo. I just migrated this issue there as well.