Cluster Cache Sync Fails When Managed Namespaces are Deleted Without Label Removal

[tricktron]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When the cluster secret has an explicit namespaces list and namespace-scoped Roles (not ClusterRoles), deleting one of those namespaces breaks cluster cache sync for the entire cluster. A 403 on the deleted namespace aborts the sync before other namespaces are processed, so apps in healthy namespaces stop detecting changes too. There is no self-recovery. Someone has to manually remove the deleted namespace from the cluster secret.

Two conditions required to trigger

1. The cluster secret must set namespaces explicitly (e.g. namespaces: "app-ns,doomed-ns"). Without it, the cache uses a cluster-scoped client and the per-namespace code path never runs.

2. RBAC must use namespace-scoped Roles, not ClusterRoles. With ClusterRoles, the apiserver returns HTTP 200 + empty list for a deleted namespace. No error, no trigger. With Roles, the Role is garbage-collected with the namespace → 403.

To Reproduce

You need ArgoCD running in-cluster with namespace-scoped RBAC. make start-local won't work. It sets ARGOCD_FAKE_IN_CLUSTER=true, which uses your kubeconfig (cluster-admin) and never gets a 403.

The simplest setup is a kind cluster with namespace-install.yaml:

# Create a kind cluster
kind create cluster --name argocd-test

# Install CRDs + namespace-scoped ArgoCD
kubectl create namespace argocd
kubectl apply --server-side --force-conflicts -k manifests/crds
kubectl apply -n argocd -f manifests/namespace-install.yaml
kubectl -n argocd rollout status statefulset/argocd-application-controller --timeout=120s

# Port-forward and log in
kubectl -n argocd port-forward svc/argocd-server 8080:443 &
PASS=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)
argocd login localhost:8080 --username admin --password "$PASS" --insecure
# Create two managed namespaces
kubectl create namespace app-ns
kubectl create namespace doomed-ns

# Grant the ArgoCD SA access via Roles (not ClusterRoles).
# These Roles get garbage-collected when the namespace is deleted.
for NS in app-ns doomed-ns; do
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argocd-managed
  namespace: $NS
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argocd-managed
  namespace: $NS
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argocd-managed
subjects:
- kind: ServiceAccount
  name: argocd-application-controller
  namespace: argocd
EOF
done

# Register the cluster with explicit namespaces
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: in-cluster-namespaced
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: in-cluster-namespaced
  server: https://kubernetes.default.svc
  namespaces: "app-ns,doomed-ns"
  clusterResources: "false"
  config: |
    {
      "tlsClientConfig": {
        "insecure": true
      }
    }
EOF

# Deploy an app to each namespace, verify Healthy/Synced
argocd app create survivor \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path guestbook \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace app-ns

argocd app sync survivor

argocd app create doomed \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path guestbook \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace doomed-ns

argocd app sync doomed

# Trigger the bug: delete the doomed namespace
kubectl delete namespace doomed-ns

# Wait for cache resync (up to 10 min) or invalidate via UI:
# Settings → Clusters → click cluster → Invalidate Cache

# Observe: survivor goes Unknown, controller logs show 403 errors
argocd app list

Expected behavior

Deleting one namespace should not break sync for the others. The controller should skip the inaccessible namespace, keep syncing the rest, and surface a warning so operators know what happened.

Screenshots

Version

All ArgoCD versions are affected but I specifically encountered the error on

v2.14.7

Logs

time="2025-01-23T09:15:04Z" level=error msg="error synchronizing cache state : failed to sync cluster https://kubernetes.default.svc:443: failed to load initial state of resource apps.Deployment: deployments.apps is forbidden: User \"system:serviceaccount:argocd:argocd-application-controller\" cannot list resource \"deployments\" in API group \"apps\" in the namespace \"test-namespace\"" application=example-app

time="2025-01-23T09:25:04Z" level=error msg="error synchronizing cache state : failed to sync cluster https://kubernetes.default.svc:443: failed to load initial state of resource core.Pod: pods is forbidden: User \"system:serviceaccount:argocd:argocd-application-controller\" cannot list resource \"pods\" in API group \"\" in the namespace \"test-namespace\"" application=example-app

[tricktron]

Update: Confirmed blocking in production

We hit this issue again in production today. All 617 apps went to Unknown state because a single deleted namespace (0000-0000-4e3a-b25c-ed31fc347aaa) remained in the cluster secret's namespaces field.

Key findings

• Restarting the controller does NOT fix the issue. The controller reads namespaces from the cluster secret verbatim and does not validate whether they still exist before attempting API calls.
• The entire cluster cache sync fails due to a single Forbidden error on a deleted namespace (fail-fast behavior in processApi()).
• Manual workaround: Remove the deleted namespace from the cluster secret's namespaces field. After that, Cluster successfully synced immediately.

Impact

Until this fix is merged, any deleted namespace that remains in the cluster secret will block the entire cluster cache sync indefinitely, making all applications show Unknown status with no automatic recovery. The only workaround is manual removal of the stale namespace from the cluster secret.