Development of a Cloud Controller Manager

[augustodsgv]

Summary

I've been running some Kubernetes clusters on top of Incus and a real pain is the lack of integration between the cluster and it's infrastructure. Currently, I use the rke2 cluster engine to manage stand alone clusters, but I would like to have this integration with Incus to enable cluster autoscaling and automatic load balancer provisioning, so I was looking for a Cloud Controller Manager that I cloud deploy in my clusters to provide these features, but I couldn't find any project developing it.

I was glad I found this project, but I don't want to migrate my architecture to ClusterAPI for now, so I would like to know if there are any plans of developing a CCM for Incus in this project, or even if it's possible to use this CAPI as an CCM.

Why is this important?

A CCM is a more flexible software that can deployed on top of any Kubernetes cluster and don't require the change of the architecture.

Are you interested in contributing to this feature?

Sure!

[markamber]

I am actively working on something that may be similar, and the maintainer, neoaggelos, mentioned they were working on something that may be similar.

What I am working on is a three-fold thing. ClusterUnderlay - which spins up an OVN network and special load balancer (it is a happy path, it mimics what this repos lxc load balancer does but also balances the workloads and provides a happy path of programming DNS with external-dns so developers can get a cluster with premade *.apps.<cl-name>.<my-domain>.net and also that serves the kubernetes API, complete with separate ingress ACL and whatnot. This also sets up Cilium CNI first on the cluster and keeps the load balancer up to day (will be moving to the in-repo ovn type load balancer once I move to incus from LXD soon)

ClusterIdentity - which takes the cluster kubeconfig and shoves it in my vault and wires up a keycloak client and delegates some groups so the dev gets access and we can use our SSO in kubectl.

ClusterDelivery - programs flux CD into the cluster itself if the dev provides adequate credentials.

neoaggelos mentioned some overlap with my work on load balancer and possibly working on something involving CSI (using the Ceph incus is on perhaps) which is exciting. I may or may not use that or just chart my own path that works best with my infra.

What are you looking to do?

[snels-sm]

Just install cluster-autoscaler to management cluster

values.yaml

cloudProvider: clusterapi
clusterAPIMode: kubeconfig-incluster
clusterAPIKubeconfigSecret: testtest-kubeconfig
autoDiscovery:
  namespace: fleet-default
  labels:
    - cluster.x-k8s.io/cluster-name: testtest
rbac:
  additionalRules:
    - apiGroups:
        - infrastructure.cluster.x-k8s.io 
      resources:
        - lxcmachinetemplates
      verbs:
        - get
        - list
        - watch

helm install testtest-autoscaler -n fleet-default cluster-autoscaler-9.58.0.tgz -f values.yaml

Add annotations to MachineDeployment:
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: 5
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: 3

• Cluster Autoscaler: Done
• Load Balancers: Creating load balancers is not the purpose of the ClusterAPI provider