Using Cilium with k0s on alpine

There's two modes we can use Cilium in when it comes to deploying on k0s. One is with Cilium as a replacement for kube-proxy, the other is working alongside it.

We're going to focus on using it as a replacement for kube-proxy and a focus on installing it as an addon so it's best followed for new clusters.

We're going to deploy cilium using k0s' helm addon feature. So lets start by editing the k0s configuration. There are two places that you can modify this config:

• The k0sctl.yaml config file you may have created using k0sctl to deploy your cluster (if you haven't already). For all intents and purposes the spec.k0s.config is the same as the config in the next step and applied with k0sctl apply k0sctl.yaml.
• Using the static/dynamic k0s configuration by either running k0s config edit or by editing /etc/k0s/k0s.yaml

Okay! So what's the config?

The first thing we need to do is change the configuration to disable kubeProxy:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  api:
    address: 192.168.1.185
    externalAddress: 159.66.22.33
  storage:
    etcd:
      peerAddress: 192.168.1.185
  network:
    kubeProxy:
       # -- The important bit
      disabled: true
      # -- End the important bit

Great, now we need to change the provider to be custom:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  api:
    address: 192.168.1.185
    externalAddress: 159.66.22.33
  storage:
    etcd:
      peerAddress: 192.168.1.185
  network:
    # -- The important bit
    provider: custom
    # -- End the important bit
    kubeProxy:
      disabled: true 

Awesome and now we're ready to setup cilium. Lets add a helm registry and helm chart:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  extensions:
    helm:
      # -- The important bit
      charts:
      - chartname: cilium/cilium
        name: cilium
        namespace: kube-system
      repositories:
      - name: cilium
        url: https://helm.cilium.io
      # -- End the important bit
  api:
    address: 192.168.1.185
    externalAddress: 159.66.22.33
  storage:
    etcd:
      peerAddress: 192.168.1.185
  network:
    provider: custom
    kubeProxy:
      disabled: true 

Awesome and now we need to tell cilium it's there to replace kube-proxy. To do that we need to know what the api address and port is. In the case of this cluster it will be 159.66.22.33:6443 as that's the external api address and the port the api server is running on:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  extensions:
    helm:
      charts:
      - chartname: cilium/cilium
        name: cilium
        namespace: kube-system
      repositories:
      - name: cilium
        url: https://helm.cilium.io
        # -- The important bit
        values: |
          kubeProxyReplacement: true
          k8sServiceHost: 159.66.22.33
          k8sServicePort: 6443
        # -- End the important bit
  api:
    address: 192.168.1.185
    externalAddress: 159.66.22.33
  storage:
    etcd:
      peerAddress: 192.168.1.185
  network:
    provider: custom
    kubeProxy:
      disabled: true 

Done! Now if you already had kube-proxy running on the cluster (assuming this is not a new install) then you'll need to hop on the node of your master and delete the kube-proxy container: kubectl delete -n kube-system kube-proxy

That's it! It should be up and running.