Kubernetes cluster (K3s) for a microservices project

My first experience of running production-ready Kubernetes clusters myself.

More information

Imagine a project coming along, which cannot just pay a cloud vendor to run a Kubernetes cluster for you. Imagine having to manage your own cluster on-prem, needing to do so on a short notice and with room to grow, so that other developers can start working on a microservice project and ship container images successfully.

What I did

After finding myself in these circumstances, I had to learn a lot quickly, to get this whole project up and running. Thankfully, I succeeded, eventually having a Kubernetes cluster up and running, with Helm charts and a custom repository both for those and container images. There was a full CI/CD cycle for getting something up and running, as well as shipping it, with an appropriate amount of instructions for others to get started. I onboarded a few developers over the coming weeks, so that they could hit the ground running, as well as helped administer the cluster long term.

What I learnt

Frankly, Kubernetes is a mess as soon as you venture off the happy path: want to run the excellent K3s distro instead of OpenShift, because your RPM server doesn't have the resources for the heavy alternative? Things might break along the way, you better be prepared to spend lots of time debugging. Want to integrate something like Portainer, so the developers can manage the Ingress in a visual manner? Well, you're going to have issues with getting the Traefik controller working and trying to replace it with Nginx will break your local development cluster due to incomplete resource cleanup.

But hey, when it works, it's okay, especially Helm charts, as long as you get a reasonable generated template to work with and then can add whatever you need on top of it. All of the sudden, deployment actually becomes easier, despite the Kubernetes YAML manifests being needlessly long, when compared to the Compose specification.