Ubuntu Server Struggles with Post-Docker Kubernetes Installs

I have a bone to pick with someone. I honestly don’t know who to point this ire toward, but there’s a big problem now with using Ubuntu Server as a base for Kubernetes.
Over the past few days, I’ve attempted, over and over, to get Kubernetes up and running on Ubuntu Server 22.04, and, no matter how many times I’ve attempted, it fails. Now, I can get Kubernetes installed on Ubuntu Server without a problem, as I’ve done so many times before. The only difference is now, instead of using Docker, I have to use a runtime like containerd. However, when attempting to initialize the cluster, I am (every time) met with the error:
1 2 3 4 5 6 7 |
[kubelet-check] Initial timeout of 40s passed. Unfortunately, an error has occurred: timed out waiting for the condition This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) |
It doesn’t matter if I’m coming from a fresh install or having done a sudo kubeadm reset, it times out and never initializes. I’ve attempted this three times (each with new instances of Ubuntu Server 22.04) and it never succeeds.
The issue sent me down a rabbit hole which offered some promise that the latest version of containerd had problems when installing on Ubuntu Server, but even after attempting a new deployment with an older version of containerd, I wound up with the same problem.
Suffice it to say, I’ve come away from this little experiment frustrated. A Kubernetes cluster on Ubuntu Server 22.04 should be a no-brainer. It’s not. Although I can get a single instance running just fine and even deploy an app with it. But the second I want to go the cluster route, things go seriously awry.
Drilldown
What’s interesting about that error is the kubelet is running. However, when running:
sudo systemctl status kubelet
I see the errors like this:
Aug 04 13:56:19 kubecontroller kubelet[550949]: E0804 13:56:19.613305 550949 kubelet.go:2424] "Error getting node" err="node \"kubecontroller\" not found"
Aug 04 13:56:19 kubecontroller kubelet[550949]: E0804 13:56:19.714099 550949 kubelet.go:2424] "Error getting node" err="node \"kubecontroller\" not found"
Aug 04 13:56:19 kubecontroller kubelet[550949]: E0804 13:56:19.814923 550949 kubelet.go:2424] "Error getting node" err="node \"kubecontroller\" not found"
The next rabbit hole had to do with the ~./kube/config file. Even after re-checking the permissions on that file, I had issues. Fixed the issues and restart the kubelet with:
sudo systemctl restart kubelet
Guess what? Now, kubelet won’t start.
Let the hair pulling commence!
A quick reboot of the system to see if it might flush out whatever nastiness remains. Once the machine rebooted, I ran the init command so I can view more troubleshooting information like so:
sudo kubeadm init
Guess what? New errors such as:
error execution phase wait-control-plane
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1 cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1 cmd/kubeadm/app/cmd/init.go:153
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute vendor/github.com/spf13/cobra/command.go:856
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC vendor/github.com/spf13/cobra/command.go:974
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run cmd/kubeadm/app/kubeadm.go:50
main.main cmd/kubeadm/kubeadm.go:25
runtime.main /usr/local/go/src/runtime/proc.go:250
runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1571
Obviously, that’s zero help. And no matter how much time I spend with my friend Google, I can’t find an answer to that which ails me.
Back to the drawing board. Another installation and the same results. And, of course, the official Kubernetes documentation is absolutely zero help.
What’s the conclusion to be drawn and what can you do (when Ubuntu Server is your go-to)?
It’s All about Docker
Once upon a time, deploying a Kubernetes cluster on Ubuntu Server was incredibly simple and rarely (if ever) failed. What’d the difference?
In a word… Docker.
The second Kubernetes removed Docker support, deploying a cluster on Ubuntu Server became an absolute nightmare. With that in mind, what can you do? Well, you can always install microk8s via snap with the command:
sudo snap install microk8s --classic --channel=1.24
Of course, as everyone knows, the snap installation can take some time and it’s not nearly as responsive as a standard installation. However… when in Rome.
Once the installation completes, add your user to the necessary group with:
sudo usermod -a -G microk8s $USER
Change the permissions of the .kube directory with:
sudo chown -f -R $USER ~/.kube
Log out and log back in and then run the command:
microk8s status --wait-ready
Big bada boom, all is working. I can deploy an NGINX app with:
microk8s kubectl create deployment nginx --image=nginx
But this isn’t a cluster. Ah, but microk8s has you covered. On the controller node, issue the command:
microk8s add-node
You will be given a join command to run on any other machine you’ve installed microk8s on that looks like this:
microk8s join 192.168.1.43:25000/bad12d3d8966b646442087d6a1edde436/6407f3e21772
Oh, but guess that that’ll error out on you as well with something like:
Contacting cluster at 192.168.1.43
Connection failed. Invalid token (500).
Rebooted both machines, rerun the add-node command again with the –skip-verify command, and no dice.
Make sure to set hostnames for both machines, that those hostnames are mapped in /etc/hosts, and double-check that the time is correct on both machines. No deal.
However, after the second reboot of both machines, for whatever reason, the node was able to join the controller. The process took far longer than it should have, but I associate that with using the snap version of the service.
After running microk8s kubectl get nodes, I can now see both nodes on my cluster.
Huzzah.
Why Make It Difficult?
To those involved… this shouldn’t be so hard. Seriously. Kubernetes is already a challenging platform to use. To make it just as hard to get up and running, without fail, sends me right back to the simplicity of Docker Swarm.
Sure, I could migrate to an RHEL-based server for my Kubernetes deployments, but Ubuntu Server has been my go-to for years. Don’t get me wrong, I don’t mind microk8s, but it won’t work with the likes of Portainer (which is my go-to for these sorts of things). For that, I have to add the Portainer addon with:
microk8s enable community
microk8s enable portainer
Outstanding…only not. What’s the problem now? After enabling Portainer, you are informed to access it via the nodeport, which you can get the address using the command:
export NODE_PORT=$(kubectl get --namespace portainer -o jsonpath="{.spec.ports[1].nodePort}" services portainer)
But wait… kubectl isn’t installed because I’m using microk8s! I have to modify the command like so:
export NODE_PORT=$(microk8s kubectl get --namespace portainer -o jsonpath="{.spec.ports[1].nodePort}" services portainer)
Then you must run the commands (again, modifying the next to include microk8s):
export NODE_IP=$(microk8s kubectl get nodes --namespace portainer -o jsonpath="{.items[0].status.addresses[0].address}") echo https://$NODE_IP:$NODE_PORT
The final command will report the address used to access Portainer.
Wouldn’t it be great if that worked? It didn’t.
Guess what? All of this was from the official documentation (minus the addition of the microk8s portion of the command which was conveniently left out).
To those responsible for these bits of technology, this shouldn’t be so hard. I realize there might be bad blood between Kubernetes and Canonical, but when that spills out to the userspace, the real frustration falls on the heads of admins and developers.
Please, please, please, fix these problems so those who prefer Ubuntu Server can get their Kubernetes on with the same ease they once could.