Kubernetes Cluster on GCE: Beyond Kubeadm
Why?
Deploying an entire application stack with a simple kubectl apply is a joy to behold. Given the expectation, I felt let down when the applying metrics-server manifest didn’t work out of the box. The Kubernetes cluster created by the default kubeadm configuration uses a self-signed certificate for the kubelet server, which doesn’t match the metrics-server requirements. We will go over the steps of creating a cluster that doesn’t rely on the kubelet-insecure-tls flag of metrics-server.
You may be wondering why one would want to run their Kubernetes cluster on GCP when you already have GKE. I have three words for you: custom node images. On GKE, you have only two choices for your worker node image, Container-Optimized OS or Ubuntu. As a result, You won’t be able to use many of the features supported by GCP compute like startup scripts, VM Manager, or nested VM. You may also be left wanting if you want to use coredns, custom autoscaler, external certificate authority, or any other feature requested on the GKE issue tracker.
What?
We will start by defining the features of our cluster:
- Worker nodes should have private IP only
- API Server should use all ETCD members
- Kubelet server and metric-server certificate should be signed by cluster CA
- Persistence Volume Claims should be backed by GCP persistent disk
- GCE Ingress controller should be able to create HTTP(S) Load balancer
How?
We will use gcloud CLI for provisioning the infrastructure. You can follow this guide if you don’t have gcloud CLI setup.
All aboard!
Provisioning Infra
We will start with creating VPC, Compute nodes, and Load balancer for our k8s master. Next, we will need to create a worker node since both the cloud controller and the ingress controller don’t expose services running on the master nodes. Finally, we will set up a cloud NAT to ensure that our instances have internet access without having a public IP. Here I am using the stock ubuntu-2004-lts image for simplicity. Of course, you can use your own custom image if you want.
Configuring master nodes
We will need to run the following command on all nodes. I would recommend enabling synchronize-panes of tmux. We will configure containerd to use systemd as its cgroup driver, or else kubeadm will complain about not following the recommendation.
Bootstrapping ETCD cluster
When using external etcd configuration, we generally use dedicated nodes. Here we will be setting up our etcd cluster on the same master nodes as k8s. kubeadm, by default, creates etcd on the same node as well. The only difference is that the API server in the default kubeadm setup uses just one etcd node and has no failure for etcd. Our API servers will use all three etcd members.
We will need to configure kubelet to run in standalone mode (not dependent on the API server) temporarily to bootstrap the etcd cluster. Following commands are largely based on kubeadm doc.
Bootstrapping Kubernetes cluster
Not using GKE doesn’t mean that we can’t use GCP persistent disk for Kubernetes StorageClass or LoadBalancer service type. All we need to do is configure the cloud provider on k8s components. Please note that cluster nodes should have admin access to GCP compute. The service account used by GCP compute by default has Editor access to the project. So it is very likely you are already covered here.
We will configure the Kubelet server to use certificates signed by our cluster CA. It will ensure that we don’t have to run metrics-server with kubelet-insecure-tls flag. Enabling serverTLSBootstrap means that you will need to manually approve the CSR for the kubelet server or set up a service like rubber-stamper. Hopefully, this will be easier when we have inbuilt support for the kubelet server certificate bootstrapping.
Deploying GCE Ingress Controller
In my experience, GCLB deployment documentation is not only confusing but also outdated. I sincerely doubt it is possible to get container-native load balancing working outside of GKE. You can still use the HTTP(S) load balancer of GCP as an ingress. Just make sure that the services you add to your ingress are of type NodePort.
Deploying Metrics Server
Besides relying on the kubelet-insecure-tls flag the metrics-server also uses insecureSkipTLSVerify in its APIService. Setting insecureSkipTLSVerify to true means that the API server won’t verify the metrics-server certificate. We will need to configure metrics-server to use a certificate signed by our cluster CA to avoid using insecure TLS. jenting/secure-metrics-server repository does an excellent job of documenting the process. I have updated it below to avoid using kustomize.
Testing
Lastly, we will make sure that the addons are working as expected.
Conclusion
Kubeadm does a great job of creating a minimum viable cluster. Regardless, you may end up scouring the internet when it comes to configuring the addons like metrics-server and GLBC. I am hoping that collecting all this information in one place will help others save some time.
Please feel free to leave a comment if you are following this post and can’t get everything working. I would be happy to help.
