Illumina Innovates with Rancher and Kubernetes
Kubernetes clusters use multiple certificates to provide both encryption of traffic to the Kubernetes components as well as authentication of these requests. These certificates are auto-generated for clusters launched by Rancher and also clusters launched by the Rancher Kubernetes Engine (RKE) CLI.
In Rancher v2.0.x and v2.1.x, the auto-generated certificates for Rancher-launched Kubernetes clusters have a validity period of one year, meaning these certificates will expire one year after the cluster is provisioned. The same applies to Kubernetes clusters provisioned by v0.1.x of the Rancher Kubernetes Engine (RKE) CLI.
If you created a Rancher-launched or RKE-provisioned Kubernetes cluster about 1 year ago, you need to rotate the certificates. If no action is taken, then when the certificates expire, the cluster will go into an error state and the Kubernetes API for the cluster will become unavailable. Rancher recommends that you rotate the certificates before they expire to avoid an unexpected service interruption. The rotation is a one time operation, and the newly-generated certificates will be valid for the next 10 years.
The instructions below detail how to rotate the certificates in both Rancher-launched and RKE-provisioned clusters, both before expiry when certificates are still valid, and also in the event that the certificates have already expired.
Rotating Kubernetes certificates may result in your cluster being temporarily unavailable as components are restarted. For production environments, it’s recommended to perform this action during a maintenance window.
Rancher v2.2.4 and higher provide UI support for certificate rotation. If you are unable to upgrade your Rancher v2.0.x or v2.1.x instances to v2.2.x, then you can upgrade them to v2.0.15 and v2.1.10 respectively. These versions contain certificate rotation support via the API, and detailed steps for this can be found in the documentation.
To rotate the certificates on a Rancher-launched cluster for which certificates are still valid, follow these steps:
Edit as YAML
Rotate all service certificates
After following these steps, the certificates will be rotated and will have a validity of 10 years.
If your Rancher-launched Kubernetes cluster is already in an error state because the certificates have expired, follow these steps to rotate the certificate:
Open a shell session to the etcd and control plane nodes for the cluster and check if the directory /etc/kubernetes/.tmp contains the file kube-apiserver-requestheader-ca.pem. If this file is absent, perform the following manual copy:
cp /etc/kubernetes/.tmp/kube-ca.pem /etc/kubernetes/.tmp/kube-apiserver-requestheader-ca.pem
cp /etc/kubernetes/.tmp/kube-ca-key.pem /etc/kubernetes/.tmp/kube-apiserver-requestheader-ca-key.pem
cp /etc/kubernetes/.tmp/kube-apiserver.pem /etc/kubernetes/.tmp/kube-apiserver-proxy-client.pem
cp /etc/kubernetes/.tmp/kube-apiserver-key.pem /etc/kubernetes/.tmp/kube-apiserver-proxy-client-key.pem
To rotate certificates, browse to the cluster in the Rancher UI, click the vertical ellipses, click Rotate Certificates, select Rotate all service certificates and click Save.
If the UI shows no activity on the cluster while the rotation is happening, and if the log still reports Expired cert, perform the steps described in Rancher Issue #20822.
After the rotation is finished, browse to the Nodes view for the cluster within the Rancher UI and check the state of Worker nodes. If the state is not Active, do the following:
Copy the following certificates from a Kubernetes control plane node to each worker node, under the same location:
Restart the kubelet and kube-proxy containers on each worker:
docker restart kubelet
docker restart kube-proxy
If you are running Rancher in High Availability (HA) mode and used a version of RKE less than v0.2.0 to provision the cluster where the Rancher server has been installed via Helm, the certificates on that management cluster have to be rotated using the RKE CLI.
Before conducting the certificate rotation, please verify the presence of the kube-apiserver-requestheader-ca.pem file.
To do so, open a shell session to the etcd and control plane nodes for the cluster and check if the directory /etc/kubernetes/.tmp contains the file kube-apiserver-requestheader-ca.pem. If this file is absent, perform the following manual copy:
To rotate the certificates on an RKE v0.1.x provisioned cluster for which certificates are still valid, follow these steps:
rke up --config cluster.yml
rke cert rotate --config cluster.yml
If your RKE v0.1.x provisioned cluster is already in an error state because the certificates have expired, follow these steps: