Continental Innovates with Rancher and Kubernetes
As of version v0.3.1 RKE adds the support for managing secret data encryption at rest, which is supported by Kubernetes since version v1.13.
v0.3.1
v1.13
At-rest data encryption is required for:
RKE provides users with two paths of configuration to enable at-rest data encryption:
Both configuration options can be added during initial cluster provisioning or by updating an existing cluster.
To utilize this feature, a new field secrets_encryption_config is added to the Kubernetes API service configuration. A full custom configuration looks like this:
secrets_encryption_config
services: kube-api: secrets_encryption_config: enabled: true custom_config: apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - aescbc: keys: - name: k-fw5hn secret: RTczRjFDODMwQzAyMDVBREU4NDJBMUZFNDhCNzM5N0I= - identity: {}
Enabling and disabling at-rest data encryption in Kubernetes is a relatively complex process that requires several steps to be performed by the Kubernetes cluster administrator. The managed configuration aims to reduce this overhead and provides a simple abstraction layer to manage the process.
Managed at-rest data encryption is disabled by default and can be enabled by using the following configuration:
services: kube-api: secrets_encryption_config: enabled: true
Once enabled, RKE will perform the following actions to enable at-rest data encryption:
aescbc
controlplane
kube-apiserver
After the kube-api server is restarted, data encryption is enabled. However, all existing secrets are still stored in plain text. RKE will rewrite all secrets to ensure encryption is fully in effect.
kube-api server
To disable encryption, you can either set the enabled flag to false, or simply remove the secrets_encryption_config block entirely from cluster.yml.
enabled
false
services: kube-api: secrets_encryption_config: enabled: false
Once encryption is disabled in cluster.yml, RKE will perform the following actions to disable encryption in your cluster:
cluster.yml
identity{}
Sometimes there is a need to rotate encryption config in your cluster. For example, the key is compromised. There are two ways to rotate the keys: with an RKE CLI command, or by disabling and re-enabling encryption in cluster.yml.
With managed configuration, RKE CLI has the ability to perform the key rotation process documented here with one command. To perform this operation, the following subcommand is used:
$ ./rke encrypt rotate-key --help NAME: rke encrypt rotate-key - Rotate cluster encryption provider key USAGE: rke encrypt rotate-key [command options] [arguments...] OPTIONS: --config value Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG] --ssh-agent-auth Use SSH Agent Auth defined by SSH_AUTH_SOCK --ignore-docker-version Disable Docker version check
This command will perform the following actions:
For a cluster with encryption enabled, you can rotate the encryption keys by updating cluster.yml. If you enable and re-enable the data encryption in the cluster.yml, RKE will not reuse old keys. Instead, it will generate new keys every time, yielding the same result as a key rotation with the RKE CLI.
With managed configuration, RKE provides the user with a very simple way to enable and disable encryption with minimal interaction and configuration. However, it doesn’t allow for any customization to the configuration.
With custom encryption configuration, RKE allows the user to provide their own configuration. Although RKE will help the user to deploy the configuration and rewrite the secrets if needed, it doesn’t provide a configuration validation on user’s behalf. It’s the user responsibility to make sure their configuration is valid.
Warning: Using invalid Encryption Provider Configuration could cause several issues with your cluster, ranging from crashing the Kubernetes API service, kube-api, to completely losing access to encrypted data.
kube-api
An example for custom configuration would be enabling an external key management system like Amazon KMS. The following is an example of the configuration for AWS KMS:
services: kube-api: extra_binds: - "/var/run/kmsplugin/:/var/run/kmsplugin/" secrets_encryption_config: enabled: true custom_config: apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - kms: name: aws-encryption-provider endpoint: unix:///var/run/kmsplugin/socket.sock cachesize: 1000 timeout: 3s - identity: {}
Documentation for AWS KMS can be found here. When Custom Configuration is set to to enable the AWS KMS provider, you should consider the following points:
extra_binds
It’s important to understand that enabling encryption for you cluster means that you can no longer access encrypted data in your etcd database and/or etcd database backups without using your encryption keys.
The encryption configuration is stored in the cluster state file cluster.rkestate, which is decoupled from the etcd backups. For example, in any of the following backup cases, the restore process will fail:
cluster.rkestate
Therefore, we recommend that when you enable or disable encryption, or when you rotate keys, you should create a snapshot so that your backup requires the same keys that you have access to.
This also means you should not rotate the keys during the restore process, because you would lose the encryption keys in cluster.rkestate.
The same applies to the custom configuration use case, however in this case it will depend on the user-provided encryption configuration.