Continental Innovates with Rancher and Kubernetes
Recurring snapshots are handled differently based on your version of RKE.
To schedule automatic recurring etcd snapshots, you can enable the etcd-snapshot service with extra configuration options. etcd-snapshot runs in a service container alongside the etcd container. By default, the etcd-snapshot service takes a snapshot for every node that has the etcd role and stores them to local disk in /opt/rke/etcd-snapshots.
etcd-snapshot
etcd
/opt/rke/etcd-snapshots
If you set up the options for S3, the snapshot will also be uploaded to the S3 backend.
When a cluster is launched with the etcd-snapshot service enabled, you can view the etcd-rolling-snapshots logs to confirm backups are being created automatically.
etcd-rolling-snapshots
$ docker logs etcd-rolling-snapshots time="2018-05-04T18:39:16Z" level=info msg="Initializing Rolling Backups" creation=1m0s retention=24h0m0s time="2018-05-04T18:40:16Z" level=info msg="Created backup" name="2018-05-04T18:40:16Z_etcd" runtime=108.332814ms time="2018-05-04T18:41:16Z" level=info msg="Created backup" name="2018-05-04T18:41:16Z_etcd" runtime=92.880112ms time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:42:16Z_etcd" runtime=83.67642ms time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms
Etcd-Snapshot
creation
etcd.retention
etcd.backup_config.retention
The --access-key and --secret-key options are not required if the etcd nodes are AWS EC2 instances that have been configured with a suitable IAM instance profile.
--access-key
--secret-key
The backup snapshot can be stored on a custom S3 backup like minio. If the S3 backend uses a self-signed or custom certificate, provide a custom certificate using the option custom_ca to connect to the S3 backend.
S3
custom_ca
In addition to API access keys, RKE supports using IAM roles for S3 authentication. The cluster etcd nodes must be assigned an IAM role that has read/write access to the designated backup bucket on S3. Also, the nodes must have network access to the S3 endpoint specified.
Below is an example IAM policy that would allow nodes to store and retrieve backups from S3:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "ListObjectsInBucket", "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::bucket-name"] }, { "Sid": "AllObjectActions", "Effect": "Allow", "Action": "s3:*Object", "Resource": ["arn:aws:s3:::bucket-name/*"] } ] }
For details on giving an application access to S3, refer to the AWS documentation on Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.
services: etcd: backup_config: interval_hours: 12 retention: 6 s3backupconfig: access_key: S3_ACCESS_KEY secret_key: S3_SECRET_KEY bucket_name: s3-bucket-name region: "" folder: "" # Optional - Available as of v0.3.0 endpoint: s3.amazonaws.com custom_ca: |- -----BEGIN CERTIFICATE----- $CERTIFICATE -----END CERTIFICATE-----
RKE saves a backup of the certificates, i.e. a file named pki.bundle.tar.gz, in the same location. The snapshot and pki bundle file are required for the restore process in versions prior to v0.2.0.
pki.bundle.tar.gz
true
5m0s
24h
retention
services: etcd: snapshot: true creation: 5m0s retention: 24h