Continental Innovates with Rancher and Kubernetes
As companies adopt container technologies, they face a significant challenge - how do we secure this new attack surface? It’s an issue that you often see backlogged in favor of solving storage, networking and monitoring issues. Add on the challenge of educating the workforce on one of the fastest-growing open source projects to date, and it’s no wonder security has lagged as the primary focus for teams. In fact, The New Stack published a survey that shows that almost 50 percent of Kubernetes users say security is their top unresolved issue.
In this blog, we aim to demystify the Kubernetes security threats, showcase best practices for securing your cluster, and provide useful tools to enable your developers. These tools include:
We encourage you to join the Kubernetes Security Master Class: Tapping Native Controls in Kubernetes to Protect your Cloud-Native Apps. This online Master Class will thoroughly flesh out the topics covered below and demonstrate how you can better secure your cloud-native workloads.
RKE is a CNCF-certified Kubernetes distribution that runs entirely within Docker containers. It addresses the installation complexity of Kubernetes by removing most host dependencies and presenting a stable path for deployment, upgrades, and rollbacks. RKE uses a declarative YAML file to configure and create Kubernetes environments. This enables reproducible on-prem or remote environments.
KubeLinter is a static analysis tool that checks Kubernetes YAML files to ensure that the declared application configuration adheres to best practices. KubeLinter is StackRox’s first open source tool designed for implementing security checks from the command line as well as a part of the CI process. KubeLinter is a binary that takes in paths to YAML files and runs a list of checks against them. Admins and developers can create their own policies to enforce, enabling quicker and more automated deployments.
StackRox Kubernetes Security Platform protects vital applications across build, deploy and runtime. StackRox deploys in your infrastructure and integrates with your DevOps tooling and workflows to deliver frictionless security and compliance. The StackRox Policy Engine includes hundreds of built-in controls to enforce DevOps and security best practices, industry standards such as CIS Benchmarks and NIST, configuration management of both containers and Kubernetes runtime security. StackRox profiles your workloads to enable you to make informed decisions about the workloads’ security.
RKE, KubeLinter, and StackRox enable you to deploy repeatable, secure clusters, visualize profile and access security vulnerabilities, and create declarative security policies. Let’s talk about the threats these applications can tackle together.
Let’s start by focusing on the Kubernetes vectors of attack. Microsoft recently published an attack matrix for Kubernetes based off of the MITRE ATT&CK framework.
The framework is adapted for Kubernetes and based on real-world observations and cases. Luckily, there are strategies to mitigate all of the various issues. First, we can start by hardening our Kubernetes control plane. After that, we will shift focus to securing our running container workloads.
The Kubernetes control plane includes the following components:
etcd will likely be on the control plane node; however, it can have a remote environment for high availability use cases. The cloud-controller-manager is installed in provider instances as well.
The Kubernetes REST API server is the core component of the control-plane. The server handles REST API calls, which include all communication between the various components and the user. This dependency makes securing the API server a top concern. There are some specific vulnerabilities in previous versions that are fixed with a simple upgrade to a newer version. However, the following hardening tasks are also within your control.
With a deployment tool such as RKE, this declarative format for setting up your cluster is easy. Below is a snippet of a default RKE config.yml file. We can see that we can enable audit logging, TLS (between Kubernetes components), and RBAC by default.
Setting up authentication for all K8s API clients is an ongoing challenge. We need to apply a zero-trust model to any workloads that will run in our cluster.
Kubernetes’ default scheduler, kube-scheduler, is designed to be pluggable. You can build your scheduler or have multiple schedulers for different workloads. Regardless of the implementation, it needs to be secure and there are a few tasks to ensure that it is.
With RKE, we can secure its connection to the API server by verifying the default scheduler addresses set to 127.0.0.1. Also, restrict file permissions by making sure the root user owns the scheduler YAML file.
stat -c %U:%G /etc/kubernetes/manifests/kube-scheduler.yaml
The Kubernetes systems regulator, the kube-controller-manager, is a daemon that regulates the system using core control loops. Securing the controller requires a similar strategy as the scheduler.
Like the scheduler, we can ensure that the communication uses the local address (not an unsecured loopback interface) and ensure the root user owns the controller YAML file.
stat -c %U:%G /etc/kubernetes/manifests/kube-controller-manager.yaml
The last core component of the control plane is its key-value store, etcd. All Kubernetes objects are located in etcd, which means all of your configuration files and secrets are stored as well. The best practice is to encrypt secrets or manage secret information with a separate secrets management solution such as Hashicorp Vault or a cloud provider secrets management service. Some key factors to remember when you are managing the database are:
We want to limit any updates or changes to the manifests to the services that are allowed access. Using RBAC controls partnered with a zero-trust model will get you started. Lastly, encryption with etcd can be cumbersome. Rancher has a unique approach where the keys are generated as part of the initial cluster configuration. Kubernetes has a similar policy, although the file with the keys needs to be secure as well. Your organization’s security requirements will dictate where and how you will secure your sensitive information.
The cloud cloud-controller-manager is unique to cloud providers or any distribution which requires our cluster to communicate with the provider’s API. When working with cloud providers, admins will not have access to the master nodes of their cluster and, therefore, will not have the ability to run the hardening steps previously outlined.
Now that our control plane is secure, it is time to work on our applications running in Kubernetes. Similar to the previous section, let’s break down the various layers of security.
In the section below, we will dive deep into the various considerations for each section. In addition, the Master Class will give us more time to demonstrate the Kubernetes functionality in a comprehensive way.
Managing your containers before they are used is the first hurdle in container adoption. In the beginning, we need to consider:
The salient points are used to select secure base images, limit unnecessary packages and to secure a registry. Nowadays, most of the registries have image scanning built in to make your life easier. The StackRox Kubernetes Security Platform can automatically enforce policies on what images can be used to launch containers and identify security issues, including vulnerabilities and problematic packages, in image layers separate from the underlying base operating system (OS) image.
If you want to learn more, read this in-depth article about container image security.
Runtime security spans different Kubernetes functionality with the core goal of ensuring that our workloads are secure. Pod Security Policies are a great place to start securing your containers with the ability to control:
Keep in mind the zero-trust approach to systems, where capabilities should be set so the container has the minimum functionality required at runtime to function. For better visualization, StackRox’s risk profiling automatically identifies containers with potentially useful tools to attackers, including bash. It also alerts the use of suspicious tools and monitors, detects and alerts concerning runtime activity, such as executing abnormal or unexpected processes within containers.
Running stateful workloads in Kubernetes creates a backdoor into your containers. The attack surface increases by attaching storage and possibly giving executables or information to a container that it should not access. Best practice in Kubernetes ensures that stateful workloads run with the least privilege required. Other considerations include:
StackRox helps mitigate these threats by delivering dynamic policy-driven admission control as part of the StackRox platform. This enables organizations to enforce security policies automatically, including limitations on host mounts and their writability before containers are ever deployed into Kubernetes clusters.
Network access is a tough challenge in Kubernetes due to the lack of visibility into your containers. By default, network policies are disabled, and every pod can reach other pods on the Kubernetes network. Without this default, newcomers would have a tough time getting started. As your organization matures, we should strive to lock down all traffic except the traffic we deem necessary. This can be done using network policies configured by namespaces. It is also important to focus on the following:
One of the significant challenges with network policies is visualization. StackRox helps protect against network mapping by monitoring active network traffic between pods and automatically generating and configuring Network Policies to restrict communications to only what is necessary for application components to operate.
RBAC is central to securing your cluster. Kubernetes RBAC permissions are additive; therefore, RBAC’s only vulnerability is if an administrator or user grants exploitable permissions. The most common problem we encounter is users having cluster-admin access when they shouldn’t. Fortunately, there are more RBAC best practices to minimize issues:
An RKE cluster uses RBAC as the default authentication option during cluster setup. StackRox expands on this default option by helping organizations limit Kubernetes RBAC permissions according to the least privilege principle. We monitor the cluster RBAC settings for users and service accounts and identify ones with overly excessive privileges on clusters.
It is challenging to take on Kubernetes security on your own. In organizations, security can get in the way of DevOps, creating hurdles that lead to abandoned security principles in the pursuit of delivery. It doesn’t have to be this way.
By proactively identifying threats and crafting reasonable policies, we further shift security left. We can assess where our time needs to be spent and avoid bogging down DevOps teams with additional responsibilities.
If you want to learn more about protecting your Kubernetes clusters and applications, sign up for the free Master Class: Tapping Native Controls in Kubernetes to Protect your Cloud-Native Apps with StackRox on January 26, 2021. We’ll use Rancher for declarative Kubernetes deployments and explore various ways to access, implement, and streamline your security code, including using KubeLinter and StackRox.