Kubernetes is a portable, extensible open-source platform for managing containerized workloads and services. The growth in containerized deployments is therefore sure to be followed by Kubernetes adoption. In this context, it’s no surprise that we’re following a security listicle on container technology like Docker with one on Kubernetes.
Like most technologies with rapidly growing adoption rates, Kubernetes has attracted a lot of malicious interest. This year has already witnessed high profile instances of Kubernetes clusters being compromised, the most prominent being a global automotive company’s cloud resources being hacked to run cryptocurrency-mining malware. So to all you developer folk new to Kubernetes, here’s a starter’s guide for securing your Kubernetes deployments.
- Securing a Cluster
- Restricting Access to kubectl
Since Kubernetes clusters are largely an API driven framework, It needs to be authenticated and authorized here. So, every request through these APIs should have some sort of authentication, either via public-private key or token-based authentication.
- Using Role-Based-Access-Controls(RBAC)
Each request should also be authorized in such a way that any user access is limited to only the necessary actions for that particular user. So, even if a man-in-the-middle gains access to some credentials, the damage will be very limited to that user's access levels. This is where the RBAC comes into play.
In RBAC, there are two types of roles you can create. One is a normal role, and another one is a cluster role. Normal role is dedicated to certain namespace. Let's assume I create a namespace for Dev, and create a role called DemoUser. This user must be able to access files within that namespace itself, and nothing outside that namespace. The other role called the Cluster Role is where the user can actually access other namespaces as well, across all namespaces.
The main point here being that the RBAC enables you to create access to users based on the users' necessity, and restrict their access to anything apart from their roles. This includes managing users' operations like creating, deleting and copying files.
- Using Network Policy
Some containers don't need access to the outside internet. Only the interconnection between the pod is necessary. They can be interlinked, made to only communicate with the internal pods. Exposure to external network only increases the attack surface.
- Using Namespaces
You can use namespaces to separate out the environments. Let's assume you are using three environments; test, staging and production. You can separate them into different name spaces. This limits the users from going outside of their designated namespaces.
- Bootstrap TLS
All the request to the API server should be encrypted using TLS encryption. This helps prevent man-in-the-middle attacks that try to sniff out un-encrypted data for malicious activity.
2. Following Security Hygiene
(It is essential to follow a certain security hygiene in a cluster deployment)
- System Update:
It is essential to regularly update Kubernetes. Since it's an open source project, vulnerabilities and security issues are found quite often. In v.1.7 for instance, the dashboard was open to public. Anybody could access this dashboard, and could request data to this API without any credentials. This was a serious vulnerability, which has been fixed in later versions. So, if you are using v.1.7 or lower versions, you should definitely update them.
- Minimal OS
It is also ideal to use a minimal OS and a lightweight image. This reduces your attack surface by a massive degree.
- Minimal identity and access management (IAM) roles
If you are trying to deploy containers using PODs, AWS, or Google Compute, you can secure your systems by using IAM roles, where the role has only the particular amount of resources to be shared, or a particular amount of activities to performed. You can authorize and restrict the users' role on the cluster. (The difference between RBAC and IAM is that one is on namespace level, IAM is on the cluster level)
- Using Private IPs
We also suggest that you use private IPs on the nodes, so that the nodes are not available to the attackers.
- Verifying binaries that are deployed regularly
It is very easy and common to forget about the binaries. You should make sure that host binaries are patched so that the containers built on top of the host is not vulnerable to attacks.
3. Taking precautions against known attacks
- Disable dashboard. (Example v1.7): Block the dashboard, and whitelist certain IPs that can access the dashboard
- Disable default service account token: Since Kubernetes is a largely an API driven framework, disable the service account token and use any sort of authentication such as Public-key/private-key.
- Protect node metadata: Run scans against container images before you deploy it to pods. You can use scanners like claire to detect vulnerabilities before its being used in pods and clusters
4. Prevent microservice compromise
This is an application level or a service level activity. This can be done with various precautions using the already built-in features.
- Pod security policy: Kubernetes has something called the pod-security policy. Here you can restrict the file system, restrict the user, and limit the number of nodes to be exposed to an application.
- Protecting Secrets: You can also protect the secrets using the secrets management system that comes built-in with Kubernetes. You can include other open-source frameworks like Vault or CyberArc. Even if your microservice is compromised, you are not leaking data which essentially leads to massive attack of the entire cluster.
- Kubernetes has something called a secret management system, where you can create a secret file and refer them in your PodSpec file, or you can base64 encode them and mention the in your podSpec file. It is highly recommended to use a third party app such as vault.
- Limiting identity for PODS
The general practice is that most of the docker images are run on root user privileges. This exposes the system to a deeper level vulnerability, which when exploited, the attacker can do any number of things to those machines on the container. So, setting the docker to run as non-root will reduce the attack surface. You can do that setting a security context in your podspec file.
- Resource management policies
Attacks can consume a lot of memory, bitcurrency mining hacks for example. Attacks can consume 99% to 100% of your resources. We can restrict such attacks by setting up the memory and CPUs. For example, in the image above, the specs only limit not more than 400 mb. Any requests that goes over that will send out a default error. (34min)
5. Tools to help secure pre-deployment
Now, let's look at some of the tools that can be used pre-deployment to help keep your Kubernetes deployment a lot more secure.
Is a YAML static analysis tool. It essentially scans your deployment YAML files and it quantifies the risk, and gives you a score. This essentially helps in integrating with your CI pipeline, before you use the YAML files in production. It checks for various parameters such as resources, security-context, policies.
- Sysdig Falco
Is an open source container runtime security. It tracks the behavior of your network, file-activity, and your system of your entire Kubernetes setup. You can also specify custom-rules, which are pretty easy to set. And, you can write logs into the log management platform.
Hope this was a helpful read. In case you want more clarity on this issue I recommend you watch the we45 webinar on “Securing Kubernetes Deployments”. It’s replete with live demos and examples. You can access the same here.