Everything you need to know about Container Runtime Security

Container orchestration systems like Kubernetes provide a simpler way to set up and manage containers at scale. As container adoption has increased, usage of these orchestration systems has skyrocketed. Kubernetes, for example, provides orchestration for more than three-quarters of containerized applications today. Of course, the popularity of these platforms makes them common targets for cyber criminals. Kubernetes combats this risk by offering pod security policies and drift management to help secure containers in development — but what happens when containers go to runtime?

While Docker and Kubernetes offer numerous tools for building security into containers, they don’t secure the runtime environment itself. As a result, a rich runtime stack of tools, processes, and policies has emerged that does the work of securing containers in production.

In this post we’ll explain the risks around container runtime and the tools and processes you can use to secure your containers in production.

The challenge of securing running containers

While practices like incorporating security into container builds and building smaller container images can help ensure no security problems are built into the container image, a number of threats can emerge once a container is running, including:

• Newly discovered application vulnerabilities in old images

• Configuration drift, such as changing user privileges without authorization

• Privilege escalation attacks that allow bad actors to access secrets, storage volumes, or other resources

• Deployment of containers using bugs in access control

• Activation of malicious code that is written inside a container

Container monitoring tools like Prometheus and Grafana are often deployed to help improve the performance and efficiency of running containers, but these tools are not designed for security.

How to find and remediate container runtime risks

To secure running containers it’s essential to monitor key events such as log-ins, yet container runtime security goes beyond simple event monitoring. It starts with incorporating security into architecture using policy enforcement tools that set and enforce policies at the Kubernetes or kernel level. Scanning tools can then analyze audit logs, infrastructure as code (IaC), configuration settings, and application code itself to uncover new vulnerabilities or misconfigurations in production.

Who is responsible for container runtime security?

Responsibility for securing the application runtime has traditionally fallen to dedicated security teams. But the speed and complexity associated with agile and cloud native development means that developers are increasingly involved in addressing security risks within their code. They are shifting left to incorporate security into the earliest stages of planning and code.

Developer security responsibilities are also spreading across the container lifecycle from planning, coding, scanning, and deployment to the runtime environment. They now must also follow the development lifecycle forward and consider what happens when their containers go to runtime.

Choosing secure container engine runtime settings

Container engine runtime settings can be configured to limit attackers’ ability to spread their reach following a network breach. Here are a few suggested settings that will give your organization additional protection:

Don’t run as root

Containers should be run as users, not root. You can do this by first defining the user ID in the Dockerfile and then running as that user. This makes it easy to limit access using role-based access controls.

Default to read-only root filesystem settings

This prevents attackers from gaining control over a machine or writing malicious code to the host.

Limit access to container hosts

Disallow SSHing into servers. This helps prevent attackers and malware from escaping and attacking the host system.

Drop Linux capabilities

Capabilities allow fine-tuning of privileges on the kernel, such as the ability to read the audit log or bypass permission checks. Dropping Linux capabilities on a container limits the ability of attackers to manipulate the kernel.

Don’t run in privileged mode

Container runtimes include robust measures to shield containers from the host system. Running in privileged mode should be avoided as it circumvents most of these checks.

Scan the runtime using IaC scanning to detect misconfigurations

It is easy for developers to introduce misconfiguration issues as they set up their cloud infrastructure and container orchestrator. IaC scanning with Snyk IaC can automatically detect Kubernetes and other misconfigurations and deliver insights back to the tools within the developer workflow.

To learn more about runtime security settings, read our post, 10 Kubernetes Security Context settings you should understand | Snyk.

6 Kubernetes tools that can help secure containers at runtime

Though Kubernetes does not come with runtime security built in, it includes tools that can contribute to the security of your containers once they hit production.

1. Network Policies

Kubernetes’ default behavior is to allow all ingress and egress involving pods. Set default behavior to deny all incoming and outgoing traffic, then use network policies to finely control network behavior around containers.

2. Role-Based Access Control (RBAC)

The Kubernetes RBAC API allows you to set permissions at the pod or cluster level. This makes it easier to fine-tune authorization policies without needing to restart the cluster.

3. Policy Admission Control

Kubernetes admission controllers allow you to enforce rules that address specific attack vectors. OPA Gatekeeper and Kyverno are two policy engines that can act as admission controllers, allowing you to evaluate calls and enforce policies or deny requests.

4. Secrets

Using a Secret allows you to hide sensitive information such as a key or password from a pod that uses it. RBAC rules can then be configured to govern behavior around the reading and writing of Secrets, along with mechanisms around which users can create or replace Secrets. Credentials should never be placed in images or configuration files, for example. Instead they should be stored using a secrets management tool.

Vault can run directly on Kubernetes, allowing you to store and manage secrets both for Kubernetes and external applications. Vault includes native integrations for Kubernetes, and any other Kubernetes tool can leverage Vault. CyberArk is another popular secrets provider for Kubernetes.

5. Audit logs

Kubernetes provides little runtime security out of the box. It does provide tooling for audit logs, however, which then can be analyzed for threats using an auditing tool like Falco.

6. Debug using ephemeral containers

Kubernetes ephemeral debug containers can help with troubleshooting without the need to load debugging utilities into your images.

Sysdig + Snyk partnership for container runtime security

There’s no single solution or magic button to secure running containers, so it’s important to integrate security protocols wherever possible. The first priority is identifying where vulnerabilities and misconfigurations are and fixing the top issues. Security itself is a form of risk management, so it’s important to identify both vulnerabilities and their ramifications. Furthermore, it’s important to leverage Kubernetes’ tools and settings to boost container security posture.

While a fully automated container runtime security solution can catch most issues, the discerning eye of developers is still needed to fix security issues. Scanning tools can uncover vulnerabilities or misconfigurations in the runtime environment, but the developer still needs to circle back to the corresponding code to apply fixes. This wastes time and resources, and can result in developers overlooking an issue. That’s where Snyk comes in. Snyk addresses this by detecting issues — and recommending fixes — within developer workflows. Tools like Snyk Container and Kubernetes Configuration Security allow you to scan Docker containers and Kubernetes configurations and receive feedback within CI/CD tools.

Additionally, Snyk has partnered with Sysdig to bring that developer-first security thinking into the runtime environment. Sysdig’s runtime security uses open source Falco to detect intrusions, threats, and vulnerabilities in real time across containers and Kubernetes. The integration of Snyk and Sysdig allows these insights to go directly back to developer teams so they can prioritize and apply fixes.

Frequently asked questions