Buildkit mount cache race: Build-time race condition container breakout (CVE-2024-23651)

Snyk has discovered a vulnerability in all versions of Docker Buildkit <=v0.12.4, as used by the Docker engine. The exploitation of this issue can result in container escape to the underlying host OS when building an image using a malicious Dockerfile or upstream image (i.e. when using FROM). This issue has been assigned CVE-2024-23651.

How does CVE-2024-23651 work?

The vulnerability occurs due to a time-of-check/time-of-use (TOCTOU) race condition when mounting a cache volume at container build time. Buildkit offers cache volumes using the RUN --mount=type=cache directive in a Dockerfile, which allows for the mounting of a persistent directory at a controlled location during Docker image build. This functionality is intended to help improve the performance of tooling, such as package managers, by keeping the persistent cache between builds.

It is possible to specify a source path inside the identified cache mount, and it was identified that the validation of this source path to ensure that it is a directory introducing a race condition. A concurrently running build step that already has the target cache directory mounted can replace the source path with a symbolic link between the check and the use (i.e. the source of the mount syscall), causing the mount syscall to incorrectly mount an arbitrary directory into the container filesystem.

In a successful attack, the now-executing process inside the container can observe that the path at which the cache filesystem was intended to be mounted now contains a bind mount of the host root filesystem. By default, access privileges will be the same as those of the Docker engine. Generally, this is the root user, and it is therefore possible to escalate from disk access to achieve full host root command execution.

How to mitigate CVE-2024-23651

Buildkit mitigated this vulnerability by ensuring that the source directory is checked using a file descriptor which is kept open and then later passed as the source parameter to the mount syscall. This effectively blocks the ability to swap the directory for a symlink in such a way that the mount syscall will traverse the symlink.

Synk recommends taking immediate action following the Buildkit advisory to mitigate this security vulnerability. The Buildkit v0.12.5 release includes patches for this issue. Users should also update technologies that bundle Buildkit (such as Docker Engine and Docker Desktop) to their respective patched versions.

Evaluating risk

This vulnerability depends on a malicious container running or building within a vulnerable infrastructure. Due to the nature of this vulnerability, detection is difficult and requires runtime for the most accurate detection. As our existing Snyk products don’t operate on the application runtime, we built two tools to make the detection of this vulnerability feasible.

Runtime detection

The new Helios team at Snyk has built an an eBPF-based runtime detection tool for this vulnerability, which can be found at leaky-vessels-runtime-detector, released under the Apache-2.0 license. eBPF is a commonly used instrumentation mechanism built into many modern Linux kernels.

This tool can identify a running container attempting to exercise this vulnerability on underlying infrastructure, placing the underlying host at risk. Note that this tool cannot prevent the exploitation of this vulnerability, only warn of exposure.

While updating your container infrastructure is our strong recommendation, you may be able to evaluate your risk or exposure using this tooling when updating is not immediately possible. Organizations should check with their container infrastructure provider to understand whether its infrastructure has been patched.

Installation and usage

To install and use our static detection tool, please do the following:

1. Clone the repository at https://github.com/snyk/leaky-vessels-dynamic-detector

2. Build the Go binary with go build.

3. Run the detector in your CI environment with: sudo ./ebpf-detector

For additional details, please check the README.md file.

Static analysis

In case you prefer not to run an external binary on your host or CI/CD pipeline, we’ve also built a static analysis detector, found at https://github.com/snyk/leaky-vessels-static-detector, that analyzes Dockerfiles and flags potential exploit attempts.

This tool can also examine images by parsing the FROM instruction or directly querying the local Docker daemon or public container registries like Dockerhub or GCR. Although the image history doesn’t include all the instructions a full Dockerfile would, it can show the WORKDIR and ONBUILD instructions of the parent image, which can indicate potentially nefarious images.

This is useful to protect against cases where a 3rd party base image has been compromised and tries to exploit the run or build commands of the child image. The static approach has a higher false positive (noise) rate, especially for exploits using the —mount flag, but can find images worthy of further investigation.

The Snyk team has performed ad hoc checks of Dockerfiles from public registries based on the images we see being used most frequently. While not exhaustive, our research did not find evidence suggesting that these vulnerabilities have been exploited. Snyk recommends that you continue monitoring your own environment and check your containers until patches are made available and deployed.

As always, using well-maintained parent images from trusted sources and staying up-to-date with the latest versions — including clearing out build caches and verifying the provenance of parent images — is a good best practice.

Installation and usage

To install and use our static detection tool, please do the following:

1. Clone the repository at https://github.com/snyk/leaky-vessels-static-detector

2. Build the Go binary with go build.

3. Run the detector with: ./static-detector dockerfile -f [PATH_TO_DOCKERFILE].

4. For additional execution options, please check the README.md file.

CVE-2024-23651 summary and advice

This container breakout vulnerability is severe and has the potential to cause damage to any underlying host infrastructure that is building containers.

Snyk recommends you update any instances of Buildkit to version v0.12.5 or later, as well as any software that depends on Buildkit.

Snyk has also disclosed several other Docker vulnerabilities in our research, which have been assigned CVEs: CVE-2024-23652, CVE-2024-21626, and CVE-2024-23653. You can read the summary of our findings across all the CVEs here.

This article is provided for informational purposes only. Snyk is not responsible for any errors or omissions, or for the results obtained from the use of this information.