The XZ Backdoor CVE-2024-3094

Unveiled on the 29th of March 2024 is the high-stakes investment and prolonged campaign by a malicious actor to plant a backdoor in the Linux software library liblzma to gain access to multiple operating systems via Linux distributions, which arguably worked out successfully. That is until a curious engineer noticed a glitch.

Currently known affected upstream software and proposed mitigation:

What is XZ?

XZ, also known as XZ Utils, provides a command-line interface for compression and decompression functions and is often conveniently bundled into Linux distributions, such as Debian, Ubuntu, and many others.

// Example usage of XZ for context:
$ xz llm_rag_context.json

// Results in a new compressed file on disk:
$ ls -al llm_tag_context.json.xz
.rw-r--r--   81k lirantal 31 Mar 10:29   -N  llm_tag_context.json.xz

What is Liblzma?

Liblzma is the software library that implements the LZMA compression and decompression algorithm and provides software bindings for other programming languages to build with it, such as the xz CLI program.

What is CVE-2024-3094?

The CVE-2024-3094 vulnerability identifier was published on March 29th, 2024, to reflect a critical 10.0 severity score in the liblzma package.

The CVE report attributes the security risk to a malicious code discovered in the liblzma software library that allowed data modification when interfaced with the library’s code, thus resulting in the loss of integrity and leading to more devastating outcomes.

What is the impact of the XZ backdoor?

The SSH protocol and its accompanying toolset on Linux allow for remote access between hosts. Due to the XZ backdoor allowing the interception and modification of data, and some Linux distributions making liblzma available for the SSH program, it is initially assumed that one impact attributed to CVE-2024-3094 could be an authentication bypass in the sshd program.

In simple terms, a person holding an RSA key could authenticate remotely to any open and backdoor’ed SSH server.

It is now speculated that the XZ backdoor could result in remote code execution. However, an ongoing investigation is being conducted to analyze the backdoor code planted in liblzma and retrace all the contributions made and connected to the malicious actor involved.

Time frame for context:

• The malicious build file was added to the Debian xz-utils on February 24, 2024.

• The 5.6.0 version of the xz-utils tarball was released on February 24, and version 5.6.1 on March 9, 2024.

• The 5.6.0 version was added to Fedora on February 27th, 2024.

The story of the XZ backdoor

A significant security breach was reported on 29 March 2024, when Andres Freund, a contributor to the Openwall's oss-security mailing list, disclosed a potential compromise within the liblzma library, part of the widely used XZ utils package.

This reveal emerged after Freund noticed unusually high CPU usage by the sshd process, prompting a deeper investigation.

The core of the issue lies in maliciously crafted compressed test files embedded within liblzma versions 5.6.0 and 5.6.1, aimed at establishing a backdoor through alterations to the tar files' configure script. The exploit, though lying dormant under normal conditions, activates in systems using a specific patch for the SSH server, allowing for the circumvention of sshd authentication and granting unauthorized remote system access.

The following screenshot is taken from Andres XZ malware analysis sent to the Openwall OSS-Security mailing list:

This sophisticated exploit leverages the GNU C Library, commonly known as glibc, IFUNC mechanism, to add a resolver for the crc64_resolve method that installs an audit hook to substitute the RSA_public_decrypt function in OpenSSH with a compromised counterpart. OpenSSH typically doesn't require liblzma; however, the exploit capitalizes on a chain-loading scenario where a third-party patch causes libsystemd to load, which subsequently loads the affected liblzma software library. At the time of writing this, the backdoor affects sshd only on a subset of Linux distributions that apply this patch to enable systemd notifications

A particularly insidious aspect of this attack was the introduction of a modified build-to-host.m4 file in the GitHub-hosted tar release, which was absent in the Git repository and thus couldn’t be tracked back to its origin in source control.

This M4 build file extracts and injects the malicious code during the build process on specific systems: x86-64 Linux platforms using glibc and GCC, built via dpkg or rpm, popular package build tools for the Debian and Red Hat Linux distributions.

The fallout from this breach led GitHub to archive the repository hosting XZ utils for violating terms of service. Investigation into the origins of the backdoor points to Jia Tan, a maintainer for the xz project, though it remains unclear if the act was intentional or if the maintainer's account had been compromised. The real identity behind this maintainer account still needs to be determined.

This supply chain attack has impacted several Linux distributions, including Debian 13 and unstable, Fedora Rawhide, Fedora 40, Kali Linux, and OpenSUSE Tumbleweed, with Arch Linux urging its users to update promptly. Notably, most distributions adhering to a stable release update protocol were spared due to using older, unaffected XZ versions. FreeBSD also remains unaffected, thanks to its inclusion of pre-incident XZ versions and the attack targeting Linux's glibc. On the flip side, if, for example, you automated your container image deployment using the latest available updates from Fedora or Debian, you’d be shipping the vulnerable XZ program.

Tracked under CVE-2024-3094 and rated with a CVSS score of 10, the highest possible, this incident underscores the critical nature of supply chain security and the complexities surrounding the trust and verification of open-source contributions.

Undoing Harm: The XZ Cleanup Operation

Upon the story unfolding, Lasse Collin, the original maintainer of the XZ Utils library, rushed to commit a source code fix to the software library’s build configuration. This code diff demonstrates the sneaky and elaborate work of the malicious actor to pull off their backdoor campaign. The dot (.) character caused a failure in the build tool, which caused the build tool to disable a security sandbox and bypass security controls.

Detecting XZ vulnerability with Snyk

There are various ways to detect the XZ vulnerability — for free — using Snyk. Using the Snyk CLI, you can test your projects locally:

• For applications, run snyk test --unmanaged from the Snyk CLI to compare unmanaged dependencies in your repository and detect individual packages and their vulnerabilities.

• For containers, run snyk container test to detect supported operating system packages that depend on vulnerable versions of XZ.

You can also run a scan across all your projects in your Git repositories, providing you with a report of all the direct and transitive dependencies you are using. 

From this report, you'll see if you're relying on XZ, and how many paths in your dependency graph it's being used in. You can also run a quick search for "CVE-2024-3094" across all the projects. 

The intersection of supply chain security and Open Source

The XZ backdoor has rippled through the open-source and cybersecurity communities, sparking debates and concerns about the integrity of open-source software and the ever-present risks of supply chain attacks. This incident is a stark reminder of the SolarWinds breach and the various malicious packages discovered in software registries like npm and PyPI, underscoring a pattern of supply chain vulnerabilities that have profound implications for global cybersecurity.

Jia Tan, who emerged under the GitHub handle JiaT75, and associated personas such as Jigar Kumar and Dennis Ens, have illustrated a sophisticated, multi-year campaign to embed malicious code within XZ Utils, an essential software utility used in numerous Linux distributions. Starting with seemingly benign contributions and escalating to direct pressure on project maintainers to gain commit access, the tactics employed were cunning, leveraging both technical and social engineering techniques to exploit the trust and collaborative nature inherent in the open source community.

The backstory of Jia Tan’s involvement with XZ Utils, beginning with suspicious activity in other projects like libarchive, paints a complex picture of premeditated actions leading up to the CVE-2024-3094 discovery. These actions, including the creation of testing infrastructure and deliberate attempts to obscure the malicious intent through the misuse of ifunc implementations, demonstrate a chilling level of forethought and manipulation.

This incident raises critical questions about the sustainability of trust and security in open-source software. The very nature of open source—its openness and reliance on community contributions—becomes its Achilles' heel when malicious actors infiltrate with the intent to harm. The incident also highlights the significant stress and mental health challenges faced by maintainers, who often volunteer their time and expertise under immense pressure to keep projects secure and up-to-date.

The parallels with past incidents, such as the sweeping SolarWinds breach or the continuous stream of poisoned packages in public software repositories, cannot be ignored. Each event shares a common thread—the exploitation of trust and the complexity of modern software supply chains. They serve as potent reminders of the sophistication of cyber adversaries and the vulnerabilities inherent in the digital infrastructures upon which we increasingly rely.

Given CVE-2024-3094 and historical precedents, the open-source community and the broader tech industry must reevaluate and reinforce the security practices surrounding software development and distribution. This includes enhancing scrutiny of contributions, implementing more robust verification processes for maintainers, and fostering greater collaboration on security best practices. Additionally, there is a pressing need for sustainable funding models for open-source projects, ensuring maintainers have the resources and tools to secure their software effectively.

As we navigate the fallout from this incident and those like it, we must engage in open, critical discussions about the balance between the open source ethos and the necessity of security. The resilience of open-source software and the broader digital ecosystem depends on our collective ability to adapt and strengthen our defenses against those who seek to undermine it.

Who is Jia Tan? Is it an individual or a nation-state sponsored actor? What other projects did they potentially compromise and backdoor? Security researchers and developers are actively tracing back the footprints of Jia Tan’s git commits activity, origin, timezone, and any other information they can scrape to put the pieces together.