How did one engineer catch a backdoor in Linux because his login was half a second slow?

A Microsoft developer named Andres Freund caught the backdoor because something felt slightly off. While investigating a performance problem in a development version of Debian, a Linux distribution, he noticed that logging in over SSH, the standard tool for reaching servers securely, was using an unexpectedly high amount of CPU and was a little slower than it should be. In one of his tests the delay was about half a second. Most people would have shrugged. Freund followed the thread, and on 29 March 2024 he announced that the upstream xz package had been backdoored.

XZ Utils is a compression tool that sits quietly inside most Linux systems. The backdoor, hidden in a library called liblzma, would have given an attacker who held a specific private key the power to run code remotely on an affected machine through OpenSSH. It scored 10.0, the highest possible severity rating. Normally OpenSSH does not use liblzma, but several distributions patch it to work with systemd, which does, and that is how the poisoned library ended up inside the login server.

The more striking part is how it got there. According to a later investigation, a user called Jia Tan spent over two years, starting in 2021, building trust. Fake accounts pressured the original maintainer of XZ Utils until Jia Tan was given maintainer permissions. Then the code went in, partly through release files that differed from what was in the public repository.

The timing was lucky. The infected versions, 5.6.0 and 5.6.1, were mostly only in pre-release distributions, so few systems had them yet. Security expert Alex Stamos said it could have been the most widespread and effective backdoor ever planted in any software product. A defender who cared about half a second prevented that.