On Friday, March 29, Microsoft engineer Andres Freund saved the world. He found malware shortly before it was due to be distributed to the vast majority of the world’s servers, including those used by banks, hospitals, governments and Fortune 500 companies.
The bit about saving the world – that’s not hyperbole. An attack was foiled that was intended to create a backdoor in the global computers that run the modern world. The attackers would have been able to run malicious programs to make planes crash and turn off the lights.
We don’t know what was planned. Perhaps the backdoor would have lurked in our servers for years as the attackers bided their time until one day, bam!, a crippling cyberattack is launched and everything goes dark and survivors live in caves fighting over food scraps and desperately poking at their dead phones. “This could have been the most widespread and effective backdoor ever planted in any software product,” said Alex Stamos, a security consultant who has been involved in many of the world’s biggest cybersecurity incidents in the last twenty years.
The New York Times made it a front page story. The IT world is reeling, panicky, flooded with adrenaline and smelling of fear and sweat. I’m going to give you a very simplified short overview of what happened. This is being studied in granular detail from every possible angle. There is more detailed coverage here and here, and a deeper dive into the technical details here.
WHAT WAS DISCOVERED
The software that runs the world is constantly changing. It’s called “Linux” – just nod your head, you don’t want to know the details. The key to this story is that at its heart Linux is open source software.
The point of “open source software” is that everything happens out in the open. The software code is available for anyone to review; it’s not held in secret by protective companies. Obsessive armies of people work on it – they collaborate, suggest changes, test for vulnerabilities, argue about everything, provide feedback, test some more, and argue some more. Communities exist for every tiny nook and cranny of the code. At some level relationships and personal trust are part of the mix for each community.
Andres Freund is a Microsoft engineer who found niggling errors deep in an obscure tiny sub-pocket of a sub-pocket of a Linux system. (Linux is not a Microsoft product, but Microsoft and its engineers play important roles in the Linux world.)
Something didn’t feel right and Freund couldn’t leave it alone. He was evaluating updates that were supposed to be benign and helpful. The updates to the program were nearly at the point of being approved for general distribution. He studied and fretted and kept looking and eventually it tumbled: he was looking at a deeply disguised backdoor – malware hidden in innocent-looking code. An attacker could use it to gain unauthorized control of a server and run any program – shut it down, drain the information from it, make it malfunction, create Skynet and destroy humanity with giant robots, whatever they wanted. No one would know the backdoor was there until it was used.
“At first, Mr. Freund doubted his own findings. Had he really discovered a backdoor in one of the world’s most heavily scrutinized open-source programs?
““It felt surreal,” he said. “There were moments where I was like, I must have just had a bad night of sleep and had some fever dreams.” . . .
“In the cybersecurity world, a database engineer inadvertently finding a backdoor in a core Linux feature is a little like a bakery worker who smells a freshly baked loaf of bread, senses something is off and correctly deduces that someone has tampered with the entire global yeast supply. It’s the kind of intuition that requires years of experience and obsessive attention to detail, plus a healthy dose of luck.”
Freund sent a warning message to a forum devoted to open-source security and the attack was halted. IT security experts breathed sighs of relief, birds sang, Andres Freund costumes will be worn at Halloween, there will be more of the usual things that go along with a victory.
HOW WAS THIS DONE?
The malware was planted by someone named Jia Tan.
Over a period of more than three years, Jia Tan became a trusted member of the Linux community and specifically the corner devoted to the data compression utility where the bad stuff happened. Jia Tan contributed more than six thousand innocuous code changes between 2021 and this year. Jia Tan participated in online conversations and emailed other developers.
Meanwhile, the individual with the most responsibility for this particular Linux subroutine was under pressure and had personal issues. (He’s just a guy named Lasse Collin, the “random person” in the XKCD comic above. No one blames him for anything.) A few people in the community complained that Collin was slow to approve things and suggested that Jia Tan could take over some of the workload if he was given admin privileges. Jia Tan gained permission to approve code updates early in 2023.
Jia Tan worked diligently and over time slipped in bits of the malware, deeply disguised, in pieces that were invisible and even harmless in isolation, but added up to a backdoor when stitched together.
A different picture emerged after the attempted malware was discovered.
There is no Jia Tan.
All of the emails, all of the forum chat, all of the work done on code – all of it was a scam to build trust in someone who doesn’t really exist.
The people who promoted Jia Tan as an admin are almost certainly fake, just part of the scam.
The best scams require intensive planning and lengthy preparation. If this was a movie, it would be The Sting or Ocean’s Eleven. The groundwork for the Linux malware was laid over a period of years by a group playing a long game. Their tradecraft was impeccable.
The details of the stealthy campaign to create Jia Tan and embed him as a trusted member of the Linux community are fascinating. Here’s a description of what’s known so far.
WHO WAS RESPONSIBLE? WHAT DID THEY WANT?
No one knows who did this. No one knows what the end game was.
Every security researcher feels in their bones that this was done by a nation-state. The lengthy build-up, the sophistication of the code – it feels like China. Or Russia. Or Korea.
There’s no proof and no clear answer. Every communication by Jia Tan is being examined by its IP address (masked by VPNs), time zone on code commits (mostly matches China, except for a few that match Eastern Europe or Russia, and anyway it can be changed manually), time of messages (much of the work was done during work hours in Middle Eastern time zones), and date of messages (some were sent on the days of Chinese holidays – a clue or misdirection?)
Russia has a government sponsored hacking group with a reputation for the most sophisticated supply chain attacks on the planet, including the Solarwinds attack in 2021. Maybe it was the Russians. Or Iran. Or Israel. So far, we don’t know.
WHAT’S THE BIG PICTURE?
This isn’t the first discovery of a major supply chain hack.
A “supply chain hack” is a cyberattack that injects a Very Bad Program into something that is distributed to computers at many different companies.
Three years ago an elite Russian hacking group slipped malware into Solarwinds network monitoring software used by thousands of companies and government agencies. A huge amount of data was extracted by the Russians over a period of more than a year before the hack was discovered by chance.
I wrote a series of articles about the Solarwinds debacle in 2021. When you read about this year’s discovery of Jia Tan’s Linux attack, this is the larger perspective you need to keep in mind:
“Cyber warfare is more effective if governments keep the details secret. Today’s cyber world is in roughly the same position as the nuclear world from the 1950s to the 1990s: tense but stable global peace. The superpowers have occasionally flexed their cyber muscles in the 2000s, careful not to be provocative enough to require a drastic response. The US and Israel launched the Stuxnet virus to disable Iranian nuclear facilities. Russia tested its powers by shutting down 30% of the computers (and much of the infrastructure) in the Ukraine. North Korea took down the Sony Corporation network because it was irritated by a Seth Rogen movie. China quietly steals terabytes of data from American corporations. All sides implant backdoors and viruses in as many foreign networks as possible, little electronic nuclear bombs, and only fear of retaliation keeps them from being detonated.
“The US also does this kind of hacking against the Russians. Because these programs are shrouded in secrecy, we don’t know how the Russian hack compares to our own efforts. If the US does not come out of the gate roaring with indignation and shaking our fist at the Russian hackers, it may be that both Russian and US government officials are fully aware that we are embedded just as deeply in Russian networks.”
This time we got lucky. Thanks to the efforts of an obsessive engineer, Linux does not have a backdoor in an obscure corner of its code.
We dodged a bullet. That’s important and worth celebrating.
Maybe it’s the only bullet that was headed toward us.
Unfortunately it’s a lot more likely that we are in the middle of a hailstorm of bullets flying every direction. Russia and China may have embedded many little bombs in the programs that run our infrastructure, and it’s likely that we have done the same to them.
A number of conventional wars are flaring up around the world in Ukraine, the Middle East, and elsewhere – the kind with bullets and bombs. Those are important and distressing, of course. But don’t take your eye off the cyber world where battles are constantly being waged just out of sight.
“Cyber warfare is the modern equivalent of the global arms race from the 1950s to the 1990s. If the bits start flying, a cyber war has the potential to be just as destructive as a nuclear war. Modern life depends on our networks to power our electrical grid, run the hospitals, keep infrastructure working, and coordinate the supply chain that brings us food and water and clothes.
“Our military leaders and intelligence agencies are spending more time and resources on cyber battles than any of us realize. Every year intelligence agencies prepare a global “Threat Assessment” for Congress; for several years cyberattacks have been the number one threat on the list. The fear of devastating cyberattacks is at the center of American defense strategy. It is arguably more important to our decision-makers than anything to do with conventional warfare.“
Have a nice day!