The SolarWinds breach story continues to get worse.
The list of known victims now includes US departments of Commerce, Defense, Energy, Homeland Security, State, the Treasury, and Health.
More worrisome for those responsible for cybersecurity at enterprise data centers, however, are the technology vendors that allowed the compromised SolarWinds Orion software into their environments. Those vendors, as far as we know, now include Microsoft, Intel, Cisco, Nvidia, VMware, Belkin, and the cybersecurity firm FireEye, which was first to discover the attack.
"I think that number [of compromised vendors] is going to grow," said Greg Touhill, who served as US federal CISO under President Barack Obama and who is now president at Appgate Federal Group. "I think we're going to find, as we untangle the knot behind this, that SolarWinds was not the only victim, and that FireEye was not the only victim in its space."
Altogether, up to 18,000 organizations may have downloaded the trojan, according to SolarWinds. The number of organizations targeted for attacks that would follow the SolarWinds breach is unknown at this time. Microsoft said it has identified more than 40 organizations in that last category. It did not name any of them but said that 44 percent were technology companies.
Other researchers have been investigating the malware’s technical details to identify more second-order victims.
A list published by the cybersecurity firm TrueSec includes Cisco, Deloitte, Mount Sinai hospital, and several other hospitals, medical organizations of other kinds, local governments, educational institutions, power companies, and financial institutions.
Cisco and Deloitte are also on the list put together by cybersecurity researchers at Prevasio. In addition, their list includes Ciena, Belkin, Nvidia, NCR, SAP, Intel, and Digital Sense.
The attackers' focus on enterprise IT vendors is particularly worrying because it could mean that the SolarWinds breach is just one of many, that other tech vendors were compromised the same way SolarWinds was. There could be even more back-channel supply chain attack vectors, which are difficult to defend against.
The US Cybersecurity and Infrastructure Security Agency has warned that the attackers may have used other initial access points besides SolarWinds.
For example, attackers have been using a zero-day vulnerability in VMware's access and identity management products to attack government systems, according to the NSA. VMware confirmed that it was notified of the vulnerability by the NSA and released a patch earlier this month. VMware also confirmed that it found instances of the compromised SolarWinds software in its environment, but said that it saw no further evidence of exploitation.
The VMware attack had one other thing in common with SolarWinds. Attackers used its software's own communication channels to evade detection. According to the NSA, exploitation activity took place within a TLS-encrypted tunnel associated with the software' web-based management interface.
Cisco has also confirmed that it found instances of the compromised SolarWinds Orion product in its environment. "At this time, there is no known impact to Cisco products, services, or any company data," and its supply chain IT networks have shown no evidence of compromise, the company said.
But that doesn't mean there aren't any problems further up the chain.
"If Cisco third-party manufacturers have IT networks not associated with Cisco’s business, Cisco does not have visibility to those networks," the company said. "Cisco is actively engaging with vendors to assess any potential impacts to their business."
Renewed Focus on Supply Chain Security
Many of this year's highest-profile attacks, such as the record-breaking wave of ransomware, exploited users' willingness to click on links in phishing emails or used stolen credentials to break into systems.
The SolarWinds attack channel didn’t involve any compromised users to gain initial foothold. Instead, the attack happened completely on the back end, through a compromised software update process. A data center looking for indicators of compromise in suspicious user behaviors, malware downloads on user devices, or in unusual network activity would have nothing to find – even as the attackers used the SolarWinds Orion platform to explore the environment.
In fact, FireEye only discovered the breach when an attacker tried to use stolen credentials to register a new device for multi-factor authentication.
"Had the MFA not caught it, had the employee not reported the stolen credential, this would not have been discovered," Liz Miller, VP and principal analyst at Constellation Research, said. "It would still be providing open doors to any and all."
Appgate's Touhill recommends that data centers that use products by SolarWinds, by Cisco, by VMware, or by other potentially compromised companies need to take a look at what their potential risk exposure is.
"Take a look at those vendors you rely on," he told DCK. "You have to be in conversation with your suppliers and see if they're following best practices, like checking the integrity of their code and checking their own systems repeatedly for any indications of compromise."
And that's not a one-time conversation, he added. "One-and-done is not going to be good enough."
Helpful to data center security managers in the aftermath of the SolarWinds breach is the amount of attention the attack has received from security researchers.
"We know how it was compromised and what to look for," Ilia Kolochenko, CEO at ImmuniWeb, a cybersecurity firm, said. "But I'm confident that SolarWinds is not the most negligent company around the globe. It's reasonable to hypothesize that they're not the only victim."
The difference is that nobody knows what other IT vendors have been hacked, and what those indicators of compromise are.
ImmuniWeb recently researched about 400 major cybersecurity companies and found that 97 percent had data leaks or other security incidents exposed on the dark web – as well as 91 companies with exploitable website security vulnerabilities. As of September, when its report was published, 26 percent of those were still unfixed.
Researchers also found more than 100,000 high-risk incidents, such as login credentials, available on the dark web. "SolarWinds is probably just the tip of the iceberg of compromise of technology companies around the globe," Kolochenko told DCK.
"You cannot trust anyone, even your security vendor," Holger Mueller, an analyst at Constellation Research, said. The only solution is code review. "But who can and wants to review source code of security vendors?"
What might emerge in response is a new kind of vendor – one that provides tools that check security software for malware, he said.
The SolarWinds breach illustrates another problem faced by data center IT security – that it needs to work more closely with the broader IT teams.
According to a recent survey by Ponemon Institute on behalf of Devo, the lack of visibility in IT security infrastructure is the top barrier to the effectiveness of security operation centers, identified as a problem by 70 percent of IT and security professionals. And 64 percent say that turf issues and siloed operations are a top barrier to effectiveness.
"This attack really should be a massive wakeup call for IT and security teams to be far more aligned," Constellation’s Miller said.
Lack of visibility makes it difficult to detect and respond to threats. According to the Ponemon survey, 39 percent of organizations said that it took, on average, "months or even years" to respond to a security incident.
"This is exactly the chaos and siloed behaviors attackers, especially sophisticated ones, rely on," Miller told DCK.
Time for Defense in Depth and Zero Trust
The SolarWinds breach proves once again that anyone can be hacked, from the most security conscious government agencies to the most security conscious cybersecurity vendors.
It’s relatively easy for sophisticated attackers to stay under the radar.
"When I've done red teaming, I don't think I've ever been caught until I've gone to do some really obvious action or make some noise," said David Wolpoff, CTO and co-founder at Randori, who breaks into companies’ networks for a living.
That doesn't mean security managers shouldn't try to detect breaches.
"Of course, we're not safe," Wolpoff said. "We're never going to be safe. But we're always making these tradeoffs in life, and cyber is no different. How much risk are you willing to accept from your partners and vendors? And if something goes wrong, what is your failsafe?"
For example, he said, every security professional he talks to says that they believe in the presumption of compromise and defense in depth. "But then nobody goes and takes those actions.”
Data centers that haven't yet moved to the zero-trust security model should start making plans, several experts said.
"Frankly, I think many companies are late in implementing zero trust, and I think that's one of the very first steps," said Appgate's Touhill.
"If you haven’t already adopted a zero-trust posture across your data center networking, now would be an exceptional time to get that in gear," said Miller.
The rules of the game have changed, said Mike Lloyd, CTO at RedSeal, a cybersecurity firm.
In the past, the question that data center security managers would ask themselves was, "How do I reduce the dark space I cannot see?" Now, they have to ask themselves, "How do I contain damage caused by the things I use to look at my own network?"
"This is a vertigo-inducing perspective," said Lloyd. "If you can't trust your monitoring software, how do you monitor anything?"