The news in late December hit like a thundercrack: A hacking group breached one of SolarWinds’ signature networking management software products, Orion, to install a backdoor that could potentially target thousands of customers, including many important U.S. government agencies.

While information about this sophisticated supply-chain attacks continues to make headlines, the U.S. government and intelligence agencies investigating the incident now believe the attack is the work of a well-funded and resourced hacking group with links to Russia and bent on conducting reconnaissance and cyberespionage.

The aftermath has left cybersecurity experts to ponder what happened and how so many security policies could have failed to protect the systems and data that they were specifically designed to protect.

“A parallel stream to the current triage should be an examination of why our defenses and other early warning systems failed so miserably,” Brandon Hoffman, CISO at security firm Netenrich, told Dice. “This should be considered a critical effort. While we are busy triaging there is most certainly additional or follow-on attempts by other adversaries across the globe. There’s blood in the water and everybody smells it.”

As investigators and security analysts continue to unravel what happened to SolarWinds, experts already believe there are several security, IT and developer lessons to be learned from the breach... with many more to come as additional information about the attack becomes public. However, a full understanding of what transpired is likely to take months, if not years, to fully comprehend.

“The clean-up effort will take many months and will consume quite an amount of time and money,” Dirk Schrader, global vice president at New Net Technologies, said. “Organizations which have been using the SolarWinds Orion solution are best advised to assume that their networks and system have been infiltrated and will need to adapt to that. Some highly sensitive areas might even require a clean sweep and a fresh install of every asset involved.”

What Happened?

While not all the details of the SolarWinds breach are known, analysis released by those companies involved offer some insights into what occurred.

At some point, the hacking group behind the breach was able to gain access to SolarWinds’ network and slip a backdoor, called Sunburst, into a software update for Orion customers (which appears to have shipped in March). The attackers in this case were also able to access the software’s digital certificate, which meant that many customers automatically installed the update since it appeared to come from SolarWinds itself.

The update, coupled with the Sunburst backdoor, appears to have been sent to some 18,000 SolarWinds customers, according to financial filings with the U.S. Securities and Exchange Commission. After the initial attack, the New York Times reports that about another 250 victims were sent a secondary malware called Teardrop that could exfiltrate data, install additional malicious tools, and allow the hackers to leap to other systems within a compromised network.

The attack itself may have gone unnoticed for even longer if not for security firm FireEye, which found in early December that hackers had stolen the company’s penetration testing tools. This would lead to uncovering the SolarWinds breach and how the Orion update had been compromised.

From there, other revelations came to light. Microsoft, for example, believes that at least 40 of its customers may have been affected by the attack, and the software giant also disclosed that the hackers may have accessed source code for some undisclosed products. Other tech firms, such as Intel and Nvidia, also appear to have been affected by the attacks, which also impacted the networks of the U.S. Treasury, Commerce, Homeland Security, State and Energy departments as well as parts of the Pentagon.

On Twitter, Dmitri Alperovitch, the co-founder of security firm CrowdStrike, noted that the SolarWinds breach was “a very carefully planned, stealthy and deliberate espionage operation against (likely) a few hundred high value targets.”

Early Lessons To Learn

For cybersecurity observers, the SolarWinds breach showed how the reliance on automated software updates, even those with signed digital certificates, needs to be rethought, especially in an increasingly interconnected world with a growing digital supply chain.

“Automatic updates have been questioned for many years and the SolarWinds story is a good example why it is necessary to have this discussion,” Schrader from New Net Technologies, told Dice. “The question about being a ‘clean source’ as a software vendor is about the combination of robust, resilient development processes—and them being audited independently—and the application of core security controls along the process steps, change control and vulnerability check on the device used in the development and the output itself.”

Roy Horev, the co-founder and CTO of Vulcan Cyber, an Israel-based remediation intelligence provider, believes that the SolarWinds breach holds lessons beyond security and IT, and should make the industry rethink how to approach issues such as DevOps, continuous development and the reliance on third-party software libraries.

Horev notes the example of a Chrome plugin called “The Great Suspender,” which was shipped with the ability to connect to and execute code from third-party servers to manipulate web requests in its latest updates about how even simple updates can have long-lasting consequences when sources are not checked.

“These cases are becoming increasingly common, are hard to identify and should always be a concern,” Horev told Dice. “The key takeaway is not to update blindly, although it may seem counter intuitive. It’s critical to understand what problems and issues a specific patch solves before installation and to cherry pick the correct fixes to the correct problems.”

Vishal Jain, CTO at cloud security firm Valtix, noted that many companies, especially in the rush to finish digital transformation projects, have embraced various cloud services without thinking through the security consequences of what can happen. This includes an overreliance on automation, such as updates that are pushed from third-party software and services suppliers.

“The cloud model is—automate all you can, so people can focus on things that can’t be automated,” Jain said. “This does, however, raise another issue that we have started to see in many companies, ensuring that automation is executing tasks that are valid and appropriate, and putting the security controls in place to do just that. Many businesses today are focusing projects around egress filtering for cloud workloads.”