5 US Government Agencies Hit So Far in SolarWinds HackFellow Victim FireEye Traces Breaches to Trojanized SolarWinds Software Updates
Network intrusions at the U.S. Commerce Department, the U.S. Treasury, FireEye and more all appear to be linked to subverted software updates for a network monitoring product called Orion, made by SolarWinds.
On Sunday, the U.S. Commerce Department confirmed it had been targeted by hackers, and the U.S. Treasury has also reportedly been struck. On Monday, new victims were added to the list: the Department of Homeland Security, State Department and National Institutes of Health, The Washington Post reports.
The Post reported last week that the same group was behind an attack against cybersecurity firm FireEye (see FireEye Says Nation-State Attackers Stole Pen Test Tools).
In an update late Sunday, FireEye warned that starting around March and continuing through June, new SolarWinds’ Orion software updates included backdoors, which it has dubbed "Sunburst." The malicious software updates were signed using valid digital signatures, and could steal files, profile systems and disable system services, it says.
FireEye warns that “the actors behind this campaign gained access to numerous public and private organizations around the world.”
Numerous victims are now being identified. "We can confirm there has been a breach in one of our bureaus," the Commerce Department says in a statement. "We have asked CISA and the FBI to investigate, and we cannot comment further at this time."
The U.S. Cybersecurity and Infrastructure Agency, or CISA, on Sunday issued an emergency directive "in response to a known or reasonably suspected information security threat," noting that the affected Orion products are versions are 2019.4 through 2020.2.1 HF1.
The Post reports that the National Telecommunications and Information Administration, which advises the president on telecommunications issues, was also attacked. Reuters reports that the attacks are considered to be so serious that the National Security Council held an emergency meeting on Saturday.
Numerous aspects of the attacks remain unclear, including exactly what the attackers stole and whether the incidents pose any immediate threat to U.S. national security. The New York Times reports that the attackers had access the Treasury and Commerce Departments' email systems.
On Sunday, SolarWinds disclosed that it is investigating a “potential vulnerability” that may be linked to software updates for its Orion network monitoring platform. The updates were released between March and June, according to the company's statement, which also mentions FireEye.
“We believe that this vulnerability is the result of a highly sophisticated, targeted and manual supply chain attack by a nation state,” SolarWinds says. “We are acting in close coordination with FireEye, the FBI, the intelligence community and other law enforcement to investigate these matters. As such, we are limited as to what we can share at this time.”
FireEye CEO Kevin Mandia said in a Sunday blog post that "the campaign demonstrates top-tier operational tradecraft and resourcing consistent with state-sponsored threat actors."
Supply Chain Attack
FireEye’s description of how the attack unfolded against it suggests that organizations using SolarWinds’ software would have had little defense against being infected. Software updates are “signed” using public key cryptography, and updates that contain invalid keys would not be accepted. But any malicious software update that had a valid key would be automatically accepted by the software as being legitimate.
The key used to generate the digital signature would have been a closely guarded secret. But FireEye’s Sunday account of how these attacks unfolded suggests that attackers somehow gained access to SolarWinds’ software-signing infrastructure, which allowed them to substitute a subverted version of the company's update, then sign it to make it still appear to be legitimate.
This type of software supply chain attack has been used to devastating effect before. The NotPetya attack of 2017, which appeared to be ransomware but turned out to function instead as disk-wiping malware, began after attackers successfully altered updates for accounting software called M.E. Doc, to add backdoors (see NotPetya Patient Zero: Ukrainian Accounting Software Vendor).
FireEye says the backdoor code added to SolarWinds Orion is stealthy, and remains dormant for about two weeks. To hide its network traffic, the malicious code uses a protocol native to Orion’s software called the "Orion improvement program" and also hides "reconnaissance results" for its operators "within legitimate plugin configuration files, allowing it to blend in with legitimate SolarWinds activity,” FireEye says.
The attackers apparently took great care to avoid detection. For example, when remotely accessing their victims’ systems, they chose IP addresses in the same country as their victims, FireEye says. Attackers, it adds, also set hostnames on their command-and-control infrastructure to match a legitimate hostname within victims’ infrastructure, and they also used valid login credentials once they were inside systems, all of which would have helped attackers escape detection.
Domains appear "old" but all appear to have had noticeable changes (ownership, hosting, nameserver, etc.) between late 2019 and ealry 2020 which would align with thie #SUNBURST operation starting around March 2020: pic.twitter.com/DCin2Lgl4I— Joe Slowik (@jfslowik) December 14, 2020
Joe Slowik, a senior security researcher at DomainTools who tracks nation-state attacks, notes that based on the information released about the attack to date, after extensive prepositioning efforts, the attack campaign it appears to have begun about nine months ago - around March.
Experts say the attack against SolarWinds is sure to send a shiver through the IT security community and could have more wide-ranging consequences as the count of victims potentially increases.
SolarWinds is a popular managed service provider that provides a range of tools and services for organizations to manage their IT infrastructure. But Information security experts have long warned of the danger posed if attackers can successfully subvert a widely used product or service, since it can give an attacker one-to-many access to a number of victims, oftentimes via the hacked provider's own infrastructure.
SolarWinds' customers, according to the company's website, include the five branches of the U.S. military, the Pentagon, State Department, NASA, National Security Agency, Postal Service, National Oceanic and Atmospheric Administration, Department of Justice and the White House. It also serves hundreds of large businesses, including 425 of the 500 largest publicly traded U.S. companies.
The company's Orion platform gives IT shops the ability to pull data from various systems and display it on one console. The platform can also be used to control those systems.
SolarWinds’ products have administrative access to organization’s networks, tweets Dmitri Alperovitch, the co-founder and former CTO of cybersecurity firm CrowdStrike, which competes with FireEye.
“Monday may be a bad day for lots of security teams,” Alperovitch writes.
Network management systems such as Orion remain prime targets for attackers since they may have access to all systems on a network, writes Jake Williams, a former offensive hacker for the NSA who's now president of Atlanta-based security consultancy Rendition Infosec, via Twitter.
If you have East/West netflow, consider doing some analysis of NMS traffic and looking for outliers. That won’t be easy in most cases.— Jake Williams (@MalwareJake) December 14, 2020
I’ll close by reiterating that hitting an NMS like SolarWinds often gives attackers keys to the kingdom. It’s like domain admin++. 11/
Even if a network management system has read-only access, attackers that control the system can still use it “to read configurations, which often include enough information for attackers to laterally move to those systems,” he writes.
Williams advises companies using network management software to closely monitor access to administrator interfaces and all traffic going into the system, since any illicit access can have serious repercussions. He also notes that for other SolarWinds customers, the indicators of compromise - digital forensic clues about these attacks - should get released within a few weeks, which will help potential victims identify if they might have been breached, as well as better safeguard themselves for the future."
“I’ll close by reiterating that hitting an NMS like SolarWinds often gives attackers keys to the kingdom,” Williams writes. “It’s like domain admin++.”
CISA says in a statement that it has "been working closely with our agency partners regarding recently discovered activity on government networks" and that it is "providing technical assistance to affected entities as they work to identify and mitigate any potential compromises."
The intrusions come at a fraught moment for the U.S., which continues to face an ongoing, dangerous surge in coronavirus cases as well as a rocky presidential transition.
President Donald Trump’s efforts to overturn the election on unverified claims of fraud have met repeated defeat in court. A lawsuit by Texas against four other states that aimed to throw out election results was unanimously rejected by the U.S. Supreme Court on Friday.
The agencies charged with helping to defend the government and American businesses against foreign hack attacks have also taken a hit. In November, Trump fired former CISA Director Christopher Krebs, claiming that Krebs made false statements regarding the election. Krebs, as well as other government agencies and election officials, continue to say that the U.S. election was the most secure one ever held (see Analysis: Does Krebs' Firing Leave US Vulnerable to Attack?).
Tracking Cozy Bear
During the course of this year's election season, the U.S. government appeared to have avoided any serious hack attacks. But it's unclear if the newly discovered attack campaign might lead to any revised assessments, especially if the list of known victims continues to expand, or the attack does trace to a group such as Cozy Bear.
Security experts believe Cozy Bear is affiliated with Russia’s SVR intelligence service. A long list of intrusions have been linked to Cozy Bear, including against Democratic National Committee officials in 2016.
In July, the U.S., U.K. and Canada accused the group of targeting agencies and companies involved in COVID-19 research (see APT Groups Target Firms Working on COVID-19 Vaccines).
Executive Editor Mathew Schwartz contributed to this report.