It’s been 10 weeks since ransomware crippled the network of city workers in Atlanta, and it has almost fully recovered. But after all the news stories about the dangers of anonymous bad actors online, are we ready for next time? If this serves as our new cautionary tale about network security — what have we learned?

The attack was first discovered on March 22, and diligent IT workers (and third-party incident-response teams) sprang into action. The city of Atlanta had almost fully recovered by mid-May, chief operating officer Richard Cox told the Atlanta Journal-Constitution, with the exception of the municipal court system. But it still wasn’t clear if they’d recovered all their data. “We are still in the process of going through files to understand the status,” Cox told the newspaper, adding that “That process will continue to take quite a while.”

As Atlanta’s new COO, Cox, had been on the job for exactly four days when the attack hit. Suddenly the city’s 450,000 residents — with nearly 6 million living in the greater Atlanta metropolitan area — could no longer pay their water bills online, or their traffic tickets, according to the New York Times. “For days, city workers were not even allowed to turn on their computers.”

Some response teams were working 24-hour shifts, with city officials working over Easter weekend to try to get a jump on the arduous recovery, Reuters reported. “Police and other public servants have spent the past week trying to piece together their digital work lives, recreating audit spreadsheets and conducting business on mobile phones in response to one of the most devastating ‘ransomware’ virus attacks to hit an American city.” One councilman lost 16 years of digital records, while the city auditor complained that “Everything on my hard drive is gone.” Three staffers for the city council ended up sharing an old laptop, while the city auditor ended up bringing her personal laptop to work. (To find details on current projects, she’d resorted to old copies of emails on her smartphone.)

“The city never stopped doing business,” quipped one local newscaster. “It just stopped doing business at the speed of the 21st century.”

The attack even delayed the release of the new mayor’s first budget, since the budget planning system remained compromised for weeks. Bills went unpaid, and courts were closed.

Ironically, the city is currently trying to convince Amazon to build their new headquarters in Atlanta.

But the city auditor points out that it could’ve been worse — since 10 of the 18 computers in the auditor’s office weren’t affected. And while the police department lost access to their investigative databases, by the end of March they were at least able to file their police reports digitally again. Atlanta received help from both the FBI and the Department of Homeland Security.

In another irony, Atlanta had just finished a cyber-security audit in January and had started implementing its recommendations. (Reuters reports that the audit “called for better record-keeping and hiring more technology workers.”)

Atlanta officials stressed to the AP that the attack wasn’t affecting the emergency response systems of the police and fire departments, and was also not affecting the safety of their water supply or Atlanta’s international airport — Though the airport’s wifi system was temporarily shut down “out of an abundance of caution,” their spokesman told the AP.

There was no evidence that anyone’s personal info was taken, but Mayor Keisha Lance Bottoms still urged people with data in the system to watch their bank accounts.

Misery Loves Company

Atlanta wasn’t alone. By the end of March, the same virus had hit eight separate government or healthcare organizations, including Colorado’s Department of Transportation and two hospitals in Indiana, according to a warning issued at the end of March by the U.S. Department of Health and Human Services. Though some made a quicker recovery, the agency noted that the attacks have “material impacts” on services to patients, adding that the risk “is expected to continue for the foreseeable future.”

The report warned that attackers seem to be focusing on a few specific sectors — government, healthcare, education, and municipalities — “likely because those systems and networks are critical and any downtime cannot and will not be tolerated, which increases the chance that the victims’ will pay the ransom.” NTT Security, a cybersecurity firm, told the New York Times that more than 87 percent of the ransomware attacks in America in 2016 were targeting the health care industry. A senior intelligence analyst at another security firm, Recorded Future, cites an underground forum where one black-hat group says they’ve started targeting state and local governments — specifically because they have bad security.

The New York Times noted ransomware attacks have also hit city or state systems in Texas, Alabama, New Mexico, and North Carolina. But things took a weird twist in Atlanta after the attackers demanded a ransom of $51,000 to decrypt the files. An image of the ransom message was shared with local media, which inadvertently revealed a way to contact the attackers to the general public. Soon the attackers found themselves fielding unwanted questions from the press, leaving them with a problem of their own.

“When questioned about their actions via the exposed portal, the SamSam group first demanded payment in exchange for answers,” reports CSO, “and later deleted the contact form entirely, calling the questions and other comments spam.”

Though cities often grapple with whether or not to pay the ransom, “given the SamSam group’s actions, it isn’t clear if payment is even possible now, since they’ve deleted the communication portal.” In April, a city spokesperson confirmed to ZDNet that the ransom hasn’t been paid.

Why Did It Happen?

CSO reported that Atlanta offered several of the attackers’ favorite targets, since “The city has RDP exposed to the public, as well as VPN gateways, FTP servers, and IIS installations.”

It’s not the first time the city’s been hit through a security hole. Cybersecurity firm Rendition Infosec reports that Atlanta was also hit in May of last year by exploits leaked from the NSA — more than a month after Microsoft had released patches and urged users to install them. “Based on our data, we can say for an organization of its size, the city of Atlanta had a substandard security posture in April 2017,” the firm’s founder told ZDNet, “making the scope of the ransomware attack far from surprising.”

Just the emergency IT services had cost the city $5 million, according to CSO, though Cox insisted to the Atlanta Journal-Constitution that “if you dig into those numbers a lot of those expenses are inevitably things we were going to have to invest in regardless.” The city also had a cyber attack insurance policy, and Cox told the newspaper that the city was having “ongoing conversations” with the insurance company now, adding diplomatically that “Our expectation is that we will be able to partner with them in a very fair manner.”

This points to one of the great ironies of ransomware attacks. CSO points out that one medical center spent $10 million responding to a $30,000 ransom demand (counting staff overtime and lost revenues)

But the city of Atlanta hopes to come back stronger than ever, Cox told a local news channel last week. “The expectations were set, early on, that this is a long-term process… ”

“What I’ve learned is you can’t totally secure a network, but you can do a really good job of defending it. We’ve learned that municipalities and organizations get attacked on a regular basis, so we never claimed victory, but we feel really good about our progress.”

Lessons Learned?

Back in November, the city of San Francisco lost control of all the ticket-selling machines for its BART transit system for a day. The trains kept running, and everyone rode for free until the problem was repaired the next day. The perpetrator had wanted $73,000 in bitcoin, but “We never considered paying the ransom,” said a spokesperson for the San Francisco Municipal Transportation Agency. Citing their in-place backup systems, he added that “We have an IT team on staff who can fully restore all systems.”

It’s not always that easy, according to the New York Times. Upon compromising a system, the first thing the attackers do is hunt for backup systems or software and try to delete it, according to a vice president at the security firm Icebrg.

Security researcher Brian Krebs recommends storing backup data offline or in the cloud, since “it’s important to ensure that backups are not connected to the computers and networks they are backing up… It should be noted, however, that some instances of ransomware can lock cloud-based backups when systems are configured to continuously back up in real-time.”

The FBI also reminds users to verify the integrity of their backups, as part of an online checklist of tips. The FBI also points out that one variant had been infecting 100,000 computers a day.

It’s been interesting to see the comments left on Krebs’ story. “At my computer shop, we get about 6 ransomware infections a month. Sometimes it’s grandma with the infection, sometimes it’s a hotel’s server.”

And of course, there was some second-guessing from the sidelines. “Is there ANY valid reason that internet addresses in Iran can access systems for something like the San Francisco MTA?”

WebReduce

• Last fall a tech podcaster finally cracked the case of the phantom phone calls.
• Atari co-founder Ted Dabney dies at age 81
• How Steve Jobs gave a special lecture at MIT in 1992.
• A programmer introduces his 7-year-old son to the 27-year-old programming language QBasic.
• Could we use a robot to pollinate crops?
• The pesticides industry may be threatened by the new precision of AI-powered, weed-killing robots.

Feature image via Pixabay.