At 1:30 a.m. on the Friday before Labor Day, the Beaverton School District’s fire suppression system deployed explosively into the room that housed the heart of the district’s computer network. An error in the fire-sensing system sent gases spewing through ceiling nozzles.
The chemicals destroyed much of the equipment, leading to the eventual failure of all hard drives, says Steve Langford, chief information officer for the district near Portland, Oregon. It was the last teacher workday before the holiday weekend, and four days before the 2013-14 year was scheduled to start for the 53-school system.
Half the district’s computer infrastructure was destroyed, along with a significant amount of vital data. The loss of data center functionality cut off phone and internet service. And because of an employee oversight, the scheduled backups had not been conducted, so the district lost all data for its $500 million budget and its finance system for 5,000 employees.
Online exclusive: Top 5 cyber threats for school districts
Despite all the preventive maintenance, firewalls and other protection procedures, school district networks remain vulnerable. Freak accidents, natural disasters and expert hackers can still wreak havoc.
“The FBI groups commercial networks into two categories: those that have been compromised and those that don’t know it yet” says John Boles, a retired FBI agent who ran the bureau’s cybersecurity division and now serves as director of global technology solutions for Navigant, a cybersecurity consulting firm.
“Schools are targets because they are becoming more and more connected, they have a lot of transient users, and they typically like their networks to be open.”
Whatever the cause, school districts will likely deal with network failures or breaches. To recover effectively, district technology professionals can respond with the following strategies.
Quick thinking required
When a network failure occurs, most school district leaders don’t immediately realize what has happened. At Beaverton, because the damaging gas was odorless and colorless, tech leaders didn’t immediately understand why the hard drives failed and wouldn’t reboot.
At Hartford Union High School District in the Milwaukee suburbs, students and teachers were using the internet one day in October 2016 when “suddenly, everyone was disconnected” says Nathan Mielke, director of technology services. “The icon was just spinning and nobody could connect.”
The first steps in responding are to figure out what’s happening and why, and to determine what equipment and data have been affected. Hartford Union’s network monitoring software showed that normal midday usage of 250 megabytes of data had jumped to between 800 and 900 megabytes.
With a phone call to the district’s internet service provider, Mielke realized the district was experiencing a distributed denial of service (DDOS) attack. As soon as the ISP realized what the problem was, it redirected legitimate traffic, a tactic also known as “right-limiting the port” Mielke says. The ISP returned to normal operations within about 15 minutes.
The next step is to assess what data is at risk and how to protect it. For instance, with email phishing—a primary threat for most school districts—a district staff member may inadvertently supply a hacker with access to passwords and other confidential information. When that happens, technology leaders must immediately determine what data was compromised and take steps to stem the damage.
“You may be able to change passwords, disable accounts or add two-factor authentication so the criminal can’t get into billing or other parts of the network” Boles says.
Similarly, if a district is attacked with ransomware or malware, the tech team must find out as soon as possible—usually with the help of outside security professionals—which parts of the network have been hijacked.
“Maybe you can prevent the attack from moving to the server and going across the enterprise by changing passwords and blocking further access” Boles says. “Find out if you’re bleeding data or if it can be contained. Determine whether you should shut down the network.”
It’s also crucial to know what servers house different types of data, Boles says. For instance, if server A, containing HR records, is compromised but server B, which holds financial data, is not, you only need to focus on HR information. And by segregating backups from your network ahead of time, you’ll be able to get your data back if you have to pull the plug on the whole system.
Transparency leads to teamwork
As soon as a problem is detected, it’s important to inform users throughout the district—but word choice is crucial.
When Hartford Union experienced the denial-of-service attack, “we communicated that we were experiencing an incident and that we were working with our ISP to handle it, but we were very careful with the words we used” Mielke says. “It wasn’t a breach and it wasn’t really an attack. It was a cyberevent, and we knew that nobody’s information was at risk.”
In the case of a breach, in which students’ or staff members’ personal information may be at risk, communication is not just the right thing to do, it’s likely the law. Regulations vary in each state, but they generally require that network administrators notify affected parties about a breach of personally identifying information within 30 to 90 days, Boles says.
The timeline for health information is between 30 and 60 days, depending on the state.
When making notifications, it’s also important to avoid causing alarm for those who weren’t impacted, but make sure to inform those who were. At Beaverton, open communication with staff members throughout the district, especially those in finance and other departments affected by the loss of data, was vital for the recovery effort.
Some of Langford’s first steps were making calls to the chief human resources officer and the chief finance officer, who came in on a holiday weekend to strategize how to continue ordering supplies, making payments and meeting payroll. One group inventoried all the data needed for those processes, while another team recovered data from the lost databases, which included those of production, HR and finance.
“We had to be completely transparent because the people we needed to help us couldn’t help us if they didn’t have all the information” Langford says. “We had 19 days to pay people, and if we didn’t pay on time, we would face multimillion dollar fines. The failure affected everyone in the district and we had to work together to overcome it.”
Help from Belgium
Once Beaverton’s team was assembled, the recovery took on aspects of a police investigation. Someone recovered a database used for testing purposes that contained critical pieces of information for the rebuild. A relatively new employee had a PDF of a partial payroll run on their desktop. Though that practice didn’t meet security standards, the document was used to populate a new database with personnel data.
Then, the employee who had made the backup error finally found a copy of the lost database on heat-damaged, corrupted tapes that were about to be destroyed. He also located a Belgian company that manufactured a tool to copy damaged databases. He ordered the product, learned how to use it and rescued the lost data.
“We were not able to recover all of the reports we had written, but with the data back, we could recreate the reports we use for daily work” Langford says. Beaverton also brought in its vendor’s disaster recovery specialists, who coordinated all staff work.
“You need to have people who have been through this before working with you” Langford says. “It’s such an emotionally charged
