The Office of Information Technologies found that less hard drives than originally estimated were affected by an upgrade error that caused data loss, but the recovery effort continues three weeks later.
On Friday, Feb. 5, at 4:10 p.m., OIT installed an upgrade to a remote hard-drive imaging program, Microsoft’s System Center Configuration Manager, and in doing so inadvertently wiped the hard drives of 302 campus computers, which is 88 fewer than initially estimated.
According to Sharon Blanton, chief information officer of OIT, this was the result of human error. When an OIT employee with administrative privileges wiped the hard drive of the computer OIT intended to upgrade, they also wiped the hard-drives of every computer connected to the server at that instant.
However, the damage was limited.
Blanton said, “It could have been way, way worse than it was. The image could have gone to up to 4,500 computers.”
Campus computers check in with the server for updates only every 30 minutes and, because this occurred late on a Friday afternoon, many faculty members had already shut down their machines.
By the following Friday, OIT had fully restored 155 computers. Blanton said, “15 people worked through the weekend…We had pretty much all the classrooms and labs back up by Sunday [Feb. 7].”
Faculty computers required more time. According to Jeff Brown, office coordinator in the Department of History, it took OIT “a little bit more than a week” to return the three or four affected computers to full functionality. Although, unsure of the exact impact on faculty research or class data, Brown said, “I assume that there was data lost.”
According to Blanton, OIT “replaced all of the hard drives [of affected faculty computers] with a new disk, so that they would at least be able to use their computer.”
OIT encourages everyone to store their data on the server. Those who followed this advice lost no data, nor did many general use lab computers and Macintoshes, which the SCCM did not affect.
OIT has the disks necessary to install programs like Microsoft Office, but for the restoration of more specialized programs, it has had to work directly with affected faculty.
As of last Thursday, 98 computers are waiting for data restoration, Blanton said. Some of those 98 computers are general-use machines, but others are “really important to people and have really important data.”
“We have to do everything possible to recover that data.” Unfortunately, some data has been lost, she said.
“A lot of this stuff comes back scrambled,” Blanton said.
According to Blanton, OIT could have prevented this by setting up its implementation of the imaging differently.
“We did not go far enough in setting up those granular levels of access,” she said.
In the future, OIT plans to divide installation tasks between at least two people, so that one person submits a command and another approves it, a system Blanton compared to the launching of nuclear weapons.