A broken power line cut electricity to a substantial portion of downtown around noon on Friday, leaving parts of PSU without energy for nearly three hours, causing servers to crash and resulting in the loss of an estimated 10 percent of e-mails sent out that day.
All of Portland State’s information systems, including the school’s online account system called Banner, Banweb and e-mail failed to function after a tree broke a power line near the intersection of S.W. Broadway and S.W. Sheridan near downtown. That break subsequently caused lines to burn at Sheridan and S.W. Sixth Avenue, with more lines burning 200 feet down the road on Sixth Avenue because of the immense power surges, according to PGE spokesman Mark Fryburg.
Cathy LaTourette, associate vice president of Human Resources, said that nearly all employees were able to remain occupied while many of the school’s computerized tools, Banner most importantly, were rendered useless. She said that employees were able to catch up on paperwork that did not require computers.
The broken line knocked out power to a rectangular area north of S.W. Abernathy of the Macadam area, east of Barbur Boulevard, west of S.W. First Avenue and south of S.W. Harrison in downtown, Fryburg said.
Many e-mails were lost during the outage, according to Jim Stapleton, assistant administrator of Unix at Portland State. He said that once the power went out, Information Technologies had about an hour to properly shut down the campus computer programs safely to prevent any kind of information loss.
Sue Riegsecker, associate director for computing services, said that they remained on the phone with PGE when the power was out, and once it came back on at about 3:30 p.m., they began working to get the information systems up and running.
She said they had everything up and running by about 5:15 p.m. except for the e-mail.
Stapleton said that upon rebooting the system, everything came back up fine, except for the campus e-mail, which suffered some loss.
When the power came back on, the master server tried to send information to the backup servers, but the backups read the information wrong and began deleting e-mails, Stapleton said.
“There are pieces of Friday missing, maybe Thursday night too,” Stapleton said. “It was ultimately a software glitch.”
Stapleton said they were able to stop the problem quickly, but they did have to freeze the server, causing more work for the team. He and three other coworkers put in close to 20 hours of work over the weekend recovering the e-mails and putting the system back in order.
About 75 percent of the campus e-mail system was working by mid-afternoon Saturday, according to Riesecker, and the additional 25 percent joined the rest by Sunday morning. The reason that some e-mail restoration took longer was because the e-mail is split up into four 25 percent blocks, including faculty, staff and students, and the team went through the blocks one by one.
Shattuck Hall, the Broadway housing building and the Ondine were also affected. There was only one major problem, according to Robyn Pierce, interim director of Facilities, with an Ondine resident becoming trapped inside one of the building’s elevators.
“We have an on-campus emergency elevator service and we had him out in about 15 minutes,” Pierce said.
Shattuck Hall, the building in which the current IT servers are located, is one of the few buildings affected that does not have a generator for backup support, unlike the Broadway building, which has generators to run the elevators in case of emergencies like this.
The servers are being relocated to a newly developed area in the Fourth Avenue Building, designed specifically to prevent this kind of situation.
“It is a delicate balancing act,” Pierce said. “We’re at the fruition of what has been coming for a long time.”
Fryburg said that in all, 1,400 customers were affected by this outage. He said that the company put four total crews on getting the problem fixed.
“We realized it was a significant failure,” Fryburg said, “That is why we put four crews on it.”