Portland State hit hard by server failure

Many systems offline for more than six hours

The Portland State campus was paralyzed Wednesday afternoon when a power malfunction shut down the university’s main server.

Many systems offline for more than six hours

Kayla nguyen/VANGUARD STAFf

Computer labs in the library sat empty on Wednesday after a campus-wide server failure crippled Internet access.

The Portland State campus was paralyzed Wednesday afternoon when a power malfunction shut down the university’s main server.

Around 3 p.m., the Fourth Avenue Building Data Center suffered a catastrophic, multi-unit failure to its local Uninterruptible Power System fabric, Information Technology Director Janaka Jayawardena said.

The localized power outage caused the Internet, email, Desire2Learn, data storage and Wi-Fi to fail, said Scott Gallagher, PSU’s director of communications.

Technicians arrived on campus at 6:30 p.m. to assess the problem.

By 9:30 p.m., two of the three systems had been repaired. PSU technicians were then given the green light to begin restarting the systems.

By 11 p.m., most of the systems in the data center were back online, Jayawardena said.

The system failure only affected the PSU campus; surrounding businesses were unaffected.

“Most services, including file servers, networking, Wi-Fi, login authentication, email, printing, VPN [the library’s Virtual Private Network], web sites and others appear functional,” Jayawardena said.

The failure caused damage in some of the feeder breakers, which PSU and UPS technicians will be looking into.

The source of the power interruption is still unknown, Jayawardena said. Portland General Electric and building engineers assured PSU that no external or building-wide power event could have caused the problem.

The failure in the system was not human error but an equipment malfunction,
Gallagher said.

Jayawardena explained that remaining problems and effects of the system failure are still being assessed.

pdx.edu was down on Wednesday afternoon after a “catastrophic” failure of the university’s servers.

While the initial problem has been solved, it may still take a while to get all systems restarted and running again, Gallagher said.

Snafus in financial aid, registration

Assistant Director of Registration and Records Chris Hart said the interruption allowed his department to catch up on a few things offline. “There are a lot of things that are quite manual, like filing, that we’re always behind on,” he said.

Over in the financial aid office, there are around 21,000 student files and 10 counselors on staff.

While an outage is difficult in the middle of the day, at least it didn’t come during a critical time for the financial aid department, said James Ofsink, assistant director of student financial aid and scholarships.

“A few years ago it went down on the second day of school; that was the worst,” Ofsink said. Since the system was down overnight, around 45 students will be receiving aid disbursements a day late; but aside from that there were no other long-term problems, he said.

Library comes to a halt

Portland State library operations were incapacitated and left many students electronically stranded. Michael Bowman, the school’s assistant university librarian, described the scope of the damage.

“When things went down yesterday we were hit pretty hard,” he said.

The library was unable to check out books or use their printer systems. Students couldn’t access the vast resources that the facility holds. Despite the crash, library staff was able to look up books but still couldn’t check out books. Students couldn’t book the library study rooms.

“Luckily it’s after midterms, but it’s still not good,” Bowman said.

In response, the library gave out information to students to access its alternate server, VPN. VPN is designed to access online library materials when a student is off campus.

“Most students don’t know anything about VPN. But the people asking us are able to get in,” Bowman said.

PSU bookstore OK, D2L down

The PSU bookstore was unfazed by the crash. Store Manager Ken Brown said the store doesn’t rely on the main PSU server for the majority of its operations.

“It doesn’t affect us at all. We’re not on the university system,” Brown said. “We weren’t even aware.”

The network interruption greatly impacted students taking online courses, said Sara Smith, graduate coordinator for the Computer Science Department. Because the D2L systems were down, students couldn’t access any course content, including quizzes.

“Most of the departments in the College [of Engineering and Computer Science] had to close early. There was nothing they could do,” Smith said. “Pretty much everyone was gone
by 4 p.m.”