Comments on: IBM Generator Failure Causes Airline Chaos

By: Downtime in 2009… « JANWIERSMA.COM

Downtime in 2009… « JANWIERSMA.COM — Thu, 17 Dec 2009 16:18:45 +0000

[...] Generator uitval bij IBM datacenter; luchtvaart chaos – Auckland Nieuw Zeeland. [...]

By: Grant

Grant — Tue, 20 Oct 2009 15:39:12 +0000

As a NZer living overseas this caught my eye… I’ve seen many stories and there does seem to be a lot of detail missing… what was the resilience available? As suggested, running this sort of facility without N+1(at least) seems like a BAD idea… but what are the restrictions imposed on IBM in terms of the facility (managed or owned by IBM?), costs, etc. And it seems poor practice not to expect (demand?) this. All fingers (and articles) point at the mainframe but what else was involved in the SERVICE outage? At the end of the day a power failure will take out midrange, storage and whatever else is connected, so its hardly the mainframe’s fault. “On restart, some data corruption and reboot issues were experienced across various platforms.” – maybe it wasn’t isloated to one platform…

By: Chris

Chris — Sun, 18 Oct 2009 01:07:36 +0000

I thought no one ever got fired for buying IBM…

By: Karl Robohm

Karl Robohm — Wed, 14 Oct 2009 14:53:14 +0000

I can tell you from my years of experience that both parties have equal blame in this outage; however, I would put more blame on the airline then the data center.

Data Centers need to maintain there systems properly and to do so equipment needs to be completely shut down to do full maintenance; however, it is never good a good idea to rely on back up systems to run your site if there is a reliable utility present at the site; IBM must have a standard MOP that requires generators to run during PM’s, which should be looked at.

The customer on the other hand has put their critical systems into an N facility (unless they were told a story about the site’s redundancies), which by all accounts is a very bad idea, they not only had a single generator, they also only had a single UPS feeding their equipment. Seems to me that the CEO of the airline should be looking within his org to find out who made the decision to colo at this site and lash out there and then begin the search for higher Tier concurantly maintainable site.

By: AC

AC — Tue, 13 Oct 2009 20:44:11 +0000

Lashing out at a service provider never helps the business relationship. Why would a major airline not have redundancy within the databases and applications? Relying on one data center goes against even a 2 page BCDR policy statement…

By: Rich Miller

Rich Miller — Tue, 13 Oct 2009 13:25:21 +0000

Here’s some additional information from an interview with Air New Zealand’s CIO, Julia Raue:

The incident occurred while running the main datacentre deliberately on generator power, in order to conduct maintenance on the uninterruptible power supply (UPS) system, Raue says.

“The intention was for the IBM team to bring down the UPS for maintenance, and run all systems on generator power deliberately bypassing the UPS during this maintenance window.

One hour into the window, the generator failed leaving all systems with no power,” says Raue.
The quickest expedient was to shift the systems back to mains power and this was done within “a matter of minutes”.

Unfortunately there had been a “crude and unclean shutdown of all systems”, she says. “On restart, some data corruption and reboot issues were experienced across various platforms.” Some key systems were then brought up at the secondary site.

The account isn’t explicit, but the references are all to a single generator.

By: WF

WF — Tue, 13 Oct 2009 12:23:43 +0000

Rich,
Can you get more details about this outage, i.e. what redundancies were built into the power delivery systems such as N+1 gensets and amount of UPS strength. If this was truly a concurrent maintainable facility then a single generator failure shouldn’t have caused such an outage. Thanks

By: Falcon124

Falcon124 — Mon, 12 Oct 2009 21:47:52 +0000

“Clearly the CEO of the airline expected that by hiring IBM nothing would fail ever. Is that realistic?”

Based on their marketing speak and the amount they charge you for their services, it would certainly appear that they’re trying to project just such an image

By: IBM data center performs face plant in New Zealand « The Server Room

IBM data center performs face plant in New Zealand « The Server Room — Mon, 12 Oct 2009 21:27:25 +0000

[...] pathetic performance, blowing Air New Zealand’s computer systems right out of the water (and here). The inability of data center providers to keep the lights on is getting [...]

By: TD

TD — Mon, 12 Oct 2009 20:05:04 +0000

Ouch! I guess the lesson is if you’re going to rely on a generator don’t neglect it’s maintenance or any other part of the system no matter how trivial it may seem. Of course even well maintained equipment can fail. So have backups for the backup equipment? If the maintenance involved shutting down the main power and the generator became the primary power source then yes they should have had a backup generator.

Some questions for the system designers are how many simultaneous failures of equipment are acceptable and what are customers expectations?

Clearly the CEO of the airline expected that by hiring IBM nothing would fail ever. Is that realistic?