Delta global outage (tornado, shop, to move)

[arjay57]

Sounds like Delta may need to spring for some new IT gear.

Delta Meltdown Reflects Problems With Aging Technology - WSJ

[rcsteiner]

I can't read the article, but "aging technology" is rarely the actual problem.

Older tech in the airline industry uses IBM or Unisys mainframe hardware which is constantly refreshed by the vendor (a customer can certainly choose not to keep up with the times), and the software on critical systems tends to be relatively reliable if well designed to begin with. I think DL is mostly an IBM shop, tho I seem to remember Unisys InfoHub hosting at least one Delta system 10-15 years ago.

The airline sites I'm aware of are all well hardened to handle things like power outages. We had several diesel generators, batteries, and a pair of completely redundant power mains at Northwest's Data Center in MSP, and systems could also be switched to locations off site (mainframes and servers in other data centers) if the main location went FUBAR.

I don't know Delta's ATL operations, since I don't work for that airline directly, but it sounds like the issue was related to the power handling infrastructure and not its core systems.

It sometimes doesn't take much ... a large airline has many many interconnected systems, and even the loss of a single border router can cause massive issues since it can take a long time for something like that to rebuild its internal route tables if a failover to a redundant unit doesn't function properly and it has to start over from scratch. When I hear the words "F5" and "spanning tree", I still have nightmares.

[flamadiddle]

Well, if they are that reliant on a single data center, I hope that F5 you mention doesn't turn out to be a Tornado the next time!

[flyonpa]

F5 Makes Very High End Routers & Network Switches. (Very fault tolerate with auto fail-over) They are a very good product.


Wondering of the "Power" issue is someone in the datacenter got fired, and hit the RED EPO Button as they walked out the door from the datacenter.


EPO = Emergency Power Off Generally a Red Button near the doors (Covered by a Clear Plastic Box) that will manually kill the power to the datacenter in the event of a fire.

[rcsteiner]

Originally Posted by flyonpa

F5 Makes Very High End Routers & Network Switches. (Very fault tolerate with auto fail-over) They are a very good product.

I understand that F5 products are VERY good. However, it's the negative incidents that I tend to remember.

[AnsleyPark]

Originally Posted by flyonpa

F5 Makes Very High End Routers & Network Switches. (Very fault tolerate with auto fail-over) They are a very good product.


Wondering of the "Power" issue is someone in the datacenter got fired, and hit the RED EPO Button as they walked out the door from the datacenter.


EPO = Emergency Power Off Generally a Red Button near the doors (Covered by a Clear Plastic Box) that will manually kill the power to the datacenter in the event of a fire.

Apparently, the failure was on Georgia Power's side of things.

[arjay57]

Originally Posted by flyonpa

Wondering of the "Power" issue is someone in the datacenter got fired, and hit the RED EPO Button as they walked out the door from the datacenter.

EPO = Emergency Power Off Generally a Red Button near the doors (Covered by a Clear Plastic Box) that will manually kill the power to the datacenter in the event of a fire.

So it could have been sabotage?

[rcsteiner]

Originally Posted by arjay57

So it could have been sabotage?

Possible, I suppose, but unlikely. People don't usually sacrifice careers to make a point.

[flyonpa]

The Way Major Data Centers are powered :

Two Different power feeds to the site, from different directions,
They Feed s HUGE Battery (UPS system) room (The Datacenter runs off the batteries, batteries are recharged by the Grid/Generators).
Onsite Generation (Usually diesel),

If the UPS system see a interruption in Grid (Georgia Power) power, it triggers the Generators to start. They have 2 Times (or more) the amount of power generation that is needed with the backup generators to run the data center and other critical building services (AC, Fire Systems, Lights, wotk station, Phones, PBX's etc).

The UPS can only run the datacenter for 10-15 minutes, witch is more then enough time for the Generators to spin up and start to provide power. If everything work right.

My Guess is some fault in the UPS system, and the onsite staff did not know how to bypass the UPS and run directly off Grid Power or the Generators,

But it's not a seamless movement from one to the other if they physically have to move from one power source to another and not go thru the UPS.

To restarts, thousands of servers, mainframes, disk arrays, Routes, switches, etc, Do rollback of Database, and re-sync them, with the remote logs all takes time.

[flamadiddle]

Originally Posted by flyonpa

To restarts, thousands of servers, mainframes, disk arrays, Routes, switches, etc, Do rollback of Database, and re-sync them, with the remote logs all takes time.

That's why companies the size of Delta should never put all their eggs in one basket. Their systems should be synced and distributed worldwide via several data centers. If one data center loses power, gets hit by a Tornado/Hurricane/Meteorite or whatever, they should be able to redirect to a surviving data center and continue operations.


I imagine many will be losing their job over this one and that Delta will be hiring qualified Network personnel in the very near future!