Please register to participate in our discussions with 2 million other members - it's free and quick! Some forums can only be seen by registered members. After you create your account, you'll be able to customize options and access all our 15,000 new posts/day with fewer ads.
Location: Mableton, GA USA (NW Atlanta suburb, 4 miles OTP)
11,334 posts, read 26,086,242 times
Reputation: 3995
Advertisements
Quote:
Originally Posted by Unsettomati
Well, that's why you have the redundant data center located somewhere else, on a different power grid. Ideally, hundreds of miles away from the production data center. That's disaster recovery 101. That way, a tornado or flood or civil disobedience or hurricane or what have you won't take down both locations.
Yes. And you would have to assume that the second DC was powered up and running all the time. Otherwise, it's just a dark emergency DC backup, and you could have an issue powering it up which would render that data center useless, at least in the short term.
Also, I can see getting the databases at the spare DC site up to speed and synchronized taking a relatively lengthy period of time. Even a clean switch to a second DC takes time. Interfaces to operations groups like flight dispatch and central load control, third party comms providers like ARINC and SITA, and possible connections to other third-party systems not inside the airline's direct control would have to be switched over cleanly and in a somewhat synchronized manner. Etc.
Quote:
Oh, and power... who doesn't have battery-backup and generators, anyway?
Delta has all that, I'm sure. But when I worked at NW, I remember one incident where the switch between the batteries and generators failed (due to human error) during a TEST of the system. That ended up taking out the production systems.
It sounds "easy", and I'm just an airline applications guy and not a DC expert, but on an enterprise level, especially when you're talking about a major airline with dozens of systems on multiple platforms and a very complex communications network, the successful switchover to a backup site can take hours even if the data is replicated in real time.
Delta has operational dependencies on other systems not controlled by Delta, for example, and if those don't switch over cleanly to the new location, it can impact certain aspects of their operation. Firewall rules and such SHOULD be configured ahead of time and tested peridocally, but I've seen that sort of common sense item fail before.
You know, even before this, with the TSA crap, I was about done with flying outside of unavoidable business travel. It's sad how bad the flying experience is now compared to "the good old days". Flying vacations are strictly in my past now.
You know, even before this, with the TSA crap, I was about done with flying outside of unavoidable business travel. It's sad how bad the flying experience is now compared to "the good old days". Flying vacations are strictly in my past now.
Wish were possible to do the same but decent rail service outside of commuter largely does not exist in USA, and regular Trans-Atlantic ocean liner service is gone as well. *LOL*
Yes. And you would have to assume that the second DC was powered up and running all the time. Otherwise, it's just a dark emergency DC backup, and you could have an issue powering it up which would render that data center useless, at least in the short term.
Also, I can see getting the databases at the spare DC site up to speed and synchronized taking a relatively lengthy period of time. Even a clean switch to a second DC takes time. Interfaces to operations groups like flight dispatch and central load control, third party comms providers like ARINC and SITA, and possible connections to other third-party systems not inside the airline's direct control would have to be switched over cleanly and in a somewhat synchronized manner. Etc.
There are mirrored systems. There's no reason you can't mirror an entire data center. Expensive? Sure. But then, so is a non-mirrored system. And so is losing your data center because you didn't mirror everything.
Quote:
Delta has all that, I'm sure. But when I worked at NW, I remember one incident where the switch between the batteries and generators failed (due to human error) during a TEST of the system. That ended up taking out the production systems.
It sounds "easy", and I'm just an airline applications guy and not a DC expert, but on an enterprise level, especially when you're talking about a major airline with dozens of systems on multiple platforms and a very complex communications network, the successful switchover to a backup site can take hours even if the data is replicated in real time.
Delta has operational dependencies on other systems not controlled by Delta, for example, and if those don't switch over cleanly to the new location, it can impact certain aspects of their operation. Firewall rules and such SHOULD be configured ahead of time and tested peridocally, but I've seen that sort of common sense item fail before.
Oh, I know.
I once observed a monthly test of the generator end early... because someone forgot to fill the tank and it ran out of gas. Oooops. And I once was at a data center when a security guard walked through the computer room and went to silence an alarm and -- ooops! -- instead of the SILENCE ALARM button he hit the EMERGENCY POWER-DOWN button. That was a long day.
And I've also seen backup data centers located right next to the main data center (terrible DR practice) or on beachfront property in Miami (Burger King, 1992, & Hurricane Andrew at cat 5 came ashore there) and myriad other unforced errors that don't happen because things are too complicated, they happen because no one wanted to spend the money. And so some director of IT came out looking real good on the finances because he decided not to spend money where it was needed, while just crossing his fingers and hoping that the unlikely event didn't happen until he'd gotten a promotion for being so dang cost-effective. In my experience, these sorts of things don't happen despite everyone's best efforts. They happen because someone(s) were looking out for their little niche of the company or their next review instead of looking at the big corporate-wide picture.
If a company is willing to invest against the unlikely yet entirely possible, and do it right, they can avoid these sorts of things.
I'm assuming the systems used by "some" airlines are just so antiquated that they have never moved from a legacy system to enterprise. A place where SPOF (single point of failure) goes away and you have archives and basically a second site that runs in tandem and can jump to the task at a moments notice.
What scares me is the that it seems there is no protocol for this type of issue/recovery. It may be a very rare thing to happen but it does and will happen. So kiosks were down, apps were not updating, and people had little insight on what to do.
Personally I would have just booked myself on another flight and dealt with the credit card payment another day. Not sure how easy that is to do but it would have been my plan instead of sitting in an airport not going anywhere.
Not antiquated as much as layered. There are many old, reliable systems that run in the background of these companies, but they have never been phased out because they work. Unfortunately, we end up with patches, updates, and new systems that pull data from the legacy ones that are layered on top and it all becomes a big crap sandwich. Delta had legacy systems which didn't go down with the rest of the company, but those systems alone cannot run the airline anymore.
Now imagine what will happen if the sun has an electromagnetic burp and a huge electromagnetic storm shorts out the electrical transmission systems and the wire and satellite based communications. At least the Delta problem only fouled the scheduling. Taking out the flight traffic controls could do a lot more damage.
Now imagine what will happen if the sun has an electromagnetic burp and a huge electromagnetic storm shorts out the electrical transmission systems and the wire and satellite based communications. At least the Delta problem only fouled the scheduling. Taking out the flight traffic controls could do a lot more damage.
An interesting thought. Solar activity is accounted for when flight planning, but a massive discharge would have far more reaching effects than the loss of VHF or HF radio communication. I hope it never happens.
Please register to post and access all features of our very popular forum. It is free and quick. Over $68,000 in prizes has already been given out to active posters on our forum. Additional giveaways are planned.
Detailed information about all U.S. cities, counties, and zip codes on our site: City-data.com.
Please register to participate in our discussions with 2 million other members - it's free and quick! Some forums can only be seen by registered members. After you create your account, you'll be able to customize options and access all our 15,000 new posts/day with fewer ads.
