American had a big mess on its hands this week when the link with Sabre went down and it had to stop flights from departing. Did any of you get stuck in this? What’s your story?
American had a big mess on its hands this week when the link with Sabre went down and it had to stop flights from departing. Did any of you get stuck in this? What’s your story?
17 comments on “Topic of the Week: American’s Tech Problems”
Finding out *why* the system went down to such an extent would be *very* interesting. Alas being a big corporation I very much doubt this information would leak out to the public domain…
Could this be a prelude to problems developing after the AA-US Airways merger?
JRS – I can’t believe how many times I’ve this theory that it somehow means things will be a mess during the merger. Why? How would this even remotely be related?
Because it suggests there’s an inherent instability/flaw/vulnerability/whatever in AA’s IT which may cause further problems when the system is stressed during integration. Seems like a pretty logical concern to me.
Since AA presumably is going to find and fix the problem, however, you could also argue this may actually prevent issue down the road.
it has been reported elsewhere that it was a communications / networking issue. These things happen, so let’s not all get in a tizzy
It was all an in-house issue since other companies using Sabre were not effected so AA will be hiding the real issue of their problems such as lack of money for equipment maintance/replacement, grumpy IT employees, etc.
Recently airlines (CO and US come to mind.) have reported the Reason for the Outage of their internal outages such as this. If AA wants to gain credibility they will report that. Maya Leibman likely will push for this to be reported, as well as detailing what changes they have made to prevent this.
While I mentioned her, I just read Maya’s bio, she is an Awesome person. I hope she stays with the new AA.
Gmail was down the next day in the morning and I don’t think anyone self-immolated. This stuff happens.
Gmail was only down for a 0.007% of their users, or a total of 29,750 people. Re: http://arstechnica.com/information-technology/2013/04/gmail-goes-down-for-small-number-of-users-usual-panic-follows/
American Airlines’s outage affected many many more people.
As a Platinum AAdvantage member who was already on my way to ORD when the system went down, I was curious as to how this was going to play out. When I arrived to check my bag, the kiosks were down. I was then directed to the counter (strangely, not many people in the check-in lobby), and the agent prepared a bag tag like in the olden days (paper bag tags and literally adding my information by hand to a bag check-in list). Since they could not print boarding passes, I guess I was lucky that I already had one on my iPhone (and went right through TSA pre-check).
I went to the Admiral’s Club to wait for my flight – at this point I knew there was a systems problem, but the departure board was still showing all flights departing on time. About 1/2 an hour later, one could sense people becoming nervous, particularly in light of Boston the day before. The Admiral’s Club desk started calling names of some who were on international flights, I presume to start re-booking. AA amazingly immediately staffed the desk with 12 or more people, and the lines started to form. A telephone call to Reservations confirmed that the issue had extended to actual flight operations, and that no one could even really re-book because there was no computer access. Wow.
Some people were in shock, some angry, and others (like me) just resigned themselves to waiting a while to see what would happen. Incoming flights were landing, but nothing was going out. The bar was doing a brisk business at this point. But real kudos to AA for staffing up so very quickly.
My flight was then cancelled. And the departure board went completely black. As there were only 3 people in line at the Admiral’s Club desk, and one now could not get through to Reservations on the phone, I queue’d up to see what my options were. They were refunding and re-booking on AA (without promises, of course) and other airlines (if requested). I was put on a later flight (4 hours after my original scheduled departure) and upgraded to first class. I had internet (and chardonnay) access at the Club, so it made sense to wait a while longer to see how things progressed.
Eventually, I made it out (5 hours late), and I was not inconvenienced all that much. The following day I received an e-mail awarding me 5,000 AAdvantage bonus miles for enduring the melt-down. It seems to me that despite the system breakdown, AA immediately stepped up to the plate and avoided a public relations disaster. At least from my perspective.
I had a similar experience. I am guessing you had already checked in at home or via mobile device before arriving at the airport.
I did not check in from home and when I arrived at DFW everything appeared outwardly normal. I could not check in at the Kiosk – it displayed an error message – and was directed to the counter. There were 3 people in line ahead of me in the First Class line for the counter. They called me up to the counter and explained the problem. If I had a boarding pass, they would manually write up a bag claim. Since I did not have a boarding pass, I got back in line. The counter staff made frequent announcements and offered suggestions such as trying to check in via the web site or the AA mobile app from cell phones or laptops. The mobile app was unable to check me in. As the lines began to grow, extra staff appeared to help keep the various lines forming in some semblance of order and answer questions from people just arriving. (DFW has very limited space in the check in area.) Agents also arrived with a large cart loaded with bottled water and started passing them around. I must say the AA staff was incredibly cheerful and friendly. One agent at the counter even made announcements based on information she was getting off the CNN website using a passenger’s iPad.
Local TV showed up about 90 minutes into my wait. They set up cameras and tired to interview a few people. They seemed kind of glum that no one was in an uproar. As reported above, the monitors were all on and showing all flights on time, which the TV crews recorded. I overheard one cameraman on a cell phone complaining to someone that he “had come all the way out there for nothing”. Then they headed out just a couple minutes before the system came back up.
Once the system came back up AA had every ticket counter position staffed. The two people ahead of me in line got issued boarding passes very quickly. When it was my turn I was informed my flight was cancelled and I had already been rebooked on a later flight. Only delay at the counter was a couple of minutes to reset the bag check printer.
Inside security the terminal was crowded but not jam packed. They were already starting the boarding process at every gate I walked past. (C17 down to about C8 and then back up to about C24) Within about 30 minutes the terminal was looking fairly normal and planes were starting push back.
Local media was reporting flights waiting on the tarmac and potentially exceeding the 3 hour rule. I did not notice a large number of aircraft immediately pulling in to spots vacated by departing flights, so I am suspicious as to the actual number of flights stranded on the tarmac at DFW. Maybe someone else can comment on that.
All in all, my observation was that this was handled very well. I agree that they avoided a PR disaster
Outages such as this one have occurred before, and inevitably will occur again. There is no reason to continue to whip this subject up. I worked for a major airline (not AA) for more than 25 years, and was deeply involved in the initial automation of reservations, ticketing and flight operations. Cranky should have been around then, although one could easily revert to manual processes if needed.
I question his motives toward AA when he cites his worst flight segment as being on AA from ORD-LAX, on which he was given a courtesy upgrade to First, but lacked the electronic entertainment and had to “endure” a talkative seatmate. I expected far more from such a headline!
Bob – Oh, the drama. I would suggest re-reading the post because you have a few things wrong. I never said it was the worst flight segment. I said it was “one of the worst I’ve had in a long time.” That is absolutely true.
And nowhere do I say it’s all American’s fault. Other than the failure of the IFE, it wasn’t American’s fault. Though several commenters seem to be convinced that blame must be assigned and I’m throwing it at American, that’s not the case. Re-read the post. (And re-read the headline as well, because that says nothing about it being the worst flight, as you suggest.)
Big companies usually have back-up systems, but theirs apparently did not work. It will be interesting if we can learn what failed, and what the implications are for other large computer networks.
Well Ed, I expect that AA has redundant lines and systems, but that the fall back did not work for some reason. There are several examples from practice where things went wrong.
* The phone company sold two redundant glass fibers to the data center; unfortunately they were in the same conduit, so the back-hoe could tear them both in a single movement.
* The life system and the hot standby run the same software version. One data request makes the first system crash (and automatically reboot.) The standby system detects the crash and jumps into action, to fall over when that same data request is retried a fraction of a second later. Without human intervention the crash and reboot action can go on for months.
BNA-LGA Tuesday — delayed 90 minutes due to “suspicious package” at LGA that emptied the entire D terminal. Then 2 hours on the tarmac @ LGA waiting for a gate to open. None ever did and we taxied to hangar apron and were bussed to terminal. Half the trip lost due to some caffeine-addled TSA idiot and the other half attributable to AA systems F-up. PITA and AA gives us 7,500 miles. Make them elite qualifying and I’m satisfied.
I’ll keep it short and sweet. JetBlue, Virgin America, (then) Continental all had computer issues at one point. AA is no different.