Overshadowed by the Frontier/Spirit merger announcement was an operational meltdown at Frontier on Monday. The airline had to cancel more than a third of its flights, and of those that did operate, only a third were on-time. As I usually do when this happens, I turned to masFlight to see the carnage, but I saw something strange. Frontier recovered very quickly, and I was curious as to why.
Let’s start with a chart that shows the full story, and then we can get into the details.
Frontier 2022 Operational Performance
The year did not start off well for Frontier or for any other airline for that matter. But once the holidays passed, Frontier settled into a predictable rhythm. On good weather days, the airline ran most of its flights. On bad weather days, it struggled. You can see that in the green line where the snowstorm that ran through the southeast and up into the Northeast in mid-January hurt operations and then again at the end of January with that nasty Nor’easter.
Every time, the recovery was quick. On January 29, Frontier canceled a quarter of its flights. On January 30, it canceled less than 5 percent, and by January 31 it canceled only 1.1 percent.
This might not sound like an overly impressive feat. After all, other airlines had similar recoveries. But weather should be easy. By that, I mean airlines know weather is coming, so they have the ability to position their airplanes and crews in advance. They can cancel flights proactively and be ready to recover. Some airlines have failed at that too in the past, but during late January, none were overly terrible. This should be a basic function of an airline.
A tech outage is something else, however. Those strike without warning, and airplanes and crews are often frozen where they stand at the time the outage hits — unless, you know, they’re in the air, in which case they’ll generally prefer to go ahead and land the thing. We have seen this with a whole lot of airlines, and the recovery can be very, very ugly.
But… look at Frontier. Things went south on Monday morning when the outage occurred, and by the end of the day it had completed only 61.9 percent of scheduled flights. The flights that did operate only ran 34 percent on-time. That is a true meltdown, and if you were flying on Frontier on Monday, my condolences.
Now take a look at Tuesday. The airline sprung back to life very quickly with a 98.4 percent completion and arrivals within 14 minutes of 77.6 percent. I didn’t include on the chart, because it wasn’t a complete day, but when I checked last night, Frontier was up to 99.7 percent completion and 88.0 percent on-time. That’s honestly rather incredible for a day or two following such a terrible Monday. How did this happen?
If you ask Frontier, which I did, here’s the story.
We worked hard throughout the pandemic to build a network schedule that is resilient and more easily recoverable when disruption occurs. This contributed to our ability to recover as quickly as we did. Our goal was to contain the issue to a 24-hour period and we believe the team did a great job to make that happen. Our Systems Operation Center team made plans that were achievable to recover. Our Crew Scheduling team worked feverishly, for many hours into the night, to get crew schedules updated and make individual contact with them. Our pilots and flight attendants strived to take care of our customers, as did our Customer Service team in offering options to get them to their destinations. It was a team effort across the entire company. Foundational to every decision made throughout the recovery was to get all of our crew members, aircraft, and the operation as a whole back on schedule as quickly as possible and avoid the irregular operation becoming a multi-day event.
It’s a good story, so I’m sure the airline is happy to spread the love around to all workgroups, and kudos should go out to all of them. Naturally, I’m particularly interested in the network design so that’s what I’ll focus on here.
Airlines like Southwest and Spirit like to run their airplanes through their networks. Southwest is the posterchild of this type of operation, and that’s why you find so many “direct” flights that stop but don’t change planes.
Here’s a typical example, Southwest 737-700 N264LV’s routing from yesterday:
This airplane touched every corner of the country on its journey from Albuquerque in the morning to Sacramento at night, and if weather was bad in one place, it’s going to mess people up far away from the weather. It can also result in crews being stranded way out of place.
Spirit does the same. For example, ship N697NK went Tampa – Detroit – Las Vegas – Sacramento – Las Vegas yesterday. That creates a lot of potential pain points, but airlines like to do it because it creates better commercial opportunities by adding more frequencies on routes that may have some nonstop service and better direct opportunities in markets that have none.
Frontier, however, doesn’t do that as much. It focuses more on the out-and-back style of operation when it can. An example here is ship N349FR — an A320neo better known as Hops the Rabbit — which yesterday flew Denver – Charlotte – Denver – Cedar Rapids – Denver – San Diego. This morning it was pegged to fly back to Denver.
Sometimes it’s not that clean, but it still generally fits a pattern. Look at N383FR to see what I mean there. Another neo, Manteo the Red Wolf, flew yesterday West Palm Beach – Islip – Tampa – Buffalo – Tampa – Atlanta. This is a lot of north-south flying in the same general region. And it ends up in Atlanta which is a base for the airline.
In these cases, it does make it easier when something goes wrong. Look at what Manteo did on Monday. He flew what turned into a horrendous redeye from LaGuardia to Miami. He was then supposed to do Miami – Atlanta – Newark – Atlanta. The outage hit, and the Miami – Atlanta flight that was supposed to leave at 7am didn’t get out until 1pm. So what did Frontier do? It scrapped the Atlanta – Newark – Atlanta roundtrip. After it came in from Miami, the airplane went on a scheduled 1:50pm Atlanta – West Palm that didn’t leave until after 3. That airplane came back to Atlanta and was ready to start fresh the next morning, as were the crews.
I suppose the point is that if you cancel enough — or even TOO much if you want to be conservative — on the day of the meltdown and you have an operational design like this, it means you can recover more quickly.
That is a big part of it, but there’s one other thing here that really helped…. dumb luck.
Go back to that chart at the top. This outage happened on a Monday. Monday is a busy day for the airline, but you know what’s not? Tuesday. MasFlight tracked 478 flights on Monday but only 319 on Tuesday, and that absolutely makes it a whole lot easier to recover.
Apparently, if you combine luck with a solid plan, you can recover quickly. Go figure.
25 comments on “Frontier’s Quick Meltdown Recovery”
“Markets that have known” should be markets that have none
Fixed
CF, absolutely right. Segregating fleet types and turns to/from different HUBs is the best way to firewall one HUB from another. If spokes turn airplanes from one HUB to another HUB it allows weather problems at one HUB to take down the other HUB with it, not to mention doubling the problems for the spokes.
Can somebody explain generally why tech outages shut down airlines so completely?
I work in a very large hospital, which is similarly reliant on its computer systems. But we have a fully thought-out backup plan for outages, involving paper forms, fax machines and telephones. It’s annoying, it’s slow, and some things don’t get done. But on the whole, we are able to function.
Clearly airlines don’t have anything like this. There must be a good reason, since he’s outages have hit many airlines, and seem to paralyze all of them. But airlines functioned before computer networks were as ubiquitous as they are now, so I wonder what the reason is that they can’t have a fallback.
grichard – The airlines have plans, but the problem is the same thing you detail, it takes a lot of time, it’s slow. And you don’t have the option of having some things not get done. So while airlines can usually operate (depending upon the nature of the outage) they can’t keep themselves ontime and things spiral out of control if they don’t start canceling quickly.
Airlines use programs that aren’t always hosted locally on a server. So there needs to be a secure link from the internal network, via either a secure internet connection or a direct pipe into the host server. And many systems are integrated, so if one piece goes down, it may cause a glitch somewhere else.
This can be a link between Navitaire, their res system, and their dispatch software – its not getting the correct flight schedule. Or if they use Sabre for weight & balance, that is then tied to BOTH systems (sometimes) for passenger & bag counts. Now lets say that Navitaire’s airport check-in system (DCS) link with SITA CUTE (Common Use Terminal) goes down (since I doubt Frontier has their own computers and permanent counters), then that system may not get the passenger name list, bag counts, etc. Or lets say some dumb*ss construction crew isn’t paying attention and accidentally cuts the pipes (most internet cables are underground) somewhere, that causes that link to go down. Bam, you literally have to have a specialist come in and build a new connector for that cable, then bury it. I won’t even start with the most common issue at airline IT – the link to SecureFlight goes down. TSA will fine the CRAP out of you if a passenger isn’t vetted against the no-fly list at least 3 times during the whole process (3 days out, at check-in, and again at boarding – NO joke!).
You know that “DING” you hear when your boarding pass is scanned? That’s not just the reservation system checking your seat off.. there’s a LOT more going on! Its checking to make sure you’ve been vetted by TSA. Its checking your E-TICKET for validity and you didn’t try to change it (or cancel it, as happened to WestJet many years ago and cost them $$$). It is verifying your actual reservation. It then validates your age against your seat assignment. If all of that clears, and yes, in a matter of a second, then DING, you can board.
So, we don’t know what system went down. It sounds like it wasn’t the core dispatch system, or crew scheduling. Maybe it was the link to ACARS (which monitors aircraft position, performance, and lets dispatch ‘talk’ to the crew like a 1995 text message). But ONE system went down that was critical enough to keep planes from departing.
Heck, for all we know, it could have been the TSA SecureFlight system!
Thanks for the thorough explanation. Super helpful and interesting.
Thanks for the detailed explanation. I love your example of everything that happens behind the scenes when a boarding pass is scanned; I thought it just queried the airline’s system to make sure that the boarding pass was valid, the seat hadn’t changed, and to prompt the gate agent to remind people who are sitting in exit rows.
A topic like this would be a great topic for Cranky or another aviation blogger/vlogger to dive into a bit more, if they could translate the technical jargon into layperson speak (which you did a very good job of) and make it interesting to non-technical people, perhaps focusing on a relatively narrow piece of things.
Thank you; that was very helpful.
I’m not really surprised that a bunch of flights get canceled when there’s a big IT outage. I *am* a bit taken aback, however, when I hear headlines like “NO planes from XYZ Airlines have departed in the last 3 hours.” I would have thought that a bunch of manual work would let an airline, minimally, limp planes into position for a good start the next morning. But you do paint a convincing picture of the complexity involved.
“Its checking to make sure you’ve been vetted by TSA.”
Is that really happening? The other day I used my shiny new CLEAR membership for the first time. Their scanner had some issues with my boarding pass (gave error), and after a few times the CLEAR guy just stamped it and waved me through the TSA ID checkpoint to the X-ray machine. And there was no issue at the gate.
Also, what would happen if I enter the sterile area with a BP to for flight A (which gets scanned at TSA), and then the flight cancels and I book another flight, perhaps on another airline without going through TSA again? Is the verification linked to the person independent of the flight, i.e., does the machine just check that a person with that name and DoB go through a TSA checkpoint recently?
I’d bet the scanners at the TSA checkpoint are just reading your information off the boarding pass and displaying it.
Remember not too long ago you could just show a paper boarding pass that the agent would just read and not scan. It used to be trivial to put any name you’d want on a printed at home boarding pass.
Haolenate, thanks for the detailed explanation on the system requirements and handshakes to run an airline schedule. Prior to my retirement, I worked for an industrial equipment manufacturer in the engineering department and then in sales for the last 15 years. The control systems on this equipment is extremely complex and the handshakes between the individual sensors and systems on these machines number in the hundreds per second. Many of these machines are installed in manufacturing plants and they ‘talk’ to dozens of other machines on a network to enable a finished product to be produced. These systems fail on occasion, which usually shut down an entire manufacturing facility until a technician can be dispatched (usually on a scheduled airline flight) to the site to troubleshoot and repair the problem. My former company had over 100 service technicians, of which 75% are constantly dispatched, including weekends and holidays, to service the equipment and their control systems. On this blog about six months into the pandemic, a contributor suggested all air travel should be halted to ‘stop the spread’, to which I replied if the technicians who repair control and computer systems were not allowed to travel to facilities to keep those systems operational, a large portion of the population of the US would die of starvation within 4 weeks because of diminished food production, packaging and distribution.
My point is that a commercial aircraft itself is one of the most complex machines manufactured. Couple that with all the communication that takes place between the A/C and the ground systems, both navigation, flight control and scheduling, one can begin to understand how a single failure in one of these servers, communication lines, input systems or satellites can disrupt an entire fleet of commercial aircraft from completing their schedule. What we see and think we understand is a tiny fraction of the system control of airline operations.
I think the problem was, Frontier saw Spirits on Monday!
How did the pax on the cancelled flights recover, though?
Great question, given that Cranky’s post on the Frontier/Spirit merger highlighted how few of Frontier’s routes are flown 2x/daily or more.
I’d be curious as to how many pax (or the % of total pax on the cancelled flights) were able to get to their destinations on the same day as scheduled. I may be wrong, but it doesn’t appear that Frontier makes a habit of rebooking pax on other airlines when it cancels a flight, only on “the next available Frontier flight.” For those whose tickets were cancelled for a flight on a route that Frontier flies only a few times a week, it means that they might be stuck waiting a few days for Frontier to get them to their destination.
Kilroy – We’re never going to find that out. I would assume they put people on other airlines when they could. A meltdown is the kind of thing you don’t want to stick to your guns on the reaccomm policy. But the point of this article has nothing to do with that. It’s that a whole lot more people could have been impacted if they weren’t able to get the operation back together so quickly.
As a potential customer, one reason that has kept me from booking the likes of Avelo, Allegiant, Spirit or Frontier has been the risk of flight cancellations with no good recovery option. It may not be the topic of this article, but it would be interesting to know what these ULCCs do in those cases.
(incidentally I have an Avelo flight coming up for a family member that was sufficiently cheaper than the alternatives that it was worth taking the risk. Also, the trip isn’t mission critical and certainly wouldn’t happen if I had had to pay the majors ticket)
Many established carriers have what are called interline agreements, meaning they can book and issue tickets on each other’s flight, and they accept each other’s tickets. The tickets are then cleared and prorated through a clearing house just like bank checks. As part of this process carriers usually give priority to taking and sending customers to/from the other carriers in their Mileage programs and Alliances. These are often global agreements and apply to carriers worldwide.
There have been a few disputes between established carriers on how the tickets are prorated when ticket holders from one carrier are sent to another. This was the fact a few years ago between AA and DL, most probably because AA had a very high number of cancellations and was sending their low value tickets to DL and keeping their high value tickets, while DL almost never cancelled.
They have since worked out a new agreement.
BUT, carriers like Frontier, Spirit, Avelo and some other don’t participate in such programs and thus when they cancel you are at the mercy of whether or not they have a seat on another flight for you, or you have to wait for a flight, or you can BUY a brand new ticket on another airlines, or you can refund your ticket and cancel your trip.
Even some newer middle level carriers like Jetblue do not have agreements with many carriers, but that is changing a little as Jetblue starts its Northeast partnership with AA. Southwest also has very limited to no options to reroute customers on other airlines with much of it driven by lack of agreements and technology that can handle to rebooking/transfer process.
The only argument I’d make here is that it’s often more on the legacy carriers than on the ULCCs for not participating in interline agreements.
It’s the legacies that don’t want to help out the little guys. That’s not the case with Southwest which has pursued that as a strategy, but for others, it’s not for lack of wanting.
Disagree.1/ The ULCC’s don’t want the added cost of participating in interline clearing house operations which include currency conversion and other complex issues they avoid.2/ The ULCC’s lack the automation to transfer e-tickets to other carriers, nor due they want to spend to get such automation.3/ Southwest has always spurned participating in clearing house operations for cost reasons and any kind of complexity for cost reasons, thus their automation has always been rudimentary. However they are now finding as the grow international operations they MUST spend greater and greater amounts on IT to compete. ( check the cap spending ).4/ The ULCC’s do not want to get involved in any pro rate schedule for involuntary rerouted tickets to legacy carriers, or to any carrier, because they want the tickets accepted with face value versus an agreed upon pro rate. For example, if AA cancels and reroutes a customer with a $150 fare for LGA-CLT-ATL to a DL non stop LGA-ATL with a pro rate of $175 because it is a non-stop, AA pays the difference in the clearing house FOR the customer. These differences can be enormous for multi class markets. The ULCCs ( and Southwest ) want nothing to do with these added cost which could be especially crushing for Spirit and Frontier with their very high cancellation rates. It is all about net revenues.Sent from Samsung tablet
Oliver I was booked on a Frontier flight in July from LGA to ATL. The flight cancelled, and they sent an email stating that I could spend $300 on another carrier and request a refund. They also sent a $50 voucher that I’ve never used. Ended up on AA with a stop in CLT.
Understood and agreed.
However, I’m with @Oliver below, in that the sub-daily flights and lack of a consumer-friendly reaccomm policy has often driven me to pay significant premiums ($100/RT or more) to avoid ULCCs, even when I consider myself pretty price sensitive.
When friends had their Allegiant flight cancelled over the holidays, they were offered a refund of that segment + $500/person cash payments (enough to book one-way, last minute lights on Southwest, and to cover an Uber home and gas/mileage in their 2nd car to drive pick up the car they had parked at airport their first Allegiant segment departed from, as their WN flight flew into another airport). However, that’s not the norm and I was very surprised by how generous Allegiant was in going above and beyond its usual policy in that situation.
By the same token, I’ve had other friends (especially in situations where the number of cancellations isn’t enough to make the news, such as a plane stuck in an outstation with a mechanical malfunction) who were told that they could get a seat on the next scheduled Allegiant flight (3 days in the future) or a refund, and who basically had to cancel their trip as a result.
I guess what I’m getting at is that there’s a risk a pax takes when they book a flight on a sub-daily route with an airline (such as many ULCCs) that has general policy of not paying to put pax from cancelled flights on other airlines’ flights. For some people (and in some situations; I’ve taken that risk myself on occasion when the price was right) it’s worth taking the risk, and in other situations it’s not, but those who are not experienced travelers or avgeeks may not be very aware of that risk.
Haaaaaaa…
Do other airlines name their individual planes, like “Manteo the Red Wolf” and “Hops the Rabbit” that you mentioned?
Miles – It’s kind of an old school practice. Airlines used to do it more frequently, but you’ll still find the practice in some places.
Domestically, JetBlue and Hawaiian both still do it. JetBlue always has something include “blue” while Hawaiian uses ??lelo names, in the Hawaiian language.