The annual Airline Quality Rating (aka DOT Data for Dummies) is out, and once again, it proves itself to be nearly useless. This time, however, it’s also angry. I’ve written about how I hate surveys and rankings before, but the fact that this one continues to get a great deal of press means that clearly news editors don’t understand how silly this whole thing is. And the finger-pointing in the release from the study authors that heaps all the blame solely on airlines 
is certainly only making things worse.
The Airline Quality Rating (AQR) is quite literally a rebranding of the monthly DOT Air Travel Consumer Report. But the DOT report, however flawed, is more useful because it doesn’t try to boil down actual statistics into a meaningless ranking. Instead, you have to actually read the data and understand it. The AQR, however, takes the DOT data and assigns a weight to each of the four big metrics (on time performance, mishandled bags, denied boardings, and consumer complaints). The result… voila! A number that means absolutely nothing about quality or really anything else.
I’ve written before about my specific issues with this and how it isn’t an accurate measure of anything, but today I want to focus on the release being put out about the data this year. This dangerously bad information has enabled lazy members of the media to put out articles like “Misery in the Air: Flyer Frustration Climbs.” Good work NBC News. Keep setting the bar lower.
The title of the release “Airline performance for 2014 declines, according to Airline Quality Rating; Virgin America still No. 1” is harmless enough, but it goes downhill quickly.
The airline industry has performed well in recent years, so a decline is a red flag for consumers, according to the researchers.
Dean Headley, associate professor of marketing at the W. Frank Barton School of Business at Wichita State University, says the weaker overall performance shows that the recent round of mergers means airlines still have work to do to compete for customer loyalty.
“Bigger isn’t always better, and the downturn in performance suggests that customer perceptions of poor outcomes are warranted,” said Headley.
Study co-researcher Brent Bowen, dean of College of Aviation at Embry-Riddle Aeronautical University’s Prescott, Ariz., campus, agreed.
“Airline mergers and consolidations are taking a systemic toll that is bad for consumers,” said Bowen. “Performance by the airlines is slipping while they claimed this would make them better.”
Forget for a moment that this last quote is actually countered by the study authors themselves in the very next paragraph, and instead let’s review the thesis. This is effectively claiming that the reason for the year-over-year decline is due to mergers. That’s quite the oversimplification to say the least, and it would be downright comical if people didn’t actually just take it at face value and believe it.
Take a look at page 17 of the report which breaks down the rating by month.
You’ll notice some minor changes in April through December, but the real change is in January through March with January being especially ugly year-over-year. Anyone remember January 2014? That was one of the more miserable months in memory in regards to weather. Remember the polar vortex? It was a complete mess. That, of course, has nothing to do with airline mergers. It does mean that people who flew in the eastern half of the country last winter certainly had it worse than they’ve had before. But that certainly has nothing to do with airline mergers.
Of course, that’s not the sole reason things have gone down this year, but it’s a huge chunk. There are plenty of other things going on that had nothing to do with mergers which would have impacted this:
- Southwest overscheduled its airplanes and that hurt 2014 more than 2013. In fact, the last quarter of 2014, after the scheduling problem had been fixed, showed year-over-year improvement in the AQR for Southwest (page 37) despite the full year results being negative for the airline.
- American Eagle carrier Envoy had a terrible year, but the good news for travelers is that American is shrinking the airline dramatically (carried 18 percent fewer passengers Feb 2015 vs Feb 2014). Envoy is also being removed by American from more challenging airports. But the AQR doesn’t look at total number of people impacted, which seems like a huge issue.
- May was particularly bad for ExpressJet (page 27), and why? Newark had a runway closed that month (and the month before) and its huge operation suffered there dramatically.
- Frontier actually saw on time performance and baggage handling improve, but complaints rocketed up. That’s likely due to the airline’s shift to a different model, and people aren’t expecting what they’re getting. This year will probably be worse as the model change continues (and some ugly reservation system migration/outsourcing issues show their face).
Does this mean that mergers had nothing to do with any of this? I can’t go that far, of course. American and US Airways are in the thick of their merger now, and it wouldn’t be a surprise to see a temporary decline until they can get the operation fixed up. That’s fairly common, as noted by the authors themselves when talking about a previous merger.
“Delta is an excellent example of a merger that declined in performance and systematically has clawed its way back to a new high level of quality performance,” Bowen said. “This shows that if an airline commits to improving their AQR rating, they can do it.”
Of course, then there’s United. Ah United. That to me is less of a merger problem and more of a management issue. The point that mergers can result in a something better has been shown… but for United, well, it still hasn’t quite worked out as planned. That doesn’t mean that mergers = performance decline, as the study might have you believe.
Then, the release really gets weird…
“The challenge is whether airline performance quality improvements at this level can be maintained as more people choose to fly. Does the infrastructure and air traffic control technology limit what the airlines can actually do?”
Bowen says the answer to that question and others like it will be sought by researchers and the flying public in the years to come.
“With AQR performance factors in a state of overall decline, it is easy to understand why passengers and many of the airline employees they encounter are not happy and are significantly frustrated,” Bowen said. “With the high profits being realized by airlines, it is evident they are not investing in customer service and restoring employee concessions given up during the economic decline.
Whoa, say what? The authors basically give a complete and total pass to any infrastructure and air traffic control issues, saying those will be studied in years to come. They’ve pinned everything on the airlines with statements that actually can’t be proven. Airlines “are not investing in customer service”? Yes they are. Airlines aren’t “restoring employee concessions”? Have they see the big pay increases that employees have received the last few years? The longer this release goes on, the more it looks like some kind of hatchet job.
And sticking with the George Washington theme, here’s the cherry on top…. (Get it, hatchet? cherry? Ah, never mind.)
“The airlines have a duty to maximize service to the American public given the trust we provided them through a very low regulatory environment.”
Now, this IS comical. As I think we’ve all seen clearly from the current fight with the Middle East carriers, the US carriers are hardly in a “very low” regulatory environment.
This whole thing is perplexing. The reality is that ratings like this are useless because they use broad brush strokes. A review of the underlying data can at least provide guidance on which airlines are doing well in which places for which reasons, but an overall number like this does nothing. And then to try to heap all the blame on to airlines themselves is downright strange. You’d think a supposedly-academic study like this would present the data and not make wild accusations. You’d think.
