The annual Airline Quality Rating (aka DOT Data for Dummies) is out, and once again, it proves itself to be nearly useless. This time, however, it’s also angry. I’ve written about how I hate surveys and rankings before, but the fact that this one continues to get a great deal of press means that clearly news editors don’t understand how silly this whole thing is. And the finger-pointing in the release from the study authors that heaps all the blame solely on airlines 
is certainly only making things worse.
The Airline Quality Rating (AQR) is quite literally a rebranding of the monthly DOT Air Travel Consumer Report. But the DOT report, however flawed, is more useful because it doesn’t try to boil down actual statistics into a meaningless ranking. Instead, you have to actually read the data and understand it. The AQR, however, takes the DOT data and assigns a weight to each of the four big metrics (on time performance, mishandled bags, denied boardings, and consumer complaints). The result… voila! A number that means absolutely nothing about quality or really anything else.
I’ve written before about my specific issues with this and how it isn’t an accurate measure of anything, but today I want to focus on the release being put out about the data this year. This dangerously bad information has enabled lazy members of the media to put out articles like “Misery in the Air: Flyer Frustration Climbs.” Good work NBC News. Keep setting the bar lower.
The title of the release “Airline performance for 2014 declines, according to Airline Quality Rating; Virgin America still No. 1” is harmless enough, but it goes downhill quickly.
The airline industry has performed well in recent years, so a decline is a red flag for consumers, according to the researchers.
Dean Headley, associate professor of marketing at the W. Frank Barton School of Business at Wichita State University, says the weaker overall performance shows that the recent round of mergers means airlines still have work to do to compete for customer loyalty.
“Bigger isn’t always better, and the downturn in performance suggests that customer perceptions of poor outcomes are warranted,” said Headley.
Study co-researcher Brent Bowen, dean of College of Aviation at Embry-Riddle Aeronautical University’s Prescott, Ariz., campus, agreed.
“Airline mergers and consolidations are taking a systemic toll that is bad for consumers,” said Bowen. “Performance by the airlines is slipping while they claimed this would make them better.”
Forget for a moment that this last quote is actually countered by the study authors themselves in the very next paragraph, and instead let’s review the thesis. This is effectively claiming that the reason for the year-over-year decline is due to mergers. That’s quite the oversimplification to say the least, and it would be downright comical if people didn’t actually just take it at face value and believe it.
Take a look at page 17 of the report which breaks down the rating by month.
You’ll notice some minor changes in April through December, but the real change is in January through March with January being especially ugly year-over-year. Anyone remember January 2014? That was one of the more miserable months in memory in regards to weather. Remember the polar vortex? It was a complete mess. That, of course, has nothing to do with airline mergers. It does mean that people who flew in the eastern half of the country last winter certainly had it worse than they’ve had before. But that certainly has nothing to do with airline mergers.
Of course, that’s not the sole reason things have gone down this year, but it’s a huge chunk. There are plenty of other things going on that had nothing to do with mergers which would have impacted this:
- Southwest overscheduled its airplanes and that hurt 2014 more than 2013. In fact, the last quarter of 2014, after the scheduling problem had been fixed, showed year-over-year improvement in the AQR for Southwest (page 37) despite the full year results being negative for the airline.
- American Eagle carrier Envoy had a terrible year, but the good news for travelers is that American is shrinking the airline dramatically (carried 18 percent fewer passengers Feb 2015 vs Feb 2014). Envoy is also being removed by American from more challenging airports. But the AQR doesn’t look at total number of people impacted, which seems like a huge issue.
- May was particularly bad for ExpressJet (page 27), and why? Newark had a runway closed that month (and the month before) and its huge operation suffered there dramatically.
- Frontier actually saw on time performance and baggage handling improve, but complaints rocketed up. That’s likely due to the airline’s shift to a different model, and people aren’t expecting what they’re getting. This year will probably be worse as the model change continues (and some ugly reservation system migration/outsourcing issues show their face).
Does this mean that mergers had nothing to do with any of this? I can’t go that far, of course. American and US Airways are in the thick of their merger now, and it wouldn’t be a surprise to see a temporary decline until they can get the operation fixed up. That’s fairly common, as noted by the authors themselves when talking about a previous merger.
“Delta is an excellent example of a merger that declined in performance and systematically has clawed its way back to a new high level of quality performance,” Bowen said. “This shows that if an airline commits to improving their AQR rating, they can do it.”
Of course, then there’s United. Ah United. That to me is less of a merger problem and more of a management issue. The point that mergers can result in a something better has been shown… but for United, well, it still hasn’t quite worked out as planned. That doesn’t mean that mergers = performance decline, as the study might have you believe.
Then, the release really gets weird…
“The challenge is whether airline performance quality improvements at this level can be maintained as more people choose to fly. Does the infrastructure and air traffic control technology limit what the airlines can actually do?”
Bowen says the answer to that question and others like it will be sought by researchers and the flying public in the years to come.
“With AQR performance factors in a state of overall decline, it is easy to understand why passengers and many of the airline employees they encounter are not happy and are significantly frustrated,” Bowen said. “With the high profits being realized by airlines, it is evident they are not investing in customer service and restoring employee concessions given up during the economic decline.
Whoa, say what? The authors basically give a complete and total pass to any infrastructure and air traffic control issues, saying those will be studied in years to come. They’ve pinned everything on the airlines with statements that actually can’t be proven. Airlines “are not investing in customer service”? Yes they are. Airlines aren’t “restoring employee concessions”? Have they see the big pay increases that employees have received the last few years? The longer this release goes on, the more it looks like some kind of hatchet job.
And sticking with the George Washington theme, here’s the cherry on top…. (Get it, hatchet? cherry? Ah, never mind.)
“The airlines have a duty to maximize service to the American public given the trust we provided them through a very low regulatory environment.”
Now, this IS comical. As I think we’ve all seen clearly from the current fight with the Middle East carriers, the US carriers are hardly in a “very low” regulatory environment.
This whole thing is perplexing. The reality is that ratings like this are useless because they use broad brush strokes. A review of the underlying data can at least provide guidance on which airlines are doing well in which places for which reasons, but an overall number like this does nothing. And then to try to heap all the blame on to airlines themselves is downright strange. You’d think a supposedly-academic study like this would present the data and not make wild accusations. You’d think.
47 comments on “The 2015 Airline Quality Rating is Out, and It’s Angrier and Dumber Than Ever”
Get yourself a copy of How to Lie with Statistics by Darrell Huff and Irving Geis if you’d like a little education on how to prove anything you care to with whatever data you have on hand.
While reading the article, and before I got to the comment section the thing I remembered from my Stat Class in College (42 Years ago!!!!) was “Figures don’t lie, but liars figure”. And we needed a very flawed study to tell us the flying experience is not fun.
CF – I think you have some valid points, such as lazy journalism, but in my opinion you also come accross as being overly defensive of the airline industry.
that’s always the case, of course. this is more of a pro-industry site than a pro-consumer site
Cranky:
Some of your observations are valid. However, your neutrality is suspect here as you seem to always give airlines a great amount of benefit of the doubt.
William Johnson – Neutrality? I hope you’re not reading this site for neutrality. This is an opinion site. I have strong opinions and I put them out there three times a week. These opinions have been formed from my years of working for airlines, so of course my opinions are going to vary widely from what the general public might think. That’s the whole point of the site in the first place.
How does shrinking Envoy help overall on time performance? For the most part, these flights aren’t going away, they are just being handed over to another operator. When the weather gets ugly and they can only get a limited number of flights in the air, will AA suddenly decide to delay/cancel mainline operations rather than regional? What am I missing?
Perhaps the thought is that since Envoy statistically runs late, decreasing the size and giving the flying to a more reliable carrier would increase OT performance. Envoy doesn’t run late just because of weather, like you said would impact any airline, but Envoy has a lot of maintenance delays that could be avoided.
Million Miler – I think there are a few things here. First, Envoy just hasn’t been running well on its own. It’s had a turbulent couple of years with labor and it is going through a big transformation losing most of its small jets. There are other operators who will run better.
But more importantly, American is reducing complexity at Envoy and that should help. Pulling Envoy out of Miami and New York and having it focus on other hubs should make it easier to run a smooth operation. Crews will be less scattered and spares will be easier to bring into the operation (in theory). Of course, not all of this is Envoy’s fault. As you say, American decides when to cancel things when the airport capacity goes down. None of that is going to change.
What I’d really like to see is someone fix the baggage handling problem at Envoy. I have no doubt that’s on American’s list of things to tackle. But as we all know, it could get worse before it gets better, as has been the case in every merger.
The problem is the way statistics are reported. Envoys baggage performance is artificially high because the ‘last carrier’ is responsible for the baggage. Example…LAX-DFW-DSM, AA Mainline LAX-DFW, Envoy DFW-DSM – Envoy is responsible for the baggage and will get hit for the baggage score even if the bag is not loaded in LAX. The mainline carrier should be responsible in these cases.
While Envoy (and former American Eagle Airlines) had some operational issues, on time performance of ALL contract carriers are based on the mainline carrier. When I was at AA, it was common to hold flights fat the hubs for connecting passengers.
When I see names like ExpressJet, Envoy, TransStates, Republic, etc, I take the statistics with a grain of salt. I also compare those stats with the mainline. It is also no secret that mainline carriers would rather ‘ding’ contract carriers with the negative statistics to make themselves look better!
I appreciate the comments. But I tend to agree with Former AA. My observation over the years is that this is mostly a conscious decision by AA. (Or maybe lack of decision?)
The labor issue at Envoy seems to be improving (at least on Pilot side) – AA could have done that sooner. Not sure what kind of point they were trying to make. And why they had to try and make it so publicly.
Side note now that they are hiring mainline pilots again, you might think the Envoy flow thru agreement was a huge benefit to mainline – like baseball having minor leaguers ready to be called up on a moments notice (not to imply that Envoy crews are less skilled or less able)
The baggage issue – living in a hub city, I don’t have bags connecting so while I always check a bag, my observation is that bags from AA and Eagle arrive about the same time (except at DFW Terminal B which seems to chronically under served in terms bag claim proximity to gates). It’s been years since I had a bag go missing.
And I can’t recall a maintenance delay on Eagle in a long time – have had a couple on MD80’s in the past year. And a crew go missing on a 757 last year.
Just one guy’s observation
Million Miller – Just one point on AA being able to fix labor relations with Envoy sooner. No they couldn’t, not without giving up the farm. They repeatedly offered pretty much the same deal and the pilots kept turning it down. Only in the end did they decide to take it when they realized Envoy would likely end up actually going away. They thought they had more leverage than they did.
My comment is via Mark Twain, who wrote:
“There are three kinds of lies; lies, damned lies and statistics.”
This incoherent rambling is worse than the AQI.
At least the AQI has value as it helps making sense of the reams and reams of excellent raw DOT data by weighing it,, and therefore making it actionable. This post does nothing to help.
Shrill boy for a justifiability hated industry.
If you think Cranky is a “shrill boy for a justifiably hated industry” why are you reading this blog?
I agree with IO and William: Your articles are great but anytime there are negative comments about the airline industry, you bash them. There are operational difficulties every year, but those should also balance out. The top airlines have been doing a great job and have become more respectable (Delta, Virgin) while others have declined (AA, Envoy), which coincides with the report. The general populous may not be able to navigate the DOT stats website so this report might be an aid.
Even if this report produced results that were representative of the industry, the methodology is highly flawed (a broken clock is right 2 times a day) and the AQR needs to stop. The DOT stats are bad enough (marketing/operating carrier, non-validated DOT complaints, MBR using pax counts instead of bag counts, etc.), then Bowen and Headley make it worse.
I’m all for creating a valid methodology and seeing how the “marketing” airlines rank, but what they are doing is just irresponsible (at best). How did these jokers become the talking heads to the press on the state of the industry?
My largest issue is that these reports only paint half of the picture. By breaking out the regionals from the mainline operations most mainline numbers are significantly higher than they should be. I would love to see the DOT require, or at least base the reports, on all flights that are operated under a certain airlines auspices (not including code shares). That way would get a much better picture of how an operation is truly doing.
On a side note, I would also love to see statistics included for all 121 carriers with scheduled service. Not just ones that meet certain thresholds.
Mike – Absolutely. And actually that’s being proposed by DOT. We’re just waiting for it to be finalized (hoping nobody gets that requirement killed).
I also agree that this report is flawed, but Cranky’s post comes across as bending over backward to defend an industry that has alienated customers in recent years. No matter how defective the report, it comes across as believable because the air travel experience sucks unless you are a high roller.
“…it comes across as believable because the air travel experience sucks unless you are a high roller.”
Well, the high rollers I know use Netjets or another fractional-share service.
As for us proles…I have such low expectations when I fly ( even as an Elite on AS ), that I happy that we don’t have a gate agent sleeping in the cargo hold, oh wait.
“…that I (am) happy that we don’t have a gate agent sleeping in the cargo hold, oh wait.”
Aren’t Alaska’s ramp agents at SEA all courtesy of their wonderful subcontractor Menzies Aviation?
Not sure about “all,” but the one caught sleeping was.
Very well said, Cranky!
When I see “AQR,” I sing “Oh Joy.” There it is!
It’s like looking at my MileagePlus earnings for my last UA trip. There’s a PQS, a PQD, and a PQM. I think they added, subtracted, multiplied, and divided, (not sure, but I think there’s a square root in there, too). And there it is, my “Award Miles” for the trip. (Why haven’t they abbreviated award miles to something we can understand, like “AM?”)
I’m sure we can get the AQR down to something numeric, like a FICO score, updated daily. Ties can be broken by using each airline’s ROW.
No argument that the AQR is flawed and don’t get me started on lazy journalism but I do feel there is a need for a compiled DOT performance report. Consumers are being asked to fork out a pretty heft sum for a flight on most occasions. There needs to be good reporting on who is doing well and who not so much. Just my 2 cents.
A – Agreed. The DOT report should be more useful than it is but it’s still valuable to have. The good news is that some real changes are in progress to it and that should make it much better if it goes through. I’m hopeful.
I have to agree with cranky. He could have been less defensive of the airlines though, but he does have a point. Its an obvious political ploy to target airline customer service. This is nothing new though false information based off statistics is used everyday. Regionals play a big part in the frontline with their mainline partners with regards to customer service. Regionals do have poor customer service currently. Its due to their ever increasing restrictive contracts thus effectively decreasing regional employee pay and increasing budget cuts. Wherever their is less money you can expect customer service to be the first thing out the door!
Our airlines have, time and again, failed to provide even an acceptable level of service to the public. Yes, I know they (and you) will blame it on the weather, infrastructure, or whatever, but the airlines need to take responsibility for their services. For example, they should not have a full schedule with no wiggle room at Newark or Chicago in the middle of the winter. If they do, they have no one else to blame when the weather wreaks havoc on their operations. At that point, they will blame the “weather” rather than themselves, but they were the ones that decided to try and make a bit of extra money by scheduling that way. Why is this possible? It’s due to mergers. A decade ago, airlines that consistently provided bad service would lose passengers. Today, there are few other choices, so airlines don’t have to worry about losing passengers so much. They can continue to provide bad service and then blame it on the “weather” or whatever else they want.
Brett, I enjoy reading your blog, but this type of article makes me think of you as just another industry apologist.
But who should be forced to cut? Should it be UA since they have the most flights in EWR and ORD? Or should Delta just be forced out since they really don’t have that many customers to serve?
Saying “don’t schedule a full set of flights” is nice in theory, but when 90% of the days in the month can run that schedule without any huge problems, getting someone to give up their part of the pie is tough, not to mention against antitrust law.
Jim – Let me guess. You’re someone who bemoans high fares? People complain about airlines overscheduling at congested airports but they aren’t doing it for fun. They do it because there’s demand. So you can run a schedule that works just fine most of the time or you can run a schedule that will work 100% of the time, slash flights, and jack up fares to match demand. Which is better? And further, which is a better use of a constrained resource? If you schedule for the worst case scenario, then you’ll probably run half the flights you have today. That seems like a terrible waste.
CF – admittedly, i’ll pick/comment/opine on your comments and not Jim’s.
– “People complain about airlines overscheduling at congested airports but they aren’t doing it for fun”—- Maybe not for fun all the time, though in a dysfunctional way they may feel it is fun to overschedule with ulterior motives (ex: political, revenge, turf defense etc.). In the past, the gov’t came to their rescue (grants & BK washateria) when extreme events occur (9/11) that exposed them.
In my opinion, your comments are based on a rational (head) and sensible (heart) industry which I don’t believe the airline industry is. I still think it is dysfunctional. The AQR may be a poor attempt to share the information, but the patient still sick.
IO
Do you have any idea how a business works? Businesses are solely set up to make a profit from providing a good/ service (unless it’s a state-owned company). They don’t have the time or money to just finance these “ulterior” motives, this isn’t politics it’s business. No one’s gonna be an asshole if it’s gonna waste money.
Don’t even start on subsidies, Chapter 11 is not a monetary subsidy and if you consider it a subsidy then we need an official definition of a subsidy. The government’s do fuck all to support airlines, if they supported them we wouldn’t have any bad service to complain about. The industry was dealt a blow ith 9/11 and you may have moved on from that event years ago but financially, airlines are only getting out of it now. If it’s so dysfunctional then why can you safely board a flight and make it to your destination in one piece, eh? Ever take safety for granted in your dysfunctional little world? Yes, as many do
LG71 – I agree with you that we all safely board the planes. I’ll admit I don’t have an idea how a business works as I don’t own one, but if i were running one it would not be how I see them run (except until recently southwest).
Let the facts speak for themselves.
Cranky, thanks for replying. Of course everyone bemoans high fares, but I think it’s better to have some degree of reliability than absolute rock-bottom fares. If I had to pay an extra $8-10 per ticket to ensure that I would get decent treatment from airlines in the case of irrops, I would gladly pay it.
I would also point out that in Europe, which has much stronger consumer protections than the US, fares are not necessarily any higher, based on distance traveled.
Jim – Really this should be on the airport operator or government to make this decision, I think. And that’s what we have in places like JFK and Newark where there are slot controls. It’s only in the rarest of occasions when airlines are going to willingly decrease their flying below what’s optimal for the business. American and United jointly agreeing to do it at O’Hare a few years ago temporarily comes to mind. But the question always becomes… where do you set the bar? If you decide to only allow enough operations as would be allowed during a massive blizzard, then you aren’t going to operate many at all. If you look back to the polar vortex, that was some seriously extreme stuff. It means the rest of the year, that asset is is horribly underutilized.
Of course, we could just solve this by being more proactive about building the capacity we need. Sadly that doesn’t appear to be a real option.
So I just read through all the comments, and while I didn’t get “defensive” when I read the blog entry nine hours ago, I was left with the question: Well, How would you do it CF?
You’ve said you’re in favor of flights being rated by marketing carrier, but what else? There will always be yearly seasonal fluctuations, so how should those be handled?
One more question.. How is the DOT discussing handling marketing carrier given that on an airplane operated by Alaska Airlines some customers will have had that marketed to them as American Airlines and some will have had that marketed to them as Alaska? (And I don’t know who else is in there… is DL still putting there code on some AS flights?)
Nick – I’m not in favor of these roll up ratings at all. But the DOT report itself can be useful because it’s not trying to distill it down into a bite size chunk that says nothing. In the DOT report, you can see on time performance by airport. You can compare various quarters. There’s definitely value in that. The issue with the DOT stuff can be fixed, and DOT is working on it.
*Marketing carriers is incredibly important, but we’re not talking about codeshares here. We’re talking about regional carriers. The idea is to report it based on the name on the side of the airplane.
*The number of carriers needs to be expanded to include all (or all but the littlest guys flying in the Alaskan bush) so that you can really get a full idea of what’s happening.
*Baggage numbers need to be changed so that they’re reported based on the total number of bags checked, not on the number of passengers flown.
But even when these issues are fixed, they shouldn’t be rolled up into some silly quality score, because that doesn’t help anyone.
*Baggage numbers need to be changed so that they’re reported based on the total number of bags checked, not on the number of passengers flown.
Ah, that makes sense CF. You’ll have a higher number of checked bags per passenger on an ULCC than on a Legacy Carrier. I see how that could skew the numbers.
CF, you know I usually agree with you here. But I think there should be some rolled up score. You have the ability to dig into the other corners of the industry, whereas many people don’t.
I think what they should be doing is a severe weather adjusted ontime measurement. Do something like drop the ten worst days, or perhaps a weighting thing of some variety.
Theres value in a rolled up score, but the folks who are rolling it up shouldn’t be so lazy.
Nick – I just don’t think the rolled up score helps. People will make decisions based on something that is really fairly arbitrary. The best thing is for people to be able to see how the airline performed out of the location in which they’re flying. Route-specific stuff is one thing, but it’s also a pretty small set of data. But in mid to large cities, if you look at the airline’s performance in that city, it would be better.
So perhaps a route specific score of some variety?
Nick – Well, it’s better than nothing. But really even that isn’t great.
I mean, the AQR-style roundup might be more helpful for someone who checks bags. If you don’t, then what do you care of they lose them?
Just because cf points out the stupidity of one party, does not mean he is defending the other. I read the article as pointing out the ways the study and the critique are flawed. For example, pointing out the flaws in a study that tries to prove the death penalty is not a deterrent dies not make one a death penalty foe. The point cf makes is that the media and the aqr take a data set and warp it much like a drunk uses a lamp post- for support rather than illumination. To those that read cf’s editorial as a defense of the airlines, I think that you are the target audience for these media releases. Poor attention and reading comprehension is the reason the media is so careless. Since the target audience does not exert the cerebral energy to think critically, there is no reason to upgrade the product beyond abject nonsense and stupidity. Go back to reading the onion if you can’t handle a critical piece.
I’m not sure what the problem is here…. Yes, they ARE dumbed down rankings, but that’s the point. The target audience of the AQI is the general public – people who don’t know much (or anything) about the inner-workings of airlines. These people aren’t going to delve into DOT reports and crunch numbers to figure out which airline they should fly next. They want something high-level and succinct.
It has always been amazing to me that some community college in Wichita Kansas could become so influential to the media. This is a perfect example of their desperation to get anything negative they can get their hands on out to the public. I simply do not understand this obsession.
Great post and good information!
Keep up the good work!