It seems like we see new airline surveys and lists every other week. Which airline is best? Which has the highest quality? Which treats their customers best? 
The media may love to talk about these things, but you know what? They really aren’t worth much at all in my eyes.
Let’s take a look at the recently released Zagat Survey (.pdf) to illustrate my point. We’ll just focus on the domestic airline results for now, since that’s what’s been getting the most press. First, what do we know about the people who took the survey? Not much. From the survey itself, we know they talked to 7,498 frequent fliers and travel professionals (how they define a frequent flier or travel professional, I don’t know). Of those, 40% are women and 60% men. Only 8% are in their 20s, 22% in their 30s, 23% in their 40s, 26% in their 50s, and 21% age 60 or higher.
That’s all we know. I was happy when the company’s PR firm contacted me trying to sell the story, because I figured I could get some more info on how they actually came to these conclusions. Sadly they refused to provide any info above and beyond what’s already published in the survey. Without this information, I find the survey to be mostly unhelpful. Here’s why.
- Let’s look at the top of the list, since that’s where most of the attention is focused. In the Premium Cabin, Virgin America was #1 and they were tied for #2 in Coach. Whoa, that immediately throws up a red flag. The airline started in August and only recently added its fifth city. How many of those 7,498 could have even had the opportunity to fly Virgin America? We’ll never know since Zagat won’t say, but it can’t be that many. And if it is that many, then this is hardly a random sample. Hmmm.
- In Coach, Midwest Airlines was the big winner. These guys have historically finished at the top of the pack in most surveys, but the airline really has multiple personalities. There’s the traditional (called “Signature”) Midwest experience which has wide leather seats (four across instead of the usual five). That is what generally gets rave reviews. But then there is the ever-growing “Saver” service that the airline provides which has the usual five across seating. It’s now going to be the largest cabin on all planes with Signature service being only several rows at the front of the plane. So in a way, the traditional Midwest model should be considered Premium now. How many of those taking the survey were in Saver and how many were in the original Signature seat? It’s a very different product that should be rated differently, but it isn’t.
- And how about Frontier? They get high marks for comfort, but their comfort is changing. The airline will be adding rows of seats to its planes, trimming the seat pitch and probably ending up with a less comfortable product. Even though this was announced long ago, it probably isn’t reflected in these rankings. How many people experienced the new tighter pitch? We don’t know, but it doesn’t really matter. Comfort will be going down.
- Then there’s the “food” rating. What exactly do people use to rate their impression of food? Midwest got highest marks but you have to pay for it on them. Continental came in a distant fourth, but it’s included for free when you fly them. Some travelers may find that the cost is more important than the actual taste, but a survey won’t give you that information.
- Let’s also look at the opposite end of the spectrum. US Airways didn’t fare well at all in this survey, and that’s to be expected. They’ve had some well-publicized operational issues this year, but again we’re talking about a tale of two airlines here. The old US Airways passengers (primarily east coast) were used to a higher level of service that the new US Airways is not trying to maintain. Those people are unhappy with a lot of the changes that they’re seeing so they’re probably going to rank the airline much worse. Out in the west, I would bet that the US Airways passengers are happier (or less unhappy) because they’re used to the America West standard of service which is being rolled out throughout the operation. They’re probably getting closer to what they’d expect so ratings should be higher. Of course, we can’t get geographic breakdowns from Zagat so I can’t prove this at all.
So, as you can tell, airline ratings boil something down to a nice media snippet, but they don’t tell enough of the story to be very helpful. While the media will continue to eat up surveys and lists like this, we should all know better than to trust a single one of them. Every person has very different views on what makes a good airline, and only more in-depth personal research can determine what is right for you.
16 comments on “Why I Hate “Best Airline” Surveys and Lists”
Re: Frontier. I fly them a few times a year being my “hometown” airline.
They’ve been loved and appreciated for years for bringing competition to Denver – and have an excellent fleet with Direct TV also. I personally like them because their gates are the closest to the terminal (DEN) and the only gates you can walk to without needing the tram ride.
But reducing pitch is the worst thing you can do. I don’t care about snacks, TV, filty pillows, or entertainment. But taking away comfort overrides all. We’ll see how many “goodie” points that the community has lavished on them will be retracted once people realize that like you said, the product of comfort is reduced.
Great explanation of how to do constructive analysis. Very few people understand that if you don’t have access to the data, a smart statistician can make the numbers say just about whatever they want. Also, a sample size of 7,500-ish people might pass the grade for confidence level, but it also doesn’t get a true-cross sample size with something as biased as airlines.
To make this analysis worthwhile, you would need to know that the people performing the survey have flown a minimum number of airlines in the past 12 months (maybe 5). However, if you live in Atlanta, you will probably be flying Delta or Airtran the majority of the time. That is the major reason for the loyalty programs. So people will rate airlines highest, even though they haven’t been on many of the others. That is just one of probably 20 things wrong with the survey. While the survey gives “answers”, they aren’t comprehensive, and don’t mean a lot to most people.
These lists are totally subjective in nature.. That is why I don’t like them..
Hey “Cranky”,
You say, “How many of those 7,498 could have even had the opportunity to fly Virgin America?”
I suspect they’ve flown several hundred thousand passengers by now – it’s been nearly 5 months!
Yes, theoretically, they could have flown all of those passengers. It has been just shy of 4 months since they launched flights on August 8. If they flew their full complement of 44 flights every day since that time, they would have flown ~750,000 seats or ~565,000 passengers if they’re running 75% full. But we know they’ve had a slow ramp up, so it’s considerably less than that.
I’m getting into the weeds here. The point is that while yes, they could have flown all those passengers technically, it’s highly unlikely that a large percentage of people would have had the opportunity since they only fly to 5 cities. They’ve had enough seats but they don’t serve many routes.
Good analysis. Speculating a bit here, it sounds like these folks answered the questions based on their perceptions and not necessarily on experiences. Hear good things about Virgin America? OK, rate that one highly. Hear bad things about US Airways? OK, rate that one low.
Basically, this survey ain’t worth a damn.
Wow, that’s a great point I didn’t even think about, DR Gooch. It doesn’t say anywhere that the ratings had to come from people who flew the airline at all. Oh man, I hope that’s not the case.
CF and Dr. G,
I doubt very much that that could be the case – the suggestion you make that perhaps the Zagat raters didn’t actually fly, but filled in the survey based on reputation, rumor and innuendo…. Zagat’s has built a nice business and it would be beyond idiotic to risk it in the way you suggest.
And CF, I disagree that, “it’s highly unlikely that a large percentage of people would have had the opportunity since they only fly to 5 cities,” because NY, SF, LA, DC & Las Vegas serve a huge number of passengers. In nearly 5 full months, it would be easy in those markets to find 7,000 Zagat people – business people who fly those routes regularly and expect more.
I don’t know that it would be as easy as you think. Take Vegas, for example. They only fly to SFO and it didn’t start until last month. Even in their largest city, SFO, they only fly to LA, NY, and DC. Many of the business travelers are going to keep flying United on those routes since they’re slaves to Mileage Plus.
But the bigger question is how they go about finding these people. Do they actively solicit? If so, what’s to say that process isn’t inherently biased? Who are these people and are they truly a representative sample? We just don’t know and Zagat won’t tell us.
But I honestly don’t mean to single out Zagat here. Their survey just happened to be the most recent. Data manipulation is a problem in any survey and for something as complex as an airline, I don’t think they’re very helpful at all.
The problem with these kind of surveys is that very, very few people have wide enough experience to properly compare. This is because not only are there lots of airlines but there is lots of different product differentiation also (eg domestic transcon first is quite different to domestic shorthaul first and different again to international first and international premium first – even on a single airline). Of course, there is also the human aspect – service can vary widely just because you got a good crew, or not.
I travel more widely than most people (hundreds of flights annually on more than 20 airlines in all classes) but even so cannot rank airlines sufficiently well, in my opinion, just on my own experience. As a well informed traveller I can and do use other people’s experiences and information to gauge.
But the vast majority of those who complete these kind of surveys/awards do not, in my view, have enough experience or reliable information to make an informed opinion.
I don’t know, GT – there are a lot more road warriors out there than you think. But I don’t think Zagat’s intends to be a methodologically sound survey; rather it is designed more along the lines of asking the opinion of a friend.
When you get off a flight, whether you take one or 400 per year, you definitely have a sense of: was the plane clean/dirty; were the food/drink offerings (and how they were made) to your liking; were the crew members on-board and the gate area staff rude/polite/friendly/helpful; if there was a glitch, was the staff accommodating/did they go the extra mile/were they indifferent; etc. etc.
Each and every flight is an airline serving its customers, and competition encourages the carriers to do better for their customers. Maybe someone flying to Wisconsin doesn’t know that they bake cookies on Midwest! Little things mean a lot in the airline business, and it’s those little things that each and every traveler has a right to an opinion about.
My flight with NWA was a memorable one. The flight ended with a wait of 70 minutes to see our luggage roll down to the baggage carousel. A little too long wait for me and my children at 10:30pm No other flights noted on the floor too.