Why I Hate “Best Airline” Surveys and Lists

It seems like we see new airline surveys and lists every other week. Which airline is best? Which has the highest quality? Which treats their customers best? 07_11_26 grouchsurveyThe media may love to talk about these things, but you know what? They really aren’t worth much at all in my eyes.

Let’s take a look at the recently released Zagat Survey (.pdf) to illustrate my point. We’ll just focus on the domestic airline results for now, since that’s what’s been getting the most press. First, what do we know about the people who took the survey? Not much. From the survey itself, we know they talked to 7,498 frequent fliers and travel professionals (how they define a frequent flier or travel professional, I don’t know). Of those, 40% are women and 60% men. Only 8% are in their 20s, 22% in their 30s, 23% in their 40s, 26% in their 50s, and 21% age 60 or higher.

That’s all we know. I was happy when the company’s PR firm contacted me trying to sell the story, because I figured I could get some more info on how they actually came to these conclusions. Sadly they refused to provide any info above and beyond what’s already published in the survey. Without this information, I find the survey to be mostly unhelpful. Here’s why.

  1. Let’s look at the top of the list, since that’s where most of the attention is focused. In the Premium Cabin, Virgin America was #1 and they were tied for #2 in Coach. Whoa, that immediately throws up a red flag. The airline started in August and only recently added its fifth city. How many of those 7,498 could have even had the opportunity to fly Virgin America? We’ll never know since Zagat won’t say, but it can’t be that many. And if it is that many, then this is hardly a random sample. Hmmm.

  2. In Coach, Midwest Airlines was the big winner. These guys have historically finished at the top of the pack in most surveys, but the airline really has multiple personalities. There’s the traditional (called “Signature”) Midwest experience which has wide leather seats (four across instead of the usual five). That is what generally gets rave reviews. But then there is the ever-growing “Saver” service that the airline provides which has the usual five across seating. It’s now going to be the largest cabin on all planes with Signature service being only several rows at the front of the plane. So in a way, the traditional Midwest model should be considered Premium now. How many of those taking the survey were in Saver and how many were in the original Signature seat? It’s a very different product that should be rated differently, but it isn’t.

  3. And how about Frontier? They get high marks for comfort, but their comfort is changing. The airline will be adding rows of seats to its planes, trimming the seat pitch and probably ending up with a less comfortable product. Even though this was announced long ago, it probably isn’t reflected in these rankings. How many people experienced the new tighter pitch? We don’t know, but it doesn’t really matter. Comfort will be going down.

  4. Then there’s the “food” rating. What exactly do people use to rate their impression of food? Midwest got highest marks but you have to pay for it on them. Continental came in a distant fourth, but it’s included for free when you fly them. Some travelers may find that the cost is more important than the actual taste, but a survey won’t give you that information.

  5. Let’s also look at the opposite end of the spectrum. US Airways didn’t fare well at all in this survey, and that’s to be expected. They’ve had some well-publicized operational issues this year, but again we’re talking about a tale of two airlines here. The old US Airways passengers (primarily east coast) were used to a higher level of service that the new US Airways is not trying to maintain. Those people are unhappy with a lot of the changes that they’re seeing so they’re probably going to rank the airline much worse. Out in the west, I would bet that the US Airways passengers are happier (or less unhappy) because they’re used to the America West standard of service which is being rolled out throughout the operation. They’re probably getting closer to what they’d expect so ratings should be higher. Of course, we can’t get geographic breakdowns from Zagat so I can’t prove this at all.

So, as you can tell, airline ratings boil something down to a nice media snippet, but they don’t tell enough of the story to be very helpful. While the media will continue to eat up surveys and lists like this, we should all know better than to trust a single one of them. Every person has very different views on what makes a good airline, and only more in-depth personal research can determine what is right for you.

(Visited 71 times, 1 visits today)

Get Posts via Email When They Go Live or in a Weekly Digest

Leave a Reply

16 Comments on "Why I Hate “Best Airline” Surveys and Lists"

newest oldest most voted
Re: Frontier. I fly them a few times a year being my “hometown” airline. They’ve been loved and appreciated for years for bringing competition to Denver – and have an excellent fleet with Direct TV also. I personally like them because their gates are the closest to the terminal (DEN) and the only gates you can walk to without needing the tram ride. But reducing pitch is the worst thing you can do. I don’t care about snacks, TV, filty pillows, or entertainment. But taking away comfort overrides all. We’ll see how many “goodie” points that the community has lavished… Read more »
Great explanation of how to do constructive analysis. Very few people understand that if you don’t have access to the data, a smart statistician can make the numbers say just about whatever they want. Also, a sample size of 7,500-ish people might pass the grade for confidence level, but it also doesn’t get a true-cross sample size with something as biased as airlines. To make this analysis worthwhile, you would need to know that the people performing the survey have flown a minimum number of airlines in the past 12 months (maybe 5). However, if you live in Atlanta, you… Read more »

These lists are totally subjective in nature.. That is why I don’t like them..


Hey “Cranky”,

You say, “How many of those 7,498 could have even had the opportunity to fly Virgin America?”

I suspect they’ve flown several hundred thousand passengers by now – it’s been nearly 5 months!


Good analysis. Speculating a bit here, it sounds like these folks answered the questions based on their perceptions and not necessarily on experiences. Hear good things about Virgin America? OK, rate that one highly. Hear bad things about US Airways? OK, rate that one low.

Basically, this survey ain’t worth a damn.

CF and Dr. G, I doubt very much that that could be the case – the suggestion you make that perhaps the Zagat raters didn’t actually fly, but filled in the survey based on reputation, rumor and innuendo…. Zagat’s has built a nice business and it would be beyond idiotic to risk it in the way you suggest. And CF, I disagree that, “it’s highly unlikely that a large percentage of people would have had the opportunity since they only fly to 5 cities,” because NY, SF, LA, DC & Las Vegas serve a huge number of passengers. In nearly… Read more »
The Global Traveller
The problem with these kind of surveys is that very, very few people have wide enough experience to properly compare. This is because not only are there lots of airlines but there is lots of different product differentiation also (eg domestic transcon first is quite different to domestic shorthaul first and different again to international first and international premium first – even on a single airline). Of course, there is also the human aspect – service can vary widely just because you got a good crew, or not. I travel more widely than most people (hundreds of flights annually on… Read more »
I don’t know, GT – there are a lot more road warriors out there than you think. But I don’t think Zagat’s intends to be a methodologically sound survey; rather it is designed more along the lines of asking the opinion of a friend. When you get off a flight, whether you take one or 400 per year, you definitely have a sense of: was the plane clean/dirty; were the food/drink offerings (and how they were made) to your liking; were the crew members on-board and the gate area staff rude/polite/friendly/helpful; if there was a glitch, was the staff accommodating/did… Read more »

[…] so maybe I was wrong in condemning all surveys. I think it may just be the surveys that take themselves too seriously. When I saw this […]

William Lundquist

My flight with NWA was a memorable one. The flight ended with a wait of 70 minutes to see our luggage roll down to the baggage carousel. A little too long wait for me and my children at 10:30pm No other flights noted on the floor too.


[…] readers of the blog know that I’m not a fan of all these “Best of” surveys. That’s why I haven’t written about JD Power’s boring surveys or anything else […]


[…] I agree with The Cranky Flier’s recent entry about best airline surveys. […]


[…] out. I’ve generally avoided talking about this and every other survey out there, except for one missive back in 2007. It seems like this might be a good time to revisit and explain why I don’t like these […]