There’s been a lot of talk about the Airline Quality Rating (AQR) lately since the 2012 rankings just came out. I’ve generally avoided talking about this and every other survey out there, except for one missive back in 2007. It seems like this might be a good time to revisit and explain why I don’t like these things. Today, I’ll just focus on the AQR.

You’ll hear the AQR referred to as a study or a survey when that’s being pretty generous. What the guys behind this do is take all the information the government releases in the monthly Air Travel Consumer Report (on time performance, mishandled bags, complaints, and denied boardings) and then smash it up into a rating. How do they smash it up? Well, they assign a weight to each metric and then combine them all.

Where did those weights come from? They were “established by surveying 65 airline industry experts regarding their opinion as to what consumers would rate as important.” Awhile later, they surveyed 766 frequent fliers but didn’t find much of a difference so they just kept the weights the same.

So what we have is really just a rehashing of what the government tells us each month. Is that useful? Not particularly. Let’s go over a few of the problems. Keep in mind, most of these complaints are about the underlying data as collected by the government. But since that’s what the AQR uses, no distinction is needed.

1. Operating Airline – The biggest problem of all is that this is based on operating airline and not marketing airline. What I mean by that is this. When you buy a ticket on United, how often are you flying on United Express? There are more United Express flights than mainline United flights, so the chances are very good that you’ll end up on an Express partner, and not just on United. But the “United” we see here is just for mainline.
    
   Can you add up the airlines by learning which ones operate for which big airline? No. Think about SkyWest, an airline which flies for Alaska, Delta, United, and US Airways. There’s no way to break apart what’s related to which marketing airline.
2. The 1% – We should feel lucky that we have the opaque SkyWest data because many of the regionals don’t report at all. Airlines with more than 1 percent of domestic scheduled passenger revenue have to report, and that leaves out many of the little guys. There’s no Comair, no Pinnacle, no Trans States, etc. So it’s a very incomplete picture. Don’t go looking for Spirit or Allegiant either. (Virgin America just got big enough to start reporting.)
3. Bags per People – People often like to look at trends with this report, but that’s hard to do when it doesn’t adjust for business changes. The most glaring is in the baggage department. Mishandled bags are measured by looking at the number of bags mishandled per 1,000 passengers. In other words, it doesn’t matter how many bags are actually checked but rather only depends upon how many people fly. This is hugely problematic as airlines have started to charge for bags over the last few years. People check fewer bags, so you’d expect there to be fewer mishandled bags. But this report wouldn’t catch that because it only looks at the number of lost bags per total passengers.
4. The Few, The Loud – Complaints are another issue here. When people complain, most of them complain to the airlines directly with the hope that they will get something in return. The number of people who get so annoyed with the airlines that they bother complaining to the DOT is miniscule. There were 7,465 complaints recorded by the DOT against the US airlines last year. That may sound like a lot until you realize there were more than 630 million passengers enplaned. That means about .001% of passengers complained. With numbers so small, it’s very hard to get any sort of true trend when comparing year over year.
    
   Delta, for example, had the highest number of complaints of the airlines last year with 1,406. One angry 777 full of passengers could band together and complain to the DOT and that would show as a 20 percent increase if everything else held the same. If you’d like to read more about this, take a look at my post ripping into the Lufthansa catering union for really bad analysis.
5. Too Broad – One other issue here (and with most ranking efforts) is the broadness of the results. Hawaiian shows up as #2 in the survey, and a lot of that is going to be thanks to its stellar on time performance. But as I’ve written before, that on time performance rides on the coattails of the interisland operation while the long haul flying lags significantly.

These are just a few reasons why I don’t like the AQR, but I don’t want to make you think that there’s another survey I like better. There isn’t. I hate all these types of forced rankings because it’s rarely indicative of what you’re going to get when you fly.

Regular readers of the blog know that I’m not a fan of all these “Best of” surveys. That’s why I haven’t written about JD Power’s boring surveys or anything else like that. But, when a survey has an interesting angle, it certainly catches my eye. This includes a recent British survey showing Palmair as one of the top four airlines in the world. Who?

This year’s Which? Awards (out of the UK) features “the best companies and individuals that seek to serve the interests of consumers.” In the airline category, it’s not surprising to see Air New Zealand, Jet Airways, and Singapore Airlines (the eventual winner) make the list of finalists. But that fourth finalist? Palmair. They ended up tying with Air New Zealand for third place.

I must admit that I’ve never heard of these guys, so I thought I’d dig in a little more. The airline seems like an unlikely hero. Based in Bournemouth, on the Southern coast of England, Palmair has a single 737-200 chartered from European Air Charter. The airplane started life with Sabena nearly 35 years ago, but it’s still going strong.

The airline exists to appear solely as a way to funnel pasty white Brits down to warm and sunny places to try to get a little color. You can’t book online, and it’s recommended that you use a travel agent or call them directly. You get a hot meal, but you won’t get your seat assignment until you arrive at the airport. You can only bring 20 kilos of baggage with you as well. So what is so great about these guys?

There aren’t any reviews on SkyTrax to help us out, but there have been plenty of articles popping up since the airline made it on the list. One from the Telegraph shows that flying with a focus on personal service has really set them apart. Just a sampling of what they do well.

• “Singapore Airlines has 100 planes including the A380 which is the largest plane in service, on-board massages and a choice of dvds. All we can offer is the choice of tea or coffee . . .”
• Until 2006, [founder Peter] Bath made sure he was in the departure lounge for each flight to greet the passengers. He then stood on the Tarmac at Bournemouth Airport in Dorset where the airline is based and waved off every single flight.”
• When Mr Bath passed away, long-standing airport representative Teresia Rossello took over the role. She also ensures each customer can have their choice of seat by drawing out a plan in her kitchen every night.
• Stewardesses place fresh flowers on the plane, including the toilets, every day and the company has removed a row of seats to give customers more room.

Sounds nice, right? I’m sure it is, but it’s easy to do with only one airplane. Good luck finding anyone who can replicate this on a large scale. It really makes you wonder how this tiny airline made it on to such a big list. Apparently, there were 70,000 votes by the general public to pick the winner. Sounds like there was a heck of a write-in campaign on behalf of this airline. It seems that every person who has heard of the airline must have felt compelled to go and vote.

So this doesn’t really change my view of surveys. I don’t find this particularly useful at all, since I highly doubt I’ll ever be in Bournemouth, and if I am, I’m unlikely to be flying to a sun spot. This may be an excellent airline, but if it doesn’t fly anywhere you need to be, is it really helpful? In fact, I’d imagine most people would find this news completely worthless, but I imagine all of us would enjoy flying on an airline like this, if it existed in our own backyard.

I just found out that my fiancée and I will be heading down to Peru in the Spring. Of course, I’m excited about the chance to see Machu Picchu, but I’m also excited about getting to try a new airline.

Looking at my options, I see all kinds of exciting possibilities. Though I’ll wait to see what airfares do over the next couple of months, I’m eager to find out which option you all would recommend.

So, forget about mileage opportunities, because that’s not important. In terms of onboard experience, which airline would you choose?

Update 10/26 @ 1124a: An excellent point was raised in the comments that I didn’t mention what class I’d be flying. I’ll be in coach.

 Loading ...