If you decide to put out bad analysis, as UNITE HERE did last week regarding Lufthansa complaints, and someone calls you on it, what would 
you do? The smart answer would probably be to just let it go and stop calling attention to the work, especially since it has more holes than Swiss cheese. But fortunately for us, UNITE HERE has decided to go the opposite route.
The union is using one of oldest tricks in the book: going after my credibility to muddy the water. This is just dumb. They really shouldn’t want to bring more attention to a flawed report like this. Now I’ve just dug in deeper and found even more problems with it. While I was waffling before, now I’m not. UNITE HERE has truly earned the Cranky Jackass Award.
You can read the union’s entire response here (pdf) if you’d like, but I’ll pull out the most fun parts. Let’s start with the opening.
One of the things I appreciate about your site is you are very open about your relationship to companies in the airline industry. And just one month after Lufthansa gave you a free round-trip, business class ride on its A380 from San Francisco to Frankfurt, perhaps I should not be surprised at your dismissive response to my report.
Ah yes, the back-handed compliment. A time-honored tradition that’s used to cover bad work. If someone calls out real issues, just call his or her credibility into question but look completely pleasant while doing so. This takes the focus off your bad analysis and tries to shift the issue. (Sounds like the author may have a future in politics.) It’s true, I’m very open about these things, and I did just fly Lufthansa at the airline’s expense. That doesn’t mean I won’t gladly rip Lufthansa a new one if it’s deserved. The problem here for the union is that it’s not.
You can read the rest of the response yourself if you’re interested in more sugar-coated insults, but let’s focus on the weak defense of the report itself and break that down.
The Department of Transportation data in the report is real, and to my knowledge is the only reliable U.S. source of compiled complaint information on international
airlines. If the DOT is willing to use these numbers to “to determine the extent to which carriers are in compliance with federal aviation consumer protection regulations,” then they’re good enough for me. Even if I am just a research analyst at a union.
*sigh* The issue is not whether this is the only place to get complaint data or not but whether or not it’s statistically valid and can be used to explain a trend or not. In this case, the year-over-year change in complaints from 2009 to 2010 moved by roughly less than one-thousandth of one percent over total passengers carried by Lufthansa to and from the US (using my rough passenger estimate). Even the initial number itself is so tiny that it’s not significantly different from zero. So regardless of what the purpose of the complaint reports are in the eyes of the DOT, that doesn’t magically mean that we can consider each number valid for any kind of analysis.
You’re right, I could have used the raw numbers, but I sort of agree with you that the raw numbers themselves aren’t incredibly exciting on their own. They’re small
because, well, how many people actually go through the effort to submit their airline complaints to the U.S. government? (If you care about an answer, you can look at the DOT analysis for the new passenger rights rule, where the DOT uses the ratio that every 1 complaint submitted to the DOT represents about 61 complaints submitted to the foreign airlines.
Excellent. Let’s just forget about using raw numbers because they aren’t “exciting.” I see. So we’re not looking for statistical validity here. We’re looking for excitement. You can apply any ratio you want to these numbers, but that still doesn’t make the small change valid. And this ratio is just an estimate by the DOT anyway, so using it would make a statistically insignificant change even less valid, if that’s possible.
The result of that comparison was clear. Lufthansa complaints went up, Air France and British Airways complaints went down. Is the sample number of complaints
small? Yes. But if the increases were random, would Lufthansa have seen them in 7 out of 8 top categories from 2009 to 2010? If they were random, wouldn’t Air France and British Airways have seen more fluctuation too?
This is my favorite part. I hadn’t even touched the Air France and British Airways numbers in my initial post, so I should thank the union for giving me even more firepower to show how awful the analysis is. The result is far from “clear” as proposed.
When I spoke with the research analyst, he told me that he didn’t bother looking at the monthly complaint reports. He just looked at the year-end summary and called it a day. That makes the analysis even worse because it doesn’t look for outliers. And that’s exactly why BA’s numbers are so different. In 2009, BA saw 347 complaints while dropping to a mere 120 in 2010. That’s great improvement, right? Wrong.
A look at the monthly data shows that in October 2009, BA received an incredible 244 complaints for reservations/ticketing/boarding. Why? According to the Bureau of Transportation Statistics, “The increase in the number of October complaints is attributed primarily to British Airways’ erroneous offer of $40 fares between the United States and India.” That’s an outlier and can’t be used to judge overall performance for an entire year. Guess what happens if we just substitute a more typical monthly result that month? We see an increase in complaints year-over-year approaching 20 percent. Fun with numbers, right? (Not that this is a significant change either.)
I highly recommend reading the entire response. In particular, I like the union’s effort to call into question the safety of the engines on the A380. Enjoy.
