If you decide to put out bad analysis, as UNITE HERE did last week regarding Lufthansa complaints, and someone calls you on it, what would 
you do? The smart answer would probably be to just let it go and stop calling attention to the work, especially since it has more holes than Swiss cheese. But fortunately for us, UNITE HERE has decided to go the opposite route.
The union is using one of oldest tricks in the book: going after my credibility to muddy the water. This is just dumb. They really shouldn’t want to bring more attention to a flawed report like this. Now I’ve just dug in deeper and found even more problems with it. While I was waffling before, now I’m not. UNITE HERE has truly earned the Cranky Jackass Award.
You can read the union’s entire response here (pdf) if you’d like, but I’ll pull out the most fun parts. Let’s start with the opening.
One of the things I appreciate about your site is you are very open about your relationship to companies in the airline industry. And just one month after Lufthansa gave you a free round-trip, business class ride on its A380 from San Francisco to Frankfurt, perhaps I should not be surprised at your dismissive response to my report.
Ah yes, the back-handed compliment. A time-honored tradition that’s used to cover bad work. If someone calls out real issues, just call his or her credibility into question but look completely pleasant while doing so. This takes the focus off your bad analysis and tries to shift the issue. (Sounds like the author may have a future in politics.) It’s true, I’m very open about these things, and I did just fly Lufthansa at the airline’s expense. That doesn’t mean I won’t gladly rip Lufthansa a new one if it’s deserved. The problem here for the union is that it’s not.
You can read the rest of the response yourself if you’re interested in more sugar-coated insults, but let’s focus on the weak defense of the report itself and break that down.
The Department of Transportation data in the report is real, and to my knowledge is the only reliable U.S. source of compiled complaint information on international
airlines. If the DOT is willing to use these numbers to “to determine the extent to which carriers are in compliance with federal aviation consumer protection regulations,” then they’re good enough for me. Even if I am just a research analyst at a union.
*sigh* The issue is not whether this is the only place to get complaint data or not but whether or not it’s statistically valid and can be used to explain a trend or not. In this case, the year-over-year change in complaints from 2009 to 2010 moved by roughly less than one-thousandth of one percent over total passengers carried by Lufthansa to and from the US (using my rough passenger estimate). Even the initial number itself is so tiny that it’s not significantly different from zero. So regardless of what the purpose of the complaint reports are in the eyes of the DOT, that doesn’t magically mean that we can consider each number valid for any kind of analysis.
You’re right, I could have used the raw numbers, but I sort of agree with you that the raw numbers themselves aren’t incredibly exciting on their own. They’re small
because, well, how many people actually go through the effort to submit their airline complaints to the U.S. government? (If you care about an answer, you can look at the DOT analysis for the new passenger rights rule, where the DOT uses the ratio that every 1 complaint submitted to the DOT represents about 61 complaints submitted to the foreign airlines.
Excellent. Let’s just forget about using raw numbers because they aren’t “exciting.” I see. So we’re not looking for statistical validity here. We’re looking for excitement. You can apply any ratio you want to these numbers, but that still doesn’t make the small change valid. And this ratio is just an estimate by the DOT anyway, so using it would make a statistically insignificant change even less valid, if that’s possible.
The result of that comparison was clear. Lufthansa complaints went up, Air France and British Airways complaints went down. Is the sample number of complaints
small? Yes. But if the increases were random, would Lufthansa have seen them in 7 out of 8 top categories from 2009 to 2010? If they were random, wouldn’t Air France and British Airways have seen more fluctuation too?
This is my favorite part. I hadn’t even touched the Air France and British Airways numbers in my initial post, so I should thank the union for giving me even more firepower to show how awful the analysis is. The result is far from “clear” as proposed.
When I spoke with the research analyst, he told me that he didn’t bother looking at the monthly complaint reports. He just looked at the year-end summary and called it a day. That makes the analysis even worse because it doesn’t look for outliers. And that’s exactly why BA’s numbers are so different. In 2009, BA saw 347 complaints while dropping to a mere 120 in 2010. That’s great improvement, right? Wrong.
A look at the monthly data shows that in October 2009, BA received an incredible 244 complaints for reservations/ticketing/boarding. Why? According to the Bureau of Transportation Statistics, “The increase in the number of October complaints is attributed primarily to British Airways’ erroneous offer of $40 fares between the United States and India.” That’s an outlier and can’t be used to judge overall performance for an entire year. Guess what happens if we just substitute a more typical monthly result that month? We see an increase in complaints year-over-year approaching 20 percent. Fun with numbers, right? (Not that this is a significant change either.)
I highly recommend reading the entire response. In particular, I like the union’s effort to call into question the safety of the engines on the A380. Enjoy.
23 comments on “UNITE HERE Strikes Back: Tries to Cover Bad Analysis By Insulting Cranky, Earns the Jackass”
If Gallup puts out a poll saying that X percent of people support candidate Y, they likely only polled about 1,000 people, or .0003% of the population. Yet the results are statistically valid, because they used a fair randomization process. Similarly, if you assume that the odds that a given passenger who has a problem will complain about it are the same for each airline and in each year, then the results are valid even with a small sample.
Just wondering, does anyone know of a formula to calculate the minimum amount that would be statistically significant? There must be a way of doing that.
> Just wondering, does anyone know of a formula to calculate the minimum amount that would be statistically significant? There must be a way of doing that.
I’ve long since forgotten most of my statistics, but it really comes down to two things: the size of the sample and how carefully you select it so that the sample is as “random” as possible and so that it reflects the general population (the main group you’re studying, in this case airline pax) as closely as possible.
Ignoring sample size (not going to research all the tests needed here for that), the main issue is that this was an opt-in sample. When Gallup polls people, they don’t just post a poll on their web site and let anyone answer it, they carefully choose and call a sample of varied people who reflect the US population as a whole, and then tweak their numbers to account for any discrepancies in demographics etc of the sample relative to the general population.
It’s the same thing here- only people who have the time and energy to complain are reflected in this sample. I’d bet that leisure travelers are over-represented in this sample (assumption: businesspeople are too busy to complain), and maybe perhaps older people as well (assumption: retirees have more time to complain, and care more about service etc).
In any event, the point remains that there is just too much noise in the data to really draw conclusions from it. You want to tell me that a difference of 10 complaints among the XX millions of people who flew in the US last year is significant? Really? That could have been one bad storm, one swarmy airline agent, BA’s pricing mishap, anything, and it wouldn’t be reflective of the airline as a whole.
Things like this vary for a lot of reasons, and we shouldn’t rush to draw conclusions from such small variances, especially when it’s an opt-in system with a small sample size that isn’t reflective of the system as a whole.
Personally, I’d put much more validity in Consumer Reports’ analysis of airlines (though they probably purposely have too many leisure travelers, as that’s what their market is)- even though their surveys are opt-in, at least they’re getting thousands of responses, not just a few hundred.
Specifically because the DOT analysis is self reporting, you can’t assume that a small number of reports extrapolates for any purpose at all (or anything under a very large number of reports for that matter, like at least 25% of the whole maybe). If the DOT had a reliable way to take a random sample from all passengers (which wouldn’t be all that hard to do if they cared to do it, but I don’t see why they would without someone passing a law telling them to…), then you have something that is statistically valid. Justification of random vs non-random sampling and the proof that all you need to survey is a few over a thousand people out of 300 million is week 1 in an introductory statistics course (I don’t mean to sound condescending; that is just the case).
There is something in the Consumer Reports point. As a business traveler pretty much wedded to *A airlines, a CR report doesn’t really tell me much, but as the poster points out likely they may have something for Leisure travelers. Part of the point with CR, though, is they have a very long history of rigorous analysis behind their surveys and generally disclose their methodology. They are good enough at it that very rarely to they just whiff an analysis (and it makes the news when they do).
Jim
There is simply no correlation between a Gallup (or similar) poll and a self generated complaint line. Any discussion of same is meaningless.
The Unite “researcher” needs some training in the use (and abuse) of stats – and if Unite really wants to hammer an airline then commission a statistically valid survey sample by Gallup (or similar)!
Keep on keeping on Cranky.
FIGHT FIGHT FIGHT FIGHT!
Cranky,
A couple of points…
First off, you should seriously consider getting the kid who made that Cranky Jackass an internship… if not with you, then with another aviation company, and if not this summer than in the next few years. Gotta keep those kids interested in aviation!
As a political organization (and let’s face it, many unions, especially airline unions, really do act as political organizations as they try to influence public opinion), the union has taken a page from the “best defense is a good offense” book… And failed. Guess they’re more used to dealing with news stations and paper reporters looking for a few good sound bites than anything substantive.
I skimmed the response that Kyle Schafer (might as well let the research analyst get credit for his bad analysis, and let him be Googled for it) wrote, and without trying to attack Mr. Schafer it reminds me of the passive aggressive things I wrote as a teenager. Great response for the sake of entertainment, especially when you reach the end of it and realize that he sidestepped your main points.
“But if the increases were random, would Lufthansa have seen them in 7 out of 8 top categories from 2009 to 2010?” — Cranky, they do have a point here, albeit a very weak one. Run a binomial test on seeing an increase in complaints in 7 or more categories and you get p=0.035 (one-tailed), which is typically considered statistically significant. Now there are several reasons to not take this seriously: First of all, for a one-tailed test you have to start out with the assumption that Lufthansa is going to be bad rather than good, and there’s no a-priori reason to assume this (a two-tailed test is non-significant). Second, you can’t take many statistics and report just the one that turns out significant according to a rather lax criterion, since a small proportion of stats will turn out to satisfy the criterion by chance. And of course, the union is not even trying to claim significance — the above quote tries to argue using rhetoric as opposed to math. Nevertheless, that observation is significant according to a fairly lax (in my opinion too lax) criterion.
Well if they are coming back and attacking you, means you tailed it on what was wrong with what they were saying.
A deserving winner.
The real question is: did the analyst give you and clue as to WHY they were waging this war in the first place?? That was the real curiosity pointed-out in your post.
At the end of the day, no one cares what UNITE HERE thinks of airline service quality anyway… so even if their data analysis WAS valid (which it clearly is not), UNITE HERE pointing it out to the world is not likely to have any impact on the airline anyway.
The whole thing, while annoying for you, CF, I am sure, is entirely moot. I say, don’t waste your time.
First, thanks to everyone for chiming in on the statistical piece. I actually reached out to someone for help with this since it’s been awhile since I’ve had to use stats for this.
Now, to answer your question, Scott, the union refused to tell me why the war was being waged. I asked that point blank and was simply told that all they could say is that they just found the results interesting. The union has been in ongoing negotiations for a contract, but this tactic seems fairly strange to say the least.
… not to mention the fact, as you pointed out in your original post, isn’t the contract negotiation entirely related to LSG SkyChefs — a separate company that happens to be owned by the parent LH Group but is in no way part of the passenger airline referenced by these stats?
The whole thing just seems odd.
Yes, correct. UNITE HERE represents the LSG SkyChefs employees in North America.
The answer as to the why may be in the response after all: “For example, what impact is Lufthansa’s Climb 2011 cost-cutting program having on the passenger experience?”
And, to over-simplify things: Passenger experience would be improved if Lufthansa started back-catering. Maybe that’s what the union is fearing.
I know that the TWU has already organized Allegiant flight attendants. Now they are also going after Virgin America, Jetblue, and SkyWest flight attendants. In the meantime the pilots at Jetblue have filed (again) with the NMB to have a vote if ALPA should represent them. I also know that US Airways pilots are going on 4 years with no contract with their union (at least that’s what the big truck with sign had. And AA is having tons of negotiation problems with their contract. Plus, I remember that Southwest flight attendants union (the TWU) had a bitter battle in their last contract negotiations in 2007/2008 having the union and company management air their dirty laundry in Forbes magazine. I really don’t see how union are any good for airlines. I know unions have done good, and some still do. But Hell, unions should not be running to organize airlines or dividing airlines and management like what is going on nowdays. It really does hurt the consumer at the end (be it strikes, increased wages, layoffs/furloughs, liquidation and bankruptcy and/or setting employee discontent with management and taking it out on their customers. All of those I have mentioned have either had a direct or indirect connection with unions and airlines.
Please excuse all the grammatical errors. Between me typing and doing research for my findings I hit the “Leave comment” button before proof reading.
I’m glad to hear the union is spending their dues dollars in such a resourceful way as analyzing these Lufthansa complaints. Great use of resources & time UNITE HERE!
And in case you can’t understand American humor, that was dripping with sarcasm.
I would recommend UNITE HERE workers demand the a partial refund in their dues because of this waste of money.
I’m not sure but I think these guys were the people who came out and handed fliers to all the spotters at the Lufthansa A380 inaugural landing at SFO. The fliers questioned the safety of the Trent 900 engine and by implication Lufthansa A380 air worthiness. We read them and said, “Wha?” Some of us even laughed. We didn’t get what the reason was and the flier seemed non-sequitorial. Now I think I may have a clue why those fliers appeared in our hands.
Speaking of A380 engines, was there a resolution to the Qantas problem? I see the jets are back at LAX, so I assume they can once again operate at maximum thrust?
As far as I know, it’s been fixed and there have been no further issues.
UNITE’s response is hilarious. Almost as funny as Laura Glading and the APFA with their buttons and lanyards telling us their union. :-)
just to be sure, this is the same union that wants to hold all people at bay whey they feel that it will effect them the most. they are almost a terrorist group. they don’t care about anyone but the idiot dues paying members. the higher ups live like kings on the back of the working man.
t.j.
Oh dear, oh dear…LH (Lufthansa) use the derated version of the Rolls Royce Trent, the Trent 970 on their A380-841’s, Qantas use the Trent 972 Higher thrust variant,
There are currently 7 A380-841’s in the Lufthansa Fleet all with upgraded engines.
A few observations (not even really analysis) about the data:
1) They pointed out that the complaints increased in 7 of the top 8 categories. 2011 YTD (through April), LH has complaints in 7 categories (out of 12 total), whereas AF has complaints in 10 categories and BA in 9 categories. Misleading.
2) 2011 YTD, AF and BA have 42 total complaints, whereas LH has 41 total complaints. LH compared to AF and BA, respectively, “leads” in 2 categories: Res/Tktg/Boarding (7-5-5) and Fares (2-1-0).
3) AF had 97 complaints in 2010, but are on pace for 126 in 2011. Will Unite Here release a report claiming AF sucks b/c their complaints went up 30% from 2010 to 2011? BA had 120 complaints in 2010 (more than LH in 2010, btw). They’re also on pace for 126 complaints, a 5% increase.