It’s so common to come across an article trying to arbitrarily rank airlines or airports one way or another that I generally have little trouble ignoring them all. But when I saw Airways cite a study saying that Los Angeles International (LAX) was the least reliable in the US, I couldn’t help but let curiosity get the best of me. How is that possible? This is just more proof that you can make numbers say anything you want. This isn’t helpful for travelers at all. Instead, it’s just clickbait. (And yes, I know I’m helping fuel that fire just by writing this, which is unfortunate.)
A website called Finder.com had someone pull up the flight status information that is publicly reported through the Bureau of Transportation Statistics. The article doesn’t make it very clear what data is being used, and I can’t get it to match exactly in my initial search. (Frankly, it’s not worth wasting time on a deeper dive.) But I’m guessing that this is likely using something similar to the arrivals at each airport by all reporting carriers for full year 2017. Regardless, the “analysis” itself was pretty simple. They just added up the number of delays of 15 minutes or more by airport and divided by the total. The winner? Well, LAX was responsible for 8.03 percent of delays with exactly 52,056 reported. So it’s clearly the least reliable since it had the most delays… right?
There are so many problems with this it’s hard to know where to start. Of course, the first issue is that this only looks at the data that’s reported. That means it’s for only the biggest US-based airlines on domestic flights, so it’s far from presenting a full picture. (Smaller US airlines started reporting in January of this year, but that’s probably not in here, or at best only for part of it.)
Then there’s the bigger issue at hand that the analysis doesn’t look at the denominator.
They say LAX had 52,056 delays. My look at DOT data shows that there were closer to 54,500 delays during that time period, but that’s close enough. My data also shows during 2017 that there were 214,312 arrivals on reporting carriers giving the airport a 74.5 percent on time rate. That’s not great on-time performance, and I’ll talk about that further later, but it’s still misleading to say LAX is the least reliable airport.
Size Skews Everything
You probably just assume LaGuardia is going to be at the top of the list, right? Well it’s not even on this list, and there’s a simple reason why… it’s just not that big. According to the data I’m using, LaGuardia had just over 26,000 delays for 2017. That must mean it’s twice as reliable as LAX, obviously. Though the reality is that LaGuardia had only 93,334 arrivals recorded, and that means it had worse on-time performance, 72 percent, than LAX. Neither are good, but you get the point. You need to know the denominator to really be able to judge someone’s chances for encountering a delay when they fly.
Another example is this gem: “When choosing among airlines, note than Allegiant Airlines accounted for only 0.10% of all delays of 15 minutes or more at minor airports, whereas Southwest Airlines accounted for 25% of such delays.” First, yes, Southwest will have the most delays in minor airports, because a lot of smaller regionals that fly for the legacy airlines didn’t start reporting until 2018. (I don’t know the cut-off for considering an airport “minor,” but San Diego is considered minor, so it’s not that small.) Allegiant didn’t start reporting data until this year either (this part must have used some 2018 data, I guess), so the data is automatically going put that airline in a favorable light.
But let’s say Allegiant’s data was fully reported just for argument’s sake. It’s going to look better than Southwest no matter what, because it’s smaller. Southwest flew 1,347,893 flights last year. Allegiant flew 93,061. Let’s say Southwest pulled off the impossible and ran on time 93 percent of the time. If that happened, Allegiant could run every single flight late and still look better than Southwest in this so-called analysis.
Context Should Matter
The other issue here is that the data has no context. In Los Angeles in 2017, the first five months of the year were pretty terrible. Once the whole Delta terminal swap completed in May, however, on-time performance rates jumped up for nearly everyone. Take a look:
It’s a tale of two very different airports once the switch occurred. I mean, if you just took the number of delays from the last six months of the year and doubled them, you’d be shy of 40,000 on the year. That’s a far cry from the actual 50,000+ delays that allows Finder to call LAX the least reliable even though it’s not today’s reality. Context makes a huge difference.
So is this entirely garbage? Pretty much. There are interesting ways to slice and dice this data, but Finder apparently chose to just take the easy way out.
20 comments on “Fun with Numbers: America’s Most Unreliable Airports”
Get yourself a copy of the classic How to Lie with Statistics and you’ll be able to prove anything you want. It’s statistics that are unreliable, not LAX.
Statistics aren’t unreliable – their use, interpretation and application can be, however.
There are three kinds of lies: lies, damned lies, and statistics.
I agree that the article doesn’t go deep enough on the details but they do mention that the “worst” are all very busy airports with a lot of aircraft movements and that given the volume of traffic any delay can quickly snowball. All true. Your odds of having a delay at LaGuardia may be greater but your odds of flying through LAX are much higher so I guess I chose to see the yin yang. If ATL and ORD weren’t on the list I’d be much more suspect that someone has it out for LAX, but I don’t think that was the intent, and flying through small airports is easier and more relaxed that the mega hubs, which is basically my takeaway.
To expand on this, it would be really interesting to see clusters of delays in more specific time frames as a percentage of overall movements in said time frame to see when something goes wrong, where does it really go wrong?
Absolutely nailed it. Statistics, percentages, and numbers by themselves are meaningless unless put into proper context. I wrote a scathing letter to our local paper recently in response to a story they ran about road safety that was total BS and precisely for the reasons you mention.
Example. If one road averages 10,000 cars and 2 accidents per day and another averages 100,000 cars per day but has 4 accidents, which one is more dangerous?
As Mark Twain once observed, “There are three kinds of lies; lies, damned lies and statistics.”
It looks like those numbers are number of aircraft, not number of passengers, so that is another way to slice the data. In the IT industry a more common measure is the 95th or 99th percentile performance, which shows what about 1 in 20 and 1 in 100 get.
Brett, good analysis and clear explanation.
Length of delay is also a data-point worth considering. FiveThrityEight did a decent job with this a few years back: https://fivethirtyeight.com/features/the-best-and-worst-airlines-airports-and-flights-summer-2015-update/
The point: not all +15 minute delays are created equal. The average passenger (not just delayed passenger) is 30 mins late leaving LGA because the ~30% of flights delayed are delayed for so long – often multiple hours – that they substantially drag down the average for everyone who uses the airport, even when factoring in on time departures. The average departing passenger from LAX, for comparison, only leaves 3 minutes late.
Hi Cranky, One of the first things I learned as a young analyst is, as you say, you can get numbers to say anything you want. One of the first studies I did was whether or not my airline should get 747s. The analysis initially said that it was too big for us. Then the word came down from “upstairs” that we perhaps didn’t understand the question. Sure enough, with a little reworking of assumptions we were able to show it to be a profitable thing to do. In fact, they turned out to be an economic disaster.
My college stats professor who was Russian used to say in his thick Soviet era accent, “if you torture the data enough, it will confess.” That phrase always stuck with me.
“Fun With Numbers.” Of course, we all laugh. But, you, as a travel agent, someone who is in the travel advice business, do you, or some travel agent association, have a list of things–things we may never have never thought of, in choosing flights and avoiding problems, like, (Airport/Airline), avoid, if possible, departing or arriving between __ and __ because of overload of flights at those times; or, (Airport), avoid, if possible, departing or arriving between __ and __ because of likelihood of fog, wind, thunderstorm issues; or (at Airport or with Airline) be aware of how often that first out, early, early morning, or that late night, last flight of the day can’t find a crew or is just doing mop up for all of the day’s snafus; etc.
[I wish DOT’s “Fly Rights-A Consumer Guide to Air Travel” made an effort to help consumers and less to act as nothing more that a CYA-mouthpiece for the airlines!
Jaybru – Sure, we think of a variety of variables depending on what clients want. But there are no hard and fast rules. It differs by client needs.
This really isn’t an example of statistics gone awry. To call it that would be giving it far too much credit.
This is just plain stupid. Surprisingly so – you’d think that a person with the wherewithal to collect and process all those stats from BTS would also have some sense of how to meaningfully work with those stats.
Keep in mind, 85% if all statistics are made up.
Btw I fly EWR-LAX every 6-8 weeks and never had a delay more than 10-15 mins, therefore they are the two most reliable airports in the US, right?
I vaguely remember my MIT statistics professor (or one of his grad students) saying, “Statistically, the average American has one breast and one testicle.” That saying taught me to do the analysis myself and take someone’s data presentation with a grain of salt.
“There are three kinds of falsehood: Lies, Damned Lies, and Statistics.” –Benjamin Disraeli, 19th Century British Prime Minister
Dear Mr Flyer
Of the 18 posts so far, 3 (17% rounded up) referenced the same quote. However, you (6%) are the only one who quoted the original source. This can be construed as “you don’t read the previous posts so post the same thing” or “you’re a smarty pants”. Either way, I think you’re the winner. What was this discussion about?
You’re right JayMac; my bad. I hadn’t scrolled down to look at previous posts but simply immediately responded to the original article. That original crankiness was analyzing the flaws in a statistical analysis of airport performance. The quote just seemed very apt and as for the attribution, there are lots of incorrectly credited quotes out there.