In this post: Hide
It seems like more and more organizations are producing their own airline rankings every year. In attempt to sort through the increasing quantity of information, I thought it might be interesting to take a look at a handful of rankings all at once, and to aggregate them into a single list. The outcome is interesting, and I walked away with a few surprising takeaways.
My love-hate relationship with rankings
I have mixed feelings about airline rankings. In an ideal world, they reflect an airline’s service quality, performance, and reliability, and generally simplify complex information to help passengers make decisions without needing to assess every detail. Rankings also allow for apples-to-apples comparisons between airlines, providing a competitive benchmark that can motivate management to improve. The enduring appeal of rankings lies, at least to me, in their use as a shortcut around the difficult work of slower, deliberate thinking in favor of quick, intuitive thinking.
Shortcuts have downsides, though, and it’s important to keep in mind the whole host drawbacks that come with rankings. To name just a few:
- Reductionism: Rankings often simplify complex realities to a single number or rating, and steer readers to make poor decisions. For example, an airline that is ranked highest overall might not necessarily have the highest on-time rates. If on-time rates are my only criteria for picking an airline, then looking at these rankings won’t help me make a decision that is in line with my preferences.
- Bias and subjectivity: The criteria used for rankings may reflect the biases of the rankers. Some important factors may be overlooked while others are overemphasized.
- Incomplete data: Rankings are often based on available data, which may not cover all aspects of the entity being ranked. They may not consider qualitative factors that can significantly affect the user experience.
- Gaming and manipulation: Airlines being ranked might focus solely on metrics used in rankings at the expense of other aspects. They might manipulate metrics to climb rankings, which may not always align with better performance or quality.
The UK-based consultancy and rankings company Skytrax is a great example of how rankings can fall short and undermine their own usefulness to passengers. For more on this, Ben from One Mile at a Time wrote a great piece about the funny business involved in Lufthansa securing a coveted five-star rating from Skytrax (which it has since lost).
One way to account for some of the drawbacks of rankings is by looking at a bunch of them because combining several rankings can minimize the effect of certain biases, specifically those that are unique to a specific ranking. Keep in mind that aggregation won’t account for more systemic issues common like herd behavior (i.e. rankers converge on a particular order to avoid being an outlier). For example, if one ranking organization puts Delta in last place but notices others have Delta in first place, they might begin to doubt their methodology and be inclined to change it so their rankings become more similar to piers’.
In this analysis, I’ve taken the 2022 rankings (the most recent year available) of US airlines produced by six different organizations. Below, I’ll outline briefly how their methodologies differ before presenting each ranking and finally averaging the rankings to produce an overall ranking. For simplicity’s sake, I’ve weighted all the rankings equally.
A brief overview of the ranking organizations
- American Customer Satisfaction Index (ACSI): ACSI’s ranking focuses on customer satisfaction and evaluates airlines based on criteria like the quality of in-flight services, booking process, baggage handling, and flight crew professionalism. The index is based on interviews with thousands of customers.
- J.D. Power: J.D. Power provides rankings based on the results of large-scale customer satisfaction surveys. They evaluate airlines across seven factors: cost and fees, in-flight services, aircraft, boarding/deplaning/baggage, flight crew, check-in, and reservation. J.D. Power created separate rankings done for economy, premium economy, and business/first class, but for easier comparison’s sake, I combined all three.
- SkyTrax: SkyTrax rates airlines on their service quality across the entire flight experience, including cabin staff, in-flight entertainment, meal service, and comfort. SkyTrax’s ratings are widely regarded as a global benchmark of airline excellence, though have been marred by controversy.
- The Points Guy (TPG): TPG ranks airlines based on metrics like award program value, cabin comfort, route network, and baggage and change fees.
- WalletHub: WalletHub’s airline rankings focus primarily on the financial aspects of flying, such as pricing, rewards programs, and baggage and change fees, though they also consider factors like safety, on-time performance, and customer complaints by collecting and analyzing flight data from the U.S. Department of Transportation.
- The Wall Street Journal (WSJ): The WSJ’s airline rankings focus on operational performance, punctuality, and the passenger experience. They use metrics like on-time arrivals, canceled flights, extreme delays, passenger complaints, and mishandled baggage.
Three things stand out to me:
- Our overall ranking corresponds perfectly with on-time performance (OTP). In 2022’s OTP ranking, Delta came first with an OTP of 82%, then Alaska and United at 79%, then American at 77%, and then, you guessed it, JetBlue at a dismal 64%.
- JetBlue finishes last in the overall ranking, behind United and American–given my travel experiences with JetBlue, I would’ve put it above both those airlines. This is a good example of why it’s dicey to generalize about an airline based on a relatively small sample of personal experience.
- J.D. Power’s ranking, which is primarily based on customer satisfaction surveys, puts JetBlue first. That means that despite poor on-time performance, JetBlue is remarkably well-regarded by its customers.
There you have it folks: averaging six different rankings sees Delta ranked the number one US airline in 2022. Keep in mind that though aggregation remedies some of the issues with rankings, this is still very much not a science, so take it with a grain of salt.