From the August 2010 newsletter:
The financial industry certainly places more emphasis on style than substance. So when their work is actually evaluated, it tends to be disappointing. Wall Street’s earnings forecasts? They suck. Performance of mutual fund managers? Quite embarrassing. Do Morningstar ratings also belong in the same category? You probably see them all the time; mutual fund companies love using Morningstar ratings in their marketing materials. But is there any value in a 5-star rating?
Luckily for us, researchers recently looked into these ratings and published their results. They compared Morningstar ratings vs. fund expense ratios as a predictor of future performance. The expense ratio is one of my favorite metrics. If you assume that mutual fund managers have no value, which I find to be a very good approximation, you would expect lower costs to predict better performance. The report found just that:
Expense ratios are strong predictors of performance. In every asset class over every time period, the cheapest quintile produced higher total returns than the most expensive quintile.
And how about Morningstar ratings? 5-star ratings predicted better performance vs. 1-star ratings in 13 of 20 observations, a success rate of just 65%. That sounds pretty good on its own, but it’s still worse than a metric that anyone can look up in seconds.
Since Morningstar uses prior performance (after fees) to calculate its ratings, the ratings already include information about expense ratios indirectly. So what is Morningstar adding with its fancy algorithm? Let’s use a little high-school algebra to find out (Warning: Geek Alert!):
Rating = Expense Ratio + Morningstar's Additional Analytics
And we just found out that:
Expense Ratio > Rating
Finally, using my graduate degree in math, I get this:
Morningstar's Additional Analytics < 0
Yes, its algorithm is horrible. And that’s not all. Morningstar reserves its 1- and 5-star ratings for the top and bottom 10% of funds. However, expense ratios were split into quintiles, or as normal people would say, 20% buckets. That’s just so sneaky. So expense ratios were handicapped by using 20% buckets instead of 10%, and still beat Morningstar ratings. Ouch.
Why would the researchers do that? Well, there’s one thing I forgot to tell you. People have done this evaluation many times with similar results, so it’s not news to serious students of investing. The interesting part of the report I quoted is the publisher: Morningstar. If you read its report, it sounds like a politician answering a tough question — uncomfortable. Independent thinkers can go directly to the results here (pdf).
Note
After writing this, I noticed that Morningstar clarified that ratings are indicators of past performance, and should not be used to predict future performance. If Morningstar were concerned about substance, it would tailor its ratings to how investors actually use them — as an indicator of a good investment. If it did that, most 5-star rated funds would just be index funds. But Morningstar unfortunately emphasizes style (and money), so it ends up with an imperfect rating system that benefits one of its biggest clients, mutual funds.
Dear Edwin –
Not bad, a few digs, but mostly on track.
I do have a few comments –
1) I understand the backlash against mutual funds, but it’s greatly overstated. I have at hand a study in the February 2010 edition of Journal of Finance that finds that most actively managed mutual funds have alphas after expenses that are “at least on par with passive funds.”
Now, one academic study finds one thing, another finds another. But … there’s plenty of evidence by people who have zero, zilch to do with the mutual fund industry, that is reasonably supportive of mutual funds.
2) Good catch on the 10% vs. 20% part, nobody else caught that. The reason of course this exists is not a subtle plan on Morningstar to rig the results, but rather that there are 5 star ratings, thus the natural inclination is to use 5 expense buckets, and quintiles are the natural element that came to mind.
If Morningstar wanted to rig the results, it could have done several other things. It could have risk-adjusted the findings. In fact, from an academic stand point, it *should* have risk-adjusted the findings. That would likely have benefited star ratings, as the star ratings are risk-adjusted and expense ratios are not. Morningstar could have load-adjusted, as once again the star ratings are load adjusted and expense ratios are not. Finally — and this is a technical but important point — Morningstar could have conducted the study using one share class for each fund. It’s a long story, but using all available share class is an option that significantly improves the predictability of expense ratios.
So no, this was not an exercise in trying to make the star rating look good. If it had been, the methodology and results would have been different.
3) Finally, it should be noted that the star ratings were directionally correct 65% of the time (13 out of 20) *only if you ignored dead funds.” If you took survivorship bias into effect, which is not only the academically correct approach but is the intuitively correct method (because funds that die had investors and had track records, their results shouldn’t just be tossed out), then the star rating worked 100% of the time.
Finally, this particular study had one ending time period, March 2010. Just one. So I wouldn’t run too far with it, although arguably Morningstar did run a bit far with its own study, and my goodness others have run much much further. But anything that has exactly one fully independent time period, is far from the last word.
Best –
John Rekenthaler
VP Research
Morningstar Inc.
Hi John-
Thank you for commenting. I must say I was pleasantly surprised to get a direct response from Morningstar. Here are my thoughts:
1) If you’re referring to “False Discoveries in Mutual Fund Performance: Measuring Luck in Estimated Alphas” by Barras, Scaillet, and Wermers, I’m surprised that you would mention that paper, because that is the exact paper I quoted for my case against active mutual funds. The link in my post refers to this article in The NY Times from 2008. My favorite quote from that paper is:
“Further, 24.0% of the funds are unskilled (true α < 0), while only 0.6% are skilled (true α > 0)—the latter being statistically indistinguishable from zero.”
I am confused how this could possibly be a case for active mutual funds.
2a) 20% v 10%. Thanks! I’m surprised that no one else caught this before.
2b) Risk Adjusting. Okay, so let’s risk-adjust the findings to be more correct from an academic standpoint. Luckily for us, you already included this in your analysis by predicting future ratings (a risk-adjusted performance measure). Directionally, rating predictions using expense ratios were correct 100% of the time; past ratings were only correct 15 of 20 (75%). And even in magnitude, expense ratios beat past ratings 19-1 for the top group, and 16-4 for the differential (top group – bottom group). As mentioned in the post, this is with the 20% vs 10% handicap and the fact that ratings already include expense ratios indirectly.
2c) Load Adjusting. Like risk adjusting, load adjusting seems to make Morningstar ratings look worse. So based on the results mentioned in (2a), the expense ratio, a measure that completely ignores loads and in some cases may be negatively correlated with loads, seems to do better in predicting load and risk adjusted performance (future ratings), than prior ratings that do incorporate both expenses and loads. Wouldn’t combining (annualized) loads with expense ratios be a better predictor of ratings than using expense ratios alone, thus even (more) better than past ratings?
2d) Multiple share classes. Actually, this is a great point. By including multiple share classes with differing expense ratios, you’re guaranteeing that the performance to expense ratio relationship exists for those funds. Perhaps, one way to get around this is to weight funds by size.
3) Even when we look at the success ratio to account for survivorship bias, which as you mention is academically and intuitively correct, the expense ratio still beats ratings 23 to 17 (or 57.5% of the time). Again, this is with the 20% vs 10% handicap and the fact that ratings already include expense ratios indirectly.
I agree that the time period for this type of study was far too short (only 5 years for predictions, and up to 10 years more to calculate initial ratings). However, when a firm like Morningstar comes out with interesting research, it obviously becomes a part of investors’ discussions, so I do feel obligated to comment and share my analysis.
Finally, your colleague Don Phillips wrote in the follow up article, “Why, then, do [ratings] court such controversy when similar past-performance measures like alpha, the Sharpe ratio, and information ratios do not?”
Well, the answer is easy. The ratings are called Morningstar Ratings not risk-adjusted past performance, so the ratings explicitly has Morningstar’s recommendation. And you surely understand that investors place a lot more weight on Morningstar ratings than information ratios, just look at any mutual fund advertisement. And I would even guess that investors (incorrectly) use ratings more as a measure of a good investment (future performance) than as an evaluation of past performance.
Yes, I took some digs at Morningstar. But that’s because you are in a great position to affect investor behavior (for the better), not one at a time like me, but thousands or even millions of investors at a time. Please do use that power to its full potential.
Edwin
Edwin –
Those are an excellent comments. You have thought through the details, have a good grasp of the facts, and are attempting to be reasonable.
As for the Morningstar ratings, yes of course I see the argument that you make that they overpromise. On the other hand, the star ratings are free and widely available, they are fully independent in that no fund company is charged for receiving them, and they tend to be directionally correct. (They are particularly good at identifying funds that will go kaput, a lot better than expenses although there the 10% vs. 20% item could be a factor.)
Twenty years ago, stating that low-cost indexing is a valid and sensible strategy meant getting flamed by active managers for going too far. Today, it means getting flamed by ETF managers for not going far enough. Times certainly have changed.
Following up on the past note, you should realize that while I agree largely with your views, you are no longer solely on the side of the angels. The same people who use to say *anything* to get a buck into their actively managed funds, have now become converts to indexing, and they are saying absolutely anything for their new cause.
Take this beauty, I pulled it off an ETF advisors’ web site: “ Over a five year period, 97% of the professional money managers underperform their index after fees and taxes!”
Yeah sure. I just called up the Large Blend stock group, the Vanguard Index 500 fund (which has very few fees and taxes) finished 705th out of 1489 Large Blend funds with a 5-year record. That was of course among the best of the index funds, the others tended to be in the 800 to 900 range.
This can’t possibly be right in any real sense, maybe in some strange technical sense. But it was surely written with the intent to deceive.
I see *way* too much of that from the supporters of indexing these days.