Reasons Why Some Sophisticated Investors Don’t Buy Index Funds

a young businessman looking out a window
•••

Martin Barraud/Caiaimage/Getty Images

For the typical investor, especially those new to investing, finding a strong equity-based wealth building strategy built for the long run can be as simple as following this formula that's proven effective through bubbles and busts, inflation and deflation:

  1. Invest in passive index funds with rock-bottom expense ratios that offer widespread diversification and low turnover.
  2. Practice disciplined dollar cost averaging to take advantage of market prices over time.
  3. Compound money by reinvesting your dividends if you can afford to do so.
  4. Take advantage of tax strategies and shelters, including asset placement or asset positioning.

So why wouldn't someone invest through index funds if it's such a great deal? To understand the reasons, you must understand the three common types of index fund investors:

  1. Those who index because they understand the benefits and prefer them despite their limitations either for the sake of simplicity, peace of mind, lack of suitable alternatives (such as having to choose a mutual fund within a 401(k) plan), or avoiding detailed analysis of individual securities
  2. Those who index because they follow the common investing wisdom of the day
  3. Those with an ideological preference for indexing, who know it works though sometimes don't understand the underlying mechanics.

The first group includes Jack Bogle, patron saint of index fund investing and founder of The Vanguard Group. Millions of retirees owe their financial success to his innovation and work, and his lessons could help many more people experience more lucrative results if they studied his writing and research. A true scholar of finance, Bogle sought to find a better way to invest and identified certain characteristics that, when combined, led to high probabilities of good long-term results. These included:

  • Low turnover to keep costs such as commissions, exchange fees, market impact, and bid/ask spreads minimal
  • Small to non-existent management fees so more investor money can compound
  • Widespread diversification so no individual company or sector dominates
  • Formulaic adherence to a pattern of regular contributions, which averages the highs and lows regardless of market volatility
  • Maximizing tax efficiency, including taking advantage of deferred taxes and tax shelters such as a SEP-IRA.

The index fund, one of Bogle's many contributions to the world, was his prescribed mechanism to harness these forces for the average investor. While it is essentially just a regular mutual fund that buys stocks based upon pre-determined rules regardless of market conditions, it removes the retail investor's judgment as pertaining to selecting individual securities. It also serves as a psychological barrier for emotional investors who might panic after seeing shares of their favorite stocks fall.

Another index fund advantage is that, because they don't require judgment, it isn't necessary to pay someone to manage the funds.

Many successful investors throughout history have built wealth using the exact same underlying principles that Bogle identified through his research with one exception: They substituted valuation of individual securities with regular purchases, using skill sets in accounting to value stocks like they would private businessesreal estate, or bonds

These people are not stock traders. They are not actively chasing hot shares that rise, living like a stereotype out of a movie from the 1980s. They are people like Anne Scheiber, who left behind $22 million, or Jack MacDonald, who left $188 million, or the dairy farmers with eight-figure net worths who keep their wealth a secret from even their children. These are people who spend years putting together a portfolio of companies in which they held an ownership stake, treating public stocks as they would any other asset.

Typically, these investors were well educated, affluent, and, in many cases, entrepreneurial themselves, not hesitating to start a business if they saw a chance to make money.

For the type of sophisticated investors who harnessed the same benefits index funds offered without the index fund methodology, buying an index fund as a matter of course is less attractive for many reasons.

Purchasing Overvalued Assets

Imagine the year is 2001. At that time, you could have parked your money in a 30-year Treasury bond, backed by the sovereign taxing power of the United States government, and earned 5.46 percent interest on your money each year.

With no risk of default, an intelligent investor was going to demand at least 5.46 percent plus a risk premium plus an inflation modification factor to justify giving up what is considered one of the safest investments in the world. That meant a hurdle rate of no less than 11.5 percent, once factoring growth, before investing in an alternative asset.

At the time, Walmart Stores, Inc. was trading at $58.75 per share. It earned $1.49 per share after taxes. That means an investor was "buying" an earnings yield of only 2.54 percent after taxes, and that didn't even include the dividend taxes that would have been owed to the federal, state, and in some cases, local governments had the earnings been distributed to owners. 

Such an insane valuation would have made sense if Walmart were a fast-growing start-up rolling out locations across the country, but this was a firm that was the largest retailer in the world at the time. Owners were all but guaranteed to earn a mere 2.54 percent plus the growth rate in profits, adjusted by changes in the valuation multiple (Bogle refers to this as the "speculative return" factor). Far from enjoying a positive return, even keeping up with the inflation rate would prove difficult.

A savvy businessperson with an accounting background would not have purchased shares of Walmart at that time, at that price. This had nothing to do with market timing—he or she would not have been making a prediction about whether Walmart stock would be higher or lower in the future—it was based solely upon the mathematics of private ownership, in many ways identical to how a restauranteur determines whether he wants to risk money to open another restaurant at a given location or a real estate investor decides whether she should acquire a building at a certain price based upon probable future rents.

An index fund investor who invested at the same time, meanwhile, would have seen several percentage points of his or her index fund purchases allocated to Walmart shares. For a typical worker with no accounting background or someone who figures it will work out in the long run, that is fine. Why? With enough time, dollar cost averaging, reinvested dividends, and tax shelters, it could end up balancing out as very high prices on the top end compensate for depressing lows during collapses.

Should knowledgeable investors pretend to be ignorant? Should they be willing to buy an asset they have no desire to own at a price they think is patently stupid (and they would never pay for a private business) simply because everyone says they should? Should they then pay an ongoing mutual fund expense ratio—even though it is rock bottom the cost is still not zero—to hold said asset?

For the third group of index fund investors—the ideologically driven—the answer they choose is "yes." Because many of them do not fully understand the inner workings of an index fund, they adhere to their belief in its chances for success. This isn't necessarily a bad thing. Sophisticated investors should encourage this behavior because the type of person who is likely to make that sort of decision is the type of investor who needs indexing the most.

Index Funds and Exposure to Certain Areas of the Economy

If, for example, the investor's family's private business was in the technology sector, a cyberattack or some other horrible event that knocked out the telecommunications infrastructure might impair cash flows for weeks or even months. To diversify away from that possibility, that investor may not want to put as much money in technology firms, all of which would likely be hit just as hard in such circumstances. So even if the rate of return were somewhat better in the technology sector, this investor would choose to avoid that sector for peace of mind.

This is basic risk management; something you must learn to do once your estate reaches a certain size to avoid waking up to find your family in ruins. As the saying goes, "you only have to get rich once." After you have your basic net worth built, protecting it is more important than the absolute rate of return you earn on your assets.

Index Fund and Taxes

Imagine you wanted to mimic the S&P 500 or Dow Jones Industrial Averages. Instead of buying a mutual fund run on an index mandate, an investor with a decent-sized portfolio could just as easily build the index directly themselves by purchasing the underlying shares of stock that make up the index. This would allow advanced tax harvesting techniques, keeping more money in your pocket when the IRS bills are due. John Bogle explains this in his books on indexing, writing, "Holding individual stocks for the long term may not only be wise but be far more tax-efficient."

Methodology of the Major Index Funds

Buying an index fund is in a way outsourcing your thinking to someone else. In the case of the Dow Jones Industrial Average, it is the editorial board of The Wall Street Journal, which determines the components in the world's most famous stock market index. In the case of the S&P 500, it's other investors who set the prices of common stocks by buying or selling them directly, determining the market capitalization of each firm. Both still involve human judgment, just not yours.

What makes this somewhat untenable to sophisticated investors is that index funds ranked by market capitalization (for example, the S&P 500) are structured in a way that means you buy more shares of a certain stock as the price gets more expensive, and sell shares as it gets less expensive—buying high and selling low. 

Unfortunately, it's the only feasible mechanism because the assets invested in index funds are so massive it makes fundamental or equal weightings impossible to achieve. Vanguard or another large index fund could not have an S&P 500 equally weighted index fund with their current asset bases. A sophisticated investor could construct one directly, though.

Reasons to Use an Index Fund

There are many situations, reasons, or circumstances in or under which a sophisticated investor might turn to index funds. In some cases, he or she might be at a significant disadvantage in understanding a particular sector. For example, a banker might be able to evaluate individual bank stocks and choose investments wisely in his own industry, but might also have no idea how to analyze pharmaceutical companies. In such a case, buying a low-cost pharmaceutical index fund might be wise. Or he may not want to deal with multiple currencies, paperwork, or different accounting standards that arise from foreign investments.

With a single purchase, you buy a basket of blue-chip stocks from most anywhere in the world, from Europe to the BRIC nations.

A sophisticated investor might want to not worry about her family after she has died. Warren Buffett himself has said that upon his passing, his wife will be left with the bulk of her fortune invested 90 percent in S&P 500 index funds and 10 percent cash despite the fact that he has never personally owned a mutual fund.

None of this changes the reality that index funds remain a compelling way for most people to take advantage of equity ownership, particularly if they are neither interested nor capable of evaluating individual businesses they might want to acquire. Besides, there's nothing stopping someone from indexing a large percentage of their assets and keeping a bit on the side to buy into the companies they want to own. Who says you can't keep 90 percent of your holdings in low-cost index funds and also routinely buy shares of whatever new company in which you strongly believe?

 It is your money, after all.