New Here? Get the Free Newsletter

Oblivious Investor offers a free newsletter providing tips on low-maintenance investing, tax planning, and retirement planning. Join over 21,000 email subscribers:

Articles are published Monday and Friday. You can unsubscribe at any time.

Investment Results and Sample Size

I briefly touched on the idea of sample size a while back when discussing the likelihood that a fund manager’s performance could be the result of luck rather than skill. Today I’d like to discuss the idea in a bit more depth.

Generally speaking, the more observations made in any given study (i.e., the larger the sample size) the more confidence we can have in the study’s conclusion. And that makes perfect sense–the more consistently we see the same results occur, the more confident we can be that the results aren’t due to randomness (i.e., “noise”).

So here’s my question: In the world of investing, why do we often let this idea go completely out the window when making decisions?

Sample size and fund managers

Why, for example, would we look at a mutual fund that’s outperformed the market for 8 of the last 10 years and even begin to think that it’s due to skill? A 10-observation sample size is very small. (Example: How much faith would you put in a medical study that only included 10 patients?)

Further, a fund manager randomly picking stocks could be expected to beat the market in 5/10 years. (Well, probably 4 out of 10 after expenses.) Outperformance in 8 years out of a 10-year sample is almost meaningless statistically.

I’m not saying that this means that a fund manager who outperforms in 8/10 years isn’t skilled. I’m simply arguing that we don’t have (even close to) enough data to know for sure. And therefore betting one’s money on it seems rather imprudent.

Sample size and market forecasters

Or what about market forecasters who successfully predicted the big crash late last year? As with every big market move, the forecasters who accurately predicted it are now regarded as experts. And what’s the sample size here?

One! A person predicts one market movement correctly, and we call him an expert.

By way of comparison, imagine if last fall Gallup had telephoned precisely one person, asked him who he was planning on voting for, and then made an official prediction that that candidate would win the election.

Overestimating our knowledge

There are plenty of other examples of such behavior–got any favorites?–but I’m sure you get the point. We need to be careful not to overestimate our knowledge about something based on such small amounts of data.

New to Investing? See My Related Book:

Book6FrontCoverTiltedBlue

Investing Made Simple: Investing in Index Funds Explained in 100 Pages or Less

Topics Covered in the Book:
  • Asset Allocation: Why it's so important, and how to determine your own,
  • How to to pick winning mutual funds,
  • Roth IRA vs. traditional IRA vs. 401(k),
  • Click here to see the full list.

A Testimonial:

"A wonderful book that tells its readers, with simple logical explanations, our Boglehead Philosophy for successful investing." - Taylor Larimore, author of The Bogleheads' Guide to Investing

Comments

  1. I agree that sample size is very important. But, think about it from the vantage point of the potential customer:

    Eight out of ten years seems like such a long time.

    But when looking at the results of a survey with 10 people, the obserber just ‘knows’ that’s a small number.

  2. Hi Mark.

    I bet you’re absolutely right there. A decade sure feels like a long time. “How could anybody get lucky for an entire decade?” seems like a pretty reasonable question.

    It’s surprising how often we’re tricked by our own brains.

Disclaimer: By using this site, you explicitly agree to its Terms of Use and agree not to hold Simple Subjects, LLC or any of its members liable in any way for damages arising from decisions you make based on the information made available on this site. I am not a financial or investment advisor, and the information on this site is for informational and entertainment purposes only and does not constitute financial advice.

Copyright 2020 Simple Subjects, LLC - All rights reserved. To be clear: This means that, aside from small quotations, the material on this site may not be republished elsewhere without my express permission. Terms of Use and Privacy Policy

My new Social Security calculator (beta): Open Social Security