86 - The Tyranny of Backtesting: Why Backtests are harmful and counter-productive

Books Referenced:

Fooled by Randomness by Nassim Taleb

*If you purchase through this affiliate link, I will receive a small commission, at no additional cost to you. Your purchase will be supporting the show's continued production of free content. 

Thank you for your support!

Mental Models discussed in this podcast:

• A priori knowledge
• Deductive vs Inductive Reasoning
• Empirical Evidence
• The Scientific Method
• Curve Fitting and Extrapolation Problem
• Boundary Conditions
• Hindsight Bias
• Ergodicity

Please review and rate the podcast

If you enjoyed this podcast and found it helpful, please consider leaving me a rating and review. Your feedback helps me to improve the podcast and grow the show's audience. 

Follow me on Twitter and YouTube

Twitter Handle: @TreyHenninger

YouTube Channel: DIY Investing

Support the Podcast on Patreon

This is a podcast supported by listeners like you. If you’d like to support this podcast and help me to continue creating great investing content, please consider becoming a Patron at DIYInvesting.org/Patron.

You can find out more information by listening to episode 11 of this podcast.

Show Outline

The full show notes for this episode are available at https://www.diyinvesting.org/Episode86

Why Backtests are harmful and counterproductive 

• Empirical Evidence - backtesting is a form of empirical evidence. I.e. observation of the past
• A priori knowledge - knowledge, justifications, or arguments that exist independently from experience. (Ex: Mathematics) (Wikipedia)
  • a form of knowledge that can be derived by reason alone.
• Deductive reasoning - the process of reasoning from one or more statements to reach a logically certain conclusion. (Wikipedia)
• Inductive reasoning:
  • “inductive reasoning as the derivation of general principles from specific observations (arguing from specific to general)” - Wikipedia (same link as below)
  • “While the conclusion of a deductive argument is certain, the truth of the conclusion of an inductive argument is probable, based upon the evidence given.” -Wikipedia
• Scientific Method:
  • Form a hypothesis
  • Propose an experiment to test that hypothesis.
  • Results of the experiment either support or disprove the hypothesis. No experiment can prove a hypothesis.
• The problem with the theory:
  • A theory only follows the scientific method if it can be disproven. If a theory is unable to be disproven, then it is not a true scientific theory but something else.
• Curve Fitting and Extrapolation problem (using a curve or model beyond the range of observed data is subject to uncertainty)
• Boundary conditions.
• G Merani (Twitter) (Website) - Covers net-net study backtests in depth

Summary:

Investors use backtests in order to test whether a portfolio's asset allocation would have performed well in the past. The use of backtesting is harmful to a portfolio because it ignores uncertainty and overstates the value of empirical evidence.

It is much better to reason from first principles using deductive reasoning. This deductive reasoning is better than inductive reasoning for investors because it eliminates hindsight bias.