In-Sample / Out-of-Sample: How to analyse different Market Regimes

We all know that the biggest enemy of backtesting is curve fitting. That happens when you torture the data again and again to create the “perfect” equity curve (which does not generate profits in the future). It could be too many parameters; a favourable market regime or just the best parameter combination calculated from the full data set. Therefore, one of the most important things to do is splitting your data set into an in-sample (training) set and an out-of-sample (validation) set. This article shows you how to create a simple module which can be applied to any existing strategy in Tradesignal. You can control the in-sample and out-of-sample periods with a rolling window and compare the results.

Practical Example: Backtesting 100 years of Dow Jones Data

Here is an example: The chart shows more than 100 years of monthly bars for the Dow Jones Industrial Average. Let`s say we would like to make a proof of concept test whether the famous MACD indicator offers some value. Also, we want to know how results change across different market regimes. The latter question is very important to understand the strengths and weaknesses of any systematic strategy.

The time period highlighted in red covers the years 1910 to 1919, so we look at 10 years of results. The strategy report indicator in the upper left corner of the chart shows the most important metrics, such as net profit, average trade or maximum drawdown. Every signal is traded with 100k USD (without reinvesting), so a net profit of -19.875 equates to a loss of around 19.9%. Unsurprisingly, in a sideways market regime like this, the MACD cannot work profitably.

Now, please click on the chart below and you will see an animated version with the 10 year backtest period rolling forward. This way you can see how both the performance metrics and the behaviour of the equity curve changes across time. Of course you can modify the length of the backtest period and go back and forth to compare different market regimes.

Dow Jones monthly bars with pre-built MACD strategy applied between 1910 and 1919

Equilla Code for rolling In-Sample and Out-of-Sample

Now let us take a look at the Equilla code for that strategy module: The input section contains all you need for controlling the rolling backtest window: the start and end year and the input “yearsJump” which you can use to jump to the right or the left side on the date axis. Lines 21 to 30 make use of some trading commands which allow us to exclude signals which are outside of the selected time period.

Equilla code for rolling in sample out of sample window
Equilla Code for rolling In Sample Out of Sample Window

Using the Optimiser to analyse Results

Having implemented the input “yearsJump”, we can manually select specific time spans for the backtest. An interesting alternative is using the built-in optimiser of Tradesignal. For example, I split up the whole Dow Jones data set into chunks of 5 years and then set the parameter zone for “yearsJump” from 0 to 20. Now we can analyse how different strategy returns and risks are for 21 different time spans.

Please contact us if you would like to receive the workspace. And don`t forget to subscribe to our blog. Thank you!

Optimiser results for 21 different time periods (each 5 years long)

Tradesignal® is a registered trademark of Tradesignal GmbH. Unauthorized use or misuse is strictly prohibited. Data by Teletrader.
Copyright Terms of use:

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe to this Blog