TABLE OF CONTENTS.
- Back test and optimization – a holy grail?
- Optimization versus curve fitting
- Using the stress test for creating a robust strategy
- The stress test indicator
- Application in practice
- Robustness – the key to success
BACK TEST AND OPTIMIZATION – A HOLY GRAIL?
Algorithmic trading strategies are based purely on objectives and clear rules:
Getting into a position, the stop, exit and the position size are all subject to unique requirements. Besides this, another advantage is that any emotional traps are easily avoidable. The use of quantitative trading strategies also provides a decisive advantage: Any strategy that can be transferred into a source code and as such converted it into a machine language, can be back tested – in other words, simulated based on historical data – before it is applied to real funds.
By using the gained risk and return metrics the characteristics and quality of a strategy can be analyzed in detail and compared with alternatives. Combined with an optimization function, many powerful tools to aid the development of trading strategies are available to the user.
OPTIMIZATION VERSUS CURVE FITTING.
Used incorrectly, however, the tools just mentioned can have undesirable consequences, namely ‘curve fitting’. The following anecdote, which reiterates a conversation between scientists, Freeman Dyson and Enrico Fermi, describes this phenomenon:
It is 1953. Freeman Dyson questions Fermi in Chicago to discuss with him his own results for the meson-proton distribution. Fermi was however visibly unimpressed by the results and asks Dyson, how many selectable parameters he had used for the calculations. Dyson answers that he had used four parameters. Fermi’s retort: “I remember how my friend John von Neumann used to say, with four parameters I can adjust an elephant, and with five I can make him wiggle his trunk.”
This quote describes, what is at stake in curve fitting – to over-fitting a trading strategy to the underlying data series. During the optimization process, individual parameters are modified (eg, length of the moving average, stop or target size) in an iterative process until, finally, from a large number of different combinations those are picked, which promise “optimal” results – but only on the basis the specific historical data that was used for the calculation.
So before a back test and optimization are performed at all, the trader should have a solid idea for the trading strategy and attempt to define meaningful parameter ranges to be tested. In short, the software should validate a trading idea, not discover it.
The more parameters a trading strategy has and the higher the number of combinations tested, the higher the curve fitting effect.
The result is that the trading strategy delivers wonderful results in the back test, but fails miserably in real trading. Simulated scientific studies show that the stronger the strategy was adapted to the historical data, the poorer the results in real trading.*
A further study entitled “The probability of backtest overfitting” can be found here.
For this reason, a division of the representative historical market data in at least two sub-periods is necessary. This allows for the development and optimization of a trading strategy based on the period A (In-sample back test) in order to test it on the unknown data-terrain, period B (Out-of-sample back test).
The results of the sub-periods should not show any significant deviations. If this were the case, it is highly likely to be evidence of an over-fitted strategy.
A special form of this test method in which the optimal parameters of a sub-period for the subsequent out-of-sample period are used,it is called the walk forward test. How you can use it in practice and what features are provided by Tradesignal? We discuss this in greater detail in a future How To issue.
USING THE STRESS TEST FOR CREATING A ROBUST STRATEGY.
For a trading strategy to work, not only in the back test, but also in real trading, it must be robust. With Tradesignal the back testing and optimization function allows you to check how stable your trading strategy behaves on different markets and in different time frames – even using an entire portfolio. The testing of the strategy at different time frames can help identify changes in performance behavior, which would be due to differing market phases. The use of other securities in the back test shows, however, whether the algorithm actually identifies general patterns that can be observed in other underlying assets or if it has been trimmed on specific patterns within the underlying data series.
An easy way to test this is to monitor the change in the risk and return characteristics while the parameter values are being modified. The more varied the results, the less robust is the strategy. An extreme negative example is shown in figure 1. The graph shows the profit factor of a trading strategy in response to an entry and exit parameter; red areas represent unprofitable, green for profitable values. As can be seen, of all parameter combinations only one is profitable. If exactly this “optimal” combination of parameters is the basis for a trading strategy, one thing is inevitable in real trading – its failure.