| When designing a trading system, traders or | | | | Here is where the danger lies. These tweaks may |
| investors will test their system against a data set. | | | | not give any real advantage. Instead, you are just |
| They will then tweak the rules until they get a | | | | optimizing the system to the actual numbers. |
| profitable system. The danger is that, by maximizing | | | | To give you an extreme example, if I see that prices |
| the performance against one set of data, they might | | | | made a 52 week low on March 21, and a 52 week |
| over-optimize - where the system is now designed | | | | high on August 9, I could create a trading system |
| to trade actual historical prices, rather than exploit a | | | | that is "buy on March 21, and hold until August 9". |
| real market edge. | | | | Obviously, this "method" would work on the price set |
| The traders will start creating their system by picking | | | | I'm using - but it probably would not work every |
| an approach or idea that should have an edge. For | | | | year. |
| example, they might try moving averages or N day | | | | Once your system is complete, you should test |
| breakouts to identify trends. They might use | | | | against over-optimization by doing an out of sample |
| oscillators or re-balancing formulas to trade | | | | test. This means that you test your system on data |
| counter-trend (buy low, sell high). | | | | from another, non-overlapping time period. For |
| Traders will then add rules for entering positions, | | | | example, you may test your system on data from |
| exiting, stop losses, profit taking, position sizing, and | | | | 1989-1993 and from 2005-2009. |
| adding additional positions. They may also have rules | | | | If you had 62% winning trades in your original |
| such as re-entries or filters. | | | | sample, and the performance falls off drastically |
| Then, they will test their rules against a sample data | | | | (maybe 30%) when testing the out of sample data, |
| set - for example, 1998-2003 S&P 500 end of day | | | | it means your method is probably over-optimized. |
| prices. | | | | Ideally, you want robustness - where the percent of |
| Using the results as feed back, traders may start | | | | winning trades, average amount of a winning trade, |
| adjusting the rules (and add more, especially filters) | | | | etc. is fairly constant. |
| to maximize the expected return of their system. | | | | |