Stock Picks 101: The Dangers of Over-Optimization

y">It might be possible to have a winning system if, for
You've probably read about the importance of havingexample, the stock you buy:o has a trade volume
a trading plan. As part of the trading plan, you needbetween 97,368 and 143,768 shares per dayo opens
to have a trading system in place. Without a system,with a price between $6.42 and $11.76o rises
you have no "map" to trade stocks by.between .39% and .76% between 9:30 and 9:53
One of the hidden dangers of system developmentam.and your exit strategy is to:o wait for the stock
is over-optimization. As part of your systempick to rise an additional .64%o close the position by
development process you will be testing your system11:44 am.
to see how well it has performed in the past. SomeAnd did I mention that this is only on Wednesdays
system development tools will let you quickly testclose to a full moon and while you're holding your
thousands of combinations of system parameters inbreath during the test run? While such a system
a short period of time.COULD have 30 winning entries in a row during the
As you go about evaluating the profit and loss,hypothetical 4 year test period, would you really
drawdown and smoothness of the equity curvehave confidence that this winning streak would
increase, you're likely to focus on the one set ofcontinue?
parameters that looks best. This is the first danger.As you can see from this hypothetical example, even
It's better to look for a cluster of superior resultsthough you MIGHT have gotten some great results
than for just one outstanding set of parameters.in the past, it's unlikely they'll continue in the future. In
Choosing parameters that are in the middle of amany cases, you could apply some serious math and
good cluster of results are likely to lead to moresee at which point you're in danger of
consistent results in the future.over-optimizing.
You will also want to make sure you have enoughHowever the answer to such an exercise will still be
trades in the resulting test set so they're statisticallya probability. With experience, you can often develop
significant. It's foolish to be confident in any systemthe judgment to know if you're in danger of having
for which you have a track record with less than 30over-optimized. In general, the fewer the parameters,
trades during the chosen test period. You can havethe less danger there is of over-optimizing.
some confidence after you have more than 300Frequently, successful systems will only have 2 to 5
trades in the test period. This assumes you haven'tparameters. You're also looking for a cluster of
over-optimized those results.superior results rather than just one outstanding test
Even 300 trades don't guarantee a reliable system. Irun.
have seen any number of over-optimized systemsFor example, you might have a system that picks
that do great for over 300 trades and then losestocks based on an average volume range and a
money forever after.price range. Right there, you already have 4
One way to check your parameters is to hold backparameters. You could add an opening price move
some of your historical data as a "test set." Theminimum, but anything beyond that could lead to
danger here is that if you use the test set moreover-optimization.
than once at the very end of your systemIn the world of probabilities, you can never be certain
development, you're compromising the optimization.that you've got a reliable system. This is because you
This is because your "test set" now has become partcould always be hitting an unlikely (but still possible)
of the optimization procedure.string of good or bad results right out of the gate.
Still, it's possible to over-optimize if you have a lot ofHowever, by not over-optimizing, you stand a better
parameters.chance of having a consistently profitable system.
HYPOTHETICAL EXAMPLE: