Searching for patterns

Many years ago I was called in to take a look at a funds management trading system that, despite a great deal of investment in time and energy during its 3 year development, had failed to beat a coin toss once it went “live”.

Prior to the launch the system had undergone extensive testing on 10 years+ of market data and had consistently out-performed the ASX 200 during the trails. Theoretically the system should have been a licence to print money. So why did it fail? The answer in the end was quite simple. The development of the system and the subsequent systems tests had been carried out using ASX historical data. The problem was the historical data set was a time boxed data set based on the ASX market listings at the time of the testing. What this meant was that all those companies that had failed, merged or delisted during the 10 years prior to the back testing cycle were excluded from the test data. This meant the test data set consisted only of the winners (i.e. the survivors) from the past decade.

So in effect what had been built was a system that was perfectly optimised to pick the most profitable winners from the pool of  winners. Once the losers were introduced back into the system the highly regarded predictive algorithms failed comprehensively in their ability to choose between the winners and the losers.

The choice of data resulted in a system being developed that was optimised to find the best historical trading patterns as opposed to predicting the most profitable future trading patterns.

I use this story often to illustrate how the quality of the data source is as important as the algorithms and visualization tools when designing intelligent systems.

Predicting the future is never easy and, as we are becoming increasingly aware, although those who fail to heed history’s warnings are fated to fall into the same traps, historical success isn’t necessarily a strong predictive indicator of success in the future.

One only needs to know that GE is the only remaining stock from the founding of the US Stock Market, that 99% of all start-ups go broke within the first 10 years and that 30% of the companies in the Dow Jones Index disappear each decade to understand that failure is far more likely than success in the business world.

I readily admit that my own record on predicting the future is very patchy. I am often reminded by those ex-students who witnessed my last University Lecture in 1995 that I predicted that the Film and TV skills they were learning back then would be obsolete within 5-7 years thanks to the internet and the democratization the medium and digitization of the production tools. Today, even in an era of YouTube,  their industry skills and knowledge are still in demand.

I console myself with thoughts that my timing was a little out and these things may still come to pass but the reality of course was I was overly enthusiastic about the possibilities and potential of the new media to disrupt the established media.

As I say forecasting is a difficult game. You are not only dealing with market trends but timing as well. If you have any exposure to analytics and business intelligence you will understand implicitly that you are dealing with a 4-D construct. (See The 3 Ages of Business Intelligence)

About these ads