Stock Prediction using Decision Tree: Classification Tree

This is the fourth post in a series on using Decision Tree for Stock Prediction. For more information, feel free to read post 1, post 2 and post 3 of the series.

Once the data have been preprocessed, we obtain a matrix in which each row is a different day (since we work with daily data) and each column is one of the possible variable (close, volume, technical indicators, combination of some indicators, etc.). The reason why I started with decision tree instead of more "trendy" neural networks or support vector machines is because I prefer to begin with simple methods and then, if necessary, change to a more complex one.

One big advantage with decision tree is that one can understand the model by seeing it (i.e. by looking at the tree). It is very appreciable to understand why, at a given day, MSFT (ticker name for Microsoft) has been predicted to increase or decrease. However, this readability is only applicable as a pre-study in the project. Indeed, since the project is based on making one prediction a day (during all the backtesting period) for each selected stock, there are too many different models for a Human being to understand them.

Thus, the high number of models is due to the following processes which have to be done:

For each year to backtest
  For each open day in the year
    For each stock that has been selected
      For each hyper-parameter value of the tree
        For each fold of the cross-validation
          Build a decision and evaluate it

If we consider that building a decision tree takes 1 second, then, for a backtest on 100 stocks from 2001 to 2008, we need:

8 * 252 * 100 * (10*10) * 10 = 201'600'000 seconds

This means more than 6 years of computation on a 4 CPU computer. At this stage, there are mainly two possibilities:

• Grid computing
  
• Computing the trees each month instead of each day

By applying these two ideas, it is possible to bring the processing time to around 3 hours of calculation (with a 6 x 4 CPU grid of computers). The next post of the series will discuss the risk management of the system.

Continue reading... Sphere: Related Content

Stock Prediction using Decision Tree

This is the first post in a series on using Decision Tree for Stock Prediction. Here are the second, third, fourth and fifth posts.

I have started applying data mining to finance for a few months now. I will thus give you an insight about my main project regarding stock market prediction. While starting in my company, I have seen several projects (so-called "screener", i.e. based on technical indicators to build stock picking rules, but no use of data mining). Most of them make two assumptions:

• The rules based on technical indicators don't evolve in time
  
• Stocks are selected (and sometimes processed) differently according to the sector they belong to (e.g. health and care, industry, etc.)

Since I don't feel good with these two assumptions, I have started a new project based on the following idea:

Each technical indicator may work for a particular stock and at a certain moment in time
This means that i) rules based on indicators should evolve in time and ii) each stock should be processed independently. Note that the second point doesn't mean that there are no correlation between a particular stock and the sector it belongs to. It only means that stocks may behave differently and thus should be treated independently. However, any information from their sector could be used in the forecasting process.

When seen as a balck box, the system has information about a specific stock (such as open, high, low, close, volume, etc.) as input and a class value as output. The class is fixed this way:

1 if close[j+n] > (x% * close[j]) + close[j]
-1 otherwise

where n is the difference between the current day and the day predicted and x is a value chosen to take transaction fees into account (note that a fixed value could also be chosen instead of a percentage). The class predictions are thus made for each stock independently. One year daily data is used for training and the following month for testing. A shifting window process is made so that the system adapts itself to the current market.

Here are the different steps of the overall methodology that makes use of decision tree for stock prediction:

1. Stock filtering
2. Data preprocessing
3. Classification tree
4. Risk management

In the following posts, I will explain in details each of these steps.

Continue reading... Sphere: Related Content

Data mining useless in finance?

Quantitative Trading, a blog owned by Ernest Chan has a pessimist post about data mining. To be brief, the author writes that data mining is often useless for financial purposes, and gives an example with stock picking. To his mind, the reasons are the lack of historical data and the noise. To the denoising methods that exist, he responds that the overfitting problem persists. He certainly is true when writing that mining small quantity of data having a lot of relationships is not easy. Although I think it is not easy, I do believe there are solutions such as cross-validation, dimensionality reduction, etc. to obtain good results. Since I have no experience in the financial field, I ask your opinion about that question. So feel free to give your mind by posting a comment.

Continue reading... Sphere: Related Content