GitXplorerGitXplorer
d

deeplearning-timeseries

public
35 stars
15 forks
2 issues

Commits

List of commits on branch master.
Unverified
4614c37268d79909d4eca6f4486f7db3d875ce37

Update README.md

ddhrushilbadani committed 8 years ago
Unverified
a51ecd8480bca96cfb7d818207e48ed1955d705b

Update README.md

ddhrushilbadani committed 8 years ago
Unverified
9ad8af8429cf6a8b2d1bb8e2ccd0a80953120cac

Update README.md

ddhrushilbadani committed 8 years ago
Unverified
b0547518be312e50ef3ab0bfe837d2df33855b4a

Update README.md

ddhrushilbadani committed 8 years ago
Unverified
80ad9ec0cf0d7b2f96e83e112fdbd3f3d8cd6903

Update README.md

ddhrushilbadani committed 8 years ago
Unverified
7de51223c9109fe44115cf31fae1b0c8079d33de

Timeseries forecasting via deep learning

ddhrushilbadani committed 8 years ago

README

The README file for this repository.

Timeseries Forecasting with Deep Learning

This Python project uses LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) based Recurrent Neural Networks to forecast (predict) timeseries using Keras + Theano. We compare the results produced by each of these deep neural networks with those from a linear regression model.

Dataset: Number of daily births in Quebec, Jan. 01 '77 - Dec. 31 '90 (Hipel & McLeod, 1994)

##Usage I suggest you install Virtualenv before trying this out.

git clone https://github.com/dhrushilbadani/deeplearning-timeseries.git
cd deeplearning-timeseries
virtualenv ENV
source ENV/bin/activate
pip install --upgrade pip
pip install keras h5py pandas sklearn
python evaluate.py

##Architecture & Model Properties We use Keras' Sequential model to construct recurrent neural networks. There are 3 layers:

  • Layer 1 : Either a LSTM (with output dimension 10, and statefulness enabled) layer or a GRU (with output dimension 4) layer.
  • Layer 2 : A Dropout layer with dropout probability = 0.2, to prevent overfitting.
  • Layer 3 : A fully-connected Dense Layer with output dimension 1.
  • Default optimizer: rmsprop; Default # of epochs: 150.
  • Accuracy Metric: Mean Squared Error.
This architecture can certainly further be optimized - I just haven't had the chance to experiment too much thanks to my laptop's constraints!

##Results & Observations

  1. The LSTM-RNN model performed the best with a MSE of 1464.78 (look back = 37).
  2. Naively making the RNN "deeper" did not yield immediate results; I didn't fine-tune the parameters (output_dim, for example) though.
  3. Making the LSTM network stateful (setting stateful=true when initializing the LSTM layer) did yield a significant performance improvement though. In stateless LSTM layers, the cell states are reset at each sequence. When stateful=true however, the states are propagated onto the next batch i.e. the state of the sample located at index trainX[i] will be used in the computation of the sample trainX[i+k] in the next batch, where k is the batch size. You can read more about this at the Keras docs.
  4. Using Glorot initializations yielded a performance improvement. However, using He uniform initialization (Gaussian initialization scaled by fan_in) yielded even better results than with Glorot.

##Files

  • ```data/number-of-daily-births-in-quebec.csv``` : Dataset.
  • ```lstm_model.py```: Contains the class ```LSTM_RNN``` for LSTM-based Recurrent Neural Networks.
  • ```gru_model.py```: Contains the class ```GRU_RNN``` for GRU-based Recurrent Neural Networks.
  • ```evaluate.py```: Loads and preprocesses the dataset, creates LSTM-RNN, GRU-RNN and Linear Regression models, and outputs results.

    • ##To-do

  • K-fold cross validation.
  • Add plots to aid in visualization.

    • ##References

    1. On the use of ‘Long-Short Term Memory’ neural networks for time series prediction, Gomez-Git et. al, 2014.
    2. Dropout: A Simple Way to Prevent Neural Networks from Overfitting, Srivastava et. al 2014.
    3. Learning to forget, Gers, Schmidhuber & Cummins, 2000.
    4. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, Chung et. al, 2014.