Comparing Classical and Machine Learning Algorithms for Time Series Forecasting

1. Comparing Classical and Machine Learning Algorithms
for Time Series Forecasting
Yifan Wang

2. Time series forecast problem can be
multivariate, LSTMs can be used to model
univariate and multivariate time series
forecasting problems.
LSTM Neural network
Sales forecasting is one of the most
fundamental and common analytics need
for business. By accurately estimating
future sales, companies to make informed
business decisions and predict short-term
and long-term performance.
Sales Forecasting
Classical Method
Simple Feed-forward Neural Network
1
Table of Content
Exponential smoothing and autoregressive
integrated moving average (ARIMA) are
most common methods for univariate time
series.
In this comparison, we use nnetar() function
from forecast package in R.
Feed-forward neural networks with a single
hidden layer and lagged inputs for forecasting
univariate time series.

3. 2
Measurement
Gross Sales & Net Sales
Challenges
The Return & Net Sales
Objectives
Forecast future 12 Month
Product Categories
Digital & Traditional
Sales Forecasting: The Time Series Data
In this example, we have a monthly sales
data from a business sell their products
via distributors. (Gross Sales)
For this industry, the distributors can
also return unsold products within a
negotiated time frame:
Gross Sales – Return = Net Sales
There are two products: the new
emerging Digital Product and the legacy
Traditional Product.
From the historical data, we can see
these two products perform differently
for past 10 years.
Because of the nature of industry, the
gross sales for both product categories
have very stable seasonality.
However, because of the return, the net
sales are more challenging to forecast.
The objective is to forecast the sales of
future two years and also the remaining
month of the current year.
e
Digital Product Gross Sales Traditional Product Gross Sales
Digital Product Net Sales Traditional Product Net Sales

4. 3
Digital Product Gross Sales Forecast Traditional Product Gross Sales Forecast
Digital Product Net Sales Forecast Traditional Product Net Sales Forecast
Classic Method: ETS Model
Model Fitted vs Actual Model Fitted vs Actual

5. 4
Digital Product Gross Sales Forecast Traditional Product Gross Sales Forecast
Digital Product Net Sales Forecast Traditional Product Net Sales Forecast
Classic Method: ARIMA Model
Model Fitted vs Actual Model Fitted vs Actual

6. 5
Digital Product Gross Sales Forecast Traditional Product Gross Sales Forecast
Digital Product Net Sales Forecast Traditional Product Net Sales Forecast
Machine Learning Algorithms : Feedforward Neural Network
Model Fitted vs Actual Model Fitted vs Actual

7. Traditional
Gross Sales
Traditional
Net Sales
Digital
Gross Sales
Digital Net
Sales
ME -637038.10 -1770053.00 -380326.10 -155923.20
RMSE 7485097.00 11024208.00 4418118.00 5084155.00
MAE 4894202.00 8021170.00 3061446.00 3741741.00
MPE -3.34 61.18 -2.27 -1.76
MAPE 10.35 110.42 13.21 58.16
ACF1 -0.18 0.15 0.02 0.18
6
14.57 16.67 2.43
Model Performance Comparison
Traditional
Gross Sales
Traditional
Net Sales
Digital
Gross Sales
Digital Net
Sales
ME -594759.10 -916118.10 3686.56 353.98
RMSE 9136902.00 10307341.00 4237217.00 4358207.00
MAE 6132050.00 7248800.00 3001718.00 3166288.00
MPE 7.98 49.67 -6.51 -18.24
MAPE 20.05 108.67 15.21 28.75
ACF1 0.15 0.14 0.00 0.00
Traditional
Gross Sales
Traditional
Net Sales
Digital
Gross Sales
Digital Net
Sales
ME -999.89 17489.41 -1622.06 5287.69
RMSE 1884502.00 1739840.00 441536.00 514144.60
MAE 1345289.00 1240022.00 310168.40 335859.80
MPE -0.69 5.83 -0.41 -1.65
MAPE 4.20 17.39 2.43 4.51
ACF1 -0.05 0.07 -0.00 0.10
ETS Model
The Best MAPE
ARIMA Model
The Best MAPE
nnetar Model
The Best MAPE

8. 8
e
Mean SD
ME 656426.89 2710422.20
RMSE 9543108.05 961744.88
MAE 6711391.31 813266.35
MPE -1.90 3.16
MAPE 16.27 1.48
ACF1 0.18 0.17
Theil's U 0.26 0.08
Mean SD
ME 165959.29 1067295.57
RMSE 4660559.64 1250324.59
MAE 3431855.56 869998.87
MPE -8.44 17.99
MAPE 28.54 18.76
ACF1 -0.09 0.22
Theil's U 0.38 0.08
Mean SD
ME -199338.85 2133135.44
RMSE 10506026.58 1650492.68
MAE 7698473.24 789304.16
MPE 17.34 27.47
MAPE 53.49 36.58
ACF1 0.02 0.15
Theil's U 0.18 0.05
Mean SD
ME 201135.86 1497058.44
RMSE 4646052.77 628516.22
MAE 3475533.26 602313.80
MPE -0.21 6.83
MAPE 15.27 3.87
ACF1 0.02 0.15
Theil's U 0.40 0.07
Digital Product
Gross Sales
Traditional Product
Gross Sales
Digital Product
Net Sales
Traditional Product
Net Sales
Did nnetar model overfitted? 5 Fold Cross Validation

9. The cross validation result from nnetar model, which measure against the test data that model
have never seen, shows the model perform not as good at the training set. however, it still
perform better than the training set with ETS and ARIMA.
Theil’s U (less than 1), indicate that the forecasting technique is better than guessing.
When it equal to 1, the forecasting technique is about as good as guessing.
If it is above 1, then the forecasting technique is worse than guessing.
Traditional
Gross Sales
Traditional
Net Sales
Digital
Gross Sales
Digital Net
Sales
ME 656426.89 -199338.85 201135.86 165959.29
RMSE 9543108.05 10506026.58 4646052.77 4660559.64
MAE 6711391.31 7698473.24 3475533.26 3431855.56
MPE -1.90 17.34 -0.21 -8.44
MAPE 16.27 53.49 15.27 28.54
ACF1 0.18 0.02 0.02 -0.09
9
Model Performance Comparison
Traditional
Gross Sales
Traditional
Net Sales
Digital
Gross Sales
Digital Net
Sales
ME -637038.10 -1770053.00 -380326.10 -155923.20
RMSE 7485097.00 11024208.00 4418118.00 5084155.00
MAE 4894202.00 8021170.00 3061446.00 3741741.00
MPE -3.34 61.18 -2.27 -1.76
MAPE 10.35 110.42 13.21 58.16
ACF1 -0.18 0.15 0.02 0.18
Traditional
Gross Sales
Traditional
Net Sales
Digital
Gross Sales
Digital Net
Sales
ME -594759.10 -916118.10 3686.56 353.98
RMSE 9136902.00 10307341.00 4237217.00 4358207.00
MAE 6132050.00 7248800.00 3001718.00 3166288.00
MPE 7.98 49.67 -6.51 -18.24
MAPE 20.05 108.67 15.21 28.75
ACF1 -0.01 -0.02 -0.02 0.09
ETS Model
Training Set
ARIMA Model
Training Set
nnetar Model
Testing Set (5 folds cross validation)
Traditional
Gross Sales
Traditional
Net Sales
Digital
Gross Sales
Digital Net
Sales
ME -999.89 17489.41 -1622.06 5287.69
RMSE 1884502.00 1739840.00 441536.00 514144.60
MAE 1345289.00 1240022.00 310168.40 335859.80
MPE -0.69 5.83 -0.41 -1.65
MAPE 4.20 17.39 2.43 4.51
ACF1 -0.05 0.07 -0.00 0.10
nnetar Model
Training Set

10. 10
Multivariate Time Series
Digital Product
Gross Sales
Traditional Product
Gross Sales
Digital Product
Net Sales
Traditional Product
Net Sales
Historic Data Future Data
Digital Product
Gross Sales
Traditional Product
Gross Sales
Digital Product
Net Sales
Traditional Product
Net Sales
Historic Data Future Data
Univariate Time Series Multivariate Time Series

11. Time X1 X2 X3 X4 Y1
t-12 Var1(t-12) Var2(t-12) Var3(t-12) Var4(t-12) Var4(t)
t-11 Var1(t-11) Var2(t-11) Var3(t-11) Var4(t-11) Var4(t+1)
t-10 Var1(t-10) Var2(t-10) Var3(t-10) Var4(t-10) Var4(t+2)
t-9 Var1(t-9) Var2(t-9) Var3(t-9) Var4(t-9) Var4(t+3)
t-8 Var1(t-8) Var2(t-8) Var3(t-8) Var4(t-8) Var4(t+4)
t-7 Var1(t-7) Var2(t-7) Var3(t-7) Var4(t-7) Var4(t+5)
t-6 Var1(t-6) Var2(t-6) Var3(t-6) Var4(t-6) Var4(t+6)
t-5 Var1(t-5) Var2(t-5) Var3(t-5) Var4(t-5) Var4(t+7)
t-4 Var1(t-4) Var2(t-4) Var3(t-4) Var4(t-4) Var4(t+8)
t-3 Var1(t-3) Var2(t-3) Var3(t-3) Var4(t-3) Var4(t+9)
t-2 Var1(t-2) Var2(t-2) Var3(t-2) Var4(t-2) Var4(t+10)
t-1 Var1(t-1) Var2(t-1) Var3(t-1) Var4(t-1) Var4(t+11)
t Var1(t) Var2(t) Var3(t) Var4(t) Var4(t+12)
t+1 Var1(t+1) Var2(t+1) Var3(t+1) Var4(t+1) Var4(t+13)
t+2 Var1(t+2) Var2(t+2) Var3(t+2) Var4(t+2) Var4(t+14)
t+3 Var1(t+3) Var2(t+3) Var3(t+3) Var4(t+3) Var4(t+15)
t+4 Var1(t+4) Var2(t+4) Var3(t+4) Var4(t+4) Var4(t+16)
t+5 Var1(t+5) Var2(t+5) Var3(t+5) Var4(t+5) Var4(t+17)
1a
11
Advanced Model: Long short-term memory (LSTM)
Configure LSTM Model
An Recurrent Neural Network
Data Preparation Training Model
150 epochsSupervised Learning Dataset
MAPE vs validation MAPE

12. Traditional
Gross Sales
Traditional
Net Sales
Digital
Gross Sales
Digital Net
Sales
ME
RMSE 2207332.06 2123658.12 7831121.66 6760654.91
MAE 1509378.75 1567319.13 6734541.00 5577573.50
MPE
MAPE 7.92 5.30 12.24 13.56
ACF1
Traditional
Gross Sales
Traditional
Net Sales
Digital
Gross Sales
Digital Net
Sales
ME
RMSE 3553031.81 3535456.35 3478618.88 3656988.30
MAE 2190123.75 2435441.00 2561047.75 2594779.50
MPE
MAPE 6.84 8.25 9.08 9.33
ACF1
Traditional
Gross Sales
Traditional
Net Sales
Digital
Gross Sales
Digital Net
Sales
ME 656426.89 -199338.85 201135.86 165959.29
RMSE 9543108.05 10506026.58 4646052.77 4660559.64
MAE 6711391.31 7698473.24 3475533.26 3431855.56
MPE -1.90 17.34 -0.21 -8.44
MAPE 16.27 53.49 15.27 28.54
ACF1 0.18 0.02 0.02 -0.09
4
nnetar vs LSTM Model
LSTM Model
Testing Set
nnetar Model
Testing Set (5 folds cross validation)
Traditional
Gross Sales
Traditional
Net Sales
Digital
Gross Sales
Digital Net
Sales
ME -999.89 17489.41 -1622.06 5287.69
RMSE 1884502.00 1739840.00 441536.00 514144.60
MAE 1345289.00 1240022.00 310168.40 335859.80
MPE -0.69 5.83 -0.41 -1.65
MAPE 4.20 17.39 2.43 4.51
ACF1 -0.05 0.07 -0.00 0.10
nnetar Model
Training Set
LSTM Model
Training Set
With training dataset, the LSTM model does
not outperform nnetar model, but the
performance are more consistent across
different measurements.
The cross validation result from LSTM model,
which measure against the test data that
model have never seen, shows the model
perform much better than nnetar model with
test dataset.
MAE vs validation MAE

13. Advanced Model: Long short-term memory (LSTM)
Digital Product Gross Sales Forecast Traditional Product Gross Sales Forecast
Digital Product Net Sales Forecast Traditional Product Net Sales Forecast
Model Fitted vs Actual Model Fitted vs Actual

14. Simple to implement
Require re-requisites (stationarity,
no level shifts)
Fast to run
Limited performance
Simple to implement
No pre-requisites (stationarity, no
level shifts)
Can model non-linear function
with neural networks
Univariate model
Structure supervised training
dataset and tune the model can
be difficult
No pre-requisites (stationarity, no
level shifts)
Can model non-linear function
with neural networks
univariate and multivariate time
series
Final Conclusion
ETS & ARIMA nnetar model LSTM model

15. THANK YOU