Information Technology Reference
In-Depth Information
manufacturers' data sets. Therefore, the automatic exponential smoothing, 20% exponential smoothing
and the 6 period window moving average all provide about the same performance.
The difference when including the Statistics Canada manufacturing data was much larger since the
patterns were stronger as a result of its aggregate nature and there were more data for the automatic
version to achieve better performance. The average error of the automatic exponential smoothing for
all three datasets is 0.6096 and the average for the fixed exponential smoothing of 20% is 0.6337 and
the difference has a significance of 0.00000002159. The moving average with a window of 6 periods
had an average error of 0.6288 and a significance of 0.0000005752 with the average of the automatic
exponential smoothing. We are impartial about which method is best when testing with only the two
manufacturer's data sets. However, when we included the Statistics Canada data set in the testing, there
was a significant difference in favor of the automatic exponential smoothing and consequently we would
identify this technique to be superior since there is added value at no loss.
In the case of the Statistics Canada dataset, the results were a little different; we found the MLR (Table
3 - Rank 3), SVM (Table 3 - Rank 4 and 5) and Theta (Table 3 - Rank 6) outperformed exponential
smoothing (Table 3 - Rank 7). However, because these approaches had such poor performance on the
chocolate and toner cartridge manufacturer datasets and because the performance gain by these over
the ES method was very small, we did not consider these results convincing. They may be the result of
the very large amount of data (12 years) and the aggregate nature of the data that was less noisy.
It is interesting to note that the trend approach (an informal way of planning by extrapolating that
a certain trend will continue in the future) was by far the worst forecasting approach since it always
ranked at the bottom of all three tables (Rank 21 and 22). Also, ARMA and most of the ML approaches
other than SVM showed a relatively poor performance.
Support Vector Machines using the Super Wide Model
The overall best performance was obtained using support vector machines in combination with the Super
Wide model. Since we have previously identified that the best traditional technique was automatic expo-
nential smoothing (it performed well on both manufacturers' data, as well as the aggregate manufacturing
data), we can calculate the forecast error reduction provided by the best ML approach. For the chocolate
manufacturer's dataset (Table 1 - Rank 2 and 5), we found a 6.70% ((0.8270 - 0.7717) / 0.8270) reduction
in the overall forecasting error and for the toner cartridge manufacturer dataset (Table 2 - Rank 1 and
5) we found a 3.11% ((0.6994 - 0.6777) / 0.6994) reduction in the overall forecasting error. In the case
of the Statistics Canada manufacturing dataset (Table 3 - Rank 2 and 7), we found a 10.00% ((0.5055
- 0.4547 / 0.5055) reduction in the forecasting error as compared to automatic exponential smoothing.
This was an average of 4.90% for our two manufacturers' dataset and an average of 6.61% for all three
as compared to automatic exponential smoothing. The performance of the Super Wide models has a
potential to improve further, if more products are included beyond the limit of 100 used in our study.
We will further examine in detail four major components of results; (1) cross- validation, (2) alternative
methods, (3) t-tests and (4) sensitivity analysis.
Cross-Validation
We tested two different support vector machine cross-validation-based parameter optimization proce-
dures: the windowed (time-oriented) approach and the standard approach. For the chocolate manufacturer
Search WWH ::
Custom Search