[Review written with Fred Collopy]
Under what conditions does domain (contextual) knowledge improve short-range forecasts? Does training in forecasting methods (including such things as how to treat outliers, how to identify trends and cycles, and how to adjust for judgmental biases) improve ones judgmental forecasts? Sanders and Ritzman address these and other issues in this important study. First, they organized the literature on the use of judgement and extrapolation for short-range forecasting. They examined: (1) making judgmental adjustments to statistical extrapolations, (2) comparing judgmental and statistical extrapolations, (3) using judgment and as input to statistical extrapolations. From this literature, they formulated four hypotheses. Of particular interest, Hypothesis 1 states "Judgmental forecasts from practitioners with contextual knowledge are significantly more accurate than those from subjects without such knowledge." Hypothesis 2 states "Judgmental forecasts from subjects with technical knowledge about forecasting are not significantly more accurate than those from subjects without such knowledge." They also examined these hypotheses when considering the level of variability in the data.
Sanders and Ritzman tested the hypotheses using three years of daily data from 22 time series from a national public warehouse. Some series were relatively stable, while others fluctuated widely. The series displayed few trends. Where appropriate, Sanders and Ritzman made seasonal adjustments.
They obtained judgmental forecasts from 81 undergraduates, 49 of whom were in two sections of an elective course in forecasting. Each student was randomly assigned to four of the 22 series, and each student produced 65 one-day ahead forecasts with feedback after each period. All participants for this formidable task were volunteers. Sanders and Ritzman found that accuracy did not decrease over time, despite the length of the task.
The practitioner forecasts were prepared by the warehouses first line supervisor in consultation with the warehouse manager. These people had no formal training in forecasting methods.
The statistical forecasts were based on an equally weighted combination of forecasts from three commonly used methods: single exponential smoothing, Holts two-parameter smoothing model, and the adaptive estimation procedure. Successive updating was used and the forecast level was started from the last observation.
All of the forecasting methods were superior to the naive (no change) forecast. As expected, the combined forecast was superior to the typical component. In fact, it was superior to the best of the three components; Theils U2 for the combined forecast was 0.76, while it was 0.87 for single exponential forecasting, the best of the components.
Strong support was provided for Hypothesis 1. The practitioner forecasts were much more accurate using either the mean absolute percentage error (MAPE) or the median APE (MdAPE). The differences were statistically significant, and they were particularly large for series with much variability. This result suggests that some forecasters might be acting rationally in their reliance upon judgmental methods (Sanders and Manrodt, forthcoming).
Hypothesis 2 was also supported. Training in forecasting methods did not improve the forecast accuracy, unfortunately. Because various forecasting procedures have been shown to improve forecast accuracy, we conclude that the students are not able to apply what they learn. This conclusion supports an old finding in studies of education. For example, Culotta (1992) reports on a large National Science Foundation grant that was motivated by the finding that even students who do well in calculus courses cannot apply what they learned.