This paper looks at the various steps that should be taken when using econometric models to obtain forecasts for decision making. The steps Fildes outlines make sense and are not surprising. For example, he includes the step of developing a theoretical model prior to the operational model (or, in his terms, the data model). Also, he suggests that one should identify relevant prior research on the problem and integrate it into the model. So what, you say. Thats obvious. Well, if it is obvious, why is it so difficult to find published studies that use these steps? These are only examples; other key steps are also frequently omitted. Fildes illustrates this with an analysis of five recent studies from the Journal of the Operational Research Society. My own-literature searches support Fildes conclusion.
In the first part of the paper, Fildes describes most aspects of econometric model building and examines the treatment of these steps in five of the leading econometric texts. He also examines recent empirical studies. From the paucity of evidence on such important topics as does the method of estimation matter?, Fildes concludes that too few econometricians have devoted their energy to resolving questions that have important empirical consequences, or even establishing the importance of those questions that have concerned them. As a result, it is difficult to draw conclusions about which aspects of econometric model building are important to forecasters. For example, how do heteroscedasticity, autocorrelation, normality of errors, and simultaneous causality affect forecast accuracy? I have tended to follow Occams Razor: Keep it simple unless there is a significant amount of evidence that these problems affect forecast accuracy. Fildes is more moderate. He suggests that one follow the more sophisticated procedures, provided that this can be done at a reasonable cost and that the procedures will be acceptable to the client. For example, he advises that one should deal with autocorrelation even after he has pointed out that in three of four studies that addressed the issue directly, corrections for autocorrelation reduced forecast accuracy. (Unfortunately these references were not specifically identified in Fildes paper.)
One of the issues Fildes examines is the importance of obtaining accurate forecasts of the causal variables. In Armstrong (1985, pp. 241 242), I concluded that this was not important for short-range forecasts. Among the evidence I found was that ex-ante (unconditional) forecasts were more accurate than ex-post (conditional) forecasts in 10 of 13 comparisons (with one tie). 1 speculated that this occured because the forecasts of the causal variables were conservative. I have since found one study that favors ex-post forecasts [Rosenstone (1983)]. Fildes adds evidence from four recent studies; he did not consider any of the studies that were in Armstrong (1985). From this newer evidence, he concludes that it is important to obtain accurate forecasts of the causal variables. Unfortunately, he does not present the results from these four studies. He speculates that the ex-ante forecasts do well in some comparisons because the methods used in the comparisons have failed to hold all other things constant; in particular, the ex-ante forecasts may benefit from subjective adjustments.
The most important contribution of the paper, in my opinion, is not that it reveals our ignorance, but that it draws conclusions from the substantial body of work that has addressed important issues in econometric forecasting. Of particular interest is the issue of under what conditions do econometric forecasts provide better forecasts? In this area, there is a growing body of excellent work. Fildes does a meta-analysis of 68 studies on this issue. Of the 20 studies that he found on medium or long-range forecasting, econometric forecasts were superior to extrapolations on 15 and inferior on only 2. That is impressive evidence in favor of econometric models and it adds to what I reported in Armstrong (1985). He also finds evidence that econometric models are more accurate than extrapolations for ex-post short-range forecasting (22 to 11), with even more impressive evidence for ex-ante forecasts (11 to 4). These results conflict with Armstrong (1985), where I concluded no difference. I hope that we will see more study on this issue.