forecastingprinciples.com Reviews of Important Papers on Forecasting,
1985-1995 Reviews
Review of:

G.L. Riddington, (1993), Time varying coefficient models and their forecasting performance, Omega 21, 573-583.


Riddington examines an important problem. He provides a systematic evaluation of research on time-varying coefficients in forecasting. And he writes with much conviction. He states that his study "concludes conclusively that the [time varying coefficient models] approach significantly improves forecasting performance" and that "it should be automatically considered by any management scientist undertaking the modeling of causal relationships over time." He reached these conclusions by summarizing the results from 21 forecasting studies. In my opinion, he overstates his case.

Riddington’s evidence is based on ex post forecast accuracy, where the forecaster uses knowledge about the predictor variables from the forecast horizon. The use of ex post forecast evaluation is relevant to assessing how well models might work for predicting the effects of changes in policy variables. However, it is difficult to answer such questions as whether this procedure would lead to better decisions, so I believe that it is stating the conclusion too forcefully to say that this procedure should always be used.

As to the second conclusion, about the use of time varying coefficients in assessing causality, there is some hope that this might be true. What is needed is evidence showing that (1) time varying coefficient models provide more valid estimates, and (2) the improvements in validity are likely to have value for decision makers. Riddington does not address the issue of how this procedure relates to the quality of decisions.

If the time varying procedure provides substantially better parameter estimates, one would hope that this would also improve in ex ante forecasts, where one has no information from the period to be forecast. This is the situation faced by the manager. A common finding in this area is that refinements in the estimation of the parameters in econometric models do not contribute much to increased accuracy (Armstrong, 1985, pp. 225-232 reviews the empirical evidence on this issue). Riddington does not address this issue of ex ante forecast accuracy.

As to the review, it would have been useful to have had more details about the studies themselves and the way in which their results were coded. It would also have been helpful to have had details on how the search was conducted. Was the search reasonably complete? For example, Wildt and Winer’s (1983) review paper on application of time-varying parameter models in marketing listed 16 studies (their table 2). Of these 16 studies, only three were reviewed by Riddington. What was the explanation for ignoring the other studies? Dziechciarz (1989) provided an extensive review on this topic yet this was ignored by Riddington’s review. In general, it would have been helpful to know the explicit procedure by which the search was conducted, how they were screened, and what was the reliability of this search and screening process. The standards for the treatment of studies in a meta-analysis are parallel to those for the treatment of data in other studies.

As a general procedure, I recommend against using time varying coefficients procedures. They are harder to understand, more expensive, and may reduce the reliability of the model. They have not been shown to improve ex ante forecasts or decision making. On the other hand, I think this area has much promise. Surely there are some conditions under which variable coefficients are useful. I believe that the search for solutions has been misdirected, and that the problem should be reframed. It is not that the parameters change because of time. They change because of shifts in causal forces.

Strong, causal evidence that the parameters will change, or that they have recently changed, is unlikely to be found in the time series itself. If the structural changes are recent, then it is of particular importance to capture the change. However, in such cases one has only small samples (with, perhaps, unreliable data), and this may lead to false identification of changes in parameters. It would seem useful, then, to draw upon additional data. In particular, information from domain experts might be useful. Armstrong and Collopy (1993) found that domain knowledge could be easily obtained and coded. It produced substantial improvements in accuracy for extrapolation methods, and we expect that it might also be useful for econometric methods. For example, a company may introduce a superior product that will affect the parameters for an existing product. (It may no longer be effective to advertise the old product.) The area of "causal force-varying parameters" is one that deserves further attention. Riddington’s review and analysis should help in this endeavor.

References

Armstrong, J.S., 1985, Long-Range Forecasting. (John Wiley, New York).

Armstrong, J.S. and F. Collopy, 1993, Causal forces: Structuring knowledge for time series extrapolation, Journal of Forecasting, 12, 103-115.

Dziechciarz, J., 1989, Changing and random coefficient models: A survey, in: Peter Hackl, Statistical Analysis and Forecasting of Economic Structural Change, 217-251.

Wildt, A.R. and R.S. Winer, 1983, Modelling and estimation in changing market environments, Journal of Business, 56, 365-388.