forecastingprinciples.com Reviews of Important Papers on Forecasting,
1985-1995 Reviews
Review of:

R. Fildes and S. Makridakis, 1995, "The impact of empirical accuracy studies on time series analysis and forecasting," International Statistical Review, 63, 289-308.


Statisticians spend much time on the development of methods for time-series forecasting. We would hope that some of these procedures would prove useful for forecasting. To establish whether progress is being made, these methods need to be tested. According to Fildes and Makridakis (F&M), few statisticians even address whether their models can be applied to forecasting. In their survey of the literature, they estimated that about 21% of the time-series papers addressed forecasting issues. When they looked at the time-series forecasting papers published in the Journal of the American Statistical Association from 1971 to 1991, only about 10% looked at out-of-sample forecast accuracy, and only 11% made forecasting comparisons against reasonable alternative approaches. (Their working paper of the same title contains more detailed statistics on their citation studies.) In other words, time-series statisticians almost never conducted scientific evaluations of the procedures that they advocated.

Perhaps statisticians see their job as merely developing procedures, then rely on others to do the scientific validation. In fact, many studies have been done on the relative values of various forecasting procedures. These studies conclude that many of the assumptions used by statisticians are not valid (e.g. contrary to the statisticians’ belief, the fit to the sample data does not provide a good measure of predictive validity). Have these research findings led to substantive changes in the work undertaken by statisticians? Not at all. F&M argue that they proceed with their work as if nothing has been learned. As F&M state, this is consistent with Kuhn’s (1962) conclusion that scientists continue working with their paradigm even after much empirical evidence has discredited the espoused theory.

Fildes and Makridakis describe four of the most important validation studies. They concluded that statisticians do not cite this research. (I have also concluded that time-series textbooks typically fail to draw upon empirical research.) F&M examined a number of journals in which statisticians have published time-series studies and found that the major evaluations of forecasting procedures have seldom been cited (only three times per year, and these largely by empirical researchers who had published in the statisticians’ journals).

What is happening? Do statisticians not read these studies? Or do they chose to ignore this evidence? I suspect it is a little of each. Because much of the statisticians’ work is based on false assumptions about phenomena and about the efficacy of various procedures, and because the track record of sophisticated statistical models is poor, reading this material would be disconcerting to statisticians. It challenges the value of their work. Researchers tend to cite work that supports them and to ignore papers that conflict (for evidence on this, see Armstrong, 1996).

A substantial amount of empirical research has been conducted on forecasting methods and validation procedures since 1960. Is it possible to contribute to time-series forecasting if we ignore this body of research? I suspect that forecasting is like other areas. Science builds upon the shoulders of those who went before. Statisticians who work on forecasting, by and large, have failed to use prior empirical research even since the publication of Kuhn’s book in 1962.

F&M attempt to correct this state of affairs. They identify some areas where empirical findings clash with the procedures used by statisticians, such as the results on combining, sophisticated versus simple methods, and prediction intervals. Their list could have been expanded.

Their paper also summarizes procedures for comparisons of alternate methods. This should be especially useful for those who conduct research on time series and for those who review such work for publication. F&M also suggest ways that research on time series might be advanced.

This is an important paper, not for empiricists, but for statisticians. It is good to see that it was published in a journal for statisticians. It is a credit to the International Statistical Review that they published a paper that criticizes the type of work they publish. The criticism is done in a kindly spirit with the intent of improving the situation. It will be interesting to follow up in a few years to see if anyone is acting upon the findings.

References

Kuhn, T.S., 1962, The Structure of Scientific Revolutions (University of Chicago, Chicago).

Armstrong, J.S., 1996, Management folklore and management science – On portfolio planning, escalation bias, and such, Interfaces 26 (July – August), 25-55.