In many situations no objective criterion is available by which the forecasts and decisions can be judged. Instead, consensus is used as a proxy for accuracy. Is this valid? Little direct study of this issue has been done. Holtzman and Sells (1954) in a study on the success of aviators, found little relationship between consensus and actual outcome. Ogburn (1934) concluded that high agreement among sportswriters was not closely related to accuracy. However, a re-analysis of data from Walker (1970) by Armstrong (1985, p. 145) showed a relationship between consensus and accuracy.
Ashtons study helps to fill this gap in the literature. She examined two different forecast situations: forecasts of annual advertising sales for Time magazine by 13 Time, Inc. executives given forecast horizons for 1, 2 and 3 quarters and covering 14 years; and forecasts by 25 auditors of going concern problems, like bankruptcy, for 40 firms. Using two criteria, correlations and mean absolute differences, Ashton compared the actual degree of agreement (between forecasts by two different judges) against the accuracy of the judges. She also compared the agreement of judges (among all other judges) and related it to that judges accuracy. As she says, the results are good news. Agreement among judges does imply greater accuracy. The relationships were strong and statistically significant. This gives a bit of confidence for using consensus as a proxy for accuracy when formal assessment of accuracy is not feasible. One hopes additional research will suggest rules-of-thumb for forecasting in various situations because many studies lead one to question this relationship. Also, would it be possible to use measures of consensus to assess the uncertainty about a given forecast?