A statistical test for forecast evaluation under a discrete loss function

Abstract

We propose a new approach to evaluating the usefulness of a set of forecasts, based on the use of a discrete loss function defined on the space of data and forecasts. Existing procedures for such an evaluation either do not allow for formal testing, or use tests statistics based just on the frequency distribution of (data , forecasts)-pairs. They can easily lead to misleading conclusions in some reasonable situations, because of the way they formalize the underlying null hypothesis that "the set of forecasts is not useful". Even though the ambiguity of the underlying null hypothesis precludes us from per-forming a standard analysis of the size and power of the tests, we get results suggesting that the proposed DISC test performs better than its competitors.