The model and macroeconomic variable selection should be based on clearly defined performance criteria using a transparent selection algorithm. The final model should be able to (i) generate values that fit the historical values of the dependent variable and (ii) generate accurate predictions.
5.9.2
For each segment, institutions should choose a final model from the list of candidate models generated from the model construction step. Statistical performance should not be the only decisive factor to choose a model. Instead, the choice of the final model should be based upon the combination of various factors. At a minimum, institutions should use the criteria outlined below. It is essential that institutions include all these criteria in the selection process. The absence of one criteria could be materially detrimental to the choice of the most relevant model.
(i)
Statistical performance:
a.
The chosen model should meet minimum requirements of performance, statistical stability and robustness as shown by the statistical indicators and their associated thresholds. Model parameters and forecasts should remain stable over time.
b.
In addition, at the model development stage, it is important to examine the stability of models: out-of-sample performance and in-sample fit should be tested and compared across candidate models. A common metric employed to express model performance is the root mean square error, for which limits should be established.
(ii)
Model sensitivity: Quantitative response of the dependent variable to independent variables should be meaningful and statistically significant - both quantitatively and qualitatively. This can be examined through simulating one standard deviation change in individual dependent variables or by considering the forecast differences across alternative scenarios.
(iii)
Business intuition: The chosen model should be constructed with variables and relationships that meet logical business and economic intuitions. This means that the model should be explained by causal relationships.
(iv)
Realistic outcomes: Projected values should be consistent with historical observations and meet economic intuition. Any material jump and/or disconnect between historical values and forecasted should be explained.
(v)
Implementation: When choosing a model, institutions should be mindful of the implementation and maintenance constraints, which should form part of the choice of the most appropriate models. For instance, some variables may not be available as frequently as expected for forecasting. Also, some model formulations may require autoregressive terms that need specific treatment during implementation.
5.9.3
In order to test the business intuition, for each candidate model, institutions should forecast the dependent variables (e.g. PD, Credit Index) under a severe downside scenario. The outcome will therefore be a range of projected dependent variables (one for each model) under the same scenario. It may become apparent that some candidate models should be excluded as they generate outputs that deviate too much from economic and business expectations.
5.9.4
Forecast Uncertainty: Projected forecast are based on mean or median values, around which uncertainty (i.e. confidence interval) inherently exists. Institutions should ensure that the model forecast uncertainty are clearly estimated, documented and reported to the Model Oversight Committee. In the context of time series regression, the confidence interval around the mean can be estimated empirically or based on the standard deviation of the residuals under the assumption of normally distributed residuals.