The validation of macro models should be performed by a different and independent party from the development team, according to the validation principles articulated in the MMS. If macro models are developed by a third party consultant, then a team within the institution or another consultant should therefore perform these independent validations.
5.10.2
Monitoring macro models may be challenging due to the low frequency of macroeconomic data. Institutions are expected to monitor the performance of macro models once a year. However, exceptional monitoring is required in the case of exceptional macroeconomic events.
5.10.3
Pre-implementation validation: This step involves the validation of the chosen macro models immediately after their development, prior to using them in production. The objective is to ensure that macro models meet a minimum level of quality and that they are fit for purpose. At a minimum, the following validation steps should be performed.
(i)
Development process: The validator should review the development process as per the principles articulated in the MMS.
(ii)
Replication: The validator should replicate the final chosen model per segment and ensure that the coefficients are correctly estimated.
(iii)
Statistical tests: The validator should ensure that statistical tests are correct, that cut-offs are reasonable and that statistical assumptions are correctly interpreted. This may necessitate partial replication. Additional statistical tests may be needed.
(iv)
Model sensitivity: The validator should measure the elasticity of the model output to changes in each input variable. The model user and validator should be aware of the presence of independent variables that dominates other variables in a given model.
(v)
Model stability: The validator should test the model stability, for instance by removing data points from the original time series (at the start or the end), re-run the regressions and re-project the dependent variable. The validator should also examine the stability of the model coefficients.
(vi)
Conclusion: When deemed appropriate, the validator can make suggestions for defect remediation to be considered by the development team.
5.10.4
Post-usage validation: This is otherwise referred to as back-testing, whereby the validator should compare the realized values of the dependent variable (e.g. PD, LGD, Credit Index) against the forecasted values based on the macroeconomic scenarios employed at the time of the forecast. A conclusion should be made based pre-defined confidence intervals.
5.10.5
Upon the post-usage validation, the validator should make a clear statement regarding the suitability of the model to be used for another cycle. When deemed appropriate, the validator can make suggestions for defect remediation to be considered by the development team.