10.4.1
| The quantitative validation must assess the suitability of the model output with respect to the objective initially assigned to the model. This process must rely on numerical analyses to derive its conclusions. Such validation should include a set of dedicated research to arrive at an independent judgement. Under certain circumstances, partial model replication and/or a challenger model may be necessary to form a judgement.
|
10.4.2
| The set of metrics employed for model validation must at least include those employed for monitoring. As a first step, the validator must make a review of the monitoring reports and their observations. In addition, institutions should employ a broader spectrum of performance metrics to fully assess model performance, since the scope of the validation process is larger than that of monitoring.
|
10.4.3
| The assessment of model performance must cover, at a minimum, the following components, applicable to both statistical and deterministic models:
|
| | (i)
| Accuracy and conservatism: The ability of a model to generate predictions that are close to the realised values, observed before and after the model development phase. For models whose results are subject to material uncertainty, the validator should assess if sufficient conservatism included in the model calibration. |
| | (ii)
| Stability and robustness: Whilst there are theoretical differences between stability and robustness, for the purpose of this MMS, this refers to the ability of a model to withstand perturbations, i.e. maintain its accuracy despite variability in its inputs or when the modelling assumptions are not fully satisfied. In particular, this means the ability of a model to generate consistent and comparable results through time. |
| | (iii)
| Controlled sensitivity: This relates to the model construction. Model sensitivity refers to the relationship between a change in the model inputs and the observed change in the model results. The sensitivity of the output to a change in inputs must be logical, fully understood and controlled.
|
10.4.4
| The quantitative validation process should include a review of the suitability, relevance and accuracy of following components.
|
| | For both statistical and deterministic models: |
| | (i) | The implementation, |
| | (ii) | The adjustments and scaling factors, if any, |
| | (iii) | The ‘hard-coded’ rules and mappings, |
| | (iv) | The extrapolations and interpolations, if any, and |
| | (v) | The sensitivities to changes in inputs, |
| | In addition for statistical models only: |
| | (vi) | The model coefficients, |
| | (vii) | The statistical accuracy of the outputs, |
| | (viii) | The raw data as per the DMF requirements, and |
| | (ix) | The historical time series, |
| | In addition, for deterministic models only: |
| | (x) | A decomposition of the model drivers and their associated sensitivity, and |
| | (xi)
| A partial replication, when possible.
|