3.9 Validation of PD Models

3.9.1	Irrespective of their choice of methods, institutions should validate PD models according to the principles articulated in the MMS. In particular for PD models, both qualitative and quantitative assessments are required.
3.9.2	Institutions should ensure that the following metrics represent accurately the risk profile of their books at segment-level: TTC 1y PD, PIT 1y PD and PD term structure. For that purpose, these metrics should be validated at several granularity levels (e.g. rating grades, segments, industries). Statistical tests alone are generally not sufficient to conduct appropriate validation of PD at segment level. Consequently, institutions should combine statistical tests, judgement and checks across several metrics to ensure that the calibration of these metrics are meaningful and accurate.
3.9.3	Comprehensive techniques should be developed in order to validate PIT PDs. At a minimum, institutions and their supporting third parties should cover the comparisons articulated in the following table. This logical cross-check approach involves comparing variables estimated via several methods. In addition to these comparisons, institutions should design and compute other metrics to suit their specific PD methodology.
3.9.4	If insufficient data is available to estimate some of the metrics in the below table, institutions should demonstrate that actions are taken to collect data to produce these metrics. Given the lack of data, they should also explain their approach to assess the suitability of the PIT PD calibration currently used in production.
	Table 2: Metrics used to validate PD models

Segment level metrics	Point-in-Time metrics (PIT)	Through-the-Cycle metrics (TTC)
1-year Default Rate (“1y DR”)	PIT 1y DR are historically observed default rates per segment. Should be in a form a rolling time series, preferably with monthly intervals.	TTC 1y DR are computed as the average of PIT 1y DR through time. They can be weighted by the number of performing obligors in each time step.
Cumulative Default Rate (“CDR”)	PIT CDR are historically observed default rates over several performance windows, covering for instance 2, 3 and 4 years. The result should be several term-structures of defaults, observed at several points in time. Also computed per segment.	TTC CDR is the average of the CDR through time, per performance window, covering for instance 2, 3 and 4 years. The result should be a single PD term structure of default per segment.
1-year Probability of Default (“1y PD”)	PIT 1y PD estimated based on score-to-PD calibration and macro models. Estimated at segment level as the average across rating grades (exposure-weighted or count-weighted).	TTC 1y PD can be computed with several methods. For instance as: (i) weighted average PD based on the bank’s master scale, or (ii) if transition matrices are used, weighted average across the default column of the TTC matrix.
Cumulative Probability of Default (“CPD”)	Terms structure of PIT PD per segment and rating grades produced by models. Estimated per segment as the average across rating grades (exposure-weighted or count-weighted).	Not always available, depending on the methodology. In the case of transition matrices, it should be based on the TTC matrix computed over several time horizons.

3.9.5	Upon the estimation of the above metrics, institutions should perform the following comparisons at segment level. Institutions should implement acceptable limits to monitor each of the following comparison, i.e. the difference between each two quantities. These limits should be included in the model validation framework and agreed by the Model Oversight Committee. Frequent and material breaches should trigger actions as articulated in the governance section of the MMS.
	(i)	TTC 1y DR vs. TTC 1y PD per segment: The objective is to verify that the central tendency observed historically is in line with the PD estimated based on the institution’s master-scale.
	(ii)	PIT 1y DR vs. PIT 1y PD estimated over the same historical period, per segment: This is a back testing exercise. The objective is to verify that the default rates observed historically are falling within a reasonable confidence interval around the PD forecasted over the same period.
	(iii)	PIT 1y DR recently computed vs. PIT 1y PD estimated over the next 12 months: The objective is to verify that the default rate recently observed is coherent with the PD forecasted from the reporting date over the next 12 months. These two quantities can diverge due to the effect of economic forecasts. There should be a logical and intuitive link between the two and material differences should be explained.
	(iv)	TTC CDR vs. PIT CPD per segment: The objective is to verify that the shape of the cumulative default rates observed historically is similar with the shape of the cumulative default rate forecasted by the model from the portfolio reporting date. The shape can be expressed as a multiplier of the 1-year PD.
	(v)	TTC CDR vs. PIT PD derived analytically: A PD term structures can be estimated simply by using survival probabilities derived from the institution’s PD mater scale. This is referred as the analytical PD term structure, that serves as a theoretical benchmark. The objective is to compare this analytical benchmark vs. (a) observed CDR and (b) the PD term structure generated by the model. Material deviations should be understood and documented. If CDR and CPD are materially lower than the analytical approach, adjustments should be considered.
3.9.6	In addition to segment level validation, institutions should ensure that the PIT PD profile across rating grades is logical and consistent. This is particularly relevant in the case of transition matrices. PIT adjustments should be coherent across different ratings. Technically, for a given segment and a given horizon forecast, the log-odd ratio of the PIT PD for a given rating over the TTC PD for the same rating, should be of similar magnitude between ratings.
3.9.7	Finally, economic consistency between segments is also part of the validation process of PD models. Institutions should ensure that such considerations are included in the scope of model validation. PIT PDs generated by models should be coherent between industries and between segments. For instance, if a given portfolio displayed high historical PD volatility, then such volatility is expected to be reflected in the forecasted PIT PD. Material deviations from coherent expectations should be explained and documented.