A calibrated, cost-sensitive, explainable probability-of-default model on
Freddie Mac single-family loans — origination-only features, out-of-time tested.
How well does it rank risky loans?
AUC: the probability that a randomly chosen defaulted loan scores higher than a
randomly chosen good loan. 0.5 = random; 1.0 = perfect. Tested on 16,000 loans from 2022–2023
the model never saw. Error bars are 95% bootstrap confidence intervals.
Can you trust the probability numbers?
The dotted diagonal is perfect calibration — if the model says 10% default risk,
roughly 10% of those loans should actually default. Yellow shows the raw model; blue diamonds show
it after correction. Closer to the diagonal is better.
Performance curves across all thresholds
ROC: the trade-off between catching real defaults (true positive rate) and
over-flagging good loans (false positive rate). The dashed diagonal is random chance.
Switch to Precision-Recall for the harder test — with only 1.6% of loans defaulting,
the model must flag risk without too many false alarms.
Finding the right approval threshold
Expected portfolio loss at every possible cutoff. The green line marks the
cost-minimising threshold — approve loans below it, decline above. Drag the sliders to
change your loss severity and margin assumptions. The dashed line shows the
accuracy-maximising threshold, which ignores the asymmetry between error types.
What the threshold means in practice
The practical outcome of the cutoff selected to the left: approval rate,
defaults that still get funded, good loans turned away, and expected cost per loan
versus a naive 0.5 threshold.
Why each loan was flagged
For three loans at different risk levels, which factors the model weighted
most heavily — and by how much. Bar length shows relative influence. Under fair-lending law,
these signed factors are what a lender must cite when declining an application.
Which factors matter most overall?
Average influence of each input across all 16,000 test loans. Longer bar
means the model leaned on that factor more. The ranking should match financial
intuition: credit score, note rate, and DTI leading is expected.
Is the model finding real patterns?
Three tests a model trained on noise would fail: shuffle the labels
(performance should collapse toward 0.5), remove the top features (performance should drop),
and compare rankings to an independent model (they should agree). Subgroup AUC by vintage
and region checks that no group is dramatically underserved.