Using only information available at loan origination, how accurately and how well-calibrated can we predict the probability that a single-family mortgage becomes seriously delinquent (90+ days past due) within a fixed -month horizon — and what does a calibrated, cost-sensitive model imply for lending cut-offs and adverse-action reason codes?
What would falsify the thesis. If, on the strictly out-of-time test set, the gradient-boosted model cannot beat the transparent WOE scorecard on rank-ordering (ΔAUC ≤ ~0.01 with overlapping bootstrap CIs) and cannot be calibrated to ECE < ~0.02, then "ML adds underwriting value here" is false and the honest conclusion is that a regulated scorecard suffices. Unstable or theory-contradicting SHAP also fails the explainability claim. This report states the verdict either way — a null is a valid result.
Freddie Mac Single-Family Loan-Level Dataset (origination attributes + monthly performance for
fixed-rate mortgages). Two files per vintage: origination (one row/loan) and performance (many
rows/loan). We train/validate on originations and test
out-of-time on — a cohort the model never saw, under a different
macro regime. LendingClub is excluded (platform retired; the public downloads were removed; severe
selection). See the
dataset page and SOURCES.md.
Label. y = 1 if the loan reaches current-delinquency-status ≥ 3 (90+ DPD)
or a default zero-balance code (short-sale/charge-off 03, REO 09)
within the horizon of first payment; y = 0 if it survives the window. Prepayment
(zero-balance 01) before the event is a competing risk: flagged and labelled
y=0 for the binary head, then re-examined with a survival model in §7. Sentinels
(credit_score 9999, DTI 999, …) are mapped to NA before any numeric use. Loans without a full
observation window and no event are dropped for that horizon (never mislabelled as non-events).
Default rate by vintage (sanity: crisis books default far more than benign ones):
Two estimators, because the comparison is the point. Benchmark: an L2-regularised
logistic regression on monotonic Weight-of-Evidence bins of the main drivers (credit score,
CLTV, DTI) plus the other origination features — the regulated-industry yardstick and the
calibration baseline. Primary: XGBoost, tuned by vintage-blocked (time-aware) CV, with
scale_pos_weight for the low default base rate. Features are origination-only; an
automated test asserts the feature set is disjoint from the performance fields (no leakage).
Headline metrics come from the OOT test set, not random CV. PR-AUC and KS are reported alongside AUC because defaults are rare. CIs are i.i.d. bootstrap (loans are cross-sectional).
A risk model that ranks well but is mis-calibrated prices loans wrong. We fit isotonic and Platt recalibration on a held-out slice of the training data (never used to fit the GBM) and measure the reliability curve, Brier score, and Expected Calibration Error on OOT, before and after.
Accuracy is the wrong objective for a lender. With an explicit cost matrix — expected loss of a funded bad (LGD × EAD) vs forgone margin on a rejected good (margin × EAD) — we derive the expected-cost-minimising approve/decline threshold and the full cost curve. The dashboard lets you move LGD/EAD/margin and watch the optimal cut-off move.
Global importance and per-loan reason codes use exact tree SHAP. The top drivers should be economically sensible (credit score, CLTV, DTI) and stable across seeds. Worked adverse-action examples list the features pushing each loan's PD up — exactly the ECOA/Reg B "principal reasons" an adverse-action notice must give.
This is an explicitly predictive exercise; there is no causal claim.
(a) Permutation — shuffling the labels must collapse OOT AUC to ~0.5. (b) Drop top-3 SHAP features — performance must degrade. (c) Competing risk — a discrete-time hazard with prepayment as a competing risk should rank loans consistently with the binary model. (d) Subgroups — performance by vintage and census region, to check the headline isn't carried by one regime.