Probability of Default: Calibration, Cost, and Reason Codes

Every mortgage lender that turns someone down is required by Regulation B to tell the applicant why, in plain language, citing the specific factors that hurt the application most. That shapes what a credit model actually has to do. It cannot just sort applicants from riskiest to safest. It has to produce honest probability estimates so the approval cutoff sits at the right place for the lender’s cost structure, and it has to explain its decisions clearly enough for a fair-lending examiner to follow. This project builds a probability-of-default model on Freddie Mac single-family originations that treats all three outputs as primary deliverables, not appendices bolted on after the fact.

The data is real: 96,000 Freddie Mac mortgage originations spanning 2012 through 2023, drawn from the Single Family Loan Level Dataset under Freddie Mac’s research licence. The raw loan files never enter version control. Only derived, aggregated outputs do.

Reading the dashboard

The dashboard has eight panels, most of them interactive. Here is what each one shows and what to look for.

How well it ranks risky loans. The first panel compares two models on a metric called AUC. Think of AUC as the probability that, if you pick one loan that eventually defaulted and one that did not, the model would correctly assign the higher risk score to the defaulted loan. A value of 0.5 is random guessing. A value of 1.0 is perfect. Both models here score around 0.745. The grey bar is a traditional scorecard, the kind banks have used for decades. The blue bar is a gradient-boosted machine, a more modern approach. Their confidence intervals overlap completely, which is the honest result: on clean, well-structured mortgage data, a sophisticated model does not dramatically outperform a well-built baseline. All of these numbers come from 16,000 loans originated in 2022 and 2023 that played no role in training.

Can you trust the probability numbers? Ranking and calibration are separate questions. A model can correctly identify that loan A is riskier than loan B while assigning completely wrong probabilities to both. This panel plots the model’s predicted default rate against the actual default rate observed in each score bucket. The dotted diagonal is perfect calibration: if the model says 5%, the observed rate should be around 5%. The yellow line shows the raw model, which runs well above the diagonal, overstating risk. The blue diamonds appear when you select a calibration method and show how the correction brings predicted probabilities much closer to observed outcomes. The ECE number below the chart quantifies this: the raw model has an ECE of 0.534, meaning its predictions are off by an average of 53 percentage points. After calibration, ECE falls to 0.022.

Model performance curves. Two standard visualizations, switchable via the buttons. The ROC curve shows what happens at every possible threshold: moving right, you flag more loans as risky, catching more real defaults but also generating more false alarms on good loans. The dotted diagonal is chance. The area under this curve is the AUC shown in the first panel. Precision-recall is the harder test when defaults are rare. It asks: of the loans the model flags as high-risk, what fraction actually default? With only 1.6% of loans defaulting, a model that flags everything correctly misses most of the picture.

Finding the right approval threshold. The model outputs a probability for each loan, but the actual lending decision is binary: approve or decline. This panel shows expected portfolio loss at every possible threshold. The yellow curve dips to a minimum, marked by the green vertical line, then rises in both directions. Move the threshold too low and you reject too many profitable loans; too high and too many defaults slip through. Drag the LGD and margin sliders to change the assumed loss severity and interest spread. The optimal threshold recalculates from a pre-built grid rather than interpolating, so the numbers are exact at each grid point.

What the threshold means in practice. Once a cutoff is selected, this panel shows the practical outcome: what share of applications get approved, how many eventual defaults still get funded, how many good loans get turned away, and how the expected cost per loan compares to simply using a 0.5 threshold. The cost difference sounds small in dollar terms here because the optimal threshold is high and the default rate is low, meaning the model declines very few loans under these cost assumptions.

Why each loan was flagged. For three example loans at low, medium, and high model-estimated risk, this panel shows which specific factors pushed the score up most. The bars represent SHAP values: signed contributions from each feature to the model’s output. Longer bars mean that feature had more influence on that particular loan. Under Regulation B and the Equal Credit Opportunity Act, the reasons cited in an adverse-action notice must be the actual factors the model used, in rank order. These are those reasons. They describe what the model weighted, not what caused the borrower’s financial situation.

Which factors matter most overall. Across all 16,000 test loans, the average absolute SHAP value for each feature. The chart is sorted by importance. Credit score leads, followed by the note rate and debt-to-income ratio. This matches what practitioners would expect: borrowers with higher credit scores, lower debt burdens, and lower interest rates default less often. The footnote reports stability across different random seeds, which checks that the importance ranking is not an artifact of a particular run.

Is the model finding real patterns? Three robustness checks. First, the labels are shuffled eight times and the model’s performance is measured each time. If the model was picking up genuine signal, performance should collapse toward 0.5 on random labels. It does: the mean permutation AUC is 0.467. Second, the three most important features are removed and the model is refitted. A significant drop in AUC confirms those features were doing real work, not just correlated free-riders. Third, a separate model fitted on monthly payment history rather than origination data ranks the same loans. Agreement between those rankings and the origination-time model suggests the risk ordering is stable across modelling approaches.

Full report

---

Defining default

Defining default sounds simple until you look at the data. Mortgages go delinquent, catch up, go delinquent again, get modified, and eventually resolve over years. Counting any missed payment as a default floods the training set with noise. Waiting for final resolution means some loans never receive a label at all.

The definition used here: a loan is a default if, within 24 months of the first payment date, it reaches 90 or more days past due, or exits the dataset with a zero-balance code consistent with loss (code 03 for foreclosure or short sale, code 09 for REO disposition). A loan that exits because the borrower paid off or refinanced (code 01) before hitting that threshold is a competing risk. It is labelled as a non-event and flagged separately. Competing-risk loans are not defaults and not censored observations. The discrete-time hazard robustness check described later accounts for them explicitly.

Loans with fewer than 24 months of observable history are excluded from both the training and test sets. They are unobservable, not survivors, and including them would bias the base rate downward. The 1.6% base rate in the test set reflects only loans with complete windows.

The train-test split is out of time throughout. Models train on 2012 through 2021 originations. The headline metrics come from 2022 and 2023. No test loan appears in any training fold, and no temporal information from the test period influences any model parameter.

What the model can and cannot use

The feature set is origination-only. Credit score, combined loan-to-value ratio, debt-to-income ratio, unpaid principal balance, original interest rate, loan purpose, occupancy type, property type, number of units, first-time homebuyer status, and census region. Nothing that arrives after the loan closes enters a feature column.

This matters more than it sounds. The performance file contains hundreds of months of payment history for each loan. A feature like current delinquency status or remaining balance at any post-origination date would encode the outcome being predicted. A pipeline check asserts that no field from the performance file appears in the feature manifest before training is allowed to run.

Three origination fields carry sentinel values that look like real data but are not. Freddie Mac uses 9999 for a missing credit score and 999 for missing DTI and CLTV. Those are replaced with null before any numeric processing. A model trained on 9999 as a number would learn a signal from the sentinel code itself.

Two models: a baseline and a challenger

The first model is a regularised logistic regression on weight-of-evidence transformed features, the standard scorecard approach. Each numeric feature is divided into bins, the bins are adjusted to enforce monotonic direction (higher credit score should always lower the predicted probability; higher CLTV should always raise it), and each bin is replaced by the log-odds of default observed within it. The result is a model that is approximately additive in a way a loan officer can follow, where every factor contributes independently to the final score.

The scorecard is the benchmark. Its OOT AUC on the 2022-2023 test set is 0.747.

The second model is XGBoost, fitted with time-aware cross-validation where each fold validates on the three most recent training vintages. The best configuration from a grid over depth and learning rate is a shallow tree at a slow learning rate, which is the typical pattern for structured tabular data where overfitting is the main risk. Scale-positive-weight handles class imbalance without resampling.

The GBM OOT AUC is 0.745. The delta is negative: the scorecard is marginally better. The confidence intervals overlap entirely. This is the correct result to report, not a failure. On clean, information-rich mortgage origination data with a proper out-of-time evaluation, there is no reason to expect the more complex model to clearly outperform a well-built linear baseline. When it dramatically does, the first question should be whether the train-test boundary is contaminated.

Getting the probabilities right

AUC measures ranking quality, not probability accuracy. A model with an AUC of 0.75 can still assign probabilities that are substantially wrong, which matters directly for the cost calculation and for the threshold decision.

The raw GBM has an ECE of 0.534 on the OOT set. Expected calibration error measures the average gap between predicted probability and observed default rate across score bins. An ECE of 0.534 means the model’s predicted probabilities deviate from the true rates by an average of around 53 percentage points across the test set: a raw score of, say, 60% would correspond to an actual default rate much closer to 1-2%.

Two calibration methods are fitted on a held-out 20% slice of the training data and then evaluated on the OOT set. Isotonic regression fits a piecewise monotone function mapping raw scores to calibrated probabilities. Platt scaling fits a logistic regression on top of the raw scores. Both reduce ECE substantially: isotonic to 0.022, Platt to 0.016. The 0.02 target is nearly met. Both calibrated models produce probabilities that are much more usable for the cost calculation downstream.

Setting the cut-off for the business

The approval threshold that maximises accuracy is not the right threshold for a mortgage lender. Accuracy treats a missed default and a rejected good loan as equally costly, which they are not.

The cost structure here is explicit. A funded default costs the lender LGD times the original balance: 0.25 times $250,000, or $62,500. A rejected non-default costs the lender the forgone net interest margin on the loan: 1.5% of $250,000, or $3,750. Those two errors differ by a factor of roughly 17. The optimal threshold is the one that minimises expected cost at portfolio level given the calibrated probability distribution and the actual default prevalence.

Under this cost structure, the optimal threshold is high relative to what it would be with a higher base rate or a larger spread between error costs. The cost-curve explorer in the dashboard makes this concrete. Adjusting the LGD and margin sliders shifts the threshold and the approval rate, recomputing from the full pre-calculated sensitivity grid.

Why the model made each decision

Exact tree SHAP values give a signed contribution from each feature to each prediction, decomposing the model output additively. The top three features by mean absolute SHAP across the OOT set are credit score, original interest rate, and debt-to-income ratio. All act in directions that match theory. Higher credit score reduces the estimated default probability. Higher note rate and higher DTI increase it.

Seed stability, measured by Spearman rank correlation of feature importances across five different random seeds, is 0.968. All six theory-direction checks pass.

The reason-code panel in the dashboard shows per-loan SHAP contributions for three example loans. Those outputs are what a lender would put on an adverse-action notice under ECOA and Regulation B: the specific features, in signed order of contribution. They are not causal estimates. A model trained on observational data inherits all the confounding in that data. Credit score predicts default in part because it correlates with income stability and other factors the model does not observe directly. The report has a dedicated section on the prediction-versus-causation distinction for readers who need to reason about it carefully.

Testing the model’s limits

The refutation suite has four components.

Permutation. The outcome labels are shuffled eight times and OOT AUC is measured on each shuffle. A model that learned genuine patterns should perform near chance when the signal is destroyed. The mean permutation AUC is 0.467, and the real model’s AUC of 0.745 is 0.224 AUC points above the highest single permutation result.

Drop-top-3. The model is refitted without credit score, note rate, and DTI. OOT AUC falls from 0.745 to 0.642. That is a meaningful drop, confirming those features are contributing genuine predictive power and not just riding on correlations with other features in the matrix.

Competing risk. A discrete-time person-period logistic regression is fitted on monthly performance records, estimating default hazard at each loan age rather than treating outcome as a single binary event. The Spearman rank correlation between those hazard-based scores and the origination-model probabilities is 0.786. The two models, built on different data and different modelling frameworks, broadly agree on which loans are risky.

Subgroup. Metrics are broken out by origination vintage and by census region. The West shows the lowest AUC at 0.707. The Northeast shows the highest at 0.764. No subgroup falls below 0.70.

Data and reproducibility

The data is Freddie Mac’s Single Family Loan Level Dataset, covering fixed-rate conforming mortgage originations. Access requires agreeing to Freddie Mac’s data use terms at the Single Family Data portal. Raw files are never committed to version control: the licence prohibits redistribution, and the files are large. Only derived, aggregated outputs go into the repository.

When the raw files are absent, the pipeline generates a deterministic synthetic loan book in the same format, with embedded signal in FICO, CLTV, and DTI. Any run from synthetic data is stamped as such and shows a banner in the dashboard. The numbers here reflect the live dataset: 96,000 loans across 12 origination vintages, 8,000 sampled per vintage.

A single seed drives all random operations. The pipeline is idempotent: the same inputs always produce the same outputs to every decimal place. GitHub Actions runs the export step on every push and redeploys the dashboard automatically.

Limitations

The model is built on conforming mortgages that passed Freddie Mac’s acquisition criteria before entering the dataset. The covariate distributions are already filtered. Generalisation to non-conforming, jumbo, or FHA originations is untested.

The 24-month outcome horizon captures early defaults but misses the full distribution on recent vintages. A 36 or 48-month horizon would capture more events but would leave fewer vintages available for the out-of-time test. The 12 and 36-month robustness cuts are reported in the full report.

Loss given default is treated as a constant. In practice, loss severity varies with property-value paths, foreclosure timelines by state, and servicer behaviour. A stochastic LGD model would sharpen the cost calculation at the cost of additional modelling assumptions.

The SHAP reason codes explain the model. They are not causal estimates of how changing a borrower’s credit score or DTI would change their probability of default. That distinction matters for fair-lending analysis, model governance, and any downstream use of the reason codes in decision-making.