Uncertainty Quantification

Motivation: why claimed uncertainty often fails to match empirical uncertainty

The overconfident-classifier teaser

Before any definitions: a small numerical experiment that motivates everything else. We fit a small MLP classifier on make_moons — a classic 2-D binary-classification dataset where the two classes form interleaved crescents — and look at two things. The decision surface tells us the classifier is accurate: about 92% on a held-out test set. The reliability diagram tells us the classifier is mis-calibrated: in the bins where the model reports “I’m 90% confident,” the empirical accuracy is closer to 70%.

The reliability diagram is the canonical visual diagnostic for this kind of failure, and we’ll define it formally in §3. For now, the picture is what matters: every orange segment is a gap between what the model claimed and what the test data delivered. Average those gaps weighted by how often each bin is used and we get the expected calibration error — about 0.06 here, on a scale where 0 is perfectly calibrated and 0.5 is maximally wrong.

Two things to notice. First, neither test accuracy nor training loss told us anything was wrong; both are aggregate quantities that average over confidence levels. Calibration is a conditional property — it’s about model performance given its own claimed confidence — and conditional properties are invisible to aggregate metrics. Second, “fix the model” is not the only response. A simple post-hoc transformation (temperature scaling, §13) can absorb most of the miscalibration without retraining or changing the model’s accuracy at all.

What “uncertainty quantification” actually answers — and what it does not

The teaser frames three questions that the rest of this topic answers. The first is calibration evaluation: does the model’s claimed uncertainty match its empirical uncertainty? That’s the reliability-diagram question, formalized in §3 with the expected calibration error and friends, and refined in §4 with proper scoring rules that fold accuracy and calibration into single numbers with clean decomposition theorems.

The second is the aleatoric–epistemic decomposition: of the variance the model reports, how much is irreducible noise in the data-generating process, and how much reflects parameter uncertainty that more training data would shrink? This is the law-of-total-variance split, the subject of §2, and it organizes how we think about every UQ construction that follows.

The third is the construction question: how do we obtain a predictive distribution in the first place? §5–§9 tour the four standard paradigms — Bayesian (via posterior-predictive), frequentist (via plug-in or bootstrap), conformal (via distribution-free coverage), and ensemble-based (via variance across independently trained models). We don’t re-derive these constructions; the construction details live in dedicated topics. What we do here is compare what each paradigm guarantees, what it doesn’t, and how they behave on the same data.

Some things UQ is not equipped to do. It doesn’t fix model misspecification — if the assumed model class can’t represent the truth, no amount of careful uncertainty bookkeeping will save you, and §2.5 makes this concrete. It doesn’t quantify uncertainty about uncertainty in any principled way. And it doesn’t survive arbitrary distribution shift; §11 documents the degradation curves and catalogs which guarantees do and don’t carry over.

Scope discipline: methodology vs procedure vs umbrella

Conformal prediction is a specific procedure for constructing prediction sets with finite-sample marginal-coverage guarantees, requiring only exchangeability of the data. The split-conformal algorithm, its theorem and proof, the nonconformity-score machinery — that’s all derived on the conformal-prediction topic page. Here, when we discuss conformal UQ in §8, we treat conformal as a finishing layer that can be applied to outputs from any other paradigm to repair coverage post hoc.

Prediction intervals (coming soon) is the umbrella topic over the four PI-construction families: Bayesian posterior-predictive, frequentist plug-in, conformal, and quantile regression. Its job is the side-by-side taxonomy and the coverage-versus-efficiency comparison across families. We’ll cite that taxonomy in §5 rather than re-deriving it.

Uncertainty quantification — this topic — is the methodology layer. Its job is the cross-paradigm synthesis: the aleatoric/epistemic decomposition, the calibration-evaluation apparatus, the diagnostic toolkit for asking whether a given estimator’s claimed uncertainty matches its empirical uncertainty, and the degradation analyses under distribution shift and in the over-parameterized regime.

Roadmap

§2 introduces the variance split that organizes everything else. §3 and §4 give us the apparatus for asking whether claimed uncertainty matches empirical uncertainty. §5–§9 tour the four construction paradigms. §10–§11 examine UQ in regimes where standard reasoning gets weird. §12 puts UQ to work. §13 is the implementation appendix; §14 closes the synthesis.

Aleatoric vs epistemic decomposition

Predictive variance has two flavors. Aleatoric (from Latin alea, “dice”) is irreducible noise — even with infinite training data, $Y$ would still vary at any fixed input. Epistemic (from Greek episteme, “knowledge”) is parameter uncertainty — different parameter settings give different predictions, and the spread is what we don’t know about the model.

The split matters because the two flavors respond to different interventions. Aleatoric is what it is; more data doesn’t shrink it. Epistemic shrinks with more data, better features, or constrained model classes.

Two contrasting regression scenarios

Two synthetic problems with different aleatoric/epistemic mixes. Scenario A: dense training ($n = 500$) with homoscedastic noise — predictive band almost entirely aleatoric. Scenario B: sparse training in $[-3, -1] \cup [1, 3]$ with heteroscedastic noise — predictive band balloons in the gap (epistemic) and grows toward edges (heteroscedastic aleatoric).

The model is Bayesian polynomial regression with known noise variance and Gaussian prior on coefficients. Closed-form posterior in both cases; details in §2.2.

Law of total variance: the conditioning step, fully expanded

For the UQ specialization, $Y = Y_*$ is the response at a test input $\mathbf{x}_*$, and $\boldsymbol{\theta}$ has posterior $p(\boldsymbol{\theta} \mid \mathcal{D})$. The first term — average within-model variance — is aleatoric. The second — variance of the conditional mean across posterior samples — is epistemic.

training points n: 200seed offset: 0

As n grows the epistemic curve (purple) shrinks at rate 1/n by Bernstein–von Mises; the aleatoric curve (cyan) — a property of the data-generating process — does not.

Aleatoric uncertainty: what more data cannot remove

The first LTV term — average noise around the conditional mean, averaged over the posterior. In a well-specified model this does not shrink as $n \to \infty$ — it’s a property of the data-generating process, not of the estimator. On the running heteroscedastic toy of this section, $\sigma^2_{\mathrm{ale}}(x) = (0.1 + 0.5|x|)^2$, growing quadratically at the edges.

Epistemic uncertainty: what more data does remove

The second LTV term — spread of the conditional mean across posterior parameter samples. In a well-specified parametric model with regular likelihood, Bernstein–von Mises gives $\sigma^2_{\mathrm{epi}}(\mathbf{x}_*) \asymp 1/n$ at each fixed test point. This is exactly what more data buys you.

Identifiability and the limits under misspecification

The split is clean only when the model class can represent the truth. Under misspecification, the reported “aleatoric” absorbs misspecification bias that should ideally appear as model uncertainty — but the model has no mechanism to recognize its own inadequacy.

Concretely: fit a linear (degree-1) model to $Y = \sin(X) + \mathrm{noise}$. As $n \to \infty$:

• Epistemic vanishes as expected: the OLS coefficients concentrate at their population values.
• “Aleatoric” $\hat\sigma$ stabilizes at a value larger than the true noise level, because the residuals reflect both the actual noise and the misspecification bias $|\sin(x) - \beta^*_{\mathrm{OLS}} x|$.

Calibration

The §1 teaser showed a classifier whose stated probabilities don’t match its empirical accuracy. This section gives us the machinery to make that observation precise: when do we say a probabilistic predictor is calibrated, how do we test it, and what does a single-number summary of miscalibration actually capture?

Perfect, approximate, and conditional calibration

Reliability diagrams as the visual diagnostic

Partition $[0, 1]$ into $K$ bins; for each bin compute mean predicted probability $\mathrm{conf}(B_k)$ and empirical positive rate $\mathrm{acc}(B_k)$; plot the second against the first. Diagonal $y = x$ is perfect calibration. Two binning rules: equal-width (fixed-width intervals) and equal-frequency (equal-count quantile bins) — the latter is more robust when the classifier saturates many of its predictions near 0 or 1.

Expected and maximum calibration error

The standard $K = 10$ for $n \ge 1000$ is the heuristic compromise; §13.1 covers the bin-count sensitivity directly.

The sharpness-vs-calibration trade-off

A constant predictor $\hat p_0(\mathbf{x}) \equiv \bar y$ is perfectly calibrated trivially (one bin, ECE = 0) but useless — zero sharpness. The principled goal is calibrated and sharp: forecasts match empirical frequencies (calibration) and are as concentrated as possible away from the marginal base rate (sharpness). One sharpness summary: $\mathrm{Var}_{\mathbf{X}}[\hat p(\mathbf{X})]$.

sharpness τ (overconfidence): 2.5bins K: 10equal-frequency binsoverlay isotonic-recalibrated

Binning artifacts and the continuous-target generalization

ECE estimates depend on bin count $K$ and binning rule. Too few bins under-detect; too many over-estimate. A bootstrap confidence band on ECE-vs-$K$ shows the dependence directly.

For continuous targets, calibration generalizes via the probability integral transform: if $\hat F(y \mid \mathbf{x})$ is the predictive CDF, the PIT values $U_i = \hat F(Y_i \mid \mathbf{x}_i)$ should be uniform on $[0, 1]$ under calibration. A U-shaped PIT histogram indicates over-confidence; a hump-shape indicates under-confidence.

Proper scoring rules

The §3 calibration diagnostics are visual and interpretive. What we want is a single scalar function whose expected value is minimized exactly when the prediction matches the true conditional distribution. This is a strictly proper scoring rule.

Brier, log-loss, CRPS

Strict properness: the finite-outcome direct calculation

A useful identity falls out of the log-loss case: $\mathbb{E}_{Y \sim q}[\mathrm{LL}(p, Y)] = H(q) + D_{\mathrm{KL}}(q \,\|\, p)$. Minimizing log-loss in $p$ is minimizing the KL divergence from the truth.

For CRPS, the argument is analogous: for $Y \sim G$, $\mathbb{E}[\mathrm{CRPS}(F, Y)] - \mathbb{E}[\mathrm{CRPS}(G, Y)] = \int (F(t) - G(t))^2 \, dt \ge 0$, with equality iff $F = G$ almost everywhere. Strictly proper.

Murphy 1973: the reliability + resolution + uncertainty decomposition

REL is the squared calibration gap, zero for a calibrated predictor. RES measures how well the bins separate outcomes — higher is more informative. UNC is the marginal Bernoulli variance, fixed by the data. Lower BS comes from either lower REL or higher RES.

true positive rate q: 0.30

Drag q to verify Theorem 2 numerically: both expected losses are uniquely minimized at p = q, with closed-form minimum values q(1−q) (Brier) and H(q) (log-loss).

Practical evaluation: which rule answers which question

Brier is bounded — it won’t blow up on confident-wrong predictions. Reach for it when comparing across predictors with very different calibration profiles, or when the Murphy decomposition is the actual goal.

Log-loss is unbounded above and severely penalizes confident-wrong predictions. Reach for it when “no surprises” is the goal, when you have many predictions so outliers won’t dominate, or when the predictor will be used for downstream maximum-likelihood (log-loss is cross-entropy).

CRPS is the right thing for continuous targets. Closed form for Gaussian predictives (Hersbach 2000); trapezoidal quadrature for general ones, agreeing to $10^{-3}$ in the verify suite.

The Murphy decomposition is a diagnostic: when Brier is high, REL says it’s a calibration problem; $-$RES says it’s a resolution problem; UNC tells you the irreducible floor.

Predictive intervals across paradigms

Four standard PI families: Bayesian posterior-predictive, frequentist plug-in, conformal, quantile regression. This section is the cross-family comparison — same data, same target coverage, what each delivers. Construction details live in the dedicated conformal-prediction topic and the prediction-intervals (coming soon) umbrella.

Umbrella view

• Bayesian PP: posterior + likelihood → posterior-predictive distribution → quantile-based PI. Correct under prior + likelihood specification; input-adaptive width by construction.
• Frequentist plug-in: point estimator + parametric quantile. Constant-width band for a homoscedastic-Gaussian plug-in. The heaviest in assumptions and the cheapest to compute.
• Split conformal: train fold + calibration residuals + inflated quantile. Constant-width band with absolute-residual nonconformity. Distribution-free finite-sample marginal coverage.
• Quantile regression: directly learn conditional quantile functions via pinball-loss minimization. Adaptive-shape band; requires model class flexibility.

Coverage vs efficiency: the no-free-lunch comparison

Every paradigm trades off coverage and efficiency (PI width). Each method makes a different trade — distribution-free vs adaptive-width vs parametric.

Marginal vs conditional coverage

Marginal coverage: $\Pr_{X, Y}(Y \in \mathrm{PI}(X)) \ge 1 - \alpha$ — averaged over inputs.

Conditional coverage: $\Pr_Y(Y \in \mathrm{PI}(x) \mid X = x) \ge 1 - \alpha$ for almost every $x$ — pointwise.

Marginal is distribution-free achievable (split conformal). Conditional is not distribution-free achievable (Vovk 2012, Foygel Barber et al. 2021). The per-decile-coverage figure isolates this.

α (miscoverage target): 0.10 (target 0.90)

At α = 0.10 all three target 0.90 marginal coverage. Bayesian PP is input-adaptive (wider at the edges). Plug-in is constant-width and undercovers because its homoscedastic σ̂ misses heteroscedasticity. Conformal is constant-width but distribution-free.

When to reach for which

• Distribution-free coverage → split conformal (and the §8 finishing layer).
• Input-adaptive under correct specification → Bayesian PP (§7 NumPy approximations).
• Heteroscedastic noise + flexible model class → quantile regression.
• Cheap parametric baseline → plug-in.

The single most useful UQ result is the conformal finishing-layer wrap around any other paradigm — distribution-free marginal coverage with adaptive width. That’s §8.

Bootstrap-based UQ

Bagging — bootstrap-aggregating — was introduced as variance reduction for unstable point predictors (Breiman 1996). The same machinery produces predictive uncertainty.

Bagged predictive distributions: the bagging-as-UQ argument

Given training data $\mathcal{D}$ and learner $\hat\mu(\cdot; \mathcal{D})$. For $B$ rounds, draw bootstrap resample $\mathcal{D}^*_b$; fit $\hat\mu_b$. At test point $\mathbf{x}_*$:

$$\bar\mu(\mathbf{x}_*) \;=\; \tfrac{1}{B}\sum_b \hat\mu_b(\mathbf{x}_*), \qquad \hat\sigma^2_{\mathrm{bag}}(\mathbf{x}_*) \;=\; \tfrac{1}{B-1}\sum_b \big(\hat\mu_b(\mathbf{x}_*) - \bar\mu(\mathbf{x}_*)\big)^2.$$

The bootstrap principle says the sampling distribution under resampling approximates the sampling distribution under the data-generating process. So $\hat\sigma^2_{\mathrm{bag}}$ estimates $\mathrm{Var}_{\mathcal{D}}[\hat\mu(\mathbf{x}_*; \mathcal{D})]$ — the §2 epistemic variance. The formal statement is Theorem 3 of formalStatistics §31.3 (resampling consistency). Bagging gives epistemic only; aleatoric must be supplied separately.

Three bootstrap variants

• Pairs (nonparametric): resample $(\mathbf{x}_i, y_i)$ jointly. No noise-structure assumption.
• Residual: fit once, resample residuals from the empirical pool. Assumes exchangeable residuals — wrong under heteroscedasticity.
• Wild: $y_i^* = \hat\mu(\mathbf{x}_i) + w_i \hat\varepsilon_i$ with Rademacher $w_i \in \{-1, +1\}$. Preserves local noise magnitude — right for heteroscedastic regression.

B (bootstrap replicates): 100variantpairsresidualwild

The dashed reference is the analytic Bayesian §2.3 epistemic. Wild bootstrap tracks it best on this heteroscedastic toy; residual underestimates because exchangeable-residual assumption fails.

Coverage in finite samples

Three failure modes: Monte Carlo noise from finite $B$; downward bias from bootstrap sample sharing $\approx 63\%$ training data; residual-bootstrap fails under heteroscedasticity. Bootstrap UQ is consistent, cheap, and almost-always-applicable — but finite-sample coverage is approximate. For a finite-sample guarantee, wrap conformal around it (§8).

Comparison with the Bayesian posterior-predictive

Bayesian UQ in NumPy

Three approximations to the posterior-predictive distribution on the §2 toy, all implemented in pure NumPy in the companion notebook: Laplace (exact for linear-in-θ), MC-dropout via re-implemented forward pass, deep ensembles.

Posterior-predictive distribution as the natural UQ object

$$p(y_* \mid \mathbf{x}_*, \mathcal{D}) \;=\; \int p(y_* \mid \mathbf{x}_*, \boldsymbol{\theta})\, p(\boldsymbol{\theta} \mid \mathcal{D})\, d\boldsymbol{\theta}.$$

Decomposes via §2 LTV. Intractable for nonlinear models; every §7 method is a different approximation.

Laplace approximation

$$p(\boldsymbol{\theta} \mid \mathcal{D}) \;\approx\; \mathcal{N}\!\left(\hat{\boldsymbol{\theta}}_{\mathrm{MAP}},\; \mathbf{H}^{-1}\right), \qquad \mathbf{H} = -\nabla^2 \log p(\boldsymbol{\theta} \mid \mathcal{D})\,\Big|_{\hat{\boldsymbol{\theta}}_{\mathrm{MAP}}}.$$

Local Gaussian fit at the MAP — misses multimodality.

MC-dropout via re-implemented forward pass

Gal & Ghahramani (2016): leave dropout active at test time, average $T$ stochastic forward passes. The full Bayesian interpretation requires training with dropout for the variational argument; our test-time-only implementation is a NumPy demonstration of the mechanism, not the full Bayesian variant.

Implementation: train an MLPRegressor without dropout, extract the public coefs_ and intercepts_, run NumPy forward passes with Bernoulli masks at hidden activations:

$$\mathbf{h}^{(\ell+1)} \;=\; (1 - p)^{-1}\,\mathbf{m}^{(\ell)} \odot \phi\!\big(\mathbf{h}^{(\ell)} \mathbf{W}^{(\ell)} + \mathbf{b}^{(\ell)}\big).$$

Inverted-dropout scaling $(1-p)^{-1}$ preserves expected activation magnitude.

Deep ensembles

Fit $M$ independent MLPs with different initializations (Lakshminarayanan, Pritzel & Blundell 2017). At a test point, take the empirical mean and variance across the $M$ predictions. Deep ensembles capture multi-modal posterior structure via genuinely independent fits — Laplace and MC-dropout cannot.

Loading pre-trained ensemble (~340 KB) …

methodLaplaceMC-dropoutDeep ensembleensemble size M: 20

What each approximation captures — and what it doesn’t

Laplace. Exact when the posterior is Gaussian. Misses multimodality otherwise.

MC-dropout. Captures some epistemic variance via test-time masks, but the variance is controlled by the hand-set dropout rate $p$ — there’s no mechanism for the model to discover how uncertain it should be.

Deep ensembles. Multi-modal via independent fits. Empirically the most reliable on benchmarks (Lakshminarayanan 2017, Ovadia 2019).

Recommended default. Deep ensembles + conformal finishing layer (§8) — the combination delivers calibrated input-adaptive PIs with a distribution-free marginal-coverage guarantee.

Conformal UQ

Position conformal as a coverage-repair finishing layer applied post hoc to outputs of any other UQ paradigm to recover finite-sample distribution-free marginal coverage. The procedure derivation lives in conformal-prediction; this section shows what using conformal as a finishing layer looks like on the §7 methods.

Distribution-free coverage as a UQ tool

Split conformal recipe: train predictor on a training fold; compute nonconformity scores $S_i$ on a calibration fold; take the inflated quantile $\hat q$. Marginal coverage $\Pr(Y_* \in \mathrm{PI}(\mathbf{x}_*)) \ge 1 - \alpha$ under exchangeability — independent of the base predictor.

Conformalizing any UQ output: post-hoc coverage repair

The nonconformity score is any function of $(\mathbf{x}, y)$. For the §7 methods, take the mean prediction $\hat\mu$, calibration absolute residuals, and the inflated quantile $\hat q$. The PI is $\hat\mu \pm \hat q$ with marginal coverage at target regardless of the underlying method’s specification.

Adaptivity via locally-weighted nonconformity scores

The vanilla nonconformity score gives a constant-width band. A normalized score

$$S_i \;=\; \frac{|Y_i - \hat\mu(\mathbf{X}_i)|}{\hat\sigma(\mathbf{X}_i)}$$

gives the PI $\hat\mu(\mathbf{x}_*) \pm \hat q \cdot \hat\sigma(\mathbf{x}_*)$ — adaptive width via the base predictor’s $\hat\sigma$, same marginal-coverage guarantee. The exchangeability of $(X_i, Y_i)$ implies the exchangeability of $S_i$, so the conformal theorem still applies. (Lei et al. 2018; Romano, Patterson & Candès 2019.)

Loading pre-trained ensemble …

base predictorLaplaceDeep ensembleα: 0.10 (target 0.90)locally-weighted (adaptive width)

What conformal cannot give

Conditional coverage is impossible distribution-free (Vovk 2012, Foygel Barber et al. 2021). Individual probability calibration is meaningless distribution-free — conformal returns a set, not a probability. The right composition is: good UQ supplies $\hat\sigma(\mathbf{x})$, locally-weighted conformal wraps for a distribution-free marginal-coverage guarantee with adaptive width.

Deep-ensemble UQ

Deeper look at the framework introduced in §7.4.

Variance-across-members estimator: derivation and intuition

$$\hat\sigma^2_{\mathrm{ens}}(\mathbf{x}_*) \;=\; \frac{1}{M - 1}\sum_{m=1}^M \big(\hat\mu_m(\mathbf{x}_*) - \bar\mu(\mathbf{x}_*)\big)^2.$$

Treating each member as a posterior sample, this estimates the §2 epistemic variance. Aleatoric is supplied separately (e.g., via the §6 residual smoother). By LTV the total predictive variance is $\hat\sigma^2_{\mathrm{ens}} + \hat\sigma^2_\epsilon$.

Why ensembles outperform single-model UQ

Four reasons: (1) multi-modal posterior coverage — different initializations land in different local minima; (2) robustness to bad init; (3) empirical dominance on calibration (Lakshminarayanan 2017) and on distribution shift (Ovadia 2019); (4) trivial parallelism. The counter is the $M \times$ compute cost.

Ensemble-size–quality Pareto and compute budgets

The standard error of the variance estimator scales $1/\sqrt{M - 1}$. Calibration metrics plateau by $M = 20$ on the clean §2 toy. Recommended $M \in [10, 20]$ for clean problems; Ovadia (2019) found gains up to $M = 100$ on harder benchmarks under shift.

Loading pre-trained ensemble …

ensemble size M: 20test point x*: 1.50

Mixture-of-Gaussians readout for full predictive distributions

The full predictive distribution from a deep ensemble is the mixture

$$p(y \mid \mathbf{x}_*) \;=\; \frac{1}{M}\sum_{m=1}^M \mathcal{N}\!\big(y;\; \hat\mu_m(\mathbf{x}_*),\, \hat\sigma_\epsilon^2(\mathbf{x}_*)\big).$$

The mixture mean is the ensemble mean. The mixture variance is $\hat\sigma_\epsilon^2 + \hat\sigma^2_{\mathrm{ens}}$ by LTV. This readout supplies the inputs for CRPS evaluation, PIT histograms, and §12 decision-theoretic UQ on the full predictive distribution rather than a Gaussian summary.

UQ in the over-parameterized regime

Where double-descent answers “what does test risk do as $p/n$ crosses the interpolation threshold?”, §10 answers “what does predictive uncertainty do?” Two UQ shadows of the risk spike: Bayesian PP variance and conformal half-width.

Posterior-predictive variance through the interpolation threshold

For the linear-Gaussian model with ridge, the posterior covariance is $\boldsymbol{\Sigma}_\beta = \sigma^2(\mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I})^{-1}$, and the posterior-predictive variance at $\mathbf{x}_*$ is $\sigma^2 + \mathbf{x}_*^\top \boldsymbol{\Sigma}_\beta \mathbf{x}_*$. The matrix $(\mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I})^{-1}$ blows up when $\mathbf{X}^\top \mathbf{X}$ becomes nearly singular — exactly at $p \approx n$. The spike in PP variance is the same algebraic phenomenon as the test-risk spike.

Conformal coverage at $p/n \to 1$: the spike’s UQ shadow

The marginal-coverage theorem from conformal-prediction holds regardless of predictor quality: coverage stays at $1 - \alpha$ across the entire sweep. The conformal half-width $\hat q$ inflates at the threshold (residuals are large) and descends past it — same shape as the PP variance. Conformal coverage is theorem-protected; conformal width follows the predictor.

Reuse of the double-descent Hastie-form risk curve as underlay

Numerically replicate the Hastie–Montanari–Rosset–Tibshirani 2022 spike-plus-second-descent risk curve; overlay the UQ counterparts. Test risk, PP variance, and conformal half-width are all algebraically linked through the same matrix $\mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I}$.

ridge log₁₀(λ): -2.0 (λ = 1.0e-2)

Test risk (a), posterior-predictive variance (b), and conformal half-width (c) all spike at p/n ≈ 1. Conformal marginal coverage (d) is flat at the target 1 − α — coverage is theorem-protected, width follows predictor quality. Drag λ to see ridge regularization smooth the spike.

What UQ does at peak risk — and after

Three observations. First, peak risk is peak uncertainty in-distribution and well-specified — the model is genuinely confused near the threshold. Second, the over-parameterized regime is not uniformly bad: the second descent in PP variance and conformal width mirrors the second descent in risk. Third, the conformal coverage guarantee survives the spike — which is the cleanest illustration of the §8.4 closing observation. Bridge to §11: out-of-distribution behavior is a different problem entirely.

Evaluating UQ at distribution shift

Honest UQ failure-mode evaluation under shift. No UQ method delivers a finite-sample distribution-free guarantee under arbitrary shift. Some methods degrade more gracefully than others.

Three shift types

Covariate shift: $p(\mathbf{x})$ changes, $p(y \mid \mathbf{x})$ unchanged. Example: deploying a model trained on one demographic to another.

Label shift: $p(y)$ changes, $p(\mathbf{x} \mid y)$ unchanged. Example: deploying a disease-prevalence-aware classifier in a new population.

Concept shift: $p(y \mid \mathbf{x})$ itself changes. Example: a fraud detector when fraudsters adapt their patterns.

Coverage degradation curves

A sweep of covariate-shift magnitude $s \in [0, 2]$ on the §2 toy shows all three §7 methods degrading. Deep ensembles degrade most gracefully (Ovadia 2019); Laplace degrades fastest because the closed form doesn’t know about shift; conformal degrades roughly as fast as the underlying predictor — exchangeability is violated, so the conformal theorem doesn’t apply.

Calibration degradation: reliability before and after

On the §1 classifier, shifting the test moons in the $x_1$ direction degrades calibration monotonically. ECE at $s = 0$ is around 0.05; at $s = 1.5$ it is several times that. The reliability diagrams show systematic over-confidence growing under shift — calibration monitoring is a viable deployment health check.

The open-set problem and out-of-distribution detection

Different from coverage degradation: an input-only test asking “is this OOD?” rather than “does my PI cover the outcome?”. The deep-ensemble epistemic variance turns out to be a strong OOD score on the §2 toy — AUROC well above 0.85 separating in-distribution from far-OOD inputs. Related but complementary diagnostic.

Loading pre-trained ensemble …

LaplaceDeep ensembleLW-conformal on ensemble

What distribution-free guarantees don’t survive shift

Vanilla split conformal needs exchangeability; under shift, exchangeability is violated and the marginal-coverage guarantee fails. Weighted conformal (Tibshirani, Foygel Barber, Candès & Ramdas 2019) recovers the guarantee under known covariate shift via density-ratio weighting — the catch is that the density ratio is rarely known in practice. Bayesian PP has no formal guarantee and degrades under bad specification. Deep ensembles degrade most gracefully empirically but have no formal guarantee. Concept shift admits no general fix: detection + retraining is the only response.

Decision-theoretic UQ

Operationalizing UQ outputs. Three operational stories — Bayes-optimal point prediction under a loss; selective prediction with abstention; cost-sensitive thresholding under miscalibration.

Expected loss minimization under the predicted distribution

The Bayes-optimal point predictor at $\mathbf{x}_*$ under loss $L$ is

$$\hat y^*(\mathbf{x}_*) \;=\; \arg\min_{\hat y} \mathbb{E}_{Y \sim \hat p(\cdot \mid \mathbf{x}_*)}[L(Y, \hat y)].$$

Three canonical loss / point-prediction pairs: squared loss → mean; absolute loss → median; pinball loss at $\tau$ → $\tau$-quantile. What to report depends on what’s downstream. Reporting the median when the deployed loss is squared error is wrong; reporting the mean when the deployed loss is pinball is also wrong.

Selective prediction with abstention: the rejection threshold

Abstain when confidence falls below a threshold (El-Yaniv & Wiener 2010, Geifman & El-Yaniv 2017). For regression, threshold on predictive variance; for classification, on max-probability. The rejected fraction is the abstention rate; the rest is the selective prediction.

Coverage–accuracy trade-off and the precision-at-coverage curve

Sweep the abstention threshold to trace the (coverage, accuracy) curve. Three operating points typically matter: no abstention (the baseline accuracy), the high-precision regime (small coverage, near-perfect accuracy), and the Pareto frontier between. The area under the selective-risk curve (AURC) is a single-number summary.

Cost-sensitive decisions when calibration is imperfect

The Bayes-optimal threshold for a binary decision under asymmetric costs $c_{FP}, c_{FN}$ is

$$\hat p^* \;=\; \frac{c_{FP}}{c_{FP} + c_{FN}}.$$

This assumes calibrated probabilities. Under miscalibration, applying the formula gives a sub-optimal threshold. Isotonic recalibration (or temperature scaling, §13.2) delivers meaningful cost savings on the §1 classifier under asymmetric cost regimes — 10–20% on the demo.

Computational notes

Implementation-hygiene appendix.

Efficient ECE estimation and binning choices

Rule of thumb: at least 30 samples per bin to avoid empirical-accuracy noise dominating the calibration signal. Bootstrap percentile confidence intervals on the ECE estimate are cheap and informative. Adaptive binning (Roelofs et al. 2022) is a tighter alternative to fixed-$K$ binning. For continuous targets, PIT-based KS or Cramér–von Mises distances generalize bin-based ECE.

Temperature scaling and isotonic regression for cheap recalibration

Fit a single scalar $T > 0$ on a calibration set: $T^* = \arg\min_T -\sum_i \big[y_i \log\sigma(z_i / T) + (1 - y_i)\log(1 - \sigma(z_i / T))\big]$.

temperature T: 1.00 (fitted T* = 2.23)

Drag T to see the reliability curve respond. T > 1 smooths overconfidence (pulls high-probability bins down toward the diagonal); T < 1 sharpens. The fitted T* minimizes the calibration NLL via golden-section search; isotonic regression is the more flexible non-parametric alternative.

Ensemble-size and MC-replicate budgets

Deep ensembles: $M = 10$ for a prototype, $M = 20$ for production, $M = 50+$ when distribution shift matters. MC-dropout: $T = 100$ for ≈5% MC error, $T = 500$ for ≈2%. Verification sample sizes: $B \ge 20$ replicates for the §10 sweep, $n \ge 200$ for the §11 AUROC.

Numerical stability at extreme probabilities

Log-loss: clip predictions to $[10^{-12}, 1 - 10^{-12}]$ before taking logarithms. Mixture NLL: compute with the log-sum-exp trick (logsumexpAxis0 in the shared TS module) to avoid underflow at small mixture-component densities. CRPS: prefer the Hersbach (2000) closed form for Gaussian forecasts; reserve trapezoidal quadrature for non-Gaussian cases or as a numerical sanity check.

Reproducibility checklist

Single topic-wide RNG seed; explicit random_state for every sklearn call; inline assert statements on the headline diagnostics in every code cell; save figures before showing them so re-runs don’t lose images; report single-number metrics inline next to the asserts; re-run on a fresh kernel before committing to verify the topic is end-to-end reproducible.

Connections and limits

Semiparametric inference: efficient influence functions for variance components

The semiparametric-inference topic develops efficient influence functions (EIFs) for asymptotically optimal estimators of any functional. Two connections to UQ. First, the EIF for a variance/uncertainty parameter yields the asymptotically optimal estimator with semiparametric efficiency bound (van der Vaart 1998) — see that topic’s §11 for the variance-functional EIF derivation and the demonstration that cross-fit residuals are the semiparametric-efficient recipe. Second, EIF-based standard errors give valid UQ for plug-in estimators with ML-estimated nuisance functions — the double machine learning framework (Chernozhukov et al. 2018). The connection is direct: every UQ method here that uses a flexible nuisance estimator (the §7 MLP, the §10 ridge) is a candidate for EIF correction.

Mixed-effects models: random effects as one form of epistemic UQ

In a mixed-effects model $y_{ij} = \beta_0 + \beta_1 \mathbf{x}_{ij} + \alpha_i + \varepsilon_{ij}$, the random effect $\alpha_i$ is exactly epistemic uncertainty about group $i$‘s parameters. Within-group variance is aleatoric; between-group variance is epistemic. Mixed-effects gives a structured parametric form of the §2 decomposition — identifiable from data when the group structure is known.

Information theory: entropy as a one-number UQ summary

The §4.2 derivation showed $\mathbb{E}[\mathrm{LL}(p, Y)] = H(q) + D_{\mathrm{KL}}(q \,\|\, p)$: log-loss is KL plus entropy. The sharpness-calibration trade-off has an information-theoretic reading as mutual-information maximization subject to a calibration constraint. Predictive entropy generalizes the §11.4 variance-based OOD score to non-Gaussian predictives.

Honest limits — model misspecification, arbitrary shift, the Pareto frontier

• Model misspecification (§2.5). No UQ method can detect or fix structural misspecification on its own.
• Distribution shift (§11.5). No procedure delivers distribution-free coverage under arbitrary shift.
• Conditional coverage impossibility (§5.3, §8.4). No procedure achieves finite-sample distribution-free conditional coverage.
• Sharpness-calibration Pareto (§3.4). Perfect joint achievement requires perfectly-predictable data.
• Identifiability under misspecification. The aleatoric/epistemic split is honest only under correct specification.
• Open research. UQ for LLMs, UQ under concept shift, UQ in distillation/quantization, UQ in active learning. Recent literature is starting to address these but no consensus framework has emerged.

Every UQ practitioner reaches a residual zone of judgment where the formal apparatus runs out. Knowing where that zone begins is what distinguishes a competent UQ practice from a careless one.

Discharged forward-pointer

The bagging-as-UQ argument in §6.1 leans on Theorem 3 of formalStatistics §31.3 (resampling consistency for the variance of a parameter estimator). The reciprocal frontmatter entry on this topic discharges the deferred-reciprocity entry that has been logged on formalstatistics’ side since the bootstrap topic shipped. The other connections to formalML topics — bayesian-neural-networks (§7), conformal-prediction (§5, §8), prediction-intervals (§5), double-descent (§10), mixed-effects (§14.2), and semiparametric-inference (§14.1) — are formalML-internal and listed in the topic’s connections array.