Measure-Theoretic Probability

Overview & Motivation

Why do we need measure theory for probability? The short answer: because naive probability breaks down in continuous settings.

Consider a “uniform” random variable $X$ on $[0, 1]$. We want $P(X \in A)$ for subsets $A \subseteq [0, 1]$. For intervals, this is simple: $P(X \in [a, b]) = b - a$. But what about arbitrary subsets? Can we assign a “length” (probability) to every subset of $[0, 1]$ while preserving countable additivity?

The answer, due to Vitali (1905), is no — there exist non-measurable sets. This forces us to restrict our attention to a carefully chosen collection of “well-behaved” subsets: a sigma-algebra.

This is not an abstract curiosity. Every time we write $E[X]$, compute a conditional expectation $E[X \mid \mathcal{G}]$, invoke the law of large numbers, or price a financial derivative, we are relying on measure-theoretic machinery. The Lebesgue integral replaces the Riemann integral because it handles limits of random variables correctly (via the Monotone and Dominated Convergence Theorems). Conditional expectation, defined as a Radon–Nikodym derivative, is the mathematical backbone of filtering, Bayesian inference, and martingale theory. And martingales themselves are the language of fair pricing in mathematical finance.

What We Cover

1. Sigma-algebras & Measurable Spaces — the sets we can assign probabilities to, and why we need them.
2. Measures & Probability Measures — Kolmogorov’s axioms, Lebesgue measure, and the Cantor set.
3. Measurable Functions & Random Variables — formalizing “random quantities” as measurable maps.
4. The Lebesgue Integral & Expectation — building the integral from simple functions, with full proofs of MCT and DCT.
5. Convergence of Random Variables — the four modes, their hierarchy, the Laws of Large Numbers, and the CLT.
6. Product Measures & Fubini’s Theorem — integrating over product spaces and why $E[XY] = E[X]E[Y]$ for independent variables.
7. Conditional Expectation & Radon–Nikodym — the deepest idea: conditional expectation as an $L^2$ projection.
8. A Preview of Martingales — filtrations, adapted processes, and connections to finance.

Connections

This topic connects to the rest of the formalML curriculum in several directions:

• PCA & Low-Rank Approximation — the sample covariance $\hat{\Sigma} = \frac{1}{n-1} X^T X$ converges to the population covariance $\Sigma$ by the Law of Large Numbers; $L^2$ theory guarantees convergence of eigenvalues.
• Concentration Inequalities — builds directly on the $L^p$ spaces and convergence theory developed here, quantifying rates of convergence beyond the LLN.
• PAC Learning Framework (coming soon) — uses measure-theoretic probability to formalize learnability.
• Bayesian Nonparametrics (coming soon) — requires conditional expectation and the Radon–Nikodym theorem for prior specifications on infinite-dimensional spaces.

---

Sigma-Algebras and Measurable Spaces

Why We Need Sigma-Algebras

The fundamental question of probability is: given a sample space $\Omega$, which subsets can we assign probabilities to?

For finite $\Omega$, the answer is easy: every subset. The power set $2^\Omega$ works. But for uncountable $\Omega$ — like $\mathbb{R}$ or $[0, 1]$ — the power set is too large. Vitali’s 1905 construction shows that no translation-invariant, countably additive measure can be defined on all subsets of $[0, 1]$. We must restrict to a smaller collection of sets that is still rich enough to do calculus.

The right structure is a sigma-algebra: a collection of subsets closed under complements and countable unions. This is precisely what we need for probability — we want to say “the probability of $A$ or $B$” (unions), “the probability of not $A$” (complements), and we want these operations to work for countable sequences of events.

Examples on a Finite Set

Let $\Omega = \{1, 2, 3\}$. Three sigma-algebras on $\Omega$:

• Trivial: $\mathcal{F}_0 = \{\emptyset, \Omega\}$ — we can only say “something happens” or “nothing happens.”
• Partial: $\mathcal{F}_1 = \{\emptyset, \{1\}, \{2, 3\}, \Omega\}$ — we can distinguish element 1 from the rest.
• Power set: $\mathcal{F}_2 = 2^\Omega$ — we can distinguish every element.

The trivial sigma-algebra carries the least information; the power set carries the most. This idea — sigma-algebras as information — is the conceptual key to conditional expectation and filtrations.

Generated Sigma-Algebras and the Borel Sets

Given any collection $\mathcal{C}$ of subsets of $\Omega$, there is a smallest sigma-algebra containing $\mathcal{C}$, written $\sigma(\mathcal{C})$. It is the intersection of all sigma-algebras containing $\mathcal{C}$ — and since the power set $2^\Omega$ is always a sigma-algebra, this intersection is non-empty.

The Borel sets on $\mathbb{R}^d$ are defined analogously: $\mathcal{B}(\mathbb{R}^d) = \sigma(\text{open sets in } \mathbb{R}^d)$.

Here is a Python implementation that verifies the sigma-algebra axioms on finite sets and computes generated sigma-algebras by closure:

def is_sigma_algebra(omega, F):
    """Verify whether F is a sigma-algebra on omega."""
    omega_set = frozenset(omega)
    F_sets = {frozenset(s) for s in F}

    # Axiom 1: omega in F
    if omega_set not in F_sets:
        return False, "Omega not in F"

    # Axiom 2: closure under complements
    for A in F_sets:
        complement = omega_set - A
        if complement not in F_sets:
            return False, f"Complement of {set(A)} not in F"

    # Axiom 3: closure under (finite, here) unions
    for A in F_sets:
        for B in F_sets:
            if A | B not in F_sets:
                return False, f"Union {set(A)} ∪ {set(B)} not in F"

    return True, "Valid sigma-algebra"

Sigma-Algebra Explorer — Ω = {1, 2, 3, 4}

Trivial {∅, Ω}σ({1,2})σ({1}, {2})σ({1,2}, {2,3})Power set 2^Ω

Click elements to toggle generators

|𝓕| = 4Generators: {1,2}⊂ 2^Ω (4 of 16 subsets)

---

Measures and Probability Measures

Definition of a Measure

A sigma-algebra tells us which subsets are measurable. A measure tells us how big they are.

Fundamental Properties

Lebesgue Measure

The Cantor set is a remarkable object: it is closed, uncountable, has Lebesgue measure zero, and is totally disconnected. We construct it by iteratively removing middle thirds from $[0, 1]$. At each step, the total length removed is $\sum_{k=0}^{n-1} 2^k / 3^{k+1}$, which converges to $1$ — leaving a set of measure zero that still contains uncountably many points (every number in $[0, 1]$ with a ternary expansion using only digits 0 and 2).

Probability Measures and Kolmogorov’s Axioms

Every familiar probability distribution defines a probability measure on $(\mathbb{R}, \mathcal{B}(\mathbb{R}))$. The uniform distribution on $[0, 1]$ is simply Lebesgue measure restricted to $[0, 1]$. A Gaussian $N(\mu, \sigma^2)$ defines $P(A) = \int_A \frac{1}{\sigma\sqrt{2\pi}} e^{-(x-\mu)^2/(2\sigma^2)} dx$ for Borel sets $A$.

---

Measurable Functions and Random Variables

Measurable Functions

Proposition 6 is one of the key advantages of measurable functions over continuous functions: pointwise limits of measurable functions are measurable, while pointwise limits of continuous functions need not be continuous.

Random Variables

This is the measure-theoretic formalization of “a quantity whose value depends on the outcome of a random experiment.” The measurability condition $X^{-1}(B) \in \mathcal{F}$ ensures that $P(X \in B)$ is well-defined for every Borel set $B$.

A random vector $X : \Omega \to \mathbb{R}^d$ is $(\mathcal{F}, \mathcal{B}(\mathbb{R}^d))$-measurable. Component-wise: $X = (X_1, \ldots, X_d)$ is a random vector if and only if each $X_i$ is a random variable.

Distributions and Independence

---

The Lebesgue Integral and Expectation

Simple Functions and the Construction

The Lebesgue integral is built in three stages: simple functions → non-negative functions → general functions.

Riemann vs. Lebesgue

The Riemann integral partitions the domain into small intervals and sums $f(x_i^*) \Delta x_i$. The Lebesgue integral partitions the range into small intervals and sums $y_i \cdot \mu(\{f \in [y_i, y_{i+1})\})$.

This “horizontal slicing” is why the Lebesgue integral handles limits better: it does not care about the geometric arrangement of the domain, only about the measure of level sets.

The Monotone Convergence Theorem

Fatou’s Lemma and the Dominated Convergence Theorem

The DCT is the workhorse of probability theory. Whenever we want to exchange a limit and an integral — which happens constantly in proving convergence results — we look for a dominating function. Without one, the exchange can fail dramatically, as the next example shows.

Expectation and $L^p$ Spaces

---

Convergence of Random Variables

The four modes of convergence — almost sure, in probability, in $L^p$, and in distribution — form a hierarchy that is central to asymptotic statistics and the theoretical foundations of machine learning.

The Four Modes

The Hierarchy

The implications between these modes form two chains:

$L^p \implies \text{in probability} \implies \text{in distribution}$

$\text{a.s.} \implies \text{in probability} \implies \text{in distribution}$

And the converses are generally false, with important exceptions.

Counterexamples

The converses fail in illuminating ways:

• In probability does not imply a.s.: The “typewriter sequence” is the classic counterexample. Consider $[0, 1]$ with Lebesgue measure, and define $f_n = \mathbf{1}_{[k/m, (k+1)/m]}$ where $n$ enumerates pairs $(m, k)$ by cycling through intervals of decreasing width. Then $f_n \to 0$ in probability (the interval width shrinks), but for every $\omega \in [0, 1]$, infinitely many $f_n(\omega) = 1$.

• In distribution does not imply in probability: Let $X \sim N(0,1)$ and $Y_n = -X$. Then $Y_n \xrightarrow{d} X$ (since $-X \sim N(0,1)$ too), but $P(|Y_n - X| > 1) = P(|2X| > 1) > 0$ for all $n$.

The Laws of Large Numbers

The SLLN is strictly stronger than the WLLN: it requires only a finite first moment (not second moment), and the convergence is almost sure. The proof uses the fourth-moment method or truncation arguments and is considerably more involved than the WLLN proof.

The Central Limit Theorem

The CLT is the deepest result in elementary probability. Its measure-theoretic proof uses characteristic functions: $\varphi_{Z_n}(t) \to e^{-t^2/2}$, plus Lévy’s continuity theorem (convergence of characteristic functions if and only if convergence in distribution).

Notice the distinction: the SLLN gives almost sure convergence of $\bar{X}_n$ to $\mu$ (a constant), while the CLT gives convergence in distribution of the rescaled fluctuations $\sqrt{n}(\bar{X}_n - \mu)/\sigma$ to a Gaussian. These are complementary, not competing, results.

---

Product Measures and Fubini’s Theorem

Given measurable spaces $(\Omega_1, \mathcal{F}_1)$ and $(\Omega_2, \mathcal{F}_2)$, the product sigma-algebra $\mathcal{F}_1 \otimes \mathcal{F}_2$ is the sigma-algebra on $\Omega_1 \times \Omega_2$ generated by the measurable rectangles $\{A_1 \times A_2 : A_1 \in \mathcal{F}_1, A_2 \in \mathcal{F}_2\}$.

For probability spaces, this gives the joint distribution of independent random variables: if $X$ and $Y$ are independent with laws $\mu_X$ and $\mu_Y$, then $(X, Y)$ has law $\mu_X \otimes \mu_Y$.

Probabilistic consequence. For independent random variables $X, Y$ with densities $f_X, f_Y$:

$E[g(X, Y)] = \int \int g(x, y) f_X(x) f_Y(y) \, dx \, dy$

and the order of integration can be swapped freely. This is why we can factor joint expectations of independent variables: $E[XY] = E[X]E[Y]$.

import numpy as np
from scipy import integrate

# Verify Fubini: ∫∫ x²e^{-y} dx dy over [0,1]×[0,∞)
# Iterated integral 1: ∫₀¹ x² dx · ∫₀^∞ e^{-y} dy = (1/3)(1) = 1/3
result_1 = integrate.dblquad(lambda y, x: x**2 * np.exp(-y), 0, 1, 0, np.inf)

# Iterated integral 2 (reversed order)
result_2 = integrate.dblquad(lambda x, y: x**2 * np.exp(-y), 0, np.inf, 0, 1)

print(f"Order 1: {result_1[0]:.6f}")  # 0.333333
print(f"Order 2: {result_2[0]:.6f}")  # 0.333333

---

Conditional Expectation and Radon–Nikodym

Absolute Continuity and the Radon–Nikodym Theorem

If $X$ has density $f_X$ with respect to Lebesgue measure $\lambda$, then $\mu_X \ll \lambda$ with $\frac{d\mu_X}{d\lambda} = f_X$. The probability density function is a Radon–Nikodym derivative.

Application to finance. The Radon–Nikodym theorem enables change of measure — the foundation of risk-neutral pricing. If $P$ is the real-world measure and $Q$ is the risk-neutral measure:

$E_Q[X] = E_P\left[\frac{dQ}{dP} X\right]$

This is the mathematical core of the Fundamental Theorem of Asset Pricing.

Conditional Expectation

The measure-theoretic definition of conditional expectation is one of the deepest ideas in probability. We cannot define $E[X \mid \mathcal{G}]$ as a single number — it is a random variable that is $\mathcal{G}$-measurable, capturing the “best prediction of $X$ given the information in $\mathcal{G}$.”

Existence. This follows from the Radon–Nikodym theorem. Define $\nu(G) = \int_G X \, dP$ on $\mathcal{G}$. Then $\nu \ll P|_\mathcal{G}$, and $E[X \mid \mathcal{G}] = \frac{d\nu}{dP|_\mathcal{G}}$.

Properties of Conditional Expectation

Conditional Expectation as $L^2$ Projection

Here is the connection that ties conditional expectation to the Linear Algebra track. The conditional expectation $E[X \mid \mathcal{G}]$ is the orthogonal projection of $X$ onto $L^2(\Omega, \mathcal{G}, P)$ — the subspace of $\mathcal{G}$-measurable square-integrable random variables.

This connects directly to PCA: PCA projects data onto a low-dimensional subspace that minimizes mean squared error. Conditional expectation is the infinite-dimensional analog — projecting onto the subspace of functions measurable with respect to a sub-sigma-algebra.

For jointly normal $(X, Y)$ with correlation $\rho$, the conditional expectation takes the familiar regression form: $E[Y \mid X = x] = \mu_Y + \rho \frac{\sigma_Y}{\sigma_X}(x - \mu_X)$. The variance reduction is $1 - \rho^2$ — exactly the fraction of variance “explained” by the conditioning.

import numpy as np

# Conditional expectation as L² projection for jointly normal (X, Y)
np.random.seed(42)
n = 5000
rho = 0.7
mu_x, mu_y, sigma_x, sigma_y = 2.0, 3.0, 1.0, 1.0

# Generate bivariate normal
Z1, Z2 = np.random.randn(n), np.random.randn(n)
X = mu_x + sigma_x * Z1
Y = mu_y + sigma_y * (rho * Z1 + np.sqrt(1 - rho**2) * Z2)

# E[Y | X = x] = mu_y + rho * (sigma_y / sigma_x) * (x - mu_x)
slope = rho * sigma_y / sigma_x
Y_hat = mu_y + slope * (X - mu_x)

# Verify: MSE is minimized at the conditional expectation
mse_optimal = np.mean((Y - Y_hat)**2)
mse_mean_only = np.mean((Y - mu_y)**2)
variance_reduction = 1 - mse_optimal / mse_mean_only

print(f"MSE (conditional): {mse_optimal:.4f}")    # ≈ σ_y²(1-ρ²) = 0.51
print(f"MSE (unconditional): {mse_mean_only:.4f}")  # ≈ σ_y² = 1.0
print(f"Variance reduction: {variance_reduction:.4f}")  # ≈ ρ² = 0.49

---

A Preview of Martingales

Filtrations and Adapted Processes

Martingales

The martingale condition says: given everything we know now, our best prediction of tomorrow’s value is today’s value. The process has “no drift” — no systematic tendency to increase or decrease.

Examples

Random walk. Let $Z_1, Z_2, \ldots$ be i.i.d. with $E[Z_i] = 0$. Then $M_n = \sum_{i=1}^n Z_i$ is a martingale:

$E[M_{n+1} \mid \mathcal{F}_n] = E[M_n + Z_{n+1} \mid \mathcal{F}_n] = M_n + E[Z_{n+1} \mid \mathcal{F}_n] = M_n + E[Z_{n+1}] = M_n$

using the “taking out what is known” property and independence.

Pólya urn. Start with 1 red and 1 blue ball. At each step, draw a ball, then replace it with 2 balls of the same color. Let $M_n$ = fraction of red balls after $n$ draws. Then $(M_n)$ is a martingale — and by the martingale convergence theorem, $M_n$ converges almost surely to a $\text{Beta}(1, 1) = \text{Uniform}(0, 1)$ random variable.

Likelihood ratio. If $P$ and $Q$ are probability measures with $Q \ll P$, and $X_1, X_2, \ldots$ are i.i.d. under $P$, then the likelihood ratio $L_n = \prod_{i=1}^n \frac{dQ}{dP}(X_i)$ is a $P$-martingale. This connects to sequential hypothesis testing and the Radon–Nikodym derivative from the previous section.

Financial Interpretation

In mathematical finance, a martingale models a fair game — a process where no betting strategy can generate a positive expected profit.

• A discounted asset price is a martingale under the risk-neutral measure $Q$ (Fundamental Theorem of Asset Pricing).
• The Efficient Market Hypothesis (weak form) asserts that prices, conditioned on historical information, should be martingales.
• In regime detection, the question is whether the martingale property holds uniformly or whether the drift switches between regimes. GARCH(1,1) captures time-varying conditional variance ($\text{Var}(X_t \mid \mathcal{F}_{t-1})$ is $\mathcal{F}_{t-1}$-measurable), while the Statistical Jump Model detects changes in the conditional distribution itself.

---

Connections & Further Reading

Cross-Track and Within-Track Connections

Target	Track	Relationship
PCA & Low-Rank Approximation	Linear Algebra	$\hat{\Sigma} = \frac{1}{n-1} X^T X$ converges to $\Sigma$ by LLN; $L^2$ theory guarantees convergence of eigenvalues
Concentration Inequalities	Probability & Statistics	Builds on $L^p$ spaces and convergence theory to quantify rates of convergence beyond LLN
PAC Learning Framework	Probability & Statistics	Uses measure-theoretic probability to formalize learnability
Bayesian Nonparametrics	Probability & Statistics	Requires conditional expectation, Radon–Nikodym, and product measures for priors on infinite-dimensional spaces
Shannon Entropy & Mutual Information	Information Theory	Entropy is $E[-\log p(X)]$, directly using the expectation and Radon–Nikodym machinery developed here. Differential entropy requires the Lebesgue integral; conditional entropy uses conditional expectation.
Categories & Functors	Category Theory	The category Meas of measurable spaces and measurable functions provides the categorical framework for probability theory. Random variables are morphisms in Meas, and the pushforward of probability measures is functorial.

Financial Applications

Application	Connection
GARCH(1,1)	Conditional variance $\text{Var}(X_t \mid \mathcal{F}_{t-1})$ is filtration-adapted
Statistical Jump Model	Regime probabilities are conditional expectations given observed filtration
Option pricing (Black–Scholes)	Discounted prices are $Q$-martingales; $dQ/dP$ is state price density
Efficient Market Hypothesis	Prices form martingale w.r.t. public information filtration

Notation Reference

Symbol	Meaning
$(\Omega, \mathcal{F}, P)$	Probability space
$\mathcal{B}(\mathbb{R})$	Borel sigma-algebra on $\mathbb{R}$
$\lambda$	Lebesgue measure
$\mathbf{1}_A$	Indicator function of set $A$
$f^+ = \max(f, 0)$	Positive part
$L^p(\mu)$	Space of $p$-integrable functions
$E[X \mid \mathcal{G}]$	Conditional expectation
$\frac{d\nu}{d\mu}$	Radon–Nikodym derivative
$\xrightarrow{\text{a.s.}}$, $\xrightarrow{P}$, $\xrightarrow{L^p}$, $\xrightarrow{d}$	Modes of convergence