Expectation and Variance

Expected Value

The expectation (mean) of a random variable $X$:

$$\mathbb{E}[X] = \begin{cases} \sum_x x \cdot p(x) & \text{discrete} \ \int_{-\infty}^{\infty} x \cdot f(x),dx & \text{continuous} \end{cases}$$

Key properties (by linearity):

$\mathbb{E}[aX + b] = a\mathbb{E}[X] + b$
$\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y]$ (always, no independence needed)
$\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y]$ only if $X \perp Y$

$$\text{Var}(X) = \mathbb{E}[(X - \mu)^2] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$$

where $\mu = \mathbb{E}[X]$. The standard deviation is $\sigma = \sqrt{\text{Var}(X)}$.

Properties:

$$\text{Cov}(X, Y) = \mathbb{E}[(X - \mu_X)(Y - \mu_Y)] = \mathbb{E}[XY] - \mu_X \mu_Y$$

The Pearson correlation coefficient normalises this to $[-1, 1]$:

$$\rho_{XY} = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}$$

Correlation ≠ Causation

$\rho \approx 0$ means no linear relationship, but there could be a strong nonlinear one. And even high $|\rho|$ says nothing about causation.

The MGF of $X$ is $M_X(t) = \mathbb{E}[e^{tX}]$, when it exists in a neighbourhood of $t = 0$.

The $n$-th moment satisfies:

$$\mathbb{E}[X^n] = M_X^{(n)}(0)$$

MGFs uniquely determine distributions (when they exist), which makes them powerful for proving convergence theorems.