Probability and statistics

The third axiom extends to any countable collection of pairwise mutually exclusive events. Consequences: $P(\varnothing )=0$ and $0\le P(A)\le 1$.

Holds for any two events. Use the complement for every 'at least one' question: $P(\text{at least one})=1-P(\text{none})$.

Add singles, subtract pairs, add the triple. The same pattern counts elements of a union: replace $P$ by $n$.

Only when the sample space has finitely many outcomes and all of them are equally likely.

Permutations count ordered arrangements of $r$ of $n$ distinct objects; combinations count unordered selections. $0!=1$ and $\left(\genfrac{}{}{0ex}{}{n}{r}\right)=\left(\genfrac{}{}{0ex}{}{n}{n-r}\right)$.

In the second rule $n_1+n_2+\cdots +n_k=n$. The circular count treats rotations as the same arrangement; if reflections are also the same (a necklace), the count is $(n-1)!/2$ for $n\ge 3$.

$n$ items are drawn without replacement from $N$ items, of which $K$ are of type 1. If the draws are made with replacement, the count is binomial with $p=K/N$ instead.

Undefined when $P(B)=0$. In general $P(A\mid B)\ne P(B\mid A)$.

The chain form is how sequential draws without replacement are multiplied along a tree.

Equivalently $P(A\mid B)=P(A)$ when $P(B)>0$. If $A,B$ are independent, so are $A$ and $B^{\prime }$, $A^{\prime }$ and $B$, and $A^{\prime }$ and $B^{\prime }$. Three events are mutually independent only if all three pairwise products hold AND $P(A\cap B\cap C)=P(A)P(B)P(C)$; pairwise independence alone is not enough.

$B_1,\dots ,B_n$ must partition $S$: mutually exclusive, together exhaustive, each with $P(B_i)>0$.

Same partition conditions, and $P(A)>0$. $P(B_k)$ is the prior probability of cause $B_k$; $P(B_k\mid A)$ is its posterior probability after observing $A$.

To find an unknown constant in a PMF, set the total equal to 1.

For a continuous variable $P(X=a)=0$, so $<$ and $\le$ give the same probability.

Every CDF is non-decreasing and right-continuous, with $F(x)\to 0$ as $x\to -\infty$ and $F(x)\to 1$ as $x\to \infty$. The relation $f=dF/dx$ holds for a continuous variable at every $x$ where $F$ is differentiable. For a discrete variable $F$ is a staircase that jumps by $p(x)$ at each value $x$, so $P(X<b)=F(b)-P(X=b)$ and strict and non-strict inequalities give different answers.

The mean exists only when the sum or integral converges absolutely. $E[g(X)]$ does not need the distribution of $g(X)$ itself.

Here $\mu =E[X]$. Variance is never negative, and is zero only when $X$ is a constant with probability 1. The second form is the one to compute with.

For any constants $a,b$. Adding $b$ moves the mean but not the spread; the sign of $a$ does not affect the variance.

For a $g$ that is not monotonic, work through the CDF: find $F_Y(y)=P(g(X)\le y)$, then differentiate. For example $Y=X^2$ gives $F_Y(y)=F_X(\sqrt{y})-F_X(-\sqrt{y})$ and $f_Y(y)=\frac{f_X(\sqrt{y})+f_X(-\sqrt{y})}{2\sqrt{y}}$ for $y>0$, with $X$ continuous.

The first raw moment is the mean and the second central moment is the variance.

Where $M_X$ exists on an open interval containing $t=0$. For independent $X$ and $Y$, $M_{X+Y}(t)=M_X(t)M_Y(t)$.

For a discrete variable, a median is any $m$ with $P(X\le m)\ge \frac{1}{2}$ and $P(X\ge m)\ge \frac{1}{2}$. A mode can sit at an end-point of the range.

Number of successes in $n$ independent trials, each with the same success probability $p$. Mean $np$, variance $np(1-p)$, which is less than the mean whenever $p>0$.

Count of events that occur independently at a constant average rate. Mean = variance = $\lambda$. If events occur as a Poisson process at rate $\alpha$ per unit time, the count in a time $t$ is Poisson with $\lambda =\alpha t$.

Used as an approximation when $n$ is large and $p$ is small (rare events): put $\lambda =np$.

Convention here: $X$ is the number of the trial on which the first success occurs, and $P(X>n)=(1-p{)}^n$. If $X$ instead counts the failures before the first success, $P(X=x)=(1-p{)}^{x}p$ for $x=0,1,\dots$, the mean becomes $(1-p)/p$ and the variance is unchanged. Among distributions on the positive integers, the geometric is the only memoryless one.

$F(x)=0$ for $x<a$ and $F(x)=1$ for $x>b$. The probability of a sub-interval of $[a,b]$ is its length divided by $b-a$.

Rate $\lambda >0$; $f$ and $F$ are zero for $x<0$. Mean $1/\lambda$ and variance $1/{\lambda }^2$, so the standard deviation equals the mean; the median is $\ln 2/\lambda$. It is the waiting time to the first event, and between successive events, of a Poisson process of rate $\lambda$.

Holds for the exponential, the only continuous distribution with this property: a component that has survived to time $s$ has the same remaining-life distribution as a new one.

Proof: $P(\min >t)={\prod }_{i}P(X_i>t)=e^{-({\lambda }_1+\cdots +{\lambda }_n)t}$. Rates add, not means. The maximum is not exponential: $P(\max \le t)={\prod }_i\left(1-e^{-{\lambda }_{i}t}\right)$.

Symmetric about $\mu$; mean = median = mode = $\mu$; points of inflection at $\mu \pm \sigma$; peak height $1/(\sigma \sqrt{2\pi })$.

Convention in these notes: $\Phi (z)=P(Z\le z)$, the area to the LEFT of $z$, with $\Phi (0)=0.5$. Some tables list the area from 0 to $z$, which equals $\Phi (z)-0.5$; check which one you are given.

For a normal variable only. The rounded figures 68%, 95% and 99.7% are approximations; use the table value whenever a question supplies one. Standard quantiles: $\Phi (1.645)\approx 0.95$, $\Phi (1.96)\approx 0.975$, $\Phi (2.576)\approx 0.995$.

Discrete case: replace the integrals by sums over the joint PMF table.

Discrete: $p_X(x)={\sum }_{y}p(x,y)$ (the totals along one direction of the table) and $P(Y=y\mid X=x)=p(x,y)/p_X(x)$.

Discrete case: one pair of values where the equality fails proves dependence. Continuous case: the equality must fail on a region of positive area (a single point proves nothing, since a density can be changed at a point); if both sides are continuous, one point of failure is enough. If the set where $f>0$ is not of the product form $\{x\in A\}\times \{y\in B\}$ (for example the triangle $0<x<y<1$), $X$ and $Y$ cannot be independent.

$B_1,\dots ,B_n$ partition $S$ with each $P(B_i)>0$. $E[X\mid Y=y]$ is the mean of the conditional distribution: ${\sum }_{x}xP(X=x\mid Y=y)$ or $\int x{f}_{X\mid Y}(x\mid y)dx$. This is total probability applied to means.

$\mathrm{Cov}(X,X)=\mathrm{Var}(X)$ and $\mathrm{Cov}(aX+b,cY+d)=ac\mathrm{Cov}(X,Y)$.

Defined only when ${\sigma }_X>0$ and ${\sigma }_Y>0$. $|\rho |=1$ exactly when $Y=aX+b$ with probability 1 for some $a\ne 0$ ($\rho$ then has the sign of $a$). $\rho$ is unchanged by shifting either variable or rescaling it by a positive factor; rescaling one variable by a negative factor reverses its sign.

Always true. When $X,Y$ are uncorrelated (in particular, independent) the covariance term drops, so $\mathrm{Var}(X-Y)=\mathrm{Var}(X)+\mathrm{Var}(Y)$: variances add even for a difference.

Means always add; variances add when the variables are independent (pairwise uncorrelated is enough). Also, for independent $X,Y$: $E[XY]=E[X]E[Y]$.

Each line requires the variables to be independent. A linear combination $aX+bY$ of independent normals is normal with mean $a{\mu }_1+b{\mu }_2$ and variance $a^2{\sigma }_1^2+b^2{\sigma }_2^2$.

For grouped (class-interval) data, $x_i$ is the class mid-point and the result is an approximation. Deviations from the mean always sum to zero (weighted by frequency for frequency data).

${\sigma }^2$: population (divisor $n$); $s^2$: sample (divisor $n-1$). The standard deviation is the positive square root. Adding a constant to every value changes neither; multiplying every value by $c$ multiplies the standard deviation by $|c|$.

$L$ = lower boundary of the median (or modal) class, $h$ = class width, $N$ = total frequency, $C$ = cumulative frequency of the classes before the median class, $f$ = frequency of the median class; $f_1$ = frequency of the modal class, $f_0$ and $f_2$ those of the classes just before and after it. The mode formula assumes the modal class and its two neighbours have the same width $h$.

An approximate rule for moderately skewed, single-peaked data only; it is not an identity.

Defined when neither variable is constant ($S_{xx},S_{yy}>0$); then $-1\le r\le 1$. The divisors cancel, so $r$ is the same whether $n$ or $n-1$ is used, provided the same one is used throughout. Computational form: $S_{xy}=\sum x_i{y}_i-n\bar{x}\bar{y}$.

$d_i$ is the difference between the two ranks of item $i$. Valid when there are no tied ranks; with ties, give tied items the average of their ranks and compute $r$ from the ranks directly. Like $r$, it lies between $-1$ and $1$.

The line of $y$ on $x$ minimises the sum of squared vertical deviations and is used to predict $y$ from $x$; the line of $x$ on $y$ minimises squared horizontal deviations and predicts $x$ from $y$. Both lines pass through $(\bar{x},\bar{y})$.

$r$, $b_{yx}$ and $b_{xy}$ all have the same sign, and that common sign is taken for $r$. Since $r^2\le 1$, the two coefficients cannot both exceed 1 in magnitude.

Solving them gives $b=S_{xy}/S_{xx}$ and $a=\bar{y}-b\bar{x}$, which is the regression line of $y$ on $x$.

Needs only independence (uncorrelated is enough) and a common mean and variance, not normality. Quadrupling $n$ halves the standard error.

Dividing by $n$ instead gives an estimate whose expected value is $\frac{n-1}{n}{\sigma }^2$, slightly too small; this is why the sample variance uses $n-1$.

For independent, identically distributed $X_i$ with finite, non-zero variance, whatever their distribution. If the $X_i$ are themselves normal, the result is exact for every $n$.

A consequence of the CLT. The $+0.5$ is the continuity correction for a discrete variable; in the same way $P(X\ge k)\approx 1-\Phi \left(\frac{k-0.5-np}{\sqrt{np(1-p)}}\right)$. A common rule of thumb asks for $np$ and $n(1-p)$ both at least about 5 (some texts ask for 10).

Holds for ANY distribution with finite variance, so it is a loose bound (and says nothing for $k\le 1$): for a normal variable the true probability at $k=2$ is about $0.046$, against the bound $0.25$. Applied to $\bar{X}$, whose variance is ${\sigma }^2/n$, it gives $P(|\bar{X}-\mu |\ge \epsilon )\le \frac{{\sigma }^2}{n{\epsilon }^2}\to 0$ for every $\epsilon >0$: the weak law of large numbers.

If $F$ is a continuous, strictly increasing CDF, then $X$ has CDF $F$. Example: $X=-\frac{1}{\lambda }\ln (1-U)$ is exponential with rate $\lambda$.