The whole Engineering Mathematics formula sheet, to print

251 formulas in 7 topics, each with when to use it. Each is explained in the topic notes.

Linear algebra

REVERSAL LAWS

(A B)^{T} = B^{T} A^{T}, (A B)^{- 1} = B^{- 1} A^{- 1}, (A^{T})^{- 1} = (A^{- 1})^{T}, (A B)^{*} = B^{*} A^{*}

The order reverses for any number of factors, e.g. $(A B C)^{- 1} = C^{- 1} B^{- 1} A^{- 1}$ . The transpose and conjugate-transpose laws hold whenever $A B$ is defined, square or not; the inverse laws need every factor square and non-singular.

MORE INVERSE RULES

(A^{- 1})^{- 1} = A, (k A)^{- 1} = \frac{1}{k} A^{- 1}, (A^{m})^{- 1} = (A^{- 1})^{m}, (A^{*})^{- 1} = (A^{- 1})^{*}

$A$ non-singular, $k \neq = 0$ a scalar and $m$ a positive integer. A diagonal matrix with non-zero diagonal entries $d_{1}, \dots, d_{n}$ has inverse $diag (1/ d_{1}, \dots, 1/ d_{n})$ , and the inverse of a non-singular upper (lower) triangular matrix is again upper (lower) triangular.

INVERSE THROUGH THE ADJOINT

A (adj A) = (adj A) A = ∣ A ∣ I, A^{- 1} = \frac{adj A}{∣ A ∣}

The first identity holds for every square matrix; the inverse formula needs $∣ A ∣ \neq = 0$ .

INVERSE OF A 2 × 2 MATRIX

[a c b d]^{- 1} = \frac{1}{a d - b c} [d - c - b a]

Swap the diagonal entries, change the sign of the off-diagonal entries, divide by the determinant. Valid only when $a d - b c \neq = 0$ .

TRACE PROPERTIES

tr (A + B) = tr A + tr B, tr (k A) = k tr A, tr (A B) = tr (B A), tr A^{T} = tr A

$tr (A B) = tr (B A)$ holds whenever both products are defined, even for non-square $A$ and $B$ . The trace is unchanged by cyclic shifts, $tr (A B C) = tr (B C A) = tr (C A B)$ , but not by other reorderings: in general $tr (A B C) \neq = tr (A C B)$ . In general $tr (A B) \neq = tr A tr B$ .

DETERMINANTS OF PRODUCTS, POWERS AND MULTIPLES

∣ A B ∣ = ∣ A ∣ ∣ B ∣, ∣ A^{m} ∣ = ∣ A ∣^{m}, ∣ A^{- 1} ∣ = \frac{1}{∣ A ∣}, ∣ k A ∣ = k^{n} ∣ A ∣

$A$ and $B$ are $n \times n$ and $k$ is a scalar. $∣ A^{m} ∣ = ∣ A ∣^{m}$ holds for every positive integer $m$ (for every integer $m$ when $A$ is non-singular), and $∣ A^{- 1} ∣ = 1/∣ A ∣$ needs $∣ A ∣ \neq = 0$ . $∣ A B ∣ = ∣ B A ∣$ even when $A B \neq = B A$ .

ADJOINT IDENTITIES

∣ adj A ∣ = ∣ A ∣^{n - 1}, adj (adj A) = ∣ A ∣^{n - 2} A, adj (A B) = (adj B) (adj A), adj (k A) = k^{n - 1} adj A

For $n \times n$ matrices with $n \geq 2$ ; all four hold for singular matrices too. The first follows from taking determinants of $A adj A = ∣ A ∣ I$ , which gives $∣ A ∣ ∣ adj A ∣ = ∣ A ∣^{n}$ . Consequently $∣ adj (adj A) ∣ = ∣ A ∣^{(n - 1)^{2}}$ . Also $adj (A^{T}) = (adj A)^{T}$ , and for non-singular $A$ , $adj (A^{- 1}) = (adj A)^{- 1} = A /∣ A ∣$ .

INNER PRODUCT, LENGTH AND ORTHOGONALITY

u \cdot v = u^{T} v = i = 1 \sum n u_{i} v_{i}, ∥ u ∥ = u^{T} u, u ⊥ v ⟺ u^{T} v = 0

For real vectors in $R^{n}$ . For complex vectors use $u^{*} v$ and $∥ u ∥ = u^{*} u$ . A set of vectors is orthonormal if they are mutually orthogonal and each has length $1$ . In an orthonormal basis $e_{1}, \dots, e_{n}$ the coordinates of any $v$ are simply $c_{i} = e_{i}^{T} v$ .

DIMENSION OF A SUM OF SUBSPACES

dim (U + W) = dim U + dim W - dim (U \cap W)

$U$ and $W$ are subspaces of a finite-dimensional space and $U + W$ is the set of all sums $u + w$ . Two distinct planes through the origin in $R^{3}$ meet in a line: $3 = 2 + 2 - 1$ .

BOUNDS ON RANK

0 \leq rank A \leq min (m, n), rank A^{T} = rank A

For an $m \times n$ matrix. A square $n \times n$ matrix has rank $n$ exactly when $∣ A ∣ \neq = 0$ .

RANK OF A PRODUCT (SYLVESTER'S INEQUALITY ON THE LEFT)

rank A + rank B - n \leq rank (A B) \leq min (rank A, rank B)

$A$ is $m \times n$ and $B$ is $n \times p$ , so $n$ is the shared inner dimension. Multiplying by a non-singular matrix does not change the rank: $rank (P A) = rank (A Q) = rank A$ for invertible $P$ and $Q$ .

RANK-NULLITY THEOREM

rank A + nullity A = n

$n$ is the number of columns of $A$ (the number of unknowns in $A X = 0$ ), and $nullity A$ is the dimension of the solution space of $A X = 0$ .

CONSISTENCY CONDITION (ROUCHÉ–CAPELLI THEOREM)

A X = B is consistent ⟺ rank A = rank [A ∣ B]

When consistent, the solution is unique if this common rank equals the number of unknowns $n$ , and has $n - r$ free parameters if the common rank is $r < n$ .

NON-TRIVIAL SOLUTIONS OF A SQUARE HOMOGENEOUS SYSTEM

A X = 0 has a non-trivial solution ⟺ ∣ A ∣ = 0 ⟺ 0 is an eigenvalue of A

$A$ is $n \times n$ . The number of linearly independent solutions is $n - rank A$ .

CRAMER'S RULE

x_{i} = \frac{Δ _{i}}{Δ}, Δ = ∣ A ∣ \neq = 0

$Δ_{i}$ is the determinant of $A$ with its $i$ -th column replaced by $B$ . Applies only to a square system with $Δ \neq = 0$ . If $Δ = 0$ and some $Δ_{i} \neq = 0$ , the system is inconsistent.

CHARACTERISTIC EQUATION OF A 2 × 2 MATRIX

λ^{2} - (tr A) λ + ∣ A ∣ = 0

For $A = [a c b d]$ : $λ^{2} - (a + d) λ + (a d - b c) = 0$ , so $λ = \frac{( a + d ) \pm ( a - d ) ^{2} + 4 b c}{2}$ . The eigenvalues are real whenever $b c \geq 0$ , in particular whenever $A$ is real symmetric.

CHARACTERISTIC EQUATION OF A 3 × 3 MATRIX

λ^{3} - (tr A) λ^{2} + (C_{11} + C_{22} + C_{33}) λ - ∣ A ∣ = 0

This is $∣ A - λ I ∣ = 0$ multiplied through by $- 1$ . $C_{11} + C_{22} + C_{33}$ is the sum of the three principal $2 \times 2$ minors (the cofactors of the diagonal entries); it also equals $λ_{1} λ_{2} + λ_{2} λ_{3} + λ_{3} λ_{1}$ .

SUM AND PRODUCT OF EIGENVALUES

λ_{1} + λ_{2} + \dots + λ_{n} = tr A, λ_{1} λ_{2} \dots λ_{n} = ∣ A ∣

Each eigenvalue is counted as often as its algebraic multiplicity, complex ones included. More generally $tr (A^{m}) = λ_{1}^{m} + λ_{2}^{m} + \dots + λ_{n}^{m}$ for every positive integer $m$ ; for instance $tr (A^{2})$ is the sum of the squares of the eigenvalues. A quick check on any eigenvalue calculation.

GEOMETRIC MULTIPLICITY

GM (λ) = n - rank (A - λ I), 1 \leq GM (λ) \leq AM (λ)

An eigenvalue with AM $= 1$ (a simple root) always has GM $= 1$ . A repeated eigenvalue can have any GM from $1$ up to its AM.

POWERS AND FUNCTIONS THROUGH DIAGONALISATION

A = P D P^{- 1} \Rightarrow A^{k} = P D^{k} P^{- 1}, f (A) = P f (D) P^{- 1}

$f (D) = diag (f (λ_{1}), \dots, f (λ_{n}))$ acts entry by entry on the diagonal. $f$ is a polynomial, or a convergent power series such as the exponential: $e^{A t} = P diag (e^{λ_{1} t}, \dots, e^{λ_{n} t}) P^{- 1}$ . For a real symmetric $A$ , take orthonormal eigenvectors as the columns of $P$ (normalise each one and, inside the eigenspace of a repeated eigenvalue, also make them orthogonal); then $P^{- 1} = P^{T}$ . Normalising alone is not enough when an eigenvalue repeats: $(1, - 1, 0)^{T}$ and $(1, 0, - 1)^{T}$ in the Section 5 example are not orthogonal.

CAYLEY–HAMILTON THEOREM

A^{n} + c_{n - 1} A^{n - 1} + \dots + c_{1} A + c_{0} I = O, c_{n - 1} = - tr A, c_{0} = (- 1)^{n} ∣ A ∣

Using $∣ A - λ I ∣$ instead of $det (λ I - A)$ only multiplies the polynomial by $(- 1)^{n}$ , which does not change the conclusion.

INVERSE FROM CAYLEY–HAMILTON

A^{- 1} = - \frac{1}{c _{0}} (A^{n - 1} + c_{n - 1} A^{n - 2} + \dots + c_{1} I)

Multiply the Cayley–Hamilton identity by $A^{- 1}$ . Needs $c_{0} \neq = 0$ , that is, $∣ A ∣ \neq = 0$ .

THE 2 × 2 CASE

A^{2} - (tr A) A + ∣ A ∣ I = O, A^{- 1} = \frac{( tr A ) I - A}{∣ A ∣}

For any $2 \times 2$ matrix; the inverse needs $∣ A ∣ \neq = 0$ .

HIGHER POWERS BY THE REMAINDER METHOD

λ^{m} = q (λ) p (λ) + r (λ), de g r \leq n - 1 \Rightarrow A^{m} = r (A)

Here $p$ is the characteristic polynomial (degree $n$ ), and the result follows because $p (A) = O$ . With distinct eigenvalues, find the $n$ coefficients of $r$ from $r (λ_{i}) = λ_{i}^{m}$ . For an eigenvalue of algebraic multiplicity $k$ , also match the first $k - 1$ derivatives; for a double root this is $r^{'} (λ_{i}) = m λ_{i}^{m - 1}$ .

TRANSFORMATION LAW FOR CARTESIAN TENSORS

v_{i}^{'} = a_{ij} v_{j}, T_{ij}^{'} = a_{i p} a_{j q} T_{pq}

Under a rotation of axes, with $a_{ij}$ the cosine of the angle between the new axis $x_{i}^{'}$ and the old axis $x_{j}$ . The matrix $[a_{ij}]$ is orthogonal, $a_{ik} a_{j k} = δ_{ij}$ . A third-order tensor picks up three factors: $T_{ij k}^{'} = a_{i p} a_{j q} a_{k r} T_{pq r}$ .

Calculus: limits, mean value theorems, integrals, maxima and minima, series

Read the notes

STANDARD LIMITS AT ZERO

x \to 0 lim \frac{sin x}{x} = 1, x \to 0 lim \frac{tan x}{x} = 1, x \to 0 lim \frac{1 - cos x}{x ^{2}} = \frac{1}{2} x \to 0 lim \frac{e ^{x} - 1}{x} = 1, x \to 0 lim \frac{a ^{x} - 1}{x} = ln a, x \to 0 lim \frac{ln ( 1 + x )}{x} = 1

$x$ in radians, $a > 0$ . Each follows from the Maclaurin series of the function.

EXPONENTIAL AND POWER LIMITS

x \to 0 lim (1 + x)^{1/ x} = e, x \to \infty lim (1 + \frac{a}{x})^{x} = e^{a}, x \to a lim \frac{x ^{n} - a ^{n}}{x - a} = n a^{n - 1}

The last holds for any real $n$ when $a > 0$ .

ORDERS OF GROWTH

x \to \infty lim \frac{ln x}{x ^{p}} = 0, x \to \infty lim \frac{x ^{p}}{e ^{a x}} = 0, x \to 0^{+} lim x^{p} ln x = 0

For $p > 0$ and $a > 0$ : a logarithm grows more slowly than any positive power, and a power more slowly than any growing exponential. Each follows from L'Hospital's rule. Consequence: $lim_{x \to 0^{+}} x^{x} = lim_{x \to 0^{+}} e^{x l n x} = e^{0} = 1$ .

THE ONE-TO-THE-POWER-INFINITY RULE

f (x) \to 1, ∣ g (x) ∣ \to \infty \Rightarrow lim f (x)^{g (x)} = e^{l i m g (x) [f (x) - 1]}

Valid whenever the limit in the exponent exists; if that limit is $+ \infty$ the answer is $\infty$ , and if it is $- \infty$ the answer is $0$ . It comes from $ln f \approx f - 1$ when $f \to 1$ .

L'HOSPITAL'S RULE

x \to a lim \frac{f ( x )}{g ( x )} = x \to a lim \frac{f ^{'} ( x )}{g ^{'} ( x )}

Only for the forms $\frac{0}{0}$ or $\frac{\infty}{\infty}$ , with $f, g$ differentiable and $g^{'} (x) \neq = 0$ near $a$ (except possibly at $a$ ), and only if $lim f^{'} (x) / g^{'} (x)$ exists (finite or infinite). Differentiate numerator and denominator separately, not by the quotient rule. Also valid for $a = \pm \infty$ and for one-sided limits.

CONTINUITY AND DIFFERENTIABILITY AT A POINT

x \to a lim f (x) = f (a), f^{'} (a) = h \to 0 lim \frac{f ( a + h ) - f ( a )}{h}

Continuity needs $f (a)$ defined, the limit to exist, and the two to be equal. Differentiability needs the left derivative ( $h \to 0^{-}$ ) and the right derivative ( $h \to 0^{+}$ ) to be finite and equal. Differentiable $\Rightarrow$ continuous; the converse is false.

ROLLE'S THEOREM

f continuous on [a, b], differentiable on (a, b), f (a) = f (b) \Rightarrow f^{'} (c) = 0 for some c \in (a, b)

Geometrically, a horizontal tangent between two equal heights. $f (x) = ∣ x ∣$ on $[- 1, 1]$ has $f (- 1) = f (1)$ but no such $c$ , because it is not differentiable at 0.

LAGRANGE'S MEAN VALUE THEOREM

f^{'} (c) = \frac{f ( b ) - f ( a )}{b - a} for some c \in (a, b)

Needs $f$ continuous on $[a, b]$ and differentiable on $(a, b)$ . The tangent at $c$ is parallel to the chord. Rolle's theorem is the case $f (a) = f (b)$ . Equivalent form: $f (b) = f (a) + (b - a) f^{'} (c)$ . Consequences on an interval: $f^{'} = 0$ throughout $\Rightarrow$ $f$ is constant; $f^{'} > 0$ $\Rightarrow$ $f$ is strictly increasing; $f^{'} < 0$ $\Rightarrow$ strictly decreasing; $∣ f^{'} ∣ \leq M$ $\Rightarrow$ $∣ f (b) - f (a) ∣ \leq M (b - a)$ .

CAUCHY'S MEAN VALUE THEOREM

\frac{f ^{'} ( c )}{g ^{'} ( c )} = \frac{f ( b ) - f ( a )}{g ( b ) - g ( a )} for some c \in (a, b)

Needs $f, g$ continuous on $[a, b]$ , differentiable on $(a, b)$ , and $g^{'} (x) \neq = 0$ on $(a, b)$ (which also forces $g (b) \neq = g (a)$ ). Lagrange's theorem is the case $g (x) = x$ .

TAYLOR SERIES ABOUT X = A

f (x) = n = 0 \sum \infty \frac{f ^{(n)} ( a )}{n !} (x - a)^{n} = f (a) + f^{'} (a) (x - a) + \frac{f ^{''} ( a )}{2 !} (x - a)^{2} + \dots

Maclaurin series when $a = 0$ . Needs derivatives of every order at $a$ , and the series equals $f (x)$ exactly when the remainder $R_{n} (x) \to 0$ ; for the standard functions below this holds throughout the stated range. A convergent Taylor series can still differ from $f$ : $f (x) = e^{- 1/ x^{2}}$ with $f (0) = 0$ has every derivative zero at 0, so its Maclaurin series is 0. A power series that equals $f$ near $a$ is its Taylor series, so series may be built by substitution, multiplication, and term-by-term differentiation or integration inside the interval of convergence.

MACLAURIN SERIES VALID FOR ALL X

e^{x} sin x cos x sinh x = 1 + x + \frac{x ^{2}}{2 !} + \frac{x ^{3}}{3 !} + \dots = x - \frac{x ^{3}}{3 !} + \frac{x ^{5}}{5 !} - \dots = 1 - \frac{x ^{2}}{2 !} + \frac{x ^{4}}{4 !} - \dots = x + \frac{x ^{3}}{3 !} + \frac{x ^{5}}{5 !} + \dots, cosh x = 1 + \frac{x ^{2}}{2 !} + \frac{x ^{4}}{4 !} + \dots

Odd functions have only odd powers, even functions only even powers.

MACLAURIN SERIES WITH A RESTRICTED RANGE

ln (1 + x) \frac{1}{1 - x} (1 + x)^{k} tan^{- 1} x tan x = x - \frac{x ^{2}}{2} + \frac{x ^{3}}{3} - \dots = 1 + x + x^{2} + x^{3} + \dots = 1 + k x + \frac{k ( k - 1 )}{2 !} x^{2} + \frac{k ( k - 1 ) ( k - 2 )}{3 !} x^{3} + \dots = x - \frac{x ^{3}}{3} + \frac{x ^{5}}{5} - \dots = x + \frac{x ^{3}}{3} + \frac{2 x ^{5}}{15} + \dots - 1 < x \leq 1 ∣ x ∣ < 1 ∣ x ∣ < 1 ∣ x ∣ \leq 1 ∣ x ∣ < \frac{π}{2}

For a non-negative integer $k$ the binomial series stops after $k + 1$ terms and holds for all $x$ . Replacing $x$ by $- x$ gives $ln (1 - x) = - x - \frac{x ^{2}}{2} - \frac{x ^{3}}{3} - \dots$ for $- 1 \leq x < 1$ , and $\frac{1}{1 + x} = 1 - x + x^{2} - \dots$ for $∣ x ∣ < 1$ .

FUNDAMENTAL THEOREM OF CALCULUS

\int_{a}^{b} f (x) d x = F (b) - F (a) where F^{'} = f, \frac{d}{d x} \int_{a}^{x} f (t) d t = f (x)

Needs $f$ continuous on $[a, b]$ . Evaluating an antiderivative across a point where $f$ is unbounded is not justified and can give a wrong answer (see the tip at the end of this section): split the integral there and treat each piece as improper.

INTEGRATION BY PARTS

\int_{a}^{b} u v^{'} d x = [u v]_{a}^{b} - \int_{a}^{b} u^{'} v d x \int e^{a x} sin b x d x = \frac{e ^{a x} ( a sin b x - b cos b x )}{a ^{2} + b ^{2}} + C \int e^{a x} cos b x d x = \frac{e ^{a x} ( a cos b x + b sin b x )}{a ^{2} + b ^{2}} + C

$u$ and $v$ with continuous derivatives on $[a, b]$ . Choose as $u$ the factor that simplifies when differentiated (a power of $x$ , a logarithm, an inverse trigonometric function). The last two lines come from integrating by parts twice; $a^{2} + b^{2} \neq = 0$ and $C$ is an arbitrary constant.

KING'S PROPERTY

\int_{a}^{b} f (x) d x = \int_{a}^{b} f (a + b - x) d x, \int_{0}^{a} f (x) d x = \int_{0}^{a} f (a - x) d x

Substitute $x \to a + b - x$ . Adding the two forms often cancels the awkward part: $2 I = \int_{a}^{b} [f (x) + f (a + b - x)] d x$ .

EVEN AND ODD INTEGRANDS

\int_{- a}^{a} f (x) d x = 2 \int_{0}^{a} f (x) d x (f even), \int_{- a}^{a} f (x) d x = 0 (f odd)

Even: $f (- x) = f (x)$ ; odd: $f (- x) = - f (x)$ . Any $f$ is its even part $\frac{f ( x ) + f ( - x )}{2}$ plus its odd part $\frac{f ( x ) - f ( - x )}{2}$ , and only the even part survives a symmetric integral.

REFLECTION ABOUT THE MIDPOINT

\int_{0}^{2 a} f (x) d x = 2 \int_{0}^{a} f (x) d x if f (2 a - x) = f (x); \int_{0}^{2 a} f (x) d x = 0 if f (2 a - x) = - f (x)

Example: $\int_{0}^{π} sin^{n} x d x = 2 \int_{0}^{π /2} sin^{n} x d x$ , and $\int_{0}^{π} cos^{n} x d x = 0$ for odd $n$ .

PERIODIC INTEGRANDS

\int_{0}^{n T} f (x) d x = n \int_{0}^{T} f (x) d x, \int_{a}^{a + T} f (x) d x = \int_{0}^{T} f (x) d x

For $f (x + T) = f (x)$ and a positive integer $n$ : every full period gives the same integral. Example: $∣ sin x ∣$ has period $π$ , so $\int_{0}^{10 π} ∣ sin x ∣ d x = 10 \int_{0}^{π} sin x d x = 20$ .

REDUCTION FORMULA FOR POWERS OF SINE AND COSINE

\int_{0}^{π /2} sin^{n} x d x = \int_{0}^{π /2} cos^{n} x d x = \frac{( n - 1 ) ( n - 3 ) \dots}{n ( n - 2 ) \dots} \times {\frac{π}{2} 1 n even n odd

$n$ a positive integer. Each product keeps only its positive factors, so it runs down to 1 or 2; a product with no factors is 1, so $n = 1$ gives 1. Examples: $n = 4$ gives $\frac{3 \cdot 1}{4 \cdot 2} \cdot \frac{π}{2} = \frac{3 π}{16}$ ; $n = 5$ gives $\frac{4 \cdot 2}{5 \cdot 3 \cdot 1} = \frac{8}{15}$ .

PRODUCTS OF POWERS OF SINE AND COSINE

\int_{0}^{π /2} sin^{m} x cos^{n} x d x = \frac{[ ( m - 1 ) ( m - 3 ) \dots ] [ ( n - 1 ) ( n - 3 ) \dots ]}{( m + n ) ( m + n - 2 ) \dots} \times {\frac{π}{2} 1 m and n both even otherwise

$m, n$ non-negative integers. Each product keeps only its positive factors, running down to 1 or 2, and a product with no factors (for $m$ or $n$ equal to 0 or 1) is 1. With $m = 0$ it is the formula above. Examples: $m = n = 2$ gives $\frac{1 \cdot 1}{4 \cdot 2} \cdot \frac{π}{2} = \frac{π}{16}$ ; $m = 3, n = 2$ gives $\frac{2 \cdot 1}{5 \cdot 3 \cdot 1} = \frac{2}{15}$ . The Beta function below gives the same values.

LEIBNIZ RULE

\frac{d}{d x} \int_{a (x)}^{b (x)} f (x, t) d t = f (x, b (x)) b^{'} (x) - f (x, a (x)) a^{'} (x) + \int_{a (x)}^{b (x)} \frac{\partial f}{\partial x} (x, t) d t

Needs $f$ and $\partial f / \partial x$ continuous and $a (x), b (x)$ differentiable. If the integrand does not contain $x$ the last term is zero; if the limits are constants only the last term remains (differentiation under the integral sign).

P-TEST FOR IMPROPER INTEGRALS

\int_{1}^{\infty} \frac{d x}{x ^{p}} converges ⟺ p > 1 (value \frac{1}{p - 1}), \int_{0}^{1} \frac{d x}{x ^{p}} converges ⟺ p < 1 (value \frac{1}{1 - p})

Both diverge at $p = 1$ (logarithm). For non-negative $f$ and $g$ : comparison test, if $0 \leq f \leq g$ then $\int g$ convergent $\Rightarrow$ $\int f$ convergent, and $\int f$ divergent $\Rightarrow$ $\int g$ divergent; limit comparison, if $f / g$ tends to a finite non-zero limit, both converge or both diverge. For any $f$ : if $\int ∣ f ∣$ converges, so does $\int f$ (absolute convergence).

STANDARD IMPROPER INTEGRALS

\int_{0}^{\infty} e^{- a x} d x = \frac{1}{a}, \int_{0}^{\infty} x^{n} e^{- a x} d x = \frac{n !}{a ^{n + 1}}, \int_{- \infty}^{\infty} e^{- a x^{2}} d x = \frac{π}{a} \int_{0}^{\infty} e^{- a x} cos b x d x = \frac{a}{a ^{2} + b ^{2}}, \int_{0}^{\infty} e^{- a x} sin b x d x = \frac{b}{a ^{2} + b ^{2}}, \int_{0}^{\infty} \frac{sin x}{x} d x = \frac{π}{2}

$a > 0$ , $b$ real and $n$ a non-negative integer; for real $n > - 1$ the second result is $\frac{Γ ( n + 1 )}{a ^{n + 1}}$ . The last integral converges, but not absolutely.

GAMMA FUNCTION

Γ (n) = \int_{0}^{\infty} x^{n - 1} e^{- x} d x (n > 0), Γ (n + 1) = n Γ (n), Γ (\frac{1}{2}) = π

$Γ (n + 1) = n!$ for non-negative integers $n$ , so $Γ (1) = 1$ . Example: $Γ (\frac{5}{2}) = \frac{3}{2} \cdot \frac{1}{2} π = \frac{3 π}{4}$ .

BETA FUNCTION

B (m, n) = \int_{0}^{1} x^{m - 1} (1 - x)^{n - 1} d x = \frac{Γ ( m ) Γ ( n )}{Γ ( m + n )} = 2 \int_{0}^{π /2} sin^{2 m - 1} θ cos^{2 n - 1} θ d θ

$m, n > 0$ and $B (m, n) = B (n, m)$ . The trigonometric form gives $\int_{0}^{π /2} sin^{p} θ cos^{q} θ d θ = \frac{1}{2} B (\frac{p + 1}{2}, \frac{q + 1}{2})$ for $p, q > - 1$ .

AREA, ARC LENGTH AND VOLUME

A L V = \int_{a}^{b} ∣ f (x) - g (x) ∣ d x, A = \frac{1}{2} \int_{α}^{β} r^{2} d θ = \int_{a}^{b} 1 + (\frac{d y}{d x})^{2} d x = \int_{t_{1}}^{t_{2}} (\frac{d x}{d t})^{2} + (\frac{d y}{d t})^{2} d t = \int_{α}^{β} r^{2} + (\frac{d r}{d θ})^{2} d θ = π \int_{a}^{b} y^{2} d x, V_{y} = 2 π \int_{a}^{b} x f (x) d x

Area between two curves and area swept by a polar curve $r (θ)$ ; arc length for $y = f (x)$ , for a parametric curve, and for a polar curve. $V$ is the volume of the solid formed by revolving the region under $y = f (x)$ , $a \leq x \leq b$ , about the $x$ -axis; $V_{y}$ (shell method, for $f \geq 0$ and $0 \leq a < b$ ) is the volume when the same region is revolved about the $y$ -axis.

TOTAL DIFFERENTIAL AND CHAIN RULE

d z = \frac{\partial z}{\partial x} d x + \frac{\partial z}{\partial y} d y, \frac{d z}{d t} = \frac{\partial z}{\partial x} \frac{d x}{d t} + \frac{\partial z}{\partial y} \frac{d y}{d t}

For $z = f (x, y)$ with $x = x (t)$ , $y = y (t)$ . For $x = x (u, v)$ , $y = y (u, v)$ : $z_{u} = z_{x} x_{u} + z_{y} y_{u}$ and $z_{v} = z_{x} x_{v} + z_{y} y_{v}$ . The total differential also estimates small changes: $Δ z \approx z_{x} Δ x + z_{y} Δ y$ .

IMPLICIT DIFFERENTIATION

f (x, y) = 0 \Rightarrow \frac{d y}{d x} = - \frac{f _{x}}{f _{y}}

Valid where $f_{y} \neq = 0$ . Note the minus sign.

TAYLOR SERIES IN TWO VARIABLES

f (a + h, b + k) = f (a, b) + (h f_{x} + k f_{y}) + \frac{1}{2 !} (h^{2} f_{xx} + 2 hk f_{x y} + k^{2} f_{y y}) + \dots

All derivatives are evaluated at $(a, b)$ , and $f$ needs continuous partial derivatives of every order used. Stopping after the first bracket gives the linear approximation $f (a + h, b + k) \approx f (a, b) + h f_{x} + k f_{y}$ , the total differential again. At a stationary point the first bracket is zero and the second decides the nature of the point, which is where the $r t - s^{2}$ test comes from.

EULER'S THEOREM ON HOMOGENEOUS FUNCTIONS

x f_{x} + y f_{y} = n f, x^{2} f_{xx} + 2 x y f_{x y} + y^{2} f_{y y} = n (n - 1) f

For $f$ differentiable and homogeneous of degree $n$ , meaning $f (t x, t y) = t^{n} f (x, y)$ for all $t > 0$ (continuous second partial derivatives are needed for the second identity). $n$ may be negative or fractional. With more variables: $\sum_{i} x_{i} \partial f / \partial x_{i} = n f$ .

TWO VARIABLES: SECOND DERIVATIVE TEST

r = f_{xx}, s = f_{x y}, t = f_{y y}, D = r t - s^{2} at a point where f_{x} = f_{y} = 0

Classify with the table below; $f$ needs continuous second partial derivatives near the point. When $D > 0$ , $r$ and $t$ have the same sign, so either may be used.

LAGRANGE MULTIPLIERS

\nabla f = λ \nabla g, g = 0 ⟺ f_{x} = λ g_{x}, f_{y} = λ g_{y}, f_{z} = λ g_{z}, g (x, y, z) = 0

Extremise $f (x, y, z)$ subject to the constraint $g (x, y, z) = 0$ , at points where $\nabla g \neq = 0$ . Solve the four equations for $x, y, z, λ$ , evaluate $f$ at each candidate and compare. The method finds candidates; it does not by itself say which is a maximum.

AREA AND VOLUME

A = \iint_{R} d A, V = \iint_{R} f (x, y) d A (f \geq 0), V = ∭_{E} d V

Area of a plane region $R$ , volume under the surface $z = f (x, y)$ over $R$ , and volume of a solid $E$ . Replacing 1 by a density gives mass.

CHANGE OF VARIABLES AND THE JACOBIAN

\iint_{R} f (x, y) d x d y = \iint_{R^{'}} f (x (u, v), y (u, v)) \frac{\partial ( x , y )}{\partial ( u , v )} d u d v, \frac{\partial ( x , y )}{\partial ( u , v )} = x_{u} y_{u} x_{v} y_{v}

Valid when the map $(u, v) \mapsto (x, y)$ takes $R^{'}$ one-to-one onto $R$ (except possibly on the boundary), with continuous partial derivatives and a non-zero Jacobian inside. The absolute value of the Jacobian is the area scale factor. The Jacobians of a transformation and its inverse multiply to 1: $\frac{\partial ( x , y )}{\partial ( u , v )} \cdot \frac{\partial ( u , v )}{\partial ( x , y )} = 1$ .

STANDARD RESULTS

\int_{- \infty}^{\infty} e^{- x^{2}} d x = π, ellipse area = π ab, sphere volume = \frac{4}{3} π a^{3}, tetrahedron \frac{x}{a} + \frac{y}{b} + \frac{z}{c} \leq 1, x, y, z \geq 0 : V = \frac{ab c}{6}

The first follows by squaring the integral and switching to polar coordinates: $(\int_{- \infty}^{\infty} e^{- x^{2}} d x)^{2} = \int_{0}^{2 π} \int_{0}^{\infty} e^{- r^{2}} r d r d θ = π$ (with $a, b, c > 0$ in the others).

FOURIER SERIES ON (-L, L)

f (x) = \frac{a _{0}}{2} + n = 1 \sum \infty [a_{n} cos \frac{nπ x}{L} + b_{n} sin \frac{nπ x}{L}]

Period $2 L$ . For period $2 π$ put $L = π$ , giving $a_{n} cos n x + b_{n} sin n x$ .

FOURIER COEFFICIENTS

a_{n} = \frac{1}{L} \int_{- L}^{L} f (x) cos \frac{nπ x}{L} d x (n = 0, 1, 2, \dots), b_{n} = \frac{1}{L} \int_{- L}^{L} f (x) sin \frac{nπ x}{L} d x (n = 1, 2, \dots)

Any interval of length $2 L$ , such as $(0, 2 L)$ , may be used. They follow from orthogonality: over one period, $cos \frac{mπ x}{L} cos \frac{nπ x}{L}$ and $sin \frac{mπ x}{L} sin \frac{nπ x}{L}$ integrate to $L$ if $m = n \geq 1$ and to $0$ if $m \neq = n$ , and every sine-cosine product integrates to $0$ .

HALF-RANGE SERIES ON (0, L)

cosine: a_{n} = \frac{2}{L} \int_{0}^{L} f (x) cos \frac{nπ x}{L} d x, sine: b_{n} = \frac{2}{L} \int_{0}^{L} f (x) sin \frac{nπ x}{L} d x

A function given only on $(0, L)$ is extended evenly (cosine series) or oddly (sine series), each with period $2 L$ . Both converge to $f$ at every point of $(0, L)$ where $f$ is continuous. They differ outside $(0, L)$ and can differ at its end points: the sine series is $0$ at $x = 0$ and $x = L$ , while the cosine series converges to $f (0^{+})$ at $x = 0$ and to $f (L^{-})$ at $x = L$ .

PARSEVAL'S IDENTITY

\frac{1}{L} \int_{- L}^{L} f^{2} (x) d x = \frac{a _{0}^{2}}{2} + n = 1 \sum \infty (a_{n}^{2} + b_{n}^{2})

Written for the $\frac{a _{0}}{2}$ convention; holds whenever $\int f^{2}$ over a period is finite, in particular under the Dirichlet conditions above. Halving both sides: mean square value $= (\frac{a _{0}}{2})^{2} + \frac{1}{2} \sum_{n \geq 1} (a_{n}^{2} + b_{n}^{2})$ .

Vector calculus: gradient, divergence, curl and the integral theorems

Read the notes

DOT (SCALAR) PRODUCT

A \cdot B = ∣ A ∣ ∣ B ∣ cos θ = A_{x} B_{x} + A_{y} B_{y} + A_{z} B_{z}

A scalar; $θ \in [0, π]$ is the angle between the vectors. Commutative. For non-zero vectors, $A \cdot B = 0$ means perpendicular. $\hat{i} \cdot \hat{i} = 1$ , $\hat{i} \cdot \hat{j} = 0$ .

ANGLE, COMPONENT AND PROJECTION

cos θ = \frac{A \cdot B}{∣ A ∣ ∣ B ∣}, comp_{B} A = \frac{A \cdot B}{∣ B ∣}, proj_{B} A = \frac{A \cdot B}{∣ B ∣ ^{2}} B

The scalar component of $A$ along $B$ is signed (negative when $θ > 9 0^{\circ}$ ); the vector projection lies along $\pm B$ . The direction cosines of a non-zero $A$ are $A_{x} /∣ A ∣$ , $A_{y} /∣ A ∣$ , $A_{z} /∣ A ∣$ , and their squares add to 1.

CROSS (VECTOR) PRODUCT

A \times B = \hat{i} A_{x} B_{x} \hat{j} A_{y} B_{y} \hat{k} A_{z} B_{z}, ∣ A \times B ∣ = ∣ A ∣ ∣ B ∣ sin θ

A vector perpendicular to both $A$ and $B$ , direction by the right-hand rule. Anti-commutative: $B \times A = - A \times B$ . $∣ A \times B ∣$ is the area of the parallelogram on $A$ and $B$ . For non-zero vectors, $A \times B = 0$ means parallel. $\hat{i} \times \hat{j} = \hat{k}$ , $\hat{j} \times \hat{k} = \hat{i}$ , $\hat{k} \times \hat{i} = \hat{j}$ .

SCALAR TRIPLE PRODUCT

[A B C] = A \cdot (B \times C) = A_{x} B_{x} C_{x} A_{y} B_{y} C_{y} A_{z} B_{z} C_{z}

Unchanged by a cyclic shift, $A \cdot (B \times C) = B \cdot (C \times A) = C \cdot (A \times B)$ ; changes sign when two vectors are swapped; dot and cross may be interchanged. $∣ [A B C] ∣$ is the volume of the parallelepiped on the three vectors, and it is zero exactly when they are coplanar (linearly dependent).

VECTOR TRIPLE PRODUCT

A \times (B \times C) = (A \cdot C) B - (A \cdot B) C

The result lies in the plane of $B$ and $C$ . The cross product is not associative: $(A \times B) \times C = (A \cdot C) B - (B \cdot C) A$ , which differs in general.

GRADIENT

\nabla ϕ = \frac{\partial ϕ}{\partial x} \hat{i} + \frac{\partial ϕ}{\partial y} \hat{j} + \frac{\partial ϕ}{\partial z} \hat{k}

Defined for a differentiable scalar field. Linear, with $\nabla (ϕ ψ) = ϕ \nabla ψ + ψ \nabla ϕ$ and $d ϕ = \nabla ϕ \cdot d r$ .

DIRECTIONAL DERIVATIVE

D_{\overset{a}{^}} ϕ = \nabla ϕ \cdot \overset{a}{^}, \overset{a}{^} = \frac{a}{∣ a ∣}

Rate of change of $ϕ$ per unit distance in the direction of $a$ . The direction must be a unit vector, so divide by $∣ a ∣$ first. 'From $P$ towards the point $Q$ ' means $a = P Q$ . 'Along the tangent to the curve $r (t)$ at $t = t_{0}$ ' means $a = r^{'} (t_{0})$ ; 'along the normal to the surface $ψ = c$ ' means $a = \nabla ψ$ at the point.

MAXIMUM AND MINIMUM RATE OF CHANGE

\overset{a}{^} max D_{\overset{a}{^}} ϕ = ∣\nabla ϕ ∣ along \frac{\nabla ϕ}{∣\nabla ϕ ∣}, \overset{a}{^} min D_{\overset{a}{^}} ϕ = - ∣\nabla ϕ ∣

Because $D_{\overset{a}{^}} ϕ = ∣\nabla ϕ ∣ cos θ$ ( $θ$ = angle between $\overset{a}{^}$ and $\nabla ϕ$ ), the rate is largest along the gradient, most negative opposite to it, and zero along any direction tangent to the level surface.

UNIT NORMAL TO A SURFACE

\overset{n}{^} = \pm \frac{\nabla ϕ}{∣\nabla ϕ ∣} on the surface ϕ (x, y, z) = c

Valid where $\nabla ϕ \neq = 0$ . The sign picks a side; for a closed surface (a sphere, say) the outward normal points away from the enclosed region. For $z = g (x, y)$ take $ϕ = z - g (x, y)$ , so $\nabla ϕ = - g_{x} \hat{i} - g_{y} \hat{j} + \hat{k}$ , which points upward.

ANGLE BETWEEN TWO SURFACES

cos θ = \frac{\nabla ϕ _{1} \cdot \nabla ϕ _{2}}{∣\nabla ϕ _{1} ∣ ∣\nabla ϕ _{2} ∣}

The angle between the surfaces $ϕ_{1} = c_{1}$ and $ϕ_{2} = c_{2}$ at a common point is the angle between their normals there, both gradients evaluated at that point. Each normal is fixed only up to sign, so the angle between the surfaces is usually quoted as the acute one: take the absolute value of the numerator. They cut orthogonally when $\nabla ϕ_{1} \cdot \nabla ϕ_{2} = 0$ .

GRADIENTS OF RADIAL FUNCTIONS

\nabla r = \frac{r}{r} = \overset{r}{^}, \nabla f (r) = f^{'} (r) \overset{r}{^}, \nabla r^{n} = n r^{n - 2} r

Valid for $r \neq = 0$ . Examples: $\nabla (r^{2}) = 2 r$ and $\nabla (1/ r) = - r / r^{3}$ .

DIVERGENCE

\nabla \cdot F = \frac{\partial F _{1}}{\partial x} + \frac{\partial F _{2}}{\partial y} + \frac{\partial F _{3}}{\partial z}

A scalar field. $\nabla \cdot r = 3$ ; a constant field has zero divergence.

CURL

\nabla \times F = \hat{i} \frac{\partial}{\partial x} F_{1} \hat{j} \frac{\partial}{\partial y} F_{2} \hat{k} \frac{\partial}{\partial z} F_{3} = (\frac{\partial F _{3}}{\partial y} - \frac{\partial F _{2}}{\partial z}) \hat{i} + (\frac{\partial F _{1}}{\partial z} - \frac{\partial F _{3}}{\partial x}) \hat{j} + (\frac{\partial F _{2}}{\partial x} - \frac{\partial F _{1}}{\partial y}) \hat{k}

A vector field. Expand the determinant along the first row; the $\hat{j}$ term, written here in its cyclic form, is where sign slips happen. $\nabla \times r = 0$ .

CONSERVATIVE FIELD: EQUIVALENT CONDITIONS

F = \nabla ϕ ⟺ \oint_{C} F \cdot d r = 0 for every closed C ⟺ \int_{A}^{B} F \cdot d r depends only on A and B

Equivalent for a continuous $F$ on any connected open region. If $F$ also has continuous first partial derivatives, each implies $\nabla \times F = 0$ . The converse, curl zero implies conservative, needs $F$ to have continuous first partial derivatives on a simply connected region (every closed curve in it can be shrunk to a point without leaving it).

FUNDAMENTAL THEOREM FOR LINE INTEGRALS

\int_{A}^{B} \nabla ϕ \cdot d r = ϕ (B) - ϕ (A)

For any piecewise smooth path from $A$ to $B$ on which $ϕ$ is continuously differentiable. Sign convention: here a potential means $F = \nabla ϕ$ ; electromagnetics writes $E = - \nabla V$ , which flips the sign of the potential but not the test.

EXACTNESS TEST IN THE PLANE

P d x + Q d y = d ϕ ⟺ \frac{\partial P}{\partial y} = \frac{\partial Q}{\partial x}

On a simply connected region with $P$ , $Q$ continuously differentiable. This is the $\hat{k}$ component of the curl test for $F = P \hat{i} + Q \hat{j}$ .

THE TWO ZERO IDENTITIES

\nabla \times (\nabla ϕ) = 0, \nabla \cdot (\nabla \times F) = 0

Every gradient field is irrotational and every curl field is solenoidal. So a field with non-zero divergence cannot be the curl of anything, and a field with non-zero curl cannot be a gradient.

PRODUCT RULES

\nabla \cdot (ϕ F) \nabla \times (ϕ F) = ϕ (\nabla \cdot F) + F \cdot \nabla ϕ = ϕ (\nabla \times F) + \nabla ϕ \times F

The order in $\nabla ϕ \times F$ matters: swapping the factors flips the sign.

LAPLACIAN

\nabla^{2} ϕ = \nabla \cdot (\nabla ϕ) = \frac{\partial ^{2} ϕ}{\partial x ^{2}} + \frac{\partial ^{2} ϕ}{\partial y ^{2}} + \frac{\partial ^{2} ϕ}{\partial z ^{2}}

A scalar. $ϕ$ is harmonic when $\nabla^{2} ϕ = 0$ (Laplace's equation). If $F = \nabla ϕ$ , then $\nabla \cdot F = \nabla^{2} ϕ$ .

CURL OF A CURL

\nabla \times (\nabla \times F) = \nabla (\nabla \cdot F) - \nabla^{2} F

Here $\nabla^{2} F = \nabla^{2} F_{1} \hat{i} + \nabla^{2} F_{2} \hat{j} + \nabla^{2} F_{3} \hat{k}$ , taking the Laplacian component by component; this componentwise rule holds for Cartesian components only. For a solenoidal field it reduces to $\nabla \times (\nabla \times F) = - \nabla^{2} F$ .

RADIAL FIELDS

\nabla \cdot (r^{n} r) = (n + 3) r^{n}, \nabla \times (f (r) r) = 0, \nabla^{2} (r^{n}) = n (n + 1) r^{n - 2}

Valid for $r \neq = 0$ . Special cases: $\nabla \cdot r = 3$ ; $\nabla \cdot (r / r^{3}) = 0$ and $\nabla^{2} (1/ r) = 0$ away from the origin.

LINE INTEGRAL OF A VECTOR FIELD

\int_{C} F \cdot d r = \int_{C} (F_{1} d x + F_{2} d y + F_{3} d z) = \int_{a}^{b} F (r (t)) \cdot r^{'} (t) d t

Depends on orientation: reversing the direction of travel changes the sign. It does not depend on which parametrisation is used, as long as the direction is the same.

LINE INTEGRAL WITH RESPECT TO ARC LENGTH

\int_{C} f d s = \int_{a}^{b} f (r (t)) ∣ r^{'} (t) ∣ d t

Independent of orientation, because $d s > 0$ . With $f = 1$ it gives the length of $C$ ; with $f$ a line density it gives the mass of a wire.

STANDARD PARAMETRISATIONS

segment P_{0} \to P_{1} : circle x^{2} + y^{2} = a^{2} : curve y = g (x) : r (t) = r_{0} + t (r_{1} - r_{0}), 0 \leq t \leq 1 x = a cos t, y = a sin t, 0 \leq t \leq 2 π x = t, y = g (t)

The circle parametrisation runs counter-clockwise, which keeps the enclosed disc on the left of the direction of travel. For the ellipse $x^{2} / a^{2} + y^{2} / b^{2} = 1$ use $x = a cos t$ , $y = b sin t$ .

FLUX THROUGH A PARAMETRISED SURFACE

\iint_{S} F \cdot \overset{n}{^} d S = \iint_{D} F (r (u, v)) \cdot (r_{u} \times r_{v}) d u d v

$r_{u} \times r_{v}$ is a normal whose length is the area scale factor, so $d S = ∣ r_{u} \times r_{v} ∣ d u d v$ . If it points to the wrong side, use $r_{v} \times r_{u}$ , which flips the sign. For a scalar $f$ , $\iint_{S} f d S = \iint_{D} f (r (u, v)) ∣ r_{u} \times r_{v} ∣ d u d v$ ; with $f = 1$ this is the area of $S$ , and it does not depend on the side chosen.

SURFACE Z = G(X, Y) OVER A REGION R OF THE XY-PLANE

\iint_{S} F \cdot \overset{n}{^} d S = \iint_{R} (- F_{1} g_{x} - F_{2} g_{y} + F_{3}) d x d y, d S = 1 + g_{x}^{2} + g_{y}^{2} d x d y

The flux formula is for the upward normal ( $\overset{n}{^} \cdot \hat{k} > 0$ ); negate it for the downward normal. In general, projecting onto the $x y$ -plane uses $d S = \frac{d x d y}{∣ n ^ \cdot k ^ ∣}$ , valid where $\overset{n}{^} \cdot \hat{k} \neq = 0$ .

SPHERE AND CYLINDER SURFACE ELEMENTS

sphere r = a : cylinder ρ = a : d S = a^{2} sin θ d θ d φ, \overset{n}{^} = \frac{r}{a} d S = a d φ d z, \overset{n}{^} = \frac{x i ^ + y j ^}{a}

Outward normals. $θ$ is measured from the $+ z$ axis ( $0 \leq θ \leq π$ ) and $φ$ around it ( $0 \leq φ < 2 π$ ). Areas: $4 π a^{2}$ for the sphere, $2 π ah$ for the curved side of a cylinder of height $h$ .

VOLUME ELEMENTS

d V = d x d y d z = ρ d ρ d φ d z = r^{2} sin θ d r d θ d φ

Cartesian, cylindrical and spherical. Volumes: sphere $\frac{4}{3} π a^{3}$ , cylinder $π a^{2} h$ , cone $\frac{1}{3} π a^{2} h$ .

GREEN'S THEOREM

\oint_{C} (P d x + Q d y) = \iint_{R} (\frac{\partial Q}{\partial x} - \frac{\partial P}{\partial y}) d x d y

$C$ is a piecewise smooth simple closed curve bounding $R$ , traversed counter-clockwise; $P$ , $Q$ and their first partial derivatives are continuous on an open region containing $R$ . Clockwise traversal changes the sign. For a region with holes, the outer boundary runs counter-clockwise and each inner boundary clockwise.

AREA AS A LINE INTEGRAL

A = \oint_{C} x d y = - \oint_{C} y d x = \frac{1}{2} \oint_{C} (x d y - y d x)

Green's theorem with $\partial Q / \partial x - \partial P / \partial y = 1$ , $C$ counter-clockwise. For the ellipse $x = a cos t$ , $y = b sin t$ it gives $π ab$ .

STOKES' THEOREM

\oint_{C} F \cdot d r = \iint_{S} (\nabla \times F) \cdot \overset{n}{^} d S

$S$ is a piecewise smooth oriented surface whose boundary is the closed curve $C$ ; $F$ has continuous first partial derivatives on a region containing $S$ . Right-hand rule: with the thumb along $\overset{n}{^}$ , the fingers curl in the direction of $C$ (for $\overset{n}{^} = \hat{k}$ , $C$ runs counter-clockwise seen from above). Every surface that meets these conditions and has the same boundary and orientation gives the same value.

GAUSS'S DIVERGENCE THEOREM

\iint_{S} F \cdot \overset{n}{^} d S = ∭_{V} (\nabla \cdot F) d V

$S$ is a closed piecewise smooth surface enclosing the volume $V$ , $\overset{n}{^}$ is the outward unit normal, and $F$ has continuous first partial derivatives throughout $V$ (no singularity inside).

Differential equations

Read the notes

LINEAR ODE OF ORDER N

a_{0} (x) y^{(n)} + a_{1} (x) y^{(n - 1)} + \dots + a_{n - 1} (x) y^{'} + a_{n} (x) y = r (x)

Linear means $y$ and every derivative appear only to the first power, are never multiplied together, and never sit inside another function. The coefficients may depend on $x$ only, and the equation has order $n$ wherever $a_{0} (x) \neq = 0$ . The equation is homogeneous when $r (x) = 0$ and non-homogeneous otherwise.

VARIABLES SEPARABLE

\frac{d y}{d x} = f (x) g (y) \Rightarrow \int \frac{d y}{g ( y )} = \int f (x) d x + C

Valid where $g (y) \neq = 0$ . Each root $y = y_{0}$ of $g (y) = 0$ is a separate constant solution, and dividing by $g (y)$ can lose it (sometimes it reappears for a particular value of the constant). For $\frac{d y}{d x} = f (a x + b y + c)$ , substitute $z = a x + b y + c$ to get the separable equation $\frac{d z}{d x} = a + b f (z)$ .

HOMOGENEOUS FIRST-ORDER EQUATION

\frac{d y}{d x} = F (\frac{y}{x}), y = v x \Rightarrow \int \frac{d v}{F ( v ) - v} = ln ∣ x ∣ + C

Applies when $f (x, y)$ is unchanged by $x \to k x, y \to k y$ , that is, when $M$ and $N$ are homogeneous functions of the same degree. It uses $\frac{d y}{d x} = v + x \frac{d v}{d x}$ . Valid where $F (v) \neq = v$ ; each root $v_{0}$ of $F (v) = v$ gives the straight-line solution $y = v_{0} x$ .

EQUATIONS REDUCIBLE TO HOMOGENEOUS FORM

\frac{d y}{d x} = \frac{a _{1} x + b _{1} y + c _{1}}{a _{2} x + b _{2} y + c _{2}}

If $a_{1} b_{2} - a_{2} b_{1} \neq = 0$ , put $x = X + h$ , $y = Y + k$ , where $(h, k)$ solves $a_{1} h + b_{1} k + c_{1} = 0$ and $a_{2} h + b_{2} k + c_{2} = 0$ . The equation becomes $\frac{d Y}{d X} = \frac{a _{1} X + b _{1} Y}{a _{2} X + b _{2} Y}$ , which is homogeneous. If $a_{1} b_{2} - a_{2} b_{1} = 0$ , the two linear expressions are proportional apart from their constants, so the right-hand side is a function of one combination $a x + b y$ . Substitute $z = a x + b y$ as in the separable case above.

EXACT EQUATION

M d x + N d y = 0 is exact ⟺ \frac{\partial M}{\partial y} = \frac{\partial N}{\partial x}; \int_{y const} M d x + \int (terms of N free of x) d y = C

The test holds on a simply connected region where $M$ , $N$ and their first partial derivatives are continuous. The solution is $ϕ (x, y) = C$ , where $ϕ_{x} = M$ and $ϕ_{y} = N$ . The shortcut on the right needs $\int M d x$ written without hidden $x$ -free parts (write $ln x$ , not $ln (x y)$ ). The safe route is $ϕ = \int M d x + h (y)$ with $h^{'} (y) = N - \frac{\partial}{\partial y} \int M d x$ .

INTEGRATING FACTORS FOR A NON-EXACT EQUATION

\frac{M _{y} - N _{x}}{N} = f (x) \Rightarrow μ = e^{\int f (x) d x}; \frac{N _{x} - M _{y}}{M} = g (y) \Rightarrow μ = e^{\int g (y) d y}

Multiply the whole equation by $μ$ to make it exact. Two more rules: if $M$ and $N$ are homogeneous of the same degree and $M x + N y \neq = 0$ , then $μ = \frac{1}{M x + N y}$ ; if $M = y f_{1} (x y)$ and $N = x f_{2} (x y)$ with $M x - N y \neq = 0$ , then $μ = \frac{1}{M x - N y}$ .

LINEAR FIRST-ORDER EQUATION

\frac{d y}{d x} + P (x) y = Q (x) : IF = e^{\int P d x}, y \cdot IF = \int Q \cdot IF d x + C

First divide by the coefficient of $\frac{d y}{d x}$ so that it becomes 1. $P$ keeps its sign. If the equation is linear in $x$ instead, $\frac{d x}{d y} + P (y) x = Q (y)$ , use IF $= e^{\int P d y}$ and $x \cdot IF = \int Q \cdot IF d y + C$ .

BERNOULLI EQUATION

\frac{d y}{d x} + P (x) y = Q (x) y^{n} (n \neq = 0, 1) : z = y^{1 - n} \Rightarrow \frac{d z}{d x} + (1 - n) P z = (1 - n) Q

Divide by $y^{n}$ first. The substitution turns the equation into a linear one in $z$ . When $n > 0$ , $y = 0$ is also a solution, which the division by $y^{n}$ loses.

CLAIRAUT'S EQUATION

y = x p + f (p), p = \frac{d y}{d x} : y = c x + f (c)

Differentiating gives $\frac{d p}{d x} [x + f^{'} (p)] = 0$ . The factor $\frac{d p}{d x} = 0$ gives the general solution, the family of straight lines $y = c x + f (c)$ (replace $p$ by $c$ ). The factor $x + f^{'} (p) = 0$ , with $p$ eliminated between it and the equation, gives the singular solution, which is the envelope of those lines. Example: $y = x p + \frac{1}{p}$ has general solution $y = c x + \frac{1}{c}$ ; $x - \frac{1}{p ^{2}} = 0$ gives the singular solution $y^{2} = 4 x$ .

ORTHOGONAL TRAJECTORIES

Cartesian: \frac{d y}{d x} \to - \frac{d x}{d y}; polar: \frac{d r}{d θ} \to - r^{2} \frac{d θ}{d r}

Form the family's ODE with the parameter eliminated, make the replacement, then solve. Example: $y = c x^{2}$ gives $\frac{d y}{d x} = \frac{2 y}{x}$ . The trajectories satisfy $- \frac{d x}{d y} = \frac{2 y}{x}$ , which gives the ellipses $x^{2} + 2 y^{2} = k$ .

STRUCTURE OF THE SOLUTION

f (D) y = r (x), f (D) = a_{0} D^{n} + a_{1} D^{n - 1} + \dots + a_{n}; y = y_{c} + y_{p}

$y_{c}$ comes from the roots of $f (m) = 0$ and is the general solution of $f (D) y = 0$ . $y_{p}$ is any one solution of the full equation.

WRONSKIAN

W (y_{1}, y_{2}) = y_{1} y_{2}^{'} - y_{2} y_{1}^{'} = det [y_{1} y_{1}^{'} y_{2} y_{2}^{'}]

For $n$ functions, use the $n \times n$ determinant whose rows are the functions and their derivatives up to order $n - 1$ . If $W \neq = 0$ at some point of an interval, the functions are linearly independent on that interval.

ABEL'S FORMULA

y^{''} + p (x) y^{'} + q (x) y = 0 \Rightarrow W (x) = K e^{- \int p (x) d x}

Divide by the coefficient of $y^{''}$ first so that it is 1. Holds for two solutions of a linear homogeneous ODE with $p, q$ continuous on an interval. There $W$ is either identically zero (the solutions are dependent) or never zero (they are independent and form a fundamental set).

REDUCTION OF ORDER

y^{''} + p (x) y^{'} + q (x) y = 0, y_{1} known : y_{2} = y_{1} \int \frac{e ^{- \int p d x}}{y _{1}^{2}} d x

Gives a second, independent solution on an interval where $y_{1} \neq = 0$ . It follows from Abel's formula, because $(\frac{y _{2}}{y _{1}})^{'} = \frac{W}{y _{1}^{2}}$ . Example: $x^{2} y^{''} - 3 x y^{'} + 4 y = 0$ ( $x > 0$ ) has $y_{1} = x^{2}$ . Dividing by $x^{2}$ gives $p = - \frac{3}{x}$ , so $e^{- \int p d x} = x^{3}$ and $y_{2} = x^{2} \int \frac{x ^{3}}{x ^{4}} d x = x^{2} ln x$ .

PI FOR AN EXPONENTIAL

\frac{1}{f ( D )} e^{a x} = \frac{e ^{a x}}{f ( a )} (f (a) \neq = 0)

Replace $D$ by $a$ . This also covers a constant ( $a = 0$ ) and $cosh a x$ or $sinh a x$ written as exponentials.

PI FOR AN EXPONENTIAL WHEN A IS A ROOT

\frac{1}{f ( D )} e^{a x} = \frac{x ^{k} e ^{a x}}{f ^{(k)} ( a )} if a is a root of f (m) = 0 of multiplicity k

Differentiate $f$ with respect to $D$ $k$ times. For a simple root this gives $\frac{x e ^{a x}}{f ^{'} ( a )}$ , and for a double root $\frac{x ^{2} e ^{a x}}{f ^{''} ( a )}$ .

PI FOR SIN AX AND COS AX

\frac{1}{ϕ ( D ^{2} )} sin a x = \frac{sin a x}{ϕ ( - a ^{2} )}, \frac{1}{ϕ ( D ^{2} )} cos a x = \frac{cos a x}{ϕ ( - a ^{2} )} (ϕ (- a^{2}) \neq = 0)

Replace $D^{2}$ by $- a^{2}$ . Never replace $D$ itself. If odd powers of $D$ remain, as in $\frac{1}{p D + q}$ , multiply numerator and denominator by $p D - q$ so that $D^{2}$ appears. Replace it by $- a^{2}$ again, then apply the $D$ now in the numerator as a derivative.

PI FOR SIN AX AND COS AX AT RESONANCE (D^2 + A^2 = 0)

\frac{1}{D ^{2} + a ^{2}} sin a x = - \frac{x}{2 a} cos a x, \frac{1}{D ^{2} + a ^{2}} cos a x = \frac{x}{2 a} sin a x

Used when $\pm j a$ are roots of the auxiliary equation, so that $ϕ (- a^{2}) = 0$ . Note the minus sign in the sine case.

PI FOR A POLYNOMIAL

\frac{1}{f ( D )} x^{m} = [f (D)]^{- 1} x^{m}, \frac{1}{1 + t} = 1 - t + t^{2} - \dots, \frac{1}{1 - t} = 1 + t + t^{2} + \dots

Take out any factor $D^{k}$ first. Then write the rest as $c [1 \pm ϕ (D)]$ , expand in ascending powers of $D$ , and stop at $D^{m}$ , because higher derivatives of $x^{m}$ vanish. Example: $\frac{1}{1 + D} x^{2} = (1 - D + D^{2}) x^{2} = x^{2} - 2 x + 2$ .

EXPONENTIAL SHIFT

\frac{1}{f ( D )} [e^{a x} V (x)] = e^{a x} \frac{1}{f ( D + a )} V (x)

Move $e^{a x}$ to the left and replace $D$ by $D + a$ . Then handle $V$ (a polynomial, sine or cosine) with the rules above. This covers $x^{n} e^{a x}$ and $e^{a x} sin b x$ .

VARIATION OF PARAMETERS

y_{p} = - y_{1} \int \frac{y _{2} r ( x )}{W} d x + y_{2} \int \frac{y _{1} r ( x )}{W} d x, W = y_{1} y_{2}^{'} - y_{2} y_{1}^{'}

For $y^{''} + p (x) y^{'} + q (x) y = r (x)$ with the coefficient of $y^{''}$ equal to 1, so divide through first. $y_{1}$ and $y_{2}$ are independent solutions of the homogeneous equation. It works for any $r (x)$ continuous on an interval where $p$ and $q$ are continuous.

CAUCHY-EULER EQUATION AND SUBSTITUTION

x^{2} y^{''} + a x y^{'} + b y = r (x), x > 0 : x = e^{z}, z = ln x, θ \equiv \frac{d}{d z}

$a$ and $b$ are constants. For $x < 0$ use $x = - e^{z}$ . This amounts to replacing $ln x$ by $ln ∣ x ∣$ and $x^{m}$ by $∣ x ∣^{m}$ .

OPERATOR REPLACEMENTS

x D = θ, x^{2} D^{2} = θ (θ - 1), x^{3} D^{3} = θ (θ - 1) (θ - 2)

Here $D = \frac{d}{d x}$ and $θ = \frac{d}{d z}$ . Rewrite the right-hand side in $z$ as well: $x^{k} = e^{k z}$ and $ln x = z$ .

AUXILIARY EQUATION FROM Y = X^M

m (m - 1) + a m + b = 0

Obtained by trying $y = x^{m}$ directly (equivalently, $θ \to m$ ). The $x^{2} y^{''}$ term contributes $m (m - 1)$ , not $m^{2}$ .

EIGENVALUE BVP (FIXED ENDS)

y^{''} + λ y = 0, y (0) = y (L) = 0 : λ_{n} = (\frac{nπ}{L})^{2}, y_{n} = sin \frac{nπ x}{L}, n = 1, 2, 3, \dots

For $λ \leq 0$ , only $y = 0$ satisfies both conditions. With $y^{'} (0) = y^{'} (L) = 0$ instead, $λ_{n} = (nπ / L)^{2}$ and $y_{n} = cos \frac{nπ x}{L}$ for $n = 0, 1, 2, \dots$ ( $n = 0$ , that is $λ = 0$ , gives a constant). These are the spatial modes in the PDE solutions below.

LAPLACE TRANSFORM METHOD FOR AN IVP

L {y^{'}} = s Y (s) - y (0), L {y^{''}} = s^{2} Y (s) - s y (0) - y^{'} (0)

One-sided (unilateral) transform, $Y (s) = \int_{0}^{\infty} y (t) e^{- s t} d t$ , with $t \geq 0$ the independent variable. In general, $L {y^{(n)}} = s^{n} Y (s) - s^{n - 1} y (0) - \dots - y^{(n - 1)} (0)$ . Transform the constant-coefficient ODE, solve for $Y (s)$ with the initial conditions already included, then invert by partial fractions. Example: $y^{'} + 3 y = 0$ , $y (0) = 2$ gives $s Y - 2 + 3 Y = 0$ , so $Y = \frac{2}{s + 3}$ and $y = 2 e^{- 3 t}$ .

CLASSIFICATION OF SECOND-ORDER LINEAR PDES

B^{2} - 4 A C < 0 : elliptic, B^{2} - 4 A C = 0 : parabolic, B^{2} - 4 A C > 0 : hyperbolic

For $A u_{xx} + B u_{x y} + C u_{y y} + \dots = 0$ . Lower-order terms and the right-hand side do not affect the type.

HEAT EQUATION, ENDS HELD AT ZERO

u (x, t) = n = 1 \sum \infty b_{n} sin \frac{nπ x}{L} e^{- c^{2} n^{2} π^{2} t / L^{2}}, b_{n} = \frac{2}{L} \int_{0}^{L} f (x) sin \frac{nπ x}{L} d x

For $u_{t} = c^{2} u_{xx}$ , $0 < x < L$ , with $u (0, t) = u (L, t) = 0$ and $u (x, 0) = f (x)$ . Every mode decays, and higher modes decay faster.

HEAT EQUATION, INSULATED ENDS

u (x, t) = a_{0} + n = 1 \sum \infty a_{n} cos \frac{nπ x}{L} e^{- c^{2} n^{2} π^{2} t / L^{2}}, a_{0} = \frac{1}{L} \int_{0}^{L} f (x) d x, a_{n} = \frac{2}{L} \int_{0}^{L} f (x) cos \frac{nπ x}{L} d x

For $u_{x} (0, t) = u_{x} (L, t) = 0$ . As $t \to \infty$ , $u \to a_{0}$ , the average initial temperature.

WAVE EQUATION, STRING FIXED AT BOTH ENDS

u (x, t) = n = 1 \sum \infty (B_{n} cos \frac{c nπ t}{L} + B_{n}^{*} sin \frac{c nπ t}{L}) sin \frac{nπ x}{L}

For $u_{tt} = c^{2} u_{xx}$ with $u (0, t) = u (L, t) = 0$ , $u (x, 0) = f (x)$ and $u_{t} (x, 0) = g (x)$ . Mode $n$ oscillates at $ω_{n} = \frac{c nπ}{L}$ rad/s, that is, $f_{n} = \frac{n c}{2 L}$ Hz.

WAVE EQUATION COEFFICIENTS

B_{n} = \frac{2}{L} \int_{0}^{L} f (x) sin \frac{nπ x}{L} d x, B_{n}^{*} = \frac{2}{c nπ} \int_{0}^{L} g (x) sin \frac{nπ x}{L} d x

A string released from rest ( $g = 0$ ) has every $B_{n}^{*} = 0$ .

D'ALEMBERT'S SOLUTION

u (x, t) = \frac{1}{2} [f (x + c t) + f (x - c t)] + \frac{1}{2 c} \int_{x - c t}^{x + c t} g (s) d s

For the wave equation on an infinite string with $u (x, 0) = f (x)$ and $u_{t} (x, 0) = g (x)$ . It shows two waves travelling at speed $c$ in opposite directions.

LAPLACE EQUATION IN A RECTANGLE

u (x, y) = n = 1 \sum \infty A_{n} sin \frac{nπ x}{a} sinh \frac{nπ y}{a}, A_{n} = \frac{2}{a sinh ( nπ b / a )} \int_{0}^{a} f (x) sin \frac{nπ x}{a} d x

For $u_{xx} + u_{y y} = 0$ on $0 \leq x \leq a$ , $0 \leq y \leq b$ , with $u = 0$ on $x = 0$ , $x = a$ and $y = 0$ , and $u (x, b) = f (x)$ . The sine in $x$ comes from the two zero side conditions, and $sinh$ is the combination in $y$ that vanishes at $y = 0$ .

Complex analysis

Read the notes

CARTESIAN, POLAR AND EXPONENTIAL FORMS

z = x + j y = r (cos θ + j sin θ) = r e^{j θ}, r = ∣ z ∣ = x^{2} + y^{2}, e^{j θ} = cos θ + j sin θ

Conjugate $\overset{z}{ˉ} = x - j y = r e^{- j θ}$ , with $z \overset{z}{ˉ} = ∣ z ∣^{2}$ , $Re z = \frac{z + z ˉ}{2}$ and $Im z = \frac{z - z ˉ}{2 j}$ . The argument is fixed only up to adding $2 k π$ .

PRINCIPAL ARGUMENT

Arg z = ⎩ ⎨ ⎧ tan^{- 1} (y / x), tan^{- 1} (y / x) + π, tan^{- 1} (y / x) - π, π /2, - π /2, x > 0 x < 0, y \geq 0 x < 0, y < 0 x = 0, y > 0 x = 0, y < 0

Here $tan^{- 1}$ returns a value in $(- π /2, π /2)$ , and $Arg 0$ is undefined. Some texts take the principal range as $[0, 2 π)$ ; these notes use $(- π, π]$ throughout, so $Arg (- 1) = π$ .

PRODUCTS, QUOTIENTS AND THE TRIANGLE INEQUALITY

∣ z_{1} z_{2} ∣ = ∣ z_{1} ∣ ∣ z_{2} ∣, \frac{z _{1}}{z _{2}} = \frac{∣ z _{1} ∣}{∣ z _{2} ∣}, ar g (z_{1} z_{2}) = ar g z_{1} + ar g z_{2}, ar g \frac{z _{1}}{z _{2}} = ar g z_{1} - ar g z_{2}

The argument rules hold only up to multiples of $2 π$ : $Arg (z_{1} z_{2})$ can differ from $Arg z_{1} + Arg z_{2}$ by $\pm 2 π$ . Triangle inequality: $∣ z_{1} ∣ - ∣ z_{2} ∣ \leq ∣ z_{1} + z_{2} ∣ \leq ∣ z_{1} ∣ + ∣ z_{2} ∣$ .

DE MOIVRE'S THEOREM

(cos θ + j sin θ)^{n} = cos n θ + j sin n θ, (r e^{j θ})^{n} = r^{n} e^{j n θ}

For every integer $n$ . For a fractional power it gives only one of the values; use the roots formula below.

N-TH ROOTS

z^{1/ n} = r^{1/ n} e^{j (θ + 2 k π) / n}, k = 0, 1, \dots, n - 1

Every $z \neq = 0$ has exactly $n$ distinct $n$ -th roots. They lie on the circle of radius $r^{1/ n}$ (the positive real root), at angles spaced $2 π / n$ apart; for $n \geq 3$ they are the vertices of a regular $n$ -gon.

ROOTS OF UNITY

ω_{k} = e^{j 2 π k / n}, k = 0, 1, \dots, n - 1; k = 0 \sum n - 1 ω_{k} = 0 (n \geq 2), k = 0 \prod n - 1 ω_{k} = (- 1)^{n - 1}

With $ω = e^{j 2 π / n}$ the roots are $1, ω, ω^{2}, \dots, ω^{n - 1}$ and $ω^{n} = 1$ . Cube roots: $1 + ω + ω^{2} = 0$ with $ω = - \frac{1}{2} + j \frac{3}{2}$ .

EXPONENTIAL, TRIGONOMETRIC AND HYPERBOLIC FUNCTIONS

e^{z} = e^{x} (cos y + j sin y), cos z = \frac{e ^{j z} + e ^{- j z}}{2}, sin z = \frac{e ^{j z} - e ^{- j z}}{2 j}, cos (j y) = cosh y, sin (j y) = j sinh y

$∣ e^{z} ∣ = e^{x}$ , and $y$ is one value of $ar g e^{z}$ ; $e^{z} \neq = 0$ for every $z$ , and $e^{z}$ has period $2 π j$ . $sin z$ and $cos z$ are entire but unbounded, so $∣ sin z ∣$ and $∣ cos z ∣$ can exceed 1, e.g. $cos (j) = cosh 1 \approx 1.543$ . The zeros of $sin z$ are $z = nπ$ and those of $cos z$ are $z = (n + \frac{1}{2}) π$ , with $n$ an integer; all of them are real.

COMPLEX LOGARITHM

ln z = ln ∣ z ∣ + j (Arg z + 2 k π), k \in Z; Log z = ln ∣ z ∣ + j Arg z

Defined for $z \neq = 0$ and multi-valued; $Log$ is the principal value ( $k = 0$ ). The principal branch is analytic everywhere except on its branch cut, the non-positive real axis (where $Arg$ jumps by $2 π$ ), and off the cut $(Log z)^{'} = 1/ z$ . Example: $Log (- 1) = j π$ . The rule $Log (z_{1} z_{2}) = Log z_{1} + Log z_{2}$ can fail by $\pm 2 π j$ .

COMPLEX POWERS

z^{c} = e^{c l n z}, principal value e^{c Log z}; j^{j} = e^{j \cdot j π /2} = e^{- π /2} \approx 0.208

Defined for $z \neq = 0$ . Multi-valued when $c$ is not an integer: for rational $c = p / q$ in lowest terms with $q \geq 2$ there are $q$ values; for irrational or non-real $c$ there are infinitely many. All values of $j^{j}$ are $e^{- π /2 - 2 k π}$ with $k \in Z$ , every one of them real.

CAUCHY-RIEMANN EQUATIONS (CARTESIAN)

\frac{\partial u}{\partial x} = \frac{\partial v}{\partial y}, \frac{\partial u}{\partial y} = - \frac{\partial v}{\partial x}; f^{'} (z) = u_{x} + j v_{x} = v_{y} - j u_{y}

Necessary at every point where $f$ is differentiable. Sufficient for differentiability at $z_{0}$ when, in addition, $u_{x}, u_{y}, v_{x}, v_{y}$ exist in a neighbourhood of $z_{0}$ and are continuous at $z_{0}$ . $f$ is analytic in a domain where these conditions hold at every point. Subscripts denote partial derivatives.

CAUCHY-RIEMANN EQUATIONS (POLAR)

\frac{\partial u}{\partial r} = \frac{1}{r} \frac{\partial v}{\partial θ}, \frac{\partial v}{\partial r} = - \frac{1}{r} \frac{\partial u}{\partial θ}; f^{'} (z) = e^{- j θ} (u_{r} + j v_{r})

For $z = r e^{j θ}$ with $r \neq = 0$ and $f = u (r, θ) + j v (r, θ)$ . Use it when $f$ is naturally written in $r$ and $θ$ : for $Log z = ln r + j θ$ with $- π < θ < π$ (off the branch cut), $u_{r} = 1/ r = v_{θ} / r$ and $v_{r} = 0 = - u_{θ} / r$ , and $f^{'} = e^{- j θ} / r = 1/ z$ .

HARMONIC FUNCTIONS

\nabla^{2} u = \frac{\partial ^{2} u}{\partial x ^{2}} + \frac{\partial ^{2} u}{\partial y ^{2}} = 0, \nabla^{2} v = 0

If $f = u + j v$ is analytic in a domain, both $u$ and $v$ are harmonic there (the derivative of an analytic function is again analytic, so the second partials exist and are continuous). $v$ is called a harmonic conjugate of $u$ . Where $f^{'} (z) \neq = 0$ , the level curves $u = c_{1}$ and $v = c_{2}$ cut at right angles. In polar form Laplace's equation is $u_{r r} + \frac{1}{r} u_{r} + \frac{1}{r ^{2}} u_{θ θ} = 0$ .

MILNE-THOMSON CONSTRUCTION

f^{'} (z) = u_{x} (z, 0) - j u_{y} (z, 0) \Rightarrow f (z) = \int [u_{x} (z, 0) - j u_{y} (z, 0)] d z + C

Builds $f$ directly from $u$ without finding $v$ : compute $u_{x}$ and $u_{y}$ , put $x = z$ and $y = 0$ , and integrate in $z$ . If $v$ is given instead, $f^{'} (z) = v_{y} (z, 0) + j v_{x} (z, 0)$ . $C$ is a complex constant. When $u$ is given, $Re C$ is fixed by $u$ itself and $Im C$ stays arbitrary unless a value such as $f (0)$ is also given (and the other way round when $v$ is given).

BILINEAR (MOBIUS) TRANSFORMATION

w = \frac{a z + b}{cz + d}, a d - b c \neq = 0

Conformal wherever $cz + d \neq = 0$ , since $w^{'} (z) = \frac{a d - b c}{( cz + d ) ^{2}}$ . For $c \neq = 0$ , $z = - d / c$ maps to $w = \infty$ and $z = \infty$ maps to $w = a / c$ . It maps circles and straight lines (a line counts as a circle through $\infty$ ) onto circles and straight lines, and it is fixed completely by the images of three distinct points. Its inverse $z = \frac{d w - b}{- c w + a}$ is also bilinear.

FIXED POINTS AND CROSS-RATIO

fixed points: c z^{2} + (d - a) z - b = 0; \frac{( w - w _{1} ) ( w _{2} - w _{3} )}{( w - w _{3} ) ( w _{2} - w _{1} )} = \frac{( z - z _{1} ) ( z _{2} - z _{3} )}{( z - z _{3} ) ( z _{2} - z _{1} )}

Fixed points solve $z = \frac{a z + b}{cz + d}$ ; there are at most two unless the map is the identity (for $c = 0$ and $a \neq = d$ there is one finite fixed point, and $\infty$ is also fixed). The cross-ratio equation gives the unique map sending $z_{1}, z_{2}, z_{3}$ to $w_{1}, w_{2}, w_{3}$ . If one of the given points is $\infty$ , delete the two factors that contain it.

CONTOUR INTEGRAL BY PARAMETRISATION

\int_{C} f (z) d z = \int_{a}^{b} f (z (t)) z^{'} (t) d t = \int_{C} (u d x - v d y) + j \int_{C} (v d x + u d y)

$z = z (t)$ , $a \leq t \leq b$ , traces $C$ in the given direction. Circle $∣ z - z_{0} ∣ = ρ$ counter-clockwise: $z = z_{0} + ρ e^{j θ}$ , $d z = j ρ e^{j θ} d θ$ , $0 \leq θ \leq 2 π$ . Segment from $z_{1}$ to $z_{2}$ : $z = z_{1} + t (z_{2} - z_{1})$ , $0 \leq t \leq 1$ .

THE BASIC CIRCLE INTEGRAL

\oint_{∣ z - z_{0} ∣ = ρ} (z - z_{0})^{n} d z = {2 π j, 0, n = - 1 n \neq = - 1 (n an integer)

Counter-clockwise, any radius $ρ > 0$ . Every term of a Laurent series integrates to zero except the $1/ (z - z_{0})$ term, which is why residues work.

ML-INEQUALITY

\int_{C} f (z) d z \leq M L, M = z \in C max ∣ f (z) ∣, L = length of C

Used to bound an integral, and to show that the integral over a large semicircular arc tends to zero when real integrals are evaluated by residues.

CAUCHY'S INTEGRAL THEOREM

\oint_{C} f (z) d z = 0

Conditions: $C$ is a simple closed contour and $f$ is analytic at every point on $C$ and inside it (more generally, in a simply connected domain containing $C$ ). Analyticity on $C$ alone is not enough: $1/ z$ is analytic on $∣ z ∣ = 1$ , yet $\oint_{∣ z ∣ = 1} d z / z = 2 π j$ .

CAUCHY'S INTEGRAL FORMULA

f (z_{0}) = \frac{1}{2 π j} \oint_{C} \frac{f ( z )}{z - z _{0}} d z ⟺ \oint_{C} \frac{f ( z )}{z - z _{0}} d z = 2 π j f (z_{0})

$f$ analytic on and inside the simple closed contour $C$ (counter-clockwise), and $z_{0}$ strictly inside $C$ . If $z_{0}$ is outside $C$ the integral is $0$ by Cauchy's theorem; if $z_{0}$ lies on $C$ the formula does not apply (in general the integral then does not exist as an ordinary integral).

CAUCHY'S INTEGRAL FORMULA FOR DERIVATIVES

f^{(n)} (z_{0}) = \frac{n !}{2 π j} \oint_{C} \frac{f ( z )}{( z - z _{0} ) ^{n + 1}} d z ⟺ \oint_{C} \frac{f ( z )}{( z - z _{0} ) ^{n + 1}} d z = \frac{2 π j}{n !} f^{(n)} (z_{0})

Same conditions, $n = 0, 1, 2, \dots$ It shows that an analytic function has derivatives of every order, all analytic. Mind the power $n + 1$ : for $(z - z_{0})^{3}$ in the denominator use $f^{''} (z_{0})$ and divide by $2!$ .

N-TH TERM TEST

\sum a_{n} converges \Rightarrow n \to \infty lim a_{n} = 0

Only a test for divergence: if $lim a_{n} \neq = 0$ , or the limit does not exist, the series diverges. The converse is false: $\sum 1/ n$ diverges although $1/ n \to 0$ .

GEOMETRIC AND P-SERIES BENCHMARKS

n = 0 \sum \infty r^{n} = \frac{1}{1 - r} (∣ r ∣ < 1), n = 1 \sum \infty \frac{1}{n ^{p}} converges ⟺ p > 1

The geometric series (real or complex $r$ ) diverges for $∣ r ∣ \geq 1$ . $\sum 1/ n$ ( $p = 1$ ) diverges, while $\sum 1/ n^{2} = π^{2} /6$ . These two families are the usual benchmarks for the comparison tests.

RADIUS OF CONVERGENCE OF A POWER SERIES

n = 0 \sum \infty a_{n} (z - z_{0})^{n} : \frac{1}{R} = n \to \infty lim \frac{a _{n + 1}}{a _{n}} or \frac{1}{R} = n \to \infty lim ∣ a_{n} ∣^{1/ n}

Use whichever limit exists (the ratio form needs $a_{n} \neq = 0$ for all large $n$ ); $R = \infty$ if the limit is $0$ and $R = 0$ if it is $\infty$ ; in general $1/ R = lim sup ∣ a_{n} ∣^{1/ n}$ . The series converges absolutely for $∣ z - z_{0} ∣ < R$ and diverges for $∣ z - z_{0} ∣ > R$ ; points on the circle $∣ z - z_{0} ∣ = R$ must be tested separately. Inside the disc it may be differentiated or integrated term by term without changing $R$ .

TAYLOR SERIES

f (z) = n = 0 \sum \infty \frac{f ^{(n)} ( z _{0} )}{n !} (z - z_{0})^{n}, ∣ z - z_{0} ∣ < R

The series equals $f$ throughout the largest open disc about $z_{0}$ in which $f$ is analytic. When the singularity on the edge of that disc is a pole or an essential singularity, that disc is exactly the disc of convergence, so $R$ is the distance from $z_{0}$ to that singularity (a removable singularity does not limit $R$ ). $z_{0} = 0$ gives the Maclaurin series. The expansion is unique, so any method that produces a power series (substitution, multiplication, term-by-term differentiation or integration) gives the Taylor series.

STANDARD EXPANSIONS ABOUT 0

e^{z} \frac{1}{1 - z} = n = 0 \sum \infty \frac{z ^{n}}{n !}, = n = 0 \sum \infty z^{n}, sin z \frac{1}{1 + z} = n = 0 \sum \infty \frac{( - 1 ) ^{n} z ^{2 n + 1}}{( 2 n + 1 )!}, = n = 0 \sum \infty (- 1)^{n} z^{n}, cos z Log (1 + z) = n = 0 \sum \infty \frac{( - 1 ) ^{n} z ^{2 n}}{( 2 n )!} (all z) = n = 1 \sum \infty \frac{( - 1 ) ^{n + 1} z ^{n}}{n} (∣ z ∣ < 1)

$sinh z$ and $cosh z$ have the same series as $sin z$ and $cos z$ without the alternating signs. Binomial series: $(1 + z)^{α} = 1 + α z + \frac{α ( α - 1 )}{2 !} z^{2} + \dots$ for $∣ z ∣ < 1$ (principal value).

LAURENT SERIES

f (z) = n = - \infty \sum \infty a_{n} (z - z_{0})^{n} = n = 0 \sum \infty a_{n} (z - z_{0})^{n} + n = 1 \sum \infty \frac{a _{- n}}{( z - z _{0} ) ^{n}}, a_{n} = \frac{1}{2 π j} \oint_{C} \frac{f ( z )}{( z - z _{0} ) ^{n + 1}} d z

Valid in an annulus $r_{1} < ∣ z - z_{0} ∣ < r_{2}$ in which $f$ is analytic ( $r_{1}$ may be $0$ and $r_{2}$ may be $\infty$ ); $C$ is any closed counter-clockwise contour around $z_{0}$ inside the annulus. The negative-power part is the principal part. In a given annulus the expansion is unique, but each annulus bounded by the singularities has its own series. In practice the coefficients come from known series, not from the integral.

ORDER OF A POLE FROM ORDERS OF ZEROS

f = \frac{p}{q}, p has a zero of order k and q a zero of order m at z_{0} \Rightarrow {pole of order m - k, removable singularity, m > k m \leq k

Here $p$ and $q$ are analytic at $z_{0}$ ; take $k = 0$ when $p (z_{0}) \neq = 0$ . Find orders from series: $1 - cos z = \frac{z ^{2}}{2} - \dots$ has a zero of order 2 at $0$ , so $\frac{1 - cos z}{z ^{5}}$ has a pole of order 3 there, not 5.

RESIDUE

Res_{z = z_{0}} f (z) = a_{- 1} = \frac{1}{2 π j} \oint_{C} f (z) d z

$a_{- 1}$ is the coefficient of $(z - z_{0})^{- 1}$ in the Laurent series valid in $0 < ∣ z - z_{0} ∣ < r$ ; $C$ is a small counter-clockwise circle around $z_{0}$ enclosing no other singularity. At a removable singularity the residue is $0$ . At an essential singularity it must be read from the Laurent series: $Res_{z = 0} e^{1/ z} = 1$ and $Res_{z = 0} z^{2} e^{1/ z} = \frac{1}{3 !} = \frac{1}{6}$ .

RESIDUE AT A SIMPLE POLE

Res_{z = z_{0}} f = z \to z_{0} lim (z - z_{0}) f (z); f = \frac{p ( z )}{q ( z )}, p (z_{0}) \neq = 0, q (z_{0}) = 0, q^{'} (z_{0}) \neq = 0 \Rightarrow Res_{z = z_{0}} f = \frac{p ( z _{0} )}{q ^{'} ( z _{0} )}

Here $p$ and $q$ are analytic at $z_{0}$ . The $p / q^{'}$ form avoids factorising: $Res_{z = nπ} \frac{1}{sin z} = \frac{1}{cos nπ} = (- 1)^{n}$ , $Res_{z = 0} cot z = \frac{cos 0}{cos 0} = 1$ , and for $\frac{1}{z ^{4} + 1}$ at a root $z_{k}$ the residue is $\frac{1}{4 z _{k}^{3}} = - \frac{z _{k}}{4}$ , since $z_{k}^{4} = - 1$ .

RESIDUE AT A POLE OF ORDER M

Res_{z = z_{0}} f = \frac{1}{( m - 1 )!} z \to z_{0} lim \frac{d ^{m - 1}}{d z ^{m - 1}} [(z - z_{0})^{m} f (z)]

Using a larger $m$ than the true order still gives the right answer (with more work); using a smaller one does not. For high orders, or when $f$ contains $sin$ , $cos$ or $e^{z}$ , reading $a_{- 1}$ from a series is usually faster: $Res_{z = 0} \frac{sin z}{z ^{4}}$ is the coefficient of $z^{3}$ in $sin z$ , which is $- \frac{1}{6}$ .

RESIDUE AT INFINITY

Res_{z = \infty} f = - Res_{w = 0} [\frac{1}{w ^{2}} f (\frac{1}{w})]; finite z_{k} \sum Res_{z = z_{k}} f + Res_{z = \infty} f = 0

Equivalently $Res_{z = \infty} f = - b_{1}$ , where $b_{1}$ is the coefficient of $1/ z$ in the expansion of $f$ valid for large $∣ z ∣$ . The sum rule holds when $f$ has finitely many singularities. So if a contour $C$ encloses every finite singularity, $\oint_{C} f d z = - 2 π j Res_{z = \infty} f$ . For a rational $f$ whose denominator degree is at least the numerator degree plus 2, $Res_{z = \infty} f = 0$ and the finite residues add to zero.

RESIDUE THEOREM

\oint_{C} f (z) d z = 2 π j k \sum Res_{z = z_{k}} f (z)

$C$ is a simple closed contour traversed counter-clockwise; $f$ is analytic on $C$ and inside it except at finitely many isolated singularities $z_{k}$ inside $C$ . Only singularities strictly inside count. Cauchy's theorem and both integral formulas are special cases.

ARGUMENT PRINCIPLE

\frac{1}{2 π j} \oint_{C} \frac{f ^{'} ( z )}{f ( z )} d z = N - P

$C$ is a simple closed contour traversed counter-clockwise; $f$ is analytic on and inside $C$ except for poles inside it, and has no zeros or poles on $C$ . $N$ and $P$ are the numbers of zeros and poles of $f$ inside $C$ , each counted as many times as its order. Reason: at a zero of order $n$ , $f^{'} / f$ has a simple pole with residue $n$ , and at a pole of order $p$ one with residue $- p$ . The left side also equals the number of times the image curve $f (C)$ winds counter-clockwise around $w = 0$ . Related (Rouche's theorem): if $f$ and $g$ are analytic on and inside $C$ and $∣ g (z) ∣ < ∣ f (z) ∣$ on $C$ , then $f$ and $f + g$ have the same number of zeros inside $C$ ; e.g. on $∣ z ∣ = 2$ , $∣3 z^{2} + 1∣ \leq 13 < 32 = ∣ z^{5} ∣$ , so all five zeros of $z^{5} + 3 z^{2} + 1$ lie in $∣ z ∣ < 2$ .

INTEGRALS OVER 0 TO 2 PI

\int_{0}^{2 π} F (cos θ, sin θ) d θ = \oint_{∣ z ∣ = 1} F (\frac{z + z ^{- 1}}{2}, \frac{z - z ^{- 1}}{2 j}) \frac{d z}{j z}

Substitute $z = e^{j θ}$ , so $d θ = \frac{d z}{j z}$ and $θ : 0 \to 2 π$ traces the unit circle once counter-clockwise. $F$ must be a rational function of $cos θ$ and $sin θ$ , finite for all $θ$ . Keep only the poles with $∣ z ∣ < 1$ . Also useful: $cos n θ = \frac{z ^{n} + z ^{- n}}{2}$ and $sin n θ = \frac{z ^{n} - z ^{- n}}{2 j}$ .

STANDARD RESULT OVER 0 TO 2 PI

\int_{0}^{2 π} \frac{d θ}{a + b cos θ} = \int_{0}^{2 π} \frac{d θ}{a + b sin θ} = \frac{2 π}{a ^{2} - b ^{2}}, a > ∣ b ∣

$a$ and $b$ real. If $a < - ∣ b ∣$ the value is $- \frac{2 π}{a ^{2} - b ^{2}}$ . If $∣ a ∣ \leq ∣ b ∣$ the integrand is infinite somewhere on the interval and the integral does not exist as a proper integral. Over $0$ to $π$ the cosine version gives half, $\frac{π}{a ^{2} - b ^{2}}$ .

IMPROPER INTEGRALS OF RATIONAL FUNCTIONS

\int_{- \infty}^{\infty} \frac{P ( x )}{Q ( x )} d x = 2 π j Im z_{k} > 0 \sum Res_{z = z_{k}} \frac{P ( z )}{Q ( z )}

Conditions: $P$ and $Q$ polynomials, $de g Q \geq de g P + 2$ , and $Q$ has no real zeros. Close the real axis with a large semicircle in the upper half-plane; its contribution vanishes by the ML-inequality. Count only the poles with $Im z > 0$ . For an even integrand, $\int_{0}^{\infty}$ is half of $\int_{- \infty}^{\infty}$ .

FOURIER-TYPE INTEGRALS

\int_{- \infty}^{\infty} \frac{P ( x )}{Q ( x )} e^{j m x} d x = 2 π j Im z_{k} > 0 \sum Res_{z = z_{k}} [\frac{P ( z )}{Q ( z )} e^{j m z}], m > 0

Conditions: $de g Q \geq de g P + 1$ and $Q$ has no real zeros. With $P$ , $Q$ real, take real and imaginary parts to get $\int \frac{P}{Q} cos m x d x$ and $\int \frac{P}{Q} sin m x d x$ . Use $e^{j m z}$ , never $cos m z$ , on the contour: $∣ e^{j m z} ∣ = e^{- m y} \leq 1$ in the upper half-plane, whereas $cos m z$ grows there. Standard result: $\int_{- \infty}^{\infty} \frac{cos m x}{x ^{2} + a ^{2}} d x = \frac{π}{a} e^{- ma}$ for $a > 0$ , $m \geq 0$ .

SIMPLE POLES ON THE REAL AXIS (PRINCIPAL VALUE)

PV \int_{- \infty}^{\infty} g (x) d x = 2 π j Im z_{k} > 0 \sum Res_{z = z_{k}} g + π j real x_{k} \sum Res_{z = x_{k}} g, g (z) = \frac{P ( z )}{Q ( z )} e^{j m z}, m \geq 0

Use this when $Q$ has real zeros $x_{k}$ and every one of them is simple. Degree conditions as above: $de g Q \geq de g P + 1$ when $m > 0$ , and $de g Q \geq de g P + 2$ when $m = 0$ . Each real pole is bypassed by a small semicircle above it, which contributes $- π j$ times its residue; that is where the half-weight $π j$ comes from. Example: $g = e^{j m z} / z$ with $m > 0$ has residue $1$ at $z = 0$ and no other pole, so $PV \int_{- \infty}^{\infty} \frac{e ^{j m x}}{x} d x = π j$ ; its imaginary part gives $\int_{- \infty}^{\infty} \frac{sin m x}{x} d x = π$ , and hence $\int_{0}^{\infty} \frac{sin x}{x} d x = \frac{π}{2}$ .

Probability and statistics

Read the notes

COMPLEMENT AND ADDITION RULE

P (A^{'}) = 1 - P (A), P (A \cup B) = P (A) + P (B) - P (A \cap B)

Holds for any two events. Use the complement for every 'at least one' question: $P (at least one) = 1 - P (none)$ .

INCLUSION-EXCLUSION FOR THREE EVENTS

P (A \cup B \cup C) = P (A) + P (B) + P (C) - P (A \cap B) - P (B \cap C) - P (C \cap A) + P (A \cap B \cap C)

Add singles, subtract pairs, add the triple. The same pattern counts elements of a union: replace $P$ by $n$ .

CLASSICAL PROBABILITY

P (A) = \frac{n ( A )}{n ( S )}

Only when the sample space has finitely many outcomes and all of them are equally likely.

PERMUTATIONS AND COMBINATIONS

^{n} P_{r} = \frac{n !}{( n - r )!},^{n} C_{r} = (r n) = \frac{n !}{r ! ( n - r )!}, 0 \leq r \leq n

Permutations count ordered arrangements of $r$ of $n$ distinct objects; combinations count unordered selections. $0! = 1$ and $(r n) = (n - r n)$ .

OTHER COUNTING RULES

ordered selections of r from n, with replacement: n^{r} arrangements of n items with n_{1}, \dots, n_{k} alike: \frac{n !}{n _{1} ! n _{2} ! \dots n _{k} !} circular arrangements of n distinct items: (n - 1)! unordered selections of r from n types, repetition allowed: (r n + r - 1)

In the second rule $n_{1} + n_{2} + \dots + n_{k} = n$ . The circular count treats rotations as the same arrangement; if reflections are also the same (a necklace), the count is $(n - 1)! /2$ for $n \geq 3$ .

CONDITIONAL PROBABILITY

P (A ∣ B) = \frac{P ( A \cap B )}{P ( B )}, P (B) > 0

Undefined when $P (B) = 0$ . In general $P (A ∣ B) \neq = P (B ∣ A)$ .

MULTIPLICATION RULE

P (A \cap B) = P (A ∣ B) P (B) = P (B ∣ A) P (A), P (A \cap B \cap C) = P (A) P (B ∣ A) P (C ∣ A \cap B)

The chain form is how sequential draws without replacement are multiplied along a tree.

INDEPENDENCE

A, B independent ⟺ P (A \cap B) = P (A) P (B)

Equivalently $P (A ∣ B) = P (A)$ when $P (B) > 0$ . If $A, B$ are independent, so are $A$ and $B^{'}$ , $A^{'}$ and $B$ , and $A^{'}$ and $B^{'}$ . Three events are mutually independent only if all three pairwise products hold AND $P (A \cap B \cap C) = P (A) P (B) P (C)$ ; pairwise independence alone is not enough.

THEOREM OF TOTAL PROBABILITY

P (A) = i = 1 \sum n P (A ∣ B_{i}) P (B_{i})

$B_{1}, \dots, B_{n}$ must partition $S$ : mutually exclusive, together exhaustive, each with $P (B_{i}) > 0$ .

BAYES' THEOREM

P (B_{k} ∣ A) = \frac{P ( A ∣ B _{k} ) P ( B _{k} )}{\sum _{i = 1}^{n} P ( A ∣ B _{i} ) P ( B _{i} )}

Same partition conditions, and $P (A) > 0$ . $P (B_{k})$ is the prior probability of cause $B_{k}$ ; $P (B_{k} ∣ A)$ is its posterior probability after observing $A$ .

PMF (DISCRETE)

p (x) \geq 0, x \sum p (x) = 1, P (X \in A) = x \in A \sum p (x)

To find an unknown constant in a PMF, set the total equal to 1.

PDF (CONTINUOUS)

f (x) \geq 0, \int_{- \infty}^{\infty} f (x) d x = 1, P (a \leq X \leq b) = \int_{a}^{b} f (x) d x

For a continuous variable $P (X = a) = 0$ , so $<$ and $\leq$ give the same probability.

CDF

F (x) = P (X \leq x), P (a < X \leq b) = F (b) - F (a), f (x) = \frac{d F ( x )}{d x}

Every CDF is non-decreasing and right-continuous, with $F (x) \to 0$ as $x \to - \infty$ and $F (x) \to 1$ as $x \to \infty$ . The relation $f = d F / d x$ holds for a continuous variable at every $x$ where $F$ is differentiable. For a discrete variable $F$ is a staircase that jumps by $p (x)$ at each value $x$ , so $P (X < b) = F (b) - P (X = b)$ and strict and non-strict inequalities give different answers.

EXPECTATION

E [X] E [g (X)] = x \sum x p (x) (discrete), E [X] = \int_{- \infty}^{\infty} x f (x) d x (continuous) = x \sum g (x) p (x) or \int_{- \infty}^{\infty} g (x) f (x) d x

The mean exists only when the sum or integral converges absolutely. $E [g (X)]$ does not need the distribution of $g (X)$ itself.

VARIANCE AND STANDARD DEVIATION

Var (X) = σ^{2} = E [(X - μ)^{2}] = E [X^{2}] - (E [X])^{2}, σ = Var (X)

Here $μ = E [X]$ . Variance is never negative, and is zero only when $X$ is a constant with probability 1. The second form is the one to compute with.

LINEAR TRANSFORMATION

E [a X + b] = a E [X] + b, Var (a X + b) = a^{2} Var (X), σ_{a X + b} = ∣ a ∣ σ_{X}

For any constants $a, b$ . Adding $b$ moves the mean but not the spread; the sign of $a$ does not affect the variance.

DENSITY OF A FUNCTION OF A RANDOM VARIABLE

Y = g (X), g strictly monotonic and differentiable : f_{Y} (y) = f_{X} (g^{- 1} (y)) \frac{d g ^{- 1} ( y )}{d y}

For a $g$ that is not monotonic, work through the CDF: find $F_{Y} (y) = P (g (X) \leq y)$ , then differentiate. For example $Y = X^{2}$ gives $F_{Y} (y) = F_{X} (y) - F_{X} (- y)$ and $f_{Y} (y) = \frac{f _{X} ( y ) + f _{X} ( - y )}{2 y}$ for $y > 0$ , with $X$ continuous.

BINOMIAL DISTRIBUTION

P (X = x) = (x n) p^{x} (1 - p)^{n - x}, x = 0, 1, \dots, n

Number of successes in $n$ independent trials, each with the same success probability $p$ . Mean $n p$ , variance $n p (1 - p)$ , which is less than the mean whenever $p > 0$ .

POISSON DISTRIBUTION

P (X = x) = \frac{e ^{- λ} λ ^{x}}{x !}, x = 0, 1, 2, \dots

Count of events that occur independently at a constant average rate. Mean = variance = $λ$ . If events occur as a Poisson process at rate $α$ per unit time, the count in a time $t$ is Poisson with $λ = α t$ .

POISSON AS A LIMIT OF THE BINOMIAL

(x n) p^{x} (1 - p)^{n - x} \to \frac{e ^{- λ} λ ^{x}}{x !} as n \to \infty, p \to 0, n p = λ fixed

Used as an approximation when $n$ is large and $p$ is small (rare events): put $λ = n p$ .

GEOMETRIC DISTRIBUTION

P (X = x) = (1 - p)^{x - 1} p, x = 1, 2, \dots; P (X > m + n ∣ X > m) = P (X > n), m, n = 0, 1, 2, \dots

Convention here: $X$ is the number of the trial on which the first success occurs, and $P (X > n) = (1 - p)^{n}$ . If $X$ instead counts the failures before the first success, $P (X = x) = (1 - p)^{x} p$ for $x = 0, 1, \dots$ , the mean becomes $(1 - p) / p$ and the variance is unchanged. Among distributions on the positive integers, the geometric is the only memoryless one.

UNIFORM DISTRIBUTION

f (x) = \frac{1}{b - a}, a \leq x \leq b; F (x) = \frac{x - a}{b - a}, a \leq x \leq b; E [X] = \frac{a + b}{2}, Var (X) = \frac{( b - a ) ^{2}}{12}

$F (x) = 0$ for $x < a$ and $F (x) = 1$ for $x > b$ . The probability of a sub-interval of $[a, b]$ is its length divided by $b - a$ .

EXPONENTIAL DISTRIBUTION

f (x) = λ e^{- λ x}, x \geq 0; F (x) = 1 - e^{- λ x}, x \geq 0; P (X > x) = e^{- λ x}

Rate $λ > 0$ ; $f$ and $F$ are zero for $x < 0$ . Mean $1/ λ$ and variance $1/ λ^{2}$ , so the standard deviation equals the mean; the median is $ln 2/ λ$ . It is the waiting time to the first event, and between successive events, of a Poisson process of rate $λ$ .

MEMORYLESS PROPERTY

P (X > s + t ∣ X > s) = P (X > t), s, t \geq 0

Holds for the exponential, the only continuous distribution with this property: a component that has survived to time $s$ has the same remaining-life distribution as a new one.

MINIMUM OF INDEPENDENT EXPONENTIALS

X_{i} \sim Exp (λ_{i}) independent \Rightarrow min (X_{1}, \dots, X_{n}) \sim Exp (λ_{1} + \dots + λ_{n}), P (X_{1} < X_{2}) = \frac{λ _{1}}{λ _{1} + λ _{2}}

Proof: $P (min > t) = \prod_{i} P (X_{i} > t) = e^{- (λ_{1} + \dots + λ_{n}) t}$ . Rates add, not means. The maximum is not exponential: $P (max \leq t) = \prod_{i} (1 - e^{- λ_{i} t})$ .

NORMAL DISTRIBUTION

f (x) = \frac{1}{σ 2 π} e^{- \frac{( x - μ ) ^{2}}{2 σ ^{2}}}, - \infty < x < \infty

Symmetric about $μ$ ; mean = median = mode = $μ$ ; points of inflection at $μ \pm σ$ ; peak height $1/ (σ 2 π)$ .

STANDARDISATION AND THE TABLE CONVENTION

Z = \frac{X - μ}{σ} \sim N (0, 1), P (X \leq x) = Φ (\frac{x - μ}{σ}), Φ (- z) = 1 - Φ (z)

Convention in these notes: $Φ (z) = P (Z \leq z)$ , the area to the LEFT of $z$ , with $Φ (0) = 0.5$ . Some tables list the area from 0 to $z$ , which equals $Φ (z) - 0.5$ ; check which one you are given.

THE 68-95-99.7 RULE (APPROXIMATE)

P (∣ X - μ ∣ \leq σ) \approx 0.6827, P (∣ X - μ ∣ \leq 2 σ) \approx 0.9545, P (∣ X - μ ∣ \leq 3 σ) \approx 0.9973

For a normal variable only. The rounded figures 68%, 95% and 99.7% are approximations; use the table value whenever a question supplies one. Standard quantiles: $Φ (1.645) \approx 0.95$ , $Φ (1.96) \approx 0.975$ , $Φ (2.576) \approx 0.995$ .

MARGINAL AND CONDITIONAL DISTRIBUTIONS

f_{X} (x) = \int_{- \infty}^{\infty} f (x, y) d y, f_{Y ∣ X} (y ∣ x) = \frac{f ( x , y )}{f _{X} ( x )}, f_{X} (x) > 0

Discrete: $p_{X} (x) = \sum_{y} p (x, y)$ (the totals along one direction of the table) and $P (Y = y ∣ X = x) = p (x, y) / p_{X} (x)$ .

INDEPENDENCE OF RANDOM VARIABLES

f (x, y) = f_{X} (x) f_{Y} (y) for all x, y (discrete: p (x, y) = p_{X} (x) p_{Y} (y))

Discrete case: one pair of values where the equality fails proves dependence. Continuous case: the equality must fail on a region of positive area (a single point proves nothing, since a density can be changed at a point); if both sides are continuous, one point of failure is enough. If the set where $f > 0$ is not of the product form ${x \in A} \times {y \in B}$ (for example the triangle $0 < x < y < 1$ ), $X$ and $Y$ cannot be independent.

LAW OF TOTAL EXPECTATION

E [X] = i = 1 \sum n E [X ∣ B_{i}] P (B_{i}), E [X] = E [E [X ∣ Y]]

$B_{1}, \dots, B_{n}$ partition $S$ with each $P (B_{i}) > 0$ . $E [X ∣ Y = y]$ is the mean of the conditional distribution: $\sum_{x} x P (X = x ∣ Y = y)$ or $\int x f_{X ∣ Y} (x ∣ y) d x$ . This is total probability applied to means.

COVARIANCE

Cov (X, Y) = E [(X - μ_{X}) (Y - μ_{Y})] = E [X Y] - E [X] E [Y]

$Cov (X, X) = Var (X)$ and $Cov (a X + b, c Y + d) = a c Cov (X, Y)$ .

CORRELATION COEFFICIENT

ρ_{X Y} = \frac{Cov ( X , Y )}{σ _{X} σ _{Y}}, - 1 \leq ρ_{X Y} \leq 1

Defined only when $σ_{X} > 0$ and $σ_{Y} > 0$ . $∣ ρ ∣ = 1$ exactly when $Y = a X + b$ with probability 1 for some $a \neq = 0$ ( $ρ$ then has the sign of $a$ ). $ρ$ is unchanged by shifting either variable or rescaling it by a positive factor; rescaling one variable by a negative factor reverses its sign.

VARIANCE OF A LINEAR COMBINATION

Var (a X + bY) = a^{2} Var (X) + b^{2} Var (Y) + 2 ab Cov (X, Y)

Always true. When $X, Y$ are uncorrelated (in particular, independent) the covariance term drops, so $Var (X - Y) = Var (X) + Var (Y)$ : variances add even for a difference.

SUMS OF RANDOM VARIABLES

E [i \sum X_{i}] = i \sum E [X_{i}] (always), Var (i \sum X_{i}) = i \sum Var (X_{i}) (independent)

Means always add; variances add when the variables are independent (pairwise uncorrelated is enough). Also, for independent $X, Y$ : $E [X Y] = E [X] E [Y]$ .

SUMS OF INDEPENDENT STANDARD VARIABLES

X \sim Poisson (λ_{1}), Y \sim Poisson (λ_{2}) \Rightarrow X + Y \sim Poisson (λ_{1} + λ_{2}) X \sim N (μ_{1}, σ_{1}^{2}), Y \sim N (μ_{2}, σ_{2}^{2}) \Rightarrow X + Y \sim N (μ_{1} + μ_{2}, σ_{1}^{2} + σ_{2}^{2}) X_{1}, \dots, X_{n} \sim Bernoulli (p) \Rightarrow X_{1} + \dots + X_{n} \sim Binomial (n, p)

Each line requires the variables to be independent. A linear combination $a X + bY$ of independent normals is normal with mean $a μ_{1} + b μ_{2}$ and variance $a^{2} σ_{1}^{2} + b^{2} σ_{2}^{2}$ .

MEAN OF DATA

\overset{x}{ˉ} = \frac{1}{n} i = 1 \sum n x_{i}, \overset{x}{ˉ} = \frac{\sum f _{i} x _{i}}{\sum f _{i}} (frequency data)

For grouped (class-interval) data, $x_{i}$ is the class mid-point and the result is an approximation. Deviations from the mean always sum to zero (weighted by frequency for frequency data).

VARIANCE AND STANDARD DEVIATION OF DATA

σ^{2} = \frac{1}{n} i = 1 \sum n (x_{i} - \overset{x}{ˉ})^{2} = \frac{1}{n} i = 1 \sum n x_{i}^{2} - \overset{x}{ˉ}^{2}, s^{2} = \frac{1}{n - 1} i = 1 \sum n (x_{i} - \overset{x}{ˉ})^{2} = \frac{n}{n - 1} σ^{2}

$σ^{2}$ : population (divisor $n$ ); $s^{2}$ : sample (divisor $n - 1$ ). The standard deviation is the positive square root. Adding a constant to every value changes neither; multiplying every value by $c$ multiplies the standard deviation by $∣ c ∣$ .

CORRELATION COEFFICIENT OF DATA

r = \frac{S _{x y}}{S _{xx} S _{y y}}, S_{x y} = \sum (x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ}), S_{xx} = \sum (x_{i} - \overset{x}{ˉ})^{2}, S_{y y} = \sum (y_{i} - \overset{y}{ˉ})^{2}

Defined when neither variable is constant ( $S_{xx}, S_{y y} > 0$ ); then $- 1 \leq r \leq 1$ . The divisors cancel, so $r$ is the same whether $n$ or $n - 1$ is used, provided the same one is used throughout. Computational form: $S_{x y} = \sum x_{i} y_{i} - n \overset{x}{ˉ} \overset{y}{ˉ}$ .

RANK CORRELATION (SPEARMAN)

r_{s} = 1 - \frac{6 \sum _{i = 1}^{n} d _{i}^{2}}{n ( n ^{2} - 1 )}

$d_{i}$ is the difference between the two ranks of item $i$ . Valid when there are no tied ranks; with ties, give tied items the average of their ranks and compute $r$ from the ranks directly. Like $r$ , it lies between $- 1$ and $1$ .

THE TWO REGRESSION LINES

regression of y on x : regression of x on y : y - \overset{y}{ˉ} = b_{y x} (x - \overset{x}{ˉ}), b_{y x} = r \frac{σ _{y}}{σ _{x}} = \frac{S _{x y}}{S _{xx}} x - \overset{x}{ˉ} = b_{x y} (y - \overset{y}{ˉ}), b_{x y} = r \frac{σ _{x}}{σ _{y}} = \frac{S _{x y}}{S _{y y}}

The line of $y$ on $x$ minimises the sum of squared vertical deviations and is used to predict $y$ from $x$ ; the line of $x$ on $y$ minimises squared horizontal deviations and predicts $x$ from $y$ . Both lines pass through $(\overset{x}{ˉ}, \overset{y}{ˉ})$ .

PRODUCT OF THE REGRESSION COEFFICIENTS

b_{y x} b_{x y} = r^{2}, r = \pm b_{y x} b_{x y}

$r$ , $b_{y x}$ and $b_{x y}$ all have the same sign, and that common sign is taken for $r$ . Since $r^{2} \leq 1$ , the two coefficients cannot both exceed 1 in magnitude.

SAMPLING DISTRIBUTION OF THE MEAN

\overset{ˉ}{X} = \frac{1}{n} i = 1 \sum n X_{i}, E [\overset{ˉ}{X}] = μ, Var (\overset{ˉ}{X}) = \frac{σ ^{2}}{n}, standard error = \frac{σ}{n}

Needs only independence (uncorrelated is enough) and a common mean and variance, not normality. Quadrupling $n$ halves the standard error.

UNBIASED SAMPLE VARIANCE

s^{2} = \frac{1}{n - 1} i = 1 \sum n (X_{i} - \overset{ˉ}{X})^{2}, E [s^{2}] = σ^{2}

Dividing by $n$ instead gives an estimate whose expected value is $\frac{n - 1}{n} σ^{2}$ , slightly too small; this is why the sample variance uses $n - 1$ .

CENTRAL LIMIT THEOREM

\frac{X ˉ - μ}{σ / n} \sim approx. N (0, 1), i = 1 \sum n X_{i} \sim approx. N (n μ, n σ^{2}) (n large)

For independent, identically distributed $X_{i}$ with finite, non-zero variance, whatever their distribution. If the $X_{i}$ are themselves normal, the result is exact for every $n$ .

CHEBYSHEV'S INEQUALITY

P (∣ X - μ ∣ \geq k σ) \leq \frac{1}{k ^{2}}, k > 0

Holds for ANY distribution with finite variance, so it is a loose bound (and says nothing for $k \leq 1$ ): for a normal variable the true probability at $k = 2$ is about $0.046$ , against the bound $0.25$ . Applied to $\overset{ˉ}{X}$ , whose variance is $σ^{2} / n$ , it gives $P (∣ \overset{ˉ}{X} - μ ∣ \geq ε) \leq \frac{σ ^{2}}{n ε ^{2}} \to 0$ for every $ε > 0$ : the weak law of large numbers.

Numerical methods

Read the notes

LU DECOMPOSITION

A = LU, L y = b (forward substitution), U x = y (back substitution)

Doolittle form: $L$ is unit lower triangular (1s on its diagonal) and holds the multipliers $m_{ik}$ below the diagonal; $U$ is the upper-triangular matrix that Gaussian elimination without row swaps produces. Crout form puts the 1s on the diagonal of $U$ instead. For a non-singular $A$ , the factorisation exists without row swaps exactly when every leading principal minor of $A$ is non-zero. One factorisation serves any number of right-hand sides $b$ .

GAUSS-JACOBI ITERATION

x_{i}^{(k + 1)} = \frac{1}{a _{ii}} b_{i} - j \neq = i \sum a_{ij} x_{j}^{(k)}

Every new component uses only the previous sweep's values. Needs $a_{ii} \neq = 0$ (reorder the equations if necessary).

GAUSS-SEIDEL ITERATION

x_{i}^{(k + 1)} = \frac{1}{a _{ii}} (b_{i} - j < i \sum a_{ij} x_{j}^{(k + 1)} - j > i \sum a_{ij} x_{j}^{(k)})

Each new component is used immediately, later in the same sweep. Needs $a_{ii} \neq = 0$ .

ITERATION MATRICES AND THE CONVERGENCE TEST

A = D + L + U : T_{J} = - D^{- 1} (L + U), T_{GS} = - (D + L)^{- 1} U, x^{(k)} \to x for every x^{(0)} ⟺ ρ (T) < 1

Here $D$ is the diagonal of $A$ and $L$ , $U$ are its strictly lower and strictly upper triangular parts (not the LU factors). Each method has the form $x^{(k + 1)} = T x^{(k)} + c$ , and $ρ (T)$ , the spectral radius, is the largest $∣ λ ∣$ over the eigenvalues of $T$ .

STRICT DIAGONAL DOMINANCE (SUFFICIENT CONDITION)

∣ a_{ii} ∣ > j \neq = i \sum ∣ a_{ij} ∣ for every row i

If it holds, both Jacobi and Gauss-Seidel converge for every starting vector. It is sufficient, not necessary: a system that fails it may still converge. Gauss-Seidel also converges for every start when $A$ is symmetric positive definite.

ORDER OF CONVERGENCE P

n \to \infty lim \frac{∣ e _{n + 1} ∣}{∣ e _{n} ∣ ^{p}} = C, 0 < C < \infty, e_{n} = x_{n} - α

$p = 1$ is linear (it also needs $C < 1$ ; the error shrinks by a roughly constant factor each step). $p > 1$ is superlinear. $p = 2$ is quadratic: the number of correct digits roughly doubles each step.

BISECTION

x_{mid} = \frac{a + b}{2}, ∣ x_{n} - α ∣ \leq \frac{b - a}{2 ^{n}}, n \geq \frac{ln ( b - a ) - ln ε}{ln 2}

Needs $f$ continuous on $[a, b]$ with $f (a) f (b) < 0$ ; keep the half on which the sign still changes. The $n$ -th midpoint $x_{n}$ is within $(b - a) / 2^{n}$ of a root, and the bracket after $n$ halvings has that length. The error bound halves at every step, which is why bisection is counted as linearly convergent with ratio $1/2$ ; the actual error need not decrease at every step. The number of steps for a tolerance $ε$ does not depend on $f$ .

REGULA FALSI (FALSE POSITION)

c = \frac{a f ( b ) - b f ( a )}{f ( b ) - f ( a )}

The $x$ -intercept of the chord joining $(a, f (a))$ and $(b, f (b))$ . Needs $f (a) f (b) < 0$ , and the new bracket is whichever of $[a, c]$ , $[c, b]$ keeps the sign change. Always converges for continuous $f$ , but usually only linearly, because one end of the bracket often stays fixed.

NEWTON-RAPHSON

x_{n + 1} = x_{n} - \frac{f ( x _{n} )}{f ^{'} ( x _{n} )}, e_{n + 1} \approx \frac{f ^{''} ( α )}{2 f ^{'} ( α )} e_{n}^{2}

The $x$ -intercept of the tangent at $x_{n}$ . For $f^{''}$ continuous near a simple root ( $f^{'} (α) \neq = 0$ ), convergence from a start close enough to the root is at least quadratic (faster when $f^{''} (α) = 0$ ). At a root of multiplicity $m$ it is only linear, with $e_{n + 1} / e_{n} \to (m - 1) / m$ ; the modified step $x_{n + 1} = x_{n} - m f (x_{n}) / f^{'} (x_{n})$ restores quadratic convergence.

NEWTON-RAPHSON FOR ROOTS AND RECIPROCALS

N : x_{n + 1} = \frac{1}{2} (x_{n} + \frac{N}{x _{n}}), k N : x_{n + 1} = \frac{1}{k} ((k - 1) x_{n} + \frac{N}{x _{n}^{k - 1}}), \frac{1}{N} : x_{n + 1} = x_{n} (2 - N x_{n})

From $f (x) = x^{2} - N$ , $f (x) = x^{k} - N$ and $f (x) = 1/ x - N$ respectively, with $N > 0$ . The $N$ iteration converges from every $x_{0} > 0$ (after the first step every iterate is at least $N$ , and they decrease). The reciprocal iteration uses no division; since $1 - N x_{n + 1} = (1 - N x_{n})^{2}$ , it converges to $1/ N$ exactly when $0 < x_{0} < 2/ N$ .

SECANT

x_{n + 1} = x_{n} - f (x_{n}) \frac{x _{n} - x _{n - 1}}{f ( x _{n} ) - f ( x _{n - 1} )}

Newton-Raphson with $f^{'} (x_{n})$ replaced by the slope through the last two iterates. Needs two starting values but no derivative and no sign change. Order $(1 + 5) /2 \approx 1.618$ near a simple root; it can diverge because it does not keep a bracket.

FIXED-POINT ITERATION

x_{n + 1} = g (x_{n}), converges for x_{0} near α if ∣ g^{'} (α) ∣ < 1

Rewrite $f (x) = 0$ as $x = g (x)$ , with $g^{'}$ continuous near $α$ . Linear convergence with ratio $∣ g^{'} (α) ∣$ when $0 < ∣ g^{'} (α) ∣ < 1$ ; the iterates move away if $∣ g^{'} (α) ∣ > 1$ . Interval test: if $g$ maps $[a, b]$ into itself and $∣ g^{'} (x) ∣ \leq k < 1$ for all $x$ in $[a, b]$ , then $[a, b]$ contains exactly one fixed point and the iteration converges to it from every $x_{0}$ in $[a, b]$ . Newton-Raphson is the choice $g (x) = x - f (x) / f^{'} (x)$ , for which $g^{'} (α) = 0$ at a simple root, which is why it is faster than linear.

COMPOSITE TRAPEZOIDAL RULE

\int_{a}^{b} f (x) d x \approx \frac{h}{2} [y_{0} + 2 (y_{1} + y_{2} + \dots + y_{n - 1}) + y_{n}]

Any $n \geq 1$ . A straight line on each subinterval. Exact for polynomials of degree $\leq 1$ .

TRAPEZOIDAL ERROR

E_{T} = - \frac{( b - a ) h ^{2}}{12} f^{''} (ξ) (one interval: - \frac{h ^{3}}{12} f^{''} (ξ))

Global error $O (h^{2})$ : halving $h$ cuts the error by a factor of about 4. If $f^{''} > 0$ on $[a, b]$ (convex $f$ ) the rule overestimates; if $f^{''} < 0$ it underestimates.

COMPOSITE SIMPSON'S 1/3 RULE

\int_{a}^{b} f (x) d x \approx \frac{h}{3} [y_{0} + 4 (y_{1} + y_{3} + \dots + y_{n - 1}) + 2 (y_{2} + y_{4} + \dots + y_{n - 2}) + y_{n}]

$n$ must be even (an odd number of points). A parabola through each pair of subintervals. Exact for polynomials of degree $\leq 3$ .

SIMPSON'S 1/3 ERROR

E_{S} = - \frac{( b - a ) h ^{4}}{180} f^{(4)} (ξ) (one pair of intervals: - \frac{h ^{5}}{90} f^{(4)} (ξ))

Global error $O (h^{4})$ : halving $h$ cuts the error by a factor of about 16.

COMPOSITE SIMPSON'S 3/8 RULE

\int_{a}^{b} f (x) d x \approx \frac{3 h}{8} [y_{0} + 3 (y_{1} + y_{2} + y_{4} + y_{5} + \dots + y_{n - 2} + y_{n - 1}) + 2 (y_{3} + y_{6} + \dots + y_{n - 3}) + y_{n}]

$n$ must be a multiple of 3. A cubic through each group of three subintervals. Exact for polynomials of degree $\leq 3$ , the same degree as the 1/3 rule.

SIMPSON'S 3/8 ERROR

E_{3/8} = - \frac{( b - a ) h ^{4}}{80} f^{(4)} (ξ) (one group of three intervals: - \frac{3 h ^{5}}{80} f^{(4)} (ξ))

Also $O (h^{4})$ globally, with a larger constant than the 1/3 rule at the same $h$ . Useful when $n$ is odd and at least 3: apply the 3/8 rule to three subintervals and the 1/3 rule to the rest.

TWO-POINT GAUSS-LEGENDRE QUADRATURE

\int_{- 1}^{1} g (t) d t \approx g (- \frac{1}{3}) + g (\frac{1}{3}), \int_{a}^{b} f (x) d x \approx \frac{b - a}{2} [f (m - \frac{b - a}{2 3}) + f (m + \frac{b - a}{2 3})], m = \frac{a + b}{2}

The second form follows from the substitution $x = m + \frac{b - a}{2} t$ . Two function values at unequally placed points, yet exact for polynomials of degree $\leq 3$ , like Simpson's rules with three. Check: for $\int_{0}^{2} x^{3} d x$ it gives $(1 - u)^{3} + (1 + u)^{3} = 2 + 6 u^{2} = 4$ with $u = 1/ 3$ , the exact value.

EULER (EXPLICIT, FORWARD)

y_{n + 1} = y_{n} + h f (x_{n}, y_{n})

Follows the tangent line at the start of the step. Order 1: local error $O (h^{2})$ , global error $O (h)$ .

BACKWARD (IMPLICIT) EULER

y_{n + 1} = y_{n} + h f (x_{n + 1}, y_{n + 1})

Order 1, but $y_{n + 1}$ appears on both sides and must be solved for (directly when $f$ is linear in $y$ ). For $y^{'} = λ y$ with real $λ < 0$ it gives $y_{n + 1} = y_{n} / (1 - hλ)$ , which decays for every $h > 0$ .

MODIFIED EULER (HEUN), A SECOND-ORDER RUNGE-KUTTA METHOD

y^{*} = y_{n} + h f (x_{n}, y_{n}), y_{n + 1} = y_{n} + \frac{h}{2} [f (x_{n}, y_{n}) + f (x_{n + 1}, y^{*})]

An Euler predictor followed by a trapezoidal corrector. Order 2: local error $O (h^{3})$ , global error $O (h^{2})$ . The same computation written as $k_{1} = h f (x_{n}, y_{n})$ , $k_{2} = h f (x_{n} + h, y_{n} + k_{1})$ , $y_{n + 1} = y_{n} + \frac{1}{2} (k_{1} + k_{2})$ is one second-order Runge-Kutta method; the midpoint method below is another. Some texts apply the corrector repeatedly, putting each new $y_{n + 1}$ back into $f (x_{n + 1}, \cdot)$ until two successive values agree; one application is the formula above.

RUNGE-KUTTA, FOURTH ORDER (CLASSICAL)

k_{1} k_{2} k_{3} k_{4} y_{n + 1} = h f (x_{n}, y_{n}) = h f (x_{n} + \frac{h}{2}, y_{n} + \frac{k _{1}}{2}) = h f (x_{n} + \frac{h}{2}, y_{n} + \frac{k _{2}}{2}) = h f (x_{n} + h, y_{n} + k_{3}) = y_{n} + \frac{1}{6} (k_{1} + 2 k_{2} + 2 k_{3} + k_{4})

Four evaluations of $f$ per step. Order 4: local error $O (h^{5})$ , global error $O (h^{4})$ . Note that $k_{3}$ uses $k_{2}$ , not $k_{1}$ , and $k_{4}$ uses a full step with $k_{3}$ .

TAYLOR-SERIES METHOD

y_{n + 1} = y_{n} + h y_{n}^{'} + \frac{h ^{2}}{2 !} y_{n}^{''} + \frac{h ^{3}}{3 !} y_{n}^{'''} + \dots, y^{'} = f, y^{''} = f_{x} + f f_{y}

The higher derivatives come from differentiating the equation itself ( $f_{x}$ , $f_{y}$ are partial derivatives, evaluated at $(x_{n}, y_{n})$ ). Truncating after the $h^{p}$ term gives a method of order $p$ ; keeping only the $h$ term is Euler's method.

PICARD'S SUCCESSIVE APPROXIMATIONS

y_{k + 1} (x) = y_{0} + \int_{x_{0}}^{x} f (t, y_{k} (t)) d t, y_{0} (x) \equiv y_{0}

Each iterate is a function, not a number. For $y^{'} = y$ , $y (0) = 1$ : $y_{1} = 1 + x$ , $y_{2} = 1 + x + \frac{x ^{2}}{2}$ , and each further iterate adds the next term of the series of $e^{x}$ .

STABILITY OF EXPLICIT EULER

y^{'} = λ y (λ < 0) : y_{n + 1} = (1 + hλ) y_{n}, ∣1 + hλ ∣ \leq 1 ⟺ 0 < h \leq \frac{2}{∣ λ ∣}

$λ$ real. For $0 < h < 1/∣ λ ∣$ the numerical solution decays without changing sign, like the exact $e^{λ x}$ ; at $h = 1/∣ λ ∣$ it is $0$ after one step. For $1/∣ λ ∣ < h < 2/∣ λ ∣$ it alternates in sign while decaying. At $h = 2/∣ λ ∣$ it alternates with constant magnitude, and for $h > 2/∣ λ ∣$ it grows without bound.

NORMAL EQUATIONS FOR Y = A + BX

\sum y_{i} = na + b \sum x_{i}, \sum x_{i} y_{i} = a \sum x_{i} + b \sum x_{i}^{2}

Two linear equations in $a$ and $b$ . For a parabola $y = a + b x + c x^{2}$ the same method gives three: $\sum y_{i} = na + b \sum x_{i} + c \sum x_{i}^{2}$ , $\sum x_{i} y_{i} = a \sum x_{i} + b \sum x_{i}^{2} + c \sum x_{i}^{3}$ , $\sum x_{i}^{2} y_{i} = a \sum x_{i}^{2} + b \sum x_{i}^{3} + c \sum x_{i}^{4}$ . For a line through the origin, $y = b x$ , the single normal equation gives $b = \sum x_{i} y_{i} / \sum x_{i}^{2}$ .

LEAST-SQUARES SLOPE AND INTERCEPT (REGRESSION OF Y ON X)

b = \frac{S _{x y}}{S _{xx}} = \frac{n \sum x _{i} y _{i} - \sum x _{i} \sum y _{i}}{n \sum x _{i}^{2} - ( \sum x _{i} ) ^{2}}, a = \overset{y}{ˉ} - b \overset{x}{ˉ}

The line passes through $(\overset{x}{ˉ}, \overset{y}{ˉ})$ and its residuals sum to zero. It minimises vertical deviations, so it is the line for predicting $y$ from $x$ .

CORRELATION COEFFICIENT (PEARSON)

r = \frac{S _{x y}}{S _{xx} S _{y y}} = \frac{n \sum x _{i} y _{i} - \sum x _{i} \sum y _{i}}{[ n \sum x _{i}^{2} - ( \sum x _{i} ) ^{2} ] [ n \sum y _{i}^{2} - ( \sum y _{i} ) ^{2} ]}

Defined only when neither $S_{xx}$ nor $S_{y y}$ is zero. $- 1 \leq r \leq 1$ , and $∣ r ∣ = 1$ exactly when all the points lie on one (non-horizontal, non-vertical) line. $r$ is unchanged by a change of origin and scale, $u = (x - A) / h$ , $v = (y - B) / k$ with $hk > 0$ ; if $hk < 0$ only its sign changes. It measures linear association only: for $x = - 1, 0, 1$ and $y = x^{2}$ , $S_{x y} = 0$ so $r = 0$ , although $y$ is a function of $x$ .

THE TWO REGRESSION LINES

y - \overset{y}{ˉ} = b_{y x} (x - \overset{x}{ˉ}), x - \overset{x}{ˉ} = b_{x y} (y - \overset{y}{ˉ}), b_{y x} = \frac{S _{x y}}{S _{xx}} = r \frac{σ _{y}}{σ _{x}}, b_{x y} = \frac{S _{x y}}{S _{y y}} = r \frac{σ _{x}}{σ _{y}}, b_{y x} b_{x y} = r^{2}

Both lines pass through $(\overset{x}{ˉ}, \overset{y}{ˉ})$ , so when $∣ r ∣ < 1$ solving them together gives the means. The two coefficients have the sign of $r$ , and $r = \pm b_{y x} b_{x y}$ with that common sign. The lines coincide only when $∣ r ∣ = 1$ ; when $r = 0$ they are $y = \overset{y}{ˉ}$ and $x = \overset{x}{ˉ}$ , perpendicular. The coefficients are unchanged by a change of origin but not of scale: with $u = (x - A) / h$ , $v = (y - B) / k$ , $b_{y x} = (k / h) b_{v u}$ .

SPEARMAN'S RANK CORRELATION

ρ_{s} = 1 - \frac{6 \sum d _{i}^{2}}{n ( n ^{2} - 1 )}

Rank the $x$ values and the $y$ values separately; $d_{i}$ is the difference between the two ranks of item $i$ . This form is exact when there are no tied ranks (it is then Pearson's $r$ computed on the ranks). $- 1 \leq ρ_{s} \leq 1$ .

GRADIENT-DESCENT UPDATE

w \leftarrow w - η \frac{\partial L}{\partial w}

$η > 0$ is the learning rate. Every partial derivative is evaluated at the current (old) weights, and then all the weights are updated together.

SQUARED-ERROR LOSS

L = \frac{1}{2} (\overset{y}{^} - t)^{2}, \frac{\partial L}{\partial y ^} = \overset{y}{^} - t

$\overset{y}{^}$ is the network output and $t$ the target. The factor $\frac{1}{2}$ is a convention that cancels the 2 from differentiation; without it every gradient doubles, so check which form a question uses.

CHAIN RULE FOR ONE NEURON

z = w a_{in} + b, a = φ (z) : δ = \frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} φ^{'} (z), \frac{\partial L}{\partial w} = δ a_{in}, \frac{\partial L}{\partial b} = δ, \frac{\partial L}{\partial a _{in}} = δ w

$a_{in}$ is the neuron's input (the previous layer's output) and $φ$ its activation. The last result is what is passed backwards to the previous layer.

BACKPROPAGATION THROUGH A LAYER

z_{j} = i \sum w_{j i} a_{i} + b_{j}, a_{j} = φ (z_{j}) : δ_{j} = φ^{'} (z_{j}) k \sum w_{k j} δ_{k}, \frac{\partial L}{\partial w _{j i}} = δ_{j} a_{i}, \frac{\partial L}{\partial b _{j}} = δ_{j}

$w_{j i}$ is the weight from unit $i$ into unit $j$ , and the sum runs over every unit $k$ that unit $j$ feeds. A unit feeding several others collects a contribution from each. For an output unit with the squared-error loss above, $δ_{k} = (a_{k} - t_{k}) φ^{'} (z_{k})$ .

ACTIVATION FUNCTIONS AND THEIR DERIVATIVES

σ (z) = \frac{1}{1 + e ^{- z}}, σ^{'} (z) = σ (z) (1 - σ (z)); \frac{d}{d z} tanh z = 1 - tanh^{2} z; ReLU (z) = max (0, z), ReLU^{'} (z) = {1, 0, z > 0 z < 0

$σ^{'} (z)$ is largest, $1/4$ , at $z = 0$ . The ReLU derivative is undefined at $z = 0$ itself; a question that lands there should state the value it uses.

SIGMOID OUTPUT WITH CROSS-ENTROPY LOSS

\overset{y}{^} = σ (z), L = - [t ln \overset{y}{^} + (1 - t) ln (1 - \overset{y}{^})] : \frac{\partial L}{\partial z} = \overset{y}{^} - t

$t$ is 0 or 1 (natural logarithm). The chain rule gives $\frac{\partial L}{\partial y ^} = \frac{y ^ - t}{y ^ ( 1 - y ^ )}$ , and multiplying by $σ^{'} (z) = \overset{y}{^} (1 - \overset{y}{^})$ cancels the denominator.