Simple, multiple and ridge regression

Average squared residual; the sum of squared errors (SSE) is the same without the factor $1/n$. Minimising either gives the same $\mathbf{w}$.

Needs at least two distinct $x$ values ($S_{xx}>0$). The fitted line always passes through $(\bar{x},\bar{y})$.

The two forms agree for least squares with an intercept on the training data, where SST = SSR + SSE; then $0\le R^2\le 1$ (assuming SST $>0$). For simple regression $R^2=r^2={\hat{\beta }}_1^2{S}_{xx}/S_{yy}$. On new (test) data, or for a model without an intercept, $1-\text{SSE}/\text{SST}$ can be negative.

The inverse form requires $X^{T}X$ to be invertible, i.e. the columns of $X$ (including the column of ones, if present) are linearly independent. This needs $n$ to be at least the number of columns. If one column is a linear combination of others (exact collinearity, or more columns than samples), $X^{T}X$ is singular: minimisers still exist, but there are infinitely many.

$\hat{\mathbf{y}}$ is the orthogonal projection of $\mathbf{y}$ onto the column space of $X$; the residual vector is orthogonal to every column of $X$. With a column of ones, this is why the residuals sum to zero.

Assumes the model $\mathbf{y}=X\mathbf{w}+\mathbf{\epsilon }$ is correct, $X$ is fixed, and the noise has mean zero, variance ${\sigma }^2$ and is uncorrelated across samples. $p$ = number of features excluding the bias, so $n-p-1$ is $n$ minus the number of columns of $X$. For simple regression, $\mathrm{Var}({\hat{\beta }}_1)={\sigma }^2/S_{xx}$.

$p$ = number of features excluding the bias; requires $n>p+1$. On the training data $R^2$ never decreases when a feature is added; adjusted $R^2$ can decrease.

Sum of squared errors plus $\lambda$ times the squared Euclidean norm of the weights. $\lambda =0$ gives OLS. For $\lambda >0$ the objective is strictly convex, so the minimiser is unique. Equivalent constrained form: minimise $\Vert \mathbf{y}-X\mathbf{w}{\Vert }^2$ subject to $\Vert \mathbf{w}{\Vert }^2\le t$, for a $t$ that depends on $\lambda$.

For $\lambda >0$ the matrix $X^{T}X+\lambda I$ is always invertible (positive definite), even when $X^{T}X$ is singular. Here $I$ is the identity of size equal to the number of weights.

$\tilde{X}$ is the $n\times d$ matrix of feature columns (no column of ones) with each column's mean subtracted; $\bar{\mathbf{x}}$ is the vector of feature means, and $I$ is $d\times d$. Centring $\mathbf{y}$ as well gives the same $\hat{\mathbf{w}}$, because ${\tilde{X}}^{T}1=0$.

Every weight is shrunk by the same factor $1/(1+\lambda )$. This is the case ${\mu }_i=1$ of the next formula.

Here $X^{T}X=V\mathrm{diag}({\mu }_i)V^T$ is its eigendecomposition, and $X^{T}X$ must be invertible for ${\hat{\mathbf{w}}}_{\text{OLS}}$ to exist. Directions with small eigenvalues ${\mu }_i$ (low-variance, near-collinear directions) are shrunk the most. For finite $\lambda$, ridge shrinks weights towards zero but does not in general make any of them exactly zero; an L1 (lasso) penalty can.

A Gaussian prior on the weights gives ridge. A tighter prior (smaller ${\tau }^2$) means a larger $\lambda$ and more shrinkage.

$\eta >0$ is the learning rate. The iterates converge from any starting point if $0<\eta <2/{\mu }_{\max }$, where ${\mu }_{\max }$ is the largest eigenvalue of $H=\frac{2}{n}{X}^{T}X$. If $\eta >2/{\mu }_{\max }$ they generally diverge.

For the objective $\Vert \mathbf{y}-X\mathbf{w}{\Vert }^2+\lambda \Vert \mathbf{w}{\Vert }^2$ (sum, not mean, of squared errors). The extra term $2\lambda \mathbf{w}$ pulls every weight towards zero at each step. The Hessian is $2(X^{T}X+\lambda I)$, so the step-size bound is $\eta <2/{\mu }_{\max }$ of that matrix.