Regression formulas

Average squared residual; the sum of squared errors (SSE) is the same without the factor $1/n$. Minimising either gives the same $\mathbf{w}$.

Needs at least two distinct $x$ values ($S_{xx}>0$). The fitted line always passes through $(\bar{x},\bar{y})$.

The two forms agree for least squares with an intercept on the training data, where SST = SSR + SSE; then $0\le R^2\le 1$ (assuming SST $>0$). For simple regression $R^2=r^2={\hat{\beta }}_1^2{S}_{xx}/S_{yy}$. On new (test) data, or for a model without an intercept, $1-\text{SSE}/\text{SST}$ can be negative.

The inverse form requires $X^{T}X$ to be invertible, i.e. the columns of $X$ (including the column of ones, if present) are linearly independent. This needs $n$ to be at least the number of columns. If one column is a linear combination of others (exact collinearity, or more columns than samples), $X^{T}X$ is singular: minimisers still exist, but there are infinitely many.

Sum of squared errors plus $\lambda$ times the squared Euclidean norm of the weights. $\lambda =0$ gives OLS. For $\lambda >0$ the objective is strictly convex, so the minimiser is unique. Equivalent constrained form: minimise $\Vert \mathbf{y}-X\mathbf{w}{\Vert }^2$ subject to $\Vert \mathbf{w}{\Vert }^2\le t$, for a $t$ that depends on $\lambda$.

For $\lambda >0$ the matrix $X^{T}X+\lambda I$ is always invertible (positive definite), even when $X^{T}X$ is singular. Here $I$ is the identity of size equal to the number of weights.

$\eta >0$ is the learning rate. The iterates converge from any starting point if $0<\eta <2/{\mu }_{\max }$, where ${\mu }_{\max }$ is the largest eigenvalue of $H=\frac{2}{n}{X}^{T}X$. If $\eta >2/{\mu }_{\max }$ they generally diverge.