ML - Lecture 5

Normal Equations

• Linear prediction in multi-dimensional spaces
• Normal equations and projections
• Different cases and pseudo-inversion

---

Polynomial Approximation:

In case of a polynomial function:

$$f=w_2{x}^2+w_{1}xb$$

we can see it as a liner multi-variable function:

$$f=w_2{x}_2+w_1{x}_1+b$$

---

Normal Equation - Solutions

To solve a normal equation we will need more points than unknown variables, which means having enough data to learn the weights.

Given the matrix $\hat{X}$ : “example” matrix, or data matrix, we put in each row one observation, plus the bias input (usually 1). The number of columns correspond to the total number of observation we took.

• More rows = More parameters = More weights to learn but a more “complete” solution, i.e. where the edge cases of the problem are taken into account
• More columns = More data = More precise weights

Number of columns $\gg$ Number of rows

Given:

• $\hat{X}$ : information matrix
• ${\hat{W}}^{\ast }$ : “perfect” weight matrix: such that $({\hat{X}}^T\hat{X}){\hat{W}}^{\ast }=\hat{X}Y$
• $Y$ : target Matrix

We have that: $\mathrm{rank}(\hat{X})=r\le d$ where $d$ is the maximum rank And we can say that:

$$({\hat{X}}^T\hat{X}){\hat{W}}^{\ast }=\hat{X}Y$$

So:

$${\hat{W}}^{\ast }=({\hat{X}}^T\hat{X}{)}^{-1}{\hat{X}}^{T}Y$$

---