AI - Lecture 17

Fast Recap:

---

Recap:

May we use MLP as an estimate of the class-conditional PDFs? Since the output of an MLP can be seen as:

$$y_i(\underline{x})\simeq P({\omega }_i|\underline{x})=\frac{p(\underline{x}|{\omega }_i)P({\omega }_i)}{p(\underline{x})}$$

We can write:

$$\frac{y_i(\underline{x})}{P({\omega }_i)}\simeq \frac{p(\underline{x}|{\omega }_i)}{p(\underline{x})}$$

Which is knows as scaled likelihood.

• $p(\underline{x})$ is unknown, but can be estimated.
• Also $p(\underline{x})$ estimates are more robust than $p(\underline{x}|{\omega }_i)$ estimates, because we need to estimate only one estimate instead of $c$ other PDFs ($c$ : number of classes, ${\omega }_1,\dots ,{\omega }_c$ ), also with the same logic if we estimate only $p(\underline{x})$ we will have $c$ times more data.
• Also if $P({\omega }_i)$ changes over time (let’s say it assumes the new value $P^{\prime }({\omega }_i)$) , we can just reuse the same MLP, so no re-training necessary and use the following formula:

$$\begin{array}{rl}{P}^{\prime }({\omega }_i|\underline{x})& =\\ & =\frac{p(\underline{x}|{\omega }_i)}{p(\underline{x})}{P}^{\prime }({\omega }_i)\\ & \simeq \frac{y_i(\underline{x})}{P({\omega }_i)}{P}^{\prime }({\omega }_i)\end{array}$$

RBF (Radial Basis Function) Networks: A generalized linear discriminant

• All weights between the input layer and the first hidden layer are equal to $1$.
• There could be a bias terms: $b_i$.
• The RB Function (Radial Basis Function), or kernel is defined as:

$$\phi (\underline{x})=e^{-\frac{\Vert \underline{x}-\underline{{\mu }_k}\Vert }{2{\sigma }_k^2}}$$

• A simple RBF Network with just 1-hidden layer will have this form:

$$y_i=\sum _{j=1}^k{w}_{ij}\phi (\underline{x})+b_i$$

• RB Functions realize a mixture of Gaussian PDFs, hence they are particularly suitable for pdf estimation.
• Like MLPs, RBF Networks are “universal” approximators.

For the learning part, it’s supervised

$$C(\tau ,w)=\frac{1}{2}\sum _i(\widehat{y_i}-y_i{)}^2$$

And we usually consider 2 approaches:

1. Via gradient descent over $C(w)$, we learn the parameters: $w_{ij}$, $b_i$, $\underline{{\mu }_k}$ and ${\sigma }_k$ .
2. $\underline{{\mu }_k}$ and ${\sigma }_k$ are estimated statistically, then the other parameters $w_{ij}$ and $b_i$ are estimated via linear algebra methods (such as matrix inversion), or via the precedent method gradient descent.

NOTE: With RBF Networks we can apply gradient-ASCENT over ML (Maximum Likelihood) method in order to estimate PDFs.

The ML method only works if the weights between the last hidden layer and the output layer sum up to $1$.

This can’t be done in MLPs because the constraint $\int p(x)dx=1$ is violated, since they realize MLPs realize mixtures of activation functions that are not inherently pdfs.

---

Original Files: