AI - Lecture 4

Fast Recap:

• Likelihood
• Likelihood of Conditional Probabilities “Likelihood of $p(Y|\underline{\Theta })$“
• ML (Maximum Likelihood) Estimate
• Log-Likelihood
• Gaussian ML Estimate
• Validation of Classifiers
  • Normal Method: 60/20/20
  • “Leave One Out” Method
  • “Many-Fold Crossvalidation” Method
• Supervised Learning
  • Non-Parametric Estimate
  • Relative Frequency Estimate
  • Easily Estimate a PDF

---

Recap:

First 2 Principal Components: Given a set of data we can create a Multivariate Gaussian Distribution from it, the first-X principal component are the first-X eigenvalues of the covariance matrix of the distribution.

Bayes Decision Rule: Bayes decision rules relies on the maximum of the joint-probability $p(\underline{x},w_i)=p(\underline{x}|w_i)\cdot P(w_i)$ .

Bayes Decision Rule with Discriminant Functions

$$\begin{array}{rl}& D(\underline{x})=w_i\text{iff}{g}_i(\underline{x})\ge g_j(\underline{x})\\ & \to g_i(\underline{x})=p(\underline{x}|{\omega }_i)\cdot P({\omega }_i)\\ & \to g_i(\underline{x})=\log (p(\underline{x}|{\omega }_i))+\log (P({\omega }_i))\end{array}$$

We define the Maximum Likelihood Decision as:

$$g_i(\underline{x})=\log (p(\underline{x}|w_i))+\log (P({\omega }_i))$$

In case of Gaussian assumptions, the MLD will become:

$$g_i(\underline{x})=-\frac{1}{2}(\underline{x}-\underline{{\mu }_i}{)}^T{\Sigma }_i^{-1}(\underline{x}-\underline{{\mu }_i})-\frac{1}{2}\log \left|{\Sigma }_i^{-1}\right|+\log P(w_i)$$

~Ex.: Diagonal Covariance Matrix ${\Sigma }_i={\sigma }^{2}I$ :

$$\begin{array}{rl}& g_i(\underline{x})=\frac{{\underline{{\mu }_i}}^t}{{\sigma }^2}\underline{x}-\frac{1}{2{\sigma }^2}{\underline{{\mu }_i}}^t\underline{{\mu }_i}+\log P({\omega }_i)\\ & \cdot w_i=\frac{{\underline{{\mu }_i}}^t}{{\sigma }^2}{w}_i:\text{i-esim weight}\\ & \cdot b_i=\frac{1}{2{\sigma }^2}{\underline{{\mu }_i}}^t\underline{{\mu }_i}+\log P({\omega }_i)b_i:\text{i-esim bias}\\ & g_i(\underline{x})=w_i^t\underline{x}+b_i\end{array}$$

In this particular we have that the MLD (Maximum Likelihood Decision) is linear with respect to $\underline{x}$.

Naming :

• ${\omega }_i$ : $i$ classes, for example the gender (male/female) we want to identificate.
• $w_i$ : weights
• $b_i$ : bias
• $\underline{x}$ : data, could mean data in input or training data.
• $D(\underline{x})=w_i$ : decision rule.
• $P(w_i|\underline{x})$ : probability that given the data $\underline{x}$ the corresponding class will be $w_i$ .
• $g_i(\underline{x})$ : discriminant function of class $w_i$, the Bayes decision rule can be rewritten as:

---

Original Files:

==First 2 Principal Components== : the first 2 eigen-vectors of the covariance matrix of the Gaussian distribution of the data.

---

---