AI - Lecture 5

Fast Recap:

---

Recap:

Likelihood : Given $Y=\{\underline{y_1}\dots ,\underline{y_n}\}$ a set of data that is given by the distribution $p(\underline{y}|{\omega }_i)$, given that they are all given by the same distribution we will say that they are identically distributed, suppose also that they independent* between each other. ⇒ $y_i$ are iid (independent and identically distributed).

Due to the independent assumption we can say that:

$$p(Y|\underline{\Theta })=\prod _{k=1}^{n}p(y_k|\underline{\Theta })$$

this is called the likelihood of $\underline{\Theta }$ given $Y$, this is a function of $\Theta$:

We call the Maximum Likelihood (ML) Estimate $\underline{\hat{\Theta }}$ the one which maximizes $p(Y|\Theta )$.

Log-Likelihood : Where:

• ${\nabla }_{\underline{\Theta }}l(\hat{\underline{\Theta }})$ is the gradient of the log-likelihood function $l(\hat{\underline{\Theta }})$ with respect to $\Theta$

~Example : If we know that $p(Y|\Theta )$ is a Gaussian Distribution the maximum likelihood of the mean and variance are respectively the sample-mean and biased sample variance.

Sample Mean : $\frac{1}{n}{\sum }_{k=1}^n\underline{y_k}$ Biased Sample Variance : $\frac{1}{n}{\sum }_{k=1}^n(\underline{y_k}-\hat{\mu }{)}^2$

(Bonus) Unbiased Sample Variance* : $\frac{1}{n-1}{\sum }_{k=1}^n(\underline{y_k}-\hat{\mu }{)}^2$

Naming :

• $\mu ,\underline{\mu }$ : mean and vector mean
• ${\sigma }^2,\Sigma$ : variance and covariance matrix
• $\underline{\Theta }$ : parameter vector, for example: $\underline{\Theta }=(\underline{\mu },\Sigma )$
• ${\omega }_i$ : $i$ classes, for example the gender (male/female) we want to identificate.
• $w_i$ : weight
• $b_i$ : bias
• $\underline{x},\underline{y}$ : data, could mean data in input or training data.
• $c$ : number of samples used as the training set
• $P(w_i|\underline{x})$ : probability that given the data $\underline{x}$ belongs to/is identified as the class ${\omega }_i$ .
• $g_i(\underline{x})$ : discriminant function of class $w_i$, usually it is defined as: $g_i(x)=\log p(\underline{x}|{\omega }_i)+\log P({\omega }_i)$ $g_i(\underline{x})=w_i^t\underline{x}+b_i$ (in the linear case)
• $D(\underline{x})=w_i$ : decision rule, a simple decision rule could be: $D(\underline{x})={\omega }_i\text{iff}{g}_i(\underline{x})\ge g_j(\underline{x})$

NOTE: weights are named with $w$, while classes with $\omega$, it may cause confusion

---

Original Files:

Where:

• ${\nabla }_{\underline{\Theta }}l(\hat{\underline{\Theta }})$ is the gradient of the log-likelihood function $l(\hat{\underline{\Theta }})$ with respect to $\Theta$