University AI - ML (Maximum Likelihood) Estimator

Maximum Likelihood

Given $Y=\{\underline{y_1}\dots ,\underline{y_n}\}$ a set of data that is given by the distribution $p(\underline{y}|{\omega }_i)$, given that they are all given by the same distribution we will say that they are identically distributed, suppose also that they independent between each other. ⇒ $y_i$ are iid (independent and identically distributed).

Due to the independent assumption we can say that:

$$p(Y|\underline{\Theta })=\prod _{k=1}^{n}p(y_k|\underline{\Theta })$$

this is called the likelihood of $\underline{\Theta }$ given $Y$, this is a function of $\Theta$:

We call the Maximum Likelihood (ML) Estimate $\underline{\hat{\Theta }}$ the one which maximizes $p(Y|\Theta )$.

Log-Likelihood : Where:

• ${\nabla }_{\underline{\Theta }}l(\hat{\underline{\Theta }})$ is the gradient of the log-likelihood function $l(\hat{\underline{\Theta }})$ with respect to $\Theta$

~Example : If we know that $p(Y|\Theta )$ is a Gaussian Distribution the maximum likelihood of the mean and variance are respectively the sample-mean and biased sample variance.

Sample Mean : $\frac{1}{n}{\sum }_{k=1}^n\underline{y_k}$ Biased Sample Variance : $\frac{1}{n}{\sum }_{k=1}^n(\underline{y_k}-\hat{\mu }{)}^2$

(Bonus) Unbiased Sample Variance* : $\frac{1}{n-1}{\sum }_{k=1}^n(\underline{y_k}-\hat{\mu }{)}^2$

---

Original Files:

Where:

• ${\nabla }_{\underline{\Theta }}l(\hat{\underline{\Theta }})$ is the gradient of the log-likelihood function $l(\hat{\underline{\Theta }})$ with respect to $\Theta$