University AI - ML Estimations of Mixture Densities

ML Estimations of Mixture Densities

Given the unlabeled data $\tau =\{\underline{x_1},\dots ,\underline{x_n}\}$ iid (independent and identically distributed) according to an unknown pdf $p(\underline{x})$, we assume a parametric model $p(\underline{x}|\underline{\Theta })$ of $p(\underline{x})$ and we search for an estimate of $\underline{\Theta }=({\theta }_1,\dots ,{\theta }_p)$ that respect the nature of the data.

~Ex.: We say that the pdf we want to try is a linear combination of 5 Gaussian PDFs, then we will search $\underline{\Theta }=\{(\underline{{\mu }_1},{\sigma }_1^2),\dots ,(\underline{{\mu }_5},{\sigma }_5^2)\}$ such that our assumption will be as close as possible to the real pdf (we minimize the cost)

Assume now that:

• The number of classes $c$ is known.
• $P({\omega }_i)$ are all estimated.
• The forms of $p(\underline{x}|{\omega }_j,\underline{{\Theta }_j})$ are all known.
• $p(\underline{x}|\underline{\Theta })$ is identifiable, that means that $\underline{\Theta }$ is unique for this pdf: $\underline{\Theta }\ne \hat{\underline{\Theta }}\Rightarrow \exists \underline{x}\in \mathbb{X}:p(\underline{x}|\underline{\Theta })\ne p(\underline{x}|\hat{\underline{\Theta }})$
• $\underline{{\Theta }_i}$ is functionally independent of $\underline{{\Theta }_i}$ , for $i\ne j$

Then we proceed with ML Estimation.

ML (Maximum Likelihood) Estimation

Assume:

$$p(\underline{x}|\underline{\Theta })=\sum _{j=1}^{c}P({\omega }_j)p(\underline{x}|{\omega }_j,\underline{{\Theta }_j})$$

We define the Likelihood as:

$$p(\tau |\underline{\Theta })=\prod _{k=1}^{n}p(\underline{x_k}|\underline{\Theta })$$

Assuming that $p(\underline{x}|\underline{\Theta })$ is differentiable we have:

$${\nabla }_{\underline{{\Theta }_i}}l=\sum _{k=1}^n\frac{1}{p(\underline{x_k}|\underline{\Theta })}\cdot {\nabla }_{\underline{{\Theta }_i}}\left\{\sum _{j=1}^{c}P({\omega }_j)p(\underline{x}|{\omega }_j,\underline{{\Theta }_j})\right\}$$

We need to have the following constraint, respected:

$${\nabla }_{\underline{{\Theta }_i}}l=0$$

To do this $\underline{\hat{\Theta }}=\{\underline{{\hat{\Theta }}_1},\dots ,\underline{{\hat{\Theta }}_c}\}$ needs to respect the following constraint:

$$\sum _{k=1}^{n}P({\omega }_i|\underline{x_k},\underline{\hat{\Theta }})\cdot {\nabla }_{\underline{{\hat{\Theta }}_i}}\log \left\{p(\underline{x}|{\omega }_i,\underline{{\hat{\Theta }}_i})\right\}=0\forall i=1,\dots ,c$$

---

Original Files

---