Maximum Likelihood Estimation

Setting of the problem

1. Samples ${\mathcal{X}}_i=1,\dots ,c$ are i.i.d from the density $p(x|{\omega }_i)$
2. $p(\mathbf{x}|{\omega }_i)$ is of the form $p(\mathbf{x}|{\mathbf{\theta }}_i)$, in which the parameters ${\mathbf{\theta }}_i$ are unknown, deterministic vectors. We need to estimate ${\mathbf{\theta }}_i$ from the given samples.

Solution

We have the likelihood function,

$$l(\theta )=p(\mathcal{X}|\mathbf{\theta })=p({\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N|\theta )=\prod _{i=1}^{N}p({\mathbf{x}}_i|\theta )$$

Let’s simplify this by taking the $\ln$,

$$H(\theta )=\ln l(\theta )$$

We can define the optimal $\theta$ as,

$$\hat{\mathbf{\theta }}=\arg \underset{\mathbf{\theta }}{\max }H(\mathbf{\theta })$$

Examples

For Gaussian distribution, the ML estimation for mean and variance is

$$\begin{array}{rl}\hat{\mu }& ={\hat{\theta }}_1=\frac{1}{N}\sum _{k=1}^N{x}_k\\ {\hat{\sigma }}^2& ={\hat{\theta }}_2=\frac{1}{N}\sum _{k=1}^N(x_k-\hat{\mu }{)}^2\end{array}$$

But remember that the value of $\hat{\sigma }$ here is biased. The unbiased variance estimator is,

$${\hat{\sigma }}^2=\frac{1}{N-1}\sum _{k=1}^N(x_k-\hat{\mu }{)}^2$$