Bayesian Estimation

Setting

1. Samples ${\mathcal{X}}_i=1,\dots ,c$ are i.i.d from density $p(\mathbf{x}|{\omega }_i)$
2. $p(\mathbf{x}|{\omega }_i)$ is of the form $p(\mathbf{x}|{\mathbf{\theta }}_i)$ where ${\mathbf{\theta }}_i$ are random vectors with prior density of $p({\mathbf{\theta }}_i)$
   • Remember that it is different from Maximum Likelihood Estimation where the ${\mathbf{\theta }}_i$ are unknown deterministic vectors.

Solution

We need to do minimal risk estimation. Loss Function of an Estimation

$$\lambda (\hat{\mathbf{\theta }},\mathbf{\theta }),\text{e.g., }\lambda (\hat{\theta },\theta )=(\theta -\hat{\theta }{)}^2$$

Expected Risk

$$\begin{array}{rl}R& ={\int }_{E^d}{\int }_{\theta }\lambda (\hat{\mathbf{\theta }},\mathbf{\theta })p(\mathbf{x},\mathbf{\theta })d\theta d\mathbf{x}\\ & ={\int }_{E^d}{\int }_{\theta }\lambda (\hat{\mathbf{\theta }},\mathbf{\theta })p(\mathbf{\theta }|\mathbf{x})p(\mathbf{x})d\theta d\mathbf{x}\\ & \\ & ={\int }_{E^d}R(\hat{\mathbf{\theta }}|\mathbf{x})p(\mathbf{x})d\mathbf{x}\end{array}$$

Risk conditional on $\mathbf{x}$

$$R(\hat{\mathbf{\theta }}|\mathbf{x})={\int }_{\theta }\lambda (\hat{\mathbf{\theta }},\mathbf{\theta })p(\mathbf{\theta }|\mathbf{x})d\mathbf{\theta }$$

Empirical Risk on all training data

$$R(\hat{\mathbf{\theta }}|\mathcal{X})={\int }_{\theta }\lambda (\hat{\mathbf{\theta }},\mathbf{\theta })p(\mathbf{\theta }|\mathcal{X})d\mathbf{\theta }$$

Bayesian Estimation

$$\hat{\mathbf{\theta }}=\arg \underset{\theta }{\min }R(\mathbf{\theta }|\mathcal{X})$$

Example

With the loss function $\lambda (\hat{\theta },\theta )$, the optimal value of $\theta$ is,

$$\begin{array}{rl}\hat{\theta }& =E[\theta |\mathbf{x}]={\int }_{\theta }\theta p(\theta |\mathbf{x})d\theta ,\text{or}\\ & \\ \hat{\theta }& =E[\theta |\mathcal{X}]={\int }_{\theta }\theta p(\theta |\mathcal{X})d\theta \end{array}$$

But we still don’t know what is $p(\theta |\mathbf{x})$ or $p(\theta |\mathcal{X})$.

1. We set the prior $p(\theta )$
2. Write down the joint density (conditional density):

$$p(\mathcal{X}|\theta )=p(x_1,x_2,\dots ,x_N|\theta )=\prod _{i=1}^{N}p(x_i|\theta )$$

3. Calculate the posterior

$$p(\theta |\mathcal{X})=\frac{p(\mathcal{X}|\theta )p(\theta )}{{\int }_{\theta }p(\mathcal{X}|\theta )p(\theta )d\theta }$$

We do Bayesian Estimation iteratively. When we obtain a new sample,

$$\begin{array}{rl}p({\mathcal{X}}^N|\theta )& =p(x_N|\theta )p({\mathcal{X}}^{N-1}|\theta )\\ & \\ p(\theta |{\mathcal{X}}^N)& =\frac{p(x_N|\theta )p(\theta |{\mathcal{X}}^{N-1})}{{\int }_{\theta }p(x_N|\theta )p(\theta |{\mathcal{X}}^{N-1})d\theta }\end{array}$$