Bayesian Decision Theory Framework with Costs

The setup is as follow:

1. Random feature vectors $\mathbf{x}$.
2. State space $\Omega =\{{\omega }_1,{\omega }_2,\dots ,{\omega }_c\}$, $c$ possible classes.
3. Decision space $\mathcal{A}=\{{\alpha }_1,{\alpha }_2,\dots {\alpha }_k\}$, where we have $k$ possible decisions.
4. Loss function $\lambda ({\alpha }_i,{\omega }_j)$, the cost of deciding ${\alpha }_i$ when the true state is ${\omega }_j$.

The goal is to minimize the expected loss. We define the conditional risk for one sample as:

$$R({\alpha }_i,\mathbf{x})=E[\lambda ({\alpha }_i,{\omega }_j)|x]=\sum _{j=1}^c\lambda ({\alpha }_i,{\omega }_j)P({\omega }_j|\mathbf{x})$$

Now, we need to calculate the overall risk for all the samples.

$$R(\alpha )=\int R(\alpha (\mathbf{x})|\mathbf{x})p(\mathbf{x})d\mathbf{x}$$

Note that $\alpha (\mathbf{x})$ here is a decision rule that maps each $\mathbf{x}$ to a decision. Next, to find the optimal decision rule ${\alpha }^{\ast }(\mathbf{x})$,

$${\alpha }^{\ast }=\arg \underset{\alpha (\cdot )}{\min }R(\alpha )=\arg \underset{\alpha (\cdot )}{\min }\int R(\alpha (\mathbf{x})|\mathbf{x})p(\mathbf{x})d\mathbf{x}$$

This might be hard to calculate, so we minimize the integral by minimizing the integrand at each pooint. This is called Minimal Risk Decision.

$${\alpha }^{\ast }(\mathbf{x})=\arg \underset{i=1,\dots ,k}{\min }R({\alpha }_i|\mathbf{x})\text{for each x}$$

To calculate $R({\alpha }_i|\mathbf{x})$, same as the above equation actually.

$$R({\alpha }_i|\mathbf{x})=\sum _{j=1}^c\lambda ({\alpha }_i,{\omega }_j)\frac{p(\mathbf{x}|{\omega }_j)P({\omega }_j)}{{\sum }_{i=1}^{c}p(\mathbf{x}|{\omega }_i)P({\omega }_i)}$$