Ensemble Learning

Building a very good classifier is hard, but building a weak one is easy (just need to be slightly better than pure chance). So, can we combine several weak classifiers to build a better one?

We can perhaps, take the weighted average, and then do activation function?

Let’s say we have classifiers $h_1(x),\dots ,h_m(x)$, then the weighted average becomes

$$H(x)={\alpha }_1{h}_1(x)+\cdots +{\alpha }_m{h}_m(x)$$

where ${\alpha }_j$ is the voting weight assigned to classifier $h_j(\cdot )$. Then the final prediction is $y(x)=\text{sign}H(x)$.

Some ensemble learning examples:

• Ada Boost
• XGBoost