Bayesian Updating

In this minisode Kyle and Linh Da discuss the meaning of Bayesian Updating.

The root concept is Bayes Theorem:

Pr(A|B) = \dfrac{Pr(B|A) \cdot Pr(A)}{Pr(B)} Although a more helpful way of thinking about this alongside our discussion is to replace A with H for Hypothesis, and B with E for Evidence. Pr(H|E) = \dfrac{Pr(E|H) \cdot Pr(H)}{Pr(E)}

In this case Pr(H) would represent your \emph{prior} probability / belief representing the likelihood that the hypothesis is true. Given new evidence E, you can evaluate how likely that hypothesis was to generate that Evidence (i.e. Pr(E|H)).

In our farmers market example, we had...

  • H_P - Box contains 100\% (P)omegranates
  • H_M - Box contains (M)ix of 50\% pomegranates and 50\% lemons
  • H_L - Box contains 100\% (L)emons

We established our prior beliefs as H_i = \frac{1}{3} for i=0,1,2.

When the farm pulls out one example fruit which turns out to be a pomegranate, Linh Da points out that ``it's not the bad of just lemons'' (a.k.a. Pr(H_L) = 0). But that doesn't mean the other hypothesis are not 50 / 50! What are they?

Pr(H_P|E) = \dfrac{Pr(E|H_P) \cdot Pr(H_P)}{Pr(E)}
= \dfrac{1.0 \cdot \frac{1}{3}}{.50} = \frac{2}{3} = 66.\overline{6}\%

Pr(H_M|E) = \dfrac{Pr(E|H_M) \cdot Pr(H_M)}{Pr(E)}
= \dfrac{0.50 \cdot \frac{1}{3}}{.50} = \frac{1}{3} = 33.\overline{3}\%

If you're having a hard time understanding why Pr(E) = .50, it's because this is your expectation of the evidence you got independent of which hypothesis turns out to be true. Thus, it's based on your prior (original) belief over each hypothesis and their likelihood of producing the observation. In other words...

Pr(E) &= Pr(E|H_P) \cdot Pr(H_P) + Pr(E|H_M) \cdot Pr(H_M) + Pr(E|H_L) \cdot Pr(H_L) \\
 &= 1.0 \cdot \frac{1}{3} + 0.50 \cdot \frac{1}{3} + 0.0 \cdot \frac{1}{3} \\
 &= 0.5
Enjoy this post? Sign up for our mailing list and don't miss any updates.