Updating Probabilities Based on Repeated Observations Updating Probabilities Based on Repeated Observations
Conditional probabilities can be tricky, especially when you have repeated observations rather than just isolated ones. This problem is worth working through by hand to develop an understanding for how these things work, as it demonstrates how tricky conditional probabilities get when you introduce repetition.
The context is simple: there are 2 bowls of cookies, each with 12 cookies in them. The breakdown of cookies is as follows:
Bowl 1: 75% vanilla cookies, 25% chocolate cookies (9 vanilla, 3 chocolate)
Bowl 2: 50% vanilla cookies, 50% chocolate cookies (6 of each)
Problem #1 (easy): If you grab a cookie randomly from one of the bowls, and the cookie is vanilla, what is the probability that it came from Bowl #1?
Problem #2 (harder): If you grab three cookies from the same randomly chosen bowl, and they are (in order) vanilla, chocolate, and vanilla, what is the probability that they came from Bowl #1 if you replaced the cookie after each drawing (sampled with replacement)?
Problem #3 (hardest): If you grab three cookies from the same randomly chosen bowl, and they are (in order) vanilla, chocolate, and vanilla, what is the probability that they came from Bowl #1 if you ate each cookie after drawing it (sampled without replacement)?
Solutions below.
——————————————————
Problem #1: This is a straightforward computation. All of the needed inputs are given, so it’s as easy to solve as applying Bayes Theorem:
P(Bowl 1 | vanilla) = P(vanilla | Bowl 1)*P(Bowl 1)/P(vanilla) = (.75*.5)/.625 = .6 (Note: The .625 is calculated as P(vanilla | Bowl 1)*P(Bowl 1) + P(vanilla | Bowl 2)*P(Bowl 2) as there is no joint probability, thus P(vanilla) = .75*.5 + .5*.5 = .375 + .25 = .625)
Problem #2: This version gets trickier, because after each draw, the prior probability of drawing from either bowl has changed. This also affects the prior probability of choosing vanilla or chocolate, though this is not obvious at first.
After step 1, which we solved above, we have P(Bowl 1) = .6, therefore P(Bowl 2) = .4 [from this point on B1 = Bowl 1, B2 = Bowl 2, V = vanilla, and C = chocolate]
Thus, the posterior probability after the second selection is P(B1|C) = (P(C|B1)*P(B1))/P(C), but P(C) is no longer .375, because P(B1) no longer equals P(B2) = .5.
P(C) = P(C|B1)*P(B1) + P(C|B2)*P(B2) = .25*.6 + .5*.4 = .15+.2 = .35
Therefore: P(B1|C) = (.25*.6)/.35 = .4286
This probability then becomes the prior probability for the third trial, and we need to recompute P(V) based on the new prior (as the decimal has gotten messy, you can choose to take my word for the following, or calculate it yourself):
P(V) = P(V|B1)*P(B1) + P(V|B2)*P(B2) = .6071
and P(B1|V) = (P(V|B1)*P(B1))/P(V) = .5294
(An interesting observation is how much more information there appears to be in the second selection than in the first or third. To examine the magnitude of evidence, you can compare the logs of prior and posterior odds ratios. In this case, the initial condition gave no evidence in either direction, or odds of 1:1, so ln(1:1) = 0, showing the neutrality of the information. After the first vanilla cookies, we now had an odds ratio of 3:2, or 1.5, with a natural log of .405, an increase of .405 from the initial condition. After the chocolate cookie, we have log-odds of -.29, a loss of .69!! The next vanilla cookie only brings the log-odds back to .12, gaining another .405. In other words, every vanilla cookie is worth .405 log-odds, while every chocolate is worth -.69, so the chocolate cookie has more information in it than the vanilla from the perspective of bowl 1.)
Problem #3: This version is the hardest because the probabilities change even more. The sampling without replacement introduces additional complexity, though the mechanics of solving the problem are identical.
We’ve done step one, so P(B1) = .6 going into step two. But not, there are only 11 cookies in “either” bowl (because we don’t know which one we sampled the first cookie from). The updated probabilities are P(V) = P(V|B1)*P(B1)+P(V|B2)*P(B2) = 8/11*.6 + 5/11*.4 = .727*.6 + .455*.4 = .618, so P(C) = 1-P(V) = .382.
We have, P(B1|C) = (P(C|B1)*P(B1))/P(C) = .4286. This is actually the same probability obtained the second problem where we sampled without replacement, which is an interesting result. I’m not 100% sure of why that worked out the way it did, but it makes sense that the evidence so far (V then C) would not be affected much by the replacement since it dilutes the V:C ratio of bowl 2 as well as bowl 1.
As we move to step 3, we now have 8/10 vanilla cookies in Bowl 1, while Bowl 2 is back to 50/50. As a result, the new probabilities are:
P(V|B1) = .8
P(V) = .8*.4286 + .5 *.5714 = .6826
So P(B1|V) = .5455
We see that when sampling without replacement, the third cookie gives more evidence that it did when sampling with replacement. To be sure, let’s check to the log-odds. Since the first two steps gave the same results, the log-odds are still the same as well. +.405 then -.693. The third step, however, gives more evidence that it did previously, because the conditional probabilities P(V|Bx) have changed. The final vanilla cookies now gives +.47 log-odds of evidence instead of just .405 for the Bowl 1 hypothesis.