Bayes’ Theorem: idea, importance, and how to use it

What is Bayes’ Theorem?

Bayes’ Theorem can be interpreted in several ways, but it requires us to introduce more definitions and terminologies, depending on the applications. Since our goal is to keep things simple and intuitive, we say that Bayes’ Theorem is a formula for inverting conditional probability:

$$P[A|B] = \frac{P[B|A]\cdot P[A]}{P[B]}$$

Think about the following examples:

1. Suppose a company survey found that 90% of people wearing hats are men. Is it therefore reasonable to assume that 90% of men wear hats? Or
2. Suppose, in a blood test study, 90% of cancer patients got a positive result in some blood test. If you take the test and get a positive result, does that mean there’s a 90% chance you have cancer? Or this,
3. Suppose on the media you notice that 90% of terrorist acts are committed by people from a certain religious background. If you see someone from that religion, is there a 90% chance they’re a terrorist?

Intuitively, we know the answer to all three questions is NO. If we take one step further and rephrase the questions using probability notations.

1. Given \(P[\mbox{ man } | \mbox{ wearing-hat }] = 90\%\), is it true that $$P[\mbox{ wearing-hat } | \mbox{ man } ] = 90\%?$$
2. Given \(P[ \mbox{ positive blood result } | \mbox { cancer patients }] = 90\%\), is it true that $$P[ \mbox { cancer patients } | \mbox{ positive blood result } ] = 90\%?$$
3. Given \(P[\mbox{ certain religion } | \mbox{ terrorists } ] = 90\%\), is it true that $$P[\mbox{ terrorists } | \mbox{ certain religion } ] = 90\%?$$

From Bayes’ Theorem, we know we cannot just reverse the probability. Yet, we still do not see why Bayes’ theorem is so important.

Why is Bayes’ Theorem so important?

Notice that in all three examples, we are trying to find some true underlying probability, which we cannot compute directly, based on the evidence or data we can observe. Bayes’ Theorem provides us with a pathway to that true underlying probability.

In practice, it helps people make decisions when there is uncertainty such as incomplete or uncertain data, such as clinical trials. It helps people make predictions by providing a way to update probabilities based on new or additional evidence. It’s used in many fields, including finance, machine learning, and medicine. 

Different Forms of Bayes’ Formula

The formula $$P[A|B] = \frac{P[B|A]\cdot P[A]}{P[B]}$$ can also be rewritten as follows: $$P[A|B] =\frac{P[B|A]\cdot P[B]}{P[B|A]\cdot P[A]+P[B|A’]\cdot P[A’]}$$

Extended Form

If the sample space is partitioned by a set of events \(A_1, A_2, \cdots, A_n\), meaning, $$ \{\mbox{sample space} \}= A_1 \cup A_2 \cup \cdots \cup A_n \mbox{ and } A_i \cap A_j = \{\mbox{empty set}\},$$ for any \(i, j = 1, 2, \cdots, n\), then the Bayes’ formula says, for any \(i\) from \(1, 2, \cdots, n\), the conditional probability can be obtained by $$\begin{aligned}
P[A_i|B] &= \frac{P[B|A_i] \cdot P[A_i]}{P[B]}\\
&=\frac{P[B|A_i] \cdot P[A_i]}{P[B|A_1] \cdot P[A_1] + P[B|A_2] \cdot P[A_2] + \cdots + P[B|A_n] \cdot P[A_n]}
\end{aligned}$$

Examples

We will continue using the hat-wearing example. In addition, we will have some examples from the Acturay Probability Exam Sample Questions.

Example 1

Let’s go back to our first toy example. Suppose a company survey found that 90% of people wearing hats are men. Furthermore, assume that hat-wearers are 25% of the population, and men are 50% of the population. Find the probability that someone will be wearing a hat if they’re a man.

Solution

From the given information, we have

• \(P[A] = P[ \mbox{man} ] = 0.5\)
• \(P[B] = P[\mbox{wearing-hat}] = 0.25 \)
• \(P[A|B] = P[\mbox{man } | \mbox{ wearing-hat} ] = 0.9 \)

Now, Bayes’ theorem says that \(P[B|A] = \frac{P[A|B] \cdot P[B] }{ P[A]}\). Therefore, $$P[B|A] = \frac{0.9 \times 0.25}{0.5} = 0.45$$ So there’s a 45% chance that someone will be wearing a hat if they’re a man.

Remark

Notice that the 45% is a long way from the 90% that a naive approach might suggest. It is also an example why we should avoid using stereotypical assumption about things or people because we may easily be very wrong.

Example 2

A blood test indicates the presence of a particular disease 95% of the time when the disease is actually present. The same test indicates the presence of the disease 0.5% of the time when the disease is not actually present. One percent of the population actually has the disease.

Calculate the probability that a person actually has the disease given that the test indicates the presence of the disease. 

Solution

Let \(Y\) = positive test result and \(D\) = disease is present. Then, we are given that

• \(P[Y|D] = 0.95\)
• \(P[D] = 0.01\) and thus, \(P[D’] = 0.99\)
• \(P[Y|D’] = 0.05\)

The probability, that a person actually has the disease given that the test indicates the presence of the disease, is obtained by
$$\begin{aligned}
P[D|Y] &= \frac{P[Y|D]\cdot P[D]}{P[Y|D]\cdot P[D] + P[Y|D’]\cdot P[D’]} \\
&= \frac{(0.95)(0.01)}{(0.95)(0.01)+(0.005)(0.99)}\\
&=0.657
\end{aligned}$$

More Examples

More examples selected from the Acturay Probability Exam are in our next post.