Bayes’ Theorem

Bayes' Theorem takes three inputs — a prior belief, the likelihood of observed evidence under that belief, and the overall probability of the evidence — and returns a posterior belief that precisely quantifies how the evidence should shift one's confidence. Its beauty lies in its simplicity and universality. Whether applied to medical diagnosis, criminal forensics, machine learning, or the search for a lost submarine, the same equation governs rational belief revision.

It is not merely a tool within statistics; it is, as E. T. Jaynes argued, the unique extension of logic to situations of uncertainty.

Each component carries distinct meaning. The prior P(H) encodes everything known about the hypothesis before the current evidence arrives — previous experiments, expert judgment, physical constraints, or formal symmetry arguments. The likelihood P(E | H) measures how well the hypothesis predicts the specific evidence observed. The evidence P(E) normalizes the result, ensuring the posterior is a valid probability distribution. And the posterior P(H | E) is the complete output: a new probability that synthesizes prior knowledge with observed data.

Historical Origins

The theorem's history is entwined with the problem of "inverse probability" — reasoning backward from observed effects to probable causes. While direct probability is straightforward (given a fair coin, what is the probability of seven heads in ten flips?), inverse probability is subtle (given seven heads in ten flips, what can we conclude about the coin's fairness?).

The Mechanics of Updating

What makes Bayes' Theorem so powerful is its iterative nature. The posterior from one round of evidence becomes the prior for the next, creating a principled mechanism for sequential learning. With each new observation, beliefs are refined — sharpened when evidence is consistent, and pulled in new directions when it is surprising.

The odds form is especially intuitive. The likelihood ratio — how much more probable the evidence is under the hypothesis than under its negation — quantifies the diagnostic strength of the evidence. Evidence equally likely under both hypotheses (likelihood ratio = 1) does not shift the odds. Evidence ten times more likely under H multiplies the odds by ten.

Taking logarithms converts multiplication to addition. Each piece of evidence contributes a "weight of evidence" measured in units that Alan Turing called bans and decibans during his cryptanalytic work at Bletchley Park — one of the earliest and most consequential applications of sequential Bayesian reasoning.

The Normalizing Constant

The denominator P(E) — the marginal likelihood or evidence — is obtained by summing (or integrating) the numerator over all hypotheses:

For simple problems this integral is easy. For complex models — those with high-dimensional parameter spaces, hierarchical structure, or non-conjugate priors — computing P(E) is the central computational challenge of Bayesian statistics. The development of MCMC, variational inference, nested sampling, and other approximation methods has been driven largely by the need to evaluate or circumvent this integral.

When comparing two models, the marginal likelihood takes on a starring role. The Bayes factor — the ratio of marginal likelihoods — measures the evidence one model provides over another. Because a complex model must spread its prior predictions over a wider range of possible data, it pays an automatic complexity penalty: the Bayesian Occam's razor.

A Worked Example: Medical Diagnosis

A screening test for a disease has 99% sensitivity and 99% specificity. The disease affects 1 in 1,000 people. A patient tests positive. What is the probability they have the disease?

Despite 99% accuracy in both directions, a positive result means only a 9% chance of disease. The low base rate (1 in 1,000) means false positives from the healthy population vastly outnumber true positives. This result — which surprises most people, including many physicians — is a direct consequence of Bayes' Theorem.

Applications

Spam Filtering

Paul Graham's 2002 essay "A Plan for Spam" introduced Bayesian spam filtering, computing P(spam | words) for each email. The Naive Bayes classifier proved remarkably effective, establishing Bayesian reasoning as a practical tool in consumer technology.

Search and Rescue

Bayesian search theory maintains a posterior probability map over possible target locations. After each search pass, the map is updated based on detection probabilities and areas covered. This approach was used to locate the USS Scorpion (1968), Steve Fossett's crash site (2007), and Air France Flight 447's black boxes (2011).

Machine Learning

Bayesian methods underpin Gaussian processes, Bayesian neural networks, Bayesian optimization, and latent Dirichlet allocation. The posterior provides not only predictions but calibrated confidence estimates — critical for safety-sensitive applications.

Forensic Evidence

DNA evidence and fingerprint evaluation are fundamentally Bayesian problems. The likelihood ratio is the correct measure of evidential strength. Confusing the likelihood with the posterior produces the "prosecutor's fallacy," a documented source of wrongful convictions.

"The theory of probabilities is at bottom nothing but common sense reduced to calculus; it enables us to appreciate with exactness that which accurate minds feel with a sort of instinct for which they are often unable to account." — Pierre-Simon Laplace, Théorie analytique des probabilités (1812)

Philosophical Significance

Bayes' Theorem is a straightforward consequence of the axioms of probability — it is not in dispute as a mathematical identity. What is debated is whether probability should be interpreted as degree of belief (the Bayesian view) or as long-run frequency (the frequentist view).

Cox's theorem (1946) showed that any consistent system for reasoning under uncertainty must be isomorphic to probability theory. De Finetti's representation theorem showed that exchangeability implies a latent parameter with a prior distribution. Together, they suggest Bayesian updating is not one method among many — it is the unique rational method.

These results do not settle all debates. The choice of prior remains contested, and there are settings where Bayesian methods perform poorly. But as a framework for coherent reasoning under uncertainty, Bayes' Theorem has no rival.

Example

You're a college student and you've been feeling under the weather. You have a sore throat. Two illnesses are on your mind: a common cold, which 20% of students currently have, and strep throat, which only 2% of students have. You know that 40% of people with a cold get a sore throat, while 90% of people with strep throat get one. The question is: given that you have a sore throat, how likely is it that you have strep vs. a cold?

Setting Up the Problem

First, identify what you know. These are the priors — the base rates of each illness before considering the sore throat evidence:

Next, the likelihoods — how probable a sore throat is under each hypothesis:

Computing P(Sore Throat)

Before we can apply Bayes' Theorem, we need the total probability of having a sore throat — regardless of cause. This is the evidence term, computed by summing over all possibilities:

So about 11.4% of students are walking around with a sore throat right now.

Applying Bayes' Theorem

Now we update each hypothesis:

What the Numbers Tell Us

Before noticing the sore throat, you'd have put only a 2% chance on strep. After, that jumps to nearly 16% — an eightfold increase. The cold, already common, becomes the dominant explanation at 70%. The sore throat was evidence, and Bayes' Theorem told you exactly how much to update each belief.

If you then took a rapid strep test and it came back positive, you could run Bayes' Theorem again using today's posteriors as tomorrow's priors. Each new piece of evidence refines the picture — that's the iterative power of Bayesian updating.