How Bayesian inference Works

Bayesian inference begins with a prior probability distribution, which encodes your initial beliefs about all possible states of the world before collecting new evidence. For example, if you're trying to determine whether a patient has a rare disease that affects 1 in 10,000 people, your prior probability would be 0.0001 for "has disease" and 0.9999 for "doesn't have disease." This prior can come from historical data, expert knowledge, previous experiments, or even an assumption of equal likelihood when you have no information at all.

The prior is not a guess in the casual sense—it's a formal probability distribution that assigns a likelihood value to every possible hypothesis you're considering. If you're estimating an unknown parameter like the bias of a coin, your prior might be a bell curve centered at 0.5 (fair coin) with some spread reflecting your uncertainty. The shape and values of this distribution directly influence how strongly new evidence will shift your conclusions.

Critics sometimes view priors as subjective or arbitrary, but they serve a crucial role: they make your assumptions explicit and quantifiable rather than hidden. Even seemingly "neutral" statistical methods contain implicit assumptions; Bayesian inference simply requires you to state yours upfront. As more evidence accumulates, the influence of the prior typically diminishes, and the data increasingly dominates the final result.

The observation phase involves collecting data that bears on the question you're investigating. This might be a medical test result, a series of coin flips, sensor readings, survey responses, or any measurement that provides information about the unknown quantity. The key is that this evidence was not available when you formulated your prior belief—it represents genuinely new information.

For each possible hypothesis, you need to know how likely the observed data would be if that hypothesis were true. This is called the likelihood function. If a medical test is 95% accurate and you're testing for a disease, the likelihood tells you the probability of getting a positive test result given the patient actually has the disease, and separately, the probability of a positive result if they don't have it.

The likelihood is not the same as the probability that the hypothesis is true—that's what you're trying to calculate. Instead, it works in the opposite direction: it assumes each hypothesis in turn and asks how expected or surprising the observed data would be under that assumption. A hypothesis that makes the observed data highly probable has a high likelihood; one that makes the data seem improbable has a low likelihood.

The mathematical heart of Bayesian inference is the multiplication of prior probability and likelihood for each hypothesis. If a hypothesis had high prior probability and the observed data is also highly likely under that hypothesis, the product will be large. If either the prior is low or the data is unexpected under that hypothesis, the product will be small. This multiplication embodies how evidence and prior knowledge combine.

This operation follows directly from the definition of conditional probability and captures an intuitive principle: you should believe hypotheses that were already plausible and that also explain the data well. Consider the disease example—even if a test is highly accurate (high likelihood), a positive result for an extremely rare disease doesn't immediately mean you have it, because the prior probability was so low. The multiplication ensures both factors matter.

At this stage, you have what's called an unnormalized posterior—a value for each hypothesis that reflects its relative plausibility given both prior knowledge and new evidence. However, these values don't yet form a proper probability distribution because they don't necessarily sum to one. They tell you which hypotheses are more or less plausible relative to each other, but not yet the actual probability of each.

Normalization converts the unnormalized products into proper probabilities by dividing each by the total probability of observing the evidence. This total, called the marginal likelihood or evidence, is calculated by summing (or integrating) the prior-times-likelihood products across all possible hypotheses. It represents how probable the observed data is overall, averaging across all the ways it could have arisen.

Mathematically, this is the denominator in Bayes' theorem: P(hypothesis|data) = P(data|hypothesis) × P(hypothesis) / P(data). That P(data) term ensures the posterior probabilities across all hypotheses sum to exactly one, satisfying the requirement that probabilities must be valid. Without this step, you'd have relative weights but not true probabilities.

In practice, calculating the normalizing constant can be the most computationally difficult part of Bayesian inference, especially when dealing with continuous parameters or complex models. For simple problems with a few discrete hypotheses, it's just addition. For sophisticated applications, researchers use advanced algorithms like Markov Chain Monte Carlo to approximate the normalization when exact calculation is impossible.

Once you've calculated the posterior probability distribution, it represents your updated belief after incorporating the new evidence. This posterior quantifies what you now believe about each hypothesis, combining everything you knew before with what the data tells you. The disease patient now has a revised probability of being sick; the coin's bias now has a refined probability distribution.

The elegant feature of Bayesian inference is that this posterior can immediately become the prior for the next round of evidence. If you get a second test result, you don't start over from scratch—you begin with your current beliefs (the previous posterior) and update again. This creates a sequential learning process where each observation refines your understanding incrementally, with all previous evidence automatically incorporated.

This iterative updating means Bayesian inference is naturally suited to ongoing learning and real-time decision-making. As evidence accumulates, the posterior distribution typically becomes narrower and more concentrated around the true value, reflecting increasing certainty. The method provides a principled pathway from ignorance to knowledge, where each new observation moves you closer to understanding reality.