The reference class problem arises whenever we attempt to assign a probability to a single event by treating it as a member of a class of similar events. A 55-year-old man asks: "What is the probability I will develop heart disease in the next ten years?" The answer depends on the class to which we assign him. Among all men? Among 55-year-old men? Among 55-year-old men with his blood pressure, cholesterol level, exercise habits, family history, and genetic markers? Each reference class yields a different probability, and there is no obvious criterion for selecting the "correct" one.
This problem is not a mere technicality — it strikes at the foundations of probability. For frequentists, probability is the limiting relative frequency within a reference class, so the choice of class determines the probability. For Bayesians, the problem recurs in the choice of prior: which prior to assign to a single-case event depends on what one considers relevantly similar. The reference class problem thus haunts both major interpretations of probability.
≠ P(heart disease | male, age 55, smoker)
≠ P(heart disease | male, age 55, smoker, family history, ...)
Each conditioning narrows the reference class and changes the probability.
Historical Origins
The problem was identified in the 19th century by John Venn, who recognized that any particular event can be classified in multiple ways, each yielding a different frequency. Hans Reichenbach addressed the problem in his 1949 The Theory of Probability, proposing the "principle of the narrowest reference class" — use the most specific class for which reliable statistics are available. But this principle faces its own difficulties: the narrowest class is the individual event itself, which has a frequency of either 0 or 1, rendering the probability assignment trivial and useless.
John Venn's The Logic of Chance recognizes that the same event belongs to multiple reference classes, each with different frequencies.
Hans Reichenbach proposes the "narrowest reference class" principle in The Theory of Probability, acknowledging the problem while offering a pragmatic resolution.
Alan Hajek's influential paper "The Reference Class Problem is Your Problem Too" argues that the reference class problem affects virtually all accounts of probability, not just frequentism.
The Problem for Frequentists
For the frequentist, probability is defined as the limiting relative frequency in an infinite sequence of identical trials. But no two real-world events are truly identical — the 55-year-old man lives a unique life, making his case different from every other case in any reference class. The frequentist must therefore decide which similarities matter and which differences can be ignored, and this decision is not itself determined by frequencies. The reference class problem thus reveals a layer of judgment beneath the frequentist edifice that is not captured by the formal theory.
In legal settings, this has practical consequences. Expert witnesses who cite statistical probabilities — "the probability that this DNA match occurred by chance is one in a billion" — are implicitly choosing a reference class (the relevant population database). Different choices can produce dramatically different numbers, and jurors are rarely informed about this dependence.
In the famous case of People v. Collins (1968), the prosecution multiplied probabilities of independent characteristics (interracial couple, yellow car, ponytail, mustache) to argue that the defendants' matching description was astronomically unlikely by chance. The California Supreme Court overturned the conviction, noting among other issues that the reference class (couples matching this description in the Los Angeles area) was poorly defined, the independence assumption was unjustified, and the probability calculation conflated the probability of a random match with the probability of innocence.
The Bayesian Response
Bayesians often claim immunity to the reference class problem by treating probabilities as degrees of belief rather than relative frequencies. A Bayesian need not identify a reference class — they simply state their prior probability and update it via Bayes' Theorem. But critics argue that the problem reappears in prior construction. When choosing a prior for a novel situation, the Bayesian must decide which past experiences are relevant — which is, in effect, choosing a reference class for the prior.
Alan Hajek (2007) has argued that the reference class problem is universal, affecting all interpretations of probability that aim to connect probability to the empirical world. Propensity theorists face it when deciding which features of a physical setup determine the propensity. Logical probability theorists face it when choosing the language in which to formulate the evidence. No interpretation entirely escapes the need to decide what counts as relevantly similar.
Practical Resolutions
While no fully satisfying theoretical resolution exists, several practical strategies mitigate the problem. Regression and stratification systematically condition on observable covariates, producing narrow reference classes within a formal statistical framework. Propensity score methods reduce high-dimensional covariate information to a single score, creating approximate reference classes for causal inference. Hierarchical Bayesian models partially address the problem by allowing data from multiple reference classes to inform estimates at each level, with the degree of borrowing determined by the data.
Perhaps the most honest response is to acknowledge the problem and perform sensitivity analysis: compute the probability under multiple reasonable reference classes and report the range. If the conclusions are robust to the choice of reference class, the problem is practically (if not philosophically) resolved. If they are sensitive, the choice of reference class is a genuine source of uncertainty that must be acknowledged.
"Every problem of probability, in the sense of applied mathematics, is a problem of assigning the individual case to the appropriate reference class. The theory of probability does not solve this problem — it presupposes a solution." — John Venn, The Logic of Chance (1866)
Connection to Modern Machine Learning
The reference class problem manifests in machine learning as the problem of choosing features, training populations, and test distributions. A model trained on one population (the training reference class) may fail when applied to a different population (a new reference class) — a phenomenon known as distribution shift. The growing literature on fairness in machine learning is deeply entangled with reference class issues: whether to condition predictions on race, gender, or other protected attributes is precisely a reference class decision with profound ethical implications.
Example: Health Risk by Reference Class
A dataset of 30 patients records age group (young/middle/senior), gender (M/F), smoking status (yes/no), and whether a health event occurred. How does the estimated probability change with the reference class?
By age: Young = 2/8 = 25.0%, Middle = 4/12 = 33.3%, Senior = 7/10 = 70.0%
By gender: Male = 8/16 = 50.0%, Female = 5/14 = 35.7%
By smoking: Smoker = 8/11 = 72.7%, Non-smoker = 5/19 = 26.3%
Intersection — Senior + Male + Smoker: 4/4 = 100.0%
Intersection — Young + Female + Non-smoker: 0/3 = 0.0%
The "same" event probability ranges from 0% (young female non-smoker) to 100% (senior male smoker), depending entirely on which reference class is chosen. The overall rate of 43% masks enormous heterogeneity. A frequentist who assigns probability by reference class frequency gets a completely different answer depending on which features they condition on. A Bayesian faces the same dilemma in choosing what prior information to incorporate. The reference class problem demonstrates that probability is not a property of the event alone — it is a property of the event relative to a chosen context.