David M. Blei is an American computer scientist and statistician at Columbia University whose invention of Latent Dirichlet Allocation (LDA) and development of stochastic variational inference have had transformative impact on machine learning, natural language processing, and Bayesian computation. LDA introduced a principled Bayesian approach to discovering latent thematic structure in text corpora, while his subsequent work on scalable variational methods made Bayesian inference practical for datasets of unprecedented size.
Life and Career
Born in the United States. Studies mathematics and computer science before pursuing graduate work in machine learning.
Earns his Ph.D. from UC Berkeley under Michael I. Jordan, with his thesis on probabilistic topic models.
Publishes "Latent Dirichlet Allocation" with Andrew Ng and Michael I. Jordan in the Journal of Machine Learning Research. The paper becomes one of the most cited in machine learning history.
Develops the correlated topic model and dynamic topic models, extending LDA to capture topic correlations and temporal evolution.
Co-develops stochastic variational inference with Matthew Hoffman and others, enabling variational methods to scale to millions of documents.
Publishes "Variational Inference: A Review for Statisticians" with Alp Kucukelbir and Jon McAuliffe, providing a comprehensive and accessible treatment of modern variational methods.
Latent Dirichlet Allocation
LDA models each document as a mixture of latent topics, where each topic is a distribution over words. The generative process assumes that a document's topic proportions are drawn from a Dirichlet prior, and each word is generated by first choosing a topic and then choosing a word from that topic's distribution. Inference reverses this process, discovering the topics and their proportions from observed text.
θ_d ~ Dirichlet(α) (topic proportions)
For each word position n:
z_{d,n} ~ Categorical(θ_d) (topic assignment)
w_{d,n} ~ Categorical(β_{z_{d,n}}) (word)
For each topic k:
β_k ~ Dirichlet(η) (word distribution)
The beauty of LDA is that it is a fully generative Bayesian model with clear probabilistic semantics. The Dirichlet priors enforce sparsity: documents tend to be about a small number of topics, and topics tend to use a focused vocabulary. This inductive bias, encoded through the prior, is what makes the discovered topics interpretable. LDA has been applied far beyond text, to images, music, genetic sequences, congressional voting records, and any domain where discrete observations arise from latent categorical structure.
While modern large language models have largely superseded LDA for text understanding, the conceptual framework that Blei established remains deeply influential. The ideas of latent representations, generative modeling, and variational inference that LDA popularized are direct ancestors of variational autoencoders and other modern generative models. LDA demonstrated that probabilistic thinking could scale to real-world text data, paving the way for the probabilistic machine learning revolution.
Stochastic Variational Inference
Classical variational inference requires processing the entire dataset at each iteration, making it impractical for large-scale applications. Blei and collaborators developed stochastic variational inference (SVI), which uses stochastic optimization to update the variational parameters using random subsets (mini-batches) of data. This approach achieves convergence guarantees while processing only a fraction of the data at each step, enabling variational inference to scale to corpora of millions of documents.
Legacy
Blei's work established topic modeling as a field and made variational inference practical for large-scale applications. His papers are among the most cited in machine learning, and his pedagogical writing has made complex Bayesian ideas accessible to researchers across computer science and statistics. The framework he developed, combining Bayesian generative models with scalable variational inference, remains the template for much of modern probabilistic machine learning.
"Probabilistic topic models provide a suite of algorithms for discovering the hidden thematic structure in large collections of documents. They let us organize, search, and understand quantities of text that would be impossible for humans to annotate by hand." — David Blei