OpenAI Chief Scientist Argues Perplexity is Key to Measuring AI Intelligence
OpenAI's Chief Scientist, Jakub Pachocki, posits that perplexity, a measure of how well a probability model predicts a sample, is the fundamental metric for tracking AI progress.

By Clara Tan, Artificial Intelligence
8 min read
OpenAI's Chief Scientist, Jakub Pachocki, has published a new blog post titled "Perplexity and Intelligence" on November 22, 2026, advocating for perplexity as the central metric for evaluating and understanding the progress of artificial intelligence.
Pachocki argues that while modern AI might appear to use diverse techniques, the underlying driver of advancements is the training of increasingly effective "compressors" or predictors on vast datasets. He states, "I believe tracking the pace of progress on training better compressors is key to understanding the arc of advances in AI."
Perplexity, in the context of language models, quantifies how well a probability model predicts a sample. A lower perplexity score indicates that the model is better at predicting the next item in a sequence, suggesting a deeper understanding of the underlying structure of language and, by extension, the world.
The blog post draws a parallel between compression and intelligence, noting that to compress data well, a model must identify and exploit patterns. This process of pattern recognition is fundamental to what we consider intelligence. Pachocki writes, "The better a model is at predicting the next token, the more it has learned about the structure of the world that generated the data."
A key observation in the post is that specific capabilities emerge at predictable perplexity thresholds. For instance, the ability to perform arithmetic or understand syntax appears when the model reaches a certain level of predictive performance. This suggests that tracking perplexity over time provides a roadmap for when new abilities might surface.
Pachocki also addresses the relationship between perplexity and model scale. He notes that perplexity improves predictably with compute, data, and model size, following scaling laws. This predictability allows researchers to forecast the capabilities of future models based on projected perplexity improvements.
The post has sparked discussion on Hacker News, where some commenters raise concerns. One critique questions whether perplexity alone captures the full picture of intelligence, pointing to "spiky abilities" where models excel at some tasks but fail at seemingly simpler ones. Another commenter challenges the notion of "intelligence on a log scale," arguing that the metric may not map linearly to human notions of cognitive ability.
Despite these debates, Pachocki's framing offers a concrete, measurable way to track AI progress. By focusing on perplexity, the AI community gains a shared benchmark that transcends specific architectures or tasks, potentially accelerating research by providing a clear optimisation target.
What is Perplexity and How Does It Work?
Perplexity is a measurement used in information theory and natural language processing to evaluate how well a probability model predicts a sample. In the context of language models, it represents the exponentiated average negative log-likelihood of a sequence. Intuitively, it can be thought of as the weighted branching factor: a perplexity of 10 means the model is, on average, as uncertain as choosing between 10 equally likely options for the next token.
Mathematically, for a sequence of tokens x_1, x_2, ..., x_N, the perplexity PP is:
PP = exp(-1/N * sum_{i=1}^N log P(x_i | x_{<i}))
Where P(x_i | x_{<i}) is the probability the model assigns to the actual next token given the previous context. Lower perplexity indicates the model assigns higher probability to the correct tokens, meaning it is less "perplexed" by the data.
Why Perplexity Matters for AI Progress
Pachocki's argument centres on the idea that minimising perplexity on large, diverse datasets forces the model to learn generalisable patterns about the world. He posits that:
Compression is intelligence: To predict well, a model must understand the underlying structure generating the data.
Scaling laws are predictable: Perplexity decreases smoothly with compute and data, allowing capability forecasting.
Emergence is measurable: New abilities appear at specific perplexity thresholds, not randomly.
This framework suggests that the path to more capable AI systems is fundamentally about building better predictors, and perplexity is the scorecard.
Critiques and Limitations
While perplexity is a powerful metric, the Hacker News discussion highlights valid limitations:
Spiky capabilities: Models can have low perplexity overall but fail catastrophically on specific reasoning tasks.
Benchmark contamination: Training on test sets can artificially lower perplexity without improving generalisation.
Human alignment: Low perplexity does not guarantee helpful, harmless, or honest outputs.
Log-scale interpretation: A 10% perplexity reduction at 1.5 is not equivalent to the same reduction at 10 in terms of perceived capability gains.
Pachocki acknowledges some of these points, noting that perplexity is a necessary but not sufficient condition for general intelligence. He argues, however, that it remains the most fundamental and trackable metric for the field.
What to Watch
Whether OpenAI releases perplexity numbers for future model announcements, making the metric a public benchmark.
If other major labs (Anthropic, Google DeepMind, Meta) adopt perplexity as a primary progress indicator in their communications.
Research into "perplexity-to-capability" mapping: quantifying exactly which abilities emerge at which perplexity levels across different model families.
Development of evaluation suites that correlate strongly with perplexity but capture the "spiky" failure modes.
Sources
OpenAI Blog: Perplexity and Intelligence (https://openai.com/index/perplexity-and-intelligence)
Hacker News Discussion: Perplexity and Intelligence (https://news.ycombinator.com/item?id=50601525)
This story was first published by TechWhoKnows on 23 November 2026. Original: https://techwhoknows.com/ai/openai-chief-scientist-argues-perplexity-is-key-to-measuring-ai-intelligence
Sources
- 1OpenAI Blog: Perplexity and Intelligenceopenai.com
- 2Hacker News Discussionnews.ycombinator.com