We Don’t Understand What Emerges Along the Intelligence Curve

We tend to draw intelligence as a smooth curve: more compute, more data, better models, higher scores.

I increasingly think this is the wrong picture.

The interesting thing may be the phase shifts along the curve — the points where a system becomes capable of a qualitatively different kind of cognition.

There is a loose analogy here to childhood development. A child does not simply become 3% more intelligent every month. Capacities such as object permanence, language, theory of mind, abstraction, and long-horizon planning appear in recognizable stages. The underlying development may be continuous, but what the mind can do changes qualitatively.

AI seems to have something like this too.

GPT-3 made in-context learning suddenly useful at scale: you could show a model a few examples in the prompt and it could infer what task you wanted without changing its weights. Work on chain-of-thought prompting then found that sufficiently large models could use intermediate reasoning steps to solve problems that smaller models largely could not.

By GPT-4, the interesting story was no longer any one benchmark. Microsoft’s 2023 Sparks of Artificial General Intelligence explored a much broader cluster of faculties: interdisciplinary composition, coding, tool use, mathematical reasoning, common-sense grounding, and theory-of-mind-like reasoning. The point was the ability to combine previously separate capacities in novel situations.

And then there are discoveries that look less like a new skill and more like the appearance of a new cognitive structure. In 2026, Anthropic reported evidence of what it calls the J-space: a small internal workspace in Claude that appears to mediate deliberate, flexible reasoning. Anthropic found that removing it largely preserved fluent language and simple recall while severely damaging multi-step reasoning. The striking part is that this workspace was not explicitly designed into the model. It appears to have emerged during training.

I don’t mean “phase shift” too literally. There is a real scientific debate about whether some supposedly sudden emergent abilities are partly artifacts of how benchmarks are measured; Schaeffer, Miranda and Koyejo showed that discontinuous metrics can make smooth improvements look abrupt. But the broader observation still seems important: as AI systems scale and acquire better training, memory, tools and scaffolding, qualitatively new ways of solving problems become usable.

And we do not have anything close to a predictive science of when those faculties should appear.

What happens after the faculties we already recognize?

So far, most of the milestones we celebrate have names because humans already have some version of them: memory, planning, theory of mind, creativity, abstraction, self-reflection.

That creates a strange selection effect. We know how to notice an AI becoming more like us. We have a much harder time imagining what lies beyond us.

Vannevar Bush gets at something adjacent to this in his 1967 collection Science Is Not Enough. Near the end of “It Is Earlier Than We Think,” he writes that humanity “has not yet developed his full power of thought”, and imagines those who come after us thinking more deeply than we do.

That line feels newly relevant.

If the intelligence curve keeps going, why would we assume the last important cognitive faculty is one Homo sapiens happened to evolve?

There may be ways of representing causality, holding interacting hypotheses, navigating enormous search spaces, synthesizing across disciplines, or reasoning through time that we simply do not possess. These are speculative examples, but that is the point: a genuinely new faculty may be difficult to describe before we have seen it.

The question I keep coming back to is: how far apart are these phase shifts, and what comes next?

The next phase shift may not happen inside a model

There is another possibility that I think is easy to miss when we talk about model intelligence: intelligence does not necessarily live inside an individual.

Leonard Read’s famous essay I, Pencil makes the point through an ordinary wooden pencil. No single person knows how to mine and refine every material, build every machine, operate every supply chain, and perform every specialized step required to make one from scratch. And yet society makes billions of complicated objects like this constantly.

The intelligence is distributed.

Humans increase our effective intelligence through specialization, division of labor, language, markets, companies, scientific institutions, libraries, governments, software, and shared culture. As Hayek argued in The Use of Knowledge in Society, much of the knowledge needed to coordinate a complex society is dispersed among many different people rather than held centrally by anyone.

So where does the intelligence of a society actually live?

Not in any one brain. It lives partly in the network: in the protocols by which specialized minds store knowledge, communicate, coordinate, disagree, divide work, and recombine what they know.

This is why multi-agent AI systems are so interesting to me. A collection of agents can specialize, explore many paths in parallel, critique one another, maintain different contexts, share persistent memory, and project useful discoveries into a common space. The resulting system can become substantially more capable even if no individual agent becomes substantially smarter.

In that sense, multi-agent systems may represent another phase shift on the intelligence curve: from intelligence as a property of a model to intelligence as a property of an organization.

Time may become another axis of intelligence

Today’s models are also strangely frozen. They can reason during a session and store information in external memory, but the underlying model generally does not continuously absorb experience the way a person or organization does.

Continual or lifelong learning would change that. An agent that can accumulate experience, update its strategies, preserve useful knowledge, and improve across months or years is not merely a better static model. It is a different kind of system.

At that point, intelligence is being compounded across at least three dimensions:

• scale — more capable underlying models;
• organization — many specialized agents combining their knowledge;
• time — systems that keep learning from their own experience.

Each of these could unlock its own phase shifts, and they can multiply one another.

This is why I think the usual question — when will AI reach human-level intelligence? — may ultimately be too narrow.

We have spent the last few years watching machines acquire faculties that humans already recognize.

The more interesting question is what kinds of thought appear after the human repertoire runs out.