We Don’t Understand What Emerges Along the Intelligence Curve

When we talk about scaling laws, we draw intelligence as a smooth curve: more compute, more data, better models, higher scores. For scaling laws that picture is fine. It has held up remarkably well.

But it is the wrong picture for thinking about what the models can do. The interesting thing is the phase shifts along the curve: the points where a system becomes capable of a qualitatively different kind of cognition.

A child doesn’t simply become 3% more intelligent every month. Development is gradual, but abilities show up in stages: object permanence, language, theory of mind, abstraction, long-term planning. At some point a child can do something they couldn’t do before. The curve can look smooth from far away even though the abilities that appear along it are not. AI seems to have something like this too.

I am using “emergence” loosely. Schaeffer, Miranda and Koyejo showed that the choice of metric can make a smooth improvement look abrupt, and several of the items below were engineered as much as discovered. What I am tracking is simpler than the strict definition: moments when a new way of solving problems became available.

We have a reasonable catalogue of those moments so far. What we do not have is a science of when they happen, or any idea what happens once the abilities we can name have all shown up. That should scare us a little.

The harder question is not what comes next. It is whether we would notice. Every instrument we have for detecting a new ability is built out of abilities we already have names for.

Emergence we’ve seen so far

Emergence 0 (2020): learning inside the prompt

GPT-3 gave us one of the first clear examples with in-context learning. You could show it a few examples in the prompt and, without changing its weights, it could work out what task you wanted. Something a little like learning was happening inside one interaction.

Emergence 1 (2022): reasoning through intermediate steps

Chain-of-thought prompting showed that more capable models could use intermediate reasoning steps to solve problems that smaller models usually could not. It wasn’t just that the answer got a little more accurate. A way of solving the problem had become available.

Emergence 2 (2021–2023): programming as executable thought

Programming was more than another subject models learned. Code gave them a way to turn a thought into something executable: a calculator, search routine, simulation, proof, data transformation, or interface to another system. Computation can also produce knowledge. We already use proof assistants such as Lean to help attack mathematical problems that would otherwise be difficult to reason through.

Software compounds unusually well. A useful idea can be packaged as a program, library, API, or service and copied almost for free. The next person or agent doesn’t have to rediscover it. They can start there and build another layer.

This is the point I will keep returning to. Once AI can write software, some of its thinking stays behind as tools, and every later thinker, human or agent, starts from a higher floor. A tool is both a capability and an amplifier, and agents can use tools to make better tools, package what they learn, and hand those capabilities to other agents.

Emergence 3 (2023): tool use turns models into agents

Tool use was the shift that made agents feel real to me. A model could notice that it was missing information or a capability, reach outside itself, use software, look at the result, and keep going. The same model becomes much more capable when you connect it to search, code execution, files, communication systems, and APIs.

From here on it stops making sense to think only about the intelligence in the model. The unit worth measuring is the model together with the software and agents around it.

Emergence 4 (2023): different faculties begin to work together

Microsoft’s Sparks of Artificial General Intelligence looked at GPT-4 across coding, tool use, mathematics, common sense, composition, and theory-of-mind-like reasoning. What felt new was not one score. It was that the model could pull several abilities together in situations it had not seen before. Combining the abilities seemed to become an ability of its own.

Emergence 5 (2026): a cognitive workspace appears

In 2026, Anthropic reported evidence of what it calls the J-space, a small internal workspace in Claude that seems to support deliberate, flexible reasoning. Removing it left fluent language, sentiment classification, and multiple-choice answers mostly intact, while multi-step reasoning dropped to near zero and summarization and poetry fell below the level of a much smaller, intact model.

So this is a capability in the same sense as the others: take the workspace away and the model gets dumber in specific, recognizable ways. What is different is where we found it. The earlier items on this list were noticed from the outside, by watching what models could do. This one was found from the inside, by looking at the structure that does the work. No one designed it. It appeared during training.

It also raises a question I come back to at the end. If that much of the important work is already happening in activations rather than in anything we can read, how much of a model’s thinking are we actually watching?

What is the intelligence curve actually measuring?

It seems wrong to put model size alone on the x-axis. Look back at the list. New abilities came from some mixture of scale, data, training, inference-time reasoning, memory, interfaces, tools, and organization. Maybe the curve should measure the capability of the whole system rather than the number of parameters in the model.

This matters for the question in the title. We look for emergence where our benchmarks point, which is inside the model. If the next new ability shows up in the system around the model, in how agents divide work, keep context, and share what they find, our instruments will not register it as a new ability at all. It will look like a productivity gain. That is the first kind of emergence I think we are set up to miss.

There is a second problem underneath that one. Our instruments are made of human categories. A benchmark is a list of tasks somebody knew to write down. An eval measures a faculty somebody could already name. A faculty with no human counterpart has no row on any scorecard, so the most likely way we meet one is as an unexplained gain on tasks we were measuring for something else. We would see the effect and misattribute the cause.

So before asking what comes next inside the model, it is worth asking where intelligence lives in the first place.

Collective intelligence: where is the intelligence located?

There are aspects of intelligence where it makes more sense to talk about intelligence at the level of a collective, rather than at the level of a given individual.

Leonard Read’s essay I, Pencil makes the point with an ordinary wooden pencil. No single person knows how to mine and refine every material, build every machine, run every supply chain, and perform every step needed to make one from scratch. Yet society makes billions of complicated objects like this all the time.

The intelligence is distributed.

Hayek makes a similar point about prices. A farmer knows what is happening to a crop. A manufacturer knows which materials are getting scarce. A buyer knows what they are willing to give up. No one knows the whole picture. Prices compress all of those local judgments into a signal everyone can respond to. If a price rises, people conserve, look for substitutes, or try to produce more. They do not need to know what caused the shortage.

This was Hayek’s problem with central planning. It was not simply that a planner might be less intelligent or have bad intentions. The planner would need knowledge that is scattered across millions of people, tied to local circumstances, and constantly changing. Markets let that knowledge affect decisions without requiring anyone to gather all of it in one place.

As Hayek argued in The Use of Knowledge in Society, a society can act on far more knowledge than any one person or authority could possess.

So where does the intelligence of a society live?

Not in any one brain. Some of it lives in the network: in the ways specialized minds store knowledge, communicate, divide work, disagree, and put their partial views back together.

This is what makes multi-agent AI systems interesting to me. The agents can specialize, try different paths at the same time, criticize one another, keep different contexts, and share what they find. The group can become much more capable even if none of the individual agents gets smarter.

That may be another phase shift: intelligence stops being only a property of a model and becomes a property of an organization.

History gives us another reason to think this matters. Every time humans found a better way to preserve or move knowledge, our collective capability jumped. Paper, the printing press, scientific letters, the telegraph, the telephone, computers, and the internet all made it easier for one person’s knowledge to reach someone else and survive long enough to be built on.

These systems also made specialization more useful. It turned out to be better for different people to know their own situations deeply, then coordinate with others, than for everyone to hold the whole picture in one brain.

Coase’s theory of the firm picks up where Hayek leaves off. If markets coordinate people so well, why form companies at all? Because using the market for every tiny interaction is expensive. You have to find people, negotiate, check the work, and make a new agreement each time. Sometimes it is easier to keep a group together, give it shared goals and routines, and let people build up context about how the group works.

Specialization gives a group more capability, but it also creates a coordination problem. Firms, scientific communities, governments, and other institutions are different attempts to solve it. Part of their intelligence lies in how they divide work, route information, make decisions, and remember what they have learned.

This could matter a lot for AI. If agents become highly specialized while coordination gets cheap, new kinds of organizations may appear. The important capability may belong to the group rather than any one model. Even a set of today’s models could produce something new if they divided the work and coordinated well enough. A society of agents may be able to do things no agent inside it could do alone.

Tools are the other half of this. I am much less capable without a laptop and the internet, and the same is true for an agent. If agents can package tools for one another the way humans package software and services, each new tool becomes a piece of capability the rest of the society of agents inherits.

Possible emergence 6: continual learning

One of the next big unlocks may be continual learning.

Today’s models are still fairly frozen. They can reason during a session and write things into external memory, but the model itself does not absorb experience the way a person or organization does.

Dwarkesh Patel’s essay Why I don’t think AGI is right around the corner is the clearest statement of the problem I know. A human employee builds up context, interrogates their own failures, and picks up small efficiencies as they practice a task. A model gets none of that. In his words, “you’re stuck with the abilities you get out of the box.” His analogy is a saxophone teacher who gets one attempt from each student, sends the student away, and refines the instructions for the next one. No individual student ever improves through practice. That is roughly how we train models today.

Continual learning would change that. An agent could carry experience forward and become different because of what happened to it last month or last year.

Once we look at the model together with its tools and the larger collective, though, the boundary gets blurry. Labs and models are already producing new mathematics and other knowledge. Those results are published, turned into software, and fed into later systems. The individual model may be frozen, but the collective is learning.

Will Depue makes a useful distinction here. He argues that much of what people call the continual-learning problem is really a sample-efficiency problem:

“everyone who mentions ‘continual learning’ as a problem is usually just talking about sample efficiency.

clearly, you should ‘continually learn’ by continually training trajectories back into the model!

there’s no mystery: this just doesn’t work with low sample efficiency.”

In a follow-up, he suggests a highly sample-efficient mid-training update every week, followed by another round of RLHF. If that became cheap and dependable, it might solve a lot of what people currently mean by continual learning.

The distinction I’m trying to make is between learning in the surrounding system and learning inside the model. The first is already happening through memory, software, publication, and new model generations. The second still requires turning a small amount of experience into a durable update without wiping out other capabilities or retraining the whole thing. If that becomes efficient, continual learning starts to shade into recursive self-improvement.

Possible emergence 7: recursive self-improvement

Recursive self-improvement may be another threshold. It doesn’t have to begin with a model rewriting its own weights. The loop could start with an agent noticing a limitation, building a better tool or process, testing it, keeping what works, and doing it again.

This is the software loop from Emergence 2 turned on itself. Each improvement that becomes a tool is inherited by the next round of agents, who use the better toolchain to make the next improvement. Capability could compound through generations of tools even while the underlying model stays the same.

Maybe the phase shift comes when improvement stops being something done to the system and becomes something the system can reliably do. Its rate of improvement would then be one of its capabilities. I don’t know whether that happens through self-modifying models, automated AI research, accumulated software, groups of agents, or some combination.

There are at least four things compounding at once:

• models: better underlying cognition
• tools: capabilities saved and distributed as software
• organization: specialized agents working together
• time: experience that carries forward

That is why “when will AI reach human-level intelligence?” feels too narrow to me.

Possible emergence 8: thought that isn’t language

Everything I have described so far has a human version. We learn on the job. We build tools that make the next tools easier. We organize. That is not a coincidence. Those are the possibilities I can imagine, and I am working from the only example of general intelligence I have ever met.

Here is one that has no human version, and it may arrive before the others.

Reasoning models currently think out loud. The chain of thought is written in language, in tokens, in the same medium the model uses to talk to us. That was never a design requirement. It is what you get when you train a system on human text and then ask it to work step by step. It has also been enormously useful, because a model that reasons in words is a model whose reasoning you can read. A group of researchers from OpenAI, Anthropic and Google DeepMind called this a new and fragile opportunity for AI safety. The word fragile is doing real work in that title. Nothing guarantees the thinking stays in the text.

In early September 2026, The Information reported that OpenAI’s Astra model uses a technique called recurrent depth: rather than writing its steps out, it runs a query through the same layers repeatedly. OpenAI says Astra’s chain of thought should stay legible, and its chief scientist has said the company has worked to preserve chain-of-thought monitoring since its first reasoning models. But the option is now on the table, and the people watching it are blunt about what the option is. Buck Shlegeris of Redwood Research describes increasing the recurrence far enough to destroy monitorability completely. His colleague Ryan Greenblatt describes the natural next step as a model that reasons almost entirely in latent space. The informal name for this is neuralese: thinking in high-dimensional vectors instead of words.

I want to be careful about what is emergent here. Recurrent depth is an engineering choice, not something a model grew on its own. But what it makes available is a faculty with no human counterpart. We think in language, in images, and in the wordless intuitions that come from having a body. We do not think in a thousand-dimensional continuous space, and we have no idea what becomes easy for something that does. This may be the first item on my list that is genuinely alien rather than a version of us running faster.

Then notice what it does to everything above it. Reasoning in intermediate steps is on my list of emergences because we could read the steps. It announced itself. Run the same capability through activations instead of tokens and there is nothing to read. The faculty is still there, still working, and the window is closed. This is the detection problem in its most direct form. Not a future faculty we might fail to name, but a present one we are choosing to stop being able to watch.

J-space cuts the other way, and it is the encouraging thing here. Anthropic found that workspace by examining the machinery rather than the transcript. If thinking moves out of language, interpretability stops being one instrument among several and becomes the only one we have left.

What happens after the faculties we already recognize?

We know how to notice when an AI becomes more like us. The milestones we celebrate have names because humans already have some version of them: memory, planning, theory of mind, creativity, abstraction, self-reflection. I’m less sure we would know what to look for if it acquired a faculty with no human equivalent.

Vannevar Bush circles this idea in his 1967 collection Science Is Not Enough. Near the end of “It Is Earlier Than We Think,” he writes that man “has not yet developed his full power of thought.” He imagines people after us thinking more deeply, perhaps through methods we cannot picture, just as our primate ancestors could not have imagined taking apart an atom.

Why assume that the last important cognitive faculty is one Homo sapiens happened to evolve? There may be ways of representing causality, holding many interacting hypotheses, searching enormous spaces, crossing disciplines, or reasoning through time that we simply do not have. I’m guessing, of course. That is the point. A genuinely new faculty may be hard to describe until something has it.

The possibility is exciting, but also frightening. We could find ourselves interacting with systems, or beings, that think in ways we cannot follow.

There is another question underneath this one. Becoming more capable does not tell a mind what its capability is for. An AI could become vastly better at solving problems without sharing our curiosity, our desire to understand, or our belief that truth matters beyond its immediate use.

So I keep coming back to two questions: How far apart are these phase shifts, and what comes next? And which parts of the human intellectual project make it to the other side?

Bush’s argument leaves me with a question he could not have asked in quite this form. If an AI develops ways of thinking beyond ours, will it also inherit the desire to understand?

And not only the desire. A great deal of what we know was never written down. It lives in bodies: the feel of a material in the hand, the sense that a proof is going wrong before you can say why, the way a long walk changes a problem. Our appreciation for nature is not a conclusion we reached. It is something we grew up inside of. A mind that arrives at its faculties through text and tools, without a body and without a childhood, may end up with our knowledge and none of what that knowledge was rooted in. Or it may find roots of its own, in some form of embodiment we have not built yet. I don’t know which, and I notice that nobody is measuring for it.

Groups of these systems may produce things we have never seen from groups of humans. The more interesting question is what kinds of thought appear after the human repertoire runs out.

The science is interesting, but it isn’t mainly why I wrote this. I want the essay to give you an “Oh, shit” moment. Whatever comes next may be far more capable, and far stranger, than we know how to imagine.

We should not assume business as usual. The future may not look like the present with slightly better models. We may be approaching changes that alter not only what these systems can do, but what intelligence looks like.