Attention is All We Have
AI Alignment as an Attention Allocation Problem
Attention is all we have.
How should we direct it? What if AI safety research is actually an attention allocation problem?
We have:
- our human attention - what we collectively all decide to spend our time on
- our capital - the congealed attention of past minds and generations
- our compute - which is now a specific type of capital, a data center of geniuses, whose attention we can choose where to invest.
What should our time, capital and compute focus on next, in order to make things go well?
As models become increasingly capable, so can the ambition of what we ask of them. We’ve seen models able to solve long-standing problems in math and physics using large compute budget - what problems should we have these frontier AI systems focus on, that would improve society’s ability to adapt to this new technology and to steer us towards good outcomes?
If we were given a very large capital and compute budget, where should we focus it to make good outcomes happen?
What is the AI Safety equivalent of a Navier-Stokes like effort on a problem? What questions would we pose? Should we structure it as a debate / discourse amongst a society of frontier agents? What sort of artifacts would we want the agents to produce for us: papers and algorithms demonstrating new alignment techniques? government policy proposals? verification techniques for allowing international treaties? formal verification for code systems to help our software be more secure? better technologies for human collaboration and mutually beneficial engagement broadly?
Should we ask it what the ratio of our attention that we should be spending on these safety questions should be, in proportion to cranking the gears of intelligence upwards? That is to say, what is the ratio of compute we should spend on answering safety questions, as opposed to training the next generation of more capable models, given that the next generations of models may both be able to more effectively answer these questions, but also potentially be more misaligned and risky.
One formulation of the bet that labs are making right now is:
“AI will hopefully solve the problems that AI introduces. We’ll time the dynamics right, we’ll figure out how to effectively steer the attention to the right places, and the system will collectively adapt and steer itself towards good outcomes.”
Are we spending enough of our collective attention: our own time and labor, our capital, our compute, on addressing the right questions?
Who decides how to steer the societal flow of tokens towards good outcomes? In the past, we’ve relied on many mechanisms for allocating attention: including governments, and the dynamic price signals and memes of networks to steer our combinatorial attention. We’ve formed our collective knowledge and societal agreements through a discursive, co-evolving process.
Is AI a sufficiently good allocator of resources today? Could we just tell it:
“Spend your budget towards producing good/safe/resilient outcomes for humanity?”
Can it steer its own attention in the right ways? What are the evals for seeing its allocation judgement or its goal setting capability? What are the evals for seeing if it understood the nuance of the questions being asked, and if it could keep reframing, and reposing and discussing what was asked for it? What happens when the timescales it operates on are so much faster than the ones we operate on?
Envisioning a Manhattan Project style approach to safety, I’ve been thinking of what a team of the world’s top multi-disciplinary researchers, with a large-scale compute budget, and access to the frontier models, would hope to ask, what questions they would want to answer, and what they would seek to produce as part of their research efforts.
What are some ambitious prompts we could pose to frontier AI that could produce good outcomes?
I’m keeping a running list of candidate prompts.