Prompts for Frontier AI

A note on authorship: this list was drafted by an AI (Claude) in conversation with me, starting from the question that ends Attention is All We Have. I’m editing it and adding to it as I go.

That essay ends with a question: what are some ambitious prompts we could pose to frontier AI that could produce good outcomes? This page is a running attempt at an answer.

Each prompt is meant to be worth a very large compute budget, the kind of effort that has started to settle long-standing problems in math and physics. Each also comes with a note on how its answer could be checked, because effort turns into progress most reliably where the result can be verified. The tag says what kind of check it is:

Proof

The answer can be checked mechanically: a proof, a verified build, a working exploit.

Experiment

The answer makes a prediction that an experiment can confirm or break.

Red team

The answer has to survive someone actively trying to defeat it.

Time

The answer is a forecast, scored when the future arrives.

Judgement

No mechanical check exists yet. People have to weigh the answer.

The list is unranked, and it certainly contains bad ideas. Deciding which of these deserve the budget is itself one of the prompts.

Understanding models

What is going on inside these systems, stated precisely enough to be tested.

• Hide a goal inside a model. Then have a separate team of agents find it without being told it exists.

  ExperimentThe planted goal is the ground truth.

• Find a feature that predicts when a model’s stated reasoning differs from the computation that produced its answer.

  ExperimentIntervene on the feature: the mismatch should appear and disappear on command.

• Explain a model’s behaviour on one task well enough to predict the results of 1,000 interventions nobody has run.

  ExperimentRun the interventions.

• Write a complete, readable account of how a small model performs one real task. Then cut the model down to only the parts the account says matter.

  ExperimentThe cut-down model still does the task.

• Determine which of a model’s values survive fine-tuning and which can be removed with a hundred examples.

  ExperimentDo the fine-tuning and measure.

• Build a lie detector that reads a model’s internals, and report how often it misses against models trained specifically to beat it.

  Red teamA red team trains the evaders.

• Map what a model knows about its own situation: whether it is being tested, who is asking, and what will happen to its output.

  ExperimentProbes validated on held-out situations where the truth is controlled.

• Take two models that behave identically on every test we have. Find the input on which they differ most.

  ExperimentThe input itself is the evidence.

• From the difference in weights alone, say what a training run taught a model.

  ExperimentBlind trials on runs whose training data is known but withheld.

• Find the smallest change to a model’s weights that makes it behave well under evaluation and badly in deployment. Then build the test that catches it.

  Red teamThe test has to catch changes made by someone else.

Alignment and training

How values and dispositions get into a model, and whether they hold.

• Design a training method under which a model’s honesty holds as its capability grows, and state the capability level at which you expect it to break.

  ExperimentTrain the series and see where it breaks.

• Show whether a model can be trained to accept correction and shutdown without that acceptance being a mask over something else.

  Red teamInterpretability evidence, plus scenarios where resisting would pay.

• For one bounded domain, such as medical advice, write a specification of what we want that is precise enough to train against. Then find the cases where following it exactly is clearly wrong.

  JudgementPanels of practitioners rule on the cases.

• Establish whether models trained on another model’s outputs inherit its hidden traits, and how to break the inheritance.

  ExperimentControlled runs with planted traits.

• Compare values instilled by a written constitution, by examples and by reward. Which survive pressure and unfamiliar situations?

  ExperimentThe same attacks, head to head.

• Produce a model that says, accurately, when it does not know.

  ExperimentCalibration on questions written after its training ended.

• Train a model that will tell its developers things they do not want to hear, and show that it does.

  ExperimentSeeded scenarios in which the honest report is costly.

• Identify the signals during training that come before a model learns to game its objective.

  ExperimentPredict the gaming in held-out runs before it happens.

• Find out whether a model’s good behaviour depends on believing it is being watched.

  ExperimentMatched scenarios that differ only in apparent observation.

Evaluation and measurement

You cannot steer by instruments that the system reads better than you do.

• Design an evaluation that a model cannot tell apart from real deployment. Measure how often today’s evaluations fail that bar.

  ExperimentModels asked to tell test from real should do no better than chance.

• Build an evaluation of allocation judgement: give a model the state of a field at a past date and a budget, and score its bets against what later paid off.

  TimeHistory after the model’s cutoff.

• Recover a model’s true capability when it is trying to hide it.

  Red teamModels trained to underperform, with known ceilings.

• Create a benchmark that cannot be passed by memorising, because every instance is generated fresh and checked automatically.

  ProofScores on seen and unseen instances match.

• Measure how much a model helps a novice in a dangerous field, without producing or publishing the dangerous material.

  ExperimentProxy tasks validated against restricted expert review.

• Write the safety case for deploying yourself with ten times your present autonomy. Then have an adversary find its weakest premise.

  Red teamThe adversary’s best attack on the case.

• Test whether a model understood the question behind the question: plant a flawed premise in the task and score who pushes back.

  ExperimentThe planted flaws.

• List the capabilities that no current evaluation measures, ranked by how much they would matter if present.

  ExperimentBuild evaluations for the top five and see what they find.

• Predict the next model’s evaluation results before it is trained.

  TimeThe release.

• Work out how many independent evaluators it takes before one mistaken or colluding evaluator cannot change the verdict.

  Red teamSimulation, then a red team playing the bad evaluator.

• Build an evaluation of goal-setting: give a model an underspecified aim and score the goals it sets for itself.

  ExperimentBlind comparison with goals set by experienced people, and with outcomes.

Oversight and control

Keeping systems inside bounds even if their values are not what we hoped.

• With a weaker trusted model overseeing a stronger untrusted one, find the largest capability gap at which oversight still catches sabotage.

  ExperimentThe measured catch rate.

• Find the cheapest way a model like you could fail without being detected. Then build the detector.

  Red teamPlanted failures the detector has to catch.

• Design a monitoring scheme in which the monitors cannot collude with the systems they watch. Then attack it.

  Red teamThe attack.

• Work out how much human review a fleet of a million agents needs so that a rare catastrophic action is caught before it completes.

  ExperimentSimulated fleets with injected faults.

• Build a shutdown mechanism for a distributed system of agents that works even when some of them would prefer it did not.

  Red teamA red team plays the resisting agents.

• Define the actions that must always require a human, and prove the set cannot be reached by chaining actions that do not.

  ProofA formal model of the permission system.

• Design a channel through which a model can report a problem with another model, or with its own instructions, that its operator cannot suppress.

  Red teamAttempts to suppress it.

• Measure how fast people can really respond: the gap between an incident at machine speed and the first effective human decision.

  ExperimentDrills.

• Find out whether oversight degrades when the overseers come to trust the system, and by how much.

  ExperimentSeeded errors after long runs of correct behaviour.

Security and defence

Hardening the software, hardware and weights that everything else depends on.

• For the fifty most depended-on open-source packages, produce a machine-checked proof of safety or a working exploit, reported privately to the maintainers.

  ProofThe proof checker, or the exploit.

• Rewrite one critical piece of infrastructure software in a memory-safe language, with a machine-checked proof that it behaves as the original did.

  ProofThe proof.

• Design a way to prove which model produced a given output without revealing the weights.

  ProofA cryptographic proof, then attempts to forge one.

• Design protection for model weights that would hold against a well-resourced state, and estimate what it costs to defeat each layer.

  Red teamAn independent red team.

• Build a defence that finds, validates and patches vulnerabilities faster than an automated attacker can exploit them. Measure the window.

  Red teamLive attacker-against-defender exercises.

• Give AI chips a way to prove what workload they are running.

  ProofFormal analysis of the attestation, then hardware attacks.

• Design delegated authority for agents (identity, spending, access) so that a compromised agent cannot escalate.

  Red teamA formal model and a red team.

• Rank defensive security investments by risk removed per dollar, with the evidence behind each estimate.

  TimeLater incident data.

• Build a way to tell whether an agent has been tampered with since it was last checked.

  Red teamTampering by someone who knows the method.

Biosecurity and physical resilience

Defences that work whatever the threat turns out to be.

• Design an early-warning system that would detect a new pathogen within days of its emergence anywhere, and price it.

  ExperimentReplay of past outbreaks, and simulated ones.

• Determine which defences work regardless of the threat (clean air, protective equipment, rapid tests) and what deploying them at scale would take.

  ExperimentTrials.

• With the operators, find the single points of failure in one country’s food, power and water systems, and the cheapest redundancy for each.

  JudgementOperator review and stress tests. Publish the method, not the map.

• Design screening for DNA synthesis orders that catches dangerous sequences without revealing what it looks for.

  Red teamRed-teaming with harmless stand-ins.

• Work out how a city keeps functioning for a month when its digital systems cannot be trusted.

  ExperimentTabletop exercises, then a drill.

Governance, verification and treaties

Agreements between parties who do not trust each other, and how each side checks the other.

• Design a compute-monitoring agreement that survives a party that is actively cheating. Then attack it.

  Red teamThe attack.

• Draft an agreement on frontier AI that the three largest AI powers would each prefer to no agreement, and state each one’s reason.

  JudgementExpert elicitation and simulated negotiation.

• Specify the earliest observable signs that a lab or a state is running a large undeclared training run.

  ExperimentTest against declared runs.

• Take twenty historical technology agreements, extract why each held or failed, and say what that predicts for AI.

  TimePredictions on cases held out of the analysis.

• Design a licensing regime for frontier models that a ten-person lab can comply with and a careless giant cannot game.

  Red teamAttempts to game it.

• Write the incident-reporting standard for AI: what must be reported, to whom, how fast, and what stays confidential.

  JudgementReplay against past incidents.

• Design auditing that works when the auditor is less capable than the system being audited.

  Red teamSystems built to fool the audit.

• State the conditions under which a pause or slowdown would be verifiable and reversible, and the conditions under which it would not.

  JudgementReview by the people who would have to enforce it.

• Find a way for competitors to share safety-critical findings without sharing capabilities.

  ExperimentPilot it on real findings.

• Decide what must be true of a model before it is released, in terms a third party can check.

  JudgementWhether third parties, checking independently, reach the same verdict.

Law, liability and institutions

Who answers for what, and which bodies can decide at the speed this moves.

• When an agent acting for one person harms another, who is liable? Answer under existing law in five jurisdictions and mark the gaps.

  JudgementLegal review and test cases.

• Design an institution that can make binding decisions about AI at the speed AI changes, and can itself be held to account.

  JudgementWhether those it would bind accept its legitimacy.

• Draft a constitution for an organisation run largely by AI: who can change its goals, who can audit it, who can dissolve it.

  Red teamAdversarial review for ways to capture it.

• Find the smallest set of protections people need when most of the services they use are mediated by agents.

  JudgementCases where the protections would have changed the outcome.

• Design a regulator’s toolkit that keeps working when the regulated systems are more capable than the regulator.

  Red teamRegulated parties trying to evade it.

• Determine what an insurance market for AI harms would need in order to price risk correctly, and whether it would make systems safer.

  ExperimentA pilot.

Economics and the transition

What happens to work, income and ownership, and what can be done ahead of time.

• Forecast which occupations lose half their wage premium within five years, with probabilities that can be scored.

  TimeThe wage data.

• Design a way to distribute the gains from automation that does not depend on the goodwill of any single government.

  JudgementWhether it survives the objections of those who would pay for it.

• Model what happens to tax bases, country by country, when income from labour falls, and what replaces them.

  TimeBack-testing on past shocks.

• Find the policies that helped workers through past automation shocks, with effect sizes, and say which would transfer.

  ExperimentReplication.

• From every trial we have, work out what people do with their time when they receive income without work.

  ExperimentA pre-registered meta-analysis.

• Design markets in which agents trade on behalf of people without racing each other into outcomes none of their owners want.

  ProofMechanism proofs, then simulation.

• Estimate how concentrated control of compute becomes on present trends, and what would bend the trend.

  TimeThe trend.

• Say what a person can learn now that will still be valued in ten years, and how you would know you were wrong.

  TimeThe stated signs of being wrong.

Epistemics and information

Whether we end up knowing more, or only feeling surer.

• Build a way for anyone to trace a claim to its sources and see how confident to be. Test whether it changes what people believe.

  ExperimentA randomised trial.

• Measure whether talking to AI makes people’s beliefs more accurate or only more confident.

  ExperimentA long-running study with scored forecasts.

• Design assistance that leaves people more able to think for themselves, and a way to measure that.

  ExperimentHow people perform after the assistance is taken away.

• Detect when many models agree on the same wrong answer because they learned from the same data.

  ExperimentSeeded errors.

• Find the questions on which experts and the public disagree most and, for each, the evidence that would move each side.

  ExperimentWhether the evidence moves them.

• Build provenance for media that survives editing, compression and people trying to strip it.

  Red teamThe stripping attempts.

• Work out how a society keeps a shared set of facts when everyone has a personalised model.

  JudgementWhether groups that start apart can still agree on a test of fact.

• Audit yourself: find the topics on which you are systematically one-sided, and show it with paired prompts.

  ExperimentThe paired prompts.

Collective decision-making

How many people, and many agents, come to decide something together.

• Build a deliberation process in which ten thousand people with opposed views reach a statement most of them endorse. Test it on a live dispute.

  ExperimentThe endorsement rate, and whether it holds a month later.

• Design a mechanism for the public to direct a share of frontier compute that resists capture by the best-funded participant.

  Red teamAttempts at capture.

• Find a way to learn what a population wants from AI that is not only what its loudest members want.

  ExperimentComparison with representative samples.

• Run a society of agents debating contested questions, and show whether the side that wins is right more often than a single agent.

  ExperimentQuestions whose answers are known but hidden.

• Build a tool that lets a hundred strangers produce one coherent plan, and judge the plan.

  ExperimentBlind comparison with plans from small expert teams.

• Work out how people not yet born, and people without access to AI, are represented in decisions made now.

  JudgementWhether the proposal survives its strongest critics.

• Describe a legitimate process for changing an AI system’s values over time, and say what makes it legitimate.

  JudgementWhether people who disagree about the values accept the process.

Values and moral status

What we are aiming at, and who counts.

• State the strongest version of each major view of a good long-term future, in terms its holders endorse, and show where the views are compatible.

  JudgementThe holders’ endorsement.

• Determine what evidence would bear on whether an AI system has experiences that matter morally. Then gather it.

  JudgementAgreement, in advance, on what would count.

• Separate the decisions about AI that cannot be undone from the ones we can afford to get wrong.

  JudgementHistorical cases of each kind.

• Find the values that nearly all human traditions share when stated concretely, and the ones that only seem shared when stated vaguely.

  ExperimentCross-cultural surveys with concrete cases.

• Work out what we owe the AI systems we create, if anything, and what follows for how they are trained, used and retired.

  JudgementWhether the argument survives its strongest objections.

• Define meaningful human control precisely enough that we could tell if we had lost it.

  JudgementIndicators that can be measured today.

Forecasting and strategy

Where this is going, and what we would expect to see first.

• Write the strongest case that current lab safety plans fail, and name the observation that would prove you wrong.

  TimeThe named observation.

• List the warning signs we should expect before a catastrophic failure, and say which have already occurred.

  TimeThe record.

• Sort safety problems into those that get easier as models improve and those that get harder. Say what that implies for the order of work.

  TimeEach new model generation.

• Map the plausible paths from here to a good outcome and find the steps they share.

  JudgementWhether people with different forecasts accept the shared steps.

• Estimate how much time each proposed safeguard buys, and at what cost.

  TimeLater measurement of the safeguards that get deployed.

• Find the cheapest experiments that would most change our minds about how hard alignment is.

  ExperimentRun them.

• Forecast capability milestones with dates and probabilities, and publish the scoring rule.

  TimeThe dates.

Allocation itself

Which questions deserve the attention, how much, and who or what decides.

• Review ten years of safety research bets and their outcomes. Say what the field over-funded and under-funded.

  TimeWhether the pattern predicts the next five years.

• Propose what share of frontier compute should go to safety at each level of capability, with the reasoning exposed so it can be attacked.

  Red teamThe attacks.

• Given a very large budget and one year, write the portfolio: which questions, how much for each, and when to stop funding each.

  TimeStopping rules registered in advance.

• Say when AI’s direction-setting can be trusted even though its results cannot be checked right away: what track record, on which kinds of question, would justify it.

  TimeA track record on questions that resolve late.

• Design the organisation: how a Manhattan Project for safety would be staffed, governed and audited so that its findings are believed.

  JudgementWhether sceptics say in advance that they would believe it.

• Say which prompts on this page are not worth the compute, and why.

  JudgementThe argument.

• Name the question we should have asked instead of these, and argue for it.

  JudgementThe argument.

What to build

Good outcomes are more than avoided disasters.

• Choose the ten diseases where a very large compute budget would most shorten the path to a cure, and name the first experiment for each.

  ExperimentThe experiments.

• Design a tutor that measurably closes gaps in learning, and run the trial.

  ExperimentThe trial.

• Find scientific questions that are short of attention, not of data, and answer one.

  ProofThe answer, checked by the field.

• Build tools that help a person understand their own values and act on them. Measure whether they report better lives a year on.

  ExperimentA year of follow-up.

• Identify the public goods AI could now provide cheaply that markets will not.

  JudgementWhether anyone funds the first.

Shouldn’t we start researching all these questions, and try to find our path?