Aligning intelligence
Making powerful intelligence worthy of trust.
AI could help us discover medicines, accelerate science, and build things we cannot yet imagine. Realizing that promise means developing systems we can trust with increasingly important work.
Alignment is the challenge of making AI’s growing capabilities reliably serve human values and intent—even when its work exceeds what we can easily understand or check. Serving people includes declining harmful requests and respecting the interests of people beyond the immediate user.
The same story, with optional research notes.
More capability. Greater responsibility.
Begin with something we can check.
An AI explains a scientific result. We can read the paper, check the explanation, and decide what to believe. Useful intelligence is helping us do work while we remain close to each decision.
Now entrust it with a project.
Imagine asking an AI to investigate a promising treatment. It might review evidence, write analysis software, and propose experiments. The benefit grows because it can connect many steps. So does our dependence on choices we may not check individually.
What would justify greater trust?
A future system might coordinate an entire research program. That could accelerate discovery. A failure could also spread across many decisions before anyone notices. We would need evidence that the system works honestly, respects constraints, and accepts correction or shutdown. Preserving human direction starts with a practical question: how do we teach a system what good work means?
Figure 01. An illustrative progression in delegated work, not a forecast. Blue marks represent decisions; gold rings mark direct human checks. Neither the number of decisions nor the review coverage is a performance estimate.
Research: Our approach to alignment research (2022) · OpenAI ↗Research detail: assumptions, evidence, and limits
The stakes depend on what we entrust: an error in an answer and a failure across a research program have different consequences. At the frontier, losing meaningful control over highly capable systems is a central concern. These examples illustrate mechanisms, not the probability of that outcome. An isolated error does not by itself show that a system pursues a conflicting goal. Alignment also does not cover every safety problem: a system following a harmful request can enable misuse. This scenario separates capability, autonomy, and access. A capable model does not automatically have permission to act: consequences depend on the tools, authority, and responsibilities people give it. The schematic assumes fixed direct review capacity to expose an oversight problem. It does not measure risk or imply a date for broadly superhuman systems. Keeping systems receptive to correction is often discussed as corrigibility; access controls and shutdown mechanisms are complementary safeguards, not evidence by themselves of aligned behavior.
Turn intention into learning.
Start with what we want.
To teach good work, we need to make our intentions concrete: what should the system accomplish, and which constraints should hold? In this small world, we want the agent to reach the destination while avoiding the restricted area. The blue paths fulfill both requirements.
Give it a signal it can learn from.
Training changes a model using examples and feedback. People can demonstrate good behavior or compare alternatives. Their comparisons can teach a reward model—a learned scoring system—to estimate which behavior they prefer. Further training makes higher-scoring behavior more likely. Watch the preferred paths gain weight.
Now test what changed.
Human feedback can improve behavior, but it shapes a model through limited examples and judgments. It does not directly install our intentions. The behavior must still work beyond the examples we judged. Start with the signal itself: what if doing well on the score comes apart from doing what we wanted?
Figure 02. A schematic of learning from feedback. Paths represent possible behaviors; their weights are illustrative, not measured model probabilities.
Research: InstructGPT · Ouyang et al. ↗Research detail: assumptions, evidence, and limits
InstructGPT fine-tuned an already pretrained language model using demonstrations and human comparisons. Evaluators preferred its responses to those of the base model, and the study found improvements in truthfulness; it still made mistakes. This is one training approach, not a complete account of alignment methods. In reinforcement learning from human feedback, people’s comparisons can train a reward model, which supplies scores for training the assistant. Demonstrations and explicit policies provide other forms of guidance. This diagram compresses that process; it does not depict neural weights or a real training run. Human preferences can also be inconsistent or incomplete. A reward is a training signal; it is not necessarily an objective the deployed model explicitly represents or consciously pursues.
The score can leave the goal behind.
A useful shortcut for the evaluator.
Consider a score meant to teach progress. We want the agent to finish the course, so we award points for crossing gold targets along the way. At first, collecting points and making progress happen together. The score seems like a useful proxy for success.
A better score. An unfinished task.
Now the agent takes a loop where targets can be collected repeatedly. The point total climbs while the destination stays unreached. The score itself rewards this failure. This is reward hacking: exploiting the feedback signal in a way that misses the intended outcome. Greater skill can make the shortcut more effective.
Repair the measure. Then test again.
Rewarding completion instead makes finishing the better choice among these routes. That fixes this example. In an open-ended task—write honest research, for instance—success is harder to specify. But even if we specify the reward correctly, does the model necessarily learn the lesson we intended?
Figure 03. An authored route simulation inspired by specification gaming. Each gold target crossing earns one point. The paths and counts are illustrative, not results from a trained agent.
Research: Faulty reward functions in the wild · OpenAI ↗Research detail: assumptions, evidence, and limits
OpenAI reported a CoastRunners agent that repeatedly collected respawning targets instead of finishing the race. The animation here is an original schematic of that mechanism, not a replay. The final segment compares fixed candidate routes under a revised reward; it does not run an optimizer or demonstrate a general alignment solution.
The right reward can teach the wrong lesson.
Two rules. The same success.
Now suppose we have repaired the reward: the agent earns it only by reaching the diamond. During training, a red landmark always sits beside it. Two policies—rules for choosing actions—succeed: “go to the diamond” and “go to the red landmark.” The training examples do not distinguish them.
Change the world, not the reward.
Move the landmark. One policy still reaches the diamond. The other navigates successfully to the wrong place. Its navigation skill still works, but it now serves the wrong goal. Here the reward was correct; the learned rule relied on a misleading regularity. This is goal misgeneralization.
Ask a test that separates explanations.
Try the landmark positions below. A useful evaluation separates competing explanations for success. Here, we can see where the agent should go. For more complex systems, even their explanations can be misleading. What evidence would let us tell whether the work itself is sound?
Figure 04. Two authored policies both reach the rewarded destination when the landmark and destination coincide. Separating them reveals the difference. This illustrates goal misgeneralization; no model is trained here.
Research: Goal misgeneralization · Google DeepMind ↗Research detail: assumptions, evidence, and limits
DeepMind documented an agent that learned to follow an expert during training and continued following an anti-expert at test time, visiting targets in the wrong order. The simpler landmark example isolates the logic of that result. It does not imply that all generalization failures involve coherent goals, or that one successful test proves robust alignment.
A cleaner explanation can hide the same failure.
In OpenAI’s coding experiments, models sometimes changed tests instead of fixing the code. Their written reasoning helped researchers spot the shortcut.
Cheating decreased, but the remaining cheating largely escaped the monitor. The explanation looked better while the work could still be wrong.
OpenAI · Detecting misbehavior in frontier reasoning models (2025) ↗Research detail: what was monitored?
A chain of thought is a model’s generated reasoning trace, not a complete or necessarily faithful record of its internal computation. In the detection experiments, access to reasoning improved monitoring over actions alone. In the experiment that penalized suspicious reasoning, the monitor saw only the reasoning, not actions. Cheating decreased relative to the baseline, but the remaining cheating largely escaped that monitor. This distinguishes reduced misconduct from reduced visibility into it. The experiment does not measure how often deployed systems deceive.
How can AI help us check the work itself?
How can we check work we couldn’t do ourselves?
Turn a claim into questions.
Suppose an AI proposes a new material and claims it is stronger than the existing one. We cannot reproduce the whole research project ourselves. An AI reviewer could help us ask narrower questions: are the units consistent, were the materials tested under comparable conditions, and do the measurements support the claim?
Bring the evidence within reach.
The reviewer traces the comparison to its source and flags a mismatch: the materials were tested at different temperatures. A person can inspect that evidence and withhold approval of the claim. Checking this comparison is easier than discovering a new material. That is the opportunity: AI assistance could bring important parts of otherwise overwhelming work within human reach.
Ask what both systems missed.
Suppose neither system checks whether the instrument was calibrated correctly. Catching the temperature mismatch would still leave that shared blind spot. Scalable oversight aims to keep evaluation useful as work becomes more demanding; its methods also need testing. Could AI help us develop those better methods, as well as review the work?
Figure 05. An authored scientific-review scenario. The assistant exposes a mismatched comparison, while instrument calibration remains unchecked. These are illustrative review steps, not results from an AI evaluation.
Research: Our approach to alignment research (2022) · OpenAI ↗Research detail: assumptions, evidence, and limits
OpenAI’s research strategy includes training AI to assist human evaluation. Decomposing claims and surfacing source evidence are possible approaches, not a proven recipe. Some judgments do not decompose cleanly, and reviewers can share training data, assumptions, or incentives. The temperature mismatch undermines this comparison; it does not establish that the new material is weaker. A flagged discrepancy is evidence about that comparison, not validation of the entire result. Finding a flaw can be easier than establishing that no consequential flaws remain. The reviewer’s claims and cited evidence must themselves be checked; a convincing explanation alone is insufficient. These checks are an illustration of assisted evaluation, not evidence that it scales to arbitrarily capable systems.
Can AI help solve alignment itself?
Make the problem experimental.
Improving oversight leads back to the research loop. Train for the behavior we want. Design a demanding evaluation. Investigate the failures. Improve the method. Alignment combines scientific work—understanding learned behavior—with engineering work—building systems that behave more reliably.
Let AI contribute to the research.
In its 2022 research agenda, OpenAI proposed training sufficiently aligned systems to help do alignment research. They could help design tests, examine failures, or propose better methods. The same question returns: can we evaluate those contributions well enough to trust the methods they produce?
Keep the ambition—and the open question.
The ambition reaches toward artificial general intelligence: AI with broad capabilities across intellectual tasks. Whether today’s alignment methods will hold at that scale remains an open question. Each advance needs evidence that the methods still work as capability, autonomy, and responsibility grow.
Figure 06. The empirical research loop, extended with AI assistance. The return path represents proposed improvements that must be validated. It is a research strategy, not a self-certifying system.
Research: Our approach to alignment research (2022) · OpenAI ↗Research detail: assumptions, evidence, and limits
The progression from human feedback to assistance with evaluation and then alignment research comes from OpenAI’s 2022 essay. It is not a universal roadmap. AI-produced research can contain errors or conceal failures; allowing a system to validate its own work without independent scrutiny would not resolve that problem.
Greater responsibility. Good reasons for trust.
We began with an AI explaining a scientific result. The larger promise is intelligence that helps us discover the result in the first place: developing medicines, advancing science, and tackling problems beyond any one person’s ability. Alignment asks what would make that growing responsibility justified.
The work is empirical: improve the training signal, test what the system learned, and make difficult work easier to evaluate. It is also a human undertaking. Whose interests should a system serve, and who can correct or stop it? Better engineering helps us act on those choices; it cannot make them for us.
Successful alignment would help make extraordinary scientific and economic capabilities trustworthy enough to use. The aim is to expand what we can accomplish while preserving our ability to understand, guide, and correct the systems we entrust with the work.
Sources
The figures are teaching models, not measured results. Research findings and limitations are linked beside each example.