01 / The starting point
Pretraining is a beginning.
Pretraining learns patterns from vast amounts of data. That gives a model a broad starting point. But producing a plausible continuation is not the same objective as following an instruction, working through a difficult problem, or gathering enough evidence to answer a question.
Post-training continues training an existing model to develop capabilities and shape behavior. It can teach the model how to respond, improve how it reasons, or specialize how it uses tools. The starting model may already have been post-trained; this process can happen more than once.
The central idea is a lasting change in behavior. Demonstrations show what to do. Comparisons express what is preferred. Rewards score what happened. Training uses those signals to change what the model tends to do on its next task.
02 / Learn from demonstrations
Show the model what good looks like.
A demonstration pairs an input with a response worth learning from. It might show a clear explanation, a carefully worked solution, or an interaction that calls a tool before answering. Training examples teach a way of responding as well as the information in a particular answer.
In supervised fine-tuning, selected tokens in the desired response become prediction targets. Training makes those tokens more likely in their context. For a tool-using assistant, the targets can include tool calls; returned tool results can supply context without becoming targets themselves.
The examples set the curriculum. They can be written by people or generated by models and then selected or checked. Many near-identical examples may teach less than a smaller, varied collection. A demonstration shows one useful path; it does not cover every situation the model will encounter.
Supervised fine-tuning · Training targets ↗03 / Inside a training step
An update makes the lesson persist.
The model assigns probabilities to possible next tokens. In supervised training, a loss penalizes low probability on the target tokens. Backpropagation computes gradients: how changes to trainable parameters would affect that loss. An optimizer uses those gradients to update the parameters.
Repeat this over batches of examples and the model’s future responses can change. Other methods construct different objectives, but the same distinction matters: reading a document changes the current context; a training update changes the learned model.
Post-training names a stage of development. Fine-tuning means further training an existing model; supervised fine-tuning is one common post-training method. Reinforcement learning can also further train an existing model. These terms overlap—they are not names for interchangeable procedures.
Inside a parameter update ↗04 / Learn from comparisons
Sometimes judging is easier than demonstrating.
Writing an ideal response is difficult. Comparing two responses can be easier. Which follows the instruction? Which acknowledges an unsupported claim? Which uses evidence well? A preference gives a relative judgment under a chosen standard.
One approach trains a reward model from comparisons, then uses its scores in reinforcement learning. Direct preference optimization, or DPO, instead trains on preferred and rejected responses relative to a reference model, without a separate reward-model training stage. These are alternative approaches, not compulsory consecutive steps.
Feedback can come from people or AI judges. Either can be inconsistent or reward the wrong features. If polished confidence wins over careful uncertainty, that preference becomes part of the training signal. The quality of the judgment matters as much as the machinery that learns from it.
DPO · Original paper ↗05 / Learn from attempts
Practice becomes a training signal.
With reinforcement learning, the model generates attempts and receives rewards. A training objective uses those scores to update its behavior across many tasks. The feedback need not supply a worked solution: the model must explore possible responses, and the algorithm must estimate which behavior to reinforce.
Some outcomes are directly checkable: an answer matches a known result, or a program passes tests. This is often called reinforcement learning with verifiable rewards. More open-ended work may need a learned reward model or a rubric-based evaluator. Passing a checker and satisfying a judge are different kinds of evidence.
This can develop reasoning as well as response style. DeepSeek-R1’s published work documents reasoning improvements through reinforcement learning and a recipe combining supervised and RL stages. But better reward is not sufficient: incomplete tests, weak judges, and familiar training tasks can all reward shortcuts.
DeepSeek-R1 · Reasoning through reinforcement learning ↗06 / From answers to agents
Now the attempt becomes a sequence of decisions.
A research agent searches, inspects a source, runs a calculation, finds a conflict, and changes course before writing its report. Each decision changes the information available for the next. That sequence of actions and observations is a trajectory. A long horizon means dependent decisions, not simply a long answer.
OpenAI described training its original deep-research model with reinforcement learning on difficult browsing and reasoning tasks, including browser and Python use. This makes research a useful application of the methods—not a separate kind of training. Our visual is a schematic trajectory, not a recorded experiment.
Useful feedback may arrive only when the whole attempt can be judged. An early source choice can undermine the final report; an unsuccessful search can help if it prompts a better one. Which decisions deserve credit? During the attempt, tool results change context. Afterward, a training process can use the scored trajectory to update the model.
Deep research · Documented training application ↗07 / Behind the interface
Pull back. The loop becomes a system.
One attempt becomes many, across many tasks. Models generate responses, tools execute actions, evaluators score results, and training workers compute updates. Long attempts complicate this coordination: a model may have changed before an earlier attempt finishes. Handling that delay is both an algorithm and a systems problem.
Thinking Machines’ Tinker makes the boundary concrete. Researchers supply the data or environment, objective, and training logic. Its API provides model sampling, gradient computation, optimizer steps, and saved state while the service manages distributed GPU infrastructure. A small program can direct a much larger computation.
Tinker uses LoRA, which freezes original weights and learns low-rank updates. Full fine-tuning instead updates all original parameters. This is a separate choice from the learning signal: either can be used with supervised or reinforcement-learning objectives. Infrastructure makes the loop practical; choosing what it should teach remains research work.
Thinking Machines · Tinker ↗08 / Ask whether it worked
The next task is the real test.
A training run produces candidate checkpoints: saved learned states. The latest checkpoint, or the one with the highest training reward, is not automatically the best. Use validation tasks to choose between candidates, then a separate final test to assess the chosen model.
For a research agent, change the questions and evidence. Add conflicting sources, a missing fact, or a failing tool. Check factual support, task completion, uncertainty, time, and cost. Split related material carefully so near-duplicate questions or documents do not leak across training and testing.
Compare against the starting model under matched prompts, tools, and inference budgets. Also ask what better prompting or tools alone achieve. More time spent searching or reasoning at inference is a different source of improvement from training. Measure regressions too: a useful specialization can come with losses elsewhere.
Evaluation · Beyond training reward ↗09 / What changed?
A better next attempt.
Within one attempt, a model can improve its answer by finding new evidence or thinking for longer. Across training updates, its parameters can change how it approaches future attempts. A capable system may combine both. Keeping them distinct helps us understand what actually improved.
There is no single mandatory post-training recipe. Demonstrations, comparisons, and rewards can be combined and revisited. The task distribution determines what the model practices; the objective determines what is encouraged; unfamiliar tasks test whether the result transfers.
Post-training develops behavior. Alignment asks, among other things, whether that behavior and the objectives behind it serve what we intend. The subjects overlap, but they ask different questions. Understanding how a model learns is the foundation for judging what we taught it.