Post-training is the new Pretraining
A freshly pretrained model is not completely useful to talk to.
It completes text. It doesn’t answer questions.
It has all the knowledge and none of the manners.
This gap is bridged by post-training and it’s primarily done in 3 stages
Stage 1: Supervised fine-tuning (SFT)
The first fix is simple: show the model examples of the behavior you want.
Collect thousands of high-quality prompt-and-response pairs i.e. questions with good answers, instructions with good completions and fine-tune on them.
SFT is what turns a text-completer into an instruction-follower.
It’s cheap relative to everything else, and it’s still the first stage everywhere.
But it has a ceiling: the model can only get as good as the examples you show it, and writing perfect examples for every behavior is impossible.
Stage 2: Preference optimization (RLHF and DPO)
This is thesis for modern assistant era: humans can easily pick the better of two.
So instead of showing the model what to say, you show it pairs of its own outputs and tell it which one a human preferred. Then you optimize the model toward the preferred kind.
RLHF (Reinforcement Learning from Human Feedback) is the original method.
Train a separate “reward model” on human preferences, then use reinforcement learning to push the main model toward outputs the reward model scores highly.
Powerful, but operationally heavy, you’re juggling multiple models and a famously finicky RL loop.
DPO (Direct Preference Optimization) arrived as the simplification
Skip the reward model entirely and optimize directly on the preference pairs.
Much simpler to run, close enough in quality for many uses.
For most teams outside frontier labs, DPO and its variants became the practical default.
Preference optimization is what gives a model its judgment i.e. character such as helpfulness, tone, knowing when to refuse.
Stage 3: Reasoning RL
The idea: some tasks have verifiable answers.
A math problem is always right or wrong. Code passes its tests or doesn’t. For these tasks, you don’t need human preferences, you can let the model generate long chains of reasoning, automatically check whether the final answer is correct, and reinforce the reasoning paths that worked.
DeepSeek’s R1 paper (January 2025) showed reasoning capability emerging from large-scale RL on a pretrained base, with minimal supervised data.
GRPO, the RL method behind it, became the technique of choice, simpler and cheaper than classic RLHF-style training, designed for exactly this verifiable-reward setting.
The result is the “thinking/reasoning” behavior you now see everywhere. Models that work through a problem step by step before answering, because the RL rewarded chains of thought that reached correct answers.
The solutions architect lens
Post-training has a different infrastructure profile than pretraining, and it’s worth knowing why.
RL is a mixed workload. Every training step requires the model to first generate outputs (an inference workload), then score them, then update weights (a training workload). Your cluster is doing both at once.
The bottleneck is evals, not compute. Verifiable rewards need verifiers - test harnesses, checkers, graders. Preference optimization needs preference data. The teams that move fast at this stage are the ones with good evaluation infrastructure, not the ones with the most chips.
Most enterprises consume post-training rather than do it. The models you get through an API or a model catalog have all three stages baked in. Where enterprises do touch this layer, it’s usually light preference tuning (DPO on a few thousand pairs) on top of an already post-trained model.
What I’d take away
A pretrained model is knowledge without manners.
SFT teaches it to follow instructions
Preference optimization gives it judgment
Reasoning RL gives it the ability to think before answering.
Verifiable rewards changed the economics. Where answers can be checked automatically, be it math, code - capability can be trained without humans labeling.
Evals are the real bottleneck. The constraint at this layer is knowing whether outputs are good, not generating them.
Next week: evaluation - the unglamorous discipline that decides whether any of this actually worked. Benchmarks, eval harnesses, and why the gap between a model leaderboard score and your production workload is bigger than you think.
If you want it in your inbox, subscribe.