How Models Learn (It's Simpler Than You Think)
We start at the beginning — pretraining.
Here’s the thing: the core of how a frontier model is built is almost embarrassingly simple to state.
Show the model an enormous amount of text.
Cover up the next word.
Ask it to guess - Tell it whether it was right.
Adjust and Repeat, trillions of times.
That’s Pretraining.
Everything a large language model “knows” - grammar, facts, reasoning patterns, the ability to write code, emerges from this one repetitive exercise of predicting the next token.
If you’ve been following my previous articles on infrastructure series, understanding what actually happens during pretraining is what makes those infrastructure decisions make sense.
What the model is actually learning
A language model is, mechanically, a next-token predictor. You give it a sequence of tokens and it outputs a probability distribution over what the next token should be. That’s the whole interface. To predict the next token well across the entire internet, the model is forced to learn:
Grammar and syntax
Facts
Reasoning
Style and register
Nobody programs these capabilities in. They emerge as a side effect of relentless next-token prediction at scale. The mechanism for “adjusting” when the model guesses wrong is backpropagation - model computes how wrong it was (the loss), then calculates how each of its billions of parameters contributed to that error, then nudges each parameter slightly in the direction that would reduce the error. Do this across trillions of tokens and the parameters settle into values that encode all that emergent knowledge.
From an infrastructure lens, this is why training is the compute-bound, throughput-hungry workload. Every token requires a forward pass (the guess) and a backward pass (the adjustment), and the backward pass roughly doubles the compute. You’re running this across trillions of tokens. The arithmetic is enormous, regular, and parallel -exactly the workload accelerators are built for.
The data and the scale
The data. A frontier model trains on trillions of tokens - a meaningful fraction of the high-quality text that exists.So an enormous amount of work goes into the data pipeline before training even starts: deduplication, quality filtering, removing toxic content, balancing the mix between domains, handling multiple languages. The quality of this pipeline matters as much as the model architecture. Two teams with the same architecture and the same compute will get different models if their data pipelines differ.
Processing trillions of tokens is a true infrastructure problem: cleaning, filtering, tokenizing, deduplicating is a massive distributed data engineering job that runs before a single training step. Therefore the data pipeline often needs as much thought as the training cluster.
The scale. Frontier pretraining runs use tens of thousands of accelerators for weeks to months. The cost lands in the tens to hundreds of millions of dollars for the largest models, dominated by accelerator time. This is the workload that needs the scale-out network, the collective communication, the checkpointing strategy, all the hard parts I covered in infrastructure series.
A pretraining run is the single most demanding thing you can do with AI infrastructure and it’s why most organizations never pretrain from scratch. The capability landscape splits cleanly:
A handful of frontier labs and hyperscalers pretrain foundation models
Almost everyone else starts from a pretrained model and adapts it
Pretraining is the foundation, but fine-tuning and post-training is what most teams actually do.
What a training run actually looks like
A pretraining run is a long-lived distributed job across thousands of chips
Checkpoint-and-resume is the normal operating mode, not an exception. The model state is saved every few hundred steps so a failure doesn't cost weeks of progress.
Teams monitor the loss curve i.e. the measure of how wrong the model is. A sudden spike ("loss divergence") can mean a bad batch of data, a numerical instability, or a hardware fault, and catching it early saves enormous amounts of money.
Uses every parallelism trick at once. The model is split across chips with tensor parallelism, pipeline parallelism, and data parallelism simultaneously. Getting this mapping right is the difference between 30% and 90% utilization, which at frontier scale is the difference of tens of millions of dollars.
Demands careful choreography. Model learning happens automatically. The hard part is keeping expensive distributed system running continuously for weeks without losing too much progress to failures.
This is why pretraining is the domain of dedicated platform teams that possess deep operational maturity around distributed systems, failure recovery, and cluster utilization.
Two things that complicate the simple story
Most frontier models no longer use all their parameters. Most frontier models in 2026 leverage a Mixture of Experts (MoE) architecture.
The model has many specialized sub-networks (”experts”), and a small router sends each token to only a few of them.
A model like Llama 4 Scout has 109 billion total parameters but only activates about 17 billion per pass, and DeepSeek’s V-series uses 671 billion total parameters while only activating roughly 37 billion per token.
Every frontier LLM released in the past year uses Mixture-of-Experts.
This matters because it breaks the simple memory math. You still need enough memory to hold all the parameters, but the compute and bandwidth per token are set by the much smaller active count. It also introduces a new parallelism strategy - expert parallelism, where experts are spread across chips, which stresses the network.
A lot of capability now comes after pretraining.
Reasoning models get much of their capability from reinforcement learning applied after pretraining.
DeepSeek’s R1 paper in January 2025 showed that reasoning capability can emerge from large-scale RL applied to a pretrained base model, with minimal supervised fine-tuning.
By 2025, post-training had become a major cost center and at some labs, post-training compute now rivals pretraining.
Key Takeaways
Pretraining justifies the infrastructure - but no longer owns the capability.
The foundation comes from pretraining but the usefulness increasingly comes from what happens after.
Almost nobody should pretrain from scratch.
Unless you’re a frontier lab with a unique dataset and tens of millions to spend, you start from a pretrained model.
The data pipeline is infrastructure too.
Preparing the tokens is as much engineering as running the cluster.
Utilization is the metric that pays.
At frontier scale, the gap between 30% and 90% utilization is tens of millions of dollars.
Next week: fine-tuning. Once you have a pretrained model — yours or someone else’s — how do you adapt it to actually do what you need? SFT, LoRA, and the question every team eventually asks: fine-tune, prompt, or retrieve?
If you want it in your inbox, subscribe.