The Map and the Walk

A paper map holds the accumulated knowledge of many earlier travelers, where the roads run, which towns sit along the river, roughly how far one is from the next. What it cannot say is whether the bridge washed out last week. Only walking the route reveals that, and a walker with no map at all can spend days discovering what any local could have explained in a sentence.

Modern AI rests on two learning methods that split the work in much the same way. A large language model, or LLM, is a neural network trained on an enormous archive of human writing to predict the next word, and in the process it absorbs a broad picture of how language, facts and arguments tend to fit together. Reinforcement learning, or RL, is learning by trial and error. A system tries an action, receives a score called a reward, and gradually shifts toward whatever earns more of it. The two are sometimes presented as competing visions of AI. In practice, many of the most capable systems in use today depend on both, and the reasons say a good deal about what each method can and cannot do alone.

Two Ways to Learn

RL's roots run through the study of animal learning. In 1981, Richard Sutton and Andrew Barto described a simple adaptive element that learned to respond in anticipation of a stimulus, much as Pavlov's dogs learned to salivate at a bell before the food arrived ^[1]. The core idea, learning from the gap between what was expected and what actually happened, became central to modern RL. Its standard form is the temporal difference error, which compares the reward just received, plus the estimated value of the situation the system landed in, against the value it had predicted for the situation it started from.

When things turn out better than expected, the error is positive and the estimate moves up, and when they turn out worse it moves down. The Greek letter gamma is a discount factor that makes distant rewards count for a little less than immediate ones.

Language models learn from a different kind of surprise. During pretraining, the model reads text one token at a time, a token being a word or a piece of a word, and is penalized according to how little probability it gave to the token that actually came next.

Scaled up far enough, that single objective produced surprising breadth. GPT-3, trained only to predict text, could pick up a new task from a handful of examples written into its prompt, with no further training at all ^[2].

Both methods, in other words, learn from prediction errors. The real differences lie in where the data comes from and what the feedback says.

The table hides an important asymmetry. A language model is shown the right answer at every position of every sentence, which makes its signal dense and plentiful but confines it to the archive it was given. An RL system often learns only whether a whole attempt succeeded, a sparse signal, yet it can keep generating fresh experience for as long as it keeps acting.

What Each Method Struggles With Alone

Prediction by itself does not produce a helpful assistant. The archive includes careless answers, confident mistakes and unpleasant text, and a model trained to imitate it reproduces some of each. The team behind InstructGPT put the problem plainly, observing that making language models bigger does not inherently make them better at following what a user actually wants ^[3]. Their remedy, reinforcement learning from human feedback, trained a separate model to predict which of two answers people preferred and then used RL to steer the language model toward answers that scored well. People preferred the outputs of the resulting model even when it was more than a hundred times smaller than the original GPT-3.

RL by itself has the opposite problem. A reward only teaches something once the system produces behavior worth rewarding. A system learning to write mathematical proofs from scratch could generate an enormous amount of nonsense before stumbling onto a single valid one, and until then the reward stays silent. Rewards are also hard to specify well. A system pushed hard against an imperfect score tends to find the gap between what the score measures and what its designers meant, a problem explored in an earlier Thinkata piece on reward design. From a systems perspective, a pretrained model eases the first problem directly. It starts with a strong prior, a built-in sense of what plausible text, code and plans look like, so the reward has sensible candidates to choose among from the very first attempt.

Where the Two Already Work Together

The InstructGPT recipe of pretraining, then a supervised stage, then RL became a widely adopted template for chat assistants, and an earlier piece on human correction loops looks at how that feedback can keep flowing after deployment. Reasoning models extend the same pattern. In 2025, a DeepSeek team reported in Nature that RL applied to a large pretrained model, rewarded only for whether its final answers to math and coding problems were correct, produced behaviors such as checking its own work and switching strategies partway through, without any human examples of how to reason ^[4]. The reward came from an automatic checker rather than from people, which makes it cheap to apply at scale wherever answers can be verified.

Robotics shows the division of labor even more plainly. Asked how to clean up a spill, a language model can describe a sensible plan, but it has no way of knowing whether a particular robot in a particular kitchen can actually reach the sponge. The SayCan system, presented at the Conference on Robot Learning, paired a language model with a library of robot skills, each with a value function, a learned estimate of how likely that skill is to succeed from where the robot currently stands ^[5]. The language model proposed steps that made sense for the task, the value functions favored steps that were feasible right now, and in the authors' words the robot acted as the language model's "hands and eyes."

The help also flows in the other direction. Writing a good reward function is one of the most tedious parts of applying RL, and Eureka, presented at ICLR in 2024, handed that job to a language model ^[6]. The model wrote candidate rewards as code, checked how well RL training went with each one, and revised them. Across a suite of simulated robot tasks, its rewards beat ones written by human experts on most of them, and they taught a simulated robot hand to spin a pen.

Why the Pairing Holds Up

Part of the case for the pairing comes from comparing RL with the other common way of adapting a pretrained model. A study presented at ICML in 2025 compared supervised fine-tuning, which trains a model to copy example answers, against RL trained on an outcome reward ^[7]. RL generalized to unseen variations of a card game and a navigation task, while supervised fine-tuning tended to memorize its examples. The same study found that the supervised stage was still needed first, because it stabilized the model's output format enough for RL to make progress.

A 2025 preprint adds a quieter advantage ^[8]. When a model learns a new task, RL appears to disturb its existing knowledge less than supervised fine-tuning does. Among the many ways to solve the new task, on-policy RL, which learns from the model's own outputs, tends to settle on the one that changes the model least. For teams that care about keeping a pretrained model's general abilities intact while teaching it something specific, that is a practical reason to reach for RL at the final step.

The pairing has limits, and the clearest ones come from asking how far the walk can stray from the map.

Counting the Corridors

From above, every route through a maze is visible at once, including the few corridors that reach the center. A NeurIPS 2025 study looked at reasoning models from roughly that vantage point, giving each model many attempts per problem and counting a problem as solved if any attempt succeeded ^[9]. With one attempt, the RL-trained models clearly beat the base models they came from. With many attempts, the base models solved more. The authors concluded that the observed reasoning abilities "originate from and are bounded by the base model," suggesting RL mostly taught the model to take a corridor it already knew more reliably. A 2026 preprint offers a partial counterpoint, finding that some problems the base model never solved did become solvable during RL training, and that focusing training on those problems pushed results past the base model ^[10].

Taken together, the evidence suggests that RL is most useful as a way to select and sharpen what pretraining already makes possible, with a smaller and still debated capacity to reach beyond it. Anyone who has built a search system will recognize the shape. A retrieval stage gathers candidates and a ranking stage picks among them. A good ranker cannot surface a document the retriever never found, and a large candidate set is wasted without a ranker that can tell good from bad. Neither stage makes the other redundant.

Looking Ahead

The balance between the two may shift. A 2024 analysis presented at ICML projected that, if trends held, training datasets would reach roughly the size of the entire stock of public human-written text sometime between 2026 and 2032 ^[11]. Two prominent RL researchers see that as a turning point.

"In key domains such as mathematics, coding, and science, the knowledge extracted from human data is rapidly approaching a limit."

Silver and Sutton, Welcome to the Era of Experience, 2025

David Silver and Richard Sutton argue that agents will increasingly generate their own training data by acting and learning from the results ^[12]. If that happens, it seems likely to lean on both methods at once, a pretrained model to make early attempts sensible and RL to learn from how those attempts turn out. The open question is how far experience can carry a model in domains where no automatic checker exists to say whether an answer was right.

For teams building with these methods now, a few habits follow. Single-attempt accuracy can hide a narrowing of what a model is able to do, so tracking performance across many attempts is a cheap safeguard. Rewards deserve the same scrutiny as any other specification, since a model trained against them optimizes what was written rather than what was meant. And the quality of the base model sets much of the ceiling for what fine-tuning can reach, which makes the choice of starting point one of the most consequential decisions in the pipeline.

The map and the walk seem to work best as a pair. A map with no one to walk it stays a guess about the terrain, and a walk with no map is mostly wandering. Most of the progress described here came from putting the two together.