Reasoning LLM · RL · 2025 / arXiv
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
A structured reading of the paper that separates the scientific R1-Zero claim from the product-oriented DeepSeek-R1 training report.
Overview
Read this paper as two documents interleaved together. One document is a technical argument about : can reinforcement learning make a base LLM discover better reasoning behavior without first imitating human-written reasoning traces? The other document is a model report about a public product model: how DeepSeek-R1 was trained, evaluated, distilled, released, and safety-checked.
The most important technical object is . It is trained directly from using and . It is not polished, but it is the paper's cleanest evidence that RL can incentivize , self-checking, reflection, and strategy switching.
The final model, , answers a different question: after raw reasoning behavior appears, how do we turn it into a readable, useful assistant? That requires , more RL, , large-scale , preference rewards, and safety mechanisms.
Why this paper exists
The background problem is that strong reasoning models before R1 were often associated with heavy supervision, carefully crafted reasoning traces, or closed training recipes. The standard post-training story was roughly: first use SFT to teach the model how good answers look, then use RL or preference optimization to align the model further.
DeepSeek-R1 asks whether that order hides a limitation. Human-written chain-of-thought can make models readable and controllable, but it also tells the model what kind of reasoning to imitate. If human traces omit verification, backtracking, or strange but useful non-human search patterns, SFT may cap exploration.
For tasks where the final answer can be checked reliably, we may not need to teach the model every reasoning step. We can give it hard problems, a reliable verifier, and enough RL compute. Then successful reasoning behaviors may become more likely because they lead to correct final answers.
This is why the word incentivizing in the title matters. The authors are not primarily saying, "we wrote better reasoning demonstrations." They are saying, "we designed a reward environment where useful reasoning behavior becomes advantageous."
R1-Zero vs R1
A lot of confusion disappears once you separate R1-Zero from R1. R1-Zero is closer to a controlled research experiment. R1 is closer to a productized assistant model.
| Model | Role in the paper | Training recipe | Main strength | Main weakness |
|---|---|---|---|---|
| DeepSeek-R1-Zero | Scientific evidence | DeepSeek-V3-Base + GRPO, without conventional SFT before RL | Strong reasoning on verifiable tasks; long CoT and reflection emerge during RL | Poor readability, language mixing, weak general assistant behavior |
| DeepSeek-R1 | Final assistant model | Cold-start SFT + reasoning RL + rejection sampling + large SFT + final mixed RL | Reasoning plus better readability, instruction following, helpfulness, and safety alignment | More complex pipeline; no longer a pure-RL evidence model |
| R1 distilled models | Open-source transfer | SFT smaller Qwen/Llama models on R1-generated 800K samples | Transfers long-CoT behavior to smaller models at much lower deployment cost | Distillation inherits the teacher's behavior rather than proving pure RL works on small models |
The headline technical claim should be attached to R1-Zero: a strong base model can develop better reasoning behavior through RL on verifiable tasks. The headline deployment claim should be attached to R1: a reasoning model also needs readability, instruction following, helpfulness, harmlessness, and general assistant data.
Training pipeline
The full R1 pipeline is not a single training trick. It is a sequence of stages, each repairing a different failure mode. The pipeline is easiest to read as a trade-off between free exploration and human-facing usability.
The first stage is the scientific proof: R1-Zero asks whether a strong base model can develop reasoning through outcome-based RL alone. The later stages are there to make that raw reasoning usable: cold-start SFT improves readability, the next RL stage restores reasoning strength, large SFT broadens assistant behavior, and final mixed RL aligns the finished model.
| Stage | Input | Training signal | Why it exists |
|---|---|---|---|
| 1. R1-Zero RL | DeepSeek-V3-Base + reasoning prompts | Accuracy reward + format reward | Test whether reasoning can emerge from RL without SFT demonstrations. |
| 2. Cold-start SFT | Thousands of curated, human-readable long-CoT examples | Next-token imitation | Give R1 a readable and language-consistent starting style before RL. |
| 3. First R1 RL | Reasoning prompts | Rule reward + language consistency reward | Recover and improve reasoning while reducing language mixing. |
| 4. Rejection sampling + SFT | Correct model-generated reasoning outputs + non-reasoning assistant data | Next-token imitation over about 800K samples | Build a broad assistant that can reason but still write, answer, translate, and follow instructions. |
| 5. Final mixed RL | Reasoning and general prompts | Rule reward + preference reward + safety reward + language reward | Align the final R1 model for both reasoning performance and user-facing helpfulness/harmlessness. |
This is also why the paper is not simply anti-SFT. It is anti-depending entirely on human reasoning traces for the emergence of reasoning. Once the raw reasoning behavior exists, SFT becomes useful again for readability, format, instruction following, and broad assistant behavior.
GRPO mechanics
is the algorithmic engine, but the paper is not a GRPO theory paper. The important question is how GRPO makes final-answer rewards usable for long chain-of-thought training.
In for LLMs, one usually trains a and a value model. The value model estimates how much reward to expect from a partial response. That is expensive for very large models and conceptually hard for long CoT: a partial solution may look wrong, then later be corrected; or it may look right, then collapse.
GRPO removes the separate value model. Instead, for each question, it samples a group of candidate answers. Each answer gets a reward. The algorithm then asks: which sampled answer was better than the group's average?
| GRPO component | What it does | Why it matters for R1-Zero |
|---|---|---|
| Group sampling | For the same question q, sample a group of G outputs from the old policy. | The group gives a local comparison set: which response is better than the others for this exact problem? |
| Group-relative advantage | Standardize each reward by subtracting the group mean and dividing by group standard deviation. | This replaces the value model used in PPO and makes advantage relative to prompt difficulty. |
| Probability ratio | Compare how likely the current policy is to generate an output against how likely the old rollout policy was. | A positive-advantage output should become more likely; a negative-advantage output should become less likely. |
| Clipping | Limit how much the probability ratio can affect one update. | This stabilizes training so one high-reward batch does not move the huge model too far at once. |
| KL penalty | Penalize divergence from a reference policy. | It prevents the model from drifting into unnatural language or reward-exploiting behavior. |
GRPO turns one prompt into a small competition among the model's own answers. The answers that beat the local group average get reinforced; the answers that fall below the local group average get discouraged.
GRPO math
The paper's GRPO objective can look intimidating, but it is built from a few simple pieces. For a question
q, sample a group of outputs from the old policy:
{o1, o2, ..., oG} ~ πθ_old(O | q)
Score every output with the reward function:
{r1, r2, ..., rG} Then compute a standardized group-relative advantage:
Ai = (ri − mean({r1, ..., rG})) / std({r1, ..., rG})
This is the most important formula in the section. A positive means the output was better than its peers for the same prompt. A negative advantage means it was worse. This replaces PPO's learned value model with a group baseline.
The update then compares the current policy probability with the old probability:
ratioi = πθ(oi | q) / πθ_old(oi | q)
If Ai > 0, increasing this ratio is good: the model should become more likely to produce
this kind of answer. If Ai < 0, increasing the ratio is bad: the model should move away from
this answer. Clipping prevents the ratio from changing too aggressively.
min(ratioiAi, clip(ratioi, 1−ε, 1+ε)Ai) − βDKL(πθ || πref)
The final term is the KL penalty. It discourages the current policy from drifting too far from the reference policy. Without this anchor, a reward-maximizing language model can become unstable, repetitive, or overly specialized to the reward format.
Positive advantage
Ai > 0The response scored above the group average. GRPO pushes the model to make it more likely.
Negative advantage
Ai < 0The response scored below the group average. GRPO pushes the model to make it less likely.
No value model
group mean ≈ baselineThe group reward distribution substitutes for a learned critic, saving memory and simplifying long-CoT RL.
Reward design
The paper's real technical bottleneck is not the GRPO formula alone. It is the . RL only optimizes what the reward signal measures. If the reward is reliable, the model can learn useful reasoning. If the reward is flawed, the model can learn shortcuts.
| Reward | Used in | How it works | Risk or limitation |
|---|---|---|---|
| Accuracy reward | R1-Zero and R1 reasoning RL | Math answers are matched against ground truth; code is evaluated with compilers/tests; multiple-choice answers are matched to the correct option. | Strong when the answer is objectively verifiable; weak for open-ended writing or subjective tasks. |
| Format reward | R1-Zero and R1 | Encourage the model to put reasoning and final answers into required tags or formats. | Useful for parsing, but format alone does not guarantee real reasoning. |
| Language consistency reward | R1 training | Reward is based on the proportion of words in the target language within the CoT. | Improves readability and user experience, but can slightly hurt raw reasoning benchmark performance. |
| Helpfulness reward model | Final R1 mixed RL | A learned model scores preference pairs for general assistant responses. | Can be reward-hacked if optimized too long or if the reward model has systematic biases. |
| Safety reward model | Final R1 mixed RL | A classifier-style reward model distinguishes safe and unsafe responses. | Safety is harder to define than math correctness and usually needs deployment-time risk control too. |
R1-Zero uses a deliberately simple :
Rewardrule = Rewardaccuracy + Rewardformat This is why verifiable tasks matter. In math, code, and multiple-choice reasoning, the final answer can often be checked automatically by a . The model is not told exactly how to reason; it is rewarded when its final answer is correct and its output is parseable.
The appendix's example is a warning: when a learned helpfulness reward model is optimized too long, the reward can keep increasing while real Codeforces performance decreases. This is one reason the paper treats rule-based rewards as the backbone of reasoning RL.
Data and verifiers
R1-Zero is often summarized as "pure RL," but the training environment is carefully chosen. The prompts are mostly from domains where correctness can be checked. That is what makes outcome-based RL practical.
| Category | Approximate size | Task type | Verifier |
|---|---|---|---|
| Math | 26K prompts | Quantitative reasoning with numeric, expression, or equation answers | Answer matching and symbolic parsing where possible |
| Code | 17K algorithm prompts + 8K bug-fixing prompts | Competitive programming and code repair | Compiler, unit tests, hidden tests, or issue-specific tests |
| STEM | 22K prompts | Physics, biology, chemistry, medicine-style multiple choice | Correct option matching |
| Logic | 15K prompts | Real and synthetic logic puzzles, code-IO reasoning, arithmetic puzzles | Option or final-answer matching |
| General | 66K helpfulness prompts + 12K safety prompts | Writing, editing, factual QA, role play, harmlessness | Model-based preference or safety reward, not a deterministic verifier |
This table is the practical boundary of the method. Math and code are good RL domains because they produce crisp feedback. Writing and open-ended dialogue are harder because there is no single correct answer. In those domains, R1 uses SFT and reward models, but the paper is much more cautious about reward hacking.
From GRPO formula to training loop
The GRPO objective explains what is optimized, but the paper also gives an important engineering view of how reinforcement learning is actually executed at scale. The figure contains rollout workers, reward modules, training modules, and VRAM management. For understanding the paper, the essential loop is simpler: generate several attempts, score them, compare them within the group, and update the policy.
This is the operational meaning of GRPO in R1-Zero. The system does not need a human-written reasoning trace for each problem. It needs a strong base model, many sampled attempts, and reliable ways to score the final outcome. Once the rewards are available, GRPO turns the group comparison into a policy update.
The policy model receives reasoning prompts and samples multiple completions for the same question. In R1-Zero, this group of completions is the raw material for learning: some attempts will solve the problem, some will fail, and GRPO learns from the contrast between them.
The outputs are scored by a combination of rule-based checks and model-based components. For the core reasoning tasks, the most important signals are verifiable: final-answer matching for math, test execution for code, and format checking for the required response structure.
Instead of asking a value model to estimate future reward, GRPO compares outputs within the same group. A response receives positive advantage if its reward is above the group average and negative advantage if it is below the group average.
The training step increases the probability of high-advantage outputs and decreases the probability of low-advantage outputs, while a KL term keeps the model from drifting too far from a reference policy.
Self-evolution evidence
The most interesting claim is behavioral: RL does not merely raise benchmark accuracy; it changes how the model solves problems. During R1-Zero training, the average response length increases, and the model begins to use more reflective language such as wait, check, verify, wrong, and retry.
Longer thinking
The model learns to spend more tokens before final answers. Length itself is not directly the reward; longer reasoning survives because it helps produce correct answers on hard problems.
Reflection
The model increasingly generates phrases that indicate checking and correction. This suggests that RL reinforces behaviors that help the model recover from wrong intermediate paths.
Adaptive difficulty
Later analysis shows that harder problems tend to receive more thinking tokens, connecting long CoT to test-time compute rather than simple verbosity.
The famous "aha moment" should not be read as proof of human-like consciousness. It is better read as a generated behavior pattern that becomes useful under the reward environment. The model learns that pausing, reconsidering, and revising can lead to correct final answers.
Benchmarks and metrics
The experiment section can feel like a product scoreboard. Benchmarks such as , , and are useful only if we read them as evidence for different kinds of reasoning. For a technical reading, focus on what each benchmark is testing and why it matters for the R1-Zero-to-R1 story.
| Benchmark | Type | What it measures | Why it matters |
|---|---|---|---|
| AIME 2024 / AIME 2025 | Math competition | Hard multi-step quantitative reasoning with exact final answers. | AIME is the headline proof that RL improves mathematical reasoning and that repeated sampling can raise consensus accuracy. |
| MATH-500 / CNMO | Math benchmark | Broader mathematical problem solving, including contest-style reasoning. | These show the effect is not limited to one AIME split. |
| LiveCodeBench / Codeforces | Algorithmic coding | Can the model derive and implement algorithms under contest-style constraints? | They are good fits for RL because code has an executable verifier. |
| SWE-Bench Verified / Aider | Software engineering | Can the model modify real repositories or produce useful code patches? | R1 improves less here because RL feedback is expensive and slow for repo-level tasks. |
| GPQA Diamond | Graduate-level STEM | Difficult biology, physics, and chemistry questions. | It tests whether reasoning helps in knowledge-heavy scientific domains. |
| MMLU / MMLU-Pro / C-Eval / CMMLU | Knowledge and academic QA | Broad subject knowledge and exam-style reasoning, in English and Chinese. | They show whether reasoning training damages or improves general academic ability. |
| IFEval / AlpacaEval / ArenaHard | Instruction following and human preference | Formatting compliance, helpfulness, open-ended answer quality, and user preference. | They reveal why R1-Zero is not enough and why final R1 needs SFT and preference alignment. |
| SimpleQA / C-SimpleQA / FRAMES / DROP | Factuality and document reasoning | Short factual answers and reasoning over provided context. | They test whether long reasoning also helps with factual and retrieval-like tasks. |
Two metrics are especially important. measures whether one sampled answer is correct. measures whether repeated samples can be combined by consensus. R1-Zero's strong pass@1 means the model is not relying only on majority voting; consensus then adds another layer of test-time compute.
R1-Zero is strong on math and code but weak on instruction following and preference benchmarks. Cold-start and SFT improve usability but can temporarily reduce raw reasoning. Later RL restores reasoning, and final mixed RL mainly improves user-facing benchmarks such as AlpacaEval and ArenaHard.
Failed routes
the final design easier to understand: alternatives such as and may look more elegant, but GRPO plus outcome reward is more scalable.
| Attempt | Why it was attractive | What went wrong | Lesson |
|---|---|---|---|
| Process Reward Model | Give fine-grained feedback to each reasoning step instead of only the final answer. | Hard to define a universal reasoning step, hard to judge step correctness, and vulnerable to reward hacking once the process reward is model-based. | Outcome reward is coarser, but more scalable and more reliable for verifiable tasks. |
| Monte Carlo Tree Search | Search over alternative reasoning branches instead of committing to one long response. | Token generation has an enormous search space, and the value model guiding the search is difficult to train reliably. | MCTS can help at inference with a good value model, but it is hard to use as the main self-improvement training loop. |
| Small-model pure RL | Train a smaller model directly with the same RL idea. | Smaller models may produce longer outputs without better reasoning, often falling into repetition rather than useful verification. | Strong base checkpoints matter. Distillation from R1 is often cheaper and stronger for smaller models. |
This is the paper's deeper engineering philosophy: do not choose the method that gives the most beautiful supervision signal on paper; choose the method whose feedback is reliable and whose training loop can scale.
Limits and product-report parts
DeepSeek-R1 is also a public model report, not only a technical paper about R1-Zero. That is why the paper spends space on safety, deployment, distillation, cost, and release details. These parts are useful for understanding the model as a released system, but they are not the central mechanism behind how GRPO elicits reasoning.
Stronger reasoning is a double-edged sword. The same ability that helps a model break down a math problem can also make unsafe requests more dangerous, because the model may produce more concrete, executable plans.
This is why the report includes a long safety appendix and discusses an external risk-control system. For the technical story of R1-Zero, safety is not the main mechanism, but for a public model release it is part of the system-level picture.
Long chain-of-thought is one of the main behaviors encouraged by RL. On hard problems, more thinking tokens can give the model room to verify, backtrack, and try alternative strategies.
The limitation is overthinking. If the model spends thousands of tokens on simple questions, reasoning becomes slow, expensive, and sometimes less user-friendly. A better reasoning model should learn when to think long and when to answer directly.
R1 reasons mainly inside text. It does not natively use search engines, calculators, compilers, or other external tools as part of its reasoning loop.
This matters because many real tasks should not be solved by internal reasoning alone. Search can check fresh facts, calculators can prevent arithmetic mistakes, and compilers can verify code. Tool-augmented RL is therefore a natural next step beyond R1-style text-only reasoning.
is not the core proof of the paper. The core proof is R1-Zero: a strong base model can develop reasoning behavior through RL on verifiable tasks.
But distillation is a major release contribution. Once a large model has learned R1-style reasoning, its generated trajectories can train smaller Qwen and Llama models. In that sense, distillation turns the expensive RL result into something more accessible for the open-model ecosystem.
PPO vs. GRPO
DeepSeek-R1 is not a paper about inventing GRPO, but GRPO is the key optimization method that makes R1-Zero possible. The important comparison is therefore not just what GRPO is, but why the authors use it instead of PPO for long-chain reasoning.
PPO estimates advantage with a separate value model, then combines reward, value estimates, and a KL constraint to update the policy. GRPO removes the value model. For each question, it samples a group of outputs, scores them, and computes relative advantages within the group. This is why the “G” in GRPO stands for group.
PPO is a standard policy optimization method. In the LLM setting, the policy model generates an output, a reward model scores that output, and a value model estimates how good the trajectory is expected to be. Advantage is then computed with techniques such as GAE, and the policy is updated while staying close to a frozen reference model by a KL penalty.
The strength of PPO is that it is general. The weakness, for this paper, is that the value model becomes costly and brittle when reasoning traces are long. In long chain-of-thought settings, it is hard to estimate the value of a partially generated solution: a response may look wrong halfway through and later self-correct, or look promising early and still fail at the end.
GRPO keeps the policy optimization flavor of PPO, but replaces the value-model-based advantage estimator with a simpler group-relative one. For the same question q, the old policy samples a group of outputs o1, o2, …, oG. Each output gets a reward ri. GRPO then compares each reward to the group mean and normalizes by the group standard deviation:
Ai =
(ri − mean(r1, ..., rG))
/ std(r1, ..., rG)
So advantage no longer means “how good is this output in absolute terms?” It means “how good is this output relative to the other attempts for the same question?” Outputs above the group average get positive advantage and become more likely; outputs below average get negative advantage and become less likely.
This design fits R1-Zero well. The paper mainly trains on verifiable tasks such as math, code, STEM, and logic. Those tasks give relatively clear outcome rewards: the answer is correct or not, the code passes tests or not, the format is valid or not. GRPO can therefore skip a large value model and still get a usable training signal by comparing a batch of attempts to the same problem.
In other words, DeepSeek-R1 does not use GRPO because GRPO is fashionable. It uses GRPO because, for long reasoning with reliable outcome rewards, group-relative comparison is cheaper, simpler, and better matched to the problem than value estimation in PPO.
PPO asks a value model to predict how promising a trajectory is; GRPO lets several attempts at the same question compete against one another and uses that relative ranking as the learning signal.
Takeaways
- The clean scientific claim belongs to R1-Zero: reasoning can be incentivized by RL on verifiable tasks without prior SFT reasoning demonstrations.
- The final product claim belongs to R1: raw reasoning must be made readable, helpful, safe, and general through a multi-stage pipeline.
- GRPO matters because it removes the value model and estimates advantage from a group of sampled answers for the same prompt.
- Reward design is the central bottleneck. Reliable verifiers make RL powerful; weak reward models invite reward hacking.
- Long CoT is not directly the objective. It emerges because longer reasoning, checking, and revision help the model get higher final-answer reward.
- Strong base model, reliable verifier, and iterative SFT/RL pipeline are the real success conditions.
- DeepSeek-R1 is partly a research paper and partly a model report. Safety, release, and deployment details matter, but they are not the core technical mechanism.