LLM · RL for NLU · 2025 / arXiv
Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning
A structured reading of a paper that reframes natural language understanding as a reinforcement-learning problem for instruction-tuned LLMs.
Overview
Read this paper as a paper about task alignment, not as a paper about a new model architecture. The authors do not invent a new Transformer. Instead, they ask whether an instruction-tuned LLM can be made better at by training it with task rewards rather than only with supervised answer imitation.
The paper starts from a surprising observation: smaller encoder-only models such as BERT can still beat 7B-scale instruction-tuned LLMs on classic NLU benchmarks. A model like LLAMA2-7B-chat-hf can follow instructions and generate fluent text, but when it is asked to solve GLUE or SuperGLUE tasks with only prompting, it often produces unstable, biased, or poorly formatted answers.
The authors' solution is to turn NLU into a reinforcement-learning environment. The model receives a prompt and generates an answer token by token. The generated answer is extracted from a fixed format, compared with the ground-truth label, and converted into a scalar . Then updates the model so that answer paths receiving higher reward become more likely.
The problem
The paper is built around a distinction that is easy to miss: generation ability is not the same as benchmarked understanding ability. Decoder-style LLMs are trained to continue text. GLUE and SuperGLUE often ask for precise judgments: positive or negative, entailment or contradiction, acceptable or unacceptable, similarity score, correct option, answer span.
A prompted LLM must solve these tasks indirectly through generation. It may answer in the wrong format, over-explain instead of selecting a label, or show a class preference. Encoder-only models such as BERT are structurally closer to the benchmark interface: encode the input, read a representation, output a label.
Under-14B instruction-tuned LLMs have broad linguistic knowledge, but prompting does not reliably turn that knowledge into stable NLU decisions. The authors argue that direct reward optimization can align the model's generation behavior with the task metric more effectively than prompting or ordinary supervised fine-tuning.
This is why the paper compares against both BERT and LLM baselines. BERT asks, "How strong is a model designed for understanding?" asks, "How far can the LLM get with only a task instruction?" asks, "How far can the LLM get when a few labeled examples are placed inside the prompt, but no weights are updated?" asks, "Is ordinary supervised adaptation enough?" PPO asks, "What if we optimize the final task reward directly?"
Background concepts
Before getting into PPO, it helps to separate two kinds of names that appear throughout the discussion: benchmark names, which tell us what kind of understanding ability is being tested, and reinforcement-learning terms, which tell us how the model is being updated.
| Name | Role | What it is | Why it matters here |
|---|---|---|---|
| GLUE | Standard NLU benchmark suite | Sentiment, paraphrase, acceptability, semantic similarity, and inference tasks. | The main proof that PPO can improve LLM performance on classic language-understanding tasks. |
| SuperGLUE | Harder NLU benchmark suite | Reading, commonsense, coreference, word-sense, and reasoning-heavy tasks. | A stress test showing the method is not only tuned to the easier GLUE setting. |
| Financial PhraseBank / Labelled Financial News | Sentiment generalization datasets | Financial-domain sentiment classification. | They test whether SST-2 PPO fine-tuning transfers beyond movie-review sentiment. |
| Mental Health / Emotion | Affective-text generalization datasets | Sentiment or emotion-related text classification. | They broaden the out-of-domain sentiment evaluation. |
| Babi-NLI / SIGA-NLI | NLI generalization datasets | Inference-style datasets, with SIGA-NLI focusing on scalar implicature and gradable adjectives. | They test whether MNLI PPO fine-tuning transfers to other inference problems. |
| SQuAD | Reading comprehension appendix task | Question answering by selecting an answer span from a passage. | It shows that the RL framing can use task metrics such as Exact Match and F1 as rewards, not only label matching. |
The most important reinforcement-learning translation is simple: an LLM can be treated as a . In ordinary RL, a policy gives the probability of taking an action in a state. In language generation, the state is the current token prefix, and the action is the next token. PPO updates the model by changing how likely it is to generate certain tokens after certain prefixes.
Policy-based RL
pθ(a | s)
Policy-based methods directly optimize the action-selection rule above. This matches language models naturally because a decoder-only LLM already outputs a probability distribution over the next token.
Value-based RL
Q(s, a)
Value-based methods learn a value function, then choose actions according to that value. This is less convenient for LLM generation because the action space is the whole vocabulary, and one answer is a long sequence of actions rather than a single decision.
Actor-critic
Vt = V(st)
The actor is the language model that generates tokens. The estimates how much future reward is expected from the current partial answer. That estimate helps PPO decide whether a generated token was better or worse than expected.
The method
The method has two parallel fine-tuning routes. In SFT, the training sequence contains the prompt, the input, and the ground-truth answer. The model is trained by next-token prediction, so it learns to imitate the correct answer text. In PPO, the model first generates an answer, then the environment scores that answer. The model is updated according to the reward.
| Setup | Do weights change? | Input form | Purpose |
|---|---|---|---|
| Zero-shot prompting | No weight update | Task instruction + query | Tests whether instruction-following alone is enough. |
| Few-shot prompting | No weight update | Task instruction + 1–5 labeled examples + query | Tests whether in-context examples can induce the task behavior. |
| SFT | LoRA weights updated | Prompt + query + ground-truth answer | Trains the model to imitate correct answer text with next-token prediction. |
| PPO | LoRA policy weights updated; value head updated | Prompt + query; model generates answer, then receives reward | Trains the model to generate responses that score well under the task reward. |
The prompt format matters because the reward function needs an answer it can parse. The paper asks the
model to place the final decision inside tags such as <Judgement>...</Judgement>.
This turns an open-ended generation problem into a scorable answer-selection problem.
Example: SST-2 as RL
Prompt: Determine whether the sentence is positive or negative. Return only the answer in <Judgement> tags.
Query: This movie is wonderful.
Generated output: <Judgement>Positive</Judgement>
Reward: If the ground-truth label is positive, the response receives high reward. If the model outputs negative, it receives lower reward. If it fails to produce a parseable judgment, it receives the format penalty.
How PPO actually runs
PPO is easiest to understand if we separate two phases that are often mixed together in the formula: rollout and update. During rollout, the current model generates answers and records the tokens it sampled. During update, PPO reuses those recorded tokens to change the trainable parameters. The new policy is not created before training starts; it is created by gradient updates on responses generated by the previous policy.
First, the model acts. Then the answer is scored. Then PPO asks: should the model become more or less likely to generate the same tokens in the same states next time?
1. What is being compared?
There are three different model states that are easy to confuse. The current policy is the model being updated. The old policy is the snapshot that generated the current rollout batch. The reference model is a longer-term anchor used by the KL penalty, usually kept fixed.
| Name | Formula | Does it change? | What it is used for |
|---|---|---|---|
| Current policy |
πθ | Yes | The policy currently being optimized. In this setup, the main trainable part is the LoRA-adapted policy, not all original LLM weights. |
πθ′ | Yes, after each rollout | The snapshot that generated the current batch of responses. PPO compares the current policy against this snapshot so one update does not move too far. | |
πref | Usually fixed | The longer-term anchor used by the KL penalty. It prevents the fine-tuned model from drifting too far from the original language model distribution. |
This means θ′ is not simply the original model forever. It is a moving snapshot. If
θ0 generates the first rollout, then θ′ = θ0 for that update.
After PPO produces θ1, the next rollout is generated by θ1,
so the next old policy becomes θ′ = θ1.
2. Rollout: generate first, update later
During rollout, the model receives a task prompt and generates a response. For an NLU classification example, the input might ask the model to place its answer inside a fixed tag:
Input: Sentence: This movie is wonderful.
Prompt ending: <Judgement>
Generated output: Positive </Judgement>
Full answer: <Judgement>Positive</Judgement>
The rollout creates a trajectory. At each token position, the state is the current prefix and the action is the next generated token.
| Time step | State | Action | Recorded during rollout |
|---|---|---|---|
t = 1 |
s1 = prompt + <Judgement>
|
a1 = Positive
|
log pθ′(a1 | s1)
and
V(s1)
|
t = 2 |
s2 = prompt + <Judgement> Positive
|
a2 = </Judgement>
|
log pθ′(a2 | s2)
and
V(s2)
|
Nothing has been updated yet. The rollout only records what the old policy did: which tokens it sampled, how likely those tokens were under the old policy, and what value the critic assigned to each prefix.
3. Reward: score the completed answer
The reward comes after the full response has been generated. For classification tasks, the answer is extracted from the judgement tag and compared with the ground-truth label.
Correct format, correct answer
R = 1
The generated label matches the ground truth.
Correct format, wrong answer
R = 0
The model produced a parseable label, but the label is wrong.
Wrong format
R = −1
The output cannot be reliably parsed. This is the largest penalty for classification tasks.
For STS-B, the task is regression rather than classification, so the reward is based on distance from the true similarity score:
R = 2.5 − |ŷi − yi|
Since STS-B scores are between 0 and 5, the largest possible error is 5. The reward therefore ranges from
−2.5 to 2.5. A formatting failure receives −2.5, the lowest possible reward.
4. Critic: estimate the value of each prefix
The does not predict the next token. It takes the hidden representation of a generated prefix and outputs one scalar value. That value means: from this partial answer, how much future reward should we expect?
Vt = V(st)
A common confusion is how the critic can use Vt+1 if the model is currently at time
t. The answer is that PPO computes these values after rollout. Once the full response has been
generated, the sequence of prefixes already exists:
s1, s2, ..., sT
So Vt+1 is not a magical look into the future. It is the critic's value estimate for the
next prefix that already appeared in the sampled trajectory.
If st = prompt + <Judgement>, the critic might estimate
V(st) = 0.35. If the next generated token is Positive, then
st+1 = prompt + <Judgement> Positive, and the critic might estimate
V(st+1) = 0.80. That means the prefix containing Positive looks more likely
to lead to a high reward.
5. Advantage: assign credit backward
The reward usually arrives only after the full answer is finished. PPO still needs to know whether each earlier token helped or hurt. That is the role of the advantage estimate.
δt = Rt + γVt+1 − Vt This is the temporal-difference error. It compares what the critic expected at the current state with what actually happened after taking the action and moving to the next state.
Ât = δt + γλÂt+1 This is Generalized Advantage Estimation. It is computed backward from the end of the generated response:
Start with ÂT+1 = 0. Compute ÂT, then
ÂT−1, then ÂT−2, and continue until
Â1. This is why the formula can depend on Ât+1 without causing
a circular dependency.
| Term | Meaning | Intuition |
|---|---|---|
Rt | Immediate reward | Often zero for intermediate tokens and nonzero at the end of the response. |
γ
| Discount factor | Controls how much future value matters compared with immediate reward. |
λ
| GAE smoothing parameter | Controls the trade-off between short-term TD estimates and longer-range credit assignment. |
Vt | Critic estimate at the current prefix | What the critic expected before seeing the next token. |
Vt+1 | Critic estimate at the next prefix | What the critic expects after the sampled token has been appended. |
6. PPO update: change token probabilities carefully
Once advantages are computed, PPO updates the policy. It takes the same sampled state-action pairs from rollout and compares the current probability of each sampled token with the probability assigned by the old policy.
rt(θ) =
pθ(at | st)
/
pθ′(at | st)
At the beginning of the update, θ and θ′ are the same, so the ratio is close to
1. As gradient updates change θ, the numerator changes, but the denominator stays fixed
because it was recorded during rollout.
J(θ) = E[min(rt(θ)Ât,
clip(rt(θ), 1−ε, 1+ε)Ât)]
If Ât > 0, the sampled token was better than expected, so PPO tries to increase its
probability. If Ât < 0, the token was worse than expected, so PPO tries to decrease its
probability. The clipping term prevents either change from becoming too large in one update.
Positive advantage
Ât > 0
Increase the probability of the sampled token, but only within the clipped trust range.
Negative advantage
Ât < 0
Decrease the probability of the sampled token, again without letting the policy move too far.
KL penalty
KL(πθ || πref)
Keeps the updated policy close to the original reference model, preserving general language behavior.
7. Critic update: make value estimates less wrong
The critic is trained with a value loss. Its target is not the label itself; its target is a better prediction of expected future reward.
Lvalue = (Rt + γVt+1 − Vt)2 If the critic underestimates a prefix that later leads to a correct answer, this loss pushes its value upward. If it overestimates a prefix that leads to a wrong or malformed answer, the loss pushes its value downward.
8. One complete PPO iteration
| Stage | What happens | What changes? |
|---|---|---|
| Prepare |
Start from the current policy πθk, a critic/value head, and a usually fixed
reference model πref.
| Nothing yet. |
| Rollout |
Generate responses using πθk. Record sampled tokens, old log-probabilities, and
value estimates.
| Nothing yet. |
| Reward | Extract the final answer and compare it with the ground-truth label or score. | Reward is computed, but no model parameter changes. |
| Advantage |
Use critic values and rewards to compute Ât backward through the response.
| Advantages are computed, but no model parameter changes. |
| Policy update | Recompute current log-probabilities, form the ratio against old log-probabilities, and apply the clipped PPO objective. | LoRA policy parameters update. |
| Critic update | Minimize the value loss so the critic becomes better at predicting future reward from a prefix. | Value head / critic parameters update. |
| Repeat PPO epochs | Reuse the same rollout batch for several PPO epochs. In this setup, each sampled batch is used for four PPO epochs. | The current policy keeps changing, while the old log-probabilities from rollout stay fixed. |
| Next iteration | Discard the old rollout batch. Generate a new batch with the updated policy. | πθ(k+1) becomes the policy for the next rollout.
|
Suppose the current model sees This movie is wonderful and generates
<Judgement>Negative</Judgement>. The extracted answer is wrong, so the reward is low.
Tokens leading to Negative receive negative advantage, and PPO lowers their probability in similar
states. If a later rollout generates <Judgement>Positive</Judgement>, the reward is high,
the relevant tokens receive positive advantage, and PPO raises their probability. Clipping prevents either
update from turning into a large, unstable jump.
Reward design
The main paper intentionally uses simple rewards. This is part of the argument: even a basic task-correctness signal can turn a weak prompting model into a much stronger NLU model. But reward design is also the biggest limitation of the approach.
| Task type | Reward | Format penalty | Intuition |
|---|---|---|---|
| Classification tasks | R = 1 if the extracted answer matches the ground-truth label; otherwise R = 0. | Malformed output receives -1, the largest penalty in the classification setting. | Wrong but parseable is bad; unparseable is worse because the RL environment cannot score it reliably. |
| STS-B regression | R = 2.5 - |predicted score - true score|. | Malformed output receives -2.5, the lowest possible reward for the regression setting. | The reward is highest when the predicted similarity score exactly matches the true score and decreases with absolute error. |
| SQuAD appendix | Exact Match and F1 are used as reward signals. | The paper focuses on EM/F1 scoring rather than the <Judgement> classification format. | If a task has an automatic metric, PPO can optimize the model toward that metric directly. |
| PPO-RM | A learned BERT-based reward model scores generated judgments and analyses. | The reward is no longer only a simple label-matching rule. | Richer rewards can improve explanation quality, but may trade off against classification accuracy. |
For classification, the malformed-output penalty of -1 is the strongest penalty in that reward
scheme. A wrong but parseable answer receives 0; an unparseable answer receives -1.
This does not mean the paper uses a simple supervised loss equal to negative reward. PPO maximizes a clipped
policy objective. The negative reward affects the advantage estimate, which pushes the model away from the
token sequence that produced the malformed output.
For , the answer is a number from 0 to 5, so the paper
defines reward as 2.5 - |ŷ - y|. A perfect prediction receives 2.5, the worst possible
error receives -2.5, and malformed output receives the same lowest reward. This keeps the reward
centered and gives PPO a graded signal rather than a binary correct/incorrect signal.
Why LoRA
The paper uses for both SFT and PPO. The obvious reason is computational: full fine-tuning a 7B model is expensive, and PPO is already heavier than SFT because it requires generation, reward computation, advantage estimation, and policy updates.
But LoRA also fits the paper's conceptual goal. The authors are not trying to overwrite the model's language knowledge. They assume the pretrained LLM already contains useful syntax, semantics, and world knowledge. The adaptation problem is to steer output behavior toward NLU labels. Freezing the backbone and training low-rank updates lets the model move in a smaller parameter subspace.
This is also why LoRA is more natural here than simply adding a classification head. The paper wants the LLM to keep answering through generation, because PPO treats the generated tokens as actions. A separate classification head would change the interface into a discriminative classifier. LoRA changes the generation distribution while preserving the model's normal prompting interface.
Experiment design
The experiments are strong because they answer several different questions, not just one. The main table asks whether PPO beats standard NLU baselines. Other experiments ask whether the effect generalizes, whether another RL algorithm behaves similarly, whether the model keeps its original abilities, and whether richer rewards change the trade-off.
| Experiment | Question | Result / role in the argument |
|---|---|---|
| GLUE main benchmark | Does PPO improve standard NLU performance? | PPO-ST reaches 84.8 average, above SFT-ST, prompting baselines, BERT-base, and BERT-large. |
| SuperGLUE appendix | Does the method survive a harder benchmark? | PPO-ST reaches 78.3 average, above BERT-large, BERT-large++, zero-shot, and few-shot LLAMA2. |
| Cross-model check | Is the effect specific to LLAMA2? | PPO improves LLAMA2-7B, Qwen2.5-7B-Instruct, and MPT-7B-chat on STS-B/COPA-style tests. |
| PPO vs GRPO | Is PPO the only RL choice? | GRPO slightly beats PPO on average in the tested tasks, but has higher runtime because it samples multiple responses per prompt. |
| OOD generalization | Does task-specific PPO transfer? | SST-2-tuned models transfer to sentiment datasets; MNLI-tuned models transfer to NLI datasets, sometimes beating GPT-4o on these narrow evaluations. |
| Instruction following and perplexity | Does fine-tuning damage the base model? | PPO preserves instruction-following and WikiText-2 perplexity better than SFT in the paper's tests. |
| SQuAD | Can PPO optimize a reading-comprehension metric? | PPO improves over SFT when Exact Match and F1 are used as rewards. |
| Base vs chat model | Does the starting checkpoint matter? | PPO-tuned LLAMA2-7B-chat-hf averages higher than PPO-tuned LLAMA2-7B-base on GLUE. |
The difference between single-task and multi-task training is also important. In single-task training, each GLUE or SuperGLUE subtask gets its own LoRA module. In multi-task training, several datasets are mixed and one LoRA module is trained to handle them together. Single-task PPO is usually strongest, but multi-task PPO shows that the method is not only a collection of one-off task adapters.
| Appendix / setup detail | Value |
|---|---|
| PPO batch search | Batch size is selected from 4, 8, 12, and 16 for each task. |
| SFT batch size | SFT uses batch size 24 across tasks. |
| PPO epochs | Each sampled batch is used for 4 PPO epochs before new rollout data is sampled. |
| Learning rate | Initial learning rate is 9 × 10^-6. |
| Optimizers | PPO uses Adafactor; SFT uses AdamW. |
| Scheduler | Cosine annealing with 10% warmup, decaying to one-tenth of the initial learning rate. |
| LoRA rank | The LoRA rank is r = 16. |
| Evaluation generation | Multinomial sampling with temperature 1; one response per data sample; response length 12–32 tokens. |
| Few-shot sweep | The number of demonstrations is swept from 1 to 5 on validation data. |
Main results
On GLUE, the paper's headline result is that PPO turns LLAMA2-7B-chat-hf from a weak prompted NLU model into a model that beats strong encoder-only baselines on average. The important comparison is not only PPO vs prompting, but PPO vs SFT.
| Method | GLUE average | What it means |
|---|---|---|
| LLAMA2-7B zero-shot | 46.1 | Instruction prompts alone do not unlock strong GLUE performance. |
| LLAMA2-7B few-shot | 58.7 | In-context examples help, but the model still trails BERT by a large margin. |
| SFT-ST | 78.5 | Supervised LoRA fine-tuning nearly reaches BERT-base but remains below PPO. |
| PPO-ST | 84.8 | Single-task PPO is the strongest GLUE setting in the paper. |
| PPO-MT | 82.9 | Multi-task PPO is weaker than single-task PPO but still competitive with or above BERT-large. |
| BERT-base / BERT-large | 79.6 / 82.1 | The encoder-only baselines remain crucial reference points for NLU. |
The SuperGLUE strengthens the same conclusion on a harder benchmark. PPO-ST reaches an average of 78.3, while BERT-large and BERT-large++ are lower. This matters because SuperGLUE contains more tasks where shallow label matching should not be enough: reading, coreference, commonsense, word-sense, and multi-sentence reasoning.
The paper also compares with . GRPO slightly improves the average score in the small comparison table, but it is slower because it samples multiple responses per prompt. PPO remains attractive because it gives a large gain over SFT with only a modest per-step runtime increase in the authors' setup.
| Algorithm | Average | Per-step runtime | Interpretation |
|---|---|---|---|
| SFT | 76.9 | 4.124s | Simple and cheaper, but lower NLU performance. |
| PPO | 84.6 | 4.299s | Large improvement with modest per-step runtime increase. |
| GRPO | 84.9 | 5.155s | Slightly higher score, but slower because it requires multiple sampled responses. |
The out-of-domain evaluations should be read carefully. PPO-tuned LLAMA2 models outperform GPT-4o on average across the selected sentiment and NLI datasets, including datasets such as , , , , and . This does not mean LLAMA2 becomes generally stronger than GPT-4o. It means a smaller model, specifically fine-tuned with a relevant task reward, can beat a much larger general model on narrow classification-style evaluations.
Reward model experiment
This experiment asks what happens when the reward is no longer just exact label matching. The main PPO setup rewards the final label: correct, wrong, or malformed. That works well for benchmark classification, but many language-understanding tasks also care about explanation quality, consistency, factuality, and helpfulness. A is one way to turn those softer qualities into a scalar reward that PPO can optimize.
| Component | What happens |
|---|---|
| Response generation | For 5,000 SST-2 samples, LLAMA2-7B-chat-hf generates four responses per sample, each with a sentiment judgment and an analysis. |
| Ranking source | GPT-4o ranks responses using label correctness, consistency between judgment and analysis, and factual/helpful analysis quality. |
| Ranking structure | Correct answer with analysis is ranked above only correct answer, which is ranked above incorrect answer with analysis, which is ranked above only incorrect answer. |
| Reward model | A BERT-base-cased model is trained with pairwise ranking loss so better responses receive higher scores than worse responses. |
| PPO-RM result | Analysis quality improves, but SST-2 classification accuracy drops from 96.4 to 89.7. |
From label reward to learned reward
In the simple reward setup, PPO only needs to check whether the extracted label matches the ground truth. The reward is direct and task-specific:
R = 1(ŷ = y)
The reward-model setup changes the source of the reward. The LLM generates both a sentiment judgement and an analysis. Instead of assigning reward only by exact label match, a separate model scores the whole response:
rθ(x, y)
Here, x is the input example, y is the generated response, and
rθ(x, y) is the reward model's scalar score for that response. PPO can then use this score
as the reward signal.
How the reward model learns preferences
The reward model is trained with pairwise ranking. For the same input, a better response should receive a higher score than a worse response. The training data therefore contains pairs:
(x, yw, yl)
yw is the winner, or the higher-ranked response. yl is the loser,
or the lower-ranked response. The desired inequality is:
rθ(x, yw) > rθ(x, yl)
The loss turns that preference into a differentiable objective:
L(θ) = −E(x,yw,yl)∼D
[log σ(rθ(x,yw) − rθ(x,yl))]
The inside of the sigmoid is the score gap between the preferred and less preferred response:
Δr = rθ(x,yw) − rθ(x,yl)
If the reward model already gives the better response a much higher score, Δr is large,
σ(Δr) is close to 1, and the loss is small. If the worse response receives a higher
score, Δr becomes negative, σ(Δr) becomes small, and the loss becomes large. Training
therefore pushes the reward model to rank better answers above worse ones.
Responses are ranked not only by whether the sentiment label is correct, but also by whether the explanation is
consistent with the label, factually reasonable, and helpful. The intended preference order is:
correct answer with analysis > only correct answer >
incorrect answer with analysis > only incorrect answer.
How it changes PPO
With exact label matching, PPO receives a simple task reward. With the learned reward model, PPO receives a score
from rθ(x, y). The rest of the PPO machinery is the same: the generated response receives
a reward, advantages are computed, and the policy is updated to make high-reward responses more likely.
| Reward source | What PPO is pushed toward | Main trade-off |
|---|---|---|
| Exact label reward | Output the correct class label in the required format. | Strong classification accuracy, but little pressure to improve explanation quality. |
| Learned reward model | Output a response that looks better under the learned preference scorer. | Better analyses, but the reward may become less directly aligned with classification accuracy. |
Why the result matters
The result is a useful warning. PPO-RM improves generated analysis quality, but reduces SST-2 accuracy. That does not mean PPO itself failed. It means the reward changed the target. Exact label reward asks the model to be sharp at classification. A learned reward model asks for a broader behavior: correct judgement, coherent explanation, factuality, and helpfulness.
PPO is an optimizer; the reward is the specification. If the reward model is trained on limited data, or if the policy finds ways to satisfy the reward model without improving the underlying task, the model can trade away accuracy for outputs that look better to the scorer.
This is why reward design is not a side detail. For simple NLU benchmarks, exact label matching is direct and reliable. For richer language-understanding tasks, the reward must represent a more complex target, and that target must be learned carefully enough that PPO does not optimize the wrong behavior.
Limitations
The main limitation is that the paper's strongest results still rely on ground-truth labels. Even though the training is framed as reinforcement learning, the reward often comes from exact comparison with the labeled answer. That is different from SFT, but it is not fully annotation-free.
The second limitation is that simple rewards fit structured benchmarks better than open-ended understanding. GLUE and SuperGLUE are good for this paper because many tasks have clear labels or automatic metrics. Real language understanding may involve ambiguous answers, long reasoning, factual support, or multiple valid explanations. These cases need more robust reward models.
The third limitation is output format dependence. The model is rewarded after its answer is extracted from a fixed structure. That is practical and necessary for automatic scoring, but it means the method depends on a reliable interface between free-form generation and discrete task evaluation.
The paper's own conclusion is therefore balanced: PPO is a promising way to adapt already pretrained LLMs to NLU tasks, but the long-term research problem is not simply "use PPO." It is how to define environments and rewards that are reliable, generalizable, and resistant to reward hacking.
How to design convincing experiments
A strong empirical paper is not just a proposed method followed by a few scores. The experiments have to build a chain of evidence. They should prove the main claim, compare against fair alternatives, explain why the method works, test whether the gain generalizes, report the cost, and show the boundary cases. This is why the experimental section here feels unusually complete: it does not only say that PPO wins on one benchmark; it tests the claim from several angles.
A good experiment section should make a skeptical reader think: the method beats reasonable alternatives, the improvement is not accidental, the cost is understandable, and the limitations are visible.
| Evidence needed | Question to answer | How this study answers it |
|---|---|---|
| Main result | Does the method directly support the central claim? | GLUE is used as the main benchmark because the claim is about improving NLU. PPO is compared against prompting, SFT, and BERT-style encoder models on standard NLU tasks. |
| Fair baselines | Is the method compared against alternatives that a reader would actually consider? | The baselines include zero-shot prompting, few-shot prompting, SFT, BERT-base, and BERT-large. This makes the comparison stronger than only showing that PPO beats a weak prompt. |
| Mechanism | Why does the method help? | PPO is compared with SFT to test whether reward-driven optimization adds something beyond supervised next-token training. PPO is also compared with GRPO to ask whether the gain is specific to PPO or more broadly related to RL-style optimization. |
| Stress test | Does the method still work when the benchmark is harder? | SuperGLUE is used as a harder NLU benchmark. This checks whether the result is more than a GLUE-specific improvement. |
| Cross-model evidence | Is the method tied to one model family? | The experiments include LLAMA2, Qwen2.5, and MPT. This makes the argument less dependent on one particular checkpoint. |
| Generalization | Does the model only learn the training dataset, or does the behavior transfer? | Models fine-tuned on SST-2 are tested on other sentiment datasets, and models fine-tuned on MNLI are tested on other NLI datasets. This checks whether the learned behavior survives outside the original training task. |
| Cost | What is the practical price of the improvement? | LoRA reduces trainable parameters. PPO and GRPO are compared by runtime, and the study also notes that LLAMA-style inference is much slower than BERT-style inference. |
| Side effects | Does fine-tuning damage other useful abilities? | Instruction-following is tested with an out-of-distribution Amazon review task, and language modeling ability is checked with WikiText-2 perplexity. |
| Failure or trade-off | Where does the method become less straightforward? | The reward model experiment improves generated analysis quality but lowers SST-2 accuracy. This shows that richer rewards can change the target and introduce trade-offs. |
| Extension | Can the same idea apply beyond classification? | The SQuAD experiment uses Exact Match and F1 as reward signals, showing how PPO can be extended to reading comprehension. |
| Starting point | Does the initialization matter? | The comparison between LLaMA2-base and LLaMA2-chat shows that an instruction-tuned starting point is more suitable for this kind of task-specific RL adaptation. |
This is the difference between a narrow benchmark win and a convincing empirical argument. A narrow version would simply say: PPO improves GLUE. A stronger version says: PPO improves the main benchmark, remains competitive on a harder benchmark, works across several model families, transfers to related datasets, has measurable cost, keeps more of the original model's behavior than SFT, and reveals an important trade-off when the reward becomes more complex.
When reading or designing a method paper, ask: What is the main claim? Are the baselines fair? What explains the gain? Where else does it work? What does it cost? What abilities might it damage? Where does it fail or trade off? If the experiments answer these questions, the paper is no longer just reporting numbers; it is making a case.
Takeaways
- Decoder-style LLMs can be strong generators while still underperforming encoder models on classic NLU tasks.
- The paper's core move is to treat NLU adaptation as reinforcement learning over generated answers.
- SFT imitates answer strings; PPO optimizes response-level reward.
- PPO's old-policy ratio is local and dynamic; the reference model used for KL is a separate, usually fixed, global anchor.
- The critic does not know the future during generation; it estimates values after rollout has produced the state sequence.
- LoRA is not only cheaper than full fine-tuning; it also constrains where the model is allowed to adapt.
- Reward design is the central bottleneck. Simple label matching works well for benchmarks, but richer tasks need better reward models.