RL Training Notes (1): Distributed RL Training — Running PPO / GRPO From One GPU to a Cluster

0. One-Sentence Positioning

Distributed RL training = Ray for orchestration, with Actor (policy) / Critic (value) / Reference (frozen, for KL) / Reward (scoring) split onto separate GPU pools, a vLLM or SGLang rollout engine sitting in the middle to generate, and a high-speed weight sync broadcasting parameters from training back to inference. It is the shared engineering substrate behind RLHF, RLAIF, preference tuning, and robot VLA preference alignment.

It is not a new algorithm. It is the decision to stop letting one model act as both performer and judge on a single GPU.

1. Why RLHF Cannot Be One Pot

The textbook PPO-RLHF implementation (early TRL / HF Trainer) runs in a single process:

same weights: fp forward to generate -> reward scoring -> same weights: backward update

That survives below 1B parameters and collapses above 7B, for three reasons:

1. One set of weights plays two roles. It is the Actor (needs gradients) and the Reference (frozen, computes KL). Both copies must stay resident, doubling memory.
2. Inference and training fight over the same SMs. Generating a 256-token response is a bandwidth-bound inference workload where bigger batches win. Gradient descent is a compute-bound workload. Jammed together, neither runs at its operating point.
3. The time distribution is wildly uneven. Within a PPO step, rollout (generation) accounts for roughly 80% of the wall clock; DeepSpeed training accounts for 20%. Running them sequentially on the same cards means those cards idle during the training phase.

In one line: rollout and training are two different workloads and must be asynchronous.

2. Four Roles Decoupled Plus a Rollout Engine

None of this is algorithmic novelty. It is all engineering, and it answers one question: how do you spend 80% of the time on the thing that matters (generation) instead of idling cards?

3. Six Stages of a PPO Step

(1) Rollout generation       : vLLM generates responses (the longest stage)
(2) Reward scoring           : reward model or rule verifier
(3) Reference forward + KL   : the frozen model computes logp_ref per sample
(4) GAE advantage            : the Critic estimates advantages
(5) Critic / Policy gradient : the backward update
(6) Weight sync back to vLLM : broadcast parameters to the engine, next round

The hard parts are (3) and (6). Stage (3) dictates your memory budget (Reference weights plus KV), and stage (6) dictates whether the pipeline stalls waiting for a barrier.

4. The PPO and GRPO Objectives

The core of PPO in four lines:

A_t    = R_t - V(s_t)                                   # advantage (GAE in practice)
rho_t  = pi_theta(a_t|s_t) / pi_old(a_t|s_t)            # importance ratio, must be >= 0
L_CLIP = E[ min( rho_t*A_t, clip(rho_t, 1-eps, 1+eps)*A_t ) ]
L_KL   = -beta * E[ log pi_theta(a_t) - log pi_ref(a_t) ]
L_PPO  = L_CLIP - c1 * L_VF + c2 * L_KL

Here eps defaults to 0.2, beta is raised adaptively as measured KL grows (too much drift means pulling back), and c1 / c2 weight the value and KL terms. The Reference must be frozen while computing logp_ref.

GRPO’s contribution is deleting the Critic. Instead of a value head, sample a group of answers for the same prompt and use within-group relative ranking as the advantage:

A_i = (r_i - mean(r_group)) / std(r_group)      # group-normalized, replaces the Critic

One fewer GPU pool, one fewer model in memory, one fewer objective. The cost is that group size must be at least 4, otherwise the within-group variance estimate is unstable.

5. Weight Sync: Two Paths

def sync_weights_to_vllm(actor, vllm):
    for name, param in actor.named_parameters():
        if param.requires_grad:
            # zero-copy GPU-to-GPU broadcast over a side-channel NCCL group
            vllm.collective_rpc("load_weights", param.data)

vLLM 0.7+ exposes WorkerExtension and collective_rpc, so the training side can build a side-channel NCCL group with StatelessProcessGroup and broadcast zero-copy.

6. Measured Speedups

Read as a group, the trend is clear: almost all of the efficiency gain comes from decoupling rollout from training, not from a new loss function.

7. Four Common Pitfalls

1. Sync gap. If weight sync is slow, training uses v1 weights while inference uses v2, and the policy gradient becomes meaningless. Sync right after every mini-batch.
2. Actor memory blowup. The Reference cannot be offloaded to CPU, or blocked I/O during KL computation stalls the whole pipeline.
3. vLLM without sleep. When switching between rollout and training phases, call enable_sleep on the vLLM engine to release VRAM, or you OOM.
4. GRPO group variance. With group size below 4, the within-group advantage estimate is too noisy and training oscillates. Enlarge the group or fall back to PPO with a Critic.

8. How It Connects to VLA

This infrastructure serves more than text RLHF. Robot policy RLAIF runs on exactly the same stack:

VLA generates trajectories -> reward model scores -> within-group ranking (GRPO) -> policy update -> sync back to inference

• Both Octo and π0 already support RLAIF against a custom reward model.
• GRPO is the efficient reference algorithm for VLA fine-tuning: dropping the Critic can cut Octo-Small (27M) fine-tuning from 2 hours to about 20 minutes.
• It is a hard requirement for parallel multi-scenario training in a factory: train and serve robots by day, collect rollouts by night, all on one pool of 64 GPUs. That is precisely the mainline OpenRLHF / veRL scenario, and robotics companies can inherit most of the boilerplate for free.

9. Summary

One line to remember: the bottleneck in RLHF was never the algorithm. It was letting generation and training each sit in their proper place. Distributed RL training is that idea, engineered.

Series navigation: this is the opening post of the series. Related background lives in fa4-deepseek-v4 (V4 training and compression) and sys8-parallel-strategies-pd-disaggregation (parallelism strategies and PD disaggregation).