Staleness and off-policy drift in asynchronous LLM RL — related-works note

Reviewed 2026-09-17. Staleness is the version lag between the policy that sampled a trajectory and the policy whose gradient that trajectory updates. In asynchronous post-training it is the default condition, not a corner case. The 2025–2026 recipes treat it as a controlled quantity: bounded by admission control or version-drop rules, corrected at the loss with ratios taken against the inference engine's own log-probabilities, and monitored through log-probability-difference and importance-ratio statistics. No source reviewed here calls the problem solved. Two confounds must be separated before any result is attributed to staleness: training-versus-inference numerical mismatch, which is present at zero lag, and multi-gradient-step mini-batching, which creates staleness with no asynchrony at all.

This is a source-pinned reading note, not an execution study. No training runs were launched. Local pins: the SkyRL checkout at ../SkyRL (commit 152ebec0, 2026-06-25), the GLM-5 report text extracted from its PDF, and the RL-Infra-Bench blueprint. Per-source records, with which numbers were re-read on the primary page and which come from a delegated sweep, are in four companion notes: systems, estimators, mismatch and the 2026 recency sweep.

1. Definitions and the decomposition

Let μ be the sampling distribution the inference engine actually realized, π_old the trainer's recomputation of the same weights, and π_θ the policy being updated. The token-level importance ratio an on-policy surrogate needs factors as

π_θ(x_t) / μ_old(x_t)  =  [ π_old(x_t) / μ_old(x_t) ]  ×  [ π_θ(x_t) / π_old(x_t) ]
                            training–inference mismatch      policy staleness

The decomposition appears in the Qwen team's formulation paper, in SkyRL's off-policy-correction documentation (crediting verl's rollout-correction math note), and as r = r_d·r_s in the May 2026 "missing old logits" paper. The first factor is nonzero in a perfectly synchronous run because inference kernels, precision, parallelism and MoE routing differ from the trainer's. The second is what asynchrony, in-flight updates, partial rollouts and mini-batching add. The Qwen paper's first-order argument is that the token-level surrogate tracks the true sequence-level objective only while both factors stay small. Qwen, arXiv 2512.01374 · SkyRL docs · Guan et al., arXiv 2605.12070

Staleness is counted in policy versions, and the accounting differs by system. SkyRL counts the training step that consumes a group minus the step at which it was scheduled. AReaL, ROLL Flash and StaleFlow count from the version that started generation. GLM-5 records the ordered list of versions that touched a response and measures from the oldest. ESTR separates intra-trajectory staleness (versions spanned inside one rollout under in-flight updates) from inter-trajectory lag (last token's version to the target policy). Applied Compute splits mean staleness into a pre-queue part accrued during generation and an in-queue part accrued while waiting for the trainer. Numbers from different papers are therefore not comparable without restating the definition. ESTR · Applied Compute

Under in-flight updates the behavior policy is not a checkpoint. GLM-5 states that tracking exact behavior probabilities across the versions a single response spans would require keeping every intermediate checkpoint, and instead uses the rollout engine's logged log-probabilities as the behavior proxy. The missing-old-logits paper measures what recovering π_old exactly would cost (snapshots: 40–76 GB and 25–150 s per step on 4B and 30B models) and proposes an exponentially weighted parameter average as the proxy. GLM-5 §4.1.2, arXiv 2602.15763

2. Where staleness comes from

3. How systems bound staleness

Admission control. AReaL rejects new generation requests when ⌊(N_r−1)/B⌋ > i+η, with N_r trajectories generated so far, B the batch size and i the current version; η = 8 for math and 4 for code. SkyRL reimplements this as a capacity rule, accepted + running < (max_staleness_steps + current_step) × mini_batch_size, with a buffer of mini_batch_size × (S+1) finished groups and the constraint mini_batch_size ≤ workers ≤ mini_batch_size × (S+1); default S = 4. ROLL Flash caps its sample buffer at (1+α)×batch so no generated sample can violate the bound. StaleFlow reserves a version tag at admission and consumes a trajectory only if V_traj + η ≥ V_buf. All four are aggregate or per-trajectory bounds on versions, not on wall-clock. AReaL · fully_async_trainer.py:104–176 · ROLL Flash · StaleFlow

Drop rules at training time. GLM-5 discards a response when the current version minus its oldest version exceeds τ (unreported). INTELLECT-3 discards rollouts generated by more than max_off_policy_steps = 8 policies. ECHO-2 discards below v_t − S. verl's V1 sampler drops, or optionally waits on, prompt groups at max_off_policy_threshold = 8 versions; NeMo-RL evicts trajectories older than max_trajectory_age_steps (default 1); PipelineRL's source counts samples beyond max_lag rather than dropping them, although the March 2026 Hugging Face survey classifies it as version rejection. SkyRL's documentation prefers proactive admission to over-generating and dropping, on the grounds of wasted compute.

In-flight handling is where implementations diverge most. PipelineRL continues in-progress sequences with the stale KV cache and reports only slightly higher divergence than recomputing it. AReaL discards and recomputes the cache. The June SkyRL checkout freezes requests with vLLM's keep-mode pause and resumes them after the sync; its own tutorial and the Hugging Face survey still describe an abort-and-refeed loop, so the behavior changed between March and June 2026. Laguna blocks any in-flight rollout step on an update so no step straddles a refresh, and resets the KV cache. ECHO-2 workers simply switch version when a snapshot finishes installing. The Prime Intellect 1T run drops by max_off_policy_steps and salts the KV cache by version. PipelineRL · inference_engine_client.py:381–395 · Laguna, arXiv 2605.27605

What sets the mean staleness. Applied Compute derives closed forms in five parameters: trainer utilization ρ, batch size B in rollouts, sampling concurrency C, queue factor q and a response-length tailness multiplier. Rollout-bound: staleness ≈ C·M_tail/B + ρ. Train-bound: C·M_tail/(ρB) + (2q+ρ−1)/(2ρ). Predictions on real Qwen3-8B GRPO runs were within 0.06–0.27 versions of measurement; setting q = 1 cut staleness 35% at unchanged step time in one configuration. This is the same dimensionless-ratio framing the RL-Infra-Bench blueprint uses, now with a validated formula. Applied Compute, 2026-07-03

Production ceilings. Mercor's SkyRL runs use S = 3 with mini-batch 16 and 16 samples per prompt, a 1,024-trajectory ceiling, run at 550 (35B) and 300 (397B) concurrent rollouts. Laguna caps at 10 optimizer steps and never reaches it. GLM-5 runs more than 1k concurrent rollouts with τ and K unreported. Marin's 67B runs recommend a staleness allowance of 4 updates. Mercor guide, 2026-09-01

4. Loss-level corrections

All of these take the ratio against the rollout engine's logged log-probabilities. They differ in what they do with it.

SkyRL applies its masks multiplicatively in a fixed order (outlier sequence mask, token mask, sequence mask) and logs is_ratio_{mean,std,max,min}, clip fractions and rollout_train_logprobs_abs_diff_*. Every correction is silently disabled when no rollout log-probabilities are attached to the batch. off_policy_correction_utils.py:257–312

5. How much staleness is tolerable: reported evidence

The pattern: tolerated staleness moved from about one step in 2024 to 3–8 versions in 2025–2026 production recipes, and to 50–256 updates in method papers on small models, but only after ratios were taken against engine log-probabilities, masks were added, and MoE routing was replayed. Quote a tolerance only together with its loss, its mismatch controls and its staleness definition.

6. The mismatch confound

Mismatch is measurable at zero staleness and is often the larger factor. Measured magnitudes: KL(engine‖trainer) roughly 1e-4–1e-3 in bf16 on dense models, 1e-2 in FP8, up to 1e-1 for MoE; a 30B-A3B MoE flips at least one expert on 94% of tokens between SGLang and Megatron. Fixes and costs:

• Precision: FP32 LM-head logits (ScaleRL 0.52 → 0.61; MiniMax-M1 correlation 0.9x → 0.99x).
• Routing replay (R3, Keep Routing): train/infer KL 1.535e-3 → 7.5e-4 on Qwen3-30B-A3B at under 3% rollout latency; adds bias under mini-batching because routing is frozen across mini-batches.
• Deterministic operators: GLM-5 replaced a non-deterministic top-k in its sparse-attention indexer with torch.topk after the alternatives collapsed within a few steps.
• Batch-invariant or bitwise-consistent kernels: 20–60% sampling-throughput cost and 2–5× trainer cost depending on architecture; the vLLM+TorchTitan bitwise stack is 2.4× slower; IsoExec reports 25% end-to-end overhead.

Two August 2026 studies test whether removing mismatch helps under staleness. Wang's GDN study reaches an exactly zero gap at lag 0 with batch-invariant kernels, then sweeps off-policy windows of 4, 12 and 32: at window 12 on MATH the parity arm peaks near 0.7 train reward against 0.6 for vLLM native, but at window 4 the arms interleave, on Search-R1 they are indistinguishable, and at window 32 both degenerate. The author's conclusion is to use parity as a debugging control, not a production default. Panda's MoE study gets K3 to zero on Qwen3.6-35B-A3B and raises held-out solve from 63.9% to 77.4% synchronously, but an asynchronous run with stale weights and cache and no mitigation falls to 59.4%, recovering to 76.6% with router recall plus CISPO. Both confirm the ordering in §1: fix numerics first so that any residual ratio is attributable to lag, then bound and correct the lag. Wang, Aug 2026 · Panda, 2026-08-17 · ../torchtitan-batch-invarient-GDN

7. Framework knobs

Defaults for frameworks other than SkyRL were read on their main branches on 2026-09-17 and are recorded, with commits and the three defaults re-read by hand, in the configuration audit. Within the SkyRL checkout alone there are three documentation drifts: the KV-cache default, the pause semantics, and a generator.use_cache_salt option named in the live docs that does not exist in the June code. Two cross-framework facts matter for benchmark design: only SkyRL, AReaL and Miles bound staleness at admission, the rest drop or wait at the trainer or do not bound it in core; and version provenance is per token in AReaL and verl V1, per sample in PipelineRL, PRIME-RL, NeMo-RL and slime, and absent in OpenRLHF and ROLL.

8. What this means for RL-Infra-Bench

The blueprint already lists "asynchrony and staleness" as a two-GPU, one-node mechanism whose trusted signal is the version-lag distribution plus importance-ratio and clipping statistics, and argues that the magnitude is governed by generation time over update time, in-flight rollouts over batch size, and sync latency over step time. The literature supports that framing and sharpens the observable signature: Applied Compute's formula turns the ratio argument into a prediction a verifier can check, and ESTR's intra/inter split and the missing-old-logits factorization say which quantities must be logged per token. BLUEPRINT.md:716, 796–812

What exists today:

• train-five-steps enables token TIS with cap 2.0, requests rollout log-probabilities, and verifies policy versions 1–5 plus evaluation version 6. solve.sh:184 · verify.py:48
• The submitted-job qualification rejects reports whose last trial carries a stale policy version. test_submitted_rl.py:589
• gdn-parity covers the mismatch confound with exact log-probability comparison.
• SkyRL's fully asynchronous Harbor entrypoint is a listed source candidate with no task yet. README.md:145

Task shapes the sources make concrete, each with a verifier signal the candidate cannot fake:

1. Silently off-policy run. Correction is configured but rollout log-probabilities are not attached, so SkyRL's correction path computes nothing. Signal: is_ratio_* metrics absent while max_staleness_steps > 0; the verifier recomputes ratios from logged engine log-probs. Reference: the GLM-5 and Mercor practice of using engine log-probs directly.
2. Aggregate bound satisfied, per-group bound violated. Long-tail trajectories exceed S while the capacity rule holds, which SkyRL accepts with a warning. Signal: per-group trained_step − scheduled_step distribution against the Applied Compute prediction for the run's C, B, q and ρ. Reference fixes: GLM-5's τ rule, INTELLECT-3's max_off_policy_steps.
3. Version provenance lost across a pause or resume. Trajectories carry the wrong version after a keep-mode pause or a checkpoint restore. Signal: engine-reported version against trainer version at each step, the same probe the blueprint lists for weight synchronization; the missing-old-logits paper gives the cost envelope for exact recovery.
4. Stale KV cache across versions. Prefix blocks reused across a sync. Signal: log-prob difference on the shared prefix before and after sync, with the PipelineRL "slightly higher divergence" result as the expected magnitude and Laguna's reset-on-broadcast as the conservative alternative.
5. Optimizer state across weight pushes. Reproduce GLM-5's optimizer reset as a knob; verify the effect on the importance-ratio distribution, not on final reward.
6. Mismatch versus staleness attribution. Use the GDN bitwise-parity control as the positive control so that a nonzero ratio is attributable to lag alone, then inject lag through the delay knob.
7. Staleness ceiling engineering. Given a fixed GPU split, reach a target step time under a staleness bound by choosing C, B and q; the verifier checks measured mean staleness against the closed form and the ESS of the correction weights.

These are candidates, not qualified tasks; each needs the blueprint's positive and negative controls before use. Effects that only appear after thousands of updates, such as the final-reward cost of a given staleness, remain out of scope for the workhorse tier, as the blueprint already states.

9. Evidence boundaries and open questions

• Production accounts (GLM-5, Laguna, Mercor, DeepSeek, Kimi K3) report settings without a staleness sweep; the sweeps that exist use one model family each, and several induce staleness by delayed data or mini-batching rather than a live pipeline.
• "Tolerable staleness" is recipe-dependent and definition-dependent; the ordering of methods flips between studies (M2PO beats GSPO at k=256; ROLL Flash finds them comparable at α ≤ 8; VCPO finds M2PO collapsing at k=12 on GSM8K).
• Applied Compute predicts staleness but not its effect on reward; Huang's collapse threshold is on a toy MDP; the scaling law is on 1B–3B models. Nobody has published a reward-versus-staleness curve at frontier scale.
• Whether reusing a stale KV cache is harmless at scale rests on PipelineRL's divergence measurement at 7B and on defaults elsewhere; Laguna chose the opposite.
• Documentation drifts inside a single SkyRL checkout show that framework defaults must be pinned to a revision, which is the same rule this workspace applies to benchmarks.

Source map