notes · overview · raw markdown

Production LLM pipelines: what the sources say, 21 September 2026

This note backs the "life cycles" figure, the code-by-stage band and the RL-loop anatomy. It merges the companion notes written earlier the same day (../../llm_infra_review/production_lifecycle_sources_v2.md, rl_pipeline_sources_v2.md; tag note-2026-09-21), three delegated sweeps (tag agent-reported), and eight primary pages re-read in this session (tag spot-checked): the slime README, GLM-4.5 §3.5, the Olmo 3 blog, the checkpoint-engine README, the Terminal-Bench 2.1 news post and task packages, and the PostTrainBench and InferenceBench READMEs. Earlier workspace notes are prior-note. Numbers are in data/pipelines.json and data/rl_loop.json; re-read agent-reported entries before quoting them in a paper.

1. The stage vocabulary is stable even where the names differ

Source	Data	Pretrain	Midtrain	Post-train	Agentic RL	Serve
Llama 3 (report)	curation, dedup, mixture	16K H100, 4D parallelism	long-context continuation, annealing	rejection sampling, SFT, DPO rounds	—	inference optimization, FP8
DeepSeek-V3 / R1 (report; serving code)	14.8T tokens	2,048 H800, DualPipe, FP8	32K → 128K (YaRN)	cold start → reasoning RL → rejection sampling + SFT → RL for all scenarios	—	prefill unit 4 nodes, decode unit 40 nodes; DeepEP/DeepGEMM/EPLB open
Qwen3 (report)	> 30T tokens	General S1, Reasoning S2	Long Context stage	long-CoT cold start → reasoning RL → thinking-mode fusion → general RL	Qwen3.5: disaggregated async RL (blog only)	—
Kimi K2 / K3 (report; two components open)	15.5T tokens	MuonClip, zero loss spike	4K → 128K; K3 to 1M	SFT, joint RL; checkpoint-engine (1T update < 30 s)	agentic data synthesis, partial rollouts; AgentENV microVMs	K3 §5.4
GLM-4.5 (report; RL on open slime)	corpus	355B-A32B	repo-level code · synthetic reasoning · long-context & agent	expert training → unified training; SFT, reasoning/agentic/general RL on slime	disaggregated async pools, trajectory pool, Docker runtime	SGLang commands
GLM-5 (report; slime)	27T tokens	744B-A40B	4K → 128K → 200K	SFT → reasoning RL → general RL → cross-stage distillation	fully decoupled, > 1k concurrent rollouts, heartbeat fault tolerance	PD disaggregation in the loop
Llama 4 (blog)	—	> 30T tokens, FP8, 32K GPUs	mid-training for 10M context	lightweight SFT → online RL → lightweight DPO	fully async online RL framework	—
DeepSeek-V4 (report)	—	32–33T tokens, FP4 experts	1M context	domain specialists → on-policy distillation	preemptible generation service (token WAL); DSec sandboxes	on-disk KV cache
MiniMax-M2 (report)	—	29.2T tokens	8K → 32K → 192K	SFT, CISPO RL	Forge: gateway, data pool, windowed FIFO	—
Nemotron 3 Super (report + open recipe)	data-prep (subset)	Megatron-Bridge, 25T tokens	LC-phase CPT at 1M on GB200	SFT → RLVR/SWE-RL/RLHF → MTP healing (NeMo-RL)	SWE-RL, NeMo-Gym, 4,000 env instances per batch	quantization, evaluator
OLMo 3 (fully open)	Dolma 3 tools	OLMo-core, 5.9T, up to 1,024 H100	Dolmino 100B; Longmino ~50B	Open Instruct: SFT → DPO → RLVR (OlmoRL async)	—	—
SmolLM3 (fully open)	datatrove	nanotron, 3 stages to 11.1T, 384 H100	long-context 100B; reasoning mid-training 140B	TRL: SFT → APO, merging	—	—
Apertus (open pretraining)	pretrain-data	Megatron fork, up to 4,096 GPUs	4K → 64K	SFT → QRPO	—	—
LLM360 K2 / K2-V2 (fully open)	TxT360	480 A100, weekly hardware failures	four mid-training stages to 512K	SFT	—	—
Marin (open lab)	data browser, executor DAG	Levanter on TPU, 12.7T (8B)	—	SFT; RL by 2026	—	—
Arcee Trinity Large (report; RL on prime-rl)	—	2,048 B300, modified TorchTitan	~117B long-context	SFT at 64K; RL with prime-rl	—	—
Mercor 397B (open recipe)	—	—	—	—	SkyRL async, S=3, 300–550 concurrent rollouts	vLLM rollout engines

Observations that shape the page:

Every stage has public code, but the released-model reports rarely give RL cluster sizes (Kimi, GLM, Qwen, MiniMax and DeepSeek-V4 do not), and only Llama 3 (466 interruptions in 54 days, > 90% effective time) and LLM360 K2 (weekly hardware failures, 12 categories) publish failure statistics. OLMo 2 describes the health-check, cordon and auto-restart mechanism without counts. This is the gap a benchmark environment must reproduce: operating the code, not writing it.
Midtraining is a real, named stage almost everywhere (Dolmino, annealing, LC-phase, three GLM sub-stages, four K2-V2 stages), yet no audited benchmark task targets it. The grid keeps it "unaudited" rather than borrowing pretraining evidence.
Serving appears twice: as the user-facing service (vLLM, SGLang, Dynamo, llm-d, production-stack; DeepSeek's own 278-node account) and as the rollout engine inside every RL stack. A CPU parser repair in vLLM is evidence for neither role's operational behavior.
Agentic RL has become the stage with the most first-party infrastructure detail (GLM-5, Kimi K2.5/K3, DeepSeek-V4, MiniMax Forge, INTELLECT-3, Nemotron 3, Mercor) and the least benchmark coverage.

2. The RL loop, and where the implementations differ

The sweep decomposes every stack into six roles: a trainer (Megatron-Core in slime, AReaL, SkyRL/Mercor, NeMo-RL, ROLL and verl; FSDP2 in prime-rl, torchforge and OpenRLHF; hidden behind LoRA in Tinker), a rollout service with an OpenAI-style endpoint (vLLM or SGLang), a weight-sync path (NCCL broadcast in verl, SkyRL, Magistral, TRL and NeMo-RL; a checkpoint engine in Kimi at about 20 s per 1T model; HTTP relays in prime-rl; delta patches in slime; an RDMA store in torchforge), a queue or buffer (slime Data Buffer, verl MessageQueue and TransferQueue, NeMo-RL replay buffer, LongCat SampleQueue, MiniMax Data Pool), reward/verifier and sandbox services scaled on CPU (Prime Sandboxes > 4,000, LongCat 32,000 environments, Kimi K2.5 up to 100,000 tasks, DeepSeek DSec "hundreds of thousands"), and an orchestrator (Ray almost everywhere; Monarch in torchforge).

They differ in placement (colocated hybrid engines for reasoning RL in Kimi K2, Seed1.5-Thinking, verl and GLM-4.5's synchronous mode; disaggregated GPUs for agentic RL in GLM-5, INTELLECT-3's 16:44 split, Mercor's 12:8, NeMo-RL's async mode, RollArt), in synchrony (bounded-staleness knobs: AReaL η = 4/8, INTELLECT-3 max_off_policy_steps = 8, Mercor 4, NeMo-RL max_trajectory_age_steps = 1, verl staleness_threshold, SkyRL max_staleness_steps; in-flight updates and partial rollouts; 2–4× speedups reported), and in the train–inference mismatch fix (importance-sampling correction by default in TRL and required in NeMo-RL; FP32 LM head in MiniMax; batch-invariant kernels in Thinking Machines and TitanRL). This matches the workspace's staleness note (../../ASYNC_RL_STALENESS.md).

The five-part conceptual loop on the page (prompts & environments → rollout → score & admit → learn → publish weights, with checkpoint/recovery and placement/networking/sandbox rails) is the earlier companion note's proposal, labeled as a synthesis illustrated with slime and GLM-4.5. It is not any repository's class diagram.

3. What the audited tasks touch, per component

Learn: MLS-Bench's four verl hooks (measured, infrastructure fixed), RE-Bench's Foundry pipeline (measured, SFT) and GPT-2 chat RL (measured, tiny).
Rollout: two Terminal-Bench 4 parser repairs in vLLM and SGLang (bounded, CPU).
Checkpoint & recovery rail: TB4 checkpoint consolidation (bounded), Foundry export (measured).
Placement rail: MLS EPLB (simulated), DeepSWE Arcane drift detection (bounded, container management).
Prompts & environments, score & admit, publish weights: nothing in the audited set. These map onto RL-Infra-Bench's rollout-throughput, staleness, weight-synchronization and recovery families (BLUEPRINT §12.2).

4. Points to settle with the user

Canonical case study. The page uses GLM-4.5 + slime as the worked example because the report and the code line up (report §3.5 describes exactly the three modules the README names). Nemotron 3 Super is the most complete open recipe; OLMo 3 is the most complete open pipeline. Any of the three could lead.
Whether to treat "Serve" as one stage or two. Rollout engines inside RL and user-facing serving share code but not operational contracts. The page keeps one column and notes the dual role; splitting it would give agentic RL a cleaner infrastructure story.
Family entries. PostTrainBench (28), InferenceBench (4), WeirdML (11) and EdgeBench (51 public) count once each. Counting instances would triple the post-training column without adding verifier diversity.
Agent-reported numbers (DeepSeek-V4 sandbox counts, Kimi K2.5 100,000 concurrent tasks, LongCat 32,000 environments, Qwen3.5 claims) are on the page with dashed tags. They should be re-read before any of them reaches a paper.
Supplemental benchmarks (ISO-Bench, FlashInfer-Bench, CommBench, ATE-Bench, KernelBench) are rings, not counts. Promoting any of them to a primary row needs a task-level read.

5. Revision 2.1 (22 September 2026)

Applied from the user's review of the page: prose now runs at full width; six older or thinner reports (Llama 3, DeepSeek-V3/R1, Llama 4, Apertus, LLM360 K2, Arcee Trinity) are hidden behind a toggle and dimmed when shown; Mercor stays visible. Continued pretraining, annealing and long-context extension were merged into the pretraining column because they share the objective and trainer and no audited task targets them separately. On-policy distillation joined the post-training vocabulary with four code chips (NeMo-RL's on-policy and X-token distillation recipes, TRL's GKDTrainer, Thinking Machines' Tinker post, Xiaomi's MOPD/MOPD2 paper) after re-reading each source. The Marin row now cites the Open Athena 535B launch note (August 2026 start on CoreWeave GPUs with JAX/XLA:GPU, 18T tokens, MFU 21% → 24%, preregistered loss predictions; Snowball 67B-A2B in post-training). The RL loop moved to two deep-dive pages: posttrain.html (recipes as sequences for all 18 pipelines, frameworks by sub-stage, twelve stabilizers with sources) and agentic.html (the loop with 59 implementations, ten fleets, twelve staleness entries, ten ledger rows, seven RL-Infra-Bench families). The grid no longer highlights the agentic column.

6. Revision 2.2 (22 September 2026)

The user asked that the pages read as a review rather than as the case for a benchmark under construction. Every mention of RL-Infra-Bench on the pages was removed: the coverage figure's target box and key item, the loop drawer's "families" line, and the agentic page's blueprint-derived table, which is now "Mechanisms that only exist across devices" with three columns (mechanism, why it needs several devices, what documents it). The overview's post-training section became three blocks: the RL loop (new page loop.html, with the figure, drawer and a component table), post-training recipes, and agentic RL infrastructure. The mapping from incidents to benchmark mechanism families remains in this note's §3 and in mimo_v26_review.md §8 for the benchmark work, off the pages.

7. Revision 2.3 (22 September 2026)

After the user's review of the published draft: the coverage figure moved to the top of the overview, the title became "AI-Infra Environment Coverage" with the question as a subtitle, every mono subhead was removed, captions and section text were cut to about half (overview prose from roughly 700 to 480 words), and the changelog became a working-copy appendix that deploy_site.py strips before publishing. Scale's RSI Bench taxonomy (ten contributor categories, no definitions published as of 22 September) was compared with the stage × layer frame and left unadopted; a one-line note distinguishes RSI Bench from the audited RSI-Exam.