Inference: xorl-sglang

xorl-sglang is Together’s fork of SGLang, the primary supported inference backend for xorl RL training. It extends SGLang with the APIs and data export capabilities needed to integrate tightly with the xorl training server.

---

Why a Fork?

The XoRL RL integration requires the following capabilities from its pinned inference engine:

1. Weight updates from training — the training server must push new weights to the inference server after policy updates using the selected NCCL or P2P transport. The pinned revision does not contain a sparse-delta receiver.

2. Decision-time per-token logprob export — RL losses require the log probability from the distribution that selected each retained token, returned alongside the completion.

3. MoE routing data export (R3) — supported MoE paths can export expert choices so the trainer can run a route-conditioned recomputation. Reusing routes does not by itself prove logprob parity or gradient correctness.

4. Numerical-program controls — for admitted architectures, --rl-on-policy-target xorl resolves the router, LM head, RMSNorm/RoPE, attention, and related arithmetic as one model-specific program. The target name is configuration; exactness remains revision-, model-, topology-, and trace-specific evidence.

---

What Was Modified

The xorl-sglang revision pinned by this checkout carries the XoRL integration. This page describes the stable integration surfaces of that exact pin rather than assuming it matches xorl-sglang main or quoting line/file counts that can drift.

1. Weight Update Endpoints and Protocol

xorl-sglang exposes a shared two-phase receiver for NCCL broadcast and Mooncake P2P. The pinned HTTP surface is:

Endpoint	Description
POST /init_weights_update_group	Join an NCCL process group for weight sync. Forces eager NCCL communicator creation with device_id and NCCL_CUMEM_ENABLE=0 to match xorl’s training side.
POST /update_weights_from_distributed	Receive broadcasted weight tensors via NCCL dist.broadcast. Called per weight bucket.
POST /prepare_weights_update	Phase 1: for NCCL, arms background receive threads; for transport: "p2p", returns registered Mooncake tensor locators and receiver-engine metadata.
POST /complete_weights_update	Phase 2: applies a prepared NCCL receive or finalizes a P2P session, then performs the requested cache/version post-processing.
POST /destroy_weights_update_group	Tear down the NCCL group after sync completes.

Key implementation details:

• Eager NCCL init: Both sides must use device_id to force eager communicator creation. Without this, sglang uses lazy init and xorl’s rank 0 hangs waiting for peers.
• Two-phase protocol: prepare_weights_update starts background recv threads, then complete_weights_update applies them after the training-side broadcast finishes. This avoids blocking the scheduler.
• P2P receiver: With transport: "p2p", the same endpoints expose registered parameter locations so Mooncake can write directly into receiver memory. Direct-EP use remains subject to the training-side topology and receiver mapping checks.
• Health check bypass: By default, /health returns 200 without running a test generation; /health_generate exercises generation when an active check is required. This avoids control-plane timeouts while NCCL operations are in flight.

The pinned revision does not expose /receive_weights, /receive_weights_ep_scatter, /list_weights, or /update_weights_from_sparse_delta. Do not build integrations against those routes for this checkout.

The training server’s nccl_broadcast backend drives these calls — see Backend: nccl_broadcast for the full protocol.

2. MoE Routing Data Export (R3)

Routing capture is implemented by the state capturer, scheduler output, and tokenizer response paths in the pinned submodule.

For MoE models, xorl-sglang records which experts each token was routed to during generation and returns this alongside completions:

# Returned in meta_info
routed_experts = completions[i].meta_info["routed_experts"]
# Shape: [num_tokens, num_layers, top_k], dtype: int32, base64-encoded

The training server can decode and replay these routing decisions so the route-conditioned trainer computation uses the exported expert assignments. That controls a discrete selection surface; it is not independent evidence that the trainer and sampler logprobs, expert arithmetic, gradients, or updated weights agree.

Format: xorl-sglang encodes routing indices as raw base64 int32 bytes to minimize transfer overhead:

{
  "routed_experts": "<base64_encoded_int32_array>"
}

The payload does not include routing weights or separate shape metadata. XoRL’s RoutingReplayHandler decodes it and infers the shape from the retained-token count, MoE layer count, and model top-k before distributing it across context-parallel and packing dimensions.

3. Numerical Alignment

Exact on-policy work requires the trainer’s retained-token logprob bytes to agree with the decision-time sampler bytes for the tested revision pair. In the pinned xorl-sglang revision, admitted dense Qwen3, Qwen3.5-family, GLM-5.2, and DSV4-Flash programs are architecture-owned: --rl-on-policy-target xorl derives and validates their precision, topology, attention, routing, cache, graph, and sampling settings.

Reduction order is part of the exact contract. Qwen, GLM, and DSV4-Flash now share the versioned balanced adjacent-pair BF16 fold. DSV4 keeps its model-specific variable-row transport and exact Marlin chunking, but its rank-ordered partials feed the same canonical fold as the other admitted MoE programs.

--enable-fp32-lm-head remains a public generic control. Router precision for the admitted exact MoE programs is resolved internally; the pinned parser has no --enable-fp32-router option. It also has no --enable-return-expert-logits option.

# xorl-sglang launch
python -m sglang.launch_server \
    --model-path <supported-model> \
    --rl-on-policy-target xorl \
    --port 30000

The resolver rejects incompatible options for an admitted architecture rather than silently selecting a different program. A successful launch is still not a parity certificate; validate the intended revision pair and workload with decision-time logprob artifacts.

4. Exact XoRL Mode

Modified files: model_runner.py, batch_invariant_ops.py

When --rl-on-policy-target xorl is set, xorl-sglang resolves the architecture-owned exact numerical path for the loaded model. Validation for the intended model, shapes, topology, and batching states is still required; the target name alone is not a parity certificate.

5. Bug Fixes

Two additional fixes on top of the upstream merge:

• req_to_token_pool slot leak (schedule_batch.py, scheduler.py): When a max_new_tokens=0 (prefill-only) request arrives during an idle window, its ScheduleBatch gets is_prefill_only=True. If normal generation requests are later merged in, merge_batch() never cleared this flag, so get_next_batch_to_run() skipped the decode path. Requests allocated pool slots during prefill but never decoded, never finished, and never freed their slots — exhausting the pool. Fixed by clearing is_prefill_only on merge and recomputing it from actual request state.

KV Cache Flush on Weight Update

The pinned receiver does not associate KV or prefix-cache entries with weight_version; that value is control-plane metadata only. The safety invariant is that no cached or in-flight state computed under one version is consumed under another. Flushing with pause_mode: retract is the general enforcement mechanism. A no-flush update is valid only when external orchestration proves the endpoint has no in-flight work and no old-version entry that can be reused. See the weight-sync overview for both cases.

---

Installation

xorl-sglang is included as a git submodule under submodules/xorl-sglang. If you cloned with --recurse-submodules, it’s already checked out.

pip install -e "submodules/xorl-sglang/python[all]"

Or use pyproject.sglang.toml to install xorl, xorl-client, and xorl-sglang together with the pinned PyTorch 2.11/Transformers 5.12/FlashAttention 4 stack:

cp pyproject.sglang.toml pyproject.toml
UV_PROJECT_ENVIRONMENT=.venv-sglang uv sync
source .venv-sglang/bin/activate

See the installation guide for full details.

Launching xorl-sglang

Single GPU (Qwen3-8B FP8)

python -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B-FP8 \
    --port 30000 \
    --rl-on-policy-target xorl \
    --enable-fp32-lm-head \
    --mem-fraction-static 0.88

Tensor Parallel (Qwen3-30B FP8, 2 GPUs)

CUDA_VISIBLE_DEVICES=4,5 python -m sglang.launch_server \
    --model-path Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 \
    --port 30000 \
    --tp-size 2 \
    --rl-on-policy-target xorl \
    --enable-fp32-lm-head \
    --mem-fraction-static 0.88

Tensor Parallel (Qwen3-235B FP8, 4 GPUs, remote node)

python -m sglang.launch_server \
    --model-path Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 \
    --port 30000 \
    --host 0.0.0.0 \
    --tp-size 4 \
    --rl-on-policy-target xorl \
    --enable-fp32-lm-head \
    --mem-fraction-static 0.88

Key launch flags

Flag	Description
--rl-on-policy-target xorl	Select deterministic inference and the architecture-owned XoRL numerical program when the model has one. Weight-sync endpoints are independent of this flag.
--enable-fp32-lm-head	Request FP32 LM-head logits on generic paths; admitted architecture-owned programs resolve their required head precision automatically.
--enable-return-routed-experts	Return expert routing indices in response metadata (for R3)
--enable-rdma-weight-updates	Initialize the Mooncake receiver used by XoRL’s P2P weight-sync backend.
--tp-size N	Tensor parallelism across N GPUs
--mem-fraction-static 0.88	Fraction of GPU memory for KV cache (leave headroom for weight sync buffers)

Wait for ready

import requests, time
while True:
    try:
        r = requests.get("http://localhost:30000/health")
        if r.status_code == 200:
            break
    except:
        pass
    time.sleep(2)

---

Registering with the Training Server

Before weight sync can happen, register the xorl-sglang instance with the training server:

import requests

requests.post("http://training-server:6000/add_inference_endpoint", json={
    "host": "inference-node-01",
    "port": 30000,
    "worker_port": 30000,
    "world_size": 4,   # match --tp-size
})

Multiple replicas can be registered — weight sync broadcasts to all of them in parallel:

for host, port, tp_size in inference_replicas:
    requests.post("http://training-server:6000/add_inference_endpoint", json={
        "host": host,
        "port": port,
        "worker_port": port,
        "world_size": tp_size,
    })

---

Sleep / Wake for Memory Sharing

On nodes where training and inference share GPUs, use sleep/wake to hand off GPU memory:

# Before a large training step: free inference GPU memory
requests.post("http://inference-node:30000/sleep")

# Run training steps
for _ in range(n_train_steps):
    training.forward_backward(...)
    training.optim_step(...)

# Sync new weights and resume inference
training.sync_inference_weights(master_address=TRAIN_HOST, master_port=29600).result()
requests.post("http://inference-node:30000/wake_up")

---

Upstream Compatibility

xorl-sglang tracks upstream SGLang and periodically rebases. Integration work is concentrated in these areas:

Area	Files modified	Nature of change
Weight sync protocol	Weight updater, tokenizer control, HTTP server, and P2P receiver paths	Two-phase NCCL and Mooncake P2P receive/update surfaces; no sparse-delta receiver
Routing data export	State capturer and tokenizer response paths	Base64 int32 routed-expert indices; no routing-weight export
Numerical alignment	Architecture-specific model, kernel, and server-argument resolvers	Fail-closed programs selected by --rl-on-policy-target xorl
CLI args	server_args.py	Current public controls include --rl-on-policy-target, --enable-fp32-lm-head, --enable-return-routed-experts, and --enable-rdma-weight-updates
Bug fixes	schedule_batch.py, scheduler.py	Prefill-only slot leak fix

Use the submodule commit pinned by the XoRL checkout. Do not infer compatibility from an unpinned xorl-sglang or upstream SGLang branch; if a needed feature is absent from the pinned revision, file an issue in the xorl-sglang repository.

Source

Repo	Description
togethercomputer/xorl-sglang	xorl’s SGLang fork — NCCL/P2P weight-update API, R3 route export, and architecture-owned numerical programs
src/xorl/server/weight_sync/backends/nccl_broadcast.py	Training-side NCCL broadcast implementation that drives the SGLang weight update endpoints
src/xorl/server/runner/utils/routing_replay_handler.py	Decodes R3 routing data from SGLang and distributes it across SP/packing dimensions