Weight Sync

Weight sync transfers the current training model weights to one or more inference servers (for example, xorl-sglang) running on separate GPUs. The transfer backend is selected by sync_inference_method in the server configuration.

Overview

Training GPUs (XoRL server)
    │  configured transport
    ▼
Inference GPUs (xorl-sglang)

Backend	Status	Data path	Direct EP transfer
nccl_broadcast	Default	Training rank 0 broadcasts prepared buckets to inference TP workers	No
p2p	Supported when Mooncake/RDMA prerequisites are installed on both sides	One-sided P2P writes into registered receiver parameter memory	Optional, when direct-EP mode is enabled
sparse_delta	Not supported by the pinned xorl-sglang revision	The training-side sender expects /update_weights_from_sparse_delta, which the pinned receiver does not provide	N/A

Do not select sync_inference_method: sparse_delta with the checked-in submodules: the request will reach a missing receiver route. Treat the backend as unavailable until xorl-sglang includes and validates the matching receiver module and HTTP endpoint.

The diagram below shows the default nccl_broadcast path. Its dedicated NCCL group is initialized and destroyed within each sync operation; that lifecycle is not a contract shared by the P2P backend.

Registering Inference Endpoints

Before syncing, register the inference server’s address with xorl:

POST /add_inference_endpoint

{"host": "inference-node-01", "port": 30000, "worker_port": 30000, "world_size": 8}

Multiple endpoints can be registered (e.g. for multi-replica inference):

for host, port in inference_servers:
    requests.post(f"{base_url}/add_inference_endpoint", json={
        "host": host,
        "port": port,
        "worker_port": port,
        "world_size": 8,
    })

Endpoints are deduplicated by (host, port) — registering the same endpoint twice is safe. For the common same-port xorl-sglang setup, set worker_port equal to port.

Triggering a Sync

POST /api/v1/sync_inference_weights
POST /sync_inference_weights          # shorthand alias

{
  "master_address": "training-node-01",
  "master_port": 0,
  "group_name": "sync_group_0",
  "buffer_size_mb": 512,
  "pause_mode": "retract",
  "flush_cache": true,
  "cache_invalidation_mode": "flush"
}

Field	REST default	Description
master_address	""	Training-head address used for rendezvous; auto-detected when empty
master_port	0	Rendezvous port; 0 selects an ephemeral port. The xorl-client helper defaults to 29600
group_name	"weight_sync_group"	Group name used by transports that require a process group
buffer_size_mb	1024	Transfer bucket size; reduce it if sync preparation exceeds memory
pause_mode	"in_place"	in_place, retract, or abort. Use retract when cached or in-flight state must be invalidated; in_place is valid only when orchestration proves no request or reusable cache entry crosses the weight-version boundary
flush_cache	false	Explicitly request receiver KV/prefix-cache invalidation. Set this to true unless the caller enforces the no-cross-version-reuse invariant described below
cache_invalidation_mode	"auto"	auto adds a flush only for the detected FP8-weight/FP8-KV combination; it is not a general cache-safety policy. none is an explicit no-flush choice and requires the caller to prevent cross-version state reuse

The invariant is no KV, prefix-cache entry, or in-flight decode state computed under one weight version may be consumed under another. The pinned receiver does not include weight_version in cache keys; it is control-plane metadata only. A flush is the general way to enforce the invariant: use pause_mode: "retract", cache_invalidation_mode: "flush", and flush_cache: true, as shown above. A deliberate no-flush update is also valid when the caller can prove there are no in-flight requests and no old-version cache entries that later requests can reuse—for example, a freshly started, isolated endpoint that has served no generation before its one update. in_place/none are not safe substitutes for that orchestration guarantee.

The sync is synchronous from the caller’s perspective — the endpoint returns once all inference servers have received the weights.

Sync in RL Training Loop

# Every N RL steps, sync weights to inference
if rl_step % sync_every == 0:
    training_client.sync_inference_weights(
        master_address=TRAINING_HEAD_IP,
        master_port=29600,
        group_name="policy_sync",
    ).result()
    # Now SGLang is serving the latest policy weights

Quantization During Sync

Optionally quantize weights during the sync to reduce transfer bandwidth and match inference precision:

POST /api/v1/set_sync_quantization

{
  "quantization": {
    "quant_method": "fp8",
    "fmt": "e4m3",
    "weight_block_size": [128, 128],
    "modules_to_not_convert": ["lm_head", "embed_tokens"]
  }
}

Set quantization to null for BF16 transfer. Online quantization currently accepts the Slime/SGLang-compatible block-FP8 E4M3 format with FP32 inverse scales; unsupported INT4, AWQ, compressed-tensors, and fake-quant formats fail before transport starts. The receiver installs the transferred FP8 tensors and scale metadata rather than treating them as BF16 weights.

Sync with LoRA

For LoRA training, the sync merges LoRA weights into the base model before broadcasting:

• W_full = W_base + lora_B @ lora_A × scaling
• The merged BF16 weight is sent to inference

The base weights on the training side are not modified — LoRA parameters remain separate for continued training.

Sync with QLoRA

For QLoRA, the sync dequantizes and merges:

• W_full = dequant(W_packed) + correction_U @ correction_B + lora_B @ lora_A × scaling
• Optionally re-quantizes to FP8 for transfer

Removing Inference Endpoints

POST /remove_inference_endpoint
{"host": "inference-node-01", "port": 30000}

Sleep / Wake Pattern

When the training server needs all GPU memory (e.g. for a large training step), put inference servers to sleep:

# Free inference GPU memory
requests.post(f"{inference_url}/sleep")

# Do training
for _ in range(n_steps):
    training_client.forward_backward(...)
    training_client.optim_step(...)

# Sync and resume inference
training_client.sync_inference_weights(...).result()
requests.post(f"{inference_url}/wake_up")

Architecture Details

The sync handler (server/weight_sync/handler.py) proceeds in order:

1. Health check: Verify all inference endpoints are reachable
2. Backend init: Initialize the configured NCCL or P2P transport
3. Per-module transfer (sequential across PP stages if PP > 1):
   • FSDP2 all-gather to reconstruct full parameter on rank 0
   • QLoRA dequantize + LoRA merge
   • Optional FP8 requantization
   • Transfer through the selected backend
4. Resume inference: Signal inference servers to resume

Multi-PP Support

For ordinary PP-sharded training, each stage’s parameters are prepared in sequence and routed to the backend sender. Virtual/interleaved PP creates multiple model chunks per rank and is not currently supported by weight sync.

EP Support

The default NCCL backend gathers expert material to rank 0 before transfer. The P2P backend can send from multiple EP ranks only when direct-EP mode is enabled and its sender/receiver topology checks pass. The training-side sparse-delta sender is rank-0 based but is not usable with the pinned xorl-sglang receiver.

Source

File	Description
src/xorl/server/weight_sync/handler.py	WeightSyncHandler — orchestrates the full sync pipeline
src/xorl/server/weight_sync/backends/	Transport backend implementations