DeepEP

DeepEP is a GPU-resident expert-parallel dispatch backend. It uses GPU interconnects such as NVLink within a node and NVSHMEM/GPUDirect RDMA on supported multi-node deployments. Whether it outperforms the default alltoall path depends on the model, token shape, EP topology, fabric, and SM allocation.

Requirements

• Supported NVIDIA GPUs and an admitted intra-node or multi-node fabric
• deep_ep wheel installed (see Installation)
• ep_dispatch: deepep in model config
• For internode (multi-node) EP: nvidia_peermem kernel module loaded and IBGDA enabled on all nodes (see below)

Installation

pip install deep_ep-*.whl   # use a wheel compatible with the selected CUDA/PyTorch profile

Verify:

import deep_ep
print("DeepEP available")

Cluster Prerequisites (multi-node)

For single-node EP (all GPUs on one machine), the wheel alone is sufficient. For multi-node EP, NVSHMEM handles inter-node RDMA communication and requires two additional steps on every node.

1. Load nvidia_peermem

nvidia_peermem bridges the NVIDIA driver and the InfiniBand stack to enable GPUDirect RDMA. Without it, NVSHMEM cannot register GPU buffers with IB HCAs and DeepEP will crash at the first dispatch with SIGABRT and errors like:

WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...

Load it on every node:

sudo modprobe nvidia_peermem

Verify:

lsmod | grep nvidia_peermem

To persist across reboots, add it to /etc/modules:

echo nvidia_peermem | sudo tee -a /etc/modules

2. Enable IBGDA in the NVIDIA driver

IBGDA allows NVSHMEM to issue RDMA operations directly from GPU SM threads without CPU involvement. Add the following to /etc/modprobe.d/nvidia.conf on every node:

options nvidia NVreg_EnableStreamMemOPs=1 NVreg_RegistryDwords="PeerMappingOverride=1;"

Then rebuild the initramfs and reboot:

sudo update-initramfs -u
sudo reboot

Verify after reboot:

sudo cat /proc/driver/nvidia/params | grep -E "EnableStreamMemOPs|RegistryDwords"
# EnableStreamMemOPs: 1
# RegistryDwords: "PeerMappingOverride=1;"

Important: nvidia_peermem must be reloaded after every reboot (via /etc/modules or modprobe) — the IBGDA driver settings alone do not load it automatically.

Configuration

model:
  ep_dispatch: deepep
  deepep_buffer_size_gb: 2.0    # communication buffer pool per GPU (default: 2.0)
  deepep_num_sms: 20            # SMs dedicated to communication (default: 20)
  deepep_async_combine: false   # async combine overlap (experimental)

Parameter	Default	Description
ep_dispatch	alltoall	Set to deepep to enable
deepep_buffer_size_gb	2.0	Per-GPU communication buffer pool in GB. Larger buffers can reduce chunking but consume more resident memory.
deepep_num_sms	20	SMs dedicated to communication kernels. Must be even.
deepep_async_combine	false	Overlap combine with next layer’s compute (experimental, currently disabled — see Async Combine)

---

How DeepEP Works

The default path uses a device-side distributed AllToAll collective:

GPU 0 → collective send/receive buffers → GPU 1

DeepEP uses GPU-initiated transport and specialized dispatch/combine kernels:

GPU 0 → NVLink within a node, or NVSHMEM/GPUDirect RDMA across nodes → GPU 1

The ordinary XoRL AllToAll path does not stage token payloads through host memory. Compare the two backends on the intended topology rather than assuming a fixed latency multiplier.

---

SM Allocation Strategy

DeepEP dedicates a fixed number of SMs to communication kernels. This creates a direct tradeoff between communication bandwidth and compute throughput:

deepep_num_sms	Communication bandwidth	Compute SMs remaining	Best for
8	Smaller communication allocation	More	Candidate when expert compute dominates
20 (default)	Default starting point	Default starting point	Initial bring-up and measurement
32	Larger communication allocation	Fewer	Candidate when dispatch/combine dominates
48+	High communication allocation	Fewer	Use only after a shape-matched profile shows a benefit

deepep_num_sms must be even. Start with 20 and tune based on profiling — if XORL_DEBUG_EP=1 shows dispatch time > compute time, increase SMs; if compute time dominates, decrease.

---

Buffer Size Tuning

The buffer is initialized lazily and shared across MoE layers. deepep_buffer_size_gb supplies the minimum NVLink allocation. For EP sizes supported by the installed DeepEP version, XoRL also asks DeepEP for dispatch and combine size hints and uses the maximum of those hints and the configured minimum. Older DeepEP versions or unrecognized EP sizes fall back to the configured value.

For an EP group spanning nodes, the implementation also allocates the RDMA buffer from DeepEP’s hints and rejects an NVLink allocation above DeepEP’s signed-int32 limit; keep deepep_buffer_size_gb <= 2.0 for that path. An initialization or dispatch allocation failure is a sizing error—XoRL does not implement the token-volume chunking formula previously shown here.

---

Async Combine (Experimental)

Currently disabled

The async combine path is forced off in code. The combined tensor is consumed by the rest of the transformer block (residual add, norm, attention) on the default stream before the next token_pre_dispatch() runs, so deferring event.current_stream_wait() races those consumers. Setting deepep_async_combine: true alone is a no-op (logs a one-time warning). To actually run async, also export XORL_DEEPEP_UNSAFE_ASYNC_COMBINE=1 — only do this if you have separately ensured the downstream consumer waits on the combine event.

deepep_async_combine: true requests overlap of the combine communication (outputs flowing back from expert ranks) with the next layer’s compute:

Step N:    dispatch → compute → [combine starts]
Step N+1:  [combine finishes] + next layer compute (overlapped)

Benefit (when working): Hides combine latency behind useful compute, especially when combine > dispatch (typical for large output projections).

Limitations:

• Gated off behind XORL_DEEPEP_UNSAFE_ASYNC_COMBINE=1
• Requires careful ordering of CUDA streams
• Not compatible with pipeline parallelism (PP > 1)

---

When to Use DeepEP vs AllToAll

Scenario	Recommendation
Bring-up, portability, or missing DeepEP/NVSHMEM prerequisites	Start with alltoall
DeepEP prerequisites pass and dispatch/combine dominates a shape-matched profile	Benchmark DeepEP with the same model, batch, sequence, EP placement, and fabric
Multi-node deployment	Validate nvidia_peermem, IBGDA, HCA reachability, and NVSHMEM before comparing throughput
Async combine	Keep disabled unless the downstream stream-ordering contract has been independently validated

---

Troubleshooting

deep_ep not found:

AttributeError: module 'deep_ep' has no attribute 'Buffer'

Install a DeepEP wheel matching the selected CUDA, PyTorch, GPU, and fabric environment.

SIGABRT / num_recv_tokens: -1 on all ranks:

WARN: device mlx5_0 cannot allocate buffer on the specified memory type. Skipping...
Global rank: 0, num_recv_tokens: -1, num_rdma_recv_tokens: -1

nvidia_peermem is not loaded. Run sudo modprobe nvidia_peermem on all nodes. See Cluster Prerequisites.

init failed for transport: IBGDA: IBGDA driver settings are not active. Check that NVreg_EnableStreamMemOPs=1 and PeerMappingOverride=1 are set in /etc/modprobe.d/nvidia.conf, then run sudo update-initramfs -u and reboot. Verify with:

sudo cat /proc/driver/nvidia/params | grep EnableStreamMemOPs

Buffer initialization OOM: Reduce deepep_buffer_size_gb. Check available GPU memory before the EP buffer allocation.

SM contention (low compute throughput): Reduce deepep_num_sms to 8–12. Use XORL_DEBUG_EP=1 to print per-phase timing:

XORL_DEBUG_EP=1 torchrun ... -m xorl.cli.train config.yaml

Shape mismatch errors during backward: With the default gradient_checkpointing_method: recompute_full_layer, R3 routing replay is used automatically to reuse the recorded expert assignments when AllToAll is recomputed. If you see shape mismatches, verify routing replay is enabled. Using recompute_before_dispatch or no_recompute avoids AllToAll recomputation and does not require local forward/backward route replay. See MoE Routing Replay.

No fallback to AllToAll: If DeepEP fails to initialize, xorl does not fall back automatically. Set ep_dispatch: alltoall explicitly.

---

Source

File	Description
src/xorl/distributed/moe/deepep.py	DeepEP buffer lifecycle, dispatch/combine autograd boundaries, and internode preflight
src/xorl/models/layers/moe/moe_block.py	MoEBlock — DeepEP dispatch/combine integration, async combine stream management
src/xorl/models/layers/moe/experts.py	MoEExperts._ep_forward() — DeepEP dispatch, compute, combine phases; XORL_DEBUG_EP timing