RDMA Networking Walkthrough
Request and verify remote direct memory access (RDMA) endpoints for workloads, align network adapters with GPUs on non-uniform memory access (NUMA) nodes, and configure multi-tenant and cache fabric interconnects.
Contents
- Summary
- Environment
- Step 1: Request RDMA
- Step 2: Align NUMA with kubelet
- Step 3: Request interfaces per role
- Step 4: KV cache over RDMA
- Step 5: Apply AWS EFA differences
- Limitations
- Troubleshooting
Summary
Device Manager discovers network interfaces from /sys/class/net and verbs devices from
/sys/class/infiniband, including devices without a netdev. It verifies link status and advertises
shared, exclusive and single-root I/O virtualization (SR-IOV) partitioned resource keys.
A workload can request network fabric beside its accelerators.
When at least one endpoint is active, the node carries feature.gpustack.ai/rdma.capable=true (see
Network Topology
and
RDMA resource keys
).
The steps verify interface discovery, container injection, NUMA alignment, and deployment integration.
| Object | Scope | Creator | Purpose |
|---|---|---|---|
Pod |
namespace | user | a workload that requests RDMA endpoints beside its accelerator |
ModelDeployment |
namespace | user | per-role interface requests under roles[].resources.interface |
KVCacheBackend |
cluster | administrator | the cache transport carried over RDMA or EFA |
Environment
| Item | Value |
|---|---|
| Cluster | Kubernetes with GPUStack installed |
| Hardware | RDMA-capable network adapters on the workload nodes |
Step 1: Request RDMA
Inspect discovered interfaces and allocatable capacities on the target node first:
kubectl get devices <node> -o json |
jq '[.spec.interfaces[] | (., (.virtualFunctions // [])[])]
| map(select(.rdma or .link)) | map({name, pciBusId, rdmaDevice, link})'
kubectl get node <node> -o json |
jq '.status.allocatable | with_entries(select(.key | contains("gpustack.ai/rdma")))'A workload requests whole accelerators and RDMA endpoints within the same container.
Pods must carry the kueue.x-k8s.io/queue-name label to participate in admission.
Name the entrance LocalQueue of the pool that holds the accelerators.
LocalQueue names look like gpustack-fnv64-<hash>. List them with kubectl -n team-a get localqueues:
apiVersion: v1
kind: Pod
metadata:
name: trainer-0
namespace: team-a
labels:
kueue.x-k8s.io/queue-name: gpustack-fnv64-<hash>
spec:
restartPolicy: Never
containers:
- name: trainer
image: ubuntu:24.04
command:
- "sleep"
- "3600"
resources:
limits:
nvidia.com/gpu: "1"
device.gpustack.ai/rdma.shared: "1"The allocated container receives character devices, an exact NCCL_IB_HCA filter and
GPUSTACK_RDMA_DEVICES, the plain list of granted HCA names in every resource mode.
See Injected devices
for variable values,
mixed-family limits and application startup requirements.
Prefer device.gpustack.ai/rdma.shared unless the workload needs exclusive-family endpoint ownership.
GetPreferredAllocation distributes shared tokens across distinct eligible HCAs in round-robin
order before assigning a second token on one HCA. The exclusive key device.gpustack.ai/rdma
reserves whole endpoints within its family; shared workloads can still use those HCAs.
Fewer uverbs devices than requested shared units occur when the request exceeds the number of
eligible HCAs, or when policy excludes granted endpoints. Inspect the
selection policy status
if grants change
after a policy update.
Step 2: Align NUMA with kubelet
Kubernetes does not pair specific accelerators with adjacent network adapters by default.
Enable NUMA alignment by setting the TopologyManager policy in /var/lib/kubelet/config.yaml
(see Enabling NUMA alignment
):
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
+topologyManagerPolicy: restricted
+topologyManagerScope: container
Restart the kubelet after saving the configuration:
systemctl restart kubelet
kubectl get --raw /api/v1/nodes/<node>/proxy/configz |
jq -r '.kubeletconfig | {topologyManagerPolicy, topologyManagerScope}'restricted ensures that single-socket requests land on one NUMA node, while multi-GPU workloads
spanning multiple sockets are packed into the minimum number of NUMA nodes.
Step 3: Request interfaces per role
In a ModelDeployment, roles do not declare raw resource keys. Specify the number of network
interfaces under roles[].resources.interface.
The operator refuses an interface request when no RDMA or EFA transfer path uses it.
A managed prefill and decode pair with interface requests on both roles is such a path:
apiVersion: worker.gpustack.ai/v1
kind: ModelDeployment
metadata:
name: qwen-pd
namespace: team-a
spec:
model:
name: Qwen/Qwen2.5-7B-Instruct
engine:
name: vLLM
image: gpustack/runner:cuda13.0-vllm0.29.0
router:
name: llm-d-router
kvTransfer:
backend: Mooncake
roles:
- name: prefill
kind: Prefill
replicas: 1
acceleratorType: gpustack--nvidia-a10g-linux-amd64
resources:
accelerator: "2"
interface: "2"
- name: decode
kind: Decode
replicas: 1
acceleratorType: gpustack--nvidia-a10g-linux-amd64
resources:
accelerator: "2"
interface: "2"For the RDMA protocol, the operator translates interface: 1 into device.gpustack.ai/rdma.shared: 1.
When interface is greater than 1, it renders device.gpustack.ai/rdma: N.
Each allocated endpoint then corresponds to an independent physical adapter
(see A minimal deployment
).
The same request without router or a cache backend that uses RDMA fails at admission with
no effective RDMA or EFA transfer leg uses the requested interfaces.
A cache backend on RDMA is the other path, and Step 4 sets it up.
Step 4: KV cache over RDMA
To route KV cache transport over an RDMA fabric instead of TCP, edit the mooncake-dram backend
from Step 1 of the Shared Cache walkthrough
.
The transport is editable on a running backend:
apiVersion: worker.gpustack.ai/v1
kind: KVCacheBackend
metadata:
name: mooncake-dram
spec:
type: Mooncake
+ transport:
+ protocol: RDMA
connection:
managed:
leader: {}
members:
- nodeSelector:
kubernetes.io/os: linux
medium: DRAM
capacityPerMember: 8Gi
+ fabricInterfaceCount: 2
Member DaemonSet Pods automatically receive the network devices, host privileges, and limits
required for high-performance transport (see Members
).
Step 5: Apply AWS EFA differences
On AWS Elastic Fabric Adapter (EFA) instances, RDMA devices bypass /sys/class/net and do not expose
standard verbs completion queues. Declare the EFA protocol explicitly on a backend for those nodes:
apiVersion: worker.gpustack.ai/v1
kind: KVCacheBackend
metadata:
name: mooncake-efa
spec:
type: Mooncake
transport:
protocol: EFA
connection:
managed:
leader: {}
members:
- nodeSelector:
node.kubernetes.io/instance-type: c6in.32xlarge
medium: DRAM
capacityPerMember: 64Gi
fabricInterfaceCount: 2Enable the aws-efa-enabled Setting for ModelDeployment and specify custom runner images containing
libfabric and EFA-enabled Mooncake builds.
Limitations
- Kubelet NUMA alignment policies (
none,restricted,single-numa-node) andTopologyAffinityErroradmission behavior are covered in Enabling NUMA alignment and Reading the result . - Kueue network-resource metering and over-admission behavior are covered in RDMA Operations and Troubleshooting .
- AWS EFA discovery and transport configuration are covered in Step 5 and Troubleshooting .
Troubleshooting
| Symptom | Signal | Check & Mitigation |
|---|---|---|
RDMA resource keys present but all Unhealthy |
Status: links Failed or HCAs unselected by policy |
Inspect kubectl get devices <node> -o json for link verdicts and selection policy status
; verify physical cable, subnet manager and policy filters |
Workload admitted by Kueue but Pod remains Pending |
Over-admission: Kueue does not meter network resources | Inspect cluster allocatable RDMA keys (device.gpustack.ai/rdma*); add nodes or wait for active Pods to finish |
Pod rejected at placement |
Reason: TopologyAffinityError — free GPUs and NICs reside on mismatched NUMA nodes |
Ensure kubelet has topologyManagerPolicy: restricted; verify GPU and NIC requests are declared in the same container |
Fewer uverbs devices inside container than requested shared units |
Request exceeds eligible HCA count, or policy excludes granted endpoints | GetPreferredAllocation spreads rdma.shared tokens across distinct eligible HCAs first, then repeats HCAs. Check eligible capacity and policy status
. Use device.gpustack.ai/rdma for whole-endpoint ownership within the exclusive family |
Node unexpectedly loses feature.gpustack.ai/rdma.capable |
Status: every endpoint on the node reports Failed or lacks a bound RDMA device |
Check kernel dmesg and interface carrier status; operator removes the label when every endpoint reports Failed or lacks a bound RDMA device. See Network Topology
|
AWS EFA workload remains Pending indefinitely |
Workload requested device.gpustack.ai/rdma instead of EFA |
EFA nodes expose 0 allocatable for standard RDMA keys; configure transport.protocol: EFA or TCP |
ModelDeployment rejected with no effective RDMA or EFA transfer leg uses the requested interfaces |
Interface requested with no RDMA or EFA path | Request interfaces on both roles of a managed prefill and decode pair, or use a KVCacheBackend on that transport. A role with its own command has no managed direct path |