RDMA Networking Walkthrough

Request and verify remote direct memory access (RDMA) endpoints for workloads, align network adapters with GPUs on non-uniform memory access (NUMA) nodes, and configure multi-tenant and cache fabric interconnects.

Contents

• Summary
• Environment
• Step 1: Request RDMA
• Step 2: Align NUMA with kubelet
• Step 3: Request interfaces per role
• Step 4: KV cache over RDMA
• Step 5: Apply AWS EFA differences
• Limitations
• Troubleshooting

Summary

Device Manager discovers RDMA NICs from /sys/class/net, resolves bound RDMA devices, verifies link status, and advertises three extended resource keys — shared, exclusive, and single-root I/O virtualization (SR-IOV) partitioned — so a workload can request network fabric beside its accelerators.

When at least one endpoint is active, the node carries feature.gpustack.ai/rdma.capable=true (see Network Topology and RDMA resource keys ).

The steps verify interface discovery, container injection, NUMA alignment, and deployment integration.

Environment

Step 1: Request RDMA

Inspect discovered interfaces and allocatable capacities on the target node first:

kubectl get devices <node> -o json |
  jq '[.spec.interfaces[] | (., (.virtualFunctions // [])[])]
      | map(select(.rdma or .link)) | map({name, pciBusId, rdmaDevice, link})'

kubectl get node <node> -o json |
  jq '.status.allocatable | with_entries(select(.key | contains("gpustack.ai/rdma")))'

A workload requests whole accelerators and RDMA endpoints within the same container. Pods must carry the kueue.x-k8s.io/queue-name label to participate in admission. Name the entrance LocalQueue of the pool that holds the accelerators. LocalQueue names look like gpustack-fnv64-<hash>. List them with kubectl -n team-a get localqueues:

apiVersion: v1
kind: Pod
metadata:
  name: trainer-0
  namespace: team-a
  labels:
    kueue.x-k8s.io/queue-name: gpustack-fnv64-<hash>
spec:
  restartPolicy: Never
  containers:
    - name: trainer
      image: ubuntu:24.04
      command:
        - "sleep"
        - "3600"
      resources:
        limits:
          nvidia.com/gpu: "1"
          device.gpustack.ai/rdma.shared: "1"

The allocated container receives character devices and NCCL_IB_HCA configuration (see Injected devices ).

Prefer device.gpustack.ai/rdma.shared when a container needs a single endpoint. Use the exclusive key device.gpustack.ai/rdma when requesting multiple distinct network endpoints.

Step 2: Align NUMA with kubelet

Kubernetes does not pair specific accelerators with adjacent network adapters by default. Enable NUMA alignment by setting the TopologyManager policy in /var/lib/kubelet/config.yaml (see Enabling NUMA alignment ):

 apiVersion: kubelet.config.k8s.io/v1beta1
 kind: KubeletConfiguration
+topologyManagerPolicy: restricted
+topologyManagerScope: container

Restart the kubelet after saving the configuration:

systemctl restart kubelet
kubectl get --raw /api/v1/nodes/<node>/proxy/configz |
  jq -r '.kubeletconfig | {topologyManagerPolicy, topologyManagerScope}'

restricted ensures that single-socket requests land on one NUMA node, while multi-GPU workloads spanning multiple sockets are packed into the minimum number of NUMA nodes.

Step 3: Request interfaces per role

In a ModelDeployment, roles do not declare raw resource keys. Specify the number of network interfaces under roles[].resources.interface. The operator refuses an interface request when no RDMA or EFA transfer path uses it. A prefill and decode pair with spec.kvTransfer.protocol: RDMA is such a path:

apiVersion: worker.gpustack.ai/v1
kind: ModelDeployment
metadata:
  name: qwen-pd
  namespace: team-a
spec:
  model:
    name: Qwen/Qwen2.5-7B-Instruct
  engine:
    name: vLLM
    image: gpustack/runner:cuda13.0-vllm0.29.0
  router:
    name: llm-d-router
  kvTransfer:
    protocol: RDMA
  roles:
    - name: prefill
      kind: Prefill
      replicas: 1
      instanceType: gpustack--nvidia-a10g-linux-amd64
      resources:
        accelerator: "2"
        interface: "2"
    - name: decode
      kind: Decode
      replicas: 1
      instanceType: gpustack--nvidia-a10g-linux-amd64
      resources:
        accelerator: "2"
        interface: "2"

For the RDMA protocol, the operator translates interface: 1 into device.gpustack.ai/rdma.shared: 1. When interface is greater than 1, it renders device.gpustack.ai/rdma: N. Each allocated endpoint then corresponds to an independent physical adapter (see A minimal deployment ).

The same request without router, kvTransfer or a cache backend that uses RDMA fails at admission with no effective RDMA or EFA transfer leg uses the requested interfaces. A cache backend on RDMA is the other path, and Step 4 sets it up.

Step 4: KV cache over RDMA

To route KV cache transport over an RDMA fabric instead of TCP, edit the mooncake-dram backend from Step 1 of the Shared Cache walkthrough . The transport is editable on a running backend:

 apiVersion: worker.gpustack.ai/v1
 kind: KVCacheBackend
 metadata:
   name: mooncake-dram
 spec:
   type: Mooncake
+  transport:
+    protocol: RDMA
   connection:
     managed:
       leader: {}
       members:
         - nodeSelector:
             kubernetes.io/os: linux
           medium: DRAM
           capacityPerMember: 8Gi
+          fabricInterfaceCount: 2

Member DaemonSet Pods automatically receive the network devices, host privileges, and limits required for high-performance transport (see Members ).

Step 5: Apply AWS EFA differences

On AWS Elastic Fabric Adapter (EFA) instances, RDMA devices bypass /sys/class/net and do not expose standard verbs completion queues. Declare the EFA protocol explicitly on a backend for those nodes:

apiVersion: worker.gpustack.ai/v1
kind: KVCacheBackend
metadata:
  name: mooncake-efa
spec:
  type: Mooncake
  transport:
    protocol: EFA
  connection:
    managed:
      leader: {}
      members:
        - nodeSelector:
            node.kubernetes.io/instance-type: c6in.32xlarge
          medium: DRAM
          capacityPerMember: 64Gi
          fabricInterfaceCount: 2

In ModelDeployment, set kvTransfer.protocol: EFA and specify custom runner images containing libfabric and EFA-enabled Mooncake builds.

Limitations

• Kubelet NUMA alignment policies (none, restricted, single-numa-node) and TopologyAffinityError admission behavior are covered in Enabling NUMA alignment and Reading the result .
• Kueue network-resource metering and over-admission behavior are covered in RDMA Operations and Troubleshooting .
• AWS EFA discovery and transport configuration are covered in Step 5 and Troubleshooting .

Troubleshooting