External DP Benchmark

On one eight-GPU H100 SXM5 node, wide tensor parallelism (TP) delivered 2.27×–6.99× higher output throughput than External DP across twelve matched workloads. External DP had lower median first-token latency at concurrency 64 for both prompt shapes. Both configurations used eight GPUs and completed 1,088/1,088 measured requests with zero errors.

The model was deepseek-ai/DeepSeek-V2-Lite-Chat, a mixture of experts (MoE) model. Wide TP kept all eight GPUs inside one Pod and used intra-Pod NVLink. External DP used four two-GPU Pods: local TP used NVLink, while cross-Pod expert collectives used TCP. External DP describes the serving contract.

The initial suites and conclusions describe cross-Pod TCP. The separate InfiniBand RDMA comparison records the follow-up.

Contents

• Configuration
• Hardware, model and workload
• Prompt lengths and concurrency
• Per-rank registration
• Summary
• InfiniBand RDMA comparison
• Limitations

Configuration

Each configuration used one ModelDeployment on the same GPU node, gpu-node-0.

Wide TP set --tensor-parallel-size 8 without --enable-expert-parallel. Attention and expert layers used TP across the eight GPUs within that Pod.

External DP set data parallelism (DP) to 4, local TP to 2, and enabled expert parallelism (EP). Each rank exposed its own API server and used --all2all-backend allgather_reducescatter. The four ranks shared an eight-GPU expert collective; each rank kept its own attention KV cache. There was no cross-rank KV-block transfer.

NVLink was used only inside a Pod. Each Pod exposed only its assigned GPUs, with private IPC and /dev/shm, hostIPC disabled and no remote direct memory access (RDMA) interface. Cross-Pod EP communication, including allgather_reducescatter, used TCP sockets over eth0. The host’s NVLink fabric did not provide a cross-Pod transport in this recording.

Hardware, model and workload

Port allocation

The captured deployment set VLLM_PORT=19500 on every rank, and the successful run selected port 19502 for collective initialization (an observed runtime choice rather than a fixed base-plus-two offset). See Port allocation for vLLM’s port-probing, candidate-broadcast and retry behavior, and the recorded bootstrap-port race that motivates a fixed VLLM_PORT, and the walkthrough manifest for the setting.

Workload

Prompt/output targets were 1024/256 and 4096/128, at concurrency 1, 4, 8, 16, 32 and 64. The request counts were 32, 32, 32, 64, 128 and 256 per shape, respectively. Each cell had four excluded warm-up requests. One calibration request preceded each configuration. Prompts used unique request markers; early stopping produced different output-token totals.

Method

The streaming load generator ran inside the cluster and called the cluster-internal endpoint. Each configuration ran once, sequentially on the same node with warmed model weights. The 24 result files contain 12 cells per configuration and 2,176 measured requests in total. Output throughput uses actual generated tokens divided by each cell’s wall time.

Latency columns report time to first token (TTFT), time per output token (TPOT) and inter-token latency (ITL). TPOT is a per-request average; ITL measures streamed-token gaps. Bold values mark lower latency or higher throughput within each matched workload and concurrency. They do not establish statistical significance.

Prompt lengths and concurrency

Every cell completed its requested count with zero errors.

Wide TP delivered higher output throughput in all twelve matched cells. Its peak was 2,647.004 tok/s at 1024/256, concurrency 16, against 436.861 tok/s for External DP. External DP peaked at 1,017.434 tok/s at 1024/256, concurrency 64.

At concurrency 1, External DP had 6.9×–7.3× higher TPOT p50 across the two shapes. The configurations differ in expert sharding, transport, router path and API-server count. The measurements do not isolate the contribution of each difference.

At 1024/256, concurrency 64, External DP reduced TTFT p50 from 4,798.411 ms to 1,863.100 ms, or 2.58× lower. Its TTFT p95 was also lower, 4,921.601 ms against 5,215.017 ms. Wide TP still delivered 2.27× higher output throughput, with lower TPOT and ITL tails.

At 4096/128, concurrency 64, External DP reduced median TTFT from 3,814.139 ms to 3,094.017 ms, or 1.23× lower. Its tail worsened: TTFT p95 reached 17,594.349 ms, and ITL p99 reached 1,953.899 ms. Wide TP recorded 4,057.352 ms and 55.619 ms, respectively. A lower first-token median did not imply better generation latency or throughput.

Per-rank registration

The final capture combines operator status, the Router registry, per-worker counters and engine access logs. The role reported one ready replica group with four confirmed serving endpoints:

{
  "name": "server",
  "kind": "Server",
  "desired": 1,
  "ready": 1,
  "parallelism": {
    "declared": { "dataParallel": 4, "tensorParallel": 2 },
    "loadBalance": "External",
    "modes": { "expertParallel": true },
    "source": { "complete": true, "kind": "ExtraArgs" }
  },
  "endpoints": {
    "eligible": 1,
    "serving": { "state": "Confirmed", "value": 4 }
  }
}

eligible counts the replica group. serving.value counts the HTTP endpoints confirmed by the Router. All four endpoints were registered, healthy, in selection and had closed circuits. The rank names below replace endpoint IPs with their corresponding member identities.

The counters total 1,137 requests: 1,088 measured requests, 48 warm-ups and one calibration request. The wide TP engine also logged 1,137 completion lines. Router counters matched every rank’s engine completion count, establishing traffic on all four ranks. The generic router-access-line capture reported zero; it is separate from these nonzero per-worker counters.

Summary

Benefits

• External DP registered four healthy serving endpoints and spread requests evenly: 285 + 284 × 3.
• At 1024/256, concurrency 64, External DP had 2.58× lower TTFT p50 than wide TP.
• At 4096/128, concurrency 64, its TTFT p50 was 1.23× lower, despite worse tail latency.
• Both configurations completed 1,088/1,088 measured requests, with zero errors.

Trade-offs

• Wide TP delivered 2.27×–6.99× higher output throughput across all matched cells.
• Cross-Pod EP used TCP, while the wide configuration kept its full collective inside one NVLink-connected Pod.
• At 4096/128, concurrency 64, External DP had lower median TTFT but much higher first-token and token-gap tails.

Choose the fixed four-rank group when per-rank ingress or lower median TTFT at concurrency 64 fits the workload. Choose wide TP when output throughput and generation latency dominate on this single-node NVLink setup. The crossover does not establish a general advantage for multi-node or RDMA deployments.

InfiniBand RDMA comparison

A follow-up on the same ConnectX-7 InfiniBand node setup described in Prefill and Decode Benchmark retained DP = 4, TP = 2 per Pod, one API server per rank and round_robin routing. All twelve measured cells completed 1,088/1,088 requests with zero errors.

Transport verification

Update external-dp.yaml from the TCP walkthrough : The diff uses current interface allocation; the recorded run used manual device exposure.

 spec:
   roles:
     - name: server
       kind: Server
       size: 4
       resources:
         accelerator: "2"
+        interface: "2"

NCCL logged Using network IB and channel transfers via NET/IB/.../GDRDMA. This establishes InfiniBand transport for the cross-Pod expert collective. Intra-Pod TP still used NVLink; HTTP and bootstrap traffic remained separate from collective transport.

The model, engine limits and collective backend matched the TCP run. Keep VLLM_PORT=19500 on every rank; InfiniBand does not fix the bootstrap-port race .

TCP and RDMA

Bold values mark lower latency or higher throughput within each matched External DP pair. The target shapes, concurrencies and measured request counts matched the TCP sweep.

At 1024/256, concurrency 64, median TTFT fell 6.26×, with 3.85× higher output throughput than TCP. At 4096/128, concurrency 64, median TTFT fell 8.47×, with 7.01× higher output throughput. The long-prompt TTFT p95 fell from 17,594.349 to 1,477.799 ms. Its ITL p99 fell from 1,953.899 to 182.693 ms, reducing the severe TCP generation stalls.

Wide TP comparison

At concurrency 64, RDMA External DP exceeded the recorded wide-TP output throughput for both shapes. Wide TP kept all eight GPUs inside one NVLink-connected Pod and was not rerun for this follow-up. Both configurations allocated eight GPUs; the measurements are separate runs.

RDMA External DP had 16.13× lower median TTFT and 1.69× higher output throughput at 1024/256, concurrency 64. At 4096/128, the corresponding gains were 10.44× lower median TTFT and 1.36× higher output throughput. This reverses the TCP throughput disadvantage at the highest tested concurrency.

Routing verification

The captured role status confirmed four serving endpoints for one replica group. Engine completion counts were 285 + 284 + 284 + 284 = 1,137 across the four ranks. These include 1,088 measured requests, 48 warm-ups and one calibration request. The counts establish traffic on every rank; the generic Router access-line capture is a separate counter.

Remaining trade-offs

Wide TP still had higher output throughput in ten of twelve matched cells, covering concurrencies 1–32. It retained lower TPOT p50 and ITL p99 in all twelve cells. At 4096/128, concurrency 64, RDMA External DP still had 182.693 ms ITL p99, versus 55.619 ms for wide TP.

RDMA also regressed one TCP tail metric: at 1024/256, concurrency 8, ITL p99 rose from 26.162 to 64.911 ms. The faster first-token median and higher aggregate throughput do not establish uniformly smoother token generation. No repeated trials isolate the cause of that tail difference.

Choose RDMA External DP when high-concurrency ingress and first-token latency justify a distributed expert collective. Wide TP remains the measured choice for lower-concurrency throughput and token-generation latency on this single-node setup. Cross-node behavior, other models and other collective backends remain unmeasured.

Limitations

• One run per configuration, with 32–256 requests per cell. There are no repeated-run confidence intervals.
• Both configurations used eight GPUs, but batching limits were per engine. Four rank API servers differ from one wide API server.
• Parallelism, transport and routing changed together. No isolated collective or Router-overhead measurement was captured.
• The initial cross-Pod EP capture used TCP over eth0; the follow-up verified InfiniBand collective transport. Cross-Pod NVLink was not measured, and the separate runs do not isolate transport from run-to-run variation.
• All ranks shared one GPU node. Cross-node placement, other DP widths and online resizing were not measured.
• Only one MoE model and two prompt/output targets were measured. Dense-model behavior cannot be inferred from these results.
• Prompt markers and early stopping produced small prompt-length differences and different generated-token totals.
• Direct nonstreaming probes to each rank and a final released-device snapshot were not part of the TCP capture.