Elastic EP Benchmark

On one H100 SXM5 node, an expert parallelism (EP) collective expanded from four GPUs to eight. At concurrency 64, time to first token (TTFT) p50 fell from 6,509.231 ms to 2,598.867 ms, a 2.50× reduction. Single-request token generation became slower. The live expansion load completed 439/439 requests with zero errors.

The initial suites describe cross-Pod TCP. The separate InfiniBand RDMA comparison records the follow-up.

Contents

• Summary
• Environment
• Suite 1: Width comparison
• Live resize
• Verification
• Benefits
• Trade-offs
• InfiniBand RDMA comparison
• Limitations

Summary

One ModelDeployment, b2-elastic-ep in namespace gpustack-bench, kept tensor parallelism (TP) = 2 per member. Its data parallelism (DP) width grew from 2 to 4. The EP world size grew from 4 to 8 GPUs. Existing Pods retained their UIDs and restart counts. Elastic EP defines the profile and resize contract.

The wider collective reduced first-token queueing at concurrency 64, but output throughput remained lower at every measured concurrency. Doubling the GPU allocation did not double throughput. Only one API server accepted requests in both states.

Choose online width growth when retaining serving processes and reducing high-concurrency first-token delay justify the extra GPUs. For this TCP collective, width 2 favored lower concurrency, throughput and token-generation latency. The crossover at concurrency 64 concerns median first-token latency; total request latency still increased at width 4.

Environment

The deployment used one Server role with replicas: 1 and size: 1. Only elasticEp.width changed. The CPU-only Ray head remained outside the GPU allocation. The head and all GPU members ran on the same GPU node. The walkthrough provides the manifest and verification commands.

The captured GPU Pods exposed only their two assigned GPUs and used private IPC namespaces. Each had its own /dev/shm, with hostIPC disabled and no remote direct memory access (RDMA) interface. NVLink carried TP traffic within each Pod. All cross-Pod communication, including the EP collective, used TCP sockets over eth0.

Suite 1: Width comparison

Each state ran one sweep of target 1,024-input / 256-output-token requests. Concurrency 1, 4 and 8 used 32 measured requests; concurrency 16 used 64; concurrency 32 and 64 used 128. Each cell excluded four warm-up requests. All 416 measured requests per state completed with zero errors.

Observed prompt lengths ranged from 1,021 to 1,025 tokens. Output lengths were caps; some requests ended earlier. Throughput uses actual generated tokens divided by cell wall time. The same deployment ran State 1, a live expansion, then State 3.

Time per output token (TPOT) measures per-request generation time per token. Inter-token latency (ITL) measures gaps between streamed tokens. Values retain the JSON precision. Bold values mark the lower latency or higher throughput within each matched pair.

Live resize

A concurrency-8 stream ran during the width 2 → 4 change, using the same 1,024/256 target shape. It completed 439/439 requests in 264.227 seconds (264.23 s), with zero client errors. The scaling-refusals-503.txt capture recorded zero client-facing HTTP 503 responses. The serving master recorded 10 /health responses with HTTP 503 during collective re-rendezvous.

The 12-second timer measured a phase-only check, not effective-width convergence. The harness’s wait_ready returned before collective re-rendezvous, while status.phase remained Ready and serving continued. It did not require the current generation’s effective width.

The patch was submitted at 08:16:11Z. In scaled-cr.json, RoleKindsReady and EndpointEligibility transitioned to True at 08:19:20Z, about 189 seconds later. This interval covers initialization and Ray collective joining for the two added GPU Pods. The final observation.json separately confirmed effective width 4; it contains no timestamp for that convergence.

The elastic-watch-timeline.txt capture sampled approximately every 16–21 seconds, with occasional intervals up to 28 seconds.

The Starting sample occurred during collective re-rendezvous, between Ready samples 40 seconds apart. The exact readiness withdrawal duration is unknown. The observed client requests completed despite this transition.

The walkthrough checks effective width before accepting expansion.

Verification

The final observation reported generation 2 and stable width 4, with reason native ranks and forwards agree. The observation is diagnostic ConfigMap data; ordinary role status still reports one replica group. ReplicasUpToDate remained Unknown/RolloutNotObserved before and after expansion.

The serving master’s UID and zero restart count matched across both Pod snapshots. The CPU Ray head and original GPU worker also retained their identities. These observations establish retention of surviving processes; they do not count internal engine weight-loading operations.

The master’s completion access-log count grew from 880 in the pre-State-3 capture to 1,320 in the final capture. Counts include measured requests, warm-up and calibration. No Router was configured; the collection script’s missing-Router diagnostic is not a serving failure. Client JSON establishes request completion and errors.

After deletion, the Devices snapshot recorded all eight H100 accelerators with remaining = 1600000. This records released accelerator claims; it does not prove removal of model-cache or cluster infrastructure.

A later runtime-acceptance run measured the same expansion on an eight-GPU H100 node:

The operator deletes the operation record when the operation ends, so the terminal state reads from the condition and the observation. An earlier attempt on the same image did not converge and remains an open observation; this run does not explain it.

Benefits

• The deployment expanded from 4 to 8 GPUs while retaining its serving master and existing GPU worker. The concurrent load completed 439/439 requests with zero errors.
• At concurrency 64, width 4 had 2.50× lower TTFT p50 than width 2: 2,598.867 ms against 6,509.231 ms. TTFT p95 fell from 8,708.596 ms to 4,789.148 ms.

The first-token improvement appears at the highest tested concurrency. It does not establish lower total request latency: end-to-end p50 rose from 11,487.169 ms to 12,154.350 ms there.

Trade-offs

Single-request ITL p50 rose from 8.566 ms to 15.675 ms, an 83.0% increase. With one request, queueing relief is limited. The eight-GPU allgather_reducescatter collective spans four TP = 2 Pods. Communication between those Pods used TCP over eth0. The higher token cost is consistent with collective and TCP overhead; their individual contributions were not isolated.

Output throughput fell at all six concurrencies. At concurrency 1, it fell from 109.586 to 61.020 tok/s. At concurrency 64, it was 1,145.131 versus 1,121.379 tok/s despite the doubled GPU allocation. The single API server and selected collective remain part of both configurations. The measurement does not isolate the API server as a bottleneck.

At concurrency 64, ITL p50 rose from 16.522 ms to 30.270 ms. The lower first-token queueing delay came with slower token generation. Choose width from the workload’s latency targets and GPU cost, rather than assuming wider always means faster.

InfiniBand RDMA comparison

A follow-up on the same ConnectX-7 InfiniBand node setup described in Prefill and Decode Benchmark expanded the TP = 2 member layout from width 2 to width 4. The final selected cells completed 416/416 requests per width, with zero errors.

Transport verification

Update elastic-ep.yaml from the TCP walkthrough : The diff uses current interface allocation; the recorded run used manual device exposure.

 spec:
   roles:
     - name: server
       kind: Server
       size: 1
       resources:
         accelerator: "2"
+        interface: "2"
       elasticEp:
         width: 2
       extraArgs:
         - --gpu-memory-utilization
-        - "0.90"
+        - "0.85"

NCCL logs reported Using network IB and GPU Direct RDMA enabled. The cross-Pod allgather_reducescatter collective used InfiniBand instead of TCP over eth0. Intra-Pod TP still used NVLink. Ray, HTTP and bootstrap control traffic are separate from collective transport.

GPU memory utilization also changed, so the runs do not isolate transport alone.

Width comparison

Bold values mark lower latency or higher throughput within each TCP/RDMA pair at the same width. Each cell used the same target shape, concurrency and measured request count as its TCP baseline. The width-4 sweep ran twice; the selected final JSON files contain the second pass.

At width 4, concurrency 64, RDMA reduced median TTFT from 2,598.867 to 373.486 ms, a 6.96× reduction. Output throughput rose from 1,121.379 to 3,755.290 tok/s, a 3.35× increase. Median ITL fell from 30.270 to 10.838 ms.

Within the RDMA follow-up, width 4 had 8.23× lower median TTFT than width 2 at concurrency 64. The values were 373.486 versus 3,073.825 ms; output throughput was 1.64× higher at width 4. This reverses the TCP result in which width 4 had lower output throughput at every tested concurrency.

Resize verification

The expansion load completed 813/813 requests in 261.794 seconds, with zero errors. The serving master, original GPU member and CPU Ray head retained their UIDs. Two GPU Pods joined; all five final Pods had zero container restarts. The final generation-2 observation confirmed known effective width and stable width 4.

Ray registration does not establish that the expanded collective is ready. The retained harness timer polled a missing current_width field, so its 748-second value is not a convergence measurement. The observation records width under effective.Value and stableWidth. The final 2,134 completion lines span width 2, expansion load and both width-4 sweeps, including warm-up and calibration.

Remaining trade-offs

RDMA did not make width 4 faster for every workload. Single-request ITL p50 remained higher at width 4: 6.979 versus 6.111 ms at width 2. Width 2 retained higher output throughput at concurrencies 1–32; width 4 led at concurrency 64. At width 2, concurrency 4, ITL p99 worsened from 12.290 to 68.396 ms despite better median latency and throughput.

Choose width from observed demand and token-latency targets. The follow-up supports high-concurrency width growth over InfiniBand; it does not establish linear scaling or transport-only causality.

Limitations

• The TCP baseline measured one model, one H100 SXM5 node, one runner configuration and one sweep per width. No confidence intervals or repeated trials establish the size of small differences.
• State 1 used four GPUs; State 3 used eight. This is an in-place capacity comparison, not an equal-GPU efficiency comparison.
• Only width 2 → 4 at TP = 2 was captured. Cross-node expansion, other collectives and scale-down were not tested.
• The initial recording used intra-Pod NVLink and cross-Pod TCP; the follow-up verified InfiniBand collective transport. Neither measures cross-Pod NVLink or isolates transport cost from collective size and configuration changes.
• Single-API-server internal load balancing (Internal LB) was used. Per-rank HTTP serving was not measured here.
• The 12-second phase check is not an effective-width convergence timer. Zero client errors do not establish uninterrupted readiness or unchanged token gaps during every resize.
• Retained UIDs and restart counts prove Pod continuity, not unchanged internal engine state. Final Ready does not mean every status condition was true.