Model Deployment Scaling Benchmark

On one node with eight H100 GPUs, ModelDeploymentScaler increased serving replica counts under live traffic. Four vLLM configurations completed every measured request. SGLang failed under sustained load, and a separate scale-out check recorded request failures at the Router.

The measurements demonstrate that ModelDeploymentScaler observes load changes and drives ModelDeployment replicas updates promptly. End-to-end scale-out landing time in production varies with deployment-environment factors outside the scaler’s control: model weight loading, engine startup and compute-graph compilation, and accelerator resource availability or queueing.

Contents

• Summary
• Environment
• Configuration
• Suite 1: Scaling comparison
• Verification
• Benefits
• Trade-offs
• Limitations

Summary

A prefill/decode (PD) pair separates prompt processing from token generation. An encode/prefill/decode (EPD) deployment adds a separate image encoder. The text configurations compared Proportional, TokenThroughput and Predictive scaling. The multimodal configuration used Predictive. Model Deployment Scaling defines the algorithms and replica guards.

Four vLLM configurations completed every measured request. The separate lifecycle check lost requests during scale-out, so these results do not establish uninterrupted serving. Its scale-down and post-retirement phases completed without request failures.

Environment

Requests went directly from the driver to the Router’s cluster-internal endpoint. No kubectl port-forward carried inference traffic. vLLM roles set VLLM_HTTP_TIMEOUT_KEEP_ALIVE=3600; the separate lifecycle check still observed scale-out failures.

Configuration

Each role started with one replica and had bounds of one to two. The scaler used Active mode and a 10 s sampling interval.

Both predictive configurations used a 30 s horizon, two minimum observations, qLevel: "0.5" and qTrend: "0.3". Token-based configurations retained the default 180 s stable and 60 s panic windows, panic threshold 2, and latency targets of 500 ms and 50 ms.

Each configuration preferred a prefill/decode ratio of 1. The EPD configuration also preferred an encode/prefill ratio of 1. Only target was set, so these were preferences rather than fixed ratio bands. The observed EPD maxima were one encode, two prefill and two decode replicas.

The vllm-pd-proportional scaler took this form.

apiVersion: worker.gpustack.ai/v1
kind: ModelDeploymentScaler
metadata:
  name: autoscale-vllm-pd
spec:
  targetRef:
    name: autoscaling-vllm-pd
  mode: Active  # commits the recommendation to the target
  sampling:
    intervalSeconds: 10  # sampling cadence
  roles:
    - name: prefill
      minReplicas: 1
      maxReplicas: 2
    - name: decode
      minReplicas: 1
      maxReplicas: 2
  ratios:
    - numeratorRole: prefill
      denominatorRole: decode
      target: "1"  # preferred ratio, not a fixed band
  algorithm:
    type: Proportional
    proportional:
      metric: RunningRequests  # requests per ready replica
      tolerance: "0.1"  # deadband around the target
      roleTargets:
        - name: prefill
          target: "3"  # requests per ready replica
        - name: decode
          target: "3"  # requests per ready replica
    # The token throughput configurations replaced the proportional block.
    # type: TokenThroughput
    # tokenThroughput:
    #   prefillTargetTokensPerSecond: "1200"
    #   decodeTargetTokensPerSecond: "300"
    # The predictive configurations kept those token targets and added a forecast.
    # type: Predictive
    # predictive:
    #   tokenThroughput:
    #     prefillTargetTokensPerSecond: "1200"
    #     decodeTargetTokensPerSecond: "300"
    #   horizonSeconds: 30
    #   minPoints: 2
    #   qLevel: "0.5"
    #   qTrend: "0.3"
    #   prefillTokenCapacityPerReplica: "2000"
    #   decodeTokenCapacityPerReplica: "2000"

The target ModelDeployment manifest is in Scaling Walkthrough .

Parallel schedule

The two waves shared an eight-GPU ceiling. Allocations describe maximum capacity reserved for each concurrent sequence; they do not assert that eight GPUs served throughout every phase.

The admission check ran once before Wave 1. It created its own targets and did not overlap either wave. Between configurations that shared a deployment, the runner deleted the scaler, waited for binding release, then restored one replica per role.

Workload

Text configurations used a continuous streaming client with a target input of 1024 tokens and up to 256 output tokens. Phase durations are requested load durations; recorded wall time includes completion of outstanding requests.

The multimodal client used those fixed request counts, a 560-pixel generated image, shared_image: false, a target text length of 1024 tokens and up to 256 output tokens. Its phases were request-count bounded, so their wall times differ from the text load durations.

Time to first token (TTFT) measures delay before the first streamed token. Inter-token latency (ITL) measures gaps between streamed tokens. Latency percentiles use successful requests and the client’s percentile calculation. Output throughput divides successful output tokens by the phase’s recorded wall time. Failed requests remain in the request and error columns.

Suite 1: Scaling comparison

The charts show TTFT p50 and output throughput per phase for the three vLLM PD configurations. The lines are, in order, vllm-pd-proportional, vllm-pd-token-throughput and vllm-pd-predictive.

xychart-beta
    title "TTFT p50 by phase"
    x-axis ["baseline", "ramp", "sustain", "cooldown"]
    y-axis "TTFT p50 (ms)" 0 --> 3000
    line [298.564, 2540.853, 506.734, 188.231]
    line [284.481, 2325.222, 480.418, 182.044]
    line [164.044, 467.906, 447.864, 60.883]

xychart-beta
    title "Output throughput by phase"
    x-axis ["baseline", "ramp", "sustain", "cooldown"]
    y-axis "Output (tok/s)" 0 --> 2500
    line [258.325, 1117.481, 2044.782, 274.402]
    line [256.117, 1230.879, 2054.687, 276.51]
    line [282.371, 2109.251, 2128.27, 288.484]

Both ramp and sustain run at concurrency 24. In ramp, vllm-pd-proportional and vllm-pd-token-throughput exceed 2300 ms TTFT p50 and cap near single-replica throughput (1117–1231 tok/s) while the second replica starts. Once that replica is Ready in sustain, TTFT p50 drops below 510 ms and output throughput nearly doubles above 2040 tok/s at the same concurrency, while vllm-pd-predictive enters ramp already scaled and holds below 470 ms.

vLLM proportional PD

Configuration: vllm-pd-proportional.

vLLM token throughput PD

Configuration: vllm-pd-token-throughput.

vLLM predictive PD

Configuration: vllm-pd-predictive.

SGLang token throughput PD

Configuration: sglang-pd-token-throughput.

vLLM predictive EPD

Configuration: vllm-epd-predictive.

The SGLang failure signature was Failed to get kvcache from prefill instance, it might be dead. Its error-heavy phases do not establish healthy serving capacity. The recorded investigation associated the failures with the documented TCP port exhaustion limitation. The TIME-WAIT reuse setting needs kubelet configuration before use; a successful rerun with that setting was not captured.

Verification

Scale-out timing

Scale-out lead measures the interval from ramp start to the first sampled desired count above one. It uses deployment status, so it does not measure when a newly added replica first became Ready or served a request. A count already above one at ramp start is reported as zero with an explicit flag.

Both token throughput and predictive configurations scaled during baseline. The zero lead therefore does not isolate a forecasting benefit. The recorded maxima summarize each role across samples; they are not a synchronized GPU utilization trace.

Retirement

The separate single-role lifecycle check confirmed a new Ready replica served requests, then reduced the desired count from two to one. It observed Disqualified and Draining, followed by one Ready survivor. Its scale-down and post-retirement phases completed with zero failed requests.

These values come from the standalone lifecycle rerun. The earlier check embedded in Wave 2 also failed during scale-out, with 3 failures among 10579 requests. Admission passed all 45 checks; the standalone lifecycle check passed 10 checks and failed its all-phases completion check.

The benchmark configuration timelines sampled Disqualified for the proportional and token-based PD configurations. Predictive PD also sampled Admitted and Aborted. The EPD timeline captured no retirement state. These sets do not establish complete retirement sequences.

The retirement protocol uses Draining followed by Deleting before release completes. Kubernetes can display a deleting Pod as Terminating. Terminating is not a retirement state. The captures do not include every transition, and zero client errors in the two scale-down phases do not prove every in-flight request survives every retirement.

A later runtime-acceptance run scaled a vLLM multi-Server deployment by hand through one to two to one replicas. The Router’s worker set followed every edit, the pinned round_robin policy split sixteen requests eight and eight across the two workers at two replicas, both workers stayed healthy, and the deployment still served after the scale-back. The run completed 21 checks with zero failures.

A second run retired one of two replica groups of a vLLM External DP deployment at data parallelism 2. The Router’s worker set held every member throughout — two workers for one group of two, four for two groups of two, and two again after the scale-down.

The endpoint withdrawal reached the Router in 21 seconds, and the retired members were deleted 70 seconds after the patch, past the 60-second disaggregated handoff grace window, which the retirement honored. Eight checks passed; the one failing row had timed the endpoint withdrawal against that window, a measurement defect in the case rather than a product finding.

A third run drove a SGLang PD deployment of one prefill and one decode replica with the scaler active: 29 checks passed with zero failures, warnings or blocked rows. The deployment published each role’s group independently, the four content classes served through the sglang-gateway, the thinking switch held on the arithmetic, Chinese and JSON requests, and the Mooncake transfer over the direct leg moved 54 MB across six transfers with zero failures.

The scaler read the engine’s waiting-request counter during a transfer and recorded 24 waiting requests on the decode role; under load the decode role grew to two replicas, and both roles returned to their floor of one with no retirement outstanding.

The three runs cover one node, one shape each and one recorded cycle. They do not establish multi-node behavior or a general scaling claim.

Benefits

Predictive PD had lower ramp median TTFT than proportional PD: 467.906 versus 2540.853 ms. Its ramp throughput was 2109.251 versus 1117.481 tok/s. It entered the ramp with extra replicas already requested, so this comparison includes different capacity at ramp start.

Trade-offs

Cooldown lasted 90 s for text configurations, while the stored downward stabilization window was 300 s. Several final snapshots still requested two replicas per role. Cooldown completion therefore does not establish a full return to the initial allocation.

Limitations

The measurements cover one node, two models and one recorded comparison per configuration, with a separate SGLang retry. They do not establish multi-node performance or general algorithm rankings. The SGLang table uses the retry’s complete phase artifacts; its earlier interrupted run is excluded.

The text and multimodal clients perform different work and use different phase stopping rules. Their throughput values cannot establish a serving-mode speedup. Image tags identify the configured builds; the capture does not establish identical installed engine package versions across configurations.

Zero errors in the four vLLM benchmark configurations describe their measured requests. The independent scale-out failures and incomplete retirement traces prevent an availability guarantee. The Scaling Walkthrough distinguishes configuration examples from captured lifecycle evidence.