Model Deployment Scaling Walkthrough
Configure a ModelDeploymentScaler in Shadow mode and inspect its recommendations. Then permit
replica changes with Active mode. The examples use a vLLM prefill/decode (PD) pair. They also
explain how to adapt the scaler for encode/prefill/decode (EPD) serving. Captured lifecycle
observations are labeled separately from reusable configuration.
Contents
- Summary
- Environment
- Step 1: Prepare the target
- Step 2: Observe in Shadow
- Step 3: Choose an algorithm
- Step 4: Activate scaling
- Step 5: Observe retirement
- Verification
- Cleanup
- Troubleshooting
Summary
Observe a ModelDeploymentScaler in Shadow, choose an algorithm, switch to Active mode, and
watch a replica retire.
| Object | Scope | Creator | Purpose |
|---|---|---|---|
ModelArtifact |
namespace | user | the qwen3-8b weights the target serves |
AcceleratorType |
cluster | operator | the whole-GPU type each role replica requests |
ModelDeployment |
namespace | user | the PD target whose replica counts the scaler drives |
ModelDeploymentScaler |
namespace | user | the policy observed in Shadow, then allowed to act in Active |
The Scaling Benchmark used eight H100 GPUs across concurrent configurations. This procedure starts one deployment. Its examples are configuration instructions; they are not a claim that this exact public-API sequence was executed in the recorded benchmark.
Environment
| Item | Value |
|---|---|
| GPUs | two free initially, up to four after scaling; one GPU per replica |
AcceleratorType |
the whole-GPU type of the intended cluster; the example uses the discovered gpustack--nvidia-h100-80gb-hbm3-linux-amd64 |
| Namespace | gpustack-bench |
| Model | Qwen/Qwen3-8B from ModelArtifact qwen3-8b, revision b968826d9c46dd6066d109eabc6255188de91218 |
| Tools | kubectl, jq, Python 3 with aiohttp; a source checkout provides the load client |
Prerequisites
Follow cluster preparation
and
model prefetch
with namespace gpustack-bench and artifact
qwen3-8b.
Use a build that includes ModelDeploymentScaler, its controller and admission webhook.
Confirm that no other scaler owns the target. See Model Deployment Scaling for algorithm contracts and GitOps exclusions .
Step 1: Prepare the target
Save this example as scaling-target.yaml. Replace both acceleratorType values with the whole-GPU
type for the intended cluster. The benchmark used bench-gpu; the reusable example uses the
discovered H100 type.
apiVersion: worker.gpustack.ai/v1
kind: ModelDeployment
metadata:
name: autoscaling-vllm-pd
namespace: gpustack-bench
spec:
model:
name: Qwen/Qwen3-8B
artifactRef:
name: qwen3-8b
engine:
name: vLLM
image: gpustack/runner:cuda13.0-vllm0.30.0
router:
name: llm-d-router
roles:
- name: prefill
kind: Prefill
replicas: 1
size: 1
acceleratorType: gpustack--nvidia-h100-80gb-hbm3-linux-amd64
resources:
accelerator: "1"
extraArgs:
- --tensor-parallel-size
- "1"
- --max-model-len
- "16384"
- --max-num-seqs
- "64"
- --gpu-memory-utilization
- "0.90"
- --enable-scale-out
env:
- name: VLLM_HTTP_TIMEOUT_KEEP_ALIVE
value: "3600"
- name: decode
kind: Decode
replicas: 1
size: 1
acceleratorType: gpustack--nvidia-h100-80gb-hbm3-linux-amd64
resources:
accelerator: "1"
extraArgs:
- --tensor-parallel-size
- "1"
- --max-model-len
- "16384"
- --max-num-seqs
- "64"
- --gpu-memory-utilization
- "0.90"
- --enable-scale-out
env:
- name: VLLM_HTTP_TIMEOUT_KEEP_ALIVE
value: "3600"kubectl apply -f scaling-target.yaml
kubectl -n gpustack-bench get modeldeployment autoscaling-vllm-pd -wWait for status.phase: Ready. Verify one Ready replica in each role and a working Router
before creating load. The vLLM flags enable the Router’s
token-counting path
.
The longer HTTP keep-alive matches the recorded configuration; it did not eliminate every scale-out failure.
Step 2: Observe in Shadow
Save this scaler as scaling-policy.yaml. Its role names match the target exactly.
The ratio target prefers equal counts; add bounds only when a fixed ratio is required.
apiVersion: worker.gpustack.ai/v1
kind: ModelDeploymentScaler
metadata:
name: autoscale-vllm-pd
namespace: gpustack-bench
spec:
targetRef:
name: autoscaling-vllm-pd
mode: Shadow
sampling:
intervalSeconds: 10
roles:
- name: prefill
minReplicas: 1
maxReplicas: 2
- name: decode
minReplicas: 1
maxReplicas: 2
ratios:
- numeratorRole: prefill
denominatorRole: decode
target: "1"
algorithm:
type: Proportional
proportional:
metric: RunningRequests
tolerance: "0.1"
roleTargets:
- name: prefill
target: "3"
- name: decode
target: "3"kubectl apply -f scaling-policy.yaml
kubectl -n gpustack-bench get mds autoscale-vllm-pd -o yaml
kubectl -n gpustack-bench get modeldeployment autoscaling-vllm-pd -o json \
| jq '{replicas: [.spec.roles[] | {name, replicas}],
binding: .metadata.annotations["modeldeployment.gpustack.ai/scaler-binding"]}'In Shadow, expect Active=False with reason ShadowMode, unchanged target counts and no binding
owned by this scaler. Bound=True means the target is available; it does not mean a Shadow scaler
owns its replica counts. Inspect status.roles[].rawRecommendedReplicas,
constrainedReplicas, signalState and reason together.
Prepare default/bench-driver before measuring. If it is absent, create it:
kubectl -n default run bench-driver --image=python:3.11-slim --restart=Never \
--command -- sleep infinityWait for readiness and install the client dependency if the driver does not already carry it:
kubectl -n default wait pod/bench-driver --for=condition=Ready --timeout=300s
kubectl -n default exec bench-driver -- python3 -m pip install aiohttp
kubectl -n default exec bench-driver -- python3 -c 'import aiohttp'Run a small load from the source checkout. The client executes inside that driver against the
Router Service. Stop if it reports local fallback; local execution cannot measure this endpoint.
BENCH_INCLUSTER=1 python3 .agents/skills/gpustack-operator-benchmark/scripts/bench_client.py load \
--endpoint http://autoscaling-vllm-pd-router.gpustack-bench.svc:8000 \
--model Qwen/Qwen3-8B --arm shadow --suite autoscaling --label baseline \
--input-tokens 1024 --output-tokens 256 --concurrency 2 --duration 45 \
--out /tmp/scaler-shadow.json --requests-out /tmp/scaler-shadow-requests.jsonlCheck the driver’s readiness and error output if the load cannot start. Preserve both the phase JSON and request records before changing modes. Status is a snapshot of control-plane changes, so unchanged raw metrics do not establish that sampling stopped. Read engine metrics when a current signal value is needed.
Step 3: Choose an algorithm
Replace the complete spec.algorithm mapping in scaling-policy.yaml when changing algorithms.
Only the block matching type is accepted. Apply the edited file while it remains in Shadow.
Capacity values below reproduce the benchmark’s settings; tune them from measured per-replica capacity.
| Choice | Target scenario | Configuration guidance |
|---|---|---|
Predictive + capacityPrior |
Latency-sensitive online serving and traffic ramps where engine startup (~76 s on H200) must begin before sustained peak load arrives. | Set horizonSeconds above replica startup time (180–300 s), keep headroom: "0.75", and bound slope spikes with behavior.scaleUp.maxStepReplicas. |
TokenThroughput + capacityPrior |
General serving with variable prompt lengths and prefix caching when GPU replica-seconds must track sustained token demand. | Use capacityPrior ("0.75" headroom) or set manual targets at 70–75% of single-replica saturation; unmargined saturation targets fail to trigger before TTFT climbs. |
Proportional |
Uniform request lengths or simple concurrency control (RunningRequests, QueueDepth) without offline token-capacity sweeps. |
Set roleTargets[].target to the per-replica concurrency where TTFT remains within target; Proportional does not accept capacityPrior. |
Token throughput
apiVersion: worker.gpustack.ai/v1
kind: ModelDeploymentScaler
metadata:
name: autoscale-vllm-pd
namespace: gpustack-bench
spec:
algorithm:
- type: Proportional
- proportional:
- metric: RunningRequests
- tolerance: "0.1"
- roleTargets:
- - name: prefill
- target: "3"
- - name: decode
- target: "3"
+ type: TokenThroughput
+ tokenThroughput:
+ prefillTargetTokensPerSecond: "1200"
+ decodeTargetTokensPerSecond: "300"
This replacement works for both vLLM and SGLang targets; the scaler reads the target engine. Establish valid counter windows before judging recommendations. Read the SGLang TCP limitation before running sustained SGLang traffic.
Predictive
apiVersion: worker.gpustack.ai/v1
kind: ModelDeploymentScaler
metadata:
name: autoscale-vllm-pd
namespace: gpustack-bench
spec:
algorithm:
- type: Proportional
- proportional:
- metric: RunningRequests
- tolerance: "0.1"
- roleTargets:
- - name: prefill
- target: "3"
- - name: decode
- target: "3"
+ type: Predictive
+ predictive:
+ horizonSeconds: 30
+ minPoints: 2
+ qLevel: "0.5"
+ qTrend: "0.3"
+ prefillTokenCapacityPerReplica: "2000"
+ decodeTokenCapacityPerReplica: "2000"
+ tokenThroughput:
+ prefillTargetTokensPerSecond: "1200"
+ decodeTargetTokensPerSecond: "300"
The nested token targets supply the fallback while the predictor warms up.
Inspect status.predictors and the role reasons to distinguish fallback from predictive planning.
Earlier growth alone does not prove a forecast caused it; token throughput also grew during baseline
in the recorded comparison
.
Catalog priors
spec.algorithm.capacityPrior resolves omitted TokenThroughput or Predictive capacities from
a worker.gpustack.ai/v1alpha1 ModelDeploymentScalerCapacityPriorCatalog. The embedded catalog carries the verified public sweeps imported from
InferenceX; the gpustack-operator-priors skill’s provenance records list every dataset.
For other evidence, supply a reviewed same-namespace ConfigMap through catalogRef (key
priors.yaml by default; the parser accepts YAML and JSON) with a qualified profile that
matches the deployment.
Single-role Server, PD and EPD targets can use catalog priors when they satisfy
Target eligibility
. Omitting
catalogRef, as in capacityPrior: {headroom: "0.75"}, selects the embedded catalog.
A ModelArtifact, AcceleratorType and engine flags supply the identity. No ModelDeployment
annotations are required when --quantization fp8 or a single catalog quantization identifies the
profile uniquely.
If multiple quantizations match, set --quantization or annotate the deployment with
worker.gpustack.ai/precision. Use the annotation also to pin precision without an engine flag.
An empty catalogRef: {} is invalid: admission requires a name for a present reference.
When the embedded catalog covers the target model, a fixed 1P:3D scaler can omit catalogRef.
Because the committed catalog omits Qwen/Qwen3-8B, create autoscaling-priors first (as shown
below, with a matching mode: disaggregated or mode: aggregated profile) before applying this
example; align replicas to the ratio and provide enough GPU capacity.
apiVersion: worker.gpustack.ai/v1
kind: ModelDeploymentScaler
metadata:
name: autoscale-vllm-pd-prior
namespace: gpustack-bench
spec:
targetRef:
name: autoscaling-vllm-pd
mode: Shadow
roles:
- name: prefill
minReplicas: 1
maxReplicas: 2
- name: decode
minReplicas: 3
maxReplicas: 6
ratios:
- numeratorRole: decode
denominatorRole: prefill
min: "3"
max: "3"
algorithm:
type: TokenThroughput
tokenThroughput:
ttftTargetMilliseconds: "2000"
itlTargetMilliseconds: "50"
capacityPrior:
catalogRef:
name: autoscaling-priors
headroom: "0.75"The resolver first matches a mode: disaggregated profile for the coupled unit. Only when no
such profile matches can mode: aggregated profiles estimate whole-unit capacity. See
Capacity priors
for the conditions.
The engine image matches by family, so no hand-computed digests or @sha256 pinning are required.
The following separate server target and scaler changes are examples, not captured output from a
live cluster run. Replace acceleratorType and check the target against the catalog before use:
apiVersion: worker.gpustack.ai/v1
kind: ModelDeployment
metadata:
name: catalog-server
namespace: gpustack-bench
spec:
model:
name: Qwen/Qwen3-8B
artifactRef:
name: qwen3-8b
engine:
name: vLLM
image: gpustack/runner:cuda13.0-vllm0.30.0
roles:
- name: server
kind: Server
replicas: 1
size: 1
acceleratorType: gpustack--nvidia-h100-80gb-hbm3-linux-amd64
resources:
accelerator: "1"Create a reviewed ConfigMap named autoscaling-priors in gpustack-bench (or reuse and extend
the one created for autoscale-vllm-pd-prior above). Its priors.yaml key must contain a
profile matching each target
—for
example, a mode: disaggregated profile for autoscaling-vllm-pd and a mode: aggregated profile
for catalog-server. Then create a separate scaler targeting this server:
apiVersion: worker.gpustack.ai/v1
kind: ModelDeploymentScaler
metadata:
name: autoscale-server
namespace: gpustack-bench
spec:
targetRef:
name: catalog-server
roles:
- name: server
minReplicas: 1
maxReplicas: 2
algorithm:
type: Predictive
predictive:
horizonSeconds: 30
minPoints: 2
tokenThroughput:
serverTargetTokensPerSecond: "1200" # explicit fallback override
ttftTargetMilliseconds: "500" # required when Server uses Catalog
itlTargetMilliseconds: "50" # required when Server uses Catalog
capacityPrior:
catalogRef:
name: autoscaling-priors
key: priors.yaml
headroom: "0.75"In this Server partial override, serverTargetTokensPerSecond stays Explicit at "1200"
while the omitted serverTokenCapacityPerReplica forecast capacity resolves from Catalog by
the role’s identity and multiplies its profile seed capacity by headroom. Both "1200" and
"0.75" are illustrative configuration values rather than measured hardware capacities; removing
capacityPrior restores manual capacity validation.
Inspect the scaler’s resolved priors and conditions:
kubectl -n gpustack-bench get mds autoscale-server -o json \
| jq '{priors: .status.resolvedCapacityPriors,
conditions: [.status.conditions[] | select(.type == "CapacityPriorsReady" or .type == "Blocked")]}'When the target matches an embedded profile, drop the explicit fallback and the ConfigMap: the
embedded catalog supplies the capacity and the identity resolves from the target.
apiVersion: worker.gpustack.ai/v1
kind: ModelDeploymentScaler
metadata:
name: autoscale-server
namespace: gpustack-bench
spec:
algorithm:
type: Predictive
predictive:
tokenThroughput:
- serverTargetTokensPerSecond: "1200"
ttftTargetMilliseconds: "500"
itlTargetMilliseconds: "50"
capacityPrior:
- catalogRef:
- name: autoscaling-priors
- key: priors.yaml
headroom: "0.75"
To return to manual mode, remove capacityPrior and supply the required manual capacities:
apiVersion: worker.gpustack.ai/v1
kind: ModelDeploymentScaler
metadata:
name: autoscale-server
namespace: gpustack-bench
spec:
algorithm:
type: Predictive
predictive:
+ serverTokenCapacityPerReplica: "2000"
tokenThroughput:
+ serverTargetTokensPerSecond: "1200"
- capacityPrior:
- headroom: "0.75"
When the target exists, CREATE rejects an unresolved catalog prior. UPDATE admission returns a
warning without writing status so you can suspend the scaler, switch to Shadow, remove
capacityPrior or supply explicit capacities; the controller then sets
CapacityPriorsReady=False, Active=False and Blocked=True on reconciliation (eventual
consistency) and stops new replica commits.
Encode ratios
For a separate multimodal target, follow the
EPD Walkthrough
with Qwen/Qwen2.5-VL-7B-Instruct and artifact
qwen2-5-vl-7b-instruct. Create a separate scaler that names that target.
Provide bounds for every role and couple encode through a direct ratio.
apiVersion: worker.gpustack.ai/v1
kind: ModelDeploymentScaler
metadata:
name: qwen-scaler
namespace: gpustack-bench
spec:
targetRef:
name: epd-arm3-epd # the EPD deployment the walkthrough above creates
roles:
- name: encode
minReplicas: 1
maxReplicas: 2
- name: prefill
minReplicas: 1
maxReplicas: 2
- name: decode
minReplicas: 1
maxReplicas: 2
algorithm:
type: TokenThroughput # the ratios couple the encode count; no token-capacity target for encode
tokenThroughput:
prefillTargetTokensPerSecond: "2000" # prompt capacity per Prefill replica
decodeTargetTokensPerSecond: "800" # generation capacity per Decode replica
ratios:
- numeratorRole: prefill
denominatorRole: decode
target: "1"
- numeratorRole: encode
denominatorRole: prefill
target: "1"Combine this fragment with TokenThroughput or Predictive; the encode count follows ratio
coupling rather than an encode token-capacity target. With one GPU per role replica, reserve up to
six GPUs. A target without min and max is a preference.
To require equal counts, set all three to "1" and ensure the current vector satisfies that policy.
Step 4: Activate scaling
Set spec.mode: Active in scaling-policy.yaml, then apply it. Keep the local manifest aligned
with the mode so later applies do not restore Shadow.
kubectl apply -f scaling-policy.yaml
kubectl -n gpustack-bench get mds autoscale-vllm-pd -o json \
| jq '{conditions: .status.conditions, binding: .status.binding, roles: .status.roles}'Expect Bound=True and Active=True. The target’s binding identifies this scaler and its term.
Let the scaler write the target’s replica counts; the ownership contract reserves those writes
while the binding is active.
In one terminal, run higher concurrency. In another, watch the target and scaler.
BENCH_INCLUSTER=1 python3 .agents/skills/gpustack-operator-benchmark/scripts/bench_client.py load \
--endpoint http://autoscaling-vllm-pd-router.gpustack-bench.svc:8000 \
--model Qwen/Qwen3-8B --arm active --suite autoscaling --label ramp \
--input-tokens 1024 --output-tokens 256 --concurrency 24 --duration 75 \
--out /tmp/scaler-ramp.json --requests-out /tmp/scaler-ramp-requests.jsonlkubectl -n gpustack-bench get modeldeployment autoscaling-vllm-pd -w
kubectl -n gpustack-bench get mds autoscale-vllm-pd -o json \
| jq '{roles: .status.roles, commit: .status.lastCommit,
pending: .status.pendingCohort, conditions: .status.conditions}'
kubectl -n gpustack-bench get pods,workloadsCompare the committed vector with desired and Ready counts. Confirm the new replica’s own request
counter increases; a requested count alone proves neither admission nor serving.
If pendingCohort remains set, inspect Kueue quota and Pod events before increasing ceilings.
Step 5: Observe retirement
Return to low concurrency and keep traffic running through the full observation window. The default scale-down stabilization window is 300 s; a 90 s cooldown can finish before any shrink. Leave additional time for the decode grace, Router withdrawal and drain.
BENCH_INCLUSTER=1 python3 .agents/skills/gpustack-operator-benchmark/scripts/bench_client.py load \
--endpoint http://autoscaling-vllm-pd-router.gpustack-bench.svc:8000 \
--model Qwen/Qwen3-8B --arm active --suite autoscaling --label cooldown \
--input-tokens 1024 --output-tokens 256 --concurrency 2 --duration 600 \
--out /tmp/scaler-cooldown.json --requests-out /tmp/scaler-cooldown-requests.jsonlPoll both status documents while the load runs. Repeat these commands to preserve changes:
kubectl -n gpustack-bench get modeldeployment autoscaling-vllm-pd -o json \
| jq '{roles: .status.roles, retirement: .status.retirement}'
kubectl -n gpustack-bench get mds autoscale-vllm-pd -o json \
| jq '{roles: .status.roles, lastScaleDown: .status.lastScaleDown,
conditions: .status.conditions}'
kubectl -n gpustack-bench get pods,workloadsThe protocol withdraws the selected replica, enters Draining, then commits deletion with state
Deleting. Kubernetes may display its deleting Pod as Terminating.
Read Shutdown
for the complete protocol
and its bounded handoff wait. On Aborted, inspect lastScaleDown and the restored vector.
If a shorter test window is needed, choose a positive downward stabilization window longer than the
sampling interval. Zero stores the default 300 s. Keep maxReplicas at least as high as the current
count; lowering it below that count blocks planning with BoundsConflict.
Verification
The recorded standalone single-role check produced these observations:
PASS | the target's replica count rises under load | the count of the role server is 2 while the heavy phase runs
PASS | the scaler records the vector it committed | the scaler committed server=2 on term 1
PASS | the replica count falls when the scaler is narrowed | the count of the role server is 1 after the scaler was narrowed
PASS | the retirement that carried the count down is observed | the target carried the retirement states Disqualified,DrainingThese lines come from a separate Server target, not the PD example above. The check also confirmed the new replica served and one Ready survivor remained after retirement. It failed its all-phases completion criterion because scale-out lost requests. See the lifecycle result table for exact counts and measurement limits.
For a new run, retain client outcomes, new-replica counters, Ready counts and retirement transitions.
Require successful requests during the tested transition before claiming uninterrupted serving.
Missing state samples do not establish a complete Draining-to-deletion trace.
Resolving a catalog prior
Catalog.Resolve turns one measured sweep into a capacity seed by walking the trade-off every
benchmark sweep records: as concurrency rises, per-replica throughput — proportional to power
efficiency (tokens/s/MW at a fixed node power) — climbs toward saturation, while per-user
interactivity (tokens/s/user ≈ 1000 / ITL milliseconds) falls.
This example resolves the shipped public profile (Server, deepseek-ai/DeepSeek-R1-0528, fp8, sglang, amd-mi325x, gpusPerReplica: 8, tensorParallel: 8) from
pkg/worker/scaling/priors/priors.yaml. Its five measured points follow; power efficiency is
illustrative at a ~0.0115 MW eight-accelerator node, and interactivity is 1000 / ITL P90:
| Concurrency | Per-replica tokens/s |
tokens/s/MW |
TTFT P90 (ms) | ITL P90 (ms) | tokens/s/user |
|---|---|---|---|---|---|
| 4 | 1875.02 | ~163k | 665.4 | 16.95 | 59.0 |
| 8 | 3167.82 | ~275k | 1001.7 | 18.47 | 54.1 |
| 16 | 4988.96 | ~434k | 1119.7 | 20.82 | 48.0 |
| 32 | 6496.65 | ~565k | 1516.6 | 28.32 | 35.3 |
| 64 | 7630.2 | ~663k | 2636.2 | 44.44 | 22.5 |
A Server role held to P90 TTFT of 2000 ms and P90 ITL of 33 ms — a floor of about
30.3 tokens/s/user — configures as:
apiVersion: worker.gpustack.ai/v1
kind: ModelDeploymentScaler
metadata:
name: deepseek-scaler
namespace: gpustack-bench
spec:
targetRef:
name: deepseek-server # Server deployment matching the DeepSeek-R1-0528 MI325X TP8 profile above
roles:
- name: server
minReplicas: 1
maxReplicas: 4 # the plan the resolver derives below must stay inside the bounds
algorithm:
type: TokenThroughput
tokenThroughput:
latencyQuantile: "0.90" # resolve the prior at P90, not the P95 default
ttftTargetMilliseconds: "2000"
itlTargetMilliseconds: "33" # a 30.3 tokens/s/user floor
# serverTargetTokensPerSecond omitted: it resolves from the catalog
capacityPrior:
headroom: "0.75" # catalogRef omitted: the embedded catalogThe resolver then:
- Matches the profile by the target’s normalized
(role, model, quantization, engine, hardware, topology)tuple — no profile identifier is configured. - Filters the points against the P90 targets: concurrency
4,8,16and32meet both; concurrency64breaches them (TTFT P902636.2ms, ITL P9044.44ms ≈22.5 tokens/s/user), establishingHigherLoadSLOFailure. Looser targets that let every point pass would instead cap at the measured concurrency64— never extrapolated (ExplicitSaturation). - Selects the feasible point with the highest per-replica
tokens/s— thetokens/s/MWmaximum under the targets — soconcurrency: 32seeds the capacity:6496.65tokens/s, about565k tokens/s/MWat35.3 tokens/s/user. - Applies the headroom:
6496.65 * 0.75 = 4872.49effective tokens/s per replica. A live demand of15,000tokens/s therefore plansceil(15000 / 4872.49) = 4replicas — about3,750tokens/s per replica — keeping every replica inside its interactivity target.
Bounds, pacing and the other guards still apply to that recommendation; Capacity comparison then measures the dynamics these seeds produce.
Capacity comparison
The initial comparison helper accepts a single-Server ModelDeployment (len(roles) == 1 and
kind: Server) with no spec.router, spec.kvCache or spec.kvTransfer, and a target-owned
Service (<target>-<role>) publishing one HTTP port.
Its engine-family image, identity (annotations or inferred) and effective launch flags must match
the offline qualified catalog (outcome: Accepted and evaluationTraceID disjoint from
fitRunIDs). It does not run Prefill/Decode comparison or require eight GPUs.
The helper compares frozen-manual, tuned-manual and catalog across at least three serial
repeats (alternating or seeded). Apart from capacity fields, the arms share Active mode,
TokenThroughput or Predictive, latency targets, bounds, pacing, image, phases, seed and start
condition.
For Predictive, the manual arms set both forecast and fallback capacities;
the full-Catalog arm omits both. The API still supports partial overrides outside this comparison.
Every scaler sets a positive downward stabilizationWindowSeconds, and the quiet cooldown phase
(concurrency: 0) spans at least stabilizationWindowSeconds plus startup and retirement allowances.
Copy .agents/skills/gpustack-operator-benchmark/manifests/autoscaling-comparison.json
(REPLACE_WITH_* values are template placeholders, not a runnable profile) and supply your context,
clusterQueues, disjoint fitRunIDs and evaluationTraceID, prepared driver Pod, JSON
deployment/scaler manifests and offline qualified catalog ConfigMap. Priors are acquired offline
only, never from live evaluation traffic.
bash .agents/skills/gpustack-operator-benchmark/cases/case-5-autoscaling.sh \
--capacity-comparison --config FILE --out DIR --validate-only
python3 .agents/skills/gpustack-operator-benchmark/scripts/autoscaling_report.py \
ARM_DIR --out REPORT_JSON--validate-only makes zero cluster API or driver calls, writes validated-plan.json and catalog-input.json under --out, and
exits 0 when valid or 2 on validation or input error. On a live hardware gate, --execute writes per-round <NN>-<arm>/
artifacts (inputs/*.json, run.json, timeline.jsonl, prepare_reqs.jsonl, <phase>_reqs.jsonl, <phase>.json, cleanup.json,
report.json) and root comparison.json (readyReplicaSeconds and latenciesByRun).
--execute exits 0 when all rounds, parser checks and UID-preconditioned raw DELETE cleanups
succeed across at least three valid repeats per arm, 1 on round or cleanup failure (130 on
interrupt), and 2 on validation error.
autoscaling_report.py writes schema 1 diagnostics (null metrics with reasons and no win
verdict when evidence is missing) and exits 0 when valid, 1 when invalid, and 2 when
ARM_DIR is missing or the output cannot be written.
See Scaling Benchmark
for the recorded 18-round H200 comparison. Legacy invocations (case-5-autoscaling.sh [OUT_DIR] ...
or --legacy-topology) remain aliases for the five-configuration topology benchmark.
Cleanup
Delete the scaler first so its finalizer can release the binding. Deleting a scaler leaves its target serving at the current counts. Delete the target separately when it is no longer needed.
kubectl -n gpustack-bench delete modeldeploymentscaler autoscale-vllm-pd --wait=true
kubectl -n gpustack-bench get modeldeployment autoscaling-vllm-pd -o json \
| jq '.metadata.annotations["modeldeployment.gpustack.ai/scaler-binding"]'
kubectl -n gpustack-bench delete modeldeployment autoscaling-vllm-pd --wait=trueRemove a separate multimodal target and scaler the same way if created.
Keep shared prefetched artifacts and default/bench-driver when other procedures use them.
Troubleshooting
| Symptom | Signal | Check & Mitigation |
|---|---|---|
| Counts remain at one | Condition: Active=False with reason ShadowMode |
Activate only after reviewing Shadow recommendations. |
| Apply is refused | Reason: AlreadyBound |
Read the binding owner; use the existing owner or release it before applying another scaler. |
| Binding annotation is unreadable | Reason: InvalidBinding |
Fix or remove the annotation on the target; the scaler cannot tell which scaler owns the target. |
| Counts sit outside the policy | Reason: BoundsConflict or RatioConflict |
Compare the current counts with the policy; include the current vector in every bound and ratio band. |
| Recommendations wait on metrics | Reason: WaitingForMetrics on the MetricsReady condition |
Allow valid successive samples from role signals and engine metrics; investigate missing or restarted endpoints. |
| Catalog prior fails to resolve | Condition: CapacityPriorsReady=False (CapacityPriorUnresolved, InsufficientLoadCoverage or IncompatibleMetric) |
Check catalogRef, the target identity (annotations or inferred), engine-family image, single-Pod topology and explicit latency targets, or remove capacityPrior / supply explicit capacities. |
| Growth stops at admission | Reason: ResourceExhausted |
Read pendingCohort, Workloads and quota; restore admission capacity before retrying growth. |
| Scale-down ends aborted | Reason: ScaleDownAborted |
Read the retirement reason and lastScaleDown; resolve the drain failure and wait through the abort cooldown. |
| Cooldown outlives its window | Reason: PacingBounded |
Check stabilization, grace and coverage; continue observation, since a configured cooldown phase does not bypass guards. |
| Router failures during growth | Log: Router logs and client errors | Preserve evidence; the recorded keep-alive setting did not prevent all failures. |
| SGLang KV transfer failures | Log: engine logs and TCP port reuse | Follow the documented setting and verify kubelet prerequisites. |