Model Deployment Routing

A managed router selects among the HTTP endpoints of serving replicas. Every routing choice measured here ran on a server role; on a prefill/decode (PD) pair, where each half is chosen separately, none has been run.

See Prefill and Decode for the external data parallelism (External DP) boundary within PD roles.

Contents

• Which Pods receive traffic
• Encode stage routing
• Each router’s default
• Prefix affinity
• Switching to round robin
• No policy flag
• Serving observations
• Per-worker routing metrics

Which Pods receive traffic

The engine’s load balancing (LB) mode determines the candidate endpoints. The Router policy chooses among those candidates. round_robin does not change the endpoint selector.

Encode Pods are never part of the deployment’s answering set. Under llm-d-router they are discovered for stage routing, and under vllm-router their Service endpoint is passed through --epd-config, while the deployment Service never targets them. All three routers (llm-d-router, vllm-router and sglang-gateway) are admitted with an Encode role for their supported engine and platform combinations. See Encode stage routing .

For an eligible External Server role, Router discovery and ordinary Services omit member-index=0. They retain the endpoint eligibility label, so an unhealthy group cannot receive new traffic. The replica’s headless Service always publishes every member for peer communication.

Known non-Internal modes follow the External selector rule. Status retains the engine’s distinct mode name. Hybrid GPU behavior remains untested. See Prefill and Decode for PD limitations.

Managed roles expose one fixed HTTP port per Pod, defaulting to 8000. All routed roles must use the same serving port. A shared custom port remains supported. Admission rejects vLLM’s MultiPort mode because it starts endpoints on additional, incremented ports. A decoder proxy’s internal backend port remains separate from its published HTTP port. See the External DP walkthrough for a minimal per-rank configuration.

Encode stage routing

How the encoder is reached depends on the engine and router. Send requests to status.endpoint in every case.

With llm-d-router and vLLM, the router defines an encode scheduling profile over the Pods labelled modeldeployment.gpustack.ai/role-kind: encode. The profile applies no queue, KV cache or prefix scorers, because an encoder keeps no KV cache and tracks no tokens. A multimodal-aware decider sends a request with media through an encoder first. A text-only request goes straight to prefill and decode.

On the Ascend encode/prefill/decode (EPD) configuration (llm-d-router + vLLM-Ascend + ECExampleConnector), the encode role answers requests that carry media and stays idle for text-only traffic. The connector hands the result to Prefill through the shared volume both roles mount; the router is not in that path.

The operator keeps a vLLM encoder in the router’s candidate set on every platform. Its address-based encoder path applies only to SGLang.

The decode sidecar’s --ec-connector follows the admitted row: ec-example on Ascend and ec-nixl on NVIDIA and AMD. See Admitted matrix .

With vllm-router and vLLM (on NVIDIA and AMD), the operator passes --epd-config={"encoder_urls":["http://<md>-<encode-role>.<ns>.svc:<port>"]} to the router. The router calls the encoder Service first, forwards ec_transfer_params to the prefill worker so it pulls the embeddings over NIXL (ECCPUConnector), and strips ec_transfer_params before forwarding to decode. --epd-config is operator-owned and refused in spec.router.extraArgs.

With llm-d-router and SGLang, the router splits every request between prefill and decode, because an SGLang decode role cannot prefill. The sidecar runs without --ec-connector, since this engine uses no encoder-cache connector.

Encoders are also excluded from KV cache event discovery, because they produce no cache blocks.

Each router’s default

vllm-router and sglang-gateway route by cache_aware, and llm-d-router scores every candidate for each request. The operator renders no policy flag for the first two, so each runs its upstream default. The table describes the bundled vLLM router v0.1.15 and SGLang gateway gateway-v0.3.1; the llm-d row describes the operator’s scoring profile.

Both cache_aware routers log the policy once at startup, as policy: CacheAware { cache_threshold: 0.3, … }.

Prefix affinity

Under cache_aware, requests that share a long prefix and arrive one after another all go to one replica, and the others stay idle. That is the policy working, not a fault: the replica that served the prefix holds it in its prefix cache, so the next request sharing it does not recompute it.

Such a request only goes elsewhere once the load gap above opens, and traffic that sends one request at a time never has more than one in flight to open it.

Measured on two server replicas, each on its own GPU, with serial chat requests that open with the same text of about 500 characters and end in a question of their own, streaming and non-streaming in turn. No request failed:

How any of the three spreads concurrent traffic is not measured.

Switching to round robin

Add --policy round_robin to spec.router.extraArgs on vllm-router or sglang-gateway to send each request to the next replica in turn, whatever it shares with earlier ones:

 apiVersion: worker.gpustack.ai/v1
 kind: ModelDeployment
 metadata:
   name: round-robin-serving
 spec:
   router:
     name: vllm-router                    # or sglang-gateway
+    extraArgs:
+      - --policy
+      - round_robin

--policy is in neither router’s refused list , so admission passes it and the operator appends it to the command line it renders. The router then logs Starting router … | policy: RoundRobin.

Measured on the same two replicas and the same kind of traffic, with the flag set at creation: under each router, 120 requests with none failing, 60 on each replica. vllm-router counted them in vllm_router_policy_decisions_total{policy="round_robin"}, 60 per replica, and sglang-gateway in smg_worker_selection_total{policy="round_robin"}, 120 in all.

router.extraArgs is editable , so a running deployment can switch as well. The router Pod is then replaced, and the new one starts with no record of earlier prefixes. Only a flag set at creation has been run.

The value is each project’s own, and the router checks it, not admission, so a misspelled one stops the router at startup:

Only round_robin has been run. The three cache_aware thresholds pass through extraArgs the same way, as do both routers’ --prefill-policy and --decode-policy for a prefill/decode pair; none of those has been run either.

No policy flag

llm-d-router cannot be switched through extraArgs. Its scorers and their weights are in the configuration document the operator renders and mounts, and --config-file, which would point the router at another one, is refused there. No field sets them either.

Serving observations

Which replicas a Router can reach is not the same question as which replicas it is serving, and the operator reports both. The first is the qualification list, and the second is an observation of the Router itself; they are published separately in status.roles[].endpoints and never collapse into one number.

The observation is a plain HTTP GET with a short timeout against each Router Pod’s own address. It deliberately does not go through the Router Service, which would load balance across the Router’s replicas and report one of them as the whole Router. What Confirmed then means is the serving-observation contract status.roles[].endpoints defines once.

A collected view informs the answer for fifteen seconds. Past that the answer becomes Unknown, so a Router that stops being reachable is reported as unobserved and never as serving whatever it served last.

The freshness bound is what keeps a stopped Router from reading as a healthy one. A view that outlived its evidence would report a serving count for a process that is gone, and the only way back would be an edit to the deployment.

Per-worker routing metrics

The deployment’s metrics snapshot does not say which replica served a request. Each router’s own /metrics does, for the series below, all seen exported in runs. A worker label is the worker’s URL, which carries its Pod address.

Neither sglang-gateway nor llm-d-router keeps a running count per replica. Behind them, compare the engine Pods’ own counts instead, such as each Pod’s time to first token (TTFT) histogram _count.