Model Deployment Routing

A router only chooses when a role has more than one replica. Every routing choice measured here ran on a server role; on a prefill/decode pair, where each half is chosen separately, none has been run.

Contents

• Each router’s default
• Prefix affinity
• Switching to round robin
• llm-d-router takes no policy flag
• Per-worker routing metrics

Each router’s default

vllm-router and sglang-gateway route by cache_aware, and llm-d-router scores every candidate for each request. The operator renders no policy flag for the first two, so each runs its upstream default. The table describes the bundled vLLM router v0.1.15 and SGLang gateway gateway-v0.3.1; the llm-d row describes the operator’s scoring profile.

Both cache_aware routers log the policy once at startup, as policy: CacheAware { cache_threshold: 0.3, … }.

Prefix affinity

Under cache_aware, requests that share a long prefix and arrive one after another all go to one replica, and the others stay idle. That is the policy working, not a fault: the replica that served the prefix holds it in its prefix cache, so the next request sharing it does not recompute it.

Such a request only goes elsewhere once the load gap above opens, and traffic that sends one request at a time never has more than one in flight to open it.

Measured on two server replicas, each on its own GPU, with serial chat requests that open with the same text of about 500 characters and end in a question of their own, streaming and non-streaming in turn. No request failed:

How any of the three spreads concurrent traffic is not measured.

Switching to round robin

Add --policy round_robin to spec.router.extraArgs on vllm-router or sglang-gateway to send each request to the next replica in turn, whatever it shares with earlier ones:

spec:
  router:
    name: vllm-router                    # or sglang-gateway
    extraArgs:
      - --policy
      - round_robin

--policy is in neither router’s refused list , so admission passes it and the operator appends it to the command line it renders. The router then logs Starting router … | policy: RoundRobin.

Measured on the same two replicas and the same kind of traffic, with the flag set at creation: under each router, 120 requests with none failing, 60 on each replica. vllm-router counted them in vllm_router_policy_decisions_total{policy="round_robin"}, 60 per replica, and sglang-gateway in smg_worker_selection_total{policy="round_robin"}, 120 in all.

router.extraArgs is editable , so a running deployment can switch as well. The router Pod is then replaced, and the new one starts with no record of earlier prefixes. Only a flag set at creation has been run.

The value is each project’s own, and the router checks it, not admission, so a misspelled one stops the router at startup:

Only round_robin has been run. The three cache_aware thresholds pass through extraArgs the same way, as do both routers’ --prefill-policy and --decode-policy for a prefill/decode pair; none of those has been run either.

llm-d-router takes no policy flag

llm-d-router cannot be switched through extraArgs. Its scorers and their weights are in the configuration document the operator renders and mounts, and --config-file, which would point the router at another one, is refused there. No field sets them either.

Per-worker routing metrics

The deployment’s metrics snapshot does not say which replica served a request. Each router’s own /metrics does, for the series below, all seen exported in runs. A worker label is the worker’s URL, which carries its Pod address.

Neither sglang-gateway nor llm-d-router keeps a running count per replica. Behind them, compare the engine Pods’ own counts instead, such as each Pod’s TTFT histogram _count.