Model Deployment Shutdown

A ModelDeployment replica drains during the interval between Pod deletion and engine exit. The grace period bounds how long it can wait for running requests and engine shutdown.

Contents

• Drain window
• Retirement protocol
• Limitations

Drain window

A replica whose command line the operator builds drains before its engine stops, whatever deleted its Pod: a rollout, a node drain, a preemption or kubectl delete pod. A removal the spec makes — a replicas reduction, a surplus shed, a dropped role — goes through the retirement protocol first, and its Pod is deleted when that finishes.

An edit that replaces a role’s configuration deletes that role’s old replicas one at a time, and each deleted replica drains exactly as below. The protocol reads a leaving replica as it is deployed — the size and command it was created with — so an edit elsewhere does not change how it is classified: a take-over replica is still held, a prefill or decode replica still waits for its grace window.

G below is the role’s grace: roles[].terminationGracePeriodSeconds, 30 when unset, written to the Pod’s field of the same name.

The hook of an idle replica therefore returns about 6 s after its delete, whatever G is, and vLLM or an SGLang server exits a few seconds later. The first 5 s and the last 5 s are constants; only the wait between them moves with G, which admission takes from 15 to 3600.

Why — every supported router drops a replica on its deletion timestamp rather than on its readiness, but a router can pick the replica in the instant before it learns of the delete; the first 5 s is where that request lands. The wait happens before SIGTERM because vLLM aborts every running request on the signal by default, and serves nothing new while it drains if told to wait.

Raising G costs accelerators and quota. It is a ceiling, not a wait: an idle replica still leaves when its engine exits. A busy or slow one holds its accelerators and quota up to G on every delete, and a rollout waits each departing replica out one at a time per role, so the cost multiplies into every rollout. What it buys is a longer wait for running requests, and a clean exit for an engine slow to stop after SIGTERM.

An SGLang prefill or decode role needs up to about 20 s after SIGTERM, even idle, so set its terminationGracePeriodSeconds to about 45. Measured idle, a decoder left 26 s and a prefiller 28 s after its delete, 2 s short of the kill at the default 30 s. From SGLang 0.5.14 on, its split-role scheduler ignores the shutdown request SIGTERM leads to, and the engine waits a fixed 15 s for that scheduler before killing it.

Why it is left alone — no flag shortens that wait; SGL_FORCE_SHUTDOWN skips only the drain. A grace of 45 buys a clean exit rather than a kill, since the engine kills its scheduler itself either way. The replica is idle by then, so the kill at 30 s cuts no request.

The hook sums these gauges from the listener the Pod’s scrape annotations name. On a direct decoder that is the engine behind the routing proxy, not the port the Service fronts.

A listener that refuses the read, answers an HTTP error or fails the TLS handshake ends the hook at once, since there is nothing it can measure. A read that times out counts as busy, and so does a sample the hook cannot parse.

Adding the hook and the grace changed every replica’s Pod spec, so upgrading to the release that carries them turns each existing replica over once; see Rollout behavior .

Changing a role’s terminationGracePeriodSeconds turns its replicas over the same way, and each replica that leaves in that rollout leaves with the grace it was created with. Writing 30 onto a role that set none renders the same Pods, so it turns nothing over.

Retirement protocol

A replica the spec takes capacity away from is withdrawn and drained before it is deleted, and the deletion waits for that to finish. The protocol covers a replicas reduction, a shed of surplus replicas, and a replica of a role the spec has dropped. Its progress is the status.retirement field , and the replica stays in place, still counted, for as long as the protocol holds it.

A replica that was never scheduled and never admitted is the one exception: it has nothing to drain, so it is removed without the protocol. See Unadmitted replicas .

For llm-d-router, withdrawal also waits for recent dispatches reported for a target member. This applies after the Router removes that member from selection. These records do not increase the serving count. A dispatch to an unrelated survivor does not hold the target.

Without spec.router, withdrawal cannot observe established connections made directly through the role’s Service. An admitted, running replica stays held during scale-down or surplus removal. The withdrawal budget expires after 30 seconds and retirement ends as Aborted, retaining its Pods, Workload and capacity. Removing the Service endpoint does not prove those connections have ended. Groups that never executed or received admission follow their separate cleanup path.

Two mechanisms are involved and they answer different questions. The protocol decides whether the Pod may be deleted; the drain window above decides how long the engine takes to exit once it is, on the kubelet’s grace clock after the delete. A replica passes the first and still spends the second.

Reading member work

The read is one bounded, read-only pods/exec into the member’s own main container, which fetches that engine’s /metrics over loopback on the port the member’s own scrape annotations name. Reading it from inside the member is what makes the answer per-member: a scrape through the Service would land on whichever replica answered and could not say which one was idle.

The gauges read are the same ones the hook sums, and they are named once in the tree, so the protocol and the hook cannot disagree about what counts as work in flight.

A member must read idle twice, on complete envelopes, before the deletion commits. Both reads have to carry every expected gauge, and a single unlucky zero cannot carry the protocol past a member that is still working. A second read that finds work holds the operation.

Not every member of a replica is read, because not every member can answer. The protocol works out which members of the target can speak for it as a whole, and reads only those: a replica of one Pod answers through itself, and a leader-served replica of several answers through the member carrying rank 0.

Selecting them does not narrow the deletion, which still covers every member the reservation bound, so a replica is only held when the operator cannot tell which member its answer comes from. See Limitations for what a hold there means.

Drain read refusals

Every one of these holds the replica in place, and the reason on the reservation names which one applied. They are the ordinary outcomes of a replica that is busy, loading, or outside the configurations the protocol can measure.

A restarted container is held, and the hold is bounded. The gauges a restarted engine serves describe a fresh process, so they cannot speak for the work the protocol set out to account for. The drain budget and the operation’s total still apply, so a replica that keeps crash-looping ends as Aborted with its capacity retained, and never blocks the queue.

A restart throws away a loaded model and any request it was serving, so the replica is not worth less for it. The protocol cannot tell whether a restarted engine is a replacement it should wait for or a crash loop it should give up on, and the budget is what resolves that question.

A disaggregated replica is held for a grace window, then drained like any other. A prefill or decode replica has no verifiable decoder-side release, so the protocol waits before it reads such a replica. See Disaggregated replicas below.

Disaggregated replicas

A prefill or decode replica is classified as disaggregated when its member Pods carry the prefill or decode role kind, when an SGLang member runs --disaggregation-mode prefill or decode, or when a vLLM member runs a data-parallel External, Hybrid or MultiPort load balance. The classification is read from the member’s own rendered command, so it still applies to a replica whose role the spec has already dropped.

What the protocol cannot verify is the release. A decode replica reading zero is an engine about to be handed work, and nothing in either engine’s metrics proves it will not be. Both engines report a decoder’s work while it waits to receive its KV: a prefill half holds it in its transfer queue, and a decode half holds it in its own prealloc and transfer queues.

The window is 60 seconds, measured from the moment the drain phase began. The drain phase is entered only after the router withdrawal is confirmed, so the replica is already out of selection when the window starts. Once the window elapses, the ordinary two-consecutive-idle-reads check runs on the answering members, and the deletion commits when they are idle. The window sits inside the 240-second drain budget, so it does not extend the operation.

Answering-member selection is not part of this. It stays keyed on the vLLM data-parallel shapes alone, so a multi-Pod leader-served prefill or decode replica still answers through its leader rather than through every member. A member whose parallelism flags cannot be read is still held as unsupported.

Retirement adoption

A reservation is cancelled when the spec re-declares the ordinal it holds. A scale-up that restores the ordinal of a reservation in Admitted, Disqualified, Withdrawing, Draining or Aborted cancels the reservation, provided the group still holds the members the reservation names and the replica qualifies on every ground except the reservation itself.

The replica becomes eligible again through the reversible modeldeployment.gpustack.ai/endpoint-eligible label, which the reconciler restores on the members it held. The event RetirementAdopted records the cancellation.

This is a cancellation and not a duplicate: the Pods are already there and healthy, so the next pass finds the replica current and creates nothing. From Deleting on the behavior does not change, because the delete has already been committed; the delete completes and the ordinal is recreated on a later pass.

Creating replicas for other ordinals or other roles never waits on a reservation, so a reservation for one replica never holds up the rest of the deployment.

Unadmitted replicas

A replica that was never scheduled and never admitted is removed without the protocol. An undeclared replica whose members were never scheduled — no node, no address, still Pending, and without the endpoint-eligible label — and whose group Workload was never admitted has nothing to drain. The reconciler deletes its Pods and its Workload straight away.

An unscheduled Pod has no running container and no metrics endpoint, so running the protocol against it would hold for the whole drain budget and end as Aborted, retaining a Pod that was never going to serve. If admission or scheduling won the race before the pass ran, the replica enters ordinary retirement instead.

Limitations

• A request still running at G − 5 is cut, 25 s after the delete at the default. vLLM aborts it when SIGTERM arrives; SGLang keeps draining it on its own until the kill at G. No supported router moves a running request to another replica: the SGLang gateway retries only while the response has not started, and the other two routers do not retry.
• An SGLang prefill or decode replica whose hook returns later than G − 20 is killed at G, since its engine then has less than the 20 s it takes to exit: 10 s after the delete at the default, 25 s at 45. The replica is idle by that point, or its hook gave up on it, so the kill cuts no request the deadline would not.
• A take-over role (one that sets command) gets no hook, because the operator cannot claim that container serves the gauges. Its Pod gets the role’s terminationGracePeriodSeconds as given, and keeps the Kubernetes default when the role sets none.
• A role that moves the engine with its own --port gets only the first 5 s: nothing answers the hook on the port the operator rendered.
• A replica of several Pods drains through its leader, and every member is still deleted. The other members serve no metrics, so their hooks return after the first 5 s, and they receive SIGTERM while the leader is still draining. The protocol reads the members whose answer stands for the whole replica, which on a leader-served replica is the leader alone.
• A replica whose answering member cannot be named is held — none of its members carries the leader index, more than one does, its members disagree on the configuration, or its role says a size the replica does not have — and the reason names which. Selecting the leader does not narrow the deletion: as the protocol above states, it still covers every member the reservation bound.
• A prefill or decode replica is held for the disaggregated grace window, then drained like any other. It is never deleted inside the 60 s window, and a replica that reaches the window while still serving is held for the rest of the drain budget. The drain window still applies to it once something else deletes its Pod.
• A direct decoder on a cluster below Kubernetes 1.29 runs its routing proxy as a classic container, which stops at the delete while the engine behind it drains, so a request in flight through the proxy still resets. The proxy’s image has no shell to run a hook of its own.