# Model Deployment Shutdown

A `ModelDeployment` replica drains during the interval between `Pod` deletion and engine exit.
The grace period bounds how long it can wait for running requests and engine shutdown.

## Contents

- [Drain window](#drain-window)
- [Retirement protocol](#retirement-protocol)
- [Limitations](#limitations)

## Drain window

A replica whose command line the operator builds drains before its engine stops, whatever deleted
its `Pod`: a rollout, a node drain, a preemption or `kubectl delete pod`. A removal the spec makes —
a *`replicas`* reduction, a surplus shed, a dropped role — goes through
[the retirement protocol](#retirement-protocol) first, and its `Pod` is deleted when that
finishes.

An edit that replaces a role's configuration deletes that role's old replicas one at a time, and
each deleted replica drains exactly as below. The protocol reads a leaving replica as it is
deployed — the size and command it was created with — so an edit elsewhere does not change how it
is classified: a take-over replica is still held, a *prefill* or *decode* replica still waits for
its grace window.

`G` below is the role's grace: *`roles[].terminationGracePeriodSeconds`*, 30 when unset, written to
the `Pod`'s field of the same name.

| From deletion | Default timing | Action |
|---|---|---|
| 0 s | 0 s | the routers drop the replica, and the kubelet starts the engine container's preStop hook |
| 0–5 s | 0–5 s | the engine keeps serving normally, including a request routed to it just before the delete |
| 5 s to `G` − 5 | 5–25 s | the hook reads the engine's in-flight gauges once a second and returns on the second idle read |
| `G` − 3 at the latest | 27 s | the engine receives SIGTERM once the hook returns: the hook starts no read after `G` − 5, and its last read can take 2 s more |
| `G` | 30 s | the kubelet kills what is left |

The hook of an idle replica therefore returns about 6 s after its delete, whatever `G` is, and vLLM
or an SGLang server exits a few seconds later. The first 5 s and the last 5 s are constants; only
the wait between them moves with `G`, which admission takes from 15 to 3600.

> **Why** — every supported router drops a replica on its deletion timestamp rather than on its
> readiness, but a router can pick the replica in the instant before it learns of the delete; the
> first 5 s is where that request lands. The wait happens before SIGTERM because vLLM aborts every
> running request on the signal by default, and serves nothing new while it drains if told to wait.

**Raising `G` costs accelerators and quota.** It is a ceiling, not a wait: an idle replica still
leaves when its engine exits. A busy or slow one holds its accelerators and quota up to `G` on every
delete, and a rollout waits each departing replica out one at a time per role, so the cost multiplies
into every rollout. What it buys is a longer wait for running requests, and a clean exit for an
engine slow to stop after SIGTERM.

**An SGLang *prefill* or *decode* role needs up to about 20 s after SIGTERM, even idle, so set its
*`terminationGracePeriodSeconds`* to about 45.** Measured idle, a decoder left 26 s and a prefiller
28 s after its delete, 2 s short of the kill at the default 30 s. From SGLang 0.5.14 on, its
split-role scheduler ignores the shutdown request SIGTERM leads to, and the engine waits a fixed 15 s
for that scheduler before killing it.

> **Why it is left alone** — no flag shortens that wait; `SGL_FORCE_SHUTDOWN` skips only the drain.
> A grace of 45 buys a clean exit rather than a kill, since the engine kills its scheduler itself
> either way. The replica is idle by then, so the kill at 30 s cuts no request.

The hook sums these gauges from the listener the `Pod`'s
[scrape annotations](/gpustack-operator/v0.11.2/docs/modules/model-deployment/metrics/index.md#scraping-the-pods) name. On a direct decoder that is
the engine behind the routing proxy, not the port the `Service` fronts.

| Engine | Gauges summed |
|---|---|
| vLLM | `vllm:num_requests_running`, `vllm:num_requests_waiting` — the second includes a decoder's requests still waiting for their KV |
| SGLang | `sglang:num_running_reqs`, `sglang:num_queue_reqs`, `sglang:num_prefill_bootstrap_queue_reqs`, `sglang:num_prefill_inflight_queue_reqs`, `sglang:num_decode_prealloc_queue_reqs`, `sglang:num_decode_transfer_queue_reqs` |

A listener that refuses the read, answers an HTTP error or fails the TLS handshake ends the hook at
once, since there is nothing it can measure. A read that times out counts as busy, and so does a
sample the hook cannot parse.

Adding the hook and the grace changed every replica's `Pod` spec, so upgrading to the release that
carries them turns each existing replica over once; see
[Rollout behavior](/gpustack-operator/v0.11.2/docs/modules/model-deployment/configuration/index.md#rollout-behavior).

Changing a role's *`terminationGracePeriodSeconds`* turns its replicas over the same way, and each
replica that leaves in that rollout leaves with the grace it was created with. Writing 30 onto a
role that set none renders the same `Pod`s, so it turns nothing over.

## Retirement protocol

**A replica the spec takes capacity away from is withdrawn and drained before it is deleted, and
the deletion waits for that to finish.** The protocol covers a *`replicas`* reduction, a shed of
surplus replicas, and a replica of a role the spec has dropped. Its progress is the
[`status.retirement` field](/gpustack-operator/v0.11.2/docs/modules/model-deployment/status/index.md#status), and the replica stays in place, still counted,
for as long as the protocol holds it.

A replica that was never scheduled and never admitted is the one exception: it has nothing to drain,
so it is removed without the protocol. See [Unadmitted replicas](#unadmitted-replicas).

For `llm-d-router`, withdrawal also waits for recent dispatches reported for a target member.
This applies after the Router removes that member from selection. These records do not increase
the serving count. A dispatch to an unrelated survivor does not hold the target.

Without *`spec.router`*, withdrawal cannot observe established connections made directly through the
role's `Service`. An admitted, running replica stays held during scale-down or surplus removal.
The withdrawal budget expires after 30 seconds and retirement ends as `Aborted`, retaining its `Pod`s,
`Workload` and capacity. Removing the `Service` endpoint does not prove those connections have ended.
Groups that never executed or received admission follow their separate cleanup path.

The mechanisms involved answer different questions. The protocol decides **whether the
`Pod` may be deleted**; the [drain window](#drain-window) above decides **how long the engine
takes to exit once it is**, on the kubelet's grace clock after the delete. A replica passes the
first and still spends the second.

### Reading member work

The read is one bounded, read-only `pods/exec` into the member's own `main` container, which
fetches that engine's `/metrics` over loopback on the port the member's own
[scrape annotations](/gpustack-operator/v0.11.2/docs/modules/model-deployment/metrics/index.md#scraping-the-pods) name. Reading it from inside the member is what
makes the answer per-member: a scrape through the `Service` would land on whichever replica answered
and could not say which one was idle.

| Bound | Value |
|---|---|
| response size | 1 MiB |
| the engine's own answer | 2 s |
| the whole read including the connection | 5 s |

The gauges read are the same ones the hook sums, and they are named once in the tree, so the
protocol and the hook cannot disagree about what counts as work in flight.

**A member must read idle twice, on complete envelopes, before the deletion commits.** Both reads
have to carry every expected gauge, and a single unlucky zero cannot carry the protocol past a
member that is still working. A second read that finds work holds the operation.

**Not every member of a replica is read, because not every member can answer.** The protocol works
out which members of the target can speak for it as a whole, and reads only those: a replica of one
`Pod` answers through itself, and a leader-served replica of several answers through the member
carrying rank 0.

Selecting them does not narrow the deletion, which still covers every member the reservation bound,
so a replica is only held when the operator cannot tell which member its answer comes from. See
[Limitations](#limitations) for what a hold there means.

### Drain read refusals

Every one of these holds the replica in place, and the *`reason`* on the reservation names which one
applied. They are the ordinary outcomes of a replica that is busy, loading, or outside
the configurations the protocol can measure.

| Member | Verdict | Reason |
|---|---|---|
| still serving requests | held | there is work in flight, which is what the read is for |
| answers with some gauges missing | held | a partial envelope is not evidence of an empty queue, and a reader cannot sum what it did not find |
| the exec fails, times out, or returns a body that is not metrics | held | nothing was measured, and an absence of measurement is not a measurement of absence |
| the container has restarted | held | the gauges now served come from an engine that did not exist when the protocol began watching |
| a take-over role | held | the operator renders no endpoint for it, so there is nothing to read |
| an engine with no gauge list | refused | no such measurement exists for it |

**A restarted container is held, and the hold is bounded.** The gauges a
restarted engine serves describe a fresh process, so they cannot speak for the work the protocol
set out to account for. The drain budget and the operation's total still apply, so a replica that
keeps crash-looping ends as `Aborted` with its capacity retained, and never blocks the queue.

> A restart throws away a loaded model and any request it was serving, so the replica is not worth
> less for it. The protocol cannot tell whether a restarted engine is a replacement it should wait
> for or a crash loop it should give up on, and the budget is what resolves that question.

**A disaggregated replica is held for a grace window, then drained like any other.** A *prefill* or
*decode* replica has no verifiable decoder-side release, so the protocol waits before it reads such a
replica. See [Disaggregated replicas](#disaggregated-replicas) below.

### Disaggregated replicas

A *prefill* or *decode* replica is classified as disaggregated when its member `Pod`s carry the
`prefill` or `decode` role kind, when an SGLang member runs `--disaggregation-mode prefill` or
`decode`, or when a vLLM member runs a data-parallel `External`, `Hybrid` or `MultiPort` load
balance. The classification is read from the member's own rendered command, so it still applies to a
replica whose role the spec has already dropped.

What the protocol cannot verify is the release. A *decode* replica reading zero is an engine about to
be handed work, and nothing in either engine's metrics proves it will not be. Both engines report a
decoder's work while it waits to receive its KV: a *prefill* half holds it in its transfer queue, and
a *decode* half holds it in its own prealloc and transfer queues.

**The window is 60 seconds, measured from the moment the drain phase began.** The drain phase is
entered only after the router withdrawal is confirmed, so the replica is already out of selection
when the window starts. Once the window elapses, the ordinary two-consecutive-idle-reads check runs
on the answering members, and the deletion commits when they are idle. The window sits inside the
240-second drain budget, so it does not extend the operation.

Answering-member selection is not part of this. It stays keyed on the vLLM data-parallel shapes
alone, so a multi-`Pod` leader-served *prefill* or *decode* replica still answers through its leader
rather than through every member. A member whose parallelism flags cannot be read is still held as
unsupported.

### Retirement adoption

**A reservation is cancelled when the spec re-declares the ordinal it holds.** A scale-up that
restores the ordinal of a reservation in `Admitted`, `Disqualified`, `Withdrawing`, `Draining` or
`Aborted` cancels the reservation, provided the group still holds the members the reservation names
and the replica qualifies on every ground except the reservation itself.

The replica becomes eligible again through the reversible `modeldeployment.gpustack.ai/endpoint-eligible`
label, which the reconciler restores on the members it held. The event `RetirementAdopted` records the
cancellation.

This is a cancellation and not a duplicate: the `Pod`s are already there and healthy, so the next
pass finds the replica current and creates nothing. From `Deleting` on the behavior does not change,
because the delete has already been committed; the delete completes and the ordinal is recreated on a
later pass.

Creating replicas for other ordinals or other roles never waits on a reservation, so a reservation
for one replica never holds up the rest of the deployment.

### Unadmitted replicas

**A replica that was never scheduled and never admitted is removed without the protocol.** An
undeclared replica whose members were never scheduled — no node, no address, still `Pending`, and
without the `endpoint-eligible` label — and whose group `Workload` was never admitted has nothing to
drain. The reconciler deletes its `Pod`s and its `Workload` straight away.

An unscheduled `Pod` has no running container and no metrics endpoint, so running the protocol
against it would hold for the whole drain budget and end as `Aborted`, retaining a `Pod` that was
never going to serve. If admission or scheduling won the race before the pass ran, the replica enters
ordinary retirement instead.

## Limitations

- **A request still running at `G` − 5 is cut**, 25 s after the delete at the default. vLLM aborts it
  when SIGTERM arrives; SGLang keeps draining it on its own until the kill at `G`. No supported
  router moves a running request to another replica: the SGLang gateway retries only while the
  response has not started, and the other routers do not retry.
- **An SGLang *prefill* or *decode* replica whose hook returns later than `G` − 20 is killed at `G`**,
  since its engine then has less than the 20 s it takes to exit: 10 s after the delete at the
  default, 25 s at 45. The replica is idle by that point, or its hook gave up on it, so the kill cuts
  no request the deadline would not.
- **A take-over role** (one that sets *`command`*) gets no hook, because the operator cannot claim
  that container serves the gauges. Its `Pod` gets the role's *`terminationGracePeriodSeconds`* as given,
  and keeps the Kubernetes default when the role sets none.
- **A stored role with a conflicting declared port and `--port`** gets only the first 5 s if the
  hook cannot reach the engine. A role without a declared port uses its `--port` for the hook.
- **A replica of several `Pod`s drains through its leader, and every member is still deleted.** The
  other members serve no metrics, so their hooks return after the first 5 s, and they receive SIGTERM
  while the leader is still draining. The protocol reads the members whose answer stands for the whole
  replica, which on a leader-served replica is the leader alone.
- **A replica whose answering member cannot be named is held** — none of its members carries the
  leader index, more than one does, its members disagree on the configuration, or its role says a size the
  replica does not have — and the *`reason`* names which. Selecting the leader does not narrow the
  deletion: as [the protocol above](#retirement-protocol) states, it still covers every member
  the reservation bound.
- **A *prefill* or *decode* replica is held for the disaggregated grace window, then drained like any
  other.** It is never deleted inside the 60 s window, and a replica that reaches the window while
  still serving is held for the rest of the drain budget. The [drain window](#drain-window) still
  applies to it once something else deletes its `Pod`.
- **A direct decoder on a cluster below Kubernetes 1.29** runs its routing proxy as a classic
  container, which stops at the delete while the engine behind it drains, so a request in flight
  through the proxy still resets. The proxy's image has no shell to run a hook of its own.

---

**See also** — [Model Deployment](/gpustack-operator/v0.11.2/docs/modules/model-deployment/configuration/index.md) for what turns a replica over ·
[Model Deployment Scaling](/gpustack-operator/v0.11.2/docs/modules/model-deployment/scaling/index.md) for the count changes that start a retirement ·
[Model Deployment Metrics](/gpustack-operator/v0.11.2/docs/modules/model-deployment/metrics/index.md) for the scrape endpoints the hook
reads.

**Next** → [Model Deployment Status](/gpustack-operator/v0.11.2/docs/modules/model-deployment/status/index.md)
