Model Deployment Status

A ModelDeployment reports its phase, ready replicas and serving endpoint in status. Use these fields and its conditions to diagnose a deployment that is still starting or has lost capacity.

Contents

• Status
• Scaler annotations
• Group status

Status

Phase and role summary

status.phase is the field to read first: Starting, Ready, Degraded or Deleting. Degraded means some required serving capacity is ready and some is not: either some role replicas are down, or all role replicas are ready but a declared router has no ready replica. status.roles[] carries name, kind, desired, ready, quotaReserved, unmanaged, assignedFlavors, parallelism and endpoints per role.

kubectl get modeldeployments shows status.roleSummary in its ROLES column. It counts current Ready instances by kind: R is managed router Pods, S is ordinary servers, E is encoders, P is prefillers, and D is decoders. For example, 1R2S means one Ready router and two Ready servers; 1R1E1P1D means one Ready router, one Ready encoder, one Ready prefiller and one Ready decoder.

During a replacement, status combines desired and observed fields. See Rollout behavior . desired and parallelism describe the current spec; they can differ from running replicas while replacement waits. ready counts replicas still serving their deployed configuration. unmanaged and cache conditions read deployed command mode.

Without a router, the R part is absent; a declared kind with nothing Ready shows zero. Several roles of the same kind are summed. A serving instance made of several Pods still counts once; use status.roles[].desired to see the requested count.

Serving endpoints

Without a router, status.endpoint is the address the deployment-wide Service serves on, in the form <scheme>://<name>.<namespace>.svc:<port>. The scheme and port come from the first role that is not Encode, and the scheme is https only when that role passes --ssl-certfile or --ssl-keyfile. With a router, it is absent until the router is ready, then uses the router Service’s http transport. Each role, including Encode, also gets its own <deployment>-<role> Service.

status.router publishes the contract used to render the router: its implementation and address, the cache pool’s client endpoint, the metrics names and port, and one entry for every role containing the effective kind, selector, direct endpoint and any dialable KV-event endpoints. The event entries are absent when the corresponding rendered Pods do not carry publisher configuration.

This router contract names metrics sources for routing; it is not a live utilization sample. The separate Model Deployment Metrics subresource reads the current router and engine Pod endpoints and reports partial coverage explicitly.

Only --ssl-certfile or --ssl-keyfile turn the scheme to https. The rest of the --ssl-* family does not: both engines enable TLS for those two alone, so any other ssl argument passed by itself leaves an ordinary HTTP server.

Readiness gates

ready counts replicas whose engine answered, not replicas whose process started, except on the three configurations listed below, which carry no gates and keep the weaker meaning. One still loading its model counts as not ready for as long as that takes.

Both figures count replicas, which is the same as counting Pods only while a replica is one Pod. A role at size: 2 with replicas: 2 runs four Pods and reports desired: 2. A replica of several Pods is ready only when every member it declares is, so a role showing 1/2 with three of its four Pods running is reporting one whole instance, not three quarters of the role.

A gated replica carries a startup gate, a readiness gate and a liveness gate, all reading the engine’s own GET /health on the port the Service targets, so ready == desired and “the endpoint answers” are one fact rather than two.

The KV cache store’s leader answers a route of the same name that is deliberately not a readiness signal; see KV Cache Election and Failover . They are different programs, and only the engine’s route is read here.

The startup gate allows thirty minutes per attempt, not in total. A replica that has not answered by then is restarted and gets the same budget again, so a model that never finishes loading is a restart loop rather than a replica stuck at not-ready.

The liveness gate never sees that window, because the kubelet suppresses liveness and readiness alike until the startup gate succeeds.

A replica that stops answering loses readiness first, and is restarted only if it stays quiet for longer. The liveness gate’s failure threshold is wider than the readiness gate’s.

Readiness withdraws a replica from the Service and is undone by answering again; a restart throws away a model that took the startup budget to load. Different thresholds keep those two events different, and the liveness gate is what gives a quiet replica a way back, since this operator deletes a replica only to shed one the spec no longer declares or to turn over an outdated one.

The operator tells the engine where to listen. It renders --host 0.0.0.0 and --port <the port the Service targets> into the engine’s own command line, so the address traffic is sent to and the address the engine opens are one decision. The two engines do not agree on a default (vLLM opens every interface on 8000, SGLang opens 127.0.0.1:30000), and an engine left on its own would be unreachable on one of them.

Both flags are filled when missing. A role that passes either through extraArgs keeps its own value, and the operator adds only the one that is absent.

A role that enables TLS is still gated, over HTTPS, on the criterion above. Only the transport moved (the address is still the operator’s own), and the kubelet does not verify the server certificate on a probe, so a self-signed pair is graded like any other.

Three configurations carry no gates at all: a role that replaces the command through roles[].command; a role that moves where its engine listens, with --host or --port, or that demands a client certificate a probe has none to present, with --ssl-cert-reqs set to anything but 0 or 1 alongside a certificate or key; and a role declaring its port as UDP or SCTP.

The --ssl-cert-reqs values: that flag is an integer, and only 0 (CERT_NONE, the default) and 1 (CERT_OPTIONAL) let a certificate-less probe through, so a role passing either keeps all three gates. Everything else withdraws them: 2, a value outside that enum, one that is not a number, and the flag with no value at all. Losing a gate costs a signal, while fitting one to a listener that refuses it restarts a replica that is serving.

The first two fail one way and the third the other. Gating an address the operator does not know would restart a replica answering every request; an HTTP engine speaks TCP whatever the declaration says, so a gate would pass while the published endpoint forwards a protocol nothing answers. Replicas of all three are Ready as soon as their process starts.

Quota and flavors

quotaReserved is how many of the role’s replicas hold a quota reservation. Each replica is its own Workload and reserves on its own, so a role sits anywhere between zero and desired while capacity arrives; a role that shared one Workload passed all-or-nothing, and this figure could not exist.

The field is always present once written, and its zero is an observed zero: it is counted from Pod and Workload lists that succeeded, and a failed list writes no status at all rather than a zero. The two readings call for opposite actions: a visible 0 means the pass read everything and no replica holds quota, so investigate capacity; a field that is not there means the pass could not see, so wait, or look at the operator rather than the pool.

assignedFlavors answers which accelerator models did this role’s replicas actually get, read from each replica’s own Workload’s per-PodSet assignment. It is a deduplicated, sorted list, because each replica is its own Workload and Kueue assigns a flavor per Workload, so two replicas of one role can land on different flavors.

It is absent rather than empty while no assignment exists, and absent does not only mean “not admitted yet”. The field reports the flavor of a replica’s accelerator credits: a role admitted onto a pool carrying no accelerator at all reports nothing here while being perfectly healthy: its Workloads hold assignments, naming a flavor for cpu.

One entry means every assigned replica of the role names it. Several entries mean the replicas were assigned different flavors, the signal to investigate and not a degraded form of one answer.

Which replica carries which flavor is deliberately not here: the ordinal such an answer would key on is the converger’s internal slotting rather than a promise this API makes, and a reader who needs the mapping reads the replicas’ own Pods. A flavor reported here is read through the same lens the per-accelerator admission gate uses, so the two cannot disagree.

Parallelism

parallelism reports the parallelism the role’s own arguments declare, with the provenance of the reading. Every pass reads each managed role’s own argument stream — its extra arguments, under the engine’s own parser rules: registered spellings and their abbreviations, a repeated flag resolved last-wins, everything past a bare -- ignored, and the environment read only where the engine itself reads it.

Nothing is defaulted into the read: a degree the role does not declare is null, an explicit 1 and an explicit local 0 are preserved as written, and an observed mode carries its explicit boolean — the absence of a mode key is not false.

parallelism.loadBalance derives the load-balancing mode from the role’s arguments: the vLLM external-balance flag or an assigned data-parallel rank derives External, the hybrid and multi-port flags derive Hybrid and MultiPort, flags that disagree leave it Unknown with the flags named in parallelism.source.unreadableReason, and anything else balances Internal.

parallelism.source says where the reading came from: kind (ExtraArgs, Command, UnmanagedCommand), whether the reading itself was complete, and an unreadableReason naming what could not be established — the source when complete is false, or a derived fact such as the load-balancing mode when the declared flags disagree, which a complete reading can still carry.

A role that replaced its command line is not read at all — the replaced argv is the role’s own book — so its source is Unknown with complete: false and reason Unmanaged, and no degree, mode or balance is claimed from it.

A stream that cannot be parsed, including an effective environment value that is not an integer, is likewise Unknown with the flag and value named — never a silently defaulted degree.

Endpoint eligibility

endpoints separates what the operator qualified from what Routers were actually observed to select, and the two never collapse into one number:

• endpoints.eligible is how many endpoints the qualification list holds, written only when that list is complete. null means the set was never observed; an explicit 0 means it was observed and is empty. It is named for eligibility and never presents itself as serving.
• endpoints.serving.state is the actual per-Router confirmation: Confirmed means complete, fresh, identity-bound Router views agree; NotConverged means complete views disagree; Unknown means a required view is missing, stale or indeterminate; NotConfigured means the deployment declares no router at all, so there is no Router process whose answer could be missing. Only Confirmed carries endpoints.serving.value, and its zero is as real as any other number.

Replica eligibility depends on five criteria. An unknown result preserves the previous eligibility decision. MemberSetComplete (every member the role declares exists), MembersReady (each answered its engine’s health route), GroupForward (each member’s forward path was observed to carry a request), NoPlannedRetirement (no retirement operation holds the replica) and NoPendingReplacement (nothing is already on its way to replace it).

A criterion that cannot be evaluated is not evidence of a fault. An engine that is still loading, a Router view that has gone stale and a replica whose group was not observed are all cases where the operator cannot tell. endpoints.eligible stays null in each, because a short list here would be contradicted by a later pass.

A replica that was qualifying and stops is restored only when all criteria pass again and the group still holds the same members. A replica replaced in the meantime is a different group, and the answer that held for the old one does not carry over to it.

The EndpointEligibility condition says the same thing at the deployment level: while no qualification observation exists it is Unknown with reason NotObserved — not False, which would say the operator disqualified endpoints it saw. A pass that has qualified nothing reports the condition and leaves the per-role counts null.

Retirement

status.retirement is present only while a retirement operation is running, at most one at a time; absent means no operation. It names the role and replica ordinal it retires (ordinal 0 is a real position and is always encoded), the metadata.generation it was admitted against, and the target member Pod and Workload UIDs its deletions carry as preconditions.

state is the protocol step (Admitted, Disqualified, Withdrawing, Draining, Deleting, Settling, Aborted, Completed), reason names what put it there, and startedAt/deadline/phaseStartedAt carry the budgets across a controller restart. lastConsumedRetryToken is the retry directive token already consumed, persisted before the annotation that carried it is cleared.

Three kinds of removal start an operation, and they are the ones that take capacity away from a role the spec still declares. A replicas reduction starts one for the ordinals it no longer declares, a shed of surplus replicas starts one for the whole set at once, and a replica of a role the spec has dropped — by a scale-away or a role rename — starts one.

Everything else that deletes a replica deletes it without an operation, including the rollout that turns an outdated replica over.

The three are the removals for which a withdrawal and a drain are worth their cost, since each one takes away capacity the deployment had. A rollout replaces a replica with an identical one and has nothing to protect, so it does not pay for a protocol.

A second removal arriving mid-operation is refused, and the replica is held. At most one operation runs at a time, and the replica a second intent named stays where it is, so it is not taken out from under the first. Look for reason naming the refusal when a scale-down appears to stall behind another removal.

Each step has its own budget, and the operation has a total. A phase that runs out ends the operation, and the total is the earlier of the two:

Aborted is retained capacity, not a rollback: a budget exhausted before deletion leaves every member and the Workload in place, and the reason names the step that ran out. Completed is the one state that clears the field, because a reservation left behind would keep holding a slot no operation occupies.

The drain step is where a replica still holding requests is caught before anything is cut; it is described in Model Deployment Shutdown .

Conditions

Ten conditions describe states that a single phase cannot represent. They are independent: “quota reserved but cache not attached” is a real and actionable state. Nine are described below; WeightsReady is described with the artifact it reports on, in Model Artifact .

DomainRegistered

Whether the referenced Binding resolved and its domain was read.

QuotaReserved

Whether every one of the deployment’s Workloads holds Kueue quota. Every replica is its own group with its own Workload, so a deployment of N replicas is N Workloads. True is therefore an answer about the whole set and not about whichever Workload sorts first, because half a deployment holding quota is not the deployment holding quota.

A message that names groups names them by role, never by instanceType: two roles can name one type, so a type would point at two roles’ groups at once while a role names exactly the set that waits. The queue an operator has to free follows from the named role’s own instanceType.

It mirrors the Workloads’ own conditions, so it stays current after admission rather than reporting only the moment a Workload was admitted.

PodGroupIncomplete, NoQueueInReservedNamespace and AdmissionInFlight all observe no Workload but mean different things. The first clears when the missing replica exists; the second is permanent because reserved namespaces deliberately have no LocalQueue; the third clears when Kueue composes the Workload. Telling them apart is why the reason exists.

Pending, PreemptedInPart and Parked all say the set does not hold quota, and they are separate because the action differs. Pending resolves itself when capacity appears. Parked is over already: those workloads were deactivated and no longer ask for anything. PreemptedInPart is the one to act on, and the replicas that kept their quota hold accelerators until either the reclaimed replicas are admitted again or the deployment is deleted.

How long the wait is worth making depends on what preempted the deployment, which is outside this object and this operator. The condition reports the state and leaves that decision with you.

The cost of the wait lands on the queue, not on this deployment. The replicas that kept their quota hold accelerators while the set cannot serve in full, and Kueue accounts that quota as used: the next workload drawing from the same ClusterQueue is refused with insufficient unused quota. A deployment preempted in part starves whatever is queued behind it.

What to do about it:

• Add capacity. The reclaimed replicas are admitted again on their own once capacity returns. The state self-heals but is UNBOUNDED: no timeout, no give-up, nothing makes the holder yield, so capacity that never comes back leaves it standing.
• Delete and recreate the deployment. Deleting is the other release for the quota the kept replicas hold.
• Move the roles apart. Each role draws from the ClusterQueue its own instanceType resolves to; giving the roles different types keeps one preemption from reclaiming both halves out of the same queue.

CacheAttached

Whether the cache is observed to be in effect, which is a different question from whether it was configured. It is judged downstream of the engine and never on a rendered flag or a log line.

NoObservationAvailable is Unknown rather than False because an attached deployment that is simply idle looks exactly like an unread one: no supported engine signals “the connector initialized” before any traffic, so silence is the most common steady state. A connector that cannot come up keeps its replica from ever becoming Ready, which is reported there.

ReplicasUpToDate

Whether the running replicas match the desired explicit spec hash and meet the deployment’s minimum modeldeployment.gpustack.ai/render-version. The explicit spec hash includes the resolved KV cache binding, so a replica carrying the current spec still differs from the desired hash once that connector stops resolving — the state behind RolloutHeldByCache.

A pass answers only for the replicas it read or created from the render it just performed; one that could account for none reports Unknown, rather than a zero indistinguishable from “all up to date”. This happens in ordinary operation — a teardown, and the pass between a rollout’s delete and the create that answers it, both write status this way.

ReplacementInProgress and RolloutInProgress answer opposite questions — whether an edit landed, and why capacity moved when nothing was edited. While a replacement waits, watch the departed Pod: the replacement is created once no Pod for that ordinal reads on the API server.

The answer is not left standing between passes. After a steady deployment it would keep reporting True while every replica is on its way out, because a pass that finds every replica departing accounts for none of them.

RolloutHeldByCache is the condition to read when an edit appears to have no effect; it is the only condition on the object that names a held rollout.

The state does not imply an edit is waiting. During an outage the render carries no connector, so every attached replica differs from it whether or not anyone changed the deployment, and an ordinary store blink puts every deployment on the pool into this state. The message names what would be delayed, never that something is.

Not every change waits: a replicas change proceeds during an outage — the ordinals it adds do not exist yet and are created without a connector, and the ordinals it sheds leave on the ordinary path. An edit that changes a running replica’s rendered Pod is what waits. Scaling out does not release it: replicas added during the outage turn over once more when the store returns, and the running replicas still wait. The store’s own recovery bounds the wait.

The stall is deliberate: the alternative — recreating every replica whose render lost its connector — deletes every replica of every deployment on a pool for a few seconds of store unavailability. Measured, a withheld edit lands within seconds of the store returning.

RoleKindsReady

Whether every role kind the deployment declares has at least one ready replica. It is deliberately not replica completeness: “every role has all the replicas it asked for” is what phase answers, by summing every role’s counts before judging.

A deployment whose roles all share one kind (the configuration used when no role names a kind, since it defaults to Server) reports True as soon as any replica is ready. That is not a blind spot. The degradation is real and phase reports it as Degraded; this condition answers a different question and is silent on that configuration by design.

On an encode, prefill and decode deployment, RoleKindsReady is True only when Encode, Prefill and Decode each have at least one ready replica.

A role that took over its command line counts toward its kind like any other. Such a role gets no readiness probe, so its Ready says its containers started rather than that anything answered on the serving path. status.roles[].unmanaged is published per role for a reader who needs the stronger reading.

Read this condition alongside phase: it reports readiness by role kind, while phase reports replica completeness. Neither guarantees that an inference request succeeds; that also depends on the engine and Service connectivity.

KVEventsPublishing

Whether every role that produces cache blocks has publisher configuration in its rendered Pods. It reports configuration, not live traffic: a configured publisher that later crashes remains True. Encode holds no KV cache and produces no cache blocks, so it is excluded.

RouterReady

Whether the managed router rendered and has a ready replica. A render refusal is reported here rather than returned as an error, so the rest of the status (phase, the other conditions, the role counts) keeps computing with the cause named.

EndpointEligibility

Whether the deployment’s endpoints have been qualified at all. It answers for the reading, not for the endpoints: until endpoint qualification exists it is Unknown/NotObserved, because a False would say the operator disqualified endpoints it saw, and no pass that reports this has seen any.

ElasticResize

Whether an elastic scale stands blocked or terminally refused. It is written only for a deployment with an elasticEp role; Elastic EP owns the resize lifecycle it reports on.

A refusal still inside its retry bound stays True: it is not terminal yet. WorldUnprovable levels back to True on its own once a later pass proves the live world again.

Scaler annotations

Two annotations on the ModelDeployment — modeldeployment.gpustack.ai/scaler-binding and modeldeployment.gpustack.ai/scaler-decision — record which ModelDeploymentScaler owns the replica count and which decision last changed it. They sit in metadata.annotations, not in status, because the single-writer fence is enforced at admission and a status field cannot refuse a write.

Binding fence owns the payload of both annotations, the admission rules for bound and unbound deployments, and the escape hatch when a scaler is gone; GitOps sync covers how a GitOps agent ignores the fields the scaler owns.

The scaler’s own status reports what it observed, what it recommends and what blocks it; Scaler status owns that contract.

Group status

status.roles[].groups[] describes each serving group the operator supervises. A group is the set of Pods that form one logical serving replica of a role, and its identity is the role, the replica ordinal and the current sorted member UID set. A replaced member is a different set, so a previous qualification does not apply to the new one.

Each member entry carries the member Pod’s name and uid, the seat it claims, and its ready value: True, False or Unknown. A member nobody has reported on is Unknown, not False.

The group entry answers what the role’s counts cannot. endpoints.eligible counts endpoints and endpoints.serving counts a Router’s confirmation; the group entry names the members, the generation those answers were taken against, and the fact that holds the group.

The eight states are the operator’s own.

AwaitingObservation names the one hold the qualification cannot name. A group that passed every leg and waits on a valid Router observation is AwaitingObservation, while a group that passed every leg and is retained across an observation loss is Eligible, because the observation is not what holds it.

The retention is proved by the fresh membership, and that membership has to be complete, unchanged, safe and current. It is read uncached when the collection runs, not taken from the reconciled list, and every member of it has to have reported ready. A member that disappears, is replaced or becomes unready invalidates the retention and the restoration. A later failed read cannot erase a change the collection already proved.

The reason names one fact and never two. When a leg of the whole-group predicate holds the group, the reason names that leg; when the Router’s observation holds it, the reason names the observation. A member fault and a lost Router view are told apart by the reason as well as by the state, and the Router’s own reason never takes the group’s place.

The qualification time is the transition, not the latest pass. A group that stays Eligible keeps the instant it earned, and a group held in the same generation keeps it too, so an observation loss does not erase it. A group that leaves Eligible and comes back records the new instant. A new generation inherits no timestamp: the membership it names was never qualified.

Elastic EP collective

An Elastic EP role publishes one group at ordinal zero, and that entry replaces the replica-derived groups a role of several Pods would otherwise publish. Its members are the role’s current GPU workers, the API/DP master included and the ancillary Ray head excluded. A worker on its way out is not a member, and a collective with no workers publishes no group at all. Each member’s seat records its admission ordinal, not its runtime engine rank.

The collective’s state is Unknown, and the master’s own qualification does not change it. The reason says that the whole elastic collective has not been evaluated as one group. The serving answer carries no count: Unknown with a Router, and NotConfigured without one. There is no qualification time, because an instant belongs to a transition into Eligible and the collective is never eligible.

A count and a group never describe the same membership on this shape. The role’s own counts are its replica counts, computed from the master the elastic path reads, because the master is the replica the Router observes. The group is computed from the whole worker set, because the collective is what a reader asks about when a rank fails.