# Upgrading to an Enforced Binding Dtype

From this release, a Binding's *`spec.domain.dtype`* stops being a declaration and becomes the engine's
`--kv-cache-dtype`, changing objects a running cluster already holds. The rule itself is stated once,
under
[The dtype is handed to the engine](/gpustack-operator/v0.9.2/docs/modules/kv-cache/pool/index.md#engine-dtype).

## Contents

- [Changes on upgrade](#changes-on-upgrade)
- [Step 1: Check every Binding](#step-1-check-every-binding)
- [Step 2: Find roles with flags](#step-2-find-roles-with-flags)
- [Troubleshooting](#troubleshooting)
- [Verification](#verification)

## Changes on upgrade

- Every pool-attached `ModelDeployment` rolls once. Its replicas gain an argument, which moves
  their spec hash, so each is replaced at the deployment's first reconcile after the upgrade.
- The `dtype` reaches the engine verbatim. A spelling that engine rejects (`bf16` on vLLM,
  `fp8` or `float16` on SGLang) makes every new `Pod` fail argument parsing, so each replica the
  rollout replaces stops serving.
- A role's own `--kv-cache-dtype` is refused while it attaches a pool. A deployment already
  stored with one keeps running on its own value, which wins because it comes later on the command
  line, until its next update is refused.
- A `Pod` opting into injection is refused when its container passes the flag itself, so its
  owner reports `FailedCreate` until the flag is removed.
- A new Binding may not declare `auto`. One stored with it keeps working and stays updatable.

## Step 1: Check every Binding

List every Binding's dtype, then compare each with the engines of the deployments that name it:

```bash
kubectl get kvcachepoolbindings -A \
  -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,DTYPE:.spec.domain.dtype
```

Each engine's accepted spellings are listed once, under
[The dtype is handed to the engine](/gpustack-operator/v0.9.2/docs/modules/kv-cache/pool/index.md#engine-dtype). A
Binding whose deployments run both engines needs a value in both rows there. A spelling its engine
does not accept is what the next two sections recover from, and fixing it before upgrading is
cheaper.

## Step 2: Find roles with flags

These deployments keep their own value after the upgrade, and their next edit is refused until the
flag is gone from *`extraArgs`*:

```bash
kubectl get modeldeployments -A -o json | jq -r '.items[]
  | select(.spec.kvCache != null) | . as $md | .spec.roles[]
  | select((.command // []) == [] and ((.extraArgs // []) | any(test("^--kv[-_]cache[-_]dtype"))))
  | "\($md.metadata.namespace)/\($md.metadata.name) role \(.name)"'
```

Remove the flag once the Binding's dtype is the one the role should run.

## Troubleshooting

When new `Pod`s fail argument parsing, the dtype is immutable, so the Binding cannot be corrected in
place. Turn the `Setting` off first, which renders and refuses nothing and recreates the replicas
without the flag:

```bash
kubectl -n gpustack-system patch setting model-deployment-kv-cache-dtype-owned \
  --type merge -p '{"spec":{"value":"false"}}'
```

Then move the workloads to a Binding with a spelling the engine accepts, and turn the `Setting` back on:

1. Create a new Binding on the same pool with the corrected `dtype` and a new *`domain.name`*; the old
   name stays claimed until the old Binding is gone.
2. Recreate each deployment with *`spec.kvCache.poolRef.name`* naming the new Binding. *`kvCache`* is an
   identity field, so it is a new deployment rather than an edit.
3. Delete the old deployments; the old Binding's deletion completes once none of them holds it.
4. Patch the `Setting` back to `"true"`.

The cache under the old domain is not carried over. It was written without a binding dtype, so it
is not worth keeping.

## Verification

Every replica of a pool-attached role carries the Binding's value as two entries of its *`command`*,
because the operator owns that role's whole argv and folds every argument into it:

```bash
kubectl -n <namespace> get pods -l app.kubernetes.io/instance=<deployment> \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].command}{"\n"}{end}' \
  | grep -o -- '--kv-cache-dtype","[^"]*'
```

An injected `Pod` carries the same two entries at the end of its `args`, where the webhook appends.

---

**See also** — [KV Cache Pools and Bindings](/gpustack-operator/v0.9.2/docs/modules/kv-cache/pool/index.md) (the dtype rule) ·
[ModelDeployment](/gpustack-operator/v0.9.2/docs/modules/model-deployment/deployment/index.md#operator-owned-keys) (what the operator owns) ·
[KV Cache Injection](/gpustack-operator/v0.9.2/docs/modules/kv-cache/injection/index.md) (the `Pod` path) ·
[Settings](/gpustack-operator/v0.9.2/docs/reference/settings/index.md) (the escape switch)

**Next** → [Migration Troubleshooting](/gpustack-operator/v0.9.2/docs/operate/migration/troubleshooting/index.md) — when an upgrade wedges for another reason.