# Shared Cache Walkthrough Create a store, a pool, a namespace grant and a workload in that order, replacing the node selector, namespace, instance type and model name in the manifests below. The serving replicas then reuse each other's prefix KV blocks. ## Contents - [Summary](#summary) - [Environment](#environment) - [Step 1: Create the store](#step-1-create-the-store) - [Step 2: Create pool and binding](#step-2-create-pool-and-binding) - [Step 3: Create the workload](#step-3-create-the-workload) - [Step 4: Enable high availability](#step-4-enable-high-availability) - [Step 5: Add a disk tier](#step-5-add-a-disk-tier) - [Step 6: Add a second team](#step-6-add-a-second-team) - [Limitations](#limitations) - [Troubleshooting](#troubleshooting) ## Summary A shared Mooncake cache pool lets serving replicas reuse each other's prefix KV blocks and spill blocks across DRAM and local-disk tiers, so a replica answering prompts another replica already processed skips the recompute. | Object | Scope | Creator | Purpose | |---|---|---|---| | `KVCacheBackend` | cluster | administrator | which nodes contribute memory, and how much | | `KVCachePool` | cluster | administrator | how much of that store may be handed out at all | | `KVCachePoolBinding` | namespace | administrator | that THIS namespace may draw on it, and up to what | | `ModelDeployment` | namespace | user | a workload that attaches to the grant | The order is fixed because each object names the one above it, so creating them the other way round leaves references that resolve to nothing. Why the two scopes split this way is on [KV Cache Pools and Bindings](/gpustack-operator/main/docs/modules/kv-cache/pool/index.md#two-kinds-by-scope). **The Binding is the authorization point.** A namespace gets access to a store when an administrator creates a Binding in it. A user naming a pool is not a path this API has: *`poolRef`* is a same-namespace reference, so the name it accepts is a Binding's. ## Environment | Item | Value | |---|---| | Cluster | Kubernetes with GPUStack installed | | Contributing nodes | selected by the backend's *`nodeSelector`*; Step 5's disk tier selects nodes labeled `kvcache: "true"` | | `InstanceType` | one from `kubectl get instancetype` | | Model | `Qwen/Qwen2.5-7B-Instruct` | | Engine | vLLM `gpustack/runner:cuda13.0-vllm0.29.0` | ## Step 1: Create the store Replace the `kubernetes.io/os: linux` selector with the labels of the nodes that should contribute memory. ```yaml apiVersion: worker.gpustack.ai/v1 kind: KVCacheBackend metadata: name: mooncake-dram spec: type: Mooncake connection: managed: leader: {} members: - nodeSelector: kubernetes.io/os: linux medium: DRAM capacityPerMember: 8Gi ``` *`spec.image`* is left unset on purpose: the cluster-wide `kv-cache-backend-image` [Setting](/gpustack-operator/main/docs/reference/settings/index.md) supplies this project's own build, which is the one that can elect a leader later. Naming a published upstream image here works until Step 4 and then does not. See [High availability](/gpustack-operator/main/docs/modules/kv-cache/leader/index.md#high-availability). *`capacityPerMember`* is charged to each member `Pod`'s host memory request, so it is a claim on the node and not a hint. One member `Pod` runs per node the selector matches. `leader: {}` takes the field defaults, *`multiTenancy`* included (it defaults on), so this master keeps a per-tenant quota ledger and Steps 2 and 3 read against it. A backend pinned to a store image from before Mooncake 0.3.12 is the one exception and declares `multiTenancy: false` out loud. See [GPUStack build variants](/gpustack-operator/main/docs/modules/kv-cache/backend/index.md#gpustack-build-variants). Wait for it, then read what it reports: ```console $ kubectl get kvcb mooncake-dram -w NAME TYPE PHASE ENDPOINT CAPACITY mooncake-dram Mooncake Ready mooncake-dram-leader.gpustack-system.svc:50051 16Gi ``` **`CAPACITY` is read from the store, never derived from the spec.** Two nodes at `8Gi` show `16Gi` because both members registered their segments, so this figure is the evidence that the store is really assembled. A number smaller than expected means a member has not registered yet, whatever the phase says: `kubectl describe kvcb mooncake-dram` and its `MembersMounted` condition name which. ## Step 2: Create pool and binding ```yaml apiVersion: worker.gpustack.ai/v1 kind: KVCachePool metadata: name: shared-dram spec: backends: # exactly one - mooncake-dram quota: total: 16Gi --- apiVersion: worker.gpustack.ai/v1 kind: KVCachePoolBinding metadata: name: team-a namespace: team-a spec: poolRef: name: shared-dram quota: ceiling: 8Gi domain: # every field here is immutable blockSize: 64 dtype: bfloat16 ``` **The `domain` is the cache's compatibility key, and it is immutable for the reason it exists.** Two workloads sharing a domain share cached blocks, so a domain that could be edited would let a running workload start reading blocks another tokenizer wrote. Pick it to match the model and engine settings the deployments in this namespace will run; a second, different model gets a second Binding. ***`domain.name`* is left out, so it is `default`.** The backend from Step 1 runs with multi-tenancy (*`leader.multiTenancy`* defaults on), so the engines are handed the tenant id `default`, which is also the tenant the store uses for a writer that names none. Name each domain once a second Binding shares the master; the name is what keeps the two apart. **A quota ceiling is not a reservation.** It is the most this namespace may hold at once, and going over it does not fail a write. See [Full-quota behavior](/gpustack-operator/main/docs/modules/kv-cache/pool/index.md#full-quota-behavior). **The ceiling is enforced because the Step-1 leader carries its tenant ledger, which the default gave it.** A backend declared `leader.multiTenancy: false` holds no ledger instead: its pool is admitted with a warning, the Binding reports `QuotaGranted=True` with reason `Unenforced` and no `EFFECTIVE` figure, and every write lands in the store's default tenant. **A multi-tenant master refuses a tenant name absent from its ledger.** An engine that ignores the injected tenant then needs a second Binding whose domain is `default`, or that leaves `name` out. See [Tenant compatibility](/gpustack-operator/main/docs/modules/kv-cache/injection/index.md#tenant-compatibility). ## Step 3: Create the workload ```yaml apiVersion: worker.gpustack.ai/v1 kind: ModelDeployment metadata: name: qwen-chat namespace: team-a spec: model: name: Qwen/Qwen2.5-7B-Instruct engine: name: vLLM # matches the store version used by the default backend image image: gpustack/runner:cuda13.0-vllm0.29.0 kvCache: poolRef: name: team-a # KVCachePoolBinding in this namespace roles: - name: server replicas: 2 instanceType: resources: accelerator: "1" ``` *`kvCache`* is optional. Omit it and the deployment runs with no shared cache at all, which is the useful comparison to have run once before attributing anything to the pool. **The engine is configured by injection, not by anything you write here.** The operator resolves the Binding, reads the backend's endpoint and the domain, and injects the store's environment into every role's `Pod`. What is injected per engine, and every refusal, is on [KV Cache Injection](/gpustack-operator/main/docs/modules/kv-cache/injection/index.md). **The store version must match the engine's client.** The engine version above decides the client, and Step 1's unset *`spec.image`* leaves the store on the `Setting`s default. A pair on different lines starts healthy and then fails every write, so the version above is a choice. See [Version compatibility](/gpustack-operator/main/docs/modules/kv-cache/backend/index.md#version-compatibility). Check the attachment from the deployment's own status rather than from the `Pod`s: ```console $ kubectl -n team-a get md qwen-chat -o jsonpath='{.status.conditions[?(@.type=="CacheAttached")]}' ``` ## Step 4: Enable high availability Everything so far runs one leader process that already holds a Kubernetes `Lease`. An update or a node failure takes the store's metadata with it. Add standbys with one edit: ```diff apiVersion: worker.gpustack.ai/v1 kind: KVCacheBackend metadata: name: mooncake-dram spec: type: Mooncake connection: managed: - leader: {} + leader: + replicas: 3 members: - nodeSelector: kubernetes.io/os: linux medium: DRAM capacityPerMember: 8Gi ``` **The healthy steady state now reads `3 desired / 1 ready`, and that is not a broken `Deployment`.** Exactly one leader serves; the other two are standbys, deliberately not ready so the leader `Service`'s endpoints never include a process that cannot serve. During a healthy failover `2` are briefly ready as the old leader steps down. Both readings are normal. **Scaling from one to three keeps the first leader `Pod` and the member templates** because the `Lease` election and accounts were already present at one replica. See [the leader Deployment](/gpustack-operator/main/docs/modules/kv-cache/leader/index.md#health-checks) for the cost of changing *`leader.electionBackend`*. ***`leader.memberAddressing`* chooses how members find the master**, and defaults to `Service`. An explicit `Lease` value uses the member's API access to read the current holder. See [High availability](/gpustack-operator/main/docs/modules/kv-cache/leader/index.md#high-availability) for the measured failover limits. These conditions apply at one replica too; the phase does not summarize them: | Condition | True | False | |---|---|---| | `ElectionObserved` | the `Lease` names a holder, so an election happened | a ready leader and a holderless `Lease` — the image, the role binding, or a first campaign still running | | `RolloutComplete` | every replica runs the current template and none of the previous one is left | a rollout in flight, or one that has stalled | `RolloutComplete` exists because this workload disables the deployment deadline that would normally answer it: that deadline requires every replica to be available, and only one ever is here. ### Failover behavior **A failover keeps nothing in memory.** A standby holds no data, so the replica that takes over knows none of the objects held in member memory and rebuilds from member remounts alone; a single leader that restarts does the same. High availability shortens the outage, and every one of those objects misses afterwards. Plan for a cold cache after every failover and every leader restart. **The store's snapshot is not offered, and its flags are refused in *`leader.extraArgs`***, because restoring a snapshot can make the cache serve another key's bytes instead of a miss. [High availability](/gpustack-operator/main/docs/modules/kv-cache/leader/index.md#high-availability) says why. ## Step 5: Add a disk tier A running member group cannot gain a local disk tier. Admission refuses the edit, because the members would have to restart to mount the directory. Declare the tier when you create a backend. This diff turns Step 1's manifest into a second backend that selects the nodes carrying the disk: ```diff apiVersion: worker.gpustack.ai/v1 kind: KVCacheBackend metadata: - name: mooncake-dram + name: mooncake-disk spec: type: Mooncake connection: managed: leader: {} members: - nodeSelector: - kubernetes.io/os: linux + kvcache: "true" medium: DRAM capacityPerMember: 8Gi + localDisks: + - path: /var/lib/kvcache + capacity: 100Gi ``` Create the directory on every selected node before you apply the backend. The published store image runs as uid 65532, so give the directory to that uid. The uid depends on the image; see [Directory requirements](/gpustack-operator/main/docs/modules/kv-cache/local-disk-tier/index.md#directory-requirements). ```console $ install -d -o 65532 -g 0 -m 0750 /var/lib/kvcache ``` A pool names exactly one backend and cannot be re-pointed. Serving workloads from `mooncake-disk` therefore takes its own `KVCachePool` and Binding. The tier writes in buckets, so a small cache can show an empty tier while it works; see [Bucket writes](/gpustack-operator/main/docs/modules/kv-cache/local-disk-tier/index.md#bucket-writes). ## Step 6: Add a second team Each Binding is one ceiling and one reuse domain. To give a second team cache capacity, create a Binding in its namespace: ```yaml apiVersion: worker.gpustack.ai/v1 kind: KVCachePoolBinding metadata: name: team-b namespace: team-b spec: poolRef: name: shared-dram quota: ceiling: 8Gi domain: name: llama-8b-v1 blockSize: 16 dtype: bfloat16 ``` A namespace that runs a different dtype needs a second Binding, because a domain is immutable and holds one dtype. Mixing dtypes in one domain can return wrong blocks silently. The Binding below serves an `fp8_e4m3` deployment in `team-a`. Replace the instance type as in Step 3. ```yaml apiVersion: worker.gpustack.ai/v1 kind: KVCachePoolBinding metadata: name: qwen-fp8 namespace: team-a spec: poolRef: name: shared-dram quota: ceiling: 4Gi domain: name: qwen-fp8-v1 blockSize: 16 dtype: fp8_e4m3 --- apiVersion: worker.gpustack.ai/v1 kind: ModelDeployment metadata: name: qwen-chat-fp8 namespace: team-a spec: model: name: Qwen/Qwen2.5-7B-Instruct engine: name: vLLM image: gpustack/runner:cuda13.0-vllm0.29.0 kvCache: poolRef: name: qwen-fp8 roles: - name: server replicas: 1 instanceType: resources: accelerator: "1" ``` The operator renders `--kv-cache-dtype fp8_e4m3` from the Binding onto every role. A role that sets that flag itself is refused. See [Engine dtype](/gpustack-operator/main/docs/modules/kv-cache/pool/index.md#engine-dtype). **The three ceilings ask for 20Gi against a pool *`quota.total`* of 16Gi.** The pool reports `QuotaWithinTotal=False` with reason `Oversubscribed`. That is not a fault: a ceiling is an ask, and the store recuts each grant against the capacity it can serve, in proportion to what each Binding asked. *`status.effectiveQuota`* on each Binding shows the grant. See [Ceiling and grant](/gpustack-operator/main/docs/modules/kv-cache/pool/index.md#ceiling-and-grant). ## Limitations - Domain reuse boundaries and tenant-ledger requirements are covered in [Limitations](/gpustack-operator/main/docs/modules/kv-cache/pool/index.md#limitations) and [Tenant compatibility](/gpustack-operator/main/docs/modules/kv-cache/injection/index.md#tenant-compatibility). - Binding deletion holds (`Releasable=False`, including `HeldByWorkloads`) are covered in [Operating notes](/gpustack-operator/main/docs/modules/kv-cache/pool/index.md#operating-notes). - Engine-version and *`blockSize`* / `dtype` compatibility across workloads sharing a domain are covered in [Tenant compatibility](/gpustack-operator/main/docs/modules/kv-cache/injection/index.md#tenant-compatibility). ## Troubleshooting | Symptom | Signal | Check & Mitigation | |---|---|---| | `KVCacheBackend` stuck in `Provisioning` or `Error` | Condition: `LeaderAvailable=False` | Leader `Deployment` unready, image missing election backend, or external endpoint address unreachable | | `KVCacheBackend` shows `Degraded` | Condition: `MembersMounted` shortfall (`SegmentsShort`, `NoSegments`) | `DaemonSet` member `Pod`s failing to mount or allocate memory/disk segments; check node capacity and daemon logs | | `KVCachePool` shows `Error` | Condition: `BackendResolved=False` / `CapacityAllocatable=False` | Referenced backend does not exist or has zero allocatable capacity | | `KVCachePool` `QuotaLedgerAvailable=False` | Reason: `MultiTenancyDisabled` | Backend explicitly set `multiTenancy: false`; ceiling quotas are intentionally not enforced | | `KVCachePoolBinding` `effectiveQuota: 0` | Condition: `QuotaGranted=False/ZeroGranted` | The master grants this domain nothing. Read the pool's `CapacityAllocatable`: `NothingToAllocate` means no member has mounted yet, and a restarted master can take roughly 30 s to remount its segments. Otherwise a proportional recut left this domain no share | | Pool reports oversubscribed | Condition: `QuotaWithinTotal=False/Oversubscribed` | The Bindings' ceilings sum past the pool's *`quota.total`*. This is not a fault: grants are recut in proportion and shown in each *`status.effectiveQuota`*. Lower a ceiling or raise *`quota.total`* for full grants | | Binding refused at apply | Reason: `Duplicate` on *`spec.domain.name`* — the domain is already registered on the same master | Another Binding holds the name; an omitted name is `default`. Give each Binding a unique *`domain.name`* | | Two Bindings report one domain | Condition: `DomainExclusive=False/DomainClaimedByMultipleBindings` | Two creates raced past admission; delete one of them | | Member `Pod` stuck | Event: `FailedMount ... hostPath type check failed` | The tier directory does not exist on that node; create it as in Step 5 | | Member `Pod` runs but never becomes Ready | Condition: `MembersMounted=False` (`NoSegments`, or `MemberCrashLooping` if the container exits), or a `CAPACITY` below the expected figure | The directory exists but the image's user cannot write it; the container log says `no write permission on directory`. Run `install -d -o 65532 -g 0 -m 0750 ` for the default image. See [Directory requirements](/gpustack-operator/main/docs/modules/kv-cache/local-disk-tier/index.md#directory-requirements) | | Member init container restarts | Condition: `MembersMounted=False/MemberInitCrashLooping` | The directory-survey init container restarted; the condition message gives the restart count, exit code and termination message. The survey runs `sh -c`, so the member image needs a shell. An unwritable directory does not cause this | | Local disk tier reports preexisting content | Condition: `TierWasEmpty=False/PreexistingContent` | The directory held files from an earlier backend. Nothing was removed; empty it yourself. `cleanAfterDelete: true` empties it only when this backend is deleted | | HA Leader shows `3 desired / 1 ready` | Status: expected steady state | Only elected active leader is Ready; 2 standby leader replicas remain unready until failover | | Deletion of `KVCachePoolBinding` blocked | Condition: `Releasable=False/HeldByWorkloads` | Active `ModelDeployment`s or `Pod`s reference this binding; delete client workloads first | --- **See also** — [KV Cache Backend](/gpustack-operator/main/docs/modules/kv-cache/backend/index.md) (every field of the store, and what status reports) · [KV Cache Election and Failover](/gpustack-operator/main/docs/modules/kv-cache/leader/index.md) (the election, why there is no snapshot, and the member addressing choice in full) · [KV Cache Pools and Bindings](/gpustack-operator/main/docs/modules/kv-cache/pool/index.md) (quota, domains and what a full quota does) · [KV Cache Local Disk Tier](/gpustack-operator/main/docs/modules/kv-cache/local-disk-tier/index.md) (directory, bucket and cleanup rules) · [Model Deployment](/gpustack-operator/main/docs/modules/model-deployment/deployment/index.md) (roles, *prefill*/*decode*, rollout) · [Prefill and Decode](/gpustack-operator/main/docs/modules/model-deployment/prefill-decode/index.md) (router, transfer) · [KV Cache Injection](/gpustack-operator/main/docs/modules/kv-cache/injection/index.md) (what a `Pod` actually receives) **Next** → [Model Prefetch Walkthrough](/gpustack-operator/main/docs/walkthroughs/model-delivery/prefetch/index.md) — cache weights on GPU nodes before serving.