Shared Cache Walkthrough

Create a store, a pool, a namespace grant and a workload in that order, replacing the node selector, namespace, instance type and model name in the manifests below. The serving replicas then reuse each other’s prefix KV blocks.

Contents

• Summary
• Environment
• Step 1: Create the store
• Step 2: Create pool and binding
• Step 3: Create the workload
• Step 4: Enable high availability
• Step 5: Add a disk tier
• Step 6: Add a second team
• Limitations
• Troubleshooting

Summary

A shared Mooncake cache pool lets serving replicas reuse each other’s prefix KV blocks and spill blocks across DRAM and local-disk tiers, so a replica answering prompts another replica already processed skips the recompute.

The order is fixed because each object names the one above it, so creating them the other way round leaves references that resolve to nothing. Why the two scopes split this way is on KV Cache Pools and Bindings .

The Binding is the authorization point. A namespace gets access to a store when an administrator creates a Binding in it. A user naming a pool is not a path this API has: poolRef is a same-namespace reference, so the name it accepts is a Binding’s.

Environment

Step 1: Create the store

Replace the kubernetes.io/os: linux selector with the labels of the nodes that should contribute memory.

apiVersion: worker.gpustack.ai/v1
kind: KVCacheBackend
metadata:
  name: mooncake-dram
spec:
  type: Mooncake
  connection:
    managed:
      leader: {}
      members:
        - nodeSelector:
            kubernetes.io/os: linux
          medium: DRAM
          capacityPerMember: 8Gi

spec.image is left unset on purpose: the cluster-wide kv-cache-backend-image Setting supplies this project’s own build, which is the one that can elect a leader later. Naming a published upstream image here works until Step 4 and then does not. See High availability .

capacityPerMember is charged to each member Pod’s host memory request, so it is a claim on the node and not a hint. One member Pod runs per node the selector matches.

leader: {} takes the field defaults, multiTenancy included (it defaults on), so this master keeps a per-tenant quota ledger and Steps 2 and 3 read against it. A backend pinned to a store image from before Mooncake 0.3.12 is the one exception and declares multiTenancy: false out loud. See GPUStack build variants .

Wait for it, then read what it reports:

$ kubectl get kvcb mooncake-dram -w
NAME            TYPE       PHASE   ENDPOINT                                         CAPACITY
mooncake-dram   Mooncake   Ready   mooncake-dram-leader.gpustack-system.svc:50051   16Gi

CAPACITY is read from the store, never derived from the spec. Two nodes at 8Gi show 16Gi because both members registered their segments, so this figure is the evidence that the store is really assembled. A number smaller than expected means a member has not registered yet, whatever the phase says: kubectl describe kvcb mooncake-dram and its MembersMounted condition name which.

Step 2: Create pool and binding

apiVersion: worker.gpustack.ai/v1
kind: KVCachePool
metadata:
  name: shared-dram
spec:
  backends:  # exactly one
    - mooncake-dram
  quota:
    total: 16Gi
---
apiVersion: worker.gpustack.ai/v1
kind: KVCachePoolBinding
metadata:
  name: team-a
  namespace: team-a
spec:
  poolRef:
    name: shared-dram
  quota:
    ceiling: 8Gi
  domain:  # every field here is immutable
    blockSize: 64
    dtype: bfloat16

The domain is the cache’s compatibility key, and it is immutable for the reason it exists. Two workloads sharing a domain share cached blocks, so a domain that could be edited would let a running workload start reading blocks another tokenizer wrote. Pick it to match the model and engine settings the deployments in this namespace will run; a second, different model gets a second Binding.

domain.name is left out, so it is default. The backend from Step 1 runs with multi-tenancy (leader.multiTenancy defaults on), so the engines are handed the tenant id default, which is also the tenant the store uses for a writer that names none. Name each domain once a second Binding shares the master; the name is what keeps the two apart.

A quota ceiling is not a reservation. It is the most this namespace may hold at once, and going over it does not fail a write. See Full-quota behavior .

The ceiling is enforced because the Step-1 leader carries its tenant ledger, which the default gave it. A backend declared leader.multiTenancy: false holds no ledger instead: its pool is admitted with a warning, the Binding reports QuotaGranted=True with reason Unenforced and no EFFECTIVE figure, and every write lands in the store’s default tenant.

A multi-tenant master refuses a tenant name absent from its ledger. An engine that ignores the injected tenant then needs a second Binding whose domain is default, or that leaves name out. See Tenant compatibility .

Step 3: Create the workload

apiVersion: worker.gpustack.ai/v1
kind: ModelDeployment
metadata:
  name: qwen-chat
  namespace: team-a
spec:
  model:
    name: Qwen/Qwen2.5-7B-Instruct
  engine:
    name: vLLM
    # matches the store version used by the default backend image
    image: gpustack/runner:cuda13.0-vllm0.29.0
  kvCache:
    poolRef:
      name: team-a  # KVCachePoolBinding in this namespace
  roles:
    - name: server
      replicas: 2
      instanceType: <one from kubectl get instancetype>
      resources:
        accelerator: "1"

kvCache is optional. Omit it and the deployment runs with no shared cache at all, which is the useful comparison to have run once before attributing anything to the pool.

The engine is configured by injection, not by anything you write here. The operator resolves the Binding, reads the backend’s endpoint and the domain, and injects the store’s environment into every role’s Pod. What is injected per engine, and every refusal, is on KV Cache Injection .

The store version must match the engine’s client. The engine version above decides the client, and Step 1’s unset spec.image leaves the store on the Settings default. A pair on different lines starts healthy and then fails every write, so the version above is a choice. See Version compatibility .

Check the attachment from the deployment’s own status rather than from the Pods:

$ kubectl -n team-a get md qwen-chat -o jsonpath='{.status.conditions[?(@.type=="CacheAttached")]}'

Step 4: Enable high availability

Everything so far runs one leader process that already holds a Kubernetes Lease. An update or a node failure takes the store’s metadata with it. Add standbys with one edit:

 apiVersion: worker.gpustack.ai/v1
 kind: KVCacheBackend
 metadata:
   name: mooncake-dram
 spec:
   type: Mooncake
   connection:
     managed:
-      leader: {}
+      leader:
+        replicas: 3
       members:
         - nodeSelector:
             kubernetes.io/os: linux
           medium: DRAM
           capacityPerMember: 8Gi

The healthy steady state now reads 3 desired / 1 ready, and that is not a broken Deployment. Exactly one leader serves; the other two are standbys, deliberately not ready so the leader Service’s endpoints never include a process that cannot serve. During a healthy failover 2 are briefly ready as the old leader steps down. Both readings are normal.

Scaling from one to three keeps the first leader Pod and the member templates because the Lease election and accounts were already present at one replica. See the leader Deployment for the cost of changing leader.electionBackend.

leader.memberAddressing chooses how members find the master, and defaults to Service. An explicit Lease value uses the member’s API access to read the current holder. See High availability for the measured failover limits.

These conditions apply at one replica too; the phase does not summarize them:

RolloutComplete exists because this workload disables the deployment deadline that would normally answer it: that deadline requires every replica to be available, and only one ever is here.

Failover behavior

A failover keeps nothing in memory. A standby holds no data, so the replica that takes over knows none of the objects held in member memory and rebuilds from member remounts alone; a single leader that restarts does the same. High availability shortens the outage, and every one of those objects misses afterwards. Plan for a cold cache after every failover and every leader restart.

The store’s snapshot is not offered, and its flags are refused in leader.extraArgs, because restoring a snapshot can make the cache serve another key’s bytes instead of a miss. High availability says why.

Step 5: Add a disk tier

A running member group cannot gain a local disk tier. Admission refuses the edit, because the members would have to restart to mount the directory. Declare the tier when you create a backend. This diff turns Step 1’s manifest into a second backend that selects the nodes carrying the disk:

 apiVersion: worker.gpustack.ai/v1
 kind: KVCacheBackend
 metadata:
-  name: mooncake-dram
+  name: mooncake-disk
 spec:
   type: Mooncake
   connection:
     managed:
       leader: {}
       members:
         - nodeSelector:
-            kubernetes.io/os: linux
+            kvcache: "true"
           medium: DRAM
           capacityPerMember: 8Gi
+          localDisks:
+            - path: /var/lib/kvcache
+              capacity: 100Gi

Create the directory on every selected node before you apply the backend. The published store image runs as uid 65532, so give the directory to that uid. The uid depends on the image; see Directory requirements .

$ install -d -o 65532 -g 0 -m 0750 /var/lib/kvcache

A pool names exactly one backend and cannot be re-pointed. Serving workloads from mooncake-disk therefore takes its own KVCachePool and Binding. The tier writes in buckets, so a small cache can show an empty tier while it works; see Bucket writes .

Step 6: Add a second team

Each Binding is one ceiling and one reuse domain. To give a second team cache capacity, create a Binding in its namespace:

apiVersion: worker.gpustack.ai/v1
kind: KVCachePoolBinding
metadata:
  name: team-b
  namespace: team-b
spec:
  poolRef:
    name: shared-dram
  quota:
    ceiling: 8Gi
  domain:
    name: llama-8b-v1
    blockSize: 16
    dtype: bfloat16

A namespace that runs a different dtype needs a second Binding, because a domain is immutable and holds one dtype. Mixing dtypes in one domain can return wrong blocks silently. The Binding below serves an fp8_e4m3 deployment in team-a. Replace the instance type as in Step 3.

apiVersion: worker.gpustack.ai/v1
kind: KVCachePoolBinding
metadata:
  name: qwen-fp8
  namespace: team-a
spec:
  poolRef:
    name: shared-dram
  quota:
    ceiling: 4Gi
  domain:
    name: qwen-fp8-v1
    blockSize: 16
    dtype: fp8_e4m3
---
apiVersion: worker.gpustack.ai/v1
kind: ModelDeployment
metadata:
  name: qwen-chat-fp8
  namespace: team-a
spec:
  model:
    name: Qwen/Qwen2.5-7B-Instruct
  engine:
    name: vLLM
    image: gpustack/runner:cuda13.0-vllm0.29.0
  kvCache:
    poolRef:
      name: qwen-fp8
  roles:
    - name: server
      replicas: 1
      instanceType: <one from kubectl get instancetype>
      resources:
        accelerator: "1"

The operator renders --kv-cache-dtype fp8_e4m3 from the Binding onto every role. A role that sets that flag itself is refused. See Engine dtype .

The three ceilings ask for 20Gi against a pool quota.total of 16Gi. The pool reports QuotaWithinTotal=False with reason Oversubscribed. That is not a fault: a ceiling is an ask, and the store recuts each grant against the capacity it can serve, in proportion to what each Binding asked. status.effectiveQuota on each Binding shows the grant. See Ceiling and grant .

Limitations

• Domain reuse boundaries and tenant-ledger requirements are covered in Limitations and Tenant compatibility .
• Binding deletion holds (Releasable=False, including HeldByWorkloads) are covered in Operating notes .
• Engine-version and blockSize / dtype compatibility across workloads sharing a domain are covered in Tenant compatibility .

Troubleshooting