KV Cache Injection Reference

A KVCachePool is usable by any Pod, not only by workloads this operator renders. A mutating admission webhook watches for one label, reads the KVCachePoolBinding the Pod names, and writes the client configuration its inference engine expects. Nothing else about the Pod changes.

A Pod that does not carry the label is left untouched by this webhook. It is not byte-identical after admission, since the API server still defaults fields such as the service-account volume, and other webhooks may write too.

Contents

• The contract
• Injection per engine
• Refusals and fixes
• Tenant compatibility
• Verify the vLLM file vehicle
• The injection record
• Transport compatibility at binding
• Workload impact
• Engine prefix caching

The contract

A Pod opts in with a label and configures the injection with annotations. The trigger is a label because a webhook’s objectSelector can only match labels; a label value is capped at 63 characters, so everything of unbounded length is an annotation.

For a plain server, one that is not half of a prefill/decode split, leave the role annotation off. Server is not in its value domain and a Pod carrying it is refused, while the same arrangement is spelled Server on ModelDeployment.spec.roles[].kind, which even defaults to it. The value that is correct there turns a Pod away here. An absent annotation renders the read-and-write configuration a shared cache wants.

On an SGLang Pod the role changes nothing. That engine’s store client does not distinguish roles, and the prefill/decode split is rendered only for a ModelDeployment, so a Pod with the annotation and one without get the same injection.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: chat
spec:
  template:
    metadata:
      labels:
        kvcache.gpustack.ai/inject: "true"
      annotations:
        kvcache.gpustack.ai/binding: chat
        kvcache.gpustack.ai/engine: vLLM
    spec:
      containers:
        - name: server
          image: vllm/vllm-openai:v0.29.0
          command:
            - "vllm"
          args:
            - "serve"
            - "--model"
            - "Qwen/Qwen3-8B"

A bare Pod follows the same contract. The label opts it in, and the optional annotations select the role and the container:

apiVersion: v1
kind: Pod
metadata:
  name: bench
  namespace: team-a
  labels:
    kvcache.gpustack.ai/inject: "true"          # the opt-in; a LABEL, not an annotation
  annotations:
    kvcache.gpustack.ai/binding: team-a         # a KVCachePoolBinding in this namespace
    kvcache.gpustack.ai/engine: vLLM
    kvcache.gpustack.ai/role: Decode            # optional: Prefill or Decode
    kvcache.gpustack.ai/container: server       # required with more than one container
spec:
  containers:
    - name: server
      image: vllm/vllm-openai:v0.29.0
      command:
        - "vllm"
      args:
        - "serve"
        - "--model"
        - "Qwen/Qwen2.5-72B-Instruct"

The engine is declared, never guessed from the image. Engines take entirely different flags, and a wrong guess about a renamed or vendored image produces a container that starts normally and caches nothing.

There is deliberately no domain annotation. The reuse domain comes from the Binding, because the Binding is the object that registers a domain and carries its ceiling. Every domain this operator provisions therefore has something accounting for it. kvcache.gpustack.ai/domain is refused rather than ignored, so a manifest written against an escape hatch that does not exist fails where its author can see it.

That is a provisioning contract, not an isolation boundary. What a Binding does and does not bound is stated in Limitations , with issue #168 for the gap. The tenant value is Binding-owned too: on SGLang the webhook replaces every container declaration of MOONCAKE_TENANT_ID with the Binding’s domain.

Injection per engine

Each engine receives its values through a different vehicle, and the choice turns on when the value becomes known, not on which keys the engine accepts. local_hostname is an address, and a mutating webhook runs before a Pod has an IP. vLLM computes its own at startup; SGLang reads it from configuration, and only its environment path consults the process environment, where Kubernetes can supply the Pod’s IP through a fieldRef the kubelet resolves at container start.

Every engine is also given --kv-cache-dtype with the Binding’s dtype, verbatim. Why, and which spellings each engine accepts, is under Engine dtype .

The file is a downwardAPI projection of the Pod’s own kvcache.gpustack.ai/client-config annotation. No ConfigMap is created, so the configuration’s lifetime is exactly the Pod’s.

The values are the same on every engine; only the spellings differ. Both vLLM-family engines read the same file with the same key names. Where they differ is the connector selected alongside it, and which keys their readers know.

Why — every key is written explicitly rather than left to a default, because two of the defaults are GiB of host memory: 4 GiB per key on vLLM, 1 GiB on vLLM-Ascend. An absent global_segment_size makes the engine container a store member of that size; an absent local_buffer_size holds that much staging. Neither appears in the container’s resources, and the symptom is an OOM pointing at no field anybody wrote.

device_name is empty on every path, RDMA and EFA included. Empty means “use every device found”, which is the only value correct for every host in one pool. A device is named per host, mlx5_0 on one and erdma_0 on the next. The documented string auto-discovery is not special-cased anywhere in the client: it is parsed as a filter naming a device no host has.

Nothing injected here grants the Pod a fabric device. These values name a transport; a Pod using this opt-in webhook must request the appropriate device-plugin resource itself. A managed ModelDeployment can instead request a positive spec.roles[].resources.interface count, which renders an RDMA or EFA resource limit on its engine Pod according to the effective transport. The backend’s member Pods configure their own host network and device access separately.

The one exception is the Ascend prefill/decode leg, whose engine Pods mount a host driver tree read-only. See How a pair is wired .

Two observability variables, MC_TE_METRIC and MC_STORE_CLIENT_METRIC_BANDWIDTH, are set to 1 when the container does not already set them. A value you set yourself is left alone.

SGLang host-memory tier

An SGLang container also gets --enable-hierarchical-cache, after the injected arguments, unless its own arguments already name that flag. The storage backend hangs off the hierarchical cache’s host tier: naming the backend alone enables storage prefetch over a plain radix cache, and the first request fails with an AttributeError (measured at SGLang v0.5.18).

That tier is a pinned host pool of hicache_ratio (default 2.0) times the device KV pool. SGLang refuses to build it unless the node’s available memory (read host-wide, not from the container’s limit) exceeds a fixed 10 GiB reserve plus the pool, so size the node for it. The container’s memory limit still has to hold the engine and the pool, or the result is an OOM kill.

A ModelDeployment renders the same switch on every SGLang role with a store except a decode half, where SGLang forces its radix cache off and refuses the two together. A decode half gets --disaggregation-decode-retraction-backup cpu_tensor instead: left unset, SGLang infers a host pool for retraction and meets the same check with no store at all. A role’s own extraArgs naming either flag drops the operator’s.

An injected variable overrules one you declared yourself. Injection is opt-in and its opt-out is explicit, so a Pod that asked for it and then declares a Mooncake variable has given two answers to a question the Binding already answered; the injected value is written in place, leaving one entry per name rather than two.

The two observability toggles above are the exception: they change no result, so a value you set is kept.

Two kinds of key are refused at admission instead of overwritten, because overwriting them would not help.

MOONCAKE_CONFIG_PATH and --kv-transfer-config select the mechanism. A second one is an ambiguity nothing reports. SGLANG_HICACHE_MOONCAKE_CONFIG_PATH and --hicache-storage-backend-extra-config select the configuration source, which leaves the injected variables present and unread. Each key, the engine it applies to and why it is refused are under Refusals and fixes .

A flag is refused in every spelling the engine’s own parser reads as it (a unique prefix, and on the vLLM family an underscored or dotted form) by the rule under Operator-owned keys .

This applies only to env: a value supplied through envFrom is invisible to the check and will be overwritten with no symptom, so declare Mooncake variables in env.

Refusals and fixes

The webhook fails closed and refuses rather than guessing, because every case below produces a container that starts normally and does not use the cache. That result is invisible from outside the Pod.

Why the command warning — putting the launch arguments in command overrides the image’s ENTRYPOINT as well, and on this project’s accelerator images that entrypoint initializes the vendor runtime. The resulting failure is further from its cause than the discarded CMD the refusal exists to prevent.

The script-specific refusal keys off the .sh suffix, which is a convention rather than a guarantee. The general launch check also refuses a suffix-less wrapper named entrypoint or run unless its author declares that it forwards appended arguments. Admission cannot open the file, so the declaration is the only way to admit that uncertainty.

A key that selects where the engine reads its store configuration from is refused, per engine: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH and --hicache-storage-backend-extra-config on SGLang, MOONCAKE_CONFIG_PATH on vLLM. None of them collides with anything this operator writes, and that is what makes them worth refusing: the engine selects one source out of three, so either key leaves every injected variable present on the Pod and read by nothing.

The same key on an engine that does not read it is not refused. SGLang’s variable means nothing to vLLM, and a refusal there would have nothing behind it.

To take future Pods back over, set kvcache.gpustack.ai/inject: "false" on the workload’s Pod template, or drop the label there. It does not undo an existing Pod: the injected args, env and volume stay, and most of a running Pod’s spec is immutable. The change takes effect when the workload rolls.

The label and the two written annotations are frozen once the Pod exists, which is why the paragraph above says template. Editing kvcache.gpustack.ai/inject, .../injected or .../client-config on a live Pod is refused, and so is removing one. Injection runs at CREATE and only there, so a label added afterwards would leave a Pod carrying the label that says it uses a cache and none of the configuration; an edited injected would be a record of a decision nobody made; and an edited client-config would swap the master address, transport and segment sizes under a running container, since the file at /etc/gpustack/kvcache/mooncake.json is a downwardAPI projection of that annotation. Every other metadata edit is left alone, finalizers included.

The freeze covers a terminating Pod too, which is where it matters most: a Pod keeps serving through its termination grace period, and the kubelet keeps reprojecting that file, so a master address swapped after the delete call would still reach the running container.

That guard is a validating webhook on UPDATE with failurePolicy: Ignore, so it does not hold while the webhook is unreachable. The direction is deliberate: it keeps a record honest, and it must never be the reason a live Pod cannot be updated or finished. Under Fail, an unreachable webhook would block finalizer removal on every opted-in Pod, and a pool’s own teardown waits behind exactly that. Clearing a finalizer touches none of the three keys, so it is admitted whether the webhook is reachable or not.

Tenant compatibility

Use an engine image that reads and forwards the injected tenant when the Binding’s domain must isolate cache reuse. The webhook always writes a non-empty domain and never inspects or rejects the image version. Verify the image you deploy: the vLLM family must consume tenant_id from the projected file, while SGLang must consume MOONCAKE_TENANT_ID from its environment.

Keep the Mooncake library bundled by the engine unless that engine documents another compatible combination. A mismatched library may reject the tenant even when the engine reads it.

An older engine is allowed. It may ignore the injected value and use Mooncake’s literal default tenant instead, without any admission error. Such a pool needs a KVCachePoolBinding whose domain is default, which is also what a Binding that leaves spec.domain.name out registers; its quota is then shared by every client that falls back to that tenant.

tenantInjected records only that the webhook wrote the value. It does not prove the image read it or that isolation took effect.

Verify the vLLM file vehicle

On an older vLLM the file is projected, mounted, and read by nothing. No error is logged, because no code looks for it.

Some vLLM images have no reader for MOONCAKE_CONFIG_PATH. On those images the projected file is inert.

This is not validated, and it cannot be: admission never inspects the container image, so the webhook does not know which build will run. The check is yours, against the image you are actually running:

$ kubectl exec chat-0 -- python3 -c \
    "import vllm.distributed.kv_transfer.kv_connector.v1.mooncake.store.worker; print('reader present')"

An ImportError means the injection is inert on that image. Note the symptom is different from the one in the previous section, and calls for a different fix: an unregistered tenant fails loudly with TENANT_NOT_REGISTERED on every write, while a missing reader fails silently. The workload runs correctly, just with no cache at all.

The injection record

An injected Pod carries kvcache.gpustack.ai/injected, a JSON object recording what was decided:

$ kubectl get pod chat-0 -o jsonpath='{.metadata.annotations.kvcache\.gpustack\.ai/injected}'
{"binding":"chat","engine":"vllm","vehicle":"file","domain":"team-a-chat",
 "tenantInjected":true,"launchProgram":"vllm","launchArgsForwarded":false}

Pods admitted by an older operator may also carry engineVersion. New records omit it because it described the upstream source used when the injector was written, not the image in the Pod. Readers must ignore that legacy field and must not infer image compatibility from it.

vehicle is on the record so a reader can tell which shape was written without decoding the container: "file" means a projected configuration document plus its volume, "environment" means variables alone.

Transport compatibility at binding

An engine configured with one transport cannot safely bind to a pool whose member groups offer different protocols. The client reads each target segment’s protocol; a block on a group using a transport the engine did not install fails with NotSupportedTransport. Pod injection refuses that binding and names every effective offer. Make the groups use one protocol or choose another pool; the mixed pool itself remains valid for consumers that can use it.

A pool whose groups offer no ascend transport makes a vLLM-Ascend container fail to start, and the injection is what triggers it. That engine accepts one transport and raises on the rest:

NotImplementedError: MooncakeBackend does not support protocol 'tcp'.

The engine’s own file reader defaults protocol to ascend, so a file that said nothing would have worked. This project writes the resolved pool transport explicitly on every path; that value overwrites the engine default.

It is refused, not left to the container. The shared transport check rejects an injected Pod at admission. A ModelDeployment checks a new binding in its own validating webhook, since its controller path does not use Pod injection.

The transport has two spellings and the message uses both. What the pool offers and what the engine accepts are reported as the artifact spells them, because that is the value the container was handed: tcp against ascend. The value to set is the API’s, CANN, because spec.transport.protocol is a case-sensitive enum.

The failing backend is not one somebody misconfigured. spec.transport.protocol defaults to Auto, which resolves to the store’s tcp. A backend left entirely at its defaults is precisely the one this engine cannot use. Pair vLLM-Ascend with a pool that offers CANN: declared on the backend’s spec.transport.protocol, or on one member group’s transport.protocol when only one group serves the fabric.

A backend may also mix vendors across member groups (one group offering rdma to NVIDIA nodes, another ascend to Ascend nodes) through the same per-group transport.protocol override. Admission imposes no rule on that combination; the per-engine check above is untouched. Whether cross-vendor sharing then works is a property of the engine’s cache key, not of the transport:

“Cross-vendor sharing is meaningless” is a statement about vLLM only. For SGLang the one remaining precondition is that both sides lay a block’s payload bytes out identically, and that has never been measured: the experiment is one model, one page_size and one TP/PP shape, SGLang on NVIDIA against SGLang on Ascend pointed at one pool, checking whether the second side hits the first’s entries and reads back the correct bytes.

The transport is not the limit: the HIXL wiki documents Mooncake’s rdma transport moving buffers directly between an NVIDIA GPU and an Ascend NPU (Mooncake NPU guide, appendix 2 ).

The operator’s applyMemberFabric grants host network and device access to backend member groups only for rdma and efa. An ascend group receives none of those grants. This rendering alone does not establish whether that member starts or can transfer bytes on a particular Ascend node; verify both before relying on cross-vendor sharing.

Why a page instead of an admission rule — a rule becomes API semantics, and an upstream improvement on either side would then force an incompatible removal; a documented note only gets edited. Treat this section as temporary: it records where the engines and this operator stand, and upstream work on either side can obsolete it.

Workload impact

Joining a pool changes three things about a Pod that are easy to file as bugs.

Host memory the Pod never asked for (on the vLLM family only). The injected local_buffer_size is 128 MiB of staging the client registers with the transfer engine. It is charged to the container’s memory and appears in no resources field, so add 128 MiB to both the request and the limit of any vllm or vllm-ascend container you inject into.

Raise both, and not only the limit. The limit alone keeps one container off a cgroup OOM, but the scheduler places Pods by their requests. Raising only the limit therefore lets a node be filled to its request capacity while every injected Pod on it consumes 128 MiB more than that arithmetic accounted for. The result is node memory pressure and kubelet eviction, on a node whose bookkeeping says it is within budget.

SGLang is not given one: the injection writes no local_buffer_size in any spelling, and that engine uses a 16 MiB default. Budget those 16 MiB the same way, in the request as well as the limit.

Why that number, and why it is written at all — it is the value the store’s own reference uses, and it is a constant here rather than a field because it is transfer-layer staging, not a resource grant. What an absent key costs instead is under Injection per engine .

Random ports. The transfer engine binds ports nobody configured. Any NetworkPolicy or port reservation must be written as a range, not a list. The webhook cannot change this and does not try.

A 30-second lease on cached blocks. kv_lease_duration defaults to 30 seconds. It does not expire from long queueing, but it does expire when a Pod’s heartbeat is interrupted (preemption, eviction, restart), and the default failure policy then fails the request outright. Anything that kills an injected Pod destroys cache its peers may be waiting on.

One benign line appears in every client’s startup log and is not an error to chase:

E transfer_metadata.cpp:991] Local segment descriptor not found

The projected client configuration is readable by anyone who can read the Pod. It carries addresses, a protocol and a domain name, and no credential.

Engine prefix caching

Injection does not touch the engine’s own prefix caching, and switching that off is not a workaround for anything. vLLM’s enable_prefix_caching defaults on and reuses blocks in GPU HBM; SGLang’s radix cache does the same.

What the transfer engine registers is that same memory, so the network card can map it. Nothing is copied and nothing is allocated twice, so the two are not competing. Disabling prefix caching buys no memory back and loses the engine’s own reuse.

Host DRAM is where an overlap could happen, and neither engine puts KV there unless asked. vLLM’s --swap-space is deprecated and ignored, and kv_offloading_size defaults off, so on vLLM the only host DRAM this injection adds is the staging buffer above. SGLang’s host tier needs --enable-hierarchical-cache, which this injection renders because the store hangs off that tier. Its host pool is the one host allocation it adds there.

Never set kv_offloading_size on a container this injects into. Setting it makes vLLM overwrite kv_connector with OffloadingConnector, unconditionally and with no conflict check, because kv_connector holds one value. The connector this operator rendered is simply gone.

The container then starts, serves, and uses no shared cache, with nothing in it saying so. The two are alternative second-tier caches (one on the node’s own DRAM, one on the cluster’s pool), so choose between them rather than configuring both.