KV Cache Local Disk Tier

A member group may declare a directory on each of its nodes, which configures the store client’s offload keys to point at it. Declaring the entry is the whole switch: the leader has no field of its own and takes its offload flags from this list’s presence. The list holds one entry, keyed by path (status.capacity reports one pair of figures for the whole backend, so a second tier could not be attributed).

spec:
  connection:
    managed:
      members:
        - nodeSelector:
            kvcache: "true"
          medium: DRAM
          capacityPerMember: 500Gi
          localDisks:                    # declaring an entry is what turns the tier on
            - path: /var/lib/kvcache
              capacity: 4Ti              # optional; unset means the store's own ceiling
              keyLimit: 10000000         # optional; the same, on the key count
              eviction:                  # optional; unset means the store's own behaviour
                enabled: true            # default; false fills the tier and then stops writing
                policy: LRU              # FIFO | LRU; unset means the store's own, which is FIFO
                watermark:               # optional; percentages of capacity
                  high: 90
                  low: 80

The mode is not selectable: the tier always runs deferred. The leader renders -enable_offload=true -offload_on_evict=true together, and only when a group declares a tier. The flags come from the declaration rather than any field, and leader.extraArgs refuses both keys so the leader cannot drift out of step with the members’ declaration.

The store keeps an object queued for offload in memory until the disk write lands, which is the branch -offload_on_evict selects. The write-through alternative, where an eviction can destroy the sole replica of an object whose bucket has not been flushed, is not renderable by any shape of this object, so there is nothing to opt into.

A tier is a layer on a member group, never a group of its own. See Connection and medium for why the shape has to be this way, and KV Cache on Disk-Heavy Nodes for what to write on a node that is mostly disk.

Every example on this page is a DRAM group, and that is not incidental. What is and is not measured about pairing the tier with a VRAM group is stated once, beside the medium it belongs to, at VRAM group accelerator access . Read it before writing one.

Contents

• Bucket writes
• Eviction
• Directory requirements
• Cleanup on deletion
• Unaccounted disk usage

Bucket writes

The store does not write an offloaded object on its own. It assembles objects into a bucket and writes nothing until that bucket is full (by bytes or by object count), and what is short of the threshold is carried to the next attempt, indefinitely.

The store’s own thresholds are 256 MB and 500 objects, and a backend that never reaches either has a tier that holds nothing while looking healthy. The member Pods are Ready, the leader logs the mount and reports objects deferred for offload, and status.capacity shows the size the tier declared, because that figure is capacity, not usage (see Status and conditions ).

This is what issue #200 tracks.

The operator renders a smaller pair of its own, so a modest backend closes buckets. They are not in the API and extraEnv refuses them: moving them is a tuning decision that would need a field, not an escape hatch that silently defines the same variable twice.

Three bounds follow from the bucket being the unit, all enforced at apply time, all naming the same figure:

• capacityPerMember must hold one bucket on a group that declares a tier, because the bytes are held in the memory segment until the bucket is complete.
• localDisks[].capacity and localDisks[].keyLimit, when set, must hold one bucket and one bucket’s worth of keys.

Why a refusal rather than a default — the store stops taking offload work as soon as one more bucket would not fit under a declared ceiling, and it reports that by doing nothing. A tier below any of these is a configuration that cannot work under any workload.

Read master_allocated_file_size_bytes to see what the tier holds:

$ kubectl exec -n gpustack-system deploy/<backend>-leader -- \
    python3 -c "import urllib.request;print([l for l in \
    urllib.request.urlopen('http://127.0.0.1:9003/metrics').read().decode().splitlines() \
    if l.startswith('master_allocated_file_size_bytes')])"

master_allocated_file_size_bytes is bytes actually written to the tier. It moves when a bucket is closed and written, and it sums the whole backend.

A 0 is not on its own a verdict, and the section above is why. A backend that has not filled a bucket has written nothing legitimately, and that is indistinguishable at this gauge from a tier that cannot write at all.

The threshold that decides which reading applies is per member, while the gauge is not. Each member client fills its own bucket, so traffic spread over four members has to reach four buckets' worth before every one of them closes (the figure the group’s capacityPerMember floor is taken from). Judge the two together: compare the data each member has been offered since the tier started with the gauge.

The gauge updates when a bucket closes. An immediate read after a put can still show 0 on a working tier; it catches up within tens of seconds. If each member has received a bucket’s worth of data and the gauge remains 0 past that delay, the tier is not taking writes.

Eviction

localDisks[].eviction is one choice with two outcomes, not a set of knobs:

• policy is the order entries leave in: FIFO drops the oldest written, LRU the least recently read. Left unset, the store’s own applies, which is FIFO.
• watermark is when eviction runs: it starts once the tier passes high and stops once it is back under low, both percentages of capacity. low must be below high.

Four combinations are refused at apply time, each because the store would accept them and then not act on them:

Why the enum has only two values — the store maps a policy string it does not recognise onto no eviction at all, with no error, no warning and no failure to start. A neutral two-value enum is what keeps a typo from being a silently disabled cache. Turning eviction off is enabled: false rather than a third enum value, so there is exactly one way to say it.

Settings this API does not name are reachable through members[].extraEnv, which extraArgs cannot reach: its entries render as the entrypoint’s config-key overrides, and this family is read from the environment only. A name the operator already renders is refused there, because Kubernetes takes a container carrying one name twice and leaves the winner to the runtime. Every value is world-readable, on the cluster-scoped object and again in the Pod. No credential belongs there.

Directory requirements

localDisks[].path is mounted with type: Directory, so the directory must already exist on every node the group selects. This is deliberate: a directory the kubelet creates is owned by root with mode 0755, while the published store image runs as uid 65532, and the member then starts and cannot write to it. fsGroup does not help, because it does not apply to hostPath volumes.

The uid depends on the image, since members[].image may put a different vendor’s build on a group. Read it off the image you are using:

$ docker run --rm --entrypoint id <your-member-image>
uid=65532 gid=0(root) groups=0(root)

Then create the directory on each node with that uid:

$ install -d -o 65532 -g 0 -m 0750 /var/lib/kvcache

There is no switch that makes the operator create the directory for you. An init container would have to name a single uid for every group, and the uid is a property of whichever image a group runs, so the command above is the preparation.

That objection is about ownership, so it reaches creating the directory and not emptying it: removing content needs no uid. Emptying is a switch, and it is below .

Five rules the path has to satisfy, all enforced at apply time:

• It must be absolute.
• It must not be the root directory.
• It may not overlap /dev/infiniband (equal to it, inside it, or containing it). A sibling such as /dev/infiniband-cache is fine.
• It may not contain a .. component.
• It may not begin or end with whitespace, spaces and tabs alike.

Why — the root directory would mount the node’s whole filesystem into a third-party container. A host fabric’s device plugin injects the granted device nodes under /dev/infiniband in this same container; a tier on that path would shadow them or be shadowed, which nothing on the object would record. That rule holds whatever spec.transport.protocol says today, because the field is editable. The .. rule mirrors the store’s own, which refuses such a path before checking whether the directory exists. The whitespace rule exists because the path is mounted exactly as written, so a trailing space produces a different directory than the one an operator read on the screen.

The tier is frozen once a group has it: it cannot be added to a running group, removed from one, or moved to another path. Members would have to restart to mount the directory, and whatever they already wrote would stay on their nodes with nothing addressing it. What the tier may hold is not frozen: capacity, keyLimit and eviction each move either way, re-rendering the variables in the table above, and the tier’s contents survive the restart that follows.

There is exactly one exit, and it needs the tier on the last group. The immutability rule reads the list by position (the position identifies a group everywhere else, so it identifies the tier here too), and an update that drops the last group takes its tier with it.

Dropping an earlier group is refused: every position after it would be compared against a different group’s spec, which is also why reordering members is refused. The refusal message is unambiguous about which group it means.

A backend whose only group carries a tier has no exit but deletion. members requires at least one entry, so that group cannot be removed, and replacing it in place is the forbidden edit. Deleting the KVCacheBackend is what is left, and it takes the leader and every member with it. Put a tier on the last group if you want to be able to take it off.

Three failure modes, all loud:

The last one never reaches a Ready state (the member’s REST port opens only after the store mounts, so the readiness probe never passes), and MembersMounted reports the shortfall rather than the backend looking healthy.

Cleanup on deletion

Deleting a KVCacheBackend leaves the tier directory exactly as it was. Set members[].localDisks[].cleanAfterDelete: true and the operator empties it as part of the deletion, on every node that group’s nodeSelector picks at the moment you delete it.

      members:
        - medium: DRAM
          capacityPerMember: 64Gi
          localDisks:
            - path: /var/lib/kvcache
              capacity: 512Gi
              cleanAfterDelete: true

It defaults to false, which is how every release before it behaved. What is on that disk is yours; removing it is not a decision this operator takes on your behalf. Turned on, the decision is still yours and the operator only carries it out.

The content goes, the directory stays. You created it, gave it an owner, and may have mounted a filesystem there.

Declaring a tier requires a shell in the group’s image. An init container surveys the directory before the member starts, and it runs sh -c; an image without a shell keeps the member from starting. The store images this project ships have one.

When cleanup is disabled

A later backend pointed at the same path starts on whatever the previous one left. The store claims those buckets, so a key the new backend has written can read back as the old backend’s bytes instead of as a miss, which is the part that makes it hard to spot.

The operator reports this rather than acting on it. A backend whose tier directory is not empty when it first starts records it in status.conditions as TierWasEmpty=False, naming each node and the number of entries found, and emits one warning event beside it. Nothing is removed and nothing is blocked.

$ kubectl get kvcachebackend <name> -o jsonpath='{.status.conditions[?(@.type=="TierWasEmpty")]}'

The condition is written once and then left alone, unlike everything else in that status. After the first write the question stops being answerable: the backend’s own data on that disk is indistinguishable from what it found there, so a member restarting later cannot be told apart from a fresh reuse.

Because that write is permanent, TierWasEmpty=True waits until every node carrying the tier has reported. False does not: one node holding content settles it whatever the others say. So the condition being absent means the question is still open: a member still starting, or one whose survey cannot run at all. Absent is not a clean tier.

Only a backend’s first member Pods are believed. A Pod replaced later (a node reboot, an eviction, an image change) re-runs the same survey against a tier this backend has since filled, and its entries are not evidence about what the backend found. Such readings are ignored, so a backend whose condition was never settled early does not end up accusing a predecessor of its own content.

“Every node” means every node the member DaemonSet wants a Pod on, not every Pod that happens to exist yet. A node still scheduling or still pulling holds the True verdict open rather than being skipped over. A node whose only reading came from a replacement counts as one that has not answered, which is why a backend can end up with no verdict at all.

Cleanup limitations

A node it cannot reach in time keeps its content. The deletion is not held open for it: a deletion waiting on a node that is gone is an object nobody can delete. The node is left as it is and gets a warning event, recorded on the node rather than on the backend, because the backend is about to stop existing and the leftover data is not.

“In time” is five minutes, counted per node from that node’s own cleanup Pod. It is counted neither from the oldest Pod of the pass nor from the deletion: a Pod created on a later pass would otherwise be reported as abandoned with most of its five minutes unspent, and a clock started at the deletion would spend its budget on the teardown wait instead of the cleanup.

A cleanup still running at the deadline is stopped, not left to finish. Emptying a very large tier can outlast the five minutes, and ending it there leaves the directory partly emptied. Leaving it running is worse: it keeps deleting from a path this backend no longer holds, and nothing stops a new backend claiming that directory the moment the deletion completes, so its fresh data would be deleted the same way.

A node the group has stopped selecting is not cleaned, and not reported either. The spec is the only record of which nodes a group covered. Nothing keeps the selector’s history, and the members are gone by the time the cleanup runs, so a node dropped by narrowing nodeSelector, or by removing the localDisks entry, cannot be named, let alone reached. Delete the backend first and edit afterwards; editing first silently takes those nodes out of the cleanup.

A tier with no image to empty it is given up on immediately, with the same event. When neither the member group nor the backend names an image and the operator has no default, there is nothing to run and no clock to start; waiting would hold the backend open for as long as that stays true. The event names the image as the reason rather than the node.

A path another backend’s tier overlaps is skipped, with the same kind of event: a directory nested inside another backend’s tier counts, not only an identical path. Nothing refuses two backends naming one directory, and emptying it for the one being deleted would take the other one’s live data with it. A removal already under way when the path becomes shared is stopped, not left to finish.

It is a check and not a lock, re-run on every pass rather than only before the removal starts. Stopping a removal only cancels the cleanup Pod; the rm keeps running until the check catches it and on through that Pod’s own termination, so everything deleted in that span is lost. That is what the second event reports. Closing the window entirely needs an atomic claim on the path, which this operator does not take.

$ kubectl describe node <node> | grep -E 'KVCacheTierNotCleaned|KVCacheTierSharedPath|KVCacheTierPartlyEmptied'

Unaccounted disk usage

The hostPath is not counted into any resource request, and it cannot be: the kubelet’s ephemeral-storage accounting covers the container filesystem, emptyDir volumes and logs, never a hostPath. A request against it would reserve a figure nothing polices and would keep the member off the very node that has the disk.

Watching that filesystem is yours. localDisks[].capacity renders the store’s own ceiling, which is the only bound on what the tier writes; nothing in Kubernetes will evict or throttle the member when the node’s disk fills.