GPUStack Operator

Architecture

GPUStack Operator manages accelerator-backed workloads on Kubernetes. It discovers devices and turns node capacity into schedulable pools. GPU Instances and model deployments use those pools. Model delivery resolves and supplies weights. Topology placement supports locality. Remote direct memory access (RDMA) networking supports fast transfers. Shared KV cache stores reusable inference state.

Contents

Components

The gpustack-operator binary provides these subcommands:

Component Deployment Role
worker Control-plane Deployment Serves APIs and manages workloads, queues and supporting resources.
device-manager Vendor DaemonSet Discovers devices and network interfaces,
reports capacity and allocates devices.
model-manager Per-node DaemonSet Caches model weights and mounts them into workloads using node delivery.
model-transfer Vendor source DaemonSet Runs the source broker and worker control service for remote weight delivery.
worker-gateway Installed separately Combines InstanceTypes and capacity from multiple clusters into a fleet view.

The chart also includes Node Feature Discovery (NFD) , Kueue and two CSI drivers. Topograph is optional and remains disabled until an administrator selects a provider. See Installation Modes for the deployment choices and Internals for contributor details.

Workload flow

GPU Instances provide containers with optional SSH access. Model deployments run inference engines and, where configured, routers or prefill/decode groups. Both use the same device capacity and admission chain. Model delivery and KV cache are optional services alongside that chain.

Workload and scheduling flow

The diagram shows the main dependencies. It does not require every workload to use a model artifact, RDMA, a topology source or a shared cache. An ordinary Pod can request device resources directly.

Device scheduling

Device scheduling follows this chain:

  1. NFD identifies node hardware and labels the nodes.
  2. The Device Manager discovers accelerators and network interfaces and maintains the Devices inventory and allocation record.
  3. The Worker derives schedulable CPU and accelerator capacity from that inventory.
  4. The Worker derives an AcceleratorType for each accelerated pool (and InstanceTypes when enabled), sharing the canonical pool ClusterQueue while topology and admission checks constrain placement.

Device Discovery covers node labeling and device discovery. Scheduling Chain covers capacity, queues and workload pool selection. RDMA Operations explains network resource requests, while Topology-Aware Scheduling explains placement domains.

Model delivery

Model delivery separates model weights and credentials from workload configuration. A ModelArtifact resolves and pins a Hugging Face repository, ModelScope repository, container image or PersistentVolumeClaim. Workloads reference the artifact instead of repeating its source and credentials.

Depending on the delivery mode, weights are fetched in the workload Pod (Engine), cached and mounted read-only on the node by the Model Manager’s CSI driver (Node), or mounted from an existing volume or image. Remote delivery can serve weights from source nodes through the model-transfer broker and vendor worker. See Remote Weight Delivery .

ModelStore and ModelStoreBinding manage cache budgets and namespace grants. NodeModelStore tracks per-node cache state. ModelPrefetch warms weights before use. These resources operate independently of GPU admission. See Model Delivery , Model Artifact , and the Model Prefetch Walkthrough .

KV cache

A shared KV cache stores reusable inference state. A backend runs the cache service or points to an external one. Pools and namespace bindings provide access and quota grants. Compatible workloads receive the cache configuration when their Pods are created. The service is optional. Its quota is separate from accelerator quota. See KV Cache .

Model Deployment brings these services together with engine, router and replica configuration. GPU Instances support interactive containers and SSH access using the same accelerator allocation.

Request admission

The Pod webhook validates an accelerator request before Kueue reserves quota. Admission checks confirm that devices can satisfy it, and the scheduler, kubelet and allocator complete the allocation. Admission owns the gate sequence and its failure behavior.

AcceleratorType and InstanceType status report the remaining capacity as workloads allocate and release devices. Use kubectl get acctype,instancetype -w to watch those changes. The Walkthrough shows the request and its resulting Kubernetes resources.

Vocabulary

Term Meaning
Pool Nodes with compatible CPU, accelerator, operating system and architecture identities, served by an isolated queue.
AcceleratorType Cluster-scoped serving and prefetch accelerator pool; short name acctype.
InstanceType Workspace sizing and available capacity for an Instance.
Allocation mode Whole accelerators, shared accelerators, logical slices or hardware partitions.
Credits Quota accounting units .
Four capacity views Available allocation modes reported by an AcceleratorType or InstanceType.
Devices ledger The per-node inventory and record of accelerator allocations.
LocalQueue The namespace queue through which a workload enters its pool.
ModelArtifact A namespaced model source and resolved weight identity referenced by workloads.
NodeModelStore The configuration and observed cache state maintained for one node.
ModelPrefetch A request to cache weights before a workload mounts them, authorized by a model-store binding.

The hardware terms Device, Accelerator and Resource are defined in Device Discovery .