Architecture
GPUStack Operator manages accelerator-backed workloads on Kubernetes. It discovers devices and turns
node capacity into schedulable pools. GPU Instances and model deployments use those pools.
Model delivery resolves and supplies weights. Topology placement supports locality. Remote direct
memory access (RDMA) networking supports fast transfers. Shared KV cache stores reusable inference state.
Contents
- Components
- Workload flow
- Device scheduling
- Model delivery
- KV cache
- Request admission
- Vocabulary
- Related documentation
Components
The gpustack-operator binary provides these subcommands:
| Component | Deployment | Role |
|---|---|---|
worker |
Control-plane Deployment |
Serves APIs and manages workloads, queues and supporting resources. |
device-manager |
Vendor DaemonSet |
Discovers devices and network interfaces, reports capacity and allocates devices. |
model-manager |
Per-node DaemonSet |
Caches model weights and mounts them into workloads using node delivery. |
model-transfer |
Vendor source DaemonSet |
Runs the source broker and worker control service for remote weight delivery. |
worker-gateway |
Installed separately | Combines InstanceTypes and capacity from multiple clusters into a fleet view. |
The chart also includes Node Feature Discovery (NFD) , Kueue and two CSI drivers. Topograph is optional and remains disabled until an administrator selects a provider. See Installation Modes for the deployment choices and Internals for contributor details.
Workload flow
GPU Instances provide containers with optional SSH access. Model deployments run inference engines
and, where configured, routers or prefill/decode groups. Both use the same device capacity and
admission chain. Model delivery and KV cache are optional services alongside that chain.
The diagram shows the main dependencies. It does not require every workload to use a model artifact,
RDMA, a topology source or a shared cache. An ordinary Pod can request device resources directly.
Device scheduling
Device scheduling follows this chain:
- NFD identifies node hardware and labels the nodes.
- The Device Manager discovers accelerators and network interfaces and maintains the
Devicesinventory and allocation record. - The Worker derives schedulable CPU and accelerator capacity from that inventory.
- The Worker derives an
AcceleratorTypefor each accelerated pool (andInstanceTypes when enabled), sharing the canonical poolClusterQueuewhile topology and admission checks constrain placement.
Device Discovery covers node labeling and device discovery. Scheduling Chain covers capacity, queues and workload pool selection. RDMA Operations explains network resource requests, while Topology-Aware Scheduling explains placement domains.
Model delivery
Model delivery separates model weights and credentials from workload configuration. A ModelArtifact
resolves and pins a Hugging Face repository, ModelScope repository, container image or
PersistentVolumeClaim. Workloads reference the artifact instead of repeating its source and credentials.
Depending on the delivery mode, weights are fetched in the workload Pod (Engine), cached and
mounted read-only on the node by the Model Manager’s CSI driver (Node), or mounted from an
existing volume or image. Remote delivery can serve weights from source nodes through the
model-transfer broker and vendor worker. See Remote Weight Delivery
.
ModelStore and ModelStoreBinding manage cache budgets and namespace grants. NodeModelStore
tracks per-node cache state. ModelPrefetch warms weights before use. These resources operate
independently of GPU admission. See Model Delivery
,
Model Artifact
, and the
Model Prefetch Walkthrough
.
KV cache
A shared KV cache stores reusable inference state. A backend runs the cache service or points to an
external one. Pools and namespace bindings provide access and quota grants. Compatible workloads
receive the cache configuration when their Pods are created. The service is optional. Its quota
is separate from accelerator quota. See KV Cache
.
Model Deployment brings these services together with engine, router and replica configuration. GPU Instances support interactive containers and SSH access using the same accelerator allocation.
Request admission
The Pod webhook validates an accelerator request before Kueue reserves quota. Admission checks
confirm that devices can satisfy it, and the scheduler, kubelet and allocator complete the
allocation. Admission
owns the gate sequence and
its failure behavior.
AcceleratorType and InstanceType status report the remaining capacity as workloads allocate and release
devices. Use kubectl get acctype,instancetype -w to watch those changes. The
Walkthrough
shows the request and its resulting Kubernetes resources.
Vocabulary
| Term | Meaning |
|---|---|
| Pool | Nodes with compatible CPU, accelerator, operating system and architecture identities, served by an isolated queue. |
AcceleratorType |
Cluster-scoped serving and prefetch accelerator pool; short name acctype. |
InstanceType |
Workspace sizing and available capacity for an Instance. |
| Allocation mode | Whole accelerators, shared accelerators, logical slices or hardware partitions. |
| Credits | Quota accounting units . |
| Four capacity views | Available allocation modes
reported by an AcceleratorType or InstanceType. |
Devices ledger |
The per-node inventory and record of accelerator allocations. |
LocalQueue |
The namespace queue through which a workload enters its pool. |
ModelArtifact |
A namespaced model source and resolved weight identity referenced by workloads. |
NodeModelStore |
The configuration and observed cache state maintained for one node. |
ModelPrefetch |
A request to cache weights before a workload mounts them, authorized by a model-store binding. |
The hardware terms Device, Accelerator and Resource are defined in Device Discovery .
Related documentation
- Walkthrough — submit a workload and inspect the scheduling chain.
- GPU Instances — configure an interactive container and SSH access.
- Model Deployment — configure inference engines and replicas.
- Model Delivery — resolve, cache and mount model weights.
- KV Cache — connect inference workloads to a shared cache.
- Installation Modes — choose how to install the operator.