GPUStack Operator
Accelerator workloads, from discovery to serving
GPUStack Operator discovers accelerators, organizes capacity into pools, and schedules Kubernetes workloads for interactive use and model serving.
Get started ยท Read the architecture
Architecture
- Discover: NFD and Device Manager identify accelerators and RDMA interfaces.
- Schedule: The worker builds pools and InstanceTypes; Kueue and admission checks place workloads.
- Deliver: Optional model prefetch and shared KV cache support workloads.
- Run: Model deployments and instances serve workloads.
Features
Discover and allocate whole, shared, sliced, or partitioned accelerators.
Give workloads network interfaces alongside accelerators.
Place workloads within requested location or network domains.
Share inference cache across compatible workloads.
Resolve, verify, and deliver model weights to workloads.
Run serving replicas with routing and optional prefill/decode roles.
Work in accelerator-backed containers over SSH.
Installation
Use Kubernetes 1.29 or newer for scheduling and node model delivery, Helm 3.8 or newer, and cluster-admin access. Install the accelerator drivers and any required container toolkit first.
helm repo add gpustack https://docs.gpustack.ai/gpustack-operator/charts
helm install gpustack-operator gpustack/gpustack-operator \
--namespace gpustack-system --create-namespaceSee vendor prerequisites and installation modes for dependencies and deployment choices.
Usage
Choose a scenario for complete configuration and verification steps.
Request a device and follow its admission and allocation.
Request a network endpoint alongside an accelerator.
Warm a node cache before a workload starts.
Deploy prefill and decode roles and verify inference.
Create a backend, pool, and namespace binding.
Configure an accelerator-backed instance with SSH access.