GPUStack Operator

Accelerator workloads, from discovery to serving

GPUStack Operator discovers accelerators, organizes capacity into pools, and schedules Kubernetes workloads for interactive use and model serving.

Get started ยท Read the architecture

Architecture

Four stages from hardware discovery to workload serving
  1. Discover: NFD and Device Manager identify accelerators and RDMA interfaces.
  2. Schedule: The worker builds pools and InstanceTypes; Kueue and admission checks place workloads.
  3. Deliver: Optional model prefetch and shared KV cache support workloads.
  4. Run: Model deployments and instances serve workloads.

Features

Heterogeneous devices

Discover and allocate whole, shared, sliced, or partitioned accelerators.

RDMA networking

Give workloads network interfaces alongside accelerators.

Topology

Place workloads within requested location or network domains.

KV cache

Share inference cache across compatible workloads.

Model delivery

Resolve, verify, and deliver model weights to workloads.

Model deployment

Run serving replicas with routing and optional prefill/decode roles.

GPU instances

Work in accelerator-backed containers over SSH.

Installation

Use Kubernetes 1.29 or newer for scheduling and node model delivery, Helm 3.8 or newer, and cluster-admin access. Install the accelerator drivers and any required container toolkit first.

bash
helm repo add gpustack https://docs.gpustack.ai/gpustack-operator/charts
helm install gpustack-operator gpustack/gpustack-operator \
  --namespace gpustack-system --create-namespace

See vendor prerequisites and installation modes for dependencies and deployment choices.

Usage

Choose a scenario for complete configuration and verification steps.

Schedule an accelerator workload

Request a device and follow its admission and allocation.

Add RDMA networking

Request a network endpoint alongside an accelerator.

Prefetch model weights

Warm a node cache before a workload starts.

Serve a model

Deploy prefill and decode roles and verify inference.

Share a KV cache

Create a backend, pool, and namespace binding.

Start a GPU workspace

Configure an accelerator-backed instance with SSH access.