GPUStack Operator
Docs Walkthroughs Model Deployment Walkthroughs

Model Deployment Walkthroughs

Choose a serving configuration, then use its cluster profile and deployment manifest. The H100 walkthroughs distinguish recorded deployments from configuration examples.

Contents

Choose an example

Tensor parallelism (TP), data parallelism (DP) and expert parallelism (EP) describe the parallel groups below. The DP width counts data-parallel ranks; the EP world size counts GPU workers in the expert-parallel collective. With internal load balancing (Internal LB), the master distributes requests to its DP ranks.

Walkthrough Serving configuration What it covers
Prefill and Decode Disaggregated roles fronted by Router Recorded H100 deployment with one Prefill and two Decode replicas, TP = 2, TCP KV transfer, and a linked benchmark .
Encode, Prefill and Decode Disaggregated multimodal roles fronted by Router Recorded H100 deployment with one Encode, one Prefill and one Decode replica, NIXL encoder transfer, and a linked benchmark .
Scaling Independent replicas managed by ModelDeploymentScaler Shadow and Active configuration, algorithm selection, role ratios, and observation of scale-out and retirement; links the measured H100 benchmark .
Elastic EP One Internal LB instance on Ray Recorded H100 expansion: TP = 2, DP width 2 → 4, EP world size 4 → 8; 439 client requests with zero errors.
External DP Four fixed DP ranks using multiprocessing Recorded H100 deployment: TP = 2, DP = 4, EP world size 8; captured per-rank inference and Router coverage.
Multi-Host Coordinated instance spanning multiple hosts Not measured here. size: 2, sixteen GPUs across two physical nodes, leader m0 routing, and gang admission.

A Router can front the Internal LB and External LB configurations. The engine’s load-balancing mode determines which Pods accept inference requests. Increasing roles[].replicas adds complete groups; see role scaling . The Scaling Walkthrough links a recorded single-role lifecycle check and distinguishes its observed replica changes from reusable configuration examples.

Prepare the cluster

Install the operator and enable node model delivery using the installation guide and model-store operations . Use kubectl, jq, curl, and a kubeconfig for the intended cluster.

The H100 recordings use one CPU node and one GPU node with eight NVIDIA H100 80 GB HBM3 SXM5 GPUs. Read each walkthrough’s environment section for its allocation and evidence limits. The Multi-Host example instead needs two GPU nodes.

bash
kubectl get nodes
kubectl get devices -o json
kubectl get instancetypes
kubectl get nodemodelstores -o json

Confirm Model Manager readiness and free GPUs before creating a serving workload. External DP needs eight free cards. Elastic EP needs four initially and eight after expansion. Complete model prefetch with the artifact, namespace and node selection specified by the serving walkthrough.

Prepare the GPU InstanceType

The H100 manifests use the discovered whole-GPU type gpustack--nvidia-h100-80gb-hbm3-linux-amd64. The recordings used the custom type bench-gpu. Each walkthrough explains this substitution; captured queue output retains the recorded name. Confirm the selected type’s capacity before applying a manifest.

Select another whole-GPU type when using different hardware, and verify model compatibility.