Model Deployment
A ModelDeployment describes the model, engine and roles of a serving workload. Each role can have
its own replicas and resources.
Contents
Deployment configuration
Model Deployment Configuration starts with a minimal manifest and explains the role contract. Prefill and Decode covers split roles, and Encode, Prefill and Decode covers multimodal disaggregation. Engine Compatibility lists minimum releases, client and store requirements, transport support and known failures.
Encode, Prefill and Decode Benchmark measures that disaggregation against an aggregated instance. Prefill and Decode Benchmark measures a text-only prefill/decode pair against an aggregated group on equal GPUs.
Follow Prefill and Decode Walkthrough for disaggregated serving, Encode, Prefill and Decode Walkthrough for multimodal disaggregation, and Multi-Host Serving Walkthrough for distributed multi-node serving.
Service operations
Status , Metrics and Routing explain what the running service reports and how requests are directed.
Scaling plans the replica count of a deployment from the signals it publishes, and Shutdown explains the retirement protocol a scale-down runs. The Scaling Walkthrough configures observation and active scaling. The Scaling Benchmark records H100 measurements and the limits observed during scale-out and retirement.
Elastic EP keeps one serving instance and changes its collective while it runs. Elastic EP Walkthrough follows prefetch, startup with tensor parallelism (TP) = 2 and data parallelism (DP) = 2, and DP width 2 → 4 expansion. It includes captured resource output and completed inference requests.
Elastic EP Benchmark measures the same width change under a live request stream.
External DP runs one fixed DP group with an HTTP endpoint on every rank and Router-balanced traffic across all of them. The External DP Walkthrough runs a fixed group with one HTTP endpoint per rank and checks managed Router coverage.
External DP Benchmark compares that group with a monolithic wide tensor-parallel group on the same GPUs.