Prefill and Decode Walkthrough

Deploy a disaggregated large language model (LLM) topology with dedicated prefill and decode roles fronted by a Router. Observe joint admission across both roles, then send completion requests through the router’s single endpoint.

Contents

• Summary
• Environment
• Step 1: Start disaggregated serving
• Step 2: Pin the transfer protocol
• Step 3: Verify the router
• Step 4: Inspect joint admission
• Step 5: Release the application
• Limitations
• Troubleshooting

Summary

Prefill/Decode (PD) disaggregation separates compute-bound prompt prefill from memory-bandwidth-bound token decoding. A compute-heavy Prefill role processes initial prompt context. A memory-bandwidth-bound Decode role generates subsequent tokens.

A managed Router fronts both roles and exposes one cluster-internal HTTP endpoint. It parses prompt requests, forwards prefill work, and coordinates key-value (KV) cache transfer to decoders. Kueue schedules both roles as a joint unit, preventing half-admitted workloads.

The H100 comparison allocated six GPUs to each configuration. It supports PD when smoother generation under long-prompt concurrency outweighs first-token delay. At concurrency 16, long-prompt inter-token latency (ITL) p99 improved 25.6×–27.8×, while first-token latency rose substantially. The mixed steady stream had 23.3× lower maximum ITL, with no p99 improvement. See Prefill and Decode Benchmark for throughput, tails and the decode-heavy outlier.

Environment

The deployment and probe below were captured on this cluster:

Each engine Pod exposed only its two assigned GPUs, with private IPC and /dev/shm. No remote direct memory access (RDMA) interface was attached. NVLink carried TP = 2 traffic inside each Pod. Cross-Pod Mooncake KV transfer used TCP sockets over eth0, despite the host’s NVLink fabric.

Step 1: Start disaggregated serving

Enable node model delivery and follow the model prefetch procedure with namespace gpustack-bench. Create ModelArtifact qwen3-32b from Qwen/Qwen3-32B at revision 9216db5781bf21249d130ec9da846c4624c16137, then prefetch it onto the H100 node. Use that artifact and namespace throughout the procedure instead of its DeepSeek example.

The captured run used the custom bench-gpu InstanceType for the same H100 accelerator group. This reusable manifest uses the discovered H100 InstanceType name. Replace the two instanceType values with bench-gpu to reproduce the captured configuration exactly.

The deployment requests one Prefill replica and two Decode replicas, each with two GPUs. All three engine Pods and the Router ran on gpu-node-0.

Save the following manifest as pd-deployment.yaml:

apiVersion: worker.gpustack.ai/v1
kind: ModelDeployment
metadata:
  name: pd-arm2-pd
  namespace: gpustack-bench
spec:
  model:
    name: /models
    artifactRef:
      name: qwen3-32b
  engine:
    name: vLLM
    image: gpustack/runner:cuda13.0-vllm0.30.0
  router:
    name: llm-d-router
  roles:
    - name: prefill
      kind: Prefill
      replicas: 1
      size: 1
      instanceType: gpustack--nvidia-h100-80gb-hbm3-linux-amd64
      image: gpustack/runner:cuda13.0-vllm0.30.0
      imagePullPolicy: IfNotPresent
      resources:
        accelerator: "2"
      extraArgs:
        - --tensor-parallel-size
        - "2"
        - --max-model-len
        - "16384"
        - --max-num-seqs
        - "64"
        - --gpu-memory-utilization
        - "0.90"
        - --enable-scale-out
    - name: decode
      kind: Decode
      replicas: 2
      size: 1
      instanceType: gpustack--nvidia-h100-80gb-hbm3-linux-amd64
      image: gpustack/runner:cuda13.0-vllm0.30.0
      imagePullPolicy: IfNotPresent
      resources:
        accelerator: "2"
      extraArgs:
        - --tensor-parallel-size
        - "2"
        - --max-model-len
        - "16384"
        - --max-num-seqs
        - "64"
        - --gpu-memory-utilization
        - "0.90"
        - --enable-scale-out

Apply the manifest and inspect the captured status:

kubectl apply -f pd-deployment.yaml
kubectl get modeldeployment pd-arm2-pd -n gpustack-bench \
  -o custom-columns=NAME:.metadata.name,ENGINE:.spec.engine.name,ROLES:.status.roleSummary,PHASE:.status.phase

NAME        ENGINE   ROLES    PHASE
pd-arm2-pd  vLLM     1R1P2D   Ready

The recorded Pods were ready and co-located on gpu-node-0:

NAME                                READY   STATUS    NODE
pd-arm2-pd-decode-2xglg             2/2     Running   gpu-node-0
pd-arm2-pd-decode-r2zt4             2/2     Running   gpu-node-0
pd-arm2-pd-prefill-xmjvs            1/1     Running   gpu-node-0
pd-arm2-pd-router-6b9fcd6b75-5ngnj   2/2     Running   gpu-node-0

Each engine process uses tensor parallelism (TP) across its two requested accelerators. The --tensor-parallel-size 2 flag sets TP = 2. The deployment has no Encode role, so both roles set --enable-scale-out; see Counting engine tokens .

Step 2: Pin the transfer protocol

Disaggregated serving requires prompt KV tensors to flow from prefill workers to decode workers. The pair in Step 1 hands blocks over directly. An unset spec.kvTransfer.protocol renders tcp. Pin the protocol explicitly. Add the lines marked + to pd-deployment.yaml:

 apiVersion: worker.gpustack.ai/v1
 kind: ModelDeployment
 metadata:
   name: pd-arm2-pd
   namespace: gpustack-bench
 spec:
+  kvTransfer:
+    protocol: TCP
   router:
     name: llm-d-router

Apply the update:

kubectl apply -f pd-deployment.yaml

The recorded manifest leaves the protocol unset, which renders tcp by default. The protocol is deployment-wide because it describes one link between the two roles. It accepts Auto, TCP, RDMA, EFA, CANN and ROCM; Auto renders tcp too. An edit restarts every role because the value renders into both ends’ arguments. RDMA and EFA also need a matching device request on each role. See the RDMA walkthrough .

The captured engine commands rendered these transfer settings:

Prefill main:
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_connector_extra_config":{"mooncake_protocol":"tcp"}}'
  --kv-events-config enables the publisher on ports 5557 and 5558
  --port 8000

Decode main:
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"mooncake_protocol":"tcp"}}'
  --port 8200

Decode routing-proxy init container:
  --port=8000 --model-server-port=8200 --kv-connector=mooncake
  --mooncake-bootstrap-port=8998 --secure-proxy=false

The InfiniBand RDMA follow-up set spec.kvTransfer.protocol to RDMA and requested resources.interface = "2" on both roles. Runtime logs verified the InfiniBand KV path; the configuration change alone is not transport proof.

A shared store through spec.kvCache.poolRef is part of the deployment identity. Declare it in the first apply. Adding it later is refused because a different cache describes a different deployment. See deployment identity fields .

Step 3: Verify the router

The captured Router endpoint is:

http://pd-arm2-pd-router.gpustack-bench.svc:8000

Send a completion request through the Router from inside the cluster:

kubectl -n gpustack-bench run test-client --rm -i --restart=Never \
  --image=curlimages/curl -- \
  curl -s http://pd-arm2-pd-router.gpustack-bench.svc:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/models",
    "messages": [{"role": "user", "content": "Explain prefill and decode disaggregation."}],
    "max_tokens": 16
  }'

The recorded burst probe sent 16 requests at concurrency 8, with 8,192 input tokens and 64 output tokens. All 16 succeeded, with zero empty bodies or errors:

{"requests":16,"concurrency":8,"input_tokens":8192,"output_tokens":64,"ok":16,"empty":0,"bad":0}

The Router delegates prompt processing and coordinates the roles. Mooncake transfers KV tensors between the engines, and the Router returns generated tokens. status.endpoint remains empty until the Router Pod passes its readiness check.

Step 4: Inspect joint admission

The operator manages Kueue workloads across all declared roles simultaneously. The captured workloads were all admitted into the captured custom bench-gpu queue. The reusable discovered InstanceType uses its own generated queue:

kubectl -n gpustack-bench get workloads

NAME                              QUEUE                             RESERVED IN   ADMITTED
gpustack-fnv64-ab5121bc91ece976   gpustack-fnv64-889bba71d87d29e2   bench-gpu     True
gpustack-fnv64-ab5122bc91eceb29   gpustack-fnv64-889bba71d87d29e2   bench-gpu     True
gpustack-fnv64-d6b4e8ae20539d37   gpustack-fnv64-889bba71d87d29e2   bench-gpu     True

You can adjust replicas independently on either role to balance prompt versus token throughput. See Rollout behavior for editable role fields and replica replacement semantics.

Step 5: Release the application

After testing disaggregated serving, delete the deployment to release GPUs and Router services:

kubectl -n gpustack-bench delete modeldeployment pd-arm2-pd
kubectl -n gpustack-bench get pods,services -l app.kubernetes.io/instance=pd-arm2-pd

Verify that all application GPU claims are released. The post-deletion PD Devices snapshot recorded the following availability:

node: gpu-node-0
accelerator: NVIDIA H100 80GB HBM3
count: 8
remaining per accelerator: 1600000

This snapshot establishes released accelerator claims. Model-cache contents and cluster infrastructure remain outside its scope.

Limitations

• The recording uses one GPU node with eight H100 accelerators and co-locates all six serving GPUs and the Router on that node. It does not measure placement across multiple GPU nodes.
• The captured transfer path is TCP through Mooncake. RDMA, EFA, CANN and ROCM paths require their own device requests and runtime validation.
• The short completion probe covers readiness and inference. The separate benchmark measures all 16 fixed-shape cells and mixed traffic; its results remain limited to the captured H100 configuration.
• Joint gang admission across all roles (PodGroup), role-set immutability after creation and per-replica rollout replacement are covered in Prefill and decode and Rollout behavior .
• Disaggregated serving requires a ready Router (status.endpoint is populated only while the Router Pod is ready); see Router configuration .

Troubleshooting