GPUStack Operator

Engine Compatibility

Check engine and transport compatibility before choosing images for a deployment. The requirements cover minimum releases, Mooncake client and store versions, supported transports and known failures.

Engines below the listed minimum versions are unsupported. Upgrade before diagnosing failures on an older engine release.

Contents

Minimum versions

The table covers aggregated serving, prefill/decode (PD) and encode/prefill/decode (EPD) configurations.

Engine Configuration Minimum Mooncake client Store members
vLLM alone 0.29.0; behind llm-d-router, --enable-scale-out is also required on 0.30.0+ 0.3.13.post1, on the cuda12.9 and the cuda13.0 tag —
vLLM prefill/decode, direct transfer over tcp 0.29.0; behind llm-d-router, --enable-scale-out is also required on 0.30.0+ 0.3.13.post1 —
vLLM with a Mooncake store, alone or as a pair 0.29.0; behind llm-d-router, --enable-scale-out is also required on 0.30.0+ 0.3.13.post1 the 0.3.13 line; the default image is one
vLLM-Ascend alone 0.23.0; behind llm-d-router, --enable-scale-out is also required on 0.30.0rc1+ 0.3.11.post1, the NPU build —
vLLM-Ascend prefill/decode, direct transfer 0.23.0, behind llm-d-router; --enable-scale-out is also required on 0.30.0rc1+ 0.3.11.post1, the NPU build —
vLLM-Ascend with a Mooncake store not verified 0.3.11.post1, the NPU build not verified
SGLang alone 0.5.18, run as the halves of a pair 0.3.12.post1 —
SGLang prefill/decode, direct transfer over tcp 0.5.18 0.3.12.post1 —
SGLang with a Mooncake store 0.5.18, run as a pair 0.3.12.post1 the 0.3.12 line, such as kvcacheai/mooncake:0.3.12.post1
vLLM encode/prefill/decode, direct transfer over NIXL (ECCPUConnector) 0.30.0, behind llm-d-router or vllm-router: NVIDIA CUDA (verified on 4 × NVIDIA L40S) or AMD ROCm (unverified on hardware) compiles its own, not run —
vLLM-Ascend encode/prefill/decode, direct transfer (ECExampleConnector) 0.30.0rc1, behind llm-d-router (verified on Ascend 910B2) 0.3.11.post1, the NPU build —
SGLang encode/prefill/decode, transfer by address (encoder-urls) 0.5.10, behind llm-d-router or sglang-gateway (NVIDIA verified on 2 × NVIDIA L40S) 0.3.12.post1 —

A version counts as run when the configuration answered requests and, where it moves KV, the engine or the store reported blocks moving: a transfer or a store write, not a Pod reaching Ready. The vLLM-Ascend rows ran on the runner’s cann9.1-910b-vllm0.23.0 tag, named through roles[].image.

The NVIDIA vLLM EPD rows (behind llm-d-router and vllm-router) and the NVIDIA SGLang EPD rows (run on 0.5.18, behind both routers) ran live. The vLLM-Ascend EPD row behind llm-d-router ran live on Ascend 910B2. The AMD EPD rows and the SGLang Ascend rows are admitted but not verified on hardware (#763 tracks them).

Reading the table

The minimum is a property of the image you name. There is no engine.version field to set: the image a role runs resolves to roles[].image when the role names one, and to spec.engine.image otherwise (Role images ). Nothing assembles a tag and no admission check reads a version, so meeting a minimum here means naming an image whose engine build carries it.

A deployment without an Encode role needs --enable-scale-out behind llm-d-router. The router counts prompt tokens through the engine’s render endpoint, which vLLM serves only under that flag, from 0.30.0 on; the gate measured on vllm-ascend is 0.30.0rc1. Name an image at those versions and set the flag on each role (Counting engine tokens ).

The 0.29.0 and 0.23.0 minimums above predate the flag: they were measured as transfer-path runs, before the operator rendered this dependency.

The floor belongs to the engine, so an SGLang EPD row states SGLang 0.5.10 on every platform, while the vLLM Ascend row states a vllm-ascend release.

The client comes with the image. The column above is read off the published runner images, and the vLLM rows are the CUDA ones; a ROCm runner image compiles its own client and has not been run. A role naming its own roles[].image carries whatever client that image embeds: read it off the image, then pick the store from it.

A store runs on the client’s minor line. The store column is a requirement rather than a suggestion; see Version compatibility .

vLLM-Ascend’s direct transfer does not support direct tcp transfer; see Pair connections .

The configurations above move KV over tcp, or over ascend on vLLM-Ascend. Which other transport each engine can use is the matrix below .

“Not verified” means no run, not a known failure. Nothing refuses such a configuration; nothing here says it works.

An EPD encoder image must meet its row’s floor, and nothing refuses a lower one. The requirement: the image an Encode deployment runs must carry vllm-ascend 0.30.0rc1 or later for Ascend ECExampleConnector, and vLLM 0.30.0 or later for NVIDIA/AMD ECCPUConnector (the admitted matrix ). Admission does not check it — the version field and its gate are gone.

An image below the floor fails at container start with an ImagePullBackOff on a tag that does not exist, or an engine that exits on an unknown argument.

The group-forward probe has a verified version set, and nothing bars an unverified one. The requirement: the version whose probe binding was read is 0.25.1/0.29.0 for vLLM (the response id is cmpl- + the request’s request_id) and 0.5.18/0.5.21 for SGLang (the response meta_info.id is the request’s rid).

On any other version the probe still runs and its outcome still decides: Verified admits a member, Unknown grants nothing new and withdraws nothing already granted. Only a failed health check of the same qualification withdraws an eligible member. Neither probe outcome is an admission refusal.

Transport compatibility

The store path uses KVCacheBackend.spec.transport.protocol. The direct prefill/decode path selects Mooncake, NIXL, or HIXL through spec.kvTransfer.backend; see Transfer parameters for fabric selection.

The store compatibility observations below apply to the named images.

Transport (API/Mooncake) Engine Store path
TCP / tcp vLLM Works, default images; beside a direct path it keeps its store connections open too and did not run out of ports where measured
TCP / tcp SGLang Works, with a published store image on its client’s line , named by hand; beside a direct path, it runs out of ports unless model-deployment-tcp-tw-reuse is on
TCP / tcp vLLM-Ascend Not supported: refused at admission, its store client accepts CANN only
RDMA / rdma vLLM Not verified
RDMA / rdma SGLang Not verified
RDMA / rdma vLLM-Ascend Not supported: refused at admission, as for TCP
EFA / efa vLLM Own image: an EFA build of Mooncake in the engine; default store image
EFA / efa SGLang Not supported: no EFA build serves SGLang
EFA / efa vLLM-Ascend Not supported: refused at admission, as for TCP
CANN / ascend vLLM Not supported: the CUDA client has no Ascend transport
CANN / ascend SGLang Not verified
CANN / ascend vLLM-Ascend Not verified; the members need an image carrying CANN
ROCM / hip vLLM, SGLang Not verified
ROCM / hip vLLM-Ascend Not supported: refused at admission, as for TCP
MUSA, MACA / musa, maca vLLM, SGLang, vLLM-Ascend Not verified

The following direct-path observations apply to the named configurations, not every backend or image:

  • Mooncake TCP worked with the default vLLM and SGLang images. SGLang under sustained load can run out of ports ; model-deployment-tcp-tw-reuse is the documented remedy.
  • Mooncake RDMA worked with vLLM on InfiniBand H100 SXM5 in the Prefill and Decode Benchmark and Encode, Prefill and Decode Benchmark .
  • EFA requires an EFA engine build . The default store image carries EFA. The combined store and direct paths over EFA have not been run here.
  • The recorded vLLM-Ascend direct path used cann9.1-910b-vllm0.23.0 behind llm-d-router. Other direct backend and fabric combinations require separate validation.
  • A store on Ascend requires CANN; its members need an image carrying CANN .

When a store and direct path request incompatible fabrics, admission refuses the deployment.

Remote weight loading

The --load-format model-transfer value selects an engine’s model-transfer adapter, the loader that reads weights from a prepared source instead of a local checkpoint. The engine keeps its own parameter mapping, sharding and finalization; the adapter supplies verified tensors and proves its finalization before the engine profiles KV memory. Configuration and behavior live in Remote Weight Delivery ; the table gives the state per engine:

Engine Adapter entry Qualification state
vLLM NativeModelTransferLoader, --load-format model-transfer on CUDA H100 loading and inference observed for full-resident and paged 80B sources and full-resident 235B broadcast; production qualification remains open
vLLM-Ascend NativeModelTransferLoader through the vLLM general plugin, with vllm_ascend (requires CANN 9.2+ runner image with Python hixl, such as cann9.2-910b-vllm0.30.0) loading and inference measured on two Ascend 910B2 nodes for Qwen3-8B, Qwen3.6-35B-A3B and Qwen3-235B-A22B-W8A8; managed HIXL routing and production qualification remain closed
SGLang NativeSGLangModelTransferLoader, --load-format model-transfer via .pth CLI and LoadFormat enum hooks (requires CANN 9.2+ runner image on Ascend for Python hixl) H100 80B loading and inference observed; Ascend evidence covers CLI, LoadFormat, probe and session-open, with full loading qualification still open

Recorded loading and inference establish only the listed configuration. Each engine’s hardware and safety gates must pass on its own platform before production qualification; a CUDA result qualifies no Ascend or ROCm row. An unqualified combination is refused or falls back by the delivery policy: under Require, a role whose engine and pool name no qualified backend is blocked rather than rewritten.

The adapter uses vLLM’s general load-format plugin for vLLM and vLLM-Ascend. SGLang uses .pth CLI and LoadFormat extension hooks.

Ascend (HIXL)

The Ascend benchmark compares same-node and cross-node loading. On Ascend (hixl), CANN 9.2 introduces the Python hixl package (/usr/local/Ascend/cann-9.2.0/python/site-packages/hixl) alongside libcann_hixl.so. CANN 9.0 images install only llm_datadist and C++ libcann_hixl.so, so load_runtime() fails closed with RuntimeUnavailableError when a CANN 9.0 runner image is used.

Limitations

  • SGLang with a store holds a pinned host pool and needs the node’s available memory above a fixed reserve plus that pool; see SGLang’s host-memory tier .

  • SGLang prefill/decode over TCP runs out of local ports under sustained load, with or without a store. The prefill half opens a new connection for every transfer, to the decode half and to each store member, all from the same ephemeral ports of its container. Connections left in TIME-WAIT use them up; from then on every request through that pair fails and keeps failing until the engine Pods are restarted. On the measured pair without a store, the gateway then answered 503 with No available prefill workers.

    Turning on model-deployment-tcp-tw-reuse prevented failure in a measured SGLang 0.5.18 pair with a store: it answered 1905 sequential requests without failure, with about 19,500 TIME-WAIT sockets. It needs a kubelet change on every eligible node.

    The pair without a store was not tested with this setting. Widening net.ipv4.ip_local_port_range only delays exhaustion; whether a fabric transport avoids it remains unverified.

    vLLM did not show this behavior in the measured runs: its prefill half keeps its transfer connections open instead of opening one per transfer, and held at most nine sockets in TIME-WAIT over a direct path with no store and at most sixteen with a TCP store beside it.