Remote Weight Delivery
Remote weight delivery lets an engine load tensors from a prepared source node without first copying the complete checkpoint to the serving node. The source registers weights verified by its node cache, and the target mounts a metadata-only view. A cluster-wide policy enables delivery; the deployment’s model block does not.
H2H means Host-to-Host; H2D means Host-to-Device. Host-memory windows are bounded and reused, so the successful remote path does not store a complete checkpoint on the target filesystem.
Contents
- Configuration
- Source preparation
- Replica startup
- Benchmarks
- Require and Prefer
- Networks
- Observing status
- Troubleshooting
- Rollback
Configuration
One Setting chooses the cluster’s policy, and the chart ships it:
Setting |
Default | Values |
|---|---|---|
model-artifact-remote-delivery-policy |
Prefer |
Disabled, Require, Prefer |
Prefer: Default policy; uses remote delivery when a prepared source is ready and local delivery otherwise.Disabled: Turns remote delivery off; leavesPodspecs, conditions, and status fields unchanged.RequireandPrefer: Require themodel.csi.gpustack.aiCSIDriverso remote replicas can mount metadata views.
kubectl -n gpustack-system patch setting model-artifact-remote-delivery-policy \
--type merge -p '{"spec":{"value":"Disabled"}}'The source broker’s ServiceAccount can review audience-scoped tokens and read live workload,
artifact, placement and grant state. It pins each Binding’s quota authority and updates that
authority’s ConfigMap in the operator namespace. It can also acknowledge drain on source Pods in
that namespace. The model-manager remains the only writer of NodeModelStore status.
The chart’s modelTransfer.images map configures full source-worker image references per supported
manufacturer (nvidia, amd, ascend). It renders one DaemonSet per configured manufacturer
with NFD PCI nodeSelector placement (feature.node.kubernetes.io/pci-<vendor>.present: "true").
Missing entries ship no source runtime for that manufacturer.
Without cert-manager, Helm preserves the generated serving certificate by looking up its Secret.
Renderers that cannot perform this lookup, including offline GitOps rendering and
helm template | kubectl apply, generate a new trust anchor on every render. Configure
cert-manager for these workflows so repeated syncs do not rotate the source certificate and
interrupt sessions while tenant discovery catches up.
Nodes without accelerator labels use the NVIDIA host-memory source image; mixed-manufacturer
nodes have no implicit source selection. Image updates via Helm use OnDelete so each source Pod
is drained and replaced only after its current runtime instance proves release.
Removing a manufacturer entry or disabling the source runtime deletes its DaemonSet directly through Helm. Perform this maintenance when the affected workload is stable; it does not wait for the image-rollout drain acknowledgement. Host driver versions do not rewrite the map. Disk warm-up retains its independent Python Alpine image.
Remote delivery targets hub artifacts on managed roles. The routing candidates are nixl
for vLLM and SGLang on NVIDIA, and hixl for vLLM-Ascend and SGLang on Ascend. The current
allowlist admits only the NVIDIA candidates; Ascend is not yet admitted. Routing admission does
not establish production acceptance, which remains open.
Setting Require or Prefer starts nothing on its own. A role with its own command is never
rewritten, and claim and image artifacts keep their existing delivery paths. On Ascend (hixl),
the engine runner image must include CANN 9.2 or newer so the Python hixl module is present.
Source preparation
A Model Prefetch
pins a verified disk cache and prepares its host-memory source.
Source preparation requires explicit placement and a ModelStoreBinding covering those nodes.
Disk readiness is reported in status.readyNodes; transfer readiness is separate in status.transfer.
The source has three limits: the broker’s node budget, the per-generation resident window, and an optional shared Binding grant. The smallest applicable limit wins. Target staging RAM and model device memory are separate; configure them under Replica startup .
Source Helm values with explicit runtime defaults (the chart defaults env to {}):
modelTransfer:
source:
# Total source reservations for this broker on the node.
# 0 uses 1/16 of MemTotal at startup, floored at minMemoryBytes.
maxMemoryBytes: 0
# 1 GiB floor; smaller nonnegative values resolve to 1 GiB.
minMemoryBytes: 1073741824
# Concurrent sessions across this source Pod; 0 resolves to 64.
maxSessions: 64
# Empty by default. Explicit limits must also cover non-transfer allocations.
# Source sizing reads node RAM from /proc/meminfo.
workerResources: {}
# Applied to both broker and worker containers.
env:
# Per-generation ceiling for either backend; 0 follows admission.
# Positive values round down to 2 MiB, with a 1 GiB floor.
# Malformed or negative values are refused.
GPUSTACK_MODEL_TRANSFER_SOURCE_RESIDENT_BUDGET_BYTES: "0"
# Additional HIXL ceiling: 16 GiB; 0 retains this default.
# Nonzero values have a 1 GiB floor and round down to 2 MiB.
GPUSTACK_MODEL_TRANSFER_HIXL_RESIDENT_BUDGET_BYTES: "17179869184"Negative source values are rejected. A ceiling reserves no memory by itself: admission checks
available RAM, node capacity and the Binding grant, including 10% reservation overhead.
Both NIXL and HIXL page checkpoints larger than the admitted window. The window must fit at least
1 GiB and the largest safetensors shard, including native allocation overhead; shards are not split.
The chart creates one shared source DaemonSet per configured manufacturer. Its privileged
containers access host devices and InfiniBand; Ascend also mounts the host driver directory.
The image supplies CANN and HIXL. Preparing a model creates no exclusive device reservation.
Vendor device use and production qualification remain separate acceptance requirements.
global.imageRegistry (or global.hub), global.imageNamespace and global.imagePullPolicy
apply to vendor image references as well.
The source identity can update Binding annotations across namespaces to pin reservation authority.
Treat source Pods and their vendor images as trusted cluster infrastructure.
Source memory examples
Use the benchmark models Qwen/Qwen3-8B and Qwen/Qwen3.6-35B-A3B in namespace
gpustack-default. Both sources share the model-source-pool store and qwen-source Binding.
These are configuration examples, not captured benchmark manifests. HIXL remains outside the
managed replica routing allowlist.
Node and source limits
Set the broker’s total ceiling to 128 GiB and each HIXL generation’s ceiling to 16 GiB:
modelTransfer:
source:
maxMemoryBytes: 137438953472 # 128 GiB shared by all this broker's generations.
minMemoryBytes: 1073741824
maxSessions: 64
env:
GPUSTACK_MODEL_TRANSFER_SOURCE_RESIDENT_BUDGET_BYTES: "0"
GPUSTACK_MODEL_TRANSFER_HIXL_RESIDENT_BUDGET_BYTES: "17179869184" # 16 GiB per generation.Models and shared grant
Label the source node gpustack.ai/model-transfer-source: "true". Create the two artifacts,
the store and its Binding. Repository revisions and manifest digests are resolved automatically;
see Model Artifact
for optional digest checks.
apiVersion: worker.gpustack.ai/v1
kind: ModelArtifact
metadata:
name: qwen3-8b
namespace: gpustack-default
spec:
source:
modelScope:
repository: Qwen/Qwen3-8B
---
apiVersion: worker.gpustack.ai/v1
kind: ModelArtifact
metadata:
name: qwen3-6-35b-a3b
namespace: gpustack-default
spec:
source:
modelScope:
repository: Qwen/Qwen3.6-35B-A3B
---
apiVersion: worker.gpustack.ai/v1
kind: ModelStore
metadata:
name: model-source-pool
spec:
nodeSelector:
matchLabels:
gpustack.ai/model-transfer-source: "true"
---
apiVersion: worker.gpustack.ai/v1
kind: ModelStoreBinding
metadata:
name: qwen-source
namespace: gpustack-default
spec:
storeRefs:
- name: model-source-pool # The ModelStore above.
quota:
bytes: 128Gi # Shared disk-cache quota; separate from source RAM.
allowPinned: true # Allow both prefetches to pin their cached weights.
# Optional: omit this block to use node-derived limits instead.
# Without it: node memory ceiling, 4096 registrations and source.maxSessions.
sourceMemory:
# 64 GiB shared by both models, across source nodes and manufacturers.
# Includes reservations; grants no dedicated capacity against other Bindings.
# Bytes and registrations can only grow while this block is present.
bytes: 64Gi
registrations: 4096
# At least 1; can change online. Existing sessions may finish above a lowered cap.
sessions: 64Two model sources
Each prefetch references its own artifact and the shared qwen-source Binding:
apiVersion: worker.gpustack.ai/v1
kind: ModelPrefetch
metadata:
name: qwen3-8b-source
namespace: gpustack-default
spec:
artifactRef:
name: qwen3-8b # Qwen/Qwen3-8B above.
bindingRef:
name: qwen-source # Shares the 64 GiB source grant and 128 GiB disk quota.
placement:
# Required for source preparation; match the ModelStore's source nodes.
nodeSelector:
matchLabels:
gpustack.ai/model-transfer-source: "true"
retention:
pinned: true
source: {} # Register a host-memory source as well as warming the disk cache.
---
apiVersion: worker.gpustack.ai/v1
kind: ModelPrefetch
metadata:
name: qwen3-6-35b-a3b-source
namespace: gpustack-default
spec:
artifactRef:
name: qwen3-6-35b-a3b # Qwen/Qwen3.6-35B-A3B above.
bindingRef:
name: qwen-source # The same grant; this creates no second 64 GiB budget.
placement:
nodeSelector:
matchLabels:
gpustack.ai/model-transfer-source: "true"
retention:
pinned: true
source: {}| Model | Benchmark checkpoint storage | 16 GiB ceiling |
|---|---|---|
| Qwen3-8B | 15.27 GiB | Fits in full if admission grants enough space for native backing. |
| Qwen3.6-35B-A3B | 66.99 GiB | Uses successive resident batches; the largest shard must fit. |
The 64 GiB Binding grant cannot enlarge either 16 GiB window. Each generation and session
keeps its own charge, even for the same digest; copies on additional nodes share the grant too.
For illustration, 16 GiB of native backing charges about 17.6 GiB with reservation overhead:
three such sources fit within 64 GiB; a fourth does not, even below the node’s 128 GiB ceiling.
Smaller node budget
Keep the Binding and prefetches above, but lower the broker’s total ceiling to 16 GiB:
modelTransfer:
source:
# Total reservations for both Qwen sources, including overhead.
# The Binding's 64 GiB grant cannot override this node ceiling.
maxMemoryBytes: 17179869184
minMemoryBytes: 1073741824
maxSessions: 64
env:
GPUSTACK_MODEL_TRANSFER_SOURCE_RESIDENT_BUDGET_BYTES: "0"
GPUSTACK_MODEL_TRANSFER_HIXL_RESIDENT_BUDGET_BYTES: "17179869184"With no other charges, at most about 14.54 GiB remains for native backing after overhead and
alignment. Even 8B now needs paging; the two sources cannot each claim that much concurrently.
Admission may wait, and each admitted window must still fit its largest shard.
Readiness and release
The model-manager verifies weights before publishing the cache. Prefetch pins that immutable tree; transfer validates identity, layout and bounds without hashing payloads again. Keep source disk available for page-in and restore. A page swap drains and acknowledges active work, revokes old registrations, reuses shared files and frees evicted allocations. A queued page miss prevents newer resident reads from extending that drain.
Idle sources release resident blocks once no native operations or unknown sessions remain. Later sessions refill from disk and wait for registration. Failed restore or unknown release quarantines the generation and retains its resources and reservation. Lease expiry, Pod deletion and process exit alone do not prove release. Missing or corrupt quota authority blocks new admissions and retains existing charges.
Remote readiness requires the current artifact, worker instance, tenant reservation, store coverage
and an unexpired lease. Unresolved or zero-sized artifacts allocate nothing and report
SourceSizeUnresolved. A grant unable to back the 1 GiB minimum window reports SourceWindowFallback under
Prefer (Progressing=True) or SourceWindowBelowFloor under Require (Progressing=False).
Prefer can use peer sync or local CSI loading; Require waits.
Drain and replace the source Pod after chart argument or environment changes. An existing
generation keeps its admitted window; prepare a fresh generation for a larger budget. Binding
grants can grow online. Removing sourceMemory restores node-derived limits, not unlimited RAM.
Deploy the operator, source SDK and engine SDK together: the layout-only contract is version 2;
version 1 readers and manifests are refused.
Inline manifests are limited to 64 MiB raw JSON. Paged manifests use roots and pages of at most
4 MiB, with at most 512 MiB of pages and 1,048,576 tensors overall. Load and source journals
cap operations at 1,048,576; byte limits can refuse work sooner. Proven acknowledgements can
be compacted; unknown receipts remain charged.
See source budget calculations for admission arithmetic and available-memory sizing.
Replica startup
The operator renders three additions to a remote replica’s Pod:
- A metadata-only CSI volume. The volume carries the artifact, its UID and the digest, plus a
model-transfer.gpustack.ai/delivery: metadataattribute. The plugin serves configuration, tokenizer and safetensors headers, never the complete checkpoint. This lets the engine inspect the model configuration before loading. A target without a checkpoint fetches this bounded view through an authenticated source endpoint using the workload’s CSI token. - A projected ServiceAccount token for audience
gpustack-model-transfer, rotated within its hour. - The wheel, staged on the host. The vendor
model-transferimage ships the universalgpustack_model_transferwheel and themodel-transferDaemonSet stages it to/var/lib/gpustack/operator/lib/model-transfer; the engine container installs it offline before start, only if it is not importable already.
The engine starts with --load-format model-transfer and GPUSTACK_MODEL_TRANSFER_* environment
variables for the artifact, UID, digest, mount path, token, wheel directory and backend. These
select and configure the engine’s model-transfer adapter. The adapter checks every tensor against
its owned-output budget before opening a session. A refused plan reads no bytes.
Configure native vLLM and SGLang loading through the serving role’s env. These values
show the defaults; merge the entries with any existing role environment:
apiVersion: worker.gpustack.ai/v1
kind: ModelDeployment
metadata:
name: qwen3-8b
namespace: gpustack-default
spec:
roles:
- kind: Server
name: server
env:
# broadcast (or tp0): the TP coordinator reads, then broadcasts tensors.
# all: each TP rank opens a session and reads its own weights.
- name: GPUSTACK_MODEL_TRANSFER_TP_MODE
value: "broadcast"
# 8 GiB target staging ceiling; 32 or 64 GiB allows larger windows.
# In all mode, ranks divide this budget, with a 1 GiB minimum per rank.
- name: GPUSTACK_MODEL_TRANSFER_MAX_WINDOW_BYTES
value: "8589934592"These entries belong to spec.roles[].env, not the source chart’s modelTransfer.env.
The all aliases are all_tp, all-tp and all_tps; invalid modes fail. Each session counts
against the source session cap. Broadcast uses torch.distributed.broadcast.
In all mode, each rank’s default window is
max(GPUSTACK_MODEL_TRANSFER_MAX_WINDOW_BYTES // tp_size, 1 GiB), aligned down to 2 MiB.
The per-rank minimum means the sum can exceed the configured budget for large TP groups.
The engine’s model_loader_extra_config accepts the following reader budget object.
This is the decoded engine option structure, separate from the ModelDeployment fields;
when supplying it through a command-line argument, encode the object as JSON:
# Explicit per-reader window; overrides the environment-derived window.
# This value is not divided by TP size. Omit it to use the default above.
window_bytes: 8589934592
# Example concurrency of 4; omit to use available CPU threads, capped at 256.
max_inflight: 4
# 4 GiB default ceiling for one owned output tensor.
owned_output_bytes: 4294967296The reader coalesces contiguous safetensors spans into a registered transfer window where possible. Window and concurrency budgets bound staging; retained output tensors still contribute to peak device memory until load completion.
The target staging window uses pinned host RAM. Ascend uses aclrtMallocHost and HIXL MEM_HOST;
NVIDIA uses pinned DRAM with NIXL. The source writes into the host window, then the loader copies
to the target tensor. GPUSTACK_MODEL_TRANSFER_TARGET_MEMORY is no longer read. The loaders close
the session after load_weights; unknown native outcomes retain resources for safety.
HIXL owned host regions always use 2 MiB-aligned addresses and registration lengths, including
small or oddly sized requests. The allocator reserves the rounded extent plus 2 MiB for address
alignment and frees the original allocation. File and tensor read/write bounds keep their original
lengths; the registered tail is zeroed and never counted as model data.
Source admission, page selection and residency reports charge this complete backing allocation.
Target window budgets round down before dividing into slots; each slot reserves 2 MiB for address
alignment. A direct loader window below 4 MiB resolves to that minimum, and double buffering needs
at least 8 MiB. Configured environment windows retain their 1 GiB floor. Adaptive transfer spans
may end at an unaligned tensor boundary without changing the aligned memory registration.
The exported HixlReader also aligns its owned host windows and charges the full backing
allocation. Its per-window budget has a 4 MiB minimum. Caller-owned tensor or byte-buffer
registrations keep their actual storage bounds; the runtime never rounds them beyond their owner.
NVIDIA pinned allocations of at least 2 MiB retain their existing address alignment. NIXL window
sizes round down and needed tensor spans round up before clamping; smaller windows can keep their
original sizes.
When available RAM backs a source window, POST /v1/sessions returns sourceWindowBytes. The
client limits its window to the smaller of its own budget and that figure, with a 256 MiB
adaptive floor and a further bound by the needed tensor span. Larger client windows transfer in
chunks. The response omits the source window if RAM is unreadable or cannot clear the floor.
When session capacity is exhausted, POST /v1/sessions returns HTTP 409, Retry-After: 5 and
"retryAfter": 5. The client tries other available peers for the same digest. If none admits the
session and a peer supplies a retry delay, it waits for the shortest delay and retries until the
startup deadline.
Credentials
The session authority is the replica’s projected token. The broker reviews its audience and bound Pod identity, checks the live Pod and current artifact grant, and binds the session to that owner and target process instance. Public discovery carries source Pod identities and CA bundles; it carries no serving private key. Source isolation remains unqualified.
Generation isolation and source immutability still require backend safety tests. The candidate uses source-initiated writes into target-owned staging windows and withholds source registration metadata at the control interface. That interface alone does not establish hardware isolation; raw READ, WRITE, ATOMIC and discovery exposure require backend safety qualification.
A drained or quarantined generation stops admitting sessions. An already loaded replica keeps serving its own loaded weights.
Benchmarks
Case scope
The benchmark covers Ascend/HIXL (Qwen3-8B, Qwen3.6-35B-A3B and Qwen3-235B-A22B-W8A8),
NVIDIA/NIXL (Qwen3-Next-80B-A3B-Instruct-FP8 and Qwen3-235B-A22B-Instruct-2507-FP8), and
NVIDIA/ModelExpress peer D2D with compile-cache sharing on Qwen3-235B-A22B-Instruct-2507-FP8.
The comparison evaluates a cold target node: TCP sync to local disk plus engine loading versus cross-node remote loading and peer D2D scale-out. SGLang retains functional coverage; the tables report vLLM and vLLM-Ascend results.
Measurement contract
The baseline sums TCP peer sync and local loading. Remote loading measures network transfer from prepared source RAM or a peer replica. Both exclude source preparation, scheduling and service warm-up; the baseline is a phase sum rather than a continuous startup trace.
Weight Transfer Time is the slowest rank’s model-loading time. Speedup is the baseline sum divided by the remote loading time. Because single runs can differ in cache residency or SDK build, ratios describe the measured configurations.
Source RAM and Target RAM report host transfer buffers, excluding device memory and other
process allocations. Target Storage is checkpoint disk usage on the target node; remote rows
report 0 GiB, omitting small metadata files and unrelated engine caches.
TP2, TP4 and TP8 use two, four and eight devices. In TP0 + broadcast, rank 0 reads remotely and
broadcasts tensors to the group; in TP all, every rank opens its own remote session. Single-model
remote runs cross nodes.
Ascend (HIXL)
The runs used two Ascend 910B2 nodes, vLLM-Ascend cann9.2-910b-vllm0.30.0, eager mode and host
transfer buffers with pre-warmed source weights. These runs used an isolated qualification setup;
HIXL remains outside the managed routing allowlist.
The source host had 2014 GiB of total RAM and approximately 1514 GiB available. With
source.maxMemoryBytes: 0, the default node Source budget is 2014 / 16 (125.88 GiB), subject
to available RAM and 10% reservation overhead.
HIXL defaults to a 16 GiB source window per generation and an 8 GiB target group budget; the Mode
column states (X GiB source, Y GiB target) for each remote row. Each reading rank (1 in
TP0 + broadcast, tp_size in TP all) caps its host window at the smaller of its per-rank budget
and one safetensors shard (3.72 GiB on 8B/35B, 4.00 GiB on 235B).
8B means Qwen/Qwen3-8B; 35B means Qwen/Qwen3.6-35B-A3B; and 235B means
vllm-ascend/Qwen3-235B-A22B-W8A8. The 35B and 235B runs used TP8 with expert parallelism (EP).
Remote times are the slowest rank’s load_model seconds.
| Model | Mode | Source RAM | Target RAM | Target Storage | Weight Transfer Time |
|---|---|---|---|---|---|
| 8B (TP2) | TCP peer sync + local load (sum) | 0 GiB | 0 GiB | 15.27 GiB | 82.90 s (baseline) |
| 8B (TP2) | TP0 + broadcast (16 GiB source, 8 GiB target) | 15.26 GiB | 3.72 GiB | 0 GiB | 15.31 s (5.41×, −81.5% time) |
| 8B (TP2) | TP all (16 GiB source, 8 GiB target) | 15.26 GiB | 7.45 GiB | 0 GiB | 10.78 s (7.69×, −87.0% time) |
| 35B (TP8 + EP) | TCP peer sync + local load (sum) | 0 GiB | 0 GiB | 66.99 GiB | 282.60 s (baseline) |
| 35B (TP8 + EP) | TP0 + broadcast (16 GiB source, 8 GiB target) | 15.66 GiB | 3.72 GiB | 0 GiB | 66.33 s (4.26×, −76.5% time) |
| 35B (TP8 + EP) | TP all (16 GiB source, 8 GiB target) | 15.66 GiB | 8.00 GiB | 0 GiB | 101.39 s (2.79×, −64.1% time) |
| 35B (TP8 + EP) | TP all (80 GiB source, 64 GiB target) | 67.05 GiB | 29.81 GiB | 0 GiB | 46.39 s (6.09×, −83.6% time) |
| 235B (TP8 + EP) | TCP peer sync + local load (sum) | 0 GiB | 0 GiB | 220.54 GiB | 599.05 s (baseline) |
| 235B (TP8 + EP) | TP0 + broadcast (16 GiB source, 8 GiB target) | 15.99 GiB | 4.00 GiB | 0 GiB | 452.83 s (1.32×, −24.4% time) |
| 235B (TP8 + EP) | TP0 + broadcast (256 GiB source, 32 GiB target) | 220.67 GiB | 4.00 GiB | 0 GiB | 253.58 s (2.36×, −57.7% time) |
| 235B (TP8 + EP) | TP all (256 GiB source, 32 GiB target) | 220.67 GiB | 32.00 GiB | 0 GiB | 232.93 s (2.57×, −61.1% time) |
When a checkpoint fits the 16 GiB source window (8B at 15.27 GiB), TP all (10.78 s, 7.69×)
beats TP0 + broadcast (15.31 s, 5.41×) by parallelizing rank transfers (7.45 GiB target RAM).
When the checkpoint exceeds the 16 GiB source window (35B and 235B), TP0 + broadcast streams
sequential shards with 3.72–4.00 GiB of target RAM (66.33 s on 35B, 452.83 s on 235B), whereas 35B
TP all slows to 101.39 s under cross-rank page contention.
Sizing the source window to hold the full model (80 GiB on 35B, 256 GiB on 235B) removes source
paging and makes TP all fastest again: 46.39 s (6.09×) on 35B and 232.93 s (2.57×) on 235B
(versus 253.58 s for full-resident broadcast; the 235B paged and full-resident samples used
different SDK builds). Detailed sweeps and lifecycle checks are in the
acceptance record
.
NVIDIA (NIXL)
The runs used Nebius H100 80 GB nodes and vLLM 0.30.0 in eager mode with pre-warmed source weights.
Every case below reached Ready and passed inference probes. The
NVIDIA evidence record
retains the tested artifacts and measurement limits.
The source host reported 1574.84 GiB of total RAM and approximately 1554.06 GiB available. With
source.maxMemoryBytes: 0, the default node Source budget is 1574.84 / 16 (98.43 GiB),
bounded by available RAM, Binding limits and the 10% reservation overhead.
Each remote row states (X GiB source, Y GiB target) in Mode. Each reading rank (1 in
TP0 + broadcast, tp_size in TP all) bounds its host window by the smaller of its per-rank
budget and one safetensors shard (10.00 GiB on 80B, 9.31 GiB on 235B).
80B means Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 (TP4); 235B means
Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 (TP8 with EP). Remote times are the slowest rank’s
Model loading took seconds.
| Model | Mode | Source RAM | Target RAM | Target Storage | Weight Transfer Time |
|---|---|---|---|---|---|
| 80B (TP4) | TCP peer sync + local load (sum) | 0 GiB | 0 GiB | 76.45 GiB | 393.14 s (baseline) |
| 80B (TP4) | TP0 + broadcast (16 GiB source, 8 GiB target) | 16.00 GiB | 8.00 GiB | 0 GiB | 160.59 s (2.45×, −59.2% time) |
| 80B (TP4) | TP all (16 GiB source, 8 GiB target) | 16.00 GiB | 8.00 GiB | 0 GiB | 1380.52 s (0.28×, +251.2% time) |
| 80B (TP4) | TP all (80 GiB source, 64 GiB target) | 76.42 GiB | 40.00 GiB | 0 GiB | 81.16 s (4.84×, −79.4% time) |
| 235B (TP8 + EP) | TCP peer sync + local load (sum) | 0 GiB | 0 GiB | 220.21 GiB | 1121.30 s (baseline) |
| 235B (TP8 + EP) | TP0 + broadcast (16 GiB source, 8 GiB target) | 16.00 GiB | 8.00 GiB | 0 GiB | 287.97 s (3.89×, −74.3% time) |
| 235B (TP8 + EP) | TP0 + broadcast (256 GiB source, 64 GiB target) | 220.19 GiB | 9.31 GiB | 0 GiB | 108.89 s (10.30×, −90.3% time) |
| 235B (TP8 + EP) | TP all (256 GiB source, 32 GiB target) | 220.19 GiB | 32.00 GiB | 0 GiB | 69.16 s (16.21×, −93.8% time) |
Under a 16 GiB source window (one resident shard at a time: 10.00 GiB on 80B, 9.31 GiB on 235B),
TP0 + broadcast streams shards sequentially in 160.59 s (2.45×) on 80B and 287.97 s (3.89×) on
235B using 8.00 GiB of target RAM. Under the same 16 GiB window, 80B TP all thrashes the
single-shard source window across four concurrent ranks (1380.52 s, 0.28×); use TP0 + broadcast
whenever source RAM cannot hold the full checkpoint.
Sizing the source window for full residency (80 GiB on 80B, 256 GiB on 235B) removes source
paging (108.89 s, 10.30× with TP0 + broadcast on 235B). Pairing full source residency with
TP all lets all ranks pull concurrently, reaching 81.16 s (4.84×, 40.00 GiB target RAM) on 80B
and 69.16 s (16.21×, 32.00 GiB target RAM) on 235B with zero target disk usage.
NVIDIA (ModelExpress)
ModelExpress lets vLLM and SGLang replicas share GPU tensors over NIXL device-to-device (D2D)
transfers and reuse compiled torch.compile and Triton caches across replicas. Hardware
qualification remains separate.
The metadata server discovers peers and cache manifests while NIXL transfers both GPU tensors (D2D)
and compiled cache archives between replicas. When no ready peer is available (seed replica or
after a metadata-server restart), GpustackLoadStrategy reads the ModelPrefetch source over NIXL
H2H and H2D. ModelExpress peer D2D over HIXL is outside this design.
The runs below measured 235B (TP8 + EP) (Qwen/Qwen3-235B-A22B-Instruct-2507-FP8) on two Nebius
H100 80 GB nodes with vLLM 0.30.0 in graph mode, comparing the NIXL TCP peer sync + local load
baseline against the seed replica, a scale-out peer replica, and a replica launched after a
metadata-server restart (detailed sweeps and verification logs are in the
ModelExpress benchmark record
):
| Model | Mode | Source RAM | Target RAM | Target Storage | Weight Transfer Time |
|---|---|---|---|---|---|
| 235B (TP8 + EP) | TCP peer sync + local load (sum) | 0 GiB | 0 GiB | 220.21 GiB | 1121.30 s (baseline) |
| 235B (TP8 + EP) | Seed replica: GpustackLoadStrategy (256 GiB source, 8 GiB target) |
220.19 GiB | 8.00 GiB | 0 GiB | 232.10 s (4.83×, −79.3% time) |
| 235B (TP8 + EP) | Peer replica: RdmaStrategy (NIXL D2D + compile cache) |
0 GiB | 0 GiB | 0 GiB | 16.78 s (66.82×, −98.5% time) |
| 235B (TP8 + EP) | Restart fallback: GpustackLoadStrategy (256 GiB source, 8 GiB target) |
220.19 GiB | 8.00 GiB | 0 GiB | 109.08 s (10.28×, −90.3% time) |
The seed replica (GpustackLoadStrategy) loads from the ModelPrefetch source in 232.10 s (4.83×
versus baseline), runs cold torch.compile (77.67 s) and CUDA Graph capture (112 s, 296.82 s total
engine init), and publishes GPU tensor descriptors plus 386.07 MiB of compiled caches
(torch.compile, Triton, DeepGEMM and FlashInfer) to the metadata server.
The scale-out peer replica (RdmaStrategy) streams pre-sharded weights directly between GPUs over
NIXL D2D in 16.78 s (66.82× versus baseline, 13.83× versus seed) with 0 GiB host transfer RAM
and 0 GiB target disk. Restoring the shared compile caches over NIXL cuts torch.compile from
77.67 s to 0.28 s (277.39×) and reduces total engine initialization from 296.82 s to 90.25 s (3.29×).
After a metadata-server restart empties in-memory peer registrations, a new replica falls back to
GpustackLoadStrategy (109.08 s from the resident 220.19 GiB source window, 10.28× versus
baseline) and republishes its descriptors once ready.
Role configuration
Select the load format on a ModelDeployment role. The example uses a Hugging Face artifact and a
cluster-specific runner image and accelerator type.
apiVersion: worker.gpustack.ai/v1
kind: ModelDeployment
metadata:
name: qwen-chat
namespace: team-a
spec:
model:
name: Qwen/Qwen3-8B # required
artifactRef:
name: qwen-weights # required for the prepared source
engine:
name: vLLM # required; vLLM or SGLang
image: example/engine:tag # required; use a compatible runner image
roles:
- name: server
replicas: 2
acceleratorType: gpustack--nvidia-a10g-linux-amd64
resources:
accelerator: 1
extraArgs:
- --load-format=modelexpress # required; mx is also accepted- Load format: Select
modelexpressormxin the role’s command or extra arguments. - Artifact: ModelExpress integration requires a Hugging Face artifact.
- Overrides: Role environment values override injected defaults (set
MX_ARTIFACT_TRANSFER: "0"to disable cache transfer). - Layout: Peer replicas need matching tensor-parallel and pipeline-parallel layouts.
The Operator injects these defaults into eligible role containers. This illustrative Pod excerpt shows the rendered environment. The actual Pod includes additional fields.
apiVersion: v1
kind: Pod
metadata:
name: qwen-chat-server
namespace: team-a
spec:
containers:
- name: server
image: example/engine:tag
env:
- name: MX_SERVER_ADDRESS
value: qwen-chat-modelexpress.team-a.svc:8001 # metadata Service
- name: MX_WORKER_HOST
valueFrom:
fieldRef:
fieldPath: status.podIP # replica Pod IP
- name: MX_ARTIFACT_TRANSFER
value: "1" # enable artifact transfer
- name: MX_ARTIFACT_COMPILE_CONFIG_DIGEST
value: <role-digest> # compile-cache identity, when available
- name: MX_ARTIFACT_READY_URL
value: http://127.0.0.1:8000/health # when engine health is available
- name: MX_LOAD_PLUGIN
value: gpustack # GPUStack loading strategy
- name: MX_LOAD_PLUGIN_BEFORE
value: server-cache # strategy insertion point- Server address: Long deployment names produce a shortened Service name.
- Digest: The Operator omits the digest when no compile-cache identity is available.
- Health URL: The Operator injects a URL only when it can grade the engine and its port is known.
Strategy and policy
| Value | Meaning | Constraints |
|---|---|---|
RdmaStrategy |
Loads tensors from a ready peer over NIXL D2D. | Tried first; peer layouts must match. |
GpustackLoadStrategy |
Loads from a ready ModelPrefetch source. |
Tried after peer delivery. |
Other cache and DefaultStrategy |
Load from eligible local cache or engine volume. | Available under Prefer. |
Require |
Allows peer and prefetch source delivery. | Failure of both permitted strategies is terminal; no local checkpoint fallback. |
Prefer |
Allows peer, prefetch source and eligible local delivery. | Local strategies follow the remote strategies. |
Tensor-parallel ranks select RdmaStrategy independently before the source strategy. One rank can
use a peer while another uses the source. After a source or local load, a replica publishes tensor
descriptors and compile-cache manifests for later replicas.
Metadata server
The Operator creates an in-memory gRPC server and Service for each deployment with a ModelExpress
role. Both use the name <md-name>-modelexpress, shortened for long names. The server is an
unauthenticated rendezvous service in the deployment namespace. Restrict its network access to
trusted Pods.
The worker Setting selects the server image. The Operator exposes it in the system namespace:
apiVersion: gpustack.ai/v1
kind: Setting
metadata:
name: model-deployment-modelexpress-server-image
namespace: gpustack-system # use the system namespace
spec:
value: gpustack/mirrored-modelexpress-server:v0.7.0 # default image- Image: Use the Setting for registry redirection or image pinning.
- Restart: Server replacement loses in-memory registrations; existing replicas keep serving.
- Recovery: A new replica loads from a ready source or an eligible local path and publishes metadata.
- Scale from zero: The first replica has no ready peer.
Require and Prefer
The policy names what an eligible replica may do when its source is missing, slow or fails:
| Policy | Allowlisted, source available | Allowlisted, source unavailable | Outside allowlist | Ineligible role |
|---|---|---|---|---|
Require |
loads remote | holds before its load starts, visible in status, never downloads | blocked with UnqualifiedTransferBackend, no new Pod |
blocked with RoleIneligibleForModelTransfer, no new Pod |
Prefer |
loads remote | uses the declared local delivery if no remote replica holds it; otherwise holds remote delivery | falls back to the declared local delivery, or is blocked when that fallback would roll a loaded remote replica | falls back to the declared local delivery |
- Require never materializes a checkpoint. A Require deployment that cannot reach a source keeps waiting within its startup limits; nothing downloads behind its back.
- Prefer tracks fallback attempts. The initial remote
Poddoes not mount a complete checkpoint, so fallback after a failed remote load recreates the replica. Each attempt incrementsfallbackAttemptsfor that replica. If the count exceeds three, the reason becomesFallbackAttemptsSpentand the replica still falls back. - Verified artifacts keep verification on fallback. An artifact with
expectedDigest,allowPatternsorignorePatternsfalls back to verified Node delivery, never to an unrestricted engine download. - The refusal reason identifies the restriction.
RoleIneligibleForModelTransfermeans the role cannot use remote delivery: it has its own command, uses an unsupported engine, or runs on a pool whose manufacturer names no backend.UnqualifiedTransferBackendmeans the candidate is outside the current routing allowlist. UnderRequireboth block; underPreferan unqualified candidate falls back only if doing so would not roll a loaded remote replica. - A running remote replica is protected from fallback. Changing the desired delivery could
roll a replica that already loaded. The controller checks running
Pods as well as deployment status, which can lag or report local delivery. It holds the operator-rendered replica’s remote delivery even if the source disappears or thePodis notReady: it may already have loaded. A replica can fall back when every member’s load failed outright. This hold does not affect an independent client that connects to the source itself.
Networks
Control and payload endpoints are identified separately. The control endpoint is the broker’s
authenticated HTTP path for session negotiation. The payload endpoint is the backend’s transfer path:
UCX/InfiniBand or shared memory for nixl, and the CANN fabric for hixl. A capability record
must identify both endpoints for the backend to qualify.
Network device selection is checked during qualification, using node discovery or the acceptance
configuration. No Setting chooses a NIC today. A payload transfer on the wrong network path
fails acceptance; the operator does not automatically correct the selection.
The source chart keeps the existing modelTransfer.env map and appended modelTransfer.extraArgs.
Use backend environment variables there when a server needs explicit device selection. Client
east-west network selection is owned by the RDMA selection work; this feature adds no server-client
NIC negotiation. HIXL Device RoCE and Host RoCE require separate qualification.
Observing status
The deployment reports remote delivery per role and replica in status.model.remote, beside
the ordinary conditions:
policyechoes the effective policy, so a status reader never guesses which behavior produced the entries.roles[].replicas[]carriesdelivery(Remote,NodeorEngine),load,reasonandfallbackAttemptsper ordinal. A deployment can serve mixed deliveries; the aggregate never flattens them.loadwalksUnknown,AwaitingMetadata,AwaitingSource,Loading,Loaded.Loadedmeans weights confirmed loaded and the engine initialized, and a loaded replica keepsLoadedwhen its source later drains.WeightsReadyunder remote delivery reflects confirmed loading for the applicable replicas — metadata readiness and source eligibility permit startup but are not serving proof.
kubectl get modeldeployment <name> -o jsonpath='{.status.model.remote}' | jqSource eligibility uses the owning prefetch’s current serving observation
.
Lease expiry schedules a new deployment observation; progress counters and ordinary lease renewals
alone do not enqueue every artifact consumer. Endpoint discovery still checks source Pod and
DaemonSet identity. These observations grant no broker authorization.
Troubleshooting
Start from status.model.remote, then the WeightsReady condition and the replica Pod:
| Observation | Meaning | Check |
|---|---|---|
Progressing=True, reason: SourceWindowFallback on the prefetch |
the grant cannot back the minimum window under Prefer |
increase the Binding grant; check local delivery |
Progressing=False, reason: SourceWindowBelowFloor on the prefetch |
the grant cannot back the minimum window under Require |
increase the Binding grant; check source capacity |
Progressing=False, reason: SourceGrantMissing on the prefetch |
the referenced Binding is missing; the condition message may still mention its source-memory section | restore the ModelStoreBinding named by spec.bindingRef |
load: AwaitingSource, remote delivery |
no prepared source serves the digest yet | the prefetch’s status.transfer; the nodes’ NodeModelStore status.modelTransfer generations |
load: AwaitingMetadata |
the metadata view is not mounted yet | the Pod’s FailedMount events; the plugin’s CSIDriver |
load: Unknown, reason: RemoteLoadFailed |
a load attempt ended; the replica is being recreated | the Pod’s terminated container exit; fallbackAttempts |
reason: FallbackAttemptsSpent |
the replica’s fallback attempt count exceeded three | the fallback delivery’s own failure: the Node or Engine path it landed on |
reason: RoleIneligibleForModelTransfer, no new Pod under Require |
the role names no transfer backend at all | the eligibility rules under Configuration
, or choose Prefer |
reason: UnqualifiedTransferBackend, no new Pod under Require |
the candidate is outside the controller’s routing allowlist | the engine and backend restrictions under Configuration |
WeightsReady=False, reason: NodeDeliveryUnavailable, under Require |
the effective policy is Require and the CSIDriver model.csi.gpustack.ai is absent |
the CSIDriver exists; a Setting edit to Require or Prefer without it is rejected at admission |
Source loss does not change replicas that already loaded. Replicas still loading can wait in
AwaitingSource. Under Prefer, a replica can fall back if its load fails outright.
Rollback
To roll back, patch the model-artifact-remote-delivery-policy Setting to Disabled (or set
modelTransfer.enabled: false in chart values). New replicas use the base
delivery mode (model-artifact-delivery-mode, Node or Engine).
Replicas already loaded keep serving their weights until their next roll. A checkpoint that
Require refused to materialize was not written to the target; rollback does not migrate files
on disk.