Vendor Prerequisites
Install the manufacturer’s driver and any required container toolkit before GPUStack. If a vendor GPU Operator is installed, disable components that conflict with GPUStack before sharing the node. Vendor keys and component names depend on the release. Check the cited artifact and version for each manufacturer.
Contents
Third-party dependencies
Check these dependencies before installation and when selecting a workload configuration. Installation options and ownership are in Installation Modes .
| Dependency | Role | Configuration |
|---|---|---|
| Node Feature Discovery (NFD) | Labels nodes and publishes hardware features. | Run one NFD installation per cluster. See Single NFD per cluster . |
| Kueue | Manages workload admission and quota. | The chart includes Kueue; see Installation Modes . |
| Topograph | Supplies topology inventory from a configured provider. | Optional and disabled by default. See Topology Operations . |
| Mooncake | Supplies cache services and transfer support for compatible engines. | Optional. Match the engine, client, store and transport using Engine Compatibility . |
| ModelExpress | Coordinates NVIDIA peer tensor transfer and compile-cache sharing. | Optional; the operator creates a per-deployment metadata service. See ModelExpress . |
| CSI drivers | Mount NFS or S3 volumes, or deliver cached model weights. | NFS and S3 drivers are bundled subcharts. Node model delivery uses the Model Manager CSI driver. See Model Store Operations . |
| Serving engines | Run inference workloads. | Choose a compatible engine image and configuration. See Engine Compatibility . |
| Vendor drivers and toolkits | Provide accelerator access on each node. | Install the driver and any required toolkit. See the manufacturer guides below. |
Manufacturer guides
AMD
Read against AMD GPU Operator v1.5.1 — Overview · Helm Chart Install · Toolkit Install .
Install before GPUStack — the ROCm driver alone. Per Allocator
injection
, GPUStack’s allocator injects /dev/kfd and each granted accelerator’s
DRM nodes itself, so neither the AMD Container Toolkit nor amd-container-runtime is needed.
A non-root workload still needs video and render supplementary-group membership on its container’s
user: the device plugin API has no field to grant a supplementary group, so this is the workload’s or
the cluster’s own securityContext to set. Measured, the failure mode without it is zero enumerated
ROCm agents, not an error: rocm-smi reads sysfs and keeps reporting the accelerator regardless, so
it is not a valid check here; use rocminfo instead.
Vendor GPU Operator — the AMD GPU Operator, driven by a DeviceConfig CR the chart creates.
Keep the driver and metrics exporter. Disable the vendor device plugin, node labeller and bundled
NFD. Disable Kernel Module Management when the driver is pre-installed. To test whole-card allocation,
leave the DRA driver enabled; it is mutually exclusive with the device plugin on one DeviceConfig.
apiVersion: amd.com/v1alpha1
kind: DeviceConfig
metadata:
name: default
namespace: kube-amd-gpu
spec:
driver:
enable: true # Default.
metricsExporter:
enable: true # Enable; default false.
devicePlugin:
enableDevicePlugin: false
enableNodeLabeller: false
draDriver:
enable: true # Alternative to the device plugin.node-feature-discovery:
enabled: false
kmm:
enabled: false # When the driver is pre-installed.
watch: falseInstall it — the chart’s own switches cover the two bundled add-ons; the device plugin and the node
labeller are DeviceConfig fields, so they are turned off on the CR the chart just created.
helm repo add rocm https://rocm.github.io/gpu-operator
helm repo update
helm install amd-gpu-operator rocm/gpu-operator-charts \
--namespace kube-amd-gpu --create-namespace \
--version=v1.5.1 \
--set node-feature-discovery.enabled=false \
--set kmm.enabled=false \
--set kmm.watch=false
kubectl -n kube-amd-gpu patch deviceconfig <name> --type merge -p \
'{"spec":{"devicePlugin":{"enableDevicePlugin":false,"enableNodeLabeller":false},"metricsExporter":{"enable":true}}}'The patch above makes the following change to the chart-created DeviceConfig. Replace the name
with that of the existing object:
apiVersion: amd.com/v1alpha1
kind: DeviceConfig
metadata:
name: <name>
namespace: kube-amd-gpu
spec:
devicePlugin:
- enableDevicePlugin: true
- enableNodeLabeller: true
+ enableDevicePlugin: false
+ enableNodeLabeller: false
metricsExporter:
- enable: false
+ enable: true
Already installed — the vendor plugin and GPUStack’s plugin both advertise amd.com/gpu.
Stop the vendor plugin before installing GPUStack on that node:
- Device plugin and
NodeLabeller — turn both off on the existingDeviceConfig, the same patch as above against the CR the running release created. Do not delete theDaemonSet: the controller recreates whatever the CR still asks for. - Bundled Node Feature Discovery — if that release brought its own,
helm upgradeit with the values it was installed with plus--set node-feature-discovery.enabled=false. - Verify — wait for
amd.com/gputo leave theNodebefore installing GPUStack;kubectl get node <node> -o jsonpath='{.status.allocatable}'no longer lists it. The kubelet drops it a moment after the plugin goes, not at the same instant.
Ascend
Read against the Ascend NPU driver and firmware package —
Driver Install
— and, for the Atlas A5
statements below, ascend-docker-runtime
26.1.0, the release carried by the A5 host they were checked on.
Install before GPUStack — the Ascend NPU driver and firmware, and the Ascend Docker Runtime: per
Allocator injection
, GPUStack’s allocator emits ASCEND_VISIBLE_DEVICES and no
device node, so that runtime is what turns an allocation into /dev entries.
Register the runtime under the handler name ascend, which is the RuntimeClass this chart creates for
this manufacturer, or set deviceManager.createRuntimeClasses=false and attach your own.
Install ascend-docker-runtime
newer than 26.0.x on Atlas A5 (the 950 series). Basic A5 support landed in 26.0.0, whose notes
carry the line “Ascend Docker Runtime supports the Atlas 350 PCIe card” (this generation’s PCIe
card). 910B and earlier are unaffected: their line ends at 7.3.x, and releases run 5.x, 6.x,
7.x, then 26.x with nothing in between.
An older runtime refuses the allocation, and blames the wrong thing. It maps the Ascend950 chip
name to no device type it knows, so it falls back to the manager-device list every earlier generation
used — and that list carries /dev/devmm_svm, which an A5 host does not have. Adding a device node
that is not there is fatal, so container creation fails, naming that device rather than the version
behind it.
The cards are not the gap: /dev/davinci<N> is built from the requested index with no chip name
involved, so an older runtime would have injected them, but it never gets that far.
ascend-docker-runtime --version does not answer this. It is a runc wrapper that hands every
unrecognized argument to the host’s own runc, so --version prints runc’s banner and says nothing
about the vendor release. Read the version the installer recorded instead:
grep '^version=' /usr/local/Ascend/Ascend-Docker-Runtime/ascend_docker_runtime_install.info
# version=v7.3.0 <- predates A5 support
# version=v26.1.0 <- serves A5gpustack-operator device-manager preflight reads the same file and reports it as the
ascend-docker-runtime check
on each A5 accelerator,
so a node too old to serve them is named before a workload lands on it.
A5 host ranktable requirement
The host ranktable must be absent or version 2.0. /etc/hccl_rootinfo.json is optional host state
owned by no package. Install
ascend-docker-runtime
newer than
26.0.x (prefer 26.2+ so the runtime no longer mounts /etc/hccl_rootinfo.json unconditionally).
Releases through 26.1.x mount /etc/hccl_rootinfo.json into every container on presence alone
with no version check. Releases 26.2+ removed that default mount, and neither release mounts
/lib/route.conf.
Its format is fixed by the chip generation: 1.0 serves A2 (910/910B/310P), 1.2 serves A3
(910C), 2.0 serves A5. A5 loads libhccl_v2, which refuses anything but 2.0:
Config_Error_Ranktable(EI0014): Value 1.0 for ranktable variable version is invalid.
Expected value: 2.0.So a table left behind by an older fleet stops every multi-card job on the node: single-card allocations never build a communicator and are unaffected. No environment variable disables the check, and the path is compiled into the library, so the file must be corrected rather than worked around:
# absent is the healthy state -- HCCL then negotiates through rootInfo
test -e /etc/hccl_rootinfo.json && grep -o '"version"[^,]*' /etc/hccl_rootinfo.json
# "version": "1.0" <- breaks multi-card HCCL on A5; remove it or regenerate as 2.0gpustack-operator device-manager preflight reports this as the hccl-ranktable
check
on each A5 accelerator.
A5 conditional host mounts
On Ascend A5 (AcceleratorType status.detail.family equal to "950"), device-manager
inspects host files through /host at Allocate time and conditionally mounts them read-only. This
supports SGLang MemFabric (device_urma) and legacy multi-node UB HCCL across
ascend-docker-runtime
versions without breaking nodes that omit those files.
| Host file | Mount condition | Skip behavior |
|---|---|---|
/etc/hccl_rootinfo.json |
Regular non-symlink file (<= 1 MiB) containing JSON "version": "2.0" |
Absent, symlinked, oversized, or non-2.0 files are skipped with a diagnostic log; allocation still succeeds |
/lib/route.conf |
Regular non-symlink file; resolves merged-usr /lib → /usr/lib inside /host |
Absent, symlinked, or non-regular files are skipped with a diagnostic log; allocation still succeeds |
- Non-A5 generations: Ascend
910,910B,910C,310P, and310Bnever receive either mount. - Collision handling: Containers already declaring a mount at the same path receive no duplicate mount.
- Unavailable host root: Disabling
deviceManager.hostRootMount.enabledskips both mounts without failing allocations.
A5 Host RoCE endpoints
For Ascend A5 Host RoCE (roce:host) vLLM workloads, provision per-NPU endpoint JSON files at
/etc/hixlep/ub_endpoint_npu_<physical_id>.json on each Host RoCE node. Operator-managed Host RoCE
Pods mount /etc/hixlep read-only with HostPathDirectory.
A missing /etc/hixlep directory fails kubelet volume setup immediately, and a missing endpoint
file causes vllm-ascend startup to fail with FileNotFoundError. See Ascend A5
topologies
for workload settings.
Cambricon
Read against Cambricon driver 5.10.22 (Neuware SDK 1.15.0) — Driver Install .
Install before GPUStack — the Cambricon MLU driver alone. Per Allocator
injection
, GPUStack’s allocator injects the card’s own node and the host’s control
nodes on both the whole-card and the sliced path, so cntoolkit is not required. It still sets the
CAMBRICON_VISIBLE_DEVICES a deployment running cambricon-container-runtime keys on, naming the
same cards the injected nodes do.
Hygon
Read against the Hygon DCU Operator — its documentation tree publishes only a latest revision,
with no pinned semantic version —
Overview
.
Install before GPUStack — the Hygon DCU driver and firmware, with the device files (/dev/kfd,
/dev/mkfd, /dev/dri/card*, /dev/dri/renderD*) visible on the node. Per Allocator
injection
, GPUStack’s allocator injects those nodes directly, and the DCU Operator
ships no container runtime of its own, so no container toolkit is required.
Vendor GPU Operator — the DCU Operator, driven by a single DeviceConfig CR (kubectl explain deviceconfig.spec); every component below is a field under that one CR, not a separate CRD.
Keep DCU-Exporter and DCU-DCGM; disable DCU-Device-Plugin and DCU-Label-Node.
Replace the API version and name with those of the installed DeviceConfig:
apiVersion: <existing-api-version>
kind: DeviceConfig
metadata:
name: device-config
namespace: kube-system
spec:
metricsExporter:
enable: true # Disabled by default.
dcgm:
enable: true # Disabled by default.
devicePlugin:
enableDevicePlugin: false
enableNodeLabeller: falseInstall it — the vendor’s install chapter sits behind its authenticated documentation portal, so no
install command is reproduced here. Whatever installs the operator, turn the two components off on the
DeviceConfig it creates.
kubectl -n kube-system patch deviceconfig <name> --type merge -p \
'{"spec":{"devicePlugin":{"enableDevicePlugin":false,"enableNodeLabeller":false},"metricsExporter":{"enable":true},"dcgm":{"enable":true}}}'The patch above updates the existing DeviceConfig. Read its apiVersion and name from
kubectl get deviceconfig -n kube-system -o yaml and substitute them in the header:
apiVersion: <existing-api-version>
kind: DeviceConfig
metadata:
name: <name>
namespace: kube-system
spec:
devicePlugin:
- enableDevicePlugin: <current-value>
+ enableDevicePlugin: false
- enableNodeLabeller: <current-value>
+ enableNodeLabeller: false
metricsExporter:
- enable: <current-value>
+ enable: true
dcgm:
- enable: <current-value>
+ enable: true
Already installed — the vendor plugin and GPUStack’s plugin both advertise hygon.com/dcu.
Stop the vendor plugin before installing GPUStack on that node:
- Device plugin and
NodeLabeller — turnenableDevicePluginandenableNodeLabelleroff on the existingDeviceConfig, with the patch above against the CR the running release created. Do not delete theDaemonSets: the controller recreates whatever the CR still asks for. - Verify — wait for
hygon.com/dcuto leave theNodebefore installing GPUStack.
Iluvatar
Read against ix-GPU-Operator v4.4.0 and ix-Container-Toolkit v1.1.0 — Helm Chart Install · Toolkit Install .
Install before GPUStack — the Iluvatar driver and the ix-Container-Toolkit: per Allocator
injection
, GPUStack’s allocator emits IX_VISIBLE_DEVICES and no device node. The
two are paired by version — a v4.4.0 driver needs the v1.1.0 toolkit, and the vendor states the two
must match on host and in-container.
Register the runtime under the handler name iluvatar, which is the RuntimeClass this chart creates
for this manufacturer, or set deviceManager.createRuntimeClasses=false and attach your own.
Vendor GPU Operator — the ix-GPU-Operator, a Helm chart with one switch per component.
Keep ix-Exporter. Disable ix-Device-Plugin and ix-Feature-Discovery, which bundles upstream NFD. Disable ix-driver when the driver is pre-installed:
ixDriver:
enabled: false # Already the default.
ixExporter:
enabled: true # Default.
ixDevicePlugin:
enabled: false
ixfd:
enabled: falseInstall it — the OCI chart, with the driver left to the node and the two competing components off.
helm install --wait ix-gpu-operator \
oci://registry.iluvatar.com.cn:10443/k8s/ix-gpu-operator --version 1.0.0 \
--create-namespace --namespace ix-gpu-operator \
--insecure-skip-tls-verify \
--set global.repository="registry.iluvatar.com.cn/k8s" \
--set global.version=4.4.0 \
--set ixDriver.enabled=false \
--set ixDevicePlugin.enabled=false \
--set ixfd.enabled=falseAlready installed — ix-Device-Plugin and GPUStack’s plugin both advertise iluvatar.com/gpu.
Stop ix-Device-Plugin before installing GPUStack on that node:
- ix-Device-Plugin and ix-Feature-Discovery —
helm upgradethe release with the values it was installed with plus--set ixDevicePlugin.enabled=false --set ixfd.enabled=false. The operator removes both workloads itself; deleting one by hand only makes it come back on the next reconcile. - Verify — wait for
iluvatar.com/gputo leave theNodebefore installing GPUStack.
Below software-stack v4.3.0 the operator is deployed from YAML manifests with no per-component toggle. There the only lever is the whole manifest: delete it, and install the driver and ix-Container-Toolkit on the node yourself.
MetaX
Read against MetaX Kubernetes GPU Operator v0.15.3 — Overview · Helm Chart Install · Components .
Install before GPUStack — the MetaX driver and the MXMACA SDK. Per Allocator
injection
, GPUStack’s allocator injects the three control nodes (/dev/mxcd,
/dev/mxnd, /dev/mxgd) and each granted accelerator’s DRM nodes directly, so no container toolkit
is required.
Vendor GPU Operator — the MetaX GPU Operator. Its minimalMode is the install shape that fits
here: it deploys neither the kernel driver nor the MXMACA SDK, leaving both to the node.
| Component | Keep / disable | Mechanism |
|---|---|---|
| Kernel driver and MXMACA SDK | Not deployed | --set minimalMode=true leaves both to the node |
mx-exporter (metrics) |
Keep | optional under minimalMode |
controller-exporter |
Keep | optional under minimalMode |
gpu-device (device plugin) |
Cannot be disabled | the vendor marks it mandatory in every install mode, minimalMode included |
gpu-label (NFD-equivalent) |
Cannot be disabled | same marking |
gpu-scheduler |
Disable | optional — leave it undeployed |
topo-discovery |
Disable | optional — leave it undeployed |
Install it — substitute your own registry for DOMAIN/PROJECT. minimalMode is fixed at install
time and cannot be switched on a running release.
helm install oci://DOMAIN/PROJECT/metax-operator \
--version VERSION \
--create-namespace -n metax-operator \
--generate-name \
--wait \
--set registry=DOMAIN/PROJECT \
--set minimalMode=trueAlready installed — gpu-device and GPUStack’s plugin both advertise metax-tech.com/gpu.
The vendor requires gpu-device in every install mode, so the plugins cannot share a node.
Uninstall the vendor operator before installing GPUStack:
- Kernel driver and MXMACA SDK — make sure both are on the node independently of the operator.
Under
minimalModethey already are; otherwise install them first, or uninstalling takes them with it. - The whole release —
helm uninstallit.gpu-deviceandgpu-labelhave no switch, so there is nothing narrower to turn off. - Verify — wait for
metax-tech.com/gputo leave theNodebefore installing GPUStack.
Moore Threads
Read against MT GPU Operator v2.1.0 (KUAE Cloud Native) — Overview · Helm Chart Install .
Install before GPUStack — the Moore Threads driver and the MT Container Toolkit: per Allocator
injection
, GPUStack’s allocator emits MTHREADS_VISIBLE_DEVICES and no device node.
Both can come from the operator’s own full mode below.
Vendor GPU Operator — the MT GPU Operator, installed in full or core mode rather than by a single
toggle. full is the mt-gpu-operator Helm chart pair; core is a kubectl apply -f of plain
manifests carrying only the device plugin and the exporter.
Keep the driver, container toolkit, DCGM and its exporter. Disable NFD, the sGPU scheduler and
webhook, kured, node-problem-detector and aiOps. In full mode the device plugin ships
inside the operator’s reconciliation loop and has no disabling switch.
mtGpuClusterPolicyClusterpolicy:
driverToolkit:
enabled: true # Default.
containerToolkit:
enabled: true # Default.
dcgm:
enabled: true
dcgmExporter:
enabled: true
nfd:
enabled: false
gpuScheduler:
enabled: false # Default.
gpuWebhook:
enabled: false # Default.
kured:
enabled: false
nodeProblemDetector:
enabled: false
aiOps:
enabled: falseInstall it — from the chart directories the vendor’s bundle unpacks, with the cluster-level add-ons
off. No key disables the device plugin under full mode, so a node that must run GPUStack’s device plugin is
served by core mode with that manifest’s device plugin removed.
helm install --wait mt-gpu-operator mt-gpu-operator
helm install --wait mt-gpu-operator-custom-resources mt-gpu-operator-custom-resources \
--set mtGpuClusterPolicyClusterpolicy.nfd.enabled=false \
--set mtGpuClusterPolicyClusterpolicy.kured.enabled=false \
--set mtGpuClusterPolicyClusterpolicy.nodeProblemDetector.enabled=false \
--set mtGpuClusterPolicyClusterpolicy.aiOps.enabled=falseAlready installed — the vendor plugin and GPUStack’s plugin both advertise mthreads.com/gpu.
Stop the vendor plugin before installing GPUStack on that node.
The full mode has no switch to disable it; follow the procedure for the installed mode:
- Cluster-level add-ons (bundled NFD,
kured, node-problem-detector,aiOps) — underfull,helm upgradethemt-gpu-operator-custom-resourcesrelease with the switches above.coreships none of them. - Device plugin — no key removes it under
full, so the mode that installed it decides:full— either leave that node to the vendor stack, or move it tocore: uninstall both charts, keep the driver and MT Container Toolkit on the node by installing them outside the operator, and go on below.core— there is no controller, so edit the manifests directly: drop the device plugin fromdeployments.yaml, leave the exporter, and re-apply.
- Verify — wait for
mthreads.com/gputo leave theNodebefore installing GPUStack.
NVIDIA
Read against NVIDIA GPU Operator v26.3.3 — Overview · Helm Chart Install .
Install before GPUStack — the NVIDIA driver and the NVIDIA Container Toolkit: per Allocator
injection
, GPUStack’s allocator emits NVIDIA_VISIBLE_DEVICES by default and no
device node. The toolkit is what the other channel needs too — it is what writes the node’s CDI
specifications. Both can come from the operator itself.
The toolkit is also what MIG needs, not only workloads: the Device Manager Pod reads the management
library through it, and the chart declares NVIDIA_MIG_CONFIG_DEVICES / NVIDIA_MIG_MONITOR_DEVICES on
that Pod so the runtime stops hiding the driver’s MIG capabilities from it — see
MIG prerequisites
for the two ways to break that.
Vendor GPU Operator — the NVIDIA GPU Operator, a Helm chart with one switch per component.
Keep the driver, toolkit, exporter and GPU Feature Discovery. Disable the device plugin, MIG Manager and CDI. Disable bundled NFD or hand the job over and disable GPUStack’s bundled NFD instead.
driver:
enabled: true # Default.
toolkit:
enabled: true # Default.
dcgmExporter:
enabled: true # Default.
devicePlugin:
enabled: false
gfd:
enabled: true # Default.
nfd:
enabled: false
migManager:
enabled: false
cdi:
enabled: false # Default true from v25.10.MIG Manager changes geometry from node labels and conflicts with Allocate-time MIG actuation. One profile is an exception; see NVIDIA MIG .
Check the node is not opted out. Some GPU distributions ship their nodes already labelled
nvidia.com/gpu.deploy.operands=false, which tells this operator to place nothing on them at all. Its
ClusterPolicy still reports ready, so the only symptom is that no operand ever appears. Set it to
true on the nodes you want managed.
Install it — the driver, toolkit, exporter and feature discovery stay, the three competing components go.
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install --wait --generate-name \
-n gpu-operator --create-namespace \
nvidia/gpu-operator --version=v26.3.3 \
--set devicePlugin.enabled=false \
--set nfd.enabled=false \
--set migManager.enabled=false \
--set cdi.enabled=falseAlready installed — the vendor plugin and GPUStack’s plugin both advertise nvidia.com/gpu.
The vendor’s MIG Manager also reconfigures hardware that GPUStack’s allocator manages.
Stop both vendor components before installing GPUStack on that node.
One helm upgrade, with the values the release was installed with plus the three switches below. The
operator removes those DaemonSets itself; deleting one by hand only makes it come back, because the
ClusterPolicy still asks for it.
- Device plugin — set
--set devicePlugin.enabled=false. - MIG Manager — set
--set migManager.enabled=false. The node retains its existing geometry. GPUStack’s allocator reconfigures it during subsequentAllocatecalls. - Bundled Node Feature Discovery —
--set nfd.enabled=false. GPUStack’s bundled NFD publishes thesystem-os_release.*labels that release had been publishing, so nothing has to be carried over by hand. Keeping this one and disabling GPUStack’s bundled NFD is the other way round — One NFD per cluster . - Driver, container toolkit, DCGM exporter, GPU Feature Discovery — leave running. GPUStack needs the first two.
- Verify — wait for
nvidia.com/gputo leave theNodebefore installing GPUStack.
CDI mode. --set cdi.enabled=false is the narrower choice for a node this operator’s allocator
must control end to end. Leaving it on is workable.
Why — GPU Operator >= 25.10 defaults
cdi.enabled: true, and the oldercdi.defaultfield is deprecated and has no effect oncecdi.enabledexists. With CDI on, the toolkit resolves a bare accelerator UUID against the node’s specifications, and a name it cannot resolve fails container creation loudly rather than silently — which is what makes leaving it on workable rather than wrong.
A node serving MIG wants CDI off. GPUStack’s allocation path is safe whatever the toolkit does: a partition allocation is always carried by the variable, never by a CDI request.
Why — in
cdimode the runtime resolves whateverNVIDIA_VISIBLE_DEVICESholds against the node’s specifications, and the MIG instance GPUStack’s allocator materializes atAllocatetime is named in none of them, so a partition request has nothing to resolve against.
Device injection channel. Prefer auto, set through
GPUSTACK_NVIDIA_DEVICE_INJECTION_STRATEGY
, over naming a
channel yourself. It answers per container, independently of the switch above.
Why — two channels exist, and which one a node uses is what decides whether it needs the vendor runtime at all: the default names the accelerator in
NVIDIA_VISIBLE_DEVICES, which onlynvidia-container-runtimereads, while the other requests it as a CDI device the container engine resolves by itself under plainrunc. Whether a node can use the second is a property of its container engine, which is whatautoreads and a human would otherwise have to. It uses CDI only where the vendor’s own specifications already name the exact accelerator being granted, and writes no specification of its own. Device Discovery has the mechanism, and Settings what it checks, when it declines, and the one engine that has to be told.
auto leaves a Pod that names a runtimeClassName to that handler, whose configuration it cannot
read — correctly, since the handler is what injects. Whether that covers the Instances this operator
creates depends on the class existing: the worker attaches runtimeClassName: nvidia only where it
finds one, so with deviceManager.createRuntimeClasses=false and none supplied, an Instance runs
under the engine’s default handler and auto answers for it like any other Pod.
T-Head
Read against the T-Head SAIL SDK (also called the PPU SDK), HGGCRT v3 — Driver Install · Overview .
Install before GPUStack — the PPU SDK alone. It ships as a self-extracting Runfile from T-Head’s
download centre, installs to /usr/local/PPU_SDK by default, and is verified with ppu-smi (which
prints the driver version) and hgcc --version. Take HGGCRT v3 rather than v2: it is backward
compatible with the v2 API.
Per Allocator injection
, GPUStack’s allocator injects T-Head accelerators as /dev
nodes directly, which is the mechanism the vendor’s own container guide uses — it isolates a
container by passing those nodes to docker run --device, with no vendor runtime in the path. Which
nodes a container receives is in Device Discovery
.
Shared coexistence rules
Allocator injection
Whether a container toolkit is required at all follows directly from how GPUStack’s allocator makes the
accelerator visible: a /dev node needs nothing from the toolkit, while an env var needs the vendor’s
runtime hook to interpret it.
| Manufacturer | Allocator injection | Toolkit required |
|---|---|---|
| AMD | /dev nodes (/dev/kfd, /dev/dri/card*, /dev/dri/renderD*) |
No |
| Ascend | ASCEND_VISIBLE_DEVICES |
Yes |
| Cambricon | /dev nodes (/dev/cambricon_dev*, the control nodes), plus CAMBRICON_VISIBLE_DEVICES |
No |
| Hygon | /dev nodes (/dev/kfd, /dev/mkfd, /dev/dri/card*, /dev/dri/renderD*) |
No |
| Iluvatar | IX_VISIBLE_DEVICES |
Yes |
| MetaX | /dev nodes (/dev/mxcd, /dev/mxnd, /dev/mxgd, /dev/dri/card*, /dev/dri/renderD*) |
No |
| Moore Threads | MTHREADS_VISIBLE_DEVICES |
Yes |
| NVIDIA | NVIDIA_VISIBLE_DEVICES by default, or a CDI request — see NVIDIA
|
Yes |
| T-Head | /dev nodes (/dev/alixpu, /dev/alixpu_ctl, /dev/alixpu_ppu<N>) |
No |
Why — a device plugin that names
/deventries directly needs only whatruncalready grants; an env var needs the vendor’s own runtime hook to translate it into a device grant, so removing that hook removes the accelerator from the container.
An env var reaches the container either way, so a missing toolkit does not fail the allocation: it produces a running container with the variable set and no accelerator behind it.
Device plugin name collisions
The kubelet does not reject a second device-plugin registration for a resource name it already knows:
it replaces the endpoint. Both plugins keep writing their own device sets, so node capacity oscillates
between the two views, and a Pod can fail admission during the flap.
Allocation can also fail when kubelet sends one plugin’s device IDs to the other. Remove the duplicate plugin before submitting workloads.
Single NFD per cluster
Run exactly one NFD master. Two masters write and garbage-collect the same
feature.node.kubernetes.io/* labels, so each one’s GC pass can erase the other’s writes. Every
Device Manager DaemonSet in this operator is placed onto a node by a pci-<vendor>.present label that
NFD produces, so a lost label stops that DaemonSet from scheduling at all.
A vendor GPU Operator obtains its node labels through a NodeFeatureRule or an NFD source.
AMD’s operator installs a cluster-scoped NodeFeatureRule, which the active NFD installation evaluates.
GPUStack’s bundled NFD can evaluate it: its master denies no label namespace and accepts NodeFeature
labels from every namespace. Other operators require labels from a specific NFD source.
Enable that source in the replacement NFD installation; see NVIDIA
.
Bringing your own NFD. To use an existing NFD installation, disable GPUStack’s bundled NFD.
Set --set node-feature-discovery.enabled=false in chart mode, or
--disable-applications=node-feature-discovery on the worker in image mode.
See Installation Modes
for the mode differences.
The worker still installs the gpustack-cpu-info NodeFeatureRule in either mode.
The replacement NFD installation must evaluate it and meet these requirements:
- A worker collecting
pci, withvendoramong itsdeviceLabelFieldsand device classes02,03,0band12whitelisted. Those are what produce thepci-<vendor>.presentlabels, and thegpustack-cpu-infoNodeFeatureRulematches the same classes. - A worker collecting
system, which publishessystem-os_release.*. GPUStack’s scheduling chain does not use these labels, but the NVIDIA GPU Operator requires them. GPUStack’s bundled NFD publishes them, so the NVIDIA operator can run with its own NFD disabled. See NVIDIA . - A master that evaluates
NodeFeatureobjects from every namespace. The worker’s and the Device Managers’ own<node>-gpustack-*objects are how their detections reach theNode. restrictions.denyNodeFeatureLabels: falseanddisableExtendedResources: false, since those detections arrive as labels and as extended resources.- A
denyLabelNsthat admitsacceleratable.feature.gpustack.aiandgeneral.feature.gpustack.ai. - One owner for the NFD CRDs. Vendor charts bundle an NFD version of their own, so decide which release installs them before installing two that both want to.
A vendor chart that bundles NFD as a subchart satisfies them all, so its NFD can be the cluster’s one.
The NVIDIA GPU Operator’s already publishes vendor as its only deviceLabelFields, which is what
item 1 asks for, and collects every source, which is item 2; its device classes are the gap.
It whitelists 0300 and 0302 (the GPUs it was written for) and neither 0b nor 12, so an
accelerator in either class never gets the label its Device Manager schedules on, and that DaemonSet
stays unscheduled with nothing to say why. Both values live under that chart’s
node-feature-discovery.worker.config.sources.pci.
Item 6 is not a version problem there: both charts vendor NFD 0.19.0, so the CRDs are the same objects and all that has to be settled is which release owns them. And the class gap only bites once a second manufacturer is present: the display-controller classes it does whitelist cover NVIDIA and AMD, so a cluster holding only those is already served by its list.
Treat these requirements as a checklist. Hardware validation used the bundled NFD; a handover to an existing NFD installation has not been tested end to end.
Resource removal lag
Removing a vendor’s device plugin does not immediately remove its extended resource from the Node
object: the kubelet only drops it once it reconverges. A scheduler can keep seeing the old resource as
allocatable for a window after the plugin is gone, so “disabled” is not the same moment as “gone” from
a scheduling decision’s point of view.
Device Manager host mount
Helm value deviceManager.hostRootMount.enabled defaults to true. When enabled, every
device-manager DaemonSet mounts host / read-only (readOnly: true, HostPathDirectory) at
/host on its main container only, never on initContainers (stage-libs).
The host-root filesystem reader enforces a strict read-only boundary over /host:
- Root confinement:
os.OpenRootvalidatesetc,proc, andusrdirectories before reading. - Read-only files: Opens use
O_RDONLYwithos.SameFilechecks and zero write or exec methods. - Symlink rules: Intermediate directory symlinks resolve inside
/host; strict regular-file inspections used for A5 host mounts reject leaf symlinks.
Set deviceManager.hostRootMount.enabled=false in either situation:
- Policy-hardened clusters: OPA Gatekeeper or Kyverno policies forbid
hostPath: /mounts. - Older or custom runtime: Disabling
/hostskips A5 conditional host mounts without failing allocations on customascend-docker-runtimesetups.