GPUStack Operator

Vendor Prerequisites

Install the manufacturer’s driver and any required container toolkit before GPUStack. If a vendor GPU Operator is installed, disable components that conflict with GPUStack before sharing the node. Vendor keys and component names depend on the release. Check the cited artifact and version for each manufacturer.

Contents

Third-party dependencies

Check these dependencies before installation and when selecting a workload configuration. Installation options and ownership are in Installation Modes .

Dependency Role Configuration
Node Feature Discovery (NFD) Labels nodes and publishes hardware features. Run one NFD installation per cluster. See Single NFD per cluster .
Kueue Manages workload admission and quota. The chart includes Kueue; see Installation Modes .
Topograph Supplies topology inventory from a configured provider. Optional and disabled by default. See Topology Operations .
Mooncake Supplies cache services and transfer support for compatible engines. Optional. Match the engine, client, store and transport using Engine Compatibility .
ModelExpress Coordinates NVIDIA peer tensor transfer and compile-cache sharing. Optional; the operator creates a per-deployment metadata service. See ModelExpress .
CSI drivers Mount NFS or S3 volumes, or deliver cached model weights. NFS and S3 drivers are bundled subcharts. Node model delivery uses the Model Manager CSI driver. See Model Store Operations .
Serving engines Run inference workloads. Choose a compatible engine image and configuration. See Engine Compatibility .
Vendor drivers and toolkits Provide accelerator access on each node. Install the driver and any required toolkit. See the manufacturer guides below.

Manufacturer guides

AMD

Read against AMD GPU Operator v1.5.1 — Overview · Helm Chart Install · Toolkit Install .

Install before GPUStack — the ROCm driver alone. Per Allocator injection , GPUStack’s allocator injects /dev/kfd and each granted accelerator’s DRM nodes itself, so neither the AMD Container Toolkit nor amd-container-runtime is needed.

A non-root workload still needs video and render supplementary-group membership on its container’s user: the device plugin API has no field to grant a supplementary group, so this is the workload’s or the cluster’s own securityContext to set. Measured, the failure mode without it is zero enumerated ROCm agents, not an error: rocm-smi reads sysfs and keeps reporting the accelerator regardless, so it is not a valid check here; use rocminfo instead.

Vendor GPU Operator — the AMD GPU Operator, driven by a DeviceConfig CR the chart creates.

Keep the driver and metrics exporter. Disable the vendor device plugin, node labeller and bundled NFD. Disable Kernel Module Management when the driver is pre-installed. To test whole-card allocation, leave the DRA driver enabled; it is mutually exclusive with the device plugin on one DeviceConfig.

yaml
apiVersion: amd.com/v1alpha1
kind: DeviceConfig
metadata:
  name: default
  namespace: kube-amd-gpu
spec:
  driver:
    enable: true  # Default.
  metricsExporter:
    enable: true  # Enable; default false.
  devicePlugin:
    enableDevicePlugin: false
    enableNodeLabeller: false
  draDriver:
    enable: true  # Alternative to the device plugin.
yaml
node-feature-discovery:
  enabled: false
kmm:
  enabled: false  # When the driver is pre-installed.
  watch: false

Install it — the chart’s own switches cover the two bundled add-ons; the device plugin and the node labeller are DeviceConfig fields, so they are turned off on the CR the chart just created.

bash
helm repo add rocm https://rocm.github.io/gpu-operator
helm repo update
helm install amd-gpu-operator rocm/gpu-operator-charts \
  --namespace kube-amd-gpu --create-namespace \
  --version=v1.5.1 \
  --set node-feature-discovery.enabled=false \
  --set kmm.enabled=false \
  --set kmm.watch=false

kubectl -n kube-amd-gpu patch deviceconfig <name> --type merge -p \
  '{"spec":{"devicePlugin":{"enableDevicePlugin":false,"enableNodeLabeller":false},"metricsExporter":{"enable":true}}}'

The patch above makes the following change to the chart-created DeviceConfig. Replace the name with that of the existing object:

diff
 apiVersion: amd.com/v1alpha1
 kind: DeviceConfig
 metadata:
   name: <name>
   namespace: kube-amd-gpu
 spec:
   devicePlugin:
-    enableDevicePlugin: true
-    enableNodeLabeller: true
+    enableDevicePlugin: false
+    enableNodeLabeller: false
   metricsExporter:
-    enable: false
+    enable: true

Already installed — the vendor plugin and GPUStack’s plugin both advertise amd.com/gpu. Stop the vendor plugin before installing GPUStack on that node:

  1. Device plugin and Node Labeller — turn both off on the existing DeviceConfig, the same patch as above against the CR the running release created. Do not delete the DaemonSet: the controller recreates whatever the CR still asks for.
  2. Bundled Node Feature Discovery — if that release brought its own, helm upgrade it with the values it was installed with plus --set node-feature-discovery.enabled=false.
  3. Verify — wait for amd.com/gpu to leave the Node before installing GPUStack; kubectl get node <node> -o jsonpath='{.status.allocatable}' no longer lists it. The kubelet drops it a moment after the plugin goes, not at the same instant.

Ascend

Read against the Ascend NPU driver and firmware package — Driver Install — and, for the Atlas A5 statements below, ascend-docker-runtime 26.1.0, the release carried by the A5 host they were checked on.

Install before GPUStack — the Ascend NPU driver and firmware, and the Ascend Docker Runtime: per Allocator injection , GPUStack’s allocator emits ASCEND_VISIBLE_DEVICES and no device node, so that runtime is what turns an allocation into /dev entries.

Register the runtime under the handler name ascend, which is the RuntimeClass this chart creates for this manufacturer, or set deviceManager.createRuntimeClasses=false and attach your own.

Install ascend-docker-runtime newer than 26.0.x on Atlas A5 (the 950 series). Basic A5 support landed in 26.0.0, whose notes carry the line “Ascend Docker Runtime supports the Atlas 350 PCIe card” (this generation’s PCIe card). 910B and earlier are unaffected: their line ends at 7.3.x, and releases run 5.x, 6.x, 7.x, then 26.x with nothing in between.

An older runtime refuses the allocation, and blames the wrong thing. It maps the Ascend950 chip name to no device type it knows, so it falls back to the manager-device list every earlier generation used — and that list carries /dev/devmm_svm, which an A5 host does not have. Adding a device node that is not there is fatal, so container creation fails, naming that device rather than the version behind it.

The cards are not the gap: /dev/davinci<N> is built from the requested index with no chip name involved, so an older runtime would have injected them, but it never gets that far.

ascend-docker-runtime --version does not answer this. It is a runc wrapper that hands every unrecognized argument to the host’s own runc, so --version prints runc’s banner and says nothing about the vendor release. Read the version the installer recorded instead:

bash
grep '^version=' /usr/local/Ascend/Ascend-Docker-Runtime/ascend_docker_runtime_install.info
# version=v7.3.0    <- predates A5 support
# version=v26.1.0   <- serves A5

gpustack-operator device-manager preflight reads the same file and reports it as the ascend-docker-runtime check on each A5 accelerator, so a node too old to serve them is named before a workload lands on it.

A5 host ranktable requirement

The host ranktable must be absent or version 2.0. /etc/hccl_rootinfo.json is optional host state owned by no package. Install ascend-docker-runtime newer than 26.0.x (prefer 26.2+ so the runtime no longer mounts /etc/hccl_rootinfo.json unconditionally).

Releases through 26.1.x mount /etc/hccl_rootinfo.json into every container on presence alone with no version check. Releases 26.2+ removed that default mount, and neither release mounts /lib/route.conf.

Its format is fixed by the chip generation: 1.0 serves A2 (910/910B/310P), 1.2 serves A3 (910C), 2.0 serves A5. A5 loads libhccl_v2, which refuses anything but 2.0:

text
Config_Error_Ranktable(EI0014): Value 1.0 for ranktable variable version is invalid.
Expected value: 2.0.

So a table left behind by an older fleet stops every multi-card job on the node: single-card allocations never build a communicator and are unaffected. No environment variable disables the check, and the path is compiled into the library, so the file must be corrected rather than worked around:

bash
# absent is the healthy state -- HCCL then negotiates through rootInfo
test -e /etc/hccl_rootinfo.json && grep -o '"version"[^,]*' /etc/hccl_rootinfo.json
# "version": "1.0"   <- breaks multi-card HCCL on A5; remove it or regenerate as 2.0

gpustack-operator device-manager preflight reports this as the hccl-ranktable check on each A5 accelerator.

A5 conditional host mounts

On Ascend A5 (AcceleratorType status.detail.family equal to "950"), device-manager inspects host files through /host at Allocate time and conditionally mounts them read-only. This supports SGLang MemFabric (device_urma) and legacy multi-node UB HCCL across ascend-docker-runtime versions without breaking nodes that omit those files.

Host file Mount condition Skip behavior
/etc/hccl_rootinfo.json Regular non-symlink file (<= 1 MiB) containing JSON "version": "2.0" Absent, symlinked, oversized, or non-2.0 files are skipped with a diagnostic log; allocation still succeeds
/lib/route.conf Regular non-symlink file; resolves merged-usr /lib → /usr/lib inside /host Absent, symlinked, or non-regular files are skipped with a diagnostic log; allocation still succeeds
  • Non-A5 generations: Ascend 910, 910B, 910C, 310P, and 310B never receive either mount.
  • Collision handling: Containers already declaring a mount at the same path receive no duplicate mount.
  • Unavailable host root: Disabling deviceManager.hostRootMount.enabled skips both mounts without failing allocations.

A5 Host RoCE endpoints

For Ascend A5 Host RoCE (roce:host) vLLM workloads, provision per-NPU endpoint JSON files at /etc/hixlep/ub_endpoint_npu_<physical_id>.json on each Host RoCE node. Operator-managed Host RoCE Pods mount /etc/hixlep read-only with HostPathDirectory.

A missing /etc/hixlep directory fails kubelet volume setup immediately, and a missing endpoint file causes vllm-ascend startup to fail with FileNotFoundError. See Ascend A5 topologies for workload settings.

Cambricon

Read against Cambricon driver 5.10.22 (Neuware SDK 1.15.0) — Driver Install .

Install before GPUStack — the Cambricon MLU driver alone. Per Allocator injection , GPUStack’s allocator injects the card’s own node and the host’s control nodes on both the whole-card and the sliced path, so cntoolkit is not required. It still sets the CAMBRICON_VISIBLE_DEVICES a deployment running cambricon-container-runtime keys on, naming the same cards the injected nodes do.

Hygon

Read against the Hygon DCU Operator — its documentation tree publishes only a latest revision, with no pinned semantic version — Overview .

Install before GPUStack — the Hygon DCU driver and firmware, with the device files (/dev/kfd, /dev/mkfd, /dev/dri/card*, /dev/dri/renderD*) visible on the node. Per Allocator injection , GPUStack’s allocator injects those nodes directly, and the DCU Operator ships no container runtime of its own, so no container toolkit is required.

Vendor GPU Operator — the DCU Operator, driven by a single DeviceConfig CR (kubectl explain deviceconfig.spec); every component below is a field under that one CR, not a separate CRD.

Keep DCU-Exporter and DCU-DCGM; disable DCU-Device-Plugin and DCU-Label-Node. Replace the API version and name with those of the installed DeviceConfig:

yaml
apiVersion: <existing-api-version>
kind: DeviceConfig
metadata:
  name: device-config
  namespace: kube-system
spec:
  metricsExporter:
    enable: true  # Disabled by default.
  dcgm:
    enable: true  # Disabled by default.
  devicePlugin:
    enableDevicePlugin: false
    enableNodeLabeller: false

Install it — the vendor’s install chapter sits behind its authenticated documentation portal, so no install command is reproduced here. Whatever installs the operator, turn the two components off on the DeviceConfig it creates.

bash
kubectl -n kube-system patch deviceconfig <name> --type merge -p \
  '{"spec":{"devicePlugin":{"enableDevicePlugin":false,"enableNodeLabeller":false},"metricsExporter":{"enable":true},"dcgm":{"enable":true}}}'

The patch above updates the existing DeviceConfig. Read its apiVersion and name from kubectl get deviceconfig -n kube-system -o yaml and substitute them in the header:

diff
 apiVersion: <existing-api-version>
 kind: DeviceConfig
 metadata:
   name: <name>
   namespace: kube-system
 spec:
   devicePlugin:
-    enableDevicePlugin: <current-value>
+    enableDevicePlugin: false
-    enableNodeLabeller: <current-value>
+    enableNodeLabeller: false
   metricsExporter:
-    enable: <current-value>
+    enable: true
   dcgm:
-    enable: <current-value>
+    enable: true

Already installed — the vendor plugin and GPUStack’s plugin both advertise hygon.com/dcu. Stop the vendor plugin before installing GPUStack on that node:

  1. Device plugin and Node Labeller — turn enableDevicePlugin and enableNodeLabeller off on the existing DeviceConfig, with the patch above against the CR the running release created. Do not delete the DaemonSets: the controller recreates whatever the CR still asks for.
  2. Verify — wait for hygon.com/dcu to leave the Node before installing GPUStack.

Iluvatar

Read against ix-GPU-Operator v4.4.0 and ix-Container-Toolkit v1.1.0 — Helm Chart Install · Toolkit Install .

Install before GPUStack — the Iluvatar driver and the ix-Container-Toolkit: per Allocator injection , GPUStack’s allocator emits IX_VISIBLE_DEVICES and no device node. The two are paired by version — a v4.4.0 driver needs the v1.1.0 toolkit, and the vendor states the two must match on host and in-container.

Register the runtime under the handler name iluvatar, which is the RuntimeClass this chart creates for this manufacturer, or set deviceManager.createRuntimeClasses=false and attach your own.

Vendor GPU Operator — the ix-GPU-Operator, a Helm chart with one switch per component.

Keep ix-Exporter. Disable ix-Device-Plugin and ix-Feature-Discovery, which bundles upstream NFD. Disable ix-driver when the driver is pre-installed:

yaml
ixDriver:
  enabled: false  # Already the default.
ixExporter:
  enabled: true  # Default.
ixDevicePlugin:
  enabled: false
ixfd:
  enabled: false

Install it — the OCI chart, with the driver left to the node and the two competing components off.

bash
helm install --wait ix-gpu-operator \
  oci://registry.iluvatar.com.cn:10443/k8s/ix-gpu-operator --version 1.0.0 \
  --create-namespace --namespace ix-gpu-operator \
  --insecure-skip-tls-verify \
  --set global.repository="registry.iluvatar.com.cn/k8s" \
  --set global.version=4.4.0 \
  --set ixDriver.enabled=false \
  --set ixDevicePlugin.enabled=false \
  --set ixfd.enabled=false

Already installed — ix-Device-Plugin and GPUStack’s plugin both advertise iluvatar.com/gpu. Stop ix-Device-Plugin before installing GPUStack on that node:

  1. ix-Device-Plugin and ix-Feature-Discovery — helm upgrade the release with the values it was installed with plus --set ixDevicePlugin.enabled=false --set ixfd.enabled=false. The operator removes both workloads itself; deleting one by hand only makes it come back on the next reconcile.
  2. Verify — wait for iluvatar.com/gpu to leave the Node before installing GPUStack.

Below software-stack v4.3.0 the operator is deployed from YAML manifests with no per-component toggle. There the only lever is the whole manifest: delete it, and install the driver and ix-Container-Toolkit on the node yourself.

MetaX

Read against MetaX Kubernetes GPU Operator v0.15.3 — Overview · Helm Chart Install · Components .

Install before GPUStack — the MetaX driver and the MXMACA SDK. Per Allocator injection , GPUStack’s allocator injects the three control nodes (/dev/mxcd, /dev/mxnd, /dev/mxgd) and each granted accelerator’s DRM nodes directly, so no container toolkit is required.

Vendor GPU Operator — the MetaX GPU Operator. Its minimalMode is the install shape that fits here: it deploys neither the kernel driver nor the MXMACA SDK, leaving both to the node.

Component Keep / disable Mechanism
Kernel driver and MXMACA SDK Not deployed --set minimalMode=true leaves both to the node
mx-exporter (metrics) Keep optional under minimalMode
controller-exporter Keep optional under minimalMode
gpu-device (device plugin) Cannot be disabled the vendor marks it mandatory in every install mode, minimalMode included
gpu-label (NFD-equivalent) Cannot be disabled same marking
gpu-scheduler Disable optional — leave it undeployed
topo-discovery Disable optional — leave it undeployed

Install it — substitute your own registry for DOMAIN/PROJECT. minimalMode is fixed at install time and cannot be switched on a running release.

bash
helm install oci://DOMAIN/PROJECT/metax-operator \
  --version VERSION \
  --create-namespace -n metax-operator \
  --generate-name \
  --wait \
  --set registry=DOMAIN/PROJECT \
  --set minimalMode=true

Already installed — gpu-device and GPUStack’s plugin both advertise metax-tech.com/gpu. The vendor requires gpu-device in every install mode, so the plugins cannot share a node. Uninstall the vendor operator before installing GPUStack:

  1. Kernel driver and MXMACA SDK — make sure both are on the node independently of the operator. Under minimalMode they already are; otherwise install them first, or uninstalling takes them with it.
  2. The whole release — helm uninstall it. gpu-device and gpu-label have no switch, so there is nothing narrower to turn off.
  3. Verify — wait for metax-tech.com/gpu to leave the Node before installing GPUStack.

Moore Threads

Read against MT GPU Operator v2.1.0 (KUAE Cloud Native) — Overview · Helm Chart Install .

Install before GPUStack — the Moore Threads driver and the MT Container Toolkit: per Allocator injection , GPUStack’s allocator emits MTHREADS_VISIBLE_DEVICES and no device node. Both can come from the operator’s own full mode below.

Vendor GPU Operator — the MT GPU Operator, installed in full or core mode rather than by a single toggle. full is the mt-gpu-operator Helm chart pair; core is a kubectl apply -f of plain manifests carrying only the device plugin and the exporter.

Keep the driver, container toolkit, DCGM and its exporter. Disable NFD, the sGPU scheduler and webhook, kured, node-problem-detector and aiOps. In full mode the device plugin ships inside the operator’s reconciliation loop and has no disabling switch.

yaml
mtGpuClusterPolicyClusterpolicy:
  driverToolkit:
    enabled: true  # Default.
  containerToolkit:
    enabled: true  # Default.
  dcgm:
    enabled: true
  dcgmExporter:
    enabled: true
  nfd:
    enabled: false
  gpuScheduler:
    enabled: false  # Default.
  gpuWebhook:
    enabled: false  # Default.
  kured:
    enabled: false
  nodeProblemDetector:
    enabled: false
  aiOps:
    enabled: false

Install it — from the chart directories the vendor’s bundle unpacks, with the cluster-level add-ons off. No key disables the device plugin under full mode, so a node that must run GPUStack’s device plugin is served by core mode with that manifest’s device plugin removed.

bash
helm install --wait mt-gpu-operator mt-gpu-operator
helm install --wait mt-gpu-operator-custom-resources mt-gpu-operator-custom-resources \
  --set mtGpuClusterPolicyClusterpolicy.nfd.enabled=false \
  --set mtGpuClusterPolicyClusterpolicy.kured.enabled=false \
  --set mtGpuClusterPolicyClusterpolicy.nodeProblemDetector.enabled=false \
  --set mtGpuClusterPolicyClusterpolicy.aiOps.enabled=false

Already installed — the vendor plugin and GPUStack’s plugin both advertise mthreads.com/gpu. Stop the vendor plugin before installing GPUStack on that node. The full mode has no switch to disable it; follow the procedure for the installed mode:

  1. Cluster-level add-ons (bundled NFD, kured, node-problem-detector, aiOps) — under full, helm upgrade the mt-gpu-operator-custom-resources release with the switches above. core ships none of them.
  2. Device plugin — no key removes it under full, so the mode that installed it decides:
    • full — either leave that node to the vendor stack, or move it to core: uninstall both charts, keep the driver and MT Container Toolkit on the node by installing them outside the operator, and go on below.
    • core — there is no controller, so edit the manifests directly: drop the device plugin from deployments.yaml, leave the exporter, and re-apply.
  3. Verify — wait for mthreads.com/gpu to leave the Node before installing GPUStack.

NVIDIA

Read against NVIDIA GPU Operator v26.3.3 — Overview · Helm Chart Install .

Install before GPUStack — the NVIDIA driver and the NVIDIA Container Toolkit: per Allocator injection , GPUStack’s allocator emits NVIDIA_VISIBLE_DEVICES by default and no device node. The toolkit is what the other channel needs too — it is what writes the node’s CDI specifications. Both can come from the operator itself.

The toolkit is also what MIG needs, not only workloads: the Device Manager Pod reads the management library through it, and the chart declares NVIDIA_MIG_CONFIG_DEVICES / NVIDIA_MIG_MONITOR_DEVICES on that Pod so the runtime stops hiding the driver’s MIG capabilities from it — see MIG prerequisites for the two ways to break that.

Vendor GPU Operator — the NVIDIA GPU Operator, a Helm chart with one switch per component.

Keep the driver, toolkit, exporter and GPU Feature Discovery. Disable the device plugin, MIG Manager and CDI. Disable bundled NFD or hand the job over and disable GPUStack’s bundled NFD instead.

yaml
driver:
  enabled: true  # Default.
toolkit:
  enabled: true  # Default.
dcgmExporter:
  enabled: true  # Default.
devicePlugin:
  enabled: false
gfd:
  enabled: true  # Default.
nfd:
  enabled: false
migManager:
  enabled: false
cdi:
  enabled: false  # Default true from v25.10.

MIG Manager changes geometry from node labels and conflicts with Allocate-time MIG actuation. One profile is an exception; see NVIDIA MIG .

Check the node is not opted out. Some GPU distributions ship their nodes already labelled nvidia.com/gpu.deploy.operands=false, which tells this operator to place nothing on them at all. Its ClusterPolicy still reports ready, so the only symptom is that no operand ever appears. Set it to true on the nodes you want managed.

Install it — the driver, toolkit, exporter and feature discovery stay, the three competing components go.

bash
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install --wait --generate-name \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator --version=v26.3.3 \
  --set devicePlugin.enabled=false \
  --set nfd.enabled=false \
  --set migManager.enabled=false \
  --set cdi.enabled=false

Already installed — the vendor plugin and GPUStack’s plugin both advertise nvidia.com/gpu. The vendor’s MIG Manager also reconfigures hardware that GPUStack’s allocator manages. Stop both vendor components before installing GPUStack on that node.

One helm upgrade, with the values the release was installed with plus the three switches below. The operator removes those DaemonSets itself; deleting one by hand only makes it come back, because the ClusterPolicy still asks for it.

  1. Device plugin — set --set devicePlugin.enabled=false.
  2. MIG Manager — set --set migManager.enabled=false. The node retains its existing geometry. GPUStack’s allocator reconfigures it during subsequent Allocate calls.
  3. Bundled Node Feature Discovery — --set nfd.enabled=false. GPUStack’s bundled NFD publishes the system-os_release.* labels that release had been publishing, so nothing has to be carried over by hand. Keeping this one and disabling GPUStack’s bundled NFD is the other way round — One NFD per cluster .
  4. Driver, container toolkit, DCGM exporter, GPU Feature Discovery — leave running. GPUStack needs the first two.
  5. Verify — wait for nvidia.com/gpu to leave the Node before installing GPUStack.

CDI mode. --set cdi.enabled=false is the narrower choice for a node this operator’s allocator must control end to end. Leaving it on is workable.

Why — GPU Operator >= 25.10 defaults cdi.enabled: true, and the older cdi.default field is deprecated and has no effect once cdi.enabled exists. With CDI on, the toolkit resolves a bare accelerator UUID against the node’s specifications, and a name it cannot resolve fails container creation loudly rather than silently — which is what makes leaving it on workable rather than wrong.

A node serving MIG wants CDI off. GPUStack’s allocation path is safe whatever the toolkit does: a partition allocation is always carried by the variable, never by a CDI request.

Why — in cdi mode the runtime resolves whatever NVIDIA_VISIBLE_DEVICES holds against the node’s specifications, and the MIG instance GPUStack’s allocator materializes at Allocate time is named in none of them, so a partition request has nothing to resolve against.

Device injection channel. Prefer auto, set through GPUSTACK_NVIDIA_DEVICE_INJECTION_STRATEGY , over naming a channel yourself. It answers per container, independently of the switch above.

Why — two channels exist, and which one a node uses is what decides whether it needs the vendor runtime at all: the default names the accelerator in NVIDIA_VISIBLE_DEVICES, which only nvidia-container-runtime reads, while the other requests it as a CDI device the container engine resolves by itself under plain runc. Whether a node can use the second is a property of its container engine, which is what auto reads and a human would otherwise have to. It uses CDI only where the vendor’s own specifications already name the exact accelerator being granted, and writes no specification of its own. Device Discovery has the mechanism, and Settings what it checks, when it declines, and the one engine that has to be told.

auto leaves a Pod that names a runtimeClassName to that handler, whose configuration it cannot read — correctly, since the handler is what injects. Whether that covers the Instances this operator creates depends on the class existing: the worker attaches runtimeClassName: nvidia only where it finds one, so with deviceManager.createRuntimeClasses=false and none supplied, an Instance runs under the engine’s default handler and auto answers for it like any other Pod.

T-Head

Read against the T-Head SAIL SDK (also called the PPU SDK), HGGCRT v3 — Driver Install · Overview .

Install before GPUStack — the PPU SDK alone. It ships as a self-extracting Runfile from T-Head’s download centre, installs to /usr/local/PPU_SDK by default, and is verified with ppu-smi (which prints the driver version) and hgcc --version. Take HGGCRT v3 rather than v2: it is backward compatible with the v2 API.

Per Allocator injection , GPUStack’s allocator injects T-Head accelerators as /dev nodes directly, which is the mechanism the vendor’s own container guide uses — it isolates a container by passing those nodes to docker run --device, with no vendor runtime in the path. Which nodes a container receives is in Device Discovery .

Shared coexistence rules

Allocator injection

Whether a container toolkit is required at all follows directly from how GPUStack’s allocator makes the accelerator visible: a /dev node needs nothing from the toolkit, while an env var needs the vendor’s runtime hook to interpret it.

Manufacturer Allocator injection Toolkit required
AMD /dev nodes (/dev/kfd, /dev/dri/card*, /dev/dri/renderD*) No
Ascend ASCEND_VISIBLE_DEVICES Yes
Cambricon /dev nodes (/dev/cambricon_dev*, the control nodes), plus CAMBRICON_VISIBLE_DEVICES No
Hygon /dev nodes (/dev/kfd, /dev/mkfd, /dev/dri/card*, /dev/dri/renderD*) No
Iluvatar IX_VISIBLE_DEVICES Yes
MetaX /dev nodes (/dev/mxcd, /dev/mxnd, /dev/mxgd, /dev/dri/card*, /dev/dri/renderD*) No
Moore Threads MTHREADS_VISIBLE_DEVICES Yes
NVIDIA NVIDIA_VISIBLE_DEVICES by default, or a CDI request — see NVIDIA Yes
T-Head /dev nodes (/dev/alixpu, /dev/alixpu_ctl, /dev/alixpu_ppu<N>) No

Why — a device plugin that names /dev entries directly needs only what runc already grants; an env var needs the vendor’s own runtime hook to translate it into a device grant, so removing that hook removes the accelerator from the container.

An env var reaches the container either way, so a missing toolkit does not fail the allocation: it produces a running container with the variable set and no accelerator behind it.

Device plugin name collisions

The kubelet does not reject a second device-plugin registration for a resource name it already knows: it replaces the endpoint. Both plugins keep writing their own device sets, so node capacity oscillates between the two views, and a Pod can fail admission during the flap.

Allocation can also fail when kubelet sends one plugin’s device IDs to the other. Remove the duplicate plugin before submitting workloads.

Single NFD per cluster

Run exactly one NFD master. Two masters write and garbage-collect the same feature.node.kubernetes.io/* labels, so each one’s GC pass can erase the other’s writes. Every Device Manager DaemonSet in this operator is placed onto a node by a pci-<vendor>.present label that NFD produces, so a lost label stops that DaemonSet from scheduling at all.

A vendor GPU Operator obtains its node labels through a NodeFeatureRule or an NFD source. AMD’s operator installs a cluster-scoped NodeFeatureRule, which the active NFD installation evaluates. GPUStack’s bundled NFD can evaluate it: its master denies no label namespace and accepts NodeFeature labels from every namespace. Other operators require labels from a specific NFD source. Enable that source in the replacement NFD installation; see NVIDIA .

Bringing your own NFD. To use an existing NFD installation, disable GPUStack’s bundled NFD. Set --set node-feature-discovery.enabled=false in chart mode, or --disable-applications=node-feature-discovery on the worker in image mode. See Installation Modes for the mode differences.

The worker still installs the gpustack-cpu-info NodeFeatureRule in either mode. The replacement NFD installation must evaluate it and meet these requirements:

  1. A worker collecting pci, with vendor among its deviceLabelFields and device classes 02, 03, 0b and 12 whitelisted. Those are what produce the pci-<vendor>.present labels, and the gpustack-cpu-info NodeFeatureRule matches the same classes.
  2. A worker collecting system, which publishes system-os_release.*. GPUStack’s scheduling chain does not use these labels, but the NVIDIA GPU Operator requires them. GPUStack’s bundled NFD publishes them, so the NVIDIA operator can run with its own NFD disabled. See NVIDIA .
  3. A master that evaluates NodeFeature objects from every namespace. The worker’s and the Device Managers’ own <node>-gpustack-* objects are how their detections reach the Node.
  4. restrictions.denyNodeFeatureLabels: false and disableExtendedResources: false, since those detections arrive as labels and as extended resources.
  5. A denyLabelNs that admits acceleratable.feature.gpustack.ai and general.feature.gpustack.ai.
  6. One owner for the NFD CRDs. Vendor charts bundle an NFD version of their own, so decide which release installs them before installing two that both want to.

A vendor chart that bundles NFD as a subchart satisfies them all, so its NFD can be the cluster’s one. The NVIDIA GPU Operator’s already publishes vendor as its only deviceLabelFields, which is what item 1 asks for, and collects every source, which is item 2; its device classes are the gap.

It whitelists 0300 and 0302 (the GPUs it was written for) and neither 0b nor 12, so an accelerator in either class never gets the label its Device Manager schedules on, and that DaemonSet stays unscheduled with nothing to say why. Both values live under that chart’s node-feature-discovery.worker.config.sources.pci.

Item 6 is not a version problem there: both charts vendor NFD 0.19.0, so the CRDs are the same objects and all that has to be settled is which release owns them. And the class gap only bites once a second manufacturer is present: the display-controller classes it does whitelist cover NVIDIA and AMD, so a cluster holding only those is already served by its list.

Treat these requirements as a checklist. Hardware validation used the bundled NFD; a handover to an existing NFD installation has not been tested end to end.

Resource removal lag

Removing a vendor’s device plugin does not immediately remove its extended resource from the Node object: the kubelet only drops it once it reconverges. A scheduler can keep seeing the old resource as allocatable for a window after the plugin is gone, so “disabled” is not the same moment as “gone” from a scheduling decision’s point of view.

Device Manager host mount

Helm value deviceManager.hostRootMount.enabled defaults to true. When enabled, every device-manager DaemonSet mounts host / read-only (readOnly: true, HostPathDirectory) at /host on its main container only, never on initContainers (stage-libs).

The host-root filesystem reader enforces a strict read-only boundary over /host:

  • Root confinement: os.OpenRoot validates etc, proc, and usr directories before reading.
  • Read-only files: Opens use O_RDONLY with os.SameFile checks and zero write or exec methods.
  • Symlink rules: Intermediate directory symlinks resolve inside /host; strict regular-file inspections used for A5 host mounts reject leaf symlinks.

Set deviceManager.hostRootMount.enabled=false in either situation: