Preflight Operations

Run preflight on a bare host to check which accelerators the node can serve and what an allocation would require. It needs no cluster, CRDs, NFD labels or running Device Manager. This page covers the command, host mounts and result fields; Discovery explains the states, depths and how preflight reuses the allocator checks.

Contents

• Before you run it
• The command line
• Mounts
• Flags
• Side effects
• Emitted steps
• Output by manufacturer
• Reading the result

Before you run it

There is one mode: the run reproduces what the device-manager does in production. These steps touch the node while it does, so decide them before running where live workloads are:

• the probe containers, which hold an accelerator for as long as they run;
• the preload-library tree, copied onto the host where an init container would have put it;
• a driver mode asked on and put straight back, on manufacturers whose slicing depends on one. A mode that is off is not a node that cannot serve (the allocator turns it on itself when a slice lands), so reading it answers nothing, and asking the driver is the only way to know. The toggle happens only where the mode was already off, so nothing on the node is sharing that accelerator and nothing can notice the window. A restore that fails is reported, loudly, on the row: the accelerator is left on and the row is the only thing that can send someone to turn it off.

Pass --dry-run to see all of it without any of it happening.

A probe container is not privileged. It gets exactly the device nodes, mounts and environment the allocator’s own injection names, because what it is there to measure is the isolation that injection establishes; a privileged container would be handed every device on the host instead.

Preflight needs CAP_SYS_CHROOT to enter the host root, which the default container capability set already carries. --privileged is for the driver reads and the device nodes, not for the chroot.

The command line

Find your manufacturer below and run the block as it stands. Each one is complete and needs nothing filled in. Add --dry-run to any of them to see every step without taking one.

The blocks differ only in their last vendor argument, which is what that manufacturer’s library loads from: the same host path the device-manager DaemonSet mounts for it, at the same access. That is what makes this run reproduce production rather than resemble it.

On containerd, swap docker run for nerdctl run; it resolves its own socket and namespace on the host it is invoked on. The two blocks carrying --runtime are the exception, and each says what to use instead.

A vendor path that is not there is created, not refused. docker run -v creates a missing source directory as an empty one, so a typo (or a driver installed somewhere else) mounts nothing over the right place, and the detect pass reports zero accelerators: the same answer a node with no hardware gives. Check the path exists on the host before reading the result.

AMD

docker run --rm --privileged --network=host \
    -v /:/host -v /dev:/dev -v /sys:/sys \
    -v /opt/rocm:/opt/rocm:ro -v /opt/rocm/lib:/opt/rocm/lib:ro \
    gpustack/gpustack-operator:latest \
    gpustack-operator device-manager preflight --manufacturer=amd

The second mount is not a duplicate of the first. ROCm’s packaging puts a symlink somewhere in this path on every install, and where it falls decides whether the mount survives it: one at or above the mount source is resolved by the host and is harmless, while one inside the mounted tree dangles in the container. Naming the library directory as its own source has the host resolve it first.

Both shapes are ordinary. On a single-version host /opt/rocm is itself the link. On one carrying two ROCm versions, /opt/rocm/lib is — and mounting /opt/rocm alone then loads no ROCm library at all, so a node with an accelerator reports as having none. The device-manager DaemonSet mounts both for the same reason.

To pin one ROCm version on a host carrying two, mount the versioned tree in place of both (-v /opt/rocm/core-<ver>:/opt/rocm:ro), which is how the same host is preflighted against 7.14 and 10.0 separately.

Ascend

docker run --rm --privileged --network=host \
    -v /:/host -v /dev:/dev -v /sys:/sys \
    -v /usr/local/Ascend:/usr/local/Ascend:ro -v /usr/local/dcmi:/usr/local/dcmi:ro \
    gpustack/gpustack-operator:latest \
    gpustack-operator device-manager preflight --manufacturer=ascend

No --runtime ascend here, although Vendor Prerequisites requires that runtime in production: what it injects is device nodes, and this command is already --privileged with /dev mounted whole. The driver is the mount above. The Ascend probe containers do need it, and get it themselves; under nerdctl they cannot, and are emitted for you to run instead.

Cambricon

docker run --rm --privileged --network=host \
    -v /:/host -v /dev:/dev -v /sys:/sys \
    -v /usr/local/neuware:/usr/local/neuware:ro \
    gpustack/gpustack-operator:latest \
    gpustack-operator device-manager preflight --manufacturer=cambricon

Hygon

docker run --rm --privileged --network=host \
    -v /:/host -v /dev:/dev -v /sys:/sys \
    -v /opt/hyhal:/opt/hyhal:ro -v /opt/dtk:/opt/dtk:ro \
    -v /etc/dmi_mig_config:/etc/dmi_mig_config \
    gpustack/gpustack-operator:latest \
    gpustack-operator device-manager preflight --manufacturer=hygon

The third mount is the only writable one on this page: the vendor library materializes a partition by writing that registry, and the DaemonSet mounts it writable for the same reason. Add --probe-image <image> to measure the two slice rows; no default is claimed for a Hygon family, and what the image has to carry is in The container-probe tier .

Iluvatar

docker run --rm --privileged --network=host \
    -v /:/host -v /dev:/dev -v /sys:/sys \
    -v /usr/local/corex:/usr/local/corex:ro \
    gpustack/gpustack-operator:latest \
    gpustack-operator device-manager preflight --manufacturer=iluvatar

No --runtime iluvatar, for the same reason Ascend needs none: ix-container-runtime injects device nodes, which --privileged and /dev already cover, and the driver is the mount above.

MetaX

docker run --rm --privileged --network=host \
    -v /:/host -v /dev:/dev -v /sys:/sys \
    -v /opt/mxdriver:/opt/mxdriver:ro -v /opt/maca:/opt/maca:ro \
    gpustack/gpustack-operator:latest \
    gpustack-operator device-manager preflight --manufacturer=metax

Moore Threads

docker run --rm --privileged --network=host --runtime mthreads \
    -v /:/host -v /dev:/dev -v /sys:/sys \
    gpustack/gpustack-operator:latest \
    gpustack-operator device-manager preflight --manufacturer=mthreads

Nothing to mount, and a --runtime instead: the user-space driver here is injected by the vendor container runtime rather than installed at a path you can bind-mount. mthreads is the handler that runtime registered with your container engine, and the same name the chart’s RuntimeClass carries. Under nerdctl, whose --runtime names an OCI shim, that flag does not reach the handler.

NVIDIA

docker run --rm --privileged --network=host --runtime nvidia \
    -v /:/host -v /dev:/dev -v /sys:/sys \
    gpustack/gpustack-operator:latest \
    gpustack-operator device-manager preflight --manufacturer=nvidia

Nothing to mount, for the same reason as Moore Threads above. Under nerdctl use --gpus all in place of --runtime nvidia, whose --runtime names an OCI shim and rejects the handler name.

The flag is not optional. Measured on a host with one RTX 4090: the identical command under --runtime runc reports accelerators: 0 against a host whose own nvidia-smi reports 1, and exits 1; under --runtime nvidia it reports ok, accelerators: 1, and exits 0.

T-Head

docker run --rm --privileged --network=host \
    -v /:/host -v /dev:/dev -v /sys:/sys \
    -v /usr/local/PPU_SDK:/usr/local/PPU_SDK:ro \
    gpustack/gpustack-operator:latest \
    gpustack-operator device-manager preflight --manufacturer=thead

Add --probe-image <image> to measure the slice rows: no default is claimed for a PPU family, because the workload container brings its own SDK and there is no generation to match an image against.

Every manufacturer at once

docker run --rm --network=host -v /:/host \
    gpustack/gpustack-operator:latest \
    gpustack-operator device-manager preflight

With no vendor mounts at all, the run still reports every manufacturer and names the mounts the rest of the questions need. Only NVIDIA, Ascend and AMD carry a host CLI to cross-check that detection against, so for manufacturers without a host CLI a detection of zero is this container’s own view and not the host’s; the Host cross-check column below says which is which.

Mounts

Why the host root rather than a mounted runtime socket — mounting a runtime socket already grants host root, and needs one mount per runtime plus a CLI of ours in the image whose version need not match the daemon it talks to. Entering the host root costs one mount and gives the host’s own CLIs, at their own versions.

Flags

The runtime is resolved from the kubelet’s own CRI endpoint wherever the host states one. That is what starts a container on this node in production, and reproducing production is the point.

Which configuration states it comes from the running kubelet’s own command line, found in the host’s process table through the mounted host root: --config names the file it loaded, --config-dir the drop-ins merged over that file, and an endpoint on the command line itself overrides both.

A host with no kubelet running — a machine before it joins a cluster, or a distribution that embeds the kubelet in its agent process — has no such command line. Then /var/lib/kubelet/kubeadm-flags.env, /var/lib/kubelet/config.yaml and the distribution’s own kubelet.conf.d tree are read instead, in that order.

Why not simply probe — a host carrying both docker and containerd, with a kubelet talking to containerd, would be probed docker-first and every container answer would then describe a path no workload takes. A host that states no endpoint — a bare machine before a cluster exists, or a distribution keeping that configuration elsewhere — falls through to probing docker, then nerdctl, then ctr.

A host root mounted without the host’s /proc refuses instead of probing. Nothing there can say which kubelet is running, and the paths above are only this node’s kubelet’s if this node’s kubelet reads them. Every container step drops to being emitted; --runtime names a runtime past it.

Side effects

It starts one probe container per accelerator that can host a logical slice, and — on a manufacturer whose co-tenancy is measured — two more for that accelerator afterwards, started together to see whether one card carries two slices at once.

Each one runs the vendor image resolved for that accelerator’s family, with the injection an allocation would emit, and prints its own address map and the vendor reader’s output. The container is named for its accelerator, so two cards never render their artifacts over one path.

NVIDIA and Ascend probes are handed to a vendor runtime, without which the container gets device nodes and no user-space driver — and the flag for that differs by CLI. docker takes --runtime nvidia / --runtime ascend, naming an entry in its own daemon configuration; nerdctl’s --runtime names an OCI shim instead and rejects those names outright, so its door to the NVIDIA hook is --gpus. A pair with no known door — nerdctl and Ascend — is emitted rather than run.

Everything it starts is removed. Every probe container carries the label gpustack.ai/preflight=true and is started with --rm, and a run sweeps containers carrying that label before it starts any of its own — so one left behind by a run that was killed is collected by the next.

On containerd the containers are created in the namespace gpustack-preflight, named in the result, and the CLI is pointed at the socket that was resolved rather than at its own default.

The run writes to the host in four places, and only the first two outlive it:

Run it before the node serves workloads. The mode above is put back within the same call, but nothing serialises that against a device-manager allocating on the same card: an allocation landing in that window can be switched off under. Ascend and Cambricon write to the driver.

Run the image that matches the installed device-manager. The library tree in the first row is the one real allocations mount, not a copy of it, and staging replaces the files already there. On a node with a device-manager installed, running preflight from a different image version therefore leaves that version’s preload libraries behind for every allocation afterwards: a working node, silently changed by a command that reads. On a node being brought up there is nothing there yet and nothing to disturb, which is what this command is for. If you must run it against an installed node, use the same tag the device-manager runs, and treat any other tag as an operation that changes the node.

Why they go in a directory of their own — the neighbouring .../operator/pods is what an allocator reads as its record of what other Pods hold, and an entry there under a Pod UID no kubelet scheduled would be counted as occupancy. Preflight never writes into it, so a run killed before it can clean up costs a later allocation nothing.

Emitted steps

A container step that cannot be taken is printed complete, and the row says it was emitted. That is an answer, not a failure of the node, and it does not affect the exit code. Whether it runs as printed depends on the case: the two below that write nothing to the host name what you have to put there first, and the ctr row names a CLI this host does not have. Five cases reach it:

Why ctr never starts a probe — ctr run has no flag that passes a device node, so the only way to reach an accelerator through it is --privileged, which grants every device on the host. That would report an isolation the injection never established: a measured answer that measured the wrong thing.

Output by manufacturer

Find your manufacturer first. How much this command can tell you depends on how much your allocator reads before it hands a device out, and that differs by vendor. The tiers below say what a run on your node will and will not establish.

Every manufacturer is asked for the capability rows. All allocators can produce an injection without one being served, so every one of them answers what an allocation would grant. What differs by tier is the two things an injection alone cannot establish: whether a driver said anything before it, and whether a container was started after it.

How many of those rows come back is a fact about the accelerator, not the tier. The two sliced rows are produced only for an accelerator declaring logical slicing; a partition-backed one (a card with MIG on) reports the two management rows and no sliced rows, because a partitioned accelerator serves no other mode. One declaring neither family reports none of the four.

An absent row is a family this accelerator does not offer, not a result that went missing.

The two middle tiers are the reminder that the two questions are independent. Cambricon and MetaX read a driver and start no container; Hygon starts a container and reads no driver. Neither is “further along” than the other: they answer different halves.

sidecar-visibility is simulated on every tier, including the full one: measuring it would need the owner container still running when the sidecar starts, and the probe containers are one-shots that exit as soon as they have printed their evidence.

The injection-only tier

Your rows stop one step short of the full tier’s, and the missing step is the driver read. Your allocator consults no driver when it serves an allocation: the memory cap and compute weight come from the container’s own resource request, and the host kmod plus the vendor container runtime enforce them. There is no precondition to check ahead of time, which is a fact about your stack and not a gap in this command; the note on your group says so in your vendor’s own terms.

What you do get:

• The capability rows your accelerators offer, at simulated — the allocator really did produce the injection, and the row names what it grants: the device nodes, the mounts, the environment. What is not established is what that injection looks like from inside a container, because no container was started. See #138 for the work that would close that; Hygon left this tier that way, and the two sections below are what changed.
• The host cross-check on your detection, the one thing detect cannot do. From inside a container with no device mounts, “this machine has no accelerators” and “this machine has eight you cannot reach” are the same sentence. This enters the host and asks its own vendor CLI, so a detection of zero on a host that sees eight comes back naming the mounts you are missing. Only NVIDIA, Ascend and AMD carry a host CLI to cross-check against; the rest of the table’s Host cross-check column says so.
• The note, which says in that manufacturer’s own terms why no precondition exists — so you can tell “nobody implemented this” from “there is nothing here to implement”.

If a slice does not work on your node, the place to look is the vendor container runtime and the resource request, not a driver flag.

The container-probe tier

Hygon is here, and it is the tier where the two sliced rows mean the most. Its allocator reads no driver (the paragraph above applies to the note on your group unchanged), but a container is started, so sliced-runtime-loaded and sliced-quota-in-force come back at measured.

Before reading those rows, check the following:

• --probe-image is required. No default is claimed for a Hygon family. The image does not have to be a DTK one (the probe runs the vendor’s own BandwidthTest out of /opt/dtk, which your allocator already mounts), but it is not “any image” either. It needs sh, cat, grep, awk, mkdir, sleep and kill, plus a C library that can load a dynamically linked glibc binary. mkdir is load-bearing: the reader claims the driver record it is about to read, and an image without it reports a healthy accelerator as unavailable. Measured on a glibc image without DTK.
• The evidence is the driver’s record, not a log line. The other measured manufacturers preload a library GPUStack builds and raise its log level to make it state the cap. Hygon’s slicing runtime is the vendor’s own DTK/hyhal user space, whose vgpu diagnostics have an API to set the level and no environment variable, so there is nothing to turn on from outside. What answers instead is the per-slice record the driver publishes under the kfd vgpu sysfs, which exists only for a process that entered vgpu mode.
• hy-smi will not confirm a slice for you. It answers from the DMI layer and reports the physical card: under a container capped at 1024 MiB it still prints the card’s full VRAM. That is not a broken slice; the cap binds the HSA/HIP runtime a workload uses. To see a slice by hand, ask a HIP client.

The full table

An empty cell is “this manufacturer has none”, not a value left out. Ascend files two more rows on an A5 accelerator, which are neither a driver read nor a probe — see The A5 host-state rows .

Read the mode column, not the capability name. Every row in a report carries a mode, and it is what makes two manufacturers comparable: container-share and cu-mask-topology are different vendors’ words for the same question — can this accelerator be logically sliced? The capability name stays the vendor’s own so that searching for it finds their documentation; mode is what tells you two rows answer the same thing, and which mode nothing answered for on your node.

An empty host cross-check is an answer, not a gap. A manufacturer with none has no vendor CLI whose output shape this command has established, and the column says so rather than guessing: a wrong match counts zero, and a zero reads as “the host sees nothing either”, the one answer that sends an operator to debug the wrong layer.

What the driver-read row means, per manufacturer

Two of these are established rather than read — container-share and smlu-mode. Where the mode is off, the driver is asked to turn it on and it is put straight back; the detail says which happened, and says so explicitly if the restore failed and the accelerator was left on.

Under --dry-run the ask is withheld, because asking is a write however briefly the mode is held. The row then reports the mode as read and says it was not established — which is a different answer from a capability that was checked and found working.

The A5 host-state rows

Ascend files two more rows on an Atlas A5 (950) accelerator and on no other. Neither reads a driver and neither starts a container: each reads one file the host either carries or does not, so both come back at the declared depth. Each is one node-level fact, and it is a precondition for every mode the accelerator serves, so each is repeated on every accelerator and under every mode, and a report filtered by either still carries it.

Why each of those fails, and how to correct it, is in Vendor Prerequisites — the mechanism is stated there once, and the reason on the row itself carries it verbatim, so a report is readable without this page.

Read the following from the table rather than inferring them:

• An absent ranktable is ok, not a prerequisite nobody installed. It is the healthy state, which is why that row treats an unreadable file the opposite way to the one above it.
• A host root this command could not look in is unavailable, not ok. A missing /etc/hccl_rootinfo.json and an unmounted host root are the same absence to the filesystem, and only the first is a healthy node. A run with no host root to enter carries on as a dry run rather than stopping, so this row says the host was never looked at. A path that merely resembles a host root counts as none.
• A ranktable behind a symbolic link is unavailable too. This command resolves an absolute target against its own root rather than the host’s, so following the link would read a file inside the container while the vendor runtime follows the same link on the host. The row names the target instead; read it there.

The slice rows, and what the probe needs

Manufacturers with a container probe start one container per sliceable accelerator. The table below lists their probe images and requirements.

The rows themselves are the same four everywhere:

Hygon is the row that mounts no shared object of ours, so its sliced-runtime-loaded is answered by the driver’s own per-slice record rather than by the container’s address space — the tier above describes what that costs and what the probe image has to carry.

A slice asks for half the accelerator, so the figure to expect inside is half what the host reports: a 24564 MiB card caps at 12282, a 65536 MiB one at 32768. Half rather than a sliver because the figure is the evidence, and a cap near the card’s own size could be read as the card.

Reading the result

One YAML document goes to stdout, with three sections. This outline uses ... for omitted values and rows:

accelerators:            # one group per manufacturer asked about
  - manufacturer: nvidia
    detection:
      state: ...
      depth: ...
      accelerators: ...
    checks:
      - ...
network:                 # the node's RDMA links — belongs to no manufacturer
  timestamp: ...
  checks:
    - ...
topology:                # what this node's kubelet does with a NUMA hint
  timestamp: ...
  policy: single-numa-node
  depth: declared

Each accelerators group carries the time it was read, the detection answer, and a row per accelerator per capability — each row carrying the state and the depth it reached, the driver’s or the container’s own words, and the container command where one was involved.

The network section carries one row per RDMA-capable interface — and per RDMA-capable virtual function, named <interface>/<vf bus id>, since on an SR-IOV node those are every RDMA device there is. Each row gives the name, the RDMA device and the link verdict .

The rows come from the same pass that produces the node’s published record, so the two judge a link the same way. They need not say the same thing: this section is a fresh read taken when you invoke it, so a link that changed since the last detect pass shows here first.

An entry with no RDMA device and no link verdict gets no row: rdma: false on its own settles the question, while a verdict without a device is the unreadable-tree case this section exists to surface. A section with no rows carries a note saying which of the two reasons applies: the enumeration failed, or the node has no RDMA hardware.

The topology section names the TopologyManager policy this node’s kubelet is configured with, read off the same command line and out of the same configuration the CRI endpoint is read from.

It matters because the policy decides what the kubelet does with the NUMA hint a device plugin publishes: single-numa-node and restricted gate admission on it, best-effort admits a misaligned placement anyway, and none discards it. So a request pairing an accelerator with an RDMA interface lands them on one NUMA node only where the policy enforces it.

A node nobody configured runs none — how to set a policy, and how to confirm the kubelet took it .

A policy that no readable configuration names is reported as unknown, never as the kubelet’s default. Reporting none because nothing was found would publish a value nobody read. An unknown always carries a note saying which of the four reasons applies: nothing the kubelet reads named one, a file could not be read, two distribution trees name different policies, or the host’s process table was not visible.

That last one reports “not read”, not a value. Without the host’s /proc nothing can say which kubelet is running or which file it loaded, and a file at a standard path is this node’s kubelet’s only if this node’s kubelet reads it, so the section says it could not look rather than naming the policy of whichever file happened to sit there.

The exit code is non-zero only for an unavailable accelerator answer. A capability this generation does not declare, a manufacturer nothing is checked for, a node carrying none of its hardware, an answer that went no deeper and a step that was emitted are all answers, and a run that reports them has done its job.

A broken RDMA link does not fail the run. It withholds a node label, which changes what a flavor selects rather than what an allocator can hand out.

Neither does the topology section, whatever it says (including a policy it could not read at all). The policy decides which placements the kubelet admits, not what this node can hand out, so failing the run on it would refuse nodes that allocate perfectly well.

Why — this exit code answers whether the node can serve the allocation modes its allocators offer, and a down link stops none of them. The reasoning is recorded in the feature’s spec.

The document is written whether or not the node passed, so a failing run is still readable rather than leaving the exit code as the only thing to debug from.

Only one preflight runs on a node at a time. Every probe container carries the same label, so a second run’s stale sweep would remove the first run’s live probes, and an accelerator whose probe was killed mid-measurement reports as unable to slice. A second run therefore refuses before it sweeps anything, reports every manufacturer unavailable naming the lock, and exits non-zero. --dry-run is exempt: it starts nothing and writes nothing.

A failed detection stops that manufacturer’s other two questions. Detection is the floor the rest stands on, so when it reads unavailable (the container cannot see hardware the host reports, or the accelerators answered but could not be named) the group carries its detection block, no rows, and a note saying the remaining questions are unanswerable. Fix what the detection reason names and run it again; rows about accelerators the report cannot identify would not have helped.

One manufacturer’s crash does not take the others down. Everything reached for a manufacturer is vendor code over a driver a half-installed node can leave in any state, and that is the node this command exists to be run on.

A panic in one is contained to that one: its group keeps the detection block it had already answered, discards whatever the crashed pass had filled in, and reports one unavailable row per accelerator plus a note with what the panic said. The other manufacturers are read and reported as usual.

- manufacturer: cambricon
  checks:
    - accelerator: MLU-0
      capability: preflight-panicked
      state: unavailable
      reason: >-
        this manufacturer's preflight panicked and was contained, so none of this
        accelerator's preconditions were established: runtime error: invalid memory address
  note: 'this manufacturer''s preflight panicked and was contained: whatever it had established
    is discarded, and every accelerator it was asked about is reported unavailable below; the
    rest of this run is unaffected: runtime error: invalid memory address'

The rows are what make it exit non-zero: a crash that left only a note would be a run that verified nothing and said it passed. Treat it as a bug worth reporting, with the -v=3 output attached; the stack trace goes to the log rather than the document.

A sweep that could not be completed fails the accelerators it could not clear. Before anything is started, the run removes every container left behind by an earlier one, see Side effects . Where that sweep fails, the run carries one stale-container-sweep row per accelerator:

- accelerator: GPU-0
  capability: stale-container-sweep
  state: unavailable
  reason: 'the stale-container sweep could not be completed, so a container an earlier run left
    behind may still be holding this accelerator and every answer measured against it here is
    unsafe to trust: docker: exit status 1: permission denied'

It is a failure and not a warning because of what a leftover does: it still holds its accelerator, so the slice measured against it comes back short and the report blames the card. A run that swept nothing, measured anyway and exited zero would state the opposite of the truth about hardware that is fine. Clear the leftovers by hand (docker ps -aq --filter label=gpustack.ai/preflight=true, and the same for nerdctl with --namespace gpustack-preflight), then run it again.

Not every unhappy sweep is one of these. The sweep lists by label and then removes what it found, and the two are told apart:

The last row is why a node carrying nerdctl beside a containerd that is not listening stays green: the CLI is there, its socket file may even be there, but nothing can be behind it.