Command Reference

The gpustack-operator binary provides long-running services and one-shot commands. The services are what a deployment runs; the one-shots are what you run by hand on a node, and the first two sections choose between them.

Contents

• Command summary
• Choosing a one-shot command
• Global flags
• worker
• worker-gateway
• device-manager serve
• model-manager
• device-manager detect
• device-manager monitor
• device-manager preflight

Command summary

Choosing a one-shot command

The one-shot commands answer different questions; only preflight changes node state.

detect and monitor are reads no flag can turn into an action. preflight is the one that acts: it starts probe containers, stages the injected libraries onto the host, and, where a manufacturer’s sharing mode is off, asks the driver to turn it on to see whether it takes, putting it back either way. --dry-run is how it is made a read.

The probe containers it starts are themselves not privileged: they get the allocator’s injection and nothing more, because the isolation that injection establishes is the thing being measured.

Why detect is not folded into preflight — detect emits the whole ledger, typed as the Devices CRD’s own spec.groups, so it can be diffed against what the cluster recorded. preflight’s detection block is a verdict and a count, deliberately not a second ledger. A reader who wants both runs both.

Global flags

Every command accepts these. They come from klog and are omitted from the per-command tables below.

worker

It is the control plane: it profiles node capacity, runs the controllers that materialize the Kueue scheduling chain, serves the aggregated extension APIs and the admission webhooks, and installs the applications a cluster needs.

Runs as the operator Deployment. Flags below are the ones that change behaviour rather than tune a connection; see Settings for what is configured through the Setting CR instead.

gpustack-operator worker --secure-port=31443 --disable-applications=kueue

worker-gateway

Aggregates resources from upstream clusters and mirrors them for worker to read. Runs beside worker; by default the two speak over a unix socket rather than a port.

gpustack-operator worker-gateway --worker-conn-mode=loopback

device-manager serve

It is the per-node agent: it detects accelerators, keeps the Devices ledger current, samples utilization, registers with the kubelet as a device plugin and injects the vendor runtime into allocated containers.

Runs as the Device Manager DaemonSet. The --no-* mode flags are how a node is restricted to a subset of the allocation modes its hardware would otherwise offer.

gpustack-operator device-manager serve --manufacturer=nvidia --no-partitioned

model-manager

The per-node model cache. A CSI node plugin serving inline ephemeral volumes of the driver model.csi.gpustack.ai: it authorizes each mount against the Pod’s namespace, downloads and verifies a Hugging Face artifact’s files once per node, and writes that node’s NodeModelStore status. See Node Model Store .

Runs as the Model Manager DaemonSet beside the node-driver-registrar sidecar. Its configuration comes from its node’s NodeModelStore.spec, not from flags.

gpustack-operator model-manager --node-name="$(hostname)" --cache-root=/mnt/models

device-manager detect

Runs one detect pass and prints the result. It is a pure read: nothing is written, nothing is started, and no flag changes that.

The output is the Devices CRD’s own spec.groups, so it can be compared directly against what the cluster recorded; kubectl get devices <node> -o yaml beside it is a diff.

docker run --rm --privileged -v /dev:/dev -v /sys:/sys \
  gpustack/gpustack-operator:<tag> \
  gpustack-operator device-manager detect --manufacturer=amd

- id: radeon-rx-7800-xt
  manufacturer: amd
  name: AMD Radeon RX 7800 XT
  memory: 16368
  cores: 60
  driverVersion: 6.16.13
  runtimeVersion: "7.2"
  computeCapability: gfx1101
  family: GC 11.0.0
  accelerators:
    - id: GPU-5c88007d760374f3
      index: 0
      physicalIndexes:
        - 1
        - 128
      topology:
        pciBusId: "0000:04:00.0"
        numaAffinity: "0"
        cpuAffinity: 0-13
      status:
        unhealthy: false
        logicalSliced:
            coresPercentageOvercommit: true
            count: 128

A vendor’s user-space libraries have to be reachable. AMD has no container runtime that injects them, so a run without /opt/rocm mounted finds no accelerator on a node that has two. preflight’s host cross-check is what tells that apart from a node with none; see below .

device-manager monitor

Takes one utilization sample and prints it. A pure read, like detect, and it accepts the same flags.

docker run --rm --privileged -v /dev:/dev -v /sys:/sys \
  gpustack/gpustack-operator:<tag> \
  gpustack-operator device-manager monitor --manufacturer=amd

- manufacturer: amd
  timestamp: 2026-08-29T08:01:43.565038175+08:00
  accelerators:
    - id: GPU-5c88007d760374f3
      memory: 16368
      memoryUsage: 174
      memoryUtilization: 1
      coresUtilization: 0
      temperature: 44
      powerUsage: 8
      unhealthy: false

The same figures reach a cluster through the Instance metrics surfaces; see Instance Metrics .

device-manager preflight

It answers whether the allocation modes this node’s allocators offer actually work here. detect cannot answer that question, because a capability can be declared and still not work.

This is the only command an operator runs by hand that acts on the node. See Preflight for the procedure (what to mount, what it starts and removes, how to read each row).

Row states and depths

Each row carries a state, a depth and, where a container ran, the evidence it ran on.

A measured example

Taken from a run on two RX 7800 XT cards:

- accelerator: GPU-5c88007d760374f3
  capability: sliced-runtime-loaded
  state: ok
  depth: measured
  detail: the slicing runtime the injection mounts is loaded in the container
  command: /usr/sbin/chroot /host docker run --rm --label gpustack.ai/preflight=true … ubuntu:24.04 …
  evidence: |
    gpustack-preflight-maps-begin
    …
    7205a176b000-7205a176d000 r--p 00000000 fc:00 13006695   /usr/local/vrocm/libvrocm.so

The verdict is the mapping line, not the shim’s own greeting: a library that failed to load prints a message naming the same path, so only /proc/self/maps says it is actually there.

The same row under --dry-run carries the command and no evidence, and its depth is simulated: the allocator’s intent, proven, with nothing run to confirm it took.

Not every row on that node reaches measured. sliced-quota-in-force on the same two cards is simulated, because the vendor reader there reports nothing until an allocation has been charged and the probe allocates nothing: the injection sets a cap and no container was observed under it. A row is measured only where something was read back, never where the intent alone is known.

Runtime resolution

Resolved from the kubelet’s own CRI endpoint wherever the host names one, because that is what starts a container on this node in production. A host that names none (a bare machine, or a distribution keeping that configuration elsewhere) falls through to probing docker, nerdctl, ctr.

The sources are ordered, and the first that names an endpoint answers, so a standard kubelet configuration wins over a distribution drop-in rather than conflicting with it. The conflict that does stop a run is within one source: a glob matching two directories that disagree, where neither is the kubelet’s own by position. The steps are emitted instead, naming the endpoints in conflict, and --runtime is how you settle it.

ctr is never driven: ctr run has no --device, and its only alternative grants every device, so what it measured would not be the isolation the injection established. A node resolving to it gets the step emitted for nerdctl instead, against the socket and namespace ctr resolved, or for docker where nerdctl cannot pass this manufacturer’s vendor runtime either.

docker run --rm --privileged --network=host --runtime nvidia \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -v /:/host -v /dev:/dev -v /sys:/sys \
  gpustack/gpustack-operator:<tag> \
  gpustack-operator device-manager preflight --manufacturer=nvidia --dry-run

--network=host is not optional: the host’s CLI is entered through a chroot, which changes the root and not the network namespace, so an image pull would read the host’s resolver and find nothing answering.