Settings & Environment Variables

GPUStack Operator reads configuration from two places: the Setting resource, which you can change at runtime, and environment variables, which the process reads once at startup.

First-deploy seeding vs. runtime changes. On first deploy, settings.Initialize creates the delegated Secret gpustack-settings in the system namespace and seeds every Setting from GPUSTACK_<UPPER_SNAKE_NAME>, or the built-in default when that variable is unset. Later restarts only backfill missing Settings and never overwrite a stored value: a GPUSTACK_* variable is an initial seed only, so after the first deploy change the Setting resource, not the environment.

Contents

• Online-adjustable settings
• Deploy-time environment variables

Online-adjustable settings

A fixed catalog of named values, served as the namespaced Setting aggregated API resource (gpustack.ai/v1, short name set, category gpustack) and read at runtime with kubectl; the operator picks a new value up on its next reconcile. Fixed means the resource serves get,list,watch,apply,update,patch and no create/delete: you edit a Setting’s value, not its existence.

# List every setting and its current value
kubectl -n gpustack-system get settings          # or: kubectl -n gpustack-system get set

# Inspect / change one
kubectl -n gpustack-system get setting node-management-manual -o yaml
kubectl -n gpustack-system edit setting node-management-manual
kubectl -n gpustack-system patch setting instance-type-derived-from-node --type merge -p '{"spec":{"value":"false"}}'

These settings (node-management-manual, instance-type-mixed-on-node, instance-type-derived-from-node, instance-type-drain-when-no-flavors, instance-type-aware-cpu-manufacturer) are read per-reconcile (ShouldValueBool(ctx)): flipping one re-converges the scheduling chain on the next reconcile, with no restart.

Authoring the InstanceType

With instance-type-derived-from-node=false the operator authors no InstanceType, so the administrator owns both halves of a pool: which InstanceTypes exist, and whether what their queue admits can actually be placed. Flipping the setting off does not retire the types the operator already authored; the derived marker is provenance, and nothing auto-removes a type.

No ClusterQueue gains a reference to the node-devices feasibility gate in this mode. That AdmissionCheck has one writer, NodeQueueReconciler , which reads this setting as a cluster-wide switch, not as “was this queue derived”.

So with the switch on, every accelerated queue carries the reference once the check reports Active, whoever authored its InstanceType; with it off none is added, and a queue that already carries one drops it the next time its resource groups are refilled, not at the moment of the flip.

Why it is not attached in this mode anyway — the gate changes what a queue admits, and in the mode where the administrator authors every InstanceType that is theirs to decide.

The failure shape to expect: without that gate, a Workload is admitted on quota alone and its Pods then sit Pending, which reads as a full cluster while the usual truth is a request no node can place, because quota is a scalar total and cannot see per-accelerator fragmentation. When the Pods of an admitted Workload stay Pending, compare the request against the per-accelerator ledger (kubectl get devices <node> -o yaml) before adding capacity.

The joint-admission check is attached in both modes. Every queue backing an InstanceType references gpustack-model-deployment-joint once it reports Active, whoever authored the type and whatever this setting says.

It answers Ready at once for every Workload that is not a replica of a multi-role ModelDeployment, so the only thing it changes in this mode is that such a deployment is admitted as a set; see Prefill and decode , and Status for a set that stays short of quota.

Write the InstanceType against the InstanceTypeFlavor catalog. It is the read-only, os/arch-agnostic view of the pools that actually exist (one entry per grouping the settings above produce), and its spec carries the identity fields an InstanceType is built from: acceleratorGroup, generalGroup, acceleratable, manufacturer, product, family, memory, cores.

kubectl get instancetypeflavors                               # short name: instypeflavor
kubectl get instancetypeflavor gpustack--nvidia-a10g -o yaml

An InstanceType whose identity matches no entry there backs a ClusterQueue that selects no ResourceFlavor, so the queue is left with empty resource groups and admits nothing.

SGLang TIME-WAIT port reuse

model-deployment-tcp-tw-reuse renders net.ipv4.tcp_tw_reuse=1 into the Pod securityContext.sysctls of the prefill half of every SGLang prefill/decode pair, and of no other Pod. The sysctl lets the kernel reuse ports held by TIME-WAIT sockets for new outgoing connections, in that Pod’s own network namespace only.

The prefill half opens a new TCP connection for every transfer, so over TCP, with or without a store, its TIME-WAIT sockets use up its local ports under sustained load.

It does not reach the decode half, which accepts transfers rather than opening them; a vLLM role, which keeps its transfer connections open; or the members of a KV cache backend . A role that replaced its command gets nothing either, since the operator did not build what runs there. Whether an SGLang server with a store runs out of ports is not measured, and the setting does not reach it.

1. Allow the sysctl on the kubelet of every node that can run such a Pod. It is not on the Kubernetes safe list , so a kubelet refuses it until told otherwise. Add it to the kubelet configuration and restart the kubelet:

# KubeletConfiguration
allowedUnsafeSysctls:
  - net.ipv4.tcp_tw_reuse

The command-line form is --allowed-unsafe-sysctls=net.ipv4.tcp_tw_reuse. It is one list, so keep any name already on it.

On a managed cluster, set it where the node pool’s kubelet configuration comes from, so a node added or replaced later keeps it; an edit made on a running node leaves with that node. On each of these it takes effect on newly created nodes:

• EKS on Amazon Linux 2023 — the launch template’s nodeadm NodeConfig, under spec.kubelet.config as above or as the flag under spec.kubelet.flags; see the nodeadm API . On Amazon Linux 2, pass the flag through the bootstrap script’s --kubelet-extra-args.
• AKS — allowedUnsafeSysctls in the kubelet configuration a node pool is created with; see custom node configuration .
• GKE — kubeletConfig.allowedUnsafeSysctls in the node system configuration of a node pool; see node system configuration .
• A platform that exposes no kubelet configuration cannot run this setting; leave it off there.

2. Turn the setting on.

kubectl -n gpustack-system patch setting model-deployment-tcp-tw-reuse --type merge -p '{"spec":{"value":"true"}}'

Flipping it recreates the prefill replicas it reaches, in either direction, because the sysctl is part of the Pod spec their fingerprint covers. Each deployment picks the value up on its next reconcile, and a setting change does not wake one. To apply it at once, delete one of the deployment’s prefill Pods: the reconcile that replaces it renders the new value, and the rollout replaces the other prefill replicas. A pair that has already locked up recovers the same way.

If the kubelet was not changed, the prefill replica never starts. The node’s kubelet refuses the Pod, which ends in phase Failed with reason SysctlForbidden; the operator deletes it and logs removing replica that failed with that reason, and the replacement is refused the same way, over and over. The events outlive the Pods, and their message names the sysctl:

kubectl -n <namespace> get events --field-selector reason=SysctlForbidden
kubectl -n <namespace> get pods -w \
  -l app.kubernetes.io/instance=<deployment>,app.kubernetes.io/component=<prefill-role>

The second command shows the prefill Pods cycling through SysctlForbidden. Allow the sysctl on those nodes or turn the setting off; either ends the loop at the next replacement.

A namespace enforcing the baseline or restricted Pod Security Standard refuses the Pod at creation instead, since the sysctl is not on that standard’s allowed list either: no Pod appears, and the operator logs the refusal.

Deploy-time environment variables

Read once at process startup, so changing one means restarting or redeploying the affected component. The Worker (WK) copies every GPUSTACK_-prefixed variable from its own Pod spec onto the Device Manager (DM) DaemonSets, so setting one on the Worker Deployment reaches the DMs automatically.

General variables

Disabling an allocator safeguard

Two device-plugin safeguards default on, and each has a switch for the case where it misjudges a node. Set it through the chart, which renders deviceManager.env onto every device-manager DaemonSet:

helm upgrade gpustack-operator <chart> --namespace gpustack-system --reset-then-reuse-values \
  --set-string deviceManager.env.GPUSTACK_DEVICE_PLUGIN_SLICED_ALLOCATE_GATE=false

• GPUSTACK_DEVICE_PLUGIN_SLICED_ALLOCATE_GATE=false is for a slice refused although its accelerator has room: the Pod keeps coming back UnexpectedAdmissionError naming that accelerator. With it off, the slice is allocated and the ledger clamps that accelerator’s Remaining at zero.
• GPUSTACK_DEVICE_PLUGIN_IDENTIFY_BY_KUBELET=false is for allocations recorded on the wrong Pod while the kubelet is reachable. With it off, the device manager picks the oldest pending Pod the request could be for, which can record two Pods bound together on each other.

The change rolls the device-manager Pods, one node at a time. Running workloads are not touched: a device-manager restart does not reach a started container, and its allocation records live on the Pods, where the restarted process reads them. A Pod admitted while its node’s device manager is restarting can fail admission, as across any device-manager restart, and its controller recreates it.

To switch the safeguard back on, remove the value. Set it to null with --set, not --set-string, which would store the string "null":

helm upgrade gpustack-operator <chart> --namespace gpustack-system --reset-then-reuse-values \
  --set deviceManager.env.GPUSTACK_DEVICE_PLUGIN_SLICED_ALLOCATE_GATE=null

Per-manufacturer overrides

Override patterns expand for every known manufacturer (amd, ascend, cambricon, hygon, iluvatar, metax, mthreads, nvidia, thead); both the WK and the DM read them, so the propagation above keeps the two sides consistent.

Installed by the chart, set the global.manufacturers row, not these variables. The overrides here are fields of that row (pciVendorID, resourceName, runtimeName, partitionKind), which the chart fans out as the matching variable to the worker and the device-managers, along with what a variable cannot reach: the DaemonSet node selectors, the RuntimeClasses it creates, Kueue’s credits mapping. The variable alone leaves those stale; the next render overwrites it.

• GPUSTACK_${MANUFACTURER}_PCI_VENDOR_ID — the PCI vendor ID used for NFD node selection and device scanning. Accepts ${vendor} or ${class}_${vendor}.
• GPUSTACK_${MANUFACTURER}_ACCELERATABLE_RESOURCE_NAME — the extended resource name the scheduling chain allocates against.
• GPUSTACK_${MANUFACTURER}_ACCELERATABLE_RUNTIME_NAME — the container runtime class name used for accelerated workloads. A RuntimeClass of that name is attached only when one exists in the cluster; the chart creates one only where the global.manufacturers row sets a runtimeInjectsDriver or runtimeInjectsDevices fact (see global.manufacturers) — never for a manufacturer setting neither, whose default below holds only if something else created the class.

Defaults:

T-Head has no default runtime name, but GPUSTACK_THEAD_ACCELERATABLE_RUNTIME_NAME is still honored and can supply one.

A fourth override applies only to a manufacturer that has hardware partitioning at all:

• GPUSTACK_${MANUFACTURER}_PARTITION_KIND — the manufacturer’s own name for hardware partitioning, which becomes the segment prefix of its per-profile resource key.

nvidia is the only one with a default: mig, giving nvidia.com/gpu.partitioned.mig-${profile}. Elsewhere the variable does nothing: without hardware partitioning there is no .partitioned family, so no key segment to rename.

A fifth override is per manufacturer, and today only NVIDIA reads it:

• GPUSTACK_${MANUFACTURER}_DEVICE_INJECTION_STRATEGY — which channel that manufacturer’s allocator uses to make a granted accelerator reach a container: envvar (default, today’s *_VISIBLE_DEVICES behavior, byte-identical), cdi-annotations (the granted accelerator is requested as a cdi.k8s.io/* device-plugin annotation), or auto (opt-in detection, falling back to envvar with the reason logged whenever it cannot confirm CDI is safe on that node).

A manufacturer whose allocator injects /dev nodes itself, or that ships no CDI generator, has nothing for the CDI channel to resolve, so the variable does nothing there. The partitioned (MIG) family and partition-backed visibility always use envvar, whatever the strategy names, since a MIG instance is materialized at Allocate time and no pre-generated specification names it.

Prefer auto over naming the CDI channel yourself. The CDI channel needs a container engine that resolves CDI requests (containerd 2.x does; containerd 1.7 only with enable_cdi = true). Ask for it on an engine that does not and the request is simply ignored: no variable is set either, and the container starts with no accelerator and no error, which is the failure this setting exists to remove. auto reads the engine first and keeps the variable when the answer is no.

Two independent questions decide which value a runtime wants: does the engine resolve CDI requests at all, and can auto tell that it does? auto reads both answers out of containerd’s config.toml, so a runtime that keeps them anywhere else is invisible to it. That file is where the detection looks; it is not something the channel itself needs.

cdi-annotations is the answer on exactly one row: a runtime that does resolve CDI requests but that auto has no way to see. Everywhere else auto reaches the same channel by itself, or correctly declines to; naming the channel where the engine ignores it is how a container ends up running with nothing.

Where to set it: the name embeds the manufacturer in upper case, so it is not a literal you will find spelled out anywhere. Pass it through the chart’s deviceManager.env, which lands on every device-manager DaemonSet:

helm upgrade ... --set deviceManager.env.GPUSTACK_NVIDIA_DEVICE_INJECTION_STRATEGY=auto

It is read once, when that manufacturer’s allocator is constructed, so a change takes effect only when the DaemonSet restarts. A value that is not one of the supported choices is reported and the node keeps envvar: refusing to start the allocator over it would take the node’s accelerators with it.

auto also keeps the variable when the engine’s own default runtime is already the vendor runtime. That is the usual shape on a distribution that ships the GPU toolkit for you. There every Pod runs under that runtime whether it asks to or not, so the variable works and a CDI request would only add a second injection path. auto doing nothing on such a node is the correct answer, not a misconfiguration.

Which channel a node settled on, and why, is logged once per answer, but at the allocator’s own verbosity, above what the DaemonSet ships with. Runtime log verbosity raises it on a running Pod.

One node-level variable feeds the same decision:

• GPUSTACK_CONTAINERD_CONFIG_DIR — the directory holding the container engine’s config.toml, from the chart’s deviceManager.containerdConfigDir (default /etc/containerd). auto reads it to learn whether the engine resolves CDI requests and which runtime handler a Pod naming no runtimeClassName runs under; unreadable keeps the node on envvar.

Manufacturer toolkit paths

The DM device bindings locate manufacturer libraries through conventional toolkit-home variables, each falling back to the listed default when unset.

Kubernetes-injected variables

Populated by the Pod specs GPUStack Operator renders (Downward API or Service environment); not user-facing knobs, listed for completeness.

Proxy and internal flags