Settings & Environment Variables

GPUStack Operator reads configuration from two places: the Setting resource, which you can change at runtime, and environment variables, which the worker reads when it starts.

The worker aligns every Setting on every start. A GPUSTACK_* variable present in the worker process environment is a standing declaration rather than a one-time default: while it holds a non-blank value, the worker writes it to the Setting on every start and the stored value loses. A Setting named in settings-default-tracking is written with the operator’s built-in default on every start. Every other Setting keeps the value you stored. Who owns a value states the four rules in the order they run, and Finding the owning rule tells you which of them owns yours.

Contents

• Who owns a value
• Settings
• Troubleshooting
• Deployment environment variables

Who owns a value

Every registered Setting lives as one key of a single delegated Secret, gpustack-settings in the system namespace. The worker aligns that Secret once per worker start and asks one question about each key: does the operator own this value, or does the administrator? A value the operator owns is written on that start and logged; a value it does not own is left exactly as it was stored.

Write D for a Setting’s seed: the value of GPUSTACK_<UPPER_SNAKE_NAME> when that variable is present in the worker process environment, and the built-in default when the variable is absent. A variable present with an empty value yields a blank seed rather than the built-in default, and {} and [] read as blank too.

The alignment applies four rules, in this order:

1. Backfill. A key the Secret does not carry is added with its seed. A blank seed is a key like any other, so a Setting with a blank default still appears in kubectl get settings.
2. A declared variable wins. If the worker environment declares the variable for the Setting and the seed is not blank, the seed replaces the stored value.
3. A tracked Setting follows the operator’s default. If settings-default-tracking names the Setting and no declared variable has claimed it, the built-in default replaces the stored value.
4. Otherwise the stored value is the administrator’s. Nothing writes it, a blank one included.

A Setting that is both declared and tracked is written by rule 2, with the variable’s value. The two rules cannot disagree, because a Setting has one seed and both of them write it.

Rules 2, 3 and 4 never write a blank value, rule 4 writing nothing at all, which is why a variable set to the empty string takes a Setting over nothing and a tracked Setting whose default is blank keeps what you stored. Only backfill writes a blank, and it runs first so that a newly registered Setting appears in the same pass that decides whether the operator owns it.

Deleting the Secret is the case with nothing to preserve: the next worker start creates it again from every seed, and settings-default-tracking returns to the six names it ships with.

Every override is logged

A start that changes a value writes one line per Setting to the worker log, naming the Setting, the value it replaced, the value it wrote and the rule that fired:

setting overridden on start setting="container-registry" stored="registry.example" value="mirror.example" rule="env-declared" envName="GPUSTACK_CONTAINER_REGISTRY"

rule is env-declared for rule 2 and default-tracked for rule 3, and envName names the variable when the Setting reads one. A start that changes nothing logs nothing, and a stored value that already equals the seed is not an override.

A Setting the operator marks sensitive logs neither value. Its line carries the name, the rule and the variable — enough to tell you which Setting changed and why — and replaces both values with values="[withheld: sensitive setting]". The log reaches a wider audience than this Secret, so a value kept out of the Secret stays out of the log. No registered Setting carries that mark today; the operator sets it on a Setting whose value should not appear in a log at all.

kubectl -n gpustack-system logs deploy/gpustack-operator-worker | grep 'setting overridden on start'

Tracking list

settings-default-tracking holds a comma-separated list of Setting names. The worker reads it from the Secret, and the backfill has already written the six names the operator ships into a Secret that carries no such key, so a cluster that predates the Setting is tracked from the first start that backfills the key.

A stored value cannot silently change what is tracked. Each segment is trimmed, and a segment left empty, a duplicate, the list’s own name and a name this operator does not register are dropped, each logged once at verbosity 4. A write that names the list itself is refused outright, because a Setting tracking its own membership would reconcile that membership against itself.

Finding the owning rule

Two checks place any Setting you did not write yourself, and they are the two the rules read:

# 1. Does the worker Deployment render a variable for it, and with what value?
kubectl -n gpustack-system get deploy -o yaml | grep -A1 'GPUSTACK_'

# 2. Does the tracking list name it?
kubectl -n gpustack-system get setting settings-default-tracking -o yaml

• The Deployment renders the variable with a non-blank value, so rule 2 owns the Setting whatever is stored. The Settings list identifies the chart-rendered variables.
• The Deployment does not render it and the tracking list names it, so rule 3 owns the Setting.
• Neither, so rule 4 applies and the value you stored stands until you change it yourself.

Settings

A fixed catalog of named values, served as the namespaced Setting aggregated API resource (gpustack.ai/v1, short name set, category gpustack) and read at runtime with kubectl; when a new value applies is the Effect window on each row. Fixed means the resource serves get,list,watch,apply,update,patch and no create/delete: you edit a Setting’s value, not its existence.

The name in parentheses beside a Setting is the variable it reads. Every one of them is authoritative on every worker start while the variable is present and holds a non-blank value, whether the chart renders it or you add it yourself (who owns a value ). A variable marked chart-rendered is one the chart writes onto the worker Deployment itself, so the chart re-asserts it on every start and every upgrade.

# List every setting and its current value
kubectl -n gpustack-system get settings          # or: kubectl -n gpustack-system get set

# Inspect / change one
kubectl -n gpustack-system get setting node-management-manual -o yaml
kubectl -n gpustack-system edit setting node-management-manual
kubectl -n gpustack-system patch setting instance-type-derived-from-node --type merge -p '{"spec":{"value":"false"}}'

Effect windows — Immediate: the change does its own work. A consuming controller watches the delegated settings Secret and reconciles what the Setting governs, or an admission webhook starts judging every call by the new value. Values reach a process through a thirty-second read cache, so Immediate means within about that.

Next trigger — nothing moves on its own. The new value applies when the next natural event named in its Usage line arrives — a reconcile, a Pod render, a resolution or revalidation tick, or a request that reaches a webhook. Neither window needs a restart.

Authoring the InstanceType

With instance-type-derived-from-node=false the operator authors no InstanceType, so the administrator owns both halves of a pool: which InstanceTypes exist, and whether what their queue admits can actually be placed. Flipping the setting off does not retire the types the operator already authored; the derived marker is provenance, and nothing auto-removes a type.

No ClusterQueue gains a reference to the node-devices feasibility gate in this mode. That AdmissionCheck has one writer, NodeQueueReconciler , which reads this setting as a cluster-wide switch, not as “was this queue derived”.

So with the switch on, every accelerated queue carries the reference once the check reports Active, whoever authored its InstanceType; with it off none is added, and a queue that already carries one drops it the next time its resource groups are refilled, not at the moment of the flip.

Why it is not attached in this mode anyway — the gate changes what a queue admits, and in the mode where the administrator authors every InstanceType that is theirs to decide.

The failure shape to expect: without that gate, a Workload is admitted on quota alone and its Pods then sit Pending, which reads as a full cluster while the usual truth is a request no node can place, because quota is a scalar total and cannot see per-accelerator fragmentation. When the Pods of an admitted Workload stay Pending, compare the request against the per-accelerator ledger (kubectl get devices <node> -o yaml) before adding capacity.

The joint-admission check is attached in both modes. Every queue backing an InstanceType references gpustack-model-deployment-joint once it reports Active, whoever authored the type and whatever this setting says.

It answers Ready at once for every Workload that is not a replica of a multi-role ModelDeployment, so the only thing it changes in this mode is that such a deployment is admitted as a set; see Prefill and decode , and Status for a set that stays short of quota.

Write the InstanceType against the InstanceTypeFlavor catalog. It is the read-only, os/arch-agnostic view of the pools that actually exist (one entry per grouping the settings above produce), and its spec carries the identity fields an InstanceType is built from: acceleratorGroup, generalGroup, acceleratable, manufacturer, product, family, memory, cores.

kubectl get instancetypeflavors                               # short name: instypeflavor
kubectl get instancetypeflavor gpustack--nvidia-a10g -o yaml

An InstanceType whose identity matches no entry there backs a ClusterQueue that selects no ResourceFlavor, so the queue is left with empty resource groups and admits nothing.

SGLang TIME-WAIT reuse

model-deployment-tcp-tw-reuse renders net.ipv4.tcp_tw_reuse=1 into the Pod securityContext.sysctls of the prefill half of every SGLang prefill/decode pair, and of no other Pod. The sysctl lets the kernel reuse ports held by TIME-WAIT sockets for new outgoing connections, in that Pod’s own network namespace only.

The prefill half opens a new TCP connection for every transfer, so over TCP, with or without a store, its TIME-WAIT sockets use up its local ports under sustained load.

It does not reach the decode half, which accepts transfers rather than opening them; a vLLM role, which keeps its transfer connections open; or the members of a KV cache backend . A role that replaced its command gets nothing either, since the operator did not build what runs there. Whether an SGLang server with a store runs out of ports is not measured, and the setting does not reach it.

1. Allow the sysctl on the kubelet of every node that can run such a Pod. It is not on the Kubernetes safe list , so a kubelet refuses it until told otherwise. Add it to the kubelet configuration and restart the kubelet:

# KubeletConfiguration
allowedUnsafeSysctls:
  - net.ipv4.tcp_tw_reuse

The command-line form is --allowed-unsafe-sysctls=net.ipv4.tcp_tw_reuse. It is one list, so keep any name already on it.

On a managed cluster, set it where the node pool’s kubelet configuration comes from, so a node added or replaced later keeps it; an edit made on a running node leaves with that node. On each of these it takes effect on newly created nodes:

• EKS on Amazon Linux 2023 — the launch template’s nodeadm NodeConfig, under spec.kubelet.config as above or as the flag under spec.kubelet.flags; see the nodeadm API . On Amazon Linux 2, pass the flag through the bootstrap script’s --kubelet-extra-args.
• AKS — allowedUnsafeSysctls in the kubelet configuration a node pool is created with; see custom node configuration .
• GKE — kubeletConfig.allowedUnsafeSysctls in the node system configuration of a node pool; see node system configuration .
• A platform that exposes no kubelet configuration cannot run this setting; leave it off there.

2. Turn the setting on.

kubectl -n gpustack-system patch setting model-deployment-tcp-tw-reuse --type merge -p '{"spec":{"value":"true"}}'

Flipping it recreates the prefill replicas it reaches, in either direction, because the sysctl is part of the Pod spec their fingerprint covers. Each deployment picks the value up on its next reconcile, and a setting change does not wake one. To apply it at once, delete one of the deployment’s prefill Pods: the reconcile that replaces it renders the new value, and the rollout replaces the other prefill replicas. A pair that has already locked up recovers the same way.

If the kubelet was not changed, the prefill replica never starts. The node’s kubelet refuses the Pod, which ends in phase Failed with reason SysctlForbidden; the operator deletes it and logs removing replica that failed with that reason, and the replacement is refused the same way, over and over. The events outlive the Pods, and their message names the sysctl:

kubectl -n <namespace> get events --field-selector reason=SysctlForbidden
kubectl -n <namespace> get pods -w \
  -l app.kubernetes.io/instance=<deployment>,app.kubernetes.io/component=<prefill-role>

The second command shows the prefill Pods cycling through SysctlForbidden. Allow the sysctl on those nodes or turn the setting off; either ends the loop at the next replacement.

A namespace enforcing the baseline or restricted Pod Security Standard refuses the Pod at creation instead, since the sysctl is not on that standard’s allowed list either: no Pod appears, and the operator logs the refusal.

Troubleshooting

Both symptoms below are rules 2 and 3 seen from the other side. A value that changed without your action was written by the operator at a worker start, and the override log line says which rule wrote it, along with the variable when one did.

Stored value stopped applying

Rule 2 wrote a variable’s value over the one you stored. The log line reports it as rule="env-declared" and names the variable in envName, so you do not have to work out which declaration claimed the Setting.

The way out is the variable, not the Secret:

• A variable you set yourself, through the chart’s worker.env or by editing the worker Deployment. Change it or remove it and your stored value stands at the next worker start. A hand-edited Deployment is outside the chart’s record, so a rollback or an upgrade run with --reset-then-reuse-values drops it; a variable meant to last belongs in a chart value.
• A variable the chart renders, the five rows marked above. Change the chart value, or clear it so the chart renders an empty one, and helm upgrade. A blank declaration claims nothing, so clearing the chart value is what hands the Setting back to you.

Setting the chart value to an empty string is the same as clearing it. With modelManager.enabled: false the chart stops rendering the delivery mode entirely, and the stored value is then nobody’s to write.

Edit reverted on restart

Either rule wrote over the edit, and rule in the log line says which one:

• rule="env-declared" means a variable holds the value. Changing or clearing that variable is the way out, as above.
• rule="default-tracked" means the Setting’s name is in settings-default-tracking. Removing the name from that list is the way out.

Your edit holds until the next worker start, and nothing reverts it while the worker runs. A node drain, a crash, a restart and an upgrade each start a worker, so each of them applies the rules again.

Keeping an image pin

An image pin does not survive an operator upgrade: the six image Settings are tracked by default, so the operator writes each of them with its built-in default on every worker start, not only on the start where a release changes that default. A pin made with kubectl patch holds until the next start, and reverts to the default at that one:

kubectl -n gpustack-system patch setting kv-cache-backend-image --type merge \
  -p '{"spec":{"value":"mirror.example/kv-cache:1.2.3"}}'

Untracking the Setting is the only thing that keeps the pin, and nothing else stands in for it: a variable set to an empty value is no declaration, and a default that is blank is never written at all.

# Read the list, drop the one name whose Setting you want to keep your own value for,
# and write the rest back. This removes model-prefetch-warmup-image and tracks the other five.
kubectl -n gpustack-system get setting settings-default-tracking -o yaml
kubectl -n gpustack-system patch setting settings-default-tracking --type merge \
  -p '{"spec":{"value":"instance-ssh-server-image,kv-cache-backend-image,model-deployment-router-image,model-deployment-router-proxy-image,model-deployment-routing-sidecar-image"}}'

A pin that predates this behavior is replaced on the first worker start of the release that introduces it, and the log line names both values when it happens. Deleting the Secret has the same effect on the list, which comes back as its six default names.

Deployment environment variables

These environment variables are read when the process starts, so changing one means restarting or redeploying the affected component. A variable a Setting reads is re-read on each of those starts rather than applied once (who owns a value ); the rest are plain process configuration.

The Worker (WK) copies every GPUSTACK_-prefixed variable from its own Pod spec onto the Device Manager (DM) DaemonSets, so setting one on the Worker Deployment reaches the DMs automatically.

General variables

Disabling an allocator safeguard

Two device-plugin safeguards default on, and each has a switch for the case where it misjudges a node. Set it through the chart, which renders deviceManager.env onto every Device Manager DaemonSet:

helm upgrade gpustack-operator <chart> --namespace gpustack-system --reset-then-reuse-values \
  --set-string deviceManager.env.GPUSTACK_DEVICE_PLUGIN_SLICED_ALLOCATE_GATE=false

• GPUSTACK_DEVICE_PLUGIN_SLICED_ALLOCATE_GATE=false is for a slice refused although its accelerator has room: the Pod keeps coming back UnexpectedAdmissionError naming that accelerator. With it off, the slice is allocated and the ledger clamps that accelerator’s Remaining at zero.
• GPUSTACK_DEVICE_PLUGIN_IDENTIFY_BY_KUBELET=false is for allocations recorded on the wrong Pod while the kubelet is reachable. With it off, the Device Manager picks the oldest pending Pod the request could be for, which can record two Pods bound together on each other.

The change rolls the Device Manager Pods, one node at a time. Running workloads are not touched: a Device Manager restart does not reach a started container, and its allocation records live on the Pods, where the restarted process reads them. A Pod admitted while its node’s Device Manager is restarting can fail admission, as across any Device Manager restart, and its controller recreates it.

To switch the safeguard back on, remove the value. Set it to null with --set, not --set-string, which would store the string "null":

helm upgrade gpustack-operator <chart> --namespace gpustack-system --reset-then-reuse-values \
  --set deviceManager.env.GPUSTACK_DEVICE_PLUGIN_SLICED_ALLOCATE_GATE=null

Per-manufacturer overrides

Override patterns expand for every known manufacturer (amd, ascend, cambricon, hygon, iluvatar, metax, mthreads, nvidia, thead); both the WK and the DM read them, so the propagation above keeps the two sides consistent.

Installed by the chart, set the global.manufacturers row, not these variables. The overrides here are fields of that row (pciVendorID, resourceName, runtimeName, partitionKind), which the chart fans out as the matching variable to the worker and the Device Managers, along with what a variable cannot reach: the DaemonSet node selectors, the RuntimeClasses it creates, Kueue’s credits mapping. The variable alone leaves those stale; the next render overwrites it.

• GPUSTACK_${MANUFACTURER}_PCI_VENDOR_ID — the PCI vendor ID used for NFD node selection and device scanning. Accepts ${vendor} or ${class}_${vendor}.
• GPUSTACK_${MANUFACTURER}_ACCELERATABLE_RESOURCE_NAME — the extended resource name the scheduling chain allocates against.
• GPUSTACK_${MANUFACTURER}_ACCELERATABLE_RUNTIME_NAME — the container runtime class name used for accelerated workloads. A RuntimeClass of that name is attached only when one exists in the cluster; the chart creates one only where the global.manufacturers row sets a runtimeInjectsDriver or runtimeInjectsDevices fact (see global.manufacturers) — never for a manufacturer setting neither, whose default below holds only if something else created the class.

Defaults:

T-Head has no default runtime name, but GPUSTACK_THEAD_ACCELERATABLE_RUNTIME_NAME is still honored and can supply one.

A fourth override applies only to a manufacturer that has hardware partitioning at all:

• GPUSTACK_${MANUFACTURER}_PARTITION_KIND — the manufacturer’s own name for hardware partitioning, which becomes the segment prefix of its per-profile resource key.

nvidia is the only one with a default: mig, giving nvidia.com/gpu.partitioned.mig-${profile}. Elsewhere the variable does nothing: without hardware partitioning there is no .partitioned family, so no key segment to rename.

A fifth override is per manufacturer, and today only NVIDIA reads it:

• GPUSTACK_${MANUFACTURER}_DEVICE_INJECTION_STRATEGY — which channel that manufacturer’s allocator uses to make a granted accelerator reach a container: envvar (default, today’s *_VISIBLE_DEVICES behavior, byte-identical), cdi-annotations (the granted accelerator is requested as a cdi.k8s.io/* device-plugin annotation), or auto (opt-in detection, falling back to envvar with the reason logged whenever it cannot confirm CDI is safe on that node).

A manufacturer whose allocator injects /dev nodes itself, or that ships no CDI generator, has nothing for the CDI channel to resolve, so the variable does nothing there. The partitioned (MIG) family and partition-backed visibility always use envvar, whatever the strategy names, since a MIG instance is materialized at Allocate time and no pre-generated specification names it.

Prefer auto over naming the CDI channel yourself. The CDI channel needs a container engine that resolves CDI requests (containerd 2.x does; containerd 1.7 only with enable_cdi = true). Ask for it on an engine that does not and the request is simply ignored: no variable is set either, and the container starts with no accelerator and no error, which is the failure this setting exists to remove. auto reads the engine first and keeps the variable when the answer is no.

Two independent questions decide which value a runtime wants: does the engine resolve CDI requests at all, and can auto tell that it does? auto reads both answers out of containerd’s config.toml, so a runtime that keeps them anywhere else is invisible to it. That file is where the detection looks; it is not something the channel itself needs.

cdi-annotations is the answer on exactly one row: a runtime that does resolve CDI requests but that auto has no way to see. Everywhere else auto reaches the same channel by itself, or correctly declines to; naming the channel where the engine ignores it is how a container ends up running with nothing.

Where to set it: the name embeds the manufacturer in upper case, so it is not a literal you will find spelled out anywhere. Pass it through the chart’s deviceManager.env, which lands on every Device Manager DaemonSet:

helm upgrade ... --set deviceManager.env.GPUSTACK_NVIDIA_DEVICE_INJECTION_STRATEGY=auto

It is read once, when that manufacturer’s allocator is constructed, so a change takes effect only when the DaemonSet restarts. A value that is not one of the supported choices is reported and the node keeps envvar: refusing to start the allocator over it would take the node’s accelerators with it.

auto also keeps the variable when the engine’s own default runtime is already the vendor runtime. That is the usual shape on a distribution that ships the GPU toolkit for you. There every Pod runs under that runtime whether it asks to or not, so the variable works and a CDI request would only add a second injection path. auto doing nothing on such a node is the correct answer, not a misconfiguration.

Which channel a node settled on, and why, is logged once per answer, but at the allocator’s own verbosity, above what the DaemonSet ships with. Runtime log verbosity raises it on a running Pod.

One node-level variable feeds the same decision:

• GPUSTACK_CONTAINERD_CONFIG_DIR — the directory holding the container engine’s config.toml, from the chart’s deviceManager.containerdConfigDir (default /etc/containerd). auto reads it to learn whether the engine resolves CDI requests and which runtime handler a Pod naming no runtimeClassName runs under; unreadable keeps the node on envvar.

Manufacturer toolkit paths

The DM device bindings locate manufacturer libraries through conventional toolkit-home variables, each falling back to the listed default when unset.

Kubernetes-injected variables

Populated by the Pod specs GPUStack Operator renders (Downward API or Service environment); not user-facing settings, listed for completeness.

Proxy and internal flags