Skip to content

Environment Variables

GPUStack supports various environment variables for configuration.

Most command line parameters can also be set via environment variables with the GPUSTACK_ prefix and in uppercase format (e.g., --data-dir can be set via GPUSTACK_DATA_DIR).

For a complete list of command line parameters that can be set as environment variables, see CLI Reference. This will not be discussed here.

Priority Order

Configuration values are applied in the following priority order (highest to lowest):

  1. Command line arguments
  2. Environment variables
  3. Configuration file
  4. Default values

This means that command line arguments will always override environment variables, and environment variables will override values in the configuration file.

GPUStack Core Environment Variables

These environment variables are typically used for third-party service integrations.

The Applies to column indicates where the environment variable should be set:

  • Server - Applies to the GPUStack server.
  • Worker - Applies to GPUStack workers.
  • Model - Applies to model deployment configurations.

Proxy Configuration

Variable
 Description Default Applies to
HTTP_PROXY HTTP proxy URL. e.g., http://proxy-server:port (empty) Server & Worker
HTTPS_PROXY HTTPS proxy URL. e.g., https://proxy-server:port (empty) Server & Worker
NO_PROXY Comma-separated list of hosts to exclude. e.g., 127.0.0.1,10.0.0.0/8,192.168.0.0/16,172.16.0.0/16,localhost,cluster.local (empty) Server & Worker

Hugging Face Hub

Variable
 Description Default Applies to
HF_ENDPOINT Hugging Face Hub endpoint. e.g., https://hf-mirror.com (empty) Server & Worker
HF_TOKEN Hugging Face Hub access token. (empty) Server & Worker

Debugging

Variable Description Default Applies to
GPUSTACK_DEBUG_ENABLE_TRACEMALLOC Enable Python memory allocation tracing at startup. Independent of --debug; tracing adds CPU and memory overhead. Restart GPUStack after changing this setting. false Server & Worker

Database Configuration

Variable Description Default Applies to
GPUSTACK_DB_ECHO Enable database query logging. false Server
GPUSTACK_DB_POOL_SIZE Database connection pool size. 30 Server
GPUSTACK_DB_MAX_OVERFLOW Database connection pool max overflow. 20 Server
GPUSTACK_DB_POOL_TIMEOUT Database connection pool timeout in seconds. 30 Server
GPUSTACK_DB_POOL_RECYCLE Database connection recycle time in seconds. 1800 Server

Network Configuration

Variable Description Default Applies to
GPUSTACK_PROXY_TIMEOUT_SECONDS Absolute ceiling on a proxied inference request, in seconds. 1800 Server & Worker
GPUSTACK_PROXY_TTFT_TIMEOUT_SECONDS Backstop on the wait for the first chunk of a streaming response, in seconds. Covers queueing and prefill, so it stays generous. 300 Server & Worker
GPUSTACK_PROXY_STREAM_IDLE_TIMEOUT_SECONDS Maximum gap between chunks once a streaming response has started, in seconds. Never applied to a non-streaming response, whose body only arrives at the end. 60 Server & Worker
GPUSTACK_PROXY_UPSTREAM_IDLE_TIMEOUT_SECONDS Upstream connection-pool idle timeout for higress, in seconds. Unrelated to request duration: it reaps idle connections, it does not bound a request. 3 Server
GPUSTACK_TCP_CONNECTOR_LIMIT HTTP client TCP connector limit. 1000 Server & Worker

Server Cache Configuration

Variable Description Default Applies to
GPUSTACK_SERVER_CACHE_TTL_SECONDS Server cache TTL in seconds. 600 Server

Authentication & Security

Variable Description Default Applies to
GPUSTACK_JWT_TOKEN_EXPIRE_MINUTES JWT token expiration time in minutes. 120 Server
GPUSTACK_PROVIDER_TEST_EGRESS_ALLOWLIST Allowlist of CIDRs, IP addresses and DNS names that the provider endpoints dialing a caller-supplied base URL (/get-models, /test-model, /test-decision-model) may reach. A target is allowed only when its hostname equals or ends with one of the names (dot boundary), or every resolved address falls inside one of the CIDRs; anything else — including a non-http(s) base URL — is refused. Empty means no restriction. e.g., api.openai.com,10.0.1.0/24,vllm.internal.corp (empty) Server

Gateway Configuration

Variable Description Default Applies to
GPUSTACK_GATEWAY_PORT_CHECK_INTERVAL The interval in seconds of GPUStack Server checking embedded gateway listening port 2 Server
GPUSTACK_GATEWAY_PORT_CHECK_RETRY_COUNT The retry count of GPUStack Server checking embedded gateway listening port 300 Server
GPUSTACK_GATEWAY_AUTH_RECONCILE_INTERVAL_SECONDS How often the server recomputes, from the database, the API keys the gateway authenticates locally. Deletions that bypass the ORM emit no event, so on a public route this is the worst-case time such a key keeps working. 30 Server
GPUSTACK_GATEWAY_AUTH_ALLOW_CUSTOM_KEYS Whether a custom API key (one whose secret the user supplied) may be authenticated at the gateway. Off, it keeps working but asks the server on every request. On, the key is published into the gateway's configuration indexed by an unsalted fast hash of the secret itself — identical across deployments, so a weak secret falls to a precomputed table. custom imposes no entropy requirement, so turn this off where users choose their own keys — it is re-read on every reconcile, so it withdraws custom keys published while it was on, not just new ones. true Server
GPUSTACK_GATEWAY_AUTH_MAX_CR_BYTES Byte budget for the key tables and public-route rules the server writes into the gateway's auth plugin. Sized under etcd's ~1.5 MiB object limit; keys past it authenticate via the server on every request. 1100000 Server
GPUSTACK_GATEWAY_TLS_MIN_PROTOCOL_VERSION Lowest TLS protocol version the gateway's HTTPS listeners accept. One of TLSv1.0, TLSv1.1, TLSv1.2, TLSv1.3; anything else is refused at startup. Empty leaves Higress on its default, which accepts TLS 1.0 and 1.1. Ignored in incluster mode — see below. (empty) Server
GPUSTACK_GATEWAY_TLS_MAX_PROTOCOL_VERSION Highest version those listeners accept. Same values and same scope; rarely needed. (empty) Server

The TLS protocol version configuration is written onto the ingress named by GPUSTACK_GATEWAY_MIRROR_INGRESS_NAME, and every ingress GPUStack generates for an LLM route copies the bounds from it. In incluster mode that ingress belongs to the helm chart, so set server.ingress.tls.minProtocolVersion / maxProtocolVersion there instead — the variables are ignored, and setting one logs a warning. A change applies to that ingress immediately and to existing LLM routes at their next reconcile; restart the server to propagate at once.

Settings that end up inside a gateway plugin's configuration are set in the config file under gateway_plugin rather than here, so that each mechanism has one effective-time story — a value there takes effect on the next reconcile, a variable here on restart. The two below moved there and are deprecated. They are still honored, as the default the config file may override, so upgrading does not silently revert a deployment that had set them; setting either logs a warning.

Deprecated variable Description Replaced by Default Applies to
GPUSTACK_HIGRESS_EXT_AUTH_TIMEOUT_MS Gateway authentication call timeout, in milliseconds. gateway_plugin.gpustack-ext-auth.config.authz.timeout 30000 Server
GPUSTACK_GATEWAY_AI_STATISTICS_PLUGIN_CONTENT_TYPES Comma-separated content types metered for token usage. gateway_plugin.ai-statistics.config.enable_content_types application/json,text/event-stream Server

Usage Tracking Configuration

Variable Description Default Applies to
GPUSTACK_TIMEZONE Platform-wide IANA timezone (e.g. Asia/Shanghai) for bucketing all usage views and rendering Last Active / event times (see note below). Empty ⇒ use OS local timezone (TZ / /etc/localtime). Replaces the deprecated GPUSTACK_USAGE_ROLLUP_TIMEZONE (still honored as an alias). (empty) Server
GPUSTACK_USAGE_ESTIMATED_BYTES_PER_INPUT_TOKEN Bytes-per-token divisor used to estimate prompt tokens when a gateway report arrives with completed=false and input_token is blank. Defaults target English-leaning GPT-style tokenizers; lower it (e.g. 2) for CJK-heavy workloads. 4 Server
GPUSTACK_USAGE_ESTIMATED_TOKENS_PER_OUTPUT_CHUNK Tokens-per-chunk multiplier used to estimate completion tokens for incomplete streams when output_token is blank. 1 Server
GPUSTACK_USAGE_DETAILS_RETENTION_MONTHS Retention window for model_usage_details (the per-request audit table). Rows older than this (anchored on COALESCE(completed_at, created_at)) are moved to model_usage_details_archive by the leader-only archiver. 13 Server
GPUSTACK_USAGE_DETAILS_ARCHIVE_CRON Cron expression (UTC) for the usage-details archiver's recurring sweep. The archiver also runs once on server startup regardless of this schedule. 0 3 * * * Server
GPUSTACK_USAGE_DETAILS_ARCHIVE_BATCH_SIZE Per-batch row count for archival moves. Smaller batches keep transactions short on environments sensitive to replication lag; larger batches reduce round-trip overhead. 1000 Server
GPUSTACK_USAGE_DETAILS_BUFFER_MAX_SIZE Hard cap on the in-memory per-request details buffer held between flushes. Bounds memory growth when flushes fail persistently (DB down, schema drift). On overflow the oldest entries are dropped with a WARNING log. 100000 Server
GPUSTACK_RESOURCE_USAGE_TICK_SECONDS Interval (seconds) for the instance-usage collector's periodic tick, which flushes accumulated runtime for long-running GPU/CPU instances that have had no lifecycle event since the last tick. 300 Server
GPUSTACK_STORAGE_USAGE_TICK_SECONDS Interval (seconds) for the persistent-volume (storage) usage collector's periodic tick. 300 Server
GPUSTACK_METERED_USAGE_SEAL_GRACE_SECONDS Grace window (seconds) after an hour-bucket elapses before it is sealed (finalized). A bucket is sealed once now >= bucket_end + grace, absorbing late events / clock skew; keep it larger than the tick interval. Sealed buckets are immutable. 900 Server
GPUSTACK_METERED_USAGE_RETENTION_MONTHS Retention window for metered_usage (the hourly resource-usage rollup). Sealed rows older than this are moved to metered_usage_archive by the leader-only archiver. Must stay well above the collector's settlement horizon (hours). 13 Server
GPUSTACK_METERED_USAGE_ARCHIVE_CRON Cron expression (UTC) for the metered-usage archiver's recurring sweep. The archiver also runs once on server startup regardless of this schedule. 0 4 * * * Server
GPUSTACK_METERED_USAGE_ARCHIVE_BATCH_SIZE Per-batch row count for metered-usage archival moves. Smaller batches keep transactions short; larger batches reduce round-trip overhead. 5000 Server
GPUSTACK_USAGE_EVENTS_RETENTION_MONTHS Retention window for resource_events (the resource lifecycle / audit log). Rows older than this are moved to resource_events_archive by the leader-only archiver. 13 Server
GPUSTACK_USAGE_EVENTS_ARCHIVE_CRON Cron expression (UTC) for the resource-events archiver's recurring sweep. The archiver also runs once on server startup regardless of this schedule. 30 3 * * * Server
GPUSTACK_USAGE_EVENTS_ARCHIVE_BATCH_SIZE Per-batch row count for resource-events archival moves. 5000 Server
GPUSTACK_USAGE_BREAKDOWN_MAX_NO_PAGINATION_ROWS Max buckets an unpaginated (page=-1) breakdown may return; trend charts and exports fetch the whole series at once. A request exceeding it is rejected (HTTP 400), not silently truncated. Raise for wide dashboards, lower to cap memory. 50000 Server

GPUSTACK_TIMEZONE scope & DST

This one timezone governs every usage view: the daily model_usages token rollup, the GPU/storage time buckets, and the displayed Last Active and resource-event times. For a zone that observes DST this has two consequences worth knowing:

  • Near a DST transition an event's displayed clock time can fall on a different calendar day than the bucket it is counted in.
  • Re-running the same historical query after a DST switch can re-bucket older rows by an hour, so the result drifts over time.

Non-DST zones (UTC, Asia/Shanghai, …) are exact and unaffected.

Cluster Configuration

Variable Description Default Applies to
GPUSTACK_DEFAULT_CLUSTER_KUBERNETES If a default cluster is created automatically, it will use the Kubernetes provider when this variable is set. false Server

Content Source Configuration

Variable Description Default Applies to
GPUSTACK_BOOTSTRAP_WITH_EMBEDDED_SOURCES Bring every content kind (model catalog, community backends, built-in backend versions, cache providers) up on its embedded content instead of the published documents, so nothing is fetched from the OTA server. This sets the starting point only: a kind put in service afterwards stays there. Useful where the OTA server is unreachable, or to keep content pinned to the installed version. false (true on a development build; an empty value counts as unset) Server

Scheduler Configuration

Variable Description Default Applies to
GPUSTACK_SCHEDULER_SCALE_UP_PLACEMENT_MAX_SCORE Max placement score used by the scheduler placement scorer. 100 Server
GPUSTACK_SCHEDULER_SCALE_UP_LOCALITY_MAX_SCORE Max locality score added to placement score when model files already exist. 5 Server
GPUSTACK_SCHEDULER_SCALE_DOWN_STATUS_MAX_SCORE Scale-down max contribution for status scorer (normalized). 100 Server
GPUSTACK_SCHEDULER_SCALE_DOWN_OFFLOAD_MAX_SCORE Scale-down max contribution for offload scorer (normalized). 10 Server
GPUSTACK_SCHEDULER_SCALE_DOWN_PLACEMENT_MAX_SCORE Scale-down max contribution for placement scorer (normalized). 1 Server
GPUSTACK_SCALING_SCHEDULER_INTERVAL Interval in seconds at which scheduled scaling recomputes each model's replica count from its windows. The reconcile is level-triggered, so this bounds only how long a window boundary can go unnoticed, never correctness. Clamped to a minimum of 1 second. 30 Server

Multi-Role Group Scheduling

These weights and windows apply to deployments that declare roles — a prefill-decode disaggregated group and its router. They have no effect on a plain single-role deployment. Every score below is a maximum contribution to the scheduler's score chain; setting one to 0 removes that consideration entirely.

Variable Description Default Applies to
GPUSTACK_SCHEDULER_TOPOLOGY_PROXIMITY_MAX_SCORE Max score per topology rung a candidate worker shares with the group's already-placed members, when scaling a role out or placing the router. Set above the placement scorer's 100-point spread so that spare capacity cannot outvote a whole rung of the declared topology. 0 places later members by resource fit alone. 150 Server
GPUSTACK_SCHEDULER_PAIRING_AFFINITY_MAX_SCORE Max score per opposite-role member already on a candidate worker, when scaling one role of a disaggregated group out — this is what makes KV transfer stay on the host. It is a floor, not the weight used: the scheduler takes the larger of this value and one more than the combined ceiling of the other scorers on the chain, so that "most opposite-role siblings wins, capacity breaks the tie" holds no matter which scorers are active. 0 is read as an off switch and is never raised to the floor. 200 Server
GPUSTACK_SCHEDULER_SCALE_DOWN_PAIRING_MAX_SCORE Max contribution to a member's keep-score from the opposite-role members sharing its worker, when scaling one role of a group in. It decides the victim in the ordinary case, where every surplus member is healthy and the status and offload scores tie. Keep it below 50, the status scorer's step between "starting" and "running" — at or above that, a starting member surrounded by peers outranks a healthy one sitting alone, and the scheduler stops deleting the already-broken member first. 0 chooses the victim by status and placement alone. 20 Server
GPUSTACK_SCHEDULER_DRAIN_WINDOW_SECONDS How long a scaled-down group member keeps running after it leaves the router's registry. Not a process grace period: it is the time the decodes still fetching KV cache from that member need to finish, which the engine has no way to wait for on its own. 0 deletes immediately, which is what a role-less deployment does regardless. 60 Server
GPUSTACK_SCHEDULER_GATHER_BLOCKED_DWELL_SECONDS How long a member of a MustGather group may sit unplaced before the deployment reports gather_blocked_scale_out. Nothing expires when it passes — it only separates "the scheduler has not run yet" from "the scheduler has run and there is nowhere inside the gather domain to put this". Every scale-up spends the first of those, so without the dwell the marker would appear on all of them. 0 reports the refusal as soon as an unplaced member exists. 120 Server

GPU Instance Configuration

Variable Description Default Applies to
GPUSTACK_GPU_INSTANCE_TRANSITIONING_REQUEUE_INTERVAL Interval in seconds at which the controller re-observes a still-transitioning (non-settled) GPU instance via an in-memory requeue; the PV / PVT finalize controllers reuse it to re-probe a still-finalizing instance. Ready-row drift is picked up by the downstream watch instead. Clamped to a minimum of 1 second. 15 Server
GPUSTACK_GPU_INSTANCE_READY_SWEEP_INTERVAL Interval in seconds for an opt-in low-frequency sweep that re-observes settled Ready GPU instances, a fallback for worker-side drift that the downstream watch could miss across a reconnect gap. 0 (default) disables it; set a low frequency only if such a coverage hole is observed. 0 Server
GPUSTACK_GPU_INSTANCE_PV_FINALIZING_UNREACHABLE_TOLERANCE How long in seconds the persistent volume (PV / PVT) finalizer waits for a Kubernetes cluster whose workers are all down, before it gives up on that cluster and releases the row anyway. A cluster with no worker at all is skipped at once — it never held the volume. A cluster that stays unreachable is waited for, then released, and its volume may survive there. 0 gives up at once — the opposite of GPUSTACK_GPU_INSTANCE_UNREADABLE_CR_TOLERANCE, whose 0 disables the bound. 1800 Server

Worker and Model Configuration

Variable Description Default Applies to
GPUSTACK_WORKER_HEARTBEAT_INTERVAL Worker heartbeat interval in seconds. 30 Worker
GPUSTACK_WORKER_STATUS_SYNC_INTERVAL Worker status synchronization interval in seconds. 30 Worker
GPUSTACK_WORKER_UNREACHABLE_CHECK_MODE Worker unreachable check mode. Options: auto, enabled, disabled. auto disables check when worker count > 50. auto Server
GPUSTACK_WORKER_HEARTBEAT_GRACE_PERIOD Worker heartbeat grace period in seconds. 150 Server
GPUSTACK_MODEL_INSTANCE_RESCHEDULE_GRACE_PERIOD Model instance reschedule grace period in seconds. 300 Server
GPUSTACK_MODEL_EVALUATION_CACHE_MAX_SIZE Maximum size of model evaluation cache. 1000 Server
GPUSTACK_MODEL_EVALUATION_CACHE_TTL TTL of model evaluation cache in seconds. 3600 Server
GPUSTACK_WORKER_ORPHAN_WORKLOAD_CLEANUP_GRACE_PERIOD Worker orphan workload cleanup grace period in seconds. 300 Worker
GPUSTACK_WORKER_ORPHAN_BENCHMARK_WORKLOAD_CLEANUP_GRACE_PERIOD Worker orphan benchmark workload cleanup grace period in seconds. 300 Worker
GPUSTACK_WORKER_STATUS_COLLECTION_LOG_SLOW_SECONDS Add debug log for slow worker status collection if it exceeds this time in seconds. 180 Worker
GPUSTACK_MODEL_INSTANCE_HEALTH_CHECK_INTERVAL Model instance health check interval in seconds. 3 Worker
GPUSTACK_DISABLE_OS_FILELOCK Disable OS file lock. false Worker
GPUSTACK_ENABLE_CUDA_MINOR_VERSION_COMPATIBILITY Allow lower-minor CUDA devices to run higher-minor images. Set globally on the worker or per model; per-model takes precedence. false Worker & Model
GPUSTACK_RESTART_IN_FLIGHT_LAPSE_SECONDS How long POST /models/{id}/restart keeps rejecting a second request with HTTP 409 while the first restart is still rebuilding — without the guard, the second teardown deletes the replacements the first one just created. The guard lapses on its own so that a deployment which never reaches running can still be restarted; size it to outlast a cold start that pulls an image and reads weights off disk. 900 Server
GPUSTACK_SERVE_LOG_MAX_BYTES Byte budget for one restart of one serving log. Past it the head is kept, the tail rotates, and a marker sits between them. 0 lifts the cap. 67108864 Worker
GPUSTACK_SERVE_LOG_HEAD_BYTES How much of that budget the head keeps, written once and never rewritten. Capped at half the budget; 0 keeps only the first line. 8388608 Worker

Benchmark Configuration

Variable Description Default Applies to
GPUSTACK_BENCHMARK_DATASET_SHAREGPT_PATH ShareGPT dataset path used by the benchmark container when dataset_name is set to ShareGPT. The default image already includes this dataset. /workspace/benchmark-runner/sharegpt_data/ShareGPT_V3_unfiltered_cleaned_split.json Worker
GPUSTACK_BENCHMARK_REQUEST_TIMEOUT Timeout for each benchmark request in seconds. 3600 Worker

Model Deployment Configuration

Note

These environment variables are not set when starting GPUStack. Instead, they should be configured in the Advanced Options > Environment Variables section when deploying a model. They are used to customize the model serving behavior.

Variable
 Description Default Applies to
GPUSTACK_MODEL_SERVING_COMMAND_SCRIPT_DISABLED Disable the automatic serving command script execution. When set to 1 or true, the script that handles package installation and other setup tasks will not run. 0 Model
PYPI_PACKAGES_INSTALL Additional PyPI packages to install in the model serving environment. Multiple packages should be space-separated. The script will use uv pip install if available, otherwise pip install. (empty) Model
GPUSTACK_MODEL_RAM_CLAIM User-declared RAM requirement (in Byte) for the model, used by the scheduler for capacity planning. (empty) Model
GPUSTACK_MODEL_VRAM_CLAIM User-declared VRAM requirement (in Byte) for the model, used by the scheduler for capacity planning. (empty) Model
GPUSTACK_APPLY_QWEN3_RERANKER_TEMPLATES Apply Qwen3 reranker templates to the request body. See instructions in https://huggingface.co/Qwen/Qwen3-Reranker-0.6B. (empty) Model
GPUSTACK_SKIP_MODEL_EVALUATION Skips the model evaluation or validation step during deployment. (empty) Model
GPUSTACK_SKIP_GPU_COUNT_CHECK Skips the validation that the manually selected GPU count must equal the parallelism world size (tp×pp×dp). Only takes effect when GPUs are manually selected; auto-scheduling is never affected. Use for manual single-node deployments where the parallelism parameters intentionally differ from the selected GPU count. (empty) Model
GPUSTACK_DISABLE_METRICS Disables metric expose and collection for the model. (empty) Model
GPUSTACK_MODEL_HEALTH_CHECK_PATH Specifies the HTTP health check path exposed by the model. (empty) Model
GPUSTACK_MODEL_RUNTIME_UID Control the user permissions of processes running inside the container. (empty) Model
GPUSTACK_MODEL_RUNTIME_GID Control the group permissions of processes running inside the container. (empty) Model
GPUSTACK_MODEL_RUNTIME_SHM_SIZE_GIB Shared memory size for the container in GiB. 10.0 Model
GPUSTACK_MODEL_INFERENCE_HEALTH_CHECK_ENABLED Enable inference health check for this model. When enabled, GPUStack periodically sends minimal inference requests to verify the model is responding correctly. false Model
GPUSTACK_MODEL_INFERENCE_HEALTH_CHECK_INTERVAL Inference health check interval in seconds (minimum 60). If recent successful inference traffic is observed within this interval, the active check is skipped. 300 Model
GPUSTACK_MODEL_INFERENCE_HEALTH_CHECK_TIMEOUT Timeout in seconds for each periodic inference health check request. Defaults to 25 for SGLang PD engine roles and 15 otherwise; an explicit value takes precedence. 15/25 Model
GPUSTACK_MODEL_INFERENCE_HEALTH_CHECK_FAILURE_THRESHOLD Number of consecutive inference health check failures before marking the instance as unhealthy. 3 Model
GPUSTACK_ENABLE_CUDA_MINOR_VERSION_COMPATIBILITY Allow lower-minor CUDA devices to run higher-minor images. Set globally on the worker or per model; per-model takes precedence. false Worker & Model

Usage Example

When deploying a model, navigate to Advanced Options > Environment Variables and add:

1
2
3
4
5
# Install additional packages before model serving starts
PYPI_PACKAGES_INSTALL=torch-audio==2.0.0 transformers==4.30.0

# Disable the serving command script entirely
GPUSTACK_MODEL_SERVING_COMMAND_SCRIPT_DISABLED=1

The serving command script automatically handles:

  • Installing additional PyPI packages specified in PYPI_PACKAGES_INSTALL
  • Supporting both uv pip and pip for package installation
  • Handling custom PyPI indices via PIP_INDEX_URL and PIP_EXTRA_INDEX_URL

GPUStack Runtime Environment Variables

These environment variables are used by GPUStack runtime. Commonly used to adjust the behavior of inference backends running in Docker/Kubernetes.

They are only usable within workers. Please set the environment variables in the workers’ containers to ensure they take effect properly.

Global Variables

Variable Description Default
GPUSTACK_RUNTIME_LOG_LEVEL Log level. INFO
GPUSTACK_RUNTIME_LOG_WARNING Enable logging warnings. 0
GPUSTACK_RUNTIME_LOG_EXCEPTION Enable logging exceptions. 0

Detector Variables

Variable Description Default
GPUSTACK_RUNTIME_DETECT Detector to use. Options: Auto, AMD, ASCEND, CAMBRICON, HYGON, ILUVATAR, METAX, MTHREADS, NVIDIA. Auto
GPUSTACK_RUNTIME_DETECT_NO_PCI_CHECK Enable no PCI check during detection. Useful for WSL environments. (empty)
GPUSTACK_RUNTIME_DETECT_PCI_CLASS_PREFIXES The PCI class prefixes to check during detection. When GPUSTACK_RUNTIME_DETECT_NO_PCI_CHECK is false, the detector will check if the PCI class of the device starts with any of the given prefixes. 02,03,0b,12
GPUSTACK_RUNTIME_DETECT_BACKEND_MAP_RESOURCE_KEY The detected backend mapping to resource keys, e.g {"cuda": "nvidia.com/devices", "rocm": "amd.com/devices"}. Used to map the gpustack-runner's backend name to the corresponding resource key. The default values named by each vendor
GPUSTACK_RUNTIME_DETECT_PHYSICAL_INDEX_PRIORITY Use physical index priority at detecting devices. 1

ROCm Detector Specific Variables

Note

Also applicable to ROCm-based backends, like Hygon.

Variable Description Default
ROCM_SMI_LIB_PATH ROCm SMI library path. (empty)
ROCM_HOME ROCm home directory. (empty)
ROCM_PATH ROCm path. /opt/rocm
ROCM_CORE_LIB_PATH ROCm core library path. (empty)

Deployer Variables

Variable Description Default
GPUSTACK_RUNTIME_DEPLOY Deployer to use. Options: Auto, Docker, Kubernetes, Podman(Experimental). Auto
GPUSTACK_RUNTIME_DEPLOY_API_CALL_ERROR_DETAIL Enable detailing the API call error during deployment. 1
GPUSTACK_RUNTIME_DEPLOY_PRINT_CONVERSION Enable printing the conversion during deployment. GPUStack Runtime provides a unified Workload definition API, which will be converted to the specific Container Runtime API calls(e.g., Docker SDK, Kubernetes API, Podman SDK). Enabling this option will print the final converted API calls in INFO log for debugging purposes. 0
GPUSTACK_RUNTIME_DEPLOY_ASYNC Enable asynchronous deployment. 1
GPUSTACK_RUNTIME_DEPLOY_ASYNC_THREADS The number of threads in the threadpool. (empty)
GPUSTACK_RUNTIME_DEPLOY_MIRRORED_NAME The name of the deployer. (empty)
GPUSTACK_RUNTIME_DEPLOY_MIRRORED_DEPLOYMENT_IGNORE_ENVIRONMENTS Environment variable names to ignore during mirrored deployment. (empty)
GPUSTACK_RUNTIME_DEPLOY_MIRRORED_DEPLOYMENT_IGNORE_VOLUMES Volume mount destinations to ignore during mirrored deployment. (empty)
GPUSTACK_RUNTIME_DEPLOY_DEFAULT_CONTAINER_REGISTRY_USERNAME Username for the default container registry. If GPUStack runner images are hosted in a container registry that requires authenticated access, set this variable to the access username at the startup of every GPUStack worker (or the all-in-one GPUStack server). In practice, it is recommended to configure the access username in the actual container runtime instead. (empty)
GPUSTACK_RUNTIME_DEPLOY_DEFAULT_CONTAINER_REGISTRY_PASSWORD Password for the default container registry. If GPUStack runner images are hosted in a container registry that requires authenticated access, set this variable to the access password at the startup of every GPUStack worker (or the all-in-one GPUStack server). In practice, it is recommended to configure the access password in the actual container runtime instead. (empty)
GPUSTACK_RUNTIME_DEPLOY_DEFAULT_CONTAINER_NAMESPACE Namespace for default runner container images. If GPUStack runner images are hosted outside the gpustack namespace, e.g. gpustack-ai, set this variable to gpustack-ai. For a multi-level namespace, e.g. awesome.com/amazing-group/amazing-subgroup/gpustack-ai, declare the registry with the parent namespace via --system-default-container-registry (or the worker system_default_container_registry config key, or the GPUSTACK_SYSTEM_DEFAULT_CONTAINER_REGISTRY environment variable), e.g. --system-default-container-registry=awesome.com/amazing-group/amazing-subgroup. gpustack
GPUSTACK_RUNTIME_DEPLOY_IMAGE_PULL_POLICY Image pull policy for the deployer (e.g., Always, IfNotPresent, Never). IfNotPresent
GPUSTACK_RUNTIME_DEPLOY_CDI_SPECS_DIRECTORY During deployment, path of directory containing Container Device Interface (CDI) specifications, or the directory to generate CDI specifications into. /var/run/cdi
GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_CDI Manual mapping of container device interfaces, which is used to tell the Container Runtime which devices to inject into the container, e.g., nvidia.com/devices=nvidia.com/gpu;amd.com/devices=amd.com/gpu. The key is the resource key, and the value is the Container Device Interface(CDI) key. The default values named by each vendor
GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_RUNTIME_VISIBLE_DEVICES Manual mapping of runtime visible devices environment variables, which is used to tell the Container Runtime which devices to inject into the container, e.g., nvidia.com/devices=NVIDIA_VISIBLE_DEVICES;amd.com/devices=AMD_VISIBLE_DEVICES. The key is the resource key, and the value is the environment variable name. The default values named by each vendor
GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_BACKEND_VISIBLE_DEVICES Manual mapping of backend visible devices environment variables, which is used to tell the Device Runtime (e.g., ROCm, CUDA, OneAPI) which devices to use inside the container, e.g., nvidia.com/devices=CUDA_VISIBLE_DEVICES;amd.com/devices=ROCR_VISIBLE_DEVICES. The key is the resource key, and the value is a list of environment variable names. The default values named by each vendor
GPUSTACK_RUNTIME_DEPLOY_RUNTIME_VISIBLE_DEVICES_VALUE_UUID Use UUIDs for the given runtime visible devices environment variables. (empty)
GPUSTACK_RUNTIME_DEPLOY_BACKEND_VISIBLE_DEVICES_VALUE_ALIGNMENT Enable value alignment for the given backend visible devices environment variables. ASCEND_RT_VISIBLE_DEVICES,NPU_VISIBLE_DEVICES
GPUSTACK_RUNTIME_DEPLOY_CPU_AFFINITY Enable CPU affinity for deployed workloads. 0
GPUSTACK_RUNTIME_DEPLOY_NUMA_AFFINITY Enable NUMA affinity for deployed workloads. When enabled, GPUSTACK_RUNTIME_DEPLOY_CPU_AFFINITY is also implied. 0

Docker Deployer Specific Variables

Variable Description Default
GPUSTACK_RUNTIME_DOCKER_HOST Host for Docker connection. Used to override the default Docker host. http+unix:///var/run/docker.sock
GPUSTACK_RUNTIME_DOCKER_PAUSE_IMAGE Container image used for the pause container in Docker. gpustack/runtime:pause
GPUSTACK_RUNTIME_DOCKER_UNHEALTHY_RESTART_IMAGE Container image used for unhealthy restart container in Docker. gpustack/runtime:health
GPUSTACK_RUNTIME_DOCKER_RESOURCE_INJECTION_POLICY Resource injection policy for the Docker deployer (e.g., Env, CDI).
Env: Injects resources using standard environment variable, based on GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_RUNTIME_VISIBLE_DEVICES.
CDI: Injects resources using CDI, based on GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_CDI.		 Env
GPUSTACK_RUNTIME_DOCKER_CDI_SPECS_GENERATE Generate CDI specifications during deployment when using CDI resource injection policy, requires GPUSTACK_RUNTIME_DEPLOY_CDI_SPECS_DIRECTORY to exist. Works only when GPUSTACK_RUNTIME_DOCKER_RESOURCE_INJECTION_POLICY is set to CDI. Using internal knowledge to generate the CDI specifications for deployer, if the output file conflicts with other tools generating CDI specifications(e.g., NVIDIA Container Toolkit), please disable this and remove the output file manually. 1

Kubernetes Deployer Specific Variables

Variable Description Default
GPUSTACK_RUNTIME_KUBERNETES_NODE_NAME Name of the Kubernetes Node to deploy workloads to. (empty)
GPUSTACK_RUNTIME_KUBERNETES_NAMESPACE Namespace of the Kubernetes to deploy workloads to. default
GPUSTACK_RUNTIME_KUBERNETES_DOMAIN_SUFFIX Domain suffix for Kubernetes services. cluster.local
GPUSTACK_RUNTIME_KUBERNETES_SERVICE_TYPE Service type for Kubernetes services. Options: ClusterIP, NodePort, LoadBalancer. ClusterIP
GPUSTACK_RUNTIME_KUBERNETES_QUORUM_READ Whether to use quorum read for Kubernetes services. 0
GPUSTACK_RUNTIME_KUBERNETES_DELETE_PROPAGATION_POLICY Deletion propagation policy for Kubernetes resources. Options: Foreground, Background, Orphan. Foreground
GPUSTACK_RUNTIME_KUBERNETES_RESOURCE_INJECTION_POLICY Resource injection policy for the Kubernetes deployer. Options: Auto, Env, KDP.
Auto: Automatically choose the resource injection policy based on the environment.
Env: Injects resources using standard environment variable, depends on underlying Container Toolkit, based on GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_RUNTIME_VISIBLE_DEVICES.
KDP: Injects resources using Kubernetes Device Plugin.		 Auto
GPUSTACK_RUNTIME_KUBERNETES_KDP_PER_DEVICE_MAX_ALLOCATIONS Maximum allocations for one device in Kubernetes Device Plugin. 10
GPUSTACK_RUNTIME_KUBERNETES_KDP_DEVICE_ALLOCATION_POLICY Device allocation policy for the Kubernetes Device Plugin. Options: Auto, CDI, Env, Opaque.
Auto: Automatically choose the device allocation policy based on the environment.
Env: Allocates devices using runtime-visible environment variables, based on GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_RUNTIME_VISIBLE_DEVICES; requires Container Toolkit support.
CDI: Allocates devices using generated CDI specifications, based on GPUSTACK_RUNTIME_DEPLOY_RESOURCE_KEY_MAP_CDI, making it easy to debug and troubleshoot; requires GPUSTACK_RUNTIME_DEPLOY_CDI_SPECS_DIRECTORY to exist.
Opaque: Uses internal logic for allocation, which is convenient for deployment but difficult to troubleshoot.		 Auto
GPUSTACK_RUNTIME_KUBERNETES_KDP_CDI_SPECS_GENERATE Generate CDI specifications during deployment, requires GPUSTACK_RUNTIME_DEPLOY_CDI_SPECS_DIRECTORY to exist. Works only when GPUSTACK_RUNTIME_KUBERNETES_KDP_DEVICE_ALLOCATION_POLICY is set to CDI. Using internal knowledge to generate the CDI specifications for deployer, if the output file conflicts with other tools generating CDI specifications(e.g., NVIDIA Container Toolkit), please disable this and remove the output file manually. 1

Podman Deployer Specific Variables

Note

Podman deployer is experimental and needs version 4.9 or higher.

Variable Description Default
GPUSTACK_RUNTIME_PODMAN_HOST Host for Podman connection. Used to override the default Podman host. http+unix:///run/podman/podman.sock
GPUSTACK_RUNTIME_PODMAN_PAUSE_IMAGE Container image used for the pause container in Podman. Default is same as GPUSTACK_RUNTIME_DOCKER_PAUSE_IMAGE. gpustack/runtime:pause
GPUSTACK_RUNTIME_PODMAN_UNHEALTHY_RESTART_IMAGE Container image used for unhealthy restart container in Podman. Default is same as GPUSTACK_RUNTIME_DOCKER_UNHEALTHY_RESTART_IMAGE. gpustack/runtime:health
GPUSTACK_RUNTIME_PODMAN_CDI_SPECS_GENERATE Generate CDI specifications during deployment, requires GPUSTACK_RUNTIME_DEPLOY_CDI_SPECS_DIRECTORY to exist. Using internal knowledge to generate the CDI specifications for deployer, if the output file conflicts with other tools generating CDI specifications(e.g., NVIDIA Container Toolkit), please disable this and remove the output file manually. Default is same as GPUSTACK_RUNTIME_DOCKER_CDI_SPECS_GENERATE. 1