Troubleshooting

View GPUStack Logs

You can view GPUStack logs with the following commands for the default setup:

1

docker logs -f gpustack

Enable Debug Mode

You can enable the DEBUG mode by setting the --debug flag when running GPUStack:

1
2
3
4
5

sudo docker run -d --name gpustack \
    ...
    gpustack/gpustack \
+    --debug
    ...

You can also enable GPUStack's debug mode at runtime by running the following command inside the server container:

1

gpustack reload-config --set debug=true

--debug enables verbose logging without starting Python memory allocation tracing. To enable memory profiling, set GPUSTACK_DEBUG_ENABLE_TRACEMALLOC=true before starting GPUStack. This setting defaults to false, works independently of --debug, and requires a restart. Enable it when investigating memory usage, since tracing adds CPU and memory overhead.

Configure Log Level

The log level endpoints require authentication.

You can configure log level of the GPUStack server at runtime by running the following command inside the server container. Authenticate with an admin API key:

1
2
3

curl -X PUT http://localhost/debug/log_level \
    -H "Authorization: Bearer <YOUR_API_KEY>" \
    -d "debug"

The same applies to GPUStack workers. Authenticate with the local worker token (defaults to /var/lib/gpustack/worker_token):

1
2
3

curl -X PUT http://localhost:10150/debug/log_level \
    -H "Authorization: Bearer $(cat /var/lib/gpustack/worker_token)" \
    -d "debug"

The available log levels are: trace, debug, info, warning, error, critical.

Reset Admin Password

In case you forgot the admin password, you can reset it by running the following command inside the server container:

1

gpustack reset-admin-password

If you changed the default port using --port when starting GPUStack, specify the GPUStack URL using the --server-url parameter. It must be run locally on the server and accessed via localhost:

1

gpustack reset-admin-password --server-url http://localhost:9090

Assist in Accelerators Detection Diagnosis

After successfully deploying the GPUStack Worker as described in the installation guide,
if the Worker fails to detect any devices,
please enter the corresponding Worker container, run the following command, and report the results to GPUStack.

1

time GPUSTACK_RUNTIME_LOG_LEVEL=debug GPUSTACK_RUNTIME_LOG_EXCEPTION=1 gpustack-runtime detect --format json

Assist in Model Deployment Diagnosis

If you experience issues after deploying a model, please enter the corresponding Worker container, run the following command, and report the results to GPUStack.

1

gpustack-runtime inspect <model instance name>